Skip to main content
Elsevier Sponsored Documents logoLink to Elsevier Sponsored Documents
. 2021 Jun;134:138–149. doi: 10.1016/j.jclinepi.2021.02.009

Recommendation mapping of the World Health Organization's guidelines on tuberculosis: A new approach to digitizing and presenting recommendations

Anisa Hajizadeh a, Tamara Lotfi a,b, Dennis Falzon c, Dominik Mertz a,b,d, Robby Nieuwlaat a,b, Nebiat Gebreselassie c, Ernesto Jaramillo c, Alexei Korobitsyn c, Matteo Zignol c, Fuad Mirzayev c, Nazir Ismail c, Jan Brozek a,b,d, Mark Loeb a,b,d, Thomas Piggott a, Andrea Darzi a, Qi Wang a, Al Subhi Mahmood a, Praveen Saroey a, Micayla Matthews a, Finn Schünemann e, Bart Dietl f, Artur Nowak f, Kuba Kulesza f, Giovanna EU Muti-Schünemann a, Antonio Bognanni a, Rana Charide g, Elie A Akl g, Tereza Kasaeva c, Holger J Schünemann a,b,d,
PMCID: PMC8168829  PMID: 33762142

Abstract

Objective

Having up-to-date health policy recommendations accessible in one location is in high demand by guideline users. We developed an easy to navigate interactive approach to organize recommendations and applied it to tuberculosis (TB) guidelines of the World Health Organization (WHO).

Study Design

We used a mixed-methods study design to develop a framework for recommendation mapping with seven key methodological considerations. We define a recommendation map as an online repository of recommendations from several guidelines on a condition, providing links to the underlying evidence and expert judgments that inform them, allowing users to filter and cross-tabulate the search results. We engaged guideline developers, users, and health software engineers in an iterative process to elaborate the WHO eTB recommendation map.

Results

Applying the seven-step framework, we included 228 recommendations, linked to 103 guideline questions and organized the recommendation map according to key components of the health question, including the original recommendations and rationale (https://who.tuberculosis.recmap.org/).

Conclusion

The recommendation mapping framework provides the entire continuum of evidence mapping by framing recommendations within a guideline questions’ population, interventions, and comparators domains. Recommendation maps should allow guideline developers to organize their work meaningfully, standardize the automated publication of guidelines through links to the GRADEpro guideline development tool, and increase their accessibility and usability.

Keywords: Evidence-based practice, Guideline, Tuberculosis, GRADE

Abbreviations: RecMaps, Recommendation mapping; PICO, Population, Intervention, Comparator, Outcome; TB, Tuberculosis; DR-TB, Drug-Resistant Tuberculosis; MDR-TB, Multidrug-Resistant Tuberculosis; XDR-TB, Extensively Drug-Resistant Tuberculosis; AMR, Antimicrobial Resistance; WHO, World Health Organization; WHO/GTB, Global Tuberculosis Programme, World Health Organization, Switzerland; HEI, Department of Health Research Methods, Evidence and Impact, McMaster University, Canada; GRADE, Grading of Recommendations Assessment, Development and Evaluation; EtD, Evidence to Decision; GDG, Guideline Development Group; GRC, Guideline Review Committee; NTP, National TB Programme

Highlights

  • What this adds to what is known: This work transfers evidence mapping methods to guidelines in a process we call “recommendation mapping.”

  • What is the implication, what should change now: Recommendation mapping provides a digital curation of guidelines for a particular condition or disease. In this article we describe how we created an online map of WHO guidance on tuberculosis prevention and care to enhance accessibility of recommendation data and other key guideline components, facilitate their prompt update as needed, allow cross-tabulation of recommendations to help visualize any priority gaps and clustering and facilitate the adaptation of recommendations by users.

  • Recommendation maps are accessory to the notion of “living guidelines” that are promptly updated to reflect the state of the science, providing a digital solution to improve upon the organization, accessibility, and ultimately, uptake of guideline recommendations to benefit individual wellbeing or public health. It is directly linked to the GRADEpro guideline development tool.

Graphical abstract

Image, graphical abstract

1. Introduction

1.1. Background

Evidence mapping is an organizational method used to chart the terrain of research for a given subject of importance to health policy [1], [2], [3]. Within the health sector, there are varied definitions of what constitutes evidence mapping, and the role they play in the review of evidence. A systematic review of 39 health evidence maps, determined some common characteristics: they usually employ systematic methods in the search for evidence, identify gaps in knowledge and future research needs, and are presented in a user-friendly interface with searchable functionality [4].

Until now, evidence mapping has not been applied to guidelines and the individual recommendations from which they are composed. Guideline development typically includes prioritization of questions as an early step to guide the search for evidence to be addressed by recommendations [5]. Recommendations strive to provide answers based on the available evidence considered by guideline developers at a given time. These are most needed when there is uncertainty about what to do when faced with a range of potential policies and interventions for which the evidence is not clear, when a new public health problem emerges or when new evidence comes to light. Clinical, public health and health systems recommendations provide brief, actionable statements. However, as a result of the many players in the guideline development ecosystem, guidelines often address topics in a fragmented approach by focusing on specific aspects of a health topic. Furthermore, even within guideline developing organizations, coherence and systematic coverage of guideline topics may be limited or overlap.

Since 2006, the World Health Organization (WHO) has produced tuberculosis (TB) guidelines using the GRADE method (Grading of Recommendations Assessment, Development and Evaluation) and in recent years the World Health Organization (WHO) has made a major drive to consolidate its tuberculosis (TB) guidelines, inclusive of efforts to simplify online access [6], [7], [8]. Evidence-based guidance across the continuum of care is instrumental to the global drive to end TB - the leading infectious disease cause of death in the world today despite being largely preventable and curable [9]. This is enshrined in the WHO End TB Strategy for 2016–2035 endorsed by all Member States. Efforts to keep consolidated guidelines updated as “living guidelines” are important for WHO so that practice is based on the best available, latest evidence[10].

This paper introduces a framework for recommendation mapping, which include digital capture of and online access to all recommendations, together with their underlying evidence base [11]. We define a recommendation map as a tool to organize recommendations on a specific health topic in a digital, visual schematic, reporting features around the evidence and judgments that inform the recommendation, and highlight clusters and gaps in availability of recommendations in a digital, visual schematic. The recommendation map is organized by the components of the PICO (Population, Intervention, Comparator, Outcome) question that guided the development of the respective recommendation and can be directly linked to GRADEpro, GRADE's original tool to develop recommendations. We illustrate the utility of the recommendation mapping concept to facilitate the consolidation of recommendations into a single platform, improving access to recommendations, and supporting the identification of research gaps of global public health policies, making reference to recent creation of an online repository of WHO TB recommendations.

2. Methods

2.1. General Methods

We used a mixed-methods approach to develop and apply the approach to recommendation mapping through the iterative development of a conceptual framework. The process also involved semi-structured interviews with key stakeholders, as well as a survey (see Appendix a). When applying the recommendation mapping approach to the online repository we considered that the cross-tabulation of a recommendation by any two of the Population, Intervention and Comparator domains from the PICO framework would point towards both the clustering - as a surrogate of fragmentation of recommendations – as well as gaps – pointing to potential unmet needs in recommendations or evidence. This could help guide future efforts to consolidate guidelines and to inform research prioritization.

We applied our approach to WHO TB guidelines as a case study and refined the product through iterative consulting process through the different stages of elaboration. Guideline and GRADE methodologists, along with staff from the WHO Global TB Programme, led and coordinated the project. Health software developers and interaction designers (both from EvidencePrime, Inc.) contributed to the conceptualization of the framework and developed the digital platforms. A team of health professionals supported the extraction of information from eligible WHO guidelines focusing on TB and the coding of data elements using standardized methods (see below).

2.2. Needs Assessment

This work was informed by a qualitative assessment of perceptions, usability and potential acceptability of the recommendation mapping among staff from the Global TB Programme of WHO, National TB Programmes (NTP), technical experts and members of WHO Guideline Development Groups (GDG) and WHO on the novel approach. This was mainly conducted at the time of two WHO TB GDG meetings in November and December of 2019 [12], [13]. We identified individuals who may be potential champions and early adopters of the proposed outputs of the project, have unique insight into the nature of the problem, and have a role within the institutions that will be impacted by this work. We used nonprobability sampling to select participants. Participation was limited to English speaking individuals. We conducted semi-structured interviews using an interview guide calibrated with three external individuals who fit the inclusion criteria prior to use (Appendix A shows the Interview Guide). We also conducted an online survey including 30 questions via SurveyMonkey, Inc [14]. NVivo Qualitative Data Analysis Software was used to code transcribed interviews using inductive coding [15] Interviews data was analysed using the principles of Grounded theory [16]. The qualitative assessment received ethics approval by the Hamilton Integrated Research Ethics Board (HiREB).

3. Case Study

3.1. Eligibility Criteria

All WHO guidelines containing recommendations for TB, published after 2006, were eligible for inclusion in the mapping exercise. We excluded guidelines that did not use the GRADE approach in their development (i.e., recommendations that lacked an assessment of quality or certainty in the evidence or diagnostic accuracy, evidence profiles, or Evidence to Decision (EtD) tables) [17], [18], [19]. Eligibility for the inclusion of a recommendation depended upon the mention of tuberculosis in its text and included TB guidelines and others that are relevant to TB control for example, antiretroviral treatment in people living with HIV. However, if the latter made no reference to tuberculosis we did not include them.

3.2. Information Sources

We identified and collected recommendations pertaining to TB prevention and care from WHO compendium of TB guidelines and associated standards, WHO guidelines on HIV and nutrition and other relevant guidelines indicated by WHO staff [6], [7], [8]. The team from the WHO Global TB Programme verified the final collection of guidelines that was deemed eligible for the recommendation map and would help with consolidating guidelines.

3.3. Data collection and standardization

We extracted the verbatim text of both guideline questions and recommendations in duplicate according to data items of interest, informed by PICO ontology (Appendix B) [20], [21]. As outcomes are not typically included in the recommendation statement, we used the recommendations PIC structure. This is because each recommendation may have many outcomes of interest, and the intervention(s) presented in the recommendation may not directly result in the outcomes suggested in the PICO. For these reasons, our work draws on the learning from evidence mapping and computable guidelines. We built on the methods proposed for “computable guidelines” or computer interpretable guidelines to manage the heterogenous free-text content of guidelines [22], [23], [24], [25], [26], [27]. We considered Shiffman et al.’s Guideline Elements Model (GEM), that suggests a computable grouping (in XML) of the heterogeneous material in guidelines [23]. GEM's system of markup involves the demarcation and labeling of text, to characterize the semantic content of the guideline [22]. Our framework confirmed the need for all three of the high-level knowledge components identified in the hierarchy of elements used in GEM: recommendations, definitions, and algorithms.

We coded extracted PIC elements using SNOMED-CT, ICD-11 and the ATC/DDD Index [28], [29], [30]. We favored SNOMED-CT labeling for both the population and the intervention codes for two reasons: [28] (1) the breadth of codes seem to cover the particularities of PICO elements in recommendations more completely; and (2) unlike ICD-11, SNOMED-CT is a polyhierarchical ontology, with the option of incorporating rule-based value sets to leverage the connections between the terminology [31]. We also extracted definitions, remarks, Summary of Findings (SoF) tables where outcomes are considered, algorithms, and links to their EtD tables whenever available.

4. Results

4.1. Needs Assessment

In total, 21 individuals involved in WHO TB guideline development took part in the qualitative assessment on acceptability of recommendation mapping (Appendix C shows participant characteristics). Coded conversations resulted in three themes that were relevant to the recommendation mapping exercise: (1) there are pressing accessibility issues with the way recommendations are currently organized and presented in many separate guidelines, (2) consolidated guidelines are useful to improve accessibility: they do not facilitate the development of guidance, and (3) the current process to set priorities, that is, deciding on the scope or topic and questions, in guideline development can be made more transparent. Considering the themes individually they pointed to the acceptability of the recommendation mapping framework, as a potential tool to ease accessibility, fulfill the need for consolidation, and aid in setting priorities objectively and identify evidence gaps.

4.2. Framework for recommendation mapping

Fig. 1 summarizes the rationale and functioning of a recommendation map. The attributes in boxes two to six represent the critical steps undertaken to create the electronic WHO TB recommendation map (eTB RecMap). We will describe the general methodology for each attribute and provide examples from the WHO eTB RecMap application in the case study.

Fig. 1.

Fig 1

The Recommendation Map framework.

4.3. Clarifying the Scope of the Recommendation Mapping

We used a collaborative approach to define the scope of a recommendation map by consulting with key stakeholders in selecting the condition of interest is a defining feature of a recommendation map. In our case this consisted of all the WHO recommendations on TB prevention, screening, diagnosis, treatment and care that were valid in April 2020. These could include recommendations developed by GDGs convened by the WHO Global TB Programme or other WHO departments. The eligibility criteria for a recommendation map are in line with evidence-mapping methods which assume systematic methodologies akin to those used for systematic reviews [2], [3], [4], [32], [33]. In terms of the scope of guideline elements, we identified three components of evidence mapping methods that are relevant for recommendation mapping: guideline questions, the evidence and other elements considered when addressing them, and the final recommendations with accompanying implementation remarks, strength and judgment on quality of the evidence. Guidelines begin with the a-priori framing of questions following a PICO format, to guide the search for evidence and focus the formulation of recommendations [34]

For the WHO TB case study, the scope of the map focused primarily on the population and intervention codes that we identified from the guideline recommendations. A total of 18 WHO guidelines composed of 103 extracted and coded questions and containing 228 coded recommendations were included (Fig. 2) [35], [36], [37], [38], [39], [40], [41], [42], [43], [44], [45], [46], [47], [48], [49], [50], [51], [52].

Fig. 2.

Fig 2

Flow diagram for the identification of WHO global tuberculosis guidelines recommendation mapping.

We found that the population and intervention codes contained in guideline questions did not always capture the recommendation codes completely. At times, additional sub-populations and interventions outside of those envisaged by the original question appeared in recommendations, likely as a result of subgroups and intervention options that were present in the evidence search and that the GDG considered relevant to the final recommendation. If the population or intervention was not described a recommendation could not be included in the two-dimensional map. Thus, in order to ensure that the scope of the map accounted for the entire breadth of recommendations, we added additional population codes to the axes when these were missing. Consequently, the parameters of the map are a joint product of elements of the guideline question and the recommendation codes, after the removal of duplicate codes.

4.4. Cataloguing Questions

The framework for recommendation mapping diverges from existing frameworks for evidence mapping, because guideline elements and the related evidence require a more nuanced approach to those typically taken [3], [4]. Recommendation maps portray multiple components of the evidence and their synthesis including primary studies and the systematic reviews that relate to a recommendation that merge into a digital health tool.

Imposing standardized element structures and codes on the free text narrative in the guidelines posed challenges. During extraction and coding, we found stylistic differences between guidelines and over the years in the sectioning of the documents, and the phrasing of questions and recommendations. We found both missing and multiple PICO elements across recommendations. When we initially piloted our approach, we observed that verbatim recommendations were missing a population or had no comparator present in the statement.

4.5. Linking the Evidence

A link between the recommendation and the evidence is a requirement for transparency and for trustworthy recommendations. Recommendation maps, thus, should provide a direct link to the evidence that informs each recommendation answering the guideline questions.

For the WHO TB case study, we connected every recommendation to the SOF table(s) and the EtD framework that was used for its development, by linking each recommendation to a parallel extraction of the EtD framework and its PICO question (Fig. 3).

Fig. 3.

Fig 3

Sample detail page for a recommendation from the online Recommendation Map (https://who.tuberculosis.recmap.org/).

4.6. Describing the Certainty in the Evidence

Recommendations typically include a description of the quality or certainty in the evidence or diagnostic accuracy, alongside the strength of the recommendation. Our recommendation mapping visualization adopted the wording suggested by the GRADE working group [53], [54].

For the WHO TB case study, we graphically depicted the recommendation strength and the certainty in the underlying evidence as standard labels attached to the repository of recommendations (Figs. 3 and 4). This graphical representation includes color coding to indicate if the recommendation was for, neutral, or against the intervention [55].

Fig. 4.

Fig 4

The list and map view of the WHO recommendations on tuberculosis (https://who.tuberculosis.recmap.org/).

5. Visualization

Fig. 4 shows the two principal views of the recommendation map as developed for the WHO eTB RecMap. The tool is dynamic, with features to assist with the accessibility of content, such as hover-features with definitions of the strength and certainty ratings, and a quick-tutorial on how to use the platform and how to read the recommendation map.

Legend Fig. 4. On the upper panel, the List view shows the recommendations organized within pages accessible via thumbnails that correspond to the modules of the consolidated WHO TB guidelines as developed by the WHO (https://who.tuberculosis.recmap.org/list). Preset and free text search functions allow the content of each page to be further filtered (e.g., by age-group, year of publication, name of a medicine, site of disease). The recommendation text is accompanied by labeling on its strength and the certainty in the evidence underpinning it. By clicking on individual recommendations one can access a page with links and references to the associated evidence, remarks, definitions, justifications, original guideline source, and visual displays of pertinent information (e.g., SoF tables, and algorithms). The default order of the recommendations primarily follows the sequence in which they appear in the source guideline, that is itself usually determined by the pathway of care (e.g., for TB preventive treatment this would be risk group identification, then testing and then treatment options).

In the lower panel, a Map view presents a cross-tabulation of the recommendations grouped by key elements such as site of disease, comorbidity or age-bracket down the rows against the modules of the consolidated guidelines as columns (https://who.tuberculosis.recmap.org/grid). The matrix can be defined by the user by selecting search terms and filters similar to those in the List view. The cells of the table display a number and a shade that both reflect the density of unitary recommendations that had analogous codes for the respective specifications in both axes. A search thus results in a “heat map” The same recommendation could feature and be enumerated more than once in the columns across a row (i.e., a recommendation could be relevant to more than one module). A highlight function identifies recommendations that feature in more than one coordinate across the grid. Clicking on the cells opens a listing of the recommendations in that particular space. Cells without a number or shading may be due to implausible combinations (e.g., malnutrition and infection control), or could reflect absence or paucity of evidence precluding a recommendation, even for an important question. A clustering of recommendations does not necessarily reflect the strength of the evidence in a particular area but may be more associated with a less parsimonious styles of generating recommendations for a given issue.

5.1. Maintenance and Sustainability

A recommendation map, like any active repository of information, is only useful to the degree that content maintenance and update is automated, and its upkeep is systematized within the organization. Stakeholder engagement throughout the whole process of recommendation mapping will be a defining feature of both recommendation and evidence gap maps, and is central to the integration of these tools within the routine work of the institution, in order to maintain its relevance and utility.

For the WHO TB case study, the online repository and display features were created in a platform related to GRADEpro, being the software that the Global TB Programme uses for guideline development. While currently the links to the sources of evidence send to the online .pdf the system to the next iteration will allow a linkage to the GRADEpro original versions of the SOF and EtD tables. This will also allow prospective recommendations to automatically find their place in the map once they are published or updated, satisfying a key requirement for “living guidelines” that try to shadow closely the new evidence. Maintenance does not only involve the addition of new recommendations, but also requires careful removal of non-relevant, retired, and outdated content from the main repository. The GRADEpro guideline development tool interface, that can produce output recommendation map, allows the organization access to manipulate the presentation of recommendations and hide outdated content.

5.2. Use and usability of the Recommendation Map

Recommendation maps digitally organize recommendations into a centrally accessible hub of relevant guidance, providing an important resource for organizations that produce evidence-informed guidance, undertake evidence reviews or users of guidelines to make better decisions. This dovetails into the process of adoption of existing recommendations, adaptation of recommendations to meet the needs of their context, or the development of de novo recommendations based on shared or newly synthesized evidence, a process called adolopment [56]. Clustering of similar recommendations can be flagged to seek opportunities to consolidate and to rationalize fragmented statements. Empty cells should be screened for any potential gaps in evidence or in recommendations, that is of use for both researchers and guideline developers. There are many ways a recommendation map, like an evidence gap map, can be explored. The most straightforward is to search for relevant recommendations by entering a population or intervention and using the relevant filtering options. Alternatively the recommendations that fall within a population or module can be visually scanned for points of interest. The process towards finalization of the recommendation mapping resource could also benefit from user feedback to assess acceptability, adoption, appropriateness, penetration, sustainability and fidelity to user demand [57].

For the TB case study, WHO has made it a priority to consolidate guidelines across different topics to avoid fragmentation and overlap. During the creation of the repository of WHO TB recommendations we identified some examples of overlapping recommendations. In addition, WHO TB guidelines have a preeminent role not only to for the guideline developers but also to promote evidence-based decision making at different levels of the health systems in Member-States. Electronic access to a centralized hub of WHO TB recommendations is accessory to guideline adolopment and can support Member-States to contextualize the recommendations and their implementation on local level [56].

There are three main areas of interaction that users will engage with in a RecMap; the grid-view, the searchable database, and the recommendation landing pages. Firstly, the searchable database displays a list of recommendations that includes option(s) in the top panel of modules, and/or filters parallel to the list. To navigate the list by key words, a search bar is also available to the user. There is a search function to easily locate recommendations by key words, or filters to adjust the map to a desired scope. Secondly, the map is set along a grid, with two axes that correspond to two PICO components. The user should orient themselves with which component is displayed on the x and y axis, respectively. Selecting for a certain module (i.e. screening, diagnosis, care) within the Intervention axis will reveal additional intervention options. Any coordinate that is filled will display a number corresponding to the number of recommendations contained in the intersection of the PICO components of interest. Unlike other repositories of guidelines, the RecMap has the unique advantage of highlighting gaps and clusters in the breadth of recommendations published by a given organization. The Heat Map option magnifies this functionality, by color-coding filled coordinates from cool toned green to warm red, to indicate increasing concentration of recommendations in “hot spots.” Thirdly, selecting for any one recommendation on either the list or grid view of the map will take the user to the recommendation landing page. Here, the recommendation is displayed with its strength and certainty, corresponding remarks, PICO components, and links to the original publication.

Ultimately, recommendation maps digitally organize recommendations into a centrally accessible hub of relevant guidance, providing an important resource for organizations that produce evidence-informed guidance, undertake evidence reviews or users of guidelines to make better decisions. Electronic access to a centralized hub of WHO TB recommendations is accessory to guideline adoption and adaptation and can support Member-States to contextualize the recommendations and their implementation on local level.

6. Discussion

6.1. Summary of the Findings

We report the development of a seven-step framework to map and consolidate health recommendations, using lessons learnt from the empirical application of these methodologies to organize, streamline and enhance the accessibility of WHO TB recommendations. The work around the TB recommendations was informed by a qualitative assessment which preceded the process and identified users’ needs and priorities around the framework. We applied these methods to create the eTB RecMap, a digital platform housing 228 recommendations contained in 18 WHO guidelines.

6.2. Strengths and Limitations

The primary strength of this work lies in the needs-driven development of a user-friendly digital solution to compile and improve access to WHO recommendations, and the associated evidence base on TB prevention and care, a topical public health challenge of global proportions and for which practice guidance is in high demand. Additionally, we employed robust measures where possible to ensure the accuracy and quality of information retrieved. This included extraction and coding in duplicate of information, continuous collaboration on the progress of outputs with key stakeholders, and ongoing qualitative assessment on the usability of the features, in parallel to their development.

A limitation of this work is the absence of a standardized measure of the utility of this approach compared to the current practice of guideline publication. An evaluation of the end-products by users will be imperative for this work moving forward and therefore a protracted period of user feedback is expected. While our framework includes principles to address non-standardized guideline text, it does not account for all possible considerations that may be needed to bring order to a diverse body of guidelines. Our exercise was limited to evidence-based recommendations created using GRADE and did not include implementation statements and other normative advice that is oftentimes included in operational guidance (e.g., drug dosage tables and schedules to monitor adverse events in patients on novel treatments). Going forward, a streamlined approach to automate the presentation of recommendations for live publication in a recommendation map, would enhance the unification of an organization's recommendations, as it strives to keep pace with new evidence. GRADE provides not only the methodological underpinning for development, but also direction for structure of recommendations and its underpinning evidence and judgments that can help overcome this challenge [58]. In fact, the RecMap as a product will be linked to recommendations developed by a guideline development group in GRADEpro, GRADE's original tool to develop recommendations. For example, a COVID-19 RecMap has been developed by an international consortium headed by Cochrane Canada (https://covid19.recmap.org/), and it includes direct url links to the original EtDs that include the evidence, judgments and rationale for a recommendation.

6.3. Implications for policy and practice

Recommendation maps have a number of implications for policy and practice. Attention should be given to understanding what clusters and gaps represent, and what they do not. In a recommendation map, clusters are not always areas of sufficiency and gaps are not always areas of need. A cluster of recommendations may imply an area of ongoing need, despite the current density of concentrated guidance on the map. A gap may counter-intuitively be an area of low priority, not necessarily requiring a recommendation at this time. Likewise, a coordinate with only one recommendation, may address the cross-section of the population and intervention completely in that single statement. We are currently exploring additional steps in developing recommendation maps. For example, when compiling a recommendation map from multiple sources, for example, different organizations, an assessment of the credibility of the guidelines or individual recommendations is desirable. Such an assessment, e.g. using the AGREE II tool, can be visually displayed [59], [60], [61], [62]. Furthermore, cataloging recommendations from multiple organizations will require a choice as to which recommendations for a given PICO to display. Furthermore, the heterogeneity in the content of the free text of the guideline is difficult to manage, but we believe that our framework will harmonize this heterogeneity and facilitate the consolidation of different guidelines. In the context of work on the eCOVID-19 recommendation map (covid19.evidenceprime.ca) funded by the Canadian government with a large international consortium, these aspects of recommendation maps further are further developed.

6.4. Implications for Research

Future research testing evidence mapping methods in the realm of other guidelines, is needed. Further, studies assessing the utility of such outputs for guideline developers, stakeholders, and end-users, will also be warranted. Measures of utility could include a comparative assessment between accessibility of WHO TB recommendations as formerly presented versus the digital representation, and a qualitative assessment of practicality and usefulness of gaps or cluster visualization for guideline priority setting.

7. Conclusions

In this article, we report the development and outcomes of the application of the recommendation mapping concept to WHO recommendations on TB prevention and care, with a view to enhance access and use of evidence-based recommendations and to improve country progress towards the targets of the End TB Strategy. The approach enables mapping of recommendations and associated evidence in an interactive, centralized digital tool linked to GRADEpro as digital guideline development tool, that supports the consolidation of guidelines on thematic health topics. In doing so, the platform enables better visualization of the availability of recommendations (or lack thereof) across the continuum of care, charting the way forward on policy and research prioritization. The approach may be useful for guidance beyond TB. Future work focuses on evaluation and consumer views of the RecMap.

Author Contributions

Anisa Hajizadeh: Methodology, Validation, Formal Analysis, Investigation, Data Curation, Writing - Original Draft, Writing - Review & Editing, Visualization, Project administration. Tamara Lotfi: Validation, Investigation, Writing - Review & Editing, Project Administration. Dominik Mertz: Supervision, Writing - Review & Editing. Robby Nieuwlaat: Supervision, Writing - Review & Editing. Dennis Falzon: Investigation, Writing - Review & Editing, Resources. Ernesto Jaramillo: Investigation, Writing - Review & Editing, Resources. Alexei Korobitsyn: Investigation, Writing - Review & Editing, Resources. Tereza Kasaeva: Methodology, Investigation, Writing - Review & Editing, Resources. Matteo Zignol: Investigation, Writing - Review & Editing, Resources. Fuad Mirzayev: Investigation, Writing - Review & Editing, Resources. Nazir Ismail: Investigation, Writing - Review & Editing, Resources. Nebiat Gebreselassie: Investigation, Writing - Review & Editing, Resources. Jan Brozek: Conceptualization, Methodology, Investigation, Data Curation, Software, Visualization, Resources. Mark Loeb: Writing - Review & Editing, Supervision. Thomas Piggott: Methodology, Writing - Review & Editing. Andrea Darzi: Validation, Investigation, Writing - Review & Editing. Qi Wang: Validation, Investigation, Writing - Review & Editing. Al Subhi Mahmood: Validation, Investigation, Writing - Review & Editing. Praveen Saroey: Validation, Investigation, Writing - Review & Editing. Micayla Matthews: Validation, Investigation, Writing - Review & Editing. Finn Schünemann: Validation, Investigation, Writing - Review & Editing. Bart Dietl: Methodology, Investigation, Data Curation, Software, Visualization, Resources. Artur Nowak: Methodology, Investigation, Data Curation, Software, Visualization, Resources. Kuba Kulesza: Methodology, Investigation, Data Curation, Software, Visualization, Resources. Giovanna E.U. Muti-Schünemann: Validation, Investigation, Writing - Review & Editing. Antonio Bognanni: Validation, Investigation, Writing - Review & Editing. Rana Charide: Validation, Investigation, Writing - Review & Editing. Elie Akl: Methodology, Writing - Review & Editing. Holger J. Schünemann: Conceptualization, Methodology, Formal analysis, Investigation, Resources, Writing - Original Draft, Writing - Review & Editing, Visualization, Supervision, Project administration, Funding acquisition.

Acknowledgments

Acknowledgments

This work was partially funded by the World Health Organization

Footnotes

Funding: This work was supported by a WHO grant.

Disclosures: AH, AB, EAA, TL, AD, GMS, RC, MM, FS, DM, ML, TP, QW, JB, ALM, and PS report no interest to disclose. AK, TK, DF, FM, NG, NI, EJ are employees of the World Health Organization and have no interest to disclose. KK, BD and AN are employees of Evidence Prime report support for his employee from McMaster University. AN plans to patent automatic entity resolution and organization using medical ontologies. HJS reports grants from the World Health Organization for this work and he is Co-chair of the GRADE Working Group.

HJS reports grants from the World Health Organization for this work and he is Co-chair of the GRADE Working Group. AK, TK, DF, FM, NG, NI, EJ are employees of the World Health Organization and have no interest to disclose. KK, BD and AN are employees of Evidence Prime and report support from McMaster University. AN plans to patent automatic entity resolution and organization using medical ontologies. AB, EAA, TL, AD, GMS, RC, MM, FS, DM, ML, TP, QW, JB, AH, ALM, and PS report no interest to disclose.

Supplementary material associated with this article can be found, in the online version, at doi:10.1016/j.jclinepi.2021.02.009.

Appendix. Supplementary materials

mmc1.docx (190KB, docx)

References

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

mmc1.docx (190KB, docx)

RESOURCES