Abstract
In the third paper in a three-part series on health systems guidance, Simon Lewin and colleagues explore the challenge of assessing how much confidence to place in evidence on health systems interventions.
Summary Points
Assessing how much confidence to place in different types of research evidence is key to informing judgements regarding policy options to address health systems problems.
Systematic and transparent approaches to such assessments are particularly important given the complexity of many health systems interventions.
Useful tools are available to assess how much confidence to place in the different types of research evidence needed to support different steps in the policy-making process; those for assessing evidence of effectiveness are most developed.
Tools need to be developed to assist judgements regarding evidence from systematic reviews on other key factors such as the acceptability of policy options to stakeholders, implementation feasibility, and equity.
Research is also needed on ways to develop, structure, and present policy options within global health systems guidance.
This is the third paper in a three-part series in PLoS Medicine on health systems guidance.
This is one paper in a three-part series that sets out how evidence should be translated into guidance to inform policies on health systems and improve the delivery of clinical and public health interventions.
Introduction
Health systems interventions establish or modify governance (e.g., licensing of professionals), financial (e.g., health insurance mechanisms) and delivery (e.g., by whom care is provided) arrangements, and implementation strategies (e.g., strategies to change health provider behaviours) within health systems (which consist of “all organisations, people and actions whose primary intent is to promote, restore or maintain health”; see Box S1 for definitions of the terms used in this article). The focus of these interventions is to strengthen health systems in their own right or to get cost-effective programmes and technologies (e.g., drugs and vaccines) to those who need them. Decisions regarding health systems strengthening, including the development of recommendations by policy makers, require evidence on the effectiveness of these interventions, as well as many other forms of evidence. For example, in assessing potential policy options, reviews of economic evaluations and of qualitative studies of stakeholders' views regarding these options might be important (Table S1). Such evidence helps to address questions such as the cost-effectiveness of these options and which options are seen as appropriate by stakeholders.
Assessing how much confidence to place in the types of evidence available on health systems interventions is a key component in informing judgements regarding the use of such interventions for health systems strengthening (Box 1). This paper, which is the third of a three-part series on health systems guidance [1],[2], aims to:
Box 1. Reasons Why It Is Important to Assess How Much Confidence Can Be Placed in Evidence to Guide Decisions on Health Systems Strengthening
Users of such evidence almost always draw implicit or explicit conclusions regarding how much confidence to place in evidence
Such evidence may inform judgements regarding recommendations, including the strength of such recommendations
- Systematic and explicit approaches can be useful in:
- facilitating critical appraisal of evidence
- protecting against bias
- clarifying implementation issues
- resolving disagreements among stakeholders
- communicating information regarding the evidence, the judgements made, and recommendations drawn from it
Systematic and explicit approaches are particularly important given the complexity of many health systems interventions. These interventions may be complex in terms of the number of discrete, active components and the interactions between them; the number of behaviours to which the intervention is directed; the number of organisational levels targeted by the intervention; the degree of flexibility or tailoring permitted in intervention implementation; the level of skill required by those delivering the intervention; and the extent of context dependency [48],[49]. Given the complexity of these interventions, deciding on the contextual relevance of evidence is a crucial component.
Source: adapted from [30].
Illustrate a range of tools available to assess the different types of evidence needed to support different steps in the policy-making process;
Discuss the GRADE (Grading of Recommendations Assessment, Development and Evaluation) approach to assessing confidence in estimates of effects (“quality of evidence”) and to grading the strength of recommendations on policy options for health systems interventions;
Discuss factors that are important when developing recommendations on policy options regarding health systems interventions.
The first paper in this series makes a case for developing guidance to inform decisions on health systems questions and explores challenges in producing such guidance and how these might be addressed [1]. The second paper explores the links between guidance development and policy development at global and national levels, and examines the range of factors that can influence policy development [2].
In this paper, which like the other two papers is based on discussions of the Task Force on Developing Health Systems Guidance (Box 2; [1],[2]), we focus particularly on the GRADE approach, which provides a transparent and systematic approach to rating the quality of evidence and grading the strength of recommendations [3].
Box 2. The Task Force on Developing Health Systems Guidance
The Task Force on Developing Health Systems Guidance was established in 2009 by the World Health Organization (WHO) to improve its response to requests for guidance on health systems.
The Task Force consisted of 20 members selected by WHO for their expertise in the field of health systems research and implementation.
Through a series of face-to-face and virtual meetings, the Task Force provided input to and oversight of the development of a Handbook for Developing Health Systems Guidance and to the identification of broader issues that warranted further dialogue and debate [39].
As part of this process, the Task Force and the Handbook developers reviewed approaches to developing clinical guidelines and the instruments used for clinical guideline development.
The Task Force suggested ways in which some of these approaches and instruments could be adapted for use in the development of health systems guidance and indicated where there were important differences between these approaches.
The writing group for this paper further considered the issues raised in these discussions and produced a first draft of the manuscript for comment by the Task Force.
This paper, and the other two in the series [1],[2], were finalised after several iterations of comments by the Task Force and external reviewers.
Tools to Assess the Evidence Needed to Support the Policy-Making Process for Health Systems Strengthening
Well-conducted systematic reviews [4] can be used to identify the best available evidence to inform judgements about the effects of policy options and to inform other key steps within the policy-making process (Table S1). As discussed in the second paper of this series [2], users need to be able to assess the quality of evidence presented in such reviews in relation to each step of the policy-making process. For example, when defining the problem and the need for intervention, tools are required to assess the confidence we can place in evidence from reviews of studies highlighting different ways of conceptualising the problem (e.g., reviews of studies of people's experiences of the problem) [5]. When assessing potential policy options, tools are needed to assess the confidence that can be placed in, for example, studies assessing impact (e.g., reviews of effectiveness studies). Similarly, when identifying implementation considerations, tools are required to assess the confidence that can be placed in reviews of factors affecting implementation.
Many tools are available to assess the risk of bias in individual studies of the effects of interventions [6] and to appraise individual qualitative studies [7]. Tools are also available to assess the quality of evidence synthesised in systematic reviews [6]. Such tools need to be appropriate to the types of studies included in the review and generic enough to be applicable across a range of questions, and must allow meaningful conclusions to be drawn regarding the quality of the included evidence. Judgements on how much confidence can be placed in the evidence from a review need to be distinguished from judgements about how well the review was conducted (i.e., its reliability). Tools have been developed to assess the reporting of systematic reviews and meta-analyses (e.g., the PRISMA checklist [8]) and to assess their methodological quality or reliability (e.g., the SUPPORT tools [9] and AMSTAR [10]) (Table 1). However, we focus here on tools to assess how much confidence can be placed in the evidence identified and presented in those reviews.
Table 1. Commonly used tools to assess systematic reviews and their findings and to assess clinical guidelines.
| Systematic Reviews | |
| SUPPORT (SUPporting Policy Relevant Reviews and Trials) tool [9] | A tool to assess how much confidence to place in the methodological quality of a systematic review, and designed for reviews of health systems interventions |
| AMSTAR (A measurement tool for the “assessment of multiple systematic reviews”) [10] | A tool designed to assess the methodological quality of a systematic review. This tool has not been designed specifically to assess how much confidence to place in reviews of health systems interventions |
| GRADE (Grading of Recommendations Assessment, Development and Evaluation) [41] | An approach to assess the quality of evidence |
| PRISMA (Preferred Reporting Items for Systematic Reviews and Meta Analyses) [8] | A tool to assess the reporting of systematic reviews and meta-analyses |
| SUPPORT applicability tool [33] and Wang applicability and transferability tool [34] | Tools to assist in assessing the applicability of the findings of a systematic review to a specific setting |
Assessing How Much Confidence to Place in the Findings of Reviews of the Effects of Policy Options
Tools to assess how much confidence to place in review findings are most developed for systematic reviews of evidence on effectiveness. The GRADE approach is one such tool, but many others are available [11]. Within GRADE, the quality of evidence derived from a systematic review is related to the quality of the included studies and to a range of other factors (Table S2). This approach has many strengths (Table S3) and is now used increasingly by international organizations, including the World Health Organization (WHO), the Cochrane Collaboration, and several agencies developing guidelines [3]. (Also see http://www.gradeworkinggroup.org.) We will discuss the application of the GRADE approach to assess the quality of evidence and the strength of recommendations for health systems interventions later.
Assessing How Much Confidence to Place in the Findings of Reviews of Questions Other Than Effects
Tools to assist judgements on how much confidence to place in the findings of reviews of questions other than effects that are relevant to the policy-making process are at an early stage of development. Such questions include stakeholders' values and preferences and the feasibility of interventions (Table S1). For some of these issues, judgements might be informed by systematic reviews of qualitative studies, together with local evidence [12]. Where reviews seek a qualitative answer to understand the nature of a problem, quality appraisal aims to assess the coherence of the resulting explanation, possibly across different contexts. Although quality criteria for individual qualitative research studies commonly consider the methods of each study and the credibility and richness of their findings [7],[13],[14], thus far tools for assessing the quality of systematic reviews of qualitative research have not considered the credibility and richness of findings. A potential tool for doing this is proposed in Table S4.
Reviews exploring factors affecting the implementation of options might employ mixed methods syntheses (i.e., syntheses of both qualitative and quantitative evidence) such as realist synthesis [15], which explores the explanatory theories implicit in existing programmes or policies, or framework synthesis, which provides a highly structured, deductive approach to data analysis drawing on an existing model or framework (for example, [16]–[18]). Quality appraisal then focuses on the confidence that can be placed in each conclusion drawn from individual studies [19]. Grading the evidence as a whole can take into account the number and context of the studies contributing to each conclusion, and the appropriateness of their methods for drawing that conclusion (for illustration, see [16]). A single study might refute or qualify a theory, but multiple studies together contribute to strengthening a theory. As yet, there are no tools for appraising how well mixed methods reviews have synthesised studies to draw conclusions about the advantages or disadvantages of policy options.
Resource use is another key issue, and tools are available to assess the reliability of reviews of economic studies [20]. In addition, GRADE provides guidance on how to incorporate considerations of resource use into recommendations [21].
The GRADE Approach to Assessing Confidence in the Estimates of Effects for Health Systems Interventions
The GRADE approach clearly separates two issues: the quality of the evidence and the strength of recommendations. Quality of evidence is only one of several factors considered when assessing the strength of recommendations.
Within the context of a systematic review, GRADE defines the quality of evidence as the extent to which one can be confident in the estimate of effect. Within the context of guidelines or guidance, GRADE defines the quality of evidence as the confidence that the effect estimate supports a particular recommendation. The degree of confidence is a continuum but, for practical purposes, it is categorised into high, moderate, low, and very low quality (Table S5).
Evidence on the effectiveness of health systems interventions raises a number of challenges that may, in turn, influence assessments of the quality of this evidence and the development of recommendations using the GRADE approach. Firstly, while experimental studies (including pragmatic randomised trials [22]) are feasible for some health systems interventions, for others (particularly those related to governance and financial arrangements), evidence may come mainly from observational studies, including evaluations of national or state-wide programmes [23],[24]. Secondly, evaluations of health systems interventions often use clustered designs and these are frequently poorly conducted, analysed, and reported [25],[26]. Thirdly, health systems interventions tend to measure proxy outcomes, such as the use of services or the uptake of an incentive. Evidence users need to decide whether there is sufficiently strong evidence of a relationship between the proxy outcome and the desired health outcome. The development of an outcomes framework to assist in assessing interventions (for an example, see [27],[28]) may help those developing guidance decide whether proxy outcomes are sufficient. Finally, poorly described health systems and political systems factors and implementation considerations may make it difficult to develop contextualised recommendations on policy options [29]. GRADE attempts to make the judgements regarding these issues systematic and transparent.
Developing Recommendations on Policy Options for Consideration Regarding Health Systems Interventions
Moving from evidence to recommendations on options for consideration often necessitates the interpretation of factors other than evidence. In most cases, these interpretations require judgments, making it important to be transparent, particularly given that recommendations will sometimes need to consider multiple complex health systems interventions, each with its own assessment of quality of evidence. Another challenge is the additional complexity of assessing the wide range of health system and political system factors that will influence the choice and implementation of options for addressing a health system problem in different settings (see the other papers in this series [1],[2]). For example, a health systems problem may involve a wide range of stakeholders, each with views regarding the available options. In addition, health systems interventions may have system-wide effects that vary across settings. Consequently, rather than making a single recommendation as in clinical guidelines, it may be more useful for health systems guidance to set out the evidence and outline a range of options, appropriate to different settings, to address a given health systems problem. These options may, in turn, feed into deliberative or decision-making processes at national or sub-national levels, as discussed later and elsewhere in this series [2].
Tools such as GRADE assist in grading the strength of a recommendation regarding options [30],[31] and can be applied to health systems interventions, but may benefit from explicitly including some additional factors (Box 3). Further research is needed to explore the usefulness of these additional factors but, in general, any tool used to guide the development of recommendations should aim to improve transparency by explicitly describing the factors, and their interpretation, that contributed to the development of recommendations.
Box 3. Factors That May Inform Decisions about the Strength of Recommendations Regarding Policy Options
GRADE factors (adapted from [30] ):
Whether there is uncertainty about the balance of benefits versus harms and burdens
The quality of the evidence from the systematic review (very low, low, moderate, high)
Whether there is uncertainty or variability in values and preferences among stakeholders
Whether there is uncertainty about whether the net benefits are worth the costs or about resource use
Whether there is uncertainty about the feasibility of the intervention (or about local factors that influence the translation of evidence into practice, including equity issues)
Additional factors that it may be useful to consider for health systems interventions:
Ease of implementation at the systems level, including governance arrangements (e.g., changes needed in regulations), financial arrangements (e.g., the extent to which the options fit with financing models within settings), and implementation strategies (e.g., how to provide the skills and experience needed among implementers or facilitators)
Socio-political considerations, e.g., how the proposed options relate to existing policies, values within the political system in relation to issues such as equity or privatisation, and economic changes
Within the GRADE approach, recommendations reflect the degree of confidence that the desirable effects of applying a recommendation outweigh the undesirable effects. Specifically, a strong recommendation implies confidence that the desirable effects of applying a recommendation outweigh the undesirable effects, whereas a conditional/qualified/weak recommendation suggests that the desirable effects of applying a recommendation probably outweigh the undesirable effects, but there is uncertainty.
GRADE attempts to make all judgments regarding the factors that are considered in developing recommendations transparent (by documenting these judgments) and systematic (by using the same approach across all the questions being considered by the guideline). Tables 2, S6, and S7 provide illustrations of the application of the GRADE approach to health systems interventions involving delivery and financial arrangements and implementation strategies, respectively, and show how guidance on health systems interventions might outline a range of options appropriate to different settings—an approach on which further research is needed. In common with other grading systems, GRADE does not yet provide guidance on how to assess the level of confidence that can be placed in evidence on “acceptability” or “feasibility”. Conventionally, these judgements have been made by consensus among the guideline panel, which needs to include individuals with expertise and experience relevant to the guideline questions. Further work is needed to develop a formal way of assessing the quality of such evidence.
Table 2. Example of factors affecting decisions about strength of recommendations—Lay or community health workers to reduce childhood mortality.
| Population: Children in high mortality settingsIntervention: Lay health workers (LHWs) delivering health promotion, treatment, and referral interventionsComparison: No LHW intervention / usual careOutcome: Childhood mortality | ||
| Key factors—is there uncertainty regarding: | Decision regarding whether there is uncertainty (yes / no) | Explanation of the decision made |
| Quality of evidence | Yes | The use of LHWs in maternal and child health programmes may lead to fewer deaths among children under five (low quality evidence—GRADE). In addition, the use of LHWs probably leads to an increase in the number of women who breastfeed and to the number of children who have their immunisation schedule up to date (moderate quality evidence for both outcomes— GRADE). These additional outcomes are also related to mortality reduction |
| Balance of benefits versus harms and burdens | Yes | Potentially important benefits (mortality reduction) but confidence interval also includes harm. Additional evidence on LHWs suggests effectiveness, e.g., LHWs associated with increased uptake of interventions of proven cost-effectiveness (immunisation, breastfeeding) |
| Acceptability | Yes | • Some evidence that LHWs acceptable to service users and used widely• Varied acceptability to other services providers in different settings (e.g., [43],[44]) |
| Resource use | Yes | Potentially large investment needed over long period but alternatives likely to be more costly |
| Feasibility (or local factors that influence the translation of evidence into practice) | Yes | There may be constraints to scaling up trained LHWs and supporting them, but it is even less feasible to scale up professional cadres. There are a number of well-documented examples of LHW programmes that have been taken to scale for which monitoring has suggested some positive outcomes, e.g., in Ethiopia and Pakistan [45],[46] |
| Recommended options for consideration This assessment of evidence within a wider health system context might result in the following recommended options for consideration:• Option 1: Where child mortality is high; an infrastructure for LHWs can be developed rapidly; it is unlikely that the numbers of other cadres could be expanded; and this cadre is acceptable to other providers and to service users and has strong political support:○ Strong recommendation to implement LHWs to reduce childhood mortality (i.e., there is confidence that the desirable effects of LHWs delivering interventions to reduce childhood mortality outweigh the undesirable effects).• Option 2: Where child mortality is high; LHWs are acceptable to other providers and to service users; but governance and financing mechanisms for LHWs will need to be established, and there is little experience of running such programmes and uncertainty among policy makers:○ Conditional recommendation to implement LHWs to reduce childhood mortality (i.e., the desirable effects of LHWs delivering interventions to reduce childhood mortality probably outweigh the undesirable effects, but there is uncertainty).• Option 3: Where child mortality is moderate; an infrastructure for LHWs can be developed rapidly; but there is evidence that the scaling up of LHWs may be challenged by health care professionals:○ Conditional recommendation to implement LHWs to reduce childhood mortality, dependent on the ability to overcome professional opposition (i.e., the desirable effects of LHWs delivering interventions to reduce childhood mortality probably outweigh the undesirable effects, but there is uncertainty). | ||
Source: This table draws on evidence from [47].
Challenges in Moving from an Assessment of the Quality of Evidence to Making Recommendations on Policy Options
The move from an assessment of the quality of evidence to making recommendations on policy options involves a number of challenges. Firstly, assessments of the strength of a recommendation may require a detailed understanding of the evidence creation and evaluation process that is not always available. Secondly, categorising recommendations as “strong” and “weak” can raise difficulties. Panels developing global guidance may, for example, be reluctant to make “weak” recommendations in case policy makers fail to respond to such recommendations because they assume they are equivalent to “no recommendation”. Thirdly, the quality of evidence is typically assessed for two alternative policy options in tools such as GRADE, while many health system (and, indeed, many clinical decisions) involve multiple interventions, which adds to the complexity of interpretation and decision-making and makes it even more important to be transparent.
How might some of these challenges be addressed? Methodological expertise is needed to conduct and interpret systematic reviews and perform assessments using tools such as GRADE. Health systems guidance panels therefore need to be supported by methodology experts. Moreover, the outputs of these tools need to be “translated” into appropriate language and formats to ensure that they can be interpreted and used correctly by the panel. Research is under way within initiatives such as the DECIDE (Developing and Evaluating Communication strategies to support Informed Decisions and practice based on Evidence; http://www.decide-collaboration.eu) collaboration on ways of presenting information on GRADE assessments and policy options to policy makers.
Additional Challenges
There are other wider challenges involved in making recommendations on policy options regarding health systems interventions. Firstly, the tools used to assess the quality of evidence and develop recommendations need to be able to accommodate the wide range of study designs that is used to assess the effectiveness of health system interventions. This is possible within GRADE. Secondly, tools need to be developed to inform judgements on how much confidence to place in the other forms of evidence (e.g., evidence on acceptability) that are needed to develop recommendations regarding health systems interventions [32].
Finally, international standard-setting organisations, such as WHO, have to formulate recommendations that are applicable at a global level. However, as noted earlier, creating global recommendations on health systems questions can be difficult because of important variations in context-specific factors that influence the applicability of interventions at national and sub-national levels [2],[33]. An approach that should help to link guidance development at the global level with policy development at the national level is outlined in the second paper of this series [2].
Where it is useful to make recommendations at the global level, those developing guidance may choose to outline policy options rather than a single recommendation. Such options may encompass one or more questions and may be based on the range of interventions considered in relation to these questions. Health and political systems factors could be taken into account by linking specific options to these factors. For example, the options may describe variations in the intervention content and method of implementation, based on the evaluations that have been conducted in different settings. Further work is needed to explore how policy makers might interpret and select policy options outlined in global guidance. However, one useful approach might be to provide national decision makers with tools to assist them in making recommendations appropriate to their setting. Several such tools are available [33],[34] or in development (see http://www.decide-collaboration.eu/work-packages-strategies). Importantly, global guidance should always indicate the factors that should be considered to assess the implications of variations in intervention, context, and other conditions. Decision models may be useful in exploring the effects of these variations (for example, [35],[36]). Given the often low quality evidence available regarding policy options for health systems problems, it is also likely that in many cases the recommended option(s) will need to be evaluated.
Presenting Evidence Regarding Contextual and Implementation Issues to Guidance Panels and Policy Makers
The best way to communicate evidence on contextual and implementation issues related to health systems and political systems to guidance panels and policy makers to inform their judgements about the strength of recommended options is currently unclear. Related work on summary of findings tables for systematic reviews of effects and evidence summaries for policy makers has illustrated the importance of paying attention to both format and content in developing useful and understandable presentation approaches [37],[38]. The Handbook for Developing Health Systems Guidance sets out an approach for presenting this type of evidence to stakeholders in a user-friendly evidence profile [39]. Similarly, the second paper in this series describes the wider features of health and political systems that may need to be assessed to inform decision-making [2].
If we want to ensure that guidance panels and policy makers use evidence to inform judgements about the strength of recommended options, more research is needed to develop and test approaches (including visual formats) for presenting the available evidence to such groups. In addition, efforts are needed to build the capacity of policy makers to use evidence to inform their decisions [2],[12],[40].
Conclusions
Useful tools are available for grading quality of evidence and strength of recommendations on policy options regarding health systems interventions, but several challenges need to be addressed. Firstly, these tools involve judgements, and these need to be made systematically and transparently. Secondly, for many health systems questions, evidence is still likely to be of low quality. Better quality research in these areas is needed and would allow guidance panels to have more confidence in the evidence and to issue stronger recommendations. Thirdly, research is needed on ways to develop, structure, and present policy options for consideration within global health systems guidance. These options need to include evidence on health and political system factors and implementation considerations, and tools to assess such evidence need to be refined. Finally, greater attention needs to be given to how guidance on health systems interventions may be implemented at the local level.
Supporting Information
Translation of the Summary Points into Spanish by Xavier Bosch-Capblanch
(DOC)
Translation of the Summary Points into French by Bruno Clary, William Lenoir, and Lise Beck
(DOC)
Translation of the Summary Points into Portuguese by Bruno Viana
(DOC)
Translation of the Summary Points into Arabic by Fadi El-Jardali
(DOC)
Definitions of terms used in this paper
(PDF)
Types of systematic reviews needed for different steps in the policy-making process and the availability of tools to assess how much confidence can be placed in the evidence presented in these reviews
(PDF)
GRADE criteria for assessing the quality of evidence for each important outcome assessed in a systematic review of effects
(PDF)
Strengths of the GRADE approach
(PDF)
Assessing how much confidence can be placed in the findings of systematic reviews of qualitative studies
(PDF)
Definitions of the quality of evidence categories within GRADE
(PDF)
Example of factors affecting decisions about strength of recommendations—Changes in user fees in low- and middle-income countries
(PDF)
Example of factors affecting decisions about strength of recommendations—Continuing education programmes for rural health workers to support their retention
(PDF)
Acknowledgments
We acknowledge other members of the Task Force on Developing Health Systems Guidance, who include (with their affiliations at the time when the Task Force was initiated): Edgardo Abalos, Centro Rosarino de Estudios Perinatales (Argentina); Abdul Ghaffar, Alliance for Health Policy and Systems Research, World Health Organization (Switzerland); Timothy Evans, Assistant Director-General, Information, Evidence and Research, World Health Organization (Switzerland); Regina Kulier, Department of Research Policy and Cooperation, Information, Evidence and Research, World Health Organization (Switzerland); Pierre Ongolo-Zogo, Centre for the Development of Best Practices in Health, Yaoundé Central Hospital, and University of Yaoundé, Yaoundé (Cameroon); Tikki Pang, Department of Research Policy and Cooperation, Information, Evidence and Research, World Health Organization (Switzerland); Ulysses Panisset, Department of Research Policy and Cooperation, Information, Evidence and Research, World Health Organization (Switzerland).
Thanks also to the following for their thoughtful input: Claire Glenton and Andy Oxman, Norwegian Knowledge Centre for the Health Services (Norway).
Abbreviations
- GRADE
Grading of Recommendations Assessment, Development and Evaluation
- WHO
World Health Organization
Footnotes
The Swiss Tropical and Public Health Institute and the Norwegian Knowledge Centre for the Health Services received funds from the WHO for the contributions of DD, PS, LB, SL, and XBC to developing a Handbook to produce health systems guidance, and some of this work is reported in this article. DG is a member of the PLoS Medicine Editorial Board. EAA and GEV are members of the GRADE Working Group. All other authors declare no competing interests.
The development of the Handbook was supported by a grant to the WHO by the Rockefeller Foundation, which had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. The paper represents the views of the authors and neither WHO nor the Rockefeller Foundation.
Provenance: Not commissioned; externally peer reviewed.
References
- 1.Bosch-Capblanch X, Lavis JN, Lewin S, Atun R, Røttingen JA, et al. Guidance for evidence-informed decisions about health systems: rationale for and challenges of guidance development. PLoS Med. 2012;9:e1001185. doi: 10.1371/journal.pmed.1001185. doi: 10.1371/journal.pmed.1001185. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Lavis JN, Røttingen JA, Bosch-Capblanch X, Atun R, El-Jardali F, et al. Guidance for evidence-informed policies about health systems: linking guidance development to policy development. PLoS Med. 2012;9:e1001187. doi: 10.1371/journal.pmed.1001186. doi: 10.1371/journal.pmed.1001187. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64:383–394. doi: 10.1016/j.jclinepi.2010.04.026. [DOI] [PubMed] [Google Scholar]
- 4.Lavis JN, Oxman AD, Lewin S, Fretheim A. SUPPORT tools for evidence-informed health policymaking (STP). Health Res Policy Syst. 2009;7(Suppl 1):I1. doi: 10.1186/1478-4505-7-S1-I1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Lavis JN, Wilson MG, Oxman AD, Grimshaw J, Lewin S, et al. SUPPORT tools for evidence-informed health policymaking (STP) 5: using research evidence to frame options to address a problem. Health Res Policy Syst. 2009;7(Suppl 1):S5. doi: 10.1186/1478-4505-7-S1-S5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.West S, King V, Carey TS, Lohr KN, McKoy N, et al. Systems to rate the strength of scientific evidence. 2002. Evidence report/technology assessment No. 47 (prepared by the Research Triangle Institute–University of North Carolina Evidence-based Practice Center under Contract No. 290-97-0011). AHRQ publication no. 02-E016. [PMC free article] [PubMed]
- 7.Cohen DJ, Crabtree BF. Evaluative criteria for qualitative research in health care: controversies and recommendations. Ann Fam Med. 2008;6:331–339. doi: 10.1370/afm.818. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol. 2009;62:e1–e34. doi: 10.1016/j.jclinepi.2009.06.006. [DOI] [PubMed] [Google Scholar]
- 9.Lewin S, Oxman AD, Lavis JN, Fretheim A. SUPPORT tools for evidence-informed health policymaking (STP) 8: deciding how much confidence to place in a systematic review. Health Res Policy Syst. 2009;7(Suppl 1):S8. doi: 10.1186/1478-4505-7-S1-S8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7:10. doi: 10.1186/1471-2288-7-10. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Atkins D, Eccles M, Flottorp S, Guyatt GH, Henry D, et al. Systems for grading the quality of evidence and the strength of recommendations I: critical appraisal of existing approaches The GRADE Working Group. BMC Health Serv Res. 2004;4:38. doi: 10.1186/1472-6963-4-38. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Oxman AD, Lavis JN, Lewin S, Fretheim A. SUPPORT tools for evidence-informed health policymaking (STP) 1: What is evidence-informed policymaking? Health Res Policy Syst. 2009;7(Suppl 1):S1. doi: 10.1186/1478-4505-7-S1-S1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008;8:45. doi: 10.1186/1471-2288-8-45. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Spencer L, Ritchie J, Lewis J, Dillon L. Quality in qualitative evaluation: a framework for assessing research evidence. England: Government Chief Social Researcher's Office; 2003. [Google Scholar]
- 15.Pawson R. Evidence-based policy: the promise of ‘realist’ synthesis. Evaluation. 2002;8:340–358. [Google Scholar]
- 16.Oliver S, Clarke-Jones L, Rees R, Milne R, Buchanan P, et al. Involving consumers in research and development agenda setting for the NHS: developing an evidence-based approach. Health Technol Assess. 2004;8:1–IV. doi: 10.3310/hta8150. [DOI] [PubMed] [Google Scholar]
- 17.Kiwanuka SN, Kinengyere AA, Rutebemberwa E, Nalwadda C, Ssengooba F, et al. Dual practice regulatory mechanisms in the health sector: a systematic review of approaches and implementation. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London; 2011. [Google Scholar]
- 18.Carroll C, Booth A, Cooper K. A worked example of “best fit” framework synthesis: a systematic review of views concerning the taking of some potential chemopreventive agents. BMC Med Res Methodol. 2011;11:29. doi: 10.1186/1471-2288-11-29. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Pawson R. Digging for nuggets: how ‘bad’ research can yield ‘good’ evidence. Int J Soc Res Methodol. 2006;9:127–142. [Google Scholar]
- 20.Jefferson T, Demicheli V, Vale L. Quality of systematic reviews of economic evaluations in health care. JAMA. 2002;287:2809–2812. doi: 10.1001/jama.287.21.2809. [DOI] [PubMed] [Google Scholar]
- 21.Guyatt GH, Oxman AD, Kunz R, Jaeschke R, Helfand M, et al. Incorporating considerations of resources use into grading recommendations. BMJ. 2008;336:1170–1173. doi: 10.1136/bmj.39504.506319.80. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Treweek S, Zwarenstein M. Making trials matter: pragmatic and explanatory trials and the problem of applicability. Trials. 2009;10:37. doi: 10.1186/1745-6215-10-37. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Lewin S, Lavis JN, Oxman AD, Bastias G, Chopra M, et al. Supporting the delivery of cost-effective interventions in primary health-care systems in low-income and middle-income countries: an overview of systematic reviews. Lancet. 2008;372:928–939. doi: 10.1016/S0140-6736(08)61403-8. [DOI] [PubMed] [Google Scholar]
- 24.Chopra M, Munro S, Lavis JN, Vist G, Bennett S. Effects of policy options for human resources for health: an analysis of systematic reviews. Lancet. 2008;371:668–674. doi: 10.1016/S0140-6736(08)60305-0. [DOI] [PubMed] [Google Scholar]
- 25.Donner A, Klar N. Pitfalls of and controversies in cluster randomization trials. Am J Public Health. 2004;94:416–422. doi: 10.2105/ajph.94.3.416. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Varnell SP, Murray DM, Janega JB, Blitstein JL. Design and analysis of group-randomized trials: a review of recent practices. Am J Public Health. 2004;94:393–399. doi: 10.2105/ajph.94.3.393. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.WHO. Increasing access to health workers in remote and rural areas through improved retention: global policy recommendations. Geneva: World Health Organization; 2010. [PubMed] [Google Scholar]
- 28.Huicho L, Dieleman M, Campbell J, Codjia L, Balabanova D, et al. Increasing access to health workers in underserved areas: a conceptual framework for measuring results. Bull World Health Organ. 2010;88:357–363. doi: 10.2471/BLT.09.070920. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Glasziou P, Chalmers I, Altman DG, Bastian H, Boutron I, et al. Taking healthcare interventions from trial to practice. BMJ. 2010;341:c3852. doi: 10.1136/bmj.c3852. [DOI] [PubMed] [Google Scholar]
- 30.Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336:924–926. doi: 10.1136/bmj.39489.470347.AD. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Guyatt GH, Oxman AD, Schunemann HJ, Tugwell P, Knottnerus A. GRADE guidelines: a new series of articles in the Journal of Clinical Epidemiology. J Clin Epidemiol. 2011;64:380–382. doi: 10.1016/j.jclinepi.2010.09.011. [DOI] [PubMed] [Google Scholar]
- 32.Peters DH, El-Saharty S, Siadat B, Janovsky K, Vujicic M. Improving health service delivery in developing countries: from evidence to action. Washington (D.C.): World Bank; 2009. [Google Scholar]
- 33.Lavis JN, Oxman AD, Souza NM, Lewin S, Gruen RL, et al. SUPPORT tools for evidence-informed health policymaking (STP) 9: assessing the applicability of the findings of a systematic review. Health Res Policy Syst. 2009;7(Suppl 1):S9. doi: 10.1186/1478-4505-7-S1-S9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Wang S, Moss JR, Hiller JE. Applicability and transferability of interventions in evidence-based public health. Health Promot Int. 2006;21:76–83. doi: 10.1093/heapro/dai025. [DOI] [PubMed] [Google Scholar]
- 35.Victora CG. Commentary: LiST: using epidemiology to guide child survival policymaking and programming. Int J Epidemiol. 2010;39(Suppl 1):i1–i2. doi: 10.1093/ije/dyq044. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Bryce J, Friberg IK, Kraushaar D, Nsona H, Afenyadu GY, et al. LiST as a catalyst in program planning: experiences from Burkina Faso, Ghana and Malawi. Int J Epidemiol. 2010;39(Suppl 1):i40–i47. doi: 10.1093/ije/dyq020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Rosenbaum SE, Glenton C, Wiysonge CS, Abalos E, Mignini L, et al. Evidence summaries tailored to health policy-makers in low- and middle-income countries. Bull World Health Organ. 2011;89:54–61. doi: 10.2471/BLT.10.075481. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Rosenbaum SE, Glenton C, Nylund HK, Oxman AD. User testing and stakeholder feedback contributed to the development of understandable and useful Summary of Findings tables for Cochrane reviews. J Clin Epidemiol. 2010;63:607–619. doi: 10.1016/j.jclinepi.2009.12.013. [DOI] [PubMed] [Google Scholar]
- 39.Bosch-Capblanch X and Project Team. Handbook for developing health systems guidance: supporting informed judgements for health systems policies. Basel: Swiss Tropical and Public Health Institute; 2011. Available: http://www.swisstph.ch/?id=53; http://www.swisstph.ch/fileadmin/user_upload/Pdfs/SCIH/WHOHSG_Handbook_v04.pdf. Accessed 10 February 2012. [Google Scholar]
- 40.Bennett S, Agyepong IA, Sheikh K, Hanson K, Ssengooba F, et al. Building the field of health policy and systems research: an agenda for action. PLoS Med. 2011;8:e1001081. doi: 10.1371/journal.pmed.1001081. doi: 10.1371/journal.pmed.1001081. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64:383–394. doi: 10.1016/j.jclinepi.2010.04.026. [DOI] [PubMed] [Google Scholar]
- 42.Brouwers MC, Kho ME, Browman GP, Burgers JS, Cluzeau F, et al. AGREE II: advancing guideline development, reporting and evaluation in health care. J Clin Epidemiol. 2010;63:1308–1311. doi: 10.1016/j.jclinepi.2010.07.001. [DOI] [PubMed] [Google Scholar]
- 43.Kane SS, Gerretsen B, Scherpbier R, Dal PM, Dieleman M. A realist synthesis of randomised control trials involving use of community health workers for delivering child health interventions in low and middle income countries. BMC Health Serv Res. 2010;10:286. doi: 10.1186/1472-6963-10-286. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Glenton C, Lewin S, Scheel IB. Still too little qualitative research to shed light on results from reviews of effectiveness trials: a case study of a Cochrane review on the use of lay health workers. Implement Sci. 2011;6:53. doi: 10.1186/1748-5908-6-53. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Admassie A, Abebaw D, Woldemichael AD. Impact evaluation of the Ethiopian Health Services Extension Programme. J Dev Effect. 2009;1:430–449. [Google Scholar]
- 46.Oxford Policy Management. Lady health worker programme. External evaluation of the National Programme for Family Planning and Primary Health Care. Management review. Oxford: Oxford Policy Management; 2009. [Google Scholar]
- 47.Lewin S, Munabi-Babigumira S, Glenton C, Daniels K, Bosch-Capblanch X, et al. Lay health workers in primary and community health care for maternal and child health and the management of infectious diseases. Cochrane Database Syst Rev. 2010;CD004015 doi: 10.1002/14651858.CD004015.pub3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48.Lewin S, Oxman AD, Glenton C. Assessing healthcare interventions along the complex-simple continuum: a proposal [P100]. 2006. Cochrane Colloquium 2006.
- 49.Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008;337:a1655. doi: 10.1136/bmj.a1655. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Translation of the Summary Points into Spanish by Xavier Bosch-Capblanch
(DOC)
Translation of the Summary Points into French by Bruno Clary, William Lenoir, and Lise Beck
(DOC)
Translation of the Summary Points into Portuguese by Bruno Viana
(DOC)
Translation of the Summary Points into Arabic by Fadi El-Jardali
(DOC)
Definitions of terms used in this paper
(PDF)
Types of systematic reviews needed for different steps in the policy-making process and the availability of tools to assess how much confidence can be placed in the evidence presented in these reviews
(PDF)
GRADE criteria for assessing the quality of evidence for each important outcome assessed in a systematic review of effects
(PDF)
Strengths of the GRADE approach
(PDF)
Assessing how much confidence can be placed in the findings of systematic reviews of qualitative studies
(PDF)
Definitions of the quality of evidence categories within GRADE
(PDF)
Example of factors affecting decisions about strength of recommendations—Changes in user fees in low- and middle-income countries
(PDF)
Example of factors affecting decisions about strength of recommendations—Continuing education programmes for rural health workers to support their retention
(PDF)