Strengthening systematic reviews in public health: guidance in the Cochrane Handbook for Systematic Reviews of Interventions, 2nd edition

Abstract

Aims

Decision makers in public health practice and policy rely on access to trustworthy, relevant, synthesized evidence. The second edition of the Cochrane Handbook for Systematic Reviews of Interventions (‘the Handbook’) reflects a major revision in guidance for authors of systematic reviews, incorporating a decade of methodological development and a number of significant changes to previous recommendations. This paper aims to highlight new guidance that addresses a number of key methodological challenges for authors of systematic reviews in public health.

Results

The revised Handbook includes guidance on framing public health research questions for synthesis, considering equity, intervention complexity, risk of bias assessment and synthesis methods other than meta-analysis. Reviews of public health interventions frequently encounter the types of methodological complexity addressed in this new guidance.

Conclusion

We hope that readers will find that the Cochrane Handbook includes detailed and thoughtful guidance on both conceptualizing and executing systematic reviews relevant to public health questions. Considering the available methods guidance will, we hope, provide support for authors of public health reviews to tackle the challenges they encounter, strengthen their analysis and provide useful answers to the important questions asked by stakeholders and users of public health evidence.

Introduction

Decision makers in public health practice and policy rely on access to trustworthy, relevant, synthesized evidence. Systematic reviews are widely published on public health topics and routinely incorporated into guidelines.^1–3 As an international non-profit organization that publishes systematic reviews, Cochrane is well-known for its reviews of the effects of clinical interventions, but the Cochrane Library also includes a wide range of systematic reviews relevant to public health (www.thecochranelibrary.com). Recent examples of collaboration during the COVID-19 crisis, such as that between Cochrane, WHO and other policy agencies, highlight the potential impact of well-conducted systematic reviews that meet the needs of policy partners.⁴

Authors of Cochrane systematic reviews aim to produce reviews that are both rigorous and useful to decision makers. To do so, they require the methodological tools to handle diverse evidence appropriately. The Cochrane Handbook for Systematic Reviews of Interventions (‘the Handbook’) incorporates practical guidance on conducting a systematic review from idea to publication and has been internationally recognized as a gold standard resource since its first print publication in 2008.⁵ Updated at the end of 2019, the second edition of the Handbook⁶ represents a major revision of methodological guidance for authors of systematic reviews, reflecting a decade of methodological development and including a number of significant changes to previous recommendations. Although also available in print, the most up-to-date version of the Cochrane Handbook is accessible free of charge at https://training.cochrane.org/handbook.

In this paper, we highlight some areas of updated Cochrane guidance of particular relevance to reviews of public health questions. In each case, the guidance available provides practical strategies to enable authors of systematic reviews in public health to plan for and navigate these challenges. Many of these methods have come a long way in recent years. We encourage all those considering embarking on a systematic review to follow these signposts to more detailed guidance in the Handbook, as even those who have produced many reviews before will likely find something new.

Framing the review objectives and structuring the synthesis

To be useful, systematic reviews must address clear and meaningful objectives. Defining the objectives takes time and energy to get right, where possible involving the perspectives of stakeholders. It is common for reviews in public health to consider interventions that vary in theoretical approach, components and implementation among the included studies. Population groups may also differ in important ways. Although not every variation can be predicted in advance, the fact that variation will be found is predictable.

Planning for complexity can minimize bias arising from post hoc selection while improving clarity for readers. Authors of reviews should have a clear understanding of the important questions they seek to ask in their review, and the factors they believe may be associated with important differences in effects. Logic models and conceptual frameworks may help to think through the important concepts within the review.⁷

The eligibility criteria of the review should clearly define the review’s scope,⁷and more structure is needed to ensure the review is well-framed to investigate the effects of public health interventions. Planning at an early stage can clearly define important population or intervention groups within the review, for example, using the same PICO framework (population, intervention, comparator and outcome) that may be used to define eligibility criteria. Guidance in the updated Handbook suggests that authors then specify how these groups will be used in specific syntheses or comparisons within the review, in enough detail that the selection of studies for inclusion in a group or synthesis could be replicated.⁸

To illustrate, a review of school-based interventions to increase physical activity had as its objectives to, first, investigate the effectiveness of school-based interventions overall and, second, to identify whether any specific type of intervention was more effective than others. To achieve those objectives, the review presented an overall synthesis of the effects of any school-based intervention, followed by more specific analyses of each intervention type, grouped into enhanced physical education classes, other school-time physical activity, before or after school physical activity, or multi-component interventions.⁹ Presenting a clear definition of each group and a rationale for the groups chosen enables transparency and replicability of the decisions to group studies in particular ways and enhances understanding for readers and end users of the review’s findings.

Chapters 2 and 3 of the Handbook provide practical guidance on a structured approach for thinking through the scope and eligibility for the review as well as possible groupings of population, intervention, outcomes and study designs.^7,8

Equity

One important lens on variation among populations and settings relates to equity. Health inequity refers to differences in health outcomes that are potentially avoidable, and considered both unacceptable and unfair.¹⁰ Public health researchers are familiar with the potential for interventions to generate inequity,¹¹ for example through interaction with the factors identified in the PROGRESS-Plus framework (place of residence, race/ethnicity/culture/language, occupation, gender/sex, religion, education, socioeconomic status and social capital, plus additional factors such as age, sexual orientation and disability).¹²

Considering equity explicitly in systematic reviews from question formulation to interpretation adds value to the review’s findings, enabling end users to understand whether and how the findings of the review apply to different populations and settings, and ensuring that high-level, aggregated results do not obscure important differences in effects. Addressed for the first time in a standalone chapter, Chapter 16 of the Handbook presents strategies to consider and incorporate equity throughout the systematic review process.¹³

In question formulation, authors can consider possible differences in the prevalence, burden of disease or outcomes for specific populations, across PROGRESS-Plus factors. Again, the use of logic models can assist authors in thinking through the review question explicitly and how the intervention may work differently for specific populations. Different types of information may be needed in the review to investigate these possible differences, focused on the specific aspects of equity identified as being of most importance to each specific review, such as describing the included populations using the relevant elements of the PROGRESS-Plus framework, recording whether participants with particular characteristics are less likely to be included or more likely to withdraw from the studies and measuring different outcomes that are of importance to different populations (such as developmental outcomes for children, or measures of social participation or independence for older adults).

Finally, equity can be considered when presenting and interpreting the findings of the review. For example, a review of rotavirus vaccines presented findings separately for low-mortality and high-mortality countries, given the substantive difference in context, and reported differences in the outcomes to be expected in each setting.¹⁴ Key differences in populations or settings should be clearly communicated in summary versions of the review (such as ‘Summary of findings’ tables and abstracts).

More detailed guidance on considering and incorporating equity in systematic reviews can be found in the Handbook and on the website of the Campbell and Cochrane Equity Methods Group (www.cochranequity.org).

Intervention complexity

It is well known that public health interventions are often comprised of multiple components (what may be considered ‘complicated’ interventions). Additional complexity in interventions may also arise from interactions, between components, between interventions and participants, or with the context in which the intervention occurs. For example, mechanisms of action might depend on the social or physical context, sometimes over long causal pathways, where implementation may be modified in practice in each site or study, and replication of any specific intervention is rare. Undertaking a review with a systems perspective takes the consideration of complexity even further, considering an intervention as an event within a broader, adaptive system.¹⁵

With this complexity in mind, authors should consider how best to design their analysis to reflect the objectives of the review and the choices faced by decision makers. Exhaustively listing intervention components may complicate analysis without adding meaningful understanding to the review’s findings. Further thought may be required to identify the common elements of an intervention that are hypothesized to bring about the desired effects, whether these be specific intervention components, theoretical approaches or expected modes of action underpinning the program as a whole, or anticipated interactions.^15,16 It may be that the most useful review would not aim to estimate a universal effect, but to explore why and how an intervention may have been successful in specific cases. This may necessitate different analysis approaches, such as qualitative research on feasibility and acceptability; process and implementation outcomes; or exploration of factors associated with successful outcomes.

Chapter 17 of the Handbook outlines in further detail how authors of systematic reviews can think through what kinds of complexity might apply to their review at the planning stages, and how they might address it in the most useful way.¹⁵

Risk of bias in diverse study designs

The study designs included in any systematic review should be those appropriate to answer the review question. Although many systematic reviews of public health interventions include randomized trials, there are others that cannot, where randomized trials may be unethical, infeasible, unaffordable or simply do not exist.

Chapter 7 of the updated Handbook outlines the theoretical underpinnings of bias, using evidence-based methods to identify risk factors associated with bias in results and discussing the role of conflict of interest.¹⁷ Chapter 13 presents guidance on the risks of bias due to missing results (such as unpublished studies or unreported outcomes).¹⁸

The Handbook also introduces current tools designed to appraise risk of bias. Chapter 8 details version 2 of the Cochrane Risk of Bias tool for randomized trials (RoB 2), which incorporates considerably updated thinking about factors important to bias, including updated guidance on cluster randomized trials, such as those where schools, families or communities are randomized to receive different interventions.¹⁹ Chapter 25 details the Risk of Bias in Non-Randomized Studies of Interventions tool (ROBINS-I), which is designed to be applied to a wide range of study designs, including follow-up (cohort) studies, and controlled or uncontrolled before–after studies (including interrupted time series).²⁰ Both tools provide structured signalling questions to assist authors in thinking through each aspect to be considered as part of the assessment process. Further guidance and templates are available at https://riskofbias.info, including an Excel-based tool to assist authors with RoB 2. Examples of reviews using both tools are increasingly available in the Cochrane Library at www.cochranelibrary.com.^21,22

Synthesis methods other than meta-analysis

Around a third to half of all systematic reviews do not use meta-analysis at all,^23,24 and more do not use meta-analysis for all outcomes. Reasons why meta-analysis may not be possible are varied, but one reason may be when the included primary studies do not provide the necessary data required for a meta-analysis. For example, the studies may report statements about the direction of effect without an effect estimate, or report effect estimates without measures of variance.

For the first time, Chapter 12 of the Handbook presents a number of alternative statistical synthesis methods for this scenario, including the use of summary statistics to describe the observed range of intervention effect estimates, vote counting based on the direction of effect, or combining P values. Companion visual displays for these synthesis methods are also described, including box plots, harvest plots and albatross plots.

In some cases, synthesis may be inappropriate, for example where the included studies are at a high risk of bias or there are concerns about missing evidence. In such cases, structured tabulation or plotting the results without synthesis may be used for presentation.²⁵

Although these alternative synthesis methods yield results that are more limited for decision making in comparison to meta-analysis, they provide additional options for review authors when the required data for meta-analysis are not available. These methods may enable authors to make the most of the available data and provide a structured approach where authors may have felt limited to text-based descriptions of the findings of individual studies, which can become rapidly unwieldy where large numbers of studies or complex interventions are involved. The recently updated PRISMA²⁶ and Synthesis Without Meta-analysis (SWiM)²⁷ reporting guidelines may be of additional assistance to authors unsure of how to specify the use of these methods at the protocol stage of the review, and how to completely report which methods have been used in practice.

Conclusion

We hope that readers will find that the Cochrane Handbook includes detailed and thoughtful guidance on both conceptualizing and executing systematic reviews relevant to public health questions. Considering the available methods guidance will, we hope, provide support for authors of public health reviews to tackle the challenges they encounter, strengthen their analysis and provide useful answers to the important questions asked by stakeholders and users of public health evidence.

Data availability statement

No new data were generated or analysed in support of this research.

Miranda S. Cumpston, Senior Methods Editor

Joanne E. McKenzie, Associate Professor

Vivian A. Welch, Director and Associate Professor

Sue E. Brennan, Senior Research Fellow

Funding

This work was supported by the Australian National Health and Medical Research Council [funding for Cochrane Public Health at the University of Newcastle to M.S.C. and Career Development Fellowship 1143429 to J.E.M.]. No authors receive royalties from the sale or use of the Cochrane Handbook for Systematic Reviews of Interventions.