New Standards for Systematic Reviews Incorporate Population Health Sciences

Since its first edition in 1994, the Cochrane Handbook for Systematic Reviews of Interventions has become the standard source of methodology for identifying, appraising, and synthesizing studies about interventions to prevent, manage, and treat disease and to improve the health status of populations. It informed systematic reviews conducted by members of the Cochrane Collaboration (“Cochrane”), which focuses on health topics, and it also informed systematic reviews conducted by independent groups and other organizations. Notably, the Campbell Collaboration publishes systematic reviews and recommends that authors use the Handbook to inform research in the fields of criminal justice, disability, education, international development, and social welfare.

Cochrane Handbook for Systematic Reviews of Interventions Version 6, 2019 By Julian Higgins, James Thomas, eds Hoboken, NJ: Wiley Blackwell; 2019 728 pages; £44.99 ISBN: 978-1-119-53665-9

Cochrane began to distribute guidance on conducting reviews of health promotion and public health in 2005, and it included these fields in the edition of the Handbook published in 2008. The 2008 edition separated the chapters on health promotion and public health from chapters on clinical interventions, thus reinforcing a widely held assumption that these areas of research, policy, and practice differ categorically from each other. However, systematic reviews that meet the standards in various editions of the Handbook have been influential in health policy and in the practice of the health professions. The Cochrane Library currently categorizes 87 reviews (of a total of 8199 as of January 3, 2020) defined as addressing public health interventions. One such review in the Cochrane Library, for example, published in 1999, evaluated primary research on wearing helmets to ride bicycles.

The Handbook has been revised extensively through six editions to make more efficient methodological tools available to address why and how certain interventions work in certain contexts and not in others. The new edition of the Handbook, published in October 2019, includes added guidance for conducting reviews of population-level interventions. Particularly, it includes methodological guidance to address the complexity of population-level interventions by replacing the chapter on “Reviews in Health Promotion and Public Health” with new chapters, the most important of which address “Intervention Complexity” (chapter 17) and “Equity and Specific Populations” (chapter 16).

COMPLEX INTERVENTIONS

The new edition emphasizes that no intervention is simple. It identifies three ways of understanding intervention complexity: “(i) in terms of the number of components in the intervention; (ii) in terms of interactions between intervention components or interactions between the intervention and its context, or both; and (iii) in terms of the wider system within which the intervention is introduced.” The authors of the chapter encourage systematic reviewers to consider this complexity in their appraisal and synthesis of evidence. The Handbook emphasizes that the definition of an intervention is not static. Interventions evolve in response to the health of, or events experienced by, trial participants. Complexity, moreover, requires different approaches to assessing risk of bias. The chapter also suggests a range of methods of synthesis, both quantitative and qualitative, to examine components of interventions, mechanisms of action, and how each component and mechanism interacts with the broader context in which an intervention is implemented.

RISK OF BIAS

The new edition includes changes in how to assess risk of bias and to consider it alongside evidence of conflict of interest. A new risk of bias tool (https://www.riskofbias.info) requires that reviewers assess five domains for risk of bias (bias arising from the randomization process, bias due to deviations from intended interventions, bias due to missing outcome data, bias in measurement of the outcome, and bias in selection of the reported result). Importantly, the new tool includes questions to guide reviewers in making an overall decision for each domain.

I highlight one component that was formerly described as performance bias, which the new edition describes as “bias due to deviations from intended interventions.” Previous editions of the Handbook described performance bias as “systematic differences between groups in the care that is provided, or in exposure to factors other than the interventions of interest.” A well-known way to reduce the risk of performance bias is masking (“blinding”) study participants and personnel. The new Handbook redefines this element as “biases due to deviations from the intended intervention” and recognizes that the complexity of the intervention can lead to different ways to design or implement them in different contexts.

EQUITY AND SPECIFIC POPULATION

The Handbook has a new chapter, “Equity and Specific Populations” (chapter 16), that outlines how systematic reviewers should address health equity, defined as “absence of avoidable and unfair differences in health.” The chapter suggests that health inequities may be experienced by individuals across characteristics outlined by an acronym, PROGRESS PLUS, which defines axes of potential advantage and disadvantage: Place of residence, Race/ethnicity/culture/language, Occupation, Gender/sex, Religion, Education, Socioeconomic status, Social capital, and other characteristics (“PLUS”) such as sexual orientation, age, and disability. Systematic reviewers should consider whether certain groups are disadvantaged because of “social, political and legal structures and processes.” During synthesis, the body of evidence in primary studies can either identify disadvantaged groups that might experience different absolute or relative effects from other populations or assess the gradients of effects and gaps in absolute and relative effects between different groups with different characteristics.

GAPS IN THE HANDBOOK

Although the Handbook alludes to the issue of stakeholder engagement in setting priorities for research, it does not discuss stakeholder engagement in detail. It also does not recognize methods of priority setting as a methodological issue, although priority setting informs the selection of topics for and the conduct of systematic reviews. These topics are especially important for public health interventions that focus on populations rather than individuals. However, the new edition includes more explicit recommendations for involving stakeholders in setting priorities for review questions compared with previous editions. There is also guidance outside of the Handbook available on priority setting by the Cochrane Knowledge Translation working group 1 embedding prioritization (https://training.cochrane.org/online-learning/knowledge-translation/priority-setting). There is increasing evidence that groups, within and outside Cochrane, are working to fill these gaps.