Improving the Quality of Systematic Reviews in Public Health: Introduction to the Series

Synthesizing evidence relevant to public health is now more important than ever. The global health threat posed by the COVID-19 pandemic has stimulated research production across the spectrum of prevention, detection, treatment, and recovery. As of July 2020, the Cochrane COVID-19 Registry contained more than 11 000 ongoing primary studies started within the first three months of the pandemic. The rate of scientific publication appears to be doubling every 14 days. At the same time, the number of evidence syntheses has skyrocketed and includes rapid reviews, systematic reviews, “living” systematic reviews, and meta-analyses. Decision makers need trustworthy, rigorous primary studies and evidence syntheses.

I am launching a series of commentaries on advances in methods for designing and conducting systematic reviews relevant to public health. The series will advance AJPH’s mission of publishing rigorous intervention and policy research by focusing on methodological issues in prioritizing, conducting, and disseminating the results of systematic reviews. Using systematic reviews and experience from the Cochrane Public Health and Health Systems Network as examples, the series aims to demonstrate the value and challenges in preparing systematic reviews and using them in public health policy and practice.

The series will describe ways that systematic reviews can be improved to better meet the needs of public health policymakers. It will highlight the main advances in methods for conducting systematic reviews on public health topics. Lastly, the series will outline future directions for methods development that is needed to tackle areas of debate or uncertainty. Specific topics will include methods for stakeholder engagement in systematic reviews, use of logic frameworks to plan systematic reviews, the values and challenges of qualitative evidence synthesis and synthesis of observational studies, core outcome sets for public health questions, and synthesis without meta-analysis. We aim for the series to provide helpful tips for authors and users of public health–relevant systematic reviews.

Conducting systematic reviews on public health topics presents a number of methodological challenges. For example, the efficacy of COVID-19 public health measures—such as isolation, social supports to improve mental health during isolation, quarantine, social distancing, and travel bans—is rarely studied using randomized controlled trials. Other study designs, such as observational studies, modeling studies, and qualitative research, are often needed to address the questions most important to public health. Of the more than 11 000 studies in the Cochrane COVID-19 Registry, 8442 are of observational design, 982 are modeling studies, and 311 are qualitative research. Yet, systematic review methods have been optimized for synthesizing randomized controlled trials of interventions.¹

Systematic review methods need to evolve beyond the current methods used to identify, assess risk of bias of, and synthesize data from randomized controlled trials. Environmental health is another area of public health where traditional systematic review methods are insufficient. Environmental health researchers need to synthesize data from a variety of types of studies to assess the potential harmful effects of exposures that are not under the control of the investigator.^2,3 Thus, researchers in environmental health must address similar methodological issues and have begun to adapt traditional systematic review methods.^4,5

The stakeholders who are interested in the efficacy of public health measures are as diverse as the methods needed to study them. They include policymakers, regulators, health systems administrators, health care practitioners, and the public. In addition to efficacy effects on health outcomes, these stakeholders need evidence on harms, process outcomes, implementation, and costs. Understanding uncertainty and sources of uncertainty in public health evidence is also important for stakeholders. Public health policymakers tend to tolerate uncertainty and may even act when evidence is uncertain. The precautionary principle implies that there is a social responsibility to protect the public from exposure to harm when scientific investigation has found a plausible, though uncertain, risk. Thus, public health policymakers may be more likely to act to protect someone from an exposure even when harm is uncertain than they would to treat someone with a medicine when efficacy of an intervention is uncertain. Public health stakeholders are also interested in not only whether an intervention works but how it works, in what contexts, and why. Qualitative research can inform these implementation considerations.

To meet the needs of policymakers and fully use the types of evidence needed for policy decisions, systematic reviews must diversify their methods beyond synthesis of randomized controlled trials. Cochrane is a global organization whose mission is to promote evidence-informed health decision making by producing high-quality, relevant, accessible systematic reviews and other synthesized research evidence. Cochrane systematic reviews, published in the Cochrane Library, are not funded by commercial sponsors or created by groups with conflicts of interest. The Cochrane Public Health and Health Systems Network (https://publichealth.cochrane.org) consists of six review groups:

1. Tobacco,

2. Infectious Diseases,

3. Effective Practice and Organization of Care,

4. Public Health,

5. Consumers and Communication, and

6. Work.

For decades, these review groups have been grappling with methodological issues arising from the need to frame questions that are relevant to public health and synthesize the best evidence to answer the questions. The Cochrane Public Health and Health Systems Network review groups have often developed methodological guidance to supplement the methods in the Cochrane Handbook for Systematic Reviews of Interventions. Although sections of the recently updated handbook have incorporated new material, such as dealing with complex interventions and incorporating patient experience of adverse events, the focus has remained on synthesis of randomized controlled trials of clinical interventions.

A forthcoming article in this methods series presents four case studies from the Cochrane Public Health and Health Systems Network to illustrate different methods to involve stakeholders in evidence syntheses. The cases present methods from the Consumers and Communication, Effective Practice and Organization of Care, and Public Health review groups. These examples cover the three key stages of the review process: topic prioritization, review production, and knowledge translation. Using and evaluating these stakeholder engagement methods will help systematic reviewers understand how the exchange between different bodies of knowledge and experience contribute to policy and practice. An editorial in the series (Lansky and Wethington, p. 1687) describes how Cochrane is using living systematic review methods to more efficiently update reviews on public health topics, such as interventions to increase children’s fruit and vegetable intake.