Systematic Review: A Method at Risk for Being Corrupted

Abstract

The production of systematic reviews is increasing, but their credibility is under threat.

Although systematic reviews are an important tool for policymaking, their influence can be weakened by methodological problems and poor policy relevance.

Using Cochrane as an example, I address standards for systematic reviews, the influence of special interests on these reviews, and ways to increase their relevance for policymakers.

In his essay “Evidence and Health Policy: Using and Regulating Systematic Reviews,” Dan Fox tells a number of compelling stories to illustrate his point that policymakers can be reluctant or even actively opposed to using systematic reviews to inform health policy.¹ Fox does not admonish policymakers. Instead, he calls on researchers to take responsibility for the limited influence of systematic reviews on health policy. He proposes a number of recommendations to improve the scientific integrity and credibility of systematic reviews. Here I expand on his suggestions for improving standards for systematic reviews and meeting the needs of policymakers, using Cochrane as an example of progress that has been achieved.

STANDARDS FOR SYSTEMATIC REVIEWS

The production of systematic reviews has skyrocketed in the past two decades.² This increase in number has been accompanied by a growing “looseness” in the definition of systematic review. The author's definition is as follows: a systematic review attempts to identify, appraise, and synthesize all of the empirical evidence that meets prespecified eligibility criteria to answer a given research question. However, as regulatory requirements to use systematic reviews to support everything from food claims to environmental risk assessments have increased, special interest groups have produced “systematic” reviews that may meet regulatory requirements but do not meet methodological requirements. The “systematic review” label is at risk for being corrupted because researchers are appropriating the term without using the required systematic approach. Many journals are accepting reviews that are not systematic because publishers know that reviews are highly cited.

Cochrane has multiple standards that must be met for systematic reviews published in the Cochrane Library. The Cochrane Handbook for Systematic Reviews of Interventions provides step-by-step guidance.³ Our Methodological Expectations of Cochrane Intervention Reviews (MECIR) are detailed standards for the reporting and conduct of a Cochrane review. The MECIR standards are integrated into our Review Manager software so that an author can see what standards should be met for each section of the review. Cochrane also provides training to help our authors meet these methodological standards.

Cochrane reviews consistently rate as having higher quality than other systematic reviews.⁴ Cochrane has been criticized for publishing only a small proportion of systematic reviews,² but the Cochrane Steering Group has identified quality, as opposed to quantity, as one of its highest priorities. Although Cochrane is the leader in enforcing standards for systematic reviews of interventions, we are working to expand our methodological standards to include other types of systematic reviews such as diagnostic and prognostic reviews. One of our key objectives to meet our Strategy 2020 goal of producing evidence is continuing to develop and implement comprehensive quality assurance mechanisms for editorial and methodological standards.

In areas such as environmental health, the production of systematic reviews is booming and the standards are variable.⁵ A leading journal in the field, Environment International, has appointed an associate editor for systematic reviews and enforces the Preferred Reporting Items for Systematic Reviews (PRISMA) guidelines.⁶ Complete and accurate reporting is a necessary first step to being able to assess the methodological rigor of systematic reviews. Although the PRISMA reporting standards were introduced in 2009, many journals do not require them for systematic reviews.⁴ These standards should be enforced across all journals, as Environment International suggests.

INDUSTRY CRITICISM OF SYSTEMATIC REVIEW METHODS

Manufacturers of prescription drugs and medical devices have been critical of systematic reviews and attempt to impede their use in policy formulation.¹ In an analysis published in 2014, my coauthors and I evaluated articles that expressed opinions about the use of systematic reviews for policymaking, such as editorials and commentaries.⁷ Articles that were critical of the use of systematic reviews in policymaking were about six times more likely to have a disclosed industry tie than supportive articles; six percent of supportive articles and 40% of critical articles disclosed industry ties. As a result of the high level of nondisclosure of financial ties in these types of articles, we also searched additional databases to identify authors’ undisclosed financial ties. When we included undisclosed industry ties, critical articles had industry ties more than twice as often as supportive articles (80% vs 35%).

This analysis suggests that when policymakers or others are presented with reasons why systematic reviews are not useful for policy formulation, they should carefully consider the source. These criticisms could be part of an industry strategy to distract the public from the use of the best available evidence in health service use and pricing decisions. Our results further suggest the need for more consistent and complete conflict of interest disclosures for all article types, including opinion pieces.

INDUSTRY-SPONSORED REVIEWS AND FAVORITISM

Across a variety of health-related fields, there is also substantial evidence that industry sponsorship of systematic reviews is associated with the results and conclusions of reviews favoring the industry’s product or position.^8–10 Although strict adherence to methodological standards should narrow the gap between differences in the outcomes of non–industry-sponsored and industry-sponsored reviews, policymakers should remain skeptical about the findings of industry-sponsored reviews. Cochrane reviews are known for their independence from commercial sponsorship. According to our commercial sponsorship policy, “Cochrane reviews cannot be funded or conducted by commercial sponsors or commercial sources with a real or potential vested interest in the findings of a specific review.”¹¹

Although some journals will not publish research articles funded by the tobacco industry or educational articles funded by the pharmaceutical industry,^12,13 to our knowledge, the Cochrane Library is the only journal that prohibits the funding of reviews by commercial sponsors. Other journals should consider only publishing systematic reviews that are not funded by sponsors with a financial interest in the outcome.

PARTNERSHIPS WITH POLICYMAKERS

As one part of our Strategy 2020 goal of advocating for evidence, Cochrane is building partnerships with a number of organizations, including policy-relevant bodies such as the World Health Organization (WHO), the Pan American Health Organization, and the Guidelines International Network. Cochrane is a nongovernmental organization in official relations with WHO, and our work plan includes supporting the developers of WHO guidelines and the WHO Essential Medicines List by providing Cochrane reviews and expertise to strengthen the evidence underpinning these global policy tools. Since 2008, the number of Cochrane reviews cited in WHO guidelines has quadrupled. Eighty-seven Cochrane reviews were cited in nine of the 12 WHO guidelines published in 2015.

The WHO Essential Medicines List plays a significant role in medicine policy because it can be used by countries to prioritize purchasing decisions and national health insurance coverage or to advocate for lower prices for medicines that meet public health needs. Cochrane reviews play an increasing role in applications for including medicines in the Essential Medicines List. In total, 177 reviews from 40 Cochrane review groups were used to inform the nine reports of the WHO Expert Committee on the Selection and Use of Essential Medicines published between 2000 and 2015. Citation of Cochrane reviews has increased steadily over time, from two reviews cited in the 2000 report to 41 in the 2015 report.

Cochrane reviews are also often used as evidence of comparative efficacy and harm. This is particularly important when the Expert Committee on the Selection and Use of Essential Medicines reviews applications for drugs that can be used for the same indication but have very different prices. As an example, a Cochrane review of head-to-head comparisons of two drugs for treating age-related macular degeneration (AMD)—bevacizumab and ranibizumab—provided crucial evidence to help the committee decide which drug to list.¹⁴

Bevacizumab has been used as an “off-label” treatment for AMD; it is approved in most countries as a treatment for colorectal cancer. Ranibizumab was tested in randomized controlled trials and has been registered as an AMD treatment since 2006. Comparative studies have shown that the drugs are equally effective, but regulators questioned the safety of bevacizumab. Importantly, ranibizumab costs about 40 times more than bevacizumab. The Cochrane review showed no differences in safety between the two drugs. WHO included off-label use of bevacizumab for treatment of AMD in its Essential Medicines List in April 2013. Medicine purchasing decisions around the world have been influenced by this Cochrane review, which was conducted rapidly to support regulatory bodies and health care payers in their decisions.¹⁵

PRIORITIZING SYSTEMATIC REVIEWS

Another impediment to use of systematic reviews is that policymakers are not invited to participate in prioritizing the production of these reviews. In 2007, Cochrane funded a number of small projects that investigated different methods for prioritizing reviews.¹⁶ The projects involved a variety of stakeholders, including policymakers, clinicians, and patients. We learned that a number of different approaches to priority setting, including theoretical frameworks, consensus development, and mapping questions to existing evidence, could all be used to identify priority review topics. The projects also highlighted that it was sometimes difficult for different stakeholders to reach a consensus on the most important questions reviews should address.

Cochrane’s current list of priority reviews is partially informed by published lists of research priorities from funders, governments, and policymakers. Many of the justifications provided for priority reviews come from patient-oriented organizations such as the James Lind Alliance rather than from direct engagement with policymakers. However, there are direct examples of productive engagement with policymakers in determining the priority list. Cochrane strives to engage with a variety of stakeholders to help ensure the relevance of our reviews for community groups, as well as policymakers, in high-, low-, and middle-income countries. We aim to develop a shorter, more focused priority review list by 2017 and to make the process for developing the list more transparent. A total of 122 high-priority reviews have been published since the list was introduced in January 2015, and we will monitor the impact of these reviews.

REALIST REVIEWS

Systematic reviews of research from the policy sciences are rare, but “realist reviews” incorporate a theory-based approach to analysis and can be more applicable than quantitative methods for synthesizing policy research.¹⁷ Publication of realist reviews in health services and policy has been increasing. These reviews provide a structure for determining the characteristics of a complex policy intervention that are associated with the success (or failure) of the intervention. Realist reviews acknowledge that policy interventions may have different results depending on the context in which they are implemented, the stakeholders involved, and the resources available.

For example, we recently conducted a realist review of policy intervention studies aimed at reducing exposures to environmental hazards in the United States.¹⁸ This work was led by a political scientist, Dorie Apollonio. After a systematic search of the literature, two coders independently extracted data from the studies, assessing methods and context, the details of the interventions, outcomes, and risks of bias. Thus, the “front end” of the review was very similar to a Cochrane review. Our analysis, however, differed. We developed context–mechanism–outcome configurations for each study to draw conclusions about the circumstances in which different policy interventions were most effective. We found that regulatory interventions appeared to reduce chemical exposures.

Our analysis was hampered by the poor reporting of details on the context and proposed mechanisms of the interventions. Realist analyses must be incorporated into the structure of a rigorous systematic review with a well-defined question, a comprehensive search, and an assessment of the included studies. The RAMESES II initiative will develop reporting standards for realist reviews that can be used to ensure that these reviews meet the methodological requirements of systematic reviews as well as additional reporting requirements on details of the interventions and setting.¹⁹

DOCUMENTING THE INFLUENCE ON POLICY

Researchers who publish systematic reviews may be uncomfortable or unfamiliar with publishing case studies that document the influence of systematic reviews on policy because they are accustomed to publishing rigorous analyses of entire bodies of evidence. Cochrane is documenting case examples of working with policymakers and others to produce reviews that affect policy decisions, guidelines, clinicians, and patients. These case examples are featured on our Web site (https://www.cochrane.org), in our annual reports, and in the media.

A Cochrane review on continuity of midwife care was updated in 2016 when WHO and the United Kingdom’s Department of Health identified it as a priority review. The review revealed that women with continuity of care from a midwife had better birth outcomes and experiences than those with medical or shared care.²⁰ This review has been cited in multiple policy documents, including the Lancet Midwifery Series, informing the United Nations’ Post-2015 Development Agenda.

Cochrane is also responsive to direct requests from policymakers. For example, an Australian Law Reform Commission issue paper on elder abuse released in June 2016 asked for evidence about elder abuse in Australia and what further research was needed. Cochrane responded with a written submission, the relevant Cochrane review,²¹ an evidence summary of the review, and a podcast. The review of seven studies including 1924 elderly participants and 740 others showed that elder abuse is a critical global issue and is worse when caregivers lack training or have poor attitudes. Although some interventions appeared to improve knowledge or attitudes, the review authors concluded that further rigorous comparative evaluations of prevention strategies are needed. Cochrane not only directly addressed the questions about research needs posed by the elder abuse inquiry in Australia but contributed to a similar government inquiry in Canada.

Cochrane has also made substantial contributions to the policy discussions around obesity prevention and food consumption. The Cochrane Public Health group has been on the forefront of developing methodological approaches for conducting reviews of complex policy interventions. The group’s two reviews on preventing and treating obesity among children have been among the top-cited reviews in the Cochrane Library.^22,23 In addition, the Cochrane review “Portion, Package or Tableware Size for Changing Selection and Consumption of Food, Alcohol and Tobacco” established that people consume more food and nonalcoholic drinks when they are offered larger portion sizes or use larger tableware items.²⁴ This complex review, which included 72 randomized controlled trials testing different policy options, provides concrete advice about limiting portion and serving sizes to reduce food intake.

CONCLUSIONS

It is understandable that policymakers are hesitant to use systematic reviews that fail to meet methodological standards, have questionable credibility, and do not meet their evidence needs. Cochrane has been a leader in developing methodological standards for systematic reviews and ensuring that reviews are free of commercial influence. We are engaged with policymakers in a variety of settings and will continue to improve the relevance of our reviews for policy decisions.

As the demand for systematic reviews increases, it is crucial that researchers and journals new to systematic reviews adhere to standards as rigorous as those promulgated by Cochrane. Analytic methods that are highly suitable for policy reviews, such as realist synthesis, should also be grounded in the principles of systematic reviews. Otherwise, the reputation and usability of the entire method of systematic review will suffer.