Skip to main content
PMC home page
American Journal of Public Health logoLink to American Journal of Public Health
. 2017 Jan;107(1):88–92. doi: 10.2105/AJPH.2016.303485

Evidence and Health Policy: Using and Regulating Systematic Reviews

Daniel M Fox 1,✉
PMCID: PMC5308142  PMID: 27854522

Abstract

Systematic reviews have, increasingly, informed policy for almost 3 decades. In many countries, systematic reviews have informed policy for public and population health, paying for health care, increasing the quality and efficiency of interventions, and improving the effectiveness of health sector professionals and the organizations in which they work. Systematic reviews also inform other policy areas: criminal justice, education, social welfare, and the regulation of toxins in the environment.

Although the production and use of systematic reviews has steadily increased, many clinicians, public health officials, representatives of commercial organizations, and, consequently, policymakers who are responsive to them, have been reluctant to use these reviews to inform policy; others have actively opposed using them.

Systematic reviews could inform policy more effectively with changes to current practices and the assumptions that sustain these practices—assumptions made by researchers and the organizations that employ them, by public and private funders of systematic reviews, and by organizations that finance, set priorities and standards for, and publish them.

Conversations about using systematic reviews to inform policy for health care and public health have been occurring among researchers, health professionals, and policymakers in many countries for almost 3 decades. These conversations began between 1990 and 1992 as a consequence of widespread attention to the implications of 4 closely linked publications: The Oxford Database of Perinatal Trials1; Effective Care in Pregnancy and Childbirth2; A Guide to Effective Care in Pregnancy and Childbirth3; and Effective Care of the Newborn Infant.4 The authors of these publications applied the rapidly evolving methodology of systematic reviews and linkage of people producing them to a broad area of medicine and public health. By 1993, policymakers in Britain were funding an increasing number of systematic reviews as well as work to establish an international organization, soon called the Cochrane Collaboration, which would set standards for systematic reviews and publish them regularly in an electronic journal, which became the Cochrane Library. Policymakers in Australia, Canada, and the United States had also begun to fund the production and publication of a growing number of systematic reviews.5

I survey the history of the use of systematic reviews in making and implementing policy. My narrative and the evidence in which it is grounded augment an editorial that Iain Chalmers and I recently published in the American Journal of Public Health, “Increasing the incidence and influence of systematic reviews on health policy and practice.”6

THE PROLIFERATION OF SYSTEMATIC REVIEWS

Chalmers and I began our editorial by documenting the proliferation of systematic reviews. In the late 1980s, journals of the health sector published between 80 and 90 systematic reviews annually.7 Two decades later, approximately 2500 systematic reviews that met widely accepted international standards were published in the international literature each year. In 2015, the researchers who had documented this number reported that “more than 8000 systematic reviews” that meet international methodological standards are now published annually in the literature of the health sector (personal communication, D. Moher, e-mail, February 19, 2015).

This proliferation has been a result of work by researchers and their allies, who frequently include policymakers responsible for financing research and regulating the health sector. Participants in the Cochrane Collaboration now represent more than 120 countries.8 The Campbell Collaboration, which has similar global reach, has, since 1999, set standards for and published systematic reviews in the fields of criminal justice, education, international development, and social welfare.9

In 2007, the Navigation Guide project began to adapt the methodology standardized and promoted by the Cochrane and Campbell collaborations to systematic reviews that address issues in environmental health. Authors associated with the Navigation Guide published the first systematic reviews pertinent to policy for assessing the risks to health of chemicals in the environment in 2014.10

SYSTEMATIC REVIEWS AND POLICY

Few publications document the influence of systematic reviews on policy and, as a result, on practice. Most publications about systematic reviews and policy are evidence-based advocacy. Their authors evaluate and recommend methodology for communicating research results to policymakers, exhort policymakers and researchers to improve such communication, or report on interviews with policymakers and members of their staff about how they set priorities for funding systematic reviews and how they use them in their work.

The authors of these publications generally reach optimistic conclusions. In 2016, for example, Greenhalgh et al. concluded, “Health research in both the United States and the United Kingdom involves increasingly complex intersectoral networks in which university scientists engage with policymakers, civil society, and industry to a far greater extent than in the past.”11(p397)

There is, however, recent evidence of diminished interest among some researchers in advancing the influence of systematic reviews on policy. The editors of the third edition of Systematic Reviews on Health Care: Meta-Analysis in Context, for instance, initially decided to replace the chapter by Sir Muir Gray, “Using systematic reviews for evidence-based policymaking,” with a 150-word summary of the subject. After the author invited to supply these words (full disclosure: the author of this article) negotiated an upgrade of this derisory invitation and submitted an article of several thousand words, the editors decided not to include a chapter on the influence of systematic reviews on policy.

Another episode of ambivalence toward systematic reviews began in 2010, when an editorial board of distinguished academics issued a call for submissions to a new journal titled Public Health Reviews.12 The 9 issues of the journal published through mid-2016 included no articles linking systematic reviews to policy for public health. Similarly, most of the chapters of a book published in 2015, A Systematic Review of Key Issues in Public Health, accord much attention to research in field epidemiology but do not describe actual and potential uses of systematic reviews to inform policy.13

A final example: In 2015, the King’s Fund, a British philanthropy, published a report about an important issue for that nation’s National Health Service: Better Value in the NHS: The Role of Changes in Clinical Practice. The authors describe and suggest ways to address the “overuse, misuse, and underuse” of interventions, “unwarranted variation” in interventions clinicians recommend to patients, and what can be learned from “high performing organizations.”14 They do not, however, link improving the quality of care to policy for financing and producing rigorous research on primary data that could be the basis of systematic reviews that could inform policy and practice.

MIXED EVIDENCE

Despite these examples of the current reluctance of some academics to embrace and prioritize the use of systematic reviews to inform policy, a growing literature seems to justify the optimism of advocates for their relevance. Attention to systematic reviews among policymakers and members of the public has increased as they have proliferated. Policymakers from numerous countries have participated in meetings, both off-the-record discussions and formal conferences, in which they have learned about the methods, uses, and potential uses of systematic reviews and about evolving standards for both systematic reviews and the research on primary data that underlie them. Leading media outlets, print and electronic, frequently report the findings of systematic reviews and their implications for policy.15

A well-documented example of media attention was the considerable media coverage, in 2004 and 2005, of a systematic review that compared the effectiveness of drugs to relieve pain and of policymakers’ positive response to it. The systematic review had been produced by the Center for Evidence-based Policy, a unit of the Oregon Health and Science University that is governed and financed by policymakers from a consortium of American states.

This systematic review described the adverse effects of Vioxx, a painkiller patented and sold by Merck, a multinational manufacturer of pharmaceutical drugs. Its findings caused many public and private insurers to exclude the drug from coverage and stimulated successful lawsuits against Merck on behalf of consumers. For perhaps the first time, moreover, a systematic review had been the basis of a story on the front page of a major American newspaper, the Wall Street Journal. As a result of the Vioxx story—and others like it—many journalists and their editors no longer considered it necessary to define systematic reviews at the beginning of each story about the significance of findings from a review for policy and practice.

Organizations in the US federal government have been using systematic reviews to inform policy since the mid-1990s. The US Preventive Services Task Force, administered by the Agency for Healthcare Research and Policy, has made many recommendations—often controversial—about policy and clinical practice as a result of relying on and sometimes commissioning systematic reviews. The Community Preventive Services Task Force, a program of the Centers for Disease Control and Prevention, has, since its inception in 1996, commissioned and published several hundred systematic reviews of considerable relevance for policy. This task force, collaborating with the Centers for Disease Control and Prevention Law Program, evaluated 65 systematic reviews of the effectiveness of 52 public health laws for an article in the American Journal of Public Health in 2009. The authors (again full disclosure: they included the author of this article) found that “many . . . public health laws have beneficial impacts . . . when [judged by] widely accepted, rigorous standards of scientific evidence.”16(p23)

Despite these examples of the significance of systematic reviews for policy, associations that represent some physicians and most manufacturers of prescription drugs and medical devices have continued to impede the use of systematic reviews by policymakers. Associations of physicians who have clinical research, and financial interests in particular interventions and care processes have frequently attacked findings of systematic reviews. These critics have often relabeled what proponents of systematic reviews call “evidence-based medicine,” “evidence-based health research,” or “evidence-informed policy,” as “cook-book medicine,” “disregard of clinical judgment,” and “interference with clinical autonomy.”

Manufacturers of prescription drugs, acting through trade associations, have been the major critics of systematic reviews that threaten corporate revenue because they inform policy. These commercial organizations have, for instance, commissioned researchers to write articles criticizing particular systematic reviews and the methodology in general; financed challenges to findings from systematic reviews by organizations that advocate for patients with particular diseases; and linked the promise or withdrawal of contributions to legislators’ campaigns to their positions on policy for coverage of particular drugs and devices that have been assessed in systematic reviews.

An early demonstration of pharmaceutical companies’ efforts to thwart the use of systematic reviews to inform policy occurred in 2000. Policymakers and researchers from 6 countries had met to discuss case studies they had written about how systematic reviews had informed policy (full disclosure: the conveners were the chair of the Cochrane Collaboration and this author). The countries were Australia, Canada, Norway, South Africa, the United Kingdom, and the United States. Policymakers from the United Kingdom had included among the authors of the case study, and invited to the meeting, an employee of an international manufacturer of pharmaceutical drugs.

Several weeks after the meeting, associations of pharmaceutical manufacturers in 4 of the countries publically criticized the quality of the scientific evidence in the case studies. Their criticism persuaded the agency that regulates the introduction of new drugs in Australia to fire the pharmacologist who was an author of the case study (he was subsequently reinstated by court order and now directs a major research center in Canada). Policymakers in Canada, Norway, and the United Kingdom defended the use of systematic reviews to inform policy.

The only countries in which the drug industry did not attack the case study were South Africa and the United States. The industry did not attack South Africa because the case addressed a drug for HIV/AIDS, the prescribing of which the president of that country had prohibited.

The authors of the US study hypothesized that they were spared because of drug industry reluctance to risk retaliation from their employer, Kaiser Permanente. An integrated delivery system serving more than 10 million people, Kaiser Permanente spends billions of dollars a year on pharmaceutical drugs through a centralized purchasing system that applies the best available evidence from research.15

A recent example of the persistence of the pharmaceutical industry in impeding the influence on policy of systematic reviews and the studies of primary data that underlie them was a controversy in the United States over the drug flibanserin (also called Addyi and the “female libido pill”). The US Food and Drug Administration (FDA) approved the drug in 2015 after having twice declined to do so. Approval occurred following an advocacy campaign that its manufacturer named “Even the Score” (because the FDA had approved 26 drugs to treat male sexual dysfunction but none as yet for female dysfunction). The campaign succeeded despite the publication of peer-reviewed studies of primary data critical of the drug, as well as the publication, soon after approval by the FDA, of a systematic review that concluded that women who used it had “on average only one-half [an] additional satisfying sexual event per month [but experienced] clinically significant . . . risk of dizziness, somnolence, nausea, and fatigue.”17(p63–64)

OTHER IMPEDIMENTS

Another impediment to growth in the influence of systematic reviews on policy is that policymakers are more often consumers of published systematic reviews than participants in prioritizing their production. Early in 2015, for example, the Cochrane Collaboration published an extensive list of priorities compiled by the groups within it that conduct reviews of interventions for particular diseases or areas of health practice.18 The explanation accompanying the list did not describe any involvement of policymakers in compiling it.

There are, however, notable examples of the successful inclusion of policymakers, clinicians, and even patients in setting priorities and key questions for systematic reviews. These include the work of the James Lind Alliance in the United Kingdom and the Center for Evidence-Based Policy in the United States.

Moreover, policymakers and even prominent academics have, at times, impeded the use of systematic reviews to inform policy. Despite the achievements of the Community Preventive Services Task Force, a senior policymaker at the Centers for Disease Control and Prevention told this author that systematic reviews are “merely research summarization.” A Canadian who served as both a public official and a senior academic in public health told the authors of case studies on the use of systematic reviews in making policy for population health that, “Overreliance on systematic reviews or unquestioning use of them might stifle creativity and innovation or lead to useful programs being sidelined because of their inadequate evidence base.”19(p1)

Officials also impede the use of systematic reviews as a result of what could be called—euphemistically—bureaucratic culture. A US federal agency recently convened a meeting to discuss using systematic reviews and cost–benefit analyses to inform regulatory policy for a significant area of public health. Perhaps in response to queries about political impediments to the use of systematic reviews in making regulatory policy from the author of this article, the officials belatedly declared that discussion at the meeting was “not for attribution.”

In a subsequent private conversation with this author, one of them claimed that he and his colleagues are prohibited by law from considering, or even talking about, any effects of politics on developing and implementing regulations—whether inter- and intragovernmental, electoral, interest and advocacy group, or scientific politics. In frustrated disbelief, this author privately asked another participant in the meeting, the chief regulator for these issues in the European Union, “How much of your time do you spending analyzing and conducting politics?” He replied, “About eighty percent.” Analogously, the American officials and the economists, mostly academics, they had invited to the meeting also dismissed suggestions that surveys of consumer preferences were not the only methodology used to estimate the cost of proposed regulations by contributors to the peer-reviewed international research literature.

A methodological obstacle to the use of systematic reviews by policymakers is the absence from most of them of relevant findings from research in disciplines of the policy sciences (with the partial exception of the economics of cost effectiveness). Research in these disciplines—notably anthropology, behavioral economics, history, law, political science, and sociology—has frequently documented the effects of bureaucratic and electoral politics on the allocation of scarce resources for interventions that systematic reviews have demonstrated could contribute to the health of individuals and populations. A recent example of the potential relevance of research in the policy sciences is a systematic review of the effectiveness of interventions its authors call “advocacy for health equity.”20

Even publications about the uses of systematic reviews in policymaking accord scant attention to findings from research in the policy sciences.21–24 A partial explanation for this inattention may be the absence of criteria, devised by leading producers of systematic reviews, for including and excluding findings from research in these disciplines.

Trisha Greenhalgh and her colleagues are a significant exception to this generalization. They have repeatedly emphasized the uses of findings from history and other social science disciplines for systematic reviews about efforts to change the organization of health services.25,26

Researchers, their sponsors, and journal editors also inhibit the use of systematic reviews to inform policy by lax enforcement of standards for methodology and ethics. For instance, a growing number of researchers are appropriating—and editors are accepting—the label “systematic review” for titles and subtitles of articles that do not adhere to international standards and, as a result, cannot be fully trusted by persons who make and implement policy. The authors of such articles typically search relevant literature using methods specified in internationally accepted guides for producing systematic reviews. Then they often fail to apply accepted methods for identifying systematic bias in primary studies before deciding which of them to include and which to exclude from a systematic review.27,28

John Ioannidis reached a dismaying conclusion about this failure to implement international standards for methodology and scientific ethics in an article about the proliferation of systematic reviews published in 2016. He argues that the “mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses” has “reached epidemic proportions.” He documents, moreover, that many authors of these systematic reviews and the meta-analyses within them have “industry conflicts” that lead them to “results [that] are aligned with sponsor interests.” As a result, the “large majority” of these publications are “unnecessary, misleading or conflicted.”29(p486)

Ioannidis concludes that this situation can be remedied only by new policy to enforce international standards for methodology and ethical behavior. In a commentary accompanying this article, Page and Moher recommend that “to succeed,” the policy Ioannidis proposes, “require[s] enhanced collaboration among methodologists, clinical researchers, academic institutions, funding bodies, industry journals, and publishers.”30(p518)

CONCLUSIONS

Although systematic reviews have, for a quarter of a century, increasingly informed policy as well as practice in health care and public health, there are many impediments to the growth, and even to the continuation, of their influence. On the one hand, the number of policymakers around the world who understand the methods and appreciate the potential uses of systematic reviews has steadily increased. The proliferation of systematic reviews and policymakers’ growing trust in their findings have reduced the use of relatively ineffective interventions and promoted the use of what, in 1989, the authors of the seminal volumes Effective Care in Pregnancy and Childbirth modestly called “forms of care that reduce negative outcomes.”4

On the other hand, the influence of systematic reviews on policy and its implementation varies widely among countries and their subjurisdictions. This variation is a result of the political culture of each country and how that culture shapes the policies and practices of institutions that educate and certify health professionals, those that regulate health care and public health services, and those that fund and publish research.

Whether and how systematic reviews inform policy in the future will be most strongly influenced by how, in each country, public policymakers and organizations in the health sector address 2 clusters of issues. The first of these clusters, as Chalmers and I argued in our editorial in the American Journal of Public Health,6 includes policy for funding research; publishing research results; paying for health services and public health interventions; educating, licensing, and certifying health professions; and regulating their behavior.

The second cluster includes policy in response to challenges to the scientific integrity and practical credibility of systematic reviews. Ioannidis has identified pressing threats to the integrity and credibility of systematic reviews.29 Another challenge is expanding the methodology for conducting systematic reviews to incorporate findings from relevant research in the disciplines of the policy sciences.

Making policy to address issues in each cluster will require the mobilization of considerable political will among policymakers and researchers. Relevant policymakers work in government, universities, independent research organizations, journal publishing, and associations that represent health professionals, educational institutions, provider organizations, and patients, and for commercial interests in the health sector.

These policymakers must participate vigorously in setting priorities for the questions systematic reviews address and in establishing and enforcing the ethical and methodological standards they must meet. Similarly, researchers and their associations must collaborate with policymakers in setting priorities and standards for conducting systematic reviews. Leaders in research must also insist that they and their colleagues adhere to high standards of ethics and methodology and must police—not too strong a word—this adherence.

Footnotes

See also Bero, p. 93, and Greenhalgh and Malterud, p. 97.

REFERENCES

  • 1.Chalmers I, editor. The Oxford Database of Perinatal Trials. Oxford, UK: Oxford University Press; 1999–1992. [Google Scholar]
  • 2.Chalmers I, Enkin M, Keirse M, editors. Effective Care in Pregnancy and Childbirth. Oxford, UK: Oxford University Press; 1989. [Google Scholar]
  • 3.Enkin M, Keirse M, Chalmers I, editors. A Guide to Effective Care in Pregnancy and Childbirth. Oxford, UK: Oxford University Press; 1989. [Google Scholar]
  • 4.Sinclair C, Bracken MB, editors. Effective Care of the Newborn Infant. Oxford, UK: Oxford University Press; 1992. [Google Scholar]
  • 5.Fox DM. Systematic reviews and health policy: the influence of a project on perinatal care since 1988. Milbank Q. 2011;89(3):425–449. doi: 10.1111/j.1468-0009.2011.00635.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Chalmers I, Fox DM. Increasing the incidence and influence of systematic reviews on health policy and practice. Am J Public Health. 2016;106(1):11–13. doi: 10.2105/AJPH.2015.302915. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007;4(3):e78. doi: 10.1371/journal.pmed.0040078. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Cochrane Library. Available at: http://www.cochranelibrary.com. Accessed September 28, 2016.
  • 9.Campbell Collaboration. Available at: http://www.campbellcollaboration.org. Accessed September 28, 2016.
  • 10.Fox DM, Bero L. Systematic reviews: perhaps “the answer to policy makers’ prayers”? Environ Health Perspect. 2014;122(10):A262–A263. doi: 10.1289/ehp.1408599. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Greenhalgh T, Jackson C, Shaw S, Janamian T. Achieving research impact through co-creation in community-based health services: literature review and case study. Milbank Q. 2016;94(2):392–429. doi: 10.1111/1468-0009.12197. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Flahault A, Fried L. Why a new journal called Public Health Reviews? Public Health Rev. 2010;32:15–17. [Google Scholar]
  • 13.Boccia S, Vilari P, Ricciardi W, editors. A Systematic Review of Key Issues in Public Health. New York, NY: Springer; 2015. [Google Scholar]
  • 14.Adderwich H, Robertson R, Appleby J, Dunn P, Maguire D. Better Value in the NHS: The Role of Changes in Clinical Practice. London, UK: King’s Fund; 2015. [Google Scholar]
  • 15.Fox DM. The Convergence of Science and Governance: Research, Health Policy and American States. Berkeley, CA: University of California Press; 2010. [Google Scholar]
  • 16.Moulton AD, Mercer S, Popovic T et al. The scientific basis for law as a public health tool. Am J Public Health. 2009;99(1):17–24. doi: 10.2105/AJPH.2007.130278. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Jaspers L, Feys F, Bramer WM, Franco OH, Leusink P, Laan ET. Efficacy and safety of flibanserin for the treatment of hypoactive sexual desire disorder in women: a systematic review and meta-analysis. JAMA Intern Med. 2016;176(4):453–462. doi: 10.1001/jamainternmed.2015.8565. [DOI] [PubMed] [Google Scholar]
  • 18.Cochrane Editorial Unit. Cochrane priority review list for 2015-16. Available at: http://editorial-unit.cochrane.org/cochrane-priority-reviews-list-2015-16. Accessed September 28, 2016.
  • 19.Sweet M, Moynihan R. Improving Population Health: The Uses of Systematic Reviews. New York, NY: Milbank Memorial Fund; 2007. [Google Scholar]
  • 20.Farrer L, Marinetti C, Cavaco YK, Costongs C. Advocacy for health equity: a synthesis review. Milbank Q. 2015;93(2):392–437. doi: 10.1111/1468-0009.12112. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Orton L, Lloyd-Williams F, Taylor-Robinson D, O’Flaherty M, Campbell S. The use of research evidence in public health decision making processes: systematic review. PLOS One. 2011;6(7):e21704. doi: 10.1371/journal.pone.0021704. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Liverani M, Hawkins B, Parkhurst JO. Political and institutional influences on the use of evidence in public health policy: a systematic review. PLOS One. 2013;8(10):e77404. doi: 10.1371/journal.pone.0077404. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Oliver K, Innvar S, Lorenc T, Woodman J, Thomas J. A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Serv Res. 2014;14:2. doi: 10.1186/1472-6963-14-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Moat KA, Lavis JN, Abelson J. How contexts and issues influence the use of policy-relevant research syntheses: a critical interpretive synthesis. Milbank Q. 2013;91(3):604–648. doi: 10.1111/1468-0009.12026. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Best A, Greenhalgh T, Lewis S, Saul JE, Carroll S, Bitz J. Large-system transformation in health care: a realist review. Milbank Q. 2012;90(3):421–456. doi: 10.1111/j.1468-0009.2012.00670.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Fox DM. History matters for understanding knowledge exchange. Milbank Q. 2010;88(4):484–491. doi: 10.1111/j.1468-0009.2010.00609.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Newhouse RP, Stanikhutt J, White KM et al. Advanced nurse practice outcomes, 1990–2008: a systematic review. Nurs Econ. 2011;29(5):230–250. [PubMed] [Google Scholar]
  • 28.Jennings L. Do men need empowering too? A systematic review of entrepreneurial education and microenterprise development on health disparities among inner-city Black youth. J Urban Health. 2014;91(5):836–850. doi: 10.1007/s11524-014-9898-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Ioannidis JP. The mass production of redundant, misleading and conflicted systematic reviews and meta-analyses. Milbank Q. 2016;94(3):485–514. doi: 10.1111/1468-0009.12210. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Page MJ, Moher D. Mass production of systematic reviews and meta-analyses: an exercise in mega-silliness? Milbank Q. 2016;94(3):515–519. doi: 10.1111/1468-0009.12211. [DOI] [PMC free article] [PubMed] [Google Scholar]

Articles from American Journal of Public Health are provided here courtesy of American Public Health Association

RESOURCES