Systematic Reviews and Health Policy: The Influence of a Project on Perinatal Care since 1988

Abstract

Context: Interrelated publications between 1988 and 1992 have influenced health policy and clinical practice: The Oxford Database of Perinatal Trials (ODPT), Effective Care in Pregnancy and Childbirth (ECPC), A Guide to Effective Care in Pregnancy and Childbirth (GECPC), and Effective Care of the Newborn Infant (ECNI). These publications applied and advanced methods that had a substantial history in the medical, biological, physical, and social sciences. Their unique contribution was to demonstrate the feasibility of organizing and sustaining programs to conduct systematic reviews across an entire field of health care. The publications also influenced subsequent advances in the methodology of systematic reviews and contributed to their proliferation; in large measure, but not entirely, because their editors and many of the authors participated in organizing and developing the Cochrane Collaboration. This article describes how and why these publications attracted favorable attention and resources from policymakers in numerous countries.

Methods: This article applies historical methods to the analysis of primary sources that help explain the influence of systematic reviews, mainly on health policy. These methods guide an analysis of the rhetoric of the two volumes of ECPC and of primary sources generated as systematic reviews influenced health policy. The analysis of rhetoric employs the methods of intellectual history and social studies of science. The analysis of policymaking uses the methods of political and policy history, political science, and public administration. Because the focus of this article is how science influenced policy it alludes to but does not describe in detail the literature on the methods, production, and publication of systematic reviews.

Findings: The influence of the four publications on policy was mainly a result of (1) their powerful blending of the rhetoric of scientific and polemical discourse, especially but not exclusively in ECPC; (2) a growing constituency for systematic reviews as a source of “evidence-based” health care among clinicians, journalists, and consumers in many countries; and (3) recognition by significant policymakers who allocate resources to and within the health sector that systematic reviews could contribute to making health care more effective and to containing the growth of costs.

Conclusions: Analysis of this aspect of the history of producing and applying systematic reviews informs understanding of how knowledge derived from research informs policy.

Systematic reviews, articles, and reports that use rigorous methods to identify relevant studies, appraise their quality, and summarize their results have proliferated in the literature of health services as well as in the literatures of criminal justice, education, social and addiction services, and, recently, international development. Four interrelated publications between 1988 and 1992 contributed to the developing methodology of systematic reviews and, because they applied this methodology to interventions across an entire field of health services, attracted attention from many clinicians and policymakers. These publications are The Oxford Database of Perinatal Trials (ODPT), Effective Care in Pregnancy and Childbirth (ECPC), A Guide to Effective Care in Pregnancy and Childbirth (GECPC), and Effective Care of the Newborn Infant (ECNI) (Chalmers 1988–1992; Chalmers, Enkin, and Keirse 1989; Enkin, Keirse, and Chalmers 1989; Sinclair and Bracken 1992). This article describes the four publications as the ECPC project, emulating the usage of Frederick Mosteller, a distinguished statistician who wrote an important early article about their significance (Mosteller 1993).

A growing literature grounds the history of systematic reviews in the history of science and medicine. Although the “need to synthesize research evidence has been recognized for well over two centuries,” explicit methods for such research emerged only in the twentieth century (Chalmers, Hedges, and Cooper 2002). In the 1970s and 1980s, “systematic reviews and meta-analyses … began to appear in a variety of health fields” (Bastian, Glasziou, and Chalmers 2010). The methodology of systematic reviewing and the research infrastructure that produced them continue to evolve (Starr et al. 2009).

Other literature describes the production of systematic reviews since the publication of the ECPC project. Most of this literature describes the history of the Cochrane Collaboration, organized in 1993 by Iain Chalmers and other participants in ECPC (Chalmers 1993, 2003; Dickersin and Manheimer 1998; Starr et al. 2009). I have described elsewhere the influence of the Cochrane Collaboration and other sources of systematic reviews on a consortium of American states that began to commission reviews in 2003 (Fox 2010a).

This article links this history of science and its sponsorship to the contemporary history of health policy. Mosteller described the ECPC project as the “most advanced current example of a basis of practicing medicine founded on both empirical evidence and theory, … and it illustrates what can be done” (Mosteller 1993, 524). I focus on another aspect of the project's impact: why and how systematic reviews have influenced policy. I also briefly examine aspects of its influence on clinical practice that relate to its effects on policy.

I organize the article in six sections. First, I discuss the rhetoric of the two volumes of ECPC, what I call its “two voices.” Next I describe initial responses to that rhetoric within the medical profession. The following two sections examine responses to the ECPC project and subsequent systematic reviews that it influenced among significant groups in the health sector: first, antagonism to these reviews and, then, support for conducting and using them. Then I explain why the ECPC project and subsequent reviews that it influenced attracted the interest of some policymakers who allocate resources for health services. Finally, I summarize what these policymakers have done to subsidize the production and encourage the application of systematic reviews since the publication of the ECPC project.

Although the focus of this article is the influence of the ECPC project on policy, several reviewers asked me to summarize some other notable effects. ODPT, one reviewer wrote, “pointed out the research foundation needed for reviews (i.e., problems with reporting biases, searching and the need for trials registers) and served as the model for Cochrane's Central Database of Controlled Trials.” GECPC was a “model for extending the reach of the work to non-doctors (e.g., midwives) and consumers.” And ECNI, another reviewer observed, “could be credited with extending the approach [to a new field of care and was] a prelude to the Cochrane Collaboration.” These achievements contributed to the influence of systematic reviews in health affairs. I will now describe the politics through which that influence occurred.

The Rhetoric of ECPC

The editors and their coauthors of many of the chapters in ECPC's two volumes wrote in two rhetorical styles, or voices, both of which have helped shape the politics of conducting and using systematic reviews. These voices enhanced the resonance of the volumes with particular audiences in the health sector, in the media, among consumers, and in government. In one of their voices, the editors and their colleagues described the methods and findings of systematic reviews in the conventional uninflected and impersonal language of biomedical science. In the other, they spoke as eloquent polemicists against authoritarianism in clinical practice and on behalf of more effective perinatal care.

When I began to write this article, I lacked evidence about the extent to which the editors consciously wrote in two voices. I knew from reading, personal observation, conversations, and correspondence that the editors were deeply committed to social justice and especially to the rights of patients. In early drafts of this article, I conjectured that they may have been stimulated by contemporaneous scholarship on the rhetoric of science, much of it by scholars who shared their commitment to social justice (Gross 2006). Then Murray Enkin, an editor of ECPC (as well as GECPC), commented on one of my drafts: “I don't think Iain [Chalmers], Marc [Keirse], and I consciously chose to use the two voices of scientific ‘objectivity’ and polemic for more patient-centered care, but when you pointed this out I wanted to shout, ‘Right on, that was the way it was.’” Oral history is the history of memory, even in email format. Nevertheless, Enkin's recollection of the editors’ motives must suffice as an explanation until additional primary sources become available.

Archie Cochrane was polemical in his foreword, as he had been many times during his distinguished scientific career. He repeated an accusation he had made in 1979, that obstetricians made the “worst” use of randomized controlled trials among medical specialists. Then he proclaimed that the systematic reviews were a “new achievement” and a “milestone” (Chalmers, Enkin, and Keirse 1989, vii).

The three editors used more conciliatory words in their preface. Chalmers, Enkin, and Keirse called attention to the variation in clinicians’ opinions about the “best means” for “achieving the objectives of care.” Such variation had multiple causes, they wrote. They emphasized the variation resulting from clinicians’ “collective uncertainty” about the “effectiveness and safety of many of the elements of care” (1989, ix–x). They apparently thought it politic to introduce the volumes with the reassuring assumption that as men and women of science, clinicians would reduce this variation when persuasive evidence of effectiveness became available. Their American contemporary, John E. Wennberg, had recently made the same assumption when he described how his studies of variation in practice within geographic areas would influence clinicians (Wennberg 1984).

The editors used their two voices in the opening paragraph of the acknowledgments. They began by according equal esteem to participants in research and to scientists: “Primary credit for this book must go to the women and investigators who participated in and conducted the controlled trials of care in pregnancy and childbirth on which so much of the evidence in this book is based.” They concluded the paragraph by thanking Archie Cochrane for both his passion and his commitment to experimental methods. Cochrane, they wrote, “first drew our attention to the particular importance” of these investigators and the women they studied for an “understanding of the effects of care during pregnancy and childbirth” (1989, xi).

Next the editors listed the contributors to ECPC and their colleagues who had provided information about unpublished trials (xi–xiii). In the context created by Cochrane's provocative rhetoric, the conciliatory opening paragraph of the preface, and the two voices of the previous paragraph, acknowledging these colleagues was political as well as conventional. The list was evidence that the potential constituents of ECPC, and of systematic reviews more generally, were numerous and international.

The table of contents introduced the unconventional scope and broad evidentiary base of the volumes, the first on pregnancy and the second on childbirth. Methodology was the subject of the first five chapters of volume 1. Three of these chapters described randomized controlled trials (RCTs) and other study designs, as well as the methodology of systematic reviews. The fourth chapter discussed relevant methods in the social sciences, and the fifth, economic evaluation. The next eight chapters addressed the “social context of care during pregnancy and childbirth.” The chapters that followed discussed the “needs of child-bearing families,” “social policies and the organization of health care,” “social and psychological support during pregnancy,” and “advice for pregnancy.” Not until chapters 14 through 47 of the volume would clinicians find discussion of evidence about the effectiveness of the interventions that comprised most of their work with pregnant women and newborn children (xv–xvi). Moreover, the editors placed the ECPC's only chapter on biology at the beginning of the second volume, perhaps signaling that unlike most of their peers, they did not regard the biological sciences as the only, or necessarily the most important, disciplines underlying clinical practice.

After the chapter on biology, however, volume 2 resembled its predecessor. Three chapters addressed, respectively, “social and professional support during childbirth,” “hospital policies for labor and delivery,” and “hospital admission practices.” The next thirty-two chapters discussed aspects of care during labor and delivery and care of mothers and newborn infants, followed by chapters on “care of the bereaved after perinatal death” and “unhappiness after childbirth.” Two of three concluding chapters examined strategy and tactics for an international movement to promote effective care during pregnancy and childbirth, including possible actions by policymakers outside the health sector. In the last chapter the editors restated the book's thesis—that reliable evidence, synthesized in rigorous systematic reviews, could be used to reduce unwarranted variation—and then summarized ways in which such reviews could inform policy and practice (xvii–xviii).

The editors challenged conventional assumptions within medicine at key points in the volumes. At the end of the first chapter, for example, Chalmers emphasized the antipathy of the ECPC project to assertions of clinical authority that were not informed by experimental evidence: “A characteristic of a well-designed enquiry into the effects of clinical practice,” he wrote, “is that it is basically anti-authoritarian.” Then he identified the “variety of strategies” persons who derived authority from power rather than knowledge could use to “discredit studies that yield results that challenge their certainties.” He concluded the paragraph with an example of the transformative power of systematic reviews: the “evidence used to prepare this book” had “helped to shatter … fondly cherished certainties” of its editors and authors (31).

The editors and two other authors (Jini Hetherington and Diana Elbourne) began the second chapter of volume 1 by criticizing conventional standards for evaluating evidence regarding the effectiveness of interventions. “Some scientists,” they wrote, “have a remarkably casual attitude to the process of synthesizing research evidence.” This attitude often led them to disguise their bias and self-interest in the rhetoric of science. Such researchers draw “conclusions … based more on factors such as their gender and how they make a living then on the available evidence.” Then the editors and their colleagues accused many medical scientists of neglecting three decades of work on research synthesis by statisticians and investigators in education and psychology (39–42). Now, however, they switched to their scientific voice for a detailed summary of the methodology of systematic reviews. But they began the concluding section with an ironic quotation from Charlie Brown of Peanuts: “I am always certain about things that are a matter of opinion.” Chalmers recently recalled that this chapter was a “very rare example of a medical textbook with a Materials and Methods section” (Chalmers, email, December 24, 2010). It was also a rare example of a textbook with two voices.

The editors began the concluding chapter of the second volume by restating their “underlying thesis” in language that was less confrontational than what they had used in the first chapters of volume 1. “Evidence from well-controlled comparisons,” they wrote, “provides the best basis for choosing among alternative forms of care.” Still speaking in the voice of science, they reiterated points they had made in the first volume about the relevance of their work to policy and practice. Research “based on the study of groups” was “relevant to guiding … broad policy for care” because it “generates evidence about how people respond to particular forms of care on average” (1465).

They even invoked clinicians’ experience of care rather than repeating their challenge to authority that was based on power and prestige rather than evidence. “Tailoring care to meet the specific needs of individuals,” they wrote, “will continue to be more of an art than a science.” As a result, the “differing circumstances and values of different individuals may provoke different reactions to the same quality of evidence” (1466). One reviewer of a draft of my article complained that this rhetoric contradicted the goals of the ECPC project: “By conceding [an art of medicine] … to clinicians,” he wrote, “they were regressing into a pre-modern concept of medicine” when instead their “project [was] based on the premise that some therapeutic questions can only be answered at the population level” (Morabia, draft review, February 23, 2011).

The authors promptly revealed, however, that they had harnessed the cliché of the art of medicine to their polemical intent. They redefined the art of medicine to make it compatible with population-level evidence. This art “can be improved,” they concluded, “by listening more carefully to what women have to say, and by involving them to a greater extent in decisions about their care.”

Despite their apparent embrace of traditional rhetoric regarding the art and science of medicine, the editors concluded the chapter with four appendices that, by implication, summarized the challenge of the ECPC project to clinicians and policymakers. These appendices listed four “forms of care”: those that “reduce negative outcomes of pregnancy and childbirth,” that “appear promising but require further evaluation,” that have “unknown effects which require further evaluation,” and that “should be abandoned” (1467–77). Because these appendices summarized what clinicians should do, stop doing, and study, they soon attracted attention both inside and outside medicine.

Initial Responses to the Rhetoric of ECPC within Medicine

The blending of scientific and polemical rhetoric in ECPC evoked strong reactions. The author of a commentary in the British Journal of Obstetrics and Gynecology hailed ECPC as “probably the most important book on obstetrics to appear in this century” (Paintin 1990, 967). But a reviewer in another obstetrical journal pointed out, “The price of 225 [pounds sterling] should protect aspiring registrars [residents] from acquiring too many confused ideas from its pages” (Hawkins 1990, 344). Another senior obstetrician, in a conversation reported to Chalmers, compared the authors with terrorists: “an obstetrical Baader-Meinhof [i.e., anarchist] gang” (Chalmers, email, December 24, 2010). The rhetoric of ECPC may have encouraged these reviewers to write their own polemics.

Most reviewers commented only indirectly on the rhetoric of ECPC and its companion publications. Their reviews emphasized the explanatory power of systematic reviews of randomized evidence while ignoring the volumes’ explicit and implicit polemic. One called the ODPT a “great visionary achievement” and commended it as the “technology of the future” to “those who enjoyed” ECPC (Neilson 1991, 175). Another praised the potential of ECPC and GECPC to “make a major contribution towards improving the quality of pregnancy care” but was silent about the authors’ suggestions about how to facilitate that contribution (Zander 1990, 396). A reviewer in the Journal of the American Medical Association characterized ECPC as “innovative,” “important,” and “monumental.” He especially praised the chapter “Materials and Methods” and the appendices to the concluding chapter (Gold 1990).

I have found only two authors who acknowledged the two voices of the volumes’ prose. Michael S. Kramer began his review in Science by noting that “ECPC is very different from the traditional clinical textbook.” Then he praised its “democratic and ‘caring’ flavor that lends a refreshingly human backdrop to its scientific rigor” (Kramer 1991, 816). Similarly, Fred Mosteller described the authors’ “consistent concern … for the personal side of care” as an “endearing feature” (Mosteller 1993, 528).

Most of the authors of reviews and early articles about the project did not recount either the history of systematic reviews or contemporary international activity to improve and apply their methods. A news article in Science, “Meta-Analysis in the Breech,” is a partial exception. The author, a freelance journalist, listed contemporary contributors to the methods of systematic reviews, for instance, Gene Glass in psychology, Richard Light in education, and Thomas Chalmers, Rory Collins, Kay Dickersin, Joseph Lau, and Richard Peto in medicine. But the journalist's reason for writing the article was that ECPC was controversial. He characterized it as a “remarkable report [that] offers a window onto the possibilities raised by meta-analysis and the vitriol it evokes” (Mann 1990, 476). In the history of science ECPC was the result of methodological developments that occurred over many years. The project was an important event in the history of health policy because it stimulated both controversy and new policy.

Antagonism to Systematic Reviews in the Health Sector

The methods and findings of ECPC and many subsequent systematic reviews antagonized some influential persons in the health sector. Many clinicians became uncomfortable, even angry, when findings of particular reviews contradicted their experience of treating patients. In countries with fee-for-service payment systems, these systematic reviews could reduce clinicians’ incomes if they contributed to limiting or eliminating reimbursement for particular interventions. Examples of clinicians’ antagonism to systematic reviews since the mid-1990s include controversies over autologous bone marrow transplants, fertility services, and treatment of back pain (Chalkidou et al. 2009; Fox 2010a; Moynihan 2008; Rettig et al. 2007; Steinbrook 2008; Wall Street Journal 2009). William Silverman, a pioneer of American neonatal medicine and a friend and mentor of Iain Chalmers, suggested that such controversies revealed that the “statistical approach was anathema to free-wheeling doctors who resented any doubts about the effectiveness of their untested treatments” (Watts 2005, 257).

Executives of both the firms that manufacture pharmaceutical drugs and medical devices and their trade associations have attacked systematic reviews that threatened to reduce their companies’ market share of particular products. Statins, atypical antipsychotic drugs, and implantable devices are recent examples (Fox 2010a; Gibson 2006; Moynihan 2010). Beginning in the early 1990s, trade associations and their member firms devised strategies to blunt or prevent the influence of independent systematic reviews (i.e., of studies they did not sponsor). They have, for instance, dedicated staff and engaged consultants to challenge reviews that threatened their profits and the methodology of research synthesis more generally, falsified or hidden data from clinical trials, and conducted media campaigns to promote evidence that contradicted the findings of reviews (Fox 2010a; Jorgensen, Hilden, and Goetzsche 2006; McCloskey and Ziliak 2008; Murray 2004; Pear and Dao 2004).

The firms also directly pressured policymakers to thwart systematic reviews they opposed. For example, an international pharmaceutical firm threatened to move a factory to another state if policymakers for Medicaid in Michigan used findings from reviews to deny coverage for any of its products (Haveman 2001). After a systematic review of a drug for influenza, the chairman of Glaxo Wellcome, in a conversation at Downing Street, “threatened to take his research out of ” Britain (Timmins 2009). And many companies made campaign contributions to elected officials and political parties (Fox 2010a).

The industry has also financed criticism of particular reviews by groups that advocate for patients with particular diseases or who suffer chronic pain (Craig 2002; Fox 2010a; Moynihan 2003; Rothman et al 2011). Reviews that found insufficient evidence of the effectiveness of some interventions often dismayed these groups (Drug Effectiveness Review Project 2004–2009; Fox 2010a; Hawkes 2009). They usually insisted that insufficient evidence did not justify denying patients’ access to an intervention that could help some of them. Early in 2011, for example, a systematic review of interventions to treat traumatic brain injury offended journalists for National Public Radio and Pro Publica and stimulated action by advocacy groups and a committee of the U.S. Congress (Miller and Zwerdling 2010).

Economic evaluations based on evidence from systematic reviews have frequently appalled both advocacy and commercial organizations because they compared the cost of extending life (usually measured in “quality-adjusted life years”) with benefits to populations from alternative expenditures (Hawkes 2009; Neumann 2006). Other critics of economic evaluation, including some public officials, criticized economists for imputing values, for example, according priority to people in particular age groups (Coast 2004).

In sum, political action by clinicians, suppliers, and advocates to prevent the application of findings from particular reviews was an incentive for public policymakers to be wary of the methodology. Elected officials could be particularly vulnerable to accusations that they condoned rationing services to vulnerable voters (Fox 2010a; Jewell and Bero 2008).

The Constituency for Systematic Reviews in the Health Sector

Despite this persistent antagonism to systematic reviews, a constituency for them among clinicians and health care managers has grown since publication of the ECPC project. A steadily growing number of international journals have published articles based on systematic reviews and on “evidence-based medicine” more generally. The authors of clinical practice guidelines issued by specialty societies, public agencies, and hybrid organizations (e.g., the Scottish Intercollegiate Guidelines Network) have increasingly based their recommendations on systematic reviews (Agency for Healthcare Research and Quality 2011). Medical students and house staff have learned the skills of critically appraising clinical literature, which are fundamental to understanding the methods and uses of systematic reviews (Choudhry, Fletcher, and Soumerai 2005; Montori and Guyatt 2008). Examiners for certification and recertification in some medical specialties now expect knowledge of the findings of systematic reviews.

This growing constituency was, as Mosteller predicted in 1992, in part a response to the example set by the ECPC project and to the persuasiveness of its two voices. No less important, however, new international organizations generalized the example, recruited participants, and organized them in international networks. Like many other scientific associations, the Cochrane and Campbell Collaborations set methodological standards and publish research that meets them.

Unlike other scientific organizations, however, the Collaborations have, since their inception (Cochrane in 1993, Campbell in 1999), reinforced and projected the two voices of the ECPC project. They have defined themselves as international movements composed of thousands of volunteers who share values about research and its application to patient care (even though most participants are employed by universities, freestanding research organizations, or organizations that provide health services and reward researchers with income and/or prestige). The officers and governing bodies of the Collaborations reinforce their self-image as movements, especially during annual international colloquia.

Chalmers, with others, led the organizing work for the Cochrane Collaboration, which by 2010 comprised 28,000 people in more than one hundred countries. The Cochrane Database of Systematic Reviews, the Collaboration's electronic journal, attracted increasing numbers of readers and became a significant source of citations in other journals (Bastian, Glasziou, and Chalmers 2010; Starr et al. 2009; Tovey 2010).

The Campbell Collaboration (C2) and its Library have organized and disseminated systematic reviews in the fields of criminal justice, education, and social services (Campbell Collaboration 2011). The founders of C2 based its identity on Cochrane, as a movement “based on voluntary cooperation among researchers.” A partial indicator of both the persuasiveness of findings from reviews and the success of their promoters is that the number of published reviews rose each year, from 87 in 1987 to 2,500 in 2005 (Moher et al. 2007).

Support from influential advocacy groups for women's health and from journalists reinforced the constituency within the health sector for the ECPC project and the subsequent systematic reviews it influenced. In the mid-1990s, leading advocacy groups for women's health financed a review that challenged widely accepted medical opinion about the benefits of autologous bone marrow transplants (Lerner et al. 2010; Rettig et al. 2007). The authors of the pathbreaking book Our Bodies Ourselves began to incorporate the findings of systematic reviews in new editions after publication of the ECPC project (Boston Women's Health Book Collective 2011; Judith Norsegian, founder and leader of Our Bodies Ourselves project, telephone conversation, March 2011). In the early 1990s, journalists began to publish articles about the potential benefits of systematic reviews. For example, a noted science reporter wrote a cover article about ECPC and GECPC for Parade, the Sunday newspaper supplement with the highest circulation in the United States (Ubell 1993). Ray Moynihan, who began to cover systematic reviews for Australian newspapers and television at about the same time, subsequently earned an international reputation (Moynihan 2011). By 2000, reporters and columnists for the Wall Street Journal and other major newspapers were reporting regularly on systematic reviews (Fox 2010a).

Systematic Reviews Inform Policymakers for Health

The potential importance of systematic reviews for policy to allocate resources to and within the health sector emerged gradually. Most experts on policy who wrote about the ECPC project soon after its publication assumed that proponents of systematic reviews would accord priority to influencing clinicians’ work, for example, by cultivating champions of such reviews among them (Cleary and Fox 1993; Lomas 1993; Sisk 1993; Stocking 1993).

Nevertheless, public officials in a few countries began to consider how systematic reviews could inform policy for allocating resources. They speculated that the findings of systematic reviews could contribute to care that was more effective and efficient, mainly by identifying interventions that were unnecessary, ineffective, or harmful. Some also wondered whether reviews that met rigorous international standards could also discourage lobbying, by professional and commercial interest groups and advocacy organizations, for more spending for particular interventions (Fox 2005, 2010a; Fox and Greenfield 2006).

Most important, a small number of policymakers hoped that systematic reviews might help them contain the rate of increase in spending for health care, to spend less as well as to spend better. Since the 1970s, annual increases in health costs have usually exceeded general inflation in industrial countries and, as a result, constrained spending for other public purposes. In the United States, where health insurance for most people of working age is linked to employment, increasing health care costs also have restricted growth in wages and reduced firms’ profits by making them less competitive in the global economy.

Policymakers who paid attention to systematic reviews usually learned about them from persons who had earned their trust as advisers. Most of these advisers were academic researchers or staff of philanthropic foundations. A few were policymakers themselves. For example, Lee Greenfield, who began in 1993 to teach policymakers about the methods and uses of systematic reviews, chaired the committee of the Minnesota House of Representatives that appropriated funds for Medicaid and other health programs (Fox 2010a; Fox and Greenfield 2006).

The growing interest in systematic reviews among policymakers has had two principal results. The first is a gradual increase in public funding to conduct reviews and train reviewers, with the timing and amounts varying widely among countries. The second result is the establishment of public and quasi-public organizations to conduct, commission, and assess systematic reviews and to apply, or advise about applying, their findings. These new organizations are variously called institutes, committees, schemes, and commissions (Fox 2010a, 2010b). The oldest of them is the Australian Pharmaceutical Benefits Advisory Committee (PBAC), established many years earlier but an early adopter of systematic reviews.

Systematic reviews began to inform policy at different times in different countries. The first endorsement of systematic reviews by policymakers for health services seems to have occurred in Britain in 1991, when the new Research and Development Programme (R&D) of the National Health Service (NHS) agreed to finance, for three years, Chalmers's proposal for “a Cochrane Centre to facilitate the preparation, maintenance and dissemination of systematic reviews of randomized controlled trials of health care” (Chalmers 2003, 251).

The founding director of the R&D Programme, Michael Peckham, who had been a prominent medical academic, endorsed systematic reviews because they contributed to achieving the priorities of the NHS R&D Programme. The new Programme, he wrote, “is an essential activity—indeed, a prerequisite for achieving a cost-effective health service response to changes in needs as well as to innovation” (Peckham 1991, 368).

Peckham implicitly endorsed both voices of the ECPC project. In his polemic, he criticized “unacceptable variation in the quality of treatment” and his medical colleagues’ “reluctance to modify practice even when compelling evidence is forthcoming.” In his scientific voice, he encouraged organizations that purchase health care for patients “to take into account the evaluative status of health practice methods.” To illustrate what could be accomplished by state-of-the-art evaluation, he described the ECPC project as the “most systematic attempt to describe an area of medicine in detail.” Systematic reviews, he concluded, could help policymakers “release resources that are not at present being used productively” as well as to take account of “advances in medicine” (Peckham 1991, 367, 369, 370).

NHS funding for the Cochrane Centre also subsidized the formation of the Cochrane Collaboration. The British government financed the editorial offices for Cochrane review groups that coordinated the international production and publication of systematic reviews. British public funds are still the largest single source of subsidy for Cochrane work globally. In 2007/2008, moreover, the UK government incorporated the dissemination of information about systematic reviews and related research into its overall strategy for assisting in international development (Donaldson and Bontavala 2007).

Since 1999, the National Institute for Health and Clinical Excellence (NICE) has been influential in Britain and internationally in applying findings from systematic reviews. Its chairman, Michael Rawlins, recalled in 2008 that its “origins … go back to the creation of the Cochrane Centre in 1992 … and the establishment of Health Technology Assessment Centres in British Universities [by NHS R&D] in the early 1990s” (Timmins 2009, 1360). An expert-led public agency at “arm's length” from government, NICE conducts and appraises evidence reviews and commissions economic evaluations for use in the NHS. Senior officials and ministers, even a prime minister, have defended NICE against attacks by medical specialists, pharmaceutical manufacturers, and advocacy groups. Courts have decided lawsuits brought by patients in favor of NICE. A select committee of Parliament recommended extending NICE's scope to include issuing guidance about interventions to improve public health and disinvesting in ineffective health technologies (Elshaug et al. 2009; Fox 2010c; Timmins 2009).

Systematic reviews have informed policy to pay for pharmaceutical drugs in many countries. The Australian PBAC began in 1993 to require that systematic reviews be conducted whenever sufficient evidence from primary studies was available (Ruth Lopert, Australian civil servant with experience in policymaking for pharmaceutical drugs, interview, July 8, 2008). By 1998 such reviews had informed 28 percent of the decisions by PBAC about insurance coverage (David Henry, former PBAC research scientist, email, February 18, 2008). The New Zealand Pharmaceutical Agency has used findings from systematic reviews from its inception in 1993, and an intergovernmental agency that assesses health technology in Canada has used methods of systematic reviewing since the mid-1990s (Morgan et al. 2006). The first Reference Drug Program in that country, established in British Columbia in 1995, relied on systematic reviews, its founding director reports (Bob Nakagawa, British Columbia's assistant deputy minister of health for pharmaceutical services, email, February 18, 2008). In both Australia and Canada, senior policymakers have intervened, not always successfully, to defend systematic reviewers, as well as the organizations that commissioned them, against negative criticism and lobbying by interest groups and advocates (Fox 2010a; Henry, email, February 18, 2008; Lenzer 2010, Lopert, interview, July 8, 2008; Moynihan 2010; Nakagawa, email, February 18, 2008).

Policymakers in the United States began to apply the findings of systematic reviews in the late 1990s. Soon after the publication of ECPC and GECPC, legislative leaders and executive branch officials in state government responded enthusiastically to a talk in which I described the four appendices at the end of the second volume of ECPC and the methodology that produced them (Fox and Greenfield 2006). A year later the senior health official in President George H.W. Bush's administration endorsed the ECPC project at a conference in Washington, whose sponsors included the American College of Obstetricians and Gynecologists. Officials of federal research agencies did not act on this endorsement until 1996, when the Centers for Disease Control and Prevention began to commission reviews for its Guide to Community Preventive Services (Benedict et al. 2000). A year later, the predecessor of the Agency for Healthcare Research and Quality (AHRQ) designated the first “evidence-based practice centers” (EPCs) to conduct evidence reviews, including systematic reviews. The U.S. Preventive Services Task Force, established in 1984 and administered by the AHRQ beginning in 1998, commissioned systematic reviews from EPCs as the basis for its recommendations about the effectiveness of interventions (Fox 2010a).

Senior policymakers in state government preceded their federal colleagues in commissioning systematic reviews explicitly to inform decisions about covering services for participants in public programs. Beginning in 1993 the Milbank Memorial Fund, an endowed foundation, organized interactive meetings at which leaders of the legislative and executive branches of government in states learned about the methods and uses of systematic reviews (Fox and Greenfield 2006).

Policymakers who had participated in meetings organized by Milbank concluded during the recession of 2001 to 2003 that applying the findings of systematic reviews comparing the effectiveness of pharmaceutical drugs in various classes could help contain the growth of prices for pharmaceutical drugs in public programs (Reforming States Group 2003). By 2003, officials in three states were collaborating to commission and disseminate the findings of state-of-the-art reviews of drugs in especially expensive classes. The number of collaborating states eventually grew to twenty but has recently fallen (Drug Effectiveness Review Project 2004–2009). Nevertheless, forty-five states subsequently used these reviews in deciding which drugs to include in Medicaid's formularies (known as “preferred drug lists”), workers’ compensation, and insurance programs for public employees (Cauchi 2008). Policymakers in most of these states created administrative processes that partially insulated from the influence of interest and advocacy groups decisions that were informed by reviews about whether to pay for drugs (Fox 2010a; Winslow, McGinley, and Adams 2002).

In 2009, the U.S. Congress included $1.1 billion in appropriations for antirecession stimulus spending to conduct and disseminate research on the comparative effectiveness of health services (CER). In drafting this legislation, members of Congress and their staff relied on states’ experience in commissioning and applying systematic reviews (Mark Gibson, founding director of the Drug Effectiveness Review Project, conversations and correspondence, 2009; John McDonough, at the time, a staff member of the U.S. Senate Committee on Health, Education, Labor and Pensions, email, January 2, 2009). Nevertheless, less than 3 percent of these funds was spent on systematic reviews, and the program did not link priorities for funding primary studies to priorities for systematic reviews (Benner et al. 2010). The relative insignificance of systematic reviews in the CER program may be evidence of the persistence of antagonism to them among biomedical researchers and officials of the National Institutes of Health.

Federal legislation in 2010 (the Patient Protection and Accountable Care Act, or ACA) assigned responsibility for most research on effectiveness and comparative effectiveness to the new, quasi-governmental Patient Care Outcomes Research Institute (PCORI). This institute has a funding mechanism independent of annual appropriations and is governed by a board whose members represent industry, health professions, consumers, and researchers. Members from outside the federal government are appointed by the comptroller general, who heads the U.S. Government Accountability Office (GAO) and is insulated from partisan politics by a fifteen-year term. The legislation also established unusually specific procedures for the accountability of the PCORI to the GAO. As a result of complaints by advocacy groups and some members of Congress that CER could be the basis for rationing care, however, the ACA prohibited federal agencies from using findings from research sponsored by the PCORI as the basis for decisions about the coverage of interventions (Lerner et al. 2010).

The influence of systematic reviews on policy for allocating resources to and within the health sector has other limitations. Policymakers must consider their constituents’ economic interests. They want to be credited with easing access to treatment that could be beneficial, and they fear voter retaliation if they appear to support rationing of care (Iglehart 2010). Elected officials in competitive districts continue to accept campaign contributions from health care interest and advocacy groups and thus incur obligations to them. Moreover, policymakers cannot easily set aside their personal values, including compassion, when they make decisions that balance the costs of access to particular treatments against potential benefits.

Another limitation on the influence of systematic reviews is a paradox of the relationship between science and policy, which is that the inexorable evolution of methodology limits the usefulness of research findings to policymakers. A recent oration and article by Michael Rawlins of NICE exemplifies this paradox. Rawlins described the methodological problems in interpreting “evidence for decisions about the use of therapeutic interventions” (Rawlins 2008, 2152). The headline of a news release from the Royal College of Physicians (which sponsored the oration) was “Sir Michael Rawlins Attacks Traditional Ways of Assessing Evidence” (Royal College of Physicians 2008). The headline implied that Rawlins's oration was a polemic against systematic reviews and the primary studies on which they were based.

Rawlins actually said, however, that a “diversity of [methodological] approaches” should influence decisions about interventions and policy. He intended “a plea to investigators to continue to develop and improve their methods,” to decision makers to abjure reliance on entrenched practices for interpreting evidence, and to both to understand that the “interpretation of evidence requires judgment” (Rawlins 2008, 2160).

Rawlins endorsed a fundamental assumption of the ECPC project: the primacy of evidence over authority derived from rank and power. But the assumption that science is always provisional is not helpful to policymakers because they are preoccupied with uncertainty. They devote their careers to assessing and addressing the impact on policy of actions and events they cannot control. Policymakers who allocate resources to and within the health sector hope that research findings will help them manage uncertainty by reducing the influence of judgment on policy, especially judgment grounded in ideology, interests, advocacy, and the unsupported views of clinicians. But the usefulness of science to countervail against uncertainty is diminished because methodology evolves, findings from research change, and interpreting evidence requires judgment.

Conclusion: Rhetoric and Policy

Despite this paradox of the relationship between science and policy, the ECPC project has had substantial influence. The research on which it reported built on advances in methodology that occurred over many years in a variety of disciplines. The editors’ two voices, their blending of scientific and polemical rhetoric, helped communicate the potential significance of systematic reviews to diverse audiences, which included clinicians, consumers, journalists, and managers of health care organizations as well as policymakers.

The ECPC project was, at first, persuasive to a relatively small number of researchers, clinicians, and advocates for women's health. These people became the core of a diverse and expanding international constituency. The editors and authors of the project became leaders in organizing, focusing, and expanding the constituency for systematic reviews. But the ECPC project also sparked antagonism within the medical profession and among commercial and advocacy groups.

Soon after the publications appeared, a small number of public policymakers conjectured that systematic reviews could be useful in making health services more effective and efficient and perhaps assist in containing the rate of increase in expenditures for health services. These policymakers subsidized particular systematic reviews and an infrastructure to produce and update them. They also devised administrative processes that partially insulated deliberations about applying the findings of systematic reviews to policy from criticism and lobbying by interest and advocacy groups.

Systematic reviews have informed policy in many jurisdictions. They have done so as a result of the influence of both voices of the ECPC project, of a growing constituency for reviews in the health sector in the media and among consumers, and of support from many policymakers. Although the methodology of systematic reviews has a long history and will continue to evolve, their ongoing use to inform policy and practice began with publication of the ECPC project.