“Developing Good Taste in Evidence”: Facilitators of and Hindrances to Evidence-Informed Health Policymaking in State Government

Abstract

Context

Policymaking is a highly complex process that is often difficult to predict or influence. Most of the scholarship examining the role of research evidence in policymaking has focused narrowly on characteristics of the evidence and the interactions between scientists and government officials. The real-life context in which policymakers are situated and make decisions also is crucial to the development of evidence-informed policy.

Methods

This qualitative study expands on other studies of research utilization at the state level through interviews with twenty-eight state legislators and administrators about their real-life experiences incorporating evidence into policymaking. The interviews were coded inductively into the following categories: (1) the important or controversial issue or problem being addressed, (2) the information that was used, (3) facilitators, and (4) hindrances.

Findings

Hindrances to evidence-informed policymaking included institutional features; characteristics of the evidence supply, such as research quantity, quality, accessibility, and usability; and competing sources of influence, such as interest groups. The policymakers identified a number of facilitators to the use of evidence, including linking research to concrete impacts, costs, and benefits; reframing policy issues to fit the research; training to use evidence-based skills; and developing research venues and collaborative relationships in order to generate relevant evidence.

Conclusions

Certain hindrances to the incorporation of research into policy, like limited budgets, are systemic and not readily altered. However, some of the barriers and facilitators of evidence-informed health policymaking are amenable to change. Policymakers could benefit from evidence-based skills training to help them identify and evaluate high-quality information. Researchers and policymakers thus could collaborate to develop networks for generating and sharing relevant evidence for policy.

Policymaking follows a logic that is different from that of the scientific enterprise (Brownson et al. 2006). The role of evidence based on research is often minimal, and even when it is used by policymakers, such evidence is greatly affected by cognitive and institutional features of the political process. In a dynamic, information-laden world, government officials can attend to only a few data sources at a time. Previous research suggests that they tend to rely on common sense, personal stories, and standard operating procedures and that they are concerned primarily with recognition and reelection (Jones and Baumgartner 2006; Lindblom and Cohen 1979; Stone 2002). The bargaining costs for achieving agreement within a governing body further hinder the ability of political systems to respond to information and demands in a timely and rational fashion. As a result, policymaking is a highly complex process that is difficult to predict or for individual participants to influence, producing stable policies occasionally marked by extreme change (Jones and Baumgartner 2005; Kingdon 1995).

A considerable body of scholarship examines factors affecting how policymakers use evidence. A systematic review of twenty-four interview studies of government officials found, for example, that personal contact between researchers and policymakers, as well as the timely conveyance of germane and concise information, facilitated the use of research (Innvær et al. 2002). More recent studies have similarly noted the importance of the relevance and timing of research as well as researchers' skills in translating and communicating information (Brownson et al. 2006; Landry, Lamari, and Amara 2003; Lavis 2002; Lavis et al. 2003; Petticrew et al. 2006; Sorian and Baugh 2002). Other studies have examined analytic tools and evidence packaging to facilitate the uptake of information, including systematic reviews and health impact assessments (Atkins, Siegel, and Slutsky 2005; Fielding and Briss 2006; Lavis et al. 2004, 2005; Sweet and Moynihan 2007). Following the “two communities” approach, most of this scholarship has focused narrowly on evidence features and the interactions between scientists and government officials (Caplan 1977). Yet these “interfaces” are only part of the picture. The real-life context in which policymakers are situated and make decisions as research “receptors” also is crucial to informed policy outcomes (Hanney et al. 2003). Health policy scholarship, however, has rarely studied this setting and therefore often underestimates how features of the political process affect the incorporation of the best evidence into decision making.

The studies that have examined research utilization from the policymakers' perspective demonstrate the significance of institutional effects. For example, the role of evidence is often hindered by the fragmentation of levels of government and across service sectors that limit research reception and dissemination; by the inundation of competing forms of information, such as from the media; by committee jurisdictional monopoly over some issues; and by the domination of narrow interest groups in policy areas with low political salience (Brownson et al. 2006; Shulock 1999; Waddell et al. 2005; Weiss 1989). By contrast, factors that have been found to facilitate informed policymaking include policy entrepreneurs and in-house research units; information-savvy, politically active social movements; and the technical, content-driven nature of some policy areas (Coburn 1998; Johnson et al. 2004; Lavis et al. 2002; Waddell et al. 2005; Weiss 1989).

Most of this research, however, is based on the Canadian context, and studies that look at the United States focus primarily on the federal level. Given the ongoing devolution of health policy authority and responsibility to the U.S. states, particularly for Medicaid, understanding how evidence is mediated at this level of government is crucial. This article expands on other studies of research utilization at the state level (e.g., Coburn 1998; Sorian and Baugh 2002). Based on interviews with twenty-eight state legislators and administrators about their real-life experiences in policymaking, we identify common features of the policymaking process that affect how evidence is incorporated into American state health policy decisions. In this article, evidence-informed decision making refers to using the best available evidence and excluding spurious information, all in the context of other political and institutional features. The objective of our study is to discover both hindrances to evidence-informed health policymaking and those facilitators that may be most amenable to targeted interventions by researchers and policymakers.

Methods

Our sample constitutes the “best cases” of what we could expect from public officials in terms of research utilization. It consists of individuals who have attended the policymakers’ section of the Rocky Mountain Evidence-Based Health Care (EBHC) workshop. This program, begun in 1999 by the University of Colorado School of Medicine, built on previous effectiveness research workshops for public officials (Fox and Greenfield 2006). Through small-group, experiential learning, attendees are taught about systematic reviews and meta-analyses,1 how to use databases like the Cochrane Library,2 as well as how to assess the quality of studies of various designs. The participating, bipartisan policymakers we interviewed are not a representative sample of typical state legislators or agency officials and may be more motivated to use evidence than many of their colleagues are.

A second feature of our study is that the interviews were not focused on a particular policy issue. Because the EBHC attendees had held a variety of government positions, it was impractical to limit the discussion to specific health topics. We instead followed a more general line of inquiry, letting them talk about those issues that were most important to them. From these detailed “real-life” stories we identified general patterns of how our informants typically access and evaluate relevant research and their perspectives on how evidence is used more generally in the policymaking process.

The purpose of this qualitative study is to gain new insights into a process that is relatively poorly understood. The interviewees' political experience and their sensitivity to evidence utilization make them excellent insider informants of the political and regulatory factors that determine how information is absorbed in the institutional settings in which they work. In order to inform and generate hypotheses, we focused on depth and detail rather than a broad representative sample (Charmaz 2007; Silverman 2004). Subsequent research can use quantitative methods to test these findings and to assess the frequency and relative significance of the factors we have identified (Grbich 2007; Yin 2003).

Data Collection

After receiving human subjects approval (UCSF approval no. H2758-21541-06), we obtained contact information from EBHC organizers for all thirty-five policymaker participants who had attended from 2000 through 2005. We sent out a letter of introduction describing our research project and followed up with telephone calls asking the interviewees to provide a brief description of their current job responsibilities, relevant employment history, and biographical information. The core of the interview consisted of their describing in detail two or three health policy examples in which they had participated, including examples in which evidence was used successfully or not. We then asked the interviewees to highlight the most important issues in their example, the key players, the sources and types of evidence they used, and the most important factors leading to the policy outcome. We also asked them to identify the challenges and facilitators of evidence-informed policymaking in their examples, as well as in the policymaking process more generally, as well as to comment on the usefulness of the EBHC workshop experience to their work. The interviews lasted between thirty minutes and two hours, with the average being an hour. Most of the interviews were conducted by telephone, but six were conducted in person. All were taped and transcribed for analysis.

Twenty-eight of the thirty-five policymakers participated, an 80 percent response rate. Two declined, three agreed to participate but were ultimately unable to schedule an interview, and two could not be located. Eleven interviewees were officials from public health agencies, fifteen were state legislators, and two had worked in both settings. We interviewed three of the five participants from 2005 twice, once at the workshop itself and again six to nine months later, for a total of thirty-one interviews. The “pre-” interviews with the 2005 workshop participants provided a different perspective on the challenges of integrating evidence into decision making, even for the well intentioned with significant technical expertise. Several of these persons could describe clearly how their current decision-making approaches contrasted with what they were learning at the workshop. The legislators were about evenly divided between state senators and representatives and had worked for an average of fifteen years of government service. Six of the legislators were from the West, six from the Midwest, two from the Northeast, and two from the South. All served on a wide range of committees, most commonly health, mental health, finance, and appropriations. The administrators tended to be high-level government career health professionals. The majority were program directors or deputy directors in Medicaid, health and human services, or mental health departments. Of them, four were from the West, four from the Midwest, three from the Northeast, and two from the South. Of the twenty-eight people we interviewed, twelve had been on the Steering Group of the Reforming States Group (RSG) between 2000 and 2005 (Andersen 1998). These twelve participants were likely to have been aware of the workshop and wanted to attend. The Milbank Memorial Fund paid all the travel expenses of the participants in the workshop.

Data Analysis

We coded the interview data inductively, not knowing specifically what we would find when we began the data analysis. When we created the coding instrument for the interviews, we used the officials' policy experiences to identify factors that facilitated or hindered the use of evidence in legislative and administrative settings. We defined evidence according to an epidemiological “hierarchy.” Thus, systematic reviews and meta-analysis are considered the best evidence, followed by individual randomized controlled trials and then observational studies (cohort and case control studies). Other types of information and experience, such as anecdotes, constitute the lowest level of evidence. But we also recognized that some types of evidence that are important to health policymaking do not fit easily into this framework, for example, studies of cost-effectiveness, and implementation or evaluation studies. Accordingly, in our coding, we considered the broad array of evidence that the policymakers actually discussed.

We initially coded the interviews in a few major areas: (1) the important or controversial issue or problem being addressed, (2) the information that was used, (3) facilitators, and (4) hindrances. This, then, represented a first attempt at abstraction, to characterize, for example, what kinds of facilitators the interviewee had identified in his or her particular example.

One of us (Jewell) and a research assistant independently coded ten interviews. We then compared the codings and descriptions and discussed and resolved the discrepancies. Jewell then coded the remaining twenty-one interviews, and the research assistant reviewed them. Facilitators and hindrance codings were then further classified according to three major categories of traditional political science frameworks: ideas, institutions, and interest groups (Lavis 2004). Thus, our analysis of the interview data reports the recurring and representative themes, ideas, reactions, and expressions found throughout the interview transcripts (Neuman 2000).

Our findings are based on a synthesis of all the interviews in this study. All statements are based on the reports of multiple interviewees. Text without attribution to a specific respondent is a summary of the interviewees' comments, not our speculations. The quotations we have incorporated into the text are representative and intended only for illustrative purposes.

Results and Analysis

Health Policy Decisions

There was considerable overlap among the types of health policy issues discussed by the legislators and administrators. They can be divided into four major categories: (1) Efforts at making system-level changes, including reforming state Medicaid programs, the mental health system, workers' compensation, state insurance requirements, and malpractice. One of the most frequently mentioned health policy areas was pharmaceuticals and the states' efforts to develop Medicaid formularies and related initiatives to influence prescribing practices (e.g., drug registries and therapeutic substitution bills). (2) Changes in program coverage, such as designing the dental service plan for a state's Children's Health Insurance Program (CHIP), easing the standard of proof for workers' compensation coverage for hepatitis C, and providing state Medicaid coverage for particular drugs, procedures, or services (e.g., bariatric surgery). (3) The institution of best practices and utilization guidelines as a means of improving the cost-effectiveness, quality, and consistency of primary care and hospital services. Examples included implementing chronic disease case management standards, ensuring that beta blockers are prescribed as a regular emergency room practice for heart attack patients, and controlling hospital bed capacity. (4) Other health policy areas, including tobacco control, immunization requirements, and STD prevention programs.

Hindrances to Evidence-Informed Health Policymaking

Institutional Features

Some of the characteristics of the political setting that hinder evidence-informed decision making are more or less systemic qualities that are not readily amenable to change. These institutional features differ enough between administrators and legislators to warrant separate analyses.

Administrative Context

Administrators often are career officials who are well trained and have a lot of work experience with particular health policy issues. The “technical” nature of their responsibilities in carrying out policy development and implementation requires collecting and assessing data as a core part of their work, and offers some insulation from the kinds of political pressures and demands typical of the legislative arena. Administrative decision processes also tend to involve fewer actors, namely, administrative staff, although stakeholders can sometimes become very active participants. In many cases, this delegated authority, which falls below the political radar screen, gives administrators tremendous discretion in how health policy is decided in their state agencies.

Nearly all the administrators pointed out their limited organizational capacity to collect and evaluate research, and many reported that their agency's ability had deteriorated as a result of budget cuts, coupled with more mandates and more complex policies. Beyond resource issues, there often was a clear mismatch between organizational characteristics and the requirements of evidence-based decision making. Before attending the EBHC conference, many health officials had had little experience assessing research design and only a limited understanding of the function and location of systematic reviews. Although agency staff (such as doctors, nurses, pharmacists, and social workers) often had had medical or public health education, few were trained to appraise evidence critically. One official noted about Medicaid directors that they were “typically highly qualified physicians but not necessarily qualified in reviewing and analyzing clinical evidence in a systematic way.”

Furthermore, the health agency staff had not been taught “to continue to use research to inform their decisions, to inform their practice.” They therefore made decisions based on “common sense,” “gut level,” “standards of practice,” and comparative convenience and awareness of available data, rather than based on systematic reviews of research. One official observed that in assessing the effectiveness of a new medical procedure, “I just did exactly what … everyone … is hoping I'm not. I talked to my brother-in-law and I Googled it.”

Legislative Context

In contrast to the comparatively insulated and technical policy world of administrative officials, the work of legislative officials is shaped by forces that present different and formidable obstacles to the regular use of scientific research. Almost all the interviewed legislators were from part-time legislatures (approximately 80 percent of state legislatures are part-time), with few staff and limited compensation. Moreover, the comparatively short meeting times of most state legislatures means that the lawmaking process proceeds at a breakneck pace. One legislator estimated that the legislature contended with seven hundred bills in four months. The pressure to make decisions quickly on a wide range of policies obviously limits the legislators' ability to accumulate much information about any one issue. One manifestation of this rapid pace is a tendency to compartmentalize issues, with individual politicians understanding well only a few issues: “The only thing we have in common is that we are all overwhelmed with information. I don't have a clue, for instance, about criminal policy. I ask my colleague about how to vote on those issues, and she usually follows my advice on health.” The compartmentalized expertise in legislative bodies is especially pronounced for health and human services, policy areas so complex “that most [legislators] are either overwhelmed or are not interested in it.”

Almost all the legislators we interviewed remarked on state legislators' limited research skills, which led to a general lack of understanding about “how [evidence] can be used properly and how it can be misused” or the large differences in the quality of the information they received. Several interviewees noted that this limited understanding extended to even very basic concepts: “Many legislators do not understand the difference between cause and correlation … and it's very important in health care.” This deficit was seen as partly to blame for the strong foothold that certain interest groups could get in the legislative process, for example, with conservative groups disseminating studies that falsely claimed to demonstrate that birth control caused the spread of AIDS and other STDs. As a result, officials are often unable to distinguish between good and bad data (“to winnow out the opinions from the facts”), especially for “high-salience” policy issues, in which policymakers often are inundated with information from a wide array of interested parties. There also arises a general lack of interest in or even aversion to evidence as “too complicated” or “too boring” so that, instead, what resonates with “common sense” and “gut feeling” is most convincing.

The importance of securing resources to further one's own legislative agenda as well as to remain in office also means that officials often take positions based on little or no high-quality evidence. The respondents recalled that legislation was most often initiated on the basis of “the anecdote or single event” of a constituent or the unquestioned claims from a lobbyist organization with vested interests in the outcome. Prior commitments—an election promise to constituents, or in exchange for support from a particular lobbyist group or another legislator—also mean that legislators “can be pretty close minded about evidence” for fear of losing crucial support or even because of not wanting to appear inconsistent by publicly changing their positions. Officials are especially intransigent about their own legislation—“everything is in getting your legislation passed … they don't want to lose it”—as well as issues in which they have a direct personal economic interest. In reference to medical malpractice reform, one official discussed the problem with having a legislature with a large contingent of trial lawyers: “Right now you've got a bunch of people who could care less about the evidence unless they can manipulate it. … They know where their economic interests are … and that will trump any kind of evidence.”

Other characteristics of legislative settings that were identified as hindering evidence utilization are the twin institutional features of turnover and hierarchy. The frequent change in the members of the legislative body hinders the development of expertise and sustained attention to particular policy issues: “Without any institutional memory that this worked and that didn't work, we won't have clear policies in place and [won't] look at legislation with an eye to seeing whether it really is effective.” This limitation of institutional memory is particularly severe in states with term limits, in which legislators who have been working in health policy for fifteen or twenty years are suddenly retired, leaving relatively inexperienced officials to fill the leadership void.

You'll see somebody who is the speaker of the house who is serving their second term … who barely knows what the institution is about. Now either you make the staff powerful, the lobbyists powerful or the governor powerful. … It's not the legislature. They haven't got the experience or knowledge at this point.

Conversely, the hierarchy in the political parties concentrates much of their power in senior members and committee chairs: “When you have a strong public health committee, then the health committee sets policy. If you have a weak public health committee, the lobby sets policy.” In states without term limits, many of the senior members have been in the legislature for a long time, but the whole idea of evidence-based decision making may be new to them: “It's going to take a lot of educating of the leaders of the general assemblies to understand the implications and importance of [solid clinical evidence] before it's going to go that route.”

Evidence Supply Features

When there is interest in obtaining evidence to make health decisions, officials identified several kinds of problems with research quantity, quality, accessibility, and usability. Legislators provided far fewer and less varied comments about evidence (approximately one-quarter of the total number of comments), likely reflecting differences in their relative level of involvement in the acquisition and interpretation of research in their daily work compared with that of administrators.

Research Quantity

There are few relevant studies for many important health policy issues, much less systematic reviews of evidence. For example, there is limited research comparing drugs within therapeutic classes, durable medical equipment, health care systems, and non-drug behavioral health interventions. Often there is also little or no evidence regarding new or emergent technologies, which can present significant challenges for administrators feeling pressured by legislators, service providers, and consumers to expand coverage.

Legislators' comments centered on the lack of information about the economic impact of various interventions, including chronic disease management and health care savings accounts. As a result, “it is easy to take the position you are emotionally drawn to because there is nothing more substantive to go on.” Legislators also commented about how many research organizations, such as universities, were unable to respond rapidly enough to be useful.

Research Quality

Sometimes the existing research is of poor quality or limited applicability. In the case of Medicaid, for example, studies from other states may not be applicable because of differences in how states structure their Medicaid programs. And research based on private insurance or Medicare data does not necessarily translate well because of differences in the program populations.

Existing studies and systematic reviews commonly lack features that would make them easier for government officials to evaluate. For example, the quality of studies is often difficult for nonexperts to interpret because the explanation of research methods is long and complicated. With regard to this issue, one official thought the best change that could be made to facilitate evidence utilization would be a bullet-point evaluation or rating system of study design quality so that “for those of us who don't make our living doing that, we don't have to read a half dozen pages to ferret it out.” Similarly, while many officials considered the funding source for a study to be an important evaluative factor, few search databases, including PubMed, routinely provide this information.

Interventions in some health policy areas, such as social services, are particularly hard to evaluate. Randomized controlled trials are difficult to conduct for chronic conditions that require long-term interventions, such as in the case of disability and mental health services. Therefore it often is necessary to rely on less rigorous research designs like observational studies. Outcomes are also more difficult to define and measure in these areas.

With physical health care … somebody either dies or they don't. Or they get better or they don't. A person with developmental disabilities doesn't get better. A person with mental illness may function a little bit better. They may get a job, they may get a place of their own, so it's just the measuring that we aren't doing … and therefore we don't know whether what we're doing is doing any good.

In some cases, the available studies use small numbers or do not appear in peer-reviewed publications, significantly limiting their explanatory power or reliability. While an astute public health official may be able to recognize poor-quality information, the solution in question may be championed by advocacy groups, making it much more difficult to dismiss (more on this later). Legislative officials discussed research quality mainly as a lack of credible sources, because all information seemed to be provided by groups with vested interests in the outcome.

Guidelines or authoritative reports from prominent organizations were regarded as an important resource because of the limited in-house time and skills available to collect and synthesize pertinent information, and such reports also lent external legitimacy and authority to policy decisions that incorporated them. Nonetheless, health officials provided several examples of “consensus-based practice guidelines” from major organizations that they believed were not based on evidence. This sometimes happened when the guideline developers had clearly favored advocacy groups or service providers with a vested interest. Three examples that officials named were the Centers for Disease Control and Prevention's (CDC's) failure to distinguish physical education from physical activity; the CDC's support for Direct Observation Therapy, despite systematic reviews showing no effect; and the Substance Abuse and Mental Health Services Administration's (SAMHSA's) support of peer-based services as a best practice, albeit without solid evidence.

Accessibility

Even when evidence is available, policymakers may have problems obtaining it. Some officials do not have access to databases like the Cochrane Library because their state has not subscribed to them. In other cases, the prevalent studies have not been centralized; examples cited were publications on benefit design and state Medicaid evaluations. And in many cases, the data are never published, such as studies conducted by pharmaceutical companies: “And you wonder how it would have impacted your view of the world if they had been there, or what limitations were uncovered that someone didn't want to pursue and for what reason.”

Usability

The most commonly cited reason attributed to the limited usability of existing data was that policymakers' needs do not drive research. Instead, much of the information is produced by service providers or product makers who both have a vested interest in the implications and provide answers to narrower, business questions. A pharmaceutical company-funded head-to-head study of proton pump inhibitors illustrates this point:

Astra, when they saw the expiration of the Prilosec patents coming down the line, they said we've got to have a drug to replace it, so they created Nexium. They tested them head to head at 20 milligrams, found no difference. So they doubled the dose of Nexium over Prilosec and found a very marginal improvement for a small subset of patients with gastric reflex disease and stopped. They didn't go back and test forty of Prilosec against forty of Nexium because they're almost identical medications anyway, and there was no difference at twenty. Of course they didn't. They had a marketing question.

In addition, academic researchers generally follow their own interests when choosing what studies to conduct or tailor them to specific requests for grants. Similarly, the synthesis of existing research in the form of systematic reviews is driven by the researchers' particular interests. As a result, policymakers find that research often “sort of applies … but not quite. You know, if we had had this conversation earlier, we could have added this or we could have, you know, fired ten degrees to the west, and we'd have been right on in terms of really helping us all.” As one official aptly phrased it, researchers need to “anticipate where the hockey puck is going to be two years from now” by, for example, producing evidence to answer emerging policy questions, such as the epidemic of child and adolescent obesity.

Although decision makers need to understand the uncertainties and weaknesses in the data (“to know where the ice is thick and where it's thin”), they often are not provided. Systematic reviews do not necessarily frame the existing evidence in terms of their policy applications. As one exceptionally well trained administrator described it,

The hardest part for me is figuring out how I take the evidence here and actually put it in the context of the policy-relevant question. There's nobody on the evidence-based side thinking about how this would fly in the policy arena. … We can predict what the likely policy questions are going to be. … You'd think there'd be six policies around child nutrition and school environment … but that's never present.

Competing Sources of Influence

Even when good-quality, relevant research evidence is at hand, other features of the policymaking process can prevent the development of evidence-informed health policy.

Power of the Anecdote

One pervasive hindrance is the greater persuasiveness of specific, concrete information over that of abstract, technical information. As one official pointed out: “If the evidence leads you to something other than what an emotional or … top-of-your-head commonsense reaction would tell you … getting over that is enormously difficult.”

People generally have difficulty understanding numerical characterizations, including percentages and very small and very large numbers. When discussing a bill on formaldehyde off-gassing in plywood for mobile homes, for example, communicating the hazardous exposure levels becomes extremely difficult:

It's a very small number of molecules per cubic centimeter of air. And you go try to present that and its numbers. They don't make any sense of it. … You can do it by zeros, and it's too hard to see what it is. And if you do it times ten to the minus, they don't know what the hell it means.

Large numbers like those found in financing decisions are similarly bewildering to most people:

Millions sometimes they can understand. After millions it starts to get fuzzy. Get to billions, you've lost most of them. They won't try to deal with it. They'll just look at you. … You've got to understand this about people. And it makes it much harder to figure out how to present it.

Producing or even collecting and synthesizing high-quality research evidence also requires resources and skills that are beyond the capacities of most interest groups (Kerwin 2003). In contrast, though, personal stories are a comparatively accessible form of information that is easy for laypeople to understand. In some cases, experiential information can function as an important counterweight when the state of the evidence is uncertain. For example, the National Alliance on Mental Illness (NAMI) objected to the scientific evidence regarding the clinical equivalence of antipsychotics by arguing that their members' experiences did not confirm the findings and that, given the uncertainty, the choice of drugs should not be restricted.

Causal assertions conveyed in personal stories, however, may also keep alive a health controversy long after the causal linkage has been refuted by scientific evidence. The parents of children with autism are an example. Several officials mentioned the recurrent efforts of advocates to assert a link between the use of thimerasol in vaccines and the rise of autism as a basis for eliminating that ingredient or requirements that children be vaccinated.

Stories also are easy to manipulate or fabricate. For example, an effort to pass legislation to allow pharmacists to substitute drugs that are therapeutically equivalent failed when drug company lobbyists used anecdotes purporting to show that patients had been hurt when pharmacists made bad decisions.

And you can always find somebody on the committee that's had some bad experience and so they're perfectly willing to tell their story, and then fifteen lobbyists stand up and talk about how different people have been harmed by this sort of legislation. … And they aren't studies … the lobbyists even lie … I mean they make up stories that don't exist.

Rights Talk

While stories represent a competing source of information to scientific research, there are also powerful counterframes that do not rely explicitly on information at all, most notably by defining policy issues in terms of people's rights. Opponents often recast efforts to institute evidence-based practices or systemic reforms in terms of rights violated by restricting medical options or infringing on privacy. For example, efforts to create state drug formularies have been met with considerable opposition by the pharmaceutical industry on the grounds that such legislation “would be taking choice away from consumers.” These charges have sometimes been reiterated by advocacy groups as the moral claim of vulnerable populations to health care or in racial terms (“that this discriminates against Hispanics or people of color”). This scientific skepticism of some advocacy groups is often part of a more general distrust of governmental and medical authority to adequately recognize the needs of marginal populations. An emphasis on the experiential knowledge of its members and the centrality of individual rights arguments represents a distinctive approach to political claims making (Jewell and Bero 2007). Similarly, one state's effort to expand Medicaid insurance coverage, but only for evidence-based interventions, generated considerable opposition from advocacy groups for disabled people and people with AIDS. These groups feared that when determining the evidence-based services that provided a social benefit justifying coverage, the government would value their lives less and so their health needs would end up at the bottom of the list.

Medical professional groups, hospital associations, and insurance companies have used similar non-evidence-based arguments regarding coercion, choice, and the sanctity of the doctor-patient relationship to oppose efforts to reform Medicaid. An effort to institute a system to use encrypted state health data to analyze variations in medical practice—an issue that does not affect consumers directly—met with significant opposition on the grounds that “even if you don't identify the individual, you somehow have violated the person because you were looking at the medical record.”

Issues of privacy, choice, and the differential valuation of individuals (real or merely perceived) are crucial concerns for any health policy decision. Yet owing to its nonevidentiary way of defining policy issues, rights talk can become an obstacle to introducing research.

Interest Groups

Interest groups can greatly influence policymakers, often in ways that hinder evidence-informed decision making. Interest groups can inundate the policy setting with bad-quality evidence, champion poorly designed studies, and limit the critical analysis of information through the social relations they develop with officials.

Many health industry associations have substantial resources for lobbying efforts in individual states. Of these groups, pharmaceutical companies were mentioned the most frequently by our respondents. One legislator commented that the drug industry had recognized the need to focus more on the states, and as a result, he estimated that there was one drug lobbyist present for every two legislators. This presence was seen to hinder evidence-informed policymaking in several ways. Drug lobbyists provided legislators with large amounts of data to support their particular products. But even though this information was often presented and accepted as scientific research, it consisted primarily of seeding studies, which, “with fifteen, twenty, thirty people … are absolutely meaningless.” Seeding studies are small studies used to familiarize doctors with a product rather than rigorously testing it. In addition, some legislators' limited critical appraisal skills made it difficult for legislators with adequate training to convey to them the inadequacies of this kind of information. As another official noted in despair, “[My colleagues] don't know enough to be able to connect those things … sometimes [lobbyists] keep saying lies like that often enough and after a while people start believing them. … And I can't spend all my time trying to discredit people.”

Similarly, consumer advocacy groups active in many state health policy issues sometimes tout untested interventions as effective treatments. Indeed, these interest groups have sometimes become vehicles for bad information from charlatans selling dubious cures to desperate families, for example, treatments for autism:

You know some of them are just bizarre. Vitamin therapy, cranial sacral therapy. Anti-yeast therapy. Gluten-free diet. You know some of these are actually harmful, and while some aren't, they are just a waste. … And I think it's easy to convince a parent, “Well if you just put them on all of these vitamins,” and “Oh, by the way, I sell these.”. … I mean these families, they're all moms. They apparently spend all their time on the Internet. And they come up with these studies where there's like five kids.

The positions of professional associations can be difficult to refute as well. Their assertions are seen as credible, and their ability to mobilize constituent support gives policymakers pause in opposing them. “Everybody likes nurses and they come to an issue with a high degree of credibility … so when the lobbyist from the nurses comes around and tells you that [this bill] is going to hurt frail elderly people, a lot of my colleagues are predisposed to believe that.” Other examples mentioned were back surgeons and ob/gyn doctors who opposed removing coverage for procedures that research had showed to be ineffective.

Perhaps an even greater source of influence is the familiarity and rapport that some lobbyists are able to form with individual policymakers. These social relations give such interest groups' claims an emotional weight that may be difficult to counter by evidence alone:

I'm not sure they talk that much about specific drugs. They just take these guys and gals out to dinner over and over again, so they become, sort of more or less personal friends. And then, you know, they can walk up to them and say, “I really need your help on the pseudoephedrine bill. You know, pseudoephedrine's really not a problem. It's going to hurt our company and it's going to hurt your choices.” … They're just back there in the background. They spend a lot of money and they give a lot to campaigns.

Political Values

Conflicts over fundamental political values concerning the proper role of government also can often limit the relevance of evidence to the decision-making process. The religious views of a considerable portion of one legislature were described as responsible for a failed attempt to expand Medicaid family-planning services. Legislators' strong feelings about offering abortions at free clinics precluded their consideration of evidence that Medicaid coverage for family-planning services actually reduced the use of free clinics. States with fiscally conservative legislatures also have more difficulty enacting some health-promoting policies, such as increases in the tobacco tax or coverage of dental care for children. Likewise, the libertarian tradition of many western states was mentioned as creating a baseline skepticism of arguments for any new regulations, as well as hindering the development of collective, multistate arrangements such as drug purchasing. In its most extreme form, the minimal government view effectively precluded health policy deliberation of any kind.

If the leadership in your governing body believes that government should have as limited a role as possible in providing health services to your state population, then it doesn't matter what you say. … They don't care about evidence because they don't believe that government should have a role in providing health care.

Interviewees also often discussed political values in regard to individual legislators who cared about a single issue, such as abortion, and were not interested in any other issues. But the interviewees also acknowledged that more liberal politicians were sometimes uninterested in research evidence because of a general faith in government programs.

Attacks on an Evidence-Based Approach

Several officials also discussed instances in which the whole notion of evidence-based health care had come under direct attack, usually by pharmaceutical companies, sometimes in collaboration with advocacy groups, some of which hid their involvement with industry. This had occurred when states attempted to develop preferred drug lists using data from the Drug Effectiveness Review Program (DERP), which was established by Oregon lawmakers in 2001 to commission systematic reviews of drug classes as a basis for creating a drug formulary for their state Medicaid program. As of September 2004, DERP had become a consortium of fifteen states and two nonprofits and had completed twelve reviews, with ten more in progress (Fox 2005). Among other strategies, interest groups mischaracterized DERP's methods of analysis, asserting falsely that it ignored observational studies or that it considered a lack of evidence to be evidence of no difference. This latter argument involved conflating the scientific evidence with the policy decisions that particular state legislatures had made based on that evidence:

In the Oregon process, when they were faced with a choice among opioid analgesics, they found that there was no comparative evidence among any of them and said, “We're not going to go with the most expensive one. There's no evidence that it is any better. So we're going to go with the least expensive one” … and [the pharmaceutical industry] is trying to use what Oregon did with the information to discredit the DERP process altogether.

Drug lobbyists also attempted to conflate the scientific and policy issues by tying the evidence from DERP to Oregon's own state prescription guidelines, which are less restrictive than those in many other states (Fox 2005; Gibson and Santa 2006). As a result, officials in at least two states chose not to use DERP data, based on concerns about other Oregon policies, as a legislator from another state explained:

Within the state of Oregon … any physician can override or can simply check on the script—“No, you have to put down Vioxx or whatever I've prescribed here.” … In our preferred drug list … physicians can't override it. … And I know the industry tracks those things and they would say, “We're glad you're joining Oregon [in using DERP evidence], but by the way, senator or representative, we want you to know that in Oregon physicians can override.”

At the same time, many groups try to enhance the credibility of their arguments by using evidence-based language and formatting in a misleading way. This may be as simple as footnoting a group of slides as if it were a publication, as was reportedly done by disease management program vendors; or it may involve creating their own evidence-based website, as in the case of chiropractors. The respondents also discussed more elaborate forms of deception, in which interest groups created their own research institutes, conferred phony academic degrees on their members, and produced reports in which their data and research citations were from their own or allied organizations; that is, they “cook their own evidence and their sources are themselves … and [then] send it to people as ‘Here's the evidence.’”

Facilitators of Evidence-Informed Health Policymaking

Concretizing Impact

It will come as no surprise that officials universally characterized research-based evidence as neither a necessary nor a sufficient part of the policymaking process. It must be packaged to incite and persuade, “to translate the evidence into something that is understandable by the average legislator, average citizen.”

For example, particular kinds of research evidence are especially resonant in the political context, namely, evidence that is able to concretize impact. General health arguments tended to be less effective than those asserting particular benefits or harms. Concise statements about lives or money can infuse the political discussion with a tone of rationality, framing the trade-offs as technical and straightforward. For example, one official asserted this quantifying feature as a key asset of the Cochrane Library's systematic reviews: “The Cochrane … showed us that medications and procedures we use … can really be measured. … So I think the concrete aspect … could be used to identify where we could best spend our money.”

One of the most important strategies for effectively conveying research evidence is delineating the effects for specific individuals or groups. This approach personalizes the policy case, thereby making it easier for legislators and the public to relate to it.

In a successful effort to rescind a Medicaid spending cap, a legislator used health plan data to demonstrate that the individuals most frequently affected had chronic conditions such as mental illness, arthritis, and cancer. Officials presented personal testimony to demonstrate how the policy would affect a sympathetic group whose circumstances were not under their own control.

You take data that's good data and then you back it up with a human face … because then you've got both the logic and compassion going for you as part of the argument. And I think that combination is powerful. … Just hearing about how many people aren't being treated when they have breast cancer … versus looking at somebody in the eye, a young woman, who has paid premiums for ten years … [who] now all of a sudden can't get what she needs to save her life.

Linking Research on Health Effects to Costs and Benefits

Because of the intrinsic complexities and ambiguities associated with policymaking, political debate is often a struggle over the meanings of concepts and the dominant metaphors used to depict a policy situation (Stone 2002). The increasing interest in and visibility of evidence-based decision making as an approach coupled with acute financial situations often allowed officials to frame their initiatives as opportunities to make health services more effective for less money. In one state, the successful passage of Medicaid reform that involved implementing more standardized, evidence-based practices was able to overcome significant opposition by tying cost containment to expanded coverage opportunities. This played well with public opinion “because we were promising … to get universal health care, and we were going to keep the costs down. Both of these things the public wants badly.” Furthermore, by specifically linking the benefits of reform to expanded services for children, the legislators also facilitated the end of medical opposition (which initially saw the reform in terms of practice restrictions). “[The pediatricians] saw that we were going to make health care more available to children. And they decided that whatever else we did, it was their job to help children. So they came out in support, the first group of doctors.”

Carefully defining the costs associated with policy inaction can also reveal previously unrecognized burdens for particular groups, shifting their view of their own interests as well as their political behavior. For example, in a tobacco control effort that had failed earlier, partly because of an insufficiently compelling argument presenting statistics on the harm of tobacco, a second attempt was successful when studies of the reduction in heart attacks from smoking bans were used to specify the impact of cost on health insurance premiums. By using this information, policymakers were able “to attract a usually nonparticipant in the tobacco control dialogue within the community,” that is, business owners.

Creating accounts that link costs and benefits across disparate policy areas is another way to frame research in politically resonant terms. For example, in one state officials were able to introduce a statewide infant home nurse visitation program. The evidence base was unusually strong, with a randomized controlled trial design, and quantified impact along multiple dimensions demonstrating a substantial reduction in teenagers' violent, antisocial behavior. But the evidence alone would not likely have been sufficient in this state, which had a political culture averse to regulation and government. Rather, the successful passage of the program hinged on timing and opportunistic framing. As one policymaker argued, the future social costs that would be averted by such a program would eliminate the current need to expand prison capacity. “We were debating another prison, and how … we [were] going to pay for [it], and here I come along with something that's going to prevent that sort of thing. And with good research to prove it.”

Characterizing innovative ways in which health costs can be addressed can also be persuasive by identifying feasible and practical means of control and the actors responsible (Stone 1989). For example, one state tied risk factors of tobacco use, obesity, and lack of physical activity to their associated costs to the state health insurance plan. The legislators then argued that the governor, as the equivalent of the CEO of the “state corporation,” should encourage employees to take a more active role in their health management by tiering health insurance premiums and covering preventive care.

(Re)-Framing Policy Issues to Resonate with Existing Evidence

Because research rarely addresses pressing policy issues directly, another important strategy is reframing the issue in ways that incorporate the best available evidence. For example, in attempting to pass legislation to allow radiological technologists (rather than only MDs and RNs) to inject contrast medium, one policymaker could find no evidence concerning the relative safety of this class of medical personnel but did find information that dyes had become much safer: “For some reason, none of the groups advocating for the bill had made that point … people were interested to know. ‘You mean this stuff … occasionally used to cause allergic reactions but the new stuff doesn't? Oh that's important.’” In another case concerning vending machines in schools, rather than arguing about the narrower issue of whether vending machines cause obesity (for which opposition focused on a lack of direct evidence), one official reframed it in a way that incorporated the more general causal link between soda consumption and overweight kids, namely, that vending machines were a vector, just as mosquitoes are for the transmission of malaria:

When we discovered the cause of malaria, we did not have to do a study about whether we should kill mosquitoes or not. … At least it caused the political leaders to pause and think. … So the policy argument was that the vending machines themselves are a vector for the risk factors that we did have knowledge about.

Evidence-Based Skills Training

Educating administrative officials who can then introduce new decision-making approaches to their agency is one important way to effect systemic change. Six administrative officials talked about how evidence-based training had led them to make significant changes in the practices that agency personnel followed. Developing in-services in which staff researched actual policy issues resulted in a more systematic and sophisticated approach to internal health policy analysis.

Other ways in which administrative officials used evidence-based skills training to alter how work was done in their agencies were (1) building up in-house staffing capacity based on the official's acquired understanding of the necessary skill set for evaluating research; (2) instituting a standard response asking applicants to the agency to produce a randomized controlled trial of their product; (3) introducing formal procedural changes that require evidence in the decision-making process; and (4) conducting their own studies. Such changes had facilitated administrators' abilities to fend off coverage demands that were not supported by research evidence, including hyperbaric treatment and bariatric surgery for children.

One administrator also used her evidence-based skills training to educate an executive working group that was created to identify promising mental health practices. Completing this task posed considerable challenges owing to the size and heterogeneity of the group and the large number of putative interventions that needed to be assessed. By introducing the group to criteria for evaluating evidence from the start and applying them initially to uncontroversial matters, the official was able to make members comfortable with using an evidence-based approach.

Training legislators also was reported to be helpful. Their daily work, though, rarely included personal involvement in the collection and synthesis of research. Instead, their evidence-based skills were most useful for evaluating information presented by interest groups and other legislators. Because most politicians do not have a scientific background, simply realizing that “just because there was an article in a journal doesn't make it true” and “just how many bad studies there are out there” was described by many as being a tremendous insight in itself. But training also made them better equipped to investigate basic research quality issues such as whether a study was a randomized controlled trial, whether it was replicable, who funded it, and where it was published. This ability to challenge assertions by interest groups was deemed especially important in public settings. As one official commented, when dealing with interest groups who use the trappings of scientific credibility, such as presenting a published paper from a putative research institute, “you've got to find the inconsistencies in that article and rip it up right in front of their eyes. That's the only thing you can do. … Sometimes you might have other evidence, such as from the CDC, and people would just shut up at that point.” Another noted how a few commonsense observations could sometimes readily disarm the opposition's position:

Being able to go back to who paid for the study. … I didn't have to prove that it wasn't credible. I just raised the issue of can you trust something that was paid for by the insurance industry … and you know, why all of a sudden are these [findings coming to light now] … have they been replicated in other studies?

While evidence-based skills training would be difficult to implement on a large scale, most of the legislative officials we interviewed were quite sanguine about more modest efforts that targeted key political leaders like themselves. The interviewees were often chairs or senior members of important health and finance committees, and many discussed the advantages of residing at these fulcrums of policy power, including the greater staff/research capacity and the ability to act as an information clearinghouse and significantly influence how issues are framed and discussed in legislative sessions. The importance of having health committee chairs that were “interested in evidence” as well as political leadership “committed to an informed decision-making process” more generally was frequently mentioned as contributing to the successful passage of evidence-informed health policy. The impact of trained health policy leaders could also be felt in the political process through their ability to mentor other legislators without the requisite research skills.

Instituting Research-Focused Venues

Another major facilitator of using evidence was officials establishing decision-making processes whose core tasks were data collection and evaluation. For example, advisory committees provided a setting in which the evidence regarding drug policy issues could be more readily deliberated than typically through the legislative process directly. One official described the importance of using this more insulated means to create a state formulary:

I had heard from … colleagues of the difficulties they had had politically getting a [state formulary] done, because the drug industry doesn't like them. … The drug companies brought in a presenter [who was] talking about things like excessive utilization and conflicting ones … as an alternative to a preferred drug list. … And so what we did was put a footnote on the budget saying we wanted the Medicaid program to look into ways that you could save money in the prescription drug area … like we were talking about their guy. And then afterward we told the health department this means a preferred drug list … they put together a committee of local experts and … that's the basis for our list.

Although pharmaceutical representatives were not excluded from such processes, they were forced to argue in scientific terms. As an official from another state explained: “[The Advisory Committee] don't allow anecdotal stories. They don't allow any research projects that have not been reviewed. And it has raised [the discussion] to a different level. … The first few meetings the manufacturers brought their salespeople. … Now they bring their science officers.” Instituting a successful process was associated with several features, such as a transparent decision-making setting that openly solicited information from all interested parties; access to a large body of synthesized information, such as from DERP; and a committee with the appropriate content knowledge and methodological skills, including individuals who could do the “actual application of the evidence” (often DERP staff).

Another example of efforts to create technical capacity in the legislative process was the creation of policy working groups or commissions. These provided officially recognized settings for research gathering, and their official status enabled their findings to be more easily incorporated into the political debate. For example, one state's health care commission was responsible for reporting on the medical malpractice system, with contentious vested interests on all sides. The commission amassed evidence demonstrating significant problems with the system, in particular that victims received only a small percentage of the money generated by the process and that the determinations had a high rate of error. Perhaps even more important was the commission's ability to identify innovative solutions, such as administrative compensation systems similar to workers' compensation used in the Scandinavian countries and New Zealand. Other policy working groups mentioned designed a health reform plan and developed a new child care initiative. Still another example of strengthening a research-focused venue was a legislator's success in changing an interim committee on health policy to a “standing” permanent committee that met regularly and had additional staff.

Generating and Sharing Information through Collaborations

One of the most successful facilitators of evidence-informed health policymaking was the presence or development of collaborative efforts that addressed systemic issues limiting the pool of credible, synthesized information. Officials frequently mentioned the importance of having access to trusted groups to whom they could turn to provide data or do research or policy analysis. Legislators especially cited the importance of having sources who could respond rapidly as issues arose, “who you could pick up the phone and call,” including local universities, policy studies centers, hospitals, health plans, and some national health advocates (e.g., the National Diabetes Association).

Officials also were often involved in creating new ways of generating, disseminating, and maintaining relevant information so that they did not need to rely on service providers or the federal government. In one case, state legislators helped create a public health foundation to conduct its own independent studies on health policy issues affecting the state. For example, this foundation conducted a study demonstrating that the state's shift from a no-fault to a fault insurance scheme had created significant financial difficulties for medical providers. Whereas insurance claims had previously been paid quickly and efficiently, now many of these claims were caught up in lengthy court battles concerning liability.

The DERP collaboration had also been an important learning experience for many policymakers and had created the impetus for further interstate collaborations. Several states were using the existing research consortium in nondrug medical areas, including to develop better access to existing technology assessments and to commission new ones in emerging areas.

An even more ambitious aim of this and other interstate collaborations is using state health data to begin filling in the gaps in the evidence, to become “evidence producers as well as consumers.” Officials discussed, for example, states' growing efforts to collect information about long-term care insurance and Medicare Part D, as well as initiating medication pilot projects in state hospitals to assess the comparative effectiveness of statins.

Creating accessible sources of consolidated research evidence was seen as critical not only to how health policy was made at the legislative or agency level but also to the health service provider and the individual doctor in treatment decisions. Such information systems can seriously disrupt the non-evidence-oriented arrangements through which both policymakers and physicians receive much of their information, that is, one that allows “industrial interests to take selectively chosen research directly to [policymakers and doctors].”

Administrators and legislators also frequently mentioned studies based on other states' data as a key source for identifying new health policy issues as well as potential solutions to well-established ones. For example, an effort to promote home-based care as an alternative to nursing homes was bolstered by another state's study showing that up to 20 percent of the current nursing home population could be cared for in other places. Such evidence was also useful in the political realm to demonstrate feasibility, cost-effectiveness, and a lack of harmful or unintended consequences. This was the case, for example, in introducing best-practice guidelines that had been tested in another state and in advocating for tobacco control policy by using evidence of the continued prosperity of restaurants where smoking bans had been passed.

Often the channels for disseminating research from a single state were through administrative officials with experience in several states. For example, previous experience with research in another state's Medicaid system helped one official replicate its methodology to identify the cause of unexpectedly high expenditures for community mental health services under the state's children's health insurance program (CHIP).

Opportunities for officials to obtain high-quality research from policy experts as well as from one another also were important. One of the collaborations most often mentioned was the Milbank Memorial Fund's Reforming States Group (RSG). Founded in 1992 around five states' early efforts at health care reform, this bipartisan nonprofit group has grown into an organization of health policy leaders from executive and legislative branches in all fifty states and several provinces in Canada. They have published recommendations on a range of issues, including state oversight of integrated health systems, federalism and health system reform, and prescription drug policy, and the organization has developed a reputation as a reliable source of information in federal politics (Andersen 1998; Fox and Greenfield 2006; RSG 1998, 2003). While the impact of RSG on policymaking has not been evaluated, officials discussed its impact on various access and coverage issues and “the adopting of evidence based standards,” such as in state formularies, CHIP, and mental health coverage. The organization has also helped facilitate collective policy responses including the development of DERP and multistate drug purchasing.

Conclusion

This article has analyzed the “real-life” accounts of experienced state legislators and administrators in order to identify factors that affect how evidence informs state health policy decisions. Certain hindrances to incorporating research are systemic and not readily altered. For example, the increasing policy responsibilities and decreasing budgets of part-time state legislatures and state health agencies limits their capacity to identify, evaluate, and use research.

Our findings also suggest, however, a number of barriers and facilitators of evidence-informed health policymaking that are amenable to change. Policymakers often lack the skills to identify and use high-quality research and thus could benefit from evidence-based skills training. The best-quality evidence may sometimes be obscured by bad-quality information that is disseminated by interest groups. Accordingly, skills training could help policymakers and their aides not only identify relevant research but also distinguish research of high and low methodological quality. As one interviewee noted, such training could help them develop “good taste in evidence.” Targeted, evidence-based training of public officials in leadership positions could provide a powerful means for influencing how research is used and how policy issues are framed in larger legislative and administrative settings.

Even those individuals actively seeking the best evidence concerning an issue often have difficulty obtaining it because of problems with research quality, accessibility, and applicability. Thus, both policymakers and researchers could develop and evaluate mechanisms to improve the methodological quality, accessibility, and applicability of policy-relevant research. The policymakers we interviewed emphasized creating partnerships to generate and share information and ideas with existing research organizations, as well as tapping into existing state data structures and political knowledge to create new research efforts. Groups like the Drug Effectiveness Review Program (DERP) and the Reforming States Group (RSG) that have slowly developed over the past five to fifteen years offer institutional capacity and political legitimacy for the further development of interstate collaborations. Our findings also suggest that developing research-focused venues within the political process—for example, working groups and advisory committees—can provide important sites for thoughtful evaluation research that is pertinent to complex health policy issues.

Officials also discussed the importance of incorporating high-quality research into compelling policy accounts that highlight features for which good evidence is available, that concretize and personalize impact, and that resonate with important societal costs and benefits. Anecdotes that are not linked to evidence, appeals to emotion and common sense, bare assertions of politically credible groups, and other nonevidentiary ways of defining policy issues (rights, family values, minimal government interference) are prevalent forms of persuasive political discourse that can be directly challenged with research evidence.

The major limitation of our qualitative study is that some of our findings may not be generalizable to all policymakers. However, our interviewees' political experience and sensitivity to evidence utilization made them excellent insider informants of the political and regulatory factors facilitating or hindering the use of evidence in the policy settings in which they work. Future research can rigorously test whether some of the interventions suggested by our respondents systematically increase the use of research evidence in policy decisions.