Examining the Role of Health Services Research in Public Policymaking

Policymakers in the health care sector have begun to ask how to make better use of health services research in developing public policy (Buxton and Hanney 1996; Eisenberg 1998; National Forum on Health 1998). Researchers have begun to provide some tentative answers to this challenging question (e.g., Coburn 1998; Davis and Howden-Chapman 1996; Eisenberg 1998; Feldman, Gold, and Chu 1997; Frenk 1992; Ginzberg 1991; Gray 1997; Klein 1997; Peterson 1995, 1997; Roos and Shapiro 1999; Soumerai, Ross-Degnan, Fortress, et al. 1997). To paraphrase the title of a recent article, many policymakers and researchers now talk of “the paradox of health services research: if it is not used, why do we produce so much of it?” (Shulock 1999).

Making better use of health services research in developing public policy requires that both health services researchers and public policymakers have realistic goals. Working through the conceptual, methodological, and practical issues that confront those who study the role of health services research in public policymaking can help determine these goals. We therefore have used an issue-based framework to organize this article. We begin with “looking for research use in all the right places” (i.e., identifying a sample of policies), then turn to “knowing what you're looking for” (i.e., identifying the research use), and, finally, “finding patterns in when research is used and not used” (i.e., identifying the conditions under which the research is used). We draw on several research traditions to support our approach: organizational behavior and management research (e.g., Cohen, March, and Olsen 1972; Langley, Mintzberg, Pitcher, et al. 1995; Lindblom 1959), the multidisciplinary field of knowledge utilization (for a review of empirical studies, see Beyer and Trice 1982), and political science research (for a review of theoretical frameworks, see Lavis 1998).

We illustrate these issues with an exploratory study in which we examined the role of health services research in Canadian provincial policymaking. Policymakers in two Canadian provinces—Ontario and Saskatchewan—expressed interest in understanding whether, how, and under what conditions health services research affects provincial policymaking and in using this information to improve their interaction with government-funded independent health services research units. Many of the political and economic features of health care policymaking in the two provinces are the same, although the scale of the expenditures in Saskatchewan is much smaller, given that its population is one-tenth that of Ontario. The provinces represent the most appropriate jurisdiction for studying health care policymaking in Canada because they have formal constitutional authority over most issues pertaining to health care, especially its organization and delivery (Health Canada 1997). Compared with American state governments, Canadian provincial governments are responsible for a much larger share of health care expenditures.

Looking for Research Use in All the Right Places

Some policies and some policymaking processes may be particularly amenable to being informed by research. If the prospects for making better use of health services research in developing public policy differ by type of policy, then presumably so too should the researchers’ and policymakers’ goals for using the research. To determine whether this is so, those who study the role of research in policymaking must decide how best to identify a sample of policies. One way would be to specify a sampling strategy (e.g., random sampling) for a universe of policy decisions that is not biased toward particular types of policies or policymaking processes. This approach, however, relies on chance to achieve variation across types of policies and policymaking processes.

A second way would be to specify a sampling strategy for a typology of policies that could be applied to each policy category (table 1, row 1). Of the many policy typologies (Dubnick and Bardes 1983), Lowi's typology is one of the best known (Lowi 1964). His typology incorporates both a policy's intent and the policymaking processes that typically accompany a given intention. He distinguishes among three types of policies: a distributive policy that is a governmental decision to provide specific benefits to specific groups without regard to limited resources, a regulatory policy that is a governmental decision as to who will be indulged and who will be deprived on the basis of some general rule, and a redistributive policy that is a governmental decision involving broad categories of citizens to whom benefits are extended or from whom losses are taken. Lowi's typology, like many others, has been criticized for its ambiguity and incompleteness (e.g., Wilson 1973).

TABLE 1.

Approach to Identifying a Sample of Policies

Step	Examples of Available Options	Our Approach
1. Select or create a typology of policies (i.e., create policy categories).	• Functional (i.e., roles or services performed for society or sector).	• Functional categories relevant to health care sector:
	• Intentional (i.e., purposes, goals, or objectives).1	• Jurisdiction/governance (i.e., establishment of jurisdictional responsibilities and accountabilities).
	• Population-focused (i.e., actions and statements that benefit or harm specific groups).
	• Programmatic (i.e., part of a “package” of similar or related policies).	• Financial arrangements (i.e., financing, funding, and remuneration arrangements to support services).
		• Delivery arrangements (i.e., how services will be delivered, by whom, and in what settings and how services will be accessed).
		•Program content (i.e., which services will be provided and to whom).
2. Identify policies from each policy category using available sources.	• Policymakers (i.e., legislators or policy advisers).	• Policy advisers in each province were asked (by the senior policy advisers on our advisory group) to identify eight policies developed over the last seven years, at least one of which was from each of the four policy categories.
	• Researchers (i.e., those whose research may have been used in the policymaking process).
	• Documents (e.g., list of proclaimed legislation).
3. Select policies in each policy category.	• Purposive sample.	• Purposive sample of four policies from each province, with one policy selected from each policy category to achieve as much similarity across provinces as possible.
	• (Stratified) random sample.
4. Assess the representativeness of the selected policies across dimensions not covered by the typology.	• Type of policy (in terms of purpose, goals, or objectives) or policymaking process.	• Type of policy (coded as distributive, regulatory, or redistributive).2
	• Extent of benefit or harm to specific groups.	• Public significance, which is a concept analogous to the visibility of a policy3 (coded as high if large change, near-term impact, and/or prominent groups affected; low otherwise).
	• Part of a “package” of similar or related policies.
	•“Go” decision (i.e., policy change) versus “no go” decision (i.e., considered a policy change but decided against it).
	• Level at which decision was made (e.g., president, Congress, head of an executive agency, staff of an executive agency).	• Stakeholder impact (coded as high if large part of system affected or large change within a part of system; low otherwise)
		• Part of or embedded in a larger policy trajectory with a common policy framework (coded as embedded or stand-alone).
		•“Go” versus “no go” decision.

¹Dubnick and Bardes 1983, 85.

²Lowi 1964.

³Arnold 1990.

Even when policy categories have been selected or created, the battle is far from won, for the policies within each category still must be selected (table 1, rows 2 and 3). Both policymakers (by which we mean both legislators and policy advisers) and researchers whose research may have been used in the policymaking process are not well positioned to identify all the policy decisions made in a given time period, because of either recall bias (for policymakers) or access restrictions (for researchers). Unfortunately no routine reporting mechanism for policy decisions exists, either. Public records are available for legislative changes and media communications, but these do not cover all policies. Annual reports for all government departments also are available, but these focus largely on financial reporting and general departmental missions, not on specific policy content.

Regardless of the sampling approach chosen, the representativeness of the selected policies for each dimension not covered by the chosen typology still must be determined (table 1, row 4). The extent to which a policy is part of a “package” of similar or related policies is an example of one such dimension. The field of organizational behavior and management research has suggested that it may not be possible to isolate more specific policy decisions from broader decision trajectories (Langley et al. 1995; Mintzberg, Raisinghani, and Theoret 1976; Mintzberg and Waters 1990). Therefore, studies of discrete policy decisions may not cover the many roles of research in policymaking. Whether the sample includes “no go” decisions as well as the more easily identifiable “go” decisions is an example of another dimension. Many “no go” decisions—airing an issue at a meeting but deciding not to raise it in a briefing note, identifying an issue in a briefing note but advising not to examine it further, and/or writing a report about a set of issues but advising that no palatable policy alternatives exist beyond the status quo—can be obtained only with the policymakers’ consent and participation.

Where We Looked for Research Use

When we could not find a commonly accepted typology of health care policies, we created our own, comprising four policy categories (table 1, row 1). Although each of these categories contains a number of issues that could be informed by health services research, the existing typologies of health services research (e.g., Berwick 1989; Eisenberg 1998), which mix issues and methods, do not overlap perfectly with this typology. These policy categories ensured at least some variation in the scale of the policy change: from large-scale changes (governance, or the rules of the game) to small-scale changes (program content, or whether or not to offer a particular problem or service).

Finding no comprehensive policy inventory from which we could sample policies within a policy category, we identified policies in a novel way, through policy advisers, and examined the consequences (table 1, row 2). The main inclusion criterion—the policies had been considered and a decision reached (but not necessarily fully implemented) in the seven-year period between 1992 and 1999—was chosen as a compromise between the study team's preference for a relatively short time frame to minimize recall problems and the Saskatchewan advisory-group members’ preference for a sufficiently long time frame so that policies related to regionalization in their province would be eligible for selection. Regionalization involved the devolution of much decision-making authority over health services to elected boards at a district (i.e., subprovincial) level (Lomas, Woods, and Veenstra 1997).

We selected four policies from each province, one from each policy category, from the list of 20 potential policies submitted by the two health departments (table 1, row 3). Within each policy category, we tried to achieve as much similarity across provinces as possible (app 1). Both the policies related to jurisdiction and governance used a specific governance strategy to implement a broader policy direction. And both the policies related to financial arrangements established a specific payment formula for a health care provider (physicians in Ontario and health districts in Saskatchewan). Both the policies that established delivery arrangements used a classification system for the long-term care population to facilitate (at least in part) client assessment and service delivery. Finally, both the policies pertaining to specific programs extended an already available service provided primarily by physicians for a targeted population.

To discover any biases resulting from our sampling procedure, we examined specific characteristics of the selected policies (table 1, row 4) and concluded that for two reasons it is unlikely that these policies represented the full range of health care policies being considered or developed (app 2, row 1, bullet 4). First, all the potential and selected policies were low in profile. None of the selected policies (and only two of the 20 potential policies) was coded as publicly significant. Only two of the selected policies (and five of the 20 potential policies) were coded as having a significant impact on stakeholders: one involved delivery arrangements in Ontario; the other, financial arrangements in Saskatchewan. Second, all the selected policies, like the longer list of potential policies from which they were selected, involved “go” decisions. These “go” decisions may have represented policies that had been developed, but not those under consideration, many of which would have culminated in a “no go” decision.

Other characteristics of the selected policies were perhaps more representative than not. All were regulatory policies but because, according to Lowi's (1964) formulation, regulatory politics typically play out in specific sectors of the economy like health care, this should not be surprising. We cannot think of a distributive policy in this sector, and the available examples of redistributive policy (e.g., provincial health-insurance plan coverage and user charges) are rarely employed on a scale that would generate the kind of redistributive politics that Lowi describes. We would have had to go back more than 30 years to the birth of Canadian Medicare to find a good example of a redistributive policy. In addition, although more than half the selected policies were embedded in a larger policy trajectory (as identified in app 1), this does not strike us as atypical. Interestingly, these larger policy trajectories had a higher profile than did the constituent policies that were offered for study.

Reflections on Where to Look for Research Use

One way to balance the need to identify policies through policymakers in the absence of routine reporting mechanisms and the apparent propensity of these policymakers to offer low-profile policies—none of which were “no go” decisions—may be to study the process concurrently (i.e., in “real time”). From a policymaker's perspective, it may be easier to let researchers “watch” the decision-making process as a participant-observer than to reveal information after a decision has already been made. By attending meetings and reading briefing notes and internal reports, researchers could then begin to identify the universe of policies (or, more accurately, the universe of decisions). The act of exposing the policymaking process to research scrutiny may fundamentally alter it, however, and may raise concerns about the generalizability of the findings.

A reasonable approach to resolving the problem of mixing embedded and stand-alone policies in the same analysis would be to change the sampling unit from policies to policy trajectories. A policy trajectory can be considered a policy in its own right (albeit, a metapolicy), with policy goals and priorities that can be informed by research. A participant-observer could thus “move beyond the decision and consider the organization as a system of decisional processes” (Langley et al. 1995, 270; italics in original) while carefully considering what constitutes the components of these decisional processes. The policymaking process could then be studied at the level of both the policy trajectory and the constituent policies.

Our finding that more than half the policies being studied were embedded in broader policy trajectories has an important implication for policymakers. Policymakers should step back periodically to ask whether research could inform the particular policy (or decision) facing them at a given moment. Once a policy trajectory is set in place, the constituent policies may be considered as “givens” and not be considered as separate policies that are addressing unique issues and could be informed by research. This research could be different from the research that pertains to the policy trajectory as a whole, or it could be the same research but worth examining in a different light.

Knowing What You're Looking For

Different conclusions about the extent to which policymaking is (or should be) informed by research may arise from different views about what constitutes health services research (table 2, row 1). For example, should we include only research that is published in a publicly available form? An alternative would be to exclude research produced by particular types of organizations, such as marketing research firms, management consulting firms, and membership-based professional organizations. A less blunt but nonetheless (for some) disquieting alternative would be to require that the research meet particular methodological requirements. The default would be to avoid restrictive definitions, as Lindblom and Cohen (1979) did when they examined the role of professional social inquiry in social problem solving and included all investigatory activities.

TABLE 2.

Approach to Identifying Research Use

Step	Examples of Available Options	Our Approach
1. Identify what constitutes research.	•“Citable” research (i.e., research that is published in a publicly available form, such as journal articles, book chapters, working papers, and/or reports).	• Citable research (which covers the types of research outputs typically produced by health services research units).
	• Research produced by particular groups.	• Citable research produced by government-funded independent research units.
	• Research meeting particular methodological standards.
	• Any professional social inquiry that can aid in social problem solving.1
2. Identify explicit uses of research.	• Policymakers state in an interview or survey that research was used in some of the prioritization (i.e., agenda-setting), policy-development, or policy-implementation stages of the policymaking process.	• When describing in an interview the prioritization and policy-development stages of the policymaking process, policy advisers mention that citable research was used in one of those stages.
	• Researchers state in an interview or survey that they believe their own research was used in some of the stages of the policymaking process.	• Sources for corroboration:
		• Documents used in one of these two stages of the policymaking process cite research.
	• Documents used in some of the stages of the policymaking process cite research.	• Research-unit directors state in a survey that they believe that “citable” research produced by their unit was used in one of these two stages of the policymaking process.
3. Assess the explicit uses of research.	• Policymakers describe in an interview or survey:	• Policy advisers mention in an interview:
	• How they accessed the research.	• How they (and/or the legislators they were advising) accessed the citable research (coded as reading original research, reading reports produced by policy advisers or interest groups, media, interacting with researchers, involving researchers in a working group, or attending hearings).
	• How they used the research.
	• The proportion of policy issues addressed by the research.
		• How they used citable research (coded as instrumental, conceptual, or symbolic uses).
		• The particular policy issue(s) addressed by the citable research (coded as informed all or part of the policy issues being addressed in the policy stage).
4. Identify and assess explicit uses of other types of information (which may or may not be based on research).	• Policymakers state in an interview or survey that other types of information were used in some of the stages of the policymaking process.	• Policy advisers mention in a semistructured interview:
		• Other types of information used in the prioritization or policy-development stage of the policymaking process (coded deductively).
	• Policymakers describe in an interview or survey:
	• How they accessed the other types of information.
	• How they used the other types of information.
	• The proportion of policy issues addressed by the other types of information.	• How they accessed the other types of information (coded as reading, media, interacting with peers or stakeholders, involving peers or stakeholders in a working group, or attending hearings).
5. Identify nonexplicit uses of research.	• Researchers identify in an interview or survey that they believe their own or others’ research was embedded in other influences on the policymaking process.	• Not addressed in exploratory study.
	• Research embedded in other types of information.
	• Research embedded in “tacit knowledge.”2
	• Research embedded in the stakeholders’ positions.
	• Research embedded in existing institutional arrangements.

¹Lindblom and Cohen 1979.

²Polanyi 1966.

Different conclusions about the extent to which policymaking is (or should be) informed by research may also arise from different views about what constitutes research use. Certainly we would want to include an explicit use of research (table 2, row 2), as when a legislator cites an article in the New England Journal of Medicine as having put an issue on the policy agenda. We may also include some uses of other types of information, such as when a policy adviser draws on the experience of colleagues in other jurisdictions whose earlier decisions had been informed by research (table 2, row 4). What about tacit knowledge (or knowing more than we can tell), which plays an indispensable role in shaping or integrating our more explicit and particular knowledge (Polanyi 1966), the positions of stakeholders or existing institutional arrangements, all of which may have been informed by research and prove influential in the policymaking process (table 2, row 5)?

We have several options for identifying explicit uses of research use: asking policymakers who were directly involved in the policymaking process, asking researchers whose research may have been used in the policymaking process, and reviewing documents that were used in the policymaking process. For each approach, which ideally can be combined, political scientists typically distinguish among the prioritization (sometimes called agenda-setting), policy-development, and policy-implementation stages of the policymaking process. Research could be used in any of these stages. At the prioritization stage, for example, research may explain why an issue was put on the policymaking agenda, given the range of issues vying for the policymakers‘ limited attention (Kingdon 1995). At the policy-development stage, research may explain how the issue was formulated, why some policy alternatives were considered and not others, why one policy alternative was chosen over others, and how the final policy was justified.

Explicit uses of research can be assessed in a number of ways (table 2, row 3) but perhaps the most interesting is how the research was used. Pelz (1978) and many others in the knowledge-utilization field distinguish among instrumental, conceptual, and symbolic uses of research. Instrumental use is acting on research in specific and direct ways, such as solving a particular problem at hand. Conceptual use refers to a more general and indirect form of enlightenment. Symbolic use is using research to justify a position or action that has already been taken for other reasons (called a political use of research in Weiss 1979) or using the fact that research is being done to justify inaction on other fronts (called a tactical use of research in Weiss 1979). Conceptual and symbolic uses of research typically are said to outnumber instrumental uses (Rich 1979, (1991), although this finding may be attributable to sampling policies through researchers (e.g., Lavis 1998), who may be more likely to be called in either very early or after a decision has already been made.

Of course, the real challenge comes with establishing with some degree of certainty whether citable research has in some way informed other types of information, tacit knowledge, and broader political forces like stakeholders’ positions and existing institutional arrangements (table 2, rows 4 and 5). This task requires us to sort out similarities and differences between concepts like “tacit knowledge” (or “ordinary knowledge,” according to Lindblom and Cohen 1979) and conceptual uses of research. Both ways of “knowing” are general and indirect. But with conceptual uses of research, the source of the enlightenment is, by definition, research. This task also requires us to decide how to identify conceptual uses of research when they lie buried amid other types of information and broader political forces.

What We Looked for When We Looked for Research Use

Our approach to identifying research use is like reducing the magnification on a telescope while keeping the objects of interest constantly in view (table 2, rows 1–5). We moved from identifying the uses of citable research (i.e., the types of research outputs typically produced by health services research units) to identifying the uses of other types of information (which may have been informed by citable research), and then to identifying broader political forces like stakeholders’ positions (which may also have been informed by citable research). Our approach stopped short of deciding whether citable research did, in fact, inform these other influences on the policymaking process. For some types of information, the answer was reasonably clear. Professional guidelines or standards, for example, almost certainly drew on citable research. For other types of information, like information obtained by “scanning” other jurisdictions or sectors, the answer was much less clear.

We determined whether and how research was used in the policymaking process for each of the policies under study by interviewing the policy advisers who were directly involved in the process (table 2, row 2). We obtained the names and contact information for these key informants at the same time that we obtained the list of potential policies for study. We conducted brief telephone interviews with these individuals to ensure that we had found the most appropriate policy advisers. The interviewer (SR) asked them about the timing of their involvement in the policy process and their role in it. These interviews led us to identify a different key informant for one of the Ontario policies and an additional key informant for one of the Saskatchewan policies. In both cases, the new informants were more senior and more familiar with the full range of policy considerations than those originally chosen.

We collected data from the policy advisers through face-to-face, semi- structured interviews. We had asked them to review their files on the policy under study before the interview, and then the interviewer (SR) had them “tell the story” of how the policy change came about. When necessary, she prompted them to describe the policy change and its significance, the use of research and other types of information in bringing about the policy change, the factors influencing how and why the issue appeared on the policy agenda (i.e., the prioritization stage), and the factors influencing how the policy was developed (i.e., the development stage). She did not ask them about the policy-implementation stage because many of the policies were still in that stage. The interviews, which ranged in length from one and one-half to two hours, were transcribed by an experienced transcriber. The interviewer ended the interviews by asking for copies of internal documents related to the policy development process. We then repeated the request for internal documents several months after the interviews.

The interviews were analyzed using an interpretive approach grounded in the organizational behavior and management, knowledge-utilization, and political science research fields. For the unit of analysis, we used policy stages; that is, for each policy we examined the prioritization and policy-development stages separately. Three of us (JNL and JMH in the first round and JNL and SER in the second round) coded them and compared the results to find similarities and differences. We agreed on the differences through discussion. The accuracy of our analysis was supported by the independent assessments of two of the transcripts by another member of our research team (CW).

The data from these interviews were enriched by additional documentation when available. Unfortunately, the policymakers were usually unwilling to release internal documents like the minutes of meetings, briefing notes, and internal reports. Moreover, because only one of these policies required high-level approval (i.e., involved a legislative change), the majority had no public paper trail like the transcripts of parliamentary debates, regulations, legislative acts, or news releases. Thus, our analysis relied mainly on the data obtained from the interviews.

Four of the eight policies used citable research: three, in only one stage of the policymaking process; and one, in both stages, for a total of five cases of research use (app 2, row 2, bullet 2). The policymakers accessed the research by interacting with the researchers in three of the four policies, with two of the interactions taking place in formal working groups, either developing policy recommendations or informing policy development (app 2, row 2, bullet 3). The research was always used in instrumental ways, perhaps because of how we identified the sample of policies for study. We suspect that the policy advisers regarded as “good” the idea of using research and therefore may have been inclined to discuss those policies for which they knew research was used. For them to have known this, the use of research must have been quite explicit, and therefore they may have selected policies for which it was used instrumentally.

Citable research addressed many of the policy issues being discussed in the same three cases in which policymakers interacted with the researchers. Two of the cases were in the prioritization stage: citable research helped identify the need for (and limited uptake of) increased HIV prenatal testing in Ontario and the need for (and limited uptake of) increased pneumococcal immunization in Saskatchewan. The other case was in the development stage: citable research informed the policymakers about both whether to move to needs-based funding and which needs-based funding formula to adopt. Citable research addressed only some of the policy issues being discussed in the other two cases. In one case, citable research helped at the prioritization stage to find one long-term care resident classification criteria set for consideration, not whether the current classification system was adequate or whether to use a different system. In the other case, citable research helped at the development stage to determine whether to facilitate access to HIV prenatal testing, not how to facilitate access from a programmatic perspective.

To help us corroborate and enrich the data on research use that we collected from the policymakers, we also surveyed the directors of the research units funded by the two health departments, asking for any research conducted by their units that might be relevant to any of the four policies selected for their province. We received responses from 22 of 25 research-unit directors. Only five listed research relevant to the policies from their province: two listed research pertaining to one policy; one listed research pertaining to three policies; and two listed research pertaining to four policies. (Because there was some overlap in the policies that their research could have informed, no locally produced research may have been available for three of the eight policies. For two of the three policies, however, citable research was used in the policymaking process.) These research-unit directors listed a total of 34 research reports and/or journal articles, 31 of which were publicly available during either the prioritization or the development stage of the relevant policy. During their interviews, the key informants specifically mentioned only two of these reports, and in one of the cases, the report was identified as having not been used because “it wasn't something that the external stakeholders really gave a lot of credence to.”

All the policies used many types of information other than citable research (app 2, row 2, bullet 4), which tended to fall into three categories: (1) what people outside the health department do, (2) what people outside the health department think or want, and (3) what people inside the health department think or want. Information about the practices of other jurisdictions or sectors (i.e., what people outside the health department say they do) was the most frequently used type of information. Information in policy documents from previous or related policies (i.e., records of what people in the health department think or want) was also frequently used in policymaking, particularly in the policies involving jurisdiction and governance in both Ontario and Saskatchewan and in the policies about delivery arrangements and program content in Ontario.

Policymakers typically obtained these other types of information by interacting with their peers or with stakeholders, just as they usually found out about citable research from researchers. The practices of other jurisdictions or sectors, for example, were discovered by direct, informal contact with policymakers in other jurisdictions or sectors (that is, by “scanning,” as a number of key informants called it). Committees also provided access to information. For seven of the eight policies, committees were established as part of the policymaking process, all of them containing members from outside the health department.

While we did not specifically examine the importance of research that was influential indirectly (e.g., in the guise of other types of information), we did recognize the magnitude of the challenge. For example, the general sense that fee-for-service remuneration creates particular behavioral responses could be said to have colored the initial deliberations about establishing transfer-payment agencies for midwifery practice groups (a conceptual use of research). Policymakers facing calls to make better use of research in developing health care policy could cite this example as a conceptual use of research. They could point to a research base and a general consensus among health services researchers as proof that they had appropriately used high-quality research (Weiss 1979). But if the conceptual uses of research lay buried in the midwives’ position, and the policymakers were simply reacting to this position, the conceptual use of research would have been much more difficult to identify.

Reflections on What to Look for When Looking for Research Use

Our exploratory study led us to wonder whether future studies of the role of research in policymaking (and future efforts to establish goals about making better use of research in policymaking) might want to focus on the degree to which a policy was informed, not just on the extent to which the research is used. We were struck by our finding that for two cases in which research appears not to have been used, we considered the policymaking process particularly well informed. In both cases, structured processes gave play to a variety of research, other types of information, and values. Even more surprising to us, one of the cases in which research was used appeared to us to be one of the least informed policies. It seemed as if the effort to use as much research as possible for one policy issue (how much, “in theory,” to try to change) actually hindered a broader assessment of an equally important policy issue (what was not working before, and why). We need to establish a way of measuring the degree to which a policy was informed in the broad pluralist sense of a fully thought-out answer, not in the narrow technical sense of a “right” answer (as Lindblom and Cohen [1979] point out, “authoritative” research holds little promise in policymaking environments).

The possibility that research may be embedded in other types of information and in broader political forces like stakeholders’ positions suggests an important implication for policymakers. They should establish accountability for assessing other types of information and stakeholders’ positions for the research base on which they rest. That said, establishing accountability for conceptual uses of research poses a challenge, for both the policymakers who would have to judge some of the “givens” in the policymaking process and the researchers on whom policymakers would rely for the research base (which typically is not developed or packaged in ways conducive to policymaking). Furthermore, policymakers should actively guard against the misuse of research. That is, they should try to find out whether the research has been interpreted and applied correctly (Weiss 1979). The more explicit the use of appropriately interpreted and applied research is, the better.

Finding Patterns in When Research is Used and Not Used

Now knowing where to look for research use and what to look for, puzzle lovers’ skills are needed. Finding patterns in the conditions under which research is used and not used requires a framework for determining the range of influences with which research competes and the context in which the policymaking occurs (table 3, row 1). The traditional political science framework distinguishes three general categories of influences on the policymaking process: ideas, interests, and institutions (Goldstein and Keohane 1993; Hall 1993, 1996; Heclo 1974; Weatherford and Mayhew 1995). The first category—ideas—includes research and also other types of information and the values of legislators, policy advisers, stakeholders, and the general public. The interests category captures the perceptions of stakeholders or legislators and policy advisers about who will benefit from and who will be hurt by a given policy (i.e., who wins, who loses, and by how much).

TABLE 3.

Approach to Identifying the Conditions Under Which Research is Used (By Stage of the Policymaking Process)

Step	Examples of Available Options	Our Approach
1. Identify factors that influence the policymaking process.	• Policymakers describe in an interview or survey the factors that influenced the prioritization (i.e., agenda-setting), policy-development, or policy-implementation stages of the policymaking process.	• When describing in an interview the prioritization and policy-development stages of the policymaking process, policy advisers identify the factors influencing each stage (and these factors were then coded in the following ways):
	• Other participants in the policymaking process identify in an interview or survey their roles and influence in any of the stages of the policymaking process.	• Ideas (subcoded as citable research, other types of information, legislators’ values, policy advisers’ values, stakeholders’ values, and the public's values).
	• Documents produced in some of the stages of the policymaking process identify the factors influencing the stage(s).	• Interests (subcoded as stakeholders’ interests, legislators’ interests, and policy advisers’ interests).
		• Institutions (subcoded as past policies, the openness of the policymaking process, the time-pressured nature of the policymaking process, and the nature of approval required for the policy).
		• Documents produced in some of the stages of the policymaking process identify the factors influencing the stage (and these factors were then coded as above).
2. Identify factors that exert a major influence on the policymaking process.	• As noted above but with the degree of influence specified.	• Each of the above factors was coded as either a major or a minor influence on the policymaking process.

The final category in this framework—institutions—includes factors like policy legacies and characteristics of the policymaking process such as its openness, the degree of time pressure, and the level of approval required (e.g., legislative or staff in an executive agency). Policy legacies refer to attributes of past policies influencing whether an issue will be put on the agenda (e.g., an existing policy or program may have been designed with a phase-out or reassessment date) or how an issue will be addressed once it is on the agenda (e.g., an existing policy or program may have precipitated the development of administrative capacities that are more amenable to one policy alternative than another).

What Conditions We Found That Appear to Favor Research Use

Taking our telescope analogy one step further, our approach to determining the conditions under which research was used can be compared to increasing the magnification on a telescope while keeping the objects of interest constantly in view. For each of the prioritization and policy-development stages, we moved from identifying any uses of research and other types of information to identifying influential uses of research and other types of information, and then to determining when research and other types of information had a great influence on the policymaking process (i.e., when it was the single most important reason that an issue was put on the agenda or why the final policy took the form that it did). Our approach went beyond simple knowledge-utilization scales that do not distinguish between influential and noninfluential uses of research or, more generally, do not attempt to understand the use of research in a broader political context (e.g., Dunn 1983; Knott and Wildavsky 1980).

Citable research was a major influence in three of the four policies in which research was used in policymaking (app 2, row 3, bullet 2). In two cases, this influence was apparent at the prioritization stage: it helped establish the need for and the limited uptake of HIV prenatal testing in Ontario and the need for and the limited uptake of pneumococcal immunization in Saskatchewan. In the other case, this influence was apparent at the development stage and helped inform both the decision to move to needs-based funding and which needs-based funding formula to adopt in Saskatchewan. In all three cases in which research exerted a major influence on the policymaking process, citable research informed a large part of the policy issue. Information other than citable research also greatly influenced one of the policies (Saskatchewan's introduction of a new long-term care classification system) in the development stage. This information included the results of a pilot test of three different classification systems and would have been considered citable research had it been published by the health department in a publicly available form.

The principal influences in the prioritization stage most frequently involved the policymakers themselves, because of either the objectives that they were pursuing (i.e., their interests) or the policy legacies that they had created. Policymakers placed an issue on the policy agenda as a way of pursuing a number of interests: establishing a precedent that they believed would serve them well in the future, creating particular incentives in the health care system, and maintaining the credibility of another policy initiative. The other times that an issue appeared on the policy agenda was because of a policy legacy. Examples are a policy or program designed with a phase-out or reassessment date, a long-term reform initiative that specified the need to address one aspect of reform once other aspects had been addressed, and a policy or program that left one or more exceptional circumstances to be addressed at a later date.

Two of the policies for which policymakers themselves acted as the major influence in the prioritization stage could have been coded as either government interests or policy legacies, but in both cases we felt that the government's interests were slightly more important than the policy legacies, because it was the government's commitment to a particular position that propelled the actual policy, not the policies’ legacies per se. In other words, although the related policy legacies provided an important backdrop for the policy's timing and issue identification, they did not in themselves justify the particular policies. In both cases, this justification came more from the government's interest in developing and enhancing particular components of the broader policy trajectory.

The major influences in the development stage were more evenly distributed across categories. Three of the policy-development processes were driven primarily by stakeholders, in either pursuing their interests or blocking an effort to bring about a policy change that was not in their interests. The policy legacies in policy development included one case in which past government actions had led to the development of administrative capacities in some areas and not others, which in turn privileged one policy alternative over another.

Some patterns can be discerned in the conditions under which citable research appears more likely to be used. For the three policies in which citable research was a major influence in the policymaking process, policymakers had direct contact with researchers. In two of these three policies, this contact took place through what could be called a “receptor” for research (Lomas 1997) created by the health department. By this we mean that specific functions were established with explicit responsibility for establishing and maintaining linkages with researchers: Ontario's AIDS Bureau, which had been established to improve the department's knowledge of (and responsiveness to) HIV/AIDS-related health issues and community concerns, and Saskatchewan's “expert” working group on a needs-based funding formula. The policymakers’ access to other types of information was also facilitated by direct contact with stakeholders and other policymakers, often through “scanning” and committees.

The research-unit directors whom we surveyed listed ten knowledge-transfer activities that included contact with policymakers and stakeholders, ranging from submitting copies of a report to potential users, presenting papers at conferences with predominantly nonresearch audiences, presenting papers at meetings convened by policymakers, briefing policymakers, testifying at public hearings, and participating in expert committees. The directors rated six of these activities or combinations of activities as having unknown influence and the other four activities at the low end (1–3) of a scale ranging from not influential (1) to very influential (5). None of the key informants specifically mentioned any of these knowledge-transfer activities.

The knowledge-utilization field emphasizes the importance of interaction between researchers and policymakers. Caplan (1979) and Wingens (1990) used the “two communities” metaphor to suggest that a lack of interaction between the research and decision-making communities may explain the decision makers’ limited use of research. The producer-push and user-pull models of knowledge transfer and uptake have therefore been supplanted by an interaction model to enhance knowledge transfer and uptake (Landry, Amara, and Lamari 2001). Our exploratory study suggests that interaction between these “two communities” does influence the use of research by policymakers, although from the research-unit directors’ comments it appears that many efforts at interaction yield no tangible impact.

For the three policies in which citable research was a major influence in the policymaking process, two could be categorized as professional decisions about program content (HIV prenatal testing and pneumococcal immunization), and one could be categorized as a technical decision about financial arrangements for which stakeholders’ perceptions of the decision's credibility were deemed to be important to the policy's implementation (needs-based funding formula). These types of professional or technical “content-driven” decisions may be more amenable to the influence of research in instrumental (i.e., specific and direct) ways than are large-scale decisions concerned with, for example, jurisdictional considerations. Large-scale decisions likely require research that is much broader in scope than is typically produced by discipline-based researchers, who ask very focused research questions. Moreover, even when relevant research exists, it may be overlooked in the rush to assess other factors, like stakeholders’ interests and institutional constraints, that seem more germane as the scale of a decision increases.

Reflections on Identifying the Conditions under Which Research is Used and Not Used

The interaction between researchers and policymakers and the existence of an accountable “receptor” function in government (Lomas 1997) appear to be the conditions most favoring the use of health services research specifically and information more generally in the policymaking process. Researchers (and research funders) should create more opportunities for interactions with the potential users of their research. They should consider such activities as part of the “real” work of research, not a superfluous add-on. And they should assign a higher priority to developing and acknowledging in others the skills required to promote this interaction. Policymakers who are committed to making better use of research in developing public policy should also create more opportunities for interactions. For example, they could include a few researchers in their information-exchange network and invite researchers to participate in working groups charged with informing the policymaking process. Establishing a “receptor” function with explicit accountabilities for developing and maintaining linkages with researchers would be a more systematic way of doing this.

The observation that some policies and policymaking processes may be particularly amenable to being informed by research suggests the need to move beyond a one-size-fits-all approach to the way that health-services researchers transfer research knowledge to policymakers. More content-driven policies (e.g., needs-based funding formula, HIV prenatal testing, and pneumococcal immunization) may be better served by key messages that help solve a particular problem at hand (i.e., to be used in instrumental ways). Other policies may be better served by key messages that provide a more general and indirect form of enlightenment (i.e., to be used in conceptual ways). But our functional categories relevant to the health care sector, which range from large-scale to small-scale policies, are only a beginning in the development of sector-specific typologies that can help inform our research and knowledge-transfer efforts.

Conclusion

We used organizing frameworks and analytic insights from three research fields to study the role of health services research in public policymaking. Taken together, these organizing frameworks and analytic insights suggest that we should think carefully about where we look for research use (some policies appear to be particularly amenable to being informed by research), what we are looking for (restricting the use of research to explicit statements by policymakers about how they used citable research appears to be an oversimplification of a complex process), and the conditions in which research is used and not used (sustained interactions between researchers and policymakers appear to make a difference). We need to look at more than the use (versus nonuse) of research in isolated policy decisions and, ideally, at the way in which research is used and at its use in the context of other, competing influences on the policymaking process. We hope that better health care policy, or at least better use of health services research in developing health care policy, will be the result.

Appendix 1

Policies Selected for Study, by Policy Category and Province

Policy Category	Ontario	Saskatchewan
Jurisdiction/governance	Establishment of community-based transfer payment agencies for administering and funding midwifery practice groups (part of a larger policy trajectory that began with the decision to publicly fund midwifery services).	Establishment of a health district with multilevel funding to serve both First Nations and non–First Nations populations (part of a larger policy trajectory that began with the decision to devolve much of the decision-making authority over health services to elected boards in health districts).
Financial arrangements	Introduction of a discounted physician fee schedule for three years to provide incentives to practice in underserved geographic areas.	Introduction of a needs-based funding formula for the allocation of approximately 60% (and eventually 80%) of health district funding (part of the same devolution initiative noted above).
Delivery arrangements	Establishment of single long-term care access points for admission to long-term care facilities and the introduction of standardized admission criteria for these facilities (part of a larger policy trajectory that began with the decision to reform the long-term care sector).	Introduction of long-term care resident classification criteria for use in needs-based funding of health districts, program planning, and resident assessment (part of the same devolution initiative noted above).
Program content	Expedited (or facilitated) access to HIV prenatal screening.	Expansion and systematization of a pneumococcal immunization program over four years.

Appendix 2

Key Findings from Our Examination of the Role of Health Services Research in Canadian Provincial Policymaking

Objective	Step	Findings
1. Identify a sample of policies.	• Select or create a typology of policies.	• Functional categories relevant to the health care sector have intuitive appeal to policy advisers.
	• Identify policies from each policy category using available sources.	• Policy advisers were able to identify policies from each policy category.
	• Select policies in each policy category.	• Policies could be selected so as to provide a fair degree of similarity across provinces.
	• Assess the representativeness of the selected policies across dimensions not covered by the typology.	• All policies were regulatory policies (not distributive or redistributive policies).
		• All policies had low public significance.
		• Six of eight policies had low stakeholder impact.
		• Five of eight policies were part of or embedded in a larger policy trajectory.
		• All policies were “go” decisions.
2. Identify research use (by stage of the policymaking process).	• Determine what constitutes research.	• Citable research in general and citable research produced specifically by government-funded independent research units could be identified in interviews with policy advisers and through documents used in the policymaking process.
	• Identify the explicit uses of citable research.	• Four of eight policies had citable research used in at least one stage of the policymaking process, and one of these four policies had citable research used in both stages.
	• Assess the explicit uses of citable research.	• For the four policies in which citable research was used, policy advisers accessed the research by:
		• Reading journal articles, book chapters, working papers, and/or reports for one of these four policies.
		• Interacting with researchers for three of these four policies.
		• Involving researchers in a working group for two of these four policies.
		• All four policies in which citable research was used involved instrumental uses of research (not conceptual or symbolic uses).
		• Research addressed a large proportion of policy issues in:
		• Two of the three policies in which citable research was used at the prioritization stage.
		• One of the two policies in which citable research was used at the development stage.
	• Identify the explicit uses of other types of information (which may or may not be based on research).	• Information other than citable research was used in all eight policies, and at least four types of information other than citable research were used in seven of eight policies.
		• These other types of information could be grouped into three categories:
		• What people outside the health department do, including the practices of other jurisdictions or sectors (“scanning”), professional guidelines or standards, and the operation of other comparable initiatives (e.g., through site visits).
		• What people outside the health department think or want, including the knowledge, experience, and demands of other legislators, policy advisers, researchers, and stakeholders.
		• What people in the health department think or want, including the knowledge and experience of staff, policy documents from previous or related policies, organizational reviews, internal administrative data, and pilot demonstrations.
		• Policy advisers accessed other types of information by interacting with peers or stakeholders and, for seven of eight policies, by involving peers or stakeholders in working groups.
	Identify the nonexplicit uses of research.	Not addressed in exploratory study.
3. Identify conditions under which research is used (by stage of the policymaking process).	• Identify the factors that influence the policymaking process.	• Citable research was one of two to four influences on the policymaking process at each of the prioritization and policy-development stages.
	• Identify the factors that exert a major influence on the policymaking process.	• Citable research was a major influence in the prioritization stage in two of eight policies, and legislators’ and policy advisers’ interests and policy legacies were a major influence at this stage in the remaining six of eight policies.
		• Citable research was a major influence in the policy-development stage in one of eight policies, and other major influences at this stage included stakeholders’ interests (three of eight policies), policy legacies (two of eight policies), legislators’ and policy advisers’ interests (one of eight policies), and other types of information (one of eight policies).

Policies Selected for Study, by Policy Category and Province

Policy Category	Ontario	Saskatchewan
Jurisdiction/governance	Establishment of community-based transfer payment agencies for administering and funding midwifery practice groups (part of a larger policy trajectory that began with the decision to publicly fund midwifery services).	Establishment of a health district with multilevel funding to serve both First Nations and non–First Nations populations (part of a larger policy trajectory that began with the decision to devolve much of the decision-making authority over health services to elected boards in health districts).
Financial arrangements	Introduction of a discounted physician fee schedule for three years to provide incentives to practice in underserved geographic areas.	Introduction of a needs-based funding formula for the allocation of approximately 60% (and eventually 80%) of health district funding (part of the same devolution initiative noted above).
Delivery arrangements	Establishment of single long-term care access points for admission to long-term care facilities and the introduction of standardized admission criteria for these facilities (part of a larger policy trajectory that began with the decision to reform the long-term care sector).	Introduction of long-term care resident classification criteria for use in needs-based funding of health districts, program planning, and resident assessment (part of the same devolution initiative noted above).
Program content	Expedited (or facilitated) access to HIV prenatal screening.	Expansion and systematization of a pneumococcal immunization program over four years.