Research and Advice Giving: A Functional View of Evidence-Informed Policy Advice in a Canadian Ministry of Health

Abstract

Context: As evidence-based medicine grows in influence and scope, its applicability to health policy prompts two questions: Can the principles and, more specifically, the tools used to bring research into the clinical world apply to civil servants offering advice to politicians? If not, what approach should the evidence-oriented health policy organization take to improve the use of research?

Methods: This article reviews evidence-based medicine and models of research use in policy. Then it reports the results of interviews with civil servants in the Ontario Ministry of Health, which recently adopted a stewardship rather than an operational role, incorporating many evidence-oriented strategies. The interviews focused on functional roles for research-based evidence in policy advice.

Findings: The clinical context and tools for evidence-based medicine can rarely be generalized to policy. Most current models of research use offer lessons to researchers wishing to apply their work to policy but little help for civil servants wishing to become more evidence oriented. The interviews revealed functional roles for research in setting agendas (noting upcoming issues and screening interest groups’ claims), developing new policies (reducing uncertainty, helping speak truth to power, and preventing repetition and duplication), and monitoring or modifying existing policies (continuously improving programs and creating a culture of inquiry). Each area requires different tools to help filter the push of evidence from researchers and set agendas, to facilitate the urgent pull on relevant research by civil servants developing new policy, and to support ongoing linkage and exchange between civil servants and researchers for monitoring and modifying existing policy.

Conclusions: A functional framework for evidence-informed policy advice is useful for distinguishing the activity from evidence-based medicine and “auditing” the balance of efforts across the different functional roles of research in policy.

Given the exponential growth in the evidence-based medicine movement, it is perhaps surprising that the use of research and related evidence by career civil servants in health care agencies has received little more than exhortation and theoretical attention. Empirical or even descriptive studies are rare (Davies, Nutley, and Smith 2000; Mitton et al. 2007; Nutley, Walter, and Davies 2007) and far outnumbered in the last twenty years by “implementation research” in clinical practice settings. Apparently it is more rewarding to persuade clinicians to change their behavior in line with evidence than it is to improve the impact of research on the policy advice given by civil servants.

The world clearly becomes more complicated when moving from a clinical focus to the management of service delivery and finally to the realm of system policy. Complex forces compete with research for the attention of civil servants and politicians: the interests of stakeholders, the values of the public, the ideologies of governing parties, the constraints of prior policy, and so on. Choosing whether to use public funds to improve children's access to immunization or to insure in-vitro fertilization (IVF), and with what breadth of coverage in what subpopulations, is decidedly more complex than the “treat–don't treat” clinical decision regarding the immunization of this child or IVF for that couple. Nor does the clinical world have to grapple with the added complexity of finding the appropriate metric to compare the benefits of immunization with those of IVF or the benefits of IVF for same-sex couples compared with heterosexual couples and how either fits with prevailing societal values.

The receptor capacity for research also is less developed in management and policy than it is in medicine. But even in medicine, the gap between research and practice still is more a chasm than a crack. Nevertheless, the medical world tends more than the policy world to subscribe to a common knowledge base with common journals, common training, a common purpose of making patients better, and, through practice guidelines and protocols, common tools of implementation. Indeed, clinical governance structures are increasingly focused on ensuring compliance with and accountability to such evidence-based standards. In contrast, the disciplinary backgrounds of administrators managing health service delivery and civil servants advising elected and politically appointed officials on health policy are diverse, their sources of knowledge and research disparate, their purposes varied, and their accountability diffuse. Their main implementation tool is consensus around the acceptable rather than convergence on a loosely defined research “truth” (Walshe and Rundall 2001). Finally, the views of managers and civil servants regarding what counts as evidence diverge from those of clinicians, as they are broader and more encompassing (Culyer and Lomas 2006; Glasby, Walshe, and Harvey 2007). As one British writer noted in regard to the political realm, “what ministers call ‘evidence’ is what they get from their constituents” (Petticrew et al. 2004, p. 813). In this way, policy advice is less “evidence based” and more “evidence informed” (Bowen and Zwi 2005).

Not surprisingly, therefore, attempts by some researchers to apply without reflection the lessons of evidence-based medicine to policy have not been successful (Boaz and Pawson 2005; Davies, Nutley, and Walter 2008; Klein 2003; Lewis 2007). Indeed, Black pointed out that “evidence based policy is not simply an extension of evidence based medicine: it is qualitatively different. Research is considered less as problem solving than as a process of argument or debate” (2001, p. 277). This view sits well with the work of Weiss (1979), who holds that a principal use of research for policymaking is conceptual: a source of enlightenment and a way of thinking about an issue, not an instrumental tool defining and then determining the “right” solution to a problem.

These are major contrasts in the realistic expectations of how evidence, particularly evidence created by researchers, is treated and can be used by civil servants for policy advice rather than medical authorities for clinical guidance: evidence-informed versus evidence-based decisions, conceptual enlightenment versus instrumental solutions, and a way of thinking and a catalyst for debate versus an attenuation of thinking and diversion around disagreement. The implication is that the tools and programs of evidence-based medicine—critical appraisal, Cochrane-style systematic reviews, practice guidelines, audit and feedback, computer reminders, and so on—are of little relevance to civil servants trying to incorporate evidence in policy advice.

What, then, are the best models and tools to encourage more evidence-informed decision making, more research-based dialogue in the policy world? In the remainder of this article, we describe the approach taken by the Ontario Ministry of Health and Long-Term Care, our method for interviewing its senior civil servants, and the specific tools it has implemented in its quest to better equip them for evidence-informed policy advice. Finally, we briefly review the shortcomings of existing classes of models or frameworks in order to understand these civil servants’ use of research before developing our own functional framework based on the interviews. In conclusion, we draw some lessons from the assessment and discuss the challenges for the future.

The Ontario Ministry of Health and Long-Term Care

Overview

The Ministry of Health and Long-Term Care in Canada's largest province, Ontario, employs about four thousand people, mostly located in the provincial capital of Toronto. For the 13 million inhabitants of the province, the ministry oversees more than C$40 billion per year for first dollar coverage of hospital and physicians’ services and an array of other services on the basis of partial dollar or select population coverage. As in the rest of Canada, about 70 percent of health care expenditures are paid by the public purse, and the remaining 30 percent are largely unregulated and are paid out-of-pocket or through private insurance (by law, however, there is no private insurance for medically necessary hospital and physicians’ services).

The titular head of the ministry, the minister of health and long-term care, is an elected politician appointed by the leader of the governing party, the provincial premier. The ministry's day-to-day operations are overseen by the deputy minister of health and long-term care, who is not an elected politician but usually is a career civil servant, although talented outsiders may be appointed. He or she occupies the gray area between the political realm of elected officials and the policy support role of career civil servants, as indicated by the fact that the appointment is by the elected premier. Ontario follows the British parliamentary tradition in which these senior civil servants are not overt political appointees and usually continue in their posts even after a change in the government's political stripe.

The ministry largely avoids focusing on the operational health and long-term care systems; that is, it does not directly employ or manage service delivery personnel. Rather, over the last two years, it has devolved the day-to-day management functions of its regional offices and a range of other services that might be considered part of service delivery to fourteen geographically defined local health integration networks and is moving toward a stewardship role as defined for health systems by the World Health Organization and others in the late 1990s (Saltman and Ferroussier-Davis 2000). In this role, the ministry sees its main functions as setting priorities, developing policy, and ensuring accountability to provincial priorities by the local health integration networks and other providers. The move to stewardship is not yet complete, however, as the ministry still directs some services, like dispatching ambulances, forming public health units, and negotiating with the physicians’ association.

Increased attention to and leadership for evidence-informed decision making coincide with the transformation of the ministry into this stewardship role. Both its most senior civil servant (the deputy minister) and one of us (Brown) support the increased use of research-based evidence as part of the new stewardship role, developing a variety of programs and recruiting staff with this in mind. Thus the ministry provides a relatively “pure” example for studying the role of advice giving for policy and the potential contribution of research-based and other evidence.

Interview Methods

Given our aim of evaluating the actual and potential roles of evidence in the ministry's policy monitoring and advice functions, we interviewed civil servants, not politicians, in the ministry.

The ministry has seven divisions, each run by an assistant deputy minister, plus a chief medical officer of health and a senior medical, scientific, and health technology adviser. Reporting to the assistant deputy ministers are the executive directors and directors in charge of branches, in which the teams and units run by managers do the main work of the ministry through a variety of senior and junior analysts. The focus of our interviews was in the core divisions charged with setting agendas, developing policy advice, and monitoring the performance of programs and policy: the Health System Strategy Division, the Health System Information Management and Investment Division, and the Health System Accountability and Performance Division. We did not interview civil servants in divisions responsible for administration or internal organization, such as Transition (to the stewardship role), Corporate and Direct Services, and Health Services Information Technology.

In March and April 2009 we interviewed the assistant deputy ministers and directors in the three divisions of interest plus the senior medical, scientific, and health technology adviser. We chose this level for our interviews because it best represented the balance between active involvement in policy advice and active involvement in the information gathering that contributes to that advice.

Access to these interviewees was facilitated by the fact that one of us (Brown) is one of the ministry's assistant deputy ministers, but Lomas conducted all the interviews, as he was on a short-term three-month contract at the ministry to evaluate evidence-informed decision-making activities. Interviewees were assured of confidentiality for their individual comments and observations. Out of a possible twenty-four interviews, we concluded eighteen. The remainder were temporarily vacant positions, or we were unable to arrange a mutually convenient time.

We were not interested in obtaining another inventory of the barriers to using research in policy, as others have preceded us with this listing, including a systematic review of such studies (Innvaer et al. 2002). Rather, we asked three simple questions to find the current and potential future uses of evidence in relation to different aspects of policymaking:

1. Please give a brief description of the kind of policy advice for which you are responsible in the ministry.

2. What does the ministry currently do well to support your evidence-informed decision making in this work?

3. What else could the ministry do to facilitate the use of evidence in your work?

The first question provided some context for subsequent statements during the interviews. Aggregation of the answers to the second question, supplemented by our own review, provided a useful overview of the ministry's activities supporting research and evidence-informed decision making. Answers to the third question, once analyzed to identify consistent major themes, offered an insight into what the interviewees saw as the practical roles of research and other evidence in their work.

The Ministry's Tools Supporting Evidence-Informed Advice

The interviews uncovered a variety of tools that the ministry was using to improve its use of evidence for policy advice. Many of these were managed or implemented by its Research and Planning Branch, although other areas of the ministry also took the initiative on the mandate to improve evidence use.

Table 1 groups these tools in the three main areas of ministry activity identified by the interviewees for the application of research and other evidence: setting agendas, developing new policies, and monitoring or modifying existing policies.

TABLE 1.

Tools Available to the Ontario Ministry of Health and Long-Term Care for Evidence-Informed Advice Giving

Helping Set or Anticipating Agendas	Informing New Policy Development	Monitoring and Modifying Existing Policies
Undirected work of 17 funded external centers and the Institute for Clinical Evaluative Sciencesa Career scientists Medical Advisory Secretariat E-repository database Health Evidence Quarterly Health Horizons newsletter Health System Trends Report Long-range scenario planning	Annually eliciting upcoming evidence priorities from staff Directed work of 17 funded external centers and the Institute for Clinical Evaluative Sciencesa Commissioned studies Medical Advisory Secretariat Data and Data Analytics Branches Rapid literature reviews Research use training workshops Consolidated searchable evidence sources in The Exchange Evidence-base checklist for policy submissions	Data and Data Analytics Branches Ad hoc program evaluations Career scientist internships with ministry staff External networks

Notes: If an initiative appears more than once, the boldface indicates the activity's primary objective.

^aUniversities’ external research centers commit 75 percent of their time to a self-generated agenda and 25 percent to ministry-directed priorities. For the independent, stand-alone Institute for Clinical Evaluative Sciences, these figures are 60 percent and 40 percent.

Setting Agendas

Most of these initiatives were designed to generate applied research that might be of future value for the ministry. For instance, the ministry spent C$24.5 million in 2008/2009 to fund seventeen university-based external research centers working under multiyear agreements, with 75 percent of their time on topics of their own choosing but within broadly defined themes of relevance to the ministry. Examples are inner-city health, rehabilitation, rural and northern health, and health economics. In addition, C$5.5 million per year goes to the independent Institute for Clinical Evaluative Sciences, which houses most of the health and health care–oriented administrative databases and allocated just under half its resources to investigator-initiated topics. Finally, C$2.25 million was spent in the year to support thirty-two career scientists doing applied work potentially relevant to the ministry. The ministry also has its own technology assessment staff in its Medical Advisory Secretariat, who are using some innovative methodologies to advance current and anticipated pressures on the system (Levin et al. 2007).

A set of related initiatives transform the outputs from these ministry-funded external groups into more accessible and usable products. All the completed funded research is made readily available through the ministry's Intranet by being placed into a searchable database (the E-repository) along with a summary description. In addition, the Health Evidence Quarterly, distributed both inside the ministry and to the fourteen local health integration networks, highlights the most relevant of these with links directly to the original studies. Another quarterly summary newsletter, Health Horizons, issues theme-based reports on research for one or two emerging or current topics such as emergency room patient satisfaction and inequalities in access. For these reports, Health Horizons combines ministry-funded research with peer-reviewed research from all jurisdictions relevant to the topic.

Two initiatives help screen claims from the various interest groups trying to get their issues on the ministry's policy agenda. The first, the Health System Trends Report, is constructed through a delphi process with an expert panel outside the ministry and drawn from across the country. The report takes a long-term view of the forces likely to affect the ministry. It is updated annually and completely revised every three years, thus offering a potential screen for issues staking a claim on the agenda. The second, a long-range scenario-planning process, is modeled on the efforts of the oil company British Petroleum in the 1970s and 1980s. It identifies major unanticipated developments in the health system and robust strategies to deal with them. This process, however, tends to be focused on specific areas, most recently on the potential changes in the informal care sector and their impact on the health system and the broader economy.

Developing New Policies

To facilitate pulling research into the policy development process, the ministry annually elicits priority topics from staff on the basis of their policy development needs for the upcoming year. It then either directs these studies to the ministry-funded external research centers (where at least 25 percent of time is committed to such responsive research) or commissions them as one-off projects from other researchers. For 2009/2010, seventeen such priority projects were identified and directed externally. Similarly, the in-house Medical Advisory Secretariat occasionally responds to specific demands for research to inform a policy, particularly but not exclusively in clinical and technology areas, as do the Data and Data Analytics Branches.

The majority of the seventeen ministry-funded external research centers have been in place for a decade or more. Over time, these externally funded centers have come to accept the quid pro quo of balancing funding for research on their own interests with responding to the ministry's declared research needs and commissions. Initially, however, there were some tensions as university investigators learned how to negotiate research programs with the ministry and vice versa. For instance, some of the ministry's research needs that may be locally idiosyncratic and lack generalizable value for peer-review publication may now be incorporated in funding for larger, more generalizable studies by the researchers.

In the last few years, the ministry has come to recognize the potential value of this peer-reviewed publication as an external and credible “validation” of its advice or action and accordingly has relaxed its prior restrictions on public release and/or peer-review publication. Ultimately, however, the ministry's concern is with relevance and the timely availability of the research it funds, not peer-reviewed publication. On their part, the researchers have come to understand the potential dangers of precipitously releasing evidence contradicting policies proposed or adopted by the government and have agreed to a holding period of sixty days between the submission of results to the ministry and their publication or general release.

The second level of policy development tools addresses the more urgent needs of civil servants working on real-time policy advice. An in-house staff of eight provides rapid response literature reviews with as little as a forty-eight-hour turnaround but prefers a one- or two-week response. The product's comprehensiveness obviously is related to the time available. In 2008/2009, about forty such rapid-response reviews were conducted. In addition, some of the ministry-funded external centers have recently started to offer this kind of rapid response as well. In the longer term, more ministry staff may be able to conduct such reviews themselves with the knowledge gained from a one-day “How to Use Research” workshop offered every few months to teach them the sources and search techniques for synthesized research. This workshop was offered five times in 2008/2009 and was attended by about seventy-five staff. Also, through its Knowledge Management Branch the ministry is gradually assembling the databases and creating staffwide ready access to these searchable sources in a single location on its Intranet: The Exchange.

Finally, an overall “reminder” tool to encourage pulling in broadly defined evidence for policy development will be implemented in 2009/2010. This “evidence-base checklist” will document the degree to which a policy submission has used available research, the nature and estimated quality of that research, the experience of other governments or agencies with similar policies, and so on. This checklist will be required as an accompaniment to the policy submission as it moves through the ministerial approval process.

Monitoring and Modifying Existing Policies

Given the ministry's new stewardship role, we might expect to find extensive performance indicator and monitoring work, with ongoing exchanges between those responsible for the programs and those producing the data. This, however, has not been the case. Although the Data and Data Analytics Branches helped with the performance indicators for provincial programs, the development and monitoring processes for this were neither routine nor comprehensive. Similarly, program evaluations have been largely ad hoc and the exception rather than the rule, even for large new initiatives. Where they are in place, such as in the Mental Health Branch, ongoing exchanges have been valued by both sides, and program or policy modifications flowed from the evaluation, although they were not necessarily well aligned with or integrated into the ministry's routine monitoring capacity.

To facilitate more general linkages between researchers and staff in the ministry, the external career scientists are expected to spend six months of their five years of funded time working directly in the ministry with staff who overlap with their area of expertise. Other than this, the ministry's civil servants are largely left on their own to develop and maintain any external networks and linkages with researchers. For instance, there is no central database of researchers by area of expertise.

Across all the areas discussed, the interviews once more highlighted that what is considered to be “evidence” is broader in the policy world than in the research world. Interviewees also confirmed that the relatively narrow approach and focused tools of the evidence-based medicine movement had found little traction in the policy world. The interviewees considered peer-reviewed research alongside gray literature, raw data (and their tabular presentation), the actions or programs in other jurisdictions, the views of experts or expert advisory committees, and opinion polls to form the core components of evidence for policy. For researchers and, to a lesser extent, for evidence-based medicine proponents, the word evidence is synonymous with research, whereas for these civil servants, evidence is more synonymous with data, analysis, or investigation. Guidelines and protocols may be the mainstay of clinical applications of research, but the one-off nature of most agenda setting, policy advice, and policy monitoring seems to require tools more sensitive to the specific context in which the research is being applied, if not new skills for acquiring and incorporating research in the ministry's work.

If the clinical models of evidence-based medicine are poor guidance for better incorporating research into policy advice, might models of research use from the political and policy sciences reflect better the needs of the senior civil servants we interviewed?

Models of Research Use in Policy

Unfortunately, the existing reviews of research use models for policy were largely written from the perspective of research rather than policy, and they address the role of the researcher trying to get his or her research used more than that of the civil servant trying to do the using (Hanney et al. 2003; Landry, Amara, and Lamari 2001; Lavis et al. 2003; Lemieux-Charles and Champagne 2004; Mitton et al. 2007; Nutley, Walter, and Davies 2007; Weiss 1995), although there are some notable exceptions (e.g., McDonough 2000). This is not surprising, as it is the role of the academic to observe, analyze, and write and the role of the civil servant or politician to read, collate, and act.

The rational actor stages model (e.g., Dror 1983; Hogwood and Gunn 1990) posits sequential stages of identifying problems, assembling options, calculating relative costs and benefits, choosing and implementing the “winner,” and monitoring results. Outside health policy's unusually clearly delineated decisions regarding drug approvals or new technology coverage, the logical sequencing represented by the model is, for the civil servant, more a description of an “ideal type,” useful largely as a baseline from which to document deviations. For example, England's introduction of general practitioner fund holding in 1991 was a last-minute addition to a National Health Service reform plan. It did not respond to a specifically identified problem but implemented an appealing idea that had not been subjected to any assessment against competing possibilities (Maynard 1993). As Nutley and colleagues (2007, p. 93) pointed out, and most civil servants already know, “its neat stages fail to reflect the messy complexity that typically characterizes policy making as it really occurs.”

In contrast, the messy, constrained world models of policy (e.g., Kingdon 1984; Lindblom 1959; Simon 1957; Weiss 1995) recognize the happenstance nature of how policy is developed. Some emphasize the constraint on policy of individual cognitive ability (Simon 1957) or institutional precedent (Lindblom 1959), inevitably leading to incremental change that is determined more by the art of the possible than the science of research. Others throw up their hands in despair and present frameworks such as “policy streams” (Kingdon 1984) or “garbage cans” (Cohen, March, and Olsen 1972) for creating some kind of order out of chaos, but again offering no privileged role for research.

It is not a coincidence that many of the creators of this class of model are from the United States, where health policy has often fallen prey to constraints that led the apparently possible to become impossible: recall the demise of the Clintons’ health reform proposal in the early 1990s. The tools for civil servants to use research in this model are neither coherent nor obvious, and praying for good luck and the occasional window of opportunity is the best for which one can hope.

Somewhere between the two aforementioned classes of models are the interaction models (e.g., Caplan 1979; Davies, Nutley, and Walter 2008; Huberman 1987; Kogan and Henkel 1983; Kogan, Henkel, and Hanney 2006; Lavis et al. 2002; Lomas 2000; Sabatier and Jenkins-Smith 1993). These see the policymaking process as one of ongoing and often prolonged interaction between and within competing interests, from which may emerge feasible and acceptable policy. In fact, much of recent climate change policy can reasonably be viewed through this lens.

For civil servants, the interaction models have the advantage of recognizing the diffuse nature of the evidence demands they face; they do not rely so much on specific studies as on access to knowledge through trusted and ongoing relationships with the research community. For researchers, the implied tools for research use are structures and processes that routinely link researchers with policymakers, either face-to-face or through intermediaries such as knowledge brokers (Dobbins et al. 2009; Lomas 2007; van Kammen et al. 2006).

Finally are the policy as argumentation models (e.g., Dobrow, Goel, and Upshur 2004; Gibbons et al. 1994; Giddens 1987; Greenhalgh and Russell 2006; Russell et al. 2008; Souza and Contandriopoulos 2004), which, like the interaction models, focus on communication, but this time more on the content than the channels. These models highlight the importance of language in framing debates. Contrary to the tenets of evidence-based medicine, they do not see different forms and sources of evidence as hierarchically arranged in relation to “truth” but rather, in a postmodernist worldview, they see all as contestable (Giddens 1987). As Klein (2003, p. 431) noted about the research exercise in this context, “if we remember that evidence speaks with many voices—and that our values drive facts and shape the conclusions we draw from them—we will also conclude that any such exercise will be no more, and should be no more, than one contribution to the process of policy-making.” Policymaking is thus a rhetorical process in which power and persuasion combine within the available institutional structures to determine outcome.

As with most things, there is a little bit of truth to all the models when applied to the initiatives and views of those in the Ontario ministry. The rational actor view emphasizes how under the right circumstances, research can “push” an issue onto the policy agenda. This view, therefore, helps explain the kind of long-term investments that the Ontario ministry has made in funding external research centers and scientists. But the messy constrained world models warn researchers not to expect great leaps forward in policy, to be modest in the degree of desired impacts, but to remain attuned to timing and ready to exploit the occasional window of opportunity when it opens. The ministry's rapid response literature review service fits well with this opportunistic view of research use. The interaction models draw attention to the importance of interpersonal factors in the policy world and the need for long-term investment in social structures and processes, which are missing from most of the ministry's initiatives but are seen as important by the interviewees. Finally, the argumentation models highlight the value of resonant language and of seeking out or even creating venues for research-based debate, another approach that is poorly represented in the ministry's approaches.

The models mainly suggest skills and tools that might be helpful for the researcher but rarely take the perspective of the civil servant. They mostly fail to provide functional guidance to the evidence-oriented civil servant or politician and offer few recommendations for what class of tool to use under what circumstance or what skills to acquire to improve the use of research-based evidence. Arguably, none of the models addresses how or why researchers should establish relationships with civil servants, nor, most tellingly, do they yield many implications for how the civil servant should use evidence.

Developing a Functional View of Research Use in Policy

Is there an alternative “insider” view of these external observer perspectives from researchers on research use in the policy world? Can we use the evidence-informed advice-giving tools implemented by the Ontario ministry and the views of its senior civil servants to create an alternative “insiders” framework?

For civil servants giving advice and implementing policy, research is of interest if it can help them do their jobs. The interviewees provided a functional view of where and how research can make tailor-made contributions. In this sense, no general framework for improving the use of research is likely to offer guidance to the specific functions of research for policy advice. Our interviewees clearly differentiated among three different functions, with three different roles of or relationships to research: setting agendas, developing new policies, and monitoring and modifying existing policies (see table 2).

TABLE 2.

Using Evidence for Policy Advice: A Functional View from Civil Servants

Area of Policy Activity	Function(s) of Evidence for Civil Servants	Nature of Relationship between Civil Servants and Evidence
Setting or anticipating agendas	May signal an emerging or neglected area for attention Helps screen the validity of interest groups' competing claims on the agenda	Civil servants are mostly defensive in reaction to research and other inputs being pushed at them for the policy agenda.
Developing new policies	Reduces uncertainty Increases confidence Prevents duplication Gives external validation for recommendation	Civil servants seek information and are motivated to pull in research and other evidence for specific policy development under deadline.
Monitoring and modifying existing policies	Provides basis for ongoing program improvement Creates currency of accountability	Civil servants want an ongoing exchange that develops longer-term interpersonal relationships with researchers for collaborative production of evidence.

Evidence in regard to setting agendas, particularly for peer-reviewed research, seems to function in two ways. First, it can signal what emerging or neglected areas may need to be on the agenda, although civil servants and their political masters still resist all but the most compelling of these areas, given the usually crowded nature of policy agendas. Second and more important, research can validate (or otherwise) the issues claimed by interest groups to be worthy of inclusion on the policy agenda. In one example, the provincial medical association was insisting that access to general practitioners was a major problem in need of attention until a population survey, conducted by ministry staff, revealed that the percentage of the population without access was, in fact, quite small. The functional role of research in setting agendas is therefore relatively small and related to managing interest groups, each bringing its own forms of evidence to support often competing claims.

In regard to developing new policies, our interviewees reported that evidence improved their confidence in making recommendations and in “speak(ing) truth to power” (Wildavsky 1979), prevented them from repetition and duplication, and, in the case of peer-reviewed research, gave politicians an extra-bureaucratic source of validation for a course of action recommended by civil servants: an imprimatur from the outside world. An example here was the evolution of a government pledge to expand prostate specific antigen (PSA) testing in men for prostate cancer. In the face of this political promise, civil servants synthesized the research showing that the PSA test's high proportion of false-positives would likely lead to a universal screening program doing more harm than good. Thus the potential pitfall of a universal program was avoided, and the election pledge was realized as a targeted screening program for high-risk males only. The functional role of evidence in developing new policies is therefore to create a more efficient process and effective final policies.

Monitoring and modifying existing policies mean not only formative and summative evaluation roles for research in ensuring ongoing program improvement but also a contribution to accountability through monitoring data on performance. Unlike the prior two areas, here the civil servant is in an ongoing rather than periodic relationship with evidence, in this case, evaluative evidence. Program evaluation and performance monitoring are increasingly seen as collaborative exercises between the evaluators and the evaluated (Patton 2008), designed to recognize the need for continuous improvement. The ongoing relationship that this approach to using evidence creates between the civil servant and the researcher can lead to spin-off benefits, akin to the interpersonal linkages envisaged by the interaction models of the policy process. For instance, in the early 2000s, the ministry's Mental Health Branch created an ongoing relationship with a group of researchers to evaluate some small-scale community mental health programs. The relationship worked out well and was immediately expanded in 2004 when the funding for community mental health was increased by 50 percent to C$167 million, thereby facilitating a real-time evaluation of major new initiatives under this umbrella. In this respect, the functional role of evidence is the currency of a learning organization committed to a culture of inquiry and improvement (Crites et al. 2009).

The nature of the functional relationship between the civil servant and the evidence differs in each area, implying the need for different kinds of tools to encourage and support research use.

For setting agendas, the evidence is pushed at civil servants by interest groups (including researchers) claiming priority for their issue/s. That is, the evidence is not fulfilling any immediate policy development need. Civil servants are not brandishing a demand for research but instead are in a defensive stance to limit the entry of research and other inputs being pushed at the system. The most useful tools are those that manage this shower of claims, helping distinguish the more important from the less important. In this area, research is often considered as just another element clamoring for attention; it is seen as useful only if it can help screen out all but the most pressing issues through the rebuttal of claims, a task most often performed by civil servants themselves rather than external researchers. The ministry's Medical Advisory Secretariat sees itself in this role.

In contrast, civil servants demand research evidence when they are developing new policies. Relevant evidence is being pulled from its sources by civil servants to inform or buttress specific recommendations, usually prepared under deadline. This is the area on which most earlier analyses of research use in policy have focused. The most useful tools are those that provide one-off summaries and syntheses of evidence for the relevant theme and deliver them on short notice through readily accessible technologies and in user-friendly formats. In this case, civil servants are in a motivated and information-seeking open stance with regard to specific and relevant research, now seen as a way to speed the path to effective policy.

Finally, the function of evidence for monitoring and modifying policy is not nearly so time limited as in the earlier cases. This function is characterized by the need of civil servants to create an ongoing relationship with evidence, using it to check in and determine whether to revise a policy or adjust a continuing program implementation. Civil servants are in a more protracted linkage and exchange relationship with researchers and their products (Lomas 2000). The most useful tools are those that facilitate this relationship and feed it a steady stream of relevant data and results. In this case, civil servants are developing a trust relationship with the sources of evidence, helping researchers understand the context, and using the research as part of their own learning as well as in discharging their specific responsibilities.

The emergence from the interviews of this functional framework in which evidence is being pushed at, pulled by, or exchanged with civil servants reflects a similar analysis conducted at the country level by Lavis and colleagues (2006). Its emergence here at the level of the provincial Ministry of Health in Ontario is a validation of its expanded usefulness. In particular, the framework's functional focus can lead to the design of tools and activities, as well as the development of civil servants’ skills, to meet the different needs for research in each of the three areas.

Although this functional framework was revealed in our interviews with the civil servants, it had not been used explicitly in the ministry's efforts to improve research use, a largely ad hoc process at that time. It would be possible, therefore, to use the framework as an audit mechanism for those efforts, showing where further activities might be worthwhile in advancing evidence-informed policy advice. The framework's focus on three specific functions for evidence, and hence the need to tailor programs to best serve those functions, highlights the activities’ strengths and weaknesses. The major strengths are in the most urgent need—pulling in evidence to inform policy development—and in response to the hardest issue to resist—funding to help researchers push their studies onto the agenda.

The most striking weakness recognized by the civil servants in our interviews is in creating exchanges between evaluators or data analysts and policy staff to support improvement in existing programs and policy. This program evaluation capacity, and the personal linkages with outside expertise that it generates for civil servants, helps move the culture more toward one of inquiry and a learning organization. Although advancing well down this path, the Ministry of Health and Long-Term Care in Ontario does not yet appear to have reached the tipping point for this change in culture. Redress for this perceived weakness might include the creation of a dedicated program evaluation branch within the ministry, the adoption of a knowledge-brokering role by its Research and Planning Branch to connect relevant researchers to program staff in ongoing relationships, and even the inclusion in annual staff performance reviews of rewards and sanctions for the presence or absence of such relationships with relevant researchers.

Lessons and Challenges

This article has described a functional perspective on the use of research in the policy process. The framework was based on interviews with civil servants in a large ministry of health, one strongly committed to improving its evidence-informed approach to policy. These interviews emphasized that evidence-informed policy advice is indeed qualitatively different from evidence-based medicine and requires different tools and often acceptance of a more conceptual than instrumental use of research. They also reinforced the view that current academic models of the use of research in policy tend to help researchers understand how to influence the policy process but not to help civil servants use research better.

The framework proved to be useful in auditing activities for “balance” across the potential functions of evidence in policy. For the Ontario ministry, it is still early in the development of many of these activities and therefore difficult to assess their specific effectiveness. However, the overall impact was noticeable in the reorientation of many civil servants toward gathering and applying evidence as a core competence.

This reorientation reflects a growing appreciation for civil servants’ competence in handling evidence, not only in Canada, but also in the public services of other countries. For instance, the United Kingdom's civil service now identifies “analysis and use of evidence” as one of four core competencies, alongside financial management, people management, and project or program management (Campbell et al. 2007). Australia's prime minister, speaking to his public service shortly after his election in 2008, outlined a seven-point vision for the Australian public service in which the “third element of the Government's agenda for the public service is to ensure a robust, evidence-based policy making process” (Rudd 2008). The functional framework outlined here therefore offers policy organizations a pragmatic tool for taking stock of their own organizational capacity for evidence-informed policymaking and may help show where it needs strengthening.

Our interviews inevitably went beyond the specifics outlined in tables 1 and 2, offering some more general lessons from a policy organization seriously committed to becoming more evidence informed. Primary among these is the central importance of having champions at a senior level able to sustain the journey. Direction came from the most senior civil servant, the deputy minister, and cascaded downward. The interviewees consistently noted how important it was to have this leadership “from the top.” This observation parallels that made for the clinical world, in which the sustainability of evidence-based clinical change programs was identified with the commitment of senior opinion leaders and local champions (Dopson et al. 2001).

The challenge presented by a central role for senior champions is whether their departure leads to the gradual decline in evidence-informed policy activity with the inevitable accumulation of time and expediency pressures. This observation underscores the need to sustain activities long enough that a cultural tipping point is reached in the policy organization, that the culture of action is replaced with a culture that combines action with evidence-informed thinking. Clearly, the Ontario ministry has not yet reached this point. But as it starts to gain a reputation for its emerging competence and government-wide leadership on thoughtful agenda setting and policy development and modification, its growing self-image should become a powerful perpetuating and sustaining factor.