Mutual Distrust: Perspectives From Researchers and Policy Makers on the Research to Policy Gap in 2013 and Recommendations for the Future

Abstract

The production of health policy-relevant research is necessary, but not sufficient, to promote its utilization in policy. Our objective was to understand the perspectives of United States’ state-level policy makers and health researchers on the barriers and facilitators to the translation of health evidence into the policy process, with a particular focus on issues related to relationship building. We conducted interviews with 215 US health services and health policy researchers and 40 state-level staffers and legislators. Researchers and policy makers faced the same major barrier to research translation: lack of dedicated time to do so. Some policy makers questioned the credibility of research, and researchers questioned policy makers’ authentic desire to use evidence in decision making. For some study participants, a mutual mistrust of the other group challenges stronger relationship formation. Interventions are needed to help both groups understand a broader role that research plays in policy making and to increase personal contact, and ultimately trusted relationships, across various actors in the policy process.

Introduction

The year 2017 has ushered in a political climate in the United States where questions about the relevance of scientific research within a highly politicized policy environment loom large. Commentators have suggested that research is in danger of being relegated as less important to the policy-making process,¹ which is particularly concerning after several years of apparent progress in promoting the translation of evidence into policy making. The Obama administration was known for using scientific research in unprecedented ways, such as in its cross-agency Social and Behavioral Sciences Team to leverage the insights of behavioral economics to make governmental programs work better to solve problems.² At the same time, experts and commentators have argued that progress has been too slow. A 2012 US National Academies of Science report called for more social science research on the conditions under which research is used or not in public policy.³ New York Times columnist Nicholas Kristof, sparking debate among the US academic community in a February 2014 column, argued that academics should inject themselves more seriously into public discourse and policy making.⁴ Given the current political climate, it is critical to reconsider a key question: What are the conditions in which research evidence is, or is not, used in health policy-making settings?

A body of empirical research in the field of knowledge transfer and exchange over the last 3 decades seeks to answer this question.⁵ This research relies on surveys and interviews with government officials, agency staff, advocates, research institutions, and/or researchers to investigate the barriers and facilitators to evidence use in policy making.^6-12 One synthesis of this research, for instance, finds that the most prominent barriers include negative attitudes and mistrust, misaligned incentives, ineffective communication approaches, and different time horizons between decision makers and researchers.⁷ A consistent conclusion in this general area of research—even a truism, at this point—is that there are vast differences in the values, worldviews, and professional routines between policy makers and researchers; these cultural and relational dissimilarities serve as a roadblock to effective bidirectional communication.^3,13

In the first study to synthesize the fledgling empirical research on use of data into an organized framework, Caplan¹⁴ described the “two-communities theory” of knowledge translation. He argued that the differing perspectives, values, and ideologies of researchers compared with government officials make their worlds isolated from and mistrustful of the other, which explains low incorporation of research in policy. Other empirical research supports the notion that the 2 groups do not sufficiently understand one another, and that this contributes to communication challenges.¹⁵

The major conclusion, then, supported by a wealth of evidence in the field of knowledge transfer, is that building personal relationships is the most important way to bridge the “gap” between the 2 communities and ultimately to more effectively translate research to its end users.⁸ Studies frequently show, for instance, that personal contact between researchers and policy makers facilitates greater evidence use in policy making.¹⁵ A recent systematic review found that increased contact, collaboration, and relationships facilitate evidence use in over two-thirds of all studies considered.¹⁶ As a result of this evidence, recommendations for more effective translation strategies often include facilitating opportunities for more in-person networking, communication, and ultimately relationship building—particularly before a specific decision making need arises—between researchers and decision makers.^7,9,17-19

While there has been much work, as outlined above, on the barriers researchers face in communicating their research evidence (ie, lack of training, capacity, time, etc) and the barriers policy makers’ face in using it (ie, constituents’ demands, credibility concerns, lack of time, other political influences), relatively few studies have analyzed relational factors of both groups simultaneously (although see Bogenschneider and Corbett’s¹² work for a notable exception). Yet examining the attitudes and perspectives of both researchers and policy makers together is essential for making recommendations of how relationships between the two can be better supported. In addition, since some of the existing empirical research on evidence in health policy making is older than the past decade,^11,18 drawing conclusions from the existing body of literature for the current context of health services and policy research needs may be inappropriate. The time is ripe for a comprehensive look at researchers’ and policy makers’ mutual perceptions, in order to understand whether and how conditions related to knowledge transfer have changed in recent years and to identify new recommendations for a political context characterized by uncertainty over the value of evidence.

Our objective in this study was thus to examine US health services researchers’ and health-related policy makers’ perceptions of the role of research evidence in the policy-making process and, particularly, their attitudes about each other. The ultimate aim of this work was to identify recommendations for future interventions to improve knowledge translation. These findings and recommendations are important to a variety of stakeholders, including research funders (such as the National Institutes of Health and foundations that have made evidence-based policy a priority, including the W.T. Grant Foundation and the Laura and John Arnold Foundation), state governments and legislators (particularly legislative research functions), universities and university-based centers that fund and promote research, and intermediary organizations, such as professional societies for researchers (such as AcademyHealth) and independent research clearinghouses (such as blogs like The Incidental Economist).

New Contribution

This qualitative study provided an in-depth assessment of the commonalities and differences between applied health policy researchers and policy makers in 2013, focusing particularly on their perspectives toward each other. The results illuminate sources of potential friction between researchers and policy makers and identify recommendations for improved processes of research translation that incorporate broader perspectives on how scientific evidence can be used across the policy-making process, not just at the point of policy decisions.

Data and Methods

Study Participants

In the summer of 2013, we collected data from researchers and policy makers at the AcademyHealth’s Annual Research Meeting (ARM) and the National Conference of State Legislatures annual meeting, respectively. AcademyHealth is the largest gathering of US health policy and health services researchers (more details on recruitment and methodology are available elsewhere²⁰). The AcademyHealth sample included 215 health services and health policy researchers recruited from a random sample of 325 preregistrants for the 2013 AcademyHealth’s Annual Research Meeting in Baltimore, Maryland, in June (66% response). We limited our sample to researchers whose primary affiliation was an academic institution. We sent each of the 325 registrants an invitation to be interviewed at the meeting along with a $2 incentive. We sent a follow-up e-mail to schedule interviews with responders, and we contacted nonresponders 2 additional times by e-mail. Interviews were conducted by trained research assistants in a designated meeting area. Twenty-eight participants were unable to complete the interview during the meeting and were interviewed by telephone.

The second sample included legislators and legislative staff attending the National Conference of State Legislatures’ annual Legislative Summit in August 2013 in Atlanta, GA. The Summit is the largest national gathering of state-level legislators. Our sample was limited to legislators and legislative staff from the United States who focused on health and/or human services issues. Inclusion in the study was based on confirmation of role as legislator or legislative staff by interview, and a total of 43 policy makers participated. Because these were intercept interviews solicited as they attended conference sessions (ie, no formal sampling frame), we report no response rate.

Consent was obtained prior to all interviews. Participants were entered in raffles for a $200 retail gift card. The study was approved by the University of Pennsylvania Institutional Review Board.

Interview Instruments

The 2 instruments were designed to identify researchers’ and policy makers’ professional activities related to the dissemination of and use of research in the policy-making process. For the researcher sample, researchers first answered a structured (quantitative) survey instrument (with results previously reported²⁰). Then they were asked several open-ended interview questions concerning activities they engaged in related to the dissemination of their own research. The interview instruments for policy makers were solely qualitative. Policy makers were asked to describe specific activities they engaged in related to finding and using research evidence in the policy-making process, with probes related to their use of social media. The open-ended question that is the focus of this analysis for the researcher sample was, “What is the biggest challenge as a researcher to influencing policy?” The open-ended question that is the focus of the analysis for the policy maker sample was, “What do you see as the biggest challenge to using research evidence in health policy?”

Analysis

All interviews were transcribed and de-identified. First, all members of the author team discussed a small set (N = 5) of transcripts from both groups. Through this inductive process, we identified the main emergent themes, mentions of challenges and barriers to translation, best practices for finding and/or communicating research (how it is accessed, how it is communicated, and by whom), characteristics of the policy-making process, and differences between researchers and policy makers. While many themes were identified in both researchers’ and policy makers’ transcripts, others were unique to one group (such as “credibility of science,” which emerged only in the policy maker sample). These themes comprised the initial codes in our codebook; additional themes were identified and added to the coding scheme as needed in a pilot coding process led by 3 of the authors. Before coding all transcripts, a subset of transcripts were double-coded to ensure interrater reliability. Disagreement was reconciled and agreement exceeded 90%. Next, 2 authors coded all transcripts using the final codebook in NVivo 10.0. After all codes were entered, the study team created analysis memos for all codes, describing the major themes under each code, key outliers, and a comprehensive list of illustrative quotes. Through a comparative method, the team analyzed all the researcher and policy-maker memos for consistency and disagreement. We also examined differences in themes for researchers by their academic rank (assistant, associate, full professors) and for policy makers by whether they were elected legislators or staff. Differences in these characteristics did not stand out as important and so we report the findings at the level of the group as a whole. For the purposes of the present analysis which seeks to illuminate the role of relationships in research translation, we focused our results on the barriers and facilitators of research translation and attitudes relevant to relationships between researchers and policy makers.

Results

Table 1 describes the characteristics of the 215 researcher participants and 43 policy maker (32 legislators, 11 staff) participants. There were more female than male researchers, while the policy maker sample was nearly three-fourths male. The largest group (44%) of researchers were within the first 10 years of degree; similarly, the largest group (32%) of policy makers had been in their position for 6 to 10 years. The policy maker sample was well represented across partisan groups, with somewhat more Democrats (47%) than Republicans (33%), with the remainder nonpartisan (for staff) or unknown.

Table 1.

Demographic Characteristics of Researcher and Policy Maker Study Participants.

	n (%)
Researchers (N = 215)
Years since degree
0-10	95 (44.2)
11-20	62 (28.8)
21-30	38 (17.7)
31+	20 (9.3)
Gender
Male	90 (41.9)
Female	125 (58.1)
Academic rank
Assistant	87 (40.5)
Associate	48 (22.3)
Full	62 (28.8)
Other	18 (8.4)
Degree
JD	4 (1.9)
MD	35 (16.3)
MD-JD or MD-PhD	10 (4.8)
Master’s	3 (1.4)
PhD	163 (75.8)
Policy makers and staff (n = 43)
Age, y
20-29	1 (2.3)
30-39	4 (9.3)
40-49	8 (18.6)
50-59	13 (30.2)
60-69	12 (27.9)
70+	2 (4.7)
Unknown	3 (7.0)
Gender
Male	31 (72.1)
Female	12 (27.9)
Professional role
Legislator	32 (74.4)
Staff	11 (25.6)
Years in position
0-2	2 (4.7)
3-5	7 (16.3)
6-10	14 (32.6)
11-20	8 (18.6)
20+	7 (16.3)
Unknown	5 (11.6)
Partisanship
Democrat	20 (46.5)
Republican	14 (32.6)
Nonpartisan	7 (16.3)
Unknown	2 (4.7)
Legislature type
Full time	6 (14.0)
Part time	35 (81.4)
Unknown	2 (4.7)
Census region
Northeast	6 (14.0)
Midwest	5 (11.6)
South	22 (51.2)
West	7 (16.3)
Other (US territories)	2 (4.7)
Unknown	1 (2.3)

The analysis generated 4 broad findings related to policy makers’ and researchers’ perceptions of one another and of use of evidence in policy making (see summary in Table 2). First, despite coming from different professional backgrounds, both groups faced consistent barriers to communicating and using research effectively, barriers that impede efforts to build relationships across groups. Second, the 2 groups had varying perspectives on the value of research in policy making, with a high degree of cynicism on the role of evidence in policy making expressed by both groups. Third, for some study participants, negative perspectives about the use of research in policy making corresponded with a strongly expressed mistrust toward the other group. Fourth, both groups described a mismatch between the needs of policy makers and the approaches taken by researchers.

Table 2.

Themes From Health Researchers’ and State Policy Makers’ Perspectives on Research Use in Policy Making.

Theme	Subthemes	Illustrative quotes (source: policy maker or researcher)
Consistent barriers to translating or use of research	Lack of time Researchers not understanding policy process	“If you want to do research there’s time constraints, so if you’re working on dissemination then you’re not doing research.” (Researcher)
Attitudes about value and credibility of research	Policy makers: cynical about science and its objectivity Researchers: cynical about policy makers’ intentions to base policy on science	“Most of the information that comes to legislators is not pure science data. It’s typically biased, and so you have to take the time to figure out who is publishing the article and what their agenda might be.” (Policy maker)
Mistrust toward other group	Research institutions and researchers are biased Policy makers are influenced by money and agendas	“Well, you know unfortunately a lot of policy makers don’t base policy on truth and reason. They base it on ideology and the interests of the powerful people who pay for their campaigns or who they are themselves.” (Researcher)
Mismatch between needs of policy makers and researchers’ approaches	Face-to-face contact is best Credible evidence from existing relationships, including internal organizations and staff Researchers lack training, time to communicate and forge relationships Social media unlikely to overcome these barriers	“You know, no one’s just gonna read someone I’ve never heard of. Now if you get to be known enough, like if maybe some professional at a university where I live that I know are very knowledgeable . . . there’s a credibility and I don’t know how to say it other than there’s just time to recognize [that] okay, I’ve seen the track of where they’re going and I agree with the thinking.” (Policy maker)

Consistent Barriers

Researchers and policy makers were consistent in their acknowledgment of the main barriers that prevent research from being communicated to, and used by, policy makers. Specifically, researchers most commonly identified competing demands on their time that limited their ability to communicate science to policy makers. Many researchers noted that the current expectations of the tenure and promotion process are not compatible with taking the time to engage with policy makers. As one respondent said,

if you want to do research there’s time constraints, so if you’re working on dissemination then you’re not doing research, and if you want to do research, which is why you’re in the business probably, it’s a time cost, and there’s no promotional incentive within the university promotions structure to do that.

For their part, policy makers also most frequently identified being very busy as a reason they do not access research information more routinely, saying, for instance, “As citizen legislators, a time constraint is our biggest problem.” Many felt overwhelmed by the amount of information that they were expected to process, receiving reports and other information daily. Like researchers, they faced hard constraints in their ability to absorb research into the policy work that they do. Several policy makers, in addition to commenting on the deluge of information they receive, also mentioned difficulty finding the “right” information for their needs.

Another consistent barrier cited by both groups was that researchers do not understand the policy process sufficiently. Researchers acknowledged that the linear model they may have been taught whereby research gets created in academia (detached from the policy context) and then the results subsequently communicated to the policy setting to be used at the point of decision making is wrong and potentially misleading. A few researchers explicitly stated that some of their peers erroneously presume that policy makers are “rational decision makers working in a very logical stepwise fashion.” This misunderstanding of the policy process contributes to researchers’ feelings of frustration about or alienation by the policy process, and also leads them to misjudge their likelihood of influence. As one stated, “A lot of researchers, especially early career researchers, think that all they have to do is tell the truth and powers will listen and that just doesn’t happen.”

Policy makers also expressed frustration with researchers’ seemingly naive perceptions of the policy process, which, they argued, leads to ineffective communication approaches. As one policy maker indicated,

I’ve seen a lot of academics come in prepared to teach. No, no. This is not a teachable moment, okay? You’ve got to explain, got to develop credibility, but much more quickly than that. And then you’ve got to hit the ground with things that legislators can really figure out and understand.

Policy makers also mentioned that researchers commonly use strategies that belie their lack of understanding of policy-making processes, such as not using constituents as messengers of research and sending too lengthy reports.

Attitudes About the Value and Credibility of Research in Policy Making

More than half of policy makers stated explicitly that they believed research has value in their policy work. They think research has value because it “can cross party lines” and that they have a responsibility to approach tough problems using the most effective mechanisms. In contrast, a single outlier (an elected official) explicitly questioned whether research is valuable, commenting that policy makers may just want to please academics by saying that research has value. He reacted against the notion that research should have an elevated influence in policy making over other factors, commenting that this was antidemocratic: “Who elected those institutions of higher education to do the research? Who elected them to tell us what to do?”

Policy makers considered research evidence to be a component of politics, not removed from it. They did not perceive research evidence as neutral, but considered it potentially politicized just as any information in the political process might be. As one stated, “Most of the information that comes to legislators is not pure science data. It’s typically biased, and so you have to take the time to figure out who is publishing the article and what their agenda might be.” Another indicated that data were used for political persuasion: “Both sides have statistics.”

Most policy makers expressed concern about potential bias among researchers, sponsors, and funders; 25 elected officials and 9 staffers raised the point that science is not always credible. This concern mostly stemmed from unease about funding and interest groups; for instance, citing an experience with research about indoor tanning, a policy maker stated, “You have to be a critical reader and think about who’s really driving this. Are you gonna trust that the people who are just studying this etiology of this increase in melanoma don’t have anything against tanning booths?” Others mentioned that research is necessarily a mix of higher and lower quality work, with demands on researchers to be prolific. In assessing research credibility, policy makers mentioned details as specific as study sample size and generalizability issues as well as more general concerns about researchers’ interpretation and advocacy for a particular perspective. As one policy maker stated, “The numbers can say what you want them to say.” Concern over the credibility of science contributed to hesitancy about the misuse of research in decision making, as a policy maker voiced:

I see the biggest challenge is [policy makers] not knowing how to read and judge research. They don’t know: if someone comes up with something that’s very politicized, but someone can say anything, right, it’s a free country. But [policy makers] don’t ask, is it in a peer reviewed journal? Has the result been replicated? Can you tell me where to find that?

Researchers perceived that policy makers may not seriously care about research, but unlike the policy makers—who frequently discussed the politicization of science and evidence—researchers infrequently identified concerns about bias in research. In contrast, they often characterized the nonresearch-related factors that go into policy as “biased.” As one described,

I think there are a lot of biases, that people approach policy making from the perspective of what seems most accessible, what seems most important to you, based on what your constituents are telling you, based on what’s in the media, and it can be very biased.

Researchers were cynical about policy makers’ authenticity in their consideration of research in their decisions, relative to political considerations like lobbyists’ and constituents’ influence; several noted this concern has only gotten worse in recent years with heightened politicization of health policy issues as well as more politically polarized legislatures. As one said,

I would say it’s, the political process is . . . it feels like a completely separate world and it’s very much out of our control. So the best we can do as researchers is do the best science possible and though we have very little control over how it will be used.

Some researchers acknowledged that this cynicism about the political process could drive researchers away from wanting to even engage in research translation, because “it might seem like a waste of time to get too closely involved.” Few researchers shared the dominant view among policy makers that research should be carefully critiqued. An outlier, one researcher made the point (as did the policy makers) that research can be “good, bad, and ugly” and that researchers sometimes push out research that aligns with a particular perspective; this researcher suggested that policy makers not “jump to every little thing that researchers find.”

Mutual Mistrust

Some policy makers’ and researchers’ cautious attitudes about research use in policy making went beyond a “healthy cynicism” of research influence in policy making (such as scrutinizing research funding or sources, and being aware of the nonevidentiary yet important influences in the political process). In fact, a strong and vocal mistrust emerged from some policy makers in their perceptions of academic research; a strength of mistrust that was matched in some researchers’ perceptions of policy makers.

Some policy makers believed that research is manipulated to be persuasive:

I would also say that it is a challenge where people are trying to persuade other people, you get into cases where some people are presenting information that is out of context or not properly supported by a well-designed research, and it’s more zealotry than factual conclusions.

Policy makers’ distrust of research was based in concerns regarding the academic setting. A small number of policy makers made statements that academic institutions care more about seeking publicity or funding than promoting the common good. One participant noted,

I think sometimes universities will, you know, affiliate themselves with—or allow the relationship to continue because there’s publicity involved for the institution. Although it may not—the underlying level of credibility of the work that’s actually being done may not be all that good.

Two mentioned a concern about “liberal bias” of universities.

Some policy makers negatively noted the influence of funding on research. As one policy maker said, “You will see articles published, and at the end, even though it doesn’t say it in the article . . . you’re left going, all they’re looking for is additional funding for whatever project they’re looking for.” One researcher also described this perspective, acknowledging that academic institutions and the researchers within them often face incentives to generate fundable research, not actionable research:

If you’re in an academic setting, I think the idea that for tenure, the hurdles that are put up for tenure, they may not be that interested in research that has to do with policy change, they may be more interested in research that brings in big dollars.

On the contrary, some researchers believed that policy makers were more strongly influenced by money than scientific evidence. One researcher said, “Well, you know unfortunately a lot of policy makers don’t base policy on truth and reason. They base it on ideology and the interests of the powerful people who pay for their campaigns.” A few others suggested that politicians are not open to hearing about a particular research finding because they have predisposing policy ideas they want to pursue, which leads to a censoring of evidence (as one said: “If you’re finding something not in line with what they want, they will not even put it out there.”) Expressing a high degree of mistrust, one researcher said: “Policy makers really don’t care about what the researcher says. They just care about who’s paying their bill.” A large group of researchers, however, acknowledged that policy makers may not use research as a result of the policy makers’ busy schedule and time constraints, rather than a willful nonuse of research.

Finally, one policy maker—albeit an outlier in how vocal he was in his distrust of researchers—believed that researchers are arrogant and “belittle” policy makers for not having sufficient knowledge: “You’re saying I can’t get in the weeds in the nuances—because [researchers are] so smart that the legislators just don’t understand?” A single researcher’s comments actually supported this belief, saying, “We’re on an intellectual plane that is so different from what plane the policy makers are on.”

Mismatch Between Needs of Policy Makers and Actions of Researchers

When solutions to facilitate more research translation were offered in the interviews, researchers and policy makers both described some best practices in communicating research and in finding or using research. Both groups talked about the importance of keeping research findings brief, actionable, and to transmit them using credible messengers. Policy makers frequently noted that the best way to find “good science” is to get it from credible people and organizations. As one participant said,

You know, no one’s just gonna read someone I’ve never heard of. Now if you get to be known enough, like if maybe some professional at a university where I live that I know are very knowledgeable . . . there’s a credibility there . . . I’ve seen the track of where they’re going and I agree with the thinking.

Or as another confirmed, “So if someone I know and respect in [home state] can explain or bring me a piece of research and put into the vernacular, it’s going to be better because I can ask questions about it.” Many relied on individuals they already knew well to provide information when needed. One elected official said, “Actually, my own physician wanted to—there was a piece of legislation that he was very interested in, and so he came to the legislature and testified.”

Policy makers noted that credibility can come not just from the individual researcher but also the institution or organization that can help communicate the research, including local universities, hospitals within their district, and intermediary organizations like the National Conference of State Legislators, Pew, and Council of State Governments. Many policy makers noted that they find the most trustworthy sources of research from their internal contacts—their own staff, committees, and legislative research branches. After pursuing these internal channels, legislators and staffers noted that local researchers are the best sources of research, especially via in-person meetings (noted by half of the 24 policy makers who mentioned a specific preferred communication channel). As one policy maker noted, “I think, at least in our legislature, reaching out and making it that individual contact is probably the best but most time-consuming.”

While not a majority, some researchers also recognized that forming relationships with policy makers could be effective and some were indeed interested in building relationships and using alternative forms of communication, including face-to-face meetings. Sometimes, though, researchers did not know the best way to connect with policy makers. Some noted that traditional scholarly channels were not effective, saying, “I don’t think those high impact factor journals are making the impact they used to make because people are going so many other places. And I don’t think the policy makers are necessarily going to those journals.” However, few felt prepared to disseminate effectively using alternative mechanisms. One spoke to a lack of dissemination training: “Writing policy briefs is something that we don’t get a lot of training in and we don’t have a lot of attention towards.” With regard to forging relationships on a personal level with policy makers, researchers noted 3 major challenges. The first was that they are not necessarily working in a single field, so the people with whom they would need to cultivate relationships would frequently change (ie, across a topic area like Medicaid or long-term care); they also reinforced the lack of time they have to cultivate relationships; and they noted that the “interesting questions” that policy makers might pose may not have funding attached to them, so researchers cannot add this to their workload when they have other pressing research grants and obligations. So while most researchers understood the need to modify their typical formats and channels of communication to meet policy makers where they are, the researchers on a whole did not demonstrate much interest in doing so given their professional constraints and, as mentioned at the outset, extreme time pressures.

Social Media as a Possible Bridge Between Worlds?

Policy makers and researchers alike viewed social media with ambivalence. Policy makers, supporting their perspective that face-to-face is best, indicated that social media would be unlikely to overcome the need for trusted relationships. As one said, “I would rather speak with a person one-on-one than through Twitter.” On the contrary, there were a few positive outliers in the policy maker sample who noted using Twitter as a method for identifying topics of interest related to research:

I read linked articles off of Twitter. So if something grabs my attention that this just published about X, Y, Z topic, I’ll go read it. I read off of that more than I do out of my newspapers or anything else.

Researchers, too (as noted in more detail elsewhere),²⁰ were ambivalent that social media like Twitter could be an effective mode for communicating research results, and were concerned that it might not actually reach policy makers, that it could force researchers to “dumb down” their message, and that it could lead to reputational consequences if messages are not carefully constructed (ideally by others who take this communication function on as a designated responsibility).

Discussion

These findings do not paint a particularly sanguine picture of the potential connections between research and policy making that would, as knowledge transfer research suggests, lead to more effective translation. The notion that researchers and policy makers reside in “two communities”¹⁴—first articulated in 1979—seems alive and well almost 35 years later, as expressed in strikingly explicit terms among both researchers and policy makers in 2013. This suggests little change since some of the earliest studies on knowledge transfer first identified mistrust as a major barrier to effective research translation. Innvaer and colleagues described “perceived political naivety of scientists and scientific naivety of policymakers”^15(p241) as a barrier identified in 8 of 24 empirical papers. It was surprising to observe such overt mistrust expressed on both sides particularly among a group of relatively applied health researchers and health-informed policy makers. Given that these results from 2013 suggest strong degrees of underlying mistrust, as the broader context of politics shifts toward heightened scrutiny of science, one could anticipate even more vocal mistrust in 2017. Social psychological research suggests that predisposing negative beliefs can lead to the amplification of differences and enhanced polarization in perspectives.²¹

Both policy makers and researchers held strong negative perceptions and misunderstandings of the other, attitudes that would challenge the formation of stronger relationships. Most important, they differed in how they thought about the very concept of “research.” Policy makers think of research as routinely “biased,” used to support one perspective, and a part of the political persuasion process. This perception likely follows from how they mostly interact with research: through the testimony process as presented by lobbyists and interest groups on one side or the other of a legislative debate. A negative perception of academic institutions as biased also contributed to negative perspectives on scientific credibility. Researchers, not surprisingly, viewed research as objective, and privileged research normatively as an important influence in policy making; they expressed disillusionment when research was not used in policy decisions given the host of other influences on policy making —from financial influences to constituent demands to political agendas and coalitions.

Both of these perceptions—of research as biased or of political decisions as overly influenced by nonevidentiary factors—rely on an incomplete and inaccurate perspective on the role of research in policy making. Both groups perceived research as being most relevant at the point of a decision about some policy matter, an “instrumental use” of evidence, to apply Carol Weiss’s classic 1979 typology.²² Rarely did study participants discuss how research might be used in setting the agenda for what is important, in framing the importance of policy problems, or in providing context for understanding issues, all more common uses of research in policy making.⁶ For both groups in our study, their “top of the head” conception of what it means to use research in policy meant decision making during legislative debate. This overemphasis on the instrumental application of data or evidence into decision making is hardly new; Nathan Caplan critiqued this view as “an image of reality too narrow to provide a suitable foundation on which to premise decisions involving the more important policy issues.”¹⁴ Similarly, UK scholar Katherine Smith suggests that the ideas or concepts that emerge from public health have indeed shaped policy in numerous ways, but a narrow conception of research evidence would suggest otherwise.²³ Because both researchers and policy makers in our study still hewed to an instrumental conception of research use in policy making (and then were either disappointed in its nonuse or alienated by its seemingly politicized use), broader exposure to the many ways in which research might have a role in policy making is needed, as described below.

While the dominant finding of this study concerns the high mutual mistrust expressed by researchers and policy makers, the two groups also faced similar constraints and barriers that could be used to forge more common ground between researchers and policy makers and also inform the creation and implementation of interventions. This study makes clear, for instance, that any intervention to communicate research would need to account for the fact that both groups are vastly overcommitted in the time resources they have available. Thus, efforts to convene the groups together need to carefully account for scheduling issues in order to be successful and also recognize that their often-contradictory perceptions and beliefs would need to be carefully navigated by those convening the groups (see, eg, Kerr et al¹⁹).

Recommendations and Future Directions

The similarity between findings in this study and the level of mistrust demonstrated in research on this topic 20 or more years ago^15,16 suggests gaps in knowledge translation are unlikely to improve organically. Indeed, while “build relationships” has been the dominant recommendation in the literature, our study suggests that this conclusion may be simplistic and insufficient. Active investment and interventions are needed to infuse evidence into multiple points in the policy process across a wider range of actors. Below, we focus our recommendations on those in the research and intermediary communities.

First, others have recommended that researchers need training in dissemination and in policy making,²⁴ and indeed our work supports this. But specifically, training opportunities must describe the spectrum of ways that research can influence policy making: from prioritizing the importance of various health problems, to identifying what issues can/should be framed as policy problems where government has a role to play, supporting a particular type of policy approach, and evaluating the effects of policies after they are implemented (including their effects on behavioral outcomes, cost outcomes, and on more abstract outcomes such as citizens’ attitudes and beliefs). In addition, researchers need more training in policy making so they can appreciate the nonlinearities of the policy process and understand what politics is, rather than something to be wary of. This is consistent with the US National Academies report, which urged for training among policy analysts and evaluators to incorporate social science findings on the conditions under which research is and is not used in policy making, rather than a more traditional education which privileges the value of scientific evidence over other decision-making modalities.³ A deeper understanding of politics and policy making can illuminate more potential points of engagement and intersection, so researchers move beyond an inaccurate (and self-defeating) linear model that proposes research production on one end and decision making on the other, as implied by a problem framed as “two communities.”²⁵ In fact, there are numerous other points of interaction and intermediaries, from media to advocates to interest groups, all of whom can establish relationships with researchers and with policy makers and interact at various points throughout the nonlinear policy process.²³

Second, research institutions should take on more active roles in translation and reducing barriers for researchers.²⁶ Universities and their dedicated policy-related centers can be engaged in more active outreach, relationship building with policy makers (and those trusted by policy makers, such as important advocates), learning who the appropriate committee staff are in state or local policy bodies, and other opportunities to forge personal relationships.¹⁹ These investments are laborious, and as our findings make clear, individual researchers rarely can invest the type of time and energy needed. At one of our institutions, a health policy institute (Leonard Davis Institute of Health Economics) has developed a robust translation and dissemination program that relies on joint efforts of staff and faculty. Staff prepare briefs summarizing individual research publications and synthesize research related to policy areas—sometimes even engaging faculty outside of the institution. Twitter is used to build social media dissemination networks. A blog serves as a vehicle to deliver expert commentary. Conferences and forums focused on specific topics (eg, health insurance exchanges) help build personal relationships with policy makers. In addition, policy institutes can play another important role. In our prior research²⁰we found that researchers value having third parties promote their research to avoid perceptions of “self-promotion”— a concern that we found to be common among academic researchers. All of these efforts require substantial resources and the traditional model of grant-based funding for research is often not aligned with these efforts. To effectively serve the needs of the public, universities and funders will need to identify strategies to fund these efforts—possibly through more dedicated funding written into individual grants, philanthropy, or other mechanisms through institutional budgets.

Third, research institutions should evaluate whether tenure and promotion standards sufficiently prioritize and recognize public communication and policy impact. New measures of research visibility and impact such as Altmetric scores are being used more frequently by journals. However, we are not aware of many academic institutions using measures of public visibility when evaluating faculty for tenure. The American Sociological Association recently recommended that public impact be explicitly included as tenure criteria, to be measured by, for instance, soliciting letters from entities outside of academia to provide testimony as to the policy impact of scholars’ work.²⁷ If evaluation standards change, faculty would have more direct incentive to engage policy makers and these activities would become normative in the research community.

Fourth, individual researchers who are especially committed to having impact (and who have institutional support, as described above) should adopt research approaches that connect policy makers, researchers, and intermediaries. Commentators have suggested various research approaches that apply knowledge transfer models directly into their design, such as models of exchange or “strategic science,” but with maintaining scientific rigor and integrity.^28,29 As an example, the new US-based Robert Wood Johnson Foundation program Interdisciplinary Research Leaders connects researchers and community leaders from the beginning with the goal of producing research that, by design, will have greater impact. Funders that care about their return on research investment should be encouraged to issue similar calls for proposals that prioritize action and dissemination and thus help to transform researchers’ traditional designs from investigator-initiated to research designs that have early and substantial policy maker and practitioner input.

Fifth, more research is needed to identify effective processes for promoting stronger linkages between researchers and policy makers and to evaluate both translation mechanisms and relationships. The knowledge exchange field should move beyond describing the problem (such as in the current study) to implementing and evaluating solutions. Such interventions, and their evaluations, should be developed in partnerships among researchers, intermediaries, and policy maker end users. As a simple intervention, researchers can experiment with different approaches to presenting research on a particular topic.³⁰ More complicated, multiphase interventions based on behavioral change theory might be designed and tested with both researcher communities and policy maker communities (government agencies, elected officials, or staff) to determine how best to shift behaviors of all actors involved toward greater communication and incorporation of research. A notable example of this is the ongoing Supporting Policy In health with Research: an Intervention Trial (SPIRIT).³¹ This study is a randomized trial involving 6 health policy agencies in Australia, and employs multiple interventions (audit feedback, training, supportive assistance, information exchange with researchers) with the goal of increasing agency capacity to use research evidence in their programs and ultimately more seeking out and use of scientific evidence.³¹ Of course, other evaluation designs beyond randomized trials are also needed to assess the output of varying research-policy partnerships. Moreover, preliminary descriptive research such as developing and testing outcome measures like metrics of research use or research impact on policy making will also help push the knowledge transfer field forward.

Sixth, technologies such as social media have some appealing qualities: a low-resource-intensive, and potentially quick fix for communicating research evidence and connecting—albeit via virtual networks—with policy makers.^20,32 However, our study revealed ambivalence on both sides about the value (in 2013) of social media. State policy makers voiced their preference for face-to-face communication, and while some users of social networking sites may indeed interact in relationship-building ways, it remains an empirical question whether and how virtual social networks can substitute for “real” interactions in the policy process as well as whether researchers and their institutions are meaningfully connected with the appropriate end users of evidence. In addition, while tweeting and posting may seem to be efficient in terms of time (they are short communications, after all), crafting the right message at the right time and curating the research information takes strategic attention.

While this research has illuminated some concerns, our findings should not be interpreted to signify that change is required for all researchers or all policy makers; nor are researchers or policy makers the only appropriate targets for change. Some researchers conducting more basic health services research do not have interest in or capacity to translate their work themselves and may not need to change their approach, just as policy makers do not necessarily need to change their behaviors or relationships to make better policy. This study confirms the importance of other intermediaries or knowledge brokers as the focus for intervention,³³ especially those who are already trusted by policy makers. Moving high-quality research syntheses into the hands of advocates, lobbyists, journalists, or other similar knowledge brokers might be a more scientifically valid and politically successful approach than trying to make the individual researcher the messenger of discrete study findings. As mentioned above, future research in knowledge transfer in health policy should consider more sophisticated models of the policy process and multiple “communities” of study (see, eg, critiques in Smith²³).

Limitations

Our research had several limitations. First, policy maker study participants were recruited using a convenience sample of legislators and legislative staff attending the National Conference of State Legislatures’ annual Legislative Summit, and researcher participants were recruited from a random sample of university health services and health policy researchers attending the AcademyHealth’s Annual Research Meetings. Researchers and policy makers attending these meetings may not be representative of researchers and policy makers overall. Researchers attending the meeting have an applied focus which would lead them to have more, not less, interest in translation to policy settings. Similarly, policy makers in the study were all attending the Summit and thus have at least some interest in using evidence to inform policy making; as state-level policy makers, they are unlikely to reflect the views of policy makers at the local or federal level. In contrast, the researcher participants were drawn at random from a sampling frame, and so we are relatively confident that the researcher sample is representative of this population. That said, health services research is only one type of research relevant to health policy decision making. It will be important to assess the behaviors and attitudes of researchers in nonhealth fields that have health policy relevance (eg, education, transportation, housing) as well.

Second, we relied on self-report to measure attitudes as well as information translation and information-seeking behaviors. This method could be susceptible to recall or social desirability bias. However, both groups discussed their distrust and hesitancy openly; if their responses were socially biased, we would have anticipated them to present warmer attitudes toward researchers (for policy makers) and more enthusiasm about evidence translation and policy impact (for researchers). Other methodologies—such as examining institutional guidelines for tenure and promotion, or examining hearing testimony for documentation of research evidence—together could triangulate the barriers and facilitators that both groups face.

Conclusions

Our study provides a sobering view on the potential for researchers and policy makers to form relationships. Given the constraints and incentives both sides face, overcoming the distance between researchers and policy makers is extremely challenging, and simple approaches will not be sufficient. Multilevel approaches that expand the role and capacity of intermediaries might include the active support of research institutions, the development, implementation, and evaluation of strategic interventions, as well as more recognition and use of the range of policy actors from advocates to journalists. Relationship building across this greater set of actors is key to overcoming the persistent divides that have long plagued the research and policy communities.