Lawmakers' use of scientific evidence can be improved

Significance

This study is an experimental trial that demonstrates the potential for formal outreach strategies to change congressional use of research. Our results show that collaboration between policy and research communities can change policymakers’ value of science and result in legislation that appears to be more inclusive of research evidence. The findings of this study also demonstrated changes in researchers’ knowledge and motivation to engage with policymakers as well as their actual policy engagement behavior. Together, the observed changes in both policymakers and researchers randomized to receive an intervention for supporting legislative use of research evidence (i.e., the Research-to-Policy Collaboration model) provides support for the underlying theories around the social nature of research translation and evidence use.

Abstract

Core to the goal of scientific exploration is the opportunity to guide future decision-making. Yet, elected officials often miss opportunities to use science in their policymaking. This work reports on an experiment with the US Congress—evaluating the effects of a randomized, dual-population (i.e., researchers and congressional offices) outreach model for supporting legislative use of research evidence regarding child and family policy issues. In this experiment, we found that congressional offices randomized to the intervention reported greater value of research for understanding issues than the control group following implementation. More research use was also observed in legislation introduced by the intervention group. Further, we found that researchers randomized to the intervention advanced their own policy knowledge and engagement as well as reported benefits for their research following implementation.

Recent public crises have further illustrated the importance of policymakers using scientific research to craft effective public policies (e.g., opioid epidemic, humanitarian emergencies at the border, COVID-19 pandemic). Yet, despite the ongoing desire among the scientific community and general public to see research being utilized by lawmakers (1), little rigorous study has investigated the effectiveness of approaches to increase policymakers’ use of research evidence (URE; ref. 2). Particularly concerning is that the promising strategies currently available have yet to undergo rigorous experimental evaluation to see if they can change lawmaker behavior. Ultimately, if the scientific community truly wants to see research used, it is time to develop engagement strategies that are themselves evidence-based.

Growing scientific study of how to improve the use of scientific evidence has shed light on the “social side” of successful research translation and evidence-based policymaking (2–5). Specifically (4), theoretical work and empirical studies have demonstrated that sustaining researcher-lawmaker relationships may be essential for supporting URE throughout the policymaking process (6, 7). In particular, structures provided by intermediary organizations have the potential to support trusting relationships between the research and policy communities (4, 6). However, work is needed to experimentally test the effectiveness of approaches designed to facilitate these processes.

Although researchers’ engagement is critical for bridging research and policy, they face numerous barriers when navigating the policy arena (5, 6, 8, 9). Formal intermediary support for researchers can help improve the frequency and manner of policymakers’ URE (10, 11). This includes understanding restrictions around outreach, overcoming divergent professional norms, and adapting to the dramatically different pace of policy settings. For instance, researchers tend to engage in relatively slow decision-making, while policymakers engage in prompt policy actions in response to opportunities or crises (7, 8, 12, 13). Timeliness of researcher engagement is particularly challenging since public policy goals often shift suddenly in response to socio-political factors (9, 14). Thus, there is a need for engaging researchers in real-time during discrete, time-limited opportunities for policy change (10, 15).

Policymakers can decide to use research evidence for varied purposes or intentions. A widely used typology in URE investigations is informed by foundational work of multiple scholars (16, 17). While researchers often deplore political uses of research for persuading others, justifying, or challenging existing policy proposals (i.e., tactical use), research evidence can also be used to guide policy development itself. This includes instances in which research is used to directly inform policy decisions (i.e., instrumental use) as well as instances in which research is indirectly used by changing the way policymakers think about problems or solutions (i.e., conceptual use). While instrumental uses may be relatively observable in specific policy efforts, conceptual use may influence a broad array of decisions in a more indirect manner (16, 17).

While some experimental study of evidence use is occurring at the state level, no work has considered how to improve Congressional evidence use (18). In an effort to create evidence-based strategies for increasing policymaker-researcher engagement and supporting URE by lawmakers, we report here on a randomized controlled trial of such a strategy with the US Congress. In this study, we randomized congressional offices and researchers to receive a promising approach for improving URE known as the Research-to-Policy Collaboration (RPC).

The RPC is a theory-based and manualized intervention for supporting lawmakers URE (Fig. 1). In particular, this work corresponds with Weiss’ conceptualization of both problem-solving and interactive models of knowledge use (17) by first eliciting policymakers’ needs and then facilitating interactions with researchers. First, a formal legislative needs assessment is used to identify policymakers’ goals, priorities, and need for scientific evidence (e.g., epidemiology and etiology, examples of successful interventions). Then, researchers who have expertise corresponding with those policy domains are coalesced into rapid response networks. These networks are provided with capacity building to increase their readiness to engage with congressional offices, fluency in the policy process, and best practices for translating research. Rapid response teams of researchers are then matched with offices based on office needs assessment results. Next, facilitated meetings occur between the office and researcher teams to further address their needs for scientific evidence. The ultimate goal of the RPC model is to create durable and productive collaborations that move beyond initial requests—with offices calling on researchers for additional questions and needs. The current study examines early indicators of outcomes associated with the model in early stages of this interactive process. Specifically, we examine outcomes associated with an implementation of the RPC pertaining to US federal child and family policymaking, although there is potential for the RPC to be used to support policymaker URE in other disciplines, as well as state or international contexts.

Fig. 1.

The RPC Intervention Model. Step 1: Policy Identification involves initial outreach to legislative staff and uses a semistructured needs assessment to inquire about policymakers’ overarching policy goals for the legislative session. Step 2: Rapid Response Network Development involves identifying researchers who have expertise relevant to policymakers’ goals and are willing to contribute to research translation efforts. Their areas of expertise are cataloged in a strategic resource mapping process that builds capacity for matching researchers with policymakers. Step 3: Network Capacity Building occurs through didactic and experiential training that aims to increase policy skills and engagement. This includes training on adapting to legislative norms without violating lobbying regulations, as well as opportunities to respond to lawmakers’ interests identified in Step 1. Step 4: Legislative Needs Assessment identifies short-term priorities and needs in anticipation of matching policymakers with researchers who have corresponding experiences and scholarly interests. This semistructured assessment is action-oriented to identify ways that researchers might support legislative efforts. Step 5: Rapid Response Meetings engage legislative staff and researchers in direct interactions to discuss research, as this is a theorized mechanism for facilitating relationship development. Meetings aim to support the codevelopment of science implications, since research interpretation is a formative and iterative process. Researchers respond to initial legislative requests and plan next steps for ongoing collaboration. Researchers are invited for these meetings based on prior RPC participation, time availability, relevant scholarly interests, and geographic similarities (e.g., researchers having done work in the state the congressional member represents). Step 6: Initial Strategic Planning for rapid responses follows immediately after meetings to summarize goals, determine next steps, prioritize and create a timeline, and identify point person(s) for follow-up. Step 7: Ongoing Collaboration includes rapid responses to legislative requests. As an example, this could include collecting and summarizing research resources, planning briefing events or testimony, or publishing written products for dissemination (e.g., briefs, op-eds).

Congressional office engagement is initiated by scheduling a needs assessment conducted by a trained policy associate, which often occurs in-person, but may also be done remotely. Some offices meet multiple times prior to or for the purposes of engaging with researchers, whereas others may meet less often, especially contingent on the clarity in next steps for ongoing collaboration. This interactive process is intended to generate requests for researcher engagement, including translational deliverables such as policy briefs and factsheets, congressional briefings, and testimony, as well as requests to review or provide legislative language for bill drafting (5, 8). Requests, in turn, provide researchers with opportunities to engage in the policy process and create tangible products that align with their professional incentive structures. Throughout the process, researchers are supported with training and technical assistance that facilitates appropriate translation and exchange of research. Training is provided by RPC implementers and policy associates who have experience with legislative engagement, science communications, and the process of using research in public policy. Importantly, no lobbying occurs as part of this process, and researchers are trained in the rules and regulations pertaining to lobbying.

To experimentally evaluate this intervention, a dual-population randomized controlled trial was undertaken with congressional offices and researchers (Fig. 2). Congressional offices received either the RPC or a light-touch traditional support condition (i.e., the control condition offered support by providing publicly available, research-based resources). Researchers were randomized to receive the RPC intervention or a traditional static policy engagement training curriculum. Child and family policies were the focus of participants’ engagement; therefore, legislation reviewed in this study pertained to child and family policies.

Fig. 2.

Consort diagrams for the RPC Evaluation. (A) Ninety-six congressional offices agreed to participate in the study and were randomized to receive either the RPC or control group condition. (B) Two hundred twenty-six researchers agreed to participate in the study and were randomized to receive either the RPC or control group condition.

Results

Comparison of experimental data for intervention and control groups after implementation indicate intervention offices’ reported value of conceptual URE (i.e., URE to indirectly inform how issues are understood) was 7% greater than controls (F = 7.03, P = 0.04, root-mean-square error [RMSE] = 0.60), but there was no difference in their value of instrumental (i.e., URE to directly inform policy development; F = 0.74, P = 0.77, RMSE = 0.52) or tactical URE (i.e., URE to justify or oppose existing policy proposals; F = 2.88, P = 0.36, RMSE = 0.72; Fig. 3A). Congressional offices in the trial introduced 2,029 pieces of legislation in the six months after implementation began. This included 405 child and family bills. Of these bills, 108 were found to include language related to the use of research evidence. Of the whole sample, 92.39% of offices introduced at least one child and family bill and 65.22% introduced at least one child and family bill that included URE legislative language. Analyses indicate that 23.30% more of the intervention group offices wrote and introduced bills containing URE language than control offices. Similarly, 20.76% fewer intervention offices introduced bills lacking URE language. Trial offices did not differ in the likelihood of introducing child and family bills overall during this time period (regardless of URE language; Fig. 3B). No differences were found in introduction of child and family bills or inclusion of research evidence in legislative language in the 6 mo preceding the trial.

Fig. 3.

Effects of the RPC Intervention Model. (A) Congressional offices participating in the RPC reported greater value of conceptual research evidence than controls, but there were no significant differences in instrumental or tactical uses. (B) Offices participating in the RPC were 1) not more likely to write more child and family bills, but were 2) more likely to include research evidence terms in legislative language, and (3) were less likely to introduce bills that did not include URE research evidence terms. (C) Researchers in the RPC reported improved (1) knowledge of current lobbying restrictions and (2) belief that engaging with policymakers would improve their own research. Their level of policy engagement was sustained, whereas control group researchers declined in policy engagement.

Since the beginning of the trial, control group researchers’ reported knowledge of lobbying regulations, reported value of policy engagement for improving their own research, and their policy engagement all fell significantly. Those in the intervention group increased in their knowledge of and reported value for engaging, as well as had no significant reduction in the level of engagement during the same time period (Fig. 3C).

Discussion

This work suggests that collaboration between policy and research communities can change policymakers’ value of science and result in legislation that appears to be more inclusive of research evidence. Importantly, the intervention did not seek to increase the number of new bills introduced by Congress. Instead, the intervention effort aimed to increase the use of scientific evidence while writing new bills.

Further, this work highlights the ability of the RPC to increase policymakers’ perceived usefulness and reported value of scientific evidence. Importantly, these findings demonstrate utility in heightening the perceived value of conceptual rather than tactical use of research evidence. Specifically, congressional offices were more likely to indicate that research use is valuable for “understanding how to think about issues” and that scientific evidence “should be used as a basis for making policy decisions.” These fundamental improvements in our policymakers’ value for science have the potential to increase immediate use of research—as indicated by legislative language findings—and possibly motivating policymakers’ support for science in the future.

Interestingly, this work did not increase the value of tactical use—a common form of evidence use that makes some researchers hesitant to engage with the policy community (19). Not only does the RPC strive to serve as a nonpartisan broker of scientific knowledge, but approaching the knowledge exchange process with problem-solving and interactive models of engagement may mitigate political influences in the research translation process (19). Similarly, these initial findings do not indicate improvements in policymakers’ value of instrumental use. This may be due to the model’s efforts to build trusting relationships between research and policy stakeholders that, in turn, allow for more open and exploratory discourse and less focus on answering specific policy questions characterized by instrumental use. Some scholars suggest that instrumental use may be impractical because it does not reflect a “real picture” of research use, whereas conceptual use involves a gradual perception shift that enlightens policymakers over time and may be more often seen in practice unless direct use of evidence is imposed or required by statute (20). Additionally, this area of scholarship is continuing to develop methodologies for assessing URE that distinguish between uses that involve persuasion (21), which was the primary basis of measuring tactical use in the surveyed construct. Nevertheless, these findings indicate room to improve the RPC, with future versions actively decreasing value of tactical URE and increasing value for both instrumental and conceptual use.

The findings of this study also demonstrated effects on researchers’ policy engagement. Specifically, results indicate not only a change in researcher knowledge and motivation to engage, but also their policy engagement behavior itself. These findings are consistent with prior qualitative work in which policy-engaged scientists reported that ongoing contact with policymakers dispelled their negative stereotypes and increased the scientists’ views of policymakers as caring and committed (22). Together, the observed changes in both policymakers and researchers randomized to the RPC intervention provide support for the underlying theories around the social nature of research translation and evidence use (3–5, 17).

Notably, this trial occurred during the 116th Congress, at a time when many researchers felt science was under attack (23). In particular, researchers at baseline exhibited a high level of concern about the manner in which science was being used by federal policymakers (SI Appendix, Table S1). Such concerns could interfere with researchers’ ability to engage in ways they feel are productive uses of their time. We see empirical evidence of this in the control group’s significant reduction in policy engagement and motivation to engage. Yet, as evidenced by this experiment, even in a political climate that has at times disregarded scientific evidence, there are policymakers across party lines who will use research in their decision making.

Limited work has demonstrated the potential for formal outreach strategies to change congressional behavior. It is our hope these findings—that research use can be strengthened even in a divided Congress—will help illustrate the value of engagement to the scientific community. This study demonstrates the potential to experimentally test the RPC approach, which should also encourage efforts to test and optimize other strategies with the same rigor as we apply to other behavioral interventions. Formalized engagement strategies, such as those considered here, can certainly be improved further. For example, our intent is that the RPC approach can be replicated in other disciplines, organizations, and governments, but we recognize that without prior work and relationship building, such strategies may be out of reach for many. Despite this, we were able to successfully replicate the approach at the state level with collaborators in Texas, who implemented it on a smaller scale (i.e., ∼20 state legislators). Ultimately, if the scientific community wishes policymakers to adopt evidence-based interventions, then we should “walk the walk” and use evidence-based practices to support lawmakers in the creation of evidence-based policies.

Materials and Methods

This trial included 96 congressional offices and 226 researchers each randomized to the intervention and control conditions, respectively. Legislators from the congressional offices were representative of the 116th Congress being on average 63.28 y old (All Representatives: 57.6 y old; All Senators: 62.9 y old), 75.00% male (All of Congress = 75.8% male), and 79.35% White (All of Congress = 78% White). More than half identified as a Democrat (58.8% total; Representatives: 54% Democrat; Senators: 45% Democrat), 27.17% of the members had at least some graduate training, and 15.56% of the offices had at least one person with a doctorate (24). Offices participating in the trial were representative of congressional offices as a whole. Participants were emailed by RPC staff to ask for a meeting to discuss how we can support their work on child and family policy. During the meeting, staffers received a needs assessment. While our unit of analysis was at the office level, survey respondents were the staff who worked in the office and who worked on issues related to children and family. The majority of the staff we surveyed were legislative assistants (n = 36), legislative aides (n = 10), legislative directors (n = 10), senior legislative assistants (n = 7), and legislative correspondents (n = 5). Eight served more than one role (e.g., deputy chief of staff and legislative assistant/counsel/director). Seven served advisor or director roles, such as senior policy advisor, health policy advisor, and health policy director. Two were legislative counsel and one was senior counsel. Two were chiefs of staff and one was deputy chief of staff. Finally, three served in “professional” roles (i.e., professional policy staff and professional staff member).

Researchers were on average 43.01 y old, 14.43% male, and 84.34% White. At baseline, researchers on average were highly concerned about the federal use of research in the current political climate. No significant differences were found between groups for either the congressional or research samples at baseline (SI Appendix, Tables S1 and S2).

Public record of bills introduced to Congress formed the basis of data collected to assess URE at baseline and six months after implementation began (April 2019). Survey data from congressional staff and researchers were collected at baseline (beginning in January 2019) and subsequent to facilitated interactions with researchers (on average, 6.42 mo after baseline). Legislation written by all trial officials were coded (n = 4,372 bills and resolutions) for a child and family focus. All child and family bills were then coded for indicators of use of research evidence (e.g., direct reference to scientific evidence or research, summary or calls for research studies, discussion of experimental/quasiexperimental studies, results of statistical analyses, reference to peer reviews literature and empirical findings; SI Appendix, Table S3).

Congressional offices were surveyed to assess policy behavior and the office’s reported value of using scientific research in policymaking (14, 15, 25, 26). This included eight items that assessed offices’ value for conceptual, instrumental, and tactical use of research (measured on a 5-point Likert scale from strongly disagree to strongly agree). The researcher survey assessed their engagement with policymakers (measured on a 4-point scale, with 1 = none, 2 = 1–3 times, 3 = 4–6 times, and 4 = 7 or more times), knowledge about policy engagement (measured on 5-point Likert scale from strongly disagree to strongly agree), beliefs about policymakers’ support of research (measured on 5-point Likert scale from strongly disagree to strongly agree), and how policy engagement informs their own research (measured on 5-point Likert scale from strongly disagree to strongly agree; SI Appendix, Table S3). This work was reviewed and approved by Pennsylvania State University’s Institutional Review Board (IRB Protocol 00010061). Survey participants were informed of the purpose of this study before agreeing to participate.

Generalized linear models were used to model the trial groups on how much congressional offices value URE for 1) conceptual, 2) instrumental, and 3) tactical purposes. Logistic models were used to model the probability of an office writing 1) a child and family bill, 2) a bill that included research evidence language, and 3) a bill that did not include URE research evidence terms. Logistic models were also used to model researchers’ change from baseline in 1) knowledge of current lobbying restrictions, 2) belief that engaging with policymakers would improve their own research, and 3) level of policy engagement.

Data Availability.

Survey items may be found in the SI Appendix. Data analyzed for this study are publicly available in Github (https://github.com/Research2Policy/PNAS-Data.git). Readers may contact the corresponding author for additional requested materials.