A new measure to understand the role of science in US Congress: lessons learned from the Legislative Use of Research Survey (LURS)

Abstract

Background:

There is growing interest in and recognition of the need to use scientific evidence to inform policymaking. However, many of the existing studies on the use of research evidence (URE) have been largely qualitative, and the majority of existing quantitative measures are underdeveloped or were tested in regional or context-dependent settings. We are unaware of any quantitative measures of URE with national policymakers in the US.

Aims and objectives:

Explore how to measure URE quantitatively by validating a measure of congressional staff’s attitudes and behaviors regarding URE, the Legislative Use of Research Survey (LURS), and by discussing the lessons learned through administering the survey.

Methods:

A 68-item survey was administered to 80 congressional staff to measure their reported research use, value of research, interactions with researchers, general information sources, and research information sources. Confirmatory factor analyses were conducted on each of these five scales. We then trimmed the number of items, based on a combination of poor factor loadings and theoretical rationale, and ran the analyses on the trimmed subscales.

Findings:

We substantially improved our model fits for each scale over the original models and all items had acceptable factor loadings with our trimmed 35-item survey. We also describe the unique set of challenges and lessons learned from surveying congressional staff.

Discussion and conclusions:

This work contributes to the transdisciplinary field of URE by offering a tool for studying the mechanisms that can bridge research and policy and shedding light into best practices for measuring URE with national policymakers in the US.

Background

There is growing interest and recognition of the need to use scientific evidence to inform policymaking (Haskins and Margolis, 2014; Commission on Evidence-Based Policymaking, 2017). However, the actual translation of research into policy has been a slow process (Innvær et al, 2002; Orton et al, 2011; Oliver et al, 2014). In order to improve the use of research evidence in policymaking at the national level, the field must first be able to measure how research is used and empirically study what facilitates its use.

The ability to measure the use of research evidence (URE) relies on being able to define and conceptualise it (Neal and Neal, 2018). To date, three broad types of URE have been defined: instrumental, conceptual, and symbolic (Weiss, 1999; Nutley et al, 2007; Tseng, 2012). Instrumental use, or direct use, is the use of research to solve problems or shape policy decisions (e.g., designing new legislation, changing existing laws). Conceptual use, or indirect use, informs thoughts on an issue and shapes ideas that can shift ‘intellectual discourses, policy paradigms, and social currents of thoughts’ (Nutley et al, 2007: 301). Symbolic use, or tactical use, is the use of research to justify a position or decisions or to persuade others.

Much of the existing work on URE relies on ethnographic and qualitative approaches (e.g., Stevens, 2011; Lawrence et al, 2017), and many of the existing quantitative measures are underdeveloped and have not been rigorously tested (Frasure, 2008; Squires et al, 2011; Lawlor et al, 2019). Further, the majority of quantitative approaches to studying URE have been tested outside the context of lawmaking, such as within the settings of education (Lawlor et al, 2019), health (Frasure, 2008; Squires et al, 2011), and mental health/social services (Palinkas et al, 2016), or have only been tested in a regional context. For example, Bogenschneider and colleagues (2019) interviewed state legislators across two US states, Wisconsin and Indiana, to examine how state policymakers’ URE compares to predictions made by four theories of URE that all rely on the three types of URE defined above. To our knowledge, there have not yet been any quantitative measures of URE that have been tested with national legislators or their staff in the US.

In fact, surveys of national-level policymakers are very rare. In one of the few extant studies recently conducted in the UK, 125 Parliament Members and their staff were surveyed about their URE (Kenny et al, 2017) and the researchers acknowledged their ‘rare privileged access to Parliament’. Due to the perceived inaccessibility of policymakers at a national level, the majority of surveys with policymakers have been conducted at the regional or state level. As examples, these studies have discussed lessons learned from conducting a survey about obesity policies (Nichols et al, 2017), compared the efficiency of mail and internet surveys (Fisher and Herrick, 2013), and assessed personality traits (Dietrich et al, 2012). In the US, outside of the Congressional Management Foundation, which routinely conducts surveys with national policymakers (e.g., Jensen, 2011), there have been few scientific studies of this population.

National policymakers remain an underrepresented research population for several reasons. First, several offices have policies against participating in surveys (Hertel-Fernandez et al, 2019), thereby limiting the types of research that can be conducted with this population. Second, time is sparse and legislative demands are high, meaning that national policymakers have little time to engage in research (Hall, 1996), particularly if the topic is not directly related to policy issues on the agenda (Whiteman, 1997). Third, the number of communications that policymakers regularly receive has drastically increased. Technological advances have facilitated quick and frequent communications from colleagues, constituents, lobbyists, advocacy and grassroots organisations, and researchers (Hysom, 2008). This makes it difficult for policymakers to respond to everyone in a timely manner, including those which involve requests (e.g., participating in research). Lastly, there is frequently high turnover of staff in US federal policymaker offices, which makes it difficult to maintain points of contact and a sense of rapport for recruiting research participants. Together, all of these factors make it difficult to conduct research with national policymakers.

Accordingly, we are unaware of any studies that have quantitatively measured URE within surveys of national legislators or their staff in the US. Considering the unique context and characteristics of this population and their profession, a URE measure validated for this population is critical to advancing URE in policymaking. As such, we developed the Legislative Use of Research Survey (LURS) to assess national legislators’ attitudes and behaviour towards research use by drawing upon existing URE measurement tools. Our measure included five main constructs: Reported use of research evidence ; Value of research evidence for policy work; Interactions with researchers; General information sources; and Research information sources.

Reported use of research evidence assesses the use of research across various stages of the policy process (e.g., agenda setting, policy evaluation) and the three types of research use (i.e., instrumental, conceptual, and symbolic) (Bogenschneider et al, 2013;Makkar et al, 2016; Brennan et al, 2017). Value of research evidence for policy work measures the perceived value of research at an individual (e.g., value for understanding issues) and office level (e.g., leaders’ value engagement with researchers) (Brennan et al, 2017). Interactions with researchers measures policymakers’ involvement with researchers and the frequency of different types of interactions with researchers (Campbell et al, 2009). General information sources assesses how often information is obtained from general sources (e.g., internet searches) (Palinkas et al, 2016). Finally, Research information sources measures the different research sources policymakers use to obtain research evidence (e.g., experts from universities) (Palinkas et al, 2016).

The purpose of this paper is methodological in nature, which aims to explore how to measure URE with national-level policymakers quantitatively by: (1) validating the structural validity of this newly developed measure; and (2) discussing the lessons we learned from collecting data among this unique population. This work offers the field a tool for studying the mechanisms that can bridge research and policy and sheds light on best practices for measuring URE with national policymakers, an underrepresented research population.

Methods

Sample

Participants included staff in congressional offices working on issue areas broadly related to children and families (e.g., child welfare, human trafficking, prevention, substance use, and poverty) who agreed to meet with us. Staff were recruited instead of the Congress member because Congress members are typically less available, but rely on their staff for information throughout the legislative process (Hall, 1996; Hertel-Fernandez et al, 2019). Thus, congressional staff can be used as a proxy for assessing Congress members.

An online Client Relationship Manager that was developed specifically for congressional outreach, Quorum (Quorum, 2017), was used to identify the legislative offices and create an activity score based on how active each office is in the five issue areas identified above (range = 3–30). These scores were then used to rank offices based on 10 percentiles, such that the higher-ranked offices were contacted first (N = 246). Of the 246 legislative offices that we contacted, 50.8% (n = 125) responded to our cold email. Of those who responded to our email, 64% completed the survey and 36% declined. This response rate is greater than what has been observed in other studies (Hertel-Fernandez et al, 2019). The present analyses include data from these 80 congressional staff members (see Table 1). Our sample of legislative staff was 55.0% women and 80.0% White, with the majority having at least a Bachelor’s degree (58.8%) and having the title of legislative aide, assistant, or correspondent at the time of the meeting (65.0%). These staff represent the legislators whom they work for, who are primarily men (76.3%) and White (80.3%), which is representative of Congress (75.8% men; 80.3% White).

Table 1.

Sample Characteristics

	N (%)	Mean Activity Score (SD)
Democrat	38 (47.5)	7.08 (1.75)
Republican	40 (50.0)	7.10 (2.17)
Independent	1 (1.25)	-
Unknown	1 (1.25)	-
Total	80	7.11 (1.96)

Note. The mean activity score was not significantly different between Democrats and Republicans, t(76) = −0.05, p = 0.96.

Procedures

Starting with offices that had the highest activity scores, we first identified the staffer who handled child and family issues either by calling the office and asking the front desk clerk or by relying on Quorum, which is updated weekly and summarises publicly available information about legislators and their staff. We then emailed the identified staffer and requested a meeting to discuss how researchers could support their current policy efforts related to children and families. If and when the staffer agreed to meet (challenges of which will be discussed in the following sections), the meetings typically took about 30 minutes, during which we assessed their current policy priorities and invited them to complete the survey. This work was reviewed and approved by Pennsylvania State University’s Institutional Review Board.

Measure development

We developed a 68-item self-report measure, the LURS, that assesses congressional staff’s attitudes and behaviours regarding the use of research. First, we generated items that were adapted and expanded upon from similar measures. Items were adapted based on the need to change the context (e.g., from an educational setting to a policymaking setting) as well as the need to simplify complex and potentially confusing items to make them more accessible for this audience (see Supplemental Table 1 for a full list of the items and the specific adaptations made: https://doi.org/10.6084/m9.figshare.13604570.v2). All previous measures were validated for other populations and, to our knowledge, none were validated with national legislators or their staff. In addition to two yes/no questions to determine the research training of staff in the office, the initial measure consisted of five scales, outlined below.

For the purposes of this study, research was specified as involving systematic scientific methods to answer scientific questions and was defined as information based on scientific studies. Scientific research included the review of original research studies, including those by the Congressional Research Service, but did not include statements authored by advocacy organisations or lobbyists. We defined ‘researchers’ broadly, including academics, programme evaluators, and those who support the use of research in practice and communities.

Reported use of research evidence

The Reported use of research evidence scale consisted of 14 items on a 5-point Likert scale, ranging from 1 (not at all) to 5 (all the time), that were adapted from the Staff Assessment of enGagement with Evidence interview protocol (SAGE) (Makkar et al, 2016) and a three-item scale regarding the use of research for policy issues (a = .74) (Bogenschneider et al, 2013). An example item from this scale is: ‘In the past six months, when doing the following activities, how often have you used research to justify a decision you made?’

Value of research evidence for policy work

The Values scale was comprised of 17 items on a 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree) that were adapted from the Seeking, Engaging with, and Evaluating Research protocol (SEER; a = .80 - .85) (Brennan et al, 2017). An example item from this scale is: ‘Please rate how strongly you agree with the following: I find it valuable to use research in my work to persuade others to a point of view or course of action’.

Interactions with researchers

The Interactions with researchers scale included 11 items on a 5-point Likert scale ranging from 1 (not at all) to 5 (all the time) that were adapted from structured interviews assessing policymakers’ frequency of interactions with researchers (Campbell et al, 2009). An example of an item from this scale is: ‘In the past six months, when developing policies or bills, how often have you interacted with researchers by collaborating with researchers to develop, implement, or interpret research findings?’

General information sources

The General information sources scale consisted of 15 items on a 5-point Likert scale ranging from 1 (not at all) to 5 (all the time) that were adapted from the Standard Interview for Evidence Use (SIEU) protocol (a = .80) (Palinkas et al, 2016). An example item from this scale is: ‘Please indicate how often you rely on each of the following sources to obtain information: I hear about it from constituents’.

Research information sources

Lastly, the Research information sources scale was comprised of 11 items on a 5-point Likert scale ranging from 1 (not at all) to 5 (all the time) that were also adapted from the SIEU (a = .80) (Palinkas et al, 2016). An example item from this scale is: ‘Please indicate how often you rely on each of the following sources to obtain information: I contact an expert from a local college or university’.

The initial measure was administered to participants at baseline (N = 80). Items that staff had difficulty answering (e.g., confusing wording, not relevant) and items that did not load onto their respective constructs were removed. The final measure resulted in 35 items across the five scales, which were carried forward for use in subsequent follow-up assessments.

Data analysis

Confirmatory factor analyses (CFAs) were employed to evaluate the validity of established scales that were adapted and expanded for use with congressional staff, CFA provides a useful approach to test the application of existing theoretical constructs to new samples (Lance and Vandenberg, 2002). This statistical method requires prior research and theoretical support suggesting that factors load onto a latent construct (Schumacker and Lomax, 2010). Thus, we employed CFA to determine whether the measure we developed and adapted from existing measures was appropriate for assessing congressional staff’s attitudes and behaviours regarding the use of research.

Analyses were conducted using the ‘lavaan’ package (Rosseel, 2012) in R (R Core Team, 2019). We used maximum likelihood estimation with Full Information Maximum Likelihood (FIML) for missing data. All latent factors were standardised. A parallel series of analyses was run for each scale (Reported use of research evidence, Values, Interactions with researchers, General information sources, and Research information sources). The Comparative Fit Index (CFI), Tucker Lewis Index (TLI), and Standardized Root Mean Squared Residual (SRMR) were examined to assess the overall model fit to the data, with CFI >= 0.90, TFI >= 0.95, and SRMR < 0.08 set as criteria for good model fit (Hu and Bentler, 1999). For each scale, we first conducted a CFA on the initial scale and examined model fit. Next, we dropped select items based on poor factor loadings (<0.32) (Worthington and Whittaker, 2006) and theoretical decisions (e.g., staff’s reactions to some questions, ceiling effects). Lastly, we conducted a CFA on the trimmed scale and re-examined model fit.

Results

Policymakers’ URE

Table 2 displays the descriptive statistics for all items included in the trimmed models. The majority of participants reported using research often or all the time to develop policy (80.0%), understand how to think about an issue (78.8%), and consider policy implementation (71.3%). There were very high levels of agreement on the value of research evidence for policy work. Nearly all participants agreed or strongly agreed that research can benefit policy (98.8%) and is useful (93.8%). The majority agreed that research evidence should be used to decide about the content or direction of a policy (93.8%), write legislation (88.8%), and used as a basis for decisions (87.5%).

Table 2.

Descriptive Statistics and Factor Loadings for Trimmed Models

Variable	Mean	SD	Beta	SE
Reported Use of Evidence
Decide about content or direction of a policy	4.04	0.95	0.85**	0.09
Develop a policy	4.26	0.88	0.70**	0.09
Meet an expectation or requirement for using research	3.40	1.38	0.69**	0.14
Persuade others to a point of view or course of action	4.14	0.82	0.57**	0.09
Consider policy implementation	4.11	0.93	0.68**	0.10
Set a policy agenda	3.82	1.05	0.83**	0.10
Help you understand how to think about an issue	4.16	0.87	0.70**	0.09
Value of Research Evidence for Policy Work
Should be used as a basis for making policy decisions	4.45	0.68	0.65**	0.07
Can benefit policy decisions	4.62	0.49	0.80**	0.05
Is useful for policymaking	4.58	0.59	0.69**	0.06
Decide about content or direction of a policy or program	4.29	0.56	0.57**	0.06
Identify issues that require a policy or program response	4.24	0.87	0.57**	0.10
Persuade others to a point of view or course of action	4.37	0.62	0.54**	0.07
Understand how to think about issues	4.25	0.70	0.73**	0.07
Write legislation	4.32	0.87	0.60**	0.10
Interactions with Researchers
Inviting researchers to be an active contributor to policy development	3.09	1.16	0.80**	0.11
Attending forums (e.g. conferences, briefings, webinars) to hear about research findings	3.39	1.06	0.53**	0.11
Collaborating with researchers to develop, implement, or interpret research findings	3.10	1.16	0.75**	0.11
Offering advice to researchers	2.05	1.01	0.48**	0.11
Working with researchers to identify policy direction or priorities	3.30	1.08	0.88**	0.10
Working with researchers to identify research direction or priorities	2.77	1.20	0.75**	0.12
Inviting researchers to give a research perspective in an area of policy development	2.99	1.03	0.69**	0.10
Using research contacts as a sounding board for policy work	3.61	1.08	0.80**	0.10
General Information Sources
I hear about it from constituents.	3.54	0.95	0.56**	0.12
I obtain it from meeting with professionals in the district or state.	3.86	0.86	0.57**	0.12
I rely on a consultant (e.g., a lobbyist or non-partisan adviser) to obtain it for me.	3.14	1.09	0.44**	0.14
I rely on nonprofit organizations/foundations.	3.76	0.90	0.72**	0.12
I rely on intermediary organizations like Child Trends.	2.75	1.29	0.32+	0.17
Research Information Sources
I contact an expert from a local college or university.	3.02	1.19	0.54**	0.15
I contact someone that has already implemented this program or intervention.	3.32	0.94	0.61**	0.12
I contact someone who presented at a briefing or other research forum.	3.34	0.86	0.57**	0.11
I obtain it from conferences or training workshops.	2.64	1.04	0.45*	0.14
I rely on particular web-based clearinghouses (e.g., What Works Clearinghouse, California Evidence-Based Clearinghouse for Child Welfare).	1.82	1.06	0.32+	0.14
I rely on data that were collected by my office or someone we hired to collect data.	2.36	1.36	0.36*	0.18
I search academic journals.	3.38	1.08	0.51**	0.14

Note.

^**indicates p < .001

^*indicates p < .01

⁺indicates p < .05.

There was greater variability in endorsement of frequency of interactions with researchers, use of general information sources, and use of research information sources. Slightly over half of participants (57.5%) reported using researchers as a sounding board for policy and half (50.0%) reported attending forums often or all the time. In contrast, only 38.8% collaborate with researchers and 28.8% invite researchers to give perspectives about policy often or all the time. Participants reported gathering information from meeting with professionals (68.8%), relying on nonprofit organisations/foundations (61.3%), searching academic journals often or all the time (52.5%), hearing from constituents often or all the time (51.3%), contacting presenters (41.3%), and relying on consultants (37.5%).

Scale validity

We conducted CFAs on each of the five scales separately (initial and trimmed): Reported use of research evidence, Value of research evidence for policy work, Interactions with researchers, General information sources, and Research information sources. Table 2 contains the factor loadings from each of the trimmed scales. Results from each of the initial scales are presented in Supplemental Table 2 (https://doi.org/10.6084/m9.figshare.13604570.v2).

Reported use of research evidence

The initial model for the Reported use of research evidence scale (14 items) demonstrated adequate fit, χ²(77) = 152.44, SRMR = 0.07, TLI = 0.85, CFI = 0.88. Seven items were cut, based on poor factor loadings and lack of performance or theoretical support for URE at the four stages of the policy process and three types of research use. This resulted in a seven-item model, yielding improved model fit, χ²(14) = 40.04, SRMR = 0.07, TLI = 0.85, CFI = 0.90. All retained items demonstrated good factor loadings and the trimmed scale demonstrated strong reliability (α = .89).

Value of research evidence for policy work

The initial model for the Values scale (17 items) demonstrated poor fit, χ²(119) = 305.97, SRMR = 0.11, TLI = 0.60, CFI = 0.65. Nine items were cut for performance or theoretical reasons and for demonstrating poor factor loadings, resulting in an eight-item model. The trimmed model demonstrated improved model fit, χ²(20) = 51.83, SRMR = 0.07, TLI = 0.80, CFI = 0.86. Further, all retained items demonstrated good factor loadings and the trimmed scale had good reliability (α = .85).

Interactions with researchers

The initial model for the Interactions with researchers scale (11 items) demonstrated acceptable fit, χ²(44) = 88.85, SRMR = 0.07, TLI = 0.86, CFI = 0.89. Three items were cut, resulting in an eight-item measure. The trimmed model demonstrated good fit, χ²(20) = 41.20, SRMR = 0.06, TLI = 0.91, CFI = 0.93. All items demonstrated good factor loadings and the trimmed scale demonstrated strong reliability (α = .89).

General information sources

The initial model for the General information sources scale demonstrated adequate fit, χ²(90) = 150.02, SRMR = 0.10, TLI = 0.66, CFI = 0.70. Ten items were cut, resulting in a five-item trimmed scale. The trimmed model demonstrated improved fit, χ²(5) = 6.46, SRMR = 0.05, TLI = 0.93, CFI = 0.97. All items showed good factor loadings and the trimmed scale demonstrated adequate reliability (α = .64).

Research information sources

The initial model for the Research information sources scale demonstrated good fit, χ²(44) = 51.09, SRMR = 0.07, TLI = 0.93, CFI = 0.94. Four items were cut, resulting in a seven-item trimmed scale. The trimmed model demonstrated excellent fit, χ²(14) = 14.13, SRMR = 0.06, TLI > 0.99, CFI > 0.99. All items had good factor loadings and the trimmed scale demonstrated adequate reliability (α = .68).

We note that we examined two sets of models for the information sources scales: one in which we treated general information sources and research information sources as two subscales of a larger information sources scale, and one in which we treated them as separate scales. The combined scale with two subscales model demonstrated poor fit (i.e., SRMR = 0.10, TLI = 0.60, CFI = 0.64 for the full model; SRMR = 0.09, TLI = 0.65, CFI = 0.72 for the trimmed model) relative to the two separate scales model, which guided our decision to treat general and research information sources as separate scales.

Discussion

The overarching goal of the present study was to explore how to quantitatively measure URE with national-level policymakers. To do this, we first developed and validated a new measure for use with congressional staff to assess national policymakers’ attitudes and behaviour towards research use. Our measure was comprised of five scales: Reported use of research evidence; Value of research evidence for policy work; Interactions with researchers; General information sources; and Research information sources. We substantially improved our model fits for each scale over the original models and all our items had acceptable factor loadings (> 0.32). We note that our trimmed Values scale still did not demonstrate as good a fit as traditionally recommended for an acceptable model fit (Hu and Bentler, 1999); however, this is a novel measure that will continue to be refined to achieve structural validity. These findings indicate that it is possible to apply the same methodological rigour that is applied to research with other populations as it is with national-level policymakers.

Next, to continue discussing quantitative methodologies for measuring URE among national-level policymaking officials, we want to share our experiences of collecting survey data. We were able to achieve a 50.8% response rate to our cold emails, which is consistent with or higher than expected when emailing legislative offices (Hertel-Fernandez et al, 2019). Those engaging in legislative outreach should anticipate a laborious and persistent email procedure to obtain such a response rate. Of those who responded to the emails, 64% agreed to complete the survey. This provides information on the likelihood of agreement among responders. Overall, our response rates demonstrate an improvement over what has been reported in previous studies, which typically range from 9% to 15% (Hertel-Fernandez et al, 2019). Despite achieving a heightened response rate, we faced various challenges at different stages of the process of administering surveys to congressional staff, similar to those identified in previous studies (Whiteman, 1997; Hall, 1996; Hertel-Fernandez et al, 2019).

Previous research (Hertel-Fernandez et al, 2019) suggests that long-term relationships are critical to connecting with this understudied population. Efforts to foster long-term relationships are time-intensive and resource-consuming, requiring persistence to receive responses from policymaking office personnel. This is not an easy feat. Thus, we believe that our experiences throughout this process would be of interest and benefit to others hoping to work with this population. We present the lessons we learned through survey administration below (summarised in Table 3).

Table 3.

Lessons Learned from Surveying Congressional Staff

Challenge	Lessons Learned
Making initial contact	1) Be brief and concise: Keep emails short but as informative as possible; 3-4 sentences is ideal. Make replies as brief and easy to answer as possible (e.g., “yes or no” questions, offer specific meeting times).
	2) Tailor subject lines: Include “meeting request.” Personalize it with the legislator’s name and focus issue.
	3) Be strategic in timing: Send emails either in the morning or at the end of the workday on Mondays or Fridays because this is the time when staff likely have the time to check emails.
Scheduling meetings	1) Be persistent and flexible: Follow up as needed (within reason, without damaging rapport). Employ multiple methods of contact (e.g., emailing, dropping by the office, calling).
	2) Be aware of current events and the congressional schedule, which can affect staffers’ availability and focus: Use this knowledge tactfully to showcase understanding and support connections. As an example, offer to meet during a congressional recess or otherwise slow period.
	3) Provide value: Offer a direct benefit for meeting (e.g., research support, information, guidance).
Establishing rapport	1) Employ excellent interpersonal skills: Be respectful. Engage in active listening and demonstrate a genuine and authentic interest in their work. Have empathy and use perspective-taking skills. Thank staff for their time.
	2) Adapt and prepare for the unexpected: Make the best out of whatever environment the meeting takes place in (e.g., entryway, waiting room, hallway, cafeteria).
	3) Emphasize non-partisanship: Explain that you are there to learn and that you do not have a political agenda.
	4) Acknowledge the nature of staff’s work: Demonstrate that you recognize that their work is important and fast-paced. Acknowledge their policy expertise. This will illustrate mutual respect and demonstrate humility.
	5) Work to develop a trusting relationship: This is of utmost importance, as relationships require trust, and will encourage continuity and ongoing engagement.
Administering surveys	1) Establish rapport during the first part of the meeting: Use the above recommendations. Thoroughly explain your work and purpose for meeting. If possible, offer to support the legislator’s work by, for example, providing resources or information.
	2) Highlight a shared goal: Frame the study as an evaluation to improve external policy efforts, not an audit of the policymaker or the office.
	3) Explain the nature of the survey and its purpose: Emphasize that the responses are confidential and will be used for non-political research purposes only.
	4) Use an empowered collaboration approach: Offer to allow staff to review the questions before beginning the survey. Emphasize choice in refusing to answer any question.

Lessons learned through survey administration

Making initial contact

The first step to administering surveys to staff is to initiate contact. As noted, the number and frequency of communications that policymakers receive has drastically increased in recent decades (Hysom, 2008). Thus, it follows that the main barrier to achieving initial contact is a low response rate to emails due to crowded inboxes, busy staff schedules, or confusion over what we were asking. Being brief and concise in the email helped us to overcome this barrier. Although clarity of the request and providing enough information is essential, it needs to be balanced with writing a very succinct email. It has been our experience that an ideal length of an email is only three to four sentences. Tailoring subject lines by including the words ‘meeting request’ seemed to grab staff’s attention and also make it immediately clear that we were requesting a meeting. Personalising subject lines with the legislator’s name and focus issue also seemed to elicit more responses, as doing this may reduce the likelihood that staff think it is spam. Making strategic use of when the emails are sent is also helpful. We received the most responses from staff when we sent emails either in the morning or at the end of the workday on Mondays and Fridays. This is likely when staff have time to check emails. Future research could empirically test what specific communication approaches are most effective by testing different subject lines and timing of emails (Scott et al, 2020).

Scheduling meetings

Responses, however, did not always result in getting the meeting scheduled, making the second primary challenge scheduling the meeting. The amount of time that policymakers have, relative to their increasing legislative and non-legislative demands, (Hall, 1996) makes scheduling meetings very difficult. The most useful strategy to overcome this challenge was simply persistence and employing multiple methods of contact – sending as many follow-up emails as needed, calling, and dropping by their offices. Persistence, however, must be balanced with rapport. We did not want to risk damaging any future potential relationship with the office.

Having familiarity with the policy process and the congressional schedule is also critical (Nichols et al, 2017). This knowledge can be used to tactfully showcase understanding and support connections. For example, offer to meet during a congressional recess or otherwise slow period. Likewise, being aware of current events that may affect staff’s availability and focus is essential. As a candid example, we were trying to schedule a meeting with a staffer who works for the Appropriations Committee during a budget-related government shutdown. The staffer first replied that she could only accommodate a meeting with constituents at this time because of appropriations matters. However, recess was supposed to be shortly after her reply, so we sent a follow-up email asking if things had slowed down for her and if she could meet during recess. She replied: ‘I’m not sure ifyou’ve been following the news but matters for appropriators have not slowed down. Unfortunately, I won’t be able to meet’. We never ended up meeting with her. Accordingly, it pays to be aware of how current events may affect staffer availability.

Finally, offering staff some sort of value for meeting will likely increase their willingness to agree to meet. Examples include research support, information, or guidance. As Whiteman (1997) noted, policymakers are more likely to respond to communications and/or requests when the topic is directly related to policy issues on the agenda. In the present study, we offered research support.

Establishing rapport

Upon getting a meeting scheduled, the next goal is to establish rapport with the staff member during the meeting (Nichols et al, 2017). Employing excellent interpersonal skills is key to achieving this goal. Qualitative researchers and investigative interviewers have shown that rapport building is accomplished by treating the interviewee with respect, engaging in active listening, demonstrating a genuine interest in their work, having empathy, and using perspective-taking skills (Stein and Mankowski, 2004; St-Yves, 2006). As much as possible, we employed these skills when meeting with staff. It is also, of course, important to express gratitude for staff taking the time to meet. Although this may be common sense, stressful situations can sometimes prevent individuals from using such skills and interrupt the rapport building process.

For instance, meetings can take place in a variety of settings. The ideal setting is a private conference room, which facilitates connection and rapport building. However, meetings sometimes took place in the common area of the office or even in the hallway, both of which are noisy and can make connection difficult. Additionally, staff are sometimes late to meetings, or can be pulled out of the meeting early, due to the pressing nature of some policy work. Interviewers noted occasionally interacting with staff who were hurried or exhibiting body language indicative of distraction or dysregulation (e.g., sweating, shaking), depending on their own circumstances. The lesson to be learned here is to be adaptable and prepared for the unexpected – be ready to make the most of whatever situation you are presented with and be willing to schedule another meeting or phone call.

Another useful strategy that facilitates rapport building is to emphasise nonpartisanship. For this work, we explained that we were there to learn from them and did not have a political agenda, which often reduced defensiveness. Finally, acknowledging the nature of staff’s work as important and fast-paced, while also acknowledging their policy expertise, can illustrate mutual respect and demonstrate humility. Above all else, when meeting with staff, working to develop a trusting relationship with them is of the utmost importance. Relationships require trust and developing such a relationship with staff will encourage continuity and ongoing engagement.

Administering surveys

One of the primary purposes of meeting with staff was to administer our survey. The biggest barrier to getting the survey completed that we encountered was that many staff would refuse or be reluctant due to their office’s policy against participating in surveys (Nichols et al, 2017). We employed several strategies to try and overcome this challenge.

First, we tried to establish rapport during the first part of the meeting using the above recommendations and thoroughly explaining our work and purpose for meeting. If possible, we offered to support the legislator’s areas of interests by, for example, providing resources or information. In the present study, we asked staff about their current child and family policy priorities and how we might support those efforts.

Next, we asked staff to complete our survey by highlighting a shared goal and framing the study as an evaluation to improve external policy efforts. We carefully explained that the questions were not meant to be a survey or audit of them, their boss, or their office. We also carefully explained the nature of the survey and its purpose; we emphasised that their responses were confidential, would not be tied to their name or office and would be tied to a number only, and that their responses would be used for analytic/academic, non-political purposes only. Finally, we used an empowered collaboration approach (Pomerantz, 2012; Sieber, 2012). For instance, we offered to let staff see the questions before we asked them and stressed that they could refuse to answer any questions. These actions were taken to empower staff to make an informed decision about their participation. Although this approach was developed for use with vulnerable populations in psychological research and practice, the core tenets of it can (and arguably, should) be applied across research populations and disciplines.

Contributions to the field of URE and future directions

Much of the existing work studying URE has relied on qualitative or ethnographic approaches (Neal and Neal, 2018; Gitomer and Crouse, 2019) and has been conducted in a regional context (e.g., Kenny et al, 2017; Bogenschneider et al, 2019). Existing quantitative measures of URE (e.g., Frasure, 2008; Squires et al, 2011; Palinkas et al, 2016; Lawlor et al, 2019) are underdeveloped and have not yet been tested with national legislators. By drawing upon existing URE measurement tools, we developed and validated a quantitative measure of URE that assesses national policymakers’ attitudes and behaviour towards research use. This work fills an important gap in the URE field by offering a tool for studying the mechanisms that can bridge research and policy.

Nevertheless, there is more that needs to be done to further validate the measure, now that its structural validity has been demonstrated. For example, convergent validity of the scales (i.e., relationship with similar phenomena) and divergent validity (i.e., dissimilarity to constructs that may be unrelated to policymakers’ URE). Testing for these types of validity was beyond the goals of the current study, which were to assess structural validity and share the lessons we learned. Validation is also needed with other types of policymakers, such as state and regional policymakers and with policymakers in other countries and political systems, to examine potential differences across levels and types of government. Further validation will allow for a better understanding of the mechanisms that bridge research and policy.

Future research could also use this measure in evaluations intended to increase the URE in policymaking. This will allow us to examine change sensitivity. For example, efforts to improve the URE in policymaking, such as the UK Knowledge Exchange Unit (UK Parliament, 2020), the Research-to-Policy Collaboration (RPC) (Crowley et al, 2018, Scott et al, 2019) and Family Impact Seminars (Bogenschneider, 1995; Bogenschneider et al, 2000; Bogenschneider et al, 2002), could use this measure to evaluate whether they are effective in changing the attitudes and behaviours of policymakers’ URE. Although Family Impact Seminars have been evaluated on aspects such as success of the seminar, attendance, and ability to meet the seminar’s goals (e.g., Bogenschneider, 1995; Wilcox et al, 2005), more evaluative work needs to be done in the URE space to understand the mechanisms that affect change over time.

Limitations

This work should be considered in the context of several limitations. First, our General information sources and Research information sources scales did not demonstrate acceptable validity when analysed together as two sub-scales within a larger Information sources scale, and fit indices for the Values scale did not demonstrate as good a fit as traditionally recommended for an acceptable model fit (Hu and Bentler, 1999). Further, some items (n = 9) demonstrated a ceiling or floor effect, as evidenced by skewness greater than one or less than negative one. These ceiling effects could have negatively impacted model fit. Of note, most skewed items pertained to values, suggesting that social desirability bias may be important to consider, particularly among a policymaker sample. Nevertheless, this work provides a starting point for developing a reliable and structurally valid measure to study URE with national-level policymakers.

Second, our measure was validated for use with US congressional staff, so it is unclear if the measure is appropriate for use with other policymakers (e.g., US state legislators, Parliament members). Third, our recommendations are based on anecdotal but substantial experience in interviewing congressional staff. Based on this, we believe that our recommendations have merit and are informative for others working in this field.

Conclusions

Improving URE in national policymaking requires the ability to measure it. However, this population is understudied, and often inaccessible, which limits the field’s ability to quantitatively measure URE at the highest level of government. Accordingly, we sought to explore approaches to doing so through validation of a newly developed quantitative survey, the LURS, and by discussing the lessons we learned through its administration with congressional staff. The LURS fills an important gap in the URE field by providing a preliminary new tool for measuring URE. Through this process, we demonstrated that it is possible to quantitatively study the use of research evidence with policymakers.

We also learned and shared many lessons that can inform future work with this unique population. Survey participation is typically quite challenging, but is even more so with congressional staff. We hope that the lessons we shared, from making initial contact to establishing rapport and getting the survey completed, help to guide others who wish to survey congressional staff. Overall, this work contributes to the transdisciplinary field of URE among national policymakers, by offering a quantitative approach to assessing mechanisms for bridging research and policy, and shedding light into best practices for surveying congressional staff.

Key messages.

• Demonstrates structural validity of a quantitative measure of policymakers’ use of research evidence;

• Includes scales that assess mechanisms for bridging research and policy;

• Illustrates the potential for applying rigorous measurement designs with congressional staff;

• Discusses specific lessons that can inform successful measurement in the future.