Measuring Use of Research Evidence: The Structured Interview for Evidence Use

Abstract

Objectives

This article describes the Standard Interview for Evidence Use (SIEU), a measure to assess the level of engagement in acquiring, evaluating, and applying research evidence in health and social service settings.

Method

Three scales measuring input, process, and output of research evidence and eight subscales were identified using principal axis factor analysis and parallel analysis of data collected from 202 state and county child welfare, mental health, and juvenile justice systems leaders.

Results

The SIEU scales and subscales demonstrate strong internal consistency as well as convergent and discriminant validity.

Conclusions

The SIEU is easy to use and can be administered as a complete scale or as three smaller scales to separately examine evidence in acquisition, evaluation, or application. The measure demonstrates potential in understanding the role of research evidence in service settings and in monitoring the process of evidence-based practice and application of scientific principles in social work practice.

As with health services in general (Institute of Medicine, 2000), there remains a large gap between interventions shown to be effective in the prevention and treatment of mental health and behavioral problems among children and adolescents and their use in everyday clinical care. Although there are numerous research-supported treatments (RSTs; often referred to as evidence-based practices [EBPs] or evidence-informed treatments) for this purpose (Burns, 2003; Henggeler, Schoenwald, Borduin, Rowland, & Cunningham, 1998; Webster-Stratton, Reid, & Hammond, 2004; Weisz, Hawley, & Jensen-Doss, 2004), they are not widely used in clinical practice (Bickman, Lambert, Andrade, & Penaloza, 2000; Chorpita et al., 2002), and there is a need to accelerate their use (Beidas & Kendall, 2014), possibly with the use of research-supported implementation strategies (RSISs) like the availability, responsiveness, and continuity organizational intervention (Glisson, Hemmelgarn, Green, & Williams, 2013). In fact, it has been estimated that 90% of publicly funded child-serving systems do not use RSTs (Hoagwood & Olin, 2002). Many in the field agree that a great deal of research will be needed to identify factors that facilitate or impede RST implementation in service sectors that cater to children and adolescents, including specialty mental health, schools, juvenile justice, primary care, and child welfare (Aarons & Palinkas, 2007; Beidas & Kendall, 2014; Fixsen, Noom, Blase, Friedman, & Wallace, 2005; Garland, Kruse, & Aarons, 2003; Glisson, 2002; Hoagwood, Burns, Kiser, Ringeisen, & Schoen-wald, 2001; Schoenwald & Hoagwood, 2001).

Embedded in most models of RST implementation is the transfer of research evidence from the developers to potential users of a particular RST, referred by some as the “science push” (Belkhodja, Amara, Landry, & Ouimet, 2007; Damschroder et al., 2009; Golden-Biddle et al., 2003; Greenhalgh, Rober, Macfarlane, Bate, & Kyriakidou, 2004; Klein & Sorra, 1996; Fixsen et al., 2005; Rynes, Bartunek, & Daft, 2001; Schoenwald et al., 2008; Stamatakis et al., 2012; Tabak, Khoong, Chambers, & Brownson, 2012; Tunis, 2007). Other models focus explicitly on the use of research evidence without specific reference to RST implementation (Honig & Coburn, 2008; Kennedy, 1984; Nutley, Walter, & Davies, 2007). Many of the latter set of models represent typologies of research use. For instance, several researchers have distinguished between an instrumental model in which “use” consists of making a decision and research evidence is assumed to be instructive to the decision and a conceptual model in which use consists of thinking about the evidence. Although the central feature of the instrumental model is the decision, the central feature of the conceptual model is the human information processor. The classic representation of the conceptual model is the Newell and Simon’s (1972) human problem-solving model, based on a computer problem-solving analogy, in which the human senses or receptors obtain information from the environment (input), filter them through a series of processors with memory storage (process), and then initiate a behavior or set of behaviors in response (output). Hence, the instrumental model focuses on the application of research evidence and the outcomes of doing so, while the conceptual model focuses on the process of evidence use (Kennedy, 1984). Nutley, Walter, and Davies (2007), for instance, identify four predictors of evidence use as an outcome, that is, (1) the nature of the research to be applied, (2) the personal characteristics of both researchers and potential research users, (3) the links between research and its users, and (4) the context for the use of research. Honig and Coburn (2008) emphasize process (searching for evidence and incorporating or not incorporating it in decision making) and predictors (features of the evidence, working knowledge, social capital, organization, normative influence, political dynamics, and state and federal policies) of evidence use.

Models of research evidence use reflect distinct theoretical orientations (i.e., human information processing, distributed cognition, diffusion of innovations, decision-making theory) and organizational settings (e.g., health care, education); however, most if not all acknowledge two essential considerations to understanding when research will be used and in what ways (Nutley et al., 2007). The first consideration is the context of research use. As noted by Davies, Nutley, and Walter (2008, p. 190), “research use is a highly contingent process. Whether and how new information gets assimilated is contingent on local priorities, cultures and systems of meaning. What makes sense in one setting can make a different sense in another.” The second consideration is that “interpersonal and social interactions are often seen as key to accessing and interpreting such research knowledge, whether among policy or practice colleagues, research intermediaries or more directly with researchers themselves” (Davies, Nutley, & Walter, 2008, p. 189).

The adoption of RSTs with or without the use of an RSIS in public youth-serving systems provides an opportunity to address the challenges inherent with both the instrumental and the conceptual models of evidence use because it enables us to examine simultaneously the outcome of a specific decision (whether or not and to what extent to adopt an RST) as well as the process by which evidence is acquired, evaluated, and applied to make or support that decision. Frontline providers’ receptivity to RSTs and RSISs seems to be dependent on their perception of the fit between the RST or RSIS and the agency’s goals, values (Rosenheck, 2001), and service delivery platform (Leslie, Maciolek, Biebel, Debordes-Jackson, & Nicholson, 2014) and on the fit and commitment to the RST or RSIS among such agency leaders as the director, board, and senior managers (Aarons, Hurlburt, & Horwitz, 2011; Aarons & Palinkas, 2007; Klein & Sorra, 1996; Proctor et al., 2007). The literature suggests that agency leaders must proactively cultivate a “research attuned culture,” where evidence is valued and reinforced (Huberman, 1994; Lavis et al., 2003; Roos & Shapiro, 1999).

In spite of the theory and research pointing to the key role of agency leaders in RST implementation, little empirical research has been conducted on the use of research evidence for this purpose. In part, the absence of information can be attributed to the lack of objective, quantifiable measures of research evidence use (Grol & Grimshaw, 1999; Innvaer, Vist, Trommwald, & Oxman, 2002). Although a few attempts have been made to scale or score the degree of impact of use of research evidence (Hanney, Davies, & Buxton, 1999; Jacob & McGregor, 1997; Lavis et al., 2003), the two methods used most frequently to assess the use of research evidence are documentary analysis and in-depth interviews (Hanney, Gonzalez-Block, Buxton, & Kogan, 2003; Innvaer et al., 2002).

Use of research evidence is also an important component of EBP. Drawing from the concept of evidence-based medicine (Sackett, Rosenberg, Muir Gray, Haynes, & Richardson, 1996), EBP is defined as the process involving the interaction between, and integration of evidence gained from scientific research, patient preferences and clinical experience. EBP as a process involves the translation of the best available scientific evidence to implementation and routine practice (Soydan & Palinkas, 2014). RSTs may be viewed as products of this process.

In addition to understanding the role of use of research evidence in EBP and the implementation of specific RSTs, assessment of patterns of research evidence use is central to the task of monitoring the application of scientific principles in social work practice and potentially in the creation of an applied science of or in social work (Brekke, 2012, 2014; Soydan & Palinkas, 2014). Implied in the notion of such an applied science is that the knowledge generated through rigorous research be accessible, relevant, transparent, and practical for policy makers and practitioners (Soydan & Palinkas, 2014). A tool for monitoring the use of research evidence would be invaluable in helping to (1) identify and address barriers to access to such evidence (Gilgun, 2005; Proctor et al., 2007), (2) integrate research and practitioner criteria for evaluating the validity, reliability, and relevance of such evidence (Anastas, 2014; Rosen, 2003), and (3) develop standards for the effective and rigorous application of research evidence in policy and practice (Brekke, 2012; Palinkas, 2014).

The main objective of this study was to explore and identify dimensions of research evidence use by generating quantitative items from qualitative content domains of evidence acquisition (input), evaluation (processing), and application (output) and subjecting them to exploratory factor analysis (EFA) in order to discern their factor structure. Our goal was to develop a scale that focused on the use of research evidence when implementing RSTs and to examine its factor structure, reliability, and validity. This is in distinction from the creation of the research evidence (e.g., through observational studies or experimental clinical trials) used to validate or support a particular treatment. We drew from relevant theory and literature on the use of research evidence (Davies et al., 2008; Honig & Coburn, 2008; Kennedy, 1984; Nutley et al., 2007), and feedback from leaders of county and state-level child welfare, mental health, and juvenile justice systems.

Method

Item Generation for the Structured Interview for Evidence Use

Item generation and domain identification proceeded in three phases. First, the research team developed a set of questions based on a review of the literature relating the use of research evidence to adoption and implementation of innovative practice and EBP. Second, interview and focus group data were collected from 64 Southern California child welfare directors, probation officers, and mental health department directors or consultants. Coding of these data generated items that represent the extent to which system leaders and supervisors obtain research evidence from a variety of sources, including information that is directly accessed by the respondent (e.g., Internet or academic journals), and external resources such as consultants, intervention developers, and federal and state agencies. Questions were also asked to understand whether and by what means respondents evaluate the validity, reliability, and generalizability of research evidence, and the conditions under which the evidence was used or ignored. Third, a set of items were generated by the research team and reviewed by a group of 26 child welfare, child mental health, and juvenile justice systems leaders for face and content validity. Using the Newell and Simon (1972) model of information processing, 60 items were developed that represented 3 potential content domains of evidence use, that is, acquisition of the evidence (input); evaluation of the validity, reliability, and relevance or generalizability of the evidence (process); and application of the evidence in making a decision whether or not to implement an innovative practice or EBP (output). See Appendix for a list of all the items.

Participants

Data from three separate groups of participants were used in this study. The first group of participants (Cohort 1) was involved in a larger randomized controlled trial (RCT) comparing the use of community development teams (Sosna & Marsenich, 2006) versus technical assistance to scale-up Multidimensional Treatment Foster Care (MTFC), an EBP to reduce behavioral problems and delinquency among youth aged 8–18 placed out-of-home care in public youth-serving systems in California and Ohio (Chamberlain, Leve, & Degarmo, 2007). MTFC is 1 of the original 10 highest tier RSTs designated by the Blueprints for Healthy Youth Development (2014) whose certification standards for determining which programs are evidence based are among the highest in the field. MTFC has demonstrated effects on preventing violence, delinquency and criminal behavior, illicit drug use, depression, and teen pregnancy and because of these benefits and its high potential for cost savings (Leve et al., 2012). Data for the Structured Interview for Evidence Use (SIEU) were collected from 140 (63.3% response rate) of the 221 available mental health, child welfare, and probation system leaders, supervisors, and administrators who were participating in the RCT at the time this study was conducted (2010–2012).

The second group of participants (Cohort 2) included leaders of child welfare, juvenile justice, or mental health systems in California counties not participating in the RCT or in other states that had implemented MTFC and had received technical assistance from Treatment Foster Care Consultants, Inc., an organization dedicated to the implementation of community-based programs that are cost-effective and achieve positive outcomes for children, youth, and families. The recruitment sample was limited to those who had implemented MTFC within 1 year prior to the initiation of this study. Data were collected from 26 (89.7% response rate) of the 29 system leaders invited to participate in this study.

The third group of participants (Cohort 3) were involved in a study of the use of research evidence to develop and implement policies related to psychotropic medication use among youth in foster care (Cohort 3). We initially requested to speak to the individual most knowledgeable about the oversight of mental health care for children in foster care, within the state child welfare agency of each of the 50 states and the District of Columbia of the United States. If the first contact had insufficient information to answer questions particular to a monitoring mechanism, we asked that person to refer us to the appropriate informant in their state or to have the colleague join for the interview. Data were collected from 36 (50% response rate) of the 72 system leaders interviewed for the larger study.

Procedure

The study was approved by the appropriate institutional review boards prior to participant recruitment, and informed consent was obtained prior to administering surveys. Participants were contacted via e-mail for recruitment to the study. Data were collected using online surveys. Each participant was e-mailed an invitation to participate as well as a link to the web-based survey. Participants reviewed informed consent and after agreeing to participate were able to access the survey and proceed to the survey items. Once participants logged in to the online survey, they were able to answer questions and could pause and resume at any time. The online survey took approximately 15–20 min to complete.

Measures

The SIEU

Item development for the SIEU is described earlier. Respondents were asked to indicate the level of agreement with a series of statements using a Likert-type scale ranging from 1 (not at all) to 5 (all the time) for the items contained in the Input scale, and a similar five-point Likert-type scale ranging from 1 (not important) to 5 (very important) for the items contained in the Process and Output scales. Lower scores on all three scales indicate lower levels of agreement, while higher scores indicate higher levels of agreement with the respective statements. Five of the 20 Output items containing statements relating to circumstances in which the research evidence is ignored were reversed scored so that a higher score for the overall Output scale reflected greater use of research evidence. Each subscale and the total score are represented as an average of the scores for each item included in the subscale/total scale. Data on research evidence use were collected from all three cohorts (N = 202).

Evidence-Based Practice Attitudes Scale

This measure examines behavioral health service provider attitudes toward adoption of EBPs in the first two studies via 18 items in four domains, namely, appeal (the extent to which the provider would adopt a new practice if it is intuitively appealing), requirements (the extent to which the provider would adopt a new practice if it is required by an agency, supervisor, or state), openness (the extent to which the provider is generally open to trying new interventions), and divergence (the extent to which the provider perceives research-based interventions as not clinically useful and less important than clinical experience; Aarons, 2004). The measure shows good internal consistency with αs for the four domains ranging from .66 to .91 and an overall scale α of .76 (Aarons et al., 2010). This measure was chosen to assess the convergent validity of the SIEU, as it was hypothesized that more positive attitudes toward EBP would be correlated with greater engagement in the use of research evidence. Data on evidence-based attitudes were collected from Cohort 1 only (n = 140).

Organizational Social Context

Items from the Organizational Social Context (OSC) scale, developed by Glisson and colleagues (2008), were used along with additional items to assess the psychological climate, structure, work attitudes, and a global measure of organizational culture and climate of the agencies participating in the RCT (n = 140). These constructs were constructed using procedures taken from the Children’s Services Survey (CSS) measure developed by Glisson and James (2002). Psychological climate is employees’ perceptions of the psychological impact of the work environment on their well-being. This construct was assessed using 45 items from the OSC and 1 new item and included measures of depersonalization, emotional exhaustion, fairness, growth and advancement, personal accomplishment, role clarity, role conflict, role overload, and cooperation. Structure refers to formalization and hierarchy of authority in the organization and was assessed using 12 items from the OSC and 2 new items. Work attitudes refer to job satisfaction and organizational commitment and was assessed using 17 items from the CSS. The α for the three measures was 90, .60, and .86, respectively. The global measure of organizational culture and climate was derived from the aggregation of 12 of the original 14 CSS first-order scales (depersonalization, emotional exhaustion, growth and advancement, personal accomplishment, role clarity, role conflict, role overload, cooperation, formalization, hierarchy of authority, job satisfaction, and organizational commitment). The α for this global measure was .65. These measures were selected to examine the divergent validity of the SIEU as organizational culture and climate were hypothesized to be unrelated to the use of research evidence. Data on OSC were collected from Cohort 1 only (n = 140).

Statistical Analyses

Each SIEU variable had missing data ranging from 0.5% to 6.4%. Only 1 of the 20 items for Input (Information from clients or their parents, 3.9%), and 1 of the 20 items for Process (assess after implementation, 3.5%), had more than 1.5% of the data missing. In contrast, the missing data for the 20 items for Output ranged from 1.0% to 6.4%, averaging 3.56%. Imputation of missing values was performed using a mean method.

Four separate exploratory factor analyses were conducted with IBM SPSS Version 20.0 Statistical Software to examine the factor structure of the Total, Input, Process, and Output Scales. Principal axis factoring (PAF) with Promax oblique rotation were conducted for each set of analyses. PAF was selected for factor extraction because it allows for consideration of both systematic and random error (Fabrigar, MacCallum, Wegener, & Strahan, 1999), and Promax oblique rotation was utilized as we assumed that the derived factors would be correlated (Russell, 2002). Three criteria were used to determine the number of factors to retain, namely, (1) examination of the oblique rotated factor pattern matrix, (2) parallel analysis (Patil, Singh, Mishr, & Donovan, 2008), and (3) interpretability of the factor structure as indicated in the rotated solution. Examination of the rotated factor structure included identification of eigenvalues above 1.0 and Scree test results as well as absence of multicollinearity and presence of outliers (DeVellis, 2003). Based on Fabrigar, MacCallum, Wegener, and Strahan’s (1999) criteria, items were removed if the item loadings were low (< .40) or if there were cross-loadings of .40 or higher. Using criteria provided by Reise, Waller, and Comrey (2000), each factor had a minimum of 3 items. Parallel analysis is among the better methods for determining the number of factors based on simulation studies (O’Connor, 2000). The number of factors to retain corresponds to the number of eigenvalues generated from the EFA that are larger than the corresponding random eigenvalues (Streiner & Norman, 2008). These corresponding eigenvalues were generated based on sample size, the number of variables in the data set, and the 100 random data matrices using the eigenvalues that correspond to the 95th percentile of the distribution of random data eigenvalues (Patil et al., 2008).

Reliability of the SIEU was assessed by examining Cronbach’s α internal consistency for each of the subscales and the total scale. Cronbach’s α of .80 and above is considered good internal consistency for a newly developed scale (DeVellis, 2003; Streiner & Norman, 2008). Corrected item-total correlation tests were also conducted in order to check each item’s contribution to the total scale (DeVellis, 2003). Item analyses were also conducted, including an examination of interitem correlations and α, if items are removed.

Convergent and discriminant validity was assessed by computing Pearson’s product–moment correlations of SIEU sub-scale and total scale scores with EBPAS subscale and total scores, and the OSC subscales and total scores, respectively.

Results

The demographic characteristics of the participants of this study are described in Table 1. Of the participants in the first two cohorts (n = 166), the mean age was 49 (SD = 9.4; range = 26–67) and the majority of them were female (68.5%) and Caucasian (85.2%; these data were not available from Cohort 3 participants). Most of the participants in all three cohorts (N = 202) attained master’s level or higher degrees (77.2%; data were missing from seven respondents) and worked in child welfare (42.8%), mental health (18.6%), juvenile justice (13.9%) or “other”¹ agencies (24.7%), respectively (data were missing from eight respondents). Respondents also held a range of positions from within these agencies, specifically 25 (82.1%) identified as a director, 18 (9.2%) identified as program administrator, and 17 (8.7%) identified as a staff member (data were missing from 6 respondents).

Table 1.

Demographic Characteristics of Study Participants.

Characteristic	RCT Study participants, n = 140		MTFC Implementers, n = 26		Psychotropic Medication Study Participants, n = 36
Age, mean (SD)	48.95 (9.37)		50.03 (9.04)			N/A
Gender
Male	41	30.6	10	38.5		N/A
Female	93	69.4	16	61.5
Ethnicity
Non-Hispanic White	118	84.3	23	88.5		N/A
Other	22	15.7	3	11.5
Education
High school	15	11.2	0	0.0	0		0.0
Some college	35	26.1	2	7.7	1		2.8
College	2	1.5	1	3.8	8		22.2
Masters	70	52.2	22	84.6	16		44.4
Registered nurse	0	0.0	0	0.0	2		5.6
Nurse practitioner	0	0.0	0	0.0	1		2.8
PhD/JD/MD	12	9.0	1	3.8	8		22.2
Agency
Child welfare	49	36.8	3	12.0	31		86.1
Mental health	29	21.8	6	24.0	1		2.8
Juvenile justice	25	18.8	2	8.0	0		0.0
Other	30	22.6	14	52.0	4		11.1
Role
Directora	111	80.4	17	77.3	33		91.6
Program administrator	15	10.9	3	13.6	0		0.0
Staff	12	8.7	2	9.1	3		8.4

Note. MTFC, Multi-dimensional Treatment Foster Care; N/A = not available; RCT = randomized controlled trial.

^aIncludes a variety of positions that function in a senior management capacity, including medical directors, agency CEOs, senior managers, analysts, and specialists.

EFA

Prior to conducting factor analyses on each of the three scales, a PAF analysis was conducted on the entire 60-item scale. The adequacy of factor analysis for the scale was tested with Kaiser–Meyer–Olkin Measure of Sampling Adequacy (KMO) and Bartlett’s Test for Sphericity. The KMO for the data was .74, greater than the recommended cutoff value of .60 (Tabachnick & Fidell, 2001), and Bartlett’s Test of Sphericity was significant, χ²(1,711) = 5,181.76, p < .001, indicating that the strength of association among the input items was large enough for factor analysis. However, 18 factors were extracted and rotation failed to converge in 30 iterations, making the structure uninterpretable. We therefore decided to proceed with examining each scale separately.

Tables 2–4 display the factor means, item means, commonalities (h²), initial eigenvalues, variance accounted for by each factor, internal consistency reliabilities, and rotated factor loadings. The KMO for the 20-item Input scale was .78, and Bartlett’s Test of Sphericity was significant, χ²(190) = 1,052.57, p < .001, indicating that the strength of association among the process items was large enough for factor analysis. Seventeen items were retained from the Input scale after applying the factor loading criteria indicated earlier, and the scree plot and parallel analysis suggested retaining three factors. Item commonalities ranged from .193 (i.e., “regular staff meetings”) to .527 (i.e., “someone I heard at a conference”). Examination of the items presented in Table 2 suggests that the content of items loading on Factor 1 could best be labeled as accessing members of local or personal networks (7 items), content of items loaded on Factor 2 could be labeled as “global” (i.e., external to the personal networks) experts (5 items), and content of items loaded on Factor 3 could be labeled as externally generated or “global” documents like training manuals, books, and curricula obtained from Internet searches, web-based clearinghouses, intermediary organizations, and academic journals. Internal consistencies were high, ranging from 0.70 to 0.86. Item analysis indicated that the interitem correlations were moderate to high (range = 0.39–0.53 for Factor 1, 0.41–0.58 for Factor 2, and 0.36–0.52 for Factor 3), and the α for the subscales would not be improved by removing any items.

Table 2.

Evidence Use Input Scale, Subscale, and Item Statistics.

							Factor loadings
Input Items	Mean	SD	ev	v	a	h2	1	2	3
1. Local Networks (7 items)	2.61	0.61	4.07	23.91	0.75
An administrator in another agency in my county/state	2.28	0.96				.425	.681	−.093	−.013
Regular meetings with professionals in my county/state	3.00	0.98				.415	.681	−.144	−.011
Nonprofit organizations/foundations	2.29	0.89				.312	.527	.081	−.001
Data collected by my agency	3.03	1.07				.267	.512	.070	−.054
Federal or state government agencies	2.44	0.91				.315	.499	−.017	.130
Regular staff meetings	3.02	0.99				.193	.441	.040	−.040
A consultant	2.19	1.03				.279	.425	.199	.008
2. Global Experts (5 items)	3.08	0.65	1.96	11.51	0.73
Someone I heard at a conference	2.82	0.94				.527	−.059	.734	.025
Someone who has implemented it	3.37	0.94				.431	.106	.633	−.048
People who developed the program	2.76	0.99				.366	.028	.565	.069
Professional association meetings	3.09	1.09				.282	.110	.526	−.119
Conferences or workshops	3.35	0.72				.240	−.161	.495	.068
3. Global Documents (5 items)	3.10	0.72	1.67	9.81	0.71
Training manuals/books/curricula	3.31	0.90				.420	−.050	.036	.654
Academic journals	2.81	0.99				.449	.072	.022	.626
Intermediary organizations	2.72	1.34				.359	.116	−.075	.566
Web-based clearinghouses	2.73	1.22				.289	.001	.076	.504
Internet	3.92	0.83				.195	−.103	−.048	.490
Total Input	2.89	0.48			0.80

Note. N = 202. SD = standard deviation; ev = eigenvalue; v = variance accounted for, a = alpha; h² = commonalities. Factor loadings: pattern.

Table 4.

Evidence Use Output Scale, Subscales, and Item Statistics.

							Factor Loadings
Output items	Mean	SD	ev	v	a	h2	1	2
1. Use the Evidence	3.65	0.53	3.77	31.43	0.80
To support decision on adopting program	3.95	0.79				.518	.718	.005
To find a program that meets needs of clients	3.76	0.81				.389	.633	−.046
To determine if program could harm participants	3.91	0.87				.403	.628	.026
To eliminate ineffective programs	3.65	0.84				.347	.601	−.063
To compare multiple programs strengths and weaknesses	3.27	0.87				.318	.569	−.019
To consider information from experts/community members	3.84	0.72				.289	.530	.027
To find the money to implement if evidence is strong enough	3.24	0.85				.248	.496	.011
Review evidence as a team with partner agencies	3.57	0.85				.205	.438	.050
2. Ignore the Evidence	3.18	0.87	2.33	19.42	0.84
If program is not feasible for my county/state	3.20	1.00				.702	−.102	.857
If no resources to implement	3.21	1.13				.695	.039	.823
If program is too rigid	3.02	1.07				.455	−.082	.690
If program doesn’t match staff skill level	3.28	1.04				.504	.201	.633
Total output	3.22	0.46			0.80
Total SIEU	3.38	0.37			0.88

Note. N = 202. SIEU = Standard Interview for Evidence Use; SD = standard deviation; ev = eigenvalue; v = variance accounted for, a = alpha; h² = commonalities. Factor loadings: pattern.

The KMO for the 20-item Process scale was .84, and Bartlett’s Test of Sphericity was significant, χ²(190) = 1,527.24, p < .001, indicating that the strength of association among the process items was large enough for factor analysis. As reported in Table 3, the Process scale includes 16 items, divided into 3 factors. Item commonalities ranged from .207 (review by intermediary organizations) to .738 (methods clearly described). Factor 1 includes 9 items and refers to self-assessments of the validity and reliability of research evidence. Factor 2 includes 3 items and refers to assessments of validity, reliability, and relevance or generalizability by others. Factor 3 includes 4 items and refers to self-assessments of the evidence for relevance or generalizability to one’s service area (county/state). Internal consistencies were high, ranging from 0.71 to 0.88. Item analysis indicated that the interitem correlations were moderate to high (range = 0.43–0.76 for Factor 1, 0.43–0.67 for Factor 2, and 0.41–0.59 for Factor 3), and the α for the subscales would not be improved by removing any items, with the exception of the item assessing reliance on peers to assess relevance; removal of that item would increase the α from .74 to .80. However, we decided to retain the item as it was conceptually linked to reliance on others for validity and reliability.

Table 3.

Evidence Use Process Scale, Subscales, and Item Statistics.

							Factor Loadings
Process items	Mean	SD	ev	v	a	h2	1	2	3
1. Self-Assessment for Validity and Reliability (9 items)	3.82	0.61	5.43	33.92	0.88
Methods clearly described	3.79	0.89				.738	.884	−.089	−.029
Information from more than one source	3.85	0.84				.653	.804	−.018	.016
Strengths and weaknesses listed	4.05	0.74				.623	.796	.019	−.021
Tested in the field	4.01	0.79				.553	.727	−.038	.045
How evidence is structured	3.99	0.81				.532	.718	−.110	.051
Based on theory	3.50	0.93				.327	.578	.109	−.079
Credibility of developers	3.61	0.90				.399	.507	.358	−.092
Reviewed by intermediary organizations	3.25	1.22				.207	.473	.028	−.055
Outcomes	4.34	0.64				.321	.435	.005	.207
2. Reliance on Others (3 items)	3.42	0.74	2.13	13.29	0.74
People I know and trust to assess reliability	3.46	0.92				.680	−.080	.820	.059
People I know and trust to assess validity	3.45	0.91				.585	.055	.778	−.100
Use of peers to determine relevance	3.35	0.92				.319	−.013	.474	.208
3. Self-Assessment for Relevance (4 items)	4.06	0.59	1.50	9.39	0.71
How much it costs to implement	4.18	0.90				.366	−.168	−.011	.672
Compare own needs with that of research population	4.11	0.73				.513	.163	−.079	.643
Time required to train staff	4.02	0.80				.458	−.002	.140	.624
Service areas with similar demographics	3.94	0.81				.294	.120	.044	.459
Total process	3.80	0.48			0.86

Note. N = 202; SD = standard deviation; ev = eigenvalue; v = variance accounted for, a = alpha; h² = commonalities. Factor loadings: pattern.

The KMO for the 19-item Output scale was .80 and Bartlett’s Test of Sphericity was significant, χ²(171) = 975.22, p < .001, indicating that the strength of association among the process items was large enough for factor analysis. Table 4 shows that 12 items were retained from the Output scale. Item commonalities ranged from .205 (review evidence as a team with partner agencies) to .702 (ignore evidence if program is not feasible for my county/state). The two factors that were retained based on results from the parallel analysis focus on (1) the circumstances in which evidence is used (8 items) and (2) circumstances when evidence is ignored (4 items). Internal consistencies were high, ranging from 0.71 to 0.88. Item analysis indicated that the interitem correlations were moderate to high (range = 0.39–0.64 for Factor 1 and 0.60– 0.75 for Factor 2), and the α for the subscales would not be improved by removing any items.

As shown in Table 5, factor correlations ranged from 0.33 to 0.43 for the three subfactors of Input, 0.19–0.48 for the three subfactors of Process and 0.24 for the two subfactors of Output. All correlations were statistically significant. However, correlations of the three scales ranged from 0.73 to 0.78, suggesting a higher order construct of engagement in evidence use. The interitem α for the entire 45-item scale was .88.

Table 5.

Scale Factor Correlations.

Factor	1	2	3
Input
Local Networks	1.00
Global Experts	.33**	1.00
Global Documents	.44**	.38**	1.00
Process
Self-Assessment of Validity/	1.00
Reliability
Assessment by Others	.19*	1.00
Self-Assessment of Relevance	.49**	.29**	1.00
Output
Use Evidence	1.00
Ignore Evidence	.25*	1.00
Total evidence use	Input	Process	Output	Total
Input	1.00
Process	.43**	1.00
Output	.31**	.40**	1.00
Total	.80**	.81**	.68**	1.00

^*p < .01;

^**p < .001.

Convergent and Discriminant Validity

Table 6 shows the results of the convergent and discriminant validity analyses. Input was significantly associated with the EBPAS subscales representing requirements and openness and the total EBPAS score. Process was inversely associated with divergence, and Output was significantly associated with the EBPAS subscales of requirements, openness, and divergence, and the total EBPAS score. As predicted, the SIEU scores had low correlations with organizational climate, structure, and work attitudes. Correlations ranged from 0.04 to 0.17, indicating strong support for the discriminant validity of the SIEU in contrast to the three dimensions of organizational culture and climate.

Table 6.

Pearson’s Product Moment Correlations of SIEU Subscale Scores With Attitudes Toward Evidence-Based Practice (Discriminant Validity) Scores and Organizational Culture (Convergent Validity) Scores.

Evidence-Based Practice Attitudes	Input	Process	Output
Requirements	.11	.03	.22*
Appeal	.13	.08	.15
Openness	.27**	.08	.21*
Divergence	−.17	−.25*	−.25*
Total EBPAS	.22*	.06	.24*
Organizational culture and climate
Climate	.10	.09	.09
Structure	.07	.04	.05
Work attitudes	.13	.15	.17
Total OSC score	.00	.15	.10

Note. EBPAS = Evidence-Based Practice Attitudes Scale; n = 140.

^*p < .05;

^**p < 01.

Discussion and Application to Social Work

This study represents the initial phase of the development of a scale to measure the use of research evidence. Although there have been previous attempts to scale the outcomes associated with research utilization (Hanney et al., 1999; Jacob & McGregor, 1997; Lavis et al., 2003), to our knowledge, the SIEU is the first measure of engagement in use of research evidence ever developed. We used an iterative process to develop items representing the acquisition, evaluation, and application of research evidence and then used quantitative data reduction techniques to develop a measure that may easily and efficiently be used for research and practice. Such measures are needed to improve our understanding of the role of the use of research evidence in EBP as well as the implementation of RSTs or the use of research-supported strategies to facilitate such implementation. However, its potential use extends far beyond that to training and quality improvement.

Although the instrument has the potential of providing a global assessment of engagement in use of research evidence, it is actually comprised of three factors representing different constructs. These three measures are consistent with human information processing (Newell & Simon, 1972) and policy decision-making (Weiss, 1980) models containing three factors, namely, input, process, and output. The Input scale contained three different sources of research evidence, (1) information that is available globally through print (i.e., academic journals, training manuals), (2) electronic media (e.g., the Internet, web-based clearinghouses) or organizations whose function is to disseminate research evidence through print or electronic media (e.g., intermediary organizations like the California Institute of Mental Health), and (3) information obtained from members of local or personal networks. Based on the average scores of all three subscales, systems leaders tend to rely primarily on global documents, followed by global experts and local networks.

The Process scale also contained three separate constructs. Two of the three constructs related to self-assessments of the research evidence for validity and reliability (one construct) and generalizability or relevance to one’s own service area. As indicated by the mean scores in Table 2, system leaders appeared to pay greater attention to relevance when evaluating research evidence. This included comparing the needs and characteristics of one’s own population with the populations represented in the research studies examined. It also included assessments of the costs involved to implement proposed EPBs, including the time required to train staff in the EBP. With respect to self-assessments of validity and reliability of the evidence, greatest attention appears to have been paid to the study outcomes, followed by an assessment of the study’s strengths and weaknesses, whether the intervention or practice was tested in the field, how the evidence was structured or presented, whether it came from more than one source, the credibility of the developers of the research, whether it was grounded in theory, and whether it had been reviewed by intermediary organizations. This is consistent with other studies that have found an association between an assessment of the quality of the research and the specific researcher and a policy maker’s inclination to use the research (Buxton & Hanney, 1996; Hanney et al., 2003). Levels of utilization of research evidence have also been linked to reliability, timeliness, and comprehensiveness (Drummond & Weatherly, 2000). Reliance on others (peers, “people I know and trust”) to assess the evidence for validity, reliability, and relevance is consistent with network models of policy making (Hanney et al., 2003).

The Output scale contained two subscales representing the conditions under which research evidence was either used or ignored. The most common reason for using research evidence, based on mean scores, was to support a decision that was already made. As Hanney, Gonzalez-Block, Buxton, and Kogan (2003, p. 9) observe, “specific findings can be used to legitimate decisions when these have been formed, have hardened or when they have been implemented.” Often, such decisions are made for reasons of interest, ideology, or intellect. “Under these circumstances, research can still be used as ammunition to support the decisions made and being implemented. Science content here is used as a collection of arguments, rather than data or evidence to be weighed” (Hanney et al., 2003, p. 10). This was followed by the use to determine whether the proposed program could potentially be harmful to clients and the use to evaluate information provided by experts on the one hand and community members on the other. In some instances, evidence was used for purposes of quality improvement (eliminating ineffective programs, meeting the needs of clients); in other instances, evidence was used instrumentally as part of a collaborative endeavor (reviewing evidence as a team with partner agencies) or to seek financial support for an RST. These uses are consistent with problem-solving/engineering-policy–driven models and interactive/social interaction models of research utilization proposed by Hanney and colleagues (2003). Use of evidence to compare multiple strengths and weaknesses is related to an assessment of the evidence itself for validity and reliability, linking process with outcome. Ignoring the evidence was based on a determination that the program or practice was not feasible or insufficiently flexible (i.e., adaptable), given existing resources and staff.

The SIEU demonstrated strong internal consistency reliability with an overall α of .88 for the entire scale. There was some support for the convergent validity of the measure as indicated by the significant correlations between the EBPAS requirements, openness and divergence subscales, and total score and the three SIEU scales. There was strong support for the discriminant validity of the SIEU when examining associations with the subscales and total score for the OSC measures.

Applications to Practice

The results of this study demonstrate that the SIEU has the potential to identify patterns of the use of research evidence among different groups of practitioners. It may also enable researchers to better identify and quantify potential predictors of the use of research evidence, thereby contributing to theory development. From a practice perspective, the SIEU may help to facilitate acquisition, evaluation, and application of research evidence in policy and practice settings. For instance, analyses of patterns of acquisition and application of research settings may enable the appropriate and cost-effective matching of sources likely to provide the research evidence necessary for specific applications such as conducting needs assessments, eliminating ineffective programs and practices, and identifying the most suitable programs and practices for one’s service population. It may also be used to train practitioners in the best use of research evidence to support programmatic and clinical practice decisions and to improve decision outcomes that are based on the use of research evidence. Finally, it may prove critical in developing or advancing the use of scientific principles in the profession of social work. As Marsh (2012, p. 467) observes, “analyses that ask whether a piece of research is consistent with the core constructs, ethical code, and fundamental purposes of social work are critically important to advancing the science of social work.” It is our contention that such analyses are not possible without a way of measuring how the evidence or knowledge resulting from research is acquired, evaluated, and applied in policy and practice settings.

Limitations

There are a number of factors that limit the findings of this study. The sample used in this study included participants of three different studies. Two of the studies involved an examination of the use of a specific RST, while participants in the third study were involved in an examination of statewide practice guidelines for the use of psychotropic medications for youth in foster care. The different contexts in which the studies were conducted may have had an impact on the relevance and utility of the scales. Moreover, while there is considerable debate as to the minimal sample size for an exploratory factor analysis (Costello & Osborn, 2005), a larger sample could have produced more representative coefficients with greater stability and would have also enabled us to split the sample in order to conduct a confirmatory factor analysis. Caution must be exercised in evaluating the convergent and discriminant validity of the measure as the reliability for some of the subscales of the EBPAS and the Organizational Context Scales and the OCS global measure was less than desirable (i.e., between .60 and .65). Finally, as a tool for monitoring engagement in EBP, the SIEU can only assess the use of research evidence, and it does not assess the use of evidence from client preferences or clinical experience.

Future Steps

Future studies using confirmatory factor analyses should be conducted to evaluate the factor structure of the SIEU in independent samples to determine whether the structure is invariant to users of research evidence from different settings (e.g., local, state, and national), specializations (e.g., social work, clinical psychology, and education), and demographic characteristics (e.g., age, gender, ethnicity). Further factor analytic research is needed to determine whether the scale can be reduced to fewer items, making it easier to administer and requiring less time to complete. Once the instrument has been thoroughly validated in independent samples, the SIEU could be used in structural equation and other multivariate modeling to test hypotheses related to the use of research evidence in adoption of innovative, research-supported treatments and implementation strategies, engagement in EBP and to the application of scientific methods in social work practice.

Appendix.

Structured Interview of Evidence Use

A. Given subsequently are a series of statements about where and how you obtain information or research evidence related to a particular program or intervention. On a scale of 1–5, please indicate how much you rely on each of the following to obtain information about a particular program or intervention:

1 = Not at all; 2 = Rarely; 3 = Occasionally; 4 = Often; and 5 = All the time.

		Not at all	Rarely	Occasionally	Often	All the Time	Refuse to Answer
When I need information or research evidence related to a particular program or intervention,
1	I search the Internet (e.g Google or other general search engines)	1	2	3	4	5	9
2	I search academic journals	1	2	3	4	5	9
3	I review available training manuals/books/curricula	1	2	3	4	5	9
4	I rely on particular staff members of my agency to obtain it for me	1	2	3	4	5	9
5	I contact the people who developed the program	1	2	3	4	5	9
6	I contact an expert from a local college or university	1	2	3	4	5	9
7	I contact someone who gave a presentation about the program or intervention I heard at a conference I attended	1	2	3	4	5	9
8	I contact someone in another county that has already implemented this program or intervention	1	2	3	4	5	9
9	I rely on particular web-based clearinghouses (e.g., Children Trends Database, Social Care Institute for Excellence, Mental Health Services Archives, etc.)	1	2	3	4	5	9
10	I rely on intermediary organizations like the California Institute of Mental Health (CIMH) or Center for Innovative Practice (CIP)	1	2	3	4	5	9
11	I rely on a particular consultant to obtain it for me	1	2	3	4	5	9
12	I rely on a particular director or senior administrator from another agency in my county	1	2	3	4	5	9
13	I rely on particular federal or state government agencies	1	2	3	4	5	9
14	I rely on particular nonprofit organizations/foundations	1	2	3	4	5	9
15	I rely on data that were collected by my agency or someone we hired to collect data	1	2	3	4	5	9
16	I obtain it through attending regularly scheduled meetings of a professional association	1	2	3	4	5	9
17	I obtain it from conferences or training workshops	1	2	3	4	5	9
18	I obtain it from attending regularly scheduled meetings with my staff	1	2	3	4	5	9
19	I obtain it from attending regularly scheduled meetings with other professionals in my county	1	2	3	4	5	9
20	I hear about it from my clients or their parents	1	2	3	4	5	9

Note. Items in boldface are included in the 45-item scale.

B. Below are a series of statements about how people evaluate the validity, reliability, and relevance of research evidence they obtain from various sources. On a scale of 1–5, please indicate how important each criterion is for you to evaluate the quality of research evidence:

1 = Not important at all; 2 = Not important; 3 = Neutral; 4 = Important; 5 = Very important.

		Not Important at All	Not Important	Neutral I	Important	Very Important	Refuse to Answer
When judging the validity of research evidence supporting a particular program or intervention I’m interested in,
1	I look at program outcomes	1	2	3	4	5	9
2	I rely on subject matter experts	1	2	3	4	5	9
3	I rely on opinions of clinicians who have been using the program	1	2	3	4	5	9
4	I review the experience of other counties implementing the program	1	2	3	4	5	9
5	I assess the credibility of those people developing the program	1	2	3	4	5	9
6	I rely on the review by intermediary organizations like the California Institute of Mental Health (CIMH) or Center for Innovative Practices (CIP)	1	2	3	4	5	9
7	I rely on people I know and trust to tell me if it is valid	1	2	3	4	5	9
8	I see if it is based on theory	1	2	3	4	5	9
9	I see whether it has been tested in the field for a period of time	1	2	3	4	5	9
10	I assess its effectiveness only after it has been implemented in my county or agency, given the uniqueness of each county	1	2	3	4	5	9
When judging the reliability of research evidence supporting a particular program or intervention I’m interested in,
1	I see if the information is obtained from more than one source and is consistent	1	2	3	4	5	9
2	I see if the research methods are clearly described	1	2	3	4	5	9
3	I see if its potential strengths and weaknesses are listed	1	2	3	4	5	9
4	I see how the evidence is structured (e.g., if it is logical, or looks like superficial advertising?)	1	2	3	4	5	9
5	I rely upon people I know and trust to tell me if it is reliable	1	2	3	4	5	9
When judging the relevance to my county of research evidence supporting a particular program or intervention I’m interested in,
1	I see how much it costs to implement the program	1	2	3	4	5	9
2	I compare the needs of my county with the needs of the populations in the research studies	1	2	3	4	5	9
3	I look at the program effects in counties with similar demographics as mine	1	2	3	4	5	9
4	I see how much time is required to train staff to use the program	1	2	3	4	5	9
5	I rely on my professional peers to determine the relevance of the program	1	2	3	4	5	9

Note. Items in boldface are included in the 45-item scale.

C. Given subsequently are a series of statements regarding how people use research evidence to influence their decisions on implementing certain evidence-based practices. On a scale of 1–5, please indicate how important each characteristic is for you when deciding whether or not to adopt a new program or practice:

1 = Not important at all, 2 = Not important, 3 = Neutral, 4 = Important, and 5 = Very important.

		Not Important at All	Not Important	Neutral	Important	Very Important	Refuse to answer
When deciding to adopt a new program or intervention in my county,
1	I make site visits to counties that have already implemented the program	1	2	3	4	5	9
2	I give greater weight to any outcome data we collect in our own county than data collected elsewhere or in research studies	1	2	3	4	5	9
3	I compare more than one program or intervention simultaneously to evaluate their respective strengths and weaknesses	1	2	3	4	5	9
4	I conduct an assessment of the needs of the population my agency serves and then find a program that meets those needs	1	2	3	4	5	9
5	I assess how much adaptation is required to meet the needs of real people and real families	1	2	3	4	5	9
6	I tend to ignore the research evidence if I am not convinced that the program or intervention will work for a particular population	1	2	3	4	5	9
7	I tend to ignore the research evidence if there are no resources to implement the program	1	2	3	4	5	9
8	I tend to ignore the research evidence if the program is too rigid and cannot be adapted to meet the needs of my clients	1	2	3	4	5	9
9	I tend to ignore the research evidence if the program does not match the skill level of my staff	1	2	3	4	5	9
10	I tend to ignore the research evidence if the program is not feasible for my county, given my county’s capability	1	2	3	4	5	9
11	I rely on strategies currently in use that have proven effectiveness without having to review the scientific evidence supporting the program	1	2	3	4	5	9
12	I defer to the decisions of the people I work for (e.g., judges, county supervisors, agency directors)	1	2	3	4	5	9
13	I consider the wishes of my partner agencies and review research evidence as a team	1	2	3	4	5	9
14	I will not introduce a new program or intervention that is not supported by research evidence, even if there are funds available to sustain the program	1	2	3	4	5	9
15	I would rather implement a program or intervention that has already been adopted by my colleagues in other counties	1	2	3	4	5	9
16	I will find money to implement the program anyway if the research evidence is strong enough to support it	1	2	3	4	5	9
17	I consider research evidence along with information obtained from subject matter experts and community members	1	2	3	4	5	9
18	I use research evidence to determine whether the program could do potential harm to participants before I consider implementing it	1	2	3	4	5	9
19	I use research evidence to support a decision on adopting the program at the executive level	1	2	3	4	5	9
20	I use research evidence to eliminate programs that proved to be not effective	1	2	3	4	5	9

Note. Items in boldface are included in the 45-item scale.