Measuring Attitudes Towards Empirically Supported Treatment in Real World Addiction Services

Abstract

Mental health workers with favorable attitudes toward empirically supported treatments (ESTs) are more likely to break through implementation barriers. The Evidence-Based Practice Attitudes Scale has been shown to be reliable for mental health workers, but has not been validated with addiction workers. This study investigates the use of the scale with a convenience sample of addiction workers from four agencies in one city. Results show that compared to mental health providers, addiction workers were more likely to view ESTs favorably if they were mandated and intuitively appealing. They also tended to rely more heavily on practical experience in forming attitudes toward treatment options. These results may help addiction agencies understand which types of workers are more likely to implement ESTs and inform effective engagement approaches specific to addiction workers.

The National Institute on Drug Abuse (NIDA) of the National Institutes of Health (NIH) indicated in its 2012 publication, Evidence-based Treatments in Real World Settings, that, along with developing and adding new interventions to the list of empirically supported treatments (ESTs), the field must investigate ways to implement and sustain ESTs in real world settings. Scientifically, addressing the bridge between effective ESTs and their efficient delivery is a documented priority for the NIH, the Institute of Medicine (IOM), and the federal government (IOM, 2000, 2001, 2006; NIDA 2009; U.S. Department of Health and Human Services, 2006).

In response, researchers have devoted resources to identifying characteristics of organizations and individuals more or less likely to adopt an EST. Commonly identified organizational characteristics of those adopting an EST include adequate resources, directors with higher levels of education, and organizational involvement with research (Lundgren, Amodeo, et al., 2011; Lundgren, Krull, Zerden, & McCarty, 2011).

Just as organizational issues can facilitate or impede EST implementation, studies have also examined worker-level factors that affect attitudes toward EST implementation. For instance, emerging literature indicates that a worker’s years of work experience (Aarons, 2004; Aarons et al., 2010, 2012; Pignotti & Thyer, 2009), educational attainment (Aarons, 2004; Osborne et al., 1998; Stahmer & Aarons, 2009), and educational discipline (Aarons, 2004; Stahmer & Aarons, 2009) shape the worker’s attitudes toward using ESTs. Furthermore, whether a student has completed an internship (Aarons, 2004; Patterson, et al., 2013) also affects the worker’s attitudes toward ESTs.

Worker characteristics, such as attitudes toward ESTs, are increasingly being recognized as a reliable factor for predicting EST implementation in certain social service settings. Social workers and other mental health providers with favorable attitudes toward ESTs are more likely to break through implementation barriers (Patterson, in press).

Aarons (2004) developed a brief, reliable measure of mental health workers’ attitudes toward ESTs, the Evidence-Based Practices Attitudes Scale (EBPAS). The EBPAS consists of 15 items written to assess four dimensions of provider attitudes: 1) openness to implementing new interventions (Openness); 2) the intuitive appeal of new interventions (Appeal); 3) willingness to use required interventions (Requirements); and 4) conflict between clinical experience and research results (Divergence). The hypothesized four factor structure of the scale was supported in three large studies of mental health providers (Aarons, 2004; Aarons, McDonald, Sheehan, & Walrath-Greene, 2007; Aarons, Glisson, Hoagwood, Kelleher, Landsverk, & Cafri, 2010). In a fourth large study with mental health providers, Patterson, Dulmus, Maguin, and Fava (in press) found that a five factor structure that divided the Divergence scale into two factors provided a better fit to their data. The EBPAS has been used to investigate how the four EBPAS dimensions relate to a set of worker demographic characteristics (Aarons, 2004; Aarons & Sawitzky, 2006; Aarons, et al., 2010; Patterson et al., 2012; Stahmer & Aarons, 2009). However, while the EBPAS has proven reliable and valid in the mental health field (Aarons, 2004; Patterson et al., in press), the scale’s psychometric properties have not been examined in the limited addiction treatment settings in which it has been administered (Henggeler et al., 2008; Smith & Manfredo, 2011).

We cannot assume that addiction treatment workers’ attitudes towards ESTs are equivalent to mental health workers’ attitudes, when the latter are more likely to have training in evidence-based practice than the former (Institute of Medicine Committee on Crossing the Quality Chasm, 2006; Substance Abuse and Mental Health Services Administration [SAMHSA, 2013). The gap in evidence-based preparation has led to increasing calls to improve the training and capacity of the addiction workforce which, relative to the mental health workforce, has fewer opportunities for specialized training, lower standards for employment, and lower educational attainment (SAMHSA, 2013; Wilkey, Lundgren, & Amodeo, 2013). If the addiction treatment workforce is to meet the challenges of modern-day health care, in which addiction treatment is becoming increasingly integrated with mental health and medical services, addiction workers must be willing and able to implement ESTs (Buck, 2011; Hoge et al., 2013). Being able to reliably measure addiction agency staff’s attitudes toward ESTs is thus important in building strategies for improving implementation. This study contributes to the external validity of the EBPAS scale by addressing the generalizability of the findings in a substantively different workforce and environment.

The overall purpose of the present study is to investigate the psychometric properties of the EBPAS with a convenience sample of addiction agency staff from four agencies in one city. This study will examine whether the EBPAS items have the same factor structure as that found in samples of mental health agency staff. It will also consider whether EBPAS scale scores for the present sample differ from those observed in prior studies. Finally, we assess how staff demographic and job-related characteristics are related to EBPAS scale scores.

METHODS

Sample

We recruited a convenience sample of 120 front-line addiction workers within four agencies providing addiction services in the St. Louis area of Missouri. The four agencies’ workforces ranged between six and fifty-five workers. Respondents primarily worked in outpatient programs (n=66) with the next largest group of respondents working in inpatient settings (n=27). The remaining respondents in the sample worked in addiction recovery services, such as case management, housing and employment support.

Table 1 summarizes the sample characteristics. The sample was primarily white (61%), female (65%), and comprised of direct service providers (83%). The average age of participants was 43 (range: 22–73; SD: 12.9). Respondents had worked in the addiction services profession for an average of eight years (SD = 8.03), though the median number of years in the profession was five. Staff had been at their jobs for an average of just over four years (SD = 4.32), though again, the median was lower at 2.5, since 35% of the sample had been at their job one year or less. A large proportion of all staff (41%) had Master’s degrees or higher in multiple fields of study, with degrees in Social Work, Psychology, and Counseling collectively making up just over one-half of the sample (56%). The remaining fields of study ranged widely from Divinity to Marketing.

Table 1.

Characteristics of Sample of Addiction Workers (N = 120)

Demographic Characteristic	N	%
Type of Program
Formal addiction treatment	71	59
Recovery support	49	41
Demographics
Female	73	61
White	78	65
Age (mean; range)	43; 22–73
Level of Education
High School	10	8
Associate’s	16	13
Bachelor’s	45	38
Master’s or higher	49	41
Field of Education (recoded)
Education, business, and public fields	9	8
Social work	22	18
Nursing, allied health, and other support	7	6
Psychology and Counseling	41	34
General	28	23
None (high school graduate)	10	8
Position in Organization
Primarily a provider of direct services	100	83
Primarily a supervisor	9	8
Primarily Management	9	8
	Mean (SD)	Range
Years in addictions services profession	8.02 (8.03)	0–34
Years in current position	4.04 (4.3)	0–27

Data Collection Procedure

Upon IRB approval from?, a member of the research team traveled to each agency and administered the survey, typically during staff meetings. The EBPAS and a companion measure that included a set of demographic questions were administered to participants in paper and pencil format. Data collection occurred in groups, with no agency administrator present. Each group was read instructions that assured subjects that their responses were anonymous and data would only be reported back to the organization in aggregated form. All subjects were volunteers, signed informed consent, and were provided no compensation. The research team counted the total number of possible front-line workers in agency and continued recruitment until reaching a response rate of 80%.

Measures and Variables

Attitudes toward evidence-based practice (EBP)

The Evidence-Based Practice Attitudes Scale (EBPAS: Aarons, 2004) was used to measure attitudes toward ESTs. The EBPAS consists of 15 Likert-scale items scored on a 0–4 response format: not at all, to a slight extent, to a moderate extent, to a great extent, and to a very great extent. Scale scores were the means of the test items, provided at least 75% of scale items had valid data. The number of cases with valid data for any one of the scales ranged between 112 (Divergence and Total) and 116 (Requirements and Appeal). Higher scores indicated a greater degree of the construct, which on the Divergence scale means a less positive attitude toward EBPs. Divergence items were reverse-scored for computing the total score. Prior studies (Aarons, 2004; Aarons, McDonald, Sheehan, & Walrath-Greene, 2007; Aarons, Glisson, et al., 2010) have reported the following coefficient alpha ranges: Appeal (.74–.80), Requirements (.90–.93), Openness (.78–.84), Divergence (.55–.66), and Total (.76–.79).

Demographic, educational, and professional characteristics

Data on age, race, gender, highest level of education, field of highest education, and years of experience in the addictions services profession were collected. In addition, we collected data on program name (each agency had multiple programs), primary work setting (outpatient, inpatient/detox, residential, and other/recovery support services), position (direct service, supervisory, or management), and years at current job. Some demographic variables (primary work setting, position, education, major, and ethnicity) were recoded to collapse small response categories after reviewing their frequency distributions.

RESULTS

Measurement Structure

Table 2 presents the EBPAS item means and standard deviations. Item distributions were also examined and these data revealed that all items except 3, 5, 6, and 7 (Divergence scale items) were negatively skewed (range: −0.06 to −1.52), indicating that responses were shifted towards the high end of the scale. Items 3, 5, 6, and 7 were positively skewed (range: 0.02 to 0.46). The items with the largest (absolute value) skewness values were 14, 13, and 15. Kurtosis values ranged between −1.08 (item 7) and 2.78 (item 15); however, there was no apparent pattern to the kurtosis values.

Table 2.

EBPAS item summary statistics and measurement structure results

Item	Mean	SD	Item-Total Correlation	Alpha	Factor Loading
A. Requirements	2.97	0.89		.92
11: EBP required by supervisor	2.85	0.94	.833		.86
12: EBP required by agency	2.94	0.97	.903		.98
13: EBP required by state	3.10	1.00	.790		.87
B. Appeal	3.05	0.62		.78
09: EBP was intuitively appealing	2.91	0.72	.574		.83
10: EBP made sense to you	3.10	0.74	.620		.90
14: EBP used by colleagues	2.93	0.84	.628		.49
15: Use EBP if had enough training	3.34	0.87	.517		.33
C. Openness	2.61	0.80		.86
01: Like to use new interventions	2.48	1.01	.642		.82
02: Willing to try manualized interventions	2.76	0.92	.785		.86
04: Willing to try research-developed interventions	2.68	0.89	.701		.75
08: Would try new therapy even if different	2.60	0.90	.688		.70
D. Divergence	1.47	0.75		.66
03: Know better than academic researchers	1.76	1.04	.253		.35
05: Research based treatments not useful	1.15	1.04	.602		.86
06: Clinical experience more important	1.97	1.10	.369		.44
07: Would not use manualized therapy	1.05	1.03	.588		.78
Total Scale	2.78	0.47		.79

Note. Nominal N=120, actual N varies due to missing data.

Table 3 presents the item correlations. Items are arranged to correspond to their order in Table 2. Since we sought to identify a set of common factors, we used a principal axis (aka, common factor) model. Communalities were estimated because item variances were expected to have both unique and common factor components. The first six eigenvalues were 4.539, 2.422, 2.023, 1.251, 0.940, and 0.791, which when plotted showed a slight break in slope at the fourth factor. We extracted three, four, and five factor solutions and applied a promax rotation to each since factors were expected to correlate. We judged the four factor solution to be more interpretable than the three or five factor solutions. Compared to the four factor solution, the three factor solution combined Aarons’ (2004) Openness and Appeal factors into a factor with items ranging from .40 to .88. The five factor solution split Aarons’ Appeal factor and placed items 9 (EBP was intuitively appealing) and 10 (EBP made sense to you) in one factor and items 14 (EBP used by colleagues) and 15 (Use EBP if had enough training) in the other factor. These factors were correlated at .49.

Table 3.

Correlations of EBPAS items arranged by items within scale

Item	11	12	13	09	10	14	15	01	02	04	08	03	05	06	07
11: EBP required by supervisor	1.00	.87	.72	.17	.30	.32	.27	.05	.15	.12	.13	.02	−.12	−.01	−.18
12: EBP required by agency	.87	1.00	.81	.14	.23	.33	.25	.09	.16	.15	.13	.06	−.04	−.02	−.11
13: EBP required by the state	.72	.81	1.00	.08	.18	.38	.34	.10	.17	.15	.05	.04	−.14	−.04	−.19
09: EBP was intuitively appealing	.17	.14	.08	1.00	.70	.38	.34	.20	.40	.37	.44	.13	−.10	−.18	−.11
10: EBP made sense to you	.30	.23	.18	.70	1.00	.50	.32	.25	.37	.30	.31	.08	−.08	−.01	−.14
14: EBP used by colleagues	.32	.33	.38	.38	.50	1.00	.60	.21	.40	.25	.35	.17	−.07	−.09	−.12
15: Use EBP if had enough training	.27	.25	.34	.34	.32	.60	1.00	.19	.41	.27	.33	.11	−.30	−.05	−.23
01: Like to use new interventions	.05	.09	.10	.20	.25	.21	.19	1.00	.62	.54	.53	.35	.09	−.05	.03
02: Willing to try manualized interventions	.15	.16	.17	.40	.37	.40	.41	.62	1.00	.68	.67	.24	−.07	−.15	−.09
04: Willing to try researched interventions	.12	.15	.15	.37	.30	.25	.27	.54	.68	1.00	.59	.13	−.16	−.14	−.13
08: Would try new therapy even if different	.13	.13	.05	.44	.31	.35	.33	.53	.67	.59	1.00	.12	−.06	−.21	−.13
03: Know better than academic researchers	.02	.06	.04	.13	.08	.17	.11	.35	.24	.13	.12	1.00	.29	.15	.17
05: Research based treatments not useful	−.12	−.04	−.14	−.10	−.08	−.07	−.30	.09	−.07	−.16	−.06	.29	1.00	.31	.68
06: Clinical experience more important	−.01	−.02	−.04	−.18	−.01	−.09	−.05	−.05	−.15	−.14	−.21	.15	.31	1.00	.39
07: Would not use manualized therapy	−.18	−.11	−.19	−.11	−.14	−.12	−.23	.03	−.09	−.13	−.13	.17	.68	.39	1.00

Note. Correlations greater than .185 are significant at the .05 level.

Nominal N=120, actual N varies due to missing data.

Table 2 presents the factor loadings for the preferred four factor solution. Except for the Requirements factor on which all three items have loadings greater than .85 and the Openness factor on which all four items have loading greater than .70, the item loadings for the Appeal and Divergence factors vary considerably, ranging from .33 to .90 for Appeal and from .35 to .86 for Divergence. The proximal cause for these variations can be seen in the correlations among the items that make up each factor. With respect to Appeal, Items 9 (EBP was intuitively appealing) and 10 (EBP made sense to you) form a pair, as do items 14 (EBP used by colleagues) and 15 (Use EBP if had enough training); resultantly, items in each pair correlate much more highly with each other than they do with items in the other pair. In the Divergence factor, there is only a single item pair, as items 5 (Research based treatments not useful) and 7 (Would not use manualized therapy) have a much larger correlation with each other than with the remaining two items (3: Know better than academic researchers and 6: Clinical experience more important). With respect to cross-loadings, half (51%) were .05 or less and 96% were .20 or less.

The factor correlations (Table 4) show that the Appeal and Openness factors have a moderately large correlation (.55) and that each has a smaller correlation with Requirements, particularly Openness. All three factors have small, negative correlations with Divergence, reflecting the reversed conceptualization of the Divergence items.

Table 4.

Factor Correlations

Factor	Requirements	Appeal	Openness	Divergence
Requirements	1.00	.35	.21	−.18
Appeal	.35	1.00	.55	−.24
Openness	.21	.55	1.00	−.12
Divergence	−.18	−.24	−.12	1.00

Scale statistics (means and standard deviations, item-total correlations and coefficient alpha reliabilities) are reported in Table 2. Item-total correlations were large and reasonably similar for items in a scale, except for the Divergence scale, which exhibited marked differences across the items in the scale, reflecting the correlations among the Divergence scale items. Scale reliabilities were acceptably large for all scales except Divergence (α = .66) and all were within or above the range of values from prior studies.

Substance Abuse Staff Compared to Previous Studies of Mental Health Staff

Comparisons between this sample of substance abuse treatment staff and mental health treatment staff were made by means of one sample t-tests against the weighted average of the three studies of mental health staff conducted by Aarons (Aarons, 2004; Aarons et al., 2007; Aarons et al., 2010). The results showed that this sample had significantly higher Requirements scores (2.97 versus a weighted average of 2.46; t[115] = 6.13, p <.001, 95% CI [0.34 0. 67], d = 0.57), higher Appeal scores (3.05 versus a weighted average of 2.92; t[116] = 2.31, p <.05, 95% CI [0.02 0.25], d = 0.21), and higher Divergence scores (1.47 versus a weighted average of 1.26; t[112] = 2.97, p <.01, 95% CI [0.07 0. 35], d = 0.28).

Worker Characteristics and EBPAS Scores

The associations between EBPAS scale scores and worker demographic and job structure variables were investigated for a number of variables that prior studies have found to be related to EBPAS scores. These variables were gender; age; ethnicity (white versus nonwhite); educational attainment (high school graduate/Associates, Bachelors, Masters or higher); educational major (social work-psychology-counseling versus all else); position (service provider versus supervisor-manager); program type (inpatient, outpatient, residential, other); years working in the field; and years at present job (Aarons et al., 2010, 2012). A total of five comparisons involving three independent variables were found to be significant. Firstly, education level was related to Divergence [F(2, 109) = 4.86, p = .010, η²= .082]. Workers with a high school or Associates degree had significantly higher Divergence scores (M = 1.90, SD = 0.64, n = 22) than did workers with either a Bachelors degree [M = 1.36, SD = 0.74, n = 42; p = .014, 95% CI = (0.09 0.99)] or a Masters or higher degree [M = 1.37, SD = 0.73, n = 48; p = .015, 95% CI = (0.09 0.97)]. Secondly, nonwhite respondents (n = 33, M = 1.92, SD = 0.64) reported higher Divergence scores than did white respondents (n = 74, M = 1.27, SD = 0.70; t[105] = 4.52, p < .001, CI [0.36 0.93]). Finally, years in the present job were negatively correlated with Appeal [r(111) = −.20, p = .035], with Openness [r(110) = −.24, p = .010], and with Total EBPAS score [r (107) = −.21, p = .027].

Discussion

The present study investigated the psychometric properties of the EBPAS among substance abuse treatment workers with respect to its factor structure, scale scores, and relationships with staff demographic and job-related variables. The results showed that the factor structure reported by Aarons (2004) was replicated here for this sample with respect to the number of factors and the items defining each factor. In addition, the observed coefficient alpha values had the same pattern as reported in Aarons and colleagues’ prior studies (Aarons, 2004; Aarons et al., 2007, Aarons et al., 2010), with the lowest values for the Divergence scale. The coefficient alpha values observed in the present study differed from the average of the values in Aarons et al.’s previous work by less than six percent. A careful comparison of the correlation matrix and factor loadings could anticipate several of the modifications that Aarons (Aarons et al., 2007 and Aarons et al., 2010) and Patterson et al. (in press) identified in subsequent analyses. One modification is a residual covariance between items 9 and 10 in the Appeal scale. It may be that modeling this effect, which cannot be done in factor analysis, accounted for the five factor solution, wherein the Appeal scale items divided into two factors. Confirmatory factor analysis, which can be used to test these hypothetical modifications, was not used in the analysis because of the small sample size.

Scale scores for this sample were compared to the weighted aggregate means from prior samples of mental health staff conducted by Aarons and colleagues. Results reveal an interesting juxtaposition of views. Compared to mental health agency staff in previous studies, the treatment staff in this sample rated themselves as more likely to use ESTs if required by their agency or state regulator or if the ESTs made “intuitive” sense to them and they had been trained in their use. At the same time, the addiction agency staff rated themselves as giving a lower value to research-derived interventions, as compared to mental health staff in prior samples. Regarding effect sizes, the differences were small (Appeal, Divergence) or medium (Requirements) in magnitude. These findings suggest that there are important differences between addiction and mental health workers, differences that should be attended to when developing EST implementation strategies in the addiction treatment workforce.

For instance, while the addiction workers in this sample did not appear to have more negative attitudes toward ESTs in general, results on the individual domains suggest that certain approaches, such as mandates, might be more effective in addiction workers than mental health providers. This may reflect close regulatory oversight that is generally characteristic of the addiction service field. Additionally, results suggest that efforts to communicate the appeal and intuitive sense of specific empirically-supported treatments may be particularly effective in improving attitudes toward ESTs in addiction workers. Given their tendency to rely more on their practical experience than empirical evidence in forming attitudes toward interventions, it may be necessary to model practical success by trialing new ESTs. This approach may be most effective if the trial implementation is mandated (e.g., on an agency level), supported with adequate training, and consistently promoted as an appealing and sensible intervention.

We found that several worker characteristics were associated with three of the four EBPAS subscales, primarily Divergence. Workers with Bachelors or higher degrees or who identified as white had lower Divergence scores, that is, a more favorable attitude towards ESTs. The relationship with ethnicity may be a consequence of education differences as non-white workers were more than twice as likely as white workers to have high school or Associates degrees. Similarly, white workers were twice as likely as nonwhite workers to have Masters degrees or higher. In addition, increased retention in the present position was associated with ESTs having less appeal, workers being less open to ESTs, and an overall less favorable attitude toward ESTs. A potential explanation for this pattern is that as addiction service providers accrue more work experience relative to their educational history, they tend to lean more heavily on their practice-based experiences, as compared to scientific evidence, in forming judgments and attitudes about new approaches. In view of the sizable proportion of workers (35%) who had been in their present position for a year or less, we checked for relationship linearity and, while we found no significant non-linearity, we did observe that Appeal, Openness and Total scores were numerically lower by nearly half of a standard deviation for workers with five or more years in their present position.

Limitations

Although this study achieved a response rate of over 80%, it is unclear whether or not workers who participated differed qualitatively from those who did not. Additionally, all participants were from the same geographical area, which might have resulted in more restricted variability in attitudes than if participants had come from a larger region. Further, the relatively small sample size may have reduced overall power to detect differences that did exist. Finally, it is possible that the multiple comparisons made elevated the likelihood of false positives. Applying the Benjamini & Hochberg (1995) correction to control Type I errors leaves the Ethnicity-Divergence relationship as the only qualifying finding. However, results from both Aarons (2004) and Aarons et al. (2010) indicate that such relationships are likely to be small. Aarons (2004) noted that a set of covariates similar to those used in this study accounted for 6 to 11 percent of the variance across the four EBPAS scales. Similar percentages also were noted by Aarons et al. (2010). Placing our findings with respect to prior work is difficult as Aarons (2004) did not report analyses involving ethnicity, and Aarons et al. (2010) entered worker characteristics in a multiple regression for each scale.

Conclusion

Although smaller in scale than some previous studies of its type, this research uniquely aimed to assess various types of attitudes toward ESTs among addiction workers. The current researchers found similarly reliable scores compared to other mental health researchers using the EBPAS tool. Our interesting findings with regard to the similarities in attitudes across the spectrum of addiction services will inform possible future studies and implementation interventions with these or similar agencies. Importantly, this study also reveals that, compared to mental health providers, addiction workers may be more likely to adopt ESTs if required to do so or if they found the EST to be appealing. Further, addiction workers may be comparatively less likely to buy into an EST based solely on empirical evidence. Consequently, addiction workers may respond differently than mental health providers to various engagement approaches.

While the results of this study are useful for investigators using the EBPAS in their research, the results could also have implications for addiction agencies interested in understanding the type of worker that might be more likely to implement ESTs. As the addiction services field continues to trend towards the implementation of ESTs in real world practice, it will become increasingly important to understand the attitudes of the current workforce. The EBPAS scale is a reliable tool that can offer important insights into attitudinal profiles of EST adopters among addiction workers and other social service providers alike. In line with findings of worker attitudes in mental health agencies (Patterson et al., 2012; 2013), certain attitudes within the addiction work force could be enriched in order to cross the chasm to successful implementation of empirically supported treatments.