How Intervention and Implementation Characteristics Relate to Community Therapists’ Attitudes toward Evidence-Based Practices: A Mixed Methods Study

Abstract

Therapists’ perceptions toward evidence-based practices (EBPs) are important in implementation efforts, however little is known about characteristics of EBPs associated with more positive attitudes. This mixed-methods study examined how intervention and implementation characteristics of six EBPs related to therapist attitudes. Quantitative analysis of 793 cross-sectional surveys revealed that therapists endorsed more positive attitudes toward EBPs with 1) prescribed session content and order and 2) required consultation. Associations between these intervention and implementation characteristics and attitudes were not moderated by therapist experience or emotional exhaustion. Qualitative analyses complemented quantitative findings, indicating that “structure” was appealing for interventions and that therapists felt supported by consultation.

Therapists’ attitudes are important to consider in the context of efforts to disseminate and implement evidence-based practices (EBPs) into publicly-funded mental health systems. For example, therapist attitudes relate to adoption of EBPs (Nelson & Steele, 2007), better engagement in training and consultation (Nelson, Shanley, Funderburk, & Bard, 2012; Pemberton et al., 2015), and adherence and skill in delivering EBPs (Beidas et al., 2014). Recent research has suggested that therapist perceptions differ widely based on the specific EBP (Cook, Thompson, & Schnurr, 2015; Reding, Chorpita, Lau, & Innes-Gomberg, 2014), but it is not clear which characteristics of EBPs therapists find to be appealing. As mental health systems are unlikely to implement a single EBP to meet the range of patient needs served (Chorpita & Daleiden, 2009; Chorpita, Bernstein, & Daleiden, 2011), it is important to plan for the adoption of multiple EBPs with features associated with more positive end-user responses (Reding et al., 2014). Past research has pointed to both intervention and implementation characteristics that may be related to attitudes toward EBPs.

Intervention Characteristics Associated with Therapist Attitudes toward EBPs

Early research on therapist attitudes found that therapists were concerned that the structure of manualized treatments interfered with the therapeutic relationship and the therapists’ ability to respond to their clients’ needs (Addis, Wade, & Hatgis, 1999; Addis & Krasnow, 2000). However, these studies were carried out with therapists who had minimal experience with manualized treatments, and studies have shown that training and implementation experiences are associated with improved attitudes toward EBPs (Edmunds et al., 2014; Lim et al., 2012). Yet in some studies, community therapists with training in EBPs have still cited concerns that the prescriptive nature of EBPs makes them too restrictive to meet the individualized needs of clients (Borntrager et al., 2009; Burgess et al., 2016; Jensen-Doss, Hawley, Lopez, & Osterberg, 2009). On the other hand, in qualitative interviews, therapists have identified benefits to using interventions with prescribed session content and order, indicating that manualized treatments made it easier to prepare for sessions and helped to keep therapists focused with their clients (Godley et al., 2001). This qualitative study was the only investigation of therapists with experience implementing multiple EBPs, but this study only interviewed 19 therapists delivering EBPs that targeted substance use, which may limit the generalizability of the study’s findings to therapists treating other clinical problem areas. Further investigation is needed to assess therapist perspectives on the structure provided by many EBPs after they have experienced implementation. For the purposes of this study, EBPs with prescribed session content and order are considered to be structured protocols.

Implementation Characteristics Associated with Therapist Attitudes toward EBPs

Beyond the intervention design, the characteristics of the implementation process (e.g., training) may be associated with therapist attitudes toward an EBP. Attitudes toward EBPs have been shown to improve following consultation, with more time in consultation being related to more positive attitudes at a two-year follow-up (Edmunds et al., 2014). Mounting evidence suggests that therapists need continuing consultation after an initial training to be able to deliver an EBP with competence (Beidas & Kendall, 2010; Herschell, Kolko, Baumann, & Davis, 2010), and it is possible that consultation provides therapists with opportunities to increase mastery and therefore have more positive experiences with EBPs. However, the observed relationship between consultation and EBP attitudes is not clear (Edmunds et al., 2014), as voluntary participation in consultation may be a marker of more positive initial attitudes toward EBP (Nelson et al., 2012; Pemberton et al., 2015). Furthermore, even though therapists have expressed the need for consultation to support EBP delivery (Nelson, Steele, & Mize, 2006), the format and time requirements of consultation can be seen as burdensome (Christian, Niec, Acevedo-Polakovich, & Kassab, 2014; Lyon et al., 2013), and it is not yet clear if requirements for consultation lead to improved attitudes toward EBPs.

Therapist Characteristics as Predictors and Moderators of Attitudes toward EBPs

Although limited research has focused on features of interventions that impact therapist attitudes, there is literature on how therapist background characteristics relate to their perceptions of EBPs, particularly with regards to levels of clinical experience. While some research suggests that individuals with less experience have more positive views of EBPs (Aarons, 2004), other studies have reported more positive attitudes among more seasoned therapists, as indicated by being licensed (Nakamura, Higa-McMillan, Okamura, & Shimabukuro, 2011) or by the number of years in practice (Reding et al., 2014). Furthermore, some studies have found no relationship between therapist experience and attitudes toward EBPs (Brookman-Frazee, Garland, Taylor, & Zoffness, 2009; Burgess et al., 2016).

It is unclear why the findings concerning the influence of therapist experience are mixed, although differences in measurement of therapist experience might contribute. Studies vary both in the indicators of therapist experience examined and how EBP attitudes are measured. Aarons (2004) found that trainee therapists had more positive attitudes toward EBP than more experienced therapists when using the Evidence-Based Practice Attitudes Scale (EBPAS), which includes attitudes toward using manualized treatments. Conversely, Nakamura and colleagues (2011) assessed attitudes toward EBP using a measure that deemphasized the use of manuals, and found that licensed therapists had more positive attitudes than unlicensed therapists. It is plausible that therapists’ responses to different features of EBPs depend on their levels of experience. Thus, the current study investigates whether therapist experience interacts with characteristics of EBPs to predict attitudes. For example, it is plausible that less experienced therapists may prefer highly structured EBPs that dictate activities and content to be delivered in each session, thereby reducing ambiguity and simplifying planning. In contrast, more experienced therapists (e.g., licensed therapists) may prefer more principle-based approaches that allow them to employ a range of techniques or activities in their repertoire to achieve individualized treatment goals in any given session (Aarons, 2004; Nakamura et al., 2011). Likewise, therapists may view EBPs that require ongoing consultation activities differently as a function of experience. Less experienced therapists may prefer interventions when they are well supported through consultation, whereas more senior therapists may be less receptive to EBPs with consultation requirements.

Given the high rates of emotional exhaustion among therapists in publicly-funded mental health settings (Morse, Salyers, Rollins, Monroe-DeVita, & Pfahler, 2012), this therapist characteristic may be impactful in large-scale implementation efforts. Emotional exhaustion, or feelings of being emotionally overextended and exhausted by one’s work, is a component of workplace burnout (Maslach & Jackson, 1981) linked to turnover intentions and poor job performance (Cropanzano, Rupp, & Byrne, 2003). Encouragingly, child welfare providers who received training and consultation in an EBP had lower rates of emotional exhaustion compared to providers who provided services as usual (Aarons, Fettes, Flores, & Sommerfield, 2009). Aarons and colleagues (2009) attributed these findings to the structure provided by the EBP protocol and the instrumental and emotional support received from consultants on the job. However, these hypotheses were not empirically tested. Furthermore, it is unclear how therapist emotional exhaustion might impact perceptions of EBPs that do not include a structured protocol or the support of consultation. It is plausible that therapists suffering from workplace exhaustion under high workloads would be more receptive to structured EBP protocols that limit preparation time. It is unclear whether therapists experiencing emotional exhaustion would be more or less receptive to implementation requirements for consultation, which may be time-consuming on the one hand, yet supportive on the other. Thus, the current study examined the interaction of EBP structure (i.e., prescribed order and content), consultation, therapist emotional exhaustion, and attitudes toward EBPs.

Current Study

Using a mixed-methods approach, this study investigates the interaction of intervention and implementation characteristics of six EBPs and therapist attitudes within a large-scale implementation effort in Los Angeles County Department of Mental Health (LACDMH), and if therapist experience and emotional exhaustion moderated these relationships. We categorized EBPs as: 1) having prescribed session content and order and 2) requiring consultation. The overarching question of this study was: How are intervention and implementation characteristics of EBPs associated with therapists’ attitudes? For the primary aim of the study, we analyzed quantitative data from a large survey distributed to therapists throughout Los Angeles County to examine whether EBP intervention and implementation characteristics related to therapist attitudes toward six specific EBPs. We conducted qualitative analyses of therapist interviews to provide depth of understanding to the quantitative findings. In an exploratory aim, we examined whether therapist characteristics moderated the associations between EBP characteristics and therapist attitudes.

Method

Study Context

In 2009, LACDMH underwent a large system-driven reform, the Prevention and Early Intervention (PEI) Transformation, which fiscally mandated the use of select evidence-based and evidence-informed practices. Further details on the PEI Transformation are provided elsewhere (Lau & Brookman-Frazee, 2016). The current study investigates the six initial EBPs for which LACDMH provided implementation support via training and consultation: Cognitive Behavioral Intervention for Trauma in Schools (CBITS), Child-Parent Psychotherapy (CPP), Managing and Adapting Practice (MAP), Seeking Safety, Trauma Focused Cognitive Behavior Therapy (TF-CBT) and Positive Parenting Program (Triple P). MAP is technically not considered an EBP, but rather a system to identify, select and track the use of components of EBPs (Chorpita, Daleidan, & Collins, 2014). However, for brevity, all six interventions will be referred to as EBPs in this study. Agencies were funded to train therapists in the EBPs, and agency leaders selected EBPs to implement based on the perceived needs and preferences of the children and families served in their agencies. Between the years of 2010 and 2014, 8,514 therapists provided one of these six EBPs to 87,100 clients, demonstrating the scope of this reform (Brookman-Frazee et al., 2016).

The six EBPs vary in their client needs served, their clinical approach, and their training and implementation requirements for consultation (see Table 1). First, the EBPs differ on the dimension of structure. CBITS, TF-CBT, and Triple P all have a prescribed session content and a suggested order for therapists to follow. The CBITS treatment manual includes outlines of the content and ordering of ten group sessions and one individual session (Jaycox, 2003). Similarly, Triple P, which can be delivered in a group or individual format, has eight recommended sessions with a specified ordering (Turner, Markie-Dadds, & Sanders, 2002). TF-CBT is recommended to take place in 12 to 16 sessions. Although the treatment manual describes that it is a components-based treatment that can be delivered in a flexible manner, there is both suggested content and a default ordering of sessions, with the manual explaining that the components are “based on a logical sequence of skill building,” and that the efficacy of the model is based on delivering the intervention in this order with all of the content included (Cohen, Mannarino, & Deblinger, 2006, p. 57). On the other hand, CPP, MAP, and Seeking Safety allow for more flexibility in the selection and ordering of therapy activities across an episode of treatment. Seeking Safety has 25 topics that can be covered in any order and can include as few or as many topics as the therapist deems helpful (Najavits, 2002). MAP is not a single treatment model, but instead a suite of therapist support tools used to select, coordinate, guide and adapt the use of EBPs for children across a range of clinical problems (Chorpita et al., 2014). CPP focuses on improving attachment and the parent-child relationship within the context of parent-child play interactions and individual caregiver sessions, and does not dictate the session activities, content or order (Lieberman & Van Horn, 2005).

Table 1.

Description of EBPs and Implementation Characteristics

EBP	Age Range	Problem Target	Format	Session Content and Order	Implementation Characteristics*
CBITS	10–15	Trauma	Group, school based, youth sessions	Session outlines provided for 11 sessions (e.g., relaxation, combating negative thoughts) with a specified order.	2-day initial training, 13 consultation calls
CPP	0–6	Trauma, Attachment	Caregiver or caregiver-youth sessions	Guidance for 12 clinical domains (e.g., play, child fearful behavior; ghosts in the nursery) that can be used across sessions with no specified order.	2.5-day initial training, 2-day booster training, 18 months of consultation
MAP	0–21	Anxiety, Trauma, Depression, Conduct	Youth, caregiver, or caregiver-youth sessions	A suite of decision support tools to identify, select, and monitor evidence-based practice. 33 practice modules (e.g., activity selection, exposure) that can be used in any order and combination.	5-day initial training, 6 months of consultation, portfolio submission
Seeking Safety	13–20	Trauma, Substance Use	Group or individual youth sessions	25 topics (e.g, recovery thinking, coping with triggers) that can be addressed in any order and combination.	1-day initial training
TF-CBT	3–18	Trauma	Individual youth, caregiver, and caregiver-youth sessions	10 treatment components (e.g., cognitive coping, trauma narrative) with a suggested order.	2-day initial training, 16 consultation calls, booster training, 2 audio tape reviews by certified trainer
Triple P	0–18	Conduct	Group or individual sessions with caregivers	Level 4 provides 8 session outlines (e.g., managing misbehaviors) with a specified order.	3-day initial training

Note:

^*= Los Angeles County Department of Mental Health (LACDMH) specified the requirements that appear in Implementation Characteristics column

Second, the training requirements established by LACDMH differed for the six EBPs, particularly on the requirement for ongoing consultation during implementation. In consultation with the treatment developers, LACDMH established requirements about the level of training that was required of therapists (LACDMH, 2010a–e). Seeking Safety and Triple P did not require ongoing consultation, whereas CBITS, CPP, MAP, and TF-CBT did. Of the EBPs that required consultation, the length and possible formats varied. CBITS required 13 consultation calls with external trainers. CPP required 18 months of consultation calls with external trainers for therapists and supervisors in the agency. After the initial training, MAP required six months of ongoing consultation, which could be provided by a certified external or internal trainer. As part of consultation, MAP therapists also submitted case-based portfolios for certification. TF-CBT required 16 consultation calls with external trainers along with audiotape review of two sessions.

Recruitment and Procedures

Ninety-eight agencies that were directly operated or contracted by LACDMH to deliver at least one of the six EBPs of interest to children or transition-age youth were eligible for inclusion in the 4KEEPS Therapist Survey. Eligible therapists included any therapist who billed for core psychotherapy services to at least one of the six EBPs of interest to children or transition-age youth. Because there is no centralized directory of therapists providing these EBPs in LACDMH directly-operated and contracted agencies, we identified eligible therapists for the survey through management at individual agencies. Of the 98 agencies, managers at 69 agencies (70.4%) provided therapist contact information or the option for therapists to sign up directly for the survey. These 69 agencies represented contracted and directly operated agencies, and were located in a range of geographic regions within Los Angeles County.

Between March 2015 and July 2015, approximately six years after the PEI Transformation was initiated, invitations to participate were sent via email to 1796 community-based mental health therapists. A total of 793 therapists completed the survey, for a response rate of 44.2 %. This rate of participation is within the range of previous surveys of community mental health therapists, which have had response rates that range from 25–51% (e.g., Hawley, Cook, and Jensen-Doss, 2009; Cashel, 2002; Piotrowski & Keller, 1989; Rosenberg & Beck, 1986). Participants received a $20 incentive for completing the survey and an additional $20 incentive if they completed the survey within two weeks.

Additionally, agencies volunteered to be part of “in-depth” data collection, which provided greater depth of understanding to the survey data from interviews with agency leaders and therapists, along with session audio recordings of therapists delivering one of the six EBPs under investigation. The current study included interviews from two selected agencies with seven program sites (i.e., clinics) that represented urban, suburban, and rural settings in Los Angeles County. Therapists were recruited for participation in the in-depth portion of the study at staff meetings. Therapists were eligible to participate if they were willing to participate in an interview with study staff and record sessions for three clients receiving one of the six EBPs. Therapist interviews were used in the qualitative portion of the current study. Therapists received a $40 incentive for completing the interview. Institutional Review Boards at LACDMH and the University of California, Los Angeles approved all procedures for this study.

Participants

Of the 793 therapists who completed the 4KEEPS Therapist Survey, the majority were female (88%), and the sample was racially/ethnically diverse (e.g., 42.9% Hispanic/Latino, 10.9% Asian/Pacific Islander, 7.1% African American). Professionally, therapists were predominately Master’s-Level therapists (86.4%), and the majority were unlicensed (55.4%). The most common mental health discipline was Marriage and Family Therapy (56.5%), followed by Social Work (29.3%) and Psychology (12.6%). Therapists reported average caseloads of 14.72 clients (SD = 10.41) and provided direct clinical services an average of 17.68 hours (SD = 8.60) a week. Of the six EBPs of interest, therapists, on average, had been trained in 2.39 (SD = 1.03) EBPs, had ever used 2.31 (SD = 1.07) EBPs, and currently used (at the time of survey completion) 1.56 EBPs (SD = .99).

Qualitative interviews were conducted with a subset of 25 therapists from two agencies. Therapists who were interviewed were representative of the larger survey sample, with no significant differences in the percentage of female (83.3%), racially/ethnically diverse (42.9% Hispanic/Latino, 12.5% African American, 8.3% Asian/Pacific Islander), licensed (32.0%), or Master’s-level (96.0%) therapists (see Table 3).

Table 3.

Characteristics of Therapists in the Survey-Only and Survey + Interview Samples

	Survey (n = 768)	Survey + Interview (n = 25)	χ2	p
Gender	–	–	.34	.56
Female	87.9%	84.0%
Male	12.1%	16.0%
Licensed	44.9%	32.0%	1.63	.20
Race/Ethnicity	–	–	4.48	.11
Non-Hispanic White	35.7%	16.0%
Hispanic/Latino	42.3%	60.0%
Other	22.0%	24.0%
Level of Education	–	–	3.65	.16
B.A. & under	2.0%	4.0%
Master’s	86.1%	96.0%
Doctorate	11.8%	0%

Measures

Therapist characteristics

Therapists completed the Therapist Background Questionnaire (Brookman-Frazee et al., 2012) concerning personal and professional characteristics. Demographic variables included: age, gender, and race/ethnicity. Professional background variables included: licensure status, mental health discipline, and highest degree obtained (i.e., Associates, Bachelor’s, Master’s, Doctorate). To index therapist professional experience in multivariate analyses, we used the variable of licensure status, as this variable has been used in past research on EBP attitudes and is an objective measure (Nakamura et al. 2011). Workload variables included: number of hours in direct service per week, and the number of clients in caseload. These two variables were converted to z-scores and combined to make a composite measuring Workload used in multivariate analyses. Therapists indicated which of the six EBPs they had 1) received training in, 2) ever delivered, and 3) delivered in the previous two months.

Emotional exhaustion

Therapist emotional exhaustion was assessed using five of the original six items on the Emotional Exhaustion subscale of the Organizational Social Context Questionnaire (Glisson et al. 2008; Glisson, Green, & Williams, 2012). The scale has demonstrated strong psychometric properties, particularly among healthcare professionals (Glisson et al. 2008; Glisson et al., 2012). Participants were asked to rate their agreement with each item on a 7-point Likert scale from (0 = strongly disagree, 6 = strongly agree). The total scale had acceptable internal consistency in the current study, Cronbach’s alpha = .89.

Perceptions toward EBPs

The Perceived Characteristics of Intervention Scale (PCIS; Cook et al., 2015) was developed to measure the theory-based model of 10 perceived characteristics of innovation (Rogers, 1962; Rogers, 2003, Greenhalgh et al., 2004), which may impact likelihood of therapist uptake of different EBPs. The original measure included 10, two-item scales to capture each of these characteristics, although psychometric properties indicated that the scale measured a unidimensional construct of attitudes (Cook et al., 2015). We examined therapist perceptions toward any EBP in which they had received training, regardless of whether they had ever or were currently delivering the EBP. To reduce measurement fatigue given the repeated nature of this measure, we administered 8 items tapping four dimensions included in the original PCIS, including Relative Advantage (e.g., “[The EBP] is more effective than other therapies I have used.”), Compatibility (e.g., “[The EBP] is aligned with my clinical judgement.”), Complexity (e.g., “[The EBP] is easy to use.”), and Potential for Reinvention (e.g., “[The EBP] can be adapted to meet the needs of my patients.”). Therapists rated each item on a 5-point Likert scale (1 = not at all, 5 = a very great extent). We used a composite mean score to measure therapists’ perceptions of each EBP. The total 8-item scale had excellent internal consistency for all EBPs, with Cronbach’s alphas ranging from .92 to .96.

General attitudes toward EBPs

The Evidence-Based Practice Attitudes Scale (EBPAS; Aarons, 2004) is a 15-item self-report measure used to assess therapist attitudes toward the adoption of EBPs. The original EBPAS yields a total scale score and four subscales: Appeal, Requirements, Openness, and Divergence. The current study included two complete subscales from the EBPAS: Openness and Divergence, each of which consisted of four items. The Openness scale assesses the therapist’s openness to trying new interventions and willingness to use EBPs, and includes items such as “I like to use new types of therapy/interventions to help my clients.” The Divergence scale assesses the therapist’s perception of EBPs as not clinically useful and less important than clinical experience, and includes items such as “Clinical experience is more important than using manualized therapy/interventions.” Therapists rated each item on a 5-point Likert scale (0 = not at all, 4 = very great extent). In the current sample, the Cronbach’ s alpha indicated that the internal consistency was acceptable for the Openness scale (α = .79) and for the Divergence scale (α = .71).

Intervention and implementation characteristics of EBPs

Based on the LACDMH PEI Implementation Handbook (LACDMH, 2010a-e) and the six EBP manuals, the EBPs were characterized on the following two dimensions: 1) the intervention characteristic of prescribed session content and order and 2) the implementation characteristic of required consultation. Five child clinical psychologists, with expertise in EBPs, participated in a group consensus process to determine how to characterize each EBP under investigation. EBPs that had prescribed session content and ordering were defined as those that provided explicit guidance in their treatment manuals as to what content should be covered (e.g., didactics scripts, specified skills training, discussion guides, activities) and had a suggested order for when specified treatment content should be delivered. EBPs that required consultation were defined as those that had mandatory follow-up consultation with an EBP expert after the initial didactic training as stated in the LACDMH training guidelines. Consultation took various forms, including phone calls, audiotape review, or in-person consultation with certified trainers. The characteristics for the six EBPs are specified in Table 2.

Table 2.

Planned Comparisons of Intervention and Implementation Characteristics

Evidence-Based Practice	Prescribed Session Content and Order	Consultation Required
CBITS (n = 65)	✓	✓
CPP (n = 140)		✓
MAP (n = 527)		✓
Seeking Safety (n = 491)
TF-CBT (n = 582)	✓	✓
Triple P (n = 184)	✓

Semi-structured interview guides

As part of a larger interview related to perceptions of the EBPs, therapists were asked a series of questions related to their attitudes toward the individual EBPs. They were asked questions about their attitudes toward EBPs that they had ever used, regardless of whether they were currently delivering the EBP. The interview followed a funnel approach with broader questions asked first followed by more specific follow-up probes (Spradley, 1979). The following topics were covered:

1. EBP appeal: “What do you like most about [EBP]?” Follow-up probes included, “Anything about [EBP] itself?” and “Anything about the training/implementation requirements?” Therapists were given examples of implementation requirements (e.g., “ongoing consultation”) if they needed clarification with this prompt.

2. EBP challenges: “What did you find most challenging about using [EBP]?” with the same probes related to the EBP and implementation supports.

Data Analytic Plan

Mixed-methods design

This study used a QUAN ➔ qual approach with sequential collection and analyses, beginning with the quantitative data (Palinkas et al., 2011). The quantitative data were to examine specific hypotheses concerning EBP characteristics associated with therapist attitudes. Given that we examined a finite set of EBPs that differ in many ways but were classified along two specific dimensions, the findings from the quantitative analyses, although indicative of the constructs of interest, are subject to other interpretations. As such, the qualitative data provide an important means of triangulation to understand whether the quantitative findings could be confirmed or elaborated by the therapist narrative.

Quantitative data analysis

Therapist EBP-specific attitude scores on the PCIS were predicted using planned comparisons of the intervention (i.e., prescribed session content and order) and implementation (i.e., consultation required) characteristics of interest. Two separate models were conducted for the intervention and implementation characteristic, with different combinations of the six EBPs in each planned comparison (Table 2). We specified multi-level models with random intercepts using SPSS v. 20 MIXED commands to account for the non-independence of (Level 1) EBPs nested within (Level 2) therapists who were nested within (Level 3) agencies. To determine the levels of nesting that must be accounted for, we followed conventions to account for nesting at an ICC ≥ .05 (Hayes, 2006). An unconditional model was estimated including only the intercept to compute the intraclass correlation (ICC) and variance at the therapist and agency levels to determine nesting structure. Based on these models, therapist-level nesting was included in the model (ICC = .18), but agency-level nesting (ICC = .02) was not.

The Level 1 outcome being predicted was the EBP-specific attitude rating from the PCIS. Level 1 predictors included whether a therapist had ever used the EBP (0=no, 1=yes), whether the therapist was currently using the EBP (0=no, 1=yes), and the EBP characteristic under examination (either session content and order or ongoing consultation coded as 0=no, 1=yes). Level 2 predictors were individual therapist characteristics, including licensure status, gender, race/ethnicity (Latino/Hispanic, Non-Hispanic White, Other Minority), level of education (B.A. and below, Master’s, Doctorate), workload, emotional exhaustion, and general attitudes toward EBPs. Exploratory cross-level moderation analyses were conducted to determine if the relationships between intervention and implementation characteristics were moderated by therapist characteristics of licensure status and emotional exhaustion.

Qualitative data analysis

Qualitative transcripts were entered into ATLAS.ti version 7.5.11. Six members of the research team conducted a rapid qualitative assessment to identify broadly emerging themes (Hamilton, 2013). First, the team developed a template to summarize interview questions under corresponding domain names, then they summarized notes from the interviews and transferred these summaries into a matrix, which was reviewed for crosscutting themes (Miles & Huberman, 1994). These themes informed a codebook, along with a process of open coding completed by three members of the team (M. B., J. R., D. S.). Once codes were finalized, these members proceeded to code all of the interview transcripts and resolve any discrepancies through consensus. Co-occurring codes were analyzed and discussed with the research team to finalize themes.

Integration of quantitative and qualitative findings

The primary functions of the mixed-methods analyses were: 1) convergence – both sources reached the same conclusion (i.e., triangulation), 2) complementarity – surveys provided a breadth of information and the interviews provided depth of understanding, 3) expansion – qualitative analyses explained unanticipated quantitative findings (Palinkas et al., 2011).

Results

Aim 1: Do Intervention and Implementation Characteristics Relate to Therapist Attitudes?

Given that separate models were conducted for intervention and implementation characteristics, there were slight differences in the predictive values of therapist characteristics that were entered as covariates. Each predictor for each model can be found in the summary of multilevel model statistics provided in Table 4. Therapist licensure, emotional exhaustion, workload, race/ethnicity, and Openness to EBP were significantly related to EBP-specific attitudes in both models. Therapists who were licensed reported less positive EBP-specific attitudes. Therapists with higher perceived emotional exhaustion and higher workloads also reported less positive EBP-specific attitudes across the two models. Non-Hispanic White therapists reported less positive EBP-specific attitudes in comparison to Latino/Hispanic and other racial/ethnic minority therapists. Therapists’ Openness toward EBPs was associated with more positive EBP-specific attitudes. Therapists’ gender, educational level, and perceived Divergence of EBPs were not significantly related to EBP-specific attitudes.

Table 4.

Predictors of EBP-Specific Attitudes

	Model 1: Prescribed Session Content and Order		Model 2: Consultation Required
	F	Estimate	F	Estimate
Intercept	306.56***	3.02***	291.97***	2.81***
EBP-Level Predictors:
Planned Comparisons
Prescribed Session Content and Order	96.77***	.37***	–	–
Consultation Required	–	–	4.68*	.09*
History of Use (Initial Training)
Ever Used	14.03***	.26***	17.55***	.30***
Current Use	202.00***	.60***	152.47***	.55***
Therapist-Level Predictors:
Gender (Female)	.01	.01	.03	.01
Licensed+	9.12**	−.15**	7.39**	−.14**
Race/Ethnicity (Non-Hispanic White)	3.45*	–	3.94*	–
Latino/Hispanic	–	.13*	–	.15*
Other Minority	–	.12	–	.13*
Education (Master’s degree)	.47	–	.63	–
B.A. & under	–	.11	–	.18
Doctorate	–	.06	–	.05
Emotional Exhaustion+	26.06***	−.08***	24.83***	−.08***
Workload (direct service hours & caseload)	11.11**	−.05***	11.13**	−.05**
General EBP Attitudes
EBPAS Openness	95.49***	.35***	97.22***	.35***
EBPAS Divergence	2.87	−.06	3.03	−.06

Notes. + = tested as a moderator,

^*= p < .05,

^**= p < .01,

^***= p < .001.

EBP-level covariates entered into each model included whether therapists had ever used the EBP and whether they currently used the EBP. Therapists had more positive EBP-specific attitudes when they had ever used the EBP compared to only having been trained in the EBP, and similarly had more positive attitudes when they currently used the EBP compared to only having used or been trained in the EBP in the past.

Prescribed session content and order

Therapists endorsed more positive attitudes for EBPs with prescribed session content and order (EMM = 3.47, SE = .08) than EBPs without this characteristic (EMM = 3.11, SE = .08), = .37, p < .001. Qualitative analyses converged with this finding, as therapists frequently identified the “structure” of interventions as being appealing (e.g., “I like the sequence and things. It facilitates a process, and it reduced the anxiety about how to go about it. ”). In contrast, therapists made relatively few comments about how structure can be challenging (e.g., “You know how it has to be very structured in order to be following the fidelity of it… [this] has been challenging”). Therapist responses also expanded on why they found structure to be appealing, explaining that it helped with treatment planning (e.g., “It’s kind of nice the way the structure is, there’s a little bit less treatment planning because, you know, okay, this week we ’re doing the feelings, affective modulation, or whatever.”) and kept the therapist and clients focused in treatment (e.g., “So I like the structure in that regards [sic] and it keeps me on track and it just feels a little bit more – I think the clients seem to like kind of being able to see the steps and know where they ’re going and know that there’s the end.”).

Interestingly, therapists identified structure as being a feature they liked across various EBPs including those with modular designs without prescribed session content and order (e.g., “Well, I love that [EBP 1]¹ gives you that structure where you kind of know what you could be doing with the kids. So I think it kind of helps to keep me organized because I can be very process-oriented, and so sometimes it takes away from me going towards solutions.”). Flexibility, which may at times be perceived as the opposite of structure, was also discussed by therapists, but less frequently than structure. Therapists identified flexibility as being appealing (e.g., “I like that there’s flexibility as far as which modules to use. So it’s not like I have to go in order.”), although there were multiple comments indicating that therapists would have desired more structure (e.g., “I like and hate at the same time the looseness of it, because I’ve noticed when I work with [EBP 2] the structure keeps me on track a little bit better.”). Therapists only discussed flexibility as being appealing for a single EBP, as opposed to having it be a theme that cut across multiple EBPs.

Consultation required

Therapists rated their attitudes toward EBPs that required consultation more positively (EMM = 3.29, SE = .08) than EBPs that did not require consultation (EMM = 3.20, SE = .08), Estimate = .09, SE = .04, p < .05. Qualitative themes complemented this finding, such that therapists expressed overwhelmingly positive attitudes toward consultation when they discussed it, and their answers provided a depth of understanding about what they found appealing (e.g., “I really like the consultations, and when I enjoyed a lot of it was just being able to present case examples… different people had to present at different times. I think we did the audios. That was nerve-racking, but it was good nerve-racking, because you got feedback as to how you applied the skills, how you could improve your skills….”). Therapists reported that they enjoyed learning from the trainers that provided consultation (e.g., “Whenever the trainer does the role plays or pretends that she’s the therapist doing the EBP, it’s been helpful.”) and having the opportunity to share experiences with other therapists in the consultation group (e.g., “I think it was good to hear challenges that other people experienced to know that I wasn’t the only one feeling challenged.”). Even though therapists described liking consultation overall, some therapists found certain aspects of consultation to be challenging, suggesting that not all formats were enjoyed by all therapists (e.g., “I mean, the consultation, you know, a lot of our consultations were done via telephone. I think if they would have done it to smaller groups and did it via Skype, that we could see each other and bounce ideas off, it would have helped.”). Therapists reported that the limited availability of consultation made the implementation of EBPs challenging (e.g., “…so we were trained and then there was nothing for two years. So that was challenging.”).

Exploratory Aim: Do Therapist Characteristics Moderate the Relationship between Intervention and Implementation Characteristics and Attitudes?

Being licensed did not moderate the relationship between prescribed session structure and order and attitudes (F(1, 1502.03) = .22, p = .643) or required consultation and attitudes (F(1, 1580.59) = 3.38, p = .066). In the second set of models, we tested whether therapist-reported emotional exhaustion moderated the relationship between 1) prescribed session structure and order and attitudes and 2) requiring ongoing consultation and attitudes. No significant interactions were found between emotional exhaustion and prescribed session content and order (F(1, 1531.76) = .69, p = .406) or between emotional exhaustion and requiring consultation (F(1, 1594.99) = .06, p = .807) in associations with attitudes.

The consistency in therapist narratives about the value of consultation and the predominantly positive appraisals of the structured aspects of the EBPs supported the quantitative findings that therapist attitudes did not vary as a function of characteristics such as licensure and emotional exhaustion. In regards to intervention design, therapists most frequently mentioned structure as an appealing characteristic, with considerably fewer comments about the flexibility of an EBP or challenges that related to structure. When therapists discussed flexibility, they frequently also discussed structured elements of EBPs, with an equal number of therapists from the qualitative sample being grouped into those who preferred structure, those who preferred flexibility, and those who found both features to be positive. Regarding implementation requirements, therapists’ comments on consultation were overwhelmingly positive. The main concern was the scarcity of consultation which they identified as a challenge for their delivery of the EBP.

Discussion

The current study took place within the unique context of a system-driven reform of children’s mental health services in Los Angeles County involving implementation of multiple EBPs, which launched in 2009–2010. This context provided the unique opportunity to investigate therapist perceptions of multiple EBPs after they had been implemented within a system of care for over five years. We used a mixed-methods approach to address gaps in the literature regarding our understanding of how intervention, implementation, and therapist characteristics relate to attitudes. The large survey sample provided a breadth of understanding about how a diverse sample of therapists perceived different EBPs, while the interviews complemented, converged, and expanded on quantitative findings.

Contrary to past concerns that community therapists perceive the structure of EBPs as too rigid and inflexible (Addis et al., 1999; Nelson et al., 2006), quantitative and qualitative findings from our study indicated that therapists considered prescriptive structure appealing. In a planned comparison, therapists rated their attitudes toward EBPs with prescribed session content and order more positively than EBPs without this characteristic. Qualitative interviews converged with and expanded on this finding, with therapists reporting that the structure of manualized treatments helped them stay focused in therapy and feel efficiently prepared for sessions. These themes were consistent with those of Godley et al. (2001), who also sampled from therapists with lived experience implementing different EBPs. Past research studies noting negative therapist attitudes toward structured or manualized interventions had been conducted primarily with therapists with limited exposure to EBPs or early in implementation (Addis et al., 1999; Borntrager et al., 2009; Jensen-Doss et al., 2009). Our findings provide insight into how attitudes toward structured EBPs may differ over the maturation of dissemination and implementation efforts. Additionally, these findings might indicate how attitudes are impacted by having experience delivering EBPs with varying level of structure, as opposed to comparing EBPs to practice as usual. The importance of the depth of exposure to EBPs for influencing attitudes is suggested by the finding that therapists had more positive attitudes toward EBPs that they had ever used or currently use as opposed to those in which they had only received training. However, it is also plausible that therapists elect to actually deliver and continue to deliver those EBPs that they find most appealing.

Past studies have suggested that therapists prefer modular EBPs that offer flexibility in delivering session content, as opposed to standard manualized treatments (Borntrager et al., 2009; Chorpita at el., 2015). In qualitative analyses for this study, although flexibility emerged as an appealing characteristic for one modular EBP, this characteristic was discussed by therapists less frequently and for fewer EBPs than structure. Interestingly, therapists also discussed how they liked the structure provided by modular EBPs, indicating that these EBPs provide more structure in regards to treatment planning than usual care services, which is appealing for therapists. Whereas past research has clearly pointed to a preference for flexibility (Borntrager et al., 2009; Nelson et al., 2006), our findings suggest that the balance of both structure and flexibility might contribute to favorable therapist attitudes, especially when therapists are routinely implementing multiple EBPs. Further research is needed to elucidate how to maximize the appeal of structure and flexibility. For example, therapists may find the structure provided for individual session content (e.g., session outlines) appealing, while desiring flexibility in how they deliver care across sessions.

This study provided additional support to the value of consultation in the implementation of EBPs in community settings, as therapists reported more positive attitudes toward EBPs that required consultation in quantitative findings. Qualitative themes complemented these findings, with therapists identifying consultation as an appealing part of the implementation experience, whereas they found it challenging to implement EBPs without consultation. Similar to past research, which found that therapists’ views of consultation varied based on the format (Christian et al., 2014; Funderburk, Ware, Altshuler, Chaffin, 2008), therapists in our sample expressed that they did not like all formats of consultation. This study and past research suggest that therapists may prefer in-person consultation to phone-based consultation (Beidas et al., 2014; Christian et al., 2014). Consultation provides an opportunity for case conceptualization, skill building, problem-solving implementation barriers, and support in making appropriate adaptations to treatments (Nadeem, Gleacher, & Beidas, 2014), all of which therapists in our study identified as being helpful, with the majority of comments related to didactic and discussion-based activities. Relatedly, therapists in past studies have expressed a preference for discussion and guidance in case conceptualization over activities that promote skill development, such as role plays and live or recorded observation and feedback (Beidas et al., 2013; Funderburk et al., 2008). However, skill-building activities involving behavioral rehearsal have been identified as critical for successful implementation (Bearman et al., 2013; Beidas & Kendall, 2010). Our findings clearly support the importance of consultation, although further research is warranted to better understand the best formats to improve both behavioral and attitudinal outcomes.

Finally, this study provides additional insights into therapist characteristics that are related to perceptions of specific EBPs. Past research has raised the concern that Latino/Hispanic therapists have less favorable attitudes toward EBPs than Non-Hispanic White therapists (Aarons, Cafri, Lugo, & Sawitzky, 2012), but our study found that ethnic minority therapists had more positive attitudes toward EBPs. This is an encouraging finding for implementation efforts within diverse community settings such as Los Angeles County. Although it has been suggested that the structure and support provided in EBP implementation can lower emotional exhaustion (Aarons et al., 2009), our findings did not support this. Similar to a past study that took place within the LACDMH’s large-system reform (Reding et al., 2014), higher levels of therapist emotional exhaustion were associated with poorer attitudes toward EBPs, as was having a larger workload. Emotional exhaustion did not moderate the relationship between therapist attitudes and EBP structure or required consultation. The influence of clinical experience on EBP attitudes has been unclear from research to date, with some indication that the effect depends on whether therapists perceive EBPs to be more or less structured (Nakamura et al., 2011). Our study found that unlicensed therapists had more positive perceptions of EBPs, with no interaction with implementation or intervention characteristics in relation to therapist attitudes. The consistency in qualitative reports of the value of EBP structure and consultation also suggested little variation in valuation of these EBP characteristics. Thus, it did not appear from either quantitative or qualitative analyses that attitudes toward EBP structure or consultation varied based on therapist experience or reported emotional exhaustion.

Limitations

This study had several limitations, which need to be considered when interpreting findings. First, the intervention and implementation characteristics analyzed only capture some of the differences in the six EBPs under investigation, and it is possible that other, unidentified characteristics contributed to the differences in attitudes found in the planned comparisons. The qualitative findings regarding the appeal of structure and consultation did corroborate these interpretations, but further investigation of additional EBPs may help confirm if these characteristics are widely appealing in implementation efforts. Second, although our findings supported that therapists prefer EBPs with required consultation, we do not know the extent of their participation in ongoing consultation activities. Additionally, although this study was one of very few that examined attitudes after multiple EBPs had been implemented within a system for multiple years, it is cross-sectional and, therefore, we are not able to say if attitudes of therapists changed over time. The finding that current use was associated with more positive attitudes suggests that perceptions toward specific EBPs may improve with experience. However, a past study found that therapists’ attitudes decreased to pre-implementation levels after two years of experience using the EBP (Edmunds et al., 2014); therefore, additional longitudinal studies are needed to understand how continued use impacts therapist attitudes. Finally, although with a reasonable response rate, we recruited a large number of therapists for this survey, it is not possible to determine if participating therapists are representative of other therapists within LACDMH or other systems of care. It is unknown whether likelihood of participation might have varied systematically with views on the implementation and intervention characteristics under study. Even with these limitations, this study makes important contributions to our understanding of characteristics that might make EBPs more appealing to therapists in implementation efforts.

Table 5.

Qualitative Themes and Illustrative Quotes about Intervention & Implementation Characteristics Associated with Therapist Attitudes

	Illustrative Quotes
Intervention Design
The structure of EBPs is appealing	I think I kind of like the structure of it. It’s kind of nice the way the structure is, there’s a little bit less treatment planning because, you know, okay, this week we’re doing the feelings, effective modulation, or whatever. Okay, now we’re going to start the trauma narrative and you know that you’re going to be doing that for a couple weeks, and then that takes some of the pressure off.
Structural elements of EBPs can be challenging	One thing, it does not have very much leeway at all, even the very first session, it’s not about rapport building, it’s right off the bat psycho-education, which kind of cuts against my grain, because I’m one of those that pretty much believes that people tell you their true self when they feel comfortable and trust you and whatever, and that’s more than just explaining what this modality and the steps and all that.
Flexibility of a modular treatment is appealing	Probably the flexibility that I can just use more of my clinical judgment and just, like, okay, this is what this kid needs right now and I can…yes, it’s a little less regimented.
Implementation Support
Having consultation is appealing	I thought it was kind of cool that they did the phone consultations after the training. Because they kind of…you know, as we were using the model, we were able to ask questions during, so I thought that was really neat. Because then you got to ask like the person who created it.
Format of consultation can be challenging	I actually don’t really prefer the consultation calls. I mean it’s like okay, but I feel like I learn more when I see the person and I see them pretending to conduct a session.
Limited consultation is challenging	But with [EBP 3], it’s just kind of like we got trained, go through the model, and like, you know, so what’s next. That’s the big thing for me. What’s next? I would really like to see them have a training on how you can fit it with youth.