Therapist Report of Adaptations to Delivery of Evidence-Based Practices within a System-Driven Reform of Publicly-Funded Children’s Mental Health Services

Abstract

Objective

This study examined clinical adaptations reported by community therapists to multiple evidence-based practices (EBPs) currently implemented in children’s mental health services. Based on an item set informed by Stirman and colleagues’ model (2015), two factors emerged describing Augmenting adaptations and Reducing/Reordering adaptations. We employed multilevel modeling to examine therapist- and practice-level predictors of therapist reports of each type of adaptation.

Method

Data were drawn from an online survey, including a novel therapist report measure of EBP adaptations, completed by 572 therapists (89.2% female, M_age = 37.08 years, 33.4% Non-Hispanic White) delivering EBPs in the context of a system-driven, fiscally mandated implementation effort.

Results

Analyses revealed that the two types of therapist adaptations (Augmenting and Reducing/Reordering) could be readily discriminated, with therapists reporting significantly more Augmenting than Reducing/Reordering adaptations. Therapists of Hispanic/Latino ethnicity and with fewer years of experience reported more extensive Augmenting adaptations, but no therapist background characteristics were associated with Reducing/Reordering adaptations. Therapists’ general attitudes that EBPs diverged from their personal approach to therapy were associated with reporting more Augmenting and Reducing/Reordering adaptations. In contrast, negative perceptions toward the specific EBP predicted Reducing/Reordering adaptations, but not Augmenting adaptations.

Conclusions

Community therapist reports suggest that most adaptations undertaken involve engaging with the practice to augment the fit of the EBPs for local contexts; however, when practices were perceived negatively, therapists were more likely to make adaptations reducing or rearranging components.

As evidence-based practices (EBPs) spread across diverse community practice settings, it is essential to understand how they are modified by end users (Chambers & Norton, 2016). Indeed, Rogers’ (1962, 2003) seminal Diffusion of Innovation theory posits that innovations are inevitably adapted, with the extent and types of modifications depending on characteristics of both the innovation and the adopter. Implementatation outcomes may be shaped by adaptations of EBPs that serve to align the practice with the local needs of providers (Chambers & Norton, 2016). Innovations with good potential for reinvention may be more appealing to adopters, ultimately resulting in greater uptake and penetration (Greenhalgh, Robert, Macfarlane, Bate, & Kyriakidou, 2004). Furthermore, providers who invest their local expertise and energy into adapting an EBP may feel more ownership over it, which could promote sustained use (Racine, 2006). Yet, is also important to understand the nature of common adaptations, as some may represent greater engagement with the innovation or enhanced outcomes, while others may reflect the opposite leading to “slippage” in EBP delivery by community providers (Stirman, Calloway, et al., 2013). Chambers and Norton (2016) have advocated the development of an “adaptome” data platform to identify and aggregrate adaptations to EBPs and their impacts on patient and implementation outcomes. The study of how organizations and communities are integrating EBPs into routine care provides critical opportunities to learn from local implementation efforts and mandates.

Thus, the current study uses a brief measure developed to systematically identify and examine the adaptations that therapists report making to multiple EBPs implemented in community mental health settings within a large-scale system reform of children’s mental health services. Here, we define adaptation as the degree to which therapists modify the delivery, structure or content of the EBP to address population-specific needs (Aarons et al., 2012; Lau, 2006). It is widely recognized that EBPs, typically developed in research settings, may require adaptation to fit local needs when delivered to clients in community settings (Chambers & Norton, 2016; Southam-Gerow, Ringeisen, & Sherrill, 2006). Indeed, community therapists commonly raise the concern that EBPs may not fit the clinical or cultural needs of the diverse clients they serve (Aarons et al., 2010). For example, therapists may feel that adaptations are warranted in settings in which EBPs are delivered to ethnic minority families that have been inadequately represented in controlled trials (Aisenberg, 2008; Bernal & Scharró-del-Río, 2001; Cabassa & Baumann, 2013). Concerns about EBPs were found to be higher among Hispanic/Latino therapists compared to non-Hispanic White therapists in a national sample of community clinicians (Aarons et al., 2010).

Adaptations based on clinical expertise and judgment are encouraged in the Institute of Medicine (2001) and American Psychological Association (2006) definitions of evidence-based practice (e.g., “integration of the best available research with clinical expertise in the context of patient characteristics, culture, and preferences”). Based on concerns about fit for clients from diverse backgrounds, therapists may adapt EBPs in a variety of ways such as, reframing the way interventions are presented, or incorporating techniques to promote acceptance of the EBP (Lau, 2006). Most research on adaptations to EBPs has been conducted through community-partnered efforts, in which researchers design systematic adaptations to EBP protocols based on community stakeholder feedback (e.g., Bigfoot & Schmidt, 2010; Domenech Rodríguez, Baumann, & Schwartz, 2011; Lau et al., 2012). These efforts have largely taken an augmentation approach to cultural adaptations, with a focus on increasing client engagement or including content to address culturally salient risk and protective processes (Chu & Leino, 2017; Lau, 2006). Common adaptations include using relevant idioms or metaphors to frame interventions, adding treatment components to attend to immigrant or minority status stressors that contextualize presenting problems, addressing challenges with client engagment in treatment, and increasing opportunities for rehearsal and consolidation of culturally unfamiliar skills (Bigfoot & Schmidt, 2010; Domenech Rodríguez et al., 2011; Lau et al., 2012; Resnicow, Baranowski, Ahluwalia, & Braithwaite, 1998; Szapocznik & Williams, 2000). Though considerable effort has been invested in adapting and testing EBPs for diverse groups, these adapted protocols are rarely disseminated (Cabassa & Baumann, 2013). It is unknown whether community therapists serving ethnically diverse clients make adaptations in line with those that have been tested in research. Active engagement in individualizing EBPs for clients may ultimately increase therapist investment in and commitment to delivering the EBP (Forehand, Dorsey, Jones, Long, & McMahon, 2010; Mazzuchelli & Sanders, 2010).

In contrast, other types of adaptations may represent disengagement from elements of EBPs, in content, structure, or process. Although community therapists increasingly report using EBPs in their practice, the delivery of complete protocols is uncommon while the omission of selected components is common (Walrath, Sheehan, Holden, Hernandez, & Blau, 2006). For example, in the area of EBP content, even among therapists with expertise in cognitive behavioral therapy with trauma-exposed populations, reported use of imaginal exposure is infrequent (Becker, Zayfert, & Anderson, 2004) as many therapists cite concerns that the practice is unacceptable or risky for clients in vulnerable states (Foa, Gillihan, & Bryant, 2013). In terms of EBP structure and process, community implementation of parent training interventions for child conduct problems is often characterized by lower use of process components, such as role play rehearsal of skills (e.g., Stern, Alaggia, Watson & Morton, 2007). Therapist reasons for omission of behavioral practice included time constraints, perceived irrelevance of some skill components, and such omissions were also associated with therapist inexperience (Asgary-Eden & Lee, 2011; Taylor, Asgary-Eden, Lee & LaRoche, 2015).

Consistent with the notion that therapist adaptations may range from those that augment versus reduce EBP process and content, Stirman and colleagues (2015) developed a typology of modifications to EBPs described in the literature. In their conceptualization, they considered adaptations to be consistent with fidelity of EBP delivery when they could be made without altering the core elements of the intervention. For the cognitive behavioral therapies for depression or anxiety that clinicians in their study delivered, these “fidelity-consistent” adaptations included tailoring presentation, adding new material or strategies that were consistent with the EBP, and shortening or lengthening the sessions or treatment episode without removing core elements. In contrast, they considered modifications that resulted in removing components or loosening the structure, as protocol drifts that potentially threaten fidelity. A mixed-methods study of community therapists who delivered cognitive behavioral therapy found that therapists with greater openness towards EBPs in general were more likely to make “fidelity-consistent” adaptations. Interestingly, therapists who reported finding EBPs appealing were nonetheless more likely to report “fidelity-inconsistent” modifications (Stirman et al., 2015).

While the reasoning of the Stirman typology is sound, their conceptualization of fidelity-consistent and -inconsistent was based on a single form of psychotherapy. It is important to recognize that EBPs range in degree of prescriptive structure, meaning that reducing or reordering treatment elements may not be incompatible with integrity to some protocols. Certainly, for some manualized EBPs, specific content and process components are laid out session-by-session, thus adaptations that result in deviations in session order or subtractions might represent threats to delivery “as intended.” However, other types of EBPs that do not standardize these structural or content features permit therapist flexibility in planning activities based on client needs and preferences. In modular treatments, decision-making about inclusion of components is shared with end-users, often guided by a formal algorithm (Borntrager, Chorpita, Higa-McMillan & Weisz, 2009; Chorpita & Daleiden, 2009). Similarly, other EBPs emphasize a ‘toolbox’ approach, where therapists and clients can select from an array of skills that may be applied to address the particulars of a client’s presenting problem (e.g., Najavits, 2002). Still other EBPs provide guidance to direct therapeutic activities that align with the intervention’s theory of change, without specific differentiated content outlined for each session (e.g., Lieberman & Van Horn, 2011). For these types of EBPs, rearranging or omitting content or process elements may be consistent with high fidelity delivery, and conceptualizations of fidelity-consistent and -inconsistent adaptations would vary depending on the protocol.

Thus, in the current study we examined therapist adaptations to multiple EBPs that range in prescriptiveness and specificity in structure. Extending the Stirman et al. (2015) typology, we anticipated two types of adaptations that may either be formally allowed for within the protocol, or may represent “off-label” applications, depending on the EBP. First, therapists may engage with EBP protocols to augment the intervention to meet the needs of their clients by: (1) introducing elements from other interventions, (2) tailoring the presentation of treatment elements, or (3) slowing the pace of the intervention, which may involve delivering additional sessions to cover the intervention content. Second, given concerns about EBP components, therapists may refrain from engaging with selected elements or procesess in ways that reduce in number, or re-order, the intervention components delivered. This would include adaptations that: (1) omit content areas, (2) speed the pace of the intervention, which likely involves reducing the time spent on treatment elements, or (3) re-arranging session order.

An important step in improving community implementation outcomes is understanding the factors associated with different forms of adaptations. The observation that therapist adaptations are associated with general attitudes towards EBPs is informative, and suggests that some therapists are more inclined to make adaptations. However, it is also apparent that therapists have distinctive perceptions of different EBPs. In contexts where therapists are trained in multiple EBPs, therapist opinions concerning the appeal, utility and limitations of specific interventions vary (Barnett et al., in press; Reding et al., 2014). Therefore, it is plausible that the likelihood that a therapist makes a modification to an EBP that augments or reduces or reorders the content is determined by their perceptions toward that specific practice, rather than their attitudes toward EBPs in general.

Within a large-scale reform of children’s mental health services in Los Angeles County, the largest county mental health system in the United States, we examined therapists’ reports of adaptations to practices on a brief online survey instrument. We addressed the following three research objectives. In objective 1, we examined the properties of a novel therapist report measure of adaptations to EBPs and the extent to which the data confirmed a two-factor structure of Augmenting and Reducing/Reordering Adaptations informed by the Stirman et al. (2015) framework. Then, in objective 2, we described the frequency of therapist reports of Augmenting and Reducing/Reordering adaptations in a large diverse sample of community therapists implementing multiple evidence-based and evidence-informed practices. Finally, in objective 3, we examined how therapist-level and practice-level characteristics predicted therapist reports of both types of adaptations.

Method

Context of the Current Study

The opportunity for this observational study of the implementation of multiple practices in children’s mental health emerged following a series of state-level developments in California. The Mental Health Services Act (MHSA) was enacted from a ballot measure initiative passed in November 2004 that assessed a 1% tax on the income of individuals who earn in excess of $1 million per year. The funding generated by this tax was earmarked for specific types of mental health services. This revenue stream resulted in changes in workforce training, systems of care for clients across the age spectrum and an increased focus on delivery of effective practices with measurable outcomes statewide.

The Los Angeles County Department of Mental Health (LACDMH) is the nation’s largest county mental health department, serving, on average, more than 250,000 clients of all ages every year (LACDMH, 2015). LACDMH elected to utilize MHSA Prevention and Early Intervention (PEI) funding to serve as a catalyst to implement Evidence-Based, Promising and Community Defined Practices to intervene earlier in the course of mental illness. In the context of an overall state budget shortfall in 2010 that limited other funding streams, LACDMH offered agencies reimbursement through PEI for the delivery of practices approved in the county’s PEI Plan beginning July 2010. Consequently, the contracts for agencies that served children and transition-age youth were amended to include PEI funds and implementation guidelines. LACDMH initially furnished provider training and implementation support for an initial set of six evidence-based/informed practices (hereafter referred to as practices) to address a range of prevalent youth mental health problems, including Cognitive Behavioral Interventions for Trauma in Schools (CBITS), Child-Parent Psychotherapy (CPP), Managing and Adapting Practice (MAP), Seeking Safety, Trauma Focused Cognitive Behavior Therapy (TF-CBT), and Triple P - Positive Parenting Program (Triple P). Trainings commenced in May 2010, and in fiscal year 2010–2011, over 32,000 children and transition-age youth were served in PEI programs (LACDMH, 2011).

Recruitment and Procedures

Data for the current study were collected as part of the broader Knowledge Exchange on Evidence-based Practice Sustainment (4KEEPS) study (Lau & Brookman-Frazee, 2015) examining sustainment of the initial six PEI practices and determinants of sustainment. The sampling frame for the 4KEEPS Therapist Survey included 98 agencies that were directly operated or contracted by LACDMH to deliver at least one of the six practices of interest to children or transition-age youth in Fiscal Year 2013–2014. Eligible therapists included any therapist who billed for core psychotherapy services to at least one of the six practices of interest to children or transition-age youth. Because there is no centralized directory of therapists providing PEI services in LACDMH directly operated and contracted agencies, we enumerated therapists into the survey through management at individual agencies. Of the 98 agencies in the sampling frame, 69 agencies (70.4%) were recruited into the study.

A total of 777 therapists completed an online survey between March 2015 and July 2015. Most therapists were recruited through a direct email campaign, for which agency management provided contact information for eligible therapists employed at agencies directly operated or contracted by LACDMH. Direct email invitations were sent with personalized survey links to 1656 community-based mental health therapists in 63 agencies. A total of 688 of these therapists completed the survey, for a response rate of 41.5% for the direct email campaign. Management in an additional six agencies elected not to release staff contact information, but instead agreed to forward an email to therapists that would allow them to provide their contact information to the research team to opt-in to the survey; 89 community therapists participated in the survey through this opt-in process. The response rate for the direct recruitment portion of the sample is consistent with past surveys of mental health therapists, with response rates ranging from 25–51% (e.g., Hawley, Cook, and Jensen-Doss, 2009; Cashel, 2002; Piotrowski & Keller, 1989; Rosenberg & Beck, 1986). Participants received a $20 incentive for completing the survey and an additional $20 incentive if they completed the survey within two weeks. Institutional Review Boards at University of California, Los Angeles and LACDMH approved all procedures for this study.

The current study investigated therapist reports of adaptations in the delivery of five practices: CBITS, CPP, Seeking Safety, TF-CBT, and Triple P. MAP was not included in this study, as it is a system for supporting EBP application and not itself a manualized treatment program. MAP is designed to be a decision-guidance framework, intended to help therapists manage and adapt a range of evidence-informed practices for specific cases (Southam-Gerow et. al, 2014). Therefore, therapists who provided responses only concerning adaptations to MAP were excluded in the current study (n = 106).

Participants

Thus, out of the 777 participants who completed the 4KEEPS Therapist survey, 572 (73.6%) qualified for the current study because they reported that they had delivered one of the five practices of interest (CBITS, CPP, Seeking Safety, TF-CBT, Triple P) within the past two months (99 additional therapists were excluded because they were not currently delivering these practices). Participants were community-based mental health therapists employed in 63 agencies directly operated or contracted by LACDMH (see Table 1 for therapist characteristics). Therapists were predominately female (89.2%) and were 37.11 years (SD = 9.31) on average. Therapists were racially/ ethnically diverse (44.6% Hispanic/Latino, 33.4% non-Hispanic White, 9.1% Asian/Pacific Islander, 6.6% African American, 2.1% Multiracial, 4.2% Other). The majority of therapists reported that they were able to deliver services in another language, with 49.0% reporting that they could deliver services in Spanish, 7.7% reporting another language (e.g., Mandarin Chinese, Farsi, Armenian, and Korean), and 1.1% Spanish and another language.

Table 1.

Participant Characteristics

Variable	Mean (SD) or %
Age	37.11 (9.31)
Gender
Male	10.8%
Female	89.2%
Race/Ethnicity
Non-Hispanic White	33.4%
Hispanic/ Latino	44.6%
Asian/ Pacific Islander	9.1%
African American	6.6%
American Indian or Alaska Native	0%
Multiracial	2.1%
Other	4.2%
Deliver Services in > 1 Language
Spanish	49.0%
Other Language	7.7%
Spanish and Other Language	1.1%
Clinical Experience (years)	6.59 (5.92)
Years at Current Agency	4.22 (4.32)
Current Caseload	15.97 (9.50)
Caseload Ethnicity/Race Composition
Hispanic/Latino	72.3% (27.7%)
Non-Hispanic White	10.4% (16.5%)
Asian/ Pacific Islander	1.9% (7.7%)
African American	13.2% (18.9%)
American Indian or Alaska Native	0.4% (4.9%)
Other	1.9% (8.1%)
Licensure Status
Licensed	43.3%
Unlicensed	56.7%
Education Level
Bachelor’s Degree or less	2.6%
Master’s Degree	85.7%
Doctoral Degree	11.7%
Mental Health Discipline
Marriage and Family Therapy	55.8%
Social Work	29.4%
Psychology	13.1%
Other	1.7%

Note. N = 572

In terms of professional training, therapists were predominately Master’s-level therapists (85.7%) and unlicensed (56.7%). The most common mental health discipline was Marriage and Family Therapy (55.8%), followed by Social Work (29.4%), and Psychology (13.1%). On average, therapists had been practicing for 6.59 years (SD = 5.92) and had been employed at their current agency for 4.22 years (SD = 4.32). Therapists reported an average caseload size of 15.97 clients (SD = 9.50) with predominately Hispanic/Latino clients (M = 72.3%, SD = 27.7%). Of the five practices of interest, on average, therapists in this sample had been trained in 1.93 practices (SD = .78), had ever used 1.86 practices (SD = .81), and currently used 1.42 practices (SD = .61).

To identify if the therapists who opted-in to participation differed from therapists directly contacted by e-mail, chi square and t-tests were calculated. There were no significant differences by recruitment method for age, gender, race/ethnicity, bilingualism, theoretical orientation, years in practice, years at the agency, caseload, theoretical orientation, or EBPAS Openness or Divergence Scores. The only differences between the groups reveal a generally higher level of education for the opt-in therapists compared to directly contacted therapists (licensed = χ(1)² = 11.26, p = .001, education level, χ(2)² = 33.57, p < .001).

Measures

Therapist characteristics

Questions about therapists characteristics were derived from a measure used previously in an effectiveness and implementation trial of a treatment for children with autism spectrum disorder (Brookman-Frazee, Drahota, & Stadnick, 2012). Therapists completed questions about their personal and professional characteristics, including age, gender, race/ethnicity, licensure status, primary theoretical orientation, highest degree obtained, caseload, years of experience as a therapist, and years at their current agency. Therapists also reported in which of the five practices they had 1) received training, 2) ever delivered, and 3) delivered in the previous two-months.

Evidence-Based Practice Attitude Scale (EBPAS)

The EBPAS (Aarons, 2004) is a 15-item, self-report measure used to assess therapist attitudes towards adoption of EBPs. The original EBPAS yields a total scale score and four subscales: Appeal, Requirements, Openness, and Divergence. The current study included two complete subscales from the EBPAS: Openness and Divergence, each of which consisted of four items. The Openness scale assesses the therapist’s openness to trying new interventions and willingness to use EBPs, and includes items such as “I like to use new types of therapy/interventions to help my clients.” The Divergence scale assesses the therapist’s perception of EBPs as not clinically useful and less important than clinical experience, and includes items such as “Clinical experience is more important than using manualized therapy/interventions.” Therapists rated each item on a 5-point Likert scale (0 = not at all, 4 = very great extent). In the current sample, the Cronbach’s alpha indicated that the internal consistency was acceptable for the Openness scale (α =.79) and for the Divergence scale (α =.71), which is comparable or better than in the EBPAS development study (Openness: α = .78, Divergence: α = .59; Aarons et al., 2004).

Perceived Characteristics of Intervention Scale (PCIS)

The PCIS (Cook, Thompson, & Schnurr, 2015) was developed to allow for the comparison of therapist perceptions of different EBPs. In the original PCIS, 10, 2-item scales were developed to measure the theory-based model of 10 perceived characteristics of innovation (Rogers, 1962; Rogers, 2003, Greenhalgh et al., 2004), which may impact likelihood of therapist uptake of different EBPs. Therapists completed PCIS items for any PEI practice for which they had reported ever receiving training or ever delivering. To reduce measurement fatigue given the repeated nature of this measure, we administered four complete scales of the original PCIS, including Relative Advantage (e.g., “[The practice] is more effective than other therapies I have used.”), Compatibility (e.g., “[The practice] is aligned with my clinical judgement.”), Complexity (e.g., “[The practice] is easy to use.”), and Potential for Reinvention (e.g., “[The practice] can be adapted to meet the needs of my patients.”). Therapists rated each item on a 5-point Likert scale (1 = not at all, 5 = a very great extent). In the current study, the total 8-item scale had excellent internal consistency for all practices with Cronbach’s alphas ranging from .92 to .96, so a composite mean score was used to measure therapists’ perceptions of each practice.

Adaptations to Evidence-Based Practices Scale (AES)

Six items about specific adaptations that therapists make to practices were developed for the AES questionnaire based on the Stirman (2013) coding system for an EBP modifications typology derived from the literature describing modifications to interventions across a variety of settings. This framework was applied to code community therapist qualitative descriptions of modifications and adaptations to EBPs in child and adult mental health services (Stirman et al., 2015). The six items related to: 1) modifying the presentation, 2) shortening or condensing the pacing of the practice, 3) lengthening or extending the pacing of the practice, 4) integrating supplemental content or strategies, 5) removing or skipping components, and 6) adjusting the order of sessions or components. Therapists completed the AES for every practice that they endorsed using within the past two months. Using a 5-point Likert scale (0 = not at all, 4 = a very great extent), therapists indicated the extent to which they used the adaptation when delivering a specified practice.

Results

Objective 1: Examine the Properties of the AES

To determine the factor structure of the AES in our sample for subsequent analyses, multilevel confirmatory factor analysis (MCFA) was employed using Mplus statistical software (Muthén & Muthén, 2012). Level 1 was the practice-level and included the AES responses about up to five practices used by the therapist in the past two months. Level 2 was the therapist-level. The MCFA model was specified with two factors at each level, with three items on one factor representing Augmenting adaptations (i.e., modifying the presentation, integrating supplemental content or strategies, and lengthening or extending the pacing of the practice) and three items on a second factor representing Reducing/Reordering adaptations (i.e., removing or skipping components, adjusting the order of sessions or components, and shortening or condensing the pacing of the practice). Both factors were scaled by restricting the first factor loading to 1.0 and each item was fixed to load on only one factor. Statistically significant factor loadings were used as the criterion for item inclusion. Maximum likelihood with first order derivative standard errors was used as the estimation procedure. Model fit was evaluated using the Root Mean Square Error of Approximation (RMSEA) and the Comparative Fit Index (CFI); RMSEA of less than or equal to 0.08 and CFI of greater than .95 indicated acceptable model fit (Hu & Bentler, 1999). Observed model fit was acceptable as shown by the RMSEA value of .064 and the CFI value of .954. Factor loadings across levels were statistically significant at p <.001 (see Table 2). Multilevel composite (omega) reliability was calculated for the Augmenting and Reducing/Reordering factors using recommendations from Geldholf and colleagues (2014). Reliability was excellent for both factors at Level 1 and Level 2 (ω = .95–.98; see Table 2).

Table 2.

Unstandardized Parameter Estimates and Composite Reliability for Multilevel Confirmatory Factor Analysis

AES Items	Estimated Marginal Mean	Level 1 (Practice)	Level 2 (Therapist)	Level 1 ω	Level 2 ω
Augmenting (Factor 1)				.95	.98
I modify how I present or discuss components of…	2.24	a1.00 (--)	a1.00 (--)
I integrate supplemental content or strategies when I deliver…	2.21	.68 (0.09)	1.10 (.12)
I lengthen/ extend the pacing of…	2.03	.25 (.07)	.74 (.10)
Reducing/Reordering (Factor 2)				.95	.97
I remove/skip components of…	.92	a1.00 (--)	a1.00 (--)
I adjust the order of…	1.43	.77 (.11)	.93 (.12)
I shorten/condense pacing of…	.66	.27 (.05)	.77 (.11)

Note. All loadings were significant at p <.001. Standard errors are in parantheses.

^aFactor loading fixed to 1.0.

Objective 2: Examine the Extensiveness of Augmenting and Reducing/Reordering Adaptations

The average extensiveness rating for each type of adaptation are presented in Figure 1. Upon visual inspection, it appeared that therapists rated greater extensiveness of adaptations on the Augmenting factor than those on the Reducing/Reordering factor.

Figure 1.

Therapist Report of Extensiveness of Different Types of Adaptations

To examine factors associated with the extensiveness of reported adaptations, a multilevel model with random intercepts using SPSS v. 20, MIXED model was used to account for the non-independence of ratings of adaptations. For these analyses, a three-level model was fit, which had repeated measures of therapist reported adaptations with scores on Augmenting and Reducing/Reordering scales (level 1) nested within practice (level 2) nested within therapist (level 3). Nesting at the agency level was not included in this model, as clustering was minimal (ICC = .01), and standard conventions were followed where nesting is only accounted for when ICC ≥ .05 (Hayes, 2006). In this model, predictors of adaptation type are denoted with π, those at the practice level are denoted with β, and the therapist level is denoted with γ, as shown in the equation below:

$$\text{Level}1(\text{Adaptation}\text{Type}):\mathit{AdaptationScore}={\pi }_0+{\pi }_1(\mathit{AdaptationType})+e$$

$$\begin{array}{ll}\text{Level}2(\text{Practice}):\hfill & {\pi }_0={\beta }_{00}+r_0\hfill \\ \hfill & {\pi }_1={\beta }_{10}+r_1\hfill \end{array}$$

$$\begin{array}{ll}\text{Level}3(\text{Therapist}):\hfill & {\beta }_{00}={\gamma }_{000}+u_{00}\hfill \\ \hfill & {\beta }_{10}={\gamma }_{100}+u_{10}\hfill \end{array}$$

An omnibus test of fixed effects indicated that therapists made significantly more Augmenting adaptations (EMM = 9.32, SE = .18) than Reducing/Reordering adaptations (EMM = 5.90, SE =.18), F(1, 1,060.55) = 1306.29, p < .001.

Objective 3: Identify Practice- and Therapist-Level Predictors of Adaptations

Two separate models were run to assess practice- and therapist-level predictors of therapist reports of Augmenting and Reducing/Reordering adaptations. Missing data analyses were conducted, which revealed that only two variables of interest had missing datapoints, and for those variables only 1% were missing. The SPSS v. 20 MIXED function treated these data as missing at random. Three-level models with practices nested within therapists nested within agencies were conducted to predict the extensiveness of each type of adaptation.

In each model, practice was included as a predictor to test if the type of adaptation (Augmenting or Reducing/Reordering) varied significantly by EBP. At level 1, perceptions of specific practices were included to assess whether therapist perceptions of a specific practice were associated with reported adaptations to that same practice. Level 2 predictors in the model included therapist characteristics, including, years in practice, gender, race/ethnicity (Non-Hispanic White, Hispanic/Latino, Other Ethnic Minority), caseload, ability to deliver care in a language other than English, theoretical orientation (CBT or other), and level of education. Also at level 2, the therapist’s general attitudes towards EBP (EBPAS Openness and Divergence scales) were included in the models to examine whether attitudes were associated with reports of adaptations. As the models for the Augmenting and Reducing/ Reordering adaptation included the same predictors, only the equation for Augmenting adaptations is given as an example:

$$\begin{array}{l}\text{Level}1(\text{Practice}):\mathit{Augmenting}\mathit{Adaptation}={\pi }_0+{\pi }_1(\mathit{Experience})+{\pi }_2(\mathit{Gender})+\\ {\pi }_3(\mathit{Ethnicity})+{\pi }_4(\mathit{Caseload})+{\pi }_5(\mathit{Bilingual})+{\pi }_6(\mathit{Education})+\\ {\pi }_9(\mathit{Theoretical}\mathit{Orientation})+{\pi }_8(\mathit{EBPASDivergence})+{\pi }_9(\mathit{EBPASOpeness})+e\end{array}$$

$$\begin{array}{ll}\text{Level}2(\text{Therapist}):\hfill & {\pi }_0={\beta }_{00}+{\beta }_{01}(\mathit{PCIS})+r_0\hfill \\ \hfill & {\pi }_1={\beta }_{10}\hfill \\ \hfill & {\pi }_2={\beta }_{20}+r_1\hfill \\ \hfill & {\pi }_3={\beta }_{30}+r_1\hfill \\ \hfill & {\pi }_4={\beta }_{40}+r_1\hfill \\ \hfill & {\pi }_5={\beta }_{50}+r_1\hfill \\ \hfill & {\pi }_6={\beta }_{60}+r_1\hfill \\ \hfill & {\pi }_7={\beta }_{70}+r_1\hfill \\ \hfill & {\pi }_8={\beta }_{80}+r_1\hfill \\ \hfill & {\pi }_9={\beta }_{90}+r_1\hfill \end{array}$$

$$\begin{array}{ll}\text{Level}3(\text{Agency}):\hfill & {\beta }_{00}={\gamma }_{000}+u_{00}\hfill \\ \hfill & {\beta }_{01}={\gamma }_{010}\hfill \\ \hfill & {\beta }_{10}={\gamma }_{100}\hfill \\ \hfill & {\beta }_{20}={\gamma }_{200}\hfill \\ \hfill & {\beta }_{30}={\gamma }_{300}\hfill \\ \hfill & {\beta }_{40}={\gamma }_{400}\hfill \\ \hfill & {\beta }_{50}={\gamma }_{500}\hfill \\ \hfill & {\beta }_{60}={\gamma }_{600}\hfill \\ \hfill & {\beta }_{70}={\gamma }_{700}\hfill \\ \hfill & {\beta }_{80}={\gamma }_{800}\hfill \\ \hfill & {\beta }_{90}={\gamma }_{900}\hfill \end{array}$$

Practice-level predictors

Controlling for all other predictors, omnibus tests of fixed effects revealed that practice type was not significantly associated with Augmenting adaptations, F(4, 441.34) = .92, p = .453, but practice type did have a significant effect on their ratings of Reducing/Reordering adaptations, F(4, 460.25) = 36.99, p < .001 (see Figure 1). Specifically, therapists reported making more Reducing/Reordering adaptations to Seeking Safety than the reference group (Triple P), Estimate = 1.80, SE = .22, p < .001. No other practice had significant differences in therapist reported adaptations from the reference group.

Perceptions of practices did not relate to Augmenting adaptations, F(1, 712.49) = 2.12, p = .146, but did significantly relate to Reducing/Reordering adaptations, F(1, 745.59) = 22.78, p < .001, with therapists who had less positive perceptions of a given practice reporting more extensive Reducing/Reordering adaptations to that same practice, Estimate = −.06, SE = .01, p < .001.

Therapist-level predictors

Therapist-level background characteristics included gender, race/ethnicity, languages spoken, education level, caseload, theoretical orientation, and years in practice. As shown in Table 3, the therapist characteristics that significantly predicted Augmenting adaptations were: years in practice F(1, 564.04) = 12.00, p = .001, and race/ethnicity F(1, 534.43) = 3.36, p = .035. Specifically, model estimates indicated that therapists with more years of experiences reported making fewer Augmenting adaptations, Estimate = −.07, SE = .02, p < .001. Hispanic/Latino therapists reported making more extensive Augmenting adaptations than Non-Hispanic White therapists, Estimate = .85, SE = .33, p = .012. No therapist background characteristics examined were associated with Reducing/Reordering adaptations.

Table 3.

Unstandardized Estimates of Effects of Predictors on Augmenting and Reducing/ Reordering Adaptations

Predictor	Augmenting Adaptations		Reducing/ Reordering Adaptations
	Estimate	SE	Estimate	SE
Intercept	8.89***	.82	6.72**	.68
Level 1 Predictors
Practicea
CBITS	.54	.90	−.15	.79
CPP	−.09	.30	−.16	.27
Seeking Safety	.27	.25	1.80**	.22
TF-CBT	.29	.23	−.04	.20
Practice-Specific PCIS	−.02	.01	−.06**	.01
Level 2 Predictors
Race/Ethnicityb
Hispanic/ Latino	.85*	.34	.51	.28
Other Minority	.25	.31	−.14	.25
Bilingual Therapist	−.05	.29	−.11	.23
Female	.33	.36	−.27	.29
Caseload	.01	.01	.02	.01
Level of Educationc
B.A. or lower	−.97	.74	−1.16	.61
Doctorate	.05	.36	−.08	.30
Theoretical Orientationd	.03	.22	.11	.18
Years of Experience	−.07**	.02	−.03	.02
EBPAS Divergence	1.17**	.16	.99**	.13
EBPAS Openness	.28	.22	.10	.14

Notes.

^*p < .05,

^**p < .001;

^aTriple P is reference group,

^bNon-Hispanic White is reference group,

^cMaster’s degree is reference group,

^dCBT is the reference group

Finally, we included therapists’ general attitudes toward EBPs as predictors of adaptations. Therapists’ Openness towards EBPs was not associated with Augmenting adaptations, F(1, 563.29) = 2.52, p = .113, or Reducing/Reordering adaptations F(1, 551.22) = .49, p = .482. However, therapists’ perceived Divergence of EBPs was associated with Augmenting adaptations F(1, 559.47) = 55.83, p < .001, and Reducing/Reordering adaptations F(1, 544.12) = 60.07, p < .001. Therapists who perceived EBPs as being more divergent from their usual practice reported more Augmenting, Estimate = 1.17 SE = .16, p < .001, and Reducing/Reordering adaptations, Estimate = .99, SE = .13, p < .001.

Discussion

There is a high probability that EBPs will be adapted by therapists when delivered in community settings (e.g., Stern et al., 2007; Taylor et al., 2015). Thus, it is advisable to engage with providers in a dialogue around dissemination to systematically plan for the eventuality of local adaptations (Aarons et al., 2012; Cabassa & Baumann, 2013; Chambers & Norton, 2016; Foa et al., 2013). End-users may consider modifications made in terms of therapy length, session frequency, or format; level of skilled workers able to provide the treatment; or therapists’ willingness to adopt all components of the treatment. The current study used survey methods to characterize the nature of therapist reports of adaptations to multiple evidence-based and evidence-informed practices being implemented as the result of a system-driven reform of children’s mental health services in Los Angeles County.

Analyses related to our first objective revealed that two types of therapist adaptations could be readily discriminated – those that reflect therapists’ active engagement with the practice (i.e., tailoring presentation, integrating supplemental content, and lengthening the pacing), and those that reflect disengagement from the elements or structure of the original practice (i.e., omitting components, re-ordering components, or shortening the pacing). These approaches of Augmenting versus Reducing/Reordering adaptations were discerned consistently across the five practices of interest in the current study, and largely conformed to a typology previously proposed by Stirman and colleagues (2015).

In objective two, we found that therapists more commonly reported making Augmenting adaptations rather than Reducing/Reordering adaptations in this implementation-as-usual context. This supports past qualitative findings that community therapists frequently tailor the terminology or language used to meet the needs of their clients and improve client engagement (Gibbs, Krieger, Cutbush, Clinton-Sherrod, & Miller, 2016; Stirman et al., 2013). However, in past studies of community implementation of EBPs, therapists also commonly reported omitting core components from practices or making other adaptations that might represent drift from fidelity (Cooper, Shrestha, Hyman, & Hill, 2015; Stirman et al., 2013; Stirman et al., 2015). Whether such adaptations have an impact on clinical outcomes is an important question for further study.

One exception to the generally low levels of reporting Reducing/Reordering adaptations in this sample was for the practice Seeking Safety, for which therapists reported extensive omission and rearranging of components. However, these reports likely do not reflect drift from Seeking Safety model, which by design allows therapists and clients to select from among 25 treatment topics that may be covered in any preferred order (Najavits, 2002). This approach is consistent with modular designs that provide flexibility in selecting the order and topics delivered in a treatment session or episode. Describing these decisions as adaptations may be misleading with this type of intervention (Chorpita et al., 2013; Park et al., 2015).

The third objective of the study was to identify factors related to therapist reports of both types of adaptations to EBPs. There was within-therapist variability in the extensiveness of Reducing/Reordering adaptations based on practice, with one practice notably having reports of more extensive adaptations. This relates to the issue of Seeking Safety’s flexible protocol as noted above. Beyond the current study, it is probable that there are variations in the extensiveness of therapist adaptations depending on features related to the intervention. In fact, our findings indicated that therapist perceptions of specific interventions were associated with the likelihood of Reducing/Reordering adaptations, specifically. Therapists who held more negative perceptions concerning the effectiveness, fit, user friendliness, and applicability of the practice were more likely to report making adaptations that may compromise the integrity of the original intervention. Perceptions of specific practices were unrelated to the likelihood of making Augmenting adaptations. Findings that attitudes differently relate to the type of adaptations therapist make is consistent with past research (Cook et al., 2015; Stirman et al., 2015), though this was the first study to investigate the relationship between percpetions of a specific practice and adaptations reportedly made to that practice versus others.

We also noted that there were some therapist characteristics associated with reports of adaptations. We found that therapists with fewer years of professional therapy experience were more likely to report Augmenting adaptations, which may indicate that more novice therapists have greater motivation to adapt treatments to their perceived clients’ needs. On the other hand, it is also possible that less experienced therapists perceive more of their delivery of EBP to be adaptations, whereas more experienced therapists perceive any tailoring of a practice to be consistent with the high quality delivery of EBPs. We also found that Hispanic/Latino therapists reported more extensive use of Augmenting adaptations, including making modifications to the presentation, adding supplemental therapy content, and slowing down the pacing of the intervention. Although we controlled for delivering therapy in a language other than English, these findings may suggest that Hispanic/Latino therapists delivering care to a largely Hispanic/Latino client population may more consistently adapt interventions to increase perceived relevance and engagement.

Cultural adaptations are considered to be important to the implementation process, especially in communities that serve ethnic minority groups (Cabassa & Baumann, 2013). Making surface adaptations, such as tailoring the presentation of intervention materials and messages, has been identified as an adaptation strategy to maintain the fidelity of models while making them more relevant and appealing to cultural groups (Resnicow et al., 1998). Within the implementation framework, Consolidated Framework for Implementation Research (CFIR), these types of adaptations are considered to be part of the “adaptable periphery,” elements or structures that can be adapted to increase the fit of the intervention with an organization or setting without undermining the intervention’s integrity (Damschroder et al., 2009). Furthermore, growing evidence supports community clinicians adapting and matching core components of EBPs to meet the needs of their culturally diverse clients, using models such as the Distillation and Matching Model (Chorpita & Daleiden, 2009; Chorpita et al., 2013; Lyon, Lau, McCauley, Vander Stoep & Chorpita et al., 2014). Minimal research has investigated whether community therapists make recommended or systematic adaptations to EBPs for their ethnic minority clients, and some evidence suggests that therapists might inappropriately omit components that they perceive as culturally unacceptable (Lau, 2012; Morawska et al., 2012). Our finding that Hispanic/Latino therapists made more Augmenting but not Reducing/Reordering adaptations suggests that the modifications made by diverse therapists for ethnic minority clients may be consistent with those tested in community partnered effectiveness studies.

Finally, therapists’ general attitudes towards EBPs was also related to reports of both types of adaptations. In particular, controlling for other background variables, therapists who felt that EBPs were generally not aligned with their usual approach to psychotherapy were more likely to report both Augmenting and Reducing/Reordering adaptations. This relationship may be explained by findings that show that therapists who believe that EBPs diverge from their clinical practice are likely to use intuitive approaches to clinical decision-making, which may lead to more overall changes to an EBP protocol (Gaudiano, Brown, & Miller, 2011). Specific to trauma treatments for children, therapists who see EBPs as diverging from their own approach are less likely to be directive in treatment and are more likely to choose psychodynamic and therapeutic techniques without an evidence base (e.g., use of touch in therapy; Allen & Crosby, 2014). It is possible that therapists who perceive EBPs as being divergent from their own practice are more likely to improvise or follow their client’s lead in session, rather than closely follow an EBP protocol. This could explain high levels of both Augmenting adaptations, such as tailoring how session content is presented, and Reducing/Reordering adaptations, that may involve removing core components of the practice not well aligned with a therapists’ orientation. Alternatively, people with divergent views may not actually make more adaptations, they may just see their approach as more original or unique so that they endorse extensive adaptations.

These findings have important implications for training, especially within contexts where therapists are delivering and adapting multiple EBPs. As ample evidence points to the inevitability of adaptations within community contexts, it is clear that training and consultation needs to focus on how therapists can adapt EBPs to their individual client needs while maintaining core components of treatment (Aarons et al., 2012; Cooper et al., 2015; Hansen et al., 2013; Stirman et al., 2015). Implementation support surrounding adaptation could include dynamic processes where developers identify components in treatment that can and cannot be adapted without compromising clinical outcomes, while community therapists provide expertise on their own personal and clients’ experiences with the practices (Southam-Gerow, Hourigan, & Allin, 2008). Collaborative approaches could allow therapists to share practice-based adaptations they have devised with other therapists and with EBP developers. This knowledge exchange may increase therapists’ feelings of investment in EBPs and perceptions of their relevance. It is still not clear how Augmenting and Reducing/Reordering adaptations impact implementation and client-level outcomes. One study found that when teachers made minor augmenting adaptations to a substance use prevention program, a higher percentage of students remained non-drug users. However, when teachers made extensive adaptations of any type, student outcomes were compromised (Hansen et al., 2013). Therefore, it is important that any implementation strategy that focuses on adaptation includes evaluations on how these adaptations impact client outcomes, along with implementation outcomes such as sustainment of practices.

This study had several limitations, which need to be considered when interpreting findings. First, though the development and use of the AES allowed for the evaluation of adaptations in a larger sample of community therapists than had previously been investigated, self-report may be biased by therapist recall and social desirability (Stirman et al., 2015). To address concerns about social desirability, we were careful to present the study as non-evaluative in developing our survey instrument and in framing the study in our recruitment materials. We conveyed three messages: (1) we wished to learn from therapists as experts on implementation in their local contexts, (2) we assumed that adaptations were occurring in the PEI implementation, and (3) we emphasized that the purpose of the overall study was to understand how community therapists “make practices work” for the diverse families they serve everyday. However, as indicated by our alternative interpretations of the findings above, the self-reports indicate therapists’ perceptions of how they modified practices. It is unclear whether the self-reported Augmenting adaptations, in particular, actually represent modifications of the original practices or clinically and culturally competent delivery of the practices. It is possible that treatment developers may not view the adaptations that therapists reported making as actual modifications to the EBP protocol, but rather as delivery inline with the treatment model. In the future, it will be important to validate therapist report against other methods, including observational data or practice expert ratings. Second, this study provided a breadth of data within the nation’s largest county mental health system, but the survey items do not illuminate the nature of the Augmenting and Reducing/Reordering adaptations or the therapists’ rationale for making the adaptations. Integrating these quantitative findings with qualitative therapist interviews may elucidate the adaptation process, and inclusion of illustrative examples of these types of adaptations may shed light on implications for implementation outcomes. Finally, although we examined several therapist characteristics associated with self-reported adaptations, there were many plausible predictors not examined in the current study at the client- and organizational-levels. It is possible that therapists were motivated to adapt practices based on client caseload characteristics such as literacy level or cultural background. In addition, it will be important to investigate the role of organizational context variables such as leadership, constraints related to resources or service delivery structure, and local implementation support for therapist adaptations to EBPs.

Even with these limitations, this study makes important contributions to our current understanding of community therapist adaptations to EBPs, given the large and diverse sample of therapists who predominately serve ethnic minority clients, the investigation of multiple EBPs, and the unique context of this fiscal mandate. This study provides insight into how intervention and therapist characteristics impact the types of adaptations made, which are important aspects to consider as EBPs are rolled out. As the first study to examine community therapist reported adaptation of multiple EBPs, it is clear that therapists’ attitudes towards and adaptations to different practices varied, suggesting that the implementation supports provided may need to vary based on the intervention. Attention to the adaptation process and how it varies across EBPs is an important step to improve future implementation efforts that maximize the public health benefits of EBP dissemination. By developing and investigating a brief measure that allows for systematic measurement of therapist adaptations, this study represents a move towards building Chamber and Norton’s (2016) “adaptome,” with the ultimate goal of expanding understanding of how adaptation impacts implementation outcomes. Next steps will examine how therapist adaptations are associated with client-level outcomes of practice delivery and therapist implementation outcomes including fidelity/adherence, and sustained implementation of practices.

Public health significance.

Increased efforts have focused on implementing EBPs in publicly-funded mental health settings, but research is needed to understand the inevitable process of community therapists’ adaptation of practices. This study is the first to identify how therapist reported adaptations vary across multiple EBPs within a mental health system reform. Findings suggest most therapists report augmenting EBPs to fit the local context, whereas adaptations reducing or rearranging EBP components are less common but more likely when therapists hold a negative view of the EBP.