Therapist Adaptations to Evidence-based Practices and Associations with Implementation Outcomes in Child Therapy Sessions

Abstract

Objective:

Community therapists inevitably adapt evidence-based practices (EBPs) to meet the needs of their clients and practice settings. Yet, the implications of spontaneous, therapist-driven adaptations for EBP implementation outcomes are not well understood. We used a sequential QUAN→qual mixed methods design to examine how different types of therapist-described adaptations were associated with observer-rated extensiveness of therapist delivery of EBP content and technique strategies at the session-level.

Method:

Data were drawn from an observational study of a system-driven implementation of multiple EBPs into public children’s mental health services. Community therapists (n = 103) described adaptations they made in 680 sessions with 273 clients (50.92% female, 49.08% male, M_age = 9.72 years, 70.70% Hispanic/Latinx). Coders classified therapist-described adaptations into five types: 1) Modifying Presentation, 2) Integrating, 3) Extending, 4) Reducing, and 5) Generalizing. Independent observers rated the extensiveness of EBP strategy delivery from session recordings using the EBP Concordant Care Assessment [ECCA] Observational Coding System.

Results:

Quantitative analyses using multilevel regression revealed that Modifying Presentation adaptations were associated with higher extensiveness of EBP technique delivery, whereas Extending adaptations were associated with lower extensiveness of EBP content and technique delivery. Qualitative analysis of adaptation descriptions identified explanations for the quantitative findings.

Conclusions:

Findings suggest that Modifying Presentation adaptations, associated with higher extensiveness, involved creative use of activities and materials, language modification, and personalization of EBP content to meet clients’ diverse needs, whereas Extending adaptations, associated with lower extensiveness, involved slowing EBP pacing in response to client challenges. Implications for provider training are discussed.

Evidence-based practices (EBPs) for youth evaluated in controlled trials have been shown to produce better mental health outcomes in comparison to usual care (Weisz et al., 2017). Yet, EBPs are not reaching the families who need them, especially those from low-income, marginalized racial and ethnic groups (Alvidrez et al., 2019). Publicly-funded mental health systems serving millions of Americans are positioned to reach marginalized communities and thus have been the target of EBP implementation efforts, in attempts to reduce disparities in quality of care (Baumann & Cabassa, 2020). As EBPs are transported from controlled research settings and implemented in routine care, there is increasing recognition that adaptations are extremely commonplace and likely inevitable in community implementation contexts (Aarons et al., 2019; Barrera Jr. et al., 2017).

Children and families seen in community settings are more likely to be socioeconomically disadvantaged and racially diverse, with greater complexity of clinical presentations compounded by chronic and acute stressors – populations often underrepresented in controlled trials during the EBP testing phase (Alvidrez et al., 2019; Southam-Gerow et al., 2012). Interventions may lack fit for children served in community settings, who are more likely to end treatment prematurely (de Haan et al., 2013), suggesting potential barriers to engagement (Barrera Jr. et al., 2017). Moreover, relative to clinical research settings, public systems of care are significantly under-resourced in supporting high-fidelity EBP implementation through provider training, ongoing consultation, and performance feedback (Regan et al., 2017). Routine care settings are also often characterized by intensive workload demands (e.g., caseload, complexity of case-mix, provider shortages) that relate to therapist burnout (Kim et al., 2018). To address heterogeneity in client needs and the multitude of organizational challenges, community therapists often report adapting EBPs to enhance fit for their clients and settings (Aarons et al., 2019; Barnett et al., 2019). Given their prevalence, it is necessary to characterize the range of therapist adaptations that occur in routine care, and study their impacts on EBP implementation and client care outcomes (Chambers & Norton, 2016).

While earlier debates presumed that adaptation occurs at the expense of fidelity to erode EBP outcomes (Elliott & Mihalic, 2004), it is increasingly recognized that EBP adaptations can be compatible with protocol integrity while having differential impacts on outcomes (Anyon et al., 2019). To the extent that adaptations achieve goals of promoting fit and reach of EBPs, the value of community implementation efforts can increase even when fidelity is diminished (von Thiele Schwarz et al., 2019). Researchers have called for systematic documentation of adaptations and assessment of their impacts on EBP implementation and care outcomes across contexts and client populations (Chambers & Norton, 2016). There has been progress in describing adaptation processes, developing adaptation taxonomies, and understanding reasons for adaptation (Stirman et al., 2017). For example, the Framework for Reporting Adaptations and Modifications – Enhanced (FRAME; Stirman et al., 2019) is a taxonomy that codifies types of adaptation, their intended purpose, how they arise (i.e., planned or unplanned), and who initiates them (e.g., therapist or developer). The FRAME also distinguishes fidelity-consistent adaptations that do not reduce adherence to an EBP’s core functions as intended by the developer, from fidelity-inconsistent adaptations that omit key functions or loosen intended structure of delivery.

However, research has primarily focused on investigator-led design-time adaptations – adaptations made by intervention researchers prior to introducing an EBP into a new setting or target population (Alvidrez et al., 2019; Chorpita & Daleiden, 2014). Investigator-driven adaptations largely involve changes made to an EBP’s implementation, delivery, or context rather than its core functions or theoretical principles (Chowdhary et al., 2014; Chu & Leino, 2017). Meta-analyses suggest outcomes favoring EBPs adapted by intervention researchers to meet the cultural needs of diverse racial and ethnic groups, although few trials have compared them to non-adapted EBP control conditions (Benish et al., 2011; Hall et al., 2016). Much less is known about the impacts of spontaneous, therapist-driven run-time adaptations made as an EBP is being implemented in community settings (Barrera Jr. et al., 2017; Chorpita & Daleiden, 2014). Failing to attend to run-time needs that arise during the implementation process may limit the success of costly implementation efforts when EBPs lack fit for local contexts (Lyon & Koerner, 2016). Community therapists hold valuable expertise about the communities they serve and accrue important practice-based evidence about what may promote client response to EBPs (Alvidrez et al., 2019). Hence, there is a need to understand how therapist run-time adaptations may improve fit of EBPs to enhance implementation and clinical outcomes (Stirman et al., 2019).

Recent studies have begun to characterize local, therapist-driven adaptations, as well as reasons for, predictors of, and outcomes associated with these adaptations. Using survey methods, Lau et al. (2017) extracted two factors from therapist reports of adaptations made to multiple EBPs implemented in children’s mental health services of Los Angeles County: Augmenting and Reducing adaptations. Therapists more frequently reported making Augmenting adaptations by adding to the EBP in some way, including modifying the presentation of EBP strategies, integrating supplemental content or strategies, and lengthening or extending the pacing of the EBP. Therapists reported making fewer Reducing adaptations, reflecting disengagement from some elements or structure of the EBP, including removing or skipping EBP strategies, adjusting the order of sessions or strategies, and shortening or condensing the pacing of the EBP. Analysis of therapists’ descriptions of their EBP adaptations at the session-level fit largely within these categories, with additions including pausing the EBP (a type of Reducing adaptation), and generalizing use of the EBP to new settings or contexts (Kim et al., 2020). Through qualitative interviews in the same Los Angeles County implementation context, Barnett et al. (2019) found that therapists made adaptations primarily to tailor EBP fit to client characteristics, such as developmental needs or cultural differences. Further research suggests therapist factors associated with the likelihood of EBP adaptations. Therapist openness to EBPs was linked to more Augmenting adaptations (Kim et al., 2020), whereas negative perceptions of an EBP were associated with more Reducing adaptations in the delivery of that EBP (Lau et al., 2017). In a different service context, Meza et al. (2019) found that greater clinician confidence in their ability to deliver cognitive-behavioral therapy in complex clinical situations and their intent to modify were associated with lower and higher numbers of reported modifications, respectively. Finally, Marques et al. (2019) revealed that higher numbers of observer-rated fidelity-consistent adaptations made by therapists delivering Cognitive Processing Therapy were associated with greater reductions in both posttraumatic stress and depressive symptoms in adult clients.

Indeed, observational ratings of greater fidelity (i.e., adherence and competence) and fidelity-consistent adaptations have both been associated with improved client outcomes in trauma treatment, suggesting that fidelity and adaptation might both beneficially impact clinical outcomes (Marques et al., 2019).

Given that community therapists routinely adapt EBPs in community mental health contexts, it is also critical to examine how such adaptations relate to implementation outcomes. Extensiveness of strategy delivery is one such implementation outcome that indexes the extent to which therapists are delivering EBP strategies common across multiple interventions for a given mental health target (Brookman-Frazee et al., 2020; McLeod et al., 2015). Understanding what therapists are actually delivering is essential to assessing the quality of community EBP implementation initiatives (Brookman-Frazee et al., 2010; Garland et al., 2010). Whereas studies of single interventions use EBP-specific fidelity measures to assess therapist delivery, the present study takes place in the context of a naturalistic observation of a system-driven implementation of multiple EBPs concurrently (Brookman-Frazee et al., 2020). Thus, assessing therapist delivery of cross-intervention EBP strategies (e.g., activity scheduling) is appropriate for the present study. Prior studies of children’s mental health services have found extensiveness of EBP strategy delivery to be relatively low in usual care (Brookman-Frazee et al., 2020; Garland et al., 2010).

The present study builds on previous work to examine associations between therapist descriptions of different types of adaptations and extensiveness of therapist EBP strategy delivery. In sessions where therapists reported adaptations, we examined their associations with extensiveness of EBP strategy delivery as rated by independent observers (Aim 1), and whether associations differed for content or technique strategy delivery (Aim 2). Whereas EBP content strategies refer to the issues being addressed in the EBP (e.g., trauma narrative), EBP technique strategies refer to the methods therapists use to impart content to clients (e.g., modeling). Using qualitative data from therapists’ descriptions of adaptations, mixed methods analysis expanded upon our quantitative findings to provide greater depth of understanding to explain the results.

Method

Participants

Data were drawn from the Knowledge Exchange on Evidence-based Practice Sustainment (4KEEPS) study, which examined the sustainment of multiple child EBPs in the Los Angeles County Department of Mental Health (LACDMH) following a reimbursement-driven implementation (Lau & Brookman-Frazee, 2016). County programs (24 programs within 14 agencies) were eligible to participate if they were contracted with LACDMH to deliver at least one of six EBPs of interest: 1) Cognitive Behavioral Intervention for Trauma in Schools (CBITS), 2) Child-Parent Psychotherapy (CPP), 3) Managing and Adapting Practice (MAP), 4) Seeking Safety (SS), 5) Trauma-Focused Cognitive Behavioral Therapy (TF-CBT), and/or 6) Positive Parenting Program (Triple P). Despite implementation support, no therapists reported on the delivery of CBITS in the present sample.

Community therapists were recruited during on-site program staff meetings. They were eligible to participate if they were: 1) employed in one of the participating program sites as a staff or trainee therapist, 2) trained in at least one of the six EBPs of interest, and 3) actively delivering one of the EBPs with at least one client. The study sample included 103 community therapists who were primarily female (88.35%), 34.13 years old on average (SD = 8.82), and racially/ethnically diverse with 55.34% identifying as Hispanic/Latinx, 21.36% as Non-Hispanic White, 14.56% as Asian American/Pacific Islander, and 8.74% as Black. Most therapists held Master’s degrees (85.44%) and were unlicensed agency staff (80.58%).

Therapists reported on their treatment of 273 child clients. Of these clients, 70.70% identified as Hispanic/Latinx, 16.12% as Black, 5.86% as Asian American/Pacific Islander, 4.76% as Non-Hispanic White, and 2.56% as Multiracial or race/ethnicity not listed. Clients were 9.76 years old (SD = 3.85) on average, with 50.92% identifying as female and 49.08% as male. Session, client, and therapist characteristics are presented in Table 1.

Table 1.

Descriptives of Session, Client, and Therapist-level Characteristics

Session-Level (n = 680)	n = 680
Adaptation Types, No. (%)
Modify Presentation	144 (21.18)
Integrate	88 (12.94)
Extend	46 (6.76)
Reduce	77 (11.32)
Generalize	53 (7.79)
Problem Focus, No. (%)
Internalizing	260 (38.24)
Externalizing	221 (32.50)
Trauma	188 (27.65)
Other	11 (1.62)
EBP, No. (%)
Managing and Adapting Practice	357 (52.50)
Trauma-focused CBT	209 (30.74)
Child-Parent Psychotherapy	49 (7.21)
Triple P: Positive Parenting Program	38 (5.59)
Seeking Safety	27 (3.97)
Session Participants, No. (%)
Youth Only	394 (57.94)
Youth and Caregiver	217 (31.91)
Caregiver Only	69 (10.15)
Session Language, No. (%)
English	563 (82.79)
Non-English	117 (17.21)
Client-Level (n = 273)	n = 273
Age, M (SD), Range	9.76 (3.85), 1 - 18
0 – 5 years	41 (15.02%)
6 – 12 years	164 (60.07%)
13 or older	68 (24.91%)
Gender, No. (%)
Female	139 (50.92)
Male	134 (49.08)
Race/Ethnicity, No. (%)
Hispanic/Latinx	193 (70.70)
Black	44 (16.12)
Asian American/Pacific Islander	16 (5.86)
Non-Hispanic White	13 (4.76)
Multiracial or Race/Ethnicity Not Listed	7 (2.56)
Therapist-Level (n = 103)
Gender, No. (%)
Female	91 (88.35)
Male	12 (11.65)
Race/Ethnicity, No. (%)
Hispanic/Latinx	57 (55.34)
Non-Hispanic White	22 (21.36)
Asian American/Pacific Islander	15 (14.56)
Black	9 (8.74)
Education, No. (%)
Below Master’s Degree	4 (3.88)
Master’s Degree	88 (85.44)
Doctoral Degree	11 (10.68)
Therapist Discipline, No. (%)
Marriage and Family Therapy	52 (50.49)
Social Work	35 (33.98)
Counseling, Clinical, & School Psychology	13 (12.62)
Other Discipline Not Listed	3 (2.91)
Therapeutic Orientation, No. (%)
Cognitive Behavioral/Behavioral Orientation	66 (64.07)
Family Systems	16 (15.53)
Eclectic	9 (8.74)
Psychodynamic	6 (5.83)
Humanistic	5 (4.85)
Other Therapy Orientation Not Listed	1 (0.97)
Licensed, No. (%)	20 (19.42)
EBPAS Divergence, M (SD), Range	1.46 (0.83), 0 - 3.83

Procedures

Therapists were consented to the study by research staff and obtained written permission from client caregivers to audio record sessions delivering an EBP of interest. Therapists recorded sessions via study-issued, password protected iPods and completed online post-session surveys for the recorded sessions, for up to three sessions, for up to three clients each (maximum of nine sessions). On the session survey, therapists responded to an open-ended prompt to describe any adaptations they made to the EBP. Session audio recordings were rated for extensiveness of EBP strategy delivery via the EBP Concordant Care Assessment (ECCA) Observational System (Brookman-Frazee et al., 2020). Session recordings were coded if they were at least 15 minutes in length with comprehensible audio quality, and had a corresponding session survey submitted within a week of the session.

Of the 805 session audio recordings collected, 64 were excluded due to session surveys being submitted more than a week after the session date, 43 were excluded because an accompanying session survey was not submitted at all, 16 were excluded because they were less than 15 minutes long, and two due to audio quality problems. In all, the 680 (84.5%) sessions included for analysis had usable and timely session survey plus audio recording. Therapists received a $5 incentive for each session survey and each session audio recording provided, and were allowed to keep the study-issued iPod if they submitted at least six recordings. All procedures were approved by the LACDMH and UCLA Institutional Review Boards.

Measures

Client-level Characteristics

Therapists reported their clients’ age, gender, and race/ethnicity.

Therapist-level Measures

Therapist Characteristics.

Therapists self-reported their gender, race/ethnicity, educational attainment, licensure status, and therapeutic orientation.

Therapist General Attitudes towards EBPs – Divergence.

Therapist perceptions that EBPs are not clinically useful and less important than clinical experience were assessed by the Evidence-Based Practice Attitudes Scale – Divergence Subscale (EBPAS; Aarons, 2004). Therapists rated each item (e.g., “Research based treatments/interventions are not clinically useful”) on a 5-point Likert scale (0 = not at all, 4 = very great extent). The EBPAS Divergence subscale demonstrated acceptable internal consistency in the present sample (α = 0.71).

Session-level Measures

Session Characteristics.

Therapists reported who attended each session (youth, caregiver, both), the language of treatment delivery (English, non-English), and the problem focus targeted for the session: internalizing (i.e., anxiety, mood), externalizing (i.e., disruptive behavior or conduct problems, attention or hyperactivity problems), trauma, or other presenting problem (e.g., “client medical complications”). Therapists reported the EBP delivered in session, which was then classified by intervention and implementation characteristics based on a prior study (Barnett et al., 2017). That is, two mutually exclusive dichotomous variables indexed: 1) whether the EBP used has prescribed session-to-session content and order (TF-CBT, Triple P) and 2) whether the EBP requires ongoing consultation (CPP, MAP, TF-CBT).

Session Adaptations.

Therapists responded yes or no to the following question: “In this session, did you adapt [EBP] for this client?” Therapists who responded yes received an open-ended prompt: “If yes, please describe how you adapted [EBP] for this session.” Given our aim to understand associations with therapist-perceived run-time adaptations, we did not provide definitions or examples for therapists on what does or does not constitute an adaptation.

The Session-level Adaptations Coding Manual (Kim et al., 2020) provided criteria, examples, and instructions for classifying therapist descriptions of session adaptations. Five broad types of adaptations were assessed and captured by 13 individual codes: 1) Modifying Presentation (modifying presentation of the EBP, translating materials), 2) Integrating (integrating supplemental content or strategies, combining the EBP with other services, providing additional psychoeducation), 3) Extending (repeating or reviewing EBP strategies, lengthening the pacing of the EBP), 4) Reducing (pausing EBP delivery, removing or skipping EBP strategies, adjusting the order of the EBP or session strategies, shortening or condensing the pacing of the EBP), and 5) Generalizing (applying the EBP to a novel problem focus, or with alternate settings or individuals than originally prescribed by the EBP’s protocol). Modifying, Integrating, and Extending were subtypes of Augmenting adaptations in Lau et al. (2017).

Coding Training and Procedure.

Adaptation descriptions were coded by three undergraduate research assistants trained through manual review and group didactic training. Coders were permitted to code independently once they reached a Cohen’s kappa of .65 (p < .05) indicating good agreement with master coder classifications on a set of criterion adaptation descriptions developed for coder training (Cohen, 1960). Coders determined the occurrence or non-occurrence of each of the 13 adaptation codes across therapist descriptions, which could be coded for multiple adaptation codes if indicated. If coders were unable to assign an adaptation code to a therapist description (n = 101; 14.85%), these descriptions were examined by two of the authors who were doctoral-level clinical psychology students (JJK and SHY). After independently coding these descriptions, JJK and SHY met to achieve consensus.

Five dichotomous variables indexing the occurrence or non-occurrence of each superordinate adaptation type based on the adaptation descriptions were used as the key predictors of interest in the analysis. For example, if any of the four Reducing adaptation codes were applied to a session, that session received an index score of “1” for Reducing adaptations.

Reliability.

Coders double coded 20% of adaptation descriptions to assess interrater reliability. Cohen’s kappas were acceptable to strong for the superordinate adaptation category codes indicating any Modifying Presentation (κ = .78), Integrating (κ = .50), Extending (κ = .87), Reducing (κ = .60), and Generalizing adaptation (κ = .59; Cohen, 1960).

Evidence-based Practice Concordant Care Assessment [ECCA] Observer Version

(Brookman-Frazee et al., 2020). The ECCA is a 32-item measure assessing the extent to which a therapist delivers essential EBP strategies for four common child mental health targets (anxiety, depression, conduct problems, trauma). ECCA items are categorized by Content (24 items) and Technique (8 items). Content items measure the delivery of the issue being addressed by the EBP (e.g., Praise, Exposure, Trauma Narrative), while technique items measure the active method used to impart content areas to a client (e.g., Modeling, Psychoeducation, Role Play).

An independent team of coders rated session audio recordings for extensiveness of each ECCA item on a 7-point Likert scale (0 = not at all, 6 = to a great extent). Extensiveness ratings were averaged across the problem targets identified for each session to create a composite score for each of the four problem targets. A total ECCA composite score was computed by averaging extensiveness ratings across all sessions for the problem focus of each session. ECCA content and ECCA technique scores averaged only the content and technique items for each session problem focus. Thus, ECCA composite scores ranged from 0 to 6 with higher scores indicating greater extensiveness of EBP strategy delivery for a given problem target in the session.

Coder Training and Interrater Reliability.

Please refer to Brookman-Frazee et al. (2020) for detailed descriptions of the ECCA development, coder training, and interrater reliability for each item. In brief, 13 coders were trained via detailed review of the coding manual and group didactic sessions. Prior to independent coding, coders achieved an average of 80% agreement on at least six criterion-rated session audio recordings. Independent observers double-coded 26% of session recordings and intraclass correlations (ICCs) were calculated to assess interrater reliability. Consistent with Fleiss (1986) and Cicchetti (1994), and considering recommendations from Trevethan (2017), ICCs exceeded the cutoff for fair agreement across all 32 items, with an average ICC of .74 (SD = 0.11) ranging from .44 (Monitoring) to .92 (Trauma Narrative). ICCs were above .60, indicating good agreement, for all but three items: Monitoring (ICC = .44), Ignoring/Differential Reinforcement of Other Behaviors (ICC = .48) and Exposure (ICC = .57).

Analytic Plan

Mixed Methods Design

The present study employed a sequential QUAN → qual mixed methods analysis, beginning with quantitative data for the primary purpose of confirmation/hypothesis testing (Palinkas et al., 2011). Qualitative methods were then applied for the function of expansion, to explore potential explanations for the quantitative results (Palinkas et al., 2011).

Quantitative Analyses

All models accounted for the four-level nested data structure, with sessions at level 1 (n = 680), clients at level 2 (n = 273), therapists at level 3 (n = 103), and agencies at level 4 (n = 14). Unconditional models were run for each outcome, revealing significant variance at the client and therapist levels for overall therapist EBP strategy delivery (ICC = .50, .31), EBP content strategy delivery (ICC = .35, .30), and EBP technique strategy delivery (ICC = .51, .28), as well as a small proportion of variance at the agency-level (overall ECCA ICC = .02; ECCA content ICC = .07; ECCA technique ICC = .004).

All models controlled for session characteristics (problem focus, practice structure, practice consultation needs, youth and/or caregiver attendance, language), client characteristics (age, gender, race/ethnicity), and therapist characteristics (gender, race/ethnicity, education, licensure status, CBT/behavioral orientation, EBPAS Divergence). All covariates, including categorical variables which were first dummy coded, were centered at the grand mean (i.e., grand mean subtracted from the individual values for each variable; Enders & Tofighi, 2007). The following variables were examined as potential covariates, but were not significantly associated with the outcomes and were thus excluded from the final models for parsimony: 1) whether a client had more than one presenting problem, 2) whether an emergent life event was discussed in session via observer ratings (Lind et al., 2021), 3) whether a client demonstrated limited engagement in session via therapist report, or 4) expressed concerns in session via therapist report (Gellatly et al., 2019), 5) client-therapist race/ethnicity match, 6) therapist perceptions of the EBP used in session (Cook et al., 2015), 7) therapist discipline, 8) therapist number of years practicing, and 9) number of EBPs therapists were trained in.

Three multilevel regressions examined associations between the five adaptation types (Modifying Presentation, Integrating, Extending, Reducing, Generalizing) and therapist EBP strategy delivery at the session-level (Aim 1) and disaggregated by content and technique strategy delivery (Aim 2), using the MIXED command with restricted maximum likelihood estimation in STATA/SE 15.1 (StataCorp, 2017). Given our interest in the effects of the five superordinate adaptation types at the session-level, we dummy coded and group mean centered these variables (i.e., group or cluster mean subtracted from the individual values for each variable), to isolate the unbiased estimate of the Level 1 relationship (i.e., the pooled within cluster regression coefficient; Enders & Tofighi, 2007).

Qualitative Analyses

Qualitative analysis of the therapist adaptation descriptions was conducted to provide greater depth of explanation for the quantitative findings (Palinkas et al., 2011). Coding, consensus, and comparison methodology drawn from a grounded theory approach was applied in a stepwise process (Willms et al., 1990). First, two authors (BW and SHY) independently reviewed therapist descriptions of the adaptation types significantly associated with therapist EBP strategy delivery to identify initial codes in an open coding process. Second, BW and SHY met to develop and refine codes, resulting in a final taxonomy of 23 action codes and 14 goal/rationale codes. Third, BW and SHY independently applied the codes across therapist descriptions of the adaptation types, before meeting to achieve consensus. Once codes were agreed upon, thematic analysis was conducted to synthesize codes into qualitative themes. BW and SHY independently identified emergent themes and met to achieve concordance. Themes were finalized in a consensus meeting of the research team.

Mixed Methods Integration

We first reviewed the significant quantitative associations between adaptation types and therapist EBP strategy delivery. We then expanded on these quantitative findings by extracting qualitative themes from therapists’ brief descriptions that might explain the observed associations. Themes arose to clarify how and why therapists made the different types of EBP adaptations, which then served to narrow explanations and interpretations of our observed quantitative findings.

Results

Of the 103 community therapists who submitted session surveys, 97 (94.17%) reported making at least one session-level adaptation. Across the 680 sessions, therapists reported making adaptations in 393 (57.79%) sessions. Of the adapted sessions, therapists described making at least one Modifying Presentation adaptation in 144 sessions (36.64% of adapted sessions), Integrating in 88 sessions (22.39%), Extending in 46 sessions (11.70%), Reducing in 77 sessions (19.59%), and Generalizing in 53 sessions (13.49%). Finally, 46 sessions had an adaptation description that did not provide enough information to be coded (11.70% of adapted sessions).

Table 2 presents the results of the multilevel regression models. Extending adaptations were significantly associated with lower extensiveness ratings of overall EBP strategy delivery (B = −0.30, SE = 0.10, 95% CI [−0.50, −0.10], p = .003). Extending adaptations were also significantly associated with lower extensiveness of EBP content strategy delivery (B = −0.21, SE = 0.10, 95% CI [−0.41, −0.02], p = .03). Modifying Presentation adaptations were significantly associated with higher extensiveness ratings of EBP technique strategy delivery (B = 0.28, SE = 0.09, 95% CI [0.10, 0.46], p = .002), while Extending adaptations were significantly associated with lower technique extensiveness (B = −0.41, SE = 0.16, 95% CI [−0.73, −0.09], p = .01). No other adaptation types were associated with overall, content, or technique strategy delivery.

Table 2.

Multilevel Regression Models Predicting Observed Therapist EBP Delivery (Extensiveness Ratings)

	Overall ECCA Ratings (n = 680)			ECCA Content Ratings (n = 680)			ECCA Technique Ratings (n = 680)
Predictor	B (SE)	95% CI	p	B (SE)	95% CI	p	B (SE)	95% CI	p
Session-level Adaptations
Adaptation Types (ref: non-occurrence of specific adaptation type)
Modifying Presentation	.11 (.06)	[−.00, .221	.052†	.001 (.06)	[−.11, .11]	.98	.28 (.09)	[.10, .46]	.002**
Integrating	.08 (.07)	[−.06, .22]	.25	.08 (.07)	[−.06, .22]	.25	.12 (.11)	[−.10, .34]	.29
Extending	−.30 (.10)	[−.50, −.10]	.003**	−.21 (.10)	[−.41, −.02]	.03*	−.41 (.16)	[−.73, −.09]	.01*
Reducing	−.01 (.07)	[−.15, .14]	.93	−.01 (.07)	[−.16, .13]	.86	−.03 (.12)	[−.27, .20]	.78
Generalizing	.06 (.10)	[−.14, .261	.59	.06 (.10)	[−14, .25]	.58	.07 (.16)	[−.25, .40]	.65
Session-Level Covariates
Problem Focus (ref: Internalizing)
Externalizing	−.05 (.07)	[−.18, .09]	.49	−.10 (.06)	[−.21, .01]	.07†	.12 (.12)	[−.11, .35]	.30
Trauma	.12 (.07)	[−.03, .26]	.12	.03 (.06)	[−.09, .16]	.63	.24 (.12)	[−.01, .48]	.06†
Other	−.14 (.20)	[−.52, .25]	.49	−.32 (.17)	[−.66, .01]	.06†	.25 (.33)	[−.39, .89]	.44
Session EBP Structured Content	.06 (.08)	[−.09, .22]	.44	.19 (.06)	[.06, .31]	.003**	.09 (.13)	[−.17, .35]	.48
Session EBP Ongoing Consultation	−.22 (.14)	[−.49, .05]	.12	−.09 (.11)	[−.30, .12]	.42	−.65 (.24)	[−1.12, −.19]	.006**
Session Participant (ref: Youth only)
Caregiver and Youth	−.09 (.06)	[−.21, .03]	.13	−.14 (.05)	[−.25, −.04]	.006**	.15 (.10)	[−.04, .34]	.13
Caregiver Only	−.14 (.08)	[−.30, .02]	.09t	−.14 (.08)	[−.28, .01]	.07†	−.01 (.14)	[−.28, .26]	.96
Session Language (ref: English)
Non-English	−.02 (.08)	[−17, .14]	.84	−.05 (.07)	[−.18, .08]	.44	.10 (.13)	[−.16, .36]	.44
Client-Level Covariates
Age (ref: Age 6 - 12)
Age 0 – 5	−.28 (.09)	[−.45, −.10]	.002**	−.10 (.07)	[−.23, .03]	.15	−.46 (.15)	[−.76, −.17]	.002**
Age 13 or older	−.07 (.07)	[−.22, .07]	.31	−.08 (.06)	[−.19, .03]	.14	−.02 (.13)	[−.27, .23]	.89
Gender (ref: Female)
Male	−.04 (.06)	[−.16, .07]	.45	−.05 (.04)	[−.13, .04]	.30	−.01 (.10)	[−.21, .18]	.91
Race/Ethnicity (ref: Hispanic/Latinx)
Black	−.14 (.09)	[−.32, .04]	.13	−.10 (.07)	[−.24, .05]	.18	−.19 (.16)	[−49, .12]	.23
Asian American/Pacific Islander	.04 (.16)	[−.27, .34]	.81	.07 (.13)	[−.19, .33]	.61	.00 (.26)	[−.51, .51]	1.00
Non-Hispanic White	−.18 (.13)	[−.43, .07]	.16	−.11 (.10)	[−.30, .08]	.28	−.28 (.22)	[−72, .15]	.20
Multiracial or Other	−.04 (.16)	[−.35, .27]	.80	−.02 (.13)	[−.28, .23]	.85	−.01 (.27)	[−.53, .51]	.97
Therapist-Level Covariates
Gender (ref: Female)
Male	.05 (.14)	[−.23, .34]	.70	.08 (.11)	[−.14, .30]	.46	−.04 (.23)	[−.49, .42]	.87
Race/Ethnicity (ref: not Hispanic/Latinx)a
Hispanic/Latinx	.07 (.09)	[−11, .24]	.45	.06 (.07)	[−.08, .20]	.40	.003 (.14)	[−.28, .29]	.98
Education (ref: Master’s Degree)
Doctoral Degree	−.10 (.14)	[−.38, .19]	.50	−.11 (.12)	[−.34, .13]	.37	−.14 (.23)	[−.60, .32]	.54
Below Master’s Degree	.04 (.24)	[−44, .51]	.88	.11 (.20)	[−.28, .50]	.57	−.13 (.39)	[−.90, .64]	.74
Licensure (ref: Not Licensed)
Licensed	.01 (.10)	[−.20, .22]	.94	.07 (.08)	[−.09, .23]	.40	−.09 (.17)	[−42, .25]	.62
Therapy Orientation (ref: not CBT)
Cognitive Behavioral/Behavioral	.07 (.09)	[−.10, .25]	.41	.02 (.07)	[−.11, .16]	.72	.17 (.14)	[−.11, .45]	.24
EBPAS Divergence Subscale	−.05 (.05)	[−.16, .05]	.34	−.02 (.04)	[−.10, .07]	.70	−.09 (.09)	[−.26, .08]	.29

^*p < .05

^**p < .01

^†p < .10

^aDifferentiation by therapist race/ethnicity for Asian American/Pacific Islander, Black, and Non-Hispanic White therapists was tested, but no variables were significant nor did their inclusion change the model findings.

Qualitative Results

From qualitative analysis of therapist descriptions of Modifying Presentation and Extending adaptations, two types of themes emerged to depict how and why therapists made these adaptations. Table 3 presents the primary themes that emerged and representative quotes.

Table 3.

Emergent Qualitative Themes and Exemplar Quotes

Emergent Themes	Type of Theme Described	Exemplar Quotes
Modifying Presentation
1. Therapists described creative use of activities and materials in active learning techniques such as modeling and skill practice (e.g., art activities/crafts, games/activities, toys, visuals) to facilitate client understanding and engagement.	How, Why	“Client wanted to make bracelets, so I turned it into a relaxation technique that he could use to help decrease his aggressive tendencies. I helped him do some deep breathing exercises before we started and also progressive muscle relaxation techniques. While we were making the bracelets, I discussed other ways we could relax or do things with our hands to help us relax, not just making bracelets.” “I presented an art exercise during session to further present affect modulation and help client understand the process of identifying different types of feelings.”
2. Therapists described modifying language for developmental level, to facilitate client understanding and engagement.	How, Why	“I described problem solving concepts using language specific to client’s developmental age level to assist client in gaining insight and understanding the concepts.” “Since the client is 5 years old I had to engage with her more than I would with someone who is older. In the session our role play was a bit more dramatic in the way I spoke, the words I used are much more simple and constant praise was a must.”
3. Therapists described linguistic and cultural conceptual translation for racially diverse clients to enhance client understanding, engagement, and fit.	How, Why	“I also explain to my client in her level of understanding the activity that we were going to work on, so she could follow instruction and participate in session. Most of my sessions are both Spanish and English to facilitate consumer to open up and be able to express her feelings and thoughts of her traumas.” “Translated content to Mandarin Chinese and used culturally-appropriate labels for emotions that ct was familiar with.”
4. Therapists described tailoring and personalizing EBP content to the client’s lived experiences (e.g., cultural, emergent life events, interests) to enhance fit.	How, Why	“Related it to experiences an 11 year old will encounter in the school and community setting. Cultural considerations (i.e. Neighborhood violence, primary language used) due to the area client lives in and the school population.” “Therapist applied the problem solving practice for session using soccer rules and how the same rules can apply to home, school and everyday life. Therapist normalized problems and reviewed feeling of when problems are solved versus unsolved. Therapist explored with client applying same problem solving skills to other areas of client’s life and how client is already used problem solving in soccer to pull from an already successful experience and clients success.”
Extending
1. Therapists described verbal repetition and reiteration of concepts in response to challenges such as difficulty understanding, and for learning consolidation.	How, Why	“[Client] seems to really struggle with understand what I am saying…Maybe it's the way I'm saying it, but she seems to struggle retaining the information from session to session. I have to take the psycho ed piece by piece in small amounts for her to be able to focus and retain it.” “Caregiver was redirected back to client behaviors and reminded that she needs to be consistent with rewards, praise, and active ignoring.” “I reviewed theoretically several previous practices/PWEBs, such as Self Awareness/ Self Praise, Self Monitoring, without external tracking, but more for Internal tracking, some Psychoeducation on Anxiety, while opening up the beginning insight towards maintenance to help the client solidify successes that he has made, and continue them.”
2. Therapists described taking more time to deliver the EBP in response to challenges, including difficulty understanding and engagement issues.	Why	“Therapist had to regress a little to still address feeling words, as client still appeared to have difficulty labeling appropriately feelings. Cl would vacillate in statements, one minute saying, "he didn’t feel sad this week, but then describing situations which would be typically as sad. In the end, the client did appear to grasp the concept after Th spent considerable time on it.” “Client is very young, had difficulty focusing on tasks, staying on topic and comprehending the practice skills discussed. Clinician had to go over skills several times.”

Modifying Presentation Adaptation Themes

Creative Active Teaching Techniques (How).

Therapists described tailoring active teaching techniques and delivering the EBP in novel, inventive ways. For example, therapists chronicled creative methods of teaching or practicing skills, using resources such as art, toys, and games. One therapist “presented an art exercise during session to further present affect modulation.” Another therapist described: “Client wanted to make bracelets, so I turned it into a relaxation technique that he could use to help decrease his aggressive tendencies.”

Modifying Language for Developmental Level (How).

Therapists also described tailoring language, such as “using language specific to client’s developmental age level.” When describing working with a 5-year-old client, another therapist stated, “our role play was a bit more dramatic in the way that I spoke, the words I used are much more simple.”

Linguistic and Cultural Conceptual Translation for Racially Diverse Clients (How).

Therapists described both linguistic and cultural conceptual translation for culturally diverse clients, invoking culturally appropriate terminology and explanations for concepts. One therapist stated, “Most of my sessions are both Spanish and English.” Another therapist described “us[ing] culturally appropriate labels for emotions that [the client] was familiar with.”

Personalizing Content to Clients’ Lived Experiences (How).

Therapists described personalizing EBP content, such as using examples that incorporate clients’ lived experiences related to culture, recent stressors, and personal interests. One therapist described “relat[ing] it to experiences an 11 year old will encounter,” including “cultural considerations (i.e., neighborhood violence, primary language used) due to the area client lives in and the school population.” Another therapist described applying “problem solving practice for session using soccer rules and how the same rules can apply to home, school, and everyday life.”

Enhance Client Understanding, Engagement, and Fit (Why).

Primary themes extracted for why therapists made Modifying Presentation adaptations were to promote client understanding, engagement, and intervention fit. Therapists described making these adaptations to “assist client in gaining insight and understanding the concepts,” “engage with her,” and “to facilitate consumer to open up and be able to express her feelings and thoughts of her traumas.”

Extending Adaptation Themes

Verbal Repetition and Reiteration of Concepts (How).

Therapists described taking more time to deliver EBP content through verbal repetition and reiteration of EBP concepts. One therapist stated: “[Client] seems to really struggle with understand[ing] what I am saying…I have to take the [psychoeducation] piece by piece in small amounts.” Another described needing to “redirect” and “remind” a caregiver about use of behavioral parenting skills.

Responding to Challenges with Client Engagement or Understanding (Why).

Therapists cited a range of challenges that motivated Extending adaptations, such as client difficulty understanding an intervention strategy or demonstrating limited engagement. One therapist described: “Client is very young, had difficulty focusing on tasks, staying on topic and comprehending the practice skills discussed.” Another stated that they “had to regress a little…as client still appeared to have difficulty labeling appropriately feelings.”

Learning Consolidation (Why).

Therapists also described verbally reviewing or reiterating concepts previously learned in therapy for learning consolidation, such as “to make sure the client remembered how to do them” or “to help the client solidify successes.”

Integration of Quantitative and Qualitative Findings

Qualitative themes expanded upon the quantitative findings by exploring how and why adaptations were made, thereby providing potential explanations for the significant associations. Modifying Presentation was described as largely motivated to improve intervention fit for the client and to promote engagement. A central theme for how this was accomplished involved the creative use of activities and materials to facilitate skill learning and practice. These findings illuminate why Modifying Presentation adaptations might be associated with higher ratings of therapist delivery of EBP technique strategies.

Extending adaptations were described as being prompted by challenges imparting skills to clients and taking more time to deliver content. Therapists appeared to rely on verbal reiteration of concepts with few active teaching strategies when clients had difficulty understanding or when reviewing previously learned concepts. This slowing and reiteration illuminates reasons for therapists covering EBP content and using EBP technique strategies (e.g., role play) less extensively.

Discussion

Given growing research documenting provider adaptations to EBPs to enhance fit for their clients and contexts, the present study examined the potential implications of different types of therapist adaptations on EBP implementation outcomes. In sessions where therapists reported these adaptations, we examined their associations with extensiveness of EBP strategy delivery within a multiple EBP implementation in children’s mental health services with diverse youth. We conducted qualitative coding and analysis to provide greater depth of explanation for our quantitative findings. Whereas prior studies have primarily examined adaptations to a single EBP, a strength of the present study was exploration of the landscape of adaptations described by therapists across multiple EBPs addressing multiple problem foci, and how they may relate to implementation outcomes.

Overall, Modifying Presentation adaptations were associated with more extensive therapist delivery of EBP technique strategies, while Extending adaptations were associated with less extensive delivery of EBP content and technique strategies. With Modifying Presentation adaptations, therapists may have perceived the need to adapt peripheral characteristics of the EBPs in ways that related to technique strategy delivery, but found the core EBP functions to hold applicability across clients, thereby being less likely to make adaptations associated with content strategy delivery. Qualitative analyses of Modifying Presentation descriptions revealed therapists tailoring active teaching techniques in creative ways; modifying language for developmental and cultural needs; and personalizing EBP content to clients’ lived experiences. Such adaptations are consistent with therapists using their local knowledge and multicultural competence to tailor EBP presentation for their clients (Ramos et al., 2021).

For Extending adaptations – characterized by repeating, reviewing, or spending more time to deliver EBP strategies than prescribed by the protocol – community therapists often described verbal repetition and reiteration of content, conveying the need to spend more time on material in response to challenges with client engagement or understanding. Prior research suggests that increased EBP dose may be indicated when serving families from diverse racial and ethnic groups, as might be expected when learning culturally unfamiliar skills rooted in European American middle-class behavioral traditions. For example, behavioral parenting interventions developed and tested with largely homogenous White research samples may be limited in their initial acceptability or familiarity to Latinx, Asian American, and Black families. Thus affordances likely should be made to provide additional sessions or time to engage, tailor, and support skill development and consolidation (Fernandez et al., 2011; Lau et al., 2011; McCabe & Yeh, 2009). However, based on associations with lower content and technique strategy delivery, community therapists describing Extending adaptations may have slowed the pacing of the EBP in ways that reduced the dose of content delivery and active teaching elements for skill-building in session (e.g., coaching, practice, performance feedback; Barnett et al., 2017). While prior work suggests that Extending adaptations are representative of Augmenting, the present findings suggest that Extending adaptations may actually be linked to decreased dosage of EBP content and techniques potentially more aligned with Reducing adaptations.

This suggests an opening toward a therapist-identified stuck point that signals the need for training and implementation support when clients are perceived to have difficulty grasping EBP concepts (Waltman et al., 2017). Indeed, Modifying Presentation and Extending adaptations were both often described with similar intentions of engaging or enhancing client learning of an EBP strategy. It stands to reason that when clients struggle to understand EBP content, therapists may have more success when they are able to respond flexibly by adapting their presentation of the material through novel and diverse active teaching techniques tailored to clients’ needs. In contrast, Extending adaptations that double down on didactic explanation and repetition may be indicative of less flexible or responsive therapist EBP delivery. Findings suggest the need to enhance therapist EBP training and consultation on how to deliver EBPs flexibly with fidelity (Kendall & Frank, 2018), such as with scaffolded skill practice and engaging teaching practices. Community therapists are likely to benefit from practical, evidence-based guidance on how to deliver EBPs flexibly, using appropriate adaptations that emerge from practice-based evidence (Cooper et al., 2016). Suggested strategies include providing a “menu of built-in adaptations” therapists may use when encountering barriers that are commonly cited as reasons for adaptation (Cooper et al., 2016, Castro et al., 2004), supplemental materials or tip sheets that discuss the evidence base related to making different types of adaptations (U.S. Department of Health & Human Services Administration for Children & Families, 2020; Webster-Stratton et al., 2011), and training on sanctioned or “allowable adaptations” to the EBP (Aarons et al., 2012). Future research on therapist effects, such as responsiveness and creativity, may provide insight on therapist factors that relate to EBP strategy delivery and adaptation, as well as clarify strategies that enhance therapy effectiveness (Castonguay & Hill, 2017).

Our findings also offer new data that may relate to discourse on the fidelity-adaptation relationship within the implementation science field. von Thiele Schwarz et al. (2019) highlight the overall value an adaptation produces while Kirk et al. (2020) call attention to both intended and unintended impacts on clinical and implementation outcomes for providers and systems. Aligned with these viewpoints and similar to Marques et al. (2019)’s findings of differential effects for fidelity-consistent versus fidelity-inconsistent adaptations on clinical outcomes, we found that Modifying Presentation and Extending adaptations were differently linked to EBP content and technique strategy delivery, contributing disparate impacts on implementation.

Finally, by asking therapists to describe their adaptations to EBPs, the present study may be revealing of therapist metacognitions, or therapist perceptions of their own engagement with EBP delivery. One way to regard our findings is that they convey therapist insights on what best promotes client response to EBPs and how these relate to extensiveness of EBP strategy delivery. Given that community therapists hold valuable local expertise, it is worthwhile to harness practice-based evidence on therapist intentions and views about how to optimize EBP care. Although observer measures will be important for validating therapist reports of adaptations, there may be challenges to observing adaptations in a wide range of multiple EBPs, given that coders will need to know the expected value of the intervention session for each EBP model.

Results from the present study must be carefully interpreted with limitations in mind. First, although the study’s high representation of Hispanic/Latinx therapists and clients and unlicensed therapists are true reflections of the LACDMH workforce and clientele, it may limit generalizability of findings. Second, therapist self-reported adaptations are vulnerable to biases, such as retrospective recall and social desirability (Barnett et al., 2019). For example, therapists may have been less likely to report Reducing adaptations due to concerns about maintaining fidelity to the EBP’s protocol. It is plausible that therapists made adaptations they did not report and vice versa. Third, therapists were not provided explicit definitions or examples of adaptations as defined by intervention developers or researchers. This approach had the benefit of not constraining community therapists to report only on a priori defined adaptations and enabled expansion upon researcher-defined adaptations (Kim et al., 2020). However, therapists may have had idiosyncratic definitions of what may constitute an adaptation in an EBP session, potentially influencing the frequency and accuracy of reporting. Indeed, in the 393 sessions in which therapists indicated they had made an adaption in the session, coders could not identify a classifiable adaptation in 46 (11.70%) of the written descriptions. Fourth, the study was limited by non-random sampling of sessions. As therapists self-selected sessions to submit, it is possible that therapists selected sessions in which they thought they delivered the EBP particularly well or with fidelity. Fifth, although we aimed to control for possible confounds related to characteristics of the case, we did not ask about and thus were unable to control for the specific reasons for adaptations, which may explain associations with EBP strategy delivery over and above the type of adaptation used. It remains possible that unmeasured case characteristics may have been related to the types of therapist adaptations made and resultant implications for strategy delivery. For example, our mixed methods analysis did suggest that Extending adaptations tended to be described in the context of engagement and learning challenges. As such, cases marked by greater symptom severity, developmental differences, or low client motivation may have elicited more therapist Extending adaptations and reduced extensiveness of strategy delivery. Finally, there was relatively lower agreement between coders for the Integrating and Generalizing codes compared to the other superordinate adaptation types. While kappas were deemed acceptable using conventionally cited cutoffs (Cohen, 1960; Rietveld & van Hout 1993), others recommend more conservative cutoffs. A recommendation to avoid discrepancies in future studies might be to spend more time in coder training discussing both examples and non-examples of Integrating and Generalizing adaptations. Application of these specific adaptation codes likely necessitates greater familiarity with the EBPs themselves and may be particularly challenging when coding sessions of multiple EBPs.

Despite limitations, the present study offers new information about how therapist-defined adaptations may impact the extensiveness of EBP strategy delivery. Findings highlight the need to provide more support for therapists on how to stay the course when clients struggle with EBP content. Therapists may need to be supported to lean into flexible, creative ways of delivering, teaching, or presenting EBP strategies to enhance engagement, understanding, and fit when clients do not initially respond to treatment. In terms of future directions, conclusions about the impact of therapist-driven run-time adaptations would be bolstered by pursuing observational methods to assess and characterize adaptations, examine concordance between therapist and observer ratings of adaptations to validate therapist reports, and identify associations with clinical and implementation outcomes in diverse communities. Using observational methods to capture local adaptations made by community therapists in session may help to elucidate a better-specified compendium of adaptation variants that relate to key clinical and implementation outcomes. Such findings can inform future study of EBP training strategies that encourage the use of adaptations when indicated that preserve the EBP’s core functions.

Public Health Significance:

Community therapists adapt evidence-based practices (EBPs) when implemented in public children’s mental health services, yet how such adaptations impact EBP delivery are unclear. Findings from the present study suggest that sessions in which community therapists tailored presentation of EBP concepts in creative ways to meet clients’ diverse needs were associated with higher extensiveness of EBP technique delivery. In contrast, sessions in which therapists slowed down intervention pacing and relied on repetition of didactic content were associated with lower extensiveness of EBP content and technique delivery. These findings may have implications for the targeted design of supportive training strategies in the community implementation of EBPs for diverse youth.