When do therapists stop using evidence-based practices? Findings from a mixed method study on system-driven implementation of multiple EBPs for children

Abstract

Therapist discontinuation of delivering an evidence-based practice (EBP) is a critical outcome in the community implementation of EBPs. This mixed methods study examined factors associated with therapist discontinuation within a large reimbursement-driven implementation of multiple EBPs in public children’s mental health services. The study integrated quantitative survey data from 748 therapists across 65 agencies, and qualitative interviews from a subset of 79 therapists across 14 agencies. Therapists adopted, on average, 2.41 EBPs (SD = 1.05, range = 1–5), and nearly half (n = 355, 47.5%) reported discontinuing at least one EBP. Multi-level models were used to predict the binary outcome of discontinuation, and qualitative analyses were used to expand upon quantitative findings. Quantitative models revealed that therapist factors, including fewer direct service hours per week, a greater number of EBPs adopted, higher emotional exhaustion, and more negative attitudes toward EBPs in general were associated with discontinuation. In addition, EBP-specific factors including more negative perceptions of the particular EBP and lower self-efficacy for delivering the specific EBP predicted discontinuation. Themes from interview responses highlighted the importance of fit of the EBP with the agency’s client base, as well as therapist perceptions of adequate EBP training supports, and the alignment of an EBP with therapists’ professional goals. Together, the findings suggest the need for strategic sustainment planning interventions that target EBP fit (i.e., fit between adopted EBPs and agency target population, fit between EBP and therapist preferences and career goals) and support therapist self-efficacy in delivering EBPs.

The implementation of multiple evidence-based practices (EBPs) within large-scale service systems is becoming an increasingly common strategy in efforts to improve mental health outcomes for youth and families (McHugh & Barlow, 2010; Starin et al., 2014). As the number of these major public investments grows, related research examining the predictors of adoption and implementation of multiple EBPs has emerged (Beidas et al., 2015; 2016; Beveridge et al., 2015; Hoagwood et al., 2014; Lau & Brookman-Frazee, 2016; Nakamura et al., 2014). However, few studies have investigated factors contributing to long-term sustainment of EBPs, or the extent to which newly implemented EBPs are maintained within an organization’s ongoing, stable operations (Proctor et al., 2011). A key threat to EBP sustainment is when therapists discontinue delivery of an EBP (Hunter, Han, Slaughter, Godley, & Garner, 2015; Massatti, Sweeney, Panzano, & Roth, 2008; Stirman et al., 2012). Understanding both the therapist and agency/organizational factors related to EBP discontinuation is critical to supporting the success of large-scale implementation efforts.

Important distinctions can be made among the circumstances under which EBPs fail to be sustained within routine practice. The term discontinuation can be applied when providers no longer continue delivering a practice at some point following adoption. Discontinuation can occur when support for implementation is insufficient, such as when provider turnover depletes the trained workforce and no new training investments are made (Massatti, Sweeney, Panzano, & Roth, 2008; Stirman et al., 2012). In contrast, de-adoption involves an active decision to remove an EBP from routine operations and in some cases requires a concerted effort to de-implement a low-value practice (Gnjidic & Elshaug, 2015; Niven et al., 2015). In the current study we focus on the unplanned cessation of EBP delivery, referred to as EBP discontinuation.

Prevalence and Costs Associated with EBP Discontinuation

This discontinuation of EBP use after adoption in community practice settings is common (Peterson et al., 2014). One study examining rates of EBP sustainment in community mental health agencies six years after implementation found that more than half of the sites (26 out of 49 sites) had discontinued delivering the EBPs due to identified barriers to sustainment (Bond et al., 2014). There are significant costs in terms of finances, person hours, and opportunity costs associated with the introduction of EBPs into community care settings, meaning it is critical to identify and implement associated supports to ensure their durability (McIntosh et al, 2010; Okamura et al, 2018).

Agency/Organizational Factors Contributing to Discontinuation

A variety of interconnected factors at different levels within a system may contribute to discontinued use of a practice (Nilsen, Timpka, Nordenfelt, & Lindqvist, 2005; Stirman et al., 2012). In the context of system-driven implementation, the outer context factors that affect implementation are generally determined by the organizational service system (e.g., policies or mandates, funding, system-level leadership; Aarons, Hurlburt, & Horwitz, 2011). Indeed, most research has described system and organizational-level determinants of sustainment such as continued leadership support, funding, or workforce turnover (Stirman et al., 2012; Bond et al., 2014). Agencies discontinue EBPs when staff turnover eliminates trained staff, when the EBP is perceived as a poor fit with the organizational mission, and when sustained implementation is not feasible due to cost restaints (Blasinsky, Goldman, & Unützer, 2006; Massatti, Sweeny, Panzano, Roth, 2008; Nadeem & Ringle, 2016). In addition, inner context factors including organizational culture and climate supporting implementation varies across settings. There is a higher probability of EBP sustainment in more well-resourced or larger organizations, agencies with strong morale where therapists feel supported and empowered, and in organizations with strong leadership for implementation (Glisson & Schoenwald, 2005; Rodriguez, Lau, Wright, Regan, & Brookman-Frazee, 2018).

Therapist Factors Contributing to EBP Discontinuation

Somewhat less attention has been paid to therapist-level factors and their associations with individual decisions to persist in using EBPs (Stirman et al., 2012). Even within a system-driven reform where decisions about initial implementation of EBPs are likely to come from organizational leadership levels (Regan et al., 2017), therapist factors may influence patterns of EBP discontinuation. A variety of therapist characteristics pertaining to training background and experience have been found to be associated with other EBP implementation outcomes, such as initial adoption and fidelity (Novins, Green, Legha, & Aarons, 2013; Blasinsky, Goldman, & Unützer, 2006). Consistent with findings from the literature on therapist attitudes toward EBPs, it is plausible therapists with more prior exposure to and comfort with EBPs, including those with a doctoral degree and those who align with cognitive behavioral orientation may be less likely to discontinue EBPs following training (Aarons, 2004).

Furthermore, as the primary end users of EBPs, therapists who become trained in multiple EBPs may contribute to organizational decisions about which EBPs to retain and deliver. Once an agency has gained exposure and experience with multiple models, leaders may consider therapist preferences and feedback when making decisions about which EBPs to sustain (e.g., Aarons, 2004; Rodriguez et al., 2018). Indeed, the likelihood of sustaining an EBP is increased when there is staff buy-in, and when there is minimal additional staff time needed for implementation (Friend, Flattum, Simpson, Nerderhoff, & Neumark-Sztainer, 2014). Thus, to the extent that therapists have a positive perception of an EBP they have been trained to deliver, it is reasonable that they would be less likely to discontinue that practice in comparison to others. The aforementioned studies of sustainment have largely been restricted to the implementation of single EBPs (Novins, Green, Legha, & Aarons, 2013; Stirman et al., 2013). The examination of predictors of sustainment within a system-driven implementation of multiple EBPs permits the study of discontinuation within a marketplace of implementation, when providers may have some choice of which EBP to implement (but not whether to implement EBPs).

Study Context

Within the Prevention and Early Intervention (PEI) initiative, the Los Angeles County Department of Mental Health led a large-scale implementation of multiple EBPs for children and youth by offering reimbursement to community agencies for their delivery. LACDMH provided training and implementation support for an initial six EBPs including Cognitive Behavioral Intervention for Trauma in Schools (CBITS), Child Parent Psychotherapy (CPP), Managing and Adapting Practice (MAP), Seeking Safety (SS), Trauma-Focused- Cognitive Behavioral Therapy (TF-CBT), and Positive Parenting Program (Triple P). These EBPs were selected because of their ability to address a range of presenting mental health concerns for youth and had developers with infrastructure to train large numbers of community therapists at the initiation of PEI. This system-driven multiple EBP implementation context provided an opportunity to examine therapist and agency level factors that might be associated with EBP discontinuation. A survival analysis of EBP delivery indexed by administrative claims data in the first five years of the PEI initiative indicated that, on average, therapists continued to deliver at least one EBP for approximately 22 months, with Marriage and Family Therapists (MFTs), bilingual, and racial/ethnic minority therapists having lower probability of discontinuation (Brookman-Frazee, et al., 2018).

The current study used a mixed-method approach to provide a more comprehensive analysis of therapist- and agency/organizational-level factors associated with the discontinuation of EBPs within a large system-driven implementation context where reimbursement for EBP delivery has remained in place. It uses sequential quantitative and qualitative methods (Palinkas et al., 2011) to explore long-term sustainment of multiple EBPs. Within a QUAN → qual design, we first tested predictors of therapist discontinuation using quantitative survey data and then we examined qualitative interview data to triangulate and deepen our interpretation of the quantitative findings (Palinkas et al., 2011).

Methods

Study Context

In 2009, the Los Angeles County Department of Mental Health (LACDMH) underwent the PEI Transformation, a system-driven reform in which contracted agencies received reimbursement for the delivery of select EBPs. Data collection for the current project was part of the larger Knowledge Exchange on Evidence-Based Practice Sustainment (4KEEPS) study, which examined the delivery of six EBPs for which LACDMH provided implementation support via training and consultation (Lau & Brookman-Frazee, 2016). These six EBPs included: Cognitive-Behavioral Intervention for Trauma in Schools (CBITS), Child-Parent Psychotherapy (CPP), Managing and Adapting Practice (MAP), Seeking Safety, Trauma Focused Cognitive Behavior Therapy (TF-CBT) and Positive Parenting Program (Triple P).

Procedure

The current study included therapists who participated in an online survey and those who took part an in-depth interview. Therapists were eligible to participate in the survey if they worked at agency directly operated or contracted by LACDMH, and if the therapist had been trained in and had ever delivered at least one of the six EBPs of interest. Therapists completed the survey between March and July 2015, which was approximately six years after the initial implementation of the PEI Transformation. The survey had a response rate of 44.2%, which is within the range of previous surveys of community mental health therapists (Hawley et al. 2009; Cashel 2002). Participants received a $20 incentive for completing the survey and an additional $20 incentive if they completed the survey within 2 weeks of the initial invitation.

Following the survey campaign, agencies were recruited to be part of “in-depth” data collection portion of the 4KEEPS study, which included interviews with agency leaders and therapists, and collection of session audio recordings of therapists delivering one of the six EBPs of interest. For details on recruitment and procedures see Lau & Brookman-Frazee (2016). Therapists were eligible to participate in the “in-depth” portion of the study if they were willing to participate in an interview with study staff and record sessions for three clients receiving one of the six EBPs. Researchers attended regular agency staff to recruit and consent therapists for the in-depth portion of the study. Therapists received a $40 incentive for completing the interview. Institutional Review Boards at LACDMH and the University of California, Los Angeles approved all procedures for this study.

Participants

Survey sample.

In total, 748 therapists who had been trained in and who had ever delivered at least one of the six EBPs of interest completed the online survey (88.0% female; M_age = 36.15, SD = 9.09), of which 44.5% were Hispanic, 85.7% held a Master’s degree, 39.8% were licensed, 57.9% ascribed to a Cognitive-Behavioral or Behavioral theoretical orientation (see Table 1). In the year 2015, 4,545 therapists across the county claimed for at least one of the PEI practices. In the sample that completed the survey, therapists discontinued the EBPs they had previously used at the following rates: CBITS = 85.7%; CPP = 20.4%; MAP = 14.6%; SS = 44.3%; TF-CBT = 22.4%; Triple P = 38.5%. See Table 2.

Table 1.

Descriptives of Therapist Characteristics for Survey and Interview Samples

Therapist Characteristics	Survey Sample	In-Depth Interview Sample
	(n = 748)	(n = 79)
Age (Years), M (SD)	36.15 (9.09)	35.26 (9.07)
Gender, No. (%)
Female	658 (88.0)	70 (88.6)
Male	90 (12.0)	9 (11.4)
Race/Ethnicity, No. (%)
Non-Hispanic White	249 (33.3)	14 (17.7)
Hispanic	333 (44.5)	48 (60.8)
Other Ethnic Minority	166 (22.2)	17 (21.5)
Education, No. (%)
Less than Master’s Degree	19 (2.5)	2 (2.5)
Master’s Degree	641 (85.7)	69 (87.3)
Doctoral Degree	88 (11.8)	8 (10.1)
Years of Professional Experience, M (SD)	6.22 (5.58)	4.34 (3.32)
Licensure, No. (%)
Licensed	297 (39.8)	17 (21.5)
Not Licensed	451 (60.2)	62 (78.5)
Theoretical Orientation, No. (%)
Cognitive Behavioral or Behavioral	433 (57.9)	50 (63.3)
Other Theoretical Orientation	315 (42.1)	29 (36.7)
Number of EBPs Therapist Ever Used, M (SD)	2.41 (1.05)	2.72 (1.05)
Hours per Week on Direct Service, M (SD)	18.74 (7.70)	20.86 (6.42)

Notes. EBP = evidence-based practice.

Table 2.

Rates of EBP Use and Discontinuation

Practice	Survey Sample (n = 748)			In-Depth Interview Sample (n = 79)
	Ever Used EBP, n (% of sample)	Currently Using EBP, n (% of sample)	Discontinued EBP, n (% of sample who ever used EBP)	Ever Used EBP, n (% of sample)	Currently Using EBP, n (% of sample)	Discontinued EBP, n (% of sample who ever used EBP)
CBITS	42 (5.6%)	6 (0.8%)	36 (85.7%)	7 (8.9%)	0 (100%)	7 (100%)
CPP	137 (18.3%)	109 (14.6%)	28 (20.4%)	24 (30.4%)	20 (25.3%)	4 (16.7%)
MAP	513 (68.6%)	438 (58.6%)	75 (14.6%)	60 (75.9%)	47 (59.5%)	13 (21.7%)
SS	400 (53.5%)	223 (29.8%)	177 (44.3%)	46 (58.2%)	26 (32.9%)	20 (43.5%)
TF-CBT	527 (70.5%)	409 (54.7%)	118 (22.4%)	58 (73.4%)	49 (62.0%)	9 (15.5%)
Triple P	179 (23.9%)	110 (14.7%)	69 (38.5%)	24 (30.4%)	16 (20.3%)	8 (33.3%)

Notes. EBP = evidence-based practice; CBITS = Cognitive Behavioral Intervention for Trauma in Schools; CPP = Child Parent Psychotherapy; MAP = Managing and Adapting Practice; SS = Seeking Safety; TF-CBT = Trauma-Focused-Cognitive Behavioral Therapy; Triple P = Positive Parenting Program.

In-depth interview sample.

A total of 123 therapists were interviewed from 20 program sites across 14 agencies. A two-fold sampling process was used to select interviews for qualitative coding and analysis. First, a random sample of 60 therapist interviews was drawn for transcription and coding, stratified by EBPs discussed. All 60 interviews from this stratified random sample from the full set of interviews were included in the current sample. Second, the survey item regarding whether a therapist discontinued an EBP was linked to the remaining full set of interviews. This process identified 19 additional interviews conducted with therapists who indicated that they discontinued an EBP on the survey. These 19 additional interviews were added to the current sample to ensure that a significant proportion of interviews discussed discontinuation. Thus, the final interview sample included 79 therapists (88.6% female; M_age = 35.26, SD = 9.07), of whom 60.8% were Hispanic, 87.3% held a Master’s degree, 21.5% were licensed, 63.3% ascribed to a Cognitive-Behavioral or Behavioral theoretical orientation (See Table 1). These therapists discontinued EBPs they had previously used at the following rates: CBITS = 100%; CPP = 16.7%; MAP = 21.7%; SS = 43.5%; TF-CBT = 15.5%; Triple P = 33.3%. See Table 2.

Measures

Quantitative Measures.

Outcome: Therapist discontinuation.

Discontinuation of specific EBPs by therapists was the main outcome of interest. Discontinuation was defined by therapists reporting: 1) they had ever delivered a given EBP to a client, and 2) they had not used that EBP within the past two months.

Predictors of EBP discontinuation: Therapist factors.

EBP-Specific Perceptions.

The Perceived Characteristics of Intervention Scale (PCIS; Cook, Thompson, & Schnurr, 2014) was used to measure therapist perceptions of specific EBPs. The original PCIS measure included 20 items assessing therapists’ perceptions of any mental health care intervention or practice. Therapists completed an adapted version of the PCIS for any PEI practice for which they had ever received training or ever delivered. The adapted version of the PCIS administered in the current study included 8 items that assessed therapists’ impressions of relative advantage (e.g., “[The EBP] is more effective than other therapies I have used.”), compatibility (e.g., [The EBP] is aligned with my clinical judgment.”), complexity (e.g., [The EBP] is easy to use.”), and potential for reinvention (e.g., “[The EBP] can be adapted to meet the needs of my client.”). For each EBP, therapists rated their agreement with each item on a 5-point Likert scale (1 = not at all, 5 = a very great extent). Internal consistencies of the 8-item scale for all EBPs in the current sample were strong, with Cronbach’s alphas ranging from α = 0.93 to α = 0.97.

Therapists’ Perceptions of EBP Self-Efficacy were assessed with two items developed for this study. The two items were: “I am well prepared to deliver [the EBP] even with challenging clients,” and “I am confident in my ability to implement [the EBP].” Therapists answered these items about each practice that they reported using with a client. Each item was rated on a 5-point Likert scale (1 = not at all, 5 = a very great extent). A mean of both items was used as a composite score. Internal consistencies for all EBPs were strong, with Cronbach’s alphas ranging from α = 0.83 to 0.94.

Therapist demographic and professional characteristics.

Therapists completed a background questionnaire which included therapist age, gender, race/ethnicity, level of education, years of professional experience, licensure status, theoretical orientation, number of EBPs ever used with a client, and average number of direct services hours provided per week.

Therapist emotional exhaustion.

Therapist feelings of emotional exhaustion in the workplace was measured by five items from the Emotional Exhaustion subscale of the Organizational Social Context Questionnaire (OSC; Glisson et al., 2007). Therapists rated their perceptions of stressful climates characterized by factors such as workload (e.g., “I feel used up at the end of the day.”) and work-related emotional exhaustion (e.g., “I feel emotionally drained from my work.”; Glisson et al., 2007). Responses were rated on a 7-point Likert Scale (0 = strongly disagree, 6 = strongly agree) with higher scores representing more exhaustion. The measure demonstrated good internal consistency (α = .81) in the current sample.

Therapist general attitudes towards EBPs.

Therapists’ general attitudes towards the adoption of EBPs was measured with the Evidence-Based Practice Attitudes Scale (EBPAS; Aarons 2004). The original EBPAS consisted of 15 items, which yielded four subscales: Appeal, Requirements, Openness, and Divergence. The current study included two complete subscales from the EBPAS: Openness and Divergence, each of which consisted of four items. These subscales were chosen to reduce therapist response burden. In addition, the EBPAS was used to assess attitudes towards EBPs in general (rather than perceptions of specific EBPs), and the Openness and Divergence subscales were determined to be most relevant for assessing more general views. The Openness subscale assessed the therapist’s openness to trying new interventions and willingness to use EBPs, and included items such as “I like to use new types of therapy/interventions to help my clients.” The Divergence subscale assessed the therapist’s perception that EBPs were not as clinically useful as clinical experience, and included items such as, “Research based treatments/interventions are not clinically useful.” Therapists rated each item on a 5-point Likert scale (0 = not at all, 4 = very great extent). In the current sample, the Cronbach’s alpha indicated that the internal consistency was acceptable for the Openness scale (α = 0.79) and for the Divergence scale (α = 0.71).

Predictors of EBP-specific discontinuation: Agency and organizational factors.

Agency size.

Information regarding the size of agencies was gathered from administrative claims data for the year during which the survey was conducted (2015). On average, the number of children who received PEI services in each agency was approximately 500 (M = 497, SD = 516), and ranged from 7 to 2,197 children. The number of children who received PEI services in each agency was used as a continuous variable in subsequent analyses.

Perceived organizational climate.

The Organizational Climate Measure (OCM; Patterson et al., 2005) measured therapist perceptions of their organization’s policies, practices, and procedures. For this study, three separate subscales were used: 6-items for Involvement (e.g., “People feel decisions are frequently made over their heads.”), 5-items for Autonomy (e.g., “Leaders or agencies keep too tight a reign on the way things are done around here.”), and 5-items for Performance Feedback (e.g., “People’s performance is measured on a regular basis.”). Items were rated on a 4-point Likert Scale (1 = definitely false, 4 = definitely true). Consistent with other studies that have examined organizational climate, this study averaged and aggregated the subscales to the agency level in the multilevel model to account for the organizational climate of the agency (Aarons & Sawitzky, 2006). The Involvement, Autonomy, and Performance Feedback subscales had acceptable to good reliability in the current sample (α = 0.85, 0.72, 0.80, respectively).

Qualitative Measures

Qualitative semi-structured interview.

Qualitative data for this study were drawn from part of a larger interview related to therapist perceptions of the EBPs. Specifically, the semi-structured interview guide focused on gauging therapist perceptions of each practice, experiences with initial and ongoing training, perceived impacts of the practices on the clients, agency, and the therapists themselves, general experiences with implementation, and adaptations that the therapists made while delivering each practice. The interview followed a funnel approach with broader questions asked first followed by more specific follow-up probes (Spradley 1979). Broader questions regarding therapists’ general perceptions of a given EBP were asked first, and were followed up by more specific questions regarding continued use of an EBP. Most therapists discussed discontinuation when asked about why they had not continued to use a given EBP, usually in response to the question,“Can you please tell us why you haven’t used [EBP]?” Interviews lasted approximately 60 minutes and were conducted at the agency in which the therapist worked.

Data Analytic Plan

Mixed‑methods design.

A sequential QUAN → qual mixed-method analysis was employed to examine factors associated with therapist discontinuation, wherein the primary method of quantitative survey data analysis was used to guide the secondary method of qualitative analysis (Palinkas et al., 2011). Given the focus of the multi-level model on therapist-level and agency/organizational-level factors contributing to EBP discontinuation, the qualitative analysis was similarly organized into inner context levels in understanding factors related to discontinuation.

Quantitative analyses.

Three-level mixed effects logistic regression models were used to examine therapist-level and agency/organizational-level predictors of therapist discontinuation of EBPs, with specific EBPs nested within therapists nested within agencies. Due to very low rates of missing data (0.83% missing values), listwise deletion was used when running analyses. All variables were missing at rates of < 1.0%. All multilevel analyses were run using Stata/SE 15.1.

Qualitative analyses.

The qualitative data analysis was completed using QSR International’s NVivo 11 Software. To identify codes and themes related to therapist discontinuation, a “coding, consensus, and comparison” methodology was employed (Hill, Thompson, & Williams, 1997; Willms et al., 1990; Glaser & Strauss, 1967). This is an iterative process, in which the coding team reviewed a subset of interviews to inform the development of a preliminary coding scheme which included eight broad a priori codes. This initial coding scheme was then applied to a set of four initial interviews coded by two postdoctoral researchers to ensure that all relevant themes were captured. After independent coding and a consensus analysis, emergent codes were added to the codebook to capture additional constructs not captured. Once a final coding scheme was decided on, the coding team (which consisted of two post-baccalaureate research coordinators, one graduate student, and one postdoctoral scholar) applied the final codebook to all transcripts. Regular meetings were held with the coding team to examine coding across analysts, conduct iterative refinement of code definitions, collaborate on the development of themes, and to come to consensus on any discrepancies in code application. To avoid coder drift and inconsistency, 50% of all interviews that were independently coded were randomly selected to be reviewed by the senior coder (AR).

Finally, qualitative themes were identified through analysis of co-occurring codes and text analysis. This process involved independent review by two coders to identify broad themes indicated in the coded data followed by consensus meetings to review and agree upon overall themes pertaining to discontinuation. In this manner, two therapist-level themes and three agency/organizational-level themes emerged to characterize and explain factors related to therapist EBP discontinuation. These procedures represent a conventional approach to qualitative analyses, which is frequently employed in implementation research (Aarons & Palinkas, 2007; Hamilton, Chinman, Cohen, Oberman, & Young, 2015; Palinkas, 2014). Consensus processes allow for an iterative refinement of code definitions and the logic of the coding tree, as well as collaborative development of themes (Hamilton et al., 2015; Palinkas, 2014).

Integration of quantitative and qualitative findings.

The primary functions of the mixed-methods analyses were: 1) convergence – investigating whether both sources reached the same conclusion (i.e., triangulation), 2) complementarity – utilizing the surveys to provide a breadth of information and the interviews to provide a depth of understanding, and 3) expansion – utilizing the qualitative analyses to explain and contextualize quantitative findings (Palinkas et al. 2011). Integration of quantitative and qualitative data was conducted by first examining the descriptive data regarding the outcome of interest, therapist discontinuation. Next, themes regarding predictors of discontinuation that emerged from analysis and integration of the qualitative data and corresponding quantitative data were examined in two general categories: 1) therapist-level predictors of therapist discontinuation, and 2) agency/organizational-level predictors of therapist discontinuation.

Results

Descriptives: Therapist EBP Use and Discontinuation

Therapists reported having ever used, on average, 2.41 EBPs with a client at some point (SD = 1.05). Specifically, 14.4% of therapists reporting having used four or more EBPs, 33.6% of therapists reported having used three EBPs, 29.9% reported having used two EBPs, and 20.5% reported having used only a single EBP.

Overall, nearly half of the therapists in the study (n = 355, 47.5%) reported discontinuing at least one of EBPs in the past two months. Of these, 31.6% reported discontinuing a single EBP (n = 236), 12.6% reported discontinuing two EBPs (n = 94), with smaller numbers discontinuing three (n = 22, 2.9%) or more EBPs (n = 3, 0.4%). In the sample of qualitative interviews, 38.3% of therapists reporting discontinuing at least one EBP.

Factors Predicting Discontinuation

Therapist-level quantitative results.

In the quantitative analyses, a three-level mixed effects logistic regression model was used to examine predictors of therapist EBP discontinuation, where level 1 was the EBP-level with the outcome being dichotomous (1=discontinued, 0=continued/current use), level 2 was the therapist-level, and level 3 was the program-level.

Results are presented in Table 3. Younger therapist age was associated with an increased odds of discontinuation (B = −0.03, p < 0.01, OR = 0.97). The greater the number of EBPs a therapist had ever used, the more likely they were to discontinue any specific EBP (B = 0.50, p < 0.01, OR = 1.65). In addition, fewer direct service hours each week was associated with higher odds of discontinuation (B = −0.04, p < 0.01, OR = 0.96), and lower reported emotional exhaustion was associated with higher odds of discontinuation (B = −0.11, p < 0.05, OR = 0.90).

Table 3.

Predictors of Therapist EBP-Specific Discontinuation in Full Sample (n = 748)

	B	SE B	Odds Ratio
Practice-level Predictors
Specific EBP (Ref = MAP)
CBITS	3.01**	.50	20.25
CPP	.10	.28	1.10
SS	1.47**	.17	4.36
TF-CBT	.77**	.18	2.17
Triple P	1.26**	.23	3.53
Therapist Attitudes Towards Specific EBPs
Therapist Self-Efficacy	−.51**	.09	.60
PCISa	−.20*	.10	.82
Therapist-level Predictors
Age (years)	−.03**	.01	.97
Gender (Ref = Male)	−.33	.18	.72
Race/Ethnicity (Ref = NHW)
Hispanic	.02	.14	1.02
Other Ethnic Minority	−.04	.17	.96
Education (Ref = Master’s Degree)
Less Than Master’s Degree	−.36	.66	.70
Doctoral Degree	.09	.20	1.09
Years of Professional Experience	.02	.02	1.02
Licensure	.14	.15	1.15
Cognitive-Behavioral/Behavioral Orientation	−.07	.12	.94
Number of EBPs Ever Used	.50**	.07	1.65
Hours Per Week of Direct Service	−.04**	.01	.96
Emotional Exhaustion	−.11*	.04	.90
General Attitudes Towards EBPsb
EBPAS Divergence	.18*	.08	1.19
EBPAS Openness	.08	.10	1.08
Agency-level Predictors
Number of Children Served Per Year	−.01	.01	1.00
Organizational Climate (OCM)c
Autonomy	.35	.34	1.41
Involvement	−.24	.26	.79
Performance Feedback	−.26	.31	.77

Notes. CBITS = Cognitive Behavioral Intervention for Trauma in Schools; CPP = Child Parent Psychotherapy; MAP = Managing and Adapting Practice; SS = Seeking Safety; TF-CBT = Trauma-Focused-Cognitive Behavioral Therapy; Triple P = Positive Parenting Program; NHW = Non-Hispanic White.

^aPerceived Characteristics of Intervention Scale;

^bEvidence-based Practice Attitude Scale;

^cTherapist reports on Organizational Climate Measure (OCM) were aggregated to the agency level. .

^*p < .05;

^**p < .01.

Therapist attitudes about EBPs in general and attitudes regarding specific EBPs were also associated with discontinuation. In terms of general attitudes towards EBPs, therapists who reported higher levels of divergence, or perceptions of EBPs as generally not useful clinically, had a higher likelihood of discontinuing any given EBP (B = 0.18, p < 0.05, OR = 1.19). In addition, more unfavorable therapist perceptions of a specific EBP (B = −0.20, p < 0.05, OR = 0.82) and lower ratings of therapist self-efficacy regarding a specific EBP (B = −0.51, p < 0.01, OR = 0.60) predicted a greater likelihood of discontinuation of that EBP.

Therapist gender, race/ethnicity, education level, years of professional experience, licensure status, and theoretical orientation, were not significantly associated with EBP discontinuation.

Therapist-level qualitative results.

Out of the total 79 therapists from the in-depth sample, 17 (21.5%) interviews included discussion of discontinuation within 34 unique quotes or references. These quotes were elicited in response to the question: “Can you please tell us why you haven’t used this [EBP]?” On average, therapists discussed discontinuation 1.09 times per transcript (SD = 0.37, Range = 1–3). From these comments, two themes emerged across the interviews as therapist-level reasons for EBP discontinuation. These included therapist professional goals and issues with adequacy of training. Themes are discussed next beginning with the most frequently discussed reason for discontinuation.

Fit with therapist professional goals.

Therapists discussed the ways in which their professional goals and preferences influenced EBP discontinuation. This was particularly relevant with regard to developing expertise within a particular area (e.g., trauma) or age group (e.g., adolescence). For example, one therapist discussed dropping one practice over another due to their own professional goals related to specialization, “[I’m not using MAP anymore] because I’m specializing in early childhood and I want to get rostered in CPP.”

Therapist self-efficacy with the practice.

Therapists mentioned factors related to their perceived competence following training as a reason for EBP discontinuation. Therapists noted that low feelings of competence and confidence in administering an EBP led to them stop using a practice. As described by one therapist, “I did the online training a while ago. I don’t feel the most confident to use it without going back and learning it. So if I had a client [that could benefit] it would still be challenging to use it because I’m not as familiar.” Another echoed this sentiment, “I still don’t feel formally trained in it [TF-CBT],” following the required protocol of initial workshop and consultation.

Agency/Organizational Factors Predicting Discontinuation

Agency/organizational-level quantitative results.

Table 3 shows the results of the three-level mixed effects logistic regression model used to examine agency/organizational-level predictors of therapist discontinuation of PEI practices. None of the agency-level factors examined (i.e. agency size, aggregated therapist-reports of autonomy, involvement, performance feedback) were significantly associated with discontinuation.

Agency/organizational-level qualitative results.

Three themes emerged across the therapist interviews related to agency/organizational-level factors contributing to therapist discontinuation of EBPs. These included a mismatch between EBP and clinical population served, leader directives, and funding.

Mismatch between EBP and agency client base/therapist casemix.

Most therapists described a mismatch between particular EBPs and the clinical population served by the agency as contributing to discontinuation. They highlighted instances of the inappropriateness of EBPs for a client age, presenting problem, and severity in the therapist’s caseload. One therapist described lack of fit with the EBP target age, “Most of my caseload tends to lean on the younger side, more pre-adolescents, so I know that this model [Seeking Safety] is a good fit for more of the adolescent age.” Another talked about symptom severity, noting “I haven’t had any clients [with] a history of cutting or substance abuse, so I haven’t really felt the need [to use it]…my clients don’t have those presenting problems.” Another therapist mentioned taking into consideration both client and caregiver symptom severity, “Based on the level of severity that either the client is presenting, and likewise with the parent, that would kind of dictate that maybe Triple P would not be appropriate.” These concerns appeared to relate to a therapists’ access to the appropriate clients within their agency, as well as the broader match between a particular EBP and the types of clients typically served by the agency.

Leader directives.

About one in four therapists cited agency/organizational leader directives in describing reasons for EBP discontinuation. Therapists noted that these preferences were communicated by clinical supervisors and program leaders, and influenced their decisions to continue using certain EBPs. For example, one therapist described the impact of supervisor EBP preference for one practice influencing the discontinuation of another, “We’ve been encouraged by supervisors to use of TF-CBT instead.”

Alignment with funding for services.

One in five therapists mentioned issues with funding as a reason for discontinuation. They reported that when their agency was not contracted or reimbursed for PEI practice delivery, certain EBPs were ultimately dropped. As illustrated by one therapist, “We’re not currently PEI funded, so we don’t have to [use the EBP], and, and I don’t think a lot of our clinicians are pushed to get PEI trained.” Another similarly explained how utilization of an EBP was restricted because, “Here [in this program] they’re not in our funding source, so I can’t utilize them. That’s kind of one of the drawbacks.”

Integration of Quantitative and Qualitative Findings

Table 4 depicts an integration of qualitative and qualitative findings. In some cases, the quantitative model results and the extracted qualitative themes converged such that the qualitative themes provided more detailed potential explanations for survey results. For example, more favorable therapist perceptions of the EBPs related to lower probability of discontinuation. However, the PCIS scale content includes items that relate to ease of use of the EBP, adaptability fo the EBP, and fit of the EBP for therapist preference and clients. The interview responses narrowed interpretations to suggest that the key perception was likely fit of the EBP with client needs, and that this in turn was dictated by agency-level case-mix.

Table 4.

Integration of Quantitative and Qualitative Findings on Factors Related to EBP Discontinuation.

Quantitative Findings	Qualitative Findings	QUAN → QUAL
Less favorable therapist perceptions of the EBP	Poor fit of the EBP with the agency mission and population served Hence EBP doesn’t fit therapist’s caseload EBP not aligned with therapist professional goals/interests	QUAL expands upon QUAN finding
Low therapist self-efficacy in delivering the EBP	Fewer cases to apply the EBP Training and support for the EBP was insufficient
Fewer direct service hours	Limited opportunity to apply EBPs
Therapist Emotional Exhaustion		Unique to QUAN Complementary
Having used more EBPs
EBPs in general not aligned with therapist’s approach to therapy
	No funding for PEI EBP in a specific program	Unique to QUAL Complementary
	Leadership directives to use different EBP

In other cases, the quantitative and qualitative results were more distinct and did not align across the survey and interview findings. These non-overlapping results may reflect the fact that the qualitative and quantitative methods measured distinct constructs. For example, findings unique to the qualitative measures highlighted the importance of organizational factors related to specific EBP funding parameters and leadership directives. Of note, these specific organizational factors were not assessed in our survey measures. Conversely, therapist survey responses indicated that burnout and general discomfort with EBPs were linked to EBP discontinuation but our interview prompt did not lend itself to revealing these determinants as we inquired about factors that led therapist to discontinue specific EBPs.

Discussion

As there are increasingly more large-scale efforts to implement EBPs into routine care settings, there is a growing need to understand factors that contribute to sustainment, and particularly discontinuation, of EBPs. Existing data offer some insight into therapist-level and agency/organizational-level factors that contribute to the discontinuation of a specific EBP, though there is minimal work examining discontinuation across multiple EBPs. The current study sought to address this gap by using a mixed method approach to examine the relationship between therapist and agency/organizational characteristics associated with discontinuation in the context of a system-driven implementation of multiple EBPs.

Quantitative and qualitative results suggested that therapist-related factors may play a role in understanding EBP discontinuation, particularly therapist perceptions of EBP fit. Perceptions of EBP fit included the fit for clients served, fit of EBP requirements and training resources, as well as the fit with a therapist’s preferences and career goals. Therapists who generally viewed EBPs as not clinically useful were more likely to discontinue any given EBP. Therapist attitudes about specific EBPs were also linked to their likelihood of discontinuing that particular EBP. Specifically, therapists with more positive perceptions of a specific EBP and a higher sense of self-efficacy in delivering the EBP were less likely to discontinue that EBP. These trends were also confirmed in the qualitative interviews, as a theme was therapist lack of confidence in their ability to administer an EBP as reasons for discontinuation. Therapist attitudes toward EBPs have been found to relate to other implementation outcomes including adoption, reported adaptations inconsistent with fidelity, and sustained delivery (Novins, Green, Legha, & Aarons, 2013; Blasinsky, Goldman, & Unützer, 2006). Therapist anxiety regarding their delivery of an EBP has been found to be associated with therapists’ non-use of certain EBP strategies (Mulkens, de Vos, de Graaff, & Waller, 2018). This inhibitory anxiety means that therapists are less likely to start the relevant EBP, due to not being certain about the outcome. On the opposite side of this, Nehrig and colleagues (2019) point out that the use of EBPs can help alleviate therapist anxiety by giving them confidence that they are doing the right thing and that they have the tools to be helpful in the form of an EBP (Nehrig, Prout, & Aafjes‐van Doorn, 2019).

The current findings further suggest that therapist perceptions of the fit of the intervention for their clients and the way they like to practice is associated with continuing a practice in the context of system driven multiple EBP implementation where therapists are required to deliver EBPs but may have some latitude in which ones they sustain. In addition, the qualitative data highlighted the importance of perceived EBP fit with a therapist’s preferences and their professional and career goals. For example, several therapists discussed a goal of “specializing” in a particular presenting problem type (e.g., trauma) or developmental period (e.g., early childhood). The opportunity to gain more extensive and complementary training in a focused area may contribute to a therapist’s sense of an EBP’s fit, increasing positive perceptions and self-efficacy regarding a particular EBP. It is also likely that EBP-setting fit played a role in the lower sustainment of CBITS, which was the only one of the six EBPs designed to be delivered in a school setting and in a group format. Since a majority of therapists practiced in primarily outpatient settings and primarily delivered individual therapy this reduced the fit with CBITS and contributed to discontinuation following training. A previous study examining implementation strategies used in 4KEEPS showed that some PEI EBPs, such as CBITS, required agencies to partner with local schools, which added an additional layer to the implementation process (Regan et al, 2017). Program leaders also noted that the group format of CBITS made sustained delivery more difficult, citing the logistical challenges associated with getting a group started (Rodriguez et al, 2018).

In terms of therapist background characteristics, only age was related to discontinuation such that younger therapists were found to be more likely to discontinue EBPs. It is unclear what might account for this quantitative finding. Overall, this was a sample of primarily early career community mental health professionals, and this finding emerged controlling for therapist education level, years of clinical experience, discipline, or theoretical orientation, none of which were related to EBP discontinuation. It is also cannot be attributed to different levels of self-efficacy or perceptions of the EBP by therapist age, as these variables are also in the model. Further research is needed to replicate and interrogate this pattern.

Quantitative data also suggested that lower therapist-reported emotional exhaustion was associated with a higher likelihood of therapists reporting the discontinuation of a specific EBP. These results contradict many other studies, which have generally found lower exhaustion to be linked with more positive outcomes for therapists and clients, as well improved implementation of EBPs (Morse, Salyers, Rollins, Monroe-DeVita, & Pfahler, 2012). In the current study, the cross-sectional nature of the data limits our ability to understand the directionality of this relationship. It is possible that therapists using a fewer number of EBPs are less exhausted due to lower demands. A previous study examining emotional exhaustion in 4KEEPS therapists found that greater numbers of EBPs delivered by a therapist was associated with higher levels of emotional exhaustion (Kim et al., 2018). In the current study, review of the qualitative data does not help narrow our interpretation of this finding, as there is little mention of emotional exhaustion in association with therapist discontinuation of EBPs. Future studies with prospective designs will be required to fully unpack this relationship between therapist emotional exhaustion and discontinuation of EBPs.

Quantitative analyses showed no significant link between agency size or therapist-reported perceptions of organizational climate and discontinuation. This is in contrast to previous work examining program leader perspectives on EBP sustainment, which found a higher probability of EBP sustainment in larger agencies (Rodriguez, Lau, Wright, Regan, & Brookman-Frazee, 2018). However, the quantitative analyses in the current study did show that therapists with fewer direct service hours per week were more likely to discontinue an EBP, suggesting that there may be a link between client base and discontinuation, and again highlighting the importance of EBP fit for sustainment. Therapists with a higher number of hours per week may have more opportunities to apply EBPs with clients eligible for them. These opportunities are critical for the sustainment of EBPs, as studies have shown that therapists who are able to utilize an EBP and receive feedback immediately after training are more likely to master the relevant skills (Beidas & Kendall, 2010; Miller, Yahne, Moyers, Martinez, & Pirritano, 2004).

Access to appropriate client populations also emerged from the qualitative analyses regarding EBP and clinical population mismatch. Therapists described bureaucratic or logistic challenges to continued use of certain PEI practices, especially difficulty in obtaining appropriate clients for specific EBPs. These findings are consistent with other results from this reimbursement-driven EBP implementation context, indicating that lack of steady demand from clients meeting criteria for specific EBPs is associated with discontinuation at the agency- (Rodriguez et al., 2018) and therapist-levels (Brookman-Frazee et al., 2018). Therapists who do not have the opportunity to use a particular EBP or who are only able to use it sporadically may tend to abandon it. In addition, continued frequent opportunities to use a specific EBP may contribute to therapist positive perceptions regarding the EBP and self-efficacy in administering the EBP, both of which also promote sustained use. These findings might also link to the quantitative link between more direct service hours and a lower likelihood of discontinuing an EBP. Therapists who have less opportunity to implement an EBP, and possibly see the long-term positive effects, may be more likely to discontinue using it.

Perceptions of organizational climate measured in the survey concerning performance feedback, decision-making autonomy, and involvement were not related to the probability of EBP discontinuation. It is possible that other unmeasured factors related to organizational culture and climate and implementation leadership may be more proximal to problems with sustainment. For example, positive perceptions of program leaders toward the EBPs has been found to predict EBP sustainment (Rodriguez, et al., 2018). In addition, organizational climate domains such as focus on EBP, educational support for EBP, recognition for EBP, rewards for EBP, selection for EBP, and selection for openness may influence the ongoing use of EBPs (Ehrhart, Aarons, & Farahnak, 2014). Indeed our qualitative findings suggest that fiscal support for EBP sustainment and leadership support for specific EBPs is a key organizational-level factor promoting continuation of therapist delivery even in the context of relatively centralized system-driven implementation effort. In particular, therapist perceptions of fiscal support is an important factor that bridges both the inner and outer context – perceptions of fiscal support at the agency level and perceptions of fiscal support across the broader county system both appear to impact sustainment of EBPs (Stewart et al., 2016).

Implications for Future Implementation Efforts

Collectively, these findings suggest that perceived EBP fit with client, therapist, and agency needs are a key factor in the long-term sustainment of EBPs. This is particularly critical in the context of multiple EBP implementation efforts, as strategies are needed to manage the myriad, diverse demands incurred by each individual EBP. Thus, along with building positive climate for implementation generally, strong sustainment leadership is needed to strategically select EBPs to maximize fit at the agency level to cover casemix needs, and strategic assignment of therapists to EBP training to support buy-in and retention. Information regarding an agency’s current clients, and who is “coverable” by certain EBPs based on factors such as age and presenting problems could help agency leaders select an array of EBPs that stand the best chance of long-term sustainment (Chorpita, Bernstien, & Daleiden, 2011).

Relatedly, there is evidence in the literature suggesting that longer intervals between EBP training and in vivo delivery of EBP to a client are related to lower adherence and poorer implementation outcomes (Herschell, et al., 2009). Treatment developers and trainers are becoming increasingly aware of these problems and attempting to address them in their dissemination models, noting that failure to solidify new knowledge obtained in training with practice experience causes trainees to lose learned skills (McNeil & Hembree-Kligin, 2010). As a result, several interventions require therapists to have a potential client who is eligible for a particular EBP before the beginning of training (e.g., Brookman-Frazee, et al., 2019). It may also be helpful for intervention developers and trainers to provide suggestions regarding community outreach and referral processes to ensure a relevant case mix. Sharing this knowledge may help agencies develop assessment and referral procedures that can properly support the implementation of specific EBPs, ensuring the trained clinicians receive appropriate clients (Bunger, Doogan, Hanson, & Birken, 2018; Nadeem, Weiss, Olin, Hoagwood, & Horwitz, 2016). Learning collaboratives, implementation resource teams, process control benchmarking, cascading training, and distance education may also be strategies to support these functions (Bunger et al., 2016; Herschell et al, 2015).

Limitations

Limitations of the study included the reliance on self-report of the predictor and the discontinuation variable within a cross-sectional study. Discontinuation was defined as the therapist reporting they had ever delivered at least one of the six EBPs of interest and also reporting that they had not used the selected EBP within the past two months. It is possible that the two-month time frame may not provide the best assessment of discontinuation, as it may tend to over-estimate discontinuation. For example, a therapist may intend to continue using an EBP but not have administered it in the past two months if they did not have an appropriate client on their caseload. In addition, the qualitative component of the measure required therapists to think retrospectively about why they discontinued a practice, which may add bias to the reporting (for example, rationalizing that the EBP was not a good fit if they stopped using it). Furthermore, there were relatively few interview comments on reasons for practice discontinuation, relatively few therapists in the interview sample discussed reasons for discontinuation even when their survey data indicated this had occurred. One contributing factor may be that the specific prompt occurred toward the end of the interview guide. Although, therapists sometimes discussed discontinuation in the sections of the interview concerning individual EBPs, fatigue may have been a factor contributing to less content elicited by this prompt near the end of the 45 minutes. An additional limitation is the context of this study – data was collected during a fiscally-mandated implementation of six EBPs within LACDMH. Though this context provides an opportunity to investigate certain questions regarding system-driven multiple EBP implementation, results may not generalize to other settings.

Conclusions

Notwithstanding these limitations, this study offers new contributions to the understanding of therapist- and agency-related factors in EBP discontinuation, and had the advantage of examining discontinuation across multiple EBPs. This is particularly critical as large-scale service systems make efforts to implement multiple EBPs into systems of care at significant public expense. Finally, analyses included a large sample size of therapists, across a range of practice contexts in the nation’s largest public mental health service system. Taken together, the findings again point to the importance of agency-level EBP fit to client needs in predicting EBP discontinuation. Specifically, access to appropriate clinical populations for specific EBPs and more positive therapist perceptions of an EBP significantly contributed to sustainment. These findings have significant implications for both organizational providers and policy developers as system-driven EBP implementation efforts evolve.