Employing Survival Analysis of Administrative Claims to Identify Prospective Predictors of Evidence-Based Practice Sustainment Versus Provider Turnover

Abstract

Objective:

This study described therapists’ delivery of six child mental health evidence-based practices (EBPs) over 33 months during the sustainment phase of a system-driven implementation aimed at improving access to EBPs in community settings.

Method:

Seven hundred seventy-seven therapists and 162 program leaders delivering at least one of six EBPs of interest completed surveys, and these data were matched to therapist administrative claims data. Survival analyses examined (a) therapists’ discontinuation of delivery of all Los Angeles County Department of Mental Health direct client services (i.e., turnover) and (b) therapists’ discontinuation of a specific EBP while continuing to provide psychotherapy services.

Results:

A majority of therapists (58.3%) were observed to discontinue providing any direct client services over the 33-month period, and discontinuation rates for specific EBPs ranged from 100% of therapists (Cognitive Behavioral Intervention for Trauma in Schools) to 76.4% (Managing and Adapting Practice). Different factors predicted the discontinuation of all service delivery compared to the discontinuation of a specific EBP. Greater therapist emotional exhaustion predicted higher likelihood of discontinuation of all direct client services. For EBP discontinuation, EBP-level factors (e.g., EBPs that required ongoing consultation), therapist-level factors (e.g., ability to provide services in a language other than English), and agency-level factors (e.g., more positive program leader perceptions of the EBP) predicted lower likelihood of discontinuation of the specific EBP.

Conclusions:

Different factors contribute to the risk of therapists leaving the service system versus discontinuing the delivery of specific EBPs.

Large-scale multiple evidence-based practice (EBP) implementation initiatives are an increasingly common approach designed to improve the quality of children’s mental health services in routine care settings (Hoagwood et al., 2014). Though these efforts involve large investments in time and capital, they often do not result in the sustained delivery of the EBPs following the initial trainings (Roundfield & Lang, 2017). Difficulty with therapist retention is a key impediment to EBP sustainment (Swales et al., 2012), with annual turnover rates ranging from 30% to 60% in community mental health (Beidas et al., 2016). However, few rigorous studies using prospective and multimethod data exist to understand predictors of therapist turnover and sustained delivery of EBPs (Proctor et al., 2015; Scheirer & Dearing, 2011). The present study builds on prior work to examine prospective predictors of therapist-sustained EBP delivery and therapist turnover within a system-driven implementation effort in Los Angeles County using both survey data and administrative claims data.

Context: A Large-Scale Multiple EBP Implementation Initiative in Los Angeles County

In 2009, the Los Angeles County Department of Mental Health (LACDMH) launched a large-scale implementation of multiple child EBPs in Los Angeles County through the California Prevention and Early Intervention (PEI) program (LACDMH, 2009). Starting in 2010, LACDMH supported the initial training and consultation in six evidence-based/informed practices for children and adolescents. Therapists working at LACDMH could bill the PEI program for fiscal reimbursement for delivery of these EBPs to eligible clients. This new reimbursement stream was particularly important in 2010, as it allowed mental health agencies to maintain levels of service delivery despite the state budget crisis (Regan et al., 2017). Over time, non-PEI sources of reimbursement have been restored as the state economy improved. When data collection for the present study began in 2015, routine sources of reimbursement were reestablished, and agencies and therapists were able to bill for services through PEI and non-PEI reimbursement streams that did not require delivery of EBPs.

Description of PEI EBP Sustainment in Los Angeles County

The Knowledge Exchange on Evidence-Based Practice Sustainment (4KEEPS) study examined the sustained delivery of EBPs in Los Angeles County several years after the initial implementation (Lau & Brookman-Frazee, 2016). Defined as the successful continuation of an EBP following training and implementation, EBP sustainment has been assessed with multiple methods in the 4KEEPS study, including administrative claims data as well as therapist- and program leader-report surveys (Swerissen & Crisp, 2004). An initial examination of administrative claims data in the 4KEEPS study found an overall ramp-up of EBP claims following the introduction of PEI in 2009–2010 but found a decrease in the volume of delivery of EBPs over time (Brookman-Frazee et al., 2016). In addition, the average length of time that a therapist delivered any EBP was 22 months, indicating that the majority of therapists were not continuing to deliver the EBP beyond 2 years (Brookman-Frazee et al., 2018). These findings were consistent with findings from other large-scale implementation initiatives, which showed that the delivery of EBPs tended to decrease following the initial training and implementation effort (Greenhalgh et al., 2012; Hunter et al., 2015; Swales et al., 2012).

Conceptual Framework of Predictors of EBP Sustainment

As theorized by the Exploration, Preparation, Implementation, Sustainment (EPIS) framework (Aarons et al., 2011; Moullin et al., 2019), EBP sustainment is impacted by factors in both the inner context and the outer context. In the inner context, EPIS highlights the importance of variations at multiple levels in contributing to the implementation and sustainment of EBPs, including the EBP, client, therapist, agency, and system levels (Aarons et al., 2011). Consistent with the EPIS framework, the present study sought to examine how variations of EBP-level, therapist-level, and agency-level factors predicted therapists’ continued delivery of specific EBPs during the sustainment phase of a large-scale multiple-EBP implementation initiative.

EBP Factors Related to Sustainment

Specific attributes or characteristics of the EBPs being implemented have been hypothesized to influence the successful adoption and the sustained delivery of a new practice (Damschroder & Hagedorn, 2011). These characteristics, such as a required consultation or a structured content/order, are properties of the EBPs themselves and are evidenced in the EBP’s implementation requirements and intervention materials, respectively. In the 4KEEPS study, interviews with program leaders highlighted the importance of ongoing consultation supports following the initial formal EBP training in promoting the program leader-reported sustainment of an EBP (Rodriguez et al., 2018). Additionally, EBPs with required consultation have been found to be linked with higher levels of therapist EBP administrative claim volume (Lau et al., 2022). Therapists in the 4KEEPS study have also reported more positive perceptions of EBPs that required ongoing consultation, which may contribute to increased sustained delivery (Barnett et al., 2017). Taken together, this suggests that EBPs requiring ongoing consultation may contribute to improved sustainment.

Structured content/order is another quality of specific EBPs that may influence sustainment outcomes. Data from the 4KEEPS study show that EBPs with a structured content/order were associated with higher levels of future therapist EBP administrative claim volume (Lau et al., 2022) and that therapists reported more positive perceptions of EBPs with a prescribed session content and order (Barnett et al., 2017). In contrast, however, other studies have found that EBPs without structured content or order may be more likely to be sustained as they allow for greater flexibility (Han & Weiss, 2005; Palinkas et al., 2013).

Therapist Factors Related to Sustainment

Previous examination of 4KEEPS administrative claims data has identified several therapist characteristics linked to the discontinued use of an EBP (Brookman-Frazee et al., 2016). These included therapist discipline (marriage and family therapists being less likely to discontinue than other disciplines), therapist primary language (Spanish speaking being less likely to discontinue than English speaking), and therapist race/ethnicity (non-Hispanic minority therapists being less likely to discontinue than non-Hispanic White therapists; Brookman-Frazee et al., 2018). In addition, therapists who claimed for a greater number of EBPs were more likely to discontinue the use of an EBP (Brookman-Frazee et al., 2018). In examining therapist-reported discontinuation of an EBP, factors such as fewer direct service hours per week, a greater number of EBPs adopted, higher emotional exhaustion, and more negative attitudes toward EBPs in general were associated with the discontinuation (Lau et al., 2022). Therapist factors such as higher therapist self-efficacy with the EBP were also linked to higher future volume of administrative claiming for that EBP (Lau et al., 2022). Findings of higher provider self-efficacy and increased EBP sustainment have also been reported in other studies of EBP implementation (Rohrbach et al., 2006; Scheirer, 2005).

Organizational/Agency Factors Related to Sustainment

Mixed methods analyses of program leader attitudes in the 4KEEPS study showed that more positive leader perceptions of a specific EBP were related to leader-reported sustainment of that EBP (Rodriguez et al., 2018). This trend has also been found in other studies examining the implementation of EBPs, with program leader perceptions of the EBP being an important predictor of the EBP sustainability (Swain et al., 2010; Tibbits et al., 2010). Taken together, this evidence is consistent with the theory that leadership plays a key role in developing an organizational climate that promotes the sustainment of EBPs. Other factors, such as lower levels of organizational stress and provision of regular feedback to therapists about their performance, have also been linked with sustained delivery of EBPs (Lau et al., 2022; Tommeraas & Ogden, 2017).

Exploring the Role of Therapist Turnover in the Discontinuation of EBPs

A significant limitation in previous studies is the inability to address the question of therapist turnover as a factor in EBP sustainment. Therapist turnover presents significant challenges to the sustained delivery of an EBP, as the average publicly funded mental health agency loses 30%–60% of its workforce annually (Beidas et al., 2016; Travis et al., 2016). In addition to the costs associated with recruiting and hiring new therapists, training new therapists in an EBP can be time-consuming and expensive (Saldana et al., 2014; Sigel et al., 2013). As with the sustained delivery of EBPs, therapist turnover can be influenced by both outer context factors, such as service context and funding, and inner context factors, such as therapist emotional exhaustion and organizational climate (Green et al., 2013). Many factors, such as therapist emotional exhaustion, are related to both therapist turnover (Morse et al., 2012) and sustained delivery of EBPs in publicly funded mental health systems (Lau et al., 2020).

The Present Study

In developing a full understanding of therapist EBP sustainment, it is critical to differentiate factors that contribute to a therapist leaving their workplace and service system versus factors that contribute to therapists ending delivery of an EBP but continuing to deliver psychotherapy services. The present study addresses this issue by matching therapist survey data with 33 months of administrative claims, which includes any fiscal reimbursement requests made through LACDMH agencies. Therefore, it is possible to determine whether therapists discontinued providing any services through LACDMH or whether they discontinued claiming to a specific EBP but continued providing psychotherapy.

This study has two primary aims. First, we sought to describe patterns of EBP discontinuation over 33 months using administration claims data. This involved examining (a) therapist discontinuation of all LACDMH direct client services and (b) therapist discontinuation of a specific EBP while continuing to provide psychotherapy within the LACDMH system. Second, we sought to identify prospective predictors of (a) therapist discontinuation of all LACDMH direct client services and (b) discontinuation of a specific EBP. Consistent with the EPIS framework, we hypothesized that factors at multiple levels would be associated with a therapists’ discontinuation of a specific EBP. Specifically, we hypothesized that EBP factors (e.g., required ongoing consultation, more positive therapist and program leader perceptions of specific EBPs), therapist factors (e.g., lower workload, lower emotional exhaustion, and more openness and less divergence with EBPs in general), and agency factors (e.g., higher staff autonomy, higher performance feedback, lower organizational stress, and higher organizational cohesion) would be associated with lower levels of discontinuation of a specific EBP. These findings will extend the existing literature by (a) examining sustainment in the context of a multiple EBP implementation effort; (b) using a prospective design to assess the associations between inner context conditions at the EBP, therapist, and organizational levels and sustainment; and (c) unpacking the differences in the predictors of therapist discontinuation of all LACDMH direct client services compared with the predictors of discontinuation of a specific EBP.

Method

Procedure

The present study took place during the sustainment phase of a large-scale multiple EBP implementation effort in Los Angeles County (see Lau & Brookman-Frazee, 2016, for details). It utilized two data sources: (a) survey data collected in 2015 from therapists and program leaders delivering PEI EBPs and (b) administrative claims data for participating therapists extracted for the 33 months following survey completion (April 2015 through December 2017; see Figure 1). All therapists and program leaders worked at agencies that were administered by LACDMH or contracted with LACDMH to provide services. The six EBPs that were initially supported by LACDMH were of interest in this study: Cognitive Behavioral Intervention for Trauma in Schools (CBITS), Child–Parent Psychotherapy (CPP), Managing and Adapting Practice (MAP),¹ Seeking Safety (SS), Trauma-Focused Cognitive Behavior Therapy (TF-CBT), and TP. The procedures for this study were approved by institutional review boards at both the University of California, Los Angeles and Los Angeles Department of Mental Health.

Figure 1. Timeline of PEI Initiative and Study Data Collection.

Note. Data collection for the present study is indicated in black. Therapists were eligible to participate in the study if they had been trained in and delivered at least one of the PEI EBPs at some point before March 2015, though they could have received training at any time from 2010 through recruitment. PEI = Prevention and Early Intervention; EBP = evidence-based practice.

Survey Data

Survey data collection occurred between March and July 2015, approximately 6 years after the initial implementation of the PEI initiative in 2009. Therapists were eligible to participate in the study if they had been trained in and delivered at least one of the PEI EBPs at some point before March 2015, though they could have received training at any time from 2010 through recruitment (see Figure 1). Program leaders were eligible if they provided clinical or administrative oversight for one or more of the six PEI EBPs in the same period. In response to requests from the research team, contact information for eligible participants was provided by leaders of 98 agencies in LACDMH. Eligible therapists and program leaders were emailed an invitation to participate in the survey, and they completed informed consent and survey items online. One hundred sixty program leaders and 777 therapists completed the surveys. Participants received a $20 gift card for completing the survey and an additional $20 gift card if they completed it within the first 2 weeks of receiving the study invitation. The survey had a response rate of 44.2% for therapists and 60.7% for program leaders, similar to previous surveys of community mental health therapists (Cashel, 2002; Hawley et al., 2009).

Administrative Claims Data

Administrative claims data submitted between April 2015 and December 2017 for each therapist survey participant were extracted from LACDMH records. Psychotherapy claims included information about each therapist–client encounter (e.g., procedure code, youth diagnosis, funding source). In addition, each psychotherapy claim included an “EBP code” field, in which LACDMH required therapists to report if they used an EBP, regardless of the funding source. For PEI reimbursement in Los Angeles County, therapists were required to deliver an EBP and indicate which EBP was delivered in this EBP code field (LACDMH, 2016). In the current sample, 95% of PEI claims reported a specific EBP in the EBP code field (e.g., TF-CBT), and 5% of non-PEI claims reported a specific EBP in this field. Claims were submitted at the time of service delivery. Claims could be made for any publicly funded source such as Medicaid or PEI (excluding private insurance). For analyses, survey data were matched to administrative claims data by therapists’ National Provider Identification numbers. A total of 606,374 psychotherapy claims for children 21 years old and younger at the time of service delivery were available for analysis. PEI EBP claims accounted for 41.2% of all psychotherapy claims made by therapists in this study.

Participants

Survey data from the 777 therapists were available and were matched with administrative claims data. On average, 11.8 therapists (SD = 14.8) from each agency participated in the study. Survey data from 162 program leaders were available and were aggregated to the agency level to capture organizational characteristics. Therapist and program leader demographic, training, and service delivery characteristics are included in Table 1.

Table 1.

Descriptive Statistics for Therapist and Program Leader Characteristics

Characteristic	Therapist (n = 777)	Program leader (n = 162)
Gender, n (%)
Female	685 (88.2)	136 (84.0)
Male	92 (11.8)	26 (16.0)
Race/ethnicity, n (%)
African American	52 (6.7)	13 (8.0)
Asian American/Pacific Islander	91 (11.7)	21 (13.0)
Hispanic	332 (42.7)	43 (26.5)
Non-Hispanic White	274 (35.3)	80 (49.4)
Multiracial/other	28 (3.6)	5 (3.1)
Education, n (%)
Less than master’s degree	18 (2.3)	0 (0)
Master’s degree	668 (86.0)	136 (84.0)
Doctoral degree	91 (11.7)	26 (16.0)
Licensure, n (%)
Licensed	351 (45.2)	153 (94.4)
Not licensed	336 (54.8)	9 (5.6)
Discipline, n (%)
Marriage and family therapy	431 (55.5)	85 (52.5)
Psychology/counseling	101 (13.0)	20 (12.3)
Social work	231 (29.7)	57 (35.2)
Other	14 (1.8)	0 (0)
Language, n (%)
Can deliver services in English only	349 (44.9)	99 (61.1)
Can deliver services in Spanish	361 (46.5)	45 (27.7)
Can deliver services in other non-English languagea	67 (8.6)	20 (12.2)
Primary setting,b n (%)
Outpatient clinic	615 (79.2)	134 (82.7)
School	81 (10.4)	52 (32.1)
Community/other	81 (10.4)	72 (44.4)
Theoretical orientation, n (%)
Cognitive behavioral or behavioral	446 (57.4)	67 (41.4)
Not cognitive behavioral or behavioral	331 (42.6)	95 (58.6)
Years of professional experience
M (SD)	6.8 (6.1)	12.7 (6.9)
Range	0–45	0–35
Number of the six EBPs therapist trained in
M (SD)	2.4 (1.1)
Range	1–6

Note. EBP = evidence-based practice.

^aOther languages included Korean, Mandarin, Cantonese, Vietnamese, Armenian, and French.

^bProgram leaders could select more than one setting for their program.

Measures

Outcome Variables

Therapist Discontinuation of All LACDMH Direct Client Services.

A discontinue event was coded from the administrative claims data if a therapist discontinued delivery of direct client services through all funding streams by the final 3 months of the time period. Because all of the therapist’s administrative data were available and indexed by National Provider Identifier, it was possible to evaluate service delivery at any LACDMH agency, even if they moved agencies during the time period. In addition, a therapist’s length of service provision (i.e., survival time) was calculated from the administrative claims data for the entire study period (April 2015 through December 2017).

Therapist Discontinuation of a Specific PEI EBP While Continuing to Deliver Psychotherapy.

A discontinue event was coded from administrative claims if a therapist did not deliver a specific EBP during the final 3 months of the study period (April 2015 through December 2017) but continued to claim for psychotherapy services other than the specific EBP. In addition, the length of delivery (i.e., survival times) for each of the six EBPs per therapist was calculated from the administrative claims data for the entire study period. The specific date that therapists were trained in an EBP was not available. To participate in the study, therapists must have been trained in and delivered at least one of the PEI EBPs at some point before March 2015, though they could have received training at any time from 2010 through recruitment.

Predictor Variables: EBP-Level Factors

EBP Structure: Prescribed Session Content/Order.

In accordance with Barnett et al.’s (2017) classification, CBITS (Jaycox, 2003), TF-CBT (Cohen et al., 2017), and TP (Turner et al., 2002) were classified as having prescribed session content and intended order of sessions and were coded as 1. The remaining three EBPs, SS (Najavits, 2001), CPP (Lieberman et al., 2015), and MAP (Chorpita et al., 2014), were determined to have more flexibility in the selection and ordering of therapy content and activities across an episode of treatment and were coded as 0. See Appendix A for more details.

EBP Training Requirements: Consultation.

The six EBPs were also classified based on their requirements for ongoing consultation following the initial training workshops. These implementation requirements were established by LACDMH in consultation with the EBP developers (LACDMH, 2016). Though the length and format of the required consultation varied, CBITS, CPP, MAP, and TF-CBT all required ongoing consultation and were coded as 1. In contrast, SS and TP did not require ongoing consultation and were coded as 0. See Appendix B for more details.

Therapist Perceptions of the EBP.

Therapists were asked to complete an adapted version of the Perceived Characteristics of Intervention Scale (PCIS; Cook et al., 2015) for each of the EBPs they had used with a client. Eight PCIS items were retained, assessing therapist’s perceptions of the EBP’s Relative Advantage, the EBP’s Compatibility, the EBP’s Complexity, and the EBP’s Potential for Reinvention. These items were retained due to their relevance for continued use of an EBP. Therapists rated each item on a 5-point Likert scale (1 = not at all, 5 = a very great extent). A single composite was computed from the mean of the eight items. Internal consistencies were high (α = .93–.94 across EBPs).

Predictor Variables: Therapist Factors

Therapist Demographic, Professional, and Practice Characteristics.

Therapists reported on their gender, race/ethnicity, language(s) of clinical service delivery, average number of direct service hours per week, and the number of years of professional practice on the survey. For each EBP, therapists reported whether they had received training in it, had ever delivered the EBP to a client, and whether they had delivered the EBP to a client within the last 2 months.

Therapist Emotional Exhaustion.

Therapist emotional exhaustion, or the extent to which therapists felt that their emotional resources had been depleted (Maslach & Jackson, 1981), was measured with the Organizational Social Context Questionnaire (Glisson et al., 2008). Therapists rated how their work affected them and their emotional exhaustion related to their work in general, with higher ratings indicating greater emotional exhaustion. Therapists completed five items on a 7-point Likert scale (1 = strongly disagree, 7 = strongly agree), and a total score was calculated by taking the mean of the items. Internal consistency was high (α = .81).

Therapist General Attitudes Toward EBPs.

Therapist attitudes toward the adoption EBPs in general were measured using the Openness and the Divergence subscales of the Evidence-Based Practice Attitude Scale (Aarons et al., 2012). The four-item Openness subscale assessed a therapist’s openness to new practices (α = .79), and the four-item Divergence subscale assessed the degree to which a therapist’s usual practice diverges with research-based/academically developed interventions (α = .71). Items were rated on a 5-point Likert scale (0 = not at all, 4 = very great extent), and subscale scores were calculated by taking the mean of the items.

Predictor Variables: Agency Level

Program Leader Perceptions of EBP.

Program leaders completed the same adapted version as therapists of the PCIS for each of the EBPs that they had supervised the implementation in their agency (see the Predictor Variables: Therapist Factors section for additional details). Internal consistencies were high (α = .91–.97 across EBPs), and the total score was calculated by taking the mean of the eight items. Program leaders’ PCIS scores corresponding to the EBP for which the therapist claimed were aggregated to the agency level by calculating the mean score for the agency.

Organizational Climate.

Therapist and program leader perceptions of their agency’s climate were measured with the Organizational Climate Measure (Forgatch et al., 2005). Therapists and program leaders completed the Autonomy subscale, which measured the extent to which they perceived employees were given a latitude to enact their work, and the Performance Feedback subscale, which assessed perceptions regarding the measurement and feedback of job performance. Items were answered using a 4-point Likert scale (1 = definitely false, 4 = definitely true), and subscale scores were calculated by taking the mean of the items. Internal consistencies were high (Autonomy: α = .72 for therapists, α = .80 for program leaders; Performance Feedback: α = .85 for therapists, α = .84 for program leaders). Therapist and program leader reports were not significantly correlated for Autonomy (r = .18, p = .18) or Performance Feedback (r = .15, p = .25). Scores were aggregated to the agency level by calculating the mean score for the agency.

Organizational Readiness for Change.

Program leaders completed three subscales of the Organizational Readiness for Change (Lehman et al., 2002) questionnaire, which measured leader perspectives on organizational climate and resources supporting the EBP implementation. Program leader (vs. therapist) report was selected for these constructs, as it was believed that they would have the most knowledge of overall organizational context. The Staffing subscale examines perceptions of workforce quality and capacity. The Cohesion subscale examines perceptions of group cohesion at the agency. The Stress subscale measures perceptions of stress, workplace burden, and staff strain. Program leaders rated their agreement with items on a 5-point Likert scale (1 = disagree strongly, 5 = agree strongly), and subscale scores were calculated by taking the mean of the items. Internal consistencies were as follows: Staffing: α = .64, Cohesion: α = .76, and Stress: α = .84. Program leader ratings were aggregated to the agency level by calculating the mean score for the agency.

Data Analytic Plan

The first aim of this study was to describe patterns of (a) therapist discontinuation of all LACDMH direct client services and (b) therapist discontinuation of a specific EBP. For Aim 1a, the mean and median lengths of therapist service delivery (i.e., survival time) were calculated from the months between a therapist’s first claim for any service to their last claim for any service within the study time frame (April 2015 through December 2017). For Aim 1b, survival time of a specific EBP was determined by calculating the months between a therapist’s first claim for a specific EBP to their last claim for a specific EBP within the study time frame (April 2015 through December 2017), and differences in survival time between the EBPs were examined using a log-rank test of survival function equality.

Aim 2a sought to determine the unique contribution of each predictor to therapists’ discontinuation of all LACDMH direct client services using a Cox regression survival model. The outcome was a dichotomous representation of a cessation of all LACDMH service delivery (i.e., a failure) or sustainment of service delivery (i.e., right censorship). For Aim 2b, predictors of therapist discontinuation of a specific EBP were examined using a multilevel mixed-effects survival model, with EBPs nested within therapists. This model included a subset of therapists who either discontinued the delivery of a specific EBP but continued claiming for psychotherapy other than that specific EBP (i.e., a failure) or sustained the delivery of a specific EBP (i.e., right censorship). EBPs were modeled at Level 1, and therapists were modeled at Level 2. For both of the survival models in Aim 2, hazard ratios (HRs) were calculated, indicating how high the risk of discontinuing delivery is for a therapist in one group compared to a therapist in another group for categorical predictors. For continuous predictors, hazard ratio indicates a change in the risk of discontinuing delivery if the predictor of interest is increased by one unit. All analyses were performed in Stata/SE (15.1).

Results

Aim 1a: Describe Therapist Discontinuation of All Direct Client Service Delivery Within the LACDMH System

These analyses followed therapists who had been trained in and delivered at least one of the PEI EBPs at some point before March 2015, though they could have received training at any time from 2010 through recruitment. Of the 777 therapists in the sample, 453 (58.3%) discontinued providing all direct client services within the LACDMH system by the end of the 33-month time period (from April 2015 through December 2017; see Table 2). On average, therapists delivered direct client services for 24.1 months during the study time frame (SD = 10.4, Mdn = 29.7, range = 0.2–33.0).

Table 2.

Descriptives of Survival Time for Six PEI EBPs of Interest From April 2015 to December 2017 for Participating Therapists

		Censoreda: Continued specific PEI EBP claiming	Eventb: Discontinued specific PEI EBP claiming	Eventb: Discontinued specific PEI EBP claiming, continued claiming for psychotherapy
Evidence-based practice	Total therapists trained or used	n (%)	n (%)	n (%)	Mean survival time (mos.)	Median survival time (mos.)	Min. (mos.)	Max. (mos.)
Child-Parent Psychotherapy	142	14 (9.9)	128 (90.1)	37 (26.1)	15.4	13.5	.1	32.9
Cognitive Behavioral Intervention for Trauma in Schools	70	0 (0)	70 (100)	22 (31.4)	0.1	0.1	.1	0.1
Managing and Adapting Practice	522	123 (23.6)	399 (76.4)	112 (21.5)	19.3	20.7	.2	33.0
Seeking Safety	500	20 (4.0)	480 (96.0)	175 (35.0)	10.9	7.1	.03	32.9
Trauma-Focused Cognitive Behavior Therapy	580	84 (14.5)	496 (85.5)	154 (26.6)	17.1	15.2	.1	33.0
Positive Parenting Program	190	25 (13.2)	165 (86.8)	62 (32.6)	17.2	16.1	.5	33.0
All LACDMH direct client services/claims	777	324 (41.7)	453 (58.3)		24.1	29.7	.2	33.0

Note. PEI = Prevention and Early Intervention; EBP = evidence-based practice; mos. = months; Min. = minimum; Max. = maximum; LACDMH = Los Angeles County Department of Mental Health.

^aCensored represents the number of therapists who continued to deliver a specific EBP during the final fiscal quarter. October through December 2017.

^bEvents represent the number of therapists who discontinued delivery of a specific EBP (i.e.. made no claims for the specific EBP) during the final fiscal quarter. October through December 2017.

Aim 1b: Describe Therapist Discontinuation of a Specific PEI EBP While Continuing to Claim for Psychotherapy Within the LACDMH System

Using the same sample of 777 therapists from Aim 1a, therapist discontinuation rates for specific EBPs while continuing to claim for psychotherapy were examined. Discontinuation rates for specific EBPs per therapist ranged from 100% (CBITS) to 76.4% (MAP). The mean and median survival times for each of the six EBPs of interest are depicted in Table 2. On average, over the 33-month study period, therapists delivered an EBP for 16.6 months (SD = 11.0, Mdn = 14.2, range = 0.03–33.0). In addition, Figure 2 shows the Kaplan–Meier survival functions for therapist delivery of each of the EBPs. A log-rank test of survival function equality revealed significant differences across the six EBPs (χ² = 92.9, df = 5, p < .001). Follow-up log-rank tests were performed to identify the specific EBPs that were different from the rest by comparing the survival curve of each practice to the combined survival curve of the five other EBPs. Results showed no significant differences between the survival curves for CPP (χ² = 2.8, df = 1, p = .09), TF-CBT (χ² = 2.6, df = 1, p = .11), and TP (χ² = 1.5, df = 1, p = .22) delivery and the survival curve of the other EBPs. In contrast, CBITS (χ² = 5.2,df = 1, p < .001) and SS (χ² = 69.2, df = 1, p < .001) had a significantly higher risk of delivery discontinuation than the other EBPs, and MAP (χ² = 35.0, df = 1, p < .001) had a significantly lower risk of delivery discontinuation than the other EBPs.

Figure 2. Cumulative Kaplan–Meier Survival Functions for Therapist Delivery of Each PEI EBP.

Note. This figure displays the survival (or continuation) rates for each EBP. PEI = Prevention and Early Intervention; EBP = evidence-based practice; CBITS = Cognitive Behavioral Intervention for Trauma in Schools; CPP = Child–Parent Psychotherapy; MAP = Managing and Adapting Practice; SS = Seeking Safety; TF-CBT = Trauma-Focused Cognitive Behavior Therapy; TP = Positive Parenting Program. See the online article for the color version of this figure.

Aim 2a: Prospective Predictors of Therapist Discontinuation of All Direct Client Service Delivery Within the LACDMH System

Table 3 displays the results of a multivariate Cox regression model showing the unique contribution of each predictor variable to the therapists’ discontinuation of all LACDMH direct client service delivery. Several therapist and agency factors were examined as possible prospective predictors. Therapists who reported that their usual practice tended to more greatly diverge from research-based/academically developed interventions, or EBPs, were more likely to discontinue delivery of all LACDMH direct client services (HR = 1.29; standard error, SE = 0.15; p = .02). In addition, higher levels of therapist emotional exhaustion were significantly linked to a greater likelihood of discontinuation of all LACDMH direct client services (HR = 1.27, SE = 0.08, p < .001).

Table 3.

Survival Models Predicting Therapist Discontinuation of All Services and Discontinuation of Specific PEI EBPs

Variable	Therapist discontinuation of all LACDMH direct client servicesa (from any funding type)	Therapist discontinuation of specific PEI EBPb (therapist who continued claiming for psychotherapy within LACDMH system but not PEI EBPs)
	Hazard ratio (SE)	Hazard ratio (SE)
EBP factors
EBP requires consultationc		0.38 (.06)**
EBP has structured content/orderd		1.37 (.20)*
Therapist perceptions of specific EBP (PCIS)		0.84 (.07)*
Therapist factors
Therapist gender	0.87 (.17)	1.14 (.28)
Therapist race/ethnicitye
Hispanic	0.75 (.12)	1.30 (.32)
Other minority	0.74 (.12)	1.11 (.25)
Therapist deliver services in language other than Englishf	0.99 (.15)	0.65 (.14)*
Therapist years practiced	0.99 (.01)	0.99 (.02)
Therapist direct service hours per week	0.99 (.01)	0.97 (.01)*
Therapist number of EBPs trained in	1.01 (.06)	1.10 (.10)
Therapist general attitudes toward EBPs (EBPAS)
Openness	1.04 (.12)	1.12 (.18)
Divergence	1.29 (.15)*	0.80 (.11)
Therapist emotional exhaustiong	1.27 (.08)**	0.94 (.08)
Agency factors
Program leader perceptions of specific EBP (PCIS)		0.63 (.11)*
Therapist perceptions of organizational climate (OCM)
Autonomy	0.98 (.29)	0.74 (.31)
Performance feedback	0.59 (.14)*	1.41 (.49)
Program leader perceptions of organizational climate (OCM)
Autonomy	0.71 (.16)	2.89 (.89)**
Performance feedback	0.83 (.19)	0.70 (.23)
Program leader perceptions of organizational readiness for change (ORC)
Staffing	1.12 (.24)	0.63 (.19)
Cohesion	1.01 (.24)	1.19 (.41)
Stress	0.97 (.11)	1.24 (.19)

Note. PEI = Prevention and Early Intervention; EBP = evidence-based practice; LACDMH = Los Angeles County Department of Mental Health; SE = standard error; PCIS = Perceived Characteristics of Intervention Scale; EBPAS = Evidence-Based Practice Attitudes Scale; OCM = Organizational Climate Measure; ORC = Organizational Readiness for Change.

^aSingle-level Cox regression survival model predicting therapist discontinuation of all services.

^bMultilevel mixed-effects survival model predicting therapist discontinuation of specific EBP while continuing to claim for services other than specific EBP.

^cEBP required consultation was coded as 0 = EBP did not have required consultation, 1 = EBP had required consultation.

^dEBP prescribed session content/order was coded as 0 = EBP does not have prescribed session content/order, 1 = EBP has prescribed session content/order.

^eRace/ethnicity reference group = non-Hispanic White.

^fTherapist language was coded as 0 = English only, 1 = non-English language.

^gTherapist emotional exhaustion was measured with the Organizational Social Context Questionnaire.

^*p < .05.

^**p < .01.

With regard to agency-level factors, therapist-reported perceptions of agency organization culture and climate were linked with discontinued service delivery, with higher levels of therapist perceptions of performance feedback associated with a lower likelihood of discontinuation of all LACDMH direct client services (HR = 0.59, SE = 0.14, p = .04). In contrast, program leader perceptions of agency culture and climate, including levels of performance feedback, were not significantly linked to therapist discontinued delivery of all LACDMH direct client services.

Aim 2b: Prospective Predictors of Therapist Discontinuation of a Specific EBP While Continuing to Claim for Psychotherapy Within the LACDMH System

Using a multilevel mixed-effects survival model, predictors at the EBP, therapist, and agency levels were explored (see Table 3). With regard to EBP-level variables, therapists were more likely to discontinue an EBP with a structured content/order compared with an EBP without a structured content/order (HR = 1.37, SE = 0.20, p = .047). In contrast, therapists were less likely to discontinue an EBP with required consultation compared with an EBP without required consultation (HR = 0.38, SE = 0.06, p < .001). Finally, therapists were less likely to discontinue when they had higher levels of positive perceptions of the EBP (HR = 0.84, SE = 0.07, p = .028).

Several therapist factors, including therapist language and number of direct service hours per week, were also associated with the prospective discontinuation of a specific EBP. Specifically, therapists who could deliver services in a language other than English were less likely to discontinue delivery of a specific EBP compared with therapists who could deliver services in English only (HR = 0.65, SE = 0.14, p = .037). Therapists who reported providing a greater number of direct service hours per week were also less likely to discontinue delivery of a specific EBP (HR = 0.97, SE = 0.01, p = .045).

At the agency level, more positive program leader perceptions of a specific EBP were associated with a decreased likelihood of therapists discontinuing the delivery of that EBP (HR = 0.63, SE = 0.11, p = .049). Program leader perceptions of agency climate were linked with therapist discontinuation of a specific EBP, with higher levels of program leader perceptions of autonomy associated with a higher likelihood of discontinuation of a specific EBP (HR = 2.89, SE = 0.89, p = .010). In contrast, therapist perceptions of agency culture and climate, including levels of autonomy, were not significantly linked to therapist discontinued delivery of a specific EBP.

Discussion

This study examined therapists’ delivery of child mental health services over 33 months during the sustainment phase of a system-wide, multiple EBP implementation effort and explored prospective EBP-, therapist-, and agency-level predictors of service delivery discontinuation. Importantly, this study sought to unpack the differences between (a) therapist discontinuation of all direct client services within the LACDMH system and (b) therapist discontinuation of a specific EBP while continuing to provide psychotherapy within the LACDMH system.

Discontinuation rates of all LACDMH direct services by the end of the 33-month period were high, with 58.3% of therapists discontinuing delivery of all services. This discontinuation was likely due to turnover as more than half of the therapists in the sample had left the county system as direct service providers. In addition, emotional exhaustion was associated with higher rates of therapists discontinuing delivery of all direct client services in the present study, which is consistent with the strong link between emotional exhaustion and turnover found in the literature (Weikel & Fisher, 2022; Willard-Grace et al., 2019).

This study offered the opportunity to compare the predictors of therapist turnover with therapist discontinuation of a specific EBP. Though therapist emotional exhaustion was a significant predictor of therapists leaving the LACDMH, emotional exhaustion was not significantly associated with the discontinuation of a specific EBP. Rather, different factors at multiple levels (EBP, therapist, and agency levels) were associated with therapist discontinuation of a specific EBP. Specifically, EBPs that are more flexible (rather than having prescribed session order and content) and that require ongoing consultation were less likely to be discontinued. Therapists with a more positive perception of a specific EBP, the ability to provide services in a language other than English, and who provided more direct service hours per week were less likely to discontinue claiming a specific EBP. In addition, EBPs were less likely to be discontinued when there were positive leader perceptions of the EBP at the agency level and when program leaders perceived lower levels of autonomy at the agency.

On the EBP level, EBPs with a required consultation were more likely to be sustained over time. Broadly, the importance of consultation for sustained EBP use is consistent with other studies linking consultation to continued EBP delivery (Swain et al., 2010) and more positive therapist perceptions of EBP (Barnett et al., 2017). The one notable exception to this was the rapid discontinuation of CBITS, which had low sustainment due to the poor fit between the outpatient setting in which the majority of the therapists worked and the fact that CBITS was designed to be delivered in schools (Brookman-Frazee et al., 2016). In addition, ongoing EBP fidelity monitoring has been associated with increased therapist tenure at an organization (Aarons et al., 2009). In contrast, the findings in the present study that EBPs without a prescribed content and order are more likely to be sustained must be placed in the broader context of mixed evidence. Other studies with the 4KEEPS population have shown that therapists have more positive perceptions of EBPs with a structured content and order (Barnett et al., 2017), and more positive perceptions are generally linked to improved sustainment (Aarons et al., 2009). However, it is possible that the less flexible EBPs with prescribed content and order may make it difficult for therapists to deliver an EBP in a way that fits with client needs over time, which is considered key for EBP sustainment (Rodriguez et al., 2018).

Therapist variables that were found to be linked to EBP sustainment were the number of direct service hours per week and a therapist’ ability to deliver services in a language other than English. There is some evidence that therapists with more clinical experience display higher competence in delivering a specific EBP (Henggeler et al., 2008; James et al., 2001). This could lead to higher levels of self-efficacy and positive perceptions of the EBP, which may contribute to sustained delivery. The finding that therapists who were able to deliver services in a language other than English were more likely to sustain delivery of a specific EBP is consistent with prior 4KEEPS findings regarding improved sustainment of any EBP during an early time period (Brookman-Frazee et al., 2018). Other research has shown that bilingual and Latinx therapists have positive experiences with EBP delivery and a high level of investment (Ramos et al., 2021; Rodriguez et al., 2021). It is possible that bilingual and Latinx therapists’ investment in time to translate and make adaptations to a particular EBP may help to explain increased sustainment.

With regard to agency-level factors, results from the present study suggest that program leaders’ perceptions of a specific EBP may contribute to therapists’ continued use of that EBP. In the current analyses, program leader perceptions of specific EBPs were averaged and aggregated to the agency level, thereby providing a variable that measured the general perception of leadership at an agency toward a specific EBP. These findings of significant associations with therapist-sustained EBP use were in contrast to the lack of significant findings regarding therapist attitudes toward specific EBPs and measures of overall agency organizational climate. This suggests that program leaders’ perceptions of specific EBPs may play a critical role in promoting a climate supportive of that EBP use (Aarons et al., 2016; Aarons & Sommerfeld, 2012). A climate of positive leader attitudes toward a specific EBP may also be important for other factors associated with therapist delivery of EBPs such as the amount that therapists use a specific EBP or the burden that therapists may experience (Brimhall et al., 2016).

Limitations and Future Directions

The use of administrative claims data provides both advantages and limitations. A strength of this method is that it does not rely on therapist retrospective report and that administrative claims are a by-product of everyday service provision. However, the information available from an administrative claim is limited—it only indicates whether the therapist submitted for reimbursement for a particular EBP for that visit, not the levels of fidelity, adherence, or quality indicators in the session. A therapist may claim to an EBP without having delivered components of the EBP with fidelity in any given session. Therefore, it is important to consider the limitations of administrative claims data when interpreting the results of the present study. Future studies would benefit from comparing therapist report of EBP delivery via administrative claims with multimethod indices of EBP delivery (e.g., observational coding).

In addition, multiple factors may influence whether a therapist claims a specific EBP for a particular client, including requirements for reimbursement. Prior to the data collection period for the present study, other non-PEI funding streams were restored to prerecession levels, and claims for reimbursement could be made to funding streams that did not require provision of EBPs and which did not have client PEI eligibility requirements (e.g., requirement to be in more early stages of illness vs. chronic care). Consistent with this, Lui et al. (2021) found PEI EBP claims decreased over the time period of 2009–2017, with a small increase in 2016–2017. In general, PEI EBP claims tended to decrease as state Medicaid mental health expenditure increased over time (Lui et al., 2021). Therefore, a therapist may have delivered an EBP for a child in a session but did not submit an PEI EBP-specific claim because of the child client’s eligibility for PEI and the availability of other funding sources. It is also possible that some EBPs may have been more likely to be reimbursed through other non-PEI funding streams (e.g., First Five). Therefore, it is possible that claims may underestimate therapists’ continued delivery of EBPs. However, PEI still comprised a significant portion of therapists’ reimbursement, with PEI EBP claims accounting for 41.2% of all psychotherapy claims made by therapists in this study during 2015–2017 and 95% of PEI claims reporting a specific EBP in the EBP code field. This suggests that therapists were still documenting the delivery of EBPs through PEI during the study period.

An additional limitation is that therapist turnover from service provision within the LACDMH system was indexed by the discontinuation of reimbursement claiming. This operational definition is a proxy for turnover and could reflect other circumstances. For example, a therapist may cease claiming because they assume a position that does not involve direct service (e.g., supervisory or management) or because they left LACDMH to provide direct services elsewhere (such as a publicly funded system in a neighboring county or private practice). Future studies would benefit from using additional methods to track a therapists’ exit from a service system, such as follow-up surveys or interviews. Given that these methods can be resource intensive, administrative claims data can provide valuable information regarding a therapist’s service delivery trajectory, though the limitations of this method should be kept in mind.

This study focuses on the sustainment phase of a system-wide implementation effort, with the PEI initiative being launched in 2009 and data collection for the present study occurring 6–8 years later in 2015–2017. Importantly, although the PEI initiative was in the sustainment phase during the study period, therapists continued to be trained in the PEI EBPs; thus, individual providers were very much still in the active implementation phase. Although some therapists in the present study had been in their positions since the beginning of the initiative in 2009, the majority likely joined the LACDMH workforce later. Survival time was calculated from the date of survey completion, and analyses were not able to fully account for time since EBP training. Future studies would benefit from having a more precise measurement of training timelines and entry into implementation initiatives.

The context of this study is limited to publicly funded child mental health services provided in Los Angeles County. Therefore, it is not clear whether the prospective predictors of sustained EBP delivery would generalize to other implementation contexts and service sectors (e.g., private practice mental health service settings accepting private insurance). In addition, the present study examines sustainment, but future work could adopt a broader lens to examine the scaling of EBPs within the LACDMH system (Fixsen et al., 2017; Karlin et al., 2012). Given the high rates of therapist turnover in the present study, scaling would require expanding the workforce able to deliver EBPs above and beyond replacing therapists who leave. Because the present study did not have data regarding the therapists replacing those leaving LACDMH, it was not possible to examine this question.

This study builds on previous cross-sectional and retrospective analyses that examined associations between therapist reports of EBP implementation experiences and self-reported continued or discontinued use of EBPs (Lau et al., 2020, 2022) and an earlier survival analyses of administrative claims in the first 5 years of the multiple EBP implementation initiative in Los Angeles County without prospective predictors at the EBP, therapist, and agency levels (Brookman-Frazee et al., 2018). In particular, this study highlights differences in prospective predictors of the discontinuation of all LACDMH service delivery (i.e., therapist turnover/attrition from the service system) and the discontinuation of a specific EBP when continuing to provide other direct services. With regard to the discontinuation of all LACDMH direct client services, higher therapist emotional exhaustion predicted higher likelihood of discontinuation of all LACDMH direct client services. In contrast, for EBP discontinuation, EBP-level factors (e.g., EBPs that required ongoing consultation), therapist-level factors (e.g., the ability to provide services in a language other than English, number of direct service hours provided per week), and agency-level factors (e.g., positive program leader perceptions of the EBP) predicted lower likelihood of the discontinuation of the specific EBP while continuing to claim for psychotherapy. These new findings differentiate between the general risk of exit from the service system and the discontinuation of EBP delivery to highlight potential targets for implementation strategies aimed at improving EBP sustainment in the increasingly common context of system-driven EBP implementation effort.

What is the public health significance of this article?

This study highlights targets for implementation supports to reduce therapist turnover and to reduce discontinuation of specific evidence-based practices. Supports for both of these will be necessary to improve the access to evidence-based practices in community settings.

Appendix A

Evidence-Based Practice Structure

Evidence-based practices (EBPs) were characterized based on whether or not they had a prescribed session content and an intended order of sessions. EBPs with prescribed session content and order were defined as those with explicit guidance in their manuals as to what content should be covered in which order. Based on guidance in their manuals, three of the EBPs, Cognitive Behavioral Intervention for Trauma in Schools (CBITS; Jaycox, 2003), Trauma-Focused Cognitive Behavior Therapy (TF-CBT; Cohen et al., 2017), and Positive Parenting Program (TP; Turner et al., 2002), were classified as having this structure. In contrast, Seeking Safety (Najavits, 2001), Child–Parent Psychotherapy (CPP; Lieberman et al., 2015), and Managing and Adapting Practice (MAP; Chorpita et al., 2014) were classified as having more flexibility in the selection and ordering of therapy content and activities across an episode of treatment. Following Barnett et al. (2017), EBPs were classified as having a prescribed session content and order (CBITS, TP, and TF-CBT) were coded as a 1, and the practices without a prescribed session content/order (Seeking Safety, CPP, and MAP) were coded as a 0.

Appendix B

EBP Training Requirements

EBPs were also characterized on whether or not there was mandatory follow-up consultation with an EBP expert trainer after the initial training. These implementation requirements were established by Los Angeles County Department of Mental Health in consultation with the EBP developers (Los Angeles County Department of Mental Health, 2016). Seeking Safety and TP did not require ongoing consultation whereas CBITS, CPP, MAP, and TF-CBT did, though the length and format of the required consultation varied. CBITS and TF-CBT required 13 and 16 consultation calls with external trainers, respectively. TF-CBT also required a review of two session recordings. CPP required 18 months of consultation calls with external trainers for both therapists and supervisors in the agency. Similarly, MAP required 6 months of ongoing consultation and submission of case-based portfolios. Following Barnett et al. (2017), the EBPs classified as having required consultation (CBITS, CPP, MAP, and TF-CBT) were coded as a 1, and those without required consultation (Seeking Safety and TP) were coded as a 0.