Measuring Sustainment of Multiple Practices Fiscally Mandated in Children’s Mental Health Services

Abstract

Examining sustainment of multiple evidence-based practices is crucial to understanding the processes and outcomes of system-driven implementation efforts that are increasingly common. This study used administrative claims data to characterize volume and penetration of six practices over 19 fiscal quarters following initial implementation within the context of a system-driven, fiscally mandated implementation effort in Los Angeles County Department of Mental Health’s Prevention and Early Intervention services. Patterns of volume changes over time revealed an overall ramp up of practice claims over time, but variability in patterns of volume and penetration for each practice. Findings varied by the methods used to index and analyze volume and penetration. Furthermore, a number of client case-mix and therapist characteristics were associated with the volume of therapists’ claims for each practice relative to their claims for the other practices.

Concerns about the effectiveness and quality of usual care (UC) (Garland et al., 2010; Weisz, Jensen-Doss, & Hawley, 2006) have prompted large-scale implementation efforts in state, county and city mental health (MH) systems (Hoagwood et al., 2014; McHugh & Barlow, 2010; Nakamura et al., 2011; Starin et al., 2014; Trupin & Kerns, 2015). By 2008, 90% of state MH authorities reported strategies to implement EBPs, 12 states had mandated the use of EBPs in public MH systems, with eight states promoting, supporting or requiring specific practices statewide (Cooper et al., 2008). Additionally, 10 state MH authorities had fiscal policies promoting EBP implementation through reimbursement practices. Mandating EBPs in public managed care has a long history in children’s MH services, beginning with the Empirical Basis to Services Task Force which reformed child MH services in Hawaii’s Department of Health following a 1999 consent decree (Chorpita et al., 2002). Federal agencies now require EBP use through requirements in SAMHSA Children’s Systems of Care and Administration for Children and Families program grants. These costly and large-scale efforts provide natural laboratories to understand implementation processes and outcomes.

System-driven and system-wide implementation efforts are likely to involve implementation of multiple EBPs in order to meet the range of patients who present for services (Chorpita & Daleiden, 2009; Chorpita, Bernstein, & Daleiden, 2011; Hoagwood et al., 2014). With few exceptions, however, most of what is known about implementation in public MH services is based on research on the implementation of single EBPs (Bond et al., 2014). Studying the simultaneous implementation of multiple EBPs is critically important to understanding and advancing the processes of implementation in system-wide efforts.

In addition to the focus on single EBP implementation, the majority of research has focused on what is necessary to achieve adoption of EBPs, with limited attention to what sustains the use of these practices (Greenhalgh, Robert, Macfarlane, Bate, & Kyriakidou, 2004; Stirman et al., 2012). The EPIS (Exploration, Preparation, Implementation, Sustainment) implementation framework (Aarons, Hurlburt, & Horwitz, 2011) delineates sustainment as a distinct phase of EBP implementation. In one of the few studies examining long-term sustainment within the context of implementation in routine care, Bond and colleagues (2014) examined whether sites continued to use one or more of five different psychosocial EBPs implemented in routine adult MH care settings in eight states. Results indicate that 47% of sites sustained delivery of an EBP after six years; the adequacy of funding was the most significant predictor of whether an agency sustained delivery of the EBP. While the Bond and colleagues (2014) study provides an important step towards understanding patterns of sustainment by MH systems, additional research is imperative, because costly, time-intensive implementation initiatives will have a limited public health impact if these practices are not maintained over time (Rhoades, Bumbarger, & Moore, 2012; Willging, Green, Gunderson, Chaffin, & Aarons, 2015).

Furthermore, although data-driven approaches to system coordination of multiple EBPs emphasize relevance mapping (Chorpita, Bernstein, & Daleiden, 2001), consumer demand for EBPs remains an unstudied predictor of sustainment. One approach to understanding consumer factors in sustainment is to assess provider case load characteristics over time, to examine how sustainment maps on to actual case-mix characteristics.

The research on sustainment conducted to date indicates that sustainment outcomes vary based on how and when sustainment outcomes are assessed. Based on a review of 125 studies examining sustainment, Stirman and colleagues (2012) highlighted the need to develop measures of multiple sustainment outcomes, and assess these outcomes over multiple years rather than at a single time point. Penetration refers to the integration of a practice within a service setting and can be measured at multiple levels (e.g., units of service, clients, providers, agencies) as the number of eligible individuals who use/deliver a treatment, divided by the number of eligible individuals (Proctor et al., 2011). An important limitation of this outcome is that indexing a penetration ratio alone does not track the absolute scale of an implementation effort over time. For example, although it is important to track the proportion of eligible therapists who continue to deliver an intervention, it is equally informative to assess the total impact in terms of the numbers of agencies, therapists and clients involved. A penetration rate of 30% in a very large implementation effort affecting 1000 therapists is arguably more impactful than a penetration rate of 80% in a modest initiative involving 50 therapists. Moreover, when studying sustainment of practices prospectively in systems of care, the denominators used to calculate penetration rates vary over time with changes in the workforce and system policies. Therefore, in addition to penetration, we assess volume of practices delivered over time in raw units of numbers of agencies, therapists, children, and units of service to characterize the scale of implementation impact.

To date, most implementation and sustainment studies measure penetration based on therapist surveys reporting on their ongoing use of a practice (e.g., Swain, Whitley, McHugo, & Drake, 2010). Recently, there have been increasing calls to use administrative data from MH systems to understand practice patterns and inform implementation efforts (Chambers & Rupp, 2015). States and local MH systems are increasingly turning to administrative data to drive healthcare service delivery and decision-making (Hoagwood et al., 2015). The advantages to using administrative data in sustainment research include a broader view of the entire system, combined with a more granular assessment of practice patterns over time. Furthermore, use of administrative data aligns with assessment envisioned by the Patient Protection and Affordable Care Act (2010). Administrative data are particularly relevant to implementation efforts that are system-wide and fiscally mandated.

Context of Current Study

Taken together, there is an increasing need to conduct research on the sustainment of multiple practices in the context of system-driven EBP implementation efforts. Use of administrative claims data to examine practice delivery over time has the potential to significantly advance the field’s understanding of patterns of sustainment in usual care implementation. This study sought to address gaps in the literature by investigating the patterns of volume and penetration for six practices in one large-scale implementation effort in a children’s MH system, Los Angeles County Department of Mental Health’s (LACDMH) Mental Health Services Act (MHSA), Prevention and Early Intervention (PEI) Transformation. This transformation is consistent with the principles of the MHSA, a California ballot measure passed in November, 2004 that assessed a 1% tax on the income of individuals who make in excess of $1 million per year. The funding generated by this tax funds very specific types of MH services that have driven transformation of the workforce, systems of care for clients across the age spectrum and a focus on effective practices measured by outcomes.

LACDMH is the nation’s largest county MH department, serving, on average, more than 250,000 County residents of all ages every year (LACDMH, 2015). In July, 2010, LACDMH elected to utilize MHSA PEI funding to serve as a catalyst to utilize Evidence-Based, Promising and Community Defined Practices to intervene earlier in the course of mental illness. In response to external factors including public policies and initiatives that mandated behavioral health reform in the context of an overall state budget shortfall, LACDMH offered agencies the opportunity to receive reimbursement through PEI for the delivery of 52 practices approved in the county’s PEI Plan to achieve the goals of prevention and early intervention. Consequently, the contracts for agencies that served children and transition-age youth were amended to include PEI funds and implementation guidelines. Agency leaders selected from among the 52 practices based on their perceived needs and preferences. Consistent with the MHSA, LACDMH was programmatically mandated to provide implementation support (i.e. education, training, consultation, PEI guideline development, and technical assistance) to facilitate the launch and transformation for an initial set of six evidence-based/informed practices (hereafter referred to as practices) to address a range of prevalent youth MH problems, including Cognitive Behavioral Interventions for Trauma in Schools [CBITS], Child-Parent Psychotherapy [CPP], Managing and Adapting Practices [MAP], Seeking Safety [SS], Trauma Focused Cognitive Behavior Therapy [TF-CBT], Triple P Positive Parenting Program [Triple P]. See Table 1 for a summary of these practices. LACDMH provided training and implementation supports to agencies with the goal of training therapists to effectively provide the practices. Trainings commenced in May, 2010¹, and in fiscal year 2010–2011, over 32,000 children and transition age youth were served in PEI programs (LACDMH, 2011). LACDMH conducts ongoing site visits, monitors training/certification registries, and conducts chart reviews to assess fidelity to PEI implementation policies and guidelines. For example, LACDMH assesses for compliance with the following: Are clients eligible for practices delivered based on presenting problem and level of severity?; Have therapists delivering each practice completed the initial and ongoing certification requirements established by the practice purveyor?

Table 1.

Indicated age range and target problems for each practice

Practice	Indicated Age Range	Target Problem(s)
Child-Parent Psychotherapy (CPP)	Birth −6 years	trauma; poor attachment
Cognitive Behavioral Intervention for Trauma in School Children (CBITS)	11–15 years	PTSD; traumatic stress
Managing and Adapting Practice (MAP)	Birth-21 years	anxiety, traumatic stress, depression, conduct
Seeking Safety	13–18 years	PTSD, substance use
Trauma Focused CBT (TF-CBT)	3–18 years	PTSD; traumatic stress
Triple P	2–12 years	disruptive behavior; family dysfunction

The scope and size of the PEI Transformation provides a critical opportunity to examine the sustainment of multiple practices simultaneously implemented in a large and diverse natural laboratory. The timing of the current study relative to the maturity of PEI transformation permits examination of practice sustainment up to five years after adoption. The objectives of the current paper are to (1) characterize patterns of volume and penetration of the six practices using multiple units of analysis within the LACDMH PEI Transformation; (2) characterize trajectories of volume of therapists’ claims for each practice relative to claims for other, and (3) identify inner context variables (therapist case-mix and service setting characteristics) that predict the relative volume of therapists claims for each practice. Through these objectives, we will illustrate methodological challenges inherent in characterizing changes in penetration of practices prospectively.

Method

Outcome Measures

In order to use PEI funds, clients must meet the specific eligibility criteria for each practice based on age, presenting problem and diagnosis and therapists submit practice-specific claims for each unit of service. Each claim is linked to client (demographics, diagnosis), setting, and therapist (i.e. demographic and discipline) characteristics. These administrative PEI claims data from LACDMH for the six practices over 19 fiscal quarters between FY 2009–2010 (Quarter 4) and FY 2014–2015 (Quarter 2) were used. Thus, analyses of these data allow us to characterize patterns of volume and penetration of these six PEI practices relative to one another. These data represent 3,014,353 total claims (2,331,000 psychotherapy claims) for 87,100 unique child, adolescents and transition age clients receiving MH services from 8,514 unique MH therapists within 94 unique agencies. For the present study, only claims for “Psychotherapy” services (individual, family, group) were extracted and used in analyses (i.e. claims for other non-psychotherapy services such as Medication Management, Evaluation and Assessment, and Case Management were excluded).

Results

Objective 1: Characterize Patterns of Volume and Penetration at the Claims, Therapist, and Agency levels Over Five Years

a) Gross volume over time

First, descriptive analyses and data visualization were employed to provide a system-level overview of practice volume over time. Volume was calculated for each fiscal quarter by the 1) the number of psychotherapy claims per practice, 2) the number of children with at least one psychotherapy claim to each practice, 3) the total number of unique therapists who made at least one psychotherapy claim for each practice, and 4) the number of agencies with at least one therapist billing a psychotherapy claim to each practice. Figure 1 displays the total number of psychotherapy claims, therapists, children and agencies per practice across all 19 fiscal quarters since initial implementation.

Figure 1.

Practice Volume over Time at the Claim, Child, Therapist, and Agency Levels

The data reflect the initial implementation of the six PEI practices occurred in Quarter 1 of FY 2010–2011 with a period of ramp up represented by a positive initial slope. Visual inspection of the pattern of claims over time indicates a flattening of claims volume billed over the course of the implementation from FY 2011–2012 through Quarter 2 of FY 2014–2015. After the first six to eight quarters, volume of claims for each practice appeared to level off and remain stable. Based on these raw counts of claims, clients, therapists, and agencies claiming to each practice, MAP, TF-CBT, and Seeking Safety demonstrate higher claims volume than CPP, CBITS, and Triple P. Specifically, MAP had the highest cumulative number of psychotherapy claims across fiscal quarters (n = 905,395), followed by TF-CBT (n = 662,184), Seeking Safety (n = 515,208), Triple P (n = 140,147), CPP (n = 105,231), and CBITS (n = 2,835). When examining volume at the therapist level, TF-CBT had the largest number of unique therapists who made at least one psychotherapy claim across fiscal quarters (n = 4,670), followed by MAP (n = 4,587), Seeking Safety (n = 3,681), Triple P (n = 1,865), CPP (n = 983), and CBITS (n = 148). When examining volume at the agency level, MAP had the highest number of unique agencies in which at least one therapist made at least one psychotherapy claim across fiscal quarters (n = 86), followed by TFCBT (n = 85), SS (n = 77), Triple P (n = 48), CPP (n = 47), and CBITS (n = 15).

To further support the visual inspection of the claims data, multilevel modeling using HLM 7 (Raudenbush, Bryk, Cheong, Congdon, & du Toit, 2011) was performed to examine the claim volume trajectories of each practice. Multilevel modeling was selected to manage the complex, nested structure of these data (clients receiving a specific practice nested within therapists nested within agencies across time). Six cross-classified multilevel models were performed in which the primary outcome was the total number of psychotherapy claims for each practice per fiscal quarter. Because therapists billed for psychotherapy services across multiple agencies within and between fiscal quarters, cross-classification of therapists across agencies was addressed as needed. The main predictors in this model were linear and non-linear (quadratic) functions of time². Consistent with the descriptive data provided above, there was a statistically significant linear increase in the volume of psychotherapy claims over time for MAP, (B=.01, p<.001), CBITS (B=.19, p<.001), TF-CBT (B = .03, p<.001) and Triple P (B=.17, p<.001). There was a statistically significant decrease in the volume of psychotherapy claims for SS (B=−.11, p<.001). There was no significant linear slope characterizing change in volume of claims for CPP over time. There was a statistically significant negative quadratic trajectory in the volume of psychotherapy claims for all practices (p-values for all practices <.001), indicating a gradual flattening of change in claims volume across time.

b) Gross penetration over time

Gross penetration was calculated at multiple levels as follows: 1) the number of psychotherapy claims per practice across fiscal quarters divided by the total volume of psychotherapy claims across fiscal quarters, 2) the number of children with at least one psychotherapy claim to each practice across fiscal quarters divided by the total volume of unique children receiving any of the six practices across fiscal quarters, 3) the total number of unique therapists who made at least one psychotherapy claim for each practice across fiscal quarters divided by the total volume of unique therapists who billed for at least one of practices across fiscal quarters, and 4) the number of agencies with at least one therapist billing a psychotherapy claim to each practice across fiscal quarters divided by the total volume of unique agencies claiming for at least one of the practices across fiscal quarters. Figure 2 displays gross penetration patterns at the claims, therapist, child and agency levels for each practice across time.

Figure 2.

Gross Penetration at the Claim, Child, Therapist, and Agency Levels

Visual inspection of the penetration data indicates that penetration at each level was variable during the first six to eight fiscal quarters. During the initial ramp-up period, some practices ramped up very quickly (TF-CBT) and resulted in proportionally more claims, clients, therapists and agencies. After the eighth quarter (Q4 ’11–12), there was some re-ordering and the penetration patterns stabilized and were generally similar across units of analysis (claims, clients, therapists, agencies). Between quarters 9 through 19, at the claims level, MAP represented the highest average proportion of claims (M=43%; SD=2.30), followed by TF-CBT (M=25%; SD=2.20), Seeking Safety (M=21%; SD=2.30), Triple P (M=6%; SD=0.80), CPP (M=5%; SD=0.50) and CBITS (M=.05%; SD=0.10). At the client level, MAP represented the highest average proportion of clients (M=44%; SD=2.20), followed by TF-CBT (M=26%; SD=2.00), Seeking Safety (M=17%; SD=1.40), Triple P (M=8%; SD=1.20), CPP (M=6%; SD=0.50) and CBITS (M=0.10%; SD=.06). At the therapist level, MAP represented the highest average proportion of therapists (M=59%; SD=1.80), followed by TF-CBT (M=50%; SD=1.90), Seeking Safety (M=34%; SD=2.70), Triple P (M=16%; SD=2.00), CPP (M=11%; SD=0.80) and CBITS (M=0.30%; SD=0.20). At the agency level, MAP represented the highest average proportion of agencies (M=87%; SD=2.20), followed by TF-CBT (M=85%; SD=1.50), Seeking Safety (M=68%; SD=2.90), CPP (M=42%; SD=1.40), Triple P (M=39%; SD=2.60), and CBITS (M=2%; SD=1.90).

Objective 2. Modeling trajectories for therapists’ volume of claims for each practice relative to total number of claims

Multilevel modeling using HLM 7 (Raudenbush, Bryk, Cheong, Congdon, & du Toit, 2011) was performed to examine the trajectories of therapists’ volume of claims for each practice across time (19 fiscal quarters). Multilevel modeling was selected to manage the complex, nested structure of these data (therapists nested within agencies across time).

a) Characterizing relative claims volume trajectories at the therapist level for each practice

In these six models, the number of psychotherapy claims per fiscal quarter aggregated to the therapist level were modeled. In these models, the total number of psychotherapy claims that the therapist claimed across all six practices during that fiscal quarter was included as a time varying covariate, in order to examine volume of claims for a given practice relative to total therapist claims. In addition, number of agencies from which a therapist billed was also included as a time-varying covariate. The linear and quadratic functions of time were the predictors of interest to characterize the nature of the trajectories. These findings are reported in Table 2 and Table illustrated graphically in Figure 3.

Table 2.

Practice-Specific Penetration Trajectories at the Therapist Level

Practice	Grand Mean	95% Confidence Intervals	Linear Term	Quadratic Term
CPP	7.46	6.17, 8.75	0.30*a	−0.26**c
CBITS	18.93	15.13, 22.73	−0.31a	−0.05c
TF-CBT	35.47	31.10, 39.84	−.1.86**b	−0.27**c
Triple P	51.29	48.00, 54.58	1.00*a	−0.38**d
SS	147.60	140.17, 155.03	−2.86**b	−0.79**e
MAP	148.00	143.53, 152.47	0.15a	−1.25**f

Notes.

^*p < .05;

^**p < .001.

Shared superscripts denote that parameter estimates were not significantly different.

Figure 3.

Practice-Specific Trajectories of Relative Claims Volume at the Therapist-Level

Note. The dotted line for the first fiscal quarter of MAP indicates that it was adopted later than the other practices. CBITS did not have a statistically significant quadratic trajectory and is not reflected in this figure.

There was a statistically significant linear decrease in the relative volume of claims per therapist across time for TF-CBT (B=−1.86, p<.001) SS (B=−2.86, p<.001), and CBITS (B=−0.31, p<.05). There was a statistically significant increase in relative volume of claims per therapist for Triple P (B=1.00, p<.05) and CPP (B=0.30, p<.05). There was no statistically significant linear trajectory characterizing MAP volume at the therapist level over time. For all practices except CBITS, there was a statistically significant negative quadratic trajectory (all ps < .001) indicating a gradual flattening of changes in relative claim volume at the therapist-level over time.

b) Comparing relative claims volume trajectories across the six practices

To achieve this objective, pairwise comparisons of the linear and quadratic model parameters (see Table 2) across the six trajectories were conducted. Overall, there were statistically significant differences in both the linear and quadratic trajectories of relative claim volume per therapist between practices. The linear trajectory for TFCBT indicated a decrease in relative volume and was significantly different from the linear trajectories of CPP (p<.05), CBITS (p<.05), Triple P (p<.001), and MAP (p<.001). Similarly, the linear trajectory for Seeking Safety indicated a decrease in volume and was significantly different from the linear trajectories for MAP (p<.01), CPP (p<.001), CBITS (p<.05), and Triple P (p<.001). The quadratic between-practices differences were then examined. The quadratic trajectories for MAP, SS, and Triple P significantly differed from all other practices (ps < .05) in pairwise comparisons. Specifically, MAP, SS, and Triple P showed significantly steeper downturn or flattening of therapist-level penetration over time compared to the other practices.

Objective 3: Identify Therapist Characteristics (Case-mix, Services Setting) Associated with Relative Claims Volume per Therapist for Each Practice

The final aim of the study was to identify characteristics of therapists and their caseloads associated with the relative volume of psychotherapy claims per therapist per fiscal quarter for each practice. The data assumed the same structure as in the Objective 2 models with relative claims volume aggregated to the therapist level, with observations for 19 fiscal quarters nested within therapist. However, in these models, linear and quadratic functions of time were not included. Rather, case-mix and therapist characteristics were modeled as predictors of claim volume. The following client case-mix characteristics (i.e. client characteristics aggregated to the therapist level within each fiscal quarter) were examined as predictors: 1) the proportion of clients in specific age groups (0–5 years; 6–15 years; 16–21 years), 2) the proportion of clients in each racial/ethnic category, 3) the proportion of male clients, 4) the proportion of clients whose primary language was English, Spanish, and Other, 5) the proportion of clients with designated categories of admission diagnoses, and 6) the proportion of clients served in each service location type (e.g., school, office, community mental health center). Client admission diagnoses were categorized into the following groups: Anxiety Disorders, Attention Deficit Hyperactivity Disorder (ADHD), Disruptive Behavior Disorders, Adjustment Disorders, Trauma Disorders, Mood Disorders, Substance Use Disorders, and Other Disorders. All case-mix characteristics were treated as continuous variables (i.e. % of caseload from 0 to 100). The following therapist characteristics were included as predictors: 1) therapist primary language (English as the reference group), 2) therapist race/ethnicity (Non-Hispanic White as the reference group), and 3) discipline (Marriage and Family Therapist [MFT] as the reference group.)

All analyses controlled for two time varying covariates: the number of agencies in which a therapist billed per fiscal quarter, and the number of clients for whom a therapist made at least one psychotherapy claim regardless of practice in a given quarter. For the purposes of data reduction in the context of a large number of potential predictors, we first conducted individual univariate models for each predictor for each practice. Predictors that were statistically significant at the p <.001 level in these preliminary univariate models were then entered together in subsequent multiple predictor models (one model for each practice). A summary of results from the multivariate models are reported in Table 3.

Table 3.

Significant Variables in Multiple Predictor Models

	CPP	MAP	SS	TF-CBT	Triple P
	B	B	B	B	B
Therapist Control Variables
Number of Agencies	0.40**	−0.29	1.59*	0.06	−0.62*
Number of Clients	2.72***	24.87***	16.84***	8.43***	8.77***
Case-mix Characteristics
Client Admission Diagnosis (% of clients)
ADHD	−0.10**	0.19**	--	--
Anxiety Disorders	--	--	−0.49***	--	--
Trauma Disorders	--	−0.60***	--	0.30***	−0.32***
Disruptive Behavior Disorders	--	0.10*	0.56***	−0.21***	--
Mood Disorders	−0.04	--	--	--	−0.24***
Adjustment Disorders	--	--	--	0.22***	--
Other Disorders	0.01	−0.31**	--	−0.09*	--
Client Age (% of clients)
0–5 years	0.15	0.79	−1.42*	--	0.19**
6–15 years	−0.10	1.21	−0.31	0.12***	--
16–21 years	−0.16	0.90	0.44	−0.005	−0.16**
Client Ethnicity (% of clients)
Non-Hispanic White	--	--	--	--	--
Hispanic	--	--	--	--	--
African American	--	--	0.13	--	--
American Indian/Alaskan Native	--	--	--	--	--
Native Hawaiian/Other Pacific Islander	--	--	--	--	--
Asian American	--	−0.06	−0.19	--	--
Other Ethnicity	--	--	--	--	--
Client Gender (% Males)	--	0.01	0.33***	--	0.10**
Client Language (% Clients)
English	--	--	0.22	--	--
Spanish	--	--	−0.04	--	--
Other	--	--	--	--	--
Service Setting (% Claims)
CMHC	−0.10***	--	−0.84***	--	--
Office	--	--	6.24***	--	--
Home	--	--	−0.68***	0.30***	--
Residential	--	--	2.38***	--	--
School	--	−0.07	--	0.03	--
Other	--	0.32***	--	0.13***	--
Therapist Characteristics
Therapist Primary Language (reference=English)
Spanish	--	--	−50.81***	--	--
Other	--	--	−30.35*	--	--
Unknown	--	--	−107.04	--	--
Therapist Ethnicity (reference group= Non-Hispanic White)
Hispanic	--	−10.24	10.99	−5.40*	15.24***
American Indian/Alaskan Native	--	−33.95	43.58	−4.31	188.40
Asian	--	5.10	2.79	0.75	−0.84
Native Hawaiian/Other Pacific Islander	--	19.77	164.29*	5.51	54.48
African American	--	11.63	−8.31	−2.87	19.87*
Unknown	--	−9.78	1.71	−7.54*	7.56
Other	--	−10.41	40.36*	5.19	0.76
Discipline (MFT: reference group)
Counselor	2.47	−10.77	16.81	−4.97	--
Social Work	−3.34	−9.53	−29.81*	−5.53	--
Psychologist	−1.77	−26.13**	24.42	−16.58***	--
Rehabilitation Professional	0.53	−11.11	38.83***	−14.84***	--
Case Manager	0.76	4.99	41.25*	−14.17**	--
Psychiatrist	−8.05	−56.54***	−56.26**	−38.86***	--
Other	−8.76	−72.44*	−62.73*	−32.39**	--
Trainee	7.81**	−13.85	−13.52	−9.50*	--

Notes.

^*p < .05;

^**p < .01,

^***p < .001.

^--indicates that the variable was not significant at p<.001 in bivariate models and was not included in the final multiple predictor models. A model for CBITS was unable to be performed.

A total of five models were run, one for each practice except CBITS, which had an insufficient number of claims for this set of analyses. As indicated in Table 3, a number of case-mix variables significantly predicted volume of claims as the therapist level. Specifically, client admission diagnosis variables were associated with all models. For example, a higher proportion of clients diagnosed with a trauma-related disorder was positively associated with TF-CBT penetration and negatively associated with the MAP and Triple P penetration. Having a larger of proportion of male clients was significantly associated with SS and Triple P penetration. Case mix of client age was significantly associated with SS, TF-CBT, and Triple P penetration. Service setting location was significantly associated with penetration of CPP, MAP, TF-CBT, and SS.

Therapist characteristics also significantly predicted practice-specific penetration. Therapist primary language was a significant predictor of SS penetration. Therapist ethnicity was also a significant predictor of SS, TF-CBT, and Triple P volumes. For example, Hispanic therapists had significantly lower volumes of TF-CBT claims compared to Non-Hispanic White therapists, whereas Hispanic therapists had significantly higher volumes of Triple P claims compared to Non-Hispanic White therapists. Finally, therapist discipline was a significant predictor in the CPP, MAP, TF-CBT, and SS models. For example, rehabilitation professionals had a higher volumes of SS claims compared to marriage and family therapists but lower volumes of TF-CBT claims.

Discussion

This study used administrative claims data to characterize volume and penetration of six practices over 19 fiscal quarters following initial implementation within the context of a system-driven, fiscally mandated implementation effort by the largest MH department in the United States. Results revealed variability in volume and penetration by practice and varied by the methods used to index and analyze these outcomes. Furthermore, a number of client case-mix and therapist characteristics were associated with relative volume of each practice within therapists’ caseloads.

Characterizing Volume and Penetration Over Five Years Following Adoption

The first objective of this study was to characterize volume and penetration of practices for the first five years of implementation. We used multiple methods to characterize volume and penetration including examining raw claims data over time at the system level and modeling penetration trajectories accounting for the nested nature of the data. The volume data provide a view of the gross numbers of psychotherapy visits claimed to each practice, clients served by each practice, and the gross numbers of therapists and agencies that billed to each practice per quarter. The initial increases in raw volume for most practices in the first quarters highlight the rapid impact of a fiscal policy change restricting reimbursement to specific practices and following LACDMH-supported training in each practice, and increasing experience delivering the practices. The penetration data provide a view of the volume data for each practice relative to the total volume of claims, clients, therapist, and agencies across all practices and indicate a stabilization in the proportions of claims, clients, therapists and agencies reached by each practice after the first 2 years. Overall levels of gross volume and penetration based on raw claims data differed by practice at the claims and therapist levels, with MAP, TF-CBT, and SS having the highest peak volume and penetration, and Triple P, CPP, and CBITS demonstrating lower levels of volume and penetration.

The high versus low volume and penetration practices based on these raw data do share some characteristics. The higher volume practices of MAP, TF-CBT and SS all apply to a broad age range of clients and can be delivered in multiple service settings. Furthermore, these three practices all address the needs of clients impacted by trauma, which is significant in the LACDMH population. Additionally, MAP, in particular, is designed to flexibly cover a range of presenting client MH problems using evidence-informed treatment planning and progress monitoring (Daleiden et al., 2006; Nakamura et al., 2011). In contrast, CPP and CBITS are typically applied to restricted age ranges, settings, and format (i.e. CBITS is exclusively delivered in group modality in schools; CPP targets caregivers of very young children). Lastly, CPP and Triple P are caregiver-directed practices that require high levels of caregiver involvement throughout the entire protocol, which can be a significant challenge to garner in community MH settings and thus impacting penetration.

In addition, the observed variation in practice delivery may be influenced by the availability and types or costs of training as well as coordination challenges. MAP and SS have both offered Train-the-Trainer capacity, which allowed agencies to feasibly train new therapists as in-house supervisors gained credentials to do so. Furthermore, therapists were approved to claim for SS after attending a one-day training, whereas therapists had to meet additional requirements before claiming to other practices. Accordingly, costs of training may also have factored into observed trends in practice penetration. Lastly, the low penetration of CBITS may reflect the dynamics of implementing school based interventions (e.g., needing memoranda of understanding [MOU] between agencies and school districts) as well as difficulties accessing clients in schools and retaining student participation (i.e. holding groups during school may contravene minimal instructional time mandates).

Characterizing and Comparing Relative Claims Volume Per Therapist for Each Practice

Multilevel modeling was used as a second approach to manage the complex, nested structure of these data (clients receiving a specific practice nested within therapists nested within agencies across time). Interpretation of linear and quadratic functions were used to understand the penetration trajectories. As indicated in the set of models examining trajectories in the number of claims per therapist relative to all claims for each quarter, there was a significant linear increase in the relative number of claims per therapist over time for Triple P and CPP; a decrease for TF-CBT and SS; and no significant change in number of claims per therapist for MAP and CBITS. The statistically significant negative quadratic trajectory for all practices except CBITS indicated a gradual flattening of changes in relative claim volume per therapist over time. The lack of significance for CBITS is likely due to the low number of overall claims for this practice.

To compare relative claims volume trajectories, we made pairwise comparisons of the grand means, linear and quadratic parameter estimates across the six practice trajectories. These findings demonstrate a fairly simple method for testing whether patterns of uptake and sustainment differ across practices in a multiple EBP context. These results suggest that there were differences in trends in therapist-level claiming. Indeed, two of the practices with low relative volume showed a trend of linear increase in therapist claims from quarter to quarter over the study time frame (CPP and Triple P), whereas other higher relative volume practices actually showed an average negative slope (TF-CBT and SS). One possibility is that some practices take some time to build a specialized caseload, whereas other initially higher relative volume practices were initially applied to large populations of need, but may have been supplanted by the increased penetration of other PEI practices. As described previously, all practices except CBITS showed a negative quadratic pattern in relative volume trajectory, but the magnitude of this flattening differed with the practices that ramped up to the highest peak levels showing a larger flattening effect. This appears logical as steeper ramp ups would by definition yield a larger negative quadratic function when the plateau occurs. As such, this illustration of the multilevel modeling method sheds light on what seem to be expected characteristics of trajectories in examinations of administrative claims data of multiple practices. In addition to differences in the shapes of the trajectories, average number of claims for therapists differed by practice in a way that is distinct from differences gross penetration of these practices. For example, even though Triple P was a lower penetration practice based on raw claims and number of therapists billing, the grand mean indicates that therapists who claimed to this practice submitted a higher number of claims relative to other practices. Relatedly, it is possible that the lower number of therapists trained in Triple P, for example, results in those trained being assigned a higher proportion of Triple P clients relative to other practices being delivered by more therapists.

Identifying Therapist Characteristics Associated with Relative Claims Volume per Therapist for Each Practice

The last objective of this study was to identify therapist and therapist case-mix characteristics associated with relative claimes volume per therapist. These analyses indicate that case-mix and therapist variables do predict practice use in patterns specific to the practice. For example, therapists with a large proportion of clients with trauma diagnoses had higher volume of TF-CBT claims per therapist over time. Overall, relative volume for each practice appeared rationally related to therapist case-mix and may be driven by consumer demand and clinical need. Additionally, these findings align with the PEI Guidelines for implementation (which were developed to be congruent with the practices and the presenting issues). These findings have important implications for system leaders as they identify a set of practices that cover a large proportion of the community mental health needs for children (Chorpita et al., 2011).

There were also some distinct patterns of differences in the relative volume of claims per practice based on therapist characteristics, including primary language, ethnicity, and discipline. For example, Hispanic therapists had lower relative volumes of TF-CBT claims and higher relative volumes of Triple P claims compared to Non-Hispanic White therapists. As another example, rehabilitation professionals had higher relative volumes of SS claims compared to marriage and family therapists but lower volumes of TF-CBT claims. These differences may be explained by PEI implementation guidelines in which SS is one of the few practices that may be provided by clinicians who are not mental health license-eligible. Triple P can be provided by BA-level staff, whereas TF-CBT may only be delivered by at least a Master’s level clinician who is licensed or license-eligible. Implementation guidelines concerning therapist discipline or education may function to shape patterns related to cultural and linguistic diversity in the workforce.

The data analyzed in the current study presented a number of methodological/statistical challenges. The primary complexity involved the nature of the nested data structure and the amount and pattern of missing data given the longitudinal data set. Traditional multilevel models assume a uniform nested data structure. In the current study, one could conceptualize the nested data structure as having 4-levels: repeated measures claims (level-1) nested within clients (level-2) nested within therapists (level-3) nested within agencies (level-4). In a traditional multilevel model, one would typically assume, for example, that a given client is assigned to the same therapist and that a given therapist is assigned to the same agency. Most multilevel modeling packages (e.g., HLM 7; Raudenbush et al., 2011) are able to statistically analyze 4-level models such as this accurately by accounting for the dependencies in the data from the multiple levels. However, to the authors’ knowledge, current statistical software is unable to statistically analyze data where clients are assigned to multiple therapists/staff and/or therapists/staff work in multiple agencies. These cross-classified or multiple membership models have been conceptually discussed and statistically analyzed (see Beretvas, 2011; Cafri, Hedeker, & Aarons, 2015), but are difficult to practically implement in the context of 4-level nested data structures. Because of this, adaptations were needed for use with available software to estimate relationships of interest.

The complexities of the claims data necessitated applying multiple adaptations to traditional analytic models. First, in Objective 1, the data were analyzed using a cross-classification model. Cross-classification models account for the fact therapists could be nested within multiple agencies. Thus, therapist and agency were treated as the same level of the nested data structure. This results in “cells” that represent different therapist by agency groups, much like the cells of a factorial analysis of variance (ANOVA) design. Using this method, significant negative quadratic functions indicate a gradual flattening in volume across time at the claims-level. The second approach used was to simply aggregate up to a conceptual level of interest (Objectives 2 and 3). Because practice reporting at the therapist level was of primary interest, claims at each fiscal quarter were summed for each therapist and analyzed as a 2-level model (repeated measures fiscal quarter nested within therapists). To account for the effect of agency, one can simply create a variable (or variables) and treat these as fixed effects at the therapist level, rather than treated agency as a formal level. Results using this approach illustrate the flattening or reduction in the relative volume of claims over time.

In addition to the nested nature of the data, the longitudinal nature of the data adds to the complexity. Specifically, missing data is a consistent problem with a longitudinal design such as this, but multilevel modeling is able to account for this assuming the data is missing at random (MAR; see Enders, 2010). In fact, this is a major benefit of these models compared to classic repeated measures ANOVAs (Raudenbush & Bryk, 2002). Given that no single client or therapist had complete data over the course of the 19 fiscal quarters, this added to the complexities of the nested data structure. However, there are inherent difficulties in applying the MAR assumption in these data. Certainly, we would expect that there are a variety of patterns of missing data that are not at random. For example, left censored data (i.e. missing data points at the beginning of the 19 fiscal quarter sequence) may suggest that the therapist was a new arrival to the LACDMH PEI context, whereas right censored data (i.e. missing data points at the end of the sequence) suggest that the therapist exited the system or remained in the system but was no longer served PEI clients). It is possible to explicitly examine trajectories associated with these and other types of missing data patterns through statistical techniques such as pattern-mixture models, and this represents an important future direction.

There are a few primary limitations to this study that should be noted. First, this study relied exclusively on administrative claims data. These types of data share limitations of therapist self-report. For example, observed associations between case-mix characteristics and penetration may reflect actual congruence between the practices and client needs, or clinicians may align report of client characteristics with practice types claimed. In addition, claims data do not tell us whether and how the practice was actually delivered, including assessment of therapist fidelity to the practice. The primary objective of this study were to describe and compare claims volume trajectories within the context of a fiscal mandate and not to conduct an evaluation of the success of the fiscal mandate as assessed by changes in therapist behavior change. The next step in our larger program of research is to characterize actual delivery of these practices and understand multiple stakeholders’ perspectives of sustainment patterns. Specifically, our research aims to develop feasible methods to characterize practice delivery across multiple interventions (Lau & Brookman-Frazee, 2015). Relatedly, we cannot determine from our data penetration rates for given diagnoses (i.e. it is not known what percent of children received the appropriate EBP given their diagnosis, presenting problem or age). This is an important future direction for examining system-wide impacts. Second, in this context, therapists are restricted to reimbursement for specific practices. It is not known how these findings generalize to systems employing implementation model methods other than a fiscal mandate. Third, while the adaptations of design and statistical analysis were necessary given the data sets and current state of statistical software, it should be emphasized that the general inability to accurately specify a nested data structure can result in biased standard error estimation in multilevel models (Luo & Kwok, 2009, 2012).

Summary and Next Steps

This study examined patterns in multiple practice delivery in the context of a massive, system transformation that was the stimulus to deliver EBPs in public children’s MH services. Our goal was not to evaluate this system or evaluate the delivery of any of the six practices. Rather, we intend to use systems administrative data captured within a natural experiment to advance our understanding of multi-practice implementation over time and of methods to analyze complex, multi-level claims data, aligning with recent calls for use of “big data” to inform policy and practice (Chambers & Rupp, 2015). Furthermore, studying multiple practice implementation efforts is critical to isolating intervention characteristics and intervention-provider fit as predictors of sustainment. Patterns of practice implementation over time in the current study highlight the importance of considering the sustainment of one practice within the context of patterns of others. In particular, this extends the notion of “fit” of a practice to a system, organization and therapist to include the availability of other EBPs as important contextual factors. System-level planning based on initial assumptions mapping consumer needs with available practices can be refined over time based on accumulating data showing how multiple practices can be best coordinated to avoid redundancy and maximize efficiency to provide the most efficient array of services (Chorpita, Bernstein, & Daleiden, 2011). Next steps in this research are to use multiple methods to assess sustainment outcome beyond volume and penetration (Lau & Brookman-Frazee, 2015). For example, we will characterize therapist delivery of specific practice elements and gather qualitative data from interviews about adaptations to practices and therapist perspectives on factors associated with sustained (or un-sustained) delivery of individual practices.