Community-Based Implementation of Trauma-Focused Interventions for Youth: Economic Impact of the Learning Collaborative Model

Abstract

This study investigated the economics of the learning collaborative (LC) model in the implementation of Trauma-Focused Cognitive-Behavioral Therapy (TF-CBT), an evidence-based intervention for traumatic stress in youth. We evaluated the cost-effectiveness of the LC model based on data from 13 LCs completed in the southeastern United States. Specifically, we calculated cost-effectiveness ratios (CERs) for two key service outcomes: (1) clinician TF-CBT competence, based on pre- and post-LC self-ratings (n = 574); and (2) trauma-related mental health symptoms (i.e., traumatic stress and depression), self- and caregiver-reported, for youth who received TF-CBT (n = 1,410). CERs represented the cost of achieving one standard unit of change on a measure (i.e., d = 1.0). The results indicated that (1) costs of $18,679 per clinician were associated with each unit increase in TF-CBT competency and (2) costs from $5,318 to $6,548 per youth were associated with each unit decrease in mental health symptoms. Thus, although the impact of LC participation on clinician competence did not produce a favorable CER, subsequent reductions in youth psychopathology demonstrated high cost-effectiveness. Clinicians and administrators in community provider agencies should consider these findings in their decisions about implementation of evidence-based interventions for youth with traumatic stress disorders.

Children and adolescents have high rates of trauma exposure, as evidenced by national estimates of 702,000 victims of child abuse and neglect in 2014 (U.S. Department of Health and Human Services, 2016) and of 1 in 20 youth meeting criteria for post-traumatic stress disorder (Merikangas et al., 2010). In addition, the annual economic burden of childhood trauma in expenses pertaining to mental health care, lost productivity, and participation in social services is over $120 billion (Fang, Brown, Florencea, & Mercya, 2012; Gustavsson et al., 2011). Furthermore, although a number of interventions have shown efficacy with trauma-related psychopathology in children and adolescents (National Child Traumatic Stress Network, n.d.; Silverman et al., 2008), such evidence-based interventions (EBIs) are not standard practice for youth served in many community-based mental health settings (APA Task Force on Evidence-Based Practice for Children and Adolescents, 2008; McHugh & Barlow, 2010).

Given the social and economic consequences of youth trauma exposure, it is important to determine cost-effective methods for implementation of EBIs in community service agencies (Proctor, 2012). To that end, the National Child Traumatic Stress Network has widely promoted use of the Learning Collaborative (LC; Ebert, Amaya-Jackson, Markiewicz, Kisiel, & Fairbank, 2012) model to implement various trauma-focused EBIs in community service settings. The intent of an LC is to bring together teams from different organizations and professional roles (e.g., clinicians, supervisors, senior leaders) that work together to learn an EBI and sustain its use over time. Implementation strategies used in the LC model include pre-work activities to build capacity, in-person trainings, and implementation periods between trainings that involve ongoing consultation and quality improvement strategies, such as small tests of change. More recently, Saunders and Hanson (2014) adapted the standard LC model to include broker professionals (i.e., those responsible for identification, referral, and monitoring of maltreated children) and a focus on a community rather than individual clinicians or agencies; this adaptation is known as the Community-Based Learning Collaborative (CBLC) model. Both the standard LC and CBLC models (which are collectively referred to as the LC model for the present study) are hypothesized to promote adoption and sustained use of EBIs for youth by community-based clinicians. However, although cost burden has long been identified as a critical barrier to the transport of EBIs to community settings (Addis, Wade, & Hatgis, 1999; Barlow, Levitt, & Bufka, 1999; Bond et al., 2012), researchers have yet to examine the cost-effectiveness of implementation strategies such as the LC model.

The purpose of the present study was to examine the cost-effectiveness of the LC model in the implementation of Trauma-Focused Cognitive-Behavioral Therapy (TF-CBT; J. A. Cohen, Mannarino & Deblinger, 2006). TF-CBT is a well-established EBI for trauma-related mental health symptoms in youth (i.e., in over a dozen randomized trials; Pollio, McLean, Behl, & Deblinger, 2014), making it an ideal intervention for transport to community settings. Recently, several initiatives have used the LC model with community-based service providers who wish to implement TF-CBT, including Project BEST (Bringing Evidence-Supported Treatments to South Carolina Children and Families; www.academicdepartments.musc.edu/projectbest), the Program on Adolescent Traumatic Stress (PATS; www.academicdepartments.musc.edu/PATS/), the Missouri Children’s Trauma Network (www.moctn.com/children-s-mental-health-tfcbt-lc), and the WI Trauma Project (www.uwgb.edu/bhtp/TF-CBT-Project/). Given the promise of LCs for increasing the use of trauma-focused EBIs in community settings, as well as the availability of relevant data from ongoing implementation projects, it seems important to determine the potential cost advantages of the LC model. Indeed, implementation efforts that increase the availability of clinically effective interventions for youth trauma are likely to be cost effective.

To that end, we performed an economic analysis of outcomes from LCs conducted through Project BEST and PATS. From the numerous methods that are available to evaluate the economic impact of health interventions, we selected cost-effectiveness analysis (see Edejer et al., 2003; Siegel, Weinstein, Russell, & Gold, 1996), which compares the costs (in monetary units) and outcomes (in units relevant to that outcome) of clinical services. Indeed, cost-effectiveness analysis is particularly well suited to evaluating outcomes with intrinsic value, such as those ultimately targeted by efforts to implement EBIs (e.g., reductions in mental health symptoms). It should be noted that interpretation of cost-effectiveness results requires comparison against a counterfactual scenario to estimate population (vs. no intervention) or incremental (vs. an alternative intervention) effectiveness. Although there are challenges to approaches that perform such estimation in the absence of a formal comparison condition (i.e., as is common in community-based research), such an approach is critical to understanding the economic impact of EBI implementation efforts that focus on real-world service providers.

In sum, the present study used cost-effectiveness analysis to compare all costs associated with delivery of the LC model (i.e., direct expenses for training and quality assurance, indirect costs of time spent participating) to two key service outcomes, one for clinicians (i.e., competence in use of TF-CBT) and the other for youth (i.e., trauma-related mental health symptoms). In addition, as is best practice in cost-effectiveness analysis (Edejer et al., 2003; Siegel et al., 1996), we used sensitivity analysis to examine the influence of uncertainty in key study parameters on our findings. Finally, we used a novel approach to interpret the economic impact of the LC model without a comparison condition. To our knowledge, this study is the first economic analysis of an implementation model for youth psychological therapy.

Method

Participants

At the time of the study, seven LCs had been completed through Project BEST (two standard and five community-based) and six had been completed through PATS (one standard and five community-based). The LCs were conducted in Florida (3), Georgia (1), North Carolina (1), South Carolina (7), and Tennessee (1).

Clinicians

The primary participants were 574 community-based mental health clinicians who participated in an LC through Project BEST or PATS. All clinicians who completed a set of pre-LC online questionnaires were eligible for participation. The number of clinicians per LC site varied from 24 to 58 (M = 44.2; SD = 9.6). As shown in Table 1, clinicians had worked at their current organization for an average of 3.74 years and the majority (85.5%) held a Master’s degree. Of the clinicians, 75.3 percent were therapists and 17.9 percent were clinical supervisors who also delivered psychotherapy services.

Table 1.

Demographic Characteristics of Clinicians who Delivered TF-CBT Through a Learning Collaborative

			Test Statistic for Group Differences
Variable	n (%) or M (SD)	Range	LC site	Type of LC	Clinician role
Highest degree			NC	χ2(2) = 0.17, p = .919	χ2(2) = 16.22, p = .000*
Bachelors	11 (1.9%)	0.0 – 6.3%
Masters	491 (85.5%)	21.6 – 94.4%
Doctorate	37 (6.4%)	0.0 – 20.8%
Missing	35 (6.1%)	0.0 – 75.7%
Years at current organization	M = 3.7 (SD = 3.9)	2.07 – 5.33	F(7,343) = 4.70, p = .000*	t(349) = 1.56, p = .119	t(327) = 6.39, p = .000*
Type of LC			N/A	N/A	χ2(1) = 0.385, p = .535
Original (k = 3)	129 (22.5%)	N/A
Community- based (k = 10)	445 (77.5%)	N/A
Clinician role			χ2(12) = 18.45, p = .103	χ2(1) = 0.385, p = .535	N/A
Therapist	432 (75.3%)	66.7 – 89.5%
Supervisor	103 (17.9%)	5.3 – 31.1%
Missing	39 (6.8%)	0.0 – 20.0%

Note.

^*Statistically significant at p < .05 level.

LC = learning collaborative; NC = not calculated (because expected cell counts were too low).

Table 1 also presents ANOVAs, t-tests, and chi-square tests that compared participant characteristics among LC sites, between standard LCs versus CBLCs, and between clinicians who were therapists versus supervisors. Three comparisons were significant, such that (1) years at current organization varied among LC sites; (2) a greater percentage of supervisors (15.5%) versus therapists (4.4%) had a doctorate; and (3) supervisors had been at the current organization longer than had therapists (means of 6.28 vs. 3.04 years, respectively).

Youth

During their participation in an LC, each clinician was expected to deliver the full TF-CBT protocol to at least two children or adolescents. For the present study, we included all youth who (a) were registered as training cases by eligible clinicians, (b) completed a pretreatment self/caregiver-report assessment battery, and (c) were between the ages of 3 and 18 at the start of treatment. A total of 1,410 youth met these criteria, with an average of 2.46 youth per clinician (range = 0 to 11); approximately three-quarters of clinicians (n = 422) registered at least one youth. As reported in Table 2, youth averaged 12.2 years of age and 60.7% were female; the majority received TF-CBT from a therapist (79.7%) and through a CBLC (79.2%); and the average length of TF-CBT was 4.77 months.

Table 2.

Demographic Characteristics of Youth who Received TF-CBT Through a Learning Collaborative

			Test Statistic for Group Differences
Variable	n (%) or M (SD)	Range	LC site	Type of LC (original vs. community- based)	Clinician role (therapist vs. supervisor)
Gender (% female)	858 (60.9%)	44.3 – 94.1%	χ2(12) = 55.82, p = .000*	χ2(1) = 0.76, p = .384	χ2(1) = 0.122, p = .727
Age	M = 12.22 (SD = 3.50)	11.01 – 13.65	F(12,1397) = 4.37, p = .000*	t(1408) = 1.36, p = .174	t(1355) = −1.49, p = .136
Length of treatment (in months)	M = 4.77 (SD = 2.01)	3.63 – 5.39	F(12,833) = 4.52, p = .000*	t(844) = 0.95, p = .342	t(814) = 1.59, p = .112
Type of LC			N/A	N/A	χ2(1) = 6.75, p = .009*
Original (k = 3)	295 (20.9%)	N/A
Community- based (k = 10)	1,115 (79.1%)	N/A
Clinician role			χ2(12) = 74.84, p = .000*	χ2(1) = 6.75, p = .009*	N/A
Therapist	1,124 (79.7%)	68.5 – 97.8%
Supervisor	233 (16.5%)	2.2 – 27.7%
Missing	53 (3.8%)	0.0 – 9.5%

Note.

^*Statistically significant at p < .05 level.

TF-CBT = trauma-focused cognitive-behavioral therapy; LC = learning collaborative.

Table 2 also presented results of ANOVAs, t-tests, and chi-square tests used to compare youth characteristics among LC sites, between standard LCs versus CBLCs, and between clinicians who were therapists versus supervisors. Five comparisons were significant, such that there was variation among LC sites in terms of (1) youth gender, (2) youth age, (3) length of treatment, and (4) percentage of therapists versus supervisors; as well as (5) a greater percentage of clinicians were therapists in standard LCs (87.9%) versus CBLCs (81.4%).

Clinical Activities

LC

The LC model (Ebert et al., 2012) is a method of organizing teams of community-based clinicians and agency leaders that work together to learn an EBI and sustain its use over time. The LCs in Project BEST and PATS, each of which lasted 12 to 14 months, included three major types of activities. First, pre-work activities, such as formation of an agency team, were used to build local capacity for implementation of TF-CBT. Second, participants completed several (usually two) 2-day, in-person trainings in the delivery of TF-CBT (i.e., “Learning Sessions”). Learning sessions were interactive and emphasized adult learning principles, including skills practice (e.g., behavioral rehearsal) and problem-based learning (e.g., case vignettes), as well as training in quality improvement strategies such as plan-do-study-act (PDSA) cycles. Finally, interspersed between Learning Sessions were implementation phases (i.e., “Action Periods”), during which clinicians delivered TF-CBT with training cases and all participants engaged in ongoing consultation calls and PDSA cycles (i.e., tests of incremental change). The intent of consultation calls and PDSA cycles was to promote successful delivery of TF-CBT by addressing barriers that clinicians encountered during work with training cases. The LCs also involved activities for senior administrative staff and, for CBLCs, child welfare professionals; however, given the focus of the current study on clinical services and outcomes, we do not describe those activities further.

TF-CBT

TF-CBT (J. A. Cohen et al., 2006) is an EBI that addresses trauma-related symptoms (e.g., traumatic stress, depression, disruptive behavior) among children and adolescents. It is a short-term, components-based treatment model that is delivered in three phases. The first phase includes psychoeducation for the youth and caregiver as well as skills training (e.g., relaxation, emotion regulation, cognitive coping). The second phase involves the youth producing a narrative account (e.g., written story) of his or her perspectives on the trauma, processing that narrative (e.g., challenging distorted beliefs) with the clinician, and ultimately sharing the narrative with one or more supportive caregivers. The third phase addresses strategies to enhance safety and planning for the future to promote sustained recovery. Throughout the course of TF-CBT, the clinician integrates parenting skills training to manage trauma-related behavior problems and gradual exposure components to address behavioral avoidance.

Procedures

All procedures and measures were approved by the Institutional Review Board (IRB) of the Medical University of South Carolina. The IRB granted a full waiver of informed consent to use these data because we created a unique database that contained de-identified data.

Project BEST and PATS, although primarily intended to be training and implementation projects, collected data for the purposes of program evaluation and quality improvement. Specifically, data were collected using two methods: (1) online questionnaires, completed by each clinician participant prior to starting (pretest) and following completion of (posttest) the LC, that obtained demographic information and assessed a variety of factors related to EBI implementation; and (2) standardized youth/caregiver-report measures, including (a) caregiver-report and (b) self-report (for youth ages 7 and up), administered by participating clinicians at the onset and conclusion of each TF-CBT training case (i.e., pretest and posttest).

Measures

TF-CBT competence

Although observational coding systems represent the “gold standard” for determining treatment competence/fidelity, we used self-report methods to strike a balance between efficiency (i.e., feasibility for community practice) and validity. Specifically, data on clinicians’ self-reported competence in delivering TF-CBT were collected using a version of the TF-CBT Clinical Practices Questionnaire (CPQ; Deblinger, Cohen, Mannarino, Runyon, & Hanson, 2005), which was originally created by the developers of TF-CBT and subsequently modified for program evaluation purposes by the Project BEST directors, during the Pre and Post online questionnaires. The CPQ produces an overall competency score, as well as four subscale scores for various clinical skills; those subscales include general clinical skills (e.g., established an agenda and structure for each therapy session), psychoeducation and cognitive coping (e.g., helped the child and parent expand their vocabularies to describe emotions), trauma-focused interventions (e.g., encouraged the child to describe thoughts, feelings, or sensations experienced during the traumatic event), and behavioral management (e.g., discussed with parents how to use a behavioral reward system). We used the overall competency score (Cronbach’s α = .983) for all analyses. A copy of the modified CPQ used in this study is available from the second author upon request.

Youth mental health symptoms

Traumatic stress

We initially used the University of California at Los Angeles Posttraumatic Stress Disorder Reaction Index (PTSD-RI; Steinberg, Brymer, Decker, & Pynoos, 2004), a self- or caregiver-report tool, to measure trauma exposure and PTSD symptoms. However, use of the PTSD-RI was discontinued in early 2014 following a newly imposed cost for use of the measure, which was determined to impede its sustainable use in community settings. It was replaced with the Child PTSD Symptom Scale (CPSS; Foa, Johnson, Feeny, & Treadwell, 2001), a publicly available self or caregiver-report measure with equally strong psychometric properties. Of the 1,410 youth participants, we evaluated 47.2 percent with the PTSD-RI and 52.8 percent with the CPSS.

Depression

The Short Moods and Feelings Questionnaire (SMFQ; Angold et al., 1995) is a self- or caregiver-report measure of depressive symptoms in youth. Total scores are calculated from the combined results of the youth/caregiver versions.

Costs

We calculated the costs of the LC model in terms of direct (i.e., training, consultation, and administration) and indirect (i.e., lost opportunities for alternative activities) expenses. We adjusted all monetary amounts to 2015 values using the Consumer Price Index (CPI; Bureau of Labor Statistics, 2015) to account for inflation.

Direct

Direct costs of the LC model are associated with initial training and ongoing quality improvement activities, including (a) community readiness and consultation; (b) initial orientation and senior leader training; (c) learning sessions; (d) consultation calls with clinicians (bimonthly), senior leaders (monthly), and, when applicable, child welfare professionals (monthly); (e) data collection and management; (f) administrative coordination; and (g) training materials. These activities generate costs through trainer fees, transportation and hotel expenses, staff salary and benefits, and purchase of materials. We used a representative budget from Project BEST to estimate direct costs; all expenses in the budget were summed to produce a total direct cost of $96,168 per LC.

Indirect

Individuals who participate in implementation-related activities also experience indirect (i.e., “opportunity”) costs as a result of lost time spent on usual professional activities. These costs can be estimated using the value of alternative uses of a person’s time (i.e., “shadow price;” Heckman, 1974). We separately estimated indirect costs to (a) therapists, (b) clinical supervisors, (c) senior administrative staff, (d) child welfare caseworkers and (e) child welfare supervisors; although we did not examine outcomes for the latter three roles, their participation still contributed to the costs of the LCs.

We took two steps to estimate the average indirect cost for each role. First, we summed the time requirements for all relevant activities, including all in-person training activities, an online training course, regular completion of online questionnaires for quality assurance/program evaluation, and participation in consultation calls. Average hours of lost productivity were as follows: 64 for therapists/clinical supervisors, 48 for senior administrative staff, and 45 for child welfare caseworkers/supervisors. Second, we multiplied each estimate of lost productivity by the respective hourly shadow price for a given role. For therapists, we assumed that they could have provided direct service delivery during all hours of lost productivity; thus, for the shadow price, we used the Medicaid reimbursement rate for an hour of individual psychotherapy as reported by the South Carolina Department of Mental Health (i.e., $146; D. Blalock, personal communication, October 12, 2015). For supervisors, we assumed that they spent 10% of their time in direct service delivery and 90% of their time in supervision and administration, based on policies for clinical supervisors at the South Carolina Department of Mental Health (D. Blalock, personal communication, October 12, 2015). Thus, we used shadow prices of (a) $146 for direct service delivery; and (b) $33¹ for supervision and administration, based on the average hourly wage for social and community service managers in the National Compensation Survey – South Atlantic region (NCS-SA; Bureau of Labor Statistics, 2011). For senior administrative staff, child welfare caseworkers, and child welfare supervisors, shadow prices were taken from the average hourly wages in the NCS-SA (Bureau of Labor Statistics, 2011) for the categories of general and operations managers ($48), community and social service occupations ($22), and social and community service managers ($33), respectively.

We then summed the products, for each professional role, of (a) the average estimated indirect cost and (b) the average number of individuals per LC. These products were calculated as follows: therapists ($9,344 × 36.8), clinical supervisors ($2,835 × 9.0), senior administrative staff ($2,304 × 10.8), child welfare caseworkers ($990 × 12.0), and child welfare supervisors ($1,485 × 4.6). The total indirect costs of the LC model were estimated at $412,635.

Total costs

When direct and indirect costs were summed, the total cost per LC was $508,803. Stated differently, completion of an LC was associated with an expense of $11,523 per clinician or $4,684 per youth.

Analytic Strategy

The present study applied cost-effectiveness analysis (see Edejer et al., 2003; Siegel et al., 1996) to the costs and outcomes associated with implementing TF-CBT through the LC model. We conducted the analysis from the perspective of a community mental health service organization, given that such organizations (a) incur direct and indirect costs by participating in an LC and (b) seek to obtain measureable improvements in clinical outcomes through that participation. We performed the analysis in three steps: (1) estimation of change in clinical outcomes, (2) calculation of cost-effectiveness ratios, and (3) sensitivity analyses.

First, we used SPSS Statistics to estimate changes in outcome variables from pretest to posttest. Specifically, we used the MIXED procedure to estimate a multilevel (i.e., hierarchical) model, which nested measures within clinicians/youths (depending on the measure) within LC sites, using the following specifications: (a) a fixed, repeated estimator for linear pre-post change; (b) a random estimator for the intercept of each LC site; (c) fixed estimators for characteristics of youth, clinicians, and LC sites; and (d) restricted maximum likelihood (REML) estimation to account for missing data. Following the recommendations of Harris (2009), we expressed changes in outcome measures using the effect size Cohen’s d (i.e., standardized mean difference; J. Cohen, 1988), which we calculated such that a positive number represents a beneficial effect of the LC model.

Second, we calculated a cost-effectiveness ratio (CER), which represents the cost associated with a change of one SD unit (i.e., d = 1.0) on a given measure, for each service outcome. We calculated CERs by dividing d for a given outcome by (a) per-clinician costs for TF-CBT competence and (b) per-youth costs for youth mental health symptoms. We also calculated the proportional costs associated with achieving a small (0.2), medium (0.5), or large (0.8) effect (J. Cohen, 1988), assuming a linear relationship between cost of the LC model and effect size (i.e., we multiplied the CER for each measure by 0.2, 0.5, and 0.8, respectively). We performed this latter step to represent the costs associated with effects smaller than d = 1.0 that would still be considered meaningful.

Finally, we conducted sensitivity analyses (see Briggs & Gray, 1999) to examine how CERs were influenced by variations in four key parameters: (1) effect sizes; (2) number of professionals in an LC, which varied between LC sites; (3) the addition of a third learning session, which was done at 5 of the LC sites; and (4) reimbursement rates for clinical services, which vary between providers. Specifically, we calculated alternative CERs for minimum and maximum plausible values for each parameter.

Regarding interpretation of cost-effectiveness results, we considered several strategies that can be used to determine whether a given CER is cost-effective. One common procedure is to compare the relative cost-effectiveness of multiple study conditions using incremental cost-effectiveness ratios (ICERs; Edejer et al., 2003; Siegel et al., 1996). However, as noted previously, the present study lacked a comparison condition; thus it was not possible to calculate ICERs. Another strategy is to use accepted thresholds for metrics of disease burden (e.g., quality-adjusted life years), that can be applied in the absence of a comparison condition. We chose not to use these metrics, however, because they have been criticized for devaluing the provision of health care to individuals with preexisting disabilities (Mehlman, Durchslag, & Neuhauser, 1997; Persad, 2015). Instead, we used a novel interpretative approach in which we compared the CERs in the present study to thresholds that were based on findings from previous economic research.

Results

Service outcomes

Table 3 presents pre-post changes in outcome measures in terms of (a) raw scores and (b) effect sizes (d). The table also reports percent attrition at posttest for each measure, which ranged from 37.91% to 57.32%.

Table 3.

Changes in Outcome Measures From Pretest to Posttest in Multilevel Mixed-Effects Models

Variable	REML Estimate (SD)	Test Statistic	p	Attrition at Posttest (%)
TF-CBT competence	0.61 (0.98)	t192.055 = 3.10	.002	57.32
Traumatic stress symptoms
Caregiver report	12.54 (17.54)	t734.986 = 25.39	.000	38.02
Youth report	15.44 (17.53)	t738.579 = 31.57	.000	39.86
Depression symptoms	13.08 (16.13)	t668.685 = 27.02	.000	37.91

Note.

^*Statistically significant at p < .05 level.

REML = restricted maximum likelihood; TF-CBT = Trauma-Focused Cognitive Behavioral Therapy.

The model for TF-CBT competence showed a significant increase in overall competence from pre-LC to post-LC over the course of LC implementation. This analysis controlled for type of LC, number of learning sessions, clinician role, clinician degree, number of years at current organization, and an interaction term for clinician role * timepoint (pre vs. post). Furthermore, the models for youth trauma-related mental health symptoms (i.e., traumatic stress, depression) showed significant decreases from pre- to post-treatment according to caregiver and youth reports. All analyses for youth mental health symptoms controlled for type of LC, number of learning sessions, clinician role, gender, age, and length of treatment.

Cost-effectiveness

Table 4 presents the respective per-unit costs, effect sizes, and CERs for each outcome measure. The CER for TF-CBT competence, based on the per-clinician cost of $11,523 and a moderate effect (d = 0.617), indicated that an expenditure of $18,679 was associated with each unit increase in clinician competence. The CERs for youth trauma-related mental health symptoms, based on the per-youth cost of $4,684 and the respective effect size for each measure (ds = 0.712 to 0.881), indicated that expenditures associated with each unit decrease in symptoms were $6,548 for caregiver-reported traumatic stress, $5,318 for youth-reported traumatic stress, and $5,777 for depression. In addition, Table 4 reports the proportional cost-effectiveness for small, moderate, and large effect sizes; these values represent the cost associated with the respective magnitude of effect size.

Table 4.

Cost, Effect Size, and Cost-Effectiveness of Learning Collaborative Model by Outcome Variable

	Cost					Proportional Cost-Effectivenessc
Variable	Value	Unit	ES	CERa	Percentage of Thresholdb	Small effect	Moderate effect	Large effect
TF-CBT competence	$11,523	per clinician	0.617	$18,679	623%	$3,736	$9,339	$14,943
Traumatic stress symptoms
Caregiver report	$4,684	per youth	0.715	$6,548	17%	$1,310	$3,274	$5,239
Youth report	$4,684	per youth	0.881	$5,318	14%	$1,064	$2,659	$4,254
Depression symptoms	$4,684	per youth	0.811	$5,777	15%	$1,155	$2,888	$4,621

Note. All expenses are expressed in 2015 dollars.

ES = effect size (measured with Cohen's d); CER = cost-effectiveness ratio; TF-CBT = Trauma-Focused Cognitive Behavioral Therapy.

^aThe cost divided by the effect size for a given variable.

^bThe CER for each variable divided by the respective cost-effectiveness threshold value (i.e., $3,000 for TF-CBT competence, $39,000 for traumatic stress symptoms and depression symptoms) and multiplied by 100%.

^cThe CER for each variable adjusted by a factor of 0.2 (small effect), 0.5 (moderate effect), or 0.8 (large effect).

Sensitivity analyses

To conduct the sensitivity analyses, we first calculated the minimum and maximum plausible values for each parameter. For effect sizes, we divided the upper and lower limits of the 95% confidence interval for each REML estimator by that estimator’s standard deviation to calculate minimum and maximum effect sizes. For the number of professionals in the LC, we estimated the minimum and maximum value for each role (i.e., clinical therapists/supervisors, senior administrative staff, child welfare caseworkers/supervisors) by calculating M±1SD of individuals in that role across LCs. For number of learning sessions, we estimated the maximum plausible value by including additional direct (i.e., training expenses) and indirect (i.e., lost productivity) costs associated with a third learning session. For reimbursement rates, we calculated the maximum plausible value using the reimbursement rate ($334) for an hour of family therapy (which is higher than for individual therapy) obtained from a trauma-focused mental health center in South Carolina that receives higher reimbursement rates than does South Carolina Department of Mental Health (A. Oliver, personal communication, October 9, 2016).

We then calculated the cost-effectiveness of the LC model for all outcome measures by successively substituting the minimum and maximum plausible values into our calculations. Table 5 presents CERs for each outcome measure under these conditions. Across outcome measures, CERs were lowest in the analyses that used the maximum plausible effect sizes ($5,007 to $16,492) and highest in the analyses with the maximum plausible reimbursement rate ($9,964 to $34,997).

Table 5.

Cost-Effectiveness of Learning Collaborative Model With Maximum and Minimum Plausible Values of Parameters

	CERa for Outcome Variable
	Traumatic stress symptoms
Parameter	TF-CBT competence	Caregiver report	Youth report	Depression symptoms
Effect size
Minimum	$21,534	$7,096	$5,670	$6,228
Maximum	$16,492	$6,079	$5,007	$5,386
Number of participants
Minimum	$21,185	$7,427	$6,031	$6,552
Maximum	$17,351	$6,083	$4,940	$5,366
Maximum number of learning sessions	$22,082	$7,741	$6,287	$6,829
Maximum billing rate	$34,997	$12,269	$9,964	$10,823

Note. All expenses are expressed in 2015 dollars.

CER = cost-effectiveness ratio; TF-CBT = Trauma-Focused Cognitive Behavioral Therapy.

^aThe cost divided by the effect size for a given variable.

Interpretation of CERs

In order to interpret whether a given CER was cost-effective, we first had to identify relevant studies that provided thresholds of cost-effectiveness for clinician competence and youth mental health symptoms. For clinician competence in TF-CBT, we compared the observed CERs to the estimated implementation costs for a clinician to achieve competence in motivational interviewing (i.e., $3,000 in 2015 dollars; Olmstead, Carroll, Canning-Ball, & Martino, 2011; Tober et al., 2005). We chose this threshold because (a) cost estimates for achieving competence in a standard cognitive-behavioral EBI are not yet available and (b) those costs were related to activities (i.e., didactics and expert consultation) that are similar to those used in the implementation of interventions from a variety of theoretical orientations. Regarding the interpretation of youth mental health symptoms, we considered CERs below a threshold of $39,000 to be cost-effective based on (a) studies showing that untreated PTSD results in lifetime expenses of approximately $85,000¹ (Gustavsson et al., 2011) to $172,000 (Fang et al., 2012), with an average of $130,000; and (b) the assumption that use of an EBI would reduce those expenses by 30%, based on a cost-effectiveness analysis (Issakidis et al., 2004) that estimated the extent to which EBIs reduced the symptom burden of PTSD.

Table 4 also presents the percentage of each threshold represented by the observed CERs. The CER for clinician competence was 623% of (i.e., 6.23 times greater than) the selected threshold. The CERs for traumatic stress and depression symptoms fell between 14% and 17% of the respective threshold (i.e., the threshold was 5.96 to 7.33 times greater).

Discussion

In recent years, administrators and policymakers have shown increased interest in the economic impact of mental health services for children and adolescents (see McDaid, Park, Knapp, Losert, & Kilian, 2010; Ord, 2013). Indeed, such findings have important implications for the ability of mental health service organizations to sustainably provide EBIs in community settings. In the present study, we examined the cost-effectiveness of the LC model in the implementation of TF-CBT with community-based clinicians. The study had a number of strengths, including (a) measurement of key service outcomes for clinicians (i.e., competence in use of TF-CBT) and youth (i.e., trauma-related symptoms), (b) use of actual (rather than estimated) implementation costs for LCs, (c) calculation of a comprehensive cost estimate that included direct and indirect expenses, and (d) use of sensitivity analysis to model uncertainty in study parameters.

Our findings demonstrate that the cost-effectiveness of the LC model varied depending on the outcome being considered: Improvements in clinician-reported TF-CBT competence were associated with higher expenditures ($18,679 per clinician) than were improvements in youth mental health symptoms ($5,318 to $6,548 per youth). The observed effect size for TF-CBT competence was moderate and the proportional cost-effectiveness of that moderate effect was $9,339, whereas the observed effect sizes for youth mental health symptoms were primarily large with the proportional cost-effectiveness of those effects ranging from $4,254 to $5,239. Furthermore, estimated CERs varied widely under minimum and maximum plausible values for key parameters (i.e., from $16,492 to $34,997 for TF-CBT competence and from $5,007 to $12,269 for youth mental health symptoms). In general, it makes sense that greater expenditures were required to increase TF-CBT competence, given that clinicians delivered TF-CBT to an average of 2.46 youths and thus per-clinician costs of the LC model were considerably higher than the per-youth costs. It is also important to note that each of the aforementioned CERs includes the total cost of the LC model; thus, the simple sum of these CERs does not represent the cumulative expense for achieving all outcomes, as calculation of a cumulative benefits estimate is beyond the scope of cost-effectiveness analysis.

As noted previously, we interpreted the results of the present cost-effectiveness analysis by comparing the CERs obtained for the LC model to thresholds derived from relevant economic studies. The fact that the CER for the LC model exceeded the chosen threshold indicates that it may be less cost-effective than alternative implementation strategies in increasing clinician competence. This finding is consistent with research indicating the need for continued development of high-intensity, low-cost implementation strategies (Crome, Shaw, & Baillie, 2016; Powell, McMillen, Hawley, & Proctor, 2013). Nevertheless, many of the expenses involved in LC activities (e.g., inclusion of multiple disciplines) are expected to increase the sustainability of clinician use and competence of TF-CBT after the LC ends (Ebert et al., 2012; Saunders & Hanson, 2014) in addition to promoting competence during participation. It will be useful for future research to examine the cost-effectiveness of the LC model based on its long-term effects on TF-CBT competence, as those estimates may be more favorable than the CERs observed in the present study.

In contrast to clinician competence, the CERs for reductions in traumatic stress and depression observed in the present study were a full order of magnitude below the selected threshold. This finding indicates that LCs offer a relatively cost-effective approach, as compared to the expected economic impact of standard use of EBIs, for service organizations to achieve meaningful improvements in mental health functioning for youth with trauma-related disorders. The observed pre-post symptom reductions, which were in the moderate to large range (ds = 0.715–0.881), were comparable in terms of practical significance to improvements that have been observed in clinical trials of TF-CBT (Pollio et al., 2014). Importantly, there is considerable evidence that clinician competence with a given EBI is strongly associated with clinical outcomes (Beidas & Kendall, 2010; Herschell, Kolko, Baumann, & Davis, 2010) and, thus, such competence is critical to clinical and cost-effectiveness of implementation efforts. Taken together, the findings regarding cost-effectiveness of the LC model suggest that (a) this implementation model requires service organizations to invest considerable resources into promoting clinician competence; and (b) such investment, although not cost-effective when considered in isolation, tends to translate into economically efficient, competent delivery of TF-CBT to youth with trauma-related disorders.

The present study has several methodological limitations. First, it was not possible to directly compare the cost-effectiveness of the LC model to alternative, less resource-intensive implementation strategies (e.g., didactics only, didactics plus expert consultation). Second, although we examined 13 LCs conducted across five U.S. states, all LCs were administered by the same organization and in the same region of country (i.e., southeastern); thus, the present findings may not generalize to LCs delivered by other organizations or in other regions. Third, reimbursement rates, which were used to estimate the greatest proportion of expenses in the LC model (i.e., indirect costs from lost clinician productivity), were generated from two providers in South Carolina and may not generalize to other service providers. Fourth, although this study included a broad range of benefits for the LC model, it is likely that other beneficial outcomes (e.g., increased interprofessional communication and interagency collaboration, increased sustainment of TF-CBT competence) were not captured by the present analyses but would be an important area for future inquiry. Fifth, it is possible that use of different measures (e.g., observational codes for clinician competence, diagnostic interviews for youth mental health symptoms) could suggest different levels of cost-effectiveness for the LC model. Finally, attrition rates were high in the present study, although our use of the REML approach still allowed for robust estimation of outcomes (Harville, 1977).

In sum, the present study demonstrated the cost-effectiveness of the LC model in promoting the implementation of TF-CBT, an EBI that has been recommended for use in community settings (Pollio et al., 2014). Our findings suggest that there is considerable promise in the use of comprehensive implementation strategies, such as the LC model, to promote the adoption and use of EBIs for youth with trauma-related disorders. Of course, less comprehensive implementation strategies are often less expensive and more profitable for provider organizations to use. In particular, LCs often require substantial changes in organizational structure (e.g., maintenance of interdisciplinary advisory teams) and culture (e.g., increased emphasis on EBI fidelity during supervision). Thus, funding for implementation efforts must be competitive and sustainable to ensure their success with community-based clinical service agencies.