A Statewide Common Elements Initiative for Children’s Mental Health

Abstract

Many evidence-based treatments (EBTs) for child and adolescent mental health disorders have been developed, but few are available in public mental health settings. This paper describes initial implementation outcomes for a state-funded effort in Washington State to increase EBT availability, via a common elements training and consultation approach focused on 4 major problem areas (anxiety, PTSD, depression, and behavioral problems). Clinicians (N = 180) reported significant improvement in their ability to assess and treat all problem areas at post-consultation. Clinicians from organizations with a supervisor-level “EBT champion” had higher baseline scores on a range of outcomes, but many differences disappeared at post-consultation. Outcomes suggest that a common elements initiative which includes training and consultation may positively impact clinician-level outcomes, and that having “in-house” organizational expertise may provide additional benefits.

Introduction

Over 500 evidence-based treatments (EBTs) for child and adolescent mental health disorders exist, many with multiple randomized controlled trials (RCTs) supporting their efficacy in improving outcomes and functioning.^1,2 However, few EBTs are provided in public mental health settings,^3,4 leading to recognition that community-level implementation of EBTs is proceeding at “an unacceptably slow pace.”⁵(p208) To address the current science-to-practice divide between empirical knowledge and adoption, states such as Hawaii and New York have used a variety of strategies⁶ in an attempt to address barriers to EBT adoption and to build in facilitators, with the ultimate goal of increasing child and adolescent access to and receipt of EBTs. Organizational champions – individuals who make use of their reputations and informal status to support a change effort – have been identified as an important facilitator in theoretical models of implementation.^7–9 Empirical studies support the effectiveness of local champions in facilitating implementation, particularly in its earlier phases (e.g., initial adoption).¹⁰ Frequently, champions are situated at some level of an organization’s leadership,¹¹ where they are more likely to hold status and have an influence on the behavior of others.

The most commonly used implementation strategy has been in-person training for mental health professionals.¹² Hawaii, for example, has a 10-year history of providing training and additional supports (e.g., performance monitoring) for a range of specific EBTs and for common elements of EBTs.¹³ Eighteen states have undertaken initiatives to implement Trauma-Focused Cognitive Behavioral Therapy¹⁴ (TF-CBT), offering training, a period of expert case consultation, and supervisor support among other strategies.¹⁵ In New York State, an EBT center was established that offers in-person training and case consultation in EBTs for depression, disruptive behavior disorders, and Posttraumatic Stress Disorder (PTSD).¹⁶

Despite the fact that many states are offering training and other supports to expand EBT availability, state-funded initiatives often have limited evaluations of implementation outcomes (e.g., clinician adoption, fidelity) and/or child/adolescent outcomes. Furthermore, more detailed outcomes, when available, are typically written in reports to funders (e.g., state, foundations). Implementation and/or clinical outcomes have only been included in a limited number of reports in the scientific literature (for some exceptions, see ^13,16–20).

Evaluating the impact of EBT training efforts is a priority given its role as a primary implementation strategy. Recent findings^21,22 document the importance of the method of training delivery (active vs. predominantly didactic) and the types of specific implementation supports following initial training (e.g., case consultation, supervision). Training alone, without additional supports,²³ appears insufficient for changing clinician practice.^21,22 Empirical evaluations of training efforts and other included supports, such as consultation, will inform future efforts so that investments made in training are maximized.

This paper describes initial implementation outcomes (e.g., knowledge gains, adoption) for an ongoing statewide implementation effort to increase EBT availability in child public mental health in Washington State. The initiative, CBT+, is funded by the state organization overseeing public mental health services (i.e., Washington State Department of Social and Health Services, Division of Behavioral Health and Recovery) using Federal Block Grant dollars. The CBT+ Initiative provides clinicians with in-person training and expert consultation in cognitive behavioral therapy (CBT) and parent management training (PMT) for the most common mental health problems of childhood. From 2009–2013, nearly 500 clinicians from more than 53 organizations participated. This paper first describes the development of the CBT+ Initiative and rationale for a common elements training approach in Washington State, then describes the CBT+ training and consultation approach. Lastly, this paper presents – and discusses the implications of – clinician-level outcomes for knowledge and adoption, considering consultation dose, impact of being from an organization with a supervisor-level EBT “champion,” and change over time from multipoint analyses with a subsample of trainees.

CBT+ Development

The impetus for CBT+ grew out of the early excitement and potential promise of common elements approaches developed by Weisz and Chorpita.^25,26 The CBT+ Initiative evolved from an earlier statewide, public mental health TF-CBT training and consultation initiative. Trauma-focused CBT is a well-established EBT for trauma-related sequelae, including PTSD.²⁷ The TF-CBT focus began with a National Child Traumatic Stress Network grant (second author, PI). When grant support ended in 2005, the state provided funding to sustain yearly TF-CBT training and consultation. The TF-CBT Initiative was well-received by public mental health organizations as evidenced by registration slots filling within a few days, clinicians completing initiative requirements (i.e., participation in the 2-day training and 6-months of biweekly consultation calls), and multiple requests for additional trainings.

After 2 years of state support for TF-CBT, the lead faculty (first and second authors) and participating organizations noted that an intervention with a single primary focus (i.e., trauma impact) had limited reach in public mental health. Only a small percentage of children referred for mental health services have trauma-related symptoms as the presenting or primary treatment concern.²⁸ In Washington, exposure to a specific CBT approach – TF-CBT – seemed to create interest in how to apply CBT skills to other presenting problems. However, a broader, alternative approach was needed to efficiently and effectively extend EBTs to meet provider interest and the treatment needs of the children and their caregivers.

Rationale for a Common Elements Training Approach

CBT+ was developed to broaden EBT applicability for the public mental health population. Researchers have called for new approaches to EBT dissemination and implementation to extend reach and more rapidly achieve the goal of improving outcomes on a broad scale.^29.30 Following the work of Chorpita and Weisz, one potential strategy for accomplishing this goal includes the use of a common elements approach.³¹ Most EBTs for commonly occurring child and adolescent problem areas (depression, anxiety, behavior disorders) are comprised of discrete clinical interventions or strategies, termed “practice elements,” or “kernels” (e.g., relaxation, praise, exposure^31,32). Typically, common elements approaches are modular: practice elements can be delivered independently or together to achieve specific treatment outcomes.³³ An approach that provides training in elements relevant to the treatment of depression, behavior problems, and anxiety (including PTSD) would equip clinicians to treat the majority of children seeking services in the public mental health system. In Washington State specifically, a report²⁸ on child service utilization (N = 30,055) indicated that at least 70% had diagnoses that fell in 1 of these 3 areas. Two additional advantages for a modularized common elements approach included potentially better acceptability by clinicians,^25,34 given options for treatment flexibility when faced with comorbidity or treatment interference and the potential for streamlining training and consultation.^35,36 For many states and organizations, it is not feasible to pay for training and ongoing supervision in multiple single-focus EBTs to treat the range of diagnoses and problem areas seen in public mental health settings. Focusing on clinician competency in common practice elements could be an efficient, attractive, and cost-effective alternative.³⁷

Empirical Evidence for Common Elements Approaches

Although evidence is accumulating, it is important to note that the current enthusiasm for common elements approaches has advanced more rapidly than empirical support for their effectiveness. In a recently completed RCT of Chorpita and Weisz’s Modularized Approach to Treating Children and Adolescents (MATCH),³⁸ the modularized common elements approach resulted in better client outcomes than traditional EBT approaches or usual care at both post-treatment and at a 2-year follow-up.^25,39 Chorpita and colleagues have over 10 years of history testing a common elements approach to treating anxiety disorders with positive outcomes (e.g., ^26,40). Additional RCTs of MATCH are ongoing in Maine, Massachusetts, and California (Chorpita and Weisz, McArthur-funded). In the adult area, Barlow and colleagues developed and are testing a common elements, transdiagnostic approach (i.e., Unified Protocol⁴¹), with preliminary results from 2 small open trials⁴² and 1 small RCT⁴³ showing promise. A series of pilot projects also have tested the feasibility of common elements approaches in school settings.^44,45 For instance, Lyon and colleagues⁴⁴ trained clinicians in school-based health centers in a common elements approach adapted from the work of Chorpita et al.⁴⁶ Findings indicated acceptability and feasibility at the clinician level. Globally, researchers developed a common elements intervention for delivery by lay counselors, with pilot data showing promise in Southern Iraq and the Thailand-Burma border.⁴⁷ Recently completed RCTs in both sites are demonstrating potential for positive results.

Method

CBT Plus Training and Consultation Approach

CBT+ includes a 3-day, in-person, skills-based training provided by the CBT+ developers (first and second author) and other CBT+ faculty, all of whom have CBT expertise (e.g., national trainers in other EBTs, received MATCH training, developers/trainers for other common elements approaches⁴⁷). Training is followed by 6 months of biweekly phone consultation, predominantly provided CBT+ faculty. Organizations are required to send 1 supervisor and 2–3 clinicians so that organizations have a participating supervisor and a clinician cohort. Participants also have access to a yearly, advanced, 1-day booster training and the CBT+ listserv. Supervisor-specific supports are also offered⁵⁶. Yearly trainings are announced through email distribution by the state division for mental health to regional networks that manage community mental health organization contracts and via announcement on the CBT+ listserv.

Training content for CBT+ includes a focus on assessment and treatment for 4 presenting problem areas: depression, anxiety, trauma-related anxiety (i.e., PTSD), and behavior problems. In the area of assessment, trainees practice scoring completed standardized measures for case vignettes and practice giving assessment feedback through behavioral rehearsal in small peer groups. Assessment measures used in CBT+ were selected based on the following characteristics: 1) limited items; 2) ability to score quickly by hand; 3) available in the public domain (i.e., no cost to organizations); and 4) strong psychometric properties (e.g., Child Posttraumatic Stress Scale⁴⁸).

CBT+ training focuses on emotion regulation skills, exposure, and cognitive reprocessing for anxiety and PTSD; on emotion regulation skills, behavioral activation, problem solving, and cognitive reprocessing for depression; and on PMT strategies (e.g., praise, rewards, consequences) for behavioral problems. Given the trauma-focused origins of CBT+, PTSD treatment includes positioning the common elements within the TF-CBT model.¹⁴ Following the MATCH approach,³⁸ the CBT+ model encourages a primary focus on the indicated elements for each problem area, but supports modularity – or tailoring – when clinically indicated (e.g., adding PMT to anxiety treatment when a child has comorbid anxiety and behavior problems). Training also targets CBT general competencies (e.g., agenda setting, collaborative homework assignment⁴⁹) found to occur infrequently in community mental health settings.²⁹ The process of CBT+ training is active and includes experiential learning activities (e.g., cognitive restructuring activity for a situation in the clinicians’ own life), trainer modeling and video demonstration of skills, trainee behavioral rehearsal of practice elements with both peer and trainer feedback and coaching, and small and large group work. Training is tailored to focus on the typical clients, setting constraints (e.g., 50-minute sessions), and identified EBT implementation challenges in public mental health settings (e.g., decision-making when faced with comorbidity, engagement strategies).

Within 3 weeks of the training, clinicians begin CBT+ expert-led consultation calls that focus on implementing CBT with clients on their caseloads. Calls are led by the first, second, and fifth authors, and by other CBT+ faculty. Each call group includes 3–4 organizational teams with approximately 10–15 trainees (clinicians and supervisors) per call. A specific case presentation format is used to help clinicians quickly get to the clinical question or difficulty so that the consultant and peers on the call can plan for concrete next steps. Calls involve reviewing assessment data to determine clinical focus, application of CBT+ components to cases, and problem-solving challenges with child and caregiver engagement. Trainees are expected to present at least 1 case over the consultation period and attend 9 of 12 calls in order to receive a CBT+ certificate of participation.

CBT+ emphasizes identifying and leveraging local EBT “champions” to support implementation and sustainment within their organizations. One ongoing mechanism for building and utilizing champions in CBT+ involves inviting individuals at the supervisor level who successfully completed the CBT+ Initiative to co-facilitate consultation calls with one of the CBT+ faculty. This benefits both the consultation group and the organization by bringing champion’s experience in using CBT+ to the consultation call group and by enhancing the champion’s CBT+ expertise through an opportunity to hone supervisory/consultation skills, which may advance CBT+ practice and supervision in the organization.

Procedures

Data for the current study predominantly come from an evaluation of all CBT+ trainees during the first 3 years of CBT+ (2009–2011; 4 cohorts). Trainees completed questionnaires at the beginning of training (via paper and pencil, at the training) and after the 6-month consultation period (via online survey, same content), henceforth referred to as the pre-post sample. For 3 of the 4 cohorts (2010–2011), data also included expert consultant ratings for all case presentations on each of the 12 consultation calls (4-item online survey). Participation in the pre-post evaluation was expected as part of Initiative involvement. Although no incentives were provided, participants received multiple reminders to complete the online surveys and also received telephone reminders in an attempt to increase completion rates. Clinicians and supervisors who participated in the CBT+ training did not seem to receive any systematic incentives or workload reductions from their organizations; however, the Initiative provided 18–20 hours of Continuing Education Credits for attending the training.

Supplementing these data, all trainees in these 3 cohorts were asked to also participate in a more time-intensive, longitudinal evaluation of CBT+, for which incentives were available, given the increased burden on participant time. Those who agreed received a $10 gift card for completion of additional measures, via web survey, at each of 4 assessment points: pre-training, post-training (within 1 week), after the 6-month consultation was completed, and at 3 months post-consultation. All evaluation activities were reviewed by the Washington State IRB and were exempted from review. This manuscript focuses predominantly on analyses with the pre-post sample of participants in the larger evaluation, with secondary analyses using the subsample that completed the longitudinal evaluation.

Participants

Participants were clinicians and supervisors employed at public mental health clinics in Washington State (see Table 1) who participated in the CBT+ Initiative. Of the 400 total participants in the 4 Washington State CBT+ training cohorts between 2009–2011 (fall 2009, fall 2010, spring 2011, fall 2011), 320 completed the pre-training survey. Of these, 36 were not clinicians or supervisors (i.e., administrators, case managers, or in other roles) and were excluded, resulting in a sample of 284 (78% participation rate). Of the 284, 180 completed the 6-month, post-consultation, and follow-up assessment, and constitute the primary sample for this study. Reasons for attrition (N = 104) included departure from organization (n = 27; 26%) and general non-response (n = 77; 74%) (see Missing Data Analyses section for examination of attrition bias). Pre-post sample participants (N = 180) were predominantly female (80%), Caucasian (84%), Master’s-level clinicians (93%) in their late twenties and thirties (68%) (see Table 1). The longitudinal subsample consisted of 71 participants who represent 54% of CBT+ trainees from the 3 cohorts in years that included the longitudinal evaluation (2010–2011).

Table 1.

Participant demographics

Variable	Pre-Post sample n = 180		Total sample N = 284
	n	%	n	%	p1
Clinician	164	91.1	255	89.8	.333
Supervisor	33	18.3	52	18.3	.989
Female	144	80.0	228	80.3	.751
Ethnicity or Race					.232
Caucasian	152	84.4	236	83.1	--
African American	6	3.3	7	2.5	--
Hispanic or Latino	10	5.6	23	8.1	--
Asian	6	3.3	9	3.2	--
Other	5	2.8	7	2.5	--
Educational background					.525
High school/AA	0	0	2	0.8	--
4-year college	9	5.0	13	4.6	--
MSW	44	24.4	68	23.9	--
Other Masters/PhD	127	70.6	201	70.8	--
Age					.389
Under 25	5	2.8	6	2.1
25–29	49	27.2	68	23.9
30–39	13	40.6	122	43.0
40–49	28	15.6	46	16.2
Over 50	25	13.9	41	14.4
Missing	0	0	1	0.4
Frequency of CBT use					.409
Not in last 5 years	6	3.4	10	3.5
Rarely	7	3.9	15	5.3
Occasionally	11	6.1	22	7.7
Sometimes	32	17.8	51	18.0
Often	72	40.0	111	39.1
Almost always	52	28.9	75	26.4
Receives CBT supervision	56	31.3	77	27.2	.043
Receives EBT-specific supervision	72	40.2	112	39.7	.819
Participants in champion agencies (CA)	37	21.3	33	17.3	.334
Supervisor in CA (of total supervisors)	5	16.1	4	19.0	.785
Left agency during study	8	4.4	28	9.9	< .001
	Mean	SD	Mean	SD
Consult call attendance	9.4 (n = 155)	1.9	8.7 (n = 230)	2.7	< .001
Years providing therapy	5.3 (n = 165)	5.0	5.6 (n = 262)	5.9	.248

¹Statistical tests compare pre-post sample with baseline-only sample (baseline-only data not depicted)

Measures

Demographics

Participants completed a questionnaire that included demographic (e.g., sex, ethnicity) and background information (e.g., role in the organization [clinician, supervisor], years of experience).

Self-report of skill

Participants self-reported skill/ability for individual treatment elements for each of the 4 problem areas addressed in the CBT+ training (0, do not use; 1, minimal; 2, minimal to moderate; 3, moderate; 4, moderate to advanced; 5, advanced). Participants were asked to rate their skills and understanding of elements separately for each problem area. Therefore, certain items were common across all or some problem areas (e.g., using assessment measures, psychoeducation). Other items were unique by problem area (e.g., for depression: “Pleasurable activity scheduling (help the child identify, plan for, and engage in fun activities)”; for anxiety: “Facing your fears/exposure (developing a list of feared situations/memories, helping the client gradually face them)”; for behavior problems: “Positive parenting (increase positive time together)”; and for trauma/PTSD: “Trauma narrative (developing and working with child to modify cognitive distortions throughout narrative)”).

Principal components factor analyses with Varimax orthogonal rotation was conducted showing a clear inflexion, with 4 factors accounting for 68.3% of the variance: depression/anxiety, behavioral problems, trauma, and a fourth factor signifying assessment skill. Items for depression/anxiety were divided into 2 scales, despite clustering as 1, to aid interpretation and explore the possibility of different trajectories of change. The 5 final factors included anxiety (7 items; Cronbach’s α = .89), depression (8 items; Cronbach’s α = .88), behavioral problems (10 items; Cronbach’s α = .94), PTSD (10 items; Cronbach’s α = .94), and assessment (4 items; Cronbach’s α = .90).

Evidence-based clinician-level activities

The Evidence-based Clinician Checklist is a locally developed questionnaire used to assess participant practice characteristics consistent with EBTs. Items are rated on a 4-point Likert scale (1, rarely; 2, occasionally; 3, regularly; 4, almost always). The checklist was derived from review of the EBT literature for the purpose of specifically capturing the hallmark clinician activities associated with delivery of CBT. By design it was intended to include only essential activities, described in a behaviorally specific way, so as to be readily understood by clinicians. Sample items included, “I use standardized measures or questionnaires to identify and measure specific clinical conditions (depression, PTSD, ADHD, behavior problems),” and “I give verbal or written feedback about my diagnostic or clinical impressions to the child and/or the child’s caregiver and establish agreement on the problems to address in treatment.” Because of the late introduction of this measure to the study, these data are only available for 3 of 4 cohorts of the sample (fall 2010, spring 2011 & fall 2011 training). Principal components factor analysis on the original 8 items revealed that 3 items had unacceptable component loadings (< .5), degrading scale internal consistency. The scale was reduced to one 5-item factor, accounting for 58.4% of the variance (Cronbach’s α of .82), assessing: 1) use of standardized assessment, 2) providing assessment feedback and establishing agreement on problems, 3) providing information on treatment options, 4) using a specific EBT matched to clinical need, and 5) re-administering measures.

Evidence-based organization-level activities

The Evidence-based Organizational Checklist is a questionnaire used to assess organizational characteristics that support the delivery of EBTs (REMOVED, unpublished measure). Items are rated on a 4-point Likert scale (1, never; 2, occasionally, 3; most of the time; 4, ongoing/routine). This measure was also introduced later in the study and is only available for 3 cohorts. The checklist was derived from review of the dissemination and implementation literature for the purpose of specifically capturing the hallmark organizational activities associated with EBT uptake. Again, by design it was intended to include only essential activities and describe them in a behaviorally-specific way. This measure was included because participation in CBT+, or other EBT training, is sometimes an organization’s first foray into EBTs. Organizations may make some changes to support EBTs over the course of the 6-month consultation period (e.g., instituting routine screening and assessment; setting up a CBT+ specific supervision group). Sample items for this measure include, “Executive leadership (e.g. administrators, directors) explicitly and repeatedly express support for and promote use of Evidence Based Practices,” and “Clinicians are provided with EBT training opportunities and ready access to EBT materials (manuals, handouts, equipment).” After principal components factor analysis, the original 7-item scale was reduced to 6 items (1 had a low factor loading [< .5]), assessing: 1) executive leadership support for EBT; 2) availability of EBT training and materials, 3) EBT clinical supervision; 4) policies around screening and assessment; 5) agency expectation is to use matched EBT, if available; and 6) procedures for fidelity monitoring. These items represented a single factor, accounting for 60.3% of the variance, with a Cronbach’s α = .87.

Consultation dose

Consultation call attendance was reported by the expert consultants leading the consultation calls.

Case presentation quality ratings

Clinicians’ case presentations on the consultation calls were rated by the expert consultant to examine adherence to training expectations. Consultants received an online survey on the day of their consultation call and rated each presentation for presence/absence on: 1) assessment data; 2) identification of a clinical target; 3) specification of 1 or more appropriate CBT+ component(s) (e.g., mentioning use of exposure for anxiety treatment); and 4) homework assignment. These items were summed to create a consultation call quality index for each presentation (range: 0–4).

Organizational Champion Status

Organizations were dichotomized on the basis of whether they had an internal, supervisor-level EBT champion. For purposes of this study, EBT champion was operationalized as a supervisor who was served as a co-consultant on at least 1 consultation call. Because the literature suggests that champions in leadership roles may have more influence,¹¹ only supervisor-level champions were invited to be consultation co-facilitators. CBT experts who led the consultation calls identified the EBT champion co-consultants as vocal proponents and adopters of either TF-CBT (before the CBT+ Initiative began) or CBT+ (when these individuals themselves were consultation call participants). Champions also served an advisory role for the Initiative and participated in approximately once or twice a year in-person or conference calls.

Results

Descriptive Analyses

Table 1 presents demographic and descriptive data stratified by participants (clinicians and supervisors) with only baseline data (N = 284 for most data; n = 217 or 214 for evidence-based clinician and organizational-level activities, respectively) and those with follow-up data (N = 180 for most data; n =146 or 144 for clinician and organization evidence-based activities). Call attendance data were available for 155 (86.1%) participants (attendance data were not available for two 2009 consult call groups). For these participants, the mean number of calls attended was 9.4 (SD = 1.9, range = 1–12), with 80% (n = 124) attending 9 or more calls. More than half (56%, n = 87) attended 10 or more calls, surpassing the 9 required to receive the CBT+ certificate of completion. The mean number of cases presented by participants during the 6-month consultation period was 2.0 (SD = 1.3, range = 1–7).

Missing Data Analyses

Cross tabulations with chi-square analyses and t-tests were run to examine any differences between participants with only a baseline survey and those with a follow-up. There were no significant differences on demographics or outcome variables at baseline. However, those with baseline-only data attended fewer consultation calls (M = 7.1 vs. 9.4, t_(98.5) = −6.6, p < .001), were less likely to receive clinical supervision (91.3% vs. 97.2%, χ²₍₂₎ = 4.8, p = .028) and supervision in CBT (20.2% vs. 31.3%, χ²₍₁₎= 4.09, p = .043), and were more likely to have left their organization during the 6-month period (19.2% vs. 4.4%, χ²₍₁₎ = 16.5, p < .001).

Primary Analyses

Change over time

Table 2 depicts paired t-tests indicating that participants reported significant improvements on self-reported skill in all five areas: anxiety (t₍₁₇₇₎ = −10.01, p < .001, d = 0.64), depression (t₍₁₇₃₎ = −8.38, p < .001, d = 0.76), behavioral problems (t₍₁₇₅₎ = −5.72, p < .001, d = 0.43), PTSD (t₍₁₇₆₎ = −11.10, p < .001, d = 0.84), and assessment (t₍₁₇₈₎ = −15.06, p < .001, d = 1.16). They also reported significant improvements on EBT activities at the clinician (t₍₁₄₃₎ = −10.24, p < .001, d = 0.28) and organizational level (t₍₁₄₅₎ = −3.27, p = .001, d = 0.86). Cohen’s d effect sizes (calculated as a within-subject effect size, adjusting for correlations between the means; equation 8⁵⁰) were moderate (> .5) to large (> .8) for most tests, with skill conducting assessment, clinician EBT checklist scores, and PTSD/trauma showing the largest effect sizes.

Table 2.

Pre-post sample: Paired t-tests for change in self-reported understanding and skill

	Pre-training		Post-consultation		t	df	p	d1
	Mean	SD	Mean	SD
Skill - depression	3.23	0.70	3.70	0.65	−8.38	173	< .001	0.64
Skill - anxiety	3.01	0.77	3.59	0.70	−10.02	177	< .001	0.76
Skill - behavior problems	3.37	0.83	3.73	0.80	−5.72	175	< .001	0.43
Skill – PTSD/trauma	2.90	0.94	3.57	0.76	−11.10	176	< .001	0.84
Skill - assessment	2.40	1.13	3.63	0.86	−15.06	178	< .001	1.16
Organization checklist	2.98	0.71	3.16	0.66	−3.27	145	< .001	0.28
Therapist checklist	2.66	0.71	3.21	0.60	−10.24	143	< .001	0.86

¹Cohen’s d

Association with consultation

Two-Way Repeated-Measures (RM) ANOVAs were run to explore whether changes were related to consultation call dosage or participation (operationalized as number of cases presented and as the case presentation quality index). There were no significant associations.

Case presentation quality

Table 3 provides detail from the independent case presentation quality ratings for the first 4 cases presented. Presentation quality was high across all 4 case presentations (range: M: 3.4 – 3.9). Three RM-ANOVAs were run to explore variation within each of the first 3 case presentations (the fourth was underpowered, n = 12). All 3 were significant. Pairwise between-measure tests, with Sidak’s adjustment for multiple comparisons, indicated that, of the 4 elements rated by consultants, significantly fewer participants specified assigning homework. In addition, second and later case presentations were more likely to include mention of specific CBT+ components.

Table 3.

Pre-post sample: Case presentation descriptives for the first four cases presented by each participant

	Case presented
	1st	2nd	3rd	4th
n participants presenting case	134	71	36	13
Of those who presented cases:
% presented assessment data	99.3a,b	98.6a	100a	100
% mention CBT+ clinical target	97.0c,d	95.8b	100b	100
% specify CBT+ components	85.1a,c,e	91.5c	91.7c	100
% specify client homework	57.5b,d,e	59.2a,b,c	55.6a,b,c	84.6
Mean (SD) number of targets addressed	3.39 (.78)	3.45 (.69)	3.47 (.61)	3.85 (.38)
RM-ANOVA Omnibus tests
df	3, 131	3, 68	2, 34	--
F	32.0	14.1	14.2	--
p	< .001	< .001	< .001	--

Note. Proportions with the same subscripts differ significantly at p < .001 within each case presentation using pairwise between-measure comparisons with Sidak’s adjustment for family- wise error. The 4^th presentation was underpowered for statistical testing.

Exploratory Analyses

Association with champion organization status

The potential impact of having an internal organization EBT champion was explored by examining whether the 37 participants who were in organizations with a champion had different mean scores at either assessment or different rates of change. Analyses involved t-tests and Two-Way RM-ANOVAs (time x champion status) to test for statistical significance. Participants from champion organizations reported higher scores at baseline for skill in 3 of 5 areas: anxiety (t_(106.8) = −2.26, p = .026), PTSD (t_(80.1) = −3.90, p < .001), and assessment (t₍₁₇₁₎ = −2.21, p < .039). They also had higher scores for evidence-based activities at both the clinician (t₍₁₃₇) = −2.87, p = .005) and organizational level (t₍₁₃₈₎ = −4.00, p < .001). Group differences for skill in depression and behavior problems were not significant. At post-consultation, participants from champion organizations reported higher scores only for skill in PTSD (t₍₁₇₁₎ = −2.76, p = .006) and evidence-based activities at the clinician (t₍₁₃₉₎ = −2.79, p = .006) and organizational levels (t₍₁₃₉₎ = −4.31, p < .001). There were no statistically significant time x champion status interaction terms; status was not related to rate of change on any of the outcomes measures (self-report of skill in the five areas, evidence-based clinician or organizational activities).

Associations between organization status and call ratings for the first case presentation were also explored (as most participants had at least one case presentation). There were no differences in mean quality ratings between groups. Looking at the 2 elements with more variation (i.e., specifying CBT + components, assigning homework), a greater percentage of participants from champion organizations reported assigning homework (75% vs. 52%, χ²(1) = 4.7, p = .03).

Longitudinal subsample analysis

To more sensitively examine change over multiple time points, the same set of outcome variables with the longitudinal sample who had baseline data on all outcome measures (n = 71) was examined. A series of multilevel growth models with full maximum likelihood estimated the overall linear and piecemeal rates of change between time points. For each outcome, −2 Log Likelihood deviance statistics determined the best-fitting model (see Table 4). Four outcomes significantly improved at a linear rate over all time points from pre-training through 3-months post-consultation follow-up: depression skill (β = .25, p < .001); anxiety skill (β = .30, p < .001); behavior problem skill (β = .30, p < .001); and the organization checklist (β = .10, p < .001). Three outcomes significantly improved from pre-training through consultation, but then plateaued (i.e., no significant change) from the end of consultation to the 3-month follow up: PTSD skill (β = .38, p < .001), assessment skill (β = .70, p < .001), and clinician evidence-based activities (β = .34, p < .001). This plateau does not appear related to ceiling effects; for all outcomes, standard deviations were not significantly smaller and negative skew was not significantly greater in the last two time points, compared to the first two. Across outcomes, mean scores in the last two time points ranged from 1.5 to 2.5 standard deviations lower than the highest possible score. These statistics indicate that there was additional variance possible in each of the two final time points for all outcome measures (analyses available upon request).

Table 4.

Longitudinal sample: Mixed model regression coefficient estimates for rate of change over time on all outcome variables

Variable	Pre-training estimate (SE)	Pre-training to follow-up linear slope (SE)	Slope p	--	--
Skill implementing depression	3.21 (.09)	.25 (.03)	< .001	--	--
Skill implementing anxiety	3.01 (.09)	.30 (.03)	< .001	--	--
Skill implementing behavior problems	3.30 (.09)	.30 (.10)	< .001	--	--
Organization checklist1	3.11 (.09)	.10 (.02)	< .001	--	--

	Pre-training estimate (SE)	Pre-training to post- consultation slope (SE)	Slope p	Post- consultation to follow-up slope (SE)	Slope p
Skill implementing trauma/PTSD	2.96 (.11)	.38 (.05)	< .001	.10 (.07)	.192
Skill assessing for all 4 components	2.52 (.13)	.70 (.07)	< .001	−.08 (.10)	.440
Therapist checklist1	2.73 (.09)	.34 (.04)	< .001	−.01 (.002)	.850

¹The therapist and organization checklist were not assessed at T2 (post-training), because training would be unlikely to impact these ratings

Discussion

This project was designed to evaluate implementation factors in the context of a common elements training program, delivered as part of a statewide initiative. The majority of clinicians who remained with their organizations completed the required consultation calls, suggesting some acceptability for both the CBT+ training process (in-person training, consultation calls) and potentially for the components-based approach. A number of clinicians completed more than the required number of consultation calls and case presentations, which may indicate that clinicians perceived the calls to be beneficial to their practice and worth the time investment. Based on their self-report, clinician skill improved significantly across the 4 targeted problem areas (i.e., depression, anxiety, behavior problems, and PTSD), with effect sizes – albeit calculated without a comparison condition – in the moderate to large range. However, clinician self-report of skill may or may not be representative of actual behavioral improvements.⁵¹ Independent consultant ratings of clinicians’ case presentations provide some additional information on clinician adoption of the CBT+ model, specifically that clinicians noted a CBT+ area of focus (e.g., depression), used standardized measures and mentioned CBT+ components when discussing cases. However, by independent consultant report, clinicians were significantly less likely to report having assigned therapeutic homework activities. Other research on usual care²⁹ indicates that, as in CBT+, homework assignment and review infrequently occurs, despite being an important CBT competency.⁴⁹ These findings bolster past research, suggesting a need for greater attention to client homework assignment and review during training and consultation.

Interestingly, participants from organizations with an internal champion had higher pre-training levels of self-reported skill for 3 of the 5 areas (anxiety, PTSD, and assessment), although other than PTSD, differences disappeared by the post-consultation assessment point. That is, clinicians from organizations with a champion did not demonstrate an increased rate of change. In the area of evidence-based clinician and organizational level activities, differences between clinicians from organizations with and without an organizational champion were maintained from pre-training to post-consultation. These findings suggest that having an organizational champion might be more beneficial in these areas. One promising finding, given low base rate across Initiative participants, was that those from organizations with an internal champion were more likely to specify homework activities in case presentations, suggesting that organizations with champions may be paying greater attention to and/or supporting this general CBT competency. Among other things, positive organizational culture and climate have been linked to EBT uptake, decreased clinician turnover, and improved child outcomes.^52,53 The literature promotes the idea of EBT champions within organizations for implementing and sustaining EBTs.⁵⁴ Although not the focus of this paper, the field would benefit from additional research on how to build champions within organizations, or other internal network change agents,⁵⁵ for promoting and advancing EBTs.

The multi-time point analyses provide additional insight into the pattern of CBT+ participant change over time, and in particular, the importance of consultation. Although some outcomes continued to improve through the 3-month post-consultation follow-up, a number of gains – skill in assessment, skill in treating PTSD, and evidence-based clinician activities – plateaued. Given the small sample size (n = 71), these analyses are exploratory in nature, but reflect other findings^21,23 from clinician training on the potential need for ongoing consultation or supervision. Organizations likely need to explicitly continue EBT supports to maintain improvements, either through local supervisors’ continuing to provide support around specific EBT activities⁵⁶ or potentially using newly emerging technologies that include clinician reminders⁵⁷ or other strategies.

As is common in public mental health,⁵⁸ clinician turnover in CBT+ was notable. At least 9% of participants in this study left their organization in the short 6-month window following the in-person training. As turnover rates are approximately 20% per year in public mental health, rates in the CBT+ initiative are in-line with national averages.⁵⁸ Organizations need access to efficient models for ongoing EBT training, as most cannot afford to send new hires to multiple trainings, each with their own set of post-training activities (e.g., required consultation, booster trainings) and fidelity monitoring procedures. Findings from the current study indicate that clinicians reported improved skill over time, participated in the post-training supports, and maintained skills post-consultation. A common elements approach, like CBT+, may offer a more streamlined and targeted approach (i.e., 1 training) for organizations to prepare new hires to treat the most common child mental health problems.

One limitation of many evaluations of training programs for behavioral health services – including this one – is that change in self-reported knowledge and skills may not necessarily reflect the acquisition or use of skills in actual practice,^59,60 although recent work suggests that self-report may have some validity.⁶¹ Regardless, determining whether such changes have occurred may require more rigorous and direct methods. The inclusion of consultant ratings, although more rigorous than clinician self-report, still relied on assessing clinician self-report of their sessions with clients. Among other more rigorous options, one possibility includes objectively evaluating skills in the EBT elements by either using behavioral rehearsal with a standardized patient⁶² or employing direct observation of skill use with clients.⁶³ However, both of these methods are costly and resource-intensive. Behavioral rehearsal methodology was included in 1 CBT+ cohort, with analyses underway. Preliminary findings suggest that behavioral rehearsal methods corroborate self-report of skill improvement over time. Additional efforts to devise feasible methods for evaluating the fidelity of EBT implementation following training are needed.⁶⁴ A second limitation is the high non-response rate at the 6-month follow-up point, even among those clinicians who remained at their agencies. However, analyses examining attrition indicated no demographic or baseline differences in outcomes between those with and without data at the follow-up, and findings from the longitudinal sub-sample bolster the larger pre-post sample findings. The inclusion of incentives to increase response rate at the post-assessment for the pre-post sample may have improved retention.

Implications for Behavioral Health

The large investments in EBT training in recent years may not have produced presumed benefits. Research demonstrates that training alone^21,23 is unrelated to actual practice change. Without a larger investment that includes post-training supports, investments in training may produce only poor or suboptimal outcomes.⁶⁵ The CBT+ Initiative in Washington State is an example of how this ongoing support could be provided. Although predominantly based on self-report, the study suggests that the combination of training and consultation may result in knowledge gain and implementation of a common elements model, CBT+. A common elements approach potentially offers a more streamlined, integrated, and sustainable approach to training mental health providers in EBTs.^38,39 In addition, this study offers some support for the beneficial role of organizational champions in supporting cross-cutting CBT competencies, such as homework assignment, within their organizations. As more states roll-out EBT initiatives, it will be important to include evaluation of implementation and client outcomes to ensure that limited resources are achieving targeted goals of improving clinician practice and positively impacting child and family outcomes.