The Effectiveness of Training Community Mental Health Therapists in an Evidence-Based Intervention for ASD: Findings from a Hybrid Effectiveness-Implementation Trial in Outpatient and School-Based Mental Health Services

Abstract

An Individualized Mental Health Intervention for ASD (AIM HI) was developed in collaboration with community stakeholders for delivery in mental health services in response to therapist and caregiver identified need to improve services for children with ASD. Primary findings from a cluster randomized Hybrid Type 1 effectiveness-implementation trial conducted in publicly funded mental health programs demonstrated the effectiveness of AIM HI therapist training on child and caregiver outcomes. This study examined therapist outcomes and therapist experience as a moderator of training effects. Mental health programs were randomized to immediate AIM HI training or usual care. Therapists and child/caregiver clients were recruited from participating programs. Therapists in the AIM HI training condition received consultation for 6 months while delivering AIM HI. Differences between training conditions were examined using multilevel modeling. Therapists receiving AIM HI training were observed to use more extensive active teaching strategies with caregivers, engagement strategies with children, strategies promoting continuity of care, and had more structured sessions with more extensively pursued skill-building. Therapist licensure moderated some training outcomes. The current study provides support for evidence-based practice implementation in usual care mental health services.

Publicly funded mental health services are important in providing care for school-age children with autism spectrum disorder (ASD) due to the high rates of co-occurring psychiatric disorders (Brookman-Frazee et al., 2018; de Bruin et al., 2007; Gjevik et al., 2011; Leyfer et al., 2006; Simonoff et al., 2008; Soke et al., 2018) and the common presentation of challenging behaviors in this population (Mandell, Walrath, Manteuffel, Sgro, & Pinto-Martin, 2005). Previous research conducted in these settings has shown that while therapists often have children with ASD on their caseloads, community therapists report a lack of training to effectively work with this population (Brookman-Frazee, Drahota, Stadnick, & Palinkas, 2012). Access to psychotherapy or mental health services is commonly reported as an unmet need for children with ASD and families are likely to identify a provider’s inability to treat their child with ASD as a barrier in obtaining therapy and mental health services (Chiri & Warfield, 2012). Although evidence based interventions (EBIs) using function-based behavioral approaches (Horner et al., 2002; Wong et al., 2015; Steinbrenner et al., 2020) and parent mediated approaches (Bearss et al., 2015; Postorino et al., 2017) to reduce behavior problems in children with ASD exist, there is limited delivery of EBIs to children with ASD in community mental health services and EBI approaches that are used are often delivered with low intensity (Brookman-Frazee, Taylor, & Garland, 2010).

One barrier to the delivery of EBIs in mental health services for clients across mental health conditions is the lack of effective therapist training (Frank, Becker-Haimes, & Kendall, 2020). Lack of access to effective EBI training and implementation support remains a significant factor limiting access to effective mental health care for clients in need (Kilbourne et al., 2018). For ASD clients specifically, limited therapist training has been identified as a barrier in accessing appropriate mental health services for adolescents and adults with ASD (Lake et al., 2014; Narendorf, Shattuck, & Sterzing, 2011) and parents of children with ASD view limited therapist training in ASD as a barrier to effective services for their child (Kalb, Stuart, Mandell, Olfson, & Vasa, 2017).

The recognition of the importance of therapist training across EBIs has prompted more research on the process and outcomes of therapist training. A recent systematic review of therapist EBI training by Frank and colleagues (2020) reported that, consistent with previous research (Beidas & Kendall, 2010; Herschell et al., 2010; Rakovshik & McManus, 2010), while therapist knowledge and attitudes toward EBIs improve after attending a training workshop, workshops alone were unlikely to change therapist practice or increase therapist use of an EBI. Ongoing training and consultation were needed to improve therapist fidelity and increase EBI use (Frank et al., 2020). Despite the recognition of the importance of ongoing therapist training, there are limited experimental studies examining the impact of training providers to deliver an intervention to children with ASD on observed provider behaviors, and the studies that do exist are primarily in the field of education (Alexander, Ayres, & Smith, 2015; Suhrheinrich et al., 2020). In response to the need for an effective therapist training model for psychotherapy services, a clinical intervention and therapist training model called An Individualized Mental Health Intervention for ASD (AIM HI: Brookman-Frazee & Drahota, 2010) was developed. The AIM HI protocol was designed for children with ASD presenting to mental health services with challenging behaviors and their mental health therapists providing psychotherapy in publicly funded mental health services and was developed and refined in collaboration with community stakeholders.

Hybrid Type 1 effectiveness-implementation trial designs are used to examine both the clinical effectiveness and implementation processes. Hybrid studies provide an opportunity to develop effective implementation strategies, produce relevant information for stakeholders, and facilitate the translation of research findings into clinical practice (Curran, Buaer, Mittman, Pyne, & Stetler, 2012; Landes, McBain & Curran, 2019). After a pilot feasibility study of AIM HI, a Hybrid Type 1 trial examining the clinical (child and caregiver) and implementation (therapist fidelity and intervention sustainment) outcomes of therapist training in AIM HI was conducted.

Previous reports from the Hybrid Type 1 trial indicated that the AIM HI intervention and training program was effective in reducing child challenging behaviors over 18 months (Brookman-Frazee, Roesch, Chlebowski, Baker-Ericzen, & Ganger, 2019). Increasing caregiver self-efficacy (parenting sense of competence) after 6 months of services provided to their child by a therapist trained in AIM HI was associated with improved child behavior outcomes at six months and mediated these effects at long term follow up at 12 and 18 months (Brookman-Frazee, Chlebowski, Villodas, Roesch, & Martinez, 2020). Mediation analyses highlight the importance of therapist delivery of evidence-based intervention strategies, with therapist use of continuity across sessions mediating child and caregiver outcomes, effective use of session structure mediating child outcomes, and effective therapist pursuit of a caregiver skill mediating caregiver outcomes (Brookman-Frazee et al., 2019; Brookman-Frazee et al., 2020).

Expanding on the clinical outcomes previously reported, the purpose of the current study was to examine therapist training outcomes from the parent trial specifically, observed changes in therapist use of EBI strategies and to examine potential moderators of therapist training effects.

Method

Procedure

Data were drawn from a subset of participants from a cluster randomized trial using a waitlist control design conducted in 29 publicly funded mental health programs with 172 therapist and 202 child/family participants. As described in earlier outcome reports (Brookman-Frazee et al., 2019; Brookman-Frazee et al., 2020) and using a protocol approved by the UC San Diego Institutional Review Board, outpatient clinic and school-based mental health programs in San Diego and Los Angeles counties were recruited for enrollment in the study. Programs were eligible if they were publicly funded and provided psychotherapy services to children, including children with ASD.

Enrolled programs were randomized to immediate AIM HI training or a waitlist control observation condition (usual care). After programs completed the usual care condition, they were invited to transition to the AIM HI training condition and participate in therapist training if they had new children with ASD served in their program and new therapists and clients were recruited for participation. Randomization occurred at the program level to prevent therapist level contamination. Randomization was completed by an independent statistical investigator who stratified programs by program size using randomly permuted blocks according to a computer-generated assignment sequence. Based on an imbalance in the average number of participants enrolled in usual care vs AIM HI conditions during the first year of the trial, a restricted randomization approach (Ryeznik & Sverdlov, 2018; Higham, Tharmanathan & Birks, 2015) was used for subsequent years in which randomization ratio of programs randomized to usual care was recalibrated from 1:1 to a 2:1 ratio.

Participants

Following program enrollment and randomization in the parent trial, therapists were recruited from within enrolled programs and consented for participation. Therapists were eligible for study participation if they planned to provide psychotherapy services for at least the next 6 months in an enrolled mental health program and had a least one eligible child on their caseload. Children and caregivers were enrolled from participant therapists’ caseloads. Eligible children were aged 5–13 years old at the time of recruitment, had an existing ASD diagnosis on record, and/or exhibited clinically significant ASD symptoms on at least one of two standardized ASD diagnostic measures (Autism Diagnostic Observation Schedule-2 (ADOS-2; Lord, Rutter & DiLavore, 2012) or Social Responsiveness Scale-2 (SRS-2; Constantino & Gruber, 2012)). Eligible caregivers were parents of a child meeting enrollment criteria who spoke English or Spanish as their preferred language. Informed written consent was obtained from all study participants; participants consented to have their sessions video recorded and reviewed by the research team (both conditions) and clinical trainer (AIM HI training condition).

The parent trial included 172 therapists from 29 programs. The current study included a subset of 131 therapist participants for whom video recorded session data were available. See Table 1 for participant characteristics, including demographic characteristics, for participants included in the current analyses.

Table 1.

Program, Participant and Treatment Characteristics for Study Sample

	Usual Care	AIM HI2
Program Characteristics	n=7	n=22
Setting
Outpatient clinic	85.7%	45.5%
School	14.3%	18.2%
Multiple settings (school, clinic)	0%	36.4%
Therapist Characteristics	n=16	n=115
Gender (% female)	81.2%	85.3%
Age M(SD)	36.3 (9.7)	33.9 (8.4)
Race/Ethnicity
Hispanic/Latinx	37.5%	33.6%
Non-Hispanic White	56.3%	45.7%
Asian/Pacific Islander	6.2%	14.7%
Black/African American	0.0%	3.4%
Multiracial	0.0%	0.9%
Unknown/Not Reported	0.0%	1.7%
Primary mental health discipline (%)
Marriage and Family Therapy	56.2%	41.4%
Clinical Psychology	18.8%	18.1%
School Psychology	0%	6.0%
Social Work	25.0%	28.5%
Other discipline3	0%	6.0%
Licensed in mental health discipline (%)	25.0%	28.4%
Previous behavioral parent training experience (%)	12,5%	14.7%
Child characteristics	n=16	n=115
Child age (years) M(SD)	8.8 (2.5)	9.0 (2.4)
Child gender (male)	87.5%	80.2%
Child race/ethnicity
Hispanic/Latinx	75.0%	56.9%
Non-Hispanic White	18.8%	27.6%
Black/African American	0%	4.3%
Multiracial	0%	5.2%
Asian/Pacific Islander	0%	6.0%
American Indian/Alaskan Native	6.2%	0%
Cognitive Standard Score4	88.7 (15.4)	88.9 (16.6)
ECBI Intensity5 Scale T-Score M(SD)	64.9 (8.0)	63.4 (10.5)
Treatment characteristics
Number of sessions (within 6 months)	13.9 (6.3)	15.5 (5.8)

¹Participants include a subsample of participants from the parent trial with recorded session data available.

²The AIM HI sample includes therapists from programs randomized to immediate training (wave 1; n= 13 programs) and therapists from programs who transitioned to the AIM HI training condition after completion of the usual care waitlist control condition (wave 2; n= 9 programs). No significant differences in program, therapist, or child characteristics between wave 1 and wave 2 were found.

³Includes psychiatry, licensed professional clinical counselor, and art therapist

⁴Cognitive scores are based on the Wechsler Abbreviated Scale of Intelligence-II or Differential Ability Scale-II, depending on child age

⁵The ECBI Intensity scale represents the frequency of disruptive behaviors.

Usual care control condition

Therapist participants in the usual care control condition delivered routine care to participating children and caregivers and recorded their therapy sessions over a 6-month observation period. No clinical training was provided to therapists in the usual care condition.

AIM HI training condition

Clinical intervention.

The AIM HI clinical intervention is a package of caregiver and child directed strategies designed to reduce challenging behaviors in children with ASD aged 5 to 13. AIM HI is a child focused and caregiver mediated intervention that teaches children positive alternative skills and teaches caregivers antecedent and consequence-based strategies to promote their child’s use of skills. AIM HI includes well-established behavioral intervention strategies for children with ASD (Steinbrenner et al., 2020; Wong et al., 2015) and packages the strategies for delivery in mental health services by therapists with limited experience working with an ASD population. AIM HI consists of a series of protocol steps that include Treatment Planning, Active Teaching of child and caregiver skills, and Evaluating Progress of the client towards treatment goals. Therapists are trained to actively colloborate with caregivers to identify patterns and the primary function of the child’s challenging behaviors, and actively teach child and caregiver skills in session using the “Active Teaching sequence” which includes psychoeducation, therapist modeling of the skill, in session behavioral rehearsal with targeted feedback and reinforcement, and assignment and review of between-session practice (e.g., homework). AIM HI also includes strategies to adapt psychotherapy structure for ASD clients, including child engagement strategies and session structuring, to facilitate client engagement and skill building. See Table 2 for overview of AIM HI clinical protocol.

Table 2.

AIM HI Clinical Protocol

Treatment Phase	AIM HI Protocol Step	Within Session Elements
Treatment Planning	• Integrate Assessment Information	Structuring sessions for skillbuilding and engagement • Schedules, visual/written materials Engaging caregivers and children • Collaboration/active involvement (caregivers) • ASD-specific motivational strategies (children) • Sharing in session control (children) • Preparing to teach new skills Active teaching with caregivers and children • Modeling, practice-with-feedback, reinforcement, between-session practice
	• Complete Behavior Tracking with Caregivers
	• Develop Behavior Plan
Active Teaching	• Teach skills to caregivers
	• Teach skills to child
	• Promote generalization of child skills
	• Teach additional child skills (as indicated)
Evaluating Progress	• Review treatment progress • Develop Plan for Next Steps

Clinical training.

Therapists from programs in the AIM HI training condition participated in AIM HI training and consultation from an expert trainer over the 6 month study period while they delivered AIM HI to a participating child/parent dyad. Training included the following components: (a) an 8-hour introductory workshop, (b) 11 (9 group, 2 individual) structured consultation meetings (one hour in length) over 6 months with didactic instruction and case-specific feedback led by an expert AIM HI trainer, (c) verbal and written performance feedback from AIM HI trainer based on review of recorded therapy sessions (d) therapist self-study of materials including a therapist manual, printed and electronic protocol forms, and a resource website.. Participating therapists met with the same trainer and consultation group for each meeting in the series; each consultation group had approximately 4–5 therapists. Trainers used standardized materials to lead the consultation groups following a structured protocol, received weekly supervision from a lead supervisor, and were monitored for fidelity to the training model. Therapists in the AIM HI training condition completed an average of 10.11 consultations over the 6- month period and 90.1% of therapists were rated by their AIM HI trainer as successfully completing the AIM HI training.

Data sources

Baseline data collection.

Therapists provided baseline data via an online survey including self-reported training and work experience, licensure status in their mental health discipline, previous experience with behavioral parent training (BPT), and pre-training self-perceptions of knowledge and feelings of self-efficacy in using EBI strategies for ASD. These therapist pre-training self-perceptions of knowledge and feelings of self-efficacy were gathered about both 1) use of function-based strategies across sessions and 2) use of within-session strategies.

In-person baseline assessments were conducted by the research team for referred children in both conditions. Child demographics were reported by caregivers at baseline. Caregiver report of child challenging behaviors at baseline were collected using the Eyberg Child Behavior Inventory (ECBI: Eyberg & Pincus, 1999); consistent with previous outcome reports, the ECBI Intensity score (rating frequency of challenging behaviors) was used in analyses.

Observer-rated therapist delivery of EBI strategies within sessions.

Session recordings submitted by therapists in both conditions during the 6 months of treatment were coded by trained raters, naïve to study condition, using the AIM HI Observational Coding System (Brookman-Frazee & Chlebowski, 2013). The structure of the AIM HI Observational Coding System was adapted from Practice and Research: Advancing Collaboration Therapy Process Observational Coding System for Child Psychotherapy – Specific Therapy Process Scale (PRAC TPOCS-S; Garland et al., 2010). The AIM HI Observational Coding System was designed to document the presence and intensity of 18 clinical strategies used by therapists, with ratings for strategies directed to children and caregivers coded separately. For each session, coders rated the extensiveness of EBI strategies used by therapists according to both the frequency (number of times strategy was used) and intensity (thoroughness with which the strategy was pursued). Ratings used a Likert scale ranging from 0 to 6, with ratings ranging from “not observed,” to “used with high frequency/intensity.” Recordings were randomly selected for coding from two-month intervals representing the beginning (months 1–2), middle (months 3–4), and end (months 5–6) of treatment to allow for a balanced representation of sessions from all therapists. A subset of the therapists in the trial were enrolled with more than one family. For the current analyses, only one family’s scores were included for each therapist. If session recordings for two families were available from a therapist, then the family with more available recordings was selected as the family for inclusion in the data. In three cases, two families had the same number of session recordings available; in these cases the family was randomly selected to include in analyses for the therapist.

In the current study, the 18 individually rated items were consolidated into composite scores for analyses. Composites reflecting the extent to which the therapist was observed to use within-session “active teaching” EBI strategies (i.e., psychoeducation, modeling, in session practice, reinforcement, feedback, assigning between session practice) directed towards children (Active Teaching with Child) and caregivers (Active Teaching with Caregivers), as well as a composite score measuring therapist use of child engagement strategies (Child Engagement) were calculated. In addition, three independent Effectiveness ratings reflecting the observed effectiveness of therapist within session strategies were used for analyses; these ratings included 1) Session Structure (the session appeared organized and structured) 2) Treatment Continuity (session activities and content were linked to previous or future sessions) and 3) Therapist Pursuit of Caregiver/Child Skill (there were clearly identifiable caregiver/child goals targeted during session activities).

Community Involvement Statement

AIM HI was developed based on systematic assessment of community therapist and client needs (Brookman-Frazee, Baker-Ericzen, Stadnick & Taylor, 2012; Brookman-Frazee, Drahota, Stadnick & Palinkas, 2012) and developed and refined in collaboration with community therapists and caregivers of children with autism. Children were involved in development of therapist training videos (e.g., describing their experience with ASD).

Data Analysis

To account for the nested data structure in which sessions [level 1] were nested within children/therapists [level 2] and children/therapists were nested within programs [level 3], all analyses were performed using multilevel modeling (MLM) (e.g., Raudenbush & Bryk, 2002) and the Maximum Likelihood Robust estimation procedure implemented in Mplus version 8.1.5 (Muthén & Muthén, 2017). The estimation procedure adjusts for missing data that can be treated as Missing at Random (MAR) and non-normality of the outcome variables. Though it is difficult to definitively determine that missing data meet these criteria, all study variables were tested, and none were significantly associated with missing data patterns. Data were treated as though they were MAR (Enders, 2010). Associations between setting and child characteristics and outcomes were tested using three-level models to test the bivariate associations between each potential covariate and each outcome. Treatment interval (beginning, middle, or end of 6-month period), child age, and child baseline ECBI intensity were significantly associated with one or more outcomes in preliminary bivariate models and were included as covariates in all analyses of observed EBI delivery. Differences between conditions in observed within session EBI strategy delivery were also tested using three-level models. Level 1 models included two dummy-coded predictors variables representing comparisons between the treatment interval (i.e., beginning, middle, or end of 6-month period) of the intervention period during which the observations were collected. The first timeframe was used as the reference category to which the second and third timeframes were compared. An example of the Level 1 model equation is:

$$\begin{gathered} y_{ijk}={\pi }_{0jk} \\ +{\pi }_{1jk}(MiddleSessionInterval)+{\pi }_{2jk}(Endof6-monthSessionInterval)+e_{ijk} \end{gathered}$$

Level 2 models included dichotomous predictors representing therapist training condition and licensure status, as well as continuous predictors representing therapists’ self-perceptions of knowledge of within session strategies, and confidence in using within session strategies. Level 2 models also included child age and child baseline ECBI intensity as covariates. An example of a Level 2 model equation is:

$$\begin{gathered} {\pi }_{0jk} \\ ={\beta }_{00k}+{\beta }_{01k}(ChildAge)+{\beta }_{02k}(BaselineECBIIntensity)+{\beta }_{03k} \\ (TherapistLicensed)+{\beta }_{04k}(TherapistBPTExperience)+{\beta }_{05k} \\ (TherapistKnowledgeofStrategies)+{\beta }_{06k} \\ (TherapistConfidenceinUsingStrategies)+{\beta }_{07k}(TrainingGroup)+r_{0jk} \end{gathered}$$

No program-level (Level 3) predictors were included. Therapist characteristics were statistically evaluated as moderators of training condition differences by adding interaction terms to the Level 2 models. Significant interactions were probed using simples slopes tests (Preacher et al., 2008).

Results

Therapist training outcome scores by training condition are reported in Table 3.

Table 3.

Therapist Training Outcome Scores by Training Condition

	Usual Care	AIM HI
Observer-rated in-session EBI strategy extensiveness (Possible range 0–6)	M(SD)	M(SD)
Composite Scores
Active Teaching with Caregiver 1	1.85 (1.16)	3.05 (1.27)
Active Teaching with Child 2	2.64 (1.28)	3.14 (1.34)
Child Engagement 3	3.05 (1.04)	4.20 (1.27)
Effectiveness Ratings (individual codes)
Session Structure	2.85 (1.47)	4.56 (1.20)
Treatment Continuity	2.51 (1.64)	4.65 (1.32)
Therapist Pursuit of caregiver and/or child skills	3.07 (1.98)	4.44 (1.64)

¹Active Teaching with Caregiver composite includes the following codes: Provided psychoeducation/information to caregiver; Modeled/demonstrated to caregiver; Provided opportunity for in-session practice for caregiver; Provided feedback to caregiver; Assigned or reviewed between session practice for caregiver.

²Active Teaching with Child composite includes the following codes: Provided psychoeducation/information to child; Modeled/demonstrated to child; Provided opportunity for in-session practice for child; Provided feedback to child; Provided positive reinforcement to child; Assigned or reviewed between session practice for child.

³Child Engagement composite includes the following codes: Used agenda/session schedule with child; Used visual materials with child; Incorporated child preferred interests; Shared control of activities with child

Therapist delivery of EBI strategies within sessions (observer rated)

Treatment interval (beginning, middle, or end of 6-month period), child age, and child baseline ECBI intensity were significantly associated with one or more outcomes in preliminary bivariate models and were included as covariates in all analyses of observed EBI delivery. Regression coefficients from three-level multivariate models for each outcome are presented in Tables 4 and 5.

Table 4.

Observed Therapist Use of EBI Strategies Within Sessions

	Active Teaching Composite (Caregiver) B [95% CI]	Active Teaching Composite (Child) B [95% CI]	Child Engagement Strategies Composite B [95% CI]
Training Condition (Reference: Usual care)
AIM HI	1.10* [.66,1.54]	.05 [−.52, .61]	.95* [.52, 1.38]
Therapist Factors
Licensed in discipline	.10 [−.16, .36]	.22 [−.07, .51]	−.09 [−.50, .33]
Previous BPT1 experience	.58* [.27, .89]	−.27 [−.65, .10]	−.32[−.76, .12]
Self-perception of knowledge of within session strategies	−.14 [−.41, .12]	−.12 [−.36, .12]	.07 [−.27, .41]
Self-efficacy in using within session strategies	.18 [−.04, .39]	.32* [.06, .58]	−.04 [−.25, .18]
Moderated effects
Training Condition x Licensed	−.73* [−1.17, −.29]	−.69 [−1.41, .04]	.08 [−.57, .72]
Training Condition x BPT experience	−.55 [−1.19, .09]	−.53 [−1.55, .50]	.26 [−.49, 1.00]
Training Condition x Self-perception of knowledge of within session strategies	−.14 [−.90, .62]	−.08 [−.58, .43]	−.08 [−.77, .61]
Training Condition x Self-efficacy in using within session strategies	−.23 [−.81, .35]	−.05 [−.48, .38]	.00 [−.68, .68]

These models controlled for treatment interval, child age, and child baseline ECBI Intensity.

¹BPT: Behavior Parent Training

Table 5.

Observed Therapist EBI Strategy Effectiveness Within Sessions

	Session Structure Rating B [95% CI]	Treatment Continuity Rating B [95% CI]	Pursued Caregiver or Child Skill Rating B [95% CI]
Training Condition (Reference: usual care)
AIM HI	1.69* [1.26, 2.13]	1.36* [.77, 1.96]	.87* [.05, 1.69]
Therapist Factors
Licensed in discipline	−.05 [−.28, .18]	.00 [−.34, .35]	.13 [−.16, .42]
Previous BPT1 experience	.01 [−.27, .29]	.14 [−.18, .46]	.28 [−.04, .59]
Self-perception of knowledge of within session strategies	−.02 [−.24, .21]	−.03 [−.17, .11]	−.17 [−.54, .21]
Self-efficacy in using within session strategies	−.08 [−.26, .11]	−.01 [−.17, .14]	.17 [−.08, .41]
Moderated Effects
Training Condition x Licensed	−.40 [−.91, .10]	.38 [−.11, .87]	−1.05 [−2.17, .07]
Training Condition x BPT experience	−.97 [−1.44, .48]	−.31 [−1.25, .64]	−1.04 [−2.39, .31]
Training Condition x Self-perception of knowledge of within session strategies	.15 [−.26, .56]	−.17 [−.67, .34]	.24 [−.48, .95]
Training Condition x Self-efficacy in using within session strategies	.25 [−.22, .72]	−.34 [−.91, .23]	.25 [−.57, 1.07]

These models controlled for treatment interval, child age, and child baseline ECBI Intensity.

¹BPT: Behavior Parent Training

Therapist use of within session active teaching strategies with caregiver.

Therapists in the AIM HI training condition were observed to use significantly more active teaching strategies (psychoeducation, modeling, in session practice with feedback, assigning of between session practice) with caregivers during sessions than therapists in the usual care training condition, after controlling for covariates, d = .89, 95% CI for d [.36, 1.42]. Therapists with prior behavioral parent training experience used more active teaching with caregivers during sessions than therapists with no prior behavioral parent training experience, d = .46, 95% CI for d [.21, .71]. There were no significant differences in therapists use of active teaching with caregivers by treatment interval. Training condition effects were not moderated by therapists’ prior behavioral parent training experience but were moderated by therapists’ licensure status (see Figure 1). Tests of simple slopes were conducted in each therapist licensure group (licensed versus not licensed) and revealed that among therapists who were licensed, those in the AIM HI training condition used significantly more active teaching with caregivers than those in usual care, B = .50, p = .038, d = .39, 95% CI for d [.02, .78]; however, among therapists who were not licensed, the difference in use of active teaching with caregivers between than those in the AIM HI training condition and usual care training condition was even greater, B = 1.08, p < .001, d = .88, 95% CI for d [.26, 1.49].

Figure 1.

Therapist license status moderating observed active teaching with caregivers.

Therapist use of within session active teaching strategies with children.

Therapists in the AIM HI training condition did not significantly differ from therapists in the usual care training condition in their use of active teaching strategies with children during sessions after controlling for covariates. Therapists who had greater confidence in using within session strategies at baseline used significantly more active teaching with children during sessions, B = .32, 95% CI [.06, .58] regardless of training condition. Therapists also used significantly less active teaching strategies with children who had higher ECBI scores. Therapists used significantly more active teaching strategies with child during the second (months 3–4), B = .69, p < .001, 95% CI [.42, .96]. or third (months 5–6), B = .98, p < .001, 95% CI [.70, 1.26]. treatment intervals, relative to the first (months 1–2) interval. Therapists also used significantly more active teaching strategies with children during the third (months 5–6) treatment interval relative to the second (months 3–4) interval, B = .29, p < .002, 95% CI [.11, .48].

Therapist use of within session engagement strategies with children.

Therapists in the AIM HI training condition were observed to use significantly more engagement strategies with children during sessions than therapists in the usual care training condition, after controlling for covariates, d =.75, 95% CI for d [.21, 1.28]. Training effects were not moderated by any therapist characteristics. Child age was a significant covariate as therapists used significantly more engagement strategies with younger children than with older children. There were no significant differences in therapists use of engagement strategies by treatment interval.

Therapist use of structure within session.

Therapists in the AIM HI training condition were observed to have significantly more structured sessions than therapists in the usual care training condition, after controlling for covariates, d = 1.45, 95% CI for d [.90, 2.01]. Training effects were not moderated by any therapist characteristics. No covariates were significantly associated with session structure. There were no significant differences in therapists use of session structure by treatment interval.

Therapist use of strategies promoting continuity across sessions.

Therapists in the AIM HI training condition were significantly more likely to use strategies promoting continuity of care in session (such as linking current session content to content from previous or upcoming sessions) than therapists in the usual care, after controlling for covariates, d = 1.13, 95% CI for d [.58, 1.67]. Training effects were not moderated by any therapist characteristics. No other covariates were significantly associated with treatment continuity. There were no significant differences in therapists use of strategies promoting continuity of care by treatment interval.

Pursuit of caregiver and child skills.

Therapists in the AIM HI training condition pursued teaching caregiver and child skills significantly more in sessions than therapists in the usual care training condition, after controlling for covariates, d = .53, 95% CI for d [.01, 1.06]. Training condition effects were not moderated by any therapist characteristics. Significant covariates included treatment interval, with therapists teaching caregivers and children skills significantly more in sessions during the second (months 3–4), and third (months 5–6) treatment intervals relative to the first (months 1–2), and baseline child behavior problems, with therapists teaching caregiver and children skills in session significantly less for children with more severe baseline behavior problems. Therapists pursued teaching caregiver and child skills significantly more in sessions during the second (months 3–4), B = .57, p < .001, 95% CI [.36, .79] or third (months 5–6), B = .61, p < .001, 95% CI [.39, .83] treatment intervals, relative to the first (months 1–2) interval, but not between the second and third treatment intervals.

Discussion

The current study examined the effectiveness of training therapists working in publicly funded mental services to deliver AIM HI, a package of evidence-based intervention strategies for children with ASD and challenging behaviors designed specifically for the community mental health service context. Data were drawn from a cluster randomized Hybrid Type 1 effectiveness-implementation trial. Compared to therapists with no AIM HI training delivering usual care, therapists receiving AIM HI training were observed to use more EBI strategies in session. Therapist characteristics were associated with and moderated a subset of outcomes.

AIM HI training was significantly associated with more extensive observed delivery of active teaching strategies with caregivers. Additionally, training was associated with more extensive use of strategies to promote in session skill building with children, such as use of session engagement strategies with children, and more structured therapy sessions achieved through the use of tools such as session schedules and visuals, than therapists in usual care. Ratings assessing observed effectiveness of therapist within session strategies indicate that AIM HI therapists pursued teaching caregivers and child skills in session significantly more extensively and were significantly more likely to use strategies promoting continuity of care, such as linking current session content to content from previous or upcoming sessions, than therapists in the usual care training condition. Taken together, these findings indicate that AIM HI training resulted in substantial improvement in community practice relative to usual care. They are consistent with research supporting the use of intensive training to change therapist behavior (Frank et al., 2020) and extend the broader mental health implementation literature to include therapist delivery of EBI strategies to children with autism.

Therapist experience emerged as a factor associated with in-session EBI strategy delivery with therapists with prior behavioral parent training experience using more extensive active teaching strategies with caregivers during sessions across training conditions. Furthermore, therapist licensure moderated the effect of AIM HI training on observed active teaching strategies directed to caregivers. Therapists in the AIM HI training condition used significantly more active teaching with caregivers than those in the usual care; however, the difference in use of active teaching with caregivers was even greater for unlicensed therapists, suggesting that AIM HI training may have an even greater impact in changing therapist practice in work directed to caregivers for less experienced therapists.

Models examining training effects and moderators controlled for certain child characteristics. There was an effect of child age with therapists more likely to use engagement strategies in session with younger children. Baseline child behavior problems was also observed impact in session behaviors, with therapists using significantly fewer active teaching strategies in session with children and less effectively pursuing the teaching of child and caregiver skills when working with children with more severe baseline behavior problems. It is possible that therapists devote more time during session to managing child behaviors and/or reviewing challenging behaviors that occurred since the previous session for these children and suggests that additional work is needed to help therapists effectively use evidence-based strategies with children with ASD presenting with more severe behavior problems.

The few differences observed by treatment interval (therapists using most active teaching strategies with the child in the middle and end of treatment as compared to the first two months) is consistent with the sequencing of the AIM HI intervention, which starts with treatment planning followed by caregiver skill teaching before transitioning to teaching child skills. While caregiver skills are targeted first in the protocol, the finding that active teaching strategies directed to caregiver did not differ by treatment interval suggests that therapists maintained targeted skill teaching for caregivers throughout treatment. Of note, therapist use of session structure, engagement strategies with children, and strategies promoting continuity of care did not differ by treatment interval, which is also consistent with the AIM HI protocol as these elements are designed to be integrated into all treatment sessions.

The current study was conducted in Southern California and findings may not generalize to child and adolescent mental health services in other geographic areas. Despite this potential limitation, this study makes several important contributions. This is the first community effectiveness trial of a model of therapist training and an ASD mental health intervention tested within the context of publicly funded mental health services. The study was conducted with a large and representative sample of therapists working with children with ASD presenting to routine mental health services. The use of trained raters naïve to therapist training condition to characterize therapist in session behaviors in a methodological strength of the current study. As has been described in previous outcome reports, the AIM HI intervention and training model were developed specifically for a community mental health service setting based on a comprehensive needs assessment in collaboration with relevant stakeholders who are the intervention’s “end users”. The documentation of therapist behavior change supports the effectiveness of a training model that is designed to fit the needs of the stakeholders.

Conclusions

The findings of this community trial support the effectiveness of training therapists in publicly funded mental health services, who do not specialize in ASD, to deliver evidence-based strategies to reduce challenging behaviors to children with ASD. This study provides evidence using a rigorous trial design for the positive effects of structured and ongoing therapist training.