Effectiveness of Training Therapists to Deliver An Individualized Mental Health Intervention for Children With ASD in Publicly Funded Mental Health Services

Key Points

Question

Does training community mental health therapists to deliver an intervention influence challenging behaviors in children with autism spectrum disorder?

Findings

In a cluster randomized community effectiveness trial, 202 children with autism spectrum disorder whose therapists received training and ongoing consultation in evidence-based strategies showed greater reductions in challenging behaviors across 18 months compared with children whose therapists delivered usual care. Observer ratings of therapist fidelity mediated the intervention effects.

Meaning

This study supports the effectiveness of an intervention designed specifically for the targeted community service context and highlights the need to develop and test implementation strategies targeting high-fidelity therapist delivery.

Abstract

Importance

Publicly funded mental health services play an important role in addressing co-occurring mental health problems in children with autism spectrum disorder (ASD); however, therapists report lacking training to effectively serve this complex population.

Objective

To test the effectiveness of training community therapists in An Individualized Mental Health Intervention for ASD (AIM HI) on challenging behaviors across 18 months among children with ASD and identify moderators and mediators of any intervention effects.

Design, Setting, and Participants

Cluster randomized trial conducted in 29 publicly funded outpatient and school-based mental health programs in southern California from 2012 to 2017. Programs were randomized to receive immediate AIM HI training or provide usual care followed by receipt of AIM HI training. Therapist participants were recruited from enrolled programs, and child participants were recruited from participant therapists’ caseloads. Data were analyzed from 202 children with ASD who were aged 5 to 13 years.

Interventions

The AIM HI protocol is a package of parent-mediated and child-focused strategies aimed to reduce challenging behaviors in children with ASD who are 5 to 13 years old. It was designed for delivery in publicly funded mental health services based on a systematic assessment of therapist training needs and child clinical needs. The therapist training and consultation process takes approximately 6 months and includes an introductory workshop, 11 structured consultation meetings as the therapist delivers AIM HI with a current client, and case-specific performance feedback from trainers.

Main Outcomes and Measures

Child participants were assessed for challenging behaviors using the Eyberg Child Behavior Inventory (ECBI) and Social Skills Improvement System (SSIS) Competing Problem Behaviors scales based on parent report at baseline and at 6-month intervals for 18 months. Outcomes were analyzed using intent-to-treat models.

Results

In total, 202 children with ASD (mean [SD] age, 9.1 [2.4] years; 170 [84.2%] male; 121 [59.9%] Latinx) were eligible, enrolled, and included in the analyses. Statistically significant group by time interactions for the ECBI Intensity (B = −0.38; P = .02) and ECBI Problem (B = −1.00; P = .005) scales were observed, with significantly larger decreases in ECBI Intensity scores in the AIM HI group (B = −1.36; P < .001) relative to the usual care group (B = −0.98; P < .001) and a significantly larger decrease in ECBI Problem scores in the AIM HI group (B = −1.22; P < .001) relative to the usual care group (B = −0.20; P = .29). Therapist fidelity moderated these intervention effects.

Conclusions and Relevance

The present findings support the effectiveness of training therapists to deliver the AIM HI model to children with ASD receiving publicly funded mental health services.

Trial Registration

ClinicalTrials.gov identifier: NCT02416323

This cluster randomized trial evaluates the effectiveness of training community therapists in an individualized mental health intervention program vs usual care to address challenging behaviors across 18 months in children with autism spectrum disorder and to identify moderators and mediators of any intervention effects.

Introduction

The number of individuals with autism spectrum disorder (ASD) has increased substantially during the past 2 decades, with an estimated 1 in 59 school-aged children meeting criteria for ASD.¹ Community service systems, including publicly funded mental health (MH) services, are faced with addressing the complex needs of these children. The federal Interagency Autism Coordinating Committee² has called for research to scale up evidence-based interventions in community service settings and to improve psychosocial interventions for core symptoms and co-occurring conditions in ASD. One related direction of study is community effectiveness research for school-age children with ASD, which is important given the paucity of research for this group.

Publicly funded MH services play an important role in caring for school-age children with ASD³ because approximately 70% of these children meet criteria for at least 1 non-ASD co-occurring psychiatric disorder.^4,5,6,7 Children with ASD represent 10% to 14% of children in the psychiatrically referred population,⁸ and MH therapists indicate that children with ASD represent a significant proportion of their caseloads.⁹ The primary presenting problems in MH services are challenging behaviors⁹ and more than 80% of children in MH services meet criteria for at least 1 disorder associated with challenging behaviors (eg, attention-deficit/hyperactivity disorder or oppositional defiant disorder).¹⁰ Furthermore, challenging behaviors are often the expressed symptoms of ASD and non-ASD psychiatric disorders.^11,12 There are significant concerns about the lack of MH therapist training to work with the ASD population^9,13 and broader concerns about the overall quality of MH services for children with ASD.¹⁴

Although a number of well-established, evidence-based behavioral intervention strategies have been shown to be efficacious in reducing behavior problems in children with ASD,^15,16,17,18 there is limited training and delivery in MH settings.¹⁴ There have also been no systematic, large-scale studies examining the effectiveness of such strategies within publicly funded MH services.¹⁸ The most relevant research has shown the efficacy of adapted anxiety interventions^19,20 when delivered in research settings and the effectiveness of interventions addressing social communication skills within the context of early intervention services^21,22,23 and in public schools.^24,25,26 Furthermore, understanding child characteristics associated with responses to interventions delivered in routine care is critical in ASD given the heterogeneity of ASD symptomology and variable responses to the intervention.²⁷ Finally, there is significant variability in community therapist fidelity to ASD interventions, and therapist fidelity is associated with child outcomes.^28,29

In response to the need for an intervention protocol that could be delivered in publicly funded MH services for children with ASD presenting with challenging behaviors, An Individualized Mental Health Intervention for ASD (AIM HI)³⁰ clinical intervention and corresponding therapist training protocol were developed. The AIM HI protocol was developed based on a systematic needs assessment of the clinical needs of children with ASD receiving MH services and the training needs of MH therapists.⁹ The purpose of the current cluster randomized controlled community effectiveness trial was to build on the promising findings from a pilot study supporting the feasibility of training MH therapists and positive changes in child behaviors^31,32,33 by assessing the effectiveness of training therapists in AIM HI on changes in child behaviors across 18 months and examining potential moderators (child characteristics) and mediators (therapist fidelity) of intervention effects.

Methods

A cluster randomized waitlist control design was used to examine the effectiveness of AIM HI training on child challenging behaviors (Figure 1). Publicly funded outpatient clinics and school-based MH programs in San Diego and Los Angeles counties were invited to enroll in the study (trial protocol in Supplement 1). Programs were eligible if they provided psychotherapy services to children, including those with ASD. After programs enrolled in the study, they were randomized to immediate (wave 1) AIM HI training or to a waitlist usual care (UC) condition/delayed (wave 2) AIM HI training condition. After programs in the waitlist control completed the UC condition, they were invited to participate in the wave 2 AIM HI condition if they had new children with ASD served in their program. Program randomization was conducted by an independent statistical investigator (S.R.) and occurred at the program level to prevent contamination at the therapist level. Randomization was stratified by agency size using randomly permuted blocks according to a computer-generated assignment sequence prepared in advance by the study statistician (S.R.). During the first year of recruitment (2012), programs were randomized using a 1:1 ratio of UC/wave 2 AIM HI to wave 1 AIM HI. Based on an imbalance in the mean number of participants enrolled in UC/wave 2 vs AIM HI wave 1 during the first year, we switched to a restricted randomization approach^34,35 for subsequent years. Specifically, the randomization ratio was recalibrated from 1:1 to 2:1, that is, 2 programs randomized to UC/wave 2 for every program randomized to wave 1 to achieve better balance in the sample size by condition. Following randomization, therapist/child dyads were recruited for participation. Once the UC waitlist control programs moved over to wave 2, a new round of therapist/child recruitment was conducted. Eight of the UC therapists participated in wave 2, and no child participated in both conditions. The University of California, San Diego Institutional Review Board approved this study, and informed written consent was obtained from all participants or their guardians.

Figure 1. Study Flow Diagram.

AIM HI indicates An Individualized Mental Health Intervention for ASD; UC, usual care; W1, wave 1; and W2, wave 2.

^aThere were 43 programs provisionally enrolled in the study; 13 had insufficient numbers of clients with autism spectrum disorder (ASD) and were thus ineligible.

^bIneligible: 1 left the clinic; 32 did not refer the client.

^cTwo unable to contact; ineligible: 1 left the clinic; 1 did not provide psychotherapy; 32 did not refer the client.

^dTherapist may have referred more than 1 child if the first referred child was not enrolled in the study or if there was more than 1 eligible child on the caseload.

^eIneligible: 1 left the clinic; 8 had no eligible client.

^fIneligible: 11 had no eligible client.

^gThree ineligible based on phone screen; 6 declined; 2 unable to contact.

^hFifteen ineligible after phone screen; 10 declined; 4 unable to contact.

ⁱIneligible: 6 did not have ASD; 3 had an ineligible referring therapist.

^jTwo unable to contact; ineligible: 1 left the clinic; 13 did not refer the client.

^kIneligible: 1 left the clinic; 8 had no eligible client.

^lFive ineligible based on phone screen; 2 declined; 2 unable to contact.

^mIneligible: 6 did not have ASD.

Participants

Therapists were recruited from enrolled programs, and children were recruited from the caseloads of participating therapists. See the Table for a summary of program, therapist, and child/family characteristics. Following the enrollment of 29 programs and randomization, therapists were recruited from within enrolled programs. Therapists were eligible if they (1) were a trainee or staff providing psychotherapy services in an enrolled program, (2) anticipated working in the program for at least the next 7 months, and (3) had a least 1 eligible child on their current caseload. Of 172 therapist participants, most (85.5%) were women, and the therapists had a mean (SD) age of 34.0 (8.1) years. The mean (SD) number of years of experience was 5.43 (4.77) and ranged from 0 to 24 years.

Table. Program, Participant, and Treatment Characteristics.

Characteristic	Usual Care	AIM HI (Wave 1 and Wave 2)	Totala
Program characteristic
No.	16	22	38
Setting, No. (%)
Outpatient clinic	8 (50.0)	10 (45.5)	18 (47.4)
School	8 (50.0)	4 (18.2)	12 (31.6)
Multiple settings (school, clinic)	0	8 (36.4)	8 (21.1)
No. of participating therapists, mean (SD)	4.44 (3.14)	7.55 (7.42)	6.24 (6.14)
Therapist characteristic
No.	42	130	172
Female sex, %	37 (88.1)	110 (84.6)	147 (85.5)
Age, mean (SD), y	35.3 (8.2)	33.5 (8.1)	34.0 (8.1)
Race/ethnicity, %
Hispanic/Latinx	17 (40.5)	42 (32.3)	59 (34.3)
Non-Hispanic white	24 (57.1)	60 (46.2)	84 (48.8)
Asian American	1 (2.4)	18 (13.80	19 (11.0)
African American	0	4 (3.1)	4 (2.3)
Multiracial	0	1 (0.8)	1 (0.6)
Unknown/not reported	0	5 (3.8)	5 (2.9)
Primary MH discipline, %
Marriage and family therapy	20 (47.6)	52 (40.0)	72 (41.9)
Clinical psychology	3 (7.1)	25 (19.2)	28 (16.3)
School psychology	5 (11.9)	13 (10.0)	18 (10.5)
Social work	11 (26.2)	34 (26.2)	45 (26.2)
Other disciplineb	3 (7.2)	6 (4.6)	9 (5.3)
Clinical experience, mean (SD) [range], y	5.48 (4.40) [0-20]	5.41 (4.91) [0-24]	5.43 (4.77) [0-24]
Licensed in MH discipline (%)	12 (28.6)	36 (27.7)	48 (27.9)
Child/family characteristic
No.	46	156	202
Child
Age, mean (SD) y	9.4 (2.4)	9.1 (2.5)	9.1 (2.4)
Male sex, %	38 (82.6)	132 (84.6)	170 (84.2)
Race/ethnicity, %
Hispanic/Latinx	31 (67.4)	90 (57.7)	121 (59.9)
White	8 (17.4)	43 (27.6)	51 (25.2)
African American	5 (10.9)	6 (3.8)	11 (5.4)
Multiracial	1 (2.2)	8 (5.1)	9 (4.5)
Asian/Pacific Islander	0	8 (5.1)	8 (4.0)
American Indian/Alaskan Native	1 (2.2)	1 (0.6)	2 (1.0)
Caregiver marital status (married), %	19 (42.2)	86 (55.1)	105 (52.2)
Annual household income, $, %
<25 000	26 (56.5)	65 (41.7)	91 (45.0)
25 001-75 000	14 (30.4)	62 (39.7)	76 (37.6)
>75 000	6 (13.0)	29 (18.6)	35 (17.3)
Child clinical characteristic, mean (SD)
Cognitive standard scorec	86.00 (16.23)	89.27 (16.64)	88.49 (16.56)
ADOS-2 comparison score	6.84 (1.91)	7.11 (2.06)	7.05 (2.03)
SRS-2 total t score	82.98 (9.80)	79.20 (11.45)	80.06 (11.19)
ECBI
Intensity scale scored	63.07 (9.61)	63.08 (10.73)	63.08 (10.46)
Problem scale scoree	62.53 (10.16)	64.14 (10.57)	63.78 (10.47)
SSIS Competing Problem Behaviors scalef	136.24 (14.54)	131.51 (16.33)	132.57 (16.03)
Treatment characteristic
No. of sessions	13.59 (6.70)	15.05 (6.36)	14.71 (6.45)
Therapist fidelity across 6 mo, mean (SD)g
Adherence
Child directed strategies	2.78 (0.74)	3.55 (0.78)	3.37 (0.84)
Caregiver directed strategies	1.84 (0.77)	3.22 (0.74)	3.07 (0.86)
Therapist effectiveness (competence)
Session structure	2.80 (1.0)	4.55 (0.71)	4.16 (1.07)
Continuity across sessions	2.47 (1.11)	4.52 (0.80)	4.07 (1.22)
Pursuit of child skill	3.09 (1.53)	3.77 (1.13)	3.61 (1.26)
Pursuit of caregiver skill	1.27 (1.30)	3.41 (1.11)	3.15 (1.33)

Abbreviations: ADOS-2, Autism Diagnostic Observation Schedule, Second Edition; AIM HI, An Individualized Mental Health Intervention for ASD; ECBI, Eyberg Child Behavior Inventory; MH, mental health; SRS-2, Social Responsiveness Scale, Second Edition; SSIS, Social Skills Improvement System scale.

^aBecause significant differences between groups were found for program setting, this variable was included as a covariate in all models; no other significant differences in program, therapist and child characteristics by condition or between wave 1 and wave 2 were found.

^bIncludes psychiatry, licensed professional clinical counselor, and art therapist.

^cBased on the Wechsler Abbreviated Scale of Intelligence–II or Differential Ability Scale–II, depending on child age.

^dRepresents the frequency of disruptive behaviors.

^eRepresents the total number of behaviors endorsed as being a problem for the caregiver.

^fIncludes a broad array of behaviors that can interfere with a child’s social skill development.

^gScored on a 7-point Likert scale reflecting the extent to which the strategy was used in a specific session; 0 indicates that the strategy was not observed; 1 to 2, 3 to 4, and 5 to 6 reflect low, moderate, and high extensiveness, respectively.

After initial enrollment, therapists referred children for an eligibility assessment conducted by study staff. Children were eligible if they met the following 5 criteria: (1) were aged 5 to 13 years old at the time of recruitment; (2) spoke English or Spanish as their primary language; (3) had an existing ASD diagnosis on record, and (4) exhibited clinically significant ASD symptoms on at least 1 of 2 standardized ASD diagnostic measures: the Autism Diagnostic Observation Schedule, Second Edition (ADOS-2)³⁶ or the Social Responsiveness Scale, Second Edition (SRS-2)³⁷; and (5) exhibited at a least 1 challenging behavior. Of 202 child participants, most (170 [84.2%]) were male, 121 (59.9%) were Latinx, and the children had a mean (SD) age of 9.1 (2.4) years. In total, 190 children (94%) met criteria on both the ADOS-2 and SRS-2, and 12 children (6%) met criteria based on the SRS-2 and had previously received a diagnosis of ASD from a community health care professional specializing in ASD diagnostic assessment.

Intervention

The AIM HI protocol is a package of parent and child directed strategies aimed to reduce challenging behaviors in children with ASD aged 5 to 13 years and designed specifically for delivery in publicly funded MH services. The AIM HI protocol contains well-established behavioral intervention strategies, which include collaborating with the child’s parent to identify patterns in the child’s challenging behaviors and actively teaching (through modeling, behavioral rehearsal, reinforcement, and in-home practice) positive alternative skills for children and complementary strategies for parents to help prevent the occurrence of challenging behaviors and to promote the child’s use of alternative skills. The AIM HI protocol also includes strategies to adapt psychotherapy structure for this population to facilitate engagement and skill building (eg, maximizing predictability). A minimum of 13 sessions is required, and it typically takes approximately 6 months to complete the AIM HI protocol steps.³¹

The AIM HI therapist training model includes active, direct, and explicit instructional methods applied through the following 4 components: (1) an introductory training workshop, (2) 11 (9 group and 2 individual) structured consultation/feedback meetings (1 hour in length) with an AIM HI trainer who is supervised by the AIM HI developer during a 6-month period with didactic instruction and case-specific feedback, (3) performance feedback from trainers based on review of the therapists’ therapy session recordings with participating clients, and (4) access to a therapist manual, printed and electronic forms, and resource website.

UC Waitlist Control

Therapist participants in the waitlist control condition delivered routine care to participating children. The therapists submitted recordings of their therapy sessions to the research team for the 6-month observation period and completed all study measures; no AIM HI training or clinical feedback was provided to therapists in the control condition.

Assessments

Baseline assessments were conducted by the research team (which included among others C.C. and M.B.-E.) at each participating program site. In addition to the ADOS-2 and SRS-2 eligibility assessments, the baseline assessment included a measure of cognitive functioning (the Wechsler Abbreviated Scale of Intelligence-Second Edition³⁸ or the Differential Ability Scales-II³⁹ based on the child’s age), parent report of challenging behaviors (the Eyberg Child Behavior Inventory [ECBI]⁴⁰ and the competing behavior scale of the Social Skills Improvement System [SSIS]⁴¹), and parent report of sociodemographic characteristics. The ECBI and SSIS competing problem behavior were the preidentified primary outcome measures, with higher scores indicating greater challenging behaviors. Participants received a $40 gift card for their time for completion of the baseline assessment. Research staff contacted parent participants by telephone at 6-, 12-, and 18-month follow-up visits to administer the follow-up measures, including the ECBI and SSIS; caregivers received a $25 gift card after each completed assessment for their time (up to $115 across all time points). Caregivers were contacted for each follow-up assessment regardless of whether their child was still in treatment with a study therapist; the proportion of children in treatment across conditions was 73.3%, 25.9%, and 17.8% at 6, 12, and 18 months, respectively. There was no difference in the likelihood that the child would remain in treatment by study condition.

Trained coders naive to study condition coded in-session therapist and child behaviors observed via video-recorded sessions collected during the 6 months training/consultation or usual care observation period. Recordings were randomly selected from 2-month windows representing the beginning (months 1-2), middle (months 3-4), and end (months 5-6), and the mean of the scores calculated across coded sessions for the 6-month period. The mean (SD) number of coded sessions per child was 7.5 (3.3) sessions. Therapist fidelity scores included 2 adherence composites reflecting the extent to which the therapist was observed to use strategies expected in most AIM HI sessions (1 comprising AIM HI strategies delivered to caregivers, and 1 comprising strategies directed to children) and 4 effectiveness ratings (session structure, continuity across sessions, pursuit of child skill, and pursuit of caregiver skill).

Statistical Analysis

Given the clustered nature of the data, multilevel modeling with random intercepts and slopes was used as the primary statistical analysis procedure: repeated measures (level 1) nested within children/therapists (level 2) nested within programs (level 3). To test for intervention effects across time, a group (usual care vs AIM HI) by time (baseline and 6, 12, and 18 months) cross-level interaction was assessed. If a statistically significant (P < .05) interaction term was evident, simple slopes analyses were conducted. To test for trajectory differences between groups on the primary outcomes, we conducted 2 sets of analyses. First, we compared usual care to AIM HI wave 1 only. Then, we compared usual care to the full AIM HI sample (wave 1 and wave 2 combined). This later, larger data set was used for moderation and mediation analyses. To test for moderators (eg, child characteristics) of the group by time interaction, the group by time by moderator cross-level interactions were statistically evaluated. To determine if therapist fidelity (adherence and effectiveness) scores mediated the intervention effect for outcomes at 6, 12, or 18 months, the longitudinal mediation approach of Cole and Maxwell⁴² was used (Figure 2). The asymmetric confidence interval (CI) was used to formally test for mediation⁴³; CIs that did not contain the value 0 were considered statistically significant mediated effects. These models also included unstructured autoregressive effects for the target outcomes. All analyses used an intent-to-treat approach and multilevel modeling analyses using the maximum likelihood robust estimation procedure implemented in the Mplus software.⁴⁴ The estimation procedure adjusted for missing data and non-normality of the outcome variables. Preceding the formal multilevel model analyses, we identified variables (eg, autism severity) that were significantly associated with missing data patterns. These variables were subsequently entered into the multilevel models to account for missing data under the missing at random assumption.⁴⁵ The power analysis program RMASS2⁴⁶ was used to estimate the sample size necessary to find statistically significant (α = .05) intervention effects given the prospective design. These power analyses assumed an effect size (d) of 0.40 at the last time point and an overall attrition rate of 20%. Moreover, clustering at the program level was accounted for with the design effect, with an mean number of therapists per program assumed to be 7 and an intraclass correlation coefficient of 0.05. Given these assumptions, 206 programs/therapists nested with 29 clinics/therapists were needed at the beginning of the study to detect the predicted intervention effect with 80% power.

Figure 2. Conceptual Mediation Model Examining Eyberg Child Behavior Inventory (ECBI) Outcomes at 6, 12, and 18 Months via Therapist Fidelity.

Results

Intraclass correlation coefficients (ICCs) showed significant variability at the child/therapist level for the ECBI Intensity (ICC = 0.69), ECBI Problem (ICC = 0.67), and the SSIS Competing Problem Behaviors (ICC = 0.69) scales. However, minimal variability was evident at the program level for all outcomes (0.01-0.02). Significantly higher autism severity scores at baseline for those that had missing data vs not (mean [SD], 82.49 [10.85] vs 79.06 [11.22]) and in the usual care (43.5%) relative to the AIM HI group (25%) were found (all P < .05). Moreover, statistically significant baseline differences for the outcomes were found for 2 study variables; higher autism severity scores were associated with higher scores on both the ECBI Intensity (B = 0.17, P = .003) and Problem (B = 0.16, P = .04) scales at baseline. Moreover, significant differences in Problem scale scores were found as a function of primary service setting, with higher scores in clinic settings relative to school settings (B = −3.73, P = .001). These variables were entered as covariates variables in multilevel modeling.

Differences in Outcome Trajectories by AIM HI Training Group

In the initial analysis comparing usual care to wave 1 only, there was a statistically significant group by time interaction for the Intensity (B = −0.36, P = .02, d = −0.10, 95% CI for d = −0.19 to −0.02) and Problem (B = −0.98, P = .009, d = −0.28, 95% CI for d = −0.49 to −0.07) scales. Simple slope analyses found a significantly steeper decline in ECBI Intensity scores in the AIM HI group (B = −1.32, P < .001) relative to the usual care group (B = −0.96, P < .001) and a significantly steeper decline in ECBI Problem scores in the AIM HI group (B = −1.21, P < .001) relative to the usual care group (B = −0.14, P = .78). There was no statistically significant difference in the group by time interaction for the SSIS Competing Problem Behaviors scale (P > .05).

In the subsequent analyses using both wave 1 and wave 2 data (Figure 3 and Figure 4), there was a statistically significant group by time interaction for the Intensity (B = −0.38, P = .02, d = −0.13, 95% CI for d = −0.24 to −0.03) and Problem (B = −1.00, P = .005, d = −0.36, 95% CI for d = −0.61 to −0.11) scales. Simple slope analyses found a significantly larger decrease in ECBI Intensity scores in the AIM HI group (B = −1.36, P < .001) relative to the usual care group (B = −0.98, P < .001) and a significantly larger decrease in ECBI Problem scores in the AIM HI group (B = −1.22, P < .001) relative to the usual care group (B = −0.20, P = .29). There was no statistically significant difference in the group by time interaction for the SSIS Competing Problem Behaviors scale.

Figure 3. Growth Trajectory by Group for the Eyberg Child Behavior Inventory (ECBI) Problem Scale.

Figure 4. Growth Trajectory by Group for the Eyberg Child Behavior Inventory (ECBI) Intensity Scale.

Moderators and Mediators of Intervention Effects

Child characteristics (age, sex, race/ethnicity, household income, IQ, and ASD severity) did not significantly moderate outcome trajectories on the ECBI scales (all P > .05, all R² < .01). Treatment continuity (MacKinnon 95% asymmetric CI, −5.34 to −0.67) and session structure (MacKinnon 95% asymmetric CI, −4.75 to −0.15) both significantly mediated the intervention effects for ECBI Problem behavior at 6 months. Specifically, children in the intervention group (relative to the UC group) had significantly higher continuity across sessions (B = 1.79, P < .001, R² = 0.59) and session structure (B = 2.18, P < .001, R² = 0.56), which in turn was associated with lower ECBI Problem scores at 6 months (for continuity: B = −1.54, β = −0.13 P = .04; for session structure: B = −1.35, β = −0.16, P = .01). No additional significant mediation effects were found for ECBI scores.

Discussion

In this cluster randomized trial, we tested the effectiveness of training therapists who were providing publicly funded MH services to deliver a package of evidence-based strategies for ASD. Our results showed that children whose therapists received AIM HI training and ongoing consultation showed greater decreases in the frequency (ECBI Intensity scale) and severity (ECBI Problem scale) of parent-reported challenging behaviors across 18 months compared with children whose therapists delivered usual care. Effect sizes estimates indicated small to medium differences between groups. Child characteristics showed no moderating effects. Observer-rated indicators of therapist delivery of AIM HI within sessions mediated these intervention effects. There were no significant differences in the outcome trajectories on a broader assessment of problems on the SSIS.

This study makes a number of important contributions. First, to our knowledge, this is the first community effectiveness trial of an ASD psychosocial intervention tested within the context of publicly funded MH services. The limited number of ASD effectiveness trials^29,47 and community-based studies^21,22,24,48 that have been conducted were in education, early intervention, or specialized ASD service settings. Second, the development of AIM HI is unique among ASD interventions because it was designed specifically for the end users and the targeted community service setting based on a comprehensive needs assessment in collaboration with relevant stakeholders.^9,10,11 The present study results support the effectiveness of a model designed to fit the needs and structure of community services when delivered by therapists who do not specialize in ASD. The significant decreases in frequency and severity of challenging behaviors suggests that AIM HI was effective in reducing the problem behavior symptoms it was designed to improve. Although modest, the ECBI effect sizes are consistent with or higher than those in trials of psychosocial interventions for youth⁴⁹ compared with usual care with similar designs (eg, youth who were clinically referred, the majority of sample from ethnic minority backgrounds, and parent/family focused interventions). The lack of differences in outcome trajectories for a broader assessment of problem behaviors (SSIS Competing Problem Behaviors scale) may relate to the scale content (a broad array of behaviors considered to influence social skill development).

Strengths and Limitations

A strength of this study is the large and diverse sample of therapists who are representative of the MH workforce (eg, master of arts–level licensed and trainees) and clients (primarily ethnic minority and low socioeconomic status with a large proportion of caregivers having limited English). Although the sample of 202 children is relatively small given the heterogeneity of ASD, the lack of differences in intervention effects by child characteristics (race/ethnicity, language, and income) and setting (clinic vs school) is promising and may support the effectiveness of AIM HI with diverse populations and in multiple settings.

Methodologic strengths include the use of a cluster randomized trial, which addresses pragmatic concerns related to contamination of data. In addition, collecting data on effectiveness (child symptoms) and implementation (therapist fidelity) responds to calls for hybrid research designs to accelerate the translation of evidence-based practices into routine care.⁵⁰ The finding that therapist fidelity related to session structure and continuity across sessions, using the criterion standard method of observational coding, mediated child outcome trajectories suggests the need for development and testing of implementation strategies to enhance therapists’ fidelity to deliver evidence-based interventions such as AIM HI. It is possible that therapists with higher scores on session structure and continuity across sessions engage in more focused skill-building activities, which results in reductions in challenging behaviors. Enhancing fidelity is the primary focus of 2 linked trials currently under way that aim to test implementation strategies across 2 ASD community-based interventions (AIM HI and Classroom Pivotal Response Teaching).⁵¹

The present study findings should be considered within the context of a number of limitations. First, the outcomes measured were limited to parent report of child challenging behaviors. The next steps are to examine changes in child behaviors reported by teachers and changes in parent functioning. Second, given the complexity of obtaining data within a community trial, there were a number of missing data points, particularly in the usual care group. This limitation was minimized through an analytic approach accounting for missing data. Third, although the researchers conducting the outcome assessments were blinded to the condition, clients/caregivers were not. Fourth, because the study simultaneously tested the AIM HI intervention protocol and therapist training model, it was not possible to disentangle the effectiveness of the intervention from that of the training. Future study could control for therapist training by comparing training in 2 active treatment groups to further understand the effectiveness of AIM HI on child outcomes.

Conclusions

The findings of this community trial support the effectiveness of training therapists in publicly funded MH services, who are not ASD specialists, to deliver evidence-based strategies to reduce challenging behaviors among children with ASD. The study findings highlight the importance of developing and testing implementation strategies aimed to facilitate high therapist fidelity in delivering evidence-based interventions.

Supplement 1.

Trial Protocol

Supplement 2.

Data Sharing Statement