Training Community Mental Health Therapists to Deliver a Package of Evidence-Based Practice Strategies for School-Age Children with Autism Spectrum Disorders: A Pilot Study

Abstract

Research on moving evidence-based practice (EBP) intervention strategies to community service settings for children with autism spectrum disorders (ASD) is urgently needed. The current pilot study addresses this need by examining the feasibility, acceptability and preliminary outcomes of training therapists practicing in community mental health (CMH) clinics to deliver a package of EBP strategies aimed to reduce challenging behaviors in school-age children with ASD. Results indicate that CMH therapists participated in both initial and ongoing training, were able to deliver the intervention with fidelity, and perceived the intervention strategies as useful. Parents participated in almost all sessions with their children and remained in therapy when therapists delivered the intervention. Meaningful reductions in child problem behaviors occurred over 5 months providing promising support for the intervention.

Introduction

The Centers for Disease Control and Prevention (CDC) estimate that 1 in 110 children have an autism spectrum disorder (ASD) (CDC 2009). Further, the societal cost of ASD is currently estimated to be $35–$90 billion annually, with costs between $3 and 5 million per child beyond the ordinary costs of raising a child (Ganz 2007). Children with ASD are significantly more likely to have problems regarding access to care and unmet needs, and their families have greater financial, employment, and time burdens compared with other children with special health care needs (Kogan et al. 2008). Given the increasing rates of ASD and the high costs of caring for individuals with these disorders, developing and testing methods to treat and deliver effective care in the community is critical to meet this growing public health need.

The Interagency Autism Coordinating Committee (IACC) 2011 Strategic Plan for Autism Spectrum Disorders Research calls for research on methods to improve dissemination, implementation, and sustainability of evidence-based interventions, services, and supports in diverse community settings (Interagency Autism Coordinating Committee 2011). Further, it calls for intervention research for school-age children with ASD. The community mental health (CMH) system plays a particularly important, but relatively unstudied role in caring for school-age children with ASD for treatment of behavioral and co-occurring psychiatric problems associated with ASD. CMH services are designed to address psychiatric problems through psychosocial and pharmacological interventions.

Approximately 70% of children with ASD meet criteria for at least one additional psychiatric disorder (Leyfer et al. 2006; Simonoff et al. 2008). Further, challenging behaviors (e.g., aggression, noncompliance, self-injury) are commonly associated with ASD and significantly impact child and family functioning (Horner et al. 2002). Research indicates that children with ASD represent approximately 10–14% of children in psychiatrically-referred populations (Joshi et al. 2010; Sverd et al. 2003). Most of these youth have milder forms of ASD (i.e., PDD-NOS or Asperger's Disorder). Our recent research indicates that most therapists in CMH clinics report currently serving a child with ASD and children with ASD represent, on average, 21% of therapists’ caseloads (Brookman-Frazee et al. 2011b). Children with ASD receiving CMH services are treated for similar problems (behavior problems) as children without ASD within the same system (Mandell et al. 2005). The high prevalence of psychiatric problems for children with ASD, the common occurrence of challenging behaviors associated with ASD, and data suggesting that CMH clinics serve many children with ASD all underscore the importance of the CMH system in caring for this population.

Unfortunately, recent research suggests that there are gaps in the quality of CMH services for children with ASD. Providers in CMH clinics that are generalist in practice (i.e., not specializing in a particular disorder) do not have enough specialized knowledge about ASD to effectively treat this population, particularly those who present with challenging behaviors. For example, research on community providers’ knowledge about ASD indicates that providers (including MH professionals and other medical professionals) hold a number of inaccurate beliefs about ASD (Heidergerken et al. 2005). Further, observational data on outpatient psychotherapy for children with ASD suggest that therapist practice is inconsistent with evidence-based practice (EBP) strategies (Brookman-Frazee et al. 2010). Therapists have limited training in ASD and are highly frustrated serving this population (Brookman-Frazee et al. 2011b). Likewise, parents view CMH services as ineffective and are frustrated and stressed by the slow rate of child progress and therapists’ lack of “tools” to work with this population (Brookman-Frazee et al. 2011a). Improving care in this sector is clearly warranted and implementing existing EBPs for ASD (Lord and Bishop 2010; National Standards Report 2009) in CMH clinics offers an innovative solution to the existing “quality gap.”

Several EBP strategies for children with ASD have been developed and tested in controlled settings. There is strong empirical support for child skills training and parent-mediated interventions based on principles of applied behavior analysis and cognitive behavioral therapy methods (Lord and Bishop 2010; National Standards Report 2009; Vismara and Rogers 2010). Unfortunately, existing treatment protocols were not designed for delivery within CMH clinics (i.e., providers with limited ASD experience who provide care within the traditional 1 h per week model). EBP strategies for behavior problems in children with ASD are typically “focal intervention practices” (Lord and Bishop 2010) that are not packaged as comprehensive protocols that may be readily adopted for use in CMH clinics.

An Individualized Mental Health Intervention for Children with ASD (AIM HI)

(Brookman-Frazee and Drahota 2010) is a clinical intervention that includes a corresponding therapist training protocol developed to address the lack of readily “implementable” EBPs to treat behavior problems in children with ASD by identifying and packaging appropriate EBP strategies based on the clinical needs of these children and training needs of their therapists. The AIM HI clinical intervention protocol and therapist training model were developed through a research-community partnership approach in which community stakeholders (organizational leaders, clinicians, and consumers) partnered with ASD intervention experts to integrate findings from a comprehensive mixed-methods needs assessment (Brookman-Frazee et al. 2011a, b) and existing EBP strategies for this population (National Standards Report 2009).

AIM HI is a package of parent-mediated and child-focused EBP strategies, based on the principles of applied behavior analysis, designed to reduce behavior problems in children with ASD ages 5–13 served in CMH clinics. The individual components of AIM HI (e.g., functional behavior assessment, use of antecedent and consequence-based strategies, skill modeling, direct skill practice with feedback, self-management, environmental modifications for ASD) are classified as “well established” by the National Autism Center's National Standards Project (National Standards Report 2009) which is based on a rigorous scientific merit rating system. The empirical support for the components is primarily based on single case design studies (e.g., 21 studies of self-management; 50 studies on modeling supporting the efficacy) which isolates the active ingredients in child behavior change (Kratochwill et al. 2010). Understanding the mechanism of action for individual components provides an empirically-based foundation for a “package” of intervention strategies.

Challenging behaviors are common and debilitating in this population regardless of co-occurring non-ASD psychiatric diagnoses (Kim et al. 2000; Matson et al. 2009; Wood and Gadow 2010). Data gathered from CMH settings indicate that challenging behaviors are the primary presenting problems for children with ASD across co-occurring conditions (Brookman-Frazee et al. 2011a, b; Mandell et al. 2005). We considered using individual protocols for specific co-occurring diagnoses (e.g., Adapted CBT for anxiety and ASD (Wood et al. 2009)). However, based on input from our community providers that it would not be feasible to train providers in multiple protocols for each combination of ASD co-morbidity, we decided to target the most common presenting problems (i.e., challenging behaviors) instead. This decision was also influenced by the strength of the evidence base for addressing challenging behaviors in ASD and limited evidence base for a number of co-occurring conditions (e.g., ADHD, depression). Specific co-occurring problems are addressed in the context of a behavior plan and broader treatment plan developed for each child.

The clinical intervention is combined with a multi-component therapy training approach that includes: (1) an interactive introductory workshop, (2) self-study using web-based and printed materials, and (3) ongoing supervisory consultation, practice, and feedback on delivery with an actual client. The therapist training model was designed based on research on effective provider training methods (Herschell et al. 2010) and therapists’ reported need for comprehensive training in ASD (Brookman-Frazee et al. 2011b).

Although there is growing research on the process and outcomes of implementing EBP strategies for children with mental health problems as well as those in the Child Welfare system (Schoenwald et al. 2003; Zazzali et al. 2008), there is limited information on training community therapists to work with children with ASD (Vismara et al. 2009b). Vismara and colleagues (2009b) conducted one of the few studies in this area for ASD. Results indicate that it is feasible to train community therapists specializing in early intervention for ASD to deliver an evidence-based intervention for infants and toddlers with ASD (Early Start Denver Model) (Vismara et al. 2009a). Specifically, with comprehensive training, a vast majority of therapists met fidelity of implementation and were satisfied with intervention use. Further, significant child gains occurred over time. Although this study provides initial support for training community therapists to deliver EBPs for ASD, it is not known if it is feasible to train therapists practicing in general mental health settings (i.e., not specializing in ASD or applied behavior analysis) to deliver a package of EBP strategies for school-age children with ASD and challenging behaviors.

The purpose of this study was to examine the feasibility of training CMH therapists to deliver a package of EBP strategies (AIM HI) to children with ASD and challenging behaviors, and their parents within routine services provided within CMH clinics. AIM HI was designed specifically for this service setting. Specifically, this study was designed to examine: (1) therapists’ participation in comprehensive training, (2) therapists’ use of AIM HI (fidelity of implementation), (3) therapists’ perceptions of utility of the intervention, and (4) changes in child behaviors after receiving 5 months of AIM HI from CMH therapists.

Method

Participants

Study participants included thirteen therapist/family dyads from three CMH clinics from one organization in a large, geographically diverse county. This organization was selected because it is the largest contractor for publicly funded, clinic-based outpatient care for children in one of the most populated counties in the United States. These clinics serve racially/ethnically, and diagnostically diverse children and their families. None of the clinics specialized in treatment of children with ASD or in EBP strategies for other disorders before or during the study period; however, clinic administrators and previous research collected from these clinics indicated that children with ASD are served in these clinics. Further, representatives from these clinics collaborated on the development of AIM HI. Since the purpose of this study was to examine the feasibility of training community therapists to deliver AIM HI in their routine practice, there were minimal exclusion criteria for participants.

Therapist Participants

During August–September, 2009, all practicing therapists from participating clinics were recruited through weekly treatment teams/supervision meetings. The timing of recruitment was selected to maximize the number of therapists eligible for participation based on when trainees typically begin providing care in the clinics. Of the 74 therapists present during the recruitment meetings, 76% (n = 56) agreed to be contacted about the study. We initially enrolled all fourteen therapists who met the inclusion criteria by the end of October, 2009. Inclusion criteria included: (a) employed as staff or a trainee at participating clinic; (b) employed for at least the next 9 months; (c) had an eligible client on current caseload (see client inclusion criteria below); and (d) did not collaborate on the development of AIM HI. Therapists received an honorarium ($100) for agreeing to participate in the study. One therapist was excluded because she was unable to participate in ongoing AIM HI training due to scheduling difficulties. The final 13 therapist participants included in the study were primarily female (85%), and 46% were Caucasian. The range of years of experience was 0–11 years (M = 3.75 years). Consistent with national samples of therapists in community-based mental health care (e.g., Glisson et al. 2008) and larger studies in the same service system (Brookman-Frazee et al. 2009) therapists were primarily master's level clinicians (77%), with 23% at the Bachelors level (currently in graduate training) and 77% considered trainees. Therapists came from different mental health disciplines: Marriage and Family Therapy (39%), Psychology (39%), and Social Work (23%). Therapist self-reported primary theoretical orientation included: Cognitive-behavioral (31%), Behavioral (15%), Family Systems (15%), Humanistic/Psychodynamic (8%) and Eclectic (15%). Therapists had an average of 13.61 clients on their caseloads (SD = 7.28; Range = 5–29) at baseline. Consistent with previous research in these settings (Brookman-Frazee et al. 2011b), children with ASD represented, on average, 24% of therapists’ current caseloads (SD = 0.12; Range: 8–50%).

Client/Family Participants

Inclusion criteria for child participants were (a) child age between 5 and 13 years at the time of recruitment, (b) have an existing ASD diagnosis on record, (c) meets criteria for autism or ASD on the Autism Diagnostic Observation Schedule (ADOS), (c) uses phrase speech, (d) English-speaking, (e), receiving treatment with the participant therapist for <1 year prior to study participation, and (f) caregiver consents to regularly attend therapy sessions with child (at least 1 time per month). Family recruitment was linked to therapist recruitment. All therapists interested in study participation obtained permission to be contacted by the research team from parents of all children with ASD on their current caseload who met the above-described criteria. During the 2 month recruitment period, 17 parents of children with an existing ASD diagnosis within the target age range provided permission to be contacted for the study eligibility assessment. The ADOS was administered by the second author who has received clinical and research training in ADOS administration and was supervised by an on-site certified ADOS trainer. Four children were excluded from the study for the following reasons: did not show to the baseline eligibility assessment (n = 1), did not meet criteria on the ADOS (n = 2), therapist unable to participate in training (n = 1). Due to HIPAA restrictions we could not collect data on non-participants, so information about how non-participants may have differed from participants is unavailable.

Informed written consent was provided by the parent and assent was provided by children ages 8 and older. Families were assured that their decision regarding participation would not impact their receipt of clinic services. Family participants were given financial incentives to participate in the study ($20 to the parent at the baseline and post assessments and $10 to the child for the baseline assessment). All protocols were approved by affiliated university, hospital, and county research review committees.

The 13 participating children were ages 5–13 years (M = 10.3, SD = 2.3), 100% were male and 77% were Caucasian, 15% Latino, and 8% Mixed, which is generally consistent with the literature on the population of children with ASD in mental health settings (Brookman-Frazee et al. 2010; Joshi, et al. Mandell et al. 2005). Clinician-reported ASD diagnoses were: 46% Asperger's Disorder, 39% PDD-NOS, 15% Autistic Disorder. Sixty-nine percent of children also had a non-ASD co-occurring psychiatric diagnosis according to clinician report, including 39% ADHD, 23% Anxiety Disorder, 15% Disruptive Behavior Disorder, 8% Mood Disorder. Care for the majority of families was funded through the school system (85%); the remaining families were funded through government sources (15%). Caregivers were an average of 43.42 years (SD = 6.30; Range: 31–53 years). The majority (62%) were married or living with partners. Fifty-four percent reported a college degree/some graduate school, 23% reported some college, and 23% reported a high school diploma/GED.

Procedures

AIM HI Clinical Intervention

AIM HI is a package of evidence-based parent-mediated and child focused strategies, based on the principles of applied behavior analysis, designed to reduce behavior problems in children with ASD ages 5–13 served in CMH clinics. AIM HI involves a series of 10 steps (8 required, 2 as indicated based on individual child needs and described below) and within-session elements aimed to teach parents of children with ASD to manage challenging behaviors (e.g., aggression, oppositionality, noncompliance, tantrums) and teach children positive alternative skills (e.g., self-regulation, social skills) using applied behavioral analytic EBP strategies. The specific AIM HI protocol steps are listed in Table 1. During treatment, the therapist actively collaborates (by proactively seeking and incorporating parent input) with the parent(s) and child to identify external functions and patterns of challenging behaviors, develop a treatment plan, and identify/teach functionally-equivalent positive alternative behaviors to the child. Sessions are structured to maximize engagement with parent and child and to maximize session time for active parent and child skill-building by using a predictable session, making transitions/changes predictable, visual aids, and priming. AIM HI sessions include the following elements: (1) review what the family did between sessions to practice skill targeted in therapy; (2) teach/review parent or child skills using active teaching strategies (describing skill, modeling, practice-with-feedback); (3) summarize session and assign between-session practice; and (4) end with a motivating activity. AIM HI was designed to be flexibly implemented in the context of psychotherapy by providers who have limited experience with ASD. A minimum of 13 sessions is required to complete the AIM HI protocol; however, the protocol can be repeated as necessary for a specific child as clinically indicated.

Table 1.

AIM HI protocol steps and within-session elements

Treatment Phase	AIM HI protocol steps	Approx. # sessions	Within-session elements
Treatment planning (TP)	1. Collect/Integrate assessment information	1–2	• Active parent involvement (collaboration; active teaching)
	2. Behavior tracking with parents (functional behavior assessment)	2	• May occur between or within sessions
	3. Develop/Review behavior plan	1
	4. Develop/Review comprehensive treatment plan
Active treatment (AT)a	5. Teach/Review antecedent and reinforcement strategies with parents	2–3	• Structuring sessions for ASD (session schedules, priming, visual cues)
	6. Teach first alternative skill(s) to child (through skill modeling, direct skill practice with feedback, reinforcement)	4–6	• Active parent involvement (collaboration; active teaching)
	7. Target generalization of alternative skills (self-management; prompt fading)	2–3	• Active teaching with children (motivational strategies, modeling, practice-with-feedback, between-session practice)
	8. Teach additional alternative skills to child (only as indicated)	0–6
Evaluating progress	9. Review treatment progress	1	• Active parent involvement (collaboration; active teaching)
	10. Develop revised treatment plan (only as indicated)	0-1	• May occur between or within sessions

^aThe number of sessions required to teach alternative skills to the child and target generalization of these skills varies based on the complexity of the skills or skill sequence and the child's learning acquisition rate

The clinical intervention was funded through existing clinic funding sources (i.e., there was no funding provided for therapist time or client services through the research study).

AIM HI Therapist Training

The multi-component therapist training approach included: (1) an interactive introductory workshop, (2) self-study using web-based and printed materials, and (3) ongoing supervisory consultation, practice, and feedback on delivery with an actual client. Specifically, after therapist/family dyads were recruited, therapists received initial training through an introductory workshop and self-study (web-based and printed) materials. The one-day (6 h) workshop was facilitated by the AIM HI developers to provide an overview of ASD characteristics and challenging behaviors in ASD, introduce therapists to underlying concepts of AIM HI (e.g., functional approach to challenging behaviors; treatment as active child and parent skill-building; collaboration with parents), introduce AIM HI steps and required session elements, provide an opportunity to practice treatment planning process, and facilitate interactive learning through discussion and feedback. A total of 39 people attended the workshop. Attendees included the participants in the current study (n = 13) as well as therapists who were interested in participating in the study, but were excluded because they did not have an eligible child on their caseload (n = 25), and the therapist participant who subsequently dropped out of the study due to scheduling conflicts (n = 1). The workshop included didactic lectures, video exemplars, discussion, and interactive practice opportunities. All licensed therapists received continuing education course credit for workshop attendance.

After the initial workshop, participant therapists delivered AIM HI for approximately 5 months with participating families while receiving consultation two times per month with the AIM HI developers. The purpose of the ongoing consultation and practice with an actual client was for therapists to use the AIM HI protocol and specific steps and receive feedback from their AIM HI trainer about use of the steps and master the skills involved. Therapists started at the treatment planning stage of the intervention (see Table 1), regardless of whether the child was a new client or a client continuing with his or her therapist. Each of the 10, 1-hour consultation meetings included the following activities: (1) review of status of each client (e.g., step in AIM HI protocol); (2) brief feedback to group on fidelity and/or video review; (3) review protocol step from previous week; (4) introduce new protocol step; (5) plan for next steps with clients. All 10 sessions were delivered in groups of approximately 3–5 therapists. However, when therapists missed a group consultation meeting, they were asked to participate in a “make up” individual consultation meeting with one of the trainers.

Therapist Measures

Therapist Training Attendance

Trainers tracked therapist attendance at the workshop and consultation sessions.

Therapist Fidelity of AIM HI Implementation

Multiple methods (observer, expert-rated and therapist self-report measures) were used to measure therapists’ use of the core EBP strategies included in the AIM HI protocol (i.e. protocol steps and within-session elements) during each phase of the intervention. Specifically, three measures were used to measure fidelity:

1. Treatment Planning (TP) Phase Fidelity (Expert-rated) was rated by one of the AIM HI developers based on a systematic review of the AIM HI treatment planning forms completed by therapists. As therapists completed the treatment planning forms, one of the AIM HI developers reviewed the forms for adherence to the required elements (e.g., “Was the alternative skill listed on the Behavior Plan observable/measurable?”). The inter-rater reliability of this measure is high (ICC = 0.83).

2. The Active Treatment (AT) Phase Session Fidelity (Observer-rated) included ratings on three required within-session therapist behaviors during Active Treatment sessions (session structure, parent involvement strategies, active teaching with children). Each therapist behavior domain had associated therapist strategies (e.g., used a concrete session schedule, described a specific intervention strategy) which guided the rating. All items were rated on a four-point Likert scale ranging from 0 (did not use the strategy during the session) to 3 (in-depth/thorough use of strategy during the session). The inter-rater reliability of the AT Session Score is high (ICC = 0.77 based on 26 double coded sessions). Further, the AT Session Fidelity Score for AIM HI sessions is significantly higher than usual care session videotapes from the same clinics for a linked study (Brookman-Frazee et al. 2010) providing preliminary support for the discriminant validity of the measure. The average AT Session Fidelity Score was calculated for 11 therapists based on 23 videotaped sessions.

3. The AIM HI Protocol Completion Checklist (Therapist report) measures completion of each of the eight required protocol steps. The therapists completed this measure through a web-based survey after the AIM HI consultation training period (after 5 months of consultation/delivering AIM HI). For each AIM HI step, therapists rated the extent to which they completed each step. Responses included, “Did not complete”, “Not thoroughly”, “Somewhat thoroughly”, and “Very thoroughly.” Therapist ratings were also dichotomized such that the responses of “Somewhat” or “Very” were considered to be completed and other responses were coded as not completed. Separate scores were then calculated for the three phases of AIM HI (Treatment Planning, Active Treatment, Evaluating Progress). Although this is a therapist self-report measure, the scores for the Treatment Planning and Evaluating Progress were validated by the research team since these steps include required forms which were turned into the research team.

Overall, multiple methods were used to measure fidelity to the AIM HI model. These methods were selected to accurately capture therapist delivery of AIM HI throughout treatment. Video observation was used to capture in-session therapy behaviors during the Active Treatment phase of treatment. The expert rated fidelity was used to capture treatment planning fidelity because the level of detail needed to capture treatment planning fidelity may not be observable via video and some therapists completed some of the treatment planning steps outside of the therapy room (e.g., over the phone with the parent). Lastly, therapist self-report was used to capture an overall fidelity to the protocol and sequence of AIM HI steps.

Therapist Perceptions of AIM HI

Therapist participants completed a web-based survey after the AIM HI consultation training period inquiring about their perceptions of the utility of the AIM HI intervention for the client (“How useful was AIM HI with your client?”), the applicability of the intervention to related childhood disorders (“How relevant is AIM HI for children with the following disorders: ADHD, anxiety, mood, disruptive behavior disorder?”), and likelihood of recommending AIM HI to a colleague (“Would you recommend AIM HI to a colleague?”).

Therapist Perceptions of Knowledge and Confidence about Serving Children with ASD

This survey was developed for the current study to assess changes in therapists’ perceptions of their knowledge and confidence serving children with ASD before and after training. This survey was develop based on previous research indicating that therapists in this setting reported feeling frustrated about their lack of “tools” to serve this population (Brookman-Frazee et al. 2011b). The survey included nine items grouped into the following categories: (1) characteristics of ASD, (2) treatment planning with these children, (3) modifying psychotherapy for this population, and (4) research-based intervention strategies. Therapists assigned two ratings to each item: how knowledgeable they felt about each topic and how confident they felt using specific strategies. Ratings were scored on a four-point Likert scale from 1 (Not at all knowledgeable/confident) to 4 (Very knowledgeable/confident).

Child/Family Measures

Parent/Child Treatment Attendance and Retention

Child and family attendance was reported by the therapist during each consultation session.

Observed Level of In-Session Parent Involvement

An observational rating was used to measure the level of in-session parent involvement. This measure was created for this study to obtain the most objective assessment of parent involvement in session. Two observers from the research staff independently viewed an entire videotaped session and rated the extent to which the parent(s) participated throughout the session. Involvement was rated on a five-point scale ranging from 1 (parent was absent/present for less than 10% of the session/watched passively when present) to 5 (parent actively collaborated with therapist). The inter-rater reliability of this measure is high (ICC = 0.90 based on 26 double coded sessions). Further, the average parent involvement rating for AIM HI sessions (n = 44) is significantly higher than usual care session videotapes from the same clinics for a linked study (n = 23) providing preliminary support for the discriminant validity of the measure. The average parent involvement rating was calculated separately for the Treatment Planning (n = 21 sessions) and Active Teaching (n = 23 sessions) phases of treatment.

Changes in Child Problem Behaviors

The Social Skills Improvement System (SSIS: (Elliott and Gresham 2007) Competing Problem Scale was used to measure changes in child problem behaviors. It was administered at the start of treatment and at 5 months after the start. The SSIS competing problem scale provides a targeted assessment of children's (ranging from 3 to 18 years old) problem behaviors (e.g., bullying, hyperactivity/inattention, internalizing and externalizing problems, and autism spectrum). For this study, only the parent-report version was used. The SSIS has robust psychometric properties with internal consistency on the various domains ranging from 0.73 to 0.95 for the parent-report and 0.73 to 0.95 for the child-report. Test–retest reliability coefficients range from 0.73 to 0.87 for the parent-report. Finally, convergent and discriminant validity have been found to be strong (Gresham and Elliot 2008).

Analysis Plan

Descriptive statistics were used to examine therapist attendance, therapist fidelity of implementation, client retention/session attendance, and parent level of involvement. Paired samples t-tests were used to examine changes in therapist attitudes toward treating children with ASD and changes in child behavior problems from baseline to 5 months.

Results

Therapist Outcomes

Training Attendance

Attendance at AIM HI trainings was very high. All participant therapists attended the introductory workshop and the average number consultation meetings attended was 9.69 (SD = 0.63; Range = 8–10) out of 10 meetings.

Therapist Fidelity of AIM HI Implementation

Refer to Table 2 for a summary of therapist fidelity scores and the proportion of therapists who reached fidelity according to each measure. Overall, therapists were observed to follow the AIM HI protocol and deliver AIM HI with a high degree of fidelity.

Table 2.

Therapist fidelity scores

Fidelity measure (possible score range)	Score M (SD; range)	% Therapists reaching fidelity or completing protocol step (n)
Treatment planning (expert-rated) (0–100 Score)	91 (0.06; 81 to 100)	100 (13/13)
Active treatment (observer-rated) (0–3 rating)	2.28 (0.47; 1.67 to 3.00)	100 (11/11)
Completion of required protocol steps (therapist- report) (0–8 Steps)	7.5 (0.97; 5 to 8)	69.2 (9/13)
Treatment planning phase (0–4 steps)	4.0 (0.00; 4 to 4)	100 (13/13)
Active treatment phase (0–3 steps)	2.46 (0.97; 0 to 3)	69.2 (9/13)
Evaluating progress phase (0–1 steps)	1.0 (0.00; 1 to 1)	100 (13/13)

Observer-rated data (based on 23 video-taped sessions) are available for 11 of 13 therapist participants

Therapist Perceptions of AIM HI

Overall, therapists perceived AIM HI as useful. Specifically, 100% (n = 13) of therapists reported that AIM HI was “somewhat” or “very” useful and 100% (n = 13) would recommend AIM HI to a colleague. Additionally, therapists indicated that they felt AIM HI was relevant to other childhood disorders: 100% (n = 13) disruptive behavior disorders, 92.4% (n = 12) anxiety disorders, and 76.9% (n = 10) mood disorders.

Therapist Perceptions of Knowledge and Confidence about Serving Children with ASD

Therapists reported significant increases in perceived knowledge related to working with children with ASD from baseline (M = 28.38; SD = 5.81) to 5 months of AIM HI training (M = 37.38; SD = 3.88), t(12) = –5.81, p < .001. Likewise, therapists reported significant increases in confidence applying their knowledge in their work with children with ASD from baseline (M = 22.85; SD = 5.30) to 5 months (M = 30.77; SD = 3.52), t(12) = –5.76, p < .001.

Child and Family Outcomes

Retention/Session Attendance

Retention was high; 92% of families remained in treatment at 5 months (1 terminated due to funding changes). Parent participation was high (M = 93% of sessions were attended by parents; SD = 0.10; Range = 73–100%).

Level of In-Session Parent Involvement

Parent involvement was high across both Treatment Planning (M = 4.10; SD = 1.4) and Active Treatment (M = 4.00; SD = 1.04) phases of treatment.

Changes in Child Problem Behaviors

Statistically and clinically significant decreases in overall parent-reported child problem behaviors were observed on the SSIS Competing Behavior Total Scale from baseline (M = 123.54; SD = 13.99) to 5 month follow up (M = 116.08; SD = 13.46), t(12) = 2.49, p = .03. On the individual Competing Behavior subscales, significant improvements were observed on the Hyperactivity, Internalizing, and ASD subscales. Specifically, significant decreases in Hyperactivity were reported by parents from baseline (M = 11.85; SD = 2.54) to 5 months (M = 9.08; SD = 2.90), t(12) = 3.86, p < .01. In addition, 85% of children were within the clinical range in Hyperactivity at baseline and 31% were in the clinical range after 5 months of AIM HI treatment, supporting the clinical significance of these changes. Children also decreased in their reported Internalizing problems from baseline (M = 9.77; SD = 5.23) to 5 months (M = 7.08; SD = 3.88), t(12) = 2.56, p = .03. Forty-six percent of children were in the clinical range at baseline compared to 31% after 5 months of AIM HI treatment. There were significant decreases in ASD behaviors from baseline (M = 21.62; SD = 6.27) to 5 months (M = 20.15; SD = 5.87), t(12) = 2.33, p = .04, with 92% of children in the clinical range at baseline and 77% in the clinical range at 5 months.

Discussion

There has been considerable progress in recent decades identifying efficacious interventions for ASD. However, there has been limited attention to implementing evidence-based interventions in community settings. This pilot study examined the use of a package of EBP strategies, AIM HI, designed for children with ASD and challenging behaviors receiving care in CMH clinics. This research begins to address the gap between evidence-based intervention strategies and community-based care for ASD. AIM HI was developed based on: (1) a detailed assessment of the clinical needs of children with ASD receiving outpatient therapy in CMH clinics and the training needs of the therapists (Brookman-Frazee et al. 2011a, b, 2010), and (2) evidence-based strategies to address challenging behaviors (Lord and Bishop 2010; National Standards Report 2009; Vismara and Rogers 2010). This study represents a first step in determining the feasibility of implementing such a clinical protocol and corresponding training model in CMH clinics.

A primary aim of this study was to determine whether it is feasible to train CMH therapists, with limited ASD experience, to deliver AIM HI to children with ASD being served in CMH clinics. Results indicate that CMH therapists participate in both initial and ongoing training, are able to deliver the intervention with fidelity, and perceive the intervention as useful for children with (and without) ASD. The data on training participation and fidelity are particularly important given the limited research on effective training methods (Beidas and Kendall 2010; Herschell et al. 2010; Rakovshik and McManus 2010). The use of ongoing consultation with feedback on actual cases is consistent with research suggesting that workshops alone do not impact therapist practice (Herschell et al. 2010). The current study builds on the very limited research on training community therapists to deliver intervention for ASD. Specifically, this study extends the one other study in this area (Vismara et al. 2009b). It is important to note that participant therapists in the (Vismara et al. 2009b) study specialized in providing early intervention to young children with ASD (e.g., had at least 2 years of experience) for whom treatment focused on social communication rather than behavioral outcomes. The current study adds to the literature as it provides preliminary evidence that therapists with limited ASD training can be trained in a relatively short amount of time to deliver EBP strategies to school-age children with ASD and their families with a very high degree of fidelity within routine, community care.

Therapists also perceived AIM HI as useful for children with ASD. This is important because it provides preliminary support for the approach used to develop AIM HI (i.e., integrating data from usual care with the best available evidence derived from efficacy research to package EBPs for specific, targeted service settings). Related, although this study focused on children with ASD, therapists also indicated that they found the intervention strategies useful for children with other psychiatric problems, especially disruptive behavior disorders, ADHD, and anxiety. This makes sense given that AIM HI strategies are based on operant conditioning principles, child skill-building, and parent training strategies that are also common in EBPs for other childhood disorders (Brookman-Frazee et al. 2006; Garland et al. 2010). It may be important for future research to examine whether therapists generalize the skills they learn for children with ASD to their other clients without ASD.

In the current study, parents participated in almost all sessions with their children and remain in therapy when therapists delivered AIM HI. The rates of parent involvement in this study are considerably higher than in usual care provided in the same service setting (Brookman-Frazee et al. 2010; Garland et al. 2010), providing additional support for the impact of training on therapist practices. The AIM HI clinical intervention and training protocol emphasize active collaboration between therapists and parents (and children, as applicable) throughout the treatment planning, active treatment, and evaluation phases. Actively targeting parents is considered an essential element of treatment for children with ASD (Lord and Bishop 2010) as it has the potential to facilitate generalization of child skills to other settings in individuals and increases the number of intervention hours to the child when parents use intervention strategies throughout their daily routines (Brookman-Frazee et al. 2006).

Pre-post improvements in severity of parent-report of children's behavior problems provide preliminary support for the effectiveness of AIM HI in reducing challenging behaviors. These pilot data suggest that the well-established behavioral strategies tested in research settings that comprise AIM HI may also be effective when packaged and sequenced as delivered in AIM HI. A large-scale randomized control trial is needed to confirm and expand these preliminary findings.

Limitations

Some study limitations should be noted. These pilot study findings should be viewed with caution due to the lack of control group and small sample size involved. A large-scale randomized control trial is needed to make conclusions about causal relationships between AIM HI training and therapist fidelity as well as improvements in child behaviors. Due to the sample size, difference in therapist and child characteristics were unable to be examined. Longer term follow up data are needed to examine whether therapists sustain and generalize their use of AIM HI strategies over time and with other clients. Further, information is needed on the long term impact of AIM HI on child outcomes as well as from multiple informants who can report on child behavior.

Although presence of an existing ASD diagnosis was verified by the ADOS to determine if the child met inclusion criteria, descriptive data on additional psychiatric diagnoses were based on therapist report (i.e., there was no formal assessment). It is important to note that, although current DSM-IV criteria excludes certain diagnoses in the presence of PDD (e.g., ADHD, generalized anxiety disorder), we applied a nonhierarchical approach for diagnostic classification because this allowed us to fully characterize the sample in a way that is consistent with how they are characterized in community care and is consistent with the research on psychiatric co-morbidity (Joshi et al. 2010).

Finally, AIM HI was found to be feasible within one large, geographically diverse county. While the distribution of therapist education level is generally similar to other studies of community mental health providers (Glisson et al. 2008) and level of limited ASD experience is similar to a larger sample of therapists from the same county (Brookman-Frazee et al. 2011b), therapist characteristics may differ in other settings and geographic locations. Similarly, the characteristics of children with ASD in this sample may differ from children receiving community mental health services in privately funded mental health settings and/or other locations that may have different fiscal policy factors that impact service referrals for the Mental Health, Special Education, and MR/DD systems. Although there was limited racial/ethnic diversity in the child participant sample, this distribution is similar to other studies of children with ASD in the same County receiving publicly-funded mental health services (Brookman-Frazee et al. 2010) and in national samples (Mandell et al. 2005).

Despite these limitations, the study has several strengths, which contribute to the significance of the results. The clinical characteristics of children with ASD in the current study were similar to those typically seen in CMH settings (Brookman-Frazee et al. 2011a, b, 2010; Joshi et al. 2010). For example, over two-thirds of the children with ASD had at least one psychiatric diagnosis above and beyond their ASD diagnosis. Further, this study used multiple methods to measure therapist fidelity and a standardized measure of child behavior problems was used to examine changes during therapist training. Additionally, the AIM HI intervention and corresponding training model are unique in that they were developed specifically for the CMH context based on the needs of therapists who are seeing high rates of children with ASD. Further, it was developed to address the most common presenting problems of children with ASD in CMH clinics, rather than specific co-occurring disorders, which contributes to the utility of the model. In sum, results of this study provide preliminary support for the feasibility and acceptability of training CMH therapists to deliver a package of EBP strategies to school-age children with ASD, and provide preliminary support for positive child outcomes. Next steps in this program of research involve conducting a large-scale randomized controlled trial to examine the impact of AIM HI on child and family outcomes, and to examine child, therapist and organizational factors associated with child and family outcomes. Lastly, feasibility of the large-scale adoption, implementation, and sustainability of AIM HI must be examined.