Improving Transportability of a CBT Intervention for Anxiety in Youth with ASD: Results from a US-Canada Collaboration

Abstract

Anxiety disorders frequently co-occur in youth with autism spectrum disorders (ASD). In addition to developing efficacious treatments for anxiety in children with ASD, it is important to examine the transportability of these treatments to real-world settings. Study aims were to: a) train clinicians to deliver Facing Your Fears: Group Therapy for Managing Anxiety in Children with High-Functioning ASD (FYF) to fidelity; and b) examine feasibility of the program for novel settings. A secondary aim was to examine preliminary youth treatment outcome. Results indicated that clinicians obtained excellent fidelity following a workshop and ongoing consultation. Acceptability ratings indicated that FYF was viewed favorably, and critiques were incorporated into program revisions. Meaningful reductions in anxiety were reported post-treatment for 54% of children. Results support the initial effectiveness and transportability of FYF in new clinical settings.

Psychiatric disorders frequently co-occur in children with ASD and markedly impede their treatment progress, signaling the need for accessible, individualized, and efficacious interventions (Dawson & Burner, 2011; Levy et al., 2010). Anxiety disorders are among the most commonly co-occurring conditions (Leyfer et al., 2006) and can be debilitating for youth with ASD, markedly interfering with participation in home, school, and community settings. Further, anxiety severity is associated with increased aggression, reduced participation in social activities, and poor relationships for youth with ASD (Selles & Storch, 2012).

Cognitive-behavioral treatments (CBT) are well-established, highly researched, evidenced-based treatments that are considered the gold standard psychosocial treatments for typically developing youth with anxiety (Silverman, Pina, & Viswesvaran, 2008; Walkup et al., 2008). CBT has been extended to youth with ASD, with the completion of at least seven randomized controlled trials yielding positive results (Chalfant, Rapee, & Carroll, 2007; Reaven, Blakeley-Smith, Culhane-Shelburne, & Hepburn, 2012; Sofronoff, Attwood, & Hinton, 2005; Storch et al., 2013; Sung et al., 2011; White et al., 2012a; Wood et al., 2009). Treatment has been delivered individually (Wood et al., 2009), in small groups (Chalfant et al., 2007; Reaven et al., 2012; Sofronoff et al., 2005), and in combined individual and small groups (White et al., 2012a). Treatment studies that have been conducted thus far for children with ASD and anxiety have occurred primarily in tightly controlled university-based research clinics. Thus, there have been limited efforts to deliver the interventions in real-world clinical settings using practitioners who have minimal experience using CBT with youth with ASD. To date, clinician qualifications have varied as some studies have explicitly required clinicians to have therapeutic experience in ASD and anxiety (Storch et al., 2013), while in other studies, prior training in CBT has not been a prerequisite for participation (Sofronoff et al., 2005). A range of clinical settings, including pediatric clinics and community mental health centers, can play a crucial role in the treatment of co-occurring psychiatric disorders in youth with ASD; however, lack of specialized training for providers in these settings may limit the usefulness of these front-line clinicians (Brookman-Frazee, Drahota, & Stadnick, 2012).

Although it has been critically important to develop treatments that address the serious co-occurring anxiety symptoms present in so many children with ASD, it is equally important to facilitate the portability of these treatments beyond the lab setting, into the hands of clinicians across settings (Schoenwald & Hoagwood, 2001). Although enhancing the portability of efficacious treatments outside of clinical research environments is not easy, introducing new treatments into new clinical settings early in the development process may inform protocol development, increase acceptability of the intervention, and maximize success for eventual clinical practice (Beidas et al., 2011; Weisz, Chu, & Polo, 2004).

Introducing Evidence-Based Interventions to Clinicians

Dissemination research is the “systematic study of processes and factors that lead to widespread use of an evidence-based intervention” (Rabin, Brownson, Haire-Joshu, Kreuter, & Weaver, 2008, p. 119). One aspect of dissemination research is transportability or the examination of the transfer of potentially efficacious interventions to usual-care settings (Schoenwald & Hoagwood, 2001). Important questions for transportability research to answer may include defining: a) the intervention; b) the intended target population; c) who can conduct the intervention; and d) under what circumstances the intervention may be delivered (Schoenwald & Hoagwood, 2001). Implementation research seeks to identify the processes that lead to the successful integration of an intervention into a novel setting (Rabin et al., 2008). In other words, implementation research identifies whether the core components of an intervention were delivered as originally intended (e.g., treatment fidelity) and the extent to which the intervention is feasible in a given context. Feasibility examines the acceptability of an intervention by the targeted population, as well as by the clinicians who deliver the treatment program. Factors such as participant recruitment, attendance, and treatment completion may also help determine feasibility of an intervention (Proctor et al., 2011).

Clinicians who are inexperienced with a new intervention may seek professional development opportunities that typically include studying a manual and/or attending a workshop, although this process is often seen as insufficient for true skill attainment (Herschell, Kolko, Baumann, & Davis, 2010). Evidence suggests that behaviorally oriented training techniques such as direct feedback, behavioral rehearsal, and role-play are key workshop components of skill transfer (Beidas & Kendall, 2010; Herschell et al., 2010). Furthermore, the transportability of new treatments is often hindered by a lack of critical feedback and input from practitioners during the development process (Hatgis et al., 2001). Pairing ongoing expert consultation and supervision with intensive trainings to targeted providers, and allowing for bidirectional input from these same providers to the treatment developers, may be important for the successful adoption of new interventions. Allowing for bidirectional input likely creates active and engaged providers rather than passive recipients of training (Beidas et al., 2011). Funneling provider input directly into program revisions is an invaluable step towards enhancing the portability and development of acceptable treatments (Weisz et al., 2004). Moreover, careful attention and planning in the transportability of a new intervention may be a necessary precursor for wide-scale dissemination efforts (Schoenwald & Hoagwood, 2001).

This paper describes the initial steps in the implementation of a group CBT program for managing anxiety in youth with ASD (Facing Your Fears: Group Therapy for Managing Anxiety in Children with High-Functioning Autism Spectrum Disorders (FYF; Reaven, Blakeley-Smith, Nichols, & Hepburn, 2011) in a novel clinical setting. The FYF program followed a multi-stage model essential for the development and implementation of evidence-based practice, including the completion of pilot testing, feasibility trials, and the development of treatment manuals and fidelity measures (Beidas et al., 2011; Reaven et al., 2009; Reaven et al., 2012; Smith et al., 2007). However, rather than move towards large multi-site studies to further assess efficacy, FYF was introduced into a new clinical setting with clinicians new to the FYF program to further inform protocol development and enhance adoptability of the intervention.

The two primary aims of the present study were to: 1) train mental health clinicians outside of our university research program to deliver FYF with fidelity, and 2) examine feasibility of the program (e.g., acceptability/satisfaction data; attendance, treatment completion) to inform further development of the intervention. A secondary aim was to examine preliminary youth anxiety treatment outcomes. It was hypothesized that clinicians in a pediatric hospital setting inexperienced in the FYF intervention would deliver the treatment program to fidelity, following participation in a workshop with ongoing supervision. Additionally, preliminary treatment outcome results were hypothesized to parallel findings obtained from previous FYF treatment trials (Reaven et al., 2009; Reaven et al., 2012), although treatment outcomes may not as robust as in previous trials, consistent with dissemination research (Weisz, Ugueto, Cheron & Herren, 2013).

Method

Participants

Two groups of participants entered the study: clinicians and treatment participants (i.e., 8- to 14-year-olds with ASD and clinical anxiety, and at least one parent). Participants were recruited in Halifax, Nova Scotia through Research Ethics Board (REB)-approved study announcements, clinician referral, and word of mouth. Informed consent and assent was obtained for all participants prior to collecting any data.

Clinicians

Thirteen clinicians participated in the study. Inclusion criteria were: (a) graduate student in a doctoral program in clinical psychology (Dalhousie University), or a Ph.D. level clinical psychologist; and (b) working in the outpatient service for children with ASD at a regional pediatric hospital (IWK Health Centre in Halifax, Nova Scotia). Just over 40% of the clinicians held a doctorate degree in psychology. On average, the clinicians had 3.6 years (range 1–10 years) of experience implementing CBT therapies, but were relatively inexperienced in implementing CBT with youth with ASD (M=1.1 years of experience; range 0–5) (see Table 1).

Table 1.

Clinician Characteristics

Clinician Characteristics (N=13)	M(SD) or %	Range
Gender
Female	92.3%
Age (years)	33.0 (8.5)	25–54
Ethnicity
Caucasian	100%
Education
College degree (B.A., B.S.) + some graduate training	16.7%
Terminal masters (M.A., M.S.)	41.7%
Professional degree (Ph.D.)	41.7%
Experience (years)
Experience with youth with ASD	8.4 (6.6)	2–25
CBT experience with typical populations	3.6 (3.1)	1–10
CBT experience with ASD populations	1.1 (1.6)	0–5

Note: All clinicians were graduate students in a doctoral program in clinical psychology or Ph.D. -level clinical psychologists.

Treatment Participants

Sixteen children and their parents participated in the study. Children were included with the following criteria: (a) chronological age of 8–14 years and living with someone who could give informed consent to participate; (b) diagnosis of ASD, defined by: (1) Autism Diagnostic Observation Schedule (ADOS; Lord et al., 1999) score above spectrum cutoff and (2) clinical diagnosis of an ASD as determined by a review of history and current clinical presentation by a clinical psychologist; (c) estimated Verbal IQ of 85 or above, determined through standardized cognitive testing using the Wechsler Abbreviated Scales of Intelligence (WASI; Wechsler, 2002), or an equivalent measure of intelligence administered within the past two years; (4) ability to read at a mid-2nd-grade level, determined by the Letter-Word Identification and Reading Comprehension subtests of the Woodcock Johnson Achievement Tests–Third Edition (WJ-III; Woodcock, McGrew, & Mather, 2001); and (5) clinically significant symptoms of anxiety, defined as a score above the clinical significance cutoff on separation (SEP), social (SOC), and/or generalized anxiety (GAD) subscales of the Screen for Child Anxiety and Related Emotional Disorders—Parent Version (SCARED; Birmaher et al., 1999). These three diagnoses are commonly treated together because of the considerable overlap in the therapeutic approach for these disorders (Barrett, Healy-Farrell, & March, 2004; Kendall & Hedtke, 2006).

Children were excluded if: (a) their family was not fluent in English; (b) their family could not commit to attending 12 of 14 sessions within the intervention period; (c) they presented with psychosis, severe aggressive behavior, or other severe clinical symptoms that required more intensive treatment such as day treatment or hospitalization; (d) their family intended to seek additional psychosocial treatment for anxiety during the 4 month study period; and (5) they were not able to separate from their parent for a minimum of 30 minutes within the pre-intervention assessment visits. If a child was not eligible to participate in the study, the family was referred to other therapeutic services. (See Table 2 for child characteristics.)

Table 2.

Family Characteristics

Family Characteristics	M(SD) or %	Range
Child Characteristics
Gender
Male	85.7%
Age (years)	10.4(1.5)	8–13
IQ
Verbal	100.4(15.2)	80–128
Nonverbal	101.6(11.2)	74–119
Full scale	98.8(16.1)	73–131
Ethnicity
Caucasian	100%
Mother Characteristics
Age (years)	44.8 (3.7)	40–51
Education
High school graduate	6.3%
Partial college	50.0%
College graduate	18.8%
Some graduate training/terminal masters	6.3%
Professional degree	6.3%

Setting

All group treatment sessions occurred at the IWK Health Centre in Halifax, Nova Scotia. The IWK serves as the tertiary pediatric health center for the Maritime region of Canada. Virtually all children with ASD in metropolitan Halifax and the surrounding area (population 370,000) are diagnosed at this center. All the trainings, recruitment, qualifying assessments, and treatment occurred on site at the IWK Health Centre, and the sample of participants was generally representative of the Halifax community.

Procedure

This study was completed in compliance with the Colorado Multiple Institutional Review Board (University of Colorado Anschutz Medical Campus) and the Research Ethics Board of the IWK Health Centre.

Facing Your Fears Intervention

The FYF program is delivered in multi-family groups over 14 weekly sessions, each lasting 1½ hours. Large group time, separate parent and child group meetings, and parent-child dyads comprise the weekly sessions. Three clinicians were assigned to each group cohort to provide coverage when the parent and child groups met simultaneously. Three treatment manuals were used: a parent workbook, a child workbook, and a facilitator manual (Reaven et al., 2011). (The FYF workbooks were not published until after participant feedback was incorporated into revised manuals).

Core CBT components for the treatment of childhood anxiety disorders are included in FYF (e.g., graded exposure, relaxation, deep breathing, emotion regulation strategies, and use of cognitive self-control; Kendall, 1994; Silverman et al., 2008). Given the social, emotional, communicative, and cognitive challenges of children with ASD, enhancements to the delivery of the core concepts were made, including the provision of written worksheets and multiple choice lists; an emphasis on creative and varied outlets for expression; and a focus on strengths, talents, and expanding areas of interest. Video modeling was included to support generalization of skills across settings. Because parent participation is integral to this intervention, instructions and activities for parents are interwoven throughout each group session. An in-depth description of FYF including a detailed parent curriculum is available elsewhere; see Reaven et al., 2011.

Aim 1–Training Clinicians and Assessing Treatment Fidelity

Eligible clinicians were identified and asked to participate in the study. Consenting clinicians participated in a 2½ day on-site training workshop delivered by two of the study authors (JR; ABS). Clinicians completed an assessment of CBT knowledge both pre- and post-workshop and then facilitated the 14-week FYF intervention program within 2 months of the workshops. Four treatment groups were completed over the course of the study. Clinicians received twice monthly phone consultation from the treatment developers (JR; ABS) throughout the duration of the intervention. All treatment sessions were videotaped and adherence to the treatment was coded by the FYF developers.

Training Workshop

All participating clinicians were required to attend the training workshop. The workshop was comprised of a series of didactic presentations, small group activities, videotaped examples, role-playing exercises, and session-by-session review of the FYF intervention. Several clinicians joined the project mid-way through the study (after two treatment groups had been completed). They were asked to view a videotape of the original workshop and attend a day-long teleconference reviewing the critical elements of FYF. All clinicians completed a workshop evaluation questionnaire after participating in the training.

Phone Consultation

Bi-monthly, hour-long phone consultation was provided by two of the authors (JR; ABS) during each cohort for the duration of the study. All FYF group sessions were digitally video-recorded and sent as encrypted files to the research site in Colorado. Sessions were viewed prior to phone consultation. The consultation format was consistent across the project and included the following components: 1) question-and-answer period regarding the FYF sessions that occurred since the last phone consultation; 2) feedback regarding strengths of facilitation, missing elements, and suggestions for delivery of session content; and 3) plan for the upcoming FYF sessions.

Aim 2 – Examining Feasibility

Acceptability measures were collected from all participants (e.g., parents, children and clinicians) following each session. Likert ratings were completed for each core activity, and participants were also asked to provide narrative comments about the helpfulness (or lack of helpfulness) of each activity.

Participant attendance was also recorded at each session.

Secondary Aim – Examining Preliminary Youth Anxiety Treatment Outcome

Interested families contacted the IWK Centre and a research assistant explained the goals and procedures of the study. If the parent was interested, a research assistant conducted a brief telephone screen for eligibility. If the child was a potentially eligible participant, the family was invited to the clinic, where they completed the ASD diagnostic measures, standardized cognitive testing (if unavailable either through previous research visits or clinical testing), and the SCARED. If a child met diagnostic criteria for an ASD, cognitive criteria and obtained scores above the clinical significance cutoff on SEP, SOC, and/or GAD subscales of the SCARED, the parent was invited to complete the Anxiety Disorders Interview Schedule for Children-Parent version (ADIS-P; Silverman & Albano, 1996). Parents completed the ADIS-P with a trained clinician (‘clinician examiner’ (CE), a PhD-level psychologist), who provided clinical severity ratings (CSR) across a range of psychiatric disorders of childhood. The CE reviewed the parent’s report of the child’s psychiatric symptoms and determined which Axis I diagnosis was ‘primary’ (i.e., yields the highest CSR on the ADIS-P) and which diagnoses were ‘secondary.’

Parents were asked to report the child’s medications and other treatments received prior to and throughout the study period. Children currently taking medications were eligible for the study; however, they were required to be on a stable dose of psychiatric medication (e.g., at least one month at the same dosage prior to the qualifying assessment), and maintain the same dosage of medications over the course of the study. Medication usage was tracked throughout the study. Within 2–6 weeks of the last treatment session, the ADIS-P was re-administered to the parent. The CE did not facilitate the FYF group treatment at any point, nor receive information about children’s progress in therapy.

Measures

Aim 1–Training Clinicians and Assessing Treatment Fidelity:

Clinician Measures

Demographic Questionnaire

A brief questionnaire was developed to track clinicians’ years of experience in the field of psychology, specific training with CBT, and experience working with children with ASD.

Assessment of CBT Knowledge

A 20-item multiple choice test similar to assessments used in other studies (Rees & Gillam, 2001) was developed to evaluate clinicians’ knowledge of CBT. Three similar but non-identical versions of the assessment were developed and were administered three times: 1) prior to; and 2) following the training workshop; and 3) following the completion of the first group cohort.

Treatment fidelity measure

A treatment fidelity measure was developed to assess the facilitators’ adherence to protocol. The measure utilized a checklist format to assess the presence or absence of core treatment components on a session-by-session basis. In addition, a global rating of quality was assigned at the end of every modality for every session (e.g., parent group, child group, parent/child dyads, and large group activity). Ratings were based on a Likert scale ranging from poor (1), to adequate (3), to excellent (5) quality. The quality rating was intended to be a global score encompassing therapeutic alliance, therapist expertise, and competence in the delivery of core treatment content for each modality (adapted from the Yale Adherence and Competence Scale Guidelines (YACSII); Nuro et al., 2005). All treatment sessions were video-recorded. Two of the authors (JR; ABS) viewed the first group cohort together and agreed on parameters for scoring absence or presence of core components, as well as for assigning quality ratings. For the remaining three cohorts, each rater viewed the sessions separately to rate the core components and quality of each treatment session. Because phone consultations occurred on a bi-monthly basis, adherence to protocol was not assessed as an independent entity, but rather as a part of an ongoing feedback loop (e.g., constructive comments from the consultants were provided to the clinicians, clinicians conducted a group session, consultants viewed the session and again provided feedback, etc.).

Aim 2–Examining Feasibility

All Participants

Acceptability measures

All participants provided weekly written feedback regarding the FYF treatment, including ease of administration (clinicians only), content relevance, and helpfulness of the activities. A 1–5 Likert Scale was used to rate each item (children used a 5-point pictorial scale to rate the helpfulness of each session’s activities). Participants also provided written comments regarding the most and least helpful aspects of each session. These written comments were reviewed and grouped thematically to aid interpretation. Acceptability data were collected to inform both the training workshop as well as the FYF intervention.

Group attendance and treatment completion data were obtained.

Secondary Aim: Examining Preliminary Youth Treatment Outcome

Children–Eligibility Measures

Diagnosis of ASD

Diagnostic status was determined by expert clinical review of the ADOS (Lord et al., 1999) and other clinical information documented using a symptom checklist based on the DSM-IV TR (American Psychiatric Association, 2000). The ADOS is a semi-structured, play-based direct assessment of social and communicative behaviors indicative of ASD. It is considered the “gold standard” for diagnosing autism spectrum disorders (Lord, Rutter, & Le Couteur, 1994). Clinicians administering the ADOS were trained to 80% reliability.

Screening for cognitive functioning and reading abilities

All potential participants lacking standardized cognitive assessment within the past 2 years were administered the Wechsler Abbreviated Scales of Intelligence (WASI; Weschler, 2002) and the reading subtests from the Woodcock-Johnson Tests of Achievement (WJ-III; Woodcock, McGrew, & Mather, 2001).

Screening for anxiety symptoms

The SCARED—Parent version (Birmaher et al., 1999) is a 41-item inventory comprised of five anxiety subscales (Panic, GAD, SEP, SOC and School Anxiety) and a total score. Parents reported on symptoms over the past month. The SCARED demonstrates excellent psychometric properties in typically developing youth populations (Birmaher et al., 1997, 1999; Hale, Crocetti, Raaijmakers, & Meeus, 2011). In youth with ASD, the extended 71-item version of the SCARED demonstrates strong internal consistency (α > 0.9) and moderate convergent validity with the ADIS (Van Steensel, Deutschman, & Bögels, 2012). Results from another study confirm the 41-item measure’s five-factor structure and suggest good sensitivity (.71) and specificity (.67) among parents of youth with ASD (Gadgil, Stern, Blakeley-Smith, Reaven, & Hepburn, under review).

Children–Outcome Measures

Anxiety Disorders Interview Schedule for Children–Parent Version (ADIS-P; Silverman & Albano, 1996)

The ADIS-P was the primary outcome measure and was administered pre- and post-intervention. While the ADIS-P and C (parent and child versions) are both typically administered for general pediatric populations (Walkup et al, 2008), the additive value of the ADIS-C in the ASD population has been questioned. Youth with ASD tend to under-report anxiety symptoms and, as such, clinicians tend to rely more upon parent report of symptoms on the ADIS than on child report (Storch et al., 2012). Thus, only the parent version was used in this study. The ADIS-P is a semi-structured psychiatric interview that assesses the presence of anxiety disorders as well as other co-occurring mental health conditions. The ADIS demonstrates strong psychometric properties, including good test-retest reliability for diagnoses and symptom scales (Silverman et al., 2001), as well as convergent validity with other anxiety measures (Lyneham & Rapee, 2005; Wood, Piacentini, Bergman, McCracken, & Barrios, 2002). The CE conducted the ADIS-P with the parent pre- and post-intervention, determined diagnostic classifications across DSM-IV categories, and provided summary codes of severity and interference, called ‘Clinician Severity Ratings’ (CSRs: 0=no symptoms, 8=very severe impairment).

ADIS-P administration training (of IS by ABS) involved scoring above 80% reliability on clinical diagnoses and CSRs on three videotaped administrations of the ADIS-P and three live administrations (serving as the interviewer). The Clinical Global Impressions Scale–Improvement ratings (CGIS-I; National Institute of Mental Health, 1970) were compiled by the CE after the participants had completed all study activities. These ratings were derived in a manner similar to the methods described in previous studies (Walkup et al., 2008; Wood et al., 2009). The CE reviewed the ADIS-P and the SCARED data obtained pre- and post-treatment, and assigned a rating of 1–7 concerning the overall impression of improvement (1=‘very much improved’; 4=‘no change’; 7=‘very much worse’). Children who obtained a CGIS-I of 1 or 2 were considered to be positive treatment responders. Reliability data on these diagnoses and CGIS-I ratings were collected from the University of Colorado.

Analysis Plan

Aim 1

The effectiveness of the clinician training and their subsequent ability to deliver the intervention was assessed in several ways. First, descriptive statistics were used to assess participant satisfaction with the training. Second, we compared the trainees’ knowledge of CBT before and after training with paired-sample t-tests, and again with a follow-up test several months later to check for loss of knowledge. Finally, fidelity of implementation was assessed in three ways: 1) by calculating the percentage of the core treatment components that were adhered to across all sessions, 2) calculating the intraclass correlation (ICC) of each clinician with gold standard assessments (made by ABS and JR), and 3) calculating descriptive statistics of the global quality rating for each treatment modality.

Aim 2

Feasibility of the FYF program was assessed by calculating descriptive statistics of the acceptability ratings from parents, children and clinicians, as well as by reviewing comments by the same individuals. We also calculated attendance rates from all participants.

Secondary aim

Treatment outcomes were not the primary focus of this study but are nonetheless important, and were therefore assessed. We first characterized the sample of participants by determining their demographics, medication status, complexity of their condition, and their primary anxiety diagnosis. The participants’ anxiety was assessed prior to treatment and after completion of the FYF program and these measurements were the basis for determining their outcomes. Paired-sample t-tests were calculated to compare their baseline anxiety with their post-treatment anxiety. Note that while multiple comparisons were calculated, we did not adjust alpha for inflation of Type I error. Given that this study was primarily about feasibility rather than outcomes and we are not trying to make strong claims about the effectiveness of the FYF program, all alphas were set at .05 (O’Keefe, 2003; Perneger, 1998; Tutzauer, 2003). As such, some caution is warranted while interpreting these findings.

Results

Aim 1–Training Clinicians and Assessing Treatment Fidelity

Clinician Outcomes

Evaluation of Training Workshop

Clinicians responded to a series of statements (e.g., ‘I would recommend this training to a colleague’) using a 6-point scale (1=‘strongly disagree,’ 6=‘strongly agree’). Overall, clinicians expressed high satisfaction with the training provided (M=5.34, SD=0.17, range: 5.18–5.55).

Assessment of CBT Knowledge

Clinicians demonstrated significant improvements on the assessment of CBT knowledge after attending the training. Results of a paired-samples t-test indicated significant improvements in clinicians’ CBT knowledge from pre-training (M=81% correct, SD=7%, range: 70%-90%) to post-training (M=90% correct, SD=7%, range: 75%-95%), t(9)=2.41, p=.039. These improvements were maintained at follow-up, after clinicians had facilitated treatment groups (M=92% correct, SD=4%, range: 85%-95%), with no significant difference between post and follow-up scores, t(4)=0.74, p=.501.

Treatment Fidelity

Treatment fidelity was calculated based on the percentage of core components that the facilitators implemented across the entire 14-week program. Treatment fidelity percentages ranged from 87% to 95% (M=92%) across all four group cohorts (Group 1–94%; Group 2–95%; Group 3–87% and Group 4–92%). All groups exceeded the minimum standard for acceptable treatment fidelity (80%). Inter-rater reliability for treatment fidelity in the last three group cohorts was excellent, ranging from 93%-96% (ICC=.99, p=.005).

In addition to collecting data regarding absence/presence of core components, global ratings of quality for each modality were obtained. Average quality ratings (Likert 1–5) for each modality were as follows: Large group (4.75), Parent group (4.60), Child Group (4.54), and Parent/Child Dyadic work (4.55). Ratings were similar across modalities and across all group cohorts, indicating that the clinicians delivered the FYF intervention with consistently high quality. Inter-rater reliability of quality was defined as agreement within one point on a 5-point Likert Scale. Percent agreement between the two independent raters across the last three cohorts was excellent and ranged from 88% to 100%.

Aim 2 - Feasibility

Acceptability of the FYF Intervention

Acceptability was high across all four treatment groups (M=4.15, SD=.50), although ratings dipped slightly for Group 2 (M=3.87, SD=.64). Acceptability ratings were highest among parents (M=4.35, SD=.33) and clinicians who facilitated the parent group (M=4.33, SD=.36). Positive ratings were also given by clinicians who facilitated the child group (M=4.08, SD=.37) and by the children themselves (M=3.84, SD=.66) (see Table 3 for session by session acceptability ratings).

Table 3.

Average Acceptability Ratings from Children, Parents, and Clinicians for Each FYF Session

	Average Acceptability Ratings*
Session	Child (N=2–4)	Parent (N=2–6)	Clinician (N=3–4)	Overall (N=7–14)
1. Welcome to Group: Words We Use for Worry	3.74	4.33	4.48	4.26
2. When I Worry	3.70	4.08	4.01	3.95
3. Time Spent Worrying	3.87	3.94	4.04	3.97
4. What Worry Does to My Body: Beginning to Measure Worry	3.48	4.29	4.21	4.04
5. The Mind-Body Connection	3.98	4.42	4.11	4.15
6. More Mind-Body Connections: Introduction to Exposure	3.64	4.35	4.07	4.03
7. Introduction to Exposure (Continued)	3.59	4.21	4.14	4.00
8. Practicing Exposure and Making Movies	3.94	4.49	4.24	4.23
9. Facing Fears and Making Movies	3.75	4.54	4.32	4.23
10. Facing Fears and Making Movies	4.00	4.55	4.40	4.33
11. Facing Fears and Making Movies	4.29	4.51	4.37	4.38
12. Facing Fears and Making Movies	3.73	4.20	4.09	4.03
13. Facing Fears and Making Movies	3.78	4.37	4.27	4.16
14. Graduation	4.28	4.66	4.20	4.35
All Sessions	3.84	4.35	4.21	4.15

^*Ns indicate the range of the number of participants who completed acceptability forms at the end of each session. For most sessions, acceptability forms were completed by 4 children, 5 parents, and 3 clinicians.

Narrative comments were reviewed and incorporated into an updated version of the treatment manual. Revisions occurred at several points during the study, with the majority of revisions occurring after the completion of the first two cohorts. It should be noted that the revisions did not entail substantial changes in core content, but rather were primarily changes in delivery of the material. For example, based on feedback from clinicians, instructions were modified to clarify certain activities; additional worksheets and ‘cheat sheets’ were generated to augment existing materials; videos and worksheets were updated to be more developmentally appropriate; instructional caveats were included in the facilitator manual to accommodate specific child needs; the word ‘autism’ was removed from the title of the parent and child manuals to minimize potential stress around this label; and greater flexibility was integrated into the treatment schedule so that sessions felt less ‘rushed.’

Positive comments that parents, children, and clinicians made were also documented. For example, parents responded with the following comments in response when asked what was most helpful: “seeing the anxieties that he recognizes in himself”; “reminders that talking about worries helps”; “doing the actual fear facing and seeing him go through it till he was calm”; and “great course, skill building with lots of everyday uses…was very glad to take it, my child definitely learned some skills”. Child participants made positive comments as well and these included: “I liked how a peer expressed an opinion that was different from mine but he did it respectfully”; “I really liked the worry bug activity, it was funny!”; and “very fun! I faced my fear and finished the movie.” Parents tended to find exploration and discussion-related activities most helpful, while children found the video activities to be most helpful; clinicians reported that discussion and skill-building activities were most helpful.

Attendance

Sixteen children (and their parents) participated in the FYF program. One parent-child pair dropped out early in the treatment process, citing the lack of fit between the child’s needs and the treatment program. Completion rate was 94% and no child missed more than 3 sessions. When a family was absent, they were asked to arrive early the following week to review missed content.

Secondary Aim: Examining Preliminary Youth Treatment Outcome

Medications

The majority of child participants were taking medications prior to the intervention (69%; 11/16 participants). Families were asked to report any changes in medications during the study. Of the 11 participants, 8 made no changes in their medications. Of the three participants who made changes, one participant was taken off medication altogether, a second experienced a decrease in dosage, and a third added a new medication for ADHD. Data were included for these three participants in the final analysis because none of the changes involved an increase in medication targeting anxiety symptoms.

Psychiatric complexity of child participants

Results from the ADIS-P administered pre-intervention indicated that the number of diagnoses (in addition to ASD) ranged from 2–7 (M=4.0; SD=1.15). Overall, 15 children (93.7%) met criteria for more than one anxiety diagnosis, 12 met criteria for Attention Deficit Hyperactivity Disorder (ADHD) (75.0%), and 1 (6.3%) met criteria for Oppositional Defiant Disorder (ODD). None met criteria for a co-occurring mood disorder (dysthymia/major depression). The sample appears to be comparable to other treatment studies of anxiety in ASD (Storch et al., 2013). Data collected at baseline indicated that 6 children (40%) were receiving some form of treatment for anxiety prior to entering the study.

A primary anxiety diagnosis was determined for each of the participants by examining the diagnosis with the highest clinician-assigned Clinical Severity Rating (CSR). Generalized Anxiety Disorder was the most common primary diagnosis (56.3%), followed by Social Anxiety Disorder (25.0%) and Separation Anxiety Disorder (18.8%). When two diagnoses had the same CSR, primary diagnosis was determined by having the parent designate the most concerning set of symptoms.

Treatment Outcome

Intent to treat analyses were conducted for two outcome variables (n=16): CSRs generated for the three principal anxiety diagnoses and diagnostic status of the principal anxiety diagnoses per ADIS-P criteria. One participant did not complete treatment, and the last observation was carried forward for these analyses.

Clinician Severity Ratings (CSRs)

Paired sample t-tests were used to examine pre-post differences in anxiety severity and interference. Significant decreases in severity were found for the three primary anxiety diagnoses post-intervention: SOC, t(15)=2.43, p=.03, d=.61; SEP, t(15)=2.13, p=.05, d=.53; and GAD, t(15)=3.03, p=.008, d=.76.

Diagnostic Status of Anxiety Disorders

Paired sample t-tests were used to assess differences in the number of anxiety disorders post-intervention. Child participants demonstrated a significant reduction in overall number of anxiety disorders, t(15)=2.78, p=.014, d=.70, post-intervention. Forty percent of participants lost their principal anxiety diagnosis after the completion of the intervention program.

CGIS-S Improvement Ratings

Clinical Global Impressions Scale – Improvement scores of 1 or 2 reflected ‘a clinically meaningful improvement in anxiety severity,’ consistent with previous studies (Reaven et al., 2012; Wood et al., 2009). Of the 15 youth that completed the study, over half (8/15) met criteria for a clinically meaningful outcome (53.3%), while 20% (3/15) somewhat improved, and 26.7 % (4/15) experienced no change in severity of symptoms, post-intervention. None of the children experienced a worsening of symptoms.

Discussion

The main purpose of the study was to train clinicians to deliver FYF to fidelity and to assess the feasibility of the intervention for clinical practice. A secondary purpose of the study was to examine preliminary youth anxiety treatment outcome. Clinicians demonstrated significant gains in CBT knowledge following a training workshop and these gains were maintained after completion of the first treatment cohort. Examination of adherence to protocol indicated that clinicians did deliver the program as intended and with high quality across all four group cohorts.

Group attendance and treatment completion rates were high. In addition, acceptability was high across all participant groups and group cohorts. In the two years following the end of the research project, the study authors remained in contact with the clinician facilitators. The group facilitators were asked if they were continuing to deliver FYF and if so, how many youth had participated following the conclusion of the study. The facilitators stated that they had continued to deliver FYF, and had treated an additional 26 youth with ASD and anxiety. These results may reflect the potential sustainability of FYF and suggest that ‘knowledge transfer’ may have occurred (Beidas et al., 2011). Treatment fidelity was not assessed for these youth, so the extent to which they delivered FYF to fidelity is unknown, thus representing an area for further research.

Preliminary youth anxiety outcome was examined via (a) change in anxiety severity and interference related to principal anxiety diagnoses, (b) change in diagnostic status of principal diagnoses; and (c) global ratings of improvement in anxiety. Youth participants evidenced significant reductions in anxiety symptoms after participating in the FYF intervention. Significant decreases in severity were apparent for all three primary anxiety diagnoses (GAD, SOC, and SEP). Further, youth participants had fewer anxiety diagnoses after completing the FYF program, and 40% of participants no longer met diagnostic criteria for their primary anxiety diagnosis post-treatment. Finally, over half of youth participants met criteria for a clinically meaningful improvement in anxiety symptoms following the intervention and 20% demonstrated modest improvements. These preliminary results are highly consistent with the results obtained from previous treatment trials (Reaven et al., 2012).

Overall, these results may signal the positive potential for dissemination and implementation of the FYF treatment for youth with ASD and anxiety in new clinical settings. The combination of an interactive training workshop paired with ongoing phone consultation was sufficient for clinicians to demonstrate excellent quality and adherence to the treatment protocol. Further, the high acceptability ratings, excellent attendance, and low drop-out rate, in addition to the positive, although preliminary, youth outcomes, suggest that the FYF program may be a feasible and effective treatment for reducing anxiety symptoms in youth with ASD and anxiety in clinical practice settings.

Perhaps the ease with which clinicians in a pediatric hospital setting who were unfamiliar with FYF were able to learn the intervention and deliver it to fidelity may be due in part, to the user-friendly qualities of the treatment manual (Reaven et al., 2011). For example, detailed descriptions of specific activities, video examples of core concepts, a ‘helpful hints’ section highlighting potential problems and solutions, and in-session worksheets likely strengthen the usability of the manual (Najavits, Weiss, Shaw & Dierberger, 2000). Obtaining specific feedback from the group facilitators as well as youth and family participants regarding the treatment program, and incorporating this feedback directly into a revised manual, may further enhance the accessibility of the FYF program for providers in clinical practice (Beidas et al., 2011). The integration of acceptability data into program development reflects a bidirectional approach to treatment implementation and dissemination, in which critical insights from practitioners and participants can support the efficacy and long-term sustainability of new programs (Hatgis et al., 2001). This may be a critical step in bridging the research-to-practice gap (Dingfelder & Mandell, 2011).

Limitations and Future Directions

The small sample and the quasi-experimental design are notable limitations. The extent to which the improved outcomes were a placebo effect related to involvement in FYF is unknown and speaks to the importance of conducting more rigorously designed studies. Further, the sample was primarily Caucasian; although the racial backgrounds of the sample were generally representative of the geographic area, this does limit the generalizability of the findings to other populations. Additionally, because the primary anxiety diagnoses targeted by FYF are SEP, SOC and GAD, the extent to which the results would generalize to youth presenting with other primary diagnoses such as Obsessive-Compulsive Disorder is unknown.

The assessment of anxiety symptoms in this sample was determined through the SCARED (Birmaher et al., 1999) and then through the ADIS-P (Silverman & Albano, 1996). It is important to note that neither instrument was specifically designed for youth with ASD. Although there is support for the co-occurrence of anxiety disorders in youth with ASD, agreement about how to diagnose anxiety symptoms in individuals with ASD must occur. The psychometric validation of anxiety measures originally developed for the typical population for individuals with ASD is an important next step (Kerns & Kendall, 2012).

Clinician participants varied in their years of experience with CBT but were all working in a pediatric hospital setting for youth with ASD (although several clinicians were graduate students enrolled in a university PhD program). The extent to which clinicians new to ASD (e.g., community mental health center clinicians) would obtain similar fidelity ratings is unknown and represents a direction for future research. Furthermore, the FYF program requires a minimum of three clinician facilitators to deliver the group treatment. Although there are a number of advantages to group therapies (e.g., management of long waiting lists, normalization of experiences and decreased isolation for participants; Reaven et al., 2009), not all clinical settings have the capacity to provide three facilitators to run groups, representing a potential limitation in the feasibility of offering group therapy in some environments. In addition, although all group cohorts met the threshold for excellent treatment fidelity, the extent to which good treatment fidelity is necessary for positive treatment outcome is unknown, and could also represent an additional area for further research.

Finally, future research may also include the systematic comparison of clinician training methods for evidence-based interventions for youth with ASD. For example, the use of modular trainings rather than a session-by-session approach may better equip some clinicians. Similarly, the extent to which ongoing consultation for clinicians inexperienced with a particular treatment program is necessary to achieve positive treatment outcomes for a new intervention needs to be determined. The results of additional research may provide much-needed guidelines for training of community practitioners stationed at the front lines of autism treatment and may enhance dissemination of treatment programs for youth with ASD.