Achieving Independence and Mastery in School: An Open Trial in the Outpatient Setting

Abstract

Youth with autism spectrum disorders (ASD) without intellectual disability frequently experience academic problems, in part due to executive functioning (EF) deficits. There are currently no evidence-based interventions targeting academic EF skills (e.g., organization, prioritization, etc.) for middle school youth with ASD. The need is critical given increasing demands on these skills during the transition from elementary to middle school. An intervention targeting academic EF skills was recently developed. This paper reports on an open trial of the intervention with 21 middle-schoolers with ASD. Results suggest high feasibility/satisfaction, and improved EF skills, particularly in the domain of organization and materials management. These promising results support further intervention development work and suggest that academic EF skills are malleable in youth with ASD.

Students who have autism spectrum disorder (ASD) without an intellectual disability (ID; i.e., IQ>70) often struggle academically (Keen, Webster, & Ridley, 2016) or perform below grade level relative to their IQ (Ashburner, Ziviani, & Rodger, 2010). In fact, during middle school years, the academic performance of youth with ASD without ID is approximately two to three years below their typical peers (Wagner et al., 2003). While some of these academic struggles stem from key features of ASD (e.g., social-communication deficits, narrowly defined interests, concrete/literal thinking), they are also strongly linked to deficits in higher order executive functions (EF) such as organization, time management, prioritization, and task initiation (DePaoli et al., 2015; Pennington & Ozonoff, 1996; Schall, Wehman, & McDonough, 2012; Troyb, Rosenthal, et al., 2014). As students must routinely be able to initiate tasks, perform multistep sequences of events, reflect, reason, plan and prioritize (e.g., complete different tasks for several subjects on time), sustain performance and complete tasks, be flexible in their thinking (e.g., select the learning strategy appropriate for each context), and monitor their performance (e.g., manage progress and check for mistakes (Best, Miller, & Jones, 2009; Best, Miller, & Naglieri, 2011; Endedijk, Denessen, & Hendriks, 2011; Fisher & Happe, 2005), it is not surprising that EF skills are strongly associated with academic success. However, 35–70% of youth with ASD without an ID present with EF deficits (Blijd-Hoogewys, Bezemer, & van Geert, 2014; Gioia, Isquith, Kenworthy, & Barton, 2002; Kenworthy et al., 2005; Pennington & Ozonoff, 1996). Specifically, particular difficulties in cognitive flexibility have been reported on rating scales (Blijd-Hoogewys et al., 2014; Gioia et al., 2002) and direct assessments (Pennington & Ozonoff, 1996). Deficits in organization (Kenworthy et al., 2005; Pennington & Ozonoff, 1996), time management, prioritization, initiation (Pennington & Ozonoff, 1996), multi-tasking (Hill & Bird, 2006), and planning (van den Bergh, Scheeren, Begeer, Koot, & Geurts, 2014) are also prominent (Best et al., 2011). As a result, youth with ASD and their parents may struggle to acquire and manage critical academic behaviors (e.g., material organization, tracking assignments, prioritizing tasks, getting started on tasks, managing distractions, homework completion, effectively studying, and breaking down large assignments) and experience significant homework issues (e.g., misunderstanding assignments, planning for studying). Not surprisingly, EF deficits are clear predictors of poor academic performance (Best et al., 2009; Best et al., 2011; Fisher & Happe, 2005) and poor outcomes in ASD (Adreon & Stella, 2001; Clark, Pritchard, & Woodward, 2010; Wallace et al., 2016; Wolf et al., 2009). For example, changes in routines, scheduled events or activities, the physical structure of the classroom, and substitute teachers can all have a profound effect on the student with ASD due to difficulties in shifting attention and lack of flexibility (Adreon & Stella, 2001). Specific academic challenges in ASD include problems with writing (including organizing content), reading comprehension (understanding how individual details contribute to a greater lesson, challenges with perspective taking), and math problem solving (Jones et al., 2009; Keen et al., 2016; Troyb, Orinstein, et al., 2014; Wolf et al., 2009). Other common challenges include difficulties managing distractions, planning for studying, multi-tasking, poor note-taking, faulty integration-synthesis, and prioritization (Wolf et al., 2009).

Clearly, interventions targeting academic EF skills, including planning, organization, time management, and study skills for youth with ASD are needed (Buescher, Cidav, Knapp, & Mandell, 2014). However, according to the National Research Council and National Autism Center there are currently no evidence-based interventions targeting academic EF skills for middle school youth with ASD (Whitby, 2013). There are existing evidence-based interventions targeting academic EF skills for other clinical populations (e.g., Attention-Deficit/Hyperactivity Disorder, ADHD) (Abikoff & Gallagher, 2008; Abikoff et al., 2013; Ciesielski et al., 2015; Evans, Axelrod, & Langberg, 2004; Evans, Langberg, Raggi, Allen, & Buvinger, 2005; Langberg et al., 2011; Tamm et al., 2015; Vaughn, Tamm, Loren, Ciesielski, & Cyran, 2014). These interventions emerged from growing recognition that many of the academic problems exhibited by children with ADHD may represent behavioral manifestations of poor EF (Pennington & Ozonoff, 1996), including problems with temporal and materials organization (i.e., often has difficulty organizing tasks and activities, often loses things, is often forgetful, and often fails to finish school-work, chores, or duties; Langberg et al., 2011). For example, deficits in arousal, inhibitory control, delay tolerance, working memory, and time perception likely impede self-regulatory behaviors and interfere with organization and planning (Abikoff et al., 2013). Such deficits are manifested as forgetting to complete or losing homework assignments, difficulties planning for the completion of long-term projects and studying for tests, and problems keeping class materials organized (Langberg et al., 2011). Although these interventions differ from one another somewhat in terms of target, setting, and developmental level, they universally report success in improving academic EF (e.g., planning and organization), homework behaviors, and academic performance in youth with ADHD (Bikic, Reichow, McCauley, Ibrahim, & Sukhodolsky, 2017). Notably, these interventions also provide parents with key skills to more effectively support their child while building independence (Ciesielski et al., 2015), which is critical for youth with ASD whose parents are sometimes pejoratively described as “helicopter” parents (Korbel, McGuire, Banerjee, & Saunders, 2011). These interventions could be ideal to address academic EFs in ASD as they are time-limited, amenable to group administration, and emphasize the crucial role of the parent who confronts the daily academic and behavioral struggles that often come with rearing a student with ASD (Bearss et al., 2013). Critically, these interventions target many of the same EF deficits evidenced in ASD (Courchesne & Pierce, 2005; Kenworthy et al., 2005). However, core characteristics of ASD, some of which may stem from EF deficits (Pennington & Ozonoff, 1996), include social-communication deficits (American Psychological Association, 2013), concrete and literal thinking styles, anxiety (White et al., 2010), over-selective attention, missing subtle cues, difficulty grasping the big picture (Happe & Frith, 2006; Ozonoff & Miller, 1996), overreliance on external support systems (Benson, Karlof, & Siperstein, 2008), and amotivation and avoidance (Carnahan, Hume, Clarke, & Borders, 2009), may contribute to difficulty engaging in and benefiting from typical skill-based interventions. Thus, adaptation is needed.

Achieving Independence and Mastery in School– Outpatient (AIMS-O)¹ is based on the Homework, Planning, and Organization (HOPS) intervention used with middle school youth with attention-deficit/hyperactivity disorder (ADHD) (Langberg, Epstein, Urbanowicz, Simon, & Graham, 2008) and adapted for the outpatient setting (Ciesielski et al., 2015). Although much of the content is similar, with the exception of including content specific to executive functioning and ASD in the early sessions (see Table 1), modifications such as increasing structure and predictability in delivery of treatment components (e.g., use of a visual schedule for each session), incorporation of additional visual supports (e.g., story boards, animated videos), interactive activities to demonstrate concepts, use of technology, increasing parental involvement, providing immediate feedback, directive teaching, modeling, and other evidence-based teaching approaches for ASD (Fleury et al., 2014) have been incorporated to accommodate ASD characteristics and maximize benefit from the intervention. Additionally, the AIMS-O adaptation involves more break-out sessions with the teens than the ADHD interventions. AIMS-O specifically aims to improve academic EF by directly teaching students and their parents strategies to improve organization, planning, time-management, problem solving, studying, etc., using principles of behavioral reinforcement (e.g., behavior agreements) to promote use of the strategies outside of group sessions. Treatment development steps for AIMS-O to date include 1) an initial uncontrolled trial of an early version of the intervention with three youth with ASD as a “proof of concept” and 2) separate focus groups with parents and youth with ASD to inform the tailoring of the intervention for youth with ASD and their parents. These initial steps documented that parents and teens identify academic EFs as an important treatment target and that the AIMS-O intervention is feasible to administer and may improve academic EF (Tamm, Duncan, Vaughn, McDade, Estell, Birnschein, Crosby, L., 2019; Tamm & Duncan, 2020). However, the “proof of concept” trial included only three families and some of the AIMS-O materials (e.g., animated videos, in session activities promoting content uptake) were not yet developed.

Table 1.

Comparison of sessions from the Academic Success Groups for ADHD (left) and the AIMS-O Adaptation (right)

Session	Academic Success for ADHD	AIMS-O
1	Provide overview of ADHD, including related EF deficits and treatment rationale.	Provide overview of common symptoms, academic strengths and deficits in youth with ASD. Review EF using a video. Provide orientation to AIMS-O treatment rationale and goals.
2	Review behavioral management, behavioral contracting and reinforcement systems. Coach on how to implement, monitor, and reward use of skills at home.	Teach 5-step problem solving model using a video and color-coded visual map. Practice identifying, defining, and prioritizing problems, and practice brainstorming.
3	Develop individualized behavioral contract targeting improving academic/ organizational/ homework management skills.	Discuss using a behavior agreement to target a problem behavior. Practice developing a behavior agreement and negotiating the terms. Determine appropriate and motivating rewards.
4	Review binder organization skills including having one binder for all courses, a specific location for homework, and no loose papers allowed. Discuss how to use planner for tasks such as documenting homework assignments	Discuss steps to effectively complete homework assignments and components of an effective homework system. Provide an overview of time management and organization strategies including a visual schedule and binder organization system.
5	Teach flash card skills including summarizing key concepts in three to five words.	Discuss specific, measurable study skills/strategies, including acronyms and acrostics and study cards. Practice making study cards.
6	Discuss summarizing skills including learning how to use storyboarding techniques. Teach acronyms and acrostics.	Discuss summarizing skills including chunking, learning how to use storyboarding techniques, and plot diagrams. Practice summarizing.
7	Review skills and focus on planning and problem-solving for future adherence.	Review skills and focus on planning and problem-solving for future adherence. Provide materials to promote effective school communication.

Note. Academic Success for ADHD sessions reproduced from Ciesielski et al., 2015.

In the current study, the feasibility, satisfaction, and initial evidence of efficacy of the AIMS-O intervention for youth with ASD were investigated in an open trial with 21 youth with ASD without ID. Specifically, our research questions were:

1. Is AIMS-O feasible to deliver, as indexed by fidelity, attendance, and adherence data?

2. Is AIMS-O acceptable to parents and youth, as indexed by consumer satisfaction ratings?

3. Does AIMS-O improve academic executive functions, as rated by parents and teachers?

4. Are gains, if any, on academic EF competency maintained over time?

Method

Participants

Youth with ASD without ID attending middle school and their parents were recruited to participate in the study (n=28), which was approved by the Cincinnati Children’s Hospital Medical Center Institutional Review Board (IRB). Participants were recruited via letters, flyers, and emails targeted at individuals who had received services at a hospital outpatient clinic specializing in assessment and treatment of autism. Interested parents contacted the research staff via email or phone and underwent an initial phone screen confirming concerns regarding academic executive functions and ASD. Parents were mailed the Behavior Rating Inventory of Executive Function, Second Edition or BRIEF-2 (Gioia, Isquith, Guy, & Kenworthy, 2015) and a consent to release information form providing permission to contact participants’ teachers. Teachers were subsequently mailed an IRB-approved information letter, a copy of the release form, and asked to complete a BRIEF-2.

Of the 28 participants assessed for eligibility, 21 met eligibility criteria and enrolled in one of three consecutive AIMS-O groups (n=7 per group). Eligible participants had a T-score >65 on the Planning/Organization, Monitor, or Organization of Materials subscales of the BRIEF-2 completed by a parent or teacher. A diagnosis of ASD was confirmed using the Autism Diagnostic Observation Schedule, 2^nd Edition (ADOS-2) Module 3 (Lord et al., 2012) administered by a research reliable assessor (n=17) or a review of medical and educational records (n=4). An IQ ≥ 80 was confirmed using the Kaufman Brief Intelligence Test, Second Edition (KBIT-2) (Kaufman & Kaufman, 2004). Youth receiving pullout special education services for the majority of the day were excluded. Demographic characteristics are provided in Table 2.

Table 2.

Demographic Characteristics of Youth with ASD (n=21)

Age	M=12.2, SD=1.0
Sex	76.2% male
IQ	M=109.7, SD=13.8
Race
Non-Hispanic White	95.2%
Native American/Alaska Native	4.8%
Ethnicity
Hispanic/Latino(a)	4.8%
Grade
6th	33.3%
7th	33.3%
8th	33.3%
Percent taking psychiatric medication	76.2%
Psychiatric Comorbidities
ADHD	14.3%
Anxiety Disorder	4.8%
ADHD & Anxiety Disorder	57.1%
Family Income
<$50,000	10%
$50,001-$80,000	10%
$80,001-$100,000	35%
>$100,000	35%

Procedures

After providing informed consent, middle school youth and their parents participated in an in-person eligibility evaluation, including the ADOS-2, KBIT-2, and rating scales. Teachers of eligible youth were mailed an additional packet of rating scales to complete. Eligible families attended AIMS-O, which was delivered in seven weekly, 90-minute group sessions. Three consecutive groups, each with seven families, were offered in the 2018–2019 academic school year. Groups were typically attended by both parents and teens together, but on occasion, teens and parents attended separate groups (more detail provided below). The intervention was offered weekly in the early evening at an outpatient clinic within Cincinnati Children’s Hospital Medical Center. A licensed clinical psychologist was the lead interventionist, accompanied by a graduate level psychology practicum student or another clinical psychologist who co-taught the intervention and led the teen-only sessions. Parents and teens completed an acceptability of intervention questionnaire after each session. Parents and teachers completed ratings of academic and EF functioning at the initial evaluation and last session, and parents completed a brief additional rating three months after the intervention ended. Families were compensated for completing in-person evaluations before ($100) and after ($75) the intervention. Participants were not compensated for attending AIMS-O, but dinner was provided. Parents were compensated $10 for completing the online rating scale at the 3-month follow-up. Teachers were compensated $5 for completing the BRIEF-2 at screening, and $50 for completing rating scales before and after the intervention.

AIMS-O Intervention.

AIMS-O involves teaching academic EF skills (i.e., task initiation, organization, planning, prioritizing, and working memory) using evidence-based strategies for youth with ASD (e.g., behavioral reinforcement, behavior agreement, visual supports, video modeling, technology) to promote increased independence related to academics. Each session involved a review of a real-world practice assignment and a didactic component (direct instruction using PowerPoint, video clips, activities, etc.) illustrating key concepts followed by an in-session practice of the key concepts and strategies (e.g., parents and teens working together to create study cards) with coaching from a clinician. Teens were provided a visual schedule at the beginning of each session. Teens were assigned a real-world practice (i.e., homework) assignment each session that consisted of additional practice of key concepts and strategies at home or school in order to further build and generalize academic EF skills between sessions. Small rewards (e.g., snacks, candy) were provided to teens during each session for active participation and completion of real-world practice assignments. A summary of each session’s content is provided below (also see Table 1).

Session 1: Orientation to AIMS-O and Understanding Executive Functioning.

This session was attended by parents and teens separately. The goals were to orient them to the intervention and promote engagement and buy-in. In particular, participants were oriented to what academic EFs are and why they are important for school functioning, and how AIMS-O will target EF deficits. Parents were provided content regarding how EF deficits may affect their teens at home and school in order to normalize the need to address academic EF skills while also taking into account other factors that may affect school functioning (e.g., increased independence in middle school, social maturity, etc.). Teens were introduced to the content using games (e.g., a stacking game requiring cooperation and EF skills), self-reflection activities (e.g., completing a survey about EF skills), and fun videos depicting important content. Additionally, teens developed and organized a binder for storing their AIMS-O materials that also served as a model for a similar binder organization system to be used for their school materials. The real-world practice assignment was to identify an academic or school issue that they wanted to work on or improve.

Session 2: Problem Solving.

This session, attended by parents and teens together, introduced the ABCs for problem-solving model (A=Aim, B=Brainstorm, C=Choose, D=Do it, E=Evaluate). Parents and teens completed problem cards such that they identified and defined problems on note cards and then sorted the problems by prioritizing which were the most important to work on. They also participated in an activity in which they had to brainstorm and then choose a solution for a problem. An animated video was utilized to promote understanding of the problem solving model. The real-world practice assignment was to choose a problem that they were having related to school, brainstorm solutions, and then choose a solution after identifying pros and cons.

Session 3: Behavioral Agreements.

This session was attended by parents and teens together; however at the end of the session, parents and teens separated. The main purpose of this session was to introduce the concept of using a behavior agreement (e.g., rationale, purpose, and elements) as a strategy to promote the teens’ use of strategies to address academic EFs. An animated video was utilized to teach the steps of establishing a behavior agreement between a parent and a teen. In the separate session, parents were provided with a brief discussion on their role in promoting individuation during adolescence with balanced assistance and the general principles of reinforcement. In the separate teen session, teens engaged in an activity to identify appropriate and motivating rewards to be used in a behavior agreement, and also role-played with therapists how to have a behavior agreement discussion with their parent. As their real-world practice assignment, parents and teens were asked to identify a problem that they wanted to target on their behavior agreement and determine what reward would be earned. For example, if a teen’s problem was “does not know daily homework assignments,” their behavior agreement might state they have to “write down their daily homework assignments after looking them up in their online portal by 3:30pm” in order to receive the reward of staying up 10 minutes later. Future sessions all included a review of the behavior agreement, and once developed, how the behavior agreement was working, with coaching and modifications provided as needed.

Session 4: Homework System.

This session, attended by teens and parents together, addressed organization and time management skills. Parents and teens were encouraged to develop a homework system together (e.g., visual list of assignments to be completed, set location/environment, and organizational systems). The backpacks of the adolescents were used to assess their current organizational system and develop organizational skills. The appropriate use of a binder system was also reviewed. The real-world practice assignment was to choose a component of the homework system discussed and implement it at home (e.g., clean out their backpack weekly, write down their list of daily assignments).

Session 5: Study Skills.

This session, attended by parents and teens together, addressed the rationale for and components of specific, measurable study skills/strategies. Study cards (i.e., using less than seven words to define concepts) were introduced, including steps on how to create and how to systematically study using study cards (e.g., creating “know” and “study more” piles). Participants actively engaged in making study cards in session. Acronyms and acrostics were also introduced as aids to memorization. The real-world practice assignment was to create 3 study cards to be used for studying for an upcoming test or quiz.

Session 6: Summarizing.

This session was attended by parents and teens together. After a review of content from previous sessions including coaching and problem solving in regard to behavior agreements, homework systems, etc., as needed, summarizing strategies (e.g., detecting key points, storyboards, asking who/what/when/where questions, plot diagrams, etc.) were introduced. Participants actively engaged in summarizing activity in session. The real-world practice assignment was to create a summary to be used for studying for an upcoming test or quiz.

Session 7: Maintaining Skills and School Communication Tips.

This session was attended by parents and teens together. The clinician utilized this session to present any material not covered during the previous sessions, and to review content that was already covered. Games (e.g., Jeopardy) were utilized to review AIMS-O content. Parents were provided tips for maintaining and building academic EFs as well as strategies for enhancing communication with teacher/school. For example, parents were encouraged to describe their child’s strengths and difficulties and discuss strategies that may be effective for targeting academic goals with educators.

Measures

Measures of Feasibility

Fidelity.

Clinicians audio- and video-taped each weekly AIMS-O session for the final cohort; the first two cohorts were not coded for fidelity as the manual content was still being finalized. Research assistants were trained to code the sessions using a standard dichotomously coded treatment fidelity checklist for each session. Training involved review of the manual and providing definitions for items on the fidelity checklist. Research assistants then coded a session and their work was checked by the trainer; when 100% agreement was reached, they were allowed to code additional sessions independently. Twenty percent of independently coded sessions were randomly selected for an inter-rater reliability probe. The checklists for these probes were compared directly and the number of agreements were tallied. Inter-rater reliability was 100%.

Attendance.

Attendance for each AIMS-O session was recorded from sign-in sheets completed at the beginning of each session.

Real-World Practice Completion.

From session two on, at the beginning of each AIMS-O session, parents and teens reported on their compliance with the real-world practice assignments (i.e., AIMS-O homework).

Measure of Acceptability

Acceptability of Intervention Questionnaire (AIQ).

This survey was created by the research team for the purposes of this study. Parents and youth provided feedback on a weekly basis on a 5-point Likert scale. Specifically parents and youth rated the effectiveness of the group (1= very ineffective to 5=very effective), the content of each session (1=not very useful to 5=very useful), the effectiveness of the slides, worksheets, visuals, and instructors (1=very ineffective to 5=very effective). Parents also rated how well they thought their teen had understood the content (1=not very well to 5=very well), and youth rated how well they understood attention and EF as well as the various strategies (1=not very well to 5=very well).

Measures of Academic EFs (see Table 3 for key dependent variables)

Table 3.

Measures, Dependent Variables, Informants, and Time Points

Rating Scale	Dependent Variables for the Current Study	Informant	Time point
Homework Problems Checklist (HPC)	Homework Completion Behaviors mean score Homework Management Behaviors mean score	Parent	Pre & Post
Diagnostic Checklist for School Success (DCSS)	Good Homework Behaviors mean score ses Organization Strategies mean score Uses Test-Taking Strategies mean score	Parent	Pre & Post & Follow-up
Children’s Organizational Skills Scale (COSS)	Task Planning T-score Organized Actions T-score Memory and Materials Management T-score	Parent & Teacher	Pre & Post
Academic Performance Rating Scale (APRS)	Academic Success mean score Academic Productivity mean score	Teacher	Pre & Post
Classroom Performance Survey (CPS)	Academic Competence mean score % of assignments completed and turned in on time % yes answers to “Working up to potential?”	Teacher	Pre & Post

Academic Performance Rating Scale (APRS).

The APRS is a 19-item scale completed by the teacher that measures how well the child is performing academically in the classroom (DuPaul, Rapport, & Perriello, 1991). Factor analyses of the APRS have documented a reliable three-factor structure including Academic Success (internal consistency = .94), Academic Productivity (internal consistency = .94), and Impulse Control (internal consistency =.72), the latter which was not examined in this study given the focus of AIMS-O.

Children’s Organizational Skills Scale (COSS).

The COSS is a rating scale completed by parents and teachers which yields three factors measuring Memory and Materials Management, Organized Actions, and Task Planning. (Abikoff & Gallagher, 2009). The COSS demonstrates strong psychometric properties. Internal consistency coefficients range from .70 to .98, and 2- to 4-week test-retest reliability (Cronbach’s alpha) range from .88 to .99. Convergent and divergent validity were supported by examining the relationship between COSS scores and the Conners ADHD Rating Scale (Conners, Sitarenios, Parker, & Epstein, 1998). Discriminative validity was supported by showing that COSS scores discriminate between children from relevant clinical groups (e.g., ADHD). T-scores are generated for each COSS subscale with scores >60 indicating a clinically significant problem.

Classroom Performance Survey (CPS).

The CPS, completed by teachers, is a rating scale assessing academic impairment in secondary school youth (Brady, Evans, Berlin, Bunford, & Kern, 2012). The 23-item (20 Likert-type, 1=always to 5=never, and 3 non-Likert) CPS includes questions that have been associated with poor academic outcomes in the literature. Two highly reliable factors, Academic Competence (coefficient omega = .98) and Interpersonal Competence (coefficient omega = .91), have been derived for the CPS (Brady et al., 2012); the latter which was not utilized in this study given the focus of AIMS-O. Higher scores indicate lower competence in these domains. The CPS Academic Competence factor score is highly correlated with the Impairment Rating Scale academic progress item, providing evidence of construct validity.

Diagnostic Checklist for School Success (DCSS).

This 23-item survey was created by the research team for the purposes of this study. Parents were surveyed on frequency of good homework behaviors (e.g., use an assignment book, complete homework on time), organization strategies (e.g., keep study area organized, keep track of grades/assignments), and test preparation and taking strategy use (e.g., match study strategies with types of questions/content, organized approach to studying) using a 1 (never) to 5 (always) scale. Reliability was adequate for good homework behaviors (alpha = .72), organization strategies (alpha = .86), and test preparation/test taking strategies (alpha = .79). Higher scores indicate more use of these strategies. The DCSS was also completed by parents at the 3-month follow-up.

Homework Problems Checklist (HPC).

The HPC is a 20-item parent-report instrument that is commonly used as a screening tool for and outcome measure of homework problems (Anesko, Schoiock, Ramirez, & Levine, 1987; Langberg et al., 2010). For each item, parents rate the frequency of a specific homework problem on a 4-point scale (0 = never, 1 = at times, 2 = often, 3 = very often). The HPC has excellent internal consistency, with alpha coefficients ranging from .90 to .92 and corrected item-total correlations ranging from .31 to .72 (Anesko et al., 1987). Factor analyses indicate that the HPC has two distinct factors (Langberg et al., 2010; Power, Werba, Watkins, Angelucci, & Eiraldi, 2006): Homework Completion Behaviors and Homework Materials Management Behaviors. Higher scores on the measure indicate more severe homework problems.

Data Analyses

Attendance, satisfaction, and fidelity results were summarized. Given that the study is underpowered, as is typical for a treatment development open trial assessing feasibility, formal statistical analyses were not conducted. Instead Cohen’s d effect sizes (Cohen, 1992), correcting for dependence between means using the formula σD=σ*2*1-p (Morris & DeShon, 2002), were computed using an online effect size calculator for repeated measures designs (https://www.psychometrica.de/effect_size.html). Parent and teacher rating scale scores (M, SD) obtained at pre and post for the HPC, COSS, APRS, and CPS, and at pre and post and follow-up for the DCSS, and their correlations were entered into the calculator. In general, effect sizes ≥ .2 are considered small, ≥ .5 are considered moderate, and ≥ .8 are considered large (Cohen, 1992).

Results

1). Is AIMS-O feasible to deliver, as indexed by fidelity, attendance, and adherence data?

Fidelity to the intervention content across the seven parent and teen sessions was 94.7%. Overall, the leaders adhered closely to the intervention content with the only deviations occurring due to time constraints (i.e., not passing out a handout, failing to conduct a review of take home points at the end of one session, failing to conduct a binder check, and not playing a game). Group attendance was excellent across the seven sessions (M = 6.33, SD = 1.01) and all 21 teens completed the AIMS-O intervention and post-assessments. Additionally, parents and teens reported good compliance with the six real-world practice assignments (M = 5.10, SD = 1.79).

2). Is AIMS-O acceptable to parents and youth, as indexed by consumer satisfaction ratings?

On the AIQ Likert scale items (5 being best), parents and teens gave high ratings (i.e., >4) for the various AIMS-O content areas (see Table 4). Parents also gave high ratings (i.e., >4) for the effectiveness of the worksheets/handouts, visuals, and instructors. Teens ratings were slightly lower (i.e., >3) for the effectiveness of the worksheets/handouts, and visuals, but they still rated the information as helpful (i.e., M = 4.18, SD = 0.88; see Table 4). Parents gave slightly lower ratings (i.e., M = 3.81, SD = 0.24) for how well they thought their teen had understood the content.

Table 4.

Satisfaction Ratings

Acceptability of Intervention Questionnaire (1 to 5 scale, 5 best)	Parent M ± SD	Teen M ± SD
How useful was the material on:
Problem Solving	4.57 ± 0.68	4.14 ± 0.96
Behavior Agreements	4.47 ± 0.61	4.37 ± 0.68
Organization	4.26 ± 0.93	4.06 ± 0.90
Time Management	4.16 ± 1.01	4.00 ± 1.06
Study Skills	4.67 ± 0.16	4.14 ± 0.97
Study Cards	4.78 ± 0.43	4.32 ± 1.06
Summarizing/Notetaking	4.52 ± 0.73	4.28 ± 0.83
Effectiveness of worksheets/handouts	4.15 ± 0.15	3.48 ± 1.10
Effectiveness of visuals	4.19 ± 0.23	3.82 ± 1.22
Effectiveness of instructors	4.56 ± 0.18	n/a
How well teen understood material	3.81 ± 0.24	n/a
How helpful was the material?	n/a	4.18 ± 0.88

3). Does AIMS-O improve academic executive functions as rated by parents and teachers?

Parent Ratings

As can be observed in Table 5, parents reported improvements with a large effect size for HPC Homework Completion Behaviors and DCSS Uses Organization Strategies, improvements with a moderate effect size for HPC Homework Management Behaviors, DCSS Uses Test Taking Strategies, and all three COSS subscales (i.e., Task Planning, Organized Actions, and Memory and Materials Management), and improvements with a small effect size for DCSS Good Homework Behaviors after completing AIMS-O.

Table 5.

Effect Sizes for Rating Scales Completed Before and After AIMS-O

	Pre M (SD)	Post M (SD)	Follow-up M (SD)	Cohen’s d
Homework Problems Checklist - Parent
Homework Completion Behaviors	24.24 (6.58)	15.71 (7.63)	n/a	1.19
Homework Management Behaviors	8.33 (6.09)	4.57 (4.40)	n/a	.69
Diagnostic Checklist for School Success - Parent
Good Homework Behaviors	3.13 (0.70)	3.38 (0.68)	3.65 (0.64)	.37 / .78a
Uses Organization Strategies	2.52 (0.83)	3.30 (0.78)	3.49 (0.63)	.96 / 1.29a
Uses Test-Taking Strategies	2.81 (0.61)	3.19 (0.56)	3.39 (0.48)	.64 / 1.04a
Children’s Organizational Skills Scale – Parent
Task Planning	63.90 (10.75)	56.00 (9.48)	n/a	.77
Organized Actions	60.05 (5.24)	55.62 (6.34)	n/a	.75
Memory and Materials Management	66.57 (15.06)	58.29 (12.81)	n/a	.57
Children’s Organizational Skills Scale – Teacher
Task Planning	53.58 (12.77)	51.89 (10.67)	n/a	.14
Organized Actions	54.47 (9.59)	50.95 (11.41)	n/a	.32
Memory and Materials Management	51.16 (9.75)	50.11 (10.54)	n/a	.10
Academic Performance Rating Scale – Teacher
Academic Success	20.89 (5.47)	22.94 (6.96)	n/a	.30
Academic Productivity	30.00 (9.25)	34.72 (8.77)	n/a	.55
Classroom Performance Survey - Teacher
Academic Competence	2.02 (0.95)	1.91 (0.83)	n/a	.13
% of assignments completed and turned in on time	86.68 (24.90)	94.11 (7.29)	n/a	.30
% yes responses to Working up to potential?	63.2	68.4	n/a	n/a

^aEffect size for pre vs follow-up.

Teacher Ratings

Teachers reported improvements with a moderate effect size for APRS Academic Productivity (see Table 5). Improvements with a small effect size were observed for COSS Organized Actions, APRS Academic Success, and CPS Interpersonal Competence and percent of assignments completed and turned in on time. Additionally, teachers responded “yes” more frequently to the CPS question regarding whether the student was working up to potential after AIMS-O (% yes at pre: 63.2%, % yes at post: 68.4%).

4). Are gains, if any, on academic EF competency maintained over time?

When comparing parent DCSS ratings obtained at pre versus the 3-month follow-up, effect sizes for each subscale were large (see Table 5).

Discussion

This study provides preliminary support for the AIMS-O intervention which targets academic EFs in middle school youth with ASD without ID. Findings show that the AIMS-O intervention is feasible, well received by families, and associated with improvement in key academic domains as rated by parents and teachers. Taken together with previous treatment development efforts (Tamm et al., 2019), AIMS-O appears to be a promising intervention for improving organization, planning, time-management, and study skills, in middle school youth with ASD. Future treatment development steps should include a randomized clinical trial to replicate and confirm specificity of findings.

Feasibility and Acceptability of AIMS-O

Although replication and a larger randomized clinical trial is warranted, results of the open trial suggest high feasibility and acceptability. Our assessment of treatment fidelity showed that the intervention can be implemented as planned. All intervention components were feasibly implemented by each interventionist suggesting AIMS-O can be reliably delivered. The few deviations from the AIMS-O manual were due to time constraints, which is not unexpected and likely countered by the level of individualized coaching built into the intervention. Family attendance was excellent and all 21 families completed the trial. Both parents and teens reported that the intervention content was useful, parents gave high ratings for the effectiveness of the instructor, visuals, and worksheets/handouts, and teens reported that the intervention content was helpful. Ratings were good for teen ratings of the visuals and worksheets/handouts and parent ratings of how well their child understood the material. The ratings of teen understanding of the AIMS-O intervention material (M=3.8, SD=0.2) improved somewhat from the initial proof of concept trial (M=3.4, SD=1.0) (Tamm et al., 2019). These data indicate satisfaction with the intervention by both parents and teens, and suggest that AIMS-O would be well attended if offered in outpatient clinical settings.

Effect of AIMS-O on Academic Executive Functions

Acute treatment effects were present, with moderate to large effect sizes observed for parent report of homework management, homework completion, materials management, organization, task planning, and use of test-taking strategies. These effect sizes are quite similar to those obtained in the “proof of concept” trial with three families, which reported moderate to large effect sizes for the parent-rated COSS materials management, organized actions, and task planning subscales, and HPC homework management and homework completion factors (Tamm et al., 2019). Improvements in these domains is particularly important given these are the areas identified as problematic for youth with ASD in relation to middle school functioning (Tamm et al., 2019). It is also relevant to note that effect sizes for the youth with ASD were in the range obtained for youth with ADHD in the original ADHD intervention from which AIMS-O was adapted [i.e., moderate to large effect sizes on the HPC factors (Ciesielski et al., 2015; Langberg et al., 2008)]. These findings are also consistent with multiple baseline studies providing support for specific strategies to improve organization (Dorminy, Luscre, & Gast, 2009), work initiation (Brown & Mirenda, 2006), and on-task behaviors (Wilczynski, Fusilier, Dubard, & Elliott, 2005) for youth with ASD. Nonetheless, these ratings were obtained by parents who were involved in AIMS-O; the fact that parents knew they were receiving an intervention targeting EF may have biased their reports of improved academic EFs.

This concern is somewhat mitigated by the finding that teachers, who were less likely to know the teen was receiving an academic EF intervention, also reported improvements in academic success, academic productivity, organization, and percent of assignments completed and turned in on time. However, these ratings were more modest in terms of effect sizes (i.e., small to moderate). The largest effect size for the teacher ratings was on the APRS Academic Productivity (d =.55), a factor reflecting behaviors that are important in the process of achieving classroom success (e.g., work completion, following instructions accurately, and ability to work independently in a timely fashion) (DuPaul et al., 1991). Further, small effects on the APRS Academic Success factor (e.g., quality of a child’s academic achievement, ability to learn material quickly, and recall skills) and COSS Organized Actions (e.g., ability to use practical tools and routines to stay organized, and to produce tidy, error-free work) suggest meaningful improvements in domains highly critical for academic performance. It may be beneficial to engage teachers in the intervention process to further boost the impact of AIMS-O strategies on academic EF behaviors. Notably, the effect size obtained for APRS Academic Productivity is higher than the between group effect size reported for youth in the Homework, Organization, and Planning (HOPS) intervention compared to those in a waitlist control group; i.e., d=.27 (Langberg et al., 2008).

Maintenance of Gains after AIMS-O

Although only one measure, the DCSS, was completed by parents at the 3-month follow-up, results suggest gains were maintained. In fact, effect sizes for each DCSS subscale at the three-month follow-up were large, particularly for Uses Organization Strategies and Uses Test Tasking Strategies, suggesting gains were not only maintained, but that skills continued to improve after AIMS-O ended. Certainly, it is not expected that all changes will occur in the space of a 7-week intervention, and it takes time for teens and their parents to apply their learnings. Other similar interventions in youth with ADHD similarly report additional gains between the post-assessment and later follow-up assessments (DuPaul et al., 2018; Evans et al., 2018). Thus, check-ins or booster sessions with families to either check in on how things are going, reiterate previously covered information, or discuss new information (e.g., how to break down long term assignments, additional study skills) may be particularly beneficial for future applications of AIMS-O. It may also be useful to provide AIMS-O booster sessions prior to the transition to high school.

Implications

While the transition to middle school presents challenges for all students, students with ASD are at heightened risk for experiencing difficulties given their problems with organization, EF, and social competency that are potent risk factors for academic problems in the middle school environment (Adreon & Stella, 2001; Mullins & Irvin, 2000). The transition to middle school is also a natural time for parents to demand increased independence related to academic work for students. However, parents of children with ASD often struggle to foster this increasing independence, in part because their children may need more support than typically developing children, and in part because they may lack the knowledge and skills regarding how to both facilitate the process of effectively developing age-appropriate independence while also fading any unnecessary supports or prompts. Thus, AIMS-O has the potential to positively impact academic functioning of youth with ASD at a particularly vulnerable transition. Parent involvement in AIMS-O is likely to increase generalization and maintenance of skills over time and foster independence and autonomy in homework and academic performance. Given the dearth of evidence-based treatments for adolescents with ASD targeting critical academic EFs such as homework completion, poor grades, difficulties with organization, procrastination, and/or the inability to break down assignments and projects, AIMS-O holds promise as treatment that could be implemented in clinical settings. In fact, the AIMS-O format can be easily adapted to a clinical setting (i.e., a primarily conjoint parent and teen group program); see (Ciesielski et al., 2015) for an example of implementing a similar program based on research for children with ADHD in an outpatient setting.

Limitations & Future Directions

This study is not without limitations. The sample size was small, with limited racial and socioeconomic diversity, limiting the generalizability of results, but appropriate for a feasibility open trial. There was no active control group to ensure that the gains observed were specifically due to the intervention. Results were limited to rating scale data; additional work investigating impact on real-world outcomes such as grades, individualized education program (IEP) goals, etc., is needed. Finally, given the small sample size we were not able to explore whether moderators (e.g., sex, psychiatric comorbidities, etc.) impacted uptake of the intervention. Replication is warranted to confirm the current findings of improved academic EFs after intervention and to rule out nonspecific effects related to time, development, or being in an intervention study. A randomized clinical trial with a larger sample of children with ASD, which includes a control group that does not receive intervention, is a necessary next step in exploring the efficacy of AIMS-O in improving organization, time management, planning, and study skills in children with ASD. Should results continue to show a beneficial effect of AIMS-O on academic EFs, it will be important to link these improvements to real-world academic outcomes. Additionally, this intervention focuses primarily on teaching academic EF skills related to organization, time management, and planning/prioritizing. Future work could also focus on targeting alternative higher level EF skills that also impact academic success, such as shifting attention and flexibility.

Conclusions

Overall, the open trial results provide promising evidence that AIMS-O is feasible to deliver, acceptable to parents and middle school youth with ASD without ID, and positively impacts academic EFs by both parent and teacher report, albeit with relatively modest effect sizes. Further, gains appear to be maintained and to additionally improve over time. Replication is warranted.