Comprehensive attention training system (CATS): A computerized executive-functioning training for school-aged children with autism spectrum disorder

Abstract

Objective

Theory suggests that impaired executive functioning (EF) might explain several symptoms of autism spectrum disorder (ASD) in children. However, only a few studies have examined the efficacy of EF training for the children using randomized control trial designs, and only two of them found significant benefits of the training.

Method

We designed Comprehensive Attention Training System (CATS), and tested this new EF intervention for children with ASD in a small-sampled randomized controlled trial. Twenty-five children with ASD aged six to twelve were randomly assigned to either the CATS or the control training and were assessed pre- and post-training.

Results

Relative to the control group, the CATS group improved on EF as measured by the trail-making test, avoiding perseverative errors, and forming conceptual responses in the Wisconsin Card Sorting Task. There were also indications that CATS contributed to long-term communication skills as measured by the Vineland adaptive behavior scales.

Conclusions

We report preliminary evidence that the CATS intervention may improve the EF of school-aged children with ASD compared to a control intervention. We discuss the results in terms of their generalizability to other developmental disorders.

Introduction

Treatment for children with autism spectrum disorder (ASD) has become an important topic in the field of developmental difficulties (Wei et al., 2014). Many researchers have demonstrated the relative weakness in executive functioning (EF) of children with ASD (Brady et al., 2017; Corbett et al., 2009; Hill, 2004; Ozonoff et al., 2007; McGonigle-Chalmers and McCrohan, 2018). However, there are only a few existing studies on the effectiveness of EF interventions for children with ASD (Baltruschat et al., 2011; de Vries et al., 2015; Fisher and Happé, 2005; Hilton et al., 2014; Kenworthy et al., 2014). Therefore, the current study aimed to examine the effectiveness of a new computerized EF intervention designed for children with ASD—the Comprehensive Attention Training System (CATS)—in a small-sampled randomized controlled trial.

Executive functioning and ASD

EF is an umbrella set of cognitive abilities that guide an individual’s goal- and future-oriented thoughts and actions (Hill, 2004). It includes abilities such as inhibitory control, working memory, flexibility, and planning (Wallace et al., 2016). Because EF provides an individual with the general mental and behavioral flexibility needed for navigating through daily life, Lezak et al. (2004) have argued that EF is necessary for acquiring social skills, adapting to new environments, and understanding others’ emotions. Difficulties with EF may hence contribute to non-social (e.g. behavioral rigidity) as well as social (e.g. poor social skills) symptoms of ASD (McGonigle-Chalmers and McCrohan, 2018; Hill and Frith, 2003).

Supporting the theory, empirical evidence has revealed several associations between EF impairment and ASD, especially for, but not limited to, the components that we targeted in the present research: planning, flexibility, and inhibitory control (Hill, 2004). A recent meta-analysis also documented that, with variations, people with ASD on average show reduced EF across its components (Demetriou et al., 2018). Below, we review the literature on EF impairment and ASD by key EF components.

Planning

Planning is the cognitive ability to organize thoughts and actions in advance in order to achieve goals (Dawson and Guare, 2004). The ability of planning is typically assessed in multi-step tasks (Wallace et al., 2016), such as Tower of London and Tower of Hanoi tasks. In these assessments, individuals’ goal is similarly to use the least steps to solve assigned puzzles; they need to think ahead and plan well accordingly. Focusing on ASD, Pellicano (2007) found that it is difficult for children with ASD to learn new strategies to better solve Tower of London. A wealth of research has also discovered that children with ASD perform worse on Tower of Hanoi than do normally developing children as well as children with Tourette syndrome (Bennetto et al., 1996; Ozonoff and McEvoy, 1994; Ozonoff and Jensen, 1999; Ozonoff et al., 1991).

Flexibility

lexibility is also an area of EF impairment associated with ASD. Flexibility—also referred to as task switching or shifting—is the ability to modify an original plan (i.e. the task one set for oneself) when one faces difficulties or obstacles, makes mistakes, or even just receives new relevant information (Hill, 2004; Wallace et al., 2016). The Wisconsin Card Sorting Task (WCST) is one of the most established flexibility assessments in the field (Bennetto et al., 1996; Pellicano, 2007; Prior and Hoffmann, 1990; Rumsey, 1985). Using the WCST, researchers have demonstrated that children with ASD exhibit less general cognitive flexibility (Bennetto et al., 1996; Pellicano, 2007; Prior and Hoffmann, 1990) and greater behavioral rigidity (Rumsey, 1985) than do typically developing children.

Inhibitory control

Finally, inhibitory control—also referred to as impulsive control or, simply, inhibition—is the ability to think before actions in order to ignore irrelevant stimuli in the environment and not respond to them (Dawson and Guare, 2004). There are many assessment tools for the ability of inhibitory control, such as the go/no-go test, the Stroop task, and Luria’s hand game. Unlike for planning and flexibility, evidence that individuals with ASD have weaker inhibitory control is still emerging. However, a few studies have now suggested that children with ASD show poorer inhibitory control compared to typically developed children (Hill 2004; Russell et al., 1991; Hughes and Russell, 1993). Lemon et al. (2011) have further discovered that the phenomenon might differ between genders. Different from typically developed individuals, it is females but not males with ASD who display weaker inhibitory control. This discovery points to the uniqueness and, potentially, importance of the roles of inhibitory control in ASD.

Executive functioning intervention

Despite a wealth of research on the associations between EF impairment and ASD symptoms, relatively little attention has been paid to EF interventions for children with ASD. Wallace et al. (2016) further suggest that there is a lack of methodological rigor in most existing studies in the research, which have omitted pre-training assessments, control groups, or random assignment if there was a control group. Nonetheless, these scholars also indicate that the literature is accumulating, albeit slowly, across a number of treatment modalities—including cognitive behavior therapy (CBT; Fisher and Happé, 2005; Kenworthy, 2014), mediation (Handen et al., 2011), biofeedback-based training (Kouijzer et al., 2013), and computerized training (de Vries et al., 2015)—and that some of them have shown considerable potential. As such, in order to balance the quantity with the quality of our review, below we summarize this literature on EF training for children with ASD, focusing on studies that utilized a full randomized-controlled-trial (RCT) design.

CBT

Two studies fall in the group of CBT-based intervention. First, Fisher and Happé (2005) conducted one of the earliest EF interventions for individuals with ASD, and they reported no significant EF improvement for the EF training group compared to a control group (trained in Theory of Mind). Specifically, the EF training consisted of five stages, each with a concept to learn, and the children trained had to perform to criterion in order to move from one stage to the next (e.g. the criterion to pass Stage two was to learn that ‘Different people can have different thought pictures in their heads’). Here, the researchers observed no improvement in EF for either the EF or the control group.

By contrast, the field experiment conducted by Kenworthy et al. (2014) reported that their EF intervention effectively strengthened the EF of children with ASD compared to a control intervention. In their study, elementary-school children were randomly assigned to either an EF training curriculum at school or a social-skill training control curriculum. After six months of training, although both groups improved in EF and school-social abilities, the EF training group did so more than the control group, in cognitive flexibility, non-verbal problem-solving skills, as well as classroom interpersonal behavior and adaptation.

Medication

Handen et al. (2011) published an RCT study testing the effect of donepezil on the EF of individuals with ASD. The participants were 34 average-IQ children and adolescents with ASD, randomly assigned to receive either donepezil or a placebo non-responder. The treatment sustained 10 weeks, and induced no significant difference between both groups in change of EF.

Biofeedback

Kouijzer et al. (2013) examined two biofeedback training programs on EF and found significant benefits from one compared to the other program. In particular, 38 participants with ASD were randomly assigned to either a 40-session EEG-biofeedback intervention, a 40-session skin-conductance (SC)-biofeedback intervention, or a waitlist group. The results indicated that about half of participants in the EEG group succeeded in regulating their delta or theta power during biofeedback sessions, and a significant improvement in cognitive flexibility was observed for these EEG regulators compared to those who were in the SC group and succeeded in regulating SC. The findings, therefore, revealed both potential and limitation of biofeedback-based intervention: It may be beneficial, but only for those who can learn to use it.

Computerized training

Focusing on individuals with ASD, de Vries et al. (2015) used the training ‘Brain Game Brain’ to design an intervention study composed of 25 training sessions. The results were inconclusive because the researchers failed to demonstrate that their training improved EF more than the control training did. Thus, whether the effects found represented actual improvements in EF or mere placebo effects remains unclear. However, de Vries et al. (2015) did document some indication—a significant effect on one measure among a few other insignificant ones—of improvement in EF from before to after the BrainGame Brain intervention for participants in the treatment group. In other words, EF is potentially trainable although not easily so (e.g. Kenworthy et al., 2014), and computerized intervention could be an option. As such, we describe below our newly designed computerized intervention—the Comprehensive Attention Training System (CATS).

Comprehensive attention training system (CATS)—computerized training

Based on Cohen et al.’s (1993) cognitive theory, one of the present authors independently developed CATS, a computerized EF training program. We report here a small-sampled randomized control trial of CATS for children with ASD. Cohen et al. (1993) posit that attention is comprised of four neuropsychological components closely related to EF: sustained attention, sensory selection, response selection and control, and attention capacity.

Building on the theory, CATS has trainees practice all four of these mental capabilities to improve their EF. Specifically, trainees are asked to respond to dot stimuli shown on the computer screen with assigned response keys. In several different cognitive training activities, (detailed in Table 1), the stimuli may appear in different colors, at different locations, and for different durations. Trainees must use the correct keys or no key-press (i.e. inhibiting key pressing) to respond to each stimulus according to the activity rules of each training session. Detailed information about the activities at difference levels of training in CATS is provided in Table 1 and Figure 1.

Table 1.

Contents of levels of training activities in CATS

Level	Training
1	In each trial in a given session, a red light flashes and the trainee needs to respond to the stimulus in time by pressing the number key of the block in which the light flashes. Only one block is used in a session, so only one number key is used in a session.
2	This level is the same as Level 1, except that all blocks are used in a session, so the trainee needs to choose the right number key to press.
3	As illustrated in Figure 1, in each trial, nearby red lights appear one after another and extend into a line. The trainee needs to track the extension and press the number key of the block in which the line stops extending.
4	As opposed to the extending in Level 3, a red light moves here (i.e. it extends with no trace). The trainee’s task is the same as in Level 3.
5	This level is the same as Level 1, expect that now a red light might be accompanied by a green light. If so, the trainee needs to inhibit key pressing and not respond. There are, therefore, two behaviors in the level: responding to the red light and not responding to the paired light.
6	The level is the same as Level 2, except that now the rule of paired-light inhibitory control is added in.
7	This level is the same as Level 3, except that some red lights might be accompanied by their green lights when the line is extending. If so, the trainee needs to inhibit key pressing when the line stops extending.
8	This level is the moving version (cf. Level 4) of Level 7. The trainee therefore needs not only inhibitory control ability, but also working memory for remembering green lights have shown (or not) to make the right response in the end of each light movement.
9	This level is the same as Level 1, except that now the stimuli might be red lights only or green lights only. The trainee needs to press a left-hand key to respond to a red light and a right-hand key to respond to a green light. Unlike Level 1, this level therefore uses two, but not one, keys in a given session.
10	This level is the same as Level 9, except that now stimuli can appear in any block (cf. Level 2).
11	This level is the same as Level 9, except that now the stimuli might be red lights only, green lights only, or paired red and green lights. If a pair of lights flash, the trainee needs to inhibit responding.
12	This level is the same as Level 11, except that now stimuli can appear in any block (cf. Level 2).

Figure 1.

A screenshot of CATS. Note: There are six blocks each with six pairs of red and green lights and a white box. Trainees are verbally instructed to respond to stimuli in each block with a different number key on the keyboard. See Table 1 for the different tasks of different levels of training.

To complete the CATS this way, children have to control their impulsivity, ignore irrelevant stimuli, sustain attention to the right (i.e. target) stimuli, as well as follow and switch between the rules of sessions. CATS, therefore, may train children in attention monitoring and cognitive flexibility, as the response rules change from one session to another and become more complex from one level to the next. For instance, children must sustain their attention to local stimuli at a fixed location at Level 1, and switch to monitor a larger field for stimuli that change locations at Level 2 (see Table 1). Further, starting at Level 5, trainees need to match the colors of stimuli with prescribed colors of targets of a session. CATS then adds a memory component to the process. Finally, the trainee’s ability to not only perform well within sessions but also track rule changes between sessions provides a possibility for them to exercise flexibility. In summary, CATS trains children in EF comprehensively by requiring them to not only inhibit incorrect responses but also initiate the correct ones. The process involves trainees’ sustaining their behavior, guiding their attention, and monitoring their performance within single sessions as well as across the entire training program.

Besides the above design features, all instructions in CATS are provided in simple and straightforward sentences to fit with the special needs of children with ASD. Similar to the intervention used by Kouijzer et al. (2013), CATS also gives children feedback on their performance by star ratings (e.g. ★★★☆; the more black stars the better), instead of verbal or written notes (as in the case of Fisher and Happé, 2005). In addition, we recognized the importance of children’s autonomy and motivation in training, which had not been addressed in the reviewed training programs. By contrast, children are encouraged to customize the difficulty of training sessions in CATS (e.g. by choosing easier versions of a training activity that use longer stimulus latency) and to adjust the difficulty anytime to match their interest level. We implement this feature of CATS because the freedom of designing—at least partly—their own training programs may help children actively monitor their performance and make plans for learning. In this regard, we supplement past EF training with training of planning ability and self-monitoring, arguably the highest-order and most abstract components of EF.

The current study

To test the effectiveness of CATS, we recruited elementary-school children with ASD and randomly assigned them to one of two eight-week training groups: the CATS group or an active control group. We expected the treatment, CATS, group to demonstrate significant improvements in EF compared to the control group.

Methods

Participants

We collaborated with a psychiatrist specialized in child and adolescent psychiatry at a teaching hospital in a major city in Taiwan to carry out the first-stage selection of school-aged participants with ASD. Eligible children were between the ages of six and twelve and had a diagnosis of ASD as defined by DSM-IV-TR. Those who met the following comorbid conditions according to their medical records were excluded: brain injury, psychotic symptoms, intellectual disability, sensory disorder, epilepsy, cerebral palsy, anxiety disorder, depression disorder, or obsessive-compulsive disorder.

After the psychiatrist referred eligible participants to us, the first author explained the study design and related ethical issues to the children’s legal guardians and invited the guardians and children to participate in the study. Those who agreed to participate completed a formal consent form and the screening test. Here, we aimed to exclude individuals with an intelligence quotient (IQ; assessed with Raven’s Progressive Matrices, see Measures) two standard deviations (SDs) below the average. The reason was that the current study was a small-sampled trial and, therefore, more susceptible to extreme responses such as those implied by extreme IQ. Nonetheless, no participant’s IQ actually fell below the threshold. The trainer who delivered both CATS and control intervention was a graduate student in clinical psychology, supervised by a licensed clinical psychologist. The guardians of 25 out of the 32 children referred to us agreed to participate. Participants were then randomly assigned to either the control group (n = 13, mean age = 8.42, SD = 1.35, 1 female) or the CATS group (n = 12, mean age = 8.75, SD = 1.40, 1 female). It might be worth noting that we did not base our targeted sample size on a formal a priori power analysis, because it was clear to us that, given the limited size of our sample pool, we could likely recruit and compensate everyone referred to us, and we did so to maximize the power we could achieve. We then conducted a sensitivity analysis, and found that the present research achieved a fair sensitivity, f = .23, to detect a small-to-medium effect, under power = .80 and pre-post-test test-retest reliability = .7.

Procedure

We trained both groups of participants in the same clinic room in the hospital where we recruited them. Participants’ EF levels and daily adaptation abilities were assessed before the training in a pre-test. Right after the training program, EF was assessed again in a post-test. Six months after the intervention ended, we further followed up with the participants’ guardians to measure the participants’ daily adaptation abilities, including that of social communication, by physical mail. All research activities were reviewed and approved by the ethical board of the hospital where the study was conducted.

During the training phase, participants in the same treatment conditions were paired into groups of two based on their availability. Children in the CATS condition completed the CATS training under the instructions of the trainer; children in the control group were trained in social skills under the instructions of the same trainer. Specifically, the control group read picture books with the aid of the trainer to learn the plots and characters’ feelings in the stories. For both groups, the trainer followed the same guideline of being positive, supportive, non-judgmental, and encouraging in training, and administered the pre- as well as the post-test. Lastly, the training in both groups took place once a week for eight consecutive weeks. Each session was 50 min long. In other words, participants could receive at most 400 min of training in total, and all participants indeed attended at least half of the intervention, that is, 4 sessions, and the average attendance was 7.24 sessions. No between-group difference in the attendance was found (t = 0.28, df = 23, p = .769).

Measures

Raven’s Progressive Matrices (RPM; Styles et al., 1998; Yu, 2003). RPM is a paper-and-pencil measure of conceptualization and abstract reasoning. Here, it was used to assess participants’ pre-training non-verbal cognitive functioning. Specifically, two versions of RPM were administered: the Raven’s Coloured Progressive Matrices (CPM) and the Raven’s Standard Progressive Matrices (SPM). CPM was administered for children aged 9.5 or younger, and the SPM was used for older children. The test-retest reliability of CPM and SPM in Taiwan is between .53 to .92, and the criterion-related validity is between .31 to .79.

Home Observation for Measurement of the Environment (HOME; elementary version; Jan et al., 2006). By assessing the quantity of support children receive in different life domains, HOME measures the quality of children’s general caring environment. We incorporated HOME in the pre-test to assess the background comparability between the two training groups in our study. The Cronbach’s alpha is .82, and the criterion related validity is .60.

Wisconsin Card Sorting Task (WCST; full-128-carded paper-and-pencil version). The WCST (Heaton et al., 1993) is one of the standard EF tests in the field. Solving the puzzles in the WCST requires children’s abstract reasoning ability and cognitive flexibility to conceptualize, maintain, and shift between different rules in a card-matching tasks across multiple card categories (Heaton et al., 1993). According to Cinan and Tanör (2002), the inter-rater reliability of the WCST is between .88 and .93, and the test-retest reliability is about .87. Below, we describe the indices of the WCST we employed, and it might be worth noting that, if an index had a standard-score correspondence in the assessment manual, we used the standard score in the following analyses, and the raw score otherwise.

Number of categories completed (raw score). The number of categories completed is equal to the number of rules a child successfully figures out in the WCST (Heaton et al., 1993). Because there are six such rules in WCST, the number of categories completed ranges from 0 to 6 and measures a child’s general EF—the higher, the stronger. The measure has no standard score transformation, so the raw score was used in the current study.

Trials to complete the first category (raw score). The number of trials needed to complete the first category represents a child’s ability to conceptualize a matching rule. Because the index only accounts for the first category, it is not affected by a child’s ability to shift between rules after the first category (Heaton et al., 1993). The score of this index ranges from 10 to 129. A higher score indicates more trials needed for a child to conceptualize a matching rule and, thus, weaker EF. As the total number of categories completed, total trials to complete the first category does not have a standard score; thus, the raw score was used.

Failure to maintain sets (raw score). Completing more than four consecutive correct matches of cards indicates a child’s successful conceptualization of a matching rule. If a child makes more than four consecutive matches but then makes an error before the whole category finishes, it is said that the child makes a mistake that shows failure to maintain a rule of matching (Heaton et al., 1993). As above, the number of this kind of mistakes of failure to maintain sets does not have a standard score, so the raw score was used, and the higher the score, the weaker one’s EF.

Perseverative errors (standard score). In contrast to failure to maintain correct matching rules, perseverative errors represent a child’s inability to shift away from incorrect matching rules (Heaton et al., 1993). The score is calculated as the ratio of perseverative errors to the total number of cards used. In the current study, however, a higher score indicated a lower likelihood of making perseverative errors, because we used the standard score derived from the normative data of the WCST (Heaton et al., 1993).

Non-perseverative errors (standard score). Derived from the same calculation process of perseverative errors, non-perseverative errors are the standardized ratio of non-perseverative errors to the total number of cards used, and the higher the score of non-perseverative errors, the lower the likelihood of making such errors. Unlike other WCST indices, we did not expect a significant effect of CATS on non-perseverative errors, because the behavioral difficulty of children with ASD is mainly perseverative and rigid behavior, not its non-perseverative counterpart.

Conceptual level responses (standard score). A conceptual level response is said to emerge when a child shows insights into matching rules by making more than two consecutive correct matches. Like perseverative errors and non-perseverative errors, the standard score of conceptual level responses was used, and higher scores here denoted more conceptual insights acquired in WCST (Heaton et al., 1993).

Trail-Making Test (TMT). The TMT (Reitan, 1955) is another well-known EF test used in both ASD and non-ASD research (Kleinhans et al., 2005). In the TMT, children are directed to link circles labelled with ordered symbols like numbers and English alphabets to make trails. Children complete the trail making twice and, in the first trail, the connection rule is easier. The first trail can usually be completed faster than the second trail, which has a rule that is a combination of a new rule and the rule of the first trail. The time increase in finishing the second trail is thus an index of one’s cognitive inflexibility to switch to a more complex symbol linking rule (Reitan, 1955).

For the current study, the time to finish the both trails was measured in seconds, and then adjusted by natural log as in most reaction-time studies. We then calculated the increase in logged time from the first to the second trail. Additionally, the TMT data of two participants in the control group were excluded, because they did not follow the instructions in the pre-test session and, thus, did not complete the TMT.

Vineland Adaptive Behavior Scales, Chinese version (VABS). The VABS (Sparrow et al., 1984; Wu et al., 2004) is a measure of children’s social adaptability. Specifically, the assessment offers adaptation scores for four different domains: communication skills, daily living skills, socialization skills, and motor skills. We also calculated a composite score of the domains, as commonly done in the literature. In the current study, VABS was sent to participants’ legal guardians by mail six months after the training, and we received five replies from the control group and eight from the CATS group. Finally, the norm data of Taiwan indicate that the split-half reliability of VABS is between .91 and .99, the test-retest reliability is between .62 to .95, and the inter-rater reliability is between .74 to .89.

Results

Differences between the two groups in participant demographic characteristics (age, quality of care, and non-verbal IQ) were compared using t-tests; we used SPSS 20 in all analyses. As shown in Table 2, participants in the CATS and the control groups were statistically comparable in age, quality of care received (HOME), and non-verbal intelligence (RPM). Further, prior to participating in the study, five participants in the control group and three in the CATS group reported taking prescription stimulants, including Ritalin (control group, n = 3; CATS group, n = 1) and Concerta (control group, n = 2; CATS group, n = 2). To minimize the influence of medication, participants were required to stop taking the medication at least 24 h before the pre- and the post-test. Chi-square tests also revealed no difference in medication status between groups (χ² = 1.02, df = 2, p = .60). Finally, three participants in the control and six in the CATS group had diagnoses of attention deficit hyperactivity disorder (ADHD) in addition to ASD. This comorbidity rate, 36%, was well within the range of those found in the literature (see Jang et al., 2013) and did not statistically vary between groups in the present research (χ² = 1.96, df = 1, p = .16).

Table 2.

Descriptives

Measure	Control group (pre-test n = 13)					CATS group (pre-test n = 12)
	Pre-M (SD)	Post-M (SD)	t	p		Pre-M (SD)	Post-M (SD)	t	p
Pre-test only
Age (yr)	8.42 (1.35)					8.75 (1.40)		0.61	0.55
HOME (avg)	0.72 (0.09)					0.69 (0.11)		−0.75	0.46
Raven’s progressive matrices (pr)	56.08 (32.68)					52.08 (27.09)		−0.33	0.74
Immediate post-test (same n as pre-test)
Trail-making test (ln; control n = 11)	1.22 (0.47)	1.41 (0.42)	1.33	.198		1.32 (0.57)	1.10 (0.40)	−1.67	.110
WCST
Categories completed (raw)	3.46 (2.44)	3.00 (2.04)	−0.83	.414		3.00 (1.91)	3.58 (1.93)	1.01	.323
Trials to 1st category (raw)	43.00 (49.80)	37.62 (42.16)	−0.52	.607		33.92 (39.15)	25.00 (33.85)	−0.83	.415
Failure to maintain set (raw)	1.23 (1.36)	1.23 (1.24)	0.00	1.000		1.00 (1.21)	2.33 (2.02)*	2.84	.009
Perseverative errors (std)	94.08 (18.94)	90.77 (15.11)	−0.85	.403		83.83 (14.86)	97.83 (12.09)*	3.47	.002
Non-perseverative err (std)	101.77 (29.22)	93.00 (18.45)	−1.10	.281		97.00 (21.29)	93.58 (18.70)	−0.41	.683
Conceptual level resp (std)	91.46 (21.97)	87.38 (15.33)	−0.74	.469		84.17 (14.01)	95.75 (17.97)†	2.01	.056
Post-test in 6 months		(n = 5)					(n = 8)
VABS composite (std)	79.62 (13.45)	83.00 (16.08)	0.89	.388		86.83 (13.54)	93.50 (20.76)	1.38	.191
Communication (std)	86.08 (14.62)	87.00 (15.25)	0.84	.417		90.00 (9.85)	100.13 (14.29)*	3.96	.002
Daily living (std)	78.46 (11.98)	83.00 (19.27)	1.31	.210		86.58 (13.61)	91.00 (18.76)	0.68	.506
Socialization (std)	80.23 (14.81)	82.40 (19.42)	0.71	.490		85.33 (18.86)	91.88 (24.43)	1.09	.296
Motor (std)	89.38 (9.81)	94.40 (14.08)†	2.02	.065		91.42 (13.30)	97.38 (12.41)	1.45	.174

Note: The above t-tests of pre-to-post changes are simple main effects within groups in the multilevel model shown in Table 3; * and † designate effects with p < .05 and .10, respectively.

Main analyses

As detailed in Table 3, we analyzed all pre-to-post-treatment effects and pretest-to-followup effects in a two-level random-intercept model in which repeated measures were nested within participants. Accordingly, we hypothesized that the cross-level time-by-group interaction would be significant to indicate greater improvements in EF demonstrated by the CATS group than by the control group. We also examined the simple effects—changes—of each group over the training in the same multilevel framework.

Table 3.

Analysis of Intervention Effects

Dependent variable	Group					Time					Group x Time
	γ CI95%	df	t	p		γ CI95%	df	t	p		γ CI95%	df	t	p
Pre-test only
Age	0.33 ± 1.14	23.00	0.61	.550
HOME	−0.03 ± 0.08	23.00	−0.75	.462
Raven’s matrices	−3.99 ± 24.96	23.00	−0.33	.744
Immediate post-test
Trail-making test	0.10 ± 0.40	33.05	0.51	.614		−0.02 ± 0.20	21.00	−0.19	.848		−0.41 ± 0.40*	21.00	−2.12	.047
WCST
Categories completed	−0.46 ± 1.70	35.49	−0.55	.586		0.06 ± 0.83	23.00	0.15	.881		1.04 ± 1.66	23.00	1.30	.205
Trials to 1st category	−9.08 ± 34.06	33.69	−0.54	.591		−7.15 ± 15.41	23.00	−0.96	.347		−3.53 ± 30.83	23.00	−0.24	.815
Failure to maintain set	−0.23 ± 1.20	39.63	−0.39	.700		0.67 ± 0.67†	23.00	2.05	.052		1.33 ± 1.34†	23.00	2.05	.052
Perseverative errors	−10.24 ± 12.63	34.02	−1.65	.108		5.35 ± 5.79†	23.00	1.91	.069		17.31 ± 11.59*	23.00	3.09	.005
Non-perseverative err	−4.77 ± 18.10	44.45	−0.53	.598		−6.09 ± 11.85	23.00	−1.06	.298		5.35 ± 23.70	23.00	0.47	.645
Conceptual level resp	−7.29 ± 14.27	40.71	−1.03	.308		3.75 ± 8.26	23.00	0.94	.357		15.66 ± 16.53†	23.00	1.96	.062
Post-test in 6 months
VABS composite	7.22 ± 12.67	27.01	1.17	.253		5.10 ± 7.03	13.38	1.56	.142		1.15 ± 14.06	13.38	0.18	.862
Communication	3.92 ± 11.09	24.59	0.73	.473		6.41 ± 4.47*	11.72	3.13	.009		7.43 ± 8.94†	11.72	1.82	.095
Daily living	8.12 ± 12.24	27.81	1.36	.185		4.92 ± 7.28	13.95	1.45	.169		−3.99 ± 14.57	13.95	−0.59	.566
Socialization	5.10 ± 15.35	25.95	0.68	.501		4.20 ± 7.36	12.78	1.24	.238		0.89 ± 14.71	12.78	0.13	.898
Motor	2.03 ± 10.12	25.59	0.41	.683		5.86 ± 5.14 *	12.07	2.48	.029		−3.15 ± 10.27	12.07	−0.67	.517

Note: The above effects were from a mixed-effect model DV _ti = γ₀ + γ₁ * Group _i + γ₂ * Time _ti + γ₃ * Group × Time _ti + u_i + r_ti, where Group was coded (-0.5, 0.5) for (control, CATS) and Time was (0, 1) for (pre-, post-test); see Results for coding rationales; * and † designate effects with p < .05 and .10, respectively.

As shown in Table 3, the CATS group demonstrated significantly greater improvements in the TMT and avoidance of perseverative errors than did the control group, either with or without familywise alpha-error-rate corrections (e.g. using the Bonferroni method). Specifically, the CATS group’s mean score of avoiding perseverative errors significantly increased from 1.08-SD below the population mean (100, SD = 15) to only 0.14 SD below the mean (Table 2), whereas no significant simple effect was found for the control group. This finding indicates that participants in the CATS group not only improved, but nearly caught up with typically developing children after the training. In comparison, we found no simple change for either the CATS or the control group on the TMT, even though the group-by-time interaction was significant.

In addition to significant findings, participants in the CATS group also demonstrated marginally significant improvements in conceptual level responses and long-term communication ability relative to the control group (Table 3). Specifically, the CATS group’s score of conceptual level responses rose marginally significantly from 1.06-SD below the population mean to 0.28 SD below (Table 2), demonstrating its minimal post-treatment difference from the typically developing children. At six-month follow-up, participants in the CATS group further demonstrated a significant improvement in communication ability from 0.88 to 0.43 SD below the population mean (i.e. 100, SD = 15). Again, although the group-by-time interaction was merely marginally significant, the change was not observed for the control group.

In comparison to the above findings in line with our hypothesis, results of other areas of EF were found nonsignificant and even seemingly contrary to our hypothesis. Particularly, participants in the CATS group showed no improvement in avoiding non-perseverative errors compared to the control group (Table 3). The results further indicated that the CATS group might increase in failure to maintain sets compared to the control (Tables 2 and 3); the CATS group’s score increased from 1.00 to 2.33 (raw score), while no significant simple change was found for the control group.

Finally, two main effects of time on EF were marginally significant and one main effect of time on VABS was found significant. To begin with, the first marginally significant effect of time was on failure to maintain sets, showing that participants’ score generally increased with time (Table 3). The mean scores in Table 2, however, indicate that only the CATS group actually increased, while the control group held still. Further, the time effect was found marginally significant for perseverative errors; again, only the CATS group’s score increased, whereas the control group decreased. Likewise, the six-month follow-up of communication ability showed a general main effect of time and a group-by-time interaction trending toward significance (Table 3), while only showing a corresponding simple effect for the CATS group but not the control (Table 2).

Discussion

We conducted a randomized controlled trial with a small sample of school-aged children with ASD, reporting preliminary evidence that CATS may improve some aspects of the children’s EF. Specifically, EF improvements related to CATS included a reduction of incorrect perseverative behavior, an improvement in shifting between different behavioral principles, and an increase in forming abstract concepts. Results from a six-month follow-up also indicated potential improvements in daily communication ability. Nevertheless, as detailed below, we suggest interpreting the last long-term effect with caution given the small sample size of our follow-up data.

Unanticipated findings

The present research also includes some unanticipated findings. For one, the results did not support a significant difference between the CATS and control groups in the general WCST performance as measured by the number of categories completed. However, given that the index is conceptually single-item and thus statistically insensitive, we believe the measure is less informative than are other more specific indices. Similarly, we did not find a significant intervention effect on the number of trials needed to complete the first category in the WCST. Nevertheless, a closer look at Table 2 revealed that the control group improved only by 12.51% on the index, whereas the CATS group improved by 26.30% (i.e. more than twice the effect for the control group). In other words, the results were still in line with our general prediction yet might fail to be detected because of the current low statistical power. We consequently look forward to seeing future research strengthen our test.

Furthermore, no significant treatment effect was seen on non-perseverative errors in the WCST. We believe this finding can be understood together with the phenomenon that CATS marginally increased failures to maintain sets as well as significantly decreased perseverative errors. Together, these results point to a potential big picture that CATS may reduce the behavioral rigidity and perseveration that characterizes ASD, while doing little about non-perseverative behavior. In line with the conjecture, our participants with ASD showed little-to-no non-perseverative errors in the pre-test and, as such, little to no room to improve in the post-test. By contrast, the children did express difficulty of perseveration before training, which was then significantly alleviated after receiving the CATS intervention. Likewise, the marginal increase in failure to maintain sets could be conceptualized as a side effect of decreased behavioral rigidity that characterizes ASD. Whether this really is the case, of course, requires future investigations.

Finally, we discovered that all children in our study seemed to marginally significantly increase in failure to maintain sets as well as marginally significantly decrease in perseverative errors, and to significantly improve in daily communication. However, regarding the facts that the simple changes of these effects were significant only for the CATS training but not the control and that their respective group-by-time interactions were either significant (i.e. for the two EF measures) or marginally significant (i.e. for communication), we believe these main effects of time are better understood as conditional and happened only to the CATS group. However, it is important to keep in mind that the key interactions of some of these time main effects were only trending to significance. The results should, therefore, be interpreted with caution and verified in future.

Long-Term effects

The results suggest that CATS might provide long-term benefits for daily interpersonal communication. Specifically, we found that the CATS group improved significantly in communication from pre-test to the six-month follow-up; the control group did not, and the difference between both groups in their changes were trending toward significance. However, an important limitation of these results concerns the current low statistical power for detecting intervention effects in the follow-up as well as data representativity. Therefore, these preliminary findings should be taken with caution. Even though we believe the results are promising, future research is needed to validate the efficacy of the CATS intervention on communication ability as well as general daily adaptation.

Limitations and future directions

As in all previous studies, there were several limitations in the present research. For one, children participating in our study were all considered high-functioning (i.e. their non-verbal IQs were higher than 70). Thus, an important area for future inquiry is how CATS may help children with low-functioning ASD. Secondly, it is important to test CATS in age groups other than that of our participants. Specifically, we suggest examining whether CATS can be used by children younger than our school-aged participants, as past research has demonstrated the benefits of early intervention for children with ASD (Corsello, 2005).

In addition, we would like to thank one of the anonymous reviewers of the current paper for pointing out that CATS, as can be seen in Figure 1, was not designed in a colorblindness-friendly manner. The issue will be addressed in future updates of the training program. Similarly, regarding training design, it might be worth pointing out that our participants were trained as pairs. One reason for this was that, based on our experiences, the children would not be able to stay focused for a 50-minute training . In comparison, pairing them into two and having them take turns and watch the partner completing the program might increase their motivation and help them learn about the training tasks while not actually performing the tasks themselves. This design feature also resembled Kenworthy’s (2014) decision to group participants into groups of three to six; that study reported significant intervention effects on EF. However, we acknowledge that we did not systematically investigate the impacts—positive or negative—of grouping trainees. Consequently, even though we believe it helped, we suggest that future research examines the effects of grouping participants in interventions.

Moreover, to strengthen the validity of our research, future investigations should consider having independent administrators for the training and for measures of its effects. Finally, we invite practitioners to upgrade the CATS intervention to make it more applicable to other rehabilitation settings, including clinics and schools. In addition, because the CATS training requires only one computer screen and two control keys, it could be translated into other electronic platforms such as tablets and smartphones. By administering the intervention on mobile devices, CATS may be more intuitive and usable for children and caregivers.

Conclusion

Theory suggests that executive dysfunction may account for the symptoms of ASD (Hill and Frith, 2003). Accordingly, we designed the Comprehensive Attention Training System (CATS), a computerized EF training program for children with ASD, and examined its effects in a small-sampled randomized controlled trial. We recruited and randomly assigned children with ASD aged six to twelve to either a CATS or a story reading-comprehension control group. We report preliminary evidence that CATS may improve these children’s flexibility and communication skills.

Correction Statement

This article has been republished with minor changes. These changes do not impact the academic content of the article.

Disclosure statement

No potential conflict of interest was reported by the author(s).