Abstract
Objective
Executive function deficits are well-established in ADHD. Unfortunately, replicated evidence indicates that executive function training for ADHD has been largely unsuccessful. We hypothesized that this may reflect insufficient targeting, such that extant protocols do not sufficiently and specifically target the neurocognitive systems associated with phenotypic ADHD behaviors/impairments.
Method
Children with ADHD ages 8–12 (M=10.41, SD=1.46; 12 girls; 74% Caucasian/Non-Hispanic) were randomized with allocation concealment to either central executive training (CET; n=25) or newly-developed inhibitory control training (ICT; n=29). Detailed data analytic plans were preregistered.
Results
Both treatments were feasible/acceptable based on training duration, child-reported ease of use, and parent-reported high satisfaction. CET was superior to ICT for improving its primary intervention targets: phonological and visuospatial working memory (d=0.70–0.84). CET was also superior to ICT for improving go/no-go (d=0.84) but not stop-signal inhibition. Mechanisms of change analyses indicated that CET-related working memory improvements produced significant reductions in the primary clinical endpoints (objectively-assessed hyperactivity) during working memory and inhibition testing (indirect effects: β≥−.11; 95%CIs exclude 0.0). CET was also superior to ICT on 3 of 4 secondary clinical endpoints (blinded teacher-rated ADHD symptoms; d=0.46–0.70 vs. 0.16–0.42) and 2 of 4 feasibility/acceptability clinical endpoints (parent-reported ADHD symptoms; d=0.96–1.42 vs. 0.45–0.65). CET-related gains were maintained at 2–4 month follow-up; ICT-related gains were maintained for attention problems but not hyperactivity/impulsivity per parent report.
Conclusions
Results support the use of CET for treating executive function deficits and targeting ADHD behavioral symptoms in children with ADHD. Findings for ICT were mixed at best and indicate the need for continued development/study.
Keywords: ADHD, working memory, inhibitory control, executive function training
Despite the availability of evidence-based pharmacological and psychosocial interventions for children with ADHD (Evans et al., 2018), the current status is that even the best interventions do not ‘normalize’ behavior or provide lasting benefits beyond the active treatment phase for most of these children (Chronis et al., 2003, 2004). According to the clinical model of psychopathology, interventions aimed at improving a disorder’s core psychological/cognitive features should produce the greatest breadth of therapeutic change (Rapport et al., 2001). Those aimed at peripheral symptoms/behaviors, in contrast, should show limited generalization to core features, and minimally affect other peripheral symptoms in the absence of bidirectional or transactional influences (Chacko et al., 2014). In this context, the lack of post-treatment generalization for current evidence-based ADHD treatments may be unsurprising to the extent that incentivized behavioral interventions and psychostimulants temporarily actuate but do not strengthen executive function-supporting cortical structures (Rapport et al., 2013) that are characterized by developmental lags of 3–5 years in pediatric ADHD (Shaw et al., 2007). In contrast, directly targeting impairments in the core executive functions – working memory and inhibitory control (Karr et al., 2018) – appears warranted based on replicated cross-sectional, experimental, and longitudinal evidence suggesting functional and potentially causal links between underlying executive dysfunction and ADHD-related behavioral symptoms and impairments (for review see Rapport et al., 2013). As such, systematically targeting executive dysfunction reflects a theoretically promising method for affecting broad-based behavioral and functional systems to the extent that executive dysfunction reflects a core psychological/cognitive feature for a large proportion of children with ADHD (Kofler et al., 2019).
Unfortunately, compelling and replicated evidence indicates that executive function training for ADHD has been unsuccessful, both in terms of improving the specific executive functions associated with ADHD-related symptoms/impairments (Chacko et al., 2014; Roberts et al., 2016) and producing reductions in ADHD symptoms beyond spurious gains associated with under-controlled and unblinded trials (Cortese et al., 2015; Melby-Lervåg et al., 2016; Sala & Gobet, 2017). Thus, a parsimonious explanation could be that executive functions in ADHD cannot be improved to an extent that translates into meaningful behavioral change. However, this conclusion relies on the assumption that extant protocols sufficiently and specifically target the neurocognitive systems associated with phenotypic ADHD behaviors/impairments. To that end, we undertook careful and integrative investigations that identified theoretical and foundational limitations of extant cognitive training protocols (omitted). Most critically, we identified significant mismatches between these protocols’ intervention targets and the evidence base regarding the specific neurocognitive subcomponents that are (a) impaired in ADHD and (b) linked with ADHD symptoms and/or functional impairments.
Building on these findings, the current study describes the continued development of an adaptive, cognitively-informed suite of neurocognitive training protocols for pediatric ADHD. In our previous report, we described the development of central executive training (CET), which targets the working components of working memory (continuous updating, dual-processing, serial/temporal reordering), and provided evidence that CET produced superior improvements in working memory and objective indices of ADHD-related hyperactivity relative to gold-standard behavioral parent training (omitted).1 Here, we describe the development of inhibitory control training (ICT) and report the findings from an initial randomized controlled trial of CET vs. ICT. Inhibitory control refers to a set of interrelated cognitive processes that underlie the ability to withhold (action restraint) or stop (action cancellation) an ongoing behavioral response (Verbruggen et al., 2013), and is supported by the septo-hippocampal system with associated projections to the inferior frontal cortex (Quay, 1997) and fronto-basal-ganglia circuitry (Aron et al., 2007). Working memory refers to the top-down, active manipulation of information held in short-term memory (Baddeley, 2007), and includes interrelated functions of the mid-lateral prefrontal cortex and interconnected networks (Wager & Smith, 2003).
Targeting Executive Dysfunction in ADHD
The foundational assumption of all cognitive training protocols, including CET and ICT, is that adaptive and repeated training, practice, and feedback will result in meaningful and sustained improvements in neural systems that support the trained abilities (Sala & Gobet, 2017; Shipstead et al., 2012). By extension, these improvements are expected to transfer to other skills and abilities that rely on the same neural networks (Simons et al., 2016). Applying this model to pediatric ADHD, we hypothesized that meaningful changes in neurocognitive functioning – and as an extension, behavioral functioning – are likely to be maximized by targeting specific neurocognitive systems implicated in the disorder’s phenotypic expression (Rapport et al., 2001). That is, we targeted neurocognitive systems that have been shown repeatedly to be both (1) impaired in many children with ADHD, suggesting the need for remediation; and (2) empirically linked with ADHD’s core behavioral symptoms and/or key areas of functional impairment, suggesting the potential for downstream effects on behavior (Kofler et al., 2019).
With regard to the breadth and depth of expected effects, we assumed that CET and ICT would produce larger improvements in proximal vs. more distal outcomes. For example, CET and ICT do not directly train distal outcomes such as academic, organizational, or social skills; rather, interventions that target these distal skills are expected to produce larger benefits following successful remediation of proximal underlying impairments in the neural substrates that support these skills (Chacko et al., 2017). Statistically, the magnitude of improvement on any untrained outcome will be capped by (a) the degree to which training improves the target executive function, and (b) the strength of the association between that executive function and the target outcome (Rapport et al., 2013). In addition, symptom normalization following a 10-week treatment protocol was considered unrealistic given the 3–5 year delayed maturation of cortical structures that support executive functions (Shaw et al., 2007); rather, we expected continued training to produce incremental benefits and/or ‘nudges’ in developmental trajectories that may only be realized over time (Halperin & Healey, 2011). Finally, we assumed that there would be a subset of children with ADHD who would not respond to CET or ICT because deficits in these executive functions do not underlie their behavioral presentation. This prediction was based on the well-documented neurocognitive heterogeneity in ADHD (Coghill et al., 2014; Nigg et al., 2005) and presumed multiple pathways to the ADHD phenotype (Sonuga-Barke et al., 2010). Looking ahead, optimal targeting will require a battery of interventions and personalized medicine approach to address each pathway to ADHD.
Targeting Inhibitory Control Abilities in ADHD
In our previous report, we described the theoretical and empirical basis for developing central executive training (CET) to target central executive working memory deficits in ADHD (e.g., presence of working memory deficits in 62%−85% of children with ADHD; cross-sectional, longitudinal, and experimental links with ADHD inattentive and hyperactive symptoms; associations with peer, academic, and family functioning; omitted). The theoretical basis for targeting inhibitory control is similarly strong. Briefly, inhibitory control has featured prominently in modern theoretical models of ADHD, where it has been hypothesized to produce ADHD behavioral symptoms in its role as the unifying core deficit (Barkley, 1997), as one of multiple causal pathways (Sonuga-Barke et al., 2010), or as a secondary impairment attributed to underlying deficits in state regulation (Sergeant, 2005) and/or working memory (Rapport et al., 2001). Empirically, inhibitory control has been linked with ADHD behavioral symptoms via meta-analytic evidence of medium-to-large magnitude impairments on inhibitory control tests for children with vs. without clinically elevated ADHD symptoms (Alderson et al., 2007). Of note, individual difference studies have produced more mixed results, with several studies reporting links between inhibitory control tests and informant-rated ADHD symptoms (Alderson et al., 2010; Brocki et al., 2010) but other studies failing to find significant associations cross-sectionally or longitudinally (e.g., Karalunas et al., 2017). Studies of neurocognitive heterogeneity indicate that approximately 21%−46% of children with ADHD have inhibitory control deficits (Nigg et al. 2005; Sonuga-Barke et al. 2010), making it an appealing training target for a large proportion of this population.
Notably, there have been several recent attempts to improve inhibitory control in children with ADHD (Azami et al., 2016; Dovis et al., 2015; Halperin et al., 2013; Hoekzema et al., 2010; Johnstone et al., 2010, 2012; Klingberg et al., 2002; Shalev et al., 2007; Rabiner et al., 2010; Tamm et al., 2019; van der Oord et al., 2012). Unfortunately, these efforts have been largely unsuccessful, such that extant studies either reported non-significant changes on all tests of inhibitory control, reported mixed findings (e.g., improvements on some but not most inhibitory control tests), or did not test for improvements in inhibitory control despite describing training that targeted inhibition. Indeed, meta-analytic estimates suggest minimal efficacy for improving inhibitory control in ADHD (d=0.06, ns; Rapport et al., 2013).
A parsimonious conclusion may therefore be that inhibitory control is not amenable to training in ADHD. However, to our knowledge all previous inhibitory control training tasks have been imbedded within larger protocols intended to improve multiple neurocognitive abilities. Although such an approach makes intuitive sense given the well-documented neurocognitive heterogeneity in ADHD (Coghill et al., 2014), meta-analytic evidence indicates that cognitive training protocols for ADHD are significantly less effective when their potency is decreased by targeting multiple neurocognitive functions (Rapport et al., 2013). Thus, the extent to which inhibitory control is amenable to training, and the extent to which these improvements reduce ADHD symptoms, remains unknown.
Current Study
The current study addresses this gap by developing an evidence-informed, adaptive, and specific inhibitory control training (ICT) protocol for children with ADHD and comparing it via randomized controlled trial to a neurocognitive training protocol previously shown to be feasible, acceptable, and efficacious for children with ADHD (Kofler, Sarver et al., 2018). We hypothesized that CET and ICT would be comparable in terms of feasibility/acceptability indicators, including parent and child satisfaction, high completion rates, parent expectancies, and parent-reported ADHD symptom reductions. We further hypothesized that CET would be superior to ICT for improving working memory abilities, whereas ICT would be superior to CET for improving inhibitory control abilities. Finally, we predicted that CET and ICT would both produce reductions in objectively-assessed hyperactivity.
Method
Preregistration, Open Data, and Open Science Disclosure Statement (Simmons et al., 2012)
Primary and secondary outcomes and detailed data analytic plans were preregistered at https://osf.io/abwms. There were no departures from the preregistered plan with one clearly marked exception and additional analyses added during the peer review process. The de-identified raw data (.jasp) and results output (including analysis scripts and test statistics) are available for peer review as recommended (Redick, 2015): https://osf.io/6h5e9/. We report how we determined our sample size, all data exclusions, all manipulations, and all measures in the study.
Study Timeline
We previously reported an initial trial of CET vs. behavioral parent training (omitted); that study was closed to recruitment when the current study’s inhibitory control training software was completed. Recruitment to this initial randomized controlled trial was closed based on our preregistered stopping rule of at least 20 completers per group (Simons et al., 2016). The sample reflects consecutive referrals from March 2017 to March 2019 who had consented or declined the intervention trial when this stopping rule was reached. None of the current study’s families participated in the previous CET vs. behavioral parent training trial.
Randomization, Allocation Concealment, and Blinding
Randomization was conducted by the study methodologist using unpredictable allocation stratified by medication status according to CONSORT guidelines. Study evaluators were blind to treatment group. Data screening, cleaning, and analyses were conducted blind to treatment group/target.
Participants
The CONSORT study flow diagram is shown in Figure 1. As shown in Table 1, the treated sample comprised 54 children aged 8–12 years (M=10.41, SD=1.46; 12 girls) from the Southeastern U.S., consecutively referred to a university-based research clinic through community resources. Psychoeducational evaluations were provided to caregivers. IRB approval was obtained/maintained; all parents/children gave informed consent/assent. Child race/ethnicity was 74% Caucasian/Non-Hispanic, 11% Hispanic, 9% African American, and 6% mixed race/ethnicity. All participants spoke English.
Table 1.
Variable | ICT (n=29) | CET (n=25) | Cohen’s d | BF01 | p | ||
---|---|---|---|---|---|---|---|
M | SD | M | SD | ||||
Gender (Girls/Boys) | 8/21 | 4/21 | -- | 2.25 | .31, ns | ||
Age | 10.07 | 1.46 | 10.23 | 1.39 | −0.10 | 3.39 | .68, ns |
SES | 49.48 | 9.46 | 45.08 | 12.70 | 0.30 | 1.79 | .20, ns |
WISC-V VCI | 105.93 | 12.29 | 103.16 | 12.42 | 0.18 | 2.75 | .42, ns |
Medication (No/Yes) | 20/9 | 15/10 | -- | 2.53 | .49, ns | ||
Race/ethnicity (W/B/H/M) | 20/4/3/2 | 20/1/3/1 | -- | 13.64 | .61, ns | ||
ADHD Presentation (I/H/C) | 9/1/19 | 6/1/18 | -- | 11.14 | .85, ns | ||
Comorbidity (No/Yes) | 16/13 | 11/14 | -- | 9.77 | .79, ns | ||
BASC-3 Attention Problems (T-score) | |||||||
Parent | 67.00 | 7.34 | 69.60 | 6.04 | −0.32 | 1.61 | .17, ns |
Teacher | 65.52 | 7.59 | 63.52 | 6.18 | 0.24 | 2.31 | .30, ns |
BASC-3 Hyperactivity (T-score) | |||||||
Parent | 66.52 | 13.01 | 73.52 | 10.86 | −0.49 | 0.58 | .04 * |
Teacher | 62.45 | 11.09 | 62.84 | 13.80 | −0.02 | 3.62 | .91, ns |
ADHD-RS-5 (T-Score) | |||||||
Attention Problems (Parent) | 67.66 | 5.56 | 69.00 | 4.89 | −0.20 | 2.53 | .35, ns |
Hyperactivity/Impulsivity (Parent) | 64.55 | 8.13 | 67.56 | 4.25 | −0.37 | 1.17 | .10, ns |
Inhibitory Control Performance Data | |||||||
Go/no-go Commission Errors | 3.10 | 2.04 | 2.80 | 2.52 | 0.11 | 3.30 | .62, ns |
Stop-signal Commission Errors | 14.79 | 4.44 | 15.12 | 4.31 | −0.06 | 3.53 | .79, ns |
Stop-Signal Reaction Time (ms) | 318.02 | 93.05 | 337.06 | 96.74 | −0.17 | 2.91 | .47, ns |
Stop-signal Delay (ms) | 272.04 | 65.33 | 265.75 | 55.31 | 0.08 | 3.43 | .71, ns |
Working Memory Performance Data (Stimuli Correct/Trial) | |||||||
Phonological Working Memory | 3.21 | 0.68 | 3.20 | 0.45 | 0.01 | 3.64 | .96, ns |
Visuospatial Working Memory | 2.29 | 0.70 | 2.40 | 0.41 | −0.14 | 3.05 | .51, ns |
Actigraph-measured Hyperactivity | |||||||
Go/no-go Task Hyperactivity (PIM) | 123.12 | 87.91 | 139.76 | 93.98 | −0.15 | 3.02 | .51, ns |
Stop-signal Task Hyperactivity (PIM) | 100.26 | 57.32 | 87.27 | 83.15 | 0.15 | 3.01 | .50, ns |
PHWM Task Hyperactivity (PIM) | 271.46 | 130.58 | 255.61 | 141.10 | 0.09 | 3.37 | .67, ns |
VSWM Task Hyperactivity (PIM) | 187.65 | 113.09 | 167.99 | 89.20 | 0.15 | 2.97 | .49, ns |
Paint Activity Hyperactivity (PIM) | 56.67 | 42.90 | 57.24 | 47.23 | −0.01 | 3.64 | .96, ns |
Note. Raw p-values are presented (uncorrected for multiple comparisons). BASC-3 = Behavior Assessment System for Children (T-scores); BF = Bayes Factor, BF01 is the odds ratio of the evidence favoring the null to the evidence favoring the alternative hypothesis. A value of 1 indicates that the data are equally likely under the null and alternative hypotheses, values >1 favor the null hypothesis that the groups are equivalent, and values ≥3 are considered statistically significant evidence of equivalence. BF10 can be computed as the inverse of BF01 (1/BF01); CET = Central Executive Training; ICT = Inhibitory Control Training; Medication Changes (Stop = Discontinued Medication During Study, No = No Changes Reported, Add = Started Medication During Study); ms = milliseconds; PH = Phonological Working Memory; Race/ethnicity (W = White, B = Black, H = Hispanic/English-Speaking, M = Mixed); Stop-Signal Reaction Time = iSSRT computed using the Verbruggen et al. (2013) integrated method; VCI = Verbal Comprehension Index (IQ; standard scores); VS = Visuospatial Working Memory.
Inclusion/Exclusion Criteria
All families completed a comprehensive evaluation that included detailed semi-structured clinical interviewing (K-SADS; Kaufman et al., 1997) and age/gender norm-referenced parent and teacher ADHD ratings (ADHD-RS-5 and BASC-3; DuPaul et al., 2016; Reynolds & Kamphaus, 2014). Study eligibility required: (1) DSM-5 diagnosis of ADHD (any presentation) by the directing clinical psychologist based on K-SADS (2013 update for DSM-5); and (2) clinical/borderline elevations on at least one parent and one teacher ADHD rating scale (i.e., >90th percentile), or previous psychoeducational evaluation documenting cross-informant symptoms (e.g., for children prescribed medication that reduces ADHD symptoms at school). All children had current impairment per K-SADS. Additional details regarding the psychoeducational evaluation and differential diagnosis process can be found on our preregistration website. Children with scores in the average range or higher on all pretreatment working memory tests were excluded (n=2); no inhibitory control thresholds were set as specified in our NIMH grant.
Comorbidities reflect consensus best estimates, and include anxiety (26%), autism spectrum (17%), and oppositional defiant disorders (7%)2. The ICT and CET groups did not differ in terms of comorbidities overall or within each diagnostic category (all BF01> 2.25, all p>.34). Learning disabilities in reading (n=3 per group; p=.85, ns; BF01 = 4.62) and math (ICT=6, CET=2; p=.19, ns; BF01 = 1.94) were suspected based on score(s) ≥1.5 SD below age-based norms on one or more KTEA-3 core subtests (Kaufman & Kaufman, 2014). The ICT (n=9) and CET groups (n=10) did not differ significantly in the number of children prescribed psychostimulants (p=.49, ns; BF01=2.53) and were equivalent in terms of medication changes during the study (p=.95, ns, BF01=12.70; Table 1).
Children were excluded for gross neurological, sensory, or motor impairment; seizure disorder, psychosis, or intellectual disability; or non-stimulant medications that could not be withheld for testing.
Procedures
Pre-treatment testing occurred during a larger battery of two, 3-hour sessions. Mid- and post-testing occurred during single, 3-hour sessions following treatment weeks 5 and 10, respectively. All tests were counterbalanced within/across sessions and children received preset breaks every 2–3 tasks to minimize order/fatigue effects. Families were not required to withhold psychostimulants prior to child treatment visits. Psychostimulants were withheld ≥24-hours prior to all child assessment sessions. The CET and ICT software was web-based and required a desktop/laptop computer with mouse, keyboard, and Internet access.
Treatments
ICT and CET were delivered identically according to manualized procedures in small group format or individually as needed to accommodate families’ schedules. Schedule changes were accommodated to the extent possible (e.g., make-up sessions the same week). Identical procedures were used for both groups (e.g., 1 hour in-office sessions). The 10-week protocol included weekly in-office sessions with the child (1 hour), combined with parent-supervised, in-home training (goal: 15-min/day, 2–3 days/week).
Active, credible, and adaptive control
Unfortunately, participants cannot be blind to their condition assignment in psychosocial/cognitive training interventions; families spend many hours engaged in the protocol and they know that they have done so (Simons et al., 2016). As reviewed by Simons et al. (2016), ruling out ‘enhanced placebo effects’ requires measurement of expectancies and randomization to an ideal control condition that is identical to the treatment condition in all respects – including adaptive difficulty, active engagement, the need for vigilance and effort, social contact with the researchers, and expectancies for success – except for the critical, ‘active’ ingredient of the treatment. Unfortunately, very few if any extant ADHD cognitive training studies meet these criteria. For example, many studies use passive waitlist controls or describe ‘active’ control conditions that do not adapt in difficulty and thus would not be considered active, credible controls as described above (Simons et al., 2016).
In the current study, CET and ICT served as active, credible, and adaptive controls for each other. Each intervention targets a model-driven, theoretically important neurocognitive process (Barkley, 1997; Rapport et al., 2001) that is impaired in a large proportion of children with ADHD (Sonuga-Barke et al. 2010) and considered a core executive function in children (Karr et al., 2018). Further, CET and ICT include equivalent contact with the research team and feature the same number of distinct training games to create protocols that are as identical as possible except for the intervention target (working memory vs. inhibitory control). For example, each matched pair of ICT/CET training games is identical in terms of website address, name, art, animations, storylines, layouts, interfaces, and use of adaptive training algorithms to maximize internal/construct validity. More generally, best practice guidelines for cognitive training studies were closely followed (Table 2), allowing strong conclusions regarding emerging group differences as a function of training target (Redick, 2015).
Table 2.
Criterion / Commentary | |
---|---|
Best practice recommendations from Simons et al. (2016) | |
✓ | Assess pre-treatment baseline performance for all groups |
The current study used a pre/mid/post test design in which all outcomes were assessed at all three time points. Pre-treatment performance was assessed and controlled when probing between-group differences at post-treatment. | |
✓ | Include an active, credible control group matched for expectancies |
Working memory and inhibitory control are both putative core mechanisms implicated in ADHD and featured in prominent conceptual models of the disorder’s etiology and psychopathology. The two versions are identical in all aspects except the target mechanism, and served as active, credible controls for each other. The groups were identical in terms of expectancies and did not differ in caregiver-reported feasibility or acceptability at post-treatment. | |
✓ | Include at least 20 participants in each treatment arm |
All analyses include ICT n=29 and CET n=25 participants. | |
✓ | Randomly assign children to condition |
Children were randomly assigned using unpredictable allocation concealment. | |
✓ | Pre-register the trial, and explicitly acknowledge departures from pre-registered plan |
The current study’s outcome measures and detailed data analytic plans were pre-registered. Preregistration occurred during data collection and prior to accessing the data. Data analyses were conducted blinded to treatment allocation. | |
✓ | Blind raters for all subjective outcomes measures |
Objective assessments of hyperactivity (actigraphs) during both proximal and distal activities were specified a priori as the primary clinical endpoint for assessing ADHD symptom changes. Blinded teacher ratings served as the secondary clinical endpoints for assessing ADHD symptoms changes. Caregivers were blinded to condition and all caregivers remained blind based on the post-treatment questionnaire. However, caregivers were not blind to the fact that their child was receiving an intervention because they are active participants in both treatments (Simons et al., 2016). Thus, caregiver ratings were treated as secondary outcomes and conceptualized under the feasibility/acceptability umbrella of “perceived efficacy” rather than primary evidence of efficacy (Sonuga-Barke et al., 2013). Meta-analytic evidence indicates that estimates of treatment effects on ADHD symptoms are inflated for unblinded raters vs. blinded raters by d=0.36–0.40 for neurocognitive training studies (Rapport et al., 2013). | |
✓ | Label any analyses conducted after inspecting the data as ‘exploratory’ |
The analyses reported herein did not depart from the preregistered plan with one clearly marked exception that occurred prior to accessing the data, and clearly marked analyses that were added during the peer review process. | |
✓ | Avoid subgroup analyses unless preregistered |
No subgroup analyses were preregistered; therefore, none were conducted. Within-group analyses were limited to planned comparisons to characterize the pattern of change for each group across assessment points. | |
✓ | Identify all outcome data collected, including outcomes not reported herein |
A complete list of data collected for secondary research questions can be found on the study’s OSF preregistration website. | |
Additional recommendations from Redick (2015) | |
✓ | Report full pre-test and post-test means and SDs for all groups |
Pre-treatment and post-treatment means and SDs are shown in Tables 1 and 3, respectively. | |
✓ | Provide full, subject-level data as supplementary material |
JASP (.jasp) and JAMOVI (.omv) data files posted for peer review on the study’s OSF website. | |
✓ | Use likelihood ratios, in particular Bayes Factors |
Traditional p-values are supplemented with Bayes Factors to allow stronger conclusions regarding both between-group equivalence and emerging between-group differences. | |
✓ | Examine outcomes graphically to ensure that the pattern of pre- to post-test change is theoretically consistent with the expected pattern of results |
Graphical representations of study outcomes are shown in Figure 2 and the Supplementary Figures. |
Adaptive Training
ICT and CET are both translational, evidence-informed, hybrid (in-office and at-home), and software-based treatment protocols that include gaming elements (Prins et al., 2011) and an automated token economy to reinforce training goals and improve player engagement. They were designed as competence-oriented trainings in which the child’s basal level is established and they are trained up from there, thus ensuring that each child is constantly working within their zone of proximal development (“flow state” in the serious games literature; Canon-Bowers & Bowers, 2010). Each CET and ICT training game includes hundreds of levels that dynamically and incrementally adjust multiple parameters based on iterative changes from extensive testing. These parameters incrementally increase demands on their target processes and are dependent on training target. For example, ICT tasks train the ‘action restraint’ and ‘action cancellation’ components of inhibitory control by dynamically adapting on go:stop target ratio, presentation rate, response speed (timers), and number of stimuli (Alderson et al., 2007). Stretching the target density (i.e., increasing the proportion of ‘go’ trials) increases inhibition demands by increasing prepotency, which makes it more difficult to inhibit during infrequently-occurring ‘stop’ trials (Engle & Kane, 2003). Similarly, dynamically changing targets from ‘go’ to ‘no go’ engages action preparation processes to maximize targeting of the action cancellation component of inhibitory control. Please see Kofler, Sarver, et al. (2018) for a detailed description of CET’s adaptive components and emphasis on training the ‘working’ rather than short-term memory components of working memory.
Maximizing dosage
Targeting multiple neurocognitive systems within a single intervention protocol reduces potency and thus limits efficacy (i.e., dividing training time across more targets = lower dosage per target), as shown in recent ADHD cognitive training meta-analyses (Rapport et al., 2013). Thus, CET and ICT were developed as distinct, yet complimentary, interventions. To ensure breadth of training, both treatments feature a ‘Mission Mode’ that automatically selects games that the child has not completed recently. They also feature an identical, automated token economy that awards ‘tickets’ for successful performance during each game, for completing each game, and for completing the daily Mission Mode. These tickets are exchanged for tangible prizes during the weekly in-office sessions.
Parent check-ins
The weekly parent check-ins occurred in a separate room from the child in-office training session, led by PhD- or Master’s-level study therapists (LJS, ELW). Parent check-ins were intended to promote adherence and troubleshoot difficulties with the at-home training (e.g., demonstrating login procedures, brainstorming feasible days/times for the child to complete training). Importantly, no active treatment components are included in the parent check-ins.
Measures
Intellectual Functioning (IQ) and Socioeconomic Status (SES) at Pre-Treatment
IQ was estimated using the WISC-V Verbal Comprehension Index (Wechsler, 2014). Hollingshead (1975) SES was estimated based on caregiver(s)’ education and occupation.
Feasibility, Acceptability, and Usability Outcomes
Client Satisfaction Questionnaire (CSQ-8; Nguyen et al., 1983)
The CSQ-8 is an extensively studied, 8-item generic measure of client perceptions regarding the value of services received. Parents completed the CSQ-8 at post-treatment. Higher mean scores indicate higher satisfaction (range=1–4).
System Usability Scale (SUS; Canon-Bowers & Bowers, 2010)
The SUS is a 10-item, item response theory-developed scale assessing ease of use. Children completed the SUS at post-treatment. Higher scores indicate greater usability (range = 0–100).
NICT Expectations of Cognitive Training (ECT; Rabipour et al., 2015)
The ECT is a 7-item scale completed by parents at mid-treatment to assess the extent to which they expect cognitive training to improve their child’s functioning. Higher mean scores indicate higher expectancies (range=1–7).
Training Duration
The CET/ICT software records training duration for each completed training game (time spent actively engaged); total minutes trained is reported.
Subjective ADHD symptom changes
Parents were informed that their child would be randomized into one of two executive function interventions. All parents remained blind in this way based on a study-created post-treatment blinding questionnaire. However, as described above parents could not be blind to the fact that their child was receiving an intervention, or to the specifics of the intervention their child received, because they were active participants (e.g., facilitating at-home training; Simons et al., 2016). Thus, as in our previous trial (omitted), parent ratings were treated as secondary outcomes and conceptualized under the feasibility/acceptability umbrella rather than as primary efficacy outcomes. Parent-reported Attention Problems and Hyperactivity/Impulsivity were assessed via age- and gender-normed T-scores on the BASC-3 (Reynolds & Kamphaus, 2004) and ADHD-RS-5 (DuPaul et al., 2016). Blinded teacher pre/post ratings on these measures were also collected as described below. Higher scores indicate greater symptom quantity/severity.
Primary Intervention Targets
Please see Kofler, Irwin et al. (2019) for detailed descriptions and psychometric support for each task with children with ADHD in the target age range.
Go/no-go (inhibitory control)
Children were instructed to quickly click a mouse button each time a vertical rectangle appeared, but to avoid clicking the button when a horizontal rectangle appeared. A ratio of 80:20 go:no-go stimuli was selected to maximize prepotency (Kane & Engle, 2003). Children completed 4 continuous blocks of 25 trials each. Commission errors reflect failed inhibitions and served as the primary index of inhibitory control. Lower scores indicate better inhibitory control.
Stop-signal (inhibitory control)
Children were instructed to quickly press ‘X’ or ‘O’ response buttons each time an X or O appeared, respectively, but to withhold responding when they heard an auditory ‘stop signal’ (25% of trials). Children completed 4 consecutive blocks of 32 trials each (8 adaptive stop-trials per block). Commission errors were preregistered as the primary outcome to match the outcome measure from the go/no-go task. Lower scores indicate better inhibitory control. Traditional metrics including stop-signal reaction time (iSSRT computed using the Verbruggen et al., 2013 integrated method) and stop-signal delay (SSD) are also reported.
Phonological and visuospatial reordering (working memory)
The phonological task involved mentally reordering and verbally recalling a jumbled series of sequentially presented numbers and letters (e.g., 4H62 is correctly recalled as 246H). The visuospatial task involved mentally reordering a sequentially presented series of spatial locations based on what color dot appeared in each location and responding on a modified keyboard. Children completed two 12-trial blocks per task (3–6 stimuli per trial). Higher scores (stimuli correct per trial) reflect better working memory.
Primary and Secondary Clinical Endpoints
Objective measurement
Objectively-assessed hyperactivity (actigraphy) was preregistered as the primary clinical endpoint. Micro Motionlogger actigraphs (Ambulatory Monitoring, 2014) are acceleration-sensitive devices that sample movement intensity 16 times/second (16 Hz). The reliability for actigraphs placed at the same site on the same person ranges from .90 to .99 (Tryon et al., 1991). Actigraphs show expected levels of correspondence with parent- and teacher-reported hyperactivity (r=.32-.58), have superior predictive validity relative to rating scales for differentiating children with all ADHD subtypes/presentations (including Predominantly Inattentive) from neurotypical and clinical control children at both the group and individual levels, and outperform other mechanical devices for differentiating ADHD from Non-ADHD groups (for review, see Kofler et al., 2016). Actigraphs were placed on the child’s non-dominant wrist and both ankles. Total hyperactivity scores (THS) were computed by summing activity level across the three actigraphs, separately for activity level during each of the four primary outcome tests described above, as well as during a computerized painting activity that occurred as the last task of each testing session (Rapport et al., 2009).3
Subjective measurement
Blinded pre/post teacher-reported ADHD symptoms on the BASC-3 and ADHD-RS-5 (described above) served as secondary clinical endpoints. These analyses were not preregistered for this study but were added during the peer review process; results should therefore be considered exploratory.
Bayesian Analyses
Traditional null hypothesis significance tests (p-values) were supplemented with Bayes Factors as recommended (Redick, 2015). Bayes Factors were added because they allow stronger conclusions by estimating the magnitude of support for both the alternative and null hypotheses (Rouder & Morey, 2012). BF10 is the Bayes Factor (BF) indicating how much more likely the alternative hypothesis (H1) is relative to the null hypothesis (H0). Values ≥3.0 are considered moderate support for the alternative hypothesis (Wagenmakers et al., 2016). BF01 is the inverse of BF10 (i.e., BF01=1/BF10), and is reported when the evidence favors the null hypothesis (Rouder & Morey, 2012). BF01 is interpreted identically to BF10 (≥3=moderate, >10=strong, >100=decisive evidence that ICT and CET are equivalent on an outcome). Both p-values and Bayes Factors are reported. We refer to findings of BF10 ≥ 3 as significant evidence for an effect (i.e., support for the alternative hypothesis of an effect at/above pre-specified evidentiary thresholds), and findings of BF01 ≥ 3 as significant evidence against an effect (i.e., support for the null hypothesis of no effect at/above pre-specified evidentiary thresholds). We refer to effects as ‘marginally significant’ when results indicate p<.05 but BF10 < 3.0 (i.e., when the effect is supported by null hypothesis testing but the Bayes Factor suggests evidentiary value below our prespecified threshold).
Data Analysis Overview
Data analyses were conducted with default JZS prior scales using JASP 0.10 (JASP Team, 2019) according to the preregistered plan. We initially compared pre-treatment characteristics of treated vs. untreated children with ADHD to probe the representativeness of our treatment sample. We then compared the ICT and CET groups on pre-treatment characteristics, study retention, and feasibility/ acceptability outcomes. Finally, ICT and CET were compared for effects on the primary intervention targets (working memory, inhibitory control) as well as on primary and secondary clinical endpoints (objective and subjective ADHD symptom assessments). These analyses involved residual gain scores (i.e., post-treatment covaried for pre-treatment) and group x outcome x time mixed-model ANOVAs, with post-hocs following significant interactions and preregistered planned contrasts to characterize the pattern of change over time separately for each group. Two measures for each outcome were used to maximize power and strengthen interpretation (Shipstead et al., 2012). Exploratory analyses were added to address the mechanisms of change, and involved computing changes in working memory, inhibitory control, and objectively-assessed hyperactivity across pre-mid-post (simple slopes) and analyzing bivariate correlations and formal tests of mediation.
Results
Power Analysis
Our sample size was determined by our preregistered stopping rule (detailed above), which was in turn determined by best practice recommendations for cognitive training studies (Simons et al., 2016). Power analysis using G*Power 3.1 (Faul et al., 2007) indicated that for α=.05 and β=.80, our N=54 is powered to detect within-subject effects of time at d≥.34, treatment x time interactions of d≥.34, and between-group effects of d≥.64. For the mechanism of change analyses, our N is powered to reliably detect two-tailed bivariate correlations of r>.36. Finally, bias-corrected bootstrapped mediation requires N=54 to detect significant mediation effects assuming large effects of the intervention on the intervention target (a pathway) and medium relations between the intervention target and the outcome (b pathway; Fritz & MacKinnon, 2007). These assumptions were considered reasonable given evidence that (a) CET produces large improvements in working memory (d=1.06; Kofler. Sarver et al., 2018); and (b) working memory abilities and actigraph-measured hyperactivity show medium-to-large associations (r=.50-.57; Rapport et al., 2009). Thus, the study is sufficiently powered to address its primary aims.
Treated vs. Untreated ADHD Samples: Pre-Treatment Characteristics
As shown in Figure 1, we recruited 62 children who met ADHD diagnostic criteria. Of these 62 children, 54 (87%) received treatment, 6 (10%) declined treatment, and 2 (3%) were ineligible because they demonstrated intact executive functioning. There were no significant differences between treated (n=54) and untreated (n=8) children with ADHD on any of the pre-treatment variables listed in Table 1 (all BF01>1.25, all p>.20). Untreated children were not followed beyond the pre-treatment evaluation.
ICT vs. CET Samples: Pre-Treatment Characteristics
As shown in Table 1, the ICT and CET groups did not differ demographically at pre-treatment (all BF01>1.79, p>.20). Thus, no covariates were included in the primary analyses.
Study Retention
Study retention was high for both ICT and CET. Notably, 92%−93% of children in both groups completed at least the mid-treatment testing, regardless of completer/non-completer status. Post-treatment completion was 83% for ICT and 88% for CET. Completers attended a minimum of 7 treatment sessions (89% completed all 10 sessions).
Outlier and Missing Data Handling
Outliers ≥3.0 SD were winsorized relative to the within-group distribution (ICT: 0.9% of data points, CET: 1.0% of data points). Missing data rates were low (1.8%), and Little’s MCAR test indicated that these data were missing completely at random (p=.99). Missing data were therefore imputed using the preregistered plan (expectation-maximization based on all available data).
Feasibility, acceptability, expectancies, and parent-reported ADHD symptom changes
NICT expectancies, CSQ-8, SUS, and engagement
ICT and CET were equivalent in terms of parent expectancies for success (all p>.77, BF01>3.29), with mean scores reflecting expectations that treatment will be “somewhat successful.” ICT and CET did not differ in parent-reported post-treatment satisfaction (p=.22, BF01=1.90), with mean scores indicating “good” to “excellent” service. Children in both groups rated the software as easy to use and did not differ in total training time (Table 3a).
Table 3a.
Variable | ICT (n=29) | CET (n=25) | Effect size η2p | Cohen’s d | BF01 | p | ||
---|---|---|---|---|---|---|---|---|
M | SD | M | SD | |||||
Medication Changes (Stop/No/Add) | 1/20/8 | 1/18/6 | -- | -- | 12.70 | .95, ns | ||
Caregiver Satisfaction (CSQ-8) | 3.63 | 0.44 | 3.47 | 0.54 | 0.28 | 1.90 | .22, ns | |
NICT Expectancies Questionnaire (Mean Scores) | ||||||||
Overall Expectancies | 4.65 | 1.24 | 4.61 | 0.83 | .000 | 0.03 | 3.38 | .89, ns |
Concentration/distractibility expectancies | 4.64 | 1.33 | 4.54 | 0.92 | .001 | 0.06 | 3.30 | .77, ns |
Cognitive abilities expectancies | 4.76 | 1.20 | 4.67 | 0.84 | .001 | 0.07 | 3.29 | .77, ns |
Training Time (minutes) | 495.02 | 235.72 | 658.32 | 389.04 | .044 | −0.43 | 0.84 | .06, ns |
System Usability Scale | 82.14 | 15.90 | 73.50 | 17.84 | .046 | 0.44 | 0.88 | .07, ns |
BASC-3 Attention Problems (parent T-score) | 64.10 | 7.23 | 63.72 | 7.82 | .004 | 0.13 | 1.55 | .14, ns |
BASC-3 Hyperactivity (parent T-score) | 58.72 | 10.65 | 62.20 | 10.58 | .001 | 0.06 | 3.51 | .84, ns |
ADHD-RS-5 Attention Problems (parent T-score) | 61.72 | 6.09 | 61.32 | 7.34 | .010 | 0.20 | 3.05 | .47, ns |
ADHD-RS-5 Hyperactive/Impulsive (parent T-score) | 59.34 | 8.17 | 61.08 | 6.66 | .000 | 0.02 | 3.41 | .89, ns |
Note. Effect sizes and statistical tests reflect control for pre-treatment scores on the same measure for BASC-3, ADHD-RS-5, and CSI-IV. Training time is measured by the CET/ICT software as time spent actively playing the training games. BF = Bayes Factor; CET = Central Executive Training; ICT = Inhibitory Control Training; Medication Changes (Stop = Discontinued Medication During Study, No = No Changes Reported, Add = Started Medication During Study).
BASC-3 parent-reported ADHD symptoms
Controlling for pre-treatment scores, ICT and CET did not differ significantly at post-treatment in terms of parent-reported Hyperactivity/Impulsivity (d=0.06, p=.84, BF01=3.51) or Attention Problems (d=0.13, p=.14, BF10=1.14). The group (ICT, CET) x symptom domain (Hyperactivity/Impulsivity, Attention Problems) x time (Pre, Mid, Post) mixed-model ANOVA was significant for main effects of time (p<.001; BF10=6.93 × 109) and symptom domain (p=.14; BF10=20.89), as well as the symptom x time (p<.001; BF10=12.54) and treatment x symptom interactions (p=.09; BF10=6.93); the main effect of treatment did not reach significance (p=.08; BF10=2.96). Post-hocs for the significant interactions indicated that the CET group had marginally higher pre-treatment Hyperactivity/Impulsivity symptoms (d = −0.49, p=.04, BF10=1.72); the groups’ equivalence at post-treatment was attributable to CET producing larger pre-post Hyperactivity/Impulsivity reductions (d=1.42; p<.001; BF10=2.95 × 105) than ICT (d=0.65; p<.001; BF10=81.42). A similar but less pronounced pattern emerged for Attention Problems: The CET and ICT groups did not differ at pre-treatment (d = −0.32, p=.17, BF01=1.61) but the CET group again showed larger pre-post improvements (CET: d=0.96; p<.001; BF10=1.64 × 103 vs. ICT: d=0.45; p=.006; BF10=7.35) (Figure S1).
ADHD-RS-5
Controlling for pre-treatment scores, ICT and CET were equivalent at post-treatment for Hyperactivity/Impulsivity (d=0.02; p=.89; BF01=3.41) and Attention Problems (d=0.20; p=.47; BF01=3.05). The group (ICT, CET) x symptom domain (Hyperactivity/Impulsivity, Attention Problems) x time (Pre, Mid, Post) mixed-model ANOVA was significant for main effects of symptom domain (p=.02; BF10=26.53) and time (p<.001; BF10=1.84 × 1017) only. Planned contrasts indicated that the CET group showed large magnitude pre-post improvements in Hyperactivity/Impulsivity (d=0.99; p<.001; BF10=2.27 × 103) and Attention Problems (d=1.06; p<.001; BF10=5.40 × 103); similarly, the ICT group showed medium-to-large magnitude pre-post improvements in Hyperactivity/Impulsivity (d=0.70; p<.001; BF10=178.39) and Attention Problems (d=0.94; p<.001; BF10=5.13 × 103) (Figure S1).
Primary Intervention Targets: Near- and Far-Transfer Effects on Executive Functioning Abilities
Inhibitory control
Controlling for pre-treatment scores, CET was superior to ICT at post-treatment for reducing go/no-go commission errors (d=0.84; p=.004; BF10=18.50), whereas there was no evidence for differential treatment-related reductions in stop-signal commission errors (d=0.41; p=.18; BF01=1.72) (Table 3b).4 The group (ICT, CET) x task (Go/No-Go, Stop-Signal) x time (Pre, Mid, Post) mixed-model ANOVA was significant for main effects of time (p<.001; BF10=1.63 × 104) and task (p<.001; BF10=2.68 × 1014), and the task x time interaction (p<.001; BF10=21.14), and was marginally significant for the treatment x time interaction (p=.03; BF01=2.94). Post-hocs for the interactions indicated significant evidence for improved go/no-go inhibitory control in the CET group (d=0.47; p=.008; BF10=6.47) despite evidence against improvements for the ICT group (d=0.10; p=.64; BF01=6.54). In contrast, the ICT group showed large pre-post reductions in stop-signal commission errors (d=1.12; p<.001; BF10=1.20 × 104) relative to small pre-post improvements for the CET group (d=0.38; p=.01; BF10=3.06) (Figure 2).
Table 3b.
Variable | ICT (n=29) | CET (n=25) | Effect size η2p | Cohen’s d | BF10 | p | ||
---|---|---|---|---|---|---|---|---|
M | SD | M | SD | |||||
Go/no-go Commission Errors | 3.28 | 2.25 | 1.60 | 1.68 | .15 | 0.84 | 18.50 | .004 |
Stop-signal Commission Errors | 12.79 | 4.72 | 11.56 | 5.10 | .04 | 0.41 | 0.58 | .18, ns |
Stop-Signal Reaction Time (ms) | 270.20 | 74.65 | 280.10 | 84.06 | .000 | 0.02 | 0.28 | .88, ns |
Stop-signal Delay (ms) | 301.00 | 62.37 | 315.30 | 61.37 | .03 | 0.35 | 0.50 | .21, ns |
Phonological Working Memory | 3.35 | 0.76 | 3.70 | 0.42 | .14 | 0.81 | 6.92 | < .001 |
Visuospatial Working Memory | 2.57 | 0.96 | 3.17 | 0.77 | .11 | 0.70 | 3.93 | .01 |
Go/no-go Task Hyperactivity (PIM) | 153.10 | 111.17 | 120.00 | 81.18 | .05 | 0.48 | 0.88 | .09, ns |
Stop-signal Task Hyperactivity (PIM) | 106.10 | 91.74 | 117.20 | 85.44 | .01 | 0.21 | 0.34 | .46, ns |
PHWM Task Hyperactivity (PIM) | 250.80 | 160.23 | 190.40 | 79.38 | .05 | 0.48 | 0.85 | .11, ns |
VSWM Task Hyperactivity (PIM) | 193.50 | 125.40 | 157.10 | 103.80 | .02 | 0.31 | 0.39 | .37, ns |
Baseline Activity Hyperactivity (PIM) | 61.68 | 55.94 | 48.09 | 37.45 | .02 | 0.31 | 0.46 | .27, ns |
BASC-3 Attention Problems (teacher T-score) | 59.38 | 8.27 | 54.71 | 6.30 | .09 | 0.63 | 0.54 | .03 |
BASC-3 Hyperactivity (teacher T-score) | 56.23 | 7.34 | 51.55 | 9.18 | .12 | 0.58 | 0.48 | .03 |
ADHD-RS-5 Attention Problems (teacher T-score) | 56.10 | 10.80 | 49.15 | 12.93 | .12 | 0.66 | 0.25 | .01 |
ADHD-RS-5 Hyperactive/Impulsive (teacher T-score) | 55.83 | 9.15 | 51.37 | 7.43 | .07 | 0.52 | 0.65 | .06, ns |
Note. Effect sizes and statistical tests reflect control for pre-treatment scores on the same measure (residualized gain scores). Partial eta-squared indicates the percent of variance in post-treatment scores explained by treatment group after accounting for pre-treatment scores (interpreted as small = .01; medium = .06; large = .13); BF01 can be computed as the inverse of BF10 (1/BF01). BF = Bayes Factor; CET = Central Executive Training; ICT = Inhibitory Control Training; PH = Phonological Working Memory (Stimuli Correct/Trial); PIM = proportional integrating measure (assesses movement intensity) VS = Visuospatial Working Memory (Stimuli Correct/Trial).
Working memory
Controlling for pre-treatment scores, CET was superior to ICT at post-treatment for improving PHWM (d=0.81; p<.001; BF10=6.92) and VSWM (d=0.70; p=.01; BF10=3.93) (Table 3b). The group (ICT, CET) x task (PHWM, VSWM) x time (Pre, Mid, Post) mixed-model ANOVA was significant for main effects of time (p<.001; BF10=5.52 × 107), treatment (p=.09; BF10=7.24), and task (p<.001; BF10=1.69 × 1014), as well as the treatment x time (p=.005; BF10=11.05; d=0.65) and treatment x task interactions (p=.09; BF10=3.18). Post-hocs for the interactions indicated that the CET group showed large pre-post improvements in PHWM (d=1.25; p<.001; BF10=4.91 × 104) and VSWM (d=0.96; p<.001; BF10=1.67 × 103). In contrast, the ICT group showed small pre-post improvements in VSWM (d=0.41; p=.01; BF10=4.31) but no evidence for improvements in PHWM (d=0.26; p=.08; BF01=1.05) (Figure 2).
Summary of effects on primary intervention targets
Taken together, the objective testing provided strong support for near-transfer effects of CET but mixed evidence for ICT. That is, CET was superior to ICT and evoked large magnitude improvements on both tests of working memory (d=0.70–0.81). CET also demonstrated evidence for far-transfer effects, with improvements on the go/no-go inhibition test that were superior to ICT (d=0.84). There was also potentially evidence for cognitive far-transfer on the stop-signal, where CET and ICT generally made equivalent gains (d=0.38–1.22 for CET vs. d=0.38–1.12 for ICT across stop-signal metrics); however, because gains on this task were equivalent across groups, the possibility that they reflect practice/test-retest effects rather than far-transfer effects cannot be ruled out.
Primary Clinical Endpoints: Far-Transfer Effects on Objective Behavioral Indicators
Actigraphs during inhibitory control testing
Controlling for pre-treatment scores, ICT and CET differences did not reach significance in terms of reducing post-treatment hyperactivity during go/no-go (d=0.48; p=.09; BF01= 1.14) or stop-signal testing (d=0.21; p=.46; BF01=2.94). The group (ICT, CET) x task (Go/No-Go, Stop-Signal) x time (Pre, Mid, Post) mixed-model ANOVA was significant for the main effect of task (p<.001; BF10=925.86) as well as marginal support for the treatment x time x task interaction (p=.04; BF01=1.38). Post-hocs for the interaction indicated evidence for reductions in hyperactivity only for the CET group and only during the go/no-go task. That is, there was marginal evidence that the CET group showed reductions in hyperactivity during go/no-go testing between pre- and mid-treatment (d=0.32; p=.05; BF10=1.30), but this effect was not detectable at post-treatment (d=0.23; p =.16; BF01=1.80). There was no evidence to suggest CET pre-post reductions in hyperactivity during stop-signal testing (d=0.06; p =.96; BF01=2.55). There was significant evidence against pre-post reductions in hyperactivity for the ICT group during both go/no-go (d=0.06; p =.94; BF01=12.09) and stop-signal testing (d=0.10; p=.63; BF01=6.37) (Figure S2).
Actigraphs during working memory testing
Controlling for pre-treatment, ICT and CET differences did not reach significance in terms of reducing post-treatment hyperactivity during PHWM (d=0.48; p=.11; BF01=1.18) or VSWM testing (d=0.31; p=.37; BF01=2.56). The group (ICT, CET) x task (PHWM, VSWM) x time (Pre, Mid, Post) mixed-model ANOVA was significant for task (p<.001; BF10=1.36×106) and time (p=.003; BF10=99.94), and marginal for the task x time interaction (p=.03; BF10=1.01). Post-hocs for the interaction indicated that CET produced significant pre-post hyperactivity reductions during PHWM testing (d=0.47; p=.007; BF10=6.94); this reduction was not observed for ICT (d=0.17; p=.27; BF01=2.93). There was marginal evidence for pre-mid CET hyperactivity reductions during VSWM testing (d=0.31; p=.05; BF10=1.33), but these gains were not detected at post-treatment (d=0.17; p=.30; BF01=3.00). There was evidence against ICT hyperactivity reductions during VSWM testing (d=0.10; p=.60; BF01=6.13).
Actigraphs during painting activity
Controlling for pre-treatment, ICT and CET did not differ significantly in terms of their effects on reducing post-treatment hyperactivity (d=0.31; p=.37; BF01=2.17). The group (ICT, CET) x time (Pre, Mid, Post) mixed-model ANOVA did not produce any significant effects (all p>.26; BF01>2.06). Planned contrasts indicated marginal support for pre-mid hyperactivity reductions for the CET group (d=0.37; p=.03; BF10=2.30), but these gains were not detectable at post-treatment (pre-post d=0.22; p=.17; BF01=1.87). There was significant evidence against hyperactivity reductions in the ICT group during paint (d=0.09; p=.68; BF01=7.03).
Summary of effects on primary clinical endpoints
Taken together, there was support for behavioral far-transfer effects of CET but not ICT. CET was superior to ICT for producing reductions in hyperactivity during one of the two working memory tests (d=0.47 for CET vs. d=0.17 for ICT) but failed to maintain initial pre-mid hyperactivity reductions during the other working memory test (d=0.31 for CET vs. d=0.10 for ICT). In contrast, there was generally significant evidence against behavioral far-transfer effects of ICT during inhibition testing. Finally, there was marginal support for more distal behavioral far-transfer effects of CET, but it was limited to pre-mid changes during two of the three non-working-memory tasks/activities (d=0.32–0.37 for CET vs. d=0.06–0.09 for ICT). In contrast, there was significant evidence against distal far-transfer effects for ICT.
Secondary Clinical Endpoints: Far-Transfer Effects on Blinded Teacher-Reported ADHD Symptoms
Teacher report data from the BASC-3 and ADHD-RS-5 were collected at pre- and post-treatment (Figures S6–S7, Supplementary Online Materials). These analyses were not preregistered for the current study but were added during the peer review process; results should therefore be considered exploratory. Reporting is truncated for readability; please see the Supplementary Online Materials for full reporting.
Controlling for pre-treatment scores, CET was superior to ICT at post-treatment in terms of teacher-reported ADHD-RS-5 Attention Problems (d=0.66; p=.01; BF10=3.96) and marginally superior to ICT at post-treatment in terms of teacher-reported BASC-3 Attention Problems (d=0.63, p=.03, BF10=1.87) and BASC-3 Hyperactivity/Impulsivity (d=0.58, p=.03, BF10=2.07); this contrast did not reach significance for ADHD-RS-5 Hyperactivity/Impulsivity (d=0.52; p=.06; BF10=1.54). The group (ICT, CET) x symptom domain (Hyperactivity/Impulsivity, Attention Problems) x time (Pre, Post) mixed-model ANOVAs were both significant for main effects of time (p<.001; BF10>7.23 × 103); the ADHD-RS-5 model was also significant for the treatment x time interaction (p=.01; BF10=3.98). Post-hocs for the interaction indicated that the CET group showed significant pre-post improvements in ADHD-RS-5 Hyperactivity/Impulsivity (d=0.46; p=.02; BF10=23.82) and Attention Problems (d=0.68; p<.001; BF10=98.32). In contrast, the ICT group failed to show pre-post improvements in ADHD-RS-5 Hyperactivity/Impulsivity (d=0.23; p=.99; BF01=1.11) or Attention Problems (d=0.16; p=.99; BF01=2.57) based on teacher report (Figure S6, bottom). Planned contrasts for the BASC-3 model revealed a similar but less pronounced pattern: The CET group demonstrated significant reductions in BASC-3 Hyperactivity/Impulsivity (d=0.70, p<.001, BF10=303.82) and Attention Problems symptoms (d=0.55, p=.002, BF10=673.21) between pre- and post-treatment. The ICT group also demonstrated significant reductions in BASC-3 Hyperactivity/Impulsivity (d=0.42, p=.05, BF10=3.75) and Attention Problems symptoms (d=0.41, p=.06, BF10=16.32) between pre- and post-treatment (Figure S6, top).
Summary of effects on secondary clinical endpoints
Taken together, there was stronger support for behavioral far-transfer effects for CET than for ICT. CET was superior to ICT for producing reductions in teacher-reported ADHD symptoms on 3 of the 4 measures (d=0.52–0.66). In addition, the CET group demonstrated significant pre-post reductions across all 4 measures (d=0.46–0.70), whereas the ICT group showed significant reductions on both BASC-3 subscales (d=0.41–0.42) but failed to show reductions on either of the ADHD-RS-5 scales (d=0.16–0.23).
Exploratory Analyses: Mechanisms of Change
Exploratory analyses were conducted to test the mechanisms of change (Figures S3–S4). This involved computing simple slopes that indexed overall change for each participant across the 3 time points as recommended (Sarver et al., 2015), averaged separately to estimate changes in working memory abilities, inhibitory control abilities, and objectively-assessed hyperactivity during working memory testing, inhibitory control testing, and the painting activity. Two sets of analyses were run: bivariate correlations between changes in each executive function and changes in objectively-assessed hyperactivity during each test/activity, and formal tests of mediation.
Bivariate results indicated that working memory improvements were related to reductions in objectively-measured hyperactivity during working memory testing (proximal far transfer; r= −.31, p=.01, BF10=4.56), inhibitory control testing (distal far transfer; r= −.28, p=.02, BF10=3.11), and marginally during the painting activity (distal far transfer; r= −.23, p=.04, BF10=1.59). In contrast, there was no evidence linking inhibitory control improvements with reductions in hyperactivity during inhibitory control testing (proximal far transfer; r=.15, p=.13, BF01=1.55), working memory testing (distal far transfer; r=.14, p=.16, BF01=1.76), or during the painting activity (r=.12, p=.20, BF01=2.14).
To test for formal mediation, we used bias-corrected bootstrapped mediation analyses with 5,000 resamples as implemented in Jamovi 1.0 (Jamovi Project, 2019), with treatment (CET, ICT) as the independent variable, changes in working memory or inhibitory control as the mediator, and changes in objectively-measured hyperactivity as the outcome. Separate models were run for each executive function/behavior combination; reporting is truncated for readability. Significant effects are indicated by 95% bootstrapped confidence intervals that exclude 0.0 (Hayes, 2009; Shrout & Bolger, 2002).
Results indicated that CET produced greater improvements in working memory (a pathway; β=.38, 95%CI excludes 0.0), and that greater improvements in working memory predicted greater reductions in objectively-measured hyperactivity (b pathways) during working memory testing (β= −.30, 95%CI excludes 0.0) and inhibitory control testing (β= −.29, 95%CI excludes 0.0). Critically, working memory improvements significantly mediated the link between CET and objectively-assessed hyperactivity reductions (ab pathways) during working memory testing (β= −.11) and inhibitory control testing (β= −.11, both 95%CIs exclude 0.0), such that the residual direct effects of CET on reductions in hyperactivity were nonsignificant (c’ pathways; both 95%CIs include 0.0). In other words, the effects of CET on reducing objectively-assessed hyperactivity during executive function testing were fully carried by CET’s impact on improving working memory abilities (effect ratios ≥ .82; Figure S5). There was no evidence of direct effects or mediation for hyperactivity reductions during painting (both 95%CI included 0.0).
CET treatment also produced greater improvements in inhibitory control (a pathway; β=.30, 95%CI excludes 0.0); however, there was no evidence to link improvements in inhibitory control with changes in objectively-assessed hyperactivity (b pathways; all 95%CIs include 0.0), and no evidence for mediation via changes in inhibitory control abilities (ab pathways; all 95%CIs include 0.0).
Combined with the primary analyses above, these findings provide additional evidence that CET improves its intended mechanism (working memory), and that these improvements produce, to a significant extent, reductions in objectively-measured hyperactivity as hypothesized. The evidence was mixed at best for ICT, with mixed evidence that it engages its target mechanism (inhibitory control) and no evidence linking improvements in this mechanism with objectively-measured hyperactivity.
Maintenance of Effects
Additional analyses were conducted to probe for maintenance of effects. These analyses were not preregistered for the current study but were added during the peer review process; results should therefore be considered exploratory. Reporting is truncated for readability; please see the Supplementary Online Materials for full reporting. Parent-reported ADHD symptoms (BASC-3 and ADHD-RS-5) were obtained at 2–4 month follow-up (M=77 days; ICT and CET were equivalent in follow-up duration, BF01 = 3.13, p=.93). We repeated the group (ICT, CET) x symptom domain (Hyperactivity/Impulsivity, Attention Problems) x time (Pre, Mid, Post) mixed-model ANOVAs, this time adding Follow-Up as a fourth time point to assess for maintenance of parent-perceived reductions in ADHD symptoms. Of primary interest were planned contrasts assessing (a) whether scores remained significantly below pre-treatment levels at follow-up (pre vs. follow-up), and (b) whether post-treatment gains were lost across the no-contact follow-up duration (post vs. follow-up).
The group (ICT, CET) x symptom domain (Hyperactivity/Impulsivity, Attention Problems) x time (Pre, Mid, Post, Follow-Up) mixed-model ANOVAs were both significant for main effects of time (p<.001; BF10>1.42 × 1014) and treatment (BF10>6.40), as well as the treatment x time interactions (p<.002; BF10>20.82). Post-hocs for the significant interactions indicated that the CET group continued to demonstrate significantly lower parent-reported Hyperactivity/Impulsivity (both d= −0.92 to −1.22, p<.001, BF10>290.20) and Attention Problems (both d= −0.86 to −0.89, p<.001, BF10>558.26) on all measures at follow-up relative to pre-treatment. The CET group did not differ between post- and follow-up for BASC-3 or ADHD-RS-5 Hyperactivity/Impulsivity (both d= −0.37 to −0.40, p>.32, BF01>1.10) or ADHD-RS-5 Attention Problems (d = −0.27, p=.99, BF10=2.35), but had lower BASC-3 Attention Problems scores at follow-up relative to immediate post-treatment (d= −0.42, p=.19, BF10=13.07). This pattern of results suggests that parents continued to view children who completed CET as significantly improved in terms of ADHD symptoms at 2–4 month follow-up, providing additional support for CET’s feasibility and acceptability.
The treatment x time interaction was due to a different pattern for the ICT group: The ICT group also did not change significantly between post- and follow-up for Attention Problems (both d= −0.05 to 0.25, p>.99, BF01=0.48–4.78) and remained significantly better at follow-up than at pre-treatment (both d= −0.48 to −0.56, p<.05, BF10>25.06). In contrast, the ICT group no longer demonstrated reduced Hyperactivity/Impulsivity symptoms at follow-up relative to pre-treatment (both d= −0.24 to −0.42, p>.21, BF10=0.69–2.56), despite not changing significantly between post- and follow-up (both d=0.21–0.22, p>.99, BF01>1.52) (Figure S7).
Summary of effects on parent-reported ADHD symptoms (feasibility/acceptability indicators)
Taken together, there was greater support for maintenance of perceived effects for CET than ICT. The CET group remained significantly below pre-treatment levels for both hyperactivity/impulsivity and attention problems on both the BASC-3 and ADHD-RS-5 (d= −0.86 to −1.22), with effect sizes that were descriptively, but in most cases not significantly, larger than those found at immediate post-treatment. In contrast, the ICT group did not demonstrate significant losses between post-treatment and follow-up, but maintained significant reductions only for parent-reported attention problems (d= −0.48 to −0.56); at follow-up, their hyperactivity/impulsivity symptoms were not significantly different from pre-treatment (d= −0.24 to −0.42, ns).
Sensitivity Analyses
Sensitivity analyses were conducted to probe for alternate explanations for the pattern of results. These analyses were not preregistered for the current study but were added during the peer review process; results should therefore be considered exploratory. Results are summarized here; please see the Supplementary Online Materials for full reporting.
Medication changes and medication effects
Despite the groups not differing in terms of pre-treatment medication or medication changes during the course of treatment, it was possible that the significant main effects of time were attributable to medication changes rather than the tested treatments. This hypothesis was unsupported: The pattern, significance, and interpretation of all results were unchanged when medication status or medication changes were added to the models.
Teacher blinding
All teachers remained blind to the child’s treatment group based on parent report on a study-created post-treatment blinding questionnaire. However, n=9 parents reported telling the teacher that their child was participating in an intervention. Because it was possible that this knowledge might produce expectancies that affected teacher ratings, we tested teacher knowledge of intervention (no/yes) as a covariate in all teacher-report analyses. The groups were equivalent in terms of teacher knowledge that the child was participating in an intervention (BF01=3.13, p=.91). In addition, the pattern, significance, and interpretation of all results was unchanged, with one exception: Controlling for pre-treatment scores (and teacher blinding), CET became marginally superior to ICT at post-treatment in terms of teacher-reported ADHD-RS-5 Hyperactivity/Impulsivity (d=0.52; p=.05; BF01=1.60). Thus, when controlling for potential teacher expectancies, CET was superior to ICT on all four teacher-report measures (BASC-3 and ADHD-RS-5 Hyperactivity/Impulsivity and Attention Problems subscales; d=0.52–0.66).
Intervention dosage
Despite the treatment groups not differing significantly in terms of total time actively playing the training games (training time), these values were qualitatively different for ICT vs. CET (mean 495 vs. 658 minutes actively engaged with the training games, respectively), with relatively large variability across children (SD = 236 vs. 389, respectively). We therefore repeated the study’s analyses, controlling for training time (minutes). Controlling for training time did not change the pattern, significance, or interpretation of any results. We then conducted additional exploratory analyses to probe for potential intervention-specific dosage effects. Results revealed that greater time training on CET was associated with greater improvements in working memory recall (r=.34, p=.01, BF10=3.35) and greater reductions in inhibitory control errors (r= −.32, p=.02, BF10=2.51). In contrast, ICT did not show the expected dose-response effect (please see the Supplementary Online Materials for full reporting and discussion).
Discussion
The current study described the continued development of a battery of computerized neurocognitive training protocols for children with ADHD. Study strengths include the detailed preregistration of study outcomes and analytic plans, open data, use of construct-valid outcome measures with strong predictive validity support, randomization with allocation concealment, blinded evaluators and data processing, explicit assessment of expectancies, inclusion of multiple tests per outcome, and adherence to best practices for cognitive training studies (Simons et al., 2016). Overall, central executive training (CET) and inhibitory control training (ICT) were equivalent in terms of parent expectancies and did not differ significantly in terms of high parent satisfaction, high child-reported ease of use, and total child training time. In terms of additional feasibility/acceptability evidence, parents reported significant reductions in their child’s inattentive and hyperactive/impulsive symptoms during the course of CET/ICT treatment; post-hocs following significant interaction effects indicated that CET (d=0.96–1.42) produced larger improvements than ICT on both BASC-3 ADHD symptom subscales (d=0.45–0.65). In contrast, CET and ICT showed similar magnitude reductions on the ADHD-RS-5 (CET d=0.99–1.06; ICT: d=0.70–0.94). These gains were maintained at 2–4 month follow-up for the CET group (pre/follow-up d=0.86–1.22), whereas maintenance of effects for the ICT group was detected for only 2 of the 4 parent report scales (pre/follow-up d=0.24–0.56).
CET produced large magnitude improvements in working memory (d=0.96–1.25) that were superior to the modest-to-nonsignificant gains attributable to ICT (d=0.26–0.41), indicating that CET successfully engaged its intended mechanism (working memory). In addition, CET produced reductions in objectively-assessed hyperactivity during working memory testing that were superior to those associated with ICT (d=0.47 vs. 0.17), although these gains were only maintained to post-treatment during one of the two tests. CET also produced reductions in blinded teacher ADHD symptom ratings that were superior to those associated with ICT on 3 of the 4 measures (d=0.52–0.66). In addition, the CET group demonstrated significant pre-post reductions across all 4 teacher ADHD symptom measures (d=0.46–0.70), whereas the ICT group showed significant reductions on both BASC-3 subscales (d=0.41–0.42) but failed to show reductions on either of the ADHD-RS-5 scales (d=0.16–0.23). More importantly, there was significant evidence of mediation, such that CET-related reductions in objectively assessed hyperactivity were conveyed via CET-related improvements in working memory during both proximal and distal testing situations. Together with our previous findings that CET produced superior improvements in working memory and reductions in objectively-assessed hyperactivity relative to gold-standard behavioral parent training (omitted), these findings provide strong support for the hypothesis that next-generation neurocognitive training protocols can overcome the limitations of extant protocols via improved targeting of etiologically relevant cognitive systems implicated in ADHD-related behavioral and functional impairments (Chacko et al., 2014).
In contrast, the evidence supporting ICT was mixed at best. ICT produced large magnitude improvements on the field’s premier measure of inhibitory control (stop-signal d=1.12; Verbruggen et al., 2013), but was inferior to CET on the go/no-go test. Although the finding that CET improved inhibition is consistent with evidence that the stop-signal task evokes demands on working memory (Tarle et al., 2019), differential change as a function of treatment was not observed. It is therefore unclear whether ICT (or CET) produced true changes in stop-signal performance as opposed to practice effects or other validity threats. In addition, there was no evidence for improvements in objectively-assessed hyperactivity for the ICT group. Further, improvements in teacher-reported ADHD symptoms were generally lower than those associated with CET, improvements in parent-reported ADHD symptoms were not consistently maintained at follow-up, and the mechanism of change analyses showed no evidence linking changes in inhibitory control with changes in objectively-assessed hyperactivity. These findings are consistent with a recent study that also failed to detect a relation between experimentally-induced increases in inhibition demands and actigraph-measured hyperactivity (Alderson et al., 2012).
Taken together, there was inconclusive evidence that ICT engaged its target mechanism (inhibitory control), and minimal evidence to support links between improvements in inhibitory control and reductions in objectively-assessed hyperactivity or blinded teacher ADHD symptom ratings. Thus, a parsimonious conclusion may be that inhibitory control is generally resistant to training in ADHD, which would be consistent with prior ADHD cognitive training studies that included inhibitory control as one of multiple targets and found either no evidence of improvements on tests of inhibitory control (e.g., Azami et al., 2016; Johnstone et al., 2012; Tamm et al., 2019), or reported improvements on some but not most inhibition tests (e.g., Dovis et al., 2012; Johnstone et al., 2010). However, given recent evidence regarding the importance of working memory for successful performance on inhibition tasks (Tarle et al., 2019), an alternative explanation could be that both treatments successfully improved stop-signal inhibition. Comparison of ICT with an active intervention not expected to affect executive functioning is therefore needed to conclusively address the malleability of inhibitory control in pediatric ADHD.
CET produced significant evidence of cognitive far-transfer effects, with improvements on the go/no-go inhibition test that were superior to those produced by ICT (d=0.84). As noted above, there was also the possibility that CET produced improvements in inhibitory control on the stop-signal. The finding that training working memory produced improvements in inhibitory control on at least one primary outcome measure adds to the mixed evidence regarding the extent to which working memory deficits are upstream from (Alderson et al., 2010; Tarle et al., 2019), downstream from (Alderson et al., 2017), or sit in parallel with (Kofler, Irwin et al., 2019) inhibitory control deficits in ADHD. Finally, the findings have implications for conceptual models of ADHD, and are generally consistent with hypotheses that inhibition deficits in ADHD are secondary to underlying impairments in working memory (e.g., Rapport et al., 2001), but not supportive of models positing that inhibition deficits produce working memory deficits in ADHD (e.g., Barkley, 1997).
Limitations
First, our a priori expectation that the study design would allow strong conclusions regarding ICT’s efficacy was predicated on our assumption that CET would not improve inhibitory control. This assumption seemed reasonable given (a) the limited evidence for cognitive far-transfer in previous ADHD training protocols (d=0.14; Rapport et al., 2013), and (b) that working memory and inhibitory control are separable constructs even in relatively young children (for review, see Karr et al., 2018). However, there are conceptual models (e.g., Rapport et al., 2009) and emerging evidence (e.g., Tarle et al., 2019) suggesting that CET-related improvements in inhibitory control should not have been entirely unexpected. Unfortunately, the cognitive far-transfer effects observed for CET limit conclusions regarding ICT’s efficacy, because ICT’s lack of superiority for improving inhibitory control introduces plausible alternative explanations for performance changes over time (e.g., practice effects). Second, the mixed support for ICT may be related to findings from recent heterogeneity studies suggesting that a smaller proportion of children with ADHD may have inhibition vs. working memory deficits (e.g., Kofler, Irwin et al., 2019), and/or our inclusion criteria that did not require below average inhibition but did require below average working memory. The latter possibility is unlikely given that only n=2 cases were excluded based on this criterion; however, future trials may care to implement a personalized medicine approach in which children are matched to CET or ICT based on their neurocognitive profile.
Third, assessing hyperactivity during the paint activity was not particularly sensitive to treatment-related changes, with only marginal evidence that these reductions covaried with improvements in working memory (r= −.23, p=.04, BF10=1.59). These findings contrast those from our first study (r= −.37, p=.003, BF10=14.43; omitted), where we also found that CET produced larger reductions than behavioral parent training during the same activity (d=0.74). Given meta-analytic evidence that children with ADHD show minimally elevated hyperactivity relative to typically developing children during low cognitive load activities (d=0.36), but pronounced hyperactivity during tasks with high executive function demands (d=1.39; Kofler et al., 2016), it appears that we need to rethink our use of this activity for assessing treatment outcomes due to potential floor effects at pre-treatment. Fourth, our multi-method evaluation of ADHD symptoms included both objective (actigraphy) and subjective measures (teacher and parent ratings); however, we were unable to include additional objective measures such as classroom observations (Kofler et al., 2008). Additional objective data such as grades and disciplinary records may be helpful in future studies to further test the behavioral and functional domains that CET does – and does not – affect. Similarly, the treated sample was comparable in most respects to the larger population of children with ADHD in terms of demographic characteristics and common comorbidities (e.g., ODD, anxiety, ASD), with the exception that there were no diagnosed depressive disorders in the current sample. Larger scale replications and examinations of the potential influence of comorbidities on treatment efficacy are needed despite the finding that the treated and untreated samples did not differ on any pre-treatment variables. Finally, as noted above, we assumed it was unreasonable to expect a 10-week intervention to normalize impairments that are characterized by 3–5 year delays in cortical maturation (Shaw et al., 2007), and as such our far-transfer outcomes focused on continuous measures rather than diagnostic remission.
Clinical and Research Implications
Taken together, there is evidence from two separate studies indicating that CET is superior to both gold-standard behavioral parent training (Kofler, Sarver et al., 2018) and newly-developed inhibitory control training for producing improvements in working memory and reductions in objectively-assessed hyperactivity for children with ADHD. The current findings also indicate that CET is superior to inhibitory control training for (a) maintaining parent-reported reductions 2–4 months post-termination, and (b) reducing teacher-reported ADHD symptoms in the classroom, suggesting the potential for ‘real world’ benefits beyond those detected using clinic-based assessments. As such, CET appears to meet the threshold for a ‘probably efficacious treatment’ (Chambless et al., 1998). In contrast, the unpredicted effects of CET for improving inhibitory control rendered conclusions regarding ICT tenuous at best and indicated that comparison with an active, credible, but non-executive function treatment will be needed to more conclusively evaluate the potential benefits of ICT. Future work with larger samples is needed to assess longer-term maintenance of treatment gains across settings, effects on peer, family, and academic impairments, and objective ADHD symptom reductions outside the clinic.
Supplementary Material
Public Health Significance Statement.
This study describes the development of inhibitory control training (ICT) and the continued testing of central executive training (CET) via a randomized controlled trial. Results indicate that CET is feasible, acceptable, and effective for treating executive function deficits and behavioral symptoms in ADHD. ICT was also feasible and acceptable, but additional study is needed to determine its potential efficacy.
Acknowledgments
Funding: This work was supported by a grant from the National Institutes of Health (NIH R01 MH115048, PI: Kofler). The sponsor had no role in design and conduct of the study; collection, management, analysis, and interpretation of the data; or preparation, review, or approval of the manuscript.
Footnotes
It is worth noting that behavioral parent training was not developed to improve executive functions such as working memory; as such, we interpret CET’s superiority for improving working memory as evidence that CET engages its target mechanism rather than as evidence against the use of evidence-based behavioral parent training. Indeed, behavioral parent training was selected as the comparator in the previous trial for precisely this reason – i.e., it was an active, credible comparator that operated via different mechanisms of action, which provided strong control for validity threats while maximizing the likelihood that we would be able to detect differential changes in CET’s hypothesized mechanism of action (working memory). In that trial, CET was also superior to BPT for reducing objectively-measured hyperactivity during 3 of the 4 conditions; CET and BPT were equivalent or did not differ significantly in terms of parent-reported ADHD symptom reductions and all other feasibility/acceptability measures (omitted).
As recommended in the K-SADS, oppositional-defiant disorder (ODD) was diagnosed only with evidence of multi-informant/multi-setting symptoms. ODD comorbidity is 38.9% based on parent-reported symptom counts, and 48.1% based on meeting parent or teacher-reported symptom counts.
Evaluation of actigraph scores during the inhibitory control tasks was not pre-registered but added prior to accessing the data to provide a more complete assessment of far transfer effects given that both working memory and inhibitory control were targeted in the current study.
Additional stop-signal metrics were also explored. Controlling for pre-treatment, the ICT and CET groups were equivalent at post-treatment in terms of inhibitory stopping speed (iSSRT; d=0.02, p=.88, BF01 = 3.57) and did not differ in terms of stop-signal delay (SSD; d=0.35, p=.21, BF01 = 2.00). The mixed-model ANOVAs indicated main effects of time only (both BF10>1.65 × 103) with planned contrasts indicating that the groups showed medium improvements in iSSRT (CET: d=0.65, p<.001, BF01 = 43.72; ICT: d=0.42, p=.01, BF01 = 4.92). The ICT group showed medium improvements in SSD (d=0.38, p=.02, BF01 = 3.35), whereas this effect was large for CET (d=1.22, p<.001, BF01 = 3.34 × 104).
Conflict of Interest: The principal investigator’s university has submitted a non-provisional patent application for the neurocognitive interventions described in the current study. There are no current licensing, financial, or other conflicts to report.
Ethical Approval: All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
Informed Consent: Informed consent was obtained from all individual participants included in the study.
Preregistration Link: https://osf.io/abwms
Open Data Link: https://osf.io/6h5e9/
References
- Alderson RM, Patros CH, Tarle SJ, Hudec KL, Kasper LJ, & Lea SE (2017). Working memory and behavioral inhibition in boys with ADHD: An experimental examination of competing models. Child Neuropsychology, 23, 255–272. [DOI] [PubMed] [Google Scholar]
- Alderson RM, Rapport MD, & Kofler MJ (2007) ADHD and behavioral inhibition: A meta-analytic review of the stop signal paradigm. Journal of abnormal child psychology, 35, 745–758. [DOI] [PubMed] [Google Scholar]
- Alderson RM, Rapport MD, Kasper LJ, Sarver DE, & Kofler MJ (2012). Hyperactivity in boys with ADHD: The association between deficient behavioral inhibition, attentional processes, and objectively measured activity. Child Neuropsychology, 18, 487–505. [DOI] [PubMed] [Google Scholar]
- Alderson RM, Rapport MD, Hudec KL, Sarver DE, & Kofler MJ (2010). Competing core processes in attention-deficit/hyperactivity disorder (ADHD): Do working memory deficiencies underlie behavioral inhibition deficits?. Journal of abnormal child psychology, 38, 497–507. [DOI] [PubMed] [Google Scholar]
- Aron AR (2011). From reactive to proactive and selective control: Developing a richer model for stopping inappropriate responses. Biological Psychiatry, 69, 55–68. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Azami S, Moghadas A, Sohrabi-Esmrood F, Nazifi M, Mirmohamad M,...Lakes K (2016). A pilot randomized controlled trial comparing computer-assisted cognitive rehabilitation, stimulant medication, and an active control in the treatment of ADHD. Child and Adolescent Mental Health, 21, 217–224. [DOI] [PubMed] [Google Scholar]
- Barkley RA (1997). Behavioral inhibition, sustained attention, and executive functions: Constructing a unifying theory of ADHD. Psychological Bulletin, 121, 65. [DOI] [PubMed] [Google Scholar]
- Baddeley A (2007). Working memory, thought, and action. Oxford, UK: Oxford Press. [Google Scholar]
- Brocki KC, Tillman CM and Bohlin G (2010). CPT performance, motor activity, and continuous relations to ADHD symptom domains: A developmental study. Eur J Dev Psychol, 7, 178–197. [Google Scholar]
- Chacko A, Bedard AC, Marks D, & Zwilling A (2017). Sequenced neurocognitive and behavioral parent training for the treatment of ADHD in school-age children. Child Neuropsychology, 24, 427–50. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Chacko A, Kofler M, & Jarrett M (2014). Improving outcomes for youth with ADHD: A conceptual framework for combined neurocognitive and skill-based treatment approaches. Clinical Child and Family Psychology Review, 17, 368–384. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Chambless DL, & Hollon SD (1998). Defining empirically supported therapies. Journal of Consulting and Clinical Psychology, 66, 7. [DOI] [PubMed] [Google Scholar]
- Chronis AM, Fabiano GA, Gnagy EM…Lopez-Williams A (2004). An evaluation of the summer treatment program for children with ADHD using a treatment withdrawal design. Behav Ther, 35, 561. [Google Scholar]
- Chronis AM, Pelham WE, Gnagy EM, Roberts JE, Aronoff HR (2003). Impact of late-afternoon stimulant dosing for children with ADHD on parent and parent–child domains. Journal of Clinical Child and Adolescent Psychology, 32, 121–129. [DOI] [PubMed] [Google Scholar]
- Coghill DR, Seth S, & Matthews K (2014). A comprehensive assessment of memory, delay aversion, timing, inhibition, decision making and variability in attention deficit hyperactivity disorder: Advancing beyond the three-pathway models. Psychological Medicine, 44, 1989–2001. [DOI] [PubMed] [Google Scholar]
- Cortese S, Ferrin M, Brandeis D, Buitelaar J, Daley D (2015). Cognitive training for ADHD: Meta-analysis of clinical and neuropsychological outcomes from randomized controlled trials. Journal of the American Academy of Child & Adolescent Psychiatry, 54, 164–174. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Dovis S, Van der Oord S, Wiers RW, & Prins PJ (2012). Can motivation normalize working memory and task persistence in children with attention-deficit/hyperactivity disorder? The effects of money and computer-gaming. Journal of abnormal child psychology, 40, 669–681. [DOI] [PMC free article] [PubMed] [Google Scholar]
- DuPaul GJ, Power TJ, Anastopoulos AD, & Reid R (2016). ADHD rating scale—5 for children and adolescents: Checklists, norms, and clinical interpretation. New York: Guilford Press. [Google Scholar]
- Halperin JM, Marks DJ, Bedard A, Chacko A…Healey DM (2013). Training executive, attention, and motor skills: a proof-of-concept study in preschool children with ADHD. J Atten Disord, 17, 711. [DOI] [PubMed] [Google Scholar]
- Halperin JM, & Healey DM (2011). The influences of environmental enrichment, cognitive enhancement, and physical exercise on brain development: Can we alter the developmental trajectory of ADHD?, Neuroscience & Biobehavioral Reviews, 35, 621–634. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Hawk LW Jr, Fosco WD, Colder CR, Waxmonsky JG, Pelham WE Jr, & Rosch KS (2018). How do stimulant treatments for ADHD work? Evidence for mediation by improved cognition. Journal of Child Psychology and Psychiatry, 59(12), 1271–1281. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Hayes AF (2009). Beyond Baron and Kenny: Statistical mediation analysis in the new millennium. Communication Monographs, 76, 408–420. [Google Scholar]
- Hoekzema E, Carmona S, Tremols V, …Vilaroya O (2010). Enhanced neural activity in frontal and cerebellar circuits after cognitive training in children with ADHD. Hum Brain Mapp, 31, 1942–50. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Johnstone SJ, Roodenrys S, Blackman R, Johnston E, Loveday K, Mantz S, & Barratt MF (2012). Neurocognitive training for children with and without ADHD. ADHD, 4, 11–23. [DOI] [PubMed] [Google Scholar]
- Johnstone SJ, Roodenrys S, Phillips E, Watt AJ, & Mantz S (2010). A pilot study of combined working memory and inhibition training for children with ADHD. ADHD, 2, 31–42. [DOI] [PubMed] [Google Scholar]
- Karalunas S, Gustafsson H, Dieckmann N …Nigg J (2017). Heterogeneity in development of aspects of working memory predicts longitudinal ADHD symptom change.J Abnorm Psychol,126,774. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Karr J, Areshenkoff C, Rast P, Hofer S..Garcia-Barrera M (2018). The unity and diversity of executive functions: A systematic review and re-analysis of latent variable studies. Psychol Bull, 144, 1147. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Klingberg T, Forssberg H, & Westerberg H (2002). Training of working memory in children with ADHD. J Clin Exp Neuropsychol, 24, 781–791. [DOI] [PubMed] [Google Scholar]
- Kofler MJ, Sarver DE, Austin KE, Schaefer H, Holland E, …Lonigan CJ (2018). Can working memory training work for ADHD? Development of central executive training and comparison with behavioral parent training. Journal of Consulting and Clinical Psychology, 86, 964–979. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kofler MJ, Irwin LN, Soto EF, Groves NB, Harmon SL, & Sarver DE (2019). Executive functioning heterogeneity in pediatric ADHD. Journal of abnormal child psychology, 47, 273–286. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kofler M, Raiker J, Sarver D., Wells E, & Soto E (2016). Is hyperactivity ubiquitous in ADHD or dependent on environmental demands? Evidence from meta-analysis. Clinical psych review, 46, 12–24. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Melby-Lervåg M, Redick TS, & Hulme C (2016). Working memory training does not improve performance on measures of intelligence or other measures of “far transfer”: Evidence from a meta-analytic review. Perspectives on Psychological Science, 11, 512–534. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Nigg JT, Willcutt EG, Doyle AE, & Sonuga-Barke EJ (2005). Causal heterogeneity in ADHD: Do we need neuropsychologically impaired subtypes? Biological Psychiatry, 57, 1224–1230. [DOI] [PubMed] [Google Scholar]
- Rabiner D, Murray D, Skinner A, & Malone P (2010). A randomized trial of two promising computer-based interventions for students with attention difficulties.J Abnormal Child Psychol,38, 131. [DOI] [PubMed] [Google Scholar]
- Rabipour S, & Davidson PS (2015). Do you believe in brain training? A questionnaire about expectations of computerised cognitive training. Behavioural brain research, 295, 64–70. [DOI] [PubMed] [Google Scholar]
- Raiker J, Rapport MD, Kofler MJ, & Sarver DE (2012). Objectively measured impulsivity and ADHD: Testing competing predictions from the working memory and behavioral inhibition models of ADHD. Journal of Abnormal Child Psychology, 40, 699–713. [DOI] [PubMed] [Google Scholar]
- Rapport MD, Chung KM, Shore G, & Isaacs P (2001). A conceptual model of child psycho-pathology: Implications for understanding ADHD and treatment efficacy. J Clin Child Psychol, 30, 48. [DOI] [PubMed] [Google Scholar]
- Rapport MD, Orban SA, Kofler MJ, & Friedman LM (2013). Do programs designed to train working memory, other executive functions, and attention benefit children with ADHD? A meta-analytic review of cognitive, academic, and behavioral outcomes. Clinical psychology review, 33, 1237. [DOI] [PubMed] [Google Scholar]
- Redick TS (2015). Working memory training and interpreting interactions in intelligence interventions. Intelligence, 50, 14–20. [Google Scholar]
- Roberts G, Quach J, Spencer-Smith M, & Wake M (2016). Academic outcomes 2 years after WM training for children with low WM: A randomized clinical trial. JAMA Pediatrics, 170, 1–10. [DOI] [PubMed] [Google Scholar]
- Rouder JN, & Morey RD (2012). Default Bayes factors for model selection in regression. Multivariate Behavioral Research, 47, 877–903. [DOI] [PubMed] [Google Scholar]
- Rubia K, Alegria AA, Cubillo AI, Smith AB, Brammer MJ, & Radua J (2014). Effects of stimulants on brain function in attention- deficit/hyperactivity disorder: a systematic review and meta-analysis. Biological Psychiatry, 76(8), 616–628. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Sala G, & Gobet F (2017). Working memory training in typically developing children: A meta-analysis of the available evidence. Developmental Psychology, 53, 671–685. [DOI] [PubMed] [Google Scholar]
- Sarver D, Rapport M, Kofler M, Raiker J, & Friedman L (2015). Hyperactivity in ADHD: Impairing deficit or compensatory behavior? Journal of Abnormal Child Psychology, 43,1219–1232 [DOI] [PubMed] [Google Scholar]
- Sergeant JA (2005). Modeling attention-deficit/hyperactivity disorder: A critical appraisal of the cognitive energetic model. Biol Psychiatry, 25, 1248–1255. [DOI] [PubMed] [Google Scholar]
- Shalev L, Tsal Y, & Mevorach C (2007). Computerized progressive attentional training (CPAT) program: Effective direct intervention for children with ADHD. Child Neuropsychology, 13, 382–388. [DOI] [PubMed] [Google Scholar]
- Shaw P, Eckstrand K, Sharp W, Blumenthal J, Lerch JP, Greenstein DE, … & Rapoport JL (2007). ADHD is characterized by a delay in cortical maturation. PNAS, 104, 19649–19654. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Shipstead Z, Redick TS, & Engle RW (2012). Is working memory training effective? Psychological Bulletin, 138, 628–654. [DOI] [PubMed] [Google Scholar]
- Shrout PE, & Bolger N (2002). Mediation in experimental and nonexperimental studies: New procedures and recommendations. Psychological methods, 7, 422. [PubMed] [Google Scholar]
- Simons DJ, Boot WR, Charness N, Gathercole SE, Chabris CF, …Stine-Morrow EA (2016). Do “brain-training” programs work?. Psychological Science in the Public Interest, 17, 103–186. [DOI] [PubMed] [Google Scholar]
- Sonuga-Barke E, Bitsakou P, & Thompson M (2010). Beyond the dual pathway model: Evidence for dissociation of timing, inhibitory, and delay-related impairments in ADHD. JAACAP, 49, 345–355. [DOI] [PubMed] [Google Scholar]
- Tamm L, Epstein JN & Becker SP (2019) A preliminary investigation of reaction time variability in relation to social functioning in children evaluated for ADHD. Child Neuropsychology, 25, 885–898. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Tamm L, Epstein JN, Loren RE, Becker SP, Brenner SB, Bamberger ME, … & Halperin JM (2019). Generating attention, inhibition, and memory: A pilot randomized trial for preschoolers with executive functioning deficits. Journal of Clinical Child & Adolescent Psychology, 48, S131–S145. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Tarle S, Alderson R, Patros C, Arrington E, & Roberts D (2019). Working memory and behavioral inhibition in children with attention-deficit/hyperactivity disorder (ADHD): An examination of varied central executive demands, construct overlap, and task impurity. Child Neuropsychology, 25, 664–687. [DOI] [PubMed] [Google Scholar]
- Tarle SJ, Alderson RM, Patros C, Lea S, Hudec K, & Arrington E (2017). ADHD and phonological working memory: Methodological variability. Neuropsychology, 31, 383. [DOI] [PubMed] [Google Scholar]
- The Jamovi project (2019). Jamovi (Version 1.0) [Computer Software]. [Google Scholar]
- Van der Oord S, Bögels SM, & Peijnenburg D (2012). The effectiveness of mindfulness training for children with ADHD and mindful parenting for their parents. J. Child Fam. Stud. 21, 139–147. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Verbruggen F, Chambers CD, & Logan GD (2013). Fictitious inhibitory differences: How skewness and slowing distort the estimation of stopping latencies. Psychological science, 24, 352–362. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Wager TD, & Smith EE (2003). Neuroimaging studies of working memory. Cognitive, Affective & Behavioral Neuroscience, 3, 255–274. [DOI] [PubMed] [Google Scholar]
- Wagenmakers EJ, Marsman M, Jamil T..Morey RD (2016). Bayesian statistical inference for psychological science. Part I: Theoretical advantages and practical ramifications. Psychonom Bull Rev. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.