Executive Functioning Heterogeneity in Pediatric ADHD

Abstract

Neurocognitive heterogeneity is increasingly recognized as a valid phenomenon in ADHD, with most estimates suggesting that executive dysfunction is present in only about 33%–50% of these children. However, recent critiques question the veracity of these estimates because our understanding of executive functioning in ADHD is based, in large part, on data from single tasks developed to detect gross neurological impairment rather than the specific executive processes hypothesized to underlie the ADHD phenotype. The current study is the first to comprehensively assess heterogeneity in all three primary executive functions in ADHD using a criterion battery that includes multiple tests per construct (working memory, inhibitory control, set shifting). Children ages 8–13 (M=10.37, SD=1.39) with and without ADHD (N=136; 64 girls; 62% Caucasian/Non-Hispanic) completed a counterbalanced series of executive function tests. Accounting for task unreliability, results indicated significantly improved sensitivity and specificity relative to prior estimates, with 89% of children with ADHD demonstrating objectively-defined impairment on at least one executive function (62% impaired working memory, 27% impaired inhibitory control, 38% impaired set shifting; 54% impaired on one executive function, 35% impaired on two or all three executive functions). Children with working memory deficits showed higher parent- and teacher-reported ADHD inattentive and hyperactive/impulsive symptoms (BF₁₀ = 5.23 x 10⁴), and were slightly younger (BF₁₀ = 11.35) than children without working memory deficits. Children with vs. without set shifting or inhibitory control deficits did not differ on ADHD symptoms, age, gender, IQ, SES, or medication status. Taken together, these findings confirm that ADHD is characterized by neurocognitive heterogeneity, while suggesting that contemporary, cognitively-informed criteria may provide improved precision for identifying a smaller number of neuropsychologically-impaired subtypes than previously described.

Attention-deficit/hyperactivity disorder (ADHD) is a chronic and impairing neurodevelopmental disorder that affects approximately 5% of school age-children (Polanczyk et al., 2014). Heterogeneity in the disorder’s behavioral symptoms, associated impairments, and cognitive sequelae has been increasingly recognized (e.g., Castellanos & Tannock, 2002; Coghill et al., 2014; Sonuga-Barke et al., 2008). Current estimates suggest that only approximately 33%–50% of children with ADHD exhibit impairments in executive functioning (Biederman et al., 2004; Nigg et al., 2005). At the same time, recent critiques of the clinical literature’s executive function task selection (e.g., Snyder et al., 2015) question the veracity of these estimates, such that our understanding of executive functioning in ADHD is based, in large part, on data from single tasks that may suboptimally assess their intended construct (Kofler et al., 2016; Coghill et al., 2014; Sonuga-Barke et al., 2008). The neurocognitive heterogeneity observed in ADHD is increasingly recognized as a valid phenomenon (Nigg et al., 2005); however, it is possible that cognitively-informed test batteries with greater construct precision and estimation with multiple tests may allow for the identification of a relatively small number of causal pathways to the ADHD phenotype (Coghill et al., 2005). While previous investigations have assessed multiple neurocognitive constructs with single tasks, or included estimates of one or two of the three primary executive functions (for review, see Coghill et al., 2014), the current study is the first to comprehensively assess heterogeneity in all three primary executive functions in ADHD using a criterion battery that includes multiple tests per construct (working memory, inhibitory control, set shifting; Miyake et al., 2000).

Executive Functioning

Executive functions refer to a set of interrelated, higher-order cognitive processes that enable goal directed behavior and novel problem solving (Baddeley, 2007; Miyake et al., 2000). Among the diverse models of executive functioning, factor analytic and theoretical work provides the most empirical support for models that include three primary executive function domains: working memory, inhibitory control, and set shifting (Miyake et al., 2000; Miyake & Friedman, 2012; St. Clair-Thompson & Gathercole, 2006; Van der Molen et al., 2006; Van der Sluis et al., 2007). These primary executive functions, in turn, enable goal-oriented behavior and support a host of secondary, non-executive cognitive abilities including but not limited to proactive and reactive interference control (Wiemers & Redick, 2018) and goal-maintenance (Engle & Kane, 2004), as well as performance on tasks intended to assess vigilance (Raiker et al., 2012), response variability (Kofler et al., 2014; Wiemers & Redick, 2018), planning (Jaroslawska et al., 2016; Kofler et al., 2018; Miyake et al., 2000), perseveration (Miyake et al., 2000), and delay tolerance (Patros et al., 2015).

Working memory refers to the active, top-down manipulation of information held in short-term memory (Baddeley, 2007), and includes interrelated functions of the mid-lateral prefrontal cortex and interconnected networks that involve updating, dual-processing, and temporal/serial reordering (Wager & Smith, 2003). Inhibitory control refers to a set of interrelated cognitive processes that underlie the ability to withhold (action restraint) or stop (action cancellation) an on-going response (Alderson et al., 2007) and are supported by networks involving bilateral frontal, right superior temporal and left inferior occipital gyri, right thalamic, and mid-brain structures (Cortese et al., 2012). Set shifting refers to the ability to flexibly switch back-and-forth between mental sets via activation of prefrontal and posterior parietal cortices (Miyake et al., 2000; Pa et al., 2010).

Executive Functioning in ADHD

Working memory and inhibitory control have been studied extensively in pediatric ADHD, with meta-analyses indicating medium to large group-level impairments on tasks intended to assess these executive functions (Alderson et al., 2007; Kasper et al., 2012; Lijffijt et al., 2005; Martinussen et al., 2005; Willcutt et al., 2005). At the same time, a growing number of studies suggest that these group-level differences may be carried by impairments in a relatively small proportion of children with ADHD (Sonuga-Barke et al., 2008). Specifically, estimates for the proportion of pediatric ADHD cases who exhibit any form of executive dysfunction range from 21% to 60% across studies employing a wide range of tasks and impairment criteria (Biederman et al., 2004; Coghill et al., 2014; Fair, Bathula, Nikolas, & Nigg, 2012; Geurts, van der Oord, & Crone, 2006; Nigg et al., 2005; Solanto et al., 2001; Sonuga-Barke, Bitsalou, & Thompson, 2010). Studies separating working memory and inhibitory control show similar heterogeneity: Working memory impairments are reported in 30% to 37% (Coghill et al., 2014; Fair et al., 2012), and inhibitory control deficits are detected in 21%–46% of pediatric ADHD cases (Coghill et al., 2014; Nigg et al., 2005; Solanto et al., 2001; Sonuga-Barke et al., 2010). To our knowledge, no ADHD study to date has examined heterogeneity in set shifting. However, meta-analytic effect sizes of d=0.46 to 0.55 (Willcutt et al., 2005) predict that 30%–36% of children with ADHD may demonstrate set shifting impairments based on converting effect size differences into expected population overlap proportions (Zakzanis et al., 2001).

Construct Validity and Task Impurity

These impairment estimates, combined with evidence that only a minority of ADHD cases are classified as impaired on multiple executive function tests (Biederman et al., 2004), have led in part to refined models of ADHD that emphasize causal heterogeneity (e.g., Nigg et al., 2005) and de-emphasize single-cause models that conceptualize ADHD as primarily a disorder of executive dysfunction (Coghill et al., 2005). However, as noted by Sonuga-Barke et al. (2008), these conclusions may be premature because the evidence-base is comprised, in large part, on data from clinical tests developed to detect gross neurological impairment rather than the more subtle deficits in cognitive control hypothesized to underlie the ADHD phenotype. That is, these tests may be most appropriate for screening for severe executive deficits in patients, but appear to lack sensitivity for targeting specific aspects of executive function and identifying individual differences across a wider range of abilities necessary for probing the nature of more subtle deficits associated with psychopathology (please see Snyder et al., 2015 for a review of specific tests).

Construct validity and task specificity

The broad scope of the measurement issue raised by Sonuga-Barke et al. (2008) is highlighted when juxtaposing the tasks included in recent meta-analyses of ADHD neuropsychological functioning (Alderson et al., 2007; Kasper et al., 2012; Martinussen et al., 2005; Willcutt et al., 2005) with a recent critique of executive function test selection in the clinical versus cognitive science literatures (Snyder, Miyake, & Hankin, 2015). Across ADHD meta-analyses, estimates of deficits in working memory, inhibitory control, and set shifting are based in large part, and in some cases entirely, on data from tasks that have been criticized for suboptimal construct validity (Redick & Lindsey, 2013) and structural organization (Friedman & Miyake, 2017) relative to contemporary advancements in cognitive psychology (Snyder et al., 2015). This suboptimal construct validity has demonstrated implications for our ability to identify executive dysfunction in children with ADHD. For example, Kasper and colleagues (2012) examined the evidence for working memory deficits in ADHD, and reported overall medium magnitude impairments (d=0.69–0.74) that were consistent with prior meta-analyses (Martinussen et al., 2005; Willcutt et al., 2005). At the same time, they found that the majority of tasks used to measure ‘working memory’ in the ADHD literature were better conceptualized as tests of ‘short term memory,’ with evidence from the cognitive literature suggesting that these tasks place relatively minimal demands on the executive components of working memory (Conway et al., 2005; Engle et al., 1999; Rapport et al., 2013). Using meta-regression, they found that their estimate of working memory deficits in ADHD increased from medium to very large (d=2.0–2.2) when based on working memory tasks with a prominent executive component congruent with contemporary cognitive definitions. Thus, whereas the uncorrected estimates suggest that 40%–44% of children with ADHD have working memory deficits (Zakzanis, 2001), these estimates increase to 81%–84% when using criterion, recall-based tasks with prominent executive demands (Coghill et al., 2014) and a sufficient number of administered trials (Wells et al., 2018).

Applying the construct specificity concerns raised by Sonuga-Barke et al. (2008) and Snyder et al. (2015) to evaluate the tests used to evaluate neuropsychological heterogeneity in ADHD indicates that, to our knowledge, 100% of the available literature either omitted tests of working memory or measured working memory using at least one test with poor construct specificity (Biederman et al., 2004; Coghill et al., 2014; Fair et al., 2012; Geurts et al., 2006; Nigg et al., 2005; Solanto et al., 2001; Sonuga-Barke et al., 2010). As noted above, these studies reported that 30%–37% of ADHD cases are likely to demonstrate working memory deficits based in large part on these less-specific tests. Similarly, 100% of set shifting tasks included in the most recent meta-analysis may be considered non-specific neuropsychological tests rather than specific tests of set shifting (Snyder et al., 2015; Willcutt et al., 2005), and the method used in most studies to derive estimates of inhibitory control from the stop-signal task has been criticized for producing spurious results (Verbruggen et al., 2013).

Task impurity

Further complicating attempts to identify causal neurocognitive processes in ADHD is the ‘task impurity problem’ (Conway et al., 2005). That is, no task is process pure (Shipstead, Redick, & Engle, 2010): all tasks require multiple executive and non-executive neurocognitive abilities for successful performance, and conclusions regarding effect specificity are limited when these correlated but distinct abilities are not measured and simultaneously controlled (Miyake & Friedman, 2012). In most cases, the majority of variance in any single test is attributable to processes other than the executive function of interest; multiple tests per construct are critical to isolating EF-specific performance (Willoughby et al., 2016). As noted by Coghill et al. (2014), no study of ADHD-related heterogeneity has simultaneously measured and controlled for all three primary executive functions (Miyake et al., 2000), and conclusions may be limited by the use of single tasks to assess neurocognitive construct(s) (Conway et al., 2005).

Current Study

Taken together, the literature suggests substantial neurocognitive heterogeneity in pediatric ADHD, but conclusions may be limited because no study to date has simultaneously measured all three primary executive functions (Miyake et al., 2000). In addition, the evidence base has been criticized for using non-specific tests that in many cases were not developed to assess a specific executive function but rather intended to detect gross neurological impairment (Sonuga-Barke et al., 2008). The current study addresses these issues, and is the first to simultaneously and comprehensively assess heterogeneity in executive dysfunction among children with carefully phenotyped ADHD using a counterbalanced battery that includes multiple criterion tests per construct (Coghill et al., 2014). Formative indicators of working memory, inhibitory control, and set shifting were derived, and these participant-level component scores were subjected to reliable change analyses (Jacobson & Truax, 1991) that explicitly account for measurement error to objectively define impairment (Kofler et al., 2016; Sarver et al., 2015). We hypothesized that these methodological refinements would produce higher estimates of executive dysfunction than prior studies, such that a majority of children with ADHD would be correctly classified as impaired on at least one of the three primary executive functions. Consistent with models emphasizing causal neurocognitive heterogeneity, we expected to detect a subset of children with ADHD without executive function impairments.

Method

Participants

The sample included 136 children aged 8 to 13 years (M=10.37, SD=1.39; 72 boys, 64 girls) from the Southeastern United States, consecutively recruited by or referred to a university-based Children’s Learning Clinic (CLC) through community resources (e.g., pediatricians, community mental health clinics, school system personnel, self-referral) between 2015 and 2017. The CLC is a research-practitioner training clinic known to the surrounding community for conducting developmental and clinical child research and providing pro bono comprehensive diagnostic and psychoeducational services. Its client base consists of children with suspected learning, behavioral or emotional problems, as well as typically developing children (those without a suspected psychological disorder) whose parents agreed to have them participate in developmental/clinical research studies. All parents and children gave informed consent/assent, and the Florida State University Institutional Review Board approved the study prior to and throughout data collection. Sample ethnicity was mixed with 81 Caucasian Non-Hispanic (62%), 16 African American (12%), 14 Hispanic (11%), 13 multiracial children (10%), and 7 Asian (5%) children.

Group Assignment

All children with ADHD and their parents completed a comprehensive psychoeducational and diagnostic evaluation that included a detailed, semi-structured clinical interview using the Kiddie Schedule for Affective Disorders and Schizophrenia for School-Aged Children (K-SADS; Kaufman et al., 1997). The K-SADS (2013 Update) allows differential diagnosis according to symptom onset, course, duration, quantity, severity, and impairment in children and adolescents based on DSM-5 criteria (APA, 2013), and was supplemented with parent and teacher ratings from the Behavior Assessment System for Children (BASC-2/3; Reynolds & Kamphaus, 2015) and ADHD Rating Scale-4/5 (ADHD-4/5; DuPaul et al., 2016). A psychoeducational report was provided to parents.

Fifty-five children met all of the following criteria and were included in the ADHD group (n=55; 38% girls): (1) DSM-5 diagnosis of ADHD Combined (n=39), Inattentive (n=14), or Hyperactive/Impulsive Presentation (n=2) by the directing clinical psychologist based on K-SADS; (2) borderline/clinical elevations on at least one parent and one teacher ADHD subscale; and (3) current impairment based on parent report. All ADHD subtypes/presentations were eligible given the instability of ADHD subtypes (Valo & Tannock, 2010). Psychostimulants (n_prescribed=17) were withheld ≥24 hours for testing. To improve generalizability, children with comorbidities were included. Comorbidities reflect clinical consensus best estimates (Kosten & Rounsaville, 1992), and included anxiety (22%), oppositional defiant (13%)¹, depressive (9%), and autism spectrum disorders (5%). Positive screens for reading (9%) and math disability (13%) were defined based on score(s) >1.5 SD below age-norms on one or more KTEA-3 Academic Skills Battery reading and math subtests, as specified in DSM-5 (APA, 2013).

The Non-ADHD group comprised 81 consecutive case-control referrals (43 girls) who did not meet ADHD criteria, and included both neurotypical children and children with psychiatric disorders other than ADHD. Neurotypical children (77%) had normal developmental histories and nonclinical parent/teacher ratings and were recruited through community resources. Clinically referred and evaluated children who did not meet ADHD criteria were also included in the Non-ADHD group. These Non-ADHD disorders were included to control for comorbidities in the ADHD group, and included best estimate diagnoses of anxiety (13%), autism spectrum (5%), depressive (4%), and oppositional defiant disorders (1%).¹ None of the clinically-evaluated Non-ADHD cases screened positive for learning disorders in reading or math. Importantly, the ADHD and Non-ADHD groups did not differ significantly in the proportion of children diagnosed with a clinical disorder other than ADHD (anxiety: BF₀₁ = 2.81, depression: BF₀₁ = 4.23, ODD: BF₀₁ = 0.66, ASD: BF₀₁ = 9.57, SLD reading: BF₀₁ = 1.38, SLD math BF₀₁ = 0.41). The Bayes Factor BF₀₁ is an odds ratio indicating support for the null hypothesis that the groups are equivalent (H₀) relative to the alternative hypothesis that the groups differ (H₁) (see Bayesian Analyses section below).

The first 41 Non-ADHD participants underwent an identical evaluation as the ADHD group. Due to funding constraints, the final 40 Non-ADHD participants completed an abbreviated screening evaluation that included parent BASC-3, a 1-subtest IQ screener, and detailed developmental, medical, educational, and psychiatric histories. Neurotypical children did not differ significantly based on whether they received a full or abbreviated evaluation in terms of IQ, gender, age, or BASC hyperactivity T-scores (all BF₀₁ > 1.37). The abbreviated subgroup had, on average, slightly lower BASC inattention T-scores (M=48.0 vs. 53.5, BF₁₀ = 4.71) and SES (M=45.5 vs. 53.7, BF₁₀ = 5.56).

Children were excluded if they presented with gross neurological, sensory, or motor impairment; history of seizure disorder, psychosis, or intellectual disability; or non-stimulant medications that could not be withheld for testing. Additional exclusion criteria were added a priori for the abbreviated evaluation subgroup because we were unable to clinically evaluate these cases: previous diagnosis of ADHD or other psychiatric disorders, or BASC-3 inattention/hyperactivity T-scores > 1.5 SD above the normative sample mean for age and gender.

Procedure

Neurocognitive testing occurred as part of a larger battery that involved 1–2 sessions of approximately 3 hours each. All tasks were counterbalanced to minimize order effects. Children received brief breaks after each task, and preset longer breaks every 2–3 tasks to minimize fatigue. Children were seated in a caster-wheel swivel chair. Performance was monitored at all times by the examiner, who was stationed just outside of the testing room (out of the child’s view) to provide a structured setting while minimizing performance improvements associated with examiner demand characteristics (Gomez & Sanson, 1994).

Measures

Working Memory Tasks

Working memory reordering

The Rapport et al. (2009) computerized phonological and visuospatial working memory tasks correctly classify children with vs. without ADHD at similar rates as parent and teacher ADHD rating scales (Tarle et al., 2017), and predict hyperactivity (Rapport et al., 2009), attention (Kofler et al., 2010), impulsivity (Raiker et al., 2012), and ADHD-related functional impairments (Friedman et al., 2017; Kofler et al., 2011, 2016). Reliability and validity evidence includes internal consistency (α=.82–.97; Kofler et al., 2017), 1- to 3-week test-retest reliability (.76–.90; Sarver et al., 2015), and expected magnitude relations with criterion working memory complex span (r=.69) and updating tasks (r=.61) (Wells et al., 2018). Internal consistency in the current sample was .81 (phonological) and .87 (visuospatial).

Both tasks involve serial reordering of characters presented (numbers, black dot locations), and reordering of a target stimulus (letter, red dot location) into the final serial position recalled. Six trials were administered at each set size for each task (3–6 stimuli/trial; 1 stimuli/second). The 24 total trials per task were randomized, then grouped into 2 blocks of 12 trials each, with short breaks between each block (approximately 1 minute) (Kofler et al., 2016). Five practice trials were administered before each task (80% correct required). The phonological task involved mentally reordering and verbally recalling a jumbled series of sequentially presented numbers and letters (e.g., 4H62 is correctly recalled as 246H). The visuospatial task involved mentally reordering a sequentially presented series of spatial locations based on what color dot appeared in each location (black dots in serial order, red dot last) and responding on a modified keyboard. Partial-credit unit scoring (stimuli correct per trial) at each set size (3–6) was used as recommended (Conway et al., 2005).

Letter updating

The Miyake et al. (2000) letter memory test was adapted for use with children and exemplifies working memory updating based on the Miyake et al. (2000) model. Working memory updating tasks involve the constant monitoring and rapid addition/deletion of working memory contents (Miyake & Friedman, 2012). Similar updating tasks have produced large magnitude ADHD/Non-ADHD between group differences (e.g., Friedman et al., 2017; Raiker et al., 2012). In this computerized task, letters were presented on the screen one at a time, and children were instructed to keep track of the last three letters presented. To ensure the task required continuous updating, children were instructed to rehearse out loud the last three letters by mentally adding the most recent letter and dropping the fourth letter back and then saying the new string of three letters out loud (Miyake et al., 2000). The number of letters presented (4–8 stimuli presented/trial, 1200 ms presentation, 2400 ms ISI) was varied randomly across trials to ensure that successful performance required continuous updating until the end of each trial. A practice block was administered; children advanced to the test phase following three correct trials. Four blocks of three trials each were administered. Children responded via mouse click. The dependent variables were mean stimuli correct per trial recalled in the correct serial order during each of the four task blocks. Internal consistency in the current sample was α=.75.

Inhibitory Control

Stop-signal

Task and administration instructions were identical to Alderson and colleagues (2008). Psychometric evidence includes high internal consistency (α=.83–.89), 3-week test-retest reliability (.72), and convergent validity with other inhibition tests (Soreni et al., 2009). Go-stimuli were displayed for 1000 ms as uppercase letters X and O positioned in the center of a computer screen (500 ms interstimulus interval; total trial duration=1500 ms). Xs and Os appeared with equal frequency. A 1000 Hz auditory tone (stop-stimulus) was presented randomly on 25% of trials. Stop-signal delay (SSD) – the latency between go- and stop-stimuli presentation – was initially set at 250 ms, and dynamically adjusted ±50 ms contingent on performance. The algorithm was designed to approximate successful inhibition on 50% of stop-trials. In the current study, inhibition success was 60.5%, 58.8%, 59.9%, and 57.6% across the four experimental blocks. Children used a modified response pad to complete two practice and four consecutive experimental blocks of 32 trials/block (8 stop-trials per block). SSD at each of the four blocks is the most direct measure of inhibition in stop-signal tasks that use dynamic stop-signal delays, because SSD changes systematically according to inhibitory success or failure (Alderson et al., 2007; Lijffijt et al., 2005)². Internal consistency in the current sample was α=.80.

Go/no-go

The go/no-go is a response inhibition task in which a motor response must be executed or inhibited based on a stimulus cue (Bezdjian et al., 2009). Children were presented a randomized series of vertical (go stimuli) and horizontal (no-go stimuli) rectangles in the center of a computer monitor (2000 ms presentation, jittered 800–2000 ms ISI to minimize anticipatory responding). They were instructed to quickly click a mouse button each time a vertical rectangle appeared, but to avoid clicking the button when a horizontal rectangle appeared. A ratio of 80:20 go:no-go stimuli was selected to maximize prepotency (Kane & Engle, 2003; Unsworth & Engle, 2007). Children completed a 10-trial practice (80% correct required) followed by 4 continuous blocks of 25 trials each. Commission errors reflect failed inhibitions (i.e., incorrectly responding to no-go trials), and served as the primary index of inhibitory control during each of the four task blocks. Internal consistency in the current sample was α=.95.

Set Shifting

Global-local

The Miyake et al. (2000) local-global task was adapted for use with children. This computerized task uses Navon (1977) figures, which feature a “global” shape (e.g., a circle) constructed using smaller, “local” figures (e.g., triangles). Figures were presented one at a time in one of four quadrants in a clockwise rotation on a computer monitor (jittered ISI 800–2000ms). Children were required to shift their response between global and local features depending on which quadrant the figures appeared (top quadrants: global; bottom quadrants: local). Trials with stimuli in the top left or bottom right quadrants involved set shifting (shift trials) because responses required a different rule than the previous trial; trials with stimuli in the top right or bottom left quadrants did not require shifting because they featured the same rule as the previous trial (non-shift trials). To minimize memory demands, on-screen cues (“big shape”, “small shapes”) remained on-screen next to each quadrant. Sixty trials were administered following three blocks of 6 to 8 practice trials (100% correct required). Internal consistency in the current sample was α=.86 (shift trials) and α=.90 (no-shift trials).

Children responded via mouse click. Performance data were recorded separately for ‘shift’ and ‘non-shift’ trials. Trials were divided into 4 consecutive blocks to match the number of outcome variables from the working memory and inhibitory control tasks. Reaction time (RT) data was processed following the steps outlined in Miyake et al. (2000) that winsorized the most extreme 2.2% of individual reaction times. First, all individual trial RTs greater than 9500ms were winsorized to 9500ms. Second, individual trial RTs greater than 3 standard deviations from each child’s mean RT were winsorized relative to that child’s within-task RT distribution. Shift costs for both response time (speed) and accuracy were computed (van der Ven et al., 2013), calculated separately for each task condition for each child (Speed shift cost = RT_shift – RT_non-shift; Accuracy shift cost = %Errors_shift – %Errors_non-shift).

Number-color

The Miyake et al. (2000) number-letter task was adapted for use with children. A pair of single-digit numbers appeared on the screen, and children were instructed to click either the larger or smaller value depending on the font color (blue = bigger, yellow = smaller; colors selected for maximal discrimination across individuals with all types of color vision). Both digits were the same color on any given trial. To minimize memory demands, on-screen instructions (blue bigger, yellow smaller) remained visible throughout the task. Trials were presented in a semi-random sequence to require shifting every other trial, with an equal number of bigger-smaller and smaller-bigger shifts. RT and accuracy data were recorded separately for ‘shift’ and ‘non-shift’ trials, and processed identically to the global-local data described above. Following an 8-trial practice block (100% correct required), children completed 4 consecutive blocks of 30 trials each (120 total trials; jittered ISI 80–200 ms). Internal consistency in the current sample was α=.95 (shift trials) and α=.87 (no-shift trials).

Due to a programming error, the first 44 participants (ADHD=17, Non-ADHD=27) completed a 60-trial version of the Number-Color task. These data were retained because, accounting for ADHD/Non-ADHD status, shift costs were equivalent for children completing the abbreviated versus full task for both response times (BF₀₁ = 5.90) and percent correct responses (BF₀₁ = 7.76), and task version did not interact significantly with ADHD status or task block (all BF₀₁ > 1.14).

Executive Function Dimension Reduction

Statistically, we controlled for task impurity by computing Bartlett maximum likelihood weighted averages based on the intercorrelations among task performance scores (DiStefano et al., 2009). Conceptually, this process isolates reliable variance across estimates of each executive function by removing task-specific demands associated with non-executive processes, time-on-task effects via inclusion of four blocks per task, and non-construct variance attributable to other measured executive and non-executive processes (e.g., short-term memory load). Thus, the 36 task performance variables (Supplementary Table 1) were reduced to three principal component estimates (30.06% of variance explained; Supplementary Table 2). A three-component solution was specified a priori to derive separate estimates of working memory, inhibitory control, and set shifting based on theory and previous empirical work (e.g., Miyake et al., 2000). The ratio of participants (136) to factors (3) was deemed acceptable (Hogarty et al., 2005). Executive function task data were represented as formative rather than reflective (confirmatory) indicators as recommended (Willoughby et al., 2016). Orthogonal components were specified to maximally control for task impurity (Kofler et al., 2016). By design, the intercorrelations among the varimax-rotated working memory, inhibitory control, and set shifting components were r_all=.00 (p>.99). These three executive function component scores (z-scores) were used in all analyses below. Higher scores reflect better working memory and inhibition but worse set shifting.

Intellectual Functioning (IQ)

All children were administered the WISC-V Short Form (Sattler, 2016) or WISC-V Matrix Reasoning subtest (Wechsler, 2014) to obtain an estimate of intellectual functioning.

Socioeconomic Status (SES)

Hollingshead (1975) SES was estimated based on caregiver(s)’ education and occupation.

Objectively-Defined Impairment

Following Sarver et al. (2015) and Kofler et al., (2016), impairment was objectively defined by applying the Jacobson & Truax (1991) model of reliable change to each child’s executive function component scores. This method was selected over static cut points (e.g., 10^th percentile of Non-ADHD group) because it improves precision by explicitly accounting for measurement unreliability (Jacobson & Truax, 1991). Children were classified as Impaired or Not Impaired in each executive function domain based on whether their score was reliably below the Non-ADHD sample (i.e., difference exceeded chance at p < .05). This classification was based on computation of the Reliable Change Index (RCI), or the ratio of the difference between the child’s score and the Non-ADHD group’s mean divided by standard error (computed using each measure’s reported test-retest reliability and the SD of the Non-ADHD sample; Rule B; Jacobson & Truax, 1991) individually for each child for each of the three executive functioning domains. Reported test-retest reliability across all tests was .72 to .83. The RCI is tested against the z distribution; impairment is defined as a score that is significantly worse than the Non-ADHD mean given the Non-ADHD group’s SD and the test’s reported reliability. A classification of Impaired indicates that the child is statistically more likely to come from the dysfunctional/impaired population than the functional/neurotypical population (Jacobson & Truax, 1991).

Inspection of the RCI data indicated that the impairment cut-offs centered around 1.5 SD below the normative sample mean across measures; statistical significance was obtained at different cut points across measures dependent on each measure’s test-retest reliability (i.e., for tests with lower reliability, scores further from the mean were required to conclude with p < .05 certainty that the child’s score was more likely to come from the dysfunctional/impaired population than the functional population).

Bayesian Analyses

Frequentist statistics were supplemented with Bayesian methods as recommended (Rouder & Morey, 2012; Wagenmakers et al., 2016); for our purposes, Bayesian analyses were added because they allow stronger conclusions by estimating the magnitude of support for both the alternative and null hypotheses (Rouder & Morey, 2012). JZS default prior scales were used (Rouder & Morey, 2012; Wagenmakers et al., 2016). Analyses were conducted using JASP 0.8.2 (JASP Team, 2017). Instead of a p-value, these analyses provide BF₁₀, which is the Bayes Factor of the alternative hypothesis (H₁) against the null hypothesis (H₀). BF₁₀ is an odds ratio, where values above 3.0 are considered moderate evidence supporting the alternative hypothesis (i.e., statistically significant evidence for the alternative hypothesis). BF₁₀ values above 10.0 are considered strong (>30 = very strong, >100 = decisive support; Wagenmakers et al., 2016).

Conversely, BF₀₁ is the Bayes Factor of the null hypothesis (H₀) against the alternative hypothesis (H₁). BF₀₁ is the inverse of BF₁₀ (i.e., BF₀₁ = 1/BF₁₀), and is reported when the evidence indicates a lack of an effect (i.e., favors the null hypothesis; Rouder & Morey, 2012). BF₀₁ values are interpreted identically to BF₁₀ (>3.0 = moderate, >10.0 = strong, >100 = decisive/extreme support for the null hypothesis that the ADHD and Non-ADHD groups are equivalent on an outcome; Rouder & Morey, 2012).

Data Analysis Overview

The analytic plan was executed in three tiers. The first Tier examined executive functioning heterogeneity in ADHD by quantifying the proportion of children with ADHD who exhibited impairments in each executive function relative to the local normative comparison group. In the second Tier, we compared demographic and behavioral indicators of children defined as Impaired vs. Not Impaired on each executive function. Finally, we conducted sensitivity analyses in Tier 3 to probe the extent to which our results were influenced by our impairment definition, sample demographics, and data reduction methods.

Results

Preliminary Analyses

Outliers beyond 3 SD were winsorized relative to the within-group distribution (ADHD, Non-ADHD). This process affected 0.6% (ADHD group) to 0.8% (Non-ADHD group) of data points. Missing data rates were low (0.5% ADHD, 0.2% Non-ADHD) and imputed using the SPSS expectation-maximization function based on all available data because Little’s MCAR test indicated that these data were missing completely at random (χ² [327]=108.24, p > .99). All parent and teacher ADHD rating scale scores were higher for the ADHD relative to Non-ADHD group as expected (Table 1). The ADHD group demonstrated group-level impairments in working memory (d = 1.41; BF₁₀ = 1.64 x 10¹⁰), inhibitory control (d= 0.60; BF₁₀ = 33.51), and set shifting (d = 0.46; BF₁₀ = 4.29) relative to the Non-ADHD group. In contrast, there was no significant evidence to indicate between-group differences in age (BF₀₁ = 2.34), gender (BF₀₁ = 1.10), ethnicity (BF₀₁ = 2.29), IQ (BF₁₀ = 2.00), or SES (BF₀₁ = 4.48); we therefore report simple model results with no covariates.

Table 1.

Sample and Demographic Variables

Variable	ADHD (N=55)		Non-ADHD (N=81)		Cohen’s d	BF10	BF01	p
	M	SD	M	SD
Gender (Boys/Girls)	34/21		38/43		--		1.10	.09, ns
Ethnicity (AA/A/C/H/M)	9/0/39/4/3		9/7/45/10/10		--		2.29	.06, ns
Age	10.18	1.40	10.51	1.37	−0.24		2.34	.18, ns
SES	46.92	12.38	48.27	12.35	−0.11		4.48	.53, ns
FSIQ	102.75	14.65	108.02	12.06	−0.40	2.00		.02
BASC-2/3 Attention Problems (T-score)
Parent	66.07	8.16	53.94	10.79	1.24	1.02 x 108		<.001
Teacher	63.11	9.05	54.88	11.35	0.82	154.99		<.001
BASC-2 Hyperactivity (T-score)
Parent	67.31	12.30	53.60	10.30	1.23	8.30 x 107		<.001
Teacher	61.91	14.85	53.93	12.88	0.57	5.76		.007
Executive Function Component Scores (Z-scores)
Working Memory	−0.69	0.95	0.47	0.73	−1.41	1.64 x 1010		<.001
Inhibitory Control	−0.34	1.09	0.23	0.87	−0.60	33.51		<.001
Set Shifting	−0.27	1.32	0.18	0.65	−0.46	4.29		.009

Note. P-values are not corrected for family-wise error, and are included for illustrative purposes to allow interested readers to compare Bayesian and frequentist results. BF₁₀ = Bayes Factor for the alternative hypothesis over the null hypothesis (values ≥ 3.0 indicate significant between-group differences). BF₀₁ = Bayes Factor for the null hypothesis over the alternative hypothesis (values ≥ 3.0 indicate significant between-group equivalence; BF₀₁ = 1/ BF₁₀). BASC = Behavior Assessment System for Children. Teacher BASC data were not collected for the n=40 Non-ADHD children who received the abbreviated screening evaluation (Non-ADHD n=41 for BASC teacher comparisons). Ethnicity: AA = African American, A = Asian, C = Caucasian Non-Hispanic, H = Hispanic, M = Multiracial

Tier 1: Executive Functioning Impairments in ADHD

As shown in Figure 1, 89% of children with ADHD were classified as Impaired on at least one executive function component (n = 49 of 55). This finding corresponded to a median log odds ratio (OR_log) of 3.39, with diagnostic sensitivity of 89.1% and specificity of 80.3% (Supplementary Table 3). Within specific executive function domains, 62% of children with ADHD were classified as Impaired in working memory (OR_log = 2.62), 27% Impaired in inhibitory control (OR_log = 1.49), and 38% Impaired in set shifting (OR_log = 2.36). Approximately half of the children with ADHD showed impairment on a single executive function (54%), and an additional third showed impairments on two (31%) or all three (4%) executive functions.

Figure 1.

Visual heuristic showing the proportion of children with ADHD classified as Impaired on each executive function based on the Jacobson and Truax (1991) model of reliable change. Circle sizes are proportionate to the percentage of children identified as Impaired in each domain. Values do not sum to 100% due to rounding; precise values are given in Supplementary Tables 3a–d.

Only 11% (6 of 55) of children with ADHD were classified as Not Impaired on all three executive functions. Following Coghill and colleagues (2014), we inspected these cases to determine whether they ‘just missed’ the threshold for definition of a deficit. Qualitative inspection of these children’s data suggest that 5 of the 6 Not Impaired children fell just above the impairment cut-off on one of the three executive function components (i.e., they had scores at/below the 20^th percentile relative to the Non-ADHD group; n=4 for working memory, n=1 for inhibitory control); the final Not Impaired child with ADHD showed below average performance on two of the three executive function components (27^th–30^th percentile for inhibitory control and set shifting) but above average working memory (75^th percentile). Formal statistical tests could not be conducted due to low cell counts.

Tier 2: Profiles of Impaired vs. Not Impaired Children

Exploratory analyses were conducted to probe for demographic and behavioral predictors as a function of impairment status, separately for each of the three executive function components. Results are based on the n=96 cases with teacher ratings, and should be considered tentative given the relatively small subgroup sample sizes (Supplementary Table 4).

Demographic characteristics

There was no significant evidence for, and in most cases significant evidence against, differences in age, gender, IQ, SES, or medication status as a function of impairment status on each executive function (all BF₁₀ < 1.26) with one exception: Children with working memory impairment (M_age = 9.91, SD = 1.41) were slightly younger than children without working memory impairment (M_age = 10.82, SD = 1.45) (BF₁₀ = 11.35, d=0.64).

ADHD symptom severity

ADHD symptom profiles associated with impairment in each executive function domain were assessed using a series of 2 (EF Impairment: no/yes) x 2 (informant: parent, teacher) x 2 (ADHD symptom domain: inattention, hyperactivity/impulsivity) Bayesian mixed-model ANOVAs. Informant and symptom domain were treated as within-subject factors to maximize power by including multiple indices per construct and minimizing the number of omnibus models. Main effects are corrected for multiple testing by fixing to 0.5 the prior probability that the null hypothesis holds across all comparisons (JASP Team, 2017; Westfall, Johnson, & Utts, 1997). Separate models were run for each of the three executive function domains (working memory, inhibitory control, set shifting). In all three models there was significant evidence for a main effect of informant (parent > teacher; BF₁₀ = 303.50), but significant evidence against effects of symptom domain (inattention = hyperactivity/impulsivity; BF₀₁ = 6.45) and informant x domain interactions (BF₀₁ > 6.67). The pattern and interpretation of results were unchanged when controlling for age, gender, SES, IQ, and medication status; and when examining just the ADHD sample. Cell sizes were insufficient to compare Impaired vs. Not Impaired Non-ADHD children (i.e., only 4–8 impaired Non-ADHD children per executive function as expected due to our method of defining impairment based on the Non-ADHD distribution).

Working memory

Results indicated significantly higher ADHD symptoms for children with impaired working memory (n=37) relative to children with unimpaired working memory (n=59) (BF₁₀ = 5.23 x 10⁴, d=0.36). Relative to the main effects model, there was significant evidence against all interaction effects, indicating that the impaired working memory subgroup showed higher ADHD symptoms across domains and informants (all BF₀₁ > 4.03; please see Figure accompanying Supplementary Table 4).

Inhibitory control

Results indicated significant evidence against ADHD symptom differences for children with impaired inhibitory control (n=19) relative to children with unimpaired inhibitory control (n=77) (BF₀₁ = 5.80, d=0.04). Relative to the main effects model, there was significant evidence against the inhibition deficit x symptom domain interaction (BF₀₁ = 6.31); results were inconclusive for the inhibition x informant interaction (BF₀₁ = 1.64) and the 3-way interaction (BF₀₁ = 2.19).

Set shifting

Results indicated significant evidence against ADHD symptom differences for children with impaired set shifting (n=24) relative to children with unimpaired set shifting (n=72) (BF₀₁ = 7.34, d=0.02). Relative to the main effects model, there was significant evidence against all interactions (BF₀₁ > 3.09).

Tier 3: Sensitivity Analyses

Finally, we probed the extent to which results were influenced by our use of the reliable change method for defining impairment (Jacobson & Truax, 1991), sample demographics, and specification of orthogonal components. Results based on the reliable change method reported above were highly consistent with results obtained by applying the static 10^th percentile cutoff used in several previous studies of ADHD-related neurocognitive heterogeneity. That is, defining impairment as the 10^th percentile of the Non-ADHD group resulted in 91% of ADHD children being classified as Impaired on at least one executive function (n=50 of 55; sensitivity = 90.91% [80.05, 96.98], specificity = 75.31% [64.47, 84.22]). Highly similar results were also obtained when controlling for age, gender, and SES by residualizing for these variables and then re-computing impairment estimates (82% of ADHD sample classified as impaired; sensitivity = 81.82% [69.10, 90.92], specificity = 80.25% [69.91, 88.27]). Highly similar results were also obtained when allowing covariation among the three executive functioning estimates by specifying an oblique rather than orthogonal factor solution (84% of ADHD sample classified as impaired; sensitivity = 83.64% [71.20, 92.23], specificity = 81.48% [73.30, 89.25]). Finally, we tested a 1-factor solution to probe the extent to which results were influenced by modeling separate executive function components. This overall estimate of executive functioning showed high specificity (98.77% [93.31, 99.97]) but poor sensitivity (34.55% [22.24, 48.58]), supporting ADHD-related heterogeneity across executive function domains.

Discussion

The current study was the first to comprehensively assess executive functioning heterogeneity in pediatric ADHD, using a carefully phenotyped sample of children with ADHD (Coghill et al., 2014), multiple criterion tests of each primary executive function (working memory, inhibitory control, set shifting; Miyake et al., 2000), and objectively-defined impairment (Jacobson & Truax, 1991). Results indicate that 89% of children with ADHD demonstrated executive functioning deficit(s), with sensitivity (89%) and specificity (80%) estimates that are similar to or exceed the diagnostic utility of ADHD symptom checklists (Tarle et al., 2017). Together, these estimates are considerably higher than previous reports and, if replicated, suggest that cognitively-informed test batteries may allow for the identification of a relatively small number of causal pathways to the ADHD phenotype (Coghill et al., 2005). Stated differently, the current findings confirm that neurocognitive heterogeneity is a valid phenomenon in ADHD (only 35% were impaired when defining executive function as a unitary, 1-component construct), but that this heterogeneity may be more circumscribed than previously suggested with 89% showing impairment in at least one of the three primary executive functions. The detection of neuropsychologically impaired subtypes (Nigg et al., 2005) has the potential to inform etiological models of ADHD, identify novel intervention targets, and tailor interventions to maximize efficacy (Chacko et al., 2014).

Approximately 10% of children with ADHD were classified as unimpaired on all three executive functions. We hypothesized that there would be an ADHD subgroup with intact executive functioning, and these results appear to support this prediction. Several additional neurocognitive processes have been proposed as candidate causal mechanisms in ADHD, including altered reinforcement gradients and delay aversion, temporal processing (timing) deficits, response inconsistency/variability, hypo-arousal and/or -activation, and slowed processing speed (Fair et al., 2012; Sonuga-Barke et al., 2008). Although process analysis suggests that poor performance on tests of delay aversion (Patros et al., 2015), response variability (Kofler et al., 2014), and emotion dysregulation (van Cauwenberge et al., 2015) by children with ADHD may be parsimoniously explained by associations with executive dysfunction, these abilities were not assessed in the current study and merit consideration in future work. Alternatively, inspection of subject-level data from the six incorrectly-classified children with ADHD suggested below average performance that in most cases ‘just missed’ our objectively-defined cut-offs (Coghill et al., 2014). Future work is needed to determine the threshold below which underdeveloped executive function(s) produce that hallmark ADHD phenotype, whether individual symptoms are associated with specific EF deficits, and whether children with ADHD show differential functional impairments based on their executive function profile. Biederman et al. (2004), for example, showed that executive dysfunction predicted significantly lower academic achievement, and Kofler et al. (2016) demonstrated differential associations across functional impairment domains.

A small group of 4–8 Non-ADHD children were classified as impaired on each executive function. Although this result was expected, and in large part a statistical artifact of defining impairment based on the Non-ADHD distribution, it highlights the need to consider trait variation in the typically developing population (Fair et al., 2012) when developing and refining causal models of ADHD (Coghill et al., 2005). In the current study, there were too few misclassified Non-ADHD children to conduct inferential statistics. Future work is needed to determine whether Non-ADHD children with executive dysfunction show elevated but subclinical ADHD symptoms, whether these children reflect clinical control cases, and/or whether these children possess protective personal and social assets (Lerner et al., 2009) that buffer against ADHD despite this risk factor.

A key assumption underlying the current study’s methods and conclusions is that executive function deficits are causal mechanisms responsible, in part, for manifestation of the ADHD phenotype. It is important to note that causality cannot be assumed from the current results, and to acknowledge alternate conceptualizations of executive function deficits in ADHD (e.g., epiphenomenal, third variable influences; van Lieshout et al., 2013). Indeed, although our test battery demonstrates that most children with ADHD have executive functioning deficits, it cannot inform whether these deficits occur due to impaired top-down processing or bottom-up signaling that should have recruited the top-down activity (Sonuga-Barke et al., 2008). Further, the extent to which an executive function deficit is a cause, correlate, or effect may differ from child to child and from executive function to executive function. Proximal environmental influences appear to exert significant influences on state-based ADHD behaviors as a function of their executive function demands (Kofler et al., 2016). For example, evidence from carefully controlled experimental studies suggests that ADHD-related inattentive and hyperactive behavior can be produced and rarefied by manipulating working memory demands (Hudec et al., 2015; Kofler et al., 2010; Rapport et al., 2009), whereas similar manipulations of inhibition demands have failed to systematically affect hyperactive behavior (Alderson et al., 2012). Similarly, large-scale longitudinal studies show that age-related reductions in ADHD symptoms are predicted by improvements in working memory but not inhibitory control (Karalunas et al., 2017). Future work is needed to determine whether set shifting produces similar experimental/longitudinal changes, clarify the causal role of each executive function in ADHD-related behavioral and functional impairments, and determine whether neurologically informed subtypes provide clinical utility for maximizing treatment efficacy via improved targeting (Chacko et al., 2014; Nigg et al., 2005).

Limitations

The current study was the first to comprehensively assess executive functioning in ADHD using a criterion battery of cognitively-informed tests, objectively-defined impairment criteria that explicitly accounted for measurement unreliability, and a relatively large sample of carefully phenotyped children with and without ADHD. Despite these methodological refinements, the following limitations must be considered when interpreting results. Given that co-occurring conditions are common in ADHD (Wilens et al., 2002), inclusion of children with these comorbidities was important to maximize external validity and generalizability of our findings. We attempted to balance external and internal validity threats by recruiting a Non-ADHD group matched for the number of these Non-ADHD disorders; however, controlling for the number of other disorders does not perfectly equate the groups, and as such future work is needed to determine how more ‘pure’ ADHD samples compare to non-disordered children – particularly given that neither executive dysfunction nor behavioral symptoms (e.g., difficulty concentrating, restlessness) appear unique to ADHD (e.g., Snyder, 2013; Youngstrom et al, 2010). Independent replications with larger samples, naturalistic outcomes, and a broader sampling of children with other clinical disorders are needed to assess the extent to which the executive dysfunction profiles identified herein provide meaningful prediction toward specific symptoms and ecologically-valid impairment domains.

Children with impaired working memory exhibited moderately higher parent- and teacher-reported ADHD symptoms than children with unimpaired working memory, whereas there was significant evidence against such differences as a function of inhibitory control and set shifting deficits. This pattern of results appears consistent with ADHD conceptual models highlighting equifinality, to the extent that the vast majority of the ADHD group demonstrated impairment in at least one EF domain (i.e., multiple combinations of impairments leading to the same level of ADHD symptoms; e.g., Sonuga-Barke et al., 2008). Alternatively, it may reflect restricted range associated with our stringent ADHD criteria, potentially in combination with the relatively small subgroup sample sizes for these analyses.

Importantly, our fractionation of executive functioning into distinct domains of working memory, inhibitory control, and set shifting was consistent with theory and empirical work (Miyake et al., 2000). At the same time, these executive sub-functions are themselves not unitary constructs. For example, working memory can be further fractionated into at least three central executive processes (updating, dual-processing, temporal reordering; the ‘working’ components of working memory; Rapport et al., 2013) as well as three non-executive short-term memory stores (phonological, visual-spatial, episodic; Baddeley, 2012), each of which is functionally and anatomically distinguishable (Wager & Smith, 2003). Similarly, inhibitory control can be fractionated into processes involved in action cancellation and action restraint (Alderson et al., 2007); we are unaware of any studies documenting subcomponents of set shifting. It remains to be seen whether ADHD is associated globally or locally with some/all of these executive subprocesses.

Clinical and Research Implications

The current results confirm that ADHD is characterized by neurocognitive heterogeneity (Nigg et al., 2005), while suggesting that applying contemporary criteria informed by cognitive science to executive function may provide improved precision for identifying a smaller number of neuropsychologically impaired subtypes than previously described. If replicated, these findings would indicate that executive dysfunction is more prevalent in ADHD than previously estimated, and provide a strong empirical basis for a line of basic and applied research to determine whether classifying children with ADHD based on their executive function profile provides a meaningful mechanism toward understanding functional impairments, predicting outcomes, and improving treatment efficacy via improved targeting and neurocognitively-informed modifications (Chacko et al., 2014).