Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2023 Nov 7.
Published in final edited form as: Child Neuropsychol. 2019 Dec 26;26(6):754–769. doi: 10.1080/09297049.2019.1708295

Measurement of executive functioning with the National Institute of Health Toolbox and the association to anxiety/depressive symptomatology in childhood/adolescence

Brian C Kavanaugh a,b, Mary Kathryn Cancilliere c, Alexa Fryc c, Eric Tirrell b,d, Jane Oliveira a,b, Lindsay M Oberman e, Bruce E Wexler f, Linda L Carpenter b,d, Anthony Spirito b
PMCID: PMC10629577  NIHMSID: NIHMS1941794  PMID: 31876232

Abstract

Introduction:

Despite preliminary research, there remain inconsistent findings with regard to the role of executive functioning (EF) deficits in childhood anxiety and depression. This report examined the association of The National Institute of Health (NIH) Toolbox to clinical neuropsychological measures and to childhood, anxiety/depressive symptomatology. Methods: One-hundred eight children and adolescents completed the three EF measures from the NIH Toolbox (List Sorting Working Memory Test [LSWMT], Dimensional Change Card Sorting Test [DCCST], and Flanker Test of Attention and Inhibition [Flanker]) in an outpatient neuropsychology program. These tests were compared to established measures of EF in terms of linear correlations and detection of impairment. Heaton’s Global Deficit Score (GDS) was utilized to calculate impairment. The Toolbox-EF measures were paired with parent-reported EF symptoms (Behavior Rating Inventory of Executive Function [BRIEF2]) to identify the role of EF in childhood anxiety/depressive symptomatology.

Results:

Toolbox-EF measures displayed medium sized correlations with their clinically comparable counterparts, and generally did not differ in their detection of impairment. Toolbox-GDS was associated with depression diagnosis and clinically significant child-reported anxiety and depressive symptoms. Together, Toolbox/BRIEF2 accounted for 26.8–30.9% of elevated depressive symptom variance, but only 13.2–14% of elevated anxiety symptom variance. Further, EF impairment was associated with depression across self report, parent report, and clinical diagnosis.

Discussion:

The NIH Toolbox-EF measures display comparable psychometric properties to clinically available EF measures in a pediatric (primarily psychiatric) neuropsychology setting. The Toolbox appears to display an appropriate ability to detect EF deficits secondary to self-reported depression in childhood.

Keywords: Executive functioning, NIH Toolbox, childhood, anxiety, depression

Psychiatric disorders are the leading cause of disability in the United States (Murray et al., 2013) and 22% of U.S. adolescents will experience at least one severely impairing psychiatric disorder (Merikangas et al., 2010). Psychiatric disorders are also the most costly to treat of any childhood health conditions (e.g. asthma, physical trauma, etc.), with $13.8 billion spent on treatment for psychiatric disorders in 2011 (Soni, 2014). Cognitive dysfunction is a core, transdiagnostic component of numerous psychiatric disorders (Masand & Pae, 2015; Millan et al., 2012) and occurs in as many as half of individuals with affective disorders (Godard, Grondin, Baruch, & Lafleur, 2011; Gu et al., 2016; Gualtieri & Morgan, 2008; Wagner et al., 2018). One of the most commonly affected cognitive domains in psychopathology is executive functioning (EF) (Etkin, Gyurak, & O’Hara, 2013; Snyder, Miyake, & Hankin, 2015).

EF is a collection of self-regulatory control processes that are divided into core subdomains of working memory (i.e. maintaining/manipulating data not perceptually present), inhibition (i.e. suppressing or controlling attention, thoughts, behaviors) and flexibility (i.e. shifting flexibly between tasks/sets) (Diamond, 2013; Miyake et al., 2000). Significant EF differences between cases and controls have been consistently identified across the majority of adult (Millan et al., 2012; Snyder et al., 2015) and childhood (Willcutt, Sonuga-Barke, Nigg, & Sergeant, 2008) affective disorders. EF predicts a host of clinical outcomes in studies of adult affective psychopathology, including long-term functional recovery (Gruber, Rosso, & Yurgelun-Todd, 2008; Jaeger, Berns, Loftus, Gonzalez, & Czobor, 2007), overall functioning (Martino et al., 2009), quality of life (Cotrena, Branco, Shansis, & Fonseca, 2016), and social/occupational functioning (O’Donnell et al., 2017; Withall, Harris, & Cumming, 2009). EF deficits are state-independent and persist during affective episode remission (Lee, Hermens, Porter, & Redoblado-Hodge, 2012).

Starting in early childhood, EF is a highly stable, heritable, and transdiagnostic trait-like feature of psychopathology (Bannon, Gonsalvez, Croft, & Boyce, 2006; Friedman et al., 2016; Hatoum, Rhee, Corley, Hewitt, & Friedman, 2018; Polderman et al., 2007). EF deficits in child/adolescent psychopathology has been associated with poorer academic outcomes (Biederman et al., 2004; Miller & Hinshaw, 2010), global functioning (Clark, Prior, & Kinsella, 2002; Gardiner & Iarocci, 2018; Miller & Hinshaw, 2010; Ware et al., 2012), social functioning (Miller & Hinshaw, 2010; Rinsky & Hinshaw, 2011), and long-term functional outcomes (Lee et al., 2013; Miller & Hinshaw, 2010). In children and adolescents, lower cognitive flexibility has been associated with higher depressive symptoms and predictive of higher anxiety symptoms at follow-up (Han et al., 2016). Lower inhibition has been associated with polygenetic risk for depression (Schork et al., 2018) and higher general internalizing symptoms (Vuontela et al., 2013), including specific depressive symptoms of negative mood, low self-esteem, and interpersonal problems (Kavanaugh et al., 2016; Kavanaugh & Holler, 2014a, 2014b). Lower working memory has also been associated with higher anhedonia (Kavanaugh & Holler, 2014b) and higher internalizing symptoms at follow-up (Rinsky & Hinshaw, 2011).

The National Institute of Mental Health’s Research Domain Criteria (RDoC) calls for the examination of transdiagnostic constructs across multiple levels of measurement (e.g., genetic, neural, cognitive) to improve the understanding on the underlying neurobiological underpinnings (Garvey, Avenevoli, & Anderson, 2016). The RDoC criteria therefore focuses on endophenotypes that may cut across psychiatric disorders. EF has received considerable interest as a potential endophenotype or critical underlying construct in affective psychopathology. Despite the potential for neuropsychology to integrate into neuroscience research, RDoC has specifically recommended that researchers avoid implementing clinical neuropsychological tests to measure neurocognitive constructs (National Advisory Mental Health Council Workgroup on Tasks and Measures for Research Domain Criteria (RDoC), 2016). As has been well-described by Robert Bilder, this concern from our colleagues may in part be because most commonly utilized neuropsychological tests were developed over 50 years ago (Bilder, 2011; Bilder & Reise, 2019). Neuropsychology must transition into tests that more appropriately integrate modern technology/neuroscience to maintain its relevance in a quickly changing healthcare and research climate.

The National Institute of Health Toolbox for the Assessment of Neurological and Behavioral Function (NIH Toolbox) is a new battery of neurobehavioral measures that utilizes an iPad to efficiently assess a range of functions (including cognition) in clinical and research settings across the lifespan (Gershon et al., 2013). The purpose of this study was to examine the association between EF and childhood anxiety/depression in an outpatient pediatric neuropsychology setting, with a particular focus on child/adolescent psychiatry. First, it was hypothesized that the EF measures from the NIH Toolbox (i.e. Toolbox-EF) would be closely associated with more established, clinical neuropsychological tests. Second, it was hypothesized that the Toolbox-EF and the Behavior Rating Inventory of Executive Function (BRIEF2) would independently predict unique aspects of anxiety/depressive symptomatology, specifically clinically significant parent and child-reported anxiety/depressive symptoms and clinically-assigned anxiety/depression diagnoses.

Methods

Participants

All children and adolescents (N = 108; 5 to 18 years; 5 year olds, n = 2; 65.7% Male) that completed the EF measures on the NIH Toolbox within a single pediatric neuropsychology program were included in this study (2017–2019). No exclusion criteria were implemented in order to examine the potential utility of the Toolbox-EF measures across the full range of neurocognitive/psychiatric functioning. Participants completed the measures within the outpatient neuropsychology program as either (1) clinical patients, (2) research participants, or (3) both clinical patients and research participants. Specifically, 78 participants completed the measures as part of an outpatient pediatric neuropsychological evaluation. Children referred for evaluation within this service generally experience potential neurocognitive deficits secondary to a primary psychiatric, neurodevelopmental, or medical/neurological disorder. A separate group of 25 participants completed the measures as part of participation in a cognitive training research study within the same neuropsychology program. The research study examined the feasibility of an internet-based, cognitive training program for children and adolescents with EF deficits. The study was advertised in the hospital and in the community. Finally, 5 participants were initially evaluated as outpatients but then later enrolled in the cognitive training study. IRB approval was obtained for the cognitive training study and the chart review of the clinical evaluations, both of which were conducted within the outpatient neuropsychology program.

Demographic data for the entire study sample are provided in Table 1. No research procedures were conducted to establish psychiatric diagnoses and the psychiatric diagnoses reported here were clinically assigned prior to the study. Depressive disorder criteria included major depressive disorder, depressive disorder unspecified, and disruptive mood dysregulation disorder. Anxiety disorder criteria included generalized anxiety disorder, panic disorder, specific phobias, anxiety disorder unspecified, obsessive-compulsive disorder, and post-traumatic stress disorder. Regarding medication, psychostimulants and alpha-2 agonists (i.e. ADHD-related medications) were loaded into analyses as a covariate due to the known effect of these medications on EF (Coghill et al., 2014; van Stralen, 2018).

Table 1.

Demographic and clinical data (n = 108).

Male 65.7% (n = 71)
Age 10.42 (3.05)
Full Scale Intelligence (FSIQ) 94.70 (14.65)
Outpatient Mental Health Services 81.6% (n = 21)
Grade 4.95 (3.02)
Academic Services: IEP/504 Plan 59% (n = 63)
Mother/Primary Caregiver Education n = 97
 HS Degree/GED 31.9% (n = 31)
 College Degree 35% (n = 34)
 Master’s or Doctoral Degree 27.8% (n = 27)
Father/Secondary Caregiver Education n = 85
 HS Degree/GED 42.3% (n = 36)
 College Degree 31.7% (n = 27)
 Master’s or Doctoral Degree 17.6 (n = 15)
Parent & Child Primary Language: English 100%
Child’s Race
 Asian 2.0% (n = 2)
 Black or African American 6.1% (n = 6)
 White 74.7% (n = 74)
 Other 17.2% (n = 17)
Child’s Ethnicity
 Hispanic 14.1% (n = 14)
 Non-Hispanic 85.9% (n = 85)
Psychiatric Diagnoses
 Adjustment Disorder 9% (n = 10)
 Anxiety Disorder 54% (n = 58)
 Attention Deficit Hyperactivity Disorder 81% (n = 88)
  Inattentive Type 14% (n = 15)
  Hyperactive/Impulsive Type 3% (n = 3)
  Combined Type 55% (n = 59)
  Unspecified Type 10% (n = 11)
 Autism Spectrum Disorder 9% (n = 10)
 Bipolar Disorder 1% (n = 1)
 Depressive Disorder 34% (n = 37)
 Developmental Coordination Disorder 5% (n = 5)
 Intellectual Disability 5% (n = 5)
 Language Disorder 5% (n = 5)
 Learning Disorders 25% (n = 27)
 Obsessive-Compulsive Disorder 2% (n = 2)
 Oppositional Defiant Disorder 9% (n = 10)
 Post-Traumatic Stress Disorder 6% (n = 6)
 Psychosis 1% (n = 1)
 Social Communication Disorder 5% (n = 5)
 Tic Disorder 3% (n = 3)
Psychiatric Medications
 Alpha-2 Agonist 17% (n = 18)
 Antidepressant 31% (n = 33)
 Anxiolytic 1% (n = 1)
 Atypical Antipsychotic 6% (n = 7)
 Mood/AED 6% (n = 7)
 Stimulant 38% (n = 41)
Medical/Neurological Disorders 9% (n = 10)
Open in a new tab

Note. IEP: Individualized Education Program. Medical/neurological disorders: Hippocampal dysplasia, hypothyroidism, spondyloepiphyseal dysplasia, concussion, acute lymphoblastic leukemia, brain tumor, rhabdomysarcoma, HIV, epilepsy

Measures

Self-report

The Children’s Depression Inventory-Second Edition, self-report (CDI-2) is a self-rated scale of depressive symptoms in children aged 7–17 years (Kovacs, 2011). It includes 28 multiple-choice items that are categorized into two scales of Emotional Problems and Functional Problems, and four subscales of Negative Mood/Physical Symptoms, Low Self-Esteem, Ineffectiveness, and Interpersonal Problems. Internal consistency of the self-report CDI-2 has been high, with alpha coefficients above .80 across various studies (Kovacs, 2011). The Screen for Child Anxiety Related Disorders, self-report (SCARED) is a self-rated scale of anxiety symptoms in children ages 8–18 years (Birmaher et al., 1999). It includes 41 multiple-choice items that are categorized into subscales of Panic/Somatic, Generalized Anxiety, Social Anxiety, Separation Anxiety, and School Avoidance, with excellent internal consistency (Cronbach’s alpha = .90). The total scores (T-scores: CDI; raw: SCARED) were utilized from each of these scales. Only participants within the recommended age range were administered these questionnaires.

Parent-report

The Behavior Rating Inventory of Executive Function-Second Edition (BRIEF2), parent form (Gioia, Isquith, Guy, & Kenworthy, 2015) is a parent-rated scale of executive functioning in children aged 5 to 18 years. It includes 63 multiple-choice items that are categorized into subscales of Inhibit, Shift, Emotional Control, Initiate, Working Memory, Plan/Organize, Organization of Materials, and Task Monitor. Internal consistency is high across both clinical and standardization samples, with alpha coefficients of .85 and .89, respectively (Gioia et al., 2015). As this study is specifically focused on empirically-based core EF subdomains (Diamond, 2013), only Inhibit, Working Memory, and Shift subscales were utilized (T-scores). The Behavior Assessment System for Children – Third Edition (BASC-3), parent form (Reynolds & Kamphaus, 2015) is a parent-rated scale of emotional-behavioral functioning in children aged 6–18. It includes 173 (6–11 years) to 175 (12–18 years) multiple-choice items that are categorized into index scores of Externalizing Problems, Internalizing Problems, Behavioral Symptoms, and Adaptive Skills. This study only utilized the subscales of Anxiety and Depression (T-scores) as a parent-report assessment of symptomatology. Internal-consistency reliability coefficients are good to excellent (with the majority of values above .90) and are consistent across gender, age, and clinical groups (Reynolds & Kamphaus, 2015). Only participants within the recommended age range were administered these questionnaires.

NIH toolbox

The National Institutes of Health Toolbox for the Assessment of Neurological and Behavioral Function (i.e. NIH Toolbox) is a brief neuropsychological screening battery of neurocognition across brain-based disorders (Gershon et al., 2013). The tasks are administered on an iPad, with all instructions presented orally and visually on the screen (O’Donnell et al., 2017). Working memory was assessed with the List Sorting Working Memory Test (LSWMT). Inhibition was assessed with the Flanker Inhibitory Control and Attention Test (Flanker). Flexibility was assessed with the Dimensional Change Card Sort Test (DCCST). Administration of these three EF tasks (i.e. Toolbox-EF measures) takes approximately 15 minutes (Weintraub et al., 2013). While each subdomain is only assessed with a single task, the results of all task can be averaged to create a “common EF” metric to represent the RDoC construct for each patient (Miyake et al., 2000). These tests display excellent psychometric properties in children and adolescents (Tulsky et al., 2013; Zelazo et al., 2013).

Neuropsychological measures

The Wechsler Abbreviated Scale of Intelligence-Second Edition (WASI-II), Wechsler Intelligence Scale for Children-Fourth Edition (WISC-V), or Wechsler Adult Intelligence Scale of Intelligence-Fourth Edition (WAIS-IV) were used to assess intelligence (Wechsler, 2008, 2011, 2014). When available, the Working Memory Scale on WISC-V/WAIS-IV was utilized to assess working memory. The NEPSY-Second Edition (NEPSY-II) subtest, Inhibition, was utilized to measure inhibition (i.e. Inhibition-Inhibition score) and flexibility (i.e. Inhibition-Switching)(Korkman, Kirk, & Kemp, 2007). Standardized scores were utilized for all measures. The standard protocol for the research study and clinical practice generally involves the Toolbox-EF measures, BASC-3, CDI-2/SCARED, NEPSY-II: Inhibition (Switching subtest not administered in research protocol), and WASI-II, WISC-V, or WAIS-IV. The BRIEF2 was a part of the research protocol and originally part of the clinical protocol. During the period that the BRIEF2 was used clinically, it was administered universally to all participants. As is common for standard pediatric neuropsychology practice, the BRIEF2 is administered to obtain data on rater-based EF weaknesses, i.e. EF weaknesses observed in the child’s day-to-day environment. This is complimentary data to that obtained from performance-based EF measures. The BRIEF2 was eventually removed from the standard battery. However, particularly in the clinical sample, the standard protocol was often adjusted based on the referral question and patient-specific characteristics (resulting in variable sample size in this report). Other clinical EF tests were not included here due to small sample size (e.g., Trail Making Test, n = 29).

Global deficit score

Heaton’s Global Deficit Score (GDS) is a measure of neurocognitive deficit severity that utilizes a 5-point scale to categorize test performance from intact (0) to severe impairment (5) (Carey et al., 2004; Heaton et al., 2004). Individual test scores are then averaged together to create a GDS composite score (for each domain or across the whole battery). Specifically, the GDS is rated as 0–5 (utilizing z-scores); 0 = −1.0 and above (i.e. normal); −1.5 ≤ 1 < −1.0 (mild impairment); −2.0 ≤ 2 < −1.5 (mild to moderate impairment); −2.5 ≤ 3 < −2.0 (moderate impairment); −3.0 ≤ 4 < −2.5 (moderate to severe impairment); 5 < −3.0 (severe impairment). A subsequent cutoff of ≥0.5 displays strong properties in detecting neurocognitive impairment in adult HIV samples (Carey et al., 2004; Heaton et al., 2004). This indicates that on average, the participant is at least mildly impaired on 50% of tests (Carey et al., 2004; Heaton et al., 2004). Recently, the GDS was able to predict the length of hospitalization in a children’s inpatient psychiatric program sample. A cut off of .43–.63 displayed optimal sensitivity/specificity in detecting prolonged length of hospitalization (Kavanaugh, Studeny, Cancilliere, & Holler, 2019).

Statistical analyses

The range of clinically obtained standardized scores (e.g. scaled scores, T-scores) were universally converted to z-scores and the GDS was then calculated for each test. The overall GDS was calculated for the NIH Toolbox (i.e. average GDS for the 3 tests; “Toolbox-GDS”) and the BRIEF2 (i.e. average GDS for the three scales of inhibition, working memory, and shifting; scores reversed to be in correct direction; “BRIEF-GDS”). First, Toolbox/BRIEF2 differences between samples were examined. Toolbox-EF measures were then compared to established, clinically available tests that purportedly measure the same underlying EF constructs: Wechsler: WMI and NIH Toolbox: LSWMT, NEPSY-II: Inhibition-Inhibition and the NIH Toolbox: Flanker, and NEPSY-II: Inhibition-Shifting and the NIH Toolbox: DCCST. Pearson correlation analyses examined the associations between GDS scores. Paired samples t-test analyses examined the differences in impairment detection between GDS scores. Analyses of covariance (ANCOVA; after controlling for age, sex, ADHD diagnosis (yes/no), and ADHD-related medications) examined EF differences between participants with/without depressive disorder diagnoses and between participants with/without anxiety disorder diagnoses.

A series of logistic regression analyses examined the role of EF in elevated anxiety/depressive symptoms. In each analysis, age, sex, ADHD diagnosis (yes/no), and ADHD-related medications were loaded and followed by the BRIEF-GDS and Toolbox-GDS. Dependent variables consisted of the BASC-3: Anxiety and Depression subscales, the CDI-2: Total, and the SCARED: Total. Clinically significant or elevated symptoms were defined as a T-score ≥ 60 (i.e. one standard deviation above mean) for BASC-3 and CDI-2. The SCARED does not produce T-scores, and instead has a cutoff of ≥ 25 (or 30 for more severe symptom cutoff). Of note, we could have defined cutoff of T ≥ 70 for BASC-3/CDI-2 and raw ≥ 30 for SCARED to reflect more severe symptom ranges, but we selected the milder cutoff due to concerns about small sample size. Significance levels were set at p < .05.

Results

There was no difference between research and clinic groups in age (F[1,106] = .189; p = .664), sex (F[1,106] = .973; p = .326), race (X2[4] = 6.927, p = .140), maternal education (x2[8] = 11.395, p = .180), Toolbox-GDS (F[1,95] = 2.664; p = .106), and BRIEF-GDS (F[1,58] = .174; p = .678). All subsequent analyses were conducted with the combined sample.

Correlations among measures

As noted in Table 2, there was a statistically significant association between NEPSY-II: Inhibition and NIH Toolbox: Flanker, between WISC-V: WMI and NIH Toolbox: LSWMT, and between NIH Toolbox: DCCST and NEPSY-II: Inhibition-Switching. No Toolbox-EF measures were correlated with BRIEF2 measures (Table 2).

Table 2.

Descriptive and correlational data for toolbox, BRIEF2, and clinical measures.

Descriptive Data Correlation to Toolbox
n Mean (SD) GDS (SD) Impairment/Elevated Flanker LSWMT DCCST
Toolbox: Flanker 105 87.41 (13.75) .89 (1.17) 48.6% - - -
Toolbox: LSWMT 100 94.19 (15.35) .52 (1.01) 28% .177 - -
Toolbox: DCCST 105 91.91 (13.12) .52 (1.06) 26.7% .422** .231* -
NEPSY-II: Inhibition 91 7.42 (3.27) .90 (1.14) 45.1% .332** −.048 .206
NEPSY-II: Switching 57 7.86 (3.28) .58 (1.12) 28.1% .288* .305* .406**
Wechsler: WMI 48 88.92 (16.09) .81 (1.30) 35.4% .308* .403** .381**
BRIEF2-Inhibit 60 67.32 (12.83) 2.2 (1.78) 73.3% −.070 .131 −.160
BRIEF2-WM 60 69.70 (8.90) 2.4 (1.63) 85% −.149 .000 .227
BRIEF2-Shift 60 65.85 (13.39) 2.1 (1.86) 66.7% .146 −.013 −.020
CDI-2: Total 56 57.30 (13.77) - 39.3% .180 −.012 .005
SCARED: Total (raw) 54 25.37 (15.06) - 48.4% .079 −.033 .026
BASC-3: Anxiety (parent) 89 59.24 (12.05) - 47.2% .257* −.071 −.001
BASC-3: Depression (parent) 89 62.36 (13.64) - 39.3% .151 −.148 −.008
Open in a new tab

Impairment: < 1 standard deviation below mean for performance-based EF, > 1 standard deviation above mean for rater-based EF; clinically elevated symptoms: ≥ 60 for CDI-2/BASC-3 and ≥ 25 for SCARED.

Impairment differences across measures

Paired sample t-tests indicated there was no statistically significant difference in GDS between WISC-V: WMI and NIH Toolbox: LSWMT (t [39] = .597; p = .554; GDS = .78 [1.27] & .65 [1.15], respectively) or between NEPSY-II: Inhibition and NIH Toolbox: Flanker (t [88] = .740; p = .461; GDS = .89 [1.14] & .80 [1.09], respectively). There was a statistically significant difference between NEPSY-II: Inhibition-Switching and NIH Toolbox: DCCST (t [54] = −2.083; p = .042; GDS = .53 [1.04] & .25 [.65], respectively), in that the NEPSY-II was more sensitive to impairment than Toolbox.

Differences between depression/no depression and anxiety/no anxiety

As noted in Table 3, there was a statistically significant difference between depression/no depression groups in Toolbox-GDS (F [1, 91] = 4.892, p = .029), but not BRIEF-GDS (F [1, 54] = .005, p = .942). In follow-up analyses of specific EF subdomains, only Flanker (F = 9.800; p = .002), but not LSWMT (F = 1.173; p = .282) or DCCST (F = .055; p = .815) was associated with diagnosis after controlling for previously stated variables. There was no statistically significant difference between anxiety/no anxiety groups in Toolbox-GDS (F [1, 91] = 3.512, p = .064) or BRIEF-GDS (F [1, 54] = .008, p = .930; Table 3).

Table 3.

Toolbox-GDS differences based on clinical diagnoses of anxiety and depressive disorders.

Depressive disorder (n = 32) No depressive disorder (n = 65) F
Toolbox GDS .890 (.14) .516 (.10) 4.892*
BRIEF2 GDS 2.26 (.28) 2.23 (.23) .006
Anxiety Disorder (n = 44) No Anxiety Disorder (n = 53) F
Toolbox GDS .767 (.11) .449 (.13) 3.512
BRIEF2 GDS 2.23 (.23) 2.26 (.28) .008
Open in a new tab

Estimated marginal means and standard errors are reported. F value is reported after controlling for age, sex, ADHD diagnosis, and ADHD-related medications.

*

p < .05.

EF predictors of elevated anxiety/depressive symptoms

All four overall models were statistically significant, including self-reported depressive symptoms (X2 [6] = 21.342, p = .002; 57.8% of total variance [BRIEF/Toolbox: 30.9% of variance]; correctly classified 82.1% of cases), self-reported anxiety symptoms (X2 [6] = 16.697, p = .010; 45.9% of total variance [BRIEF/Toolbox: 13.6% of variance]; correctly classified 75.0% of cases), parent-reported depressive symptoms (X2 [6] = 21.157, p = .002; 44.6% of total variance [BRIEF/Toolbox: 26.8% of variance]; correctly classified 76.9% of cases), and parent-reported anxiety symptoms (X2 [5] = 18.725, p = .005; 40.5% of total variance [BRIEF/Toolbox: 13.0% of variance]; correctly classified 78.8% of cases). As noted in Table 4, after controlling for age, sex, ADHD diagnosis, and ADHD-related medications, BRIEF-GDS independently predicted parent-reported depressive symptoms, while Toolbox-GDS independently predicted self-reported anxiety and depressive symptoms. In follow-up analyses of specific EF subdomains (when significant findings were obtained), only Toolbox: DCCST (B = 1.028; p = .041) predicted self-reported depressive symptoms. No subdomains predicted self-reported anxiety symptoms or parent-reported depressive symptoms.

Table 4.

Logistic regression analyses of EF predictors of elevated anxiety/depressive symptoms.

CDI-2: Total (n = 39) SCARED: Total (n = 35)
B Wald p B Wald p
Age .720 4.095 .043 .269 1.587 .208
Sex 2.683 4.749 .029 2.300 5.830 .016
ADHD Diagnosis 2.747 1.917 .166 −1.106 .509 .476
ADHD Medication 1.062 1.495 .222 .15 .044 .833
BRIEF-GDS .553 1.660 .198 .320 .720 .396
Toolbox-GDS 2.455 6.328 .012 1.301 3.866 .049
BASC-3: Depression (n = 52) BASC-3: Anxiety (n = 52)
B Wald p B Wald p
Age .474 7.758 .005 .472 7.109 .008
Sex −.833 .999 .318 1.413 2.797 .094
ADHD Diagnosis −.909 1.005 .316 −.282 .090 .764
ADHD Medication .563 .893 .345 .729 1.326 .250
BRIEF-GDS .982 8.893 .003 .573 3.834 .050
Toolbox-GDS .595 1.485 .223 .726 2.039 .153
Open in a new tab

EF impairment criteria

Based on these results, we examined the implementation of formal EF impairment diagnostic criteria. Mimicking the GDS approach (Carey et al., 2004; Cysique et al., 2014; Marquine et al., 2018), we set ≥ .5 as neurocognitive impairment for Toolbox-GDS. In the GDS approach (within a dementing, adult population), the neurocognitive GDS is supplemented with a reported decline in 2 to 3 domains of activities of daily living (ADLs). As a decline in ADLs is not as relevant to child psychiatry, we instead implemented the BRIEF2 into the criteria for EF impairment as reflecting the clinical impairment or daily living symptoms of EF. Criteria was set as BRIEF-GDS ≥ 1 (i.e. 1 SD above mean) on any of the inhibition, working memory, or flexibility domains. Both Toolbox and BRIEF2 criteria were required for impairment status. There were no group differences between age, sex, ADHD diagnosis, and ADHD-related medications (Table 5). Significant differences were detected between EF impaired and intact groups in self-reported depressive symptoms, parent-reported depressive symptoms, parent-reported anxiety symptoms, and depression diagnosis, but not self-reported anxiety symptoms and anxiety diagnosis.

Table 5.

Clinical differences between impaired and intact EF.

Intact EF (n = 33) Impaired EF (n = 23) F/X2
Age 10.8 (2.8) 10.1 (2.7) .891
Male 67% 61% .198
ADHD-Related Meds .8 (.7) .6 (.7) 1.142
ADHD Diagnosis 88% 74% 1.802
BASC-3: Depression 60.2 (14.0) 69.1 (15.0) 4.650*
BASC-3: Anxiety 58.0 (11.6) 66.4 (14.9) 5.220*
CDI-2:Total 52.4 (8.7) 60.9 (14.7) 5.084*
SCARED: Total 22.7 (14.1) 25.7 (13.3) .449
Depression Diagnosis 27% 61% 6.321*
Anxiety Diagnosis 48% 65% 1.536
Open in a new tab

Note.

*

p < .05.

Discussion

This study investigated the potential utility of the NIH Toolbox-executive functioning (EF) measures in detecting clinically meaningful neurocognitive impairment. First, the tests were compared to clinically available EF tests. The List Sorting Working Memory Test (LSWMT) is designed to measure working memory, the Flanker Inhibitory Control and Attention Test (Flanker) is designed to measure inhibition, and the Dimensional Change Card Sort Test (DCCST) is designed to measure flexibility. All three tests displayed statistically significant and medium-sized associations with their established clinical counterpart tests (r = .33: inhibition to r = .41; flexibility). The LSWMT and Flanker did not differ from their clinical test counterparts in the detection of impairment, while the DCCST was somewhat less sensitive in detecting flexibility impairment. The rates of impairment were also similar between tests for flexibility (26.7–28.1%), working memory (28–35.4%), and inhibition (45.1–48.6%). Taken together, results indicate that the Toolbox-EF measures are broadly comparable to clinically available EF measures when utilized within the child/adolescent psychiatric setting.

Within this context, this study then examined the role of EF in childhood anxiety and depression. Measured via Toolbox, EF deficits were associated with a clinical diagnosis of depression and self-reported anxiety/depressive symptoms, but not parent-reported symptoms or anxiety diagnosis. Alternatively, EF deficits measured via BRIEF2 were associated with parent-reported depressive symptoms, but not self-reported anxiety/depressive symptoms, parent-reported anxiety symptoms, or clinical diagnoses of anxiety/depression. When examining specific EF subdomains, response inhibition was associated with depressive disorders, while cognitive flexibility was associated with self-reported depressive symptoms. Working memory was not independently associated with any anxiety/depression measures.

These findings are consistent with a prior meta-analysis that identified EF deficits in major depression during childhood/adolescence (Wagner, Müller, Helmreich, Huss, & Tadić, 2015), and extend specific findings regarding the association between inhibition/flexibility and depressive symptoms (Han et al., 2016; Kavanaugh et al., 2016; Kavanaugh & Holler, 2014a, 2014b). One potential novelty of current findings is the utilization of a combination of performance (i.e. Toolbox) and rater (i.e. BRIEF2)-based EF measurement. These measures proved to be complimentary in identifying anxiety/depression, as BRIEF2 measures were more closely related to parent-rated symptoms while Toolbox-EF measures were more closely related to diagnoses and self-reported symptoms. In fact, after controlling for relevant variables, together the Toolbox/BRIEF2 accounted for 26.8–30.9% of elevated depressive symptom variance (along with 13.0–13.6% of elevated anxiety symptom variance). Broadly, results appear to confirm the established the role of EF deficits in depression, although the role of EF deficits in anxiety was not consistently observed. Of note, current findings were obtained within a clinical neuropsychology program (including a high rate of ADHD [81%]) and may not reflect a standard child anxiety/depression presentation.

At the neurobiological level, the link between EF and depression can be explained by their shared neuroanatomical correlates, particularly the frontoparietal network or central executive network (CEN)(Kaiser, Andrews-Hanna, Wager, & Pizzagalli, 2015; Niendam et al., 2012; Pizzagalli, 2011). In this perspective, EF measures may be conceptualized as probes of CEN integrity, with identified EF deficits reflecting dampened or suboptimal CEN activity that could contribute to depression vulnerability. At the neurocognitive level, deficits in cognitive flexibility may lead to perseverative tendencies, such struggles to ignore maladaptive thoughts or feelings. Deficits in response inhibition may similarly cause impulsivity in decision making or information processing, which could potentially lead to a negative interpretation of neutral or positive stimuli (e.g., emotional expressions of others). At the behavioral or measurement level, EF deficits and depression share many symptoms (e.g., irritability, inattention, and rigidity) that can make it hard to differentiate these two symptom clusters.

With the objective of moving toward a universal definition of neurocognitive impairment in child psychopathology, we tested one potential approach to defining EF impairment. Heaton and colleagues have long utilized the Global Deficit Score (GDS) to detect neurocognitive impairment in adult HIV, with a GDS ≥ .5 set as the optimal cutoff for detecting impairment (Carey et al., 2004; Heaton et al., 2004). In that approach, the GDS is utilized to categorize intact (GDS < .5), mildly impaired (.5 ≤ GDS < 1.5), or moderately impaired (GDS ≥ 1.5) neurocognition. This is paired with measurement of activities of daily living (ADLs) to identify asymptomatic neurocognitive impairment (no ADL decline), mild neurocognitive disorder (mild to moderate ADL decline), and dementia (severe ADL decline)(Kamminga et al., 2017; Marquine et al., 2018). In our initial attempt to integrate the GDS into the pediatric setting, we found that a GDS = .43–.63 was able to detect prolonged hospitalization within a children’s inpatient psychiatric program (Kavanaugh et al., 2019).

In an attempt to model the approach of Heaton and build on preliminary findings, we set impairment as GDS > .5 on Toolbox-EF, and to reflect clinical or functional impairment, BRIEF-GDS ≥ 1 (i.e. 1 SD above mean) on any of the inhibition, working memory, or flexibility domains. Compared to those with Intact EF, those with Impaired EF displayed higher parent-reported anxiety/depressive symptoms, higher self-reported depressive symptoms, and higher rates of depression. This provides the first attempt of our knowledge to implement formal diagnostic criteria for EF impairment in childhood psychopathology. The findings are promising and suggest that adapting the GDS model in childhood psychopathology may prove useful (at least for children evaluated within a neuropsychology program). However, this is a small first step that will need to be continuously tested, refined, and potentially replaced by a better approach in the ultimate goal of a universally accepted and empirically supported approach to identifying neurocognitive impairment in childhood psychopathology.

Current results indicate that the Toolbox-EF measures are at least comparable to established neuropsychological measures and sensitive/specific enough to detect EF deficits underlying childhood depression. There may be a few potential advantages of the Toolbox-EF measures. The measures can be administered/scored in ~15 minutes by bachelors-level professionals (e.g., testing technicians and research assistants). As the measures are administered or guided by verbal/visual cues from the iPad, there may be improved standardization as compared to standard neuropsychological measures. The measures can be administered in both clinical and research settings, and in collaboration with a neuropsychologist, can be interpreted by non-neuropsychological providers. The field of neuropsychology needs to move toward a new era of neurocognitive measurement, and the NIH Toolbox reflects one, of many, promising tools to help achieve this goal.

This study is not without limitations. First, this sample included only those children and adolescents with potential neurocognitive impairments, including those children referred for neuropsychological evaluation and those children recruited to participate in a cognitive training study. Future studies will need to obtain a spectrum of psychopathology and EF (enriched for clinically elevated symptoms) to most appropriately examine the association between these variables. Second, the missing data was not handled with complex analyses such as maximum likelihood or multiple imputation. This may have resulted in differences between children that were included and excluded from each analysis. Future studies will need to address this limitation. Third, clinical measures of overall depression and anxiety were utilized that capture a range of depression/anxiety-related symptoms. Future studies will need to utilize more specific RDoC-consistent measures of positive and negative valence to systematically investigate the EF-affect association. Fourth, no measures of performance validity (i.e. tests to establish the validity of neurocognitive performance; e.g., Test of Memory Malingering [TOMM]) were consistently implemented into the battery. Future studies must utilize stand-alone measures of performance validity to appropriately measure and interpret EF.

Funding

This work was supported by the Rhode Island Foundation [20174332]; APA Division 40; Thrasher Research Fund [14511].

Footnotes

Disclosure statement

No potential conflict of interest was reported by the authors.

References

  1. Bannon S, Gonsalvez CJ, Croft RJ, & Boyce PM (2006). Executive functions in obsessive-compulsive disorder: State or trait deficits? Australian and New Zealand Journal of Psychiatry, 40, 1031–1038. [DOI] [PubMed] [Google Scholar]
  2. Biederman J, Monuteaux MC, Doyle AE, Seidman LJ, Wilens TE, Ferrero F, … Faraone SV (2004). Impact of executive function deficits and attention-deficit/hyperactivity disorder (ADHD) on academic outcomes in children. Journal of Consulting and Clinical Psychology, 72(5), 757–766. [DOI] [PubMed] [Google Scholar]
  3. Bilder RM (2011). Neuropsychology 3.0: Evidence-based science and practice. Journal of the International Neuropsychological Society : JINS, 17(1), 7–13. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Bilder RM, & Reise SP (2019). Neuropsychological tests of the future: How do we get there from here? The Clinical Neuropsychologist, 33(2), 220–245. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Birmaher B, Brent DA, Chiappetta L, Bridge J, Monga S, & Baugher M (1999). Psychometric properties of the screen for child anxiety and related emotional disorders (SCARED): A replication study. Journal of the American Academy of Child and Adolescent Psychiatry, 38(1230–1236). [DOI] [PubMed] [Google Scholar]
  6. Carey CL, Woods SP, Gonzalez R, Conover E, Marcotte TD, Grant I, & Heaton RK; the HNRC Group. (2004). Predictive validity of global deficit scores in detcting neuropsychological impairment in HIV infection. Journal of Clinical and Experimental Neuropsychology, 26, 307–319. [DOI] [PubMed] [Google Scholar]
  7. Clark P, Prior M, & Kinsella G (2002). The relationship between executive function abilities, adaptive behavior, and academic achievement in children with externalizing behavior problems. Journal of Child Psychology and Psychiatry, 43, 785–796. [DOI] [PubMed] [Google Scholar]
  8. Coghill DR, Seth S, Pedroso S, Usala T, Currie J, & Gagliano A (2014). Effects of methylphenidate on cognitive functions in children and adolescents with attention-deficit/hyperactivity disorder: Evidence from a systematic review and a meta-analysis. Biological Psychiatry, 76(8), 603–615. [DOI] [PubMed] [Google Scholar]
  9. Cotrena C, Branco LD, Shansis FM, & Fonseca RP (2016). Executive function impairments in depression and bipolar disorder: Association with functional impairment and quality of life. Journal of Affective Disorders, 190, 744–753. [DOI] [PubMed] [Google Scholar]
  10. Cysique LA, Heaton RK, Kamminga J, Lane T, Gates TM, Moore DM, … Brew BJ (2014). HIV-associated neurocognitive disorder in Australia: A case of a high-functioning and optimally treated cohort and implications for international neuroHIV research. Journal of Neurovirology, 20(3), 258–268. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Diamond A (2013). Executive functions. Annual Review of Psychology, 64, 135–168. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Etkin A, Gyurak A, & O’Hara R (2013). A neurobiological approach to the cognitive deficits of psychiatric disorders. Dialogues in Clinical Neuroscience, 15, 419–429. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Friedman NP, Miyake A, Altamirano LJ, Corley RP, Young SE, Rhea SA, & Hewitt JK (2016). Stability and change in executive function abilities from late adolescence to early adulthood: A longitudinal twin study. Developmental Psychology, 52(2), 326–340. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Gardiner E, & Iarocci G (2018). Everyday executive function predicts adaptive and internalizing behavior among children with and without autism spectrum disorder. Autism Research, 11(2), 284–295. [DOI] [PubMed] [Google Scholar]
  15. Garvey M, Avenevoli S, & Anderson K (2016). The national institute of mental health research domain criteria and clinical research in child and adolescent psychiatry. Journal of the American Academy of Child & Adolescent Psychiatry, 55(2), 93–98. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Gershon RC, Wagster MV, Hendrie HC, Fox NA, Cook KF, & Nowinski CJ (2013). NIH Toolbox for assessment of neurological and behavioral function. Neurology, 80(11 Suppl 3), S2–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Gioia G, Isquith PK, Guy SC, & Kenworthy L (2015). Behavior rating inventory of executive function, second edition, professional manual. Lutz, FL: Psychological Assessment Resources. [Google Scholar]
  18. Godard J, Grondin S, Baruch P, & Lafleur MF (2011). Psychosocial and neurocognitive profiles in depressed patients with major depressive disorder and bipolar disorder. Psychiatry Research, 190(2–3), 244–252. [DOI] [PubMed] [Google Scholar]
  19. Gruber SA, Rosso IM, & Yurgelun-Todd D (2008). Neuropsychological performance predicts clinical recovery in bipolar patients. Journal of Affective Disorders, 105(1–3), 253–260. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Gu CZ, He HL, Duan HF, Su ZH, Chen H, & Gan JL (2016). Predictors of neurocognitive impairment at 2years after a first-episode major depressive disorder. Comprehensive Psychiatry, 68, 24–33. [DOI] [PubMed] [Google Scholar]
  21. Gualtieri CT, & Morgan DW (2008). The frequency of cognitive impairment in patients with anxiety, depression, and bipolar disorder: An unaccounted source of variance in clinical trials. Journal of Clinical Psychiatry, 68, 1122–1130. [DOI] [PubMed] [Google Scholar]
  22. Han G, Helm J, Iucha C, Zahn-Waxler C, Hastings PD, & Klimes-Dougan B (2016). Are executive functioning deficits concurrently and predictively associated with depressive and anxiety symptoms in adolescents? Journal of Clinical Child & Adolescent Psychology, 45(1), 44–58. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Hatoum AS, Rhee SH, Corley RP, Hewitt JK, & Friedman NP (2018). Etiology of stability and growth of internalizing and externalizing behavior problems across childhood and adolescence. Behavior Genetics. doi: 10.1007/s10519-018-9900-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Heaton RK, Marcotte TD, Mindt MR, Sadek J, Moore DJ, Bentley H, … Group H. (2004). The impact of HIV-associated neuropsychological impairment on everyday functioning. Journal of the International Neuropsychological Society, 10(3), 317–331. [DOI] [PubMed] [Google Scholar]
  25. Jaeger J, Berns S, Loftus S, Gonzalez C, & Czobor P (2007). Neurocognitive test performance predicts functional recovery from acute exacerbation leading to hospitalization in bipolar disorder. Bipolar Disorders, 9, 93–102. [DOI] [PubMed] [Google Scholar]
  26. Kaiser RH, Andrews-Hanna JR, Wager TD, & Pizzagalli DA (2015). Large-scale network dysfunction in major depressive disorder: A meta-analysis of resting-state functional connectivity. JAMA Psychiatry (Chicago, Ill.), 72(6), 603–611. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Kamminga J, Bloch M, Vincent T, Carberry A, Brew BJ, & Cysique LA (2017). Determining optimal impairment rating methodology for a new HIV-associated neurocognitive disorder screening procedure. Journal of Clinical and Experimental Neuropsychology, 39(8), 753–767. [DOI] [PubMed] [Google Scholar]
  28. Kavanaugh B, Studeny J, Cancilliere MK, & Holler KA (2019). Neurocognitive predictors of length of stay within a children’s psychiatric inpatient program. Child Neuropsychology. doi: 10.1080/09297049.2019.1617843 [DOI] [PubMed] [Google Scholar]
  29. Kavanaugh BC, Gaudet CE, Dupont-Frechette JA, Tellock PP, Maher ID, Haisley LD, & Holler KA (2016). Failure to maintain set as a predictor of childhood depression within a children’s psychiatric inpatient sample. Psychiatry Research, 246, 644–649. [DOI] [PubMed] [Google Scholar]
  30. Kavanaugh BC, & Holler KA (2014a). Executive functioning and self-reported depressive symptoms within an adolescent inpatient population. Applied Neuropsychology. Child, 3(2), 126–134. [DOI] [PubMed] [Google Scholar]
  31. Kavanaugh BC, & Holler KA (2014b). Executive, emotional, and language functioning following childhood maltreatment and the influence of pediatric PTSD. Journal of Child & Adolescent Trauma, 7(2), 121–130. [Google Scholar]
  32. Korkman M, Kirk U, & Kemp SL (2007). NEPSY II. Clinical and interpretative manual. San Antonio, TX: Psychological Corporation. [Google Scholar]
  33. Kovacs M (2011). Children’s depression inventory 2 (CDI 2). North Tonawanda, NY: Multi-Health Systems, Inc. [Google Scholar]
  34. Lee RS, Hermens DF, Porter MA, & Redoblado-Hodge MA (2012). A meta-analysis of cognitive deficits in first-episode major depressive disorder. Journal of Affective Disorders, 140 (2), 113–124. [DOI] [PubMed] [Google Scholar]
  35. Lee RS, Hermens DF, Redoblado-Hodge MA, Naismith SL, Porter MA, Kaur M, … Hickie IB (2013). Neuropsychological and socio-occupational functioning in young psychiatric outpatients: A longitudinal investigation. PloS One, 8(3), e58176. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Marquine MJ, Flores I, Kamat R, Johnson N, Umlauf A, Letendre S, … Heaton RK (2018). A composite of multisystem injury and neurocognitive impairment in HIV infection: Association with everyday functioning. Journal of Neurovirology, 24(5), 549–556. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Martino DJ, Marengo E, Igoa A, Scapola M, Ais ED, Perinot L, & Strejilevich SA (2009). Neurocognitive and symptomatic predictors of functional outcome in bipolar disorders: A prospective 1 year follow-up study. Journal of Affective Disorders, 116(1–2), 37–42. [DOI] [PubMed] [Google Scholar]
  38. Masand PS, & Pae CU (2015). Cognitive dysfunction in psychiatric illness: A neglected domain. The Journal of Clinical Psychiatry, 76(3), e375–377. [DOI] [PubMed] [Google Scholar]
  39. Merikangas KR, He JP, Brody D, Fisher PW, Bourdon K, & Koretz DS (2010). Prevalence and treatment of mental disorders among U.S. children in the 2001–2004 NHANES. Pediatrics, 125, 75–81. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Millan MJ, Agid Y, Brune M, Bullmore ET, Carter CS, Clayton NS, … Young LJ (2012). Cognitive dysfunction in psychiatric disorders: Characteristics, causes and the quest for improved therapy. Nature Reviews. Drug Discovery, 11(2), 141–168. [DOI] [PubMed] [Google Scholar]
  41. Miller M, & Hinshaw SP (2010). Does childhood executive function predict adolescent functional outcomes in girls with ADHD? Journal of Abnormal Child Psychology, 38(3), 315–326. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Miyake A, Friedman NP, Emerson MJ, Witzki AH, Howerter A, & Wager TD (2000). The unity and diversity of executive functions and their contributions to complex “Frontal Lobe” tasks: A latent variable analysis. Cognitive Psychology, 41(1), 49–100. [DOI] [PubMed] [Google Scholar]
  43. Murray CJ, Atkinson C, Bhalla K, Birbeck G, Burstein R, Chou D, … Collaborators, U. S. B. O. D. (2013). The state of US health, 1990–2010: Burden of diseases, injuries, and risk factors. JAMA, 310(6), 591–608. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. National Advisory Mental Health Council Workgroup on Tasks and Measures for Research Domain Criteria (RDoC). (2016). Behavioral assessment methods for RDoC constructs. Bethesda, MD. [Google Scholar]
  45. Niendam TA, Laird AR, Ray KL, Dean YM, Glahn DC, & Carter CS (2012). Meta-analytic evidence for a superordinate cognitive control network subserving diverse executive functions. Cognitive, Affective & Behavioral Neuroscience, 12(2), 241–268. [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. O’Donnell LA, Deldin PJ, Grogan-Kaylor A, McInnis MG, Weintraub J, Ryan KA, & Himle JA (2017). Depression and executive functioning deficits predict poor occupational functioning in a large longitudinal sample with bipolar disorder. Journal of Affective Disorders, 215, 135–142. [DOI] [PubMed] [Google Scholar]
  47. Pizzagalli DA (2011). Frontocingulate dysfunction in depression: Toward biomarkers of treatment response. Neuropsychopharmacology, 36(1), 183–206. [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Polderman TJ, Posthuma D, De Sonneville LM, Stins JF, Verhulst FC, & Boomsma DI (2007). Genetic analyses of the stability of executive functioning during childhood. Biological Psychology, 76(1–2), 11–20. [DOI] [PubMed] [Google Scholar]
  49. Reynolds CR, & Kamphaus RW (2015). Behavior assessment system for children: Third edition manual. Circle Pines, MN: American Guidance Services. [Google Scholar]
  50. Rinsky JR, & Hinshaw SP (2011). Linkages between childhood executive functioning and adolescent social functioning and psychopathology in girls with ADHD. Child Neuropsychology : a Journal on Normal and Abnormal Development in Childhood and Adolescence, 17(4), 368–390. [DOI] [PMC free article] [PubMed] [Google Scholar]
  51. Schork AJ, Brown TT, Hagler DJ, Thompson WK, Chen CH, Dale AM, … Genetics S (2018). Polygenic risk for psychiatric disorders correlates with executive function in typical development. Genes, Brain, and Behavior, e12480. doi: 10.1111/gbb.12480 [DOI] [PMC free article] [PubMed] [Google Scholar]
  52. Snyder HR, Miyake A, & Hankin BL (2015). Advancing understanding of executive function impairments and psychopathology: Bridging the gap between clinical and cognitive approaches. Frontiers in Psychology, 6, 328. [DOI] [PMC free article] [PubMed] [Google Scholar]
  53. Soni A (2014). The five most costly children’s conditions, 2011: Estimates for U.S. civilian non-institutionalized children, ages 0–17. Statistical Briefing. Agency for Healthcare Research and Quality. [PubMed] [Google Scholar]
  54. Tulsky DS, Carlozzi NE, Chevalier N, Espy KA, Beaumont JL, & Mungas D (2013). NIH toolbox cognition battery (cb): Measuring working memory. Monographs of the Society for Research in Child Development, 78, 70–87. [DOI] [PMC free article] [PubMed] [Google Scholar]
  55. van Stralen JPM (2018). A controlled trial of extended-release guanfacine and psychostimulants on executive function and ADHD. Journal of Attention Disorders, 1087054717751197. doi: 10.1177/1087054717751197 [DOI] [PubMed] [Google Scholar]
  56. Vuontela V, Carlson S, Troberg AM, Fontell T, Simola P, Saarinen S, & Aronen ET (2013). Working memory, attention, inhibition, and their relation to adaptive functioning and behavioral/emotional symptoms in school-aged children. Child Psychiatry and Human Development, 44(1), 105–122. [DOI] [PubMed] [Google Scholar]
  57. Wagner S, Helmreich I, Wollschlager D, Meyer K, Kaaden S, Reiff J, … Lieb K (2018). Early improvement of executive test performance during antidepressant treatment predicts treatment outcome in patients with Major Depressive Disorder. PloS One, 13(4), e0194574. [DOI] [PMC free article] [PubMed] [Google Scholar]
  58. Wagner S, Müller C, Helmreich I, Huss M, & Tadić A (2015). A meta-analysis of cognitive functions in children and adolescents with major depressive disorder. European Child & Adolescent Psychiatry, 24(1), 5–19. [DOI] [PubMed] [Google Scholar]
  59. Ware AL, Crocker N, O’Brien JW, Deweese BN, Roesch SC, Coles CD, & CIFASD. (2012). Executive function predicts adaptive behavior in children with histories of heavy prenatal alcohol exposure and attention-deficit/hyperactivity disorder. Alcoholism, Clinical and Experimental Research, 36(8), 1431–1441. [DOI] [PMC free article] [PubMed] [Google Scholar]
  60. Wechsler D (2008). Wechsler adult intelligence scale- fourth edition. San Antonio, TX: Pearson. [Google Scholar]
  61. Wechsler D (2011). Wechsler Abbreviated Scale of Intelligence (2nd ed.). San Antonio, TX: The Psychological Corporation. [Google Scholar]
  62. Wechsler D (2014). Wechsler intelligence scale for children- fifth edition. Bloomington, MN: Pearson. [Google Scholar]
  63. Weintraub S, Dikmen SS, Heaton RK, Tulsky DS, Zelazo PD, Bauer PJ, … Gershon RC (2013). Cognition assessment using the NIH toolbox. Neurology, 80, S54–S64. [DOI] [PMC free article] [PubMed] [Google Scholar]
  64. Willcutt E, Sonuga-Barke E, Nigg J, & Sergeant J (2008). Recent developments in neuropsychological models of childhood psychiatric disorders. Biological Child Psychiatry, 24, 195–226. [Google Scholar]
  65. Withall A, Harris LM, & Cumming SR (2009). The relationship between cognitive function and clinical and functional outcomes in major depressive disorder. Psychological Medicine, 39 (3), 393–402. [DOI] [PubMed] [Google Scholar]
  66. Zelazo PD, Anderson JE, Richler J, Wallner-Allen K, Beaumont JL, & Weintraub S (2013). NIH toolbox cognition battery (CB): Measuring executive function and attention. Monographs of the Society for Research in Child Development, 78, 16–33. [DOI] [PubMed] [Google Scholar]

RESOURCES