Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2020 Sep 29.
Published in final edited form as: Behav Ther. 2019 Jul 5;51(2):268–282. doi: 10.1016/j.beth.2019.06.009

The Validity of a Frustration Paradigm to Assess the Effect of Frustration on Cognitive Control in School-Age Children

Karen E Seymour 1, Keri S Rosch 2, Alyssa Tiedemann 3, Stewart H Mostofsky 4
PMCID: PMC7523008  NIHMSID: NIHMS1627489  PMID: 32138937

Abstract

Irritability refers to a proneness for anger, and is a symptom of internalizing and externalizing psychopathology. Since irritability is associated with significant cross-sectional and longitudinal impairments, research on the behavioral and neural correlates of pediatric irritability in populations at risk for significant irritability is of paramount importance. Irritability can be assessed in the laboratory using behavioral paradigms that elicit frustration. Few behavioral frustration paradigms have been designed to measure the effects of frustration on cognitive control. Therefore, the goal of the present study was to validate a behavioral frustration paradigm for use in school-age children which addressed some of the limitations of prior research. Participants included children, ages 8–12 years, who were either typically developing (TD; n = 38) or diagnosed with attention-deficit/hyperactivity disorder (ADHD; n = 67), which provided a sample of children with a range of baseline irritability. All participants completed the Frustration Go/ No-Go (GNG) task, and self-reported irritability was assessed using the Affective Reactivity Index. Results showed that across participants, self-reported frustration, commission error rate, and tau all increased with the addition of frustration, with similar effect sizes in ADHD and TD groups. Further, self-reported irritability, moreso than ADHD symptoms, predicted changes in self-reported frustration during the task. Together, these results support the construct validity of the Frustration GNG task as a means of assessing the effect of frustration on cognitive control. Clinical applications and future directions are discussed.

Keywords: attention-deficit/hyperactivity disorder, frustration, irritability, emotion regulation, cognitive control

Irritability refers to a proneness towards anger (Brotman, Kircanski, & Leibenluft, 2017; Leibenluft & Stoddard, 2013; Vidal-Ribas, Brotman, Valdivieso, Leibenluft, & Stringaris, 2016), and is a transdiagnostic symptom that cuts across both externalizing and internalizing psychopathology (Brotman, Kircanski, Stringaris, Pine, & Leibenluft, 2017; Vidal-Ribas et al., 2016). Irritability represents a dimensional spectrum ranging from low and normatively occurring to severe and atypical (Perlman, Luna, Hein, & Huppert, 2014; Wakschlag et al., 2015), and when in the severe and atypical range, irritabilty predicts significant cross-sectional and longitudinal impairments (Brotman et al., 2006; Copeland, Shanahan, Egger, Angold, & Costello, 2014; Dougherty et al., 2013; Pickles et al., 2010; Stringaris, Cohen, Pine, & Leibenluft, 2009). For example, irritability in childhood is associated with increased risk for the development of depressive and anxiety disorders, lower financial and educational attainment, increased risky and illegal behaviors, and negative health outcomes in young adulthood (Copeland et al., 2014; Stringaris et al., 2009; Vidal-Ribas et al., 2016). Even more concerning is that impairing irritability in adolescence is predictive of suicidality independent of affective disorders (Pickles et al., 2010). These deleterious outcomes stress the importance of research on irritability in pediatric populations, particularly those at risk for increased levels of clinically significant irritability.

Irritability aligns with the Research Domain Criteria (RDoC) construct of frustrative nonreward (or frustration), which is defined as an affective reaction to blocked goal-attainment (Amsel, 1958). Specifically, irritability is considered an expression of frustrative nonreward. In general, frustrative nonreward occurs when an individual’s ability to obtain an expected reward is blocked (Amsel, 1958), and like irritability, there are interindividual differences in frustrative responses following blocked goal attainment (Pawliczek et al., 2013). Frustration can be elicited in the laboratory using behavioral tasks in which expected rewards are omitted or goal-attainment is blocked via increased task difficulty. While frustration is a normative affective response, empirical studies have linked clinically significant pediatric irritability to aberrant emotional, behavioral and neural responses to frustration. For example, compared to typically developing (TD) children, children with severe irritability often display increased levels of arousal (Rich et al., 2007; Rich et al., 2011) and self-reported frustration (Deveney et al., 2013; Rich et al., 2005) during frustration tasks. Further, compared to TD children, children with severe irritability show decreased ability to successfully shift attention during frustration tasks (Deveney et al., 2013). Lastly, functional neuroimaging studies have shown that children with severe irritability have abberant activation during frustration tasks in regions critical to emotion regulation, including the amygdala, striatum, anterior cingulate cortex (ACC) and lateral prefrontal cortex (PFC; (Deveney et al., 2013; Grabell et al., 2018; Perlman et al., 2014; Perlman et al., 2015; Rich et al., 2010; Tseng et al., 2018). Taken together, these studies suggest that atypical responses to frustrative nonreward may be a core mechanism underlying clinically significant pediatric irritability (Brotman, Kircanski, & Leibenluft, 2017; Leibenluft, 2017; Vidal-Ribas et al., 2016). Therefore, it is of critical importance to elucidate the biobehavioral correlates underlying frustration in clinical populations demonstrating moderate to high levels of irritability.

Within the pediatric irritability literature, a number of tasks have been developed to probe frustrative nonreward. The most commonly used is a modified version of an attentional Posner task (i.e., Affective Posner, Affective Posner 2), which assesses the effect of frustration on spatial attention and orienting to a cue (Deveney et al., 2013; Rich et al., 2005; Rich et al., 2007; Rich et al., 2010; Rich et al., 2011; Tseng et al., 2017; Tseng et al., 2018). In contrast, fewer studies have assessed the effect of frustration on cognitive control. Cognitive control refers to “a set of superordinate functions that encode and maintain representations of the current task...marshalling to that task subordinate functions including working memory...attention...action selection and inhibition” (Botvinick & Braver, 2015, p. 85). Cognitive control allows an individual to flexibly adapt and adjust behavior in the context of ever-changing task demands and goals (Carter & Krus, 2012). Two important subprocesses of cognitive control include response inhibition, or the ability to inhibit a prepotent response to a stimulus (Barkley, 1997; Nigg, 2001), and attentional control, the ability to sustain and focus attention on goal-relevant stimuli while ignoring competing stimuli (Nigg, 2017). Behaviorally, cognitive control can be measured using go/no-go (GNG) paradigms in which participants are instructed to make responses to “Go” stimuli and withhold responses to “No-Go” stimuli. On GNG tasks, response inhibition can be measured via commission errors (i.e., incorrectly responding on no-go trials), and attentional control can be measured using tau (i.e., a measure of infrequent, abnormally slow responses thought to be reflective of attentional lapses).

Cognitive control is critical for the regulation of emotion (Gross, 2015; Posner & Rothbart, 2000). Specifically, in the context of emotional stimuli, poor cognitive control likely contributes to dysregulated emotion, including excessive emotional responses, mood lability, and atypical allocation of attention to emotional stimuli (Posner & Rothbart, 2000). That is, poor top-down cognitive control in the context of emotional stimuli may result in a disproportionate and maladaptive behavioral response to the stimuli (e.g., temper tantrums) or excessive attention directed toward the stimuli (e.g., rumination or negative attentional biases), both of which would likely increase negative affect.

To date, most studies of the effect of frustration on cognitive control in pediatric populations have been conducted with young children (Grabell et al., 2018; Grabell, Olson, Tardif, Thompson, & Gehring, 2017; Perlman et al., 2014; Perlman et al., 2015) or in children who do not demonstrate clinically significant levels of irritability and/or difficulties with cognitive control (Grabell et al., 2018; Lamm & Lewis, 2010; Lewis, Lamm, Segalowitz, Stieben, & Zelazo, 2006; Perlman et al., 2014). Using a developmentally sensitive modification of a GNG paradigm called the Frustration Emotion Task for Children (FETCH), Perlman et al. (2014) showed that in a community sample of children ages 3–5 (without clinically significant irritability), frustration on the task was associated with increased activation of the lateral PFC, and greater engagement of the lateral PFC was associated with higher parent-rated frustration tolerance. In a follow-up study using the FETCH task in children ages 6–9 years old, results showed that during the frustration condition, TD children had greater activation in the ACC relative to the children with clinically significant irritability, but children in the clinical group had greater activation in the posterior cingulate cortex. Additionally, Grabell et al. (2018) found a nonlinear association of frustration to PFC activation in 3- to 7-year-olds demonstrating a range of irritability from normative to impairing. Specifically, for children on the low end of the irritability spectrum, parent-reported irritability was associated with increased lateral PFC activation during frustration whereas for children at the high end of the irritability spectrum, parent-rated irritability was associated with decreased lateral PFC activation during frustration. Collectively, this research suggests the importance of prefrontal regions in the regulation of frustration in both TD young children and young children with impairing irritability.

In another line of research, Lewis and colleagues utilized an emotional induction GNG task in TD school-age children (Lamm & Lewis, 2010; Lewis et al., 2006) and school-age children with increased levels of externalizing problems (Lewis et al., 2008; Stieben et al., 2007). In TD children, results showed that response time increased and accuracy decreased during the block with the negative emotional induction. Further, this pattern of responding was more prevalent in younger relative to older children (Lewis et al., 2006). Similar behavioral results were found in children with externalizing problems. Moreover, relative to controls, children with externalizing problems had greater difficulties slowing their responses during the negative induction block (Stieben et al., 2007). Taken together, this literature suggests that the interaction between emotional and cognitive control may be particularly important in understanding the pathophysiology of impairing irritability.

Children with attention-deficit/hyperactivity disorder (ADHD) display both increased levels of irritability (Eyre et al., 2017; Geller et al., 2002; Karalunas et al., 2014; Karalunas, Gustafsson, Fair, Musser, & Nigg, 2018; Kircanski et al., 2017) and deficits in cognitive control (Kofler et al., 2013; Lijffijt, Kenemans, Verbaten, & van Engeland, 2005; Oosterlaan & Sergeant, 1998; Tamm et al., 2012; Willcutt, Doyle, Nigg, Faraone, and Pennington, 2005a, b), underscoring the value of examining the effect of frustration on cognitive control in this population. Regarding irritability, in a large sample of children ages 6–18 years old with ADHD, Eyre et al. (2017) found that 91% of the sample had at least one parent-reported irritability symptom (derived from the ODD section of the Child and Adolescent Psychiatric Assessment), and 53% had all three irritability symptoms. In another study, which derived subtypes of ADHD from a temperament questionnaire, Karalunas et al. (2014, 2018) found three distinct subtypes of ADHD, including: “mild ADHD” (i.e., more inattentive and hyperactive than controls, but did not differ on any temperament domains from controls), “surgent ADHD” (i.e., greater impulsivity, high intensity pleasure-seeking and increased activity level), and “irritable ADHD” (i.e., increased negative emotionality including higher levels of anger, discomfort, fear, and sadness, and lower levels of soothability than other two ADHD subtypes and controls). Longitudinal analyses showed that relative to mild or surgent ADHD children, irritable ADHD children were more likely to develop a new comorbid disorder at 1-year follow-up (Karalunas et al., 2018). Further, resting state connectivity analyses showed that irritable ADHD children had weaker functional connectivity from the amygdala to the anterior insula relative to all other groups, and weaker connectivity from the amygdala to the posterior cingulate relative to controls and the ADHD surgent type (Karalunas et al., 2014). Jointly, these results suggest important differences in neural function and outcomes in children with ADHD with elevated levels of irritability. In order to inform prevention and intervention efforts targeting irritability in children with ADHD, it is critical to understand the interplay between frustration and cognitive control deficits given the frequency with which children with ADHD encounter frustrating situations in academic, home, and social situations.

Deficits in cognitive control are a primary impairment in children with ADHD. Specifically, compared to TD children, children with ADHD show increased commission errors, reflective of deficits in response inhibition, on GNG tasks (Willcutt, Doyle, Nigg, Faraone, and Pennington, 2005a, b; Wodka et al., 2007). However, more recently, research on cognitive control in ADHD has suggested that children with ADHD may not merely make more commission errors, but may also be more variable in their responding (i.e., have greater response time [RT] variability; (Kofler et al., 2013; Tamm et al., 2012). In fact, the most consistent finding in the ADHD cognitive control literature is that individuals with ADHD show elevated response time variability, including tau, relative to controls. Tau is an ex-Gaussian measure that represents the mean of the exponential component of the RT distribution (Kofler et al., 2013). Specifically, in ex-Gaussian methods, the RT distribution is separated into the normal (Gaussian) and exponential (i.e., tail of the positive skew) components (Heathcote, Brown, & Mewhort, 2002; Hockley, 1984; Kofler et al., 2013). Tau measures the extent to which an individual displays infrequent, abnormally slow responses that may reflect attentional lapses (Hervey et al., 2006; Leth-Steensen, Elbaz, & Douglas, 2000), reflecting a specific component of RT variability not captured by the more widely used standard deviation of RT. To date, no studies of the effect of frustration on cognitive control have examined the impact of frustration on tau.

Therefore, the overall objective of the present study was to develop and test a behavioral frustration paradigm, the Frustration GNG, that assesses the effect of frustration on cognitiv control in a sample of children with and without cognitive control deficits who demonstrate a range of irritability (i.e., TD children and children with ADHD). This task was designed to address some limitations of prior frustration paradigms that assess the effect of frustration on cognitive control. First, it was designed to allow for the calculation of tau to better characterize the relationship between frustration and attentional control impairments relevant to ADHD. Additionally, it is developmentally appropriate for school-age children. While Lewis et al. (2006, 2008) also utilized a GNG design for school-age children, their task had a working memory component that is less well-suited for children with ADHD given working memory deficits in this population (Willcutt et al., 2005a, b). Moreover, their task was intended to induce negative emotions more broadly,such as anxiety and/or anger rather than frustration specifically. Last, participant ratings of negative affect on Lewis’ emotional induction GNG were relatively low (i.e., “mad” = 1.7/10 and “upset” = 2.5/10) compared to ratings on other frustration tasks (e.g., FETCH, Affective Posner), and therefore, it may not adequately induce frustration.

As such, the first aim of the study was to validate the Frustration GNG task and demonstrate construct validity. It was hypothesized that across the sample, the task would elicit increased levels of self-reported frustration during the frustrative relative to the nonfrustrative blocks. Further, it was hypothesized that across the sample, cognitive control measures, including commission error rate and tau, would increase during the frustrative relative to the nonfrustrative blocks. A second aim was to examine group differences in self-reported frustration on the task, as well as, dependent measures of cognitive control. It was hypothesized that compared to TD children, children with ADHD would report greater change in frustration on the task and would show increased cognitive control deficits with increased frustration. We further sought to examine whether change in task dependent variables was more attributable to ADHD symptoms or self-reported irritability. We hypothesized that across all variables self-reported irritability would be more predicitive of change than ADHD symptoms.

Methods

PARTICIPANTS

Participants (n = 105) were drawn from a sample of 114 children, ages 8–12 years old, who completed the Frustration GNG paradigm as part of a larger study of frustration tolerance in children with ADHD. Participants were sampled across a range of irritability from normative (i.e., TD controls, n = 38) to impairing (i.e., ADHD, n = 67). Participants were primarily recruited through local schools, with additional recruitment resources including: community-wide advertisement, volunteer organizations, medical institutions, and word of mouth. This study was approved by the Johns Hopkins Medical Institutional Review Board, and all research was conducted in accordance with the Declaration of Helsinki. Written consent was obtained from a parent/guardian and assent was obtained from the participating child.

All participants were initially screened for eligibility during a brief phone interview with the parent, and children with a history of intellectual disability, seizures, traumatic brain injury and/or other neurological illnesses were excluded due to the aims of the larger study. Eligible participants completed the remaining study procedures to assess eligibility including: (1) diagnostic interview with parents conducted by a MA-level clinician to assess for the presence of psychopathology (Kiddie Schedule for Affective Disorders and Schizophrenia for School-Age Children- Present and Lifetime version, Kiddie-SADS-PL 2016, Kaufman et al., 2016); (2) assessment of cognitive abilities and academic achievement, Wechsler Intelligence Scale for Children-5th edition (WISC-V, Wechsler, 2014) and Wechsler Individual Achievement Test-2nd edition, (WIAT-II, Wechsler, 2002) or Wechsler Individual Achievement Test-3rd edition (WIAT-III, Wechsler, 2009); (3) completion of parent-report measures of ADHD symptoms including Conners 3 (Conners, 2008) and ADHD Rating Scale-IV, home version (ADHD-RS, DuPaul, Power, Anastopoulos, & Reid, 1998); and (4) completion of the computerized frustration task the Frustration GNG. Teacher ratings on the Conners 3 and ADHD-RS were also collected to assess cross-situational impairment.

For all participants, full scale IQ scores below 80 on the WISC-V were exclusionary. One participant with ADHD was excluded for having a FSIQ less than 80. Children were also excluded for a score below 85 on the WIAT-II Word Reading to screen for possible reading disorders.

For inclusion in the ADHD group, children had to meet full DSM-5 criteria for ADHD based on the following criteria: (1) an ADHD diagnosis according to the Kiddie-SADS including presence of symptoms and cross-situational impairment criteria, and (2) T-score of 65 or higher on the Conners 3 DSM Inattention or Hyperactivity/ Impulsivity subscales or a score of 2 or 3 (i.e., symptoms rated as occurring “often” or “very often”) on at least 6/9 items on the Inattentive or Hyperactivity/Impulsivity scales of the ADHD-RS. If teacher reports were not available, cross-situational impairment was determined based on parent-report during the Kiddie-SADS (e.g., presence of academic, home, or social difficulties, etc.). All information was reviewed by a either a licensed clinical psychologist (K.E.S. or K.S.R.) or a child neurologist (S.H.M.) for diagnostic confirmation. Given that the aim of this study was to validate a frustration paradigm across participants with a range of irritability, children with ADHD and comorbid diagnoses of oppositional-defiant disorder, anxiety disorders (except PTSD), and depressive disorders were included in the sample. Children taking stimulant medication were asked to withhold medication on the day prior to and day of testing. Children taking other psychotropic medications were allowed to continue their medication as prescribed. Eleven children in the ADHD group were taking SSRIs and 1 was taking another psychtropic.

For inclusion in the TD group, participants could not meet diagnostic criteria for any psychiatric disorder based on the Kiddie-SADS, have a history of neurological disorder, have been diagnosed with a learning disability, or be taking psychotropic medication. Additionally, TD participants’ scores on parent and teacher (when available) ADHD rating scales had to be below the clinical cut-points for inclusion.

To balance groups on IQ, the TD males with the 5 highest FSIQs were dropped from analyses. Participant demographic and clinical characteristics are reported in Table 1.

Table 1.

Participant Demographic and Clinical Characteristics

TD (n = 38) ADHD (n = 67) Group Comparison
Age 10.83 (1.4) 10.91 (1.5) F(1,103) = 0.07, p = 0.800
% Male 68% (26) 75% (50) X2(1) = 0.467, p = 0.494
% Caucasian 63% (24) 60% (40) X2(1) = .122, p = 0.727
SES 55.14 (9.4) 52.43 (13.4) F(1,102) = 1.19, p = 0.278
WISC-5 GAI 109.31 (10.4) 107.00 (12.0) F(1,103) = 1.02, p = 0.316
ADHD Subtype (CT:IA:HI) N/A 31:35:1 N/A
% ODD Diagnosis N/A 16% (11) N/A
% Anxiety Diagnosis N/A 42% (28) N/A
% Depression Diagnosis N/A 4.5% (3) N/A
% Stimulant Medication N/A 61% N/A
Conners 3 IA (t-score) 46.2 (7.4) 75.3 (12.2) F(1,101) = 173.22, p<.001
Conners 3 HI (t-score) 46.2 (7.5) 76.5 (15.1) F(1,101) = 129.95, p<.001
Conners 3 ODD (t-score) 47.8 (7.4) 63.8 (14.0) F(1,101) = 41.03, p<.001
DuPaul IA (raw score) 3.6 (3.1) 18.6 (5.7) F(1,101) = 217.37, p<.001
DuPaul HI (raw score) 2.4 (3.1) 14.1 (7.1) F(1,101) = 88.67, p<.001
DuPaul Total (raw score) 6.0 (5.8) 32.8 (10.7) F(1,101) = 192.56, p<.001
ARI-S Total 1.6 (1.7) 3.7 (2.3) F(1,99) = 24.52, p<.001
ARI-P Total 0.81 (1.5) 3.3 (2.9) F(1,99) = 23.99, p<.001
Open in a new tab

Note. Results presented as Mean (SD) or %(n). ADHD = Attention-Deficit/Hyperactivity Disorder; IA = Inattentive; HI = Hyperactive/Impulsive; SES = socioeconomic status from Hollingshead total score; TD = Typically Developing Controls; WISC-V GAI = Wechsler Intelligence Scale for Children-5th Edition General Ability Index

FRUSTRATION GNG PARADIGM

Participants completed the Frustration GNG task, a task designed to assess the effect of frustration on cognitive control. In this task, participants are instructed to respond to “Go” stimuli (i.e., use their finger on the computer track pad to move the “Go” stimuli [penguin with a green hat and scarf] across the screen to a target [igloo]), and withhold response to “No-Go” stimuli (i.e., penguin with red hat and scarf, Figure 1). The GNG stimuli ratio is 4:1 (120 Go/30 No-Go per block), so as to create a strong prepotent response tendency that is then inhibited during No-Go trials. During the task, participants earn 5ȼ 10ȼ for correct No-Go trials, and lose 10ȼ for incorrect Go or No-Go trials.

FIGURE 1.

FIGURE 1

Open in a new tab

Screenshot of the Frustration GNG paradigm

The task consists of 12 practice trials and three runs of 150, 150 and 75 trials, respectively (total trials = 375). For each trial, stimuli are presented for up to 2,000 ms, feedback duration is 1,000 ms, and ISI is 1,000 ms. Total task time is approximately 25 minutes. The first block of the task (i.e., Block 1) is a nonfrustration block in which few frustration stimuli are presented (i.e., 5%, n = 6). The second block (i.e., Block 2) is the frustration block in which 50% (n = 60) of Go trials are “rigged” so the penguin will not move into the igloo despite participant effort (i.e., “frustration trials”). Following frustration trials, participants received feedback that their response was “Too Slow!” and lost 10 cents. The final block (i.e., Block 3) returns to the nonfrustration condition where only 5% of trials are frustration trials. During Blocks 1 and 3, partcipants received feedback based on their actual performance for correct Go and correct No-Go trials (You earn 5 or 10 cents, respectively) or incorrect Go and incorrect NoGo trials (You lose 10 cents). Money earned on the task was added to participant compensation. Primary task performance dependent variables include: (1) commission error rate, defined as the proportion of No-Go trials on which the participant responded, and (2) tau, with higher value indicating greater variability of responding and greater attentional lapses. After examination of the Frustration GNG variables, 2 children with ADHD and 1 TD control were dropped from analyses for being significant outliers, which resulted in 105 participants being included in the final analyses.

QUESTIONNAIRE MEASURES

To assess subjective frustration during the task, participants completed the Positive and Negative Affect Scale for Children (PANAS-C) at four points during the task as a manipulation check (baseline, after Blocks 1–3). Self-reported frustration on the task was assessed using the “Frustrated” item from the PANAS-C (range 0–10).

Affective Reactivity Index (ARI)

Children’s trait irritability was assessed using self-report on the ARI (Stringaris et al., 2012). The ARI is a six-item scale assessing the frequency, duration, and threshold of child irritability over the past 6 months, and is thought to reflect trait-like irritability (Stringaris et al., 2012). Each item is rated on a 3-point Likert scale ranging from 0 (Not True)to 2(Certainly True); therefore, total scores on the ARI range from 0–12 with higher scores reflecting greater levels of irritability. Prior research using the ARI in both clinical and community samples has demonstrated good internal consistency, strong validity, and a single-factor structure (DeSousa et al., 2013; Mulraney, Melvin, & Tonge, 2014; Stringaris et al., 2012; Tseng et al., 2017). In the current study, reliability of ARI self-report was adequate (α = .75). A histogram reflecting the distribution of ARI-S scores is presented in Fig. S1.

DATA ANALYTIC PLAN

All analyses were performed using IBM SPSS Statistics version 25. There was no missing data for the Frustration GNG variables (n = 105), but there was missing data for the ARI self-report (n = 4). Missing data were handled with pairwise deletion in all analyses.

Across the whole sample, Pearson-product moment correlations were conducted to examine the relationship between key demographic variables (age, gender, race, SES, WISC-V General Abilities Index [GAI] score), and (1) self-reported frustration during the task as well as change in self-reported frustration from baseline to the end of Block 2, and (2) commission error rate and tau as well as change in commission error rate and tau from Block 1 to Block 2. Significantly related demographic variables were then included in subsequent analyses (i.e., gender for self-reported frustration; age for commission error rate and age and race for tau).

To assess the construct validity of the task (i.e., did the manipulation result in increased self-reported frustration ratings) and whether self-reported frustration differentially increased for the diagnostic groups, a 2 Group (ADHD vs. TD) × 4 Block (baseline and after Blocks 1–3) repeated measures analyses of covariance (ANCOVA) controlling for gender was conducted. To assess the effect of frustration on commission error rate and tau, two separate 2 Group (ADHD vs. TD) x 3 Block (Blocks 1–3) repeated measures ANCOVAs were conducted with age as a covariate. For analyses of tau, race was also included as a covariate. Bonferroni corrections were applied for post-hoc comparisons, and violations of sphericity were addressed using Hunh-Feldt corrections (Field, 2009). Effect sizes were assessed using Cohen’s d with small, medium, and large effect sizes as Cohen’s d 0.3–0.5, 0.5–0.8, and ≥ 0.8, respectively (Cohen, 1988).

To examine whether change in self-reported frustration and cognitive control from the nonfrustrative to frustrative condition was more attributable to ADHD symptoms or self-reported irritability, a series of hierarchical linear regressions predicting change in task dependent variables across the whole sample were conducted in which covariates were entered in Step 1, total ADHD symptoms were entered in Step 2, and ARI self-report was entered in Step 3.

Results

CORRELATIONS TO EXAMINE THE RELATIONSHIP BETWEEN DEMOGRAPHIC VARIABLES AND SELF-REPORTED FRUSTRATION ON THE TASK AND TASK BEHAVIORAL DVs

Results showed that age, race, SES and WISC-V GAI were not significantly related to self-reported frustration on the task at baseline, after Blocks 1–3, or to change in self-reported frustration from baseline to the end of frustrative Block 2. However, gender was significantly related to Block 2 frustration (r = .195, p = .046). A follow-up GLM ANCOVA showed that during the frustration block (Block 2), boys reported greater levels of frustration than girls, F(1,103) = 4.09, p = .046, d = .44. Correlations are shown in Table S1.

Results examining the relationship between demographic variables and commission error rate and tau (as well as change in commission error rate and tau) showed that age was significantly negatively related to both nonfrustrative Block 1 and frustrative Block 2 commission error rate and tau (r’s = −.298 to −.398, p’s≥.01). Age was also negatively associated with Block 3 commission error rate (r = −.247, p = .01). There was also a significant relationship between race and Block 3 tau (r = −.200, p = .041) such that non-Caucasian participants had higher tau in Block 3 than Caucasian participants. Correlations are shown in Table S2.

CONSTRUCT VALIDITY OF THE TASK, EFFECT OF FRUSTRATION ON COGNITIVE CONTROL,AND EXAMINATION OF GROUP DIFFERENCES

Results of the 2 Group × 4 Block GLM repeated measures ANCOVA examining changes in self-reported frustration across the task and controlling for gender showed a main effect of Block, F (2.92,298.74) = 8.58, p<.001. Follow-up pairwise comparisons showed a quadratic effect across participants such that self-reported frustration increased linearly from baseline to nonfrustrative Block 1 (p≤.001, d = .48), from nonfrustrative Block 1 to frustrative Block 2 (p≤.001, d = .73), and then decreased from frustrative Block 2 to nonfrustrative Block 3 (p≤.001, d = −.61) (Figure 2). Participant ratings of frustration nearly doubled fromnonfrustrative Block 1 to the frustrative Block 2 (1.65±.26 vs. 3.89±.33). The main effects of group, F(1,102) = 2.87, p =.093, and gender, F(1,102) = 3.17, p = .078, were not significant. Therefore, while at baseline and in all blocks, the ADHD group reported higher levels of self-reported frustration compared to the TD group, the groups were not statistically different. Neither the Block × Diagnosis (p = .391) nor the Block × Gender (p = .164) interactions were significant.

FIGURE 2.

FIGURE 2

Open in a new tab

Changes in self-reported frustration during the task across participantsNote. Values for self-reported frustration include gender as a covariate. Comparisons reported are between block of interest and prior block. F= frustration, NF= non-frustration.**p≤.001

Results of the 2 Group × 3Block GLM repeated measures ANCOVA examining changes in commission error rate controlling for age showed a main effect of Block, F(1.95,199.23) = 3.17, p =.045 (Figure 3a). Follow-up pairwise comparisons showed a quadratic effect across participants such that commission error rate increased from nonfrustrative Block 1 to frustrative Block 2 (p = .012, d =.29),and remained elevated even after frustration was removed, from frustrative Block 2 to nonfrustrative Block 3 (p = 1.00, d = .02). There was a main effect of group, F(1,102) = 13.31 p<.001. Follow-up pairwise comparisons showed children with ADHD showed a higher commission error rate than TD children in all three blocks (p’s respectively: p<.001, p = .019, p = .012). There was also a main effect of age,F(1,102) = 23.41 p<.001, such that across the sample, commission error rate decreased with increasing age. The Block × Age (p = .059) and Block × Group (p= .201) interactions were not significant.

FIGURE 3.

FIGURE 3

Open in a new tab

a-b. Figure 3a shows the effect of frustration on commission error rate during the task across participants. Figure 3b shows the effect of frustration on tau during the task across participants.Note. Values for commission error rate include age as a covariate. Values for tau include age and race as covariates. Comparisons reported are between block of interest and prior block. F= frustration, NF= non-frustration.*p<.01, **p≤.001

Results of the 2 Group × 3 Block GLM repeated measures ANCOVA examining change in tau controlling for age and race showed a main effect of Block, F(1.93,194.71)= 5.22, p =.007(Figure 3b). Follow-up pairwise comparisons showed a quadratic effect across participants such that tau increased from nonfrustrative Block 1 to frustrative Block 2 (p<.001, d = .95), and then decreased frustration was removed from frustrative Block 2 to nonfrustrative Block 3 (p<.001, d =.55) (Figure 3b). There was also a main effect of group, F(1,101) = 36.15, p<.001. Follow-up pairwise comparisons showed that children with ADHD showed higher tau in all three blocks compared to TD children (all p’s<.001). There was also a main effect of age, F (1,101) = 14.34, P<.001, such that across the sample, tau decreased with increasing age. The whenBlock × Age (p = .065), Block × Race (p = .079) and Block × Group (p = .766) interactions were not significant.

REGRESSIONS PREDICTING CHANGE IN SELF-REPORTED FRUSTRATION AND COGNITIVE CONTROL FROM ADHD SYMPTOMS AND IRRITABILITY

The regression model predicting change in self-reported frustration from gender, total parent-reported ADHD symtpoms and self-reported irritability was significant, F(3,96) = 3.92, p = .01, R2 = .11. Specifically, self-reported irritability predicted change in frustration on the task (β = .32, p = .006) over and above gender (β = .17, p = .08) and total ADHD symptoms (β = −.10, p = .38).

The overall regression model predicting change in commission error rate from age, total ADHD symptoms and self-reported irritability was not significant, F(3,96) = 1.91, p = .13, R2 = .06. However, examination of each step in the regression model showed a trend for self-reported irritability to be related to change in commission error rate (β =.22, p =.07) over and above age (β = −.15,p = .13)andtotal ADHD symptoms (β = −.15,p = .20).

The regression model predicting change in tau from age, race, total ADHD symtpoms and self-reported irritability was not significant, F(3,96) = .99, p = .42, R2 = .04. None of the variables included in the model were significant predictors of change in tau: age (β = −.11, p = .30), race (β = −.10, p = .33), total ADHD symptoms (β = .14, p = .26), and self-reported irritability (β = −.04, p = .76).

Discussion

The present study examined the construct validity of a frustration paradigm, the Frustration GNG, designed specifically to assess the effect of frustration on cognitive control in school-age children. Results showed that the Frustration GNG demonstrated good construct validity as it effectively elicited negative affect across participants. Moreover, as hypothesized, the addition of a frustrative component resulted in performance decrements, specifically increased commission error rate and tau, across participants. Group comparisons in self-reported frustration showed that, at baseline and across all blocks, the ADHD group reported higher levels of frustration than the TD group, but group differences were not statistically significant. Group comparisons of commission error rate and tau showed that children with ADHD showed higher commission error rates and tau than TD participants across all blocks of the task; however, the frustrative component of the task, did not differentially impact cognitive control for children with ADHD compared to TD controls contrary to our hypotheses. Regression analyses showed that self-reported irritability, over and above ADHD symptoms, significantly predicted change in self-reported frustration during the task. Overall, these findings suggest that the Frustration GNG task is a valid measure of the effect of frustration on response control.

Regarding construct validity, results showed that there was a significant change in self-reported frustration across the task such that frustration increased during the frustrative relative to the nonfrustrative blocks. In fact, participant frustration nearly doubled (d = .73) from nonfrustrative Block 1 to frustrative Block 2, suggesting the successful manipulation of negative affect during the task. Of importance, the sample consisted of participants demonstrating a range of irritability from low/normative to high/impairing, and, regardless of ADHD symptom level, across all participants, the task elicited frustration suggesting its future utility with a wide range of participants. While change in frustration scores across the sample could be viewed as moderate (i.e., change from baseline level of .58 to post-frustration block rating of 3.89 out of a total of 10), such change is in line with other validated measures of frustration for use in children (Tseng et al., 2017). Moreover, while the overall goal of the task was to increase negative affect, particularly frustration, there are ethical limits as to the level of frustration elicited.

Our results also showed that across participants frustration negatively impacted cognitive control as measured by increases in commission error rate and tau with the addition of frustration. For commission error rate, there was a significant increase from nonfrustrative Block 1 to frustrative Block 2, and commission error rate stayed elevated despite the removal of frustration in Block 3. In contrast, for tau, an increase in frustration resulted in increased tau from Block 1 to Block 2, but removal of frustration resulted in a decrease in tau from Block 2 to Block 3, albeit not to pre-frustration levels. These results are consistent with studies that have examined the impact of negative affect on response inhibition in TD children (Lewis et al., 2006). Results showed that in the wake of negative affect, TD children demonstrated increases in reaction time and error rates. However, the tendency towards impulse responding in the context of negative affect diminished with age, suggesting the effect of negative affect on cognitive control may be more salient for younger rather than older children (Lewis et al., 2006). Given that for commission error rates the removal of frustration did not result in a subsequent decrease in task errors, our results suggest that the impact of frustration on cognitive control may continue to effect performance even after the withdrawal of the emotional component. This differential pattern of results for commission error rate and tau might suggest persistent effects of frustration on response inhibition, but not attentional control.

When comparing children with ADHD to TD children, our results did not show a Group × Block interaction for self-reported frustration, suggesting that groups reported comparable levels of frustration on the task. The literature examining group differences in self-reported negative affect specific to the frustrative component of a frustration task is mixed, with some studies reporting group differences between clinical and control groups (Rich et al., 2007; Rich et al., 2011) and others reporting no significant differences (Deveney et al., 2013; Perlman et al., 2015; Rich et al., 2005; Seymour, Macatee, & Chronis-Tuscano, 2016). In fact, in studies that have shown group differences in clinical vs. control groups, the effects occur across all blocks of the task and are rarely specific to the frustration condition itself. We speculate that while it is certainly possible that children with ADHD and TD children both experienced the same level of negative affect during the task, it is more likely that children with ADHD are less insightful of their own emotional states and therefore do not rate their negative emotions as accurately. Indeed, research utilizing ecological momentary assessment to study emotion regulation ability in children with ADHD has shown that children with ADHD have difficulty completing self-ratings of emotion when experiencing negative emotions (Rosen, Epstein, & Van Orden, 2013). In addition, child-report mood ratings differed significantly from parent ratings, and tended to be more positive despite high parent ratings of negative emotion (Rosen et al., 2013). Observations from our research staff in the testing room during the administration of the Frustration GNG suggested that, in general, children with ADHD displayed more negative emotions (i.e., eye rolling, groaning, banging on the desk, banging on the computer) and made more negative comments during the frustration block compared to TD children. However, when it came time to rate their emotions, children with ADHD did not always rate their level of frustration as commensurate with their behaviors and verbalizations. It will be important for future studies of frustration to rely on objective measures of negative emotion, including observational coding of child behaviors and measures of physiology (e.g., skin conductance and pulse), which may provide more accurate measures of children’s emotional reactivity than self-report ratings.

While our results revealed that compared to TD children, children with ADHD had higher levels of commission error rate and tau, the Group × Block interactions were not significant, suggesting that the introduction of frustration similarly impacted cognitive control in children with ADHD and TD controls. While the group differences are consistent, with prior research showing increased error rates and intrasubject variability in children with ADHD compared to TD children (Kofler et al., 2013; Tamm et al., 2012; Vaurio, Simmonds, & Mos-tofsky, 2009), they were contrary to our hypothesis that frustration would differentially impact cognitive control in children with ADHD compared to TD children. However, our results are consistent with the behavioral results of many studies using frustration tasks. Namely, the majority of studies using behavioral frustration tasks do not report group differences between clinical populations and TD controls on measures of behavioral performance during increased frustration (i.e., RT, accuracy, etc.) (Grabell et al., 2017; Perlman et al., 2015; Rich et al., 2010; Rich et al., 2011). One notable exception is that some studies using the Affective Posner have shown that patient groups with high irritability have slower RT than controls during the frustration condition (Deveney et al., 2013; Rich et al., 2005). Yet, even in lieu of behavioral group differences, the literature supports neural differences in response to frustration between TD and patient groups. For example, while Perlman et al. (2015) did not report differences in RT during a frustration GNG task between clinically irritable and TD children, fMRI analyses revealed group differences in ACC and posterior cingulate cortex activation during frustration. Therefore, an important next step will be to examine neural responses during the Frustration GNG in children with ADHD and TD children. In addition, it may be that there are “subgroups” of children with ADHD (i.e., girls vs. boys, inattentive vs. combined presentation, etc.) for whom frustration has a greater impact on cognitive control. Given that this preliminary study was underpowered to test for interactions with two between-subjects factors (e.g., Group × Sex or ADHD presentation), further investigations with larger samples are needed to parse these potential effects.

It is also possible that we did not observe behavioral group differences during frustration because children with ADHD are more accustomed to functioning in the wake of frustration, and as a result, have developed compensatory skills that assist them. In fact, one prior study comparing children with ADHD to TD children on an emotional GNG task showed no behavioral group differences on the task. Yet, examination of ERP data revealed that compared to TD children, children with ADHD showed greater activation of inhibition-related neural mechanisms (i.e., greater no-go P3 amplitudes) during emotional compared to neutral stimuli, suggesting that children with ADHD required greater inhibitory effort to reach a similar performance level as control children (Lopez-Martin, Albert, Fernandez-Jaen, & Carretie, 2015). Likewise, Grabell et al. (2018) found that frustration-related lateral PFC activation increased as parent-reported irritability increased within the normative range, but decreased with increasing irritability within the severe range, suggesting that increased activation of regulatory regions may be adaptive in individuals with high, but normative, irritability. Applied to our work, it may suggest that during frustration, children with ADHD have to expend greater cognitive control resources to manage their negative emotions, resulting in fewer resources to deploy towards other situations requiring cognitive control. Future work will need to probe the functional brain activity that occurs during frustration in children with ADHD compared to TD children, and determine if compensatory activity is invoked by children with ADHD during frustrating situations. This may be particularly true in situations where children with ADHD are motivated by the prospect of reward to adaptively inhibit their response, as in the current paradigm.

Last, our results showed that self-reported irritability predicted change in frustration on the task over and above parent-reported ADHD symptoms, but did not show that irritability was associated with changes in commission error rate and tau. Our results showing that self-reported irritability predicted change in frustration may be the product of shared method/informant variance; however, it is also possible that they reflect similiarities between “state” and “trait” irritability. Specifically, behavioral paradigm ratings are more reflective of a particular emotional “state” in the moment, whereas ratings that reflect on irritability over an extended period of time (i.e., rating scales) may be more indicative of “trait”-like features. Additional longitudinal work is needed to examine whether state versus trait irritability is more predictive of clinical outcomes in children with ADHD. Our results did not show an association between self-reported irritability and changes in commission error rate and tau. Such results are consistent with prior research examining the relationship between irritability and behavioral measures on frustration tasks. Specifically, in a large sample of 8- to 12-year-olds with well-distributed levels of irritability, including severe irritability, Tseng et al. (2018) found that self-report on the ARI was not associated with behavioral measures such as RT or accuracy during the frustration condition. However, increased ARI scores were associated with increased neural activation in several posterior and frontal regions in young children and young adolescents (ages 8–14), but not associated with neural activation in older adolescents (ages 15–18). Therefore, it may be that irritability is more related to neural rather than behavioral indices of frustration.

As with all studies, our results should be considered in light of some limitations. As this is the first study to utilize the Frustration GNG task, replication of these results with a larger, independent sample will be needed. Moreover, while the sample size was modest, it was not large enough for us to examine the effect of additional factors such as sex or comorbidity on the results. A larger sample size, particularly of children with ADHD, would allow us to also examine a greater range of irritability, which may reveal interesting within-group differences based on level of irritability. Another limitation was that the TD group was half the size of the ADHD group, so future studies should try to balance participant groups to facilitate comparison. Further, this was a cross-sectional study and does not assess how current frustration tolerance is associated with subsequent impairments or outcomes in children with A Longitudinal research assessing frustration over multiple developmental periods and examining a multitude of outcomes (e.g., development of comorbidity, academic/occupational success, prediction of treatment response etc) is required. A potential concern is that the length of the task was 25 minutes, which may have caused some children to give up due to task length. However, based on our prior experience with administering cognitive tasks to children ages 8–12 years old, we feel that our task was well within a reasonable limit to probe cognitive control in this age group. Another limitation is that children who participated in the study were not medication naïve. While no participants were taking stimulant medications the day before or during testing, it is unknown if there are underlying effects of medication.

In conclusion, the Frustration GNG task demonstrated good construct validity as a measure of the effect of frustration on cognitive control. Across participants, self-reported frustration, commission error rate, and tau increased during the frustration block relative to the non-frustration blocks. TD and ADHD participants did not differ in the magnitude of the effect of frustration on response inhibition and variability in responding during the task. Last, irritability, over and above ADHD symptoms, was shown to predict change in frustration on the task, suggesting some relationship between state and trait irritability.

Supplementary Material

Table S1
Table S2
Figure S1

Acknowledgments

This research was funded by National Institutes of Mental Health (NIMH) grant K23MH107734 (PI: Seymour) and a NARSAD Young Investigators Grant from the Brain & Behavior Foundation (PI: Seymour).

Footnotes

Conflict of Interest Statement

The authors declare that there are no conflicts of interest.

Supplementary data to this article can be found online at https://doi.org/10.1016/j.beth.2019.06.009.

Contributor Information

Karen E. Seymour, Johns Hopkins University School of Medicine, Johns Hopkins University Bloomberg School of Public Health, Kennedy Krieger Institute

Keri S. Rosch, Johns Hopkins University School of Medicine, Kennedy Krieger Institute

Alyssa Tiedemann, Kennedy Krieger Institute.

Stewart H. Mostofsky, Johns Hopkins University School of Medicine, Kennedy Krieger Institute

References

  1. Amsel A (1958). The role of frustrative nonreward in noncontinuous reward situations. Psychological Bulletin, 55(2), 102–119. [DOI] [PubMed] [Google Scholar]
  2. Barkley RA (1997). Behavioral inhibition, sustained attention, and executive functions: constructing a unifying theory of ADHD. Psychological Bulletin, 121(1), 65–94. [DOI] [PubMed] [Google Scholar]
  3. Botvinick M, & Braver T (2015). Motivation and cognitive control: from behavior to neural mechanism. Annual Review of Psychology, 66, 83–113. 10.1146/annurev-psych-010814-015044 [DOI] [PubMed] [Google Scholar]
  4. Brotman MA, Kircanski K, & Leibenluft E (2017). Irritability in Children and Adolescents. Annual Review of Clinical Psychology, 13, 317–341. 10.1146/annurev-clinpsy-032816-044941 [DOI] [PubMed] [Google Scholar]
  5. Brotman MA, Kircanski K, Stringaris A, Pine D, & Leibenluft E (2017). Irritability in Youths: A Translational Model. American Journal of Psychiatry, 174(6), 520–532. 10.1176/appi.ajp.2016.16070839 [DOI] [PubMed] [Google Scholar]
  6. Brotman MA, Schmajuk M, Rich BA, Dickstein, Guyer AE,Costello EJ... Leibenluft E (2006). Prevalence, clinical correlates, and longitudinal course of severe mood dysregulation in children. Biological Psychiatry, 60(9), 991–997. 10.1016/j.biopsych.2006.08.042 [DOI] [PubMed] [Google Scholar]
  7. Carter CS, & Krus MK (2012). Dynamic cognitive control and frontal-cingulate interactions In Posner MI (Ed.), Cognitive neuroscience of attention (pp. 88–98). New York, NY: Guilford Press. [Google Scholar]
  8. Cohen D (1988). Statistical power analyses for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Earlbaum Associates. [Google Scholar]
  9. Conners CK (2008). Conners (3rd ed.). North Tonawanda, NY: Multi-Health Systems Inc.. [Google Scholar]
  10. Copeland WE, Shanahan L, Egger H, Angold A, & Costello EJ (2014). Adult diagnostic and functional outcomes of DSM-5 disruptive mood dysregulation disorder. American Journal of Psychiatry, 171(6), 668–674. 10.1176/appi.ajp.2014.13091213 [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. DeSousa DA, Stringaris A, Leibenluft E, Koller SH, Manfro GG, & Salum GA (2013). Cross-cultural adaptation and preliminary psychometric properties of the Affective Reactivity Index in Brazilian Youth: implications for DSM-5 measured irritability. Trends in Psychiatry and Psychotherapy, 35(3), 171–180. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Deveney CM, Connolly ME, Haring CT, Bones BL, Reynolds RC, Kim P … Leibenluft E (2013). Neural mechanisms of frustration in chronically irritable children. American Journal of Psychiatry, 170(10), 1186–1194. 10.1176/appi.ajp.2013.12070917 [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Dougherty LR, Smith VC, Bufferd SJ, Stringaris A, Leibenluft E, Carlson GA, & Klein DN (2013). Preschool irritability: longitudinal associations with psychiatric disorders at age 6 and parental psychopathology. Journal of the American Academy of Child and Adolescent Psychiatry, 52(12), 1304–1313. 10.1016/j.jaac.2013.09.007 [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. DuPaul GJ, Power TJ, Anastopoulos AD, & Reid R (1998). ADHD Rating Scale-IV. New York: Guilford Press. [Google Scholar]
  15. Eyre O, Langley K, Stringaris A, Leibenluft E, Collishaw S, & Thapar A (2017). Irritability in ADHD: Associations with depression liability. Journal of Affective Disorders, 215, 281–287. 10.1016/j.jad.2017.03.050 [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Field A (2009). Discovering Statistics Using SPSS. London: Sage Publications. [Google Scholar]
  17. Geller B, Zimerman B, Williams M, Delbello MP, Bolhofner K, Craney JL … Nickelsburg MJ (2002). DSM-IV mania symptoms in a prepubertal and early adolescent bipolar disorder phenotype compared to attention-deficit hyperactive and normal controls. Journal of Child and Adolescent Psychopharmacology, 12(1), 11–25. 10.1089/10445460252943533 [DOI] [PubMed] [Google Scholar]
  18. Grabell AS, Li Y, Barker JW, Wakschlag LS, Huppert TJ, & Perlman SB (2018). Evidence of Non-Linear Associations between Frustration-Related Prefrontal Cortex Activation and the Normal:Abnormal Spectrum of Irritability in Young Children. Journal of Abnormal Child Psychology, 46(1), 137–147. 10.1007/s10802-017-0286-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Grabell AS, Olson SL, Tardif T, Thompson MC, & Gehring WJ (2017). Comparing Self-Regulation-Associated Event Related Potentials in Preschool Children with and without High Levels of Disruptive Behavior. Journal of Abnormal Child Psychology, 45(6), 1119–1132. 10.1007/s10802-016-0228-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Gross JJ (2015). Emotion regulation: current status and future prospects. Psychological Inquiry, 26, 1–26. 10.1080/1047840x.2014.940781 [DOI] [Google Scholar]
  21. Heathcote A, Brown S, & Mewhort DJ (2002). Quantile maximum likelihood estimation of response time distributions. Psychonomic Bulletin & Review, 9(2), 394–401. [DOI] [PubMed] [Google Scholar]
  22. Hervey AS, Epstein JN, Curry JF, Tonev S, Eugene Arnold L, Keith Conners C … Hechtman L. (2006). Reaction time distribution analysis of neuropsychological performance in an ADHD sample. Child Neuropsychology, 12(2), 125–140. 10.1080/09297040500499081 [DOI] [PubMed] [Google Scholar]
  23. Hockley WE (1984). Analysis of response time distributions in the study of cognitive processes. Journal of Experimental Psychology: Learning, Memory, and Cognition, 10(4), 598–615. 10.1037/0278-7393.10.4.598 [DOI] [Google Scholar]
  24. Karalunas SL, Fair D, Musser ED, Aykes K, Iyer SP, & Nigg JT (2014). Subtyping attention-deficit/ hyperactivity disorder using temperament dimensions: toward biologically based nosologic criteria. JAMA Psychiatry, 71(9), 1015–1024. 10.1001/jamapsychiatry.2014.763 [DOI] [PMC free article] [PubMed] [Google Scholar] [Retracted]
  25. Karalunas SL, Gustafsson HC, Fair D, Musser ED, & Nigg JT (2018). Do we need an irritable subtype of ADHD? Replication and extension of a promising temperament profile approach to ADHD subtyping.Psychological Assessment. 10.1037/pas000066410.1037/pas0000664.supp (Supplemental) [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Kaufman J, Birmaher B, Axelson D, Perepletchikova F, Brent D, & Ryan N (2016). Kiddie Schedule for Affective Disorders and Schizophrenia for School-Aged Children – Present and Lifetime Version (Kiddie-SADS-PL 2016). Pittsburgh Pennsylvania: Western Psychiatric Institute and Clinic and Yale University.
  27. Kircanski K, Zhang S, Stringaris A, Wiggins JL, Towbin KE, Pine DS … Brotman MA (2017). Empirically derived patterns of psychiatric symptoms in youth: A latent profile analysis. Journal of Affective Disorders, 216, 109–116. 10.1016/j.jad.2016.09.016 [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Kofler MJ, Rapport MD, Sarver DE, Raiker JS, Orban SA, Friedman LM, & Kolomeyer EG (2013). Reaction time variability in ADHD: a meta-analytic review of 319 studies. Clinical Psychology Review, 33(6), 795–811. 10.1016/j.cpr.2013.06.001 [DOI] [PubMed] [Google Scholar]
  29. Lamm C, & Lewis MD (2010). Developmental change in the neurophysiological correlates of self-regulation in highand low-emotion conditions. Developmental Neuropsychology, 35(2), 156–176. 10.1080/87565640903526512 [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Leibenluft E (2017). Pediatric Irritability: A Systems Neuroscience Approach. Trends in Cognitive Sciences, 21(4), 277–289. 10.1016/j.tics.2017.02.002 [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Leibenluft E, & Stoddard J (2013). The developmental psychopathology of irritability. Development and Psychopathology, 25(4 Pt 2), 1473–1487. 10.1017/S0954579413000722 [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Leth-Steensen C, Elbaz ZK, & Douglas VI (2000). Mean response times, variability, and skew in the responding of ADHD children: a response time distributional approach. Acta Psychologica, 104(2), 167–190. [DOI] [PubMed] [Google Scholar]
  33. Lewis MD, Granic I, Lamm C, Zelazo PD, Stieben J, Todd RM … Pepler D. (2008). Changes in the neural bases of emotion regulation associated with clinical improvement in children with behavior problems. Development and Psychopathology, 20(3), 913–939. 10.1017/s0954579408000448 [DOI] [PubMed] [Google Scholar]
  34. Lewis MD, Lamm C, Segalowitz SJ, Stieben J, & Zelazo PD (2006). Neurophysiological correlates of emotion regulation in children and adolescents. Journal of Cognitive Neuroscience, 18(3), 430–443. 10.1162/089892906775990633 [DOI] [PubMed] [Google Scholar]
  35. Lijffijt M, Kenemans JL, Verbaten MN, & van Engeland H (2005). A meta-analytic review of stopping performance in attention-deficit/hyperactivity disorder: deficient inhibitory motor control? Journal of Abnormal Psychology, 114(2), 216–222. 10.1037/0021-843x.114.2.216 [DOI] [PubMed] [Google Scholar]
  36. Lopez-Martin S, Albert J, Fernandez-Jaen A, & Carretie L (2015). Emotional response inhibition in children with attention-deficit/hyperactivity disorder: neural and behavioural data. Psychological Medicine, 45(10), 2057–2071. 10.1017/S0033291714003195 [DOI] [PubMed] [Google Scholar]
  37. Mulraney MA, Melvin GA, & Tonge BJ (2014). Psychometric properties of the affective reactivity index in Australian adults and adolescents. Psychological Assessment, 26(1), 148–155. 10.1037/a0034891 [DOI] [PubMed] [Google Scholar]
  38. Nigg JT (2001).Is ADHD a disinhibitory disorder? Psychological Bulletin, 127(5), 571–598. [DOI] [PubMed] [Google Scholar]
  39. Nigg JT (2017). Annual Research Review: On the relations among self-regulation, self-control, executive functioning, effortful control, cognitive control, impulsivity, risk-taking, and inhibition for developmental psychopathology. Journal of Child Psychology and Psychiatry and Allied Disciplines, 58(4), 361–383. 10.1111/jcpp.12675 [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Oosterlaan J, & Sergeant JA (1998). Response inhibition and response re-engagement in attention-deficit/ hyperactivity disorder, disruptive, anxious and normal children. Behavioural Brain Research, 94(1), 33–43. [DOI] [PubMed] [Google Scholar]
  41. Pawliczek CM, Derntl B, Kellermann T, Gur RC, Schneider F, & Habel U (2013). Anger under control: neural correlates of frustration as a function of trait aggression. PloS One, 8(10)e78503 10.1371/journal.pone.0078503 [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Perlman SB, Jones BM, Wakschlag LS, Axelson D, Birmaher B, & Phillips ML (2015). Neural substrates of child irritability in typically developing and psychiatric populations. Developmental Cognitive Neuroscience, 14, 71–80. 10.1016/j.dcn.2015.07.003 [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Perlman SB, Luna B, Hein TC, & Huppert TJ (2014). fNIRS evidence of prefrontal regulation of frustration in early childhood. Neuroimage, 85(Pt 1), 326–334. 10.1016/j.neuroimage.2013.04.057 [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Pickles A, Aglan A, Collishaw S, Messer J, Rutter M, & Maughan B (2010). Predictors of suicidality across the life span: the Isle of Wight study. Psychological Medicine, 40(9), 1453–1466. 10.1017/S0033291709991905 [DOI] [PubMed] [Google Scholar]
  45. Posner MI, & Rothbart MK (2000). Developing mechanisms of self-regulation. Development and Psychopathology, 12, 427–441. [DOI] [PubMed] [Google Scholar]
  46. Rich BA, Carver FW, Holroyd T, Rosen HR, Mendoza JK, Cornwell BR … Leibenluft E. (2011). Different neural pathways to negative affect in youth with pediatric bipolar disorder and severe mood dysregulation. Journal of Psychiatric Research, 45(10), 1283–1294. 10.1016/j.jpsychires.2011.04.006 [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Rich BA, Holroyd T, Carver FW, Onelio LM, Mendoza JK, Cornwell BR … Leibenluft E (2010). A preliminary study of the neural mechanisms of frustration in pediatric bipolar disorder using magnetoencephalography. Depression and Anxiety, 27(3), 276–286. 10.1002/da.20649 [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Rich BA, Schmajuk M, Perez-Edgar KE, Fox NA, Pine DS, & Leibenluft E (2007). Different psychophysiological and behavioral responses elicited by frustration in pediatric bipolar disorder and severe mood dysregulation. American Journal of Psychiatry, 164(2), 309–317. 10.1176/appi.ajp.164.2.309 [DOI] [PubMed] [Google Scholar]
  49. Rich BA, Schmajuk M, Perez-Edgar KE, Pine DS, Fox NA, & Leibenluft E (2005). The impact of reward, punishment, and frustration on attention in pediatric bipolar disorder. Biological Psychiatry, 58(7), 532–539. https://doi. org/10.1016/j.biopsych.2005.01.006 [DOI] [PubMed] [Google Scholar]
  50. Rosen PJ, Epstein JN, & Van Orden G (2013). I know it when I quantify it: ecological momentary assessment and recurrence quantification analysis of emotion dysregulation in children with ADHD. Attention Deficit and Hyperactivity Disorders, 5(3), 283–294. 10.1007/s12402-013-0101-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
  51. Seymour KE, Macatee R, & Chronis-Tuscano A (2016). Frustration Tolerance in Youth With ADHD. Journal of Attention Disorders. 10.1177/1087054716653216 [DOI] [PMC free article] [PubMed] [Google Scholar]
  52. Stieben J, Lewis MD, Granic I, Zelazo PD, Segalowitz S, &Pepler D(2007).Neurophysiologicalmechanismsof emotion regulation for subtypes of externalizing children. Development and Psychopathology, 19(2), 455–480. 10.1017/S0954579407070228 [DOI] [PubMed] [Google Scholar]
  53. Stringaris A, Cohen P, Pine DS, & Leibenluft E (2009). Adult outcomes of youth irritability: a 20-year prospective community-based study. American Journal of Psychiatry, 166(9), 1048–1054. 10.1176/appi.ajp.2009.08121849 [DOI] [PMC free article] [PubMed] [Google Scholar]
  54. Stringaris A, Goodman R, Ferdinando S, Razdan V, Muhrer E, Leibenluft E, & Brotman MA (2012). The Affective Reactivity Index: a concise irritability scale for clinical and research settings. Journal of Child Psychology and Psychiatry and Allied Disciplines, 53(11), 1109–1117. 10.1111/j.1469-7610.2012.02561.x [DOI] [PMC free article] [PubMed] [Google Scholar]
  55. Tamm L, Narad ME, Antonini TN, O'Brien KM, Hawk LW, & Epstein JN (2012). Reaction time variability in ADHD: a review. Neurotherapeutics, 9(3), 500–508. 10.1007/s13311-012-0138-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
  56. Tseng WL, Deveney CM, Stoddard J, Kircanski K, Frackman AE, Yi JY … Leibenluft E (2018). Brain Mechanisms of Attention Orienting Following Frustration: Associations With Irritability and Age in Youths.American Journal of Psychiatry. 10.1176/appi.ajp.2018.18040491appiajp201818040491 [DOI] [PMC free article] [PubMed] [Google Scholar]
  57. Tseng WL, Moroney E, Machlin L, Roberson-Nay R, Hettema JM, Carney D … Brotman MA (2017). Test-retest reliability and validity of a frustration paradigm and irritability measures. Journal of Affective Disorders, 212, 38–45. 10.1016/j.jad.2017.01.024 [DOI] [PMC free article] [PubMed] [Google Scholar]
  58. Vaurio RG, Simmonds DJ, & Mostofsky SH (2009). Increased intra-individual reaction time variability in attention-deficit/hyperactivity disorder across response inhibition tasks with different cognitive demands. Neuropsychologia, 47(12), 2389–2396. 10.1016/j.neuropsychologia.2009.01.022 [DOI] [PMC free article] [PubMed] [Google Scholar]
  59. Vidal-Ribas P, Brotman MA, Valdivieso I, Leibenluft E, & Stringaris A (2016). The Status of Irritability in Psychiatry: A Conceptual and Quantitative Review. Journal of the American Academy of Child and Adolescent Psychiatry, 55(7), 556–570. 10.1016/j.jaac.2016.04.014 [DOI] [PMC free article] [PubMed] [Google Scholar]
  60. Wakschlag LS, Estabrook R, Petitclerc A, Henry D, Burns JL, Perlman SB … Briggs-Gowan ML (2015). Clinical Implications of a Dimensional Approach: The Normal:Abnormal Spectrum of Early Irritability. Journal of the American Academy of Child and Adolescent Psychiatry, 54(8), 626–634. 10.1016/j.jaac.2015.05.016 [DOI] [PMC free article] [PubMed] [Google Scholar]
  61. Wechsler DL (2002). Wechsler Individual Achievement Test-Second Edition (WIAT-II). San Antonio, TX: The Psychological Corporation. [Google Scholar]
  62. Wechsler D (2009). Wechsler Individual Achievement Test-3rd edition (WIAT- III). San Antonio, TX: Psychological Corporation. [Google Scholar]
  63. Wechsler D (2014). Wechsler Intelligence Scale for Children-Fifth Edition (WISC-V).
  64. Willcutt E, Doyle AE, Nigg JT, Faraone SV, & Pennington BF (2005). Validity of the executive function theory of attention-deficit/hyperactivity disorder: a meta-analytic review. Biological Psychiatry, 57(11), 1336–1346. 10.1016/j.biopsych.2005.02.006 [DOI] [PubMed] [Google Scholar]
  65. Willcutt EG, Doyle AE, Nigg JT, Faraone SV, & Pennington BF (2005). Validity of the executive function theory of attention-deficit/hyperactivity disorder: a meta-analytic review. Biological Psychiatry, 57(11), 1336–1346. 10.1016/j.biopsych.2005.02.006 [DOI] [PubMed] [Google Scholar]
  66. Wodka EL, Mahone EM, Blankner JG, Larson JC, Fotedar S, Denckla MB, & Mostofsky SH (2007). Evidence that response inhibition is a primary deficit in ADHD. Journal of Clinical and Experimental Neuropsychology, 29(4), 345–356. 10.1080/13803390600678046 [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Table S1
NIHMS1627489-supplement-Table_S1.docx (13.2KB, docx)
Table S2
NIHMS1627489-supplement-Table_S2.docx (14.9KB, docx)
Figure S1
NIHMS1627489-supplement-Figure_S1.docx (21.9KB, docx)

RESOURCES