Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2018 Aug 1.
Published in final edited form as: Dev Psychopathol. 2017 Aug 14;30(2):571–579. doi: 10.1017/S0954579417001080

Altered Intrinsic Functional Connectivity of the Cingulate Cortex in Children with Severe Temper Outbursts

Amy Krain Roy 1,2, Randi Bennett 1, Jonathan Posner 3, Leslie Hulvershorn 4, F Xavier Castellanos 2, Rachel G Klein 2
PMCID: PMC5812860  NIHMSID: NIHMS902580  PMID: 28803557

Abstract

Severe temper outbursts in children are associated with impaired school and family functioning and may contribute to negative outcomes. These outbursts can be conceptualized as excessive frustration responses reflecting reduced emotion regulation capacity. The anterior cingulate cortex (ACC) has been implicated in negative affect as well as emotional control, and exhibits disrupted function in children with elevated irritability and outbursts. This study examined the intrinsic functional connectivity (iFC) of a region of the ACC, the anterior midcingulate cortex (aMCC), in 5–9 year old children with severe temper outbursts (STO; n=20), comparing them to children with ADHD without outbursts (ADHD; n=18). Additional analyses compared results to a sample of healthy children (HC; n=18) and examined specific associations with behavioral and emotional dysregulation. Compared to the ADHD group, STO children exhibited reduced iFC between the aMCC and surrounding regions of the ACC, and increased iFC between the aMCC and precuneus. These differences were also seen between the STO and HC groups; ADHD and HC groups did not differ. Specificity analyses found associations between aMCC-ACC connectivity and hyperactivity, and between aMCC-precuneus iFC and emotion dysregulation. Disruption in aMCC networks may underlie the behavioral and emotional dysregulation characteristic of children with severe temper outbursts.

Keywords: functional neuroimaging, emotion regulation, anterior cingulate cortex, temper outbursts, ADHD

Temper outbursts are common in young children (Wakschlag et al., 2012); however, when they are severe and chronic, they can interfere with children’s well-being and functioning in multiple domains, including family, peer, and teacher relationships, and may even precipitate psychiatric hospitalization (Brotman et al., 2006; Carlson, Potegal, Margulies, Gutkovich, & Basile, 2009; Leibenluft, Cohen, Gorrindo, Brook, & Pine, 2006). Severe temper outbursts are transdiagnostic, contributing to functional impairments in children with disruptive behavior disorders (Bhatia, Nigam, Bohra, & Malik, 1991; Roy et al., 2013) as well as neurodevelopmental (Dominick, Davis, Lainhart, Tager-Flusberg, & Folstein, 2007; Malone, Gratz, Delaney, & Hyman, 2005) and anxiety disorders (Johnco et al., 2015). In our own work, children clinically referred due to unmanageable temper outbursts most frequently presented with oppositional defiant disorder (ODD) and ADHD (Brotman et al., 2006; Roy et al., 2013). Several longitudinal studies of children with ODD show that symptoms of irritability, including temper outbursts, are associated with elevated risk for later depressive and anxiety disorders (Burke, Hipwell, & Loeber, 2010; Rowe, Costello, Angold, Copeland, & Maughan, 2010; Stringaris, Cohen, Pine, & Leibenluft, 2009; Stringaris & Goodman, 2009), highlighting their clinical importance. Little is known about the neurobiological basis of severe temper outbursts in young children, which could elucidate etiological trajectories of early emotion dysregulation and potentially inform preventive interventions and treatments.

Whether severe or not, temper outbursts typically occur in response to frustration. Thus, clinically significant outbursts likely reflect exaggerated frustration responses that involve crying, screaming, and yelling, as well as aggressive behaviors such as kicking, hitting, or running away (Giesbrecht, Miller, & Muller, 2010; Potegal & Davidson, 2003). Physiological studies support this dysregulated frustration response. For example, chronically irritable children with severe temper outbursts display significantly greater arousal in response to frustration than healthy comparisons (Rich et al., 2007). In addition to reflecting an elevated frustration response, evidence suggests that severe temper outbursts are associated, more generally, with poor emotional control (Gatzke-Kopp, Greenberg, & Bierman, 2015). For example, when faced with frustration, children with severe temper outbursts exhibit deficits in negative affect regulation (Roy et al., 2013) and respond relatively more slowly on an affective Posner task, implying deficits in attentional control of emotional responses (Deveney et al., 2013; Rich et al., 2007). Children rated high in temperamental anger by parents and teachers have difficulty shifting attention away from rewarding stimuli, further supporting impairments in attentional control, an aspect of emotion regulation that arises early in development (He et al., 2013).

From a neural perspective, frustration processing has been linked to multiple brain regions involved in negative affective responses, such as the anterior insula (Abler, Walter, & Erk, 2005; Rilling et al., 2008; Yu, Mobbs, Seymour, Rowe, & Calder, 2014), amygdala (Rilling et al., 2008; Yu et al., 2014), and anterior cingulate cortex (ACC; Spunt, Lieberman, Cohen, & Eisenberger, 2012; Yu et al., 2014). The ACC is of particular interest because it plays an integral role in neural networks involved in emotion regulation, beginning early in development (see Swingler, Perry, & Calkins, 2015 for a review). Specifically, dorsal regions of the ACC have been implicated in the use of attentional strategies, such as distraction, to regulate negative affective responses to visual stimuli (e.g., negatively valenced IAPS pictures) in studies of adults (Kanske, Heissler, Schonfelder, Bongers, & Wessa, 2011; McRae et al., 2010). Consistent with this, functional MRI studies implicate dorsal ACC regions in frustration processing. For example, adults with high trait aggression exhibit decreased activation of dorsal ACC in response to frustration (Pawliczek et al., 2013) while children (ages 6–9) with clinically significant irritability exhibit similar decreases in activation in a more rostral region of dorsal ACC (Perlman et al., 2015). Alternatively, decreased responding in a similar region of the ACC has been observed in a non-clinical sample of adults with high susceptibility to frustration (Siegrist et al., 2005). Finally, in a magnetoencephalography (MEG) study, a similar, but older, sample (ages 8–17) to that presented here (ages 5–9.9), showed increased activity of rostral ACC, along with higher ratings of agitation and sadness, in response to negative feedback, compared to healthy controls (Rich et al., 2011). Thus, while task-based studies provide clear evidence supporting the role of the ACC in processes presumed to underlie severe temper outbursts, frustration tolerance and emotion regulation, the direction of the findings, as well as their locations within the ACC, are inconsistent.

A recent coordinate-based meta-analysis of 192 task-based studies confirms the role of the ACC in both the subjective experience of negative affect, and in cognitive control (Shackman et al., 2011). A conjunction analysis revealed a single cingulate subregion, the anterior midcingulate cortex (aMCC), with activation across both domains. The anatomic connectivity of this region supports its role in cognitive control and negative affective states such as frustration. For example, the aMCC has reciprocal connections with subcortical dopaminergic pathways, suggesting that it may receive information about punishment signals such as those involved in the experience of frustration (Shackman et al., 2011). As a key node of the salience network, the aMCC is also tightly connected with the anterior insula, a region implicated in affect and cognitive control. Finally, Shackman and colleagues (2011) suggest that this region is linked to multiple motor centers that permit planning of motivated instrumental responses, such as temper outbursts.

We apply a broader network-based approach to understanding the role of the ACC in emotion dysregulation by using resting state functional MRI methods to evaluate the intrinsic functional connectivity (iFC) of an empirically-derived aMCC region in young children (ages 5–9.9 years) with clinically impairing outbursts. While this approach may not directly address inconsistencies in the task-based literature, evidence of disruptions in iFC of cingulate networks may inform comprehensive neural models that can be tested in future studies integrating activation paradigms and iFC approaches. In prior work, we found that the majority of children for whom severe temper outbursts are a primary concern do not exhibit chronic irritability (as defined as being in an irritable or angry mood more than 50% of the time) (Roy et al., 2013). Thus, the present study did not recruit children characterized by chronic irritability, as previous task- based studies of frustration responses have (Perlman et al., 2015; Rich et al., 2011), but rather focused on children with frequent, impairing outbursts. Further, our previous work suggests that a majority of young children with severe temper outbursts have ADHD (Roy et al., 2013), which is characterized by altered function (Dickstein, Bannon, Castellanos, & Milham, 2006), structure, (Makris et al., 2008; Seidman et al., 2006), and connectivity (Castellanos et al., 2008; Sun et al., 2012) of the ACC. Therefore, to address the aim of the study to identify neural features associated with severe temper outbursts, we compare children with severe outbursts to children with ADHD who do not have severe outbursts. To distinguish from general effects of psychopathology, we included healthy comparisons, group-matched for age and sex. The emotional dysregulation of children with severe temper outbursts may reflect a specific dysfunction, or may be due to greater overall symptom severity. Therefore, we conducted dimensional analyses to examine emotion dysregulation and ADHD symptom severity in relation to aMCC iFC.

Methods

Participants

Boys and girls, ages 5–9.9 years (7.2 ± 1.3; 21 female), were recruited across three groups: children with severe temper outbursts (85% with ADHD), children with ADHD without severe temper outbursts, and healthy comparisons. These children were recruited for two separate studies with identical entry criteria and imaging methods (n= 49 from Study 1, n = 24 from Study 2). Behavioral data from Study 1 (including 7 of the children in this study) have been published (Roy et al., 2013). Across both studies, children with severe temper outbursts (STO) had to exhibit verbal rages and/or physical aggression towards people or property at least twice a week, for at least 3 months. Outbursts had to (1) be characterized as out of proportion for the situation and the child’s developmental level, (2) of at least 15 minutes duration, during which the child was inconsolable, and (3) not occur exclusively during anxiety-provoking situations (e.g., in response to separation). Children in the ADHD group met DSM-IV criteria for ADHD without meeting criteria for STO. To ensure that the ADHD group was free of significant outbursts, children with ADHD who exhibited moderate outbursts (frequent and impairing, shorter in duration than STO) were excluded. Healthy comparisons (HC) were free of any major current DSM-IV diagnoses and of STO. Across all groups, participants were excluded if they displayed evidence of posttraumatic stress disorder, psychosis, or autism, IQ < 80 on the Kaufman Brief Intelligence Test (K-BIT; Kaufman & Kaufman, 2004), or if there were any contraindications to MRI scanning (e.g., claustrophobia, braces). Current or past (within the past three months) psychotropic use was exclusionary except for stimulants, with the provision that they were withdrawn for 72 hours prior to the initial assessment and scan. Parents/legal guardians provided written informed consent as approved by the New York University Langone School of Medicine and Fordham University Institutional Review Boards and children provided written (ages 7–9) or verbal (ages 5–6) assent. A total of 73 children met criteria for study entry and were invited to complete scan procedures. Five of these failed the mock scan training session, making them ineligible for the functional scan (3 STO, 1 ADHD, 1 HC). An additional 12 failed the scan due to excessive movement (7 STO, 2 ADHD, 3 HC). Notably, four of these were scanned before mock scanning procedures were in place and thus, did not have the benefit of a practice session. Overall, 56 children were included in the final sample.

Clinical Assessment

Tantrum severity was determined by clinical interview across the two studies. In Study 1, parents completed a Child Emotion Dysregulation Interview that asked about frequency and duration of the child’s outbursts. Parents were also asked to describe the child’s most recent tantrum, a typical tantrum, and the most severe outburst that the child had experienced and to indicate how often he/she has outbursts. These responses were used to determine tantrum severity. In Study 2, we obtained the same information using a modified version of the Temper Tantrum Grid (Giesbrecht et al., 2010) as a clinician-administered questionnaire. Parents answered the same questions about frequency and duration and were asked to describe a recent tantrum. A checklist of tantrum behaviors was provided which were rated on frequency of occurrence by the parent. For both studies, these data were then discussed during weekly diagnostic case conferences with the PI (AKR) and study group status was determined by consensus.

Diagnoses were determined by semi-structured clinical interviews of parents using the Schedule for Affective Disorders and Schizophrenia for School-Age Children- Present and Lifetime Version (K-SADS-PL; J. Kaufman et al., 1997) conducted by a child psychiatrist, child psychologists, or trained clinical psychology doctoral students. Only a brief clinical interview was conducted with the children, due to their young age. Final diagnoses were based on all available information, including teacher report when obtained, and determined through consensus among study clinicians and the PI. A diagnosis of ADHD-NOS was assigned when parents did not report ADHD-related impairments in more than one setting and teacher ratings were not available, precluding determination of cross-situational impairments.

Parents completed questionnaires about their child’s emotions and behaviors including the Behavior Assessment System for Children Parent Rating Scale (BASC-2-PRS; (Reynolds & Kamphaus, 2004)), a rating scale that assesses multiple symptom and functional domains. BASC-2 Parent Report Scales have high internal consistency with a near equivalency between clinical and general samples, and high test–retest reliability (Reynolds & Kamphaus, 2004). Hyperactivity and Attention Problems subscale raw scores were used to compare ADHD symptoms across groups. They were also used in regression analyses to disentangle associations between ADHD symptoms and emotion dysregulation and group differences in dACC iFC. The BASC-2-PRS for one participant was missing multiple items; as a result, the Attention Problems subscale score could not be computed. Parents also completed the Emotion Regulation Checklist (ERC; Shields & Cicchetti, 1997), a 24- item questionnaire that yields a highly reliable (Cronbach’s α = .91) Composite score, along with two subscales, Emotion Regulation and Lability/Negativity. The Composite score was used in the present study to compare overall emotion regulation skills across groups and to evaluate the specific relationship between emotion regulation and group differences in aMCC iFC. ERC Composite scores were not available for 6 of the participants (2 STO, 1 ADHD, 1 HC).

Scan Procedures

Approximately one week prior to imaging, most children participated in one or more “mock” scan session(s) in which they were taught how to lay still in the scanner while watching a movie and practicing the resting-state scan. Imaging was performed using the NYU Center for Brain Imaging Siemens Allegra 3.0 T Scanner (Siemens; Iselin, New Jersey). Children completed a 6-minute resting-state scan for which they were instructed to “stay as still as a statue” while a white cross was back-projected on a black background. The scan comprised 180 contiguous whole-brain functional volumes, acquired using a multi-echo echo planar imaging (EPI) sequence (repetition time = 2000 ms; echo time = 30 ms; flip angle = 90°; 33 slices; matrix = 64 × 64; voxel size = 3 × 3 × 4 mm). To minimize data loss, two EPI sequences were obtained when possible. The first EPI rest scan was used for 45 children and the second scan was used for 11 children who moved excessively during the first scan (3 STO, 5 ADHD, 3 HC). For spatial normalization and localization, a high- resolution T1-weighted anatomical image was also acquired using a magnetization prepared gradient echo sequence (MPRAGE, TR = 2500 ms; TE = 4.35 ms; TI = 900 ms; flip angle = 8; 176 slices, FOV = 256 mm).

Imaging Analyses

Functional Image Pre-processing

All brain data pre-preprocessing and group analyses were conducted using an alpha version (0.3.9) of the Configurable Pipeline for the Analysis of Connectomes (CPAC; http://fcp-indi.github.io/). CPAC is a configurable, open-source, Nipype-based (http://nipy.org/nipype/), automated processing pipeline for resting state functional MRI (R-fMRI) data. Pre-processing consisted of slice time correction, 3D motion correction (24 parameters; Friston et al., 1996), despiking (removal of extreme time series outliers), spatial smoothing (FWHM = 6mm), mean-based intensity normalization of all volumes by the same factor, and temporal band-pass filtering to isolate the low-frequency BOLD fluctuations of interest. Functional image registration was completed using Boundary Based Registration as implemented in FSL (Greve & Fischl, 2009). Structural images were registered normalized to common stereotaxic space (Montreal Neurological Institute: MNI) using Advanced Normalization Tools (Avants et al., 2011; http://www.picsl.upenn.edu/ANTS). Single participant nuisance regression included 24 Friston motion parameters (Friston et al., 1996) and five CompCor signals (Behzadi et al., 2007). Six participants (4 STO, 1 ADHD, 1 HC) had overall mean framewise displacement values greater than 0.25 and were considered for exclusion. However, visual inspection of their time series showed movements greater than 3mm only occurred within the last 100 seconds of their scans. Thus, rather than exclude them, we truncated their time series by removing the volume with motion > 3mm and all subsequent volumes.

aMCC Region of Interest Analysis

Given our interest in a cingulate region involved in both frustration responses and cognitive control, we selected a region-of-interest (ROI) based on the coordinate-based meta-analysis described earlier (Shackman et al., 2011). Using the coordinates for the center of intensity of the conjunction analysis (Talairach coordinates: 0, 12, 42, which were converted to MNI space), we created a 4mm radius sphere as our aMCC ROI. See Figure 1. The mean time series of this ROI was calculated by averaging the time series of all contained voxels. For each participant, aMCC connectivity strength with other regions of the brain was assessed using a whole brain analysis that involved correlations between the aMCC ROI time series and all other voxels in the brain. This resulted in individual participant-level maps of all voxels exhibiting significant iFC with the aMCC (Gaussian random field [GRF] correction: p < 0.05, Z > 2.3). Group-level analyses were conducted using a random-effects, ordinary least-squares model, including two group mean predictors (STO vs. ADHD) and three nuisance covariates (sex, age, and mean frame-wise displacement); all were GRF corrected at p < 0.05, Z > 2.3. We opted not to conduct a one-way ANCOVA to compare all groups at once, as this would emphasize findings related to overall psychopathology more than associations with STO.

Figure 1.

Figure 1

Open in a new tab

Additional Analyses

Normative Comparison

To clarify differences in the aMCC iFC between the STO and ADHD groups, we compared each group with the matched healthy comparisons, using independent samples t-tests in SPSS 19.0 (IBM SPSS Statistics, Version 19.0 IBM Corp: Armonk, NY).

Dimensional Analyses

The increased emotion dysregulation in the STO group might reflect greater overall severity of symptoms. To assess the unique contributions of emotion dysregulation and ADHD symptoms to aMCC iFC, we conducted multiple linear regression analyses for each of the regions showing significant group iFC differences, using SPSS 19.0. Each model included the ERC Composite Score as a measure of emotion regulation and BASC-2-PRS Hyperactivity and Attention Problems raw scores as measures of ADHD symptoms. Raw scores were used rather than T-scores to allow for greater equivalence with the ERC Composite, which does not have standardized norms. To account for putative effects of age and sex, these were included as covariates in the regression models. These regressions were conducted across STO and ADHD groups to increase power to detect significant effects.

Results

Demographic and Clinical Characteristics

Group characteristics of the 56 children who successfully completed functional MRI scans are presented in Table 1. The three groups did not differ in age (F2,55 = 0.28, p = 0.76), sex (X2[2] = 2.64, p = 0.27), IQ (F2,54 = 2.4, p = 0.10) or movement as measured by mean frame-wise displacement (F2,55 = 1.58, p = 0.22). Regarding diagnosis, the STO group exhibited significantly greater psychopathology with 85% meeting diagnostic criteria for 2 or more comorbid diagnoses. In contrast, only 16.6% (n = 3) of the children in the ADHD group met criteria for at least one other disorder (X2[2, N = 38] = 18.64, p < .001). Seven children (4 ADHD, 3 STO) were receiving pharmacological treatment at the time of the study; all stimulants were withdrawn at least three days prior to the MRI scan as per study entry criteria. One child had taken melatonin the night before the scan; otherwise, the children were medication free.

Table 1.

Demographic and clinical characteristics of the study sample.

STO (n = 20)
Mean (SD)		 ADHD (n = 18)
Mean (SD)		 HC (n = 18)
Mean (SD)		 Group
Differences (p < .05)
Age in years 6.99 (1.6) 7.25 (1.2) 7.28 (1.5) STO = ADHD = HC
Males (%) 11 (55%) 14 (77.8%) 10 (55.6%) STO = ADHD = HC
Framewise displacement in mm 0.10 (0.04) 0.08 (0.04) 0.07 (0.04) STO = ADHD = HC
Full Scale IQ 108.75 (18.0) 99.76 (12.5) 110.11 (14.0) STO = ADHD = HC
Diagnoses (%)
Attention Deficit Hyperactivity Disorder (ADHD) 17 (85%) 18 (100%) – STO = ADHD
 ADHD-Combined Type 7 (35%) 10 (55.6%) –
 ADHD-Inattentive Type 0 3 (16.7%) –
 ADHD-Hyperactive/Impulsive Type 5 (25%) 2 (11.1%) –
 ADHD-Not Otherwise Specified 5 (25%) 3 (16.7%) –
Oppositional Defiant Disorder 16 (80%) 3 (16.7%) – STO > ADHD
Any Anxiety Disorder 7 (35%) 2 (11.1%) STO = ADHD
Any Depressive Disorder 4 (20%) 1 (5.6%) STO = ADHD
Comorbidity STO > ADHD
Single diagnosis 3 (15%) 11 (61.1%) –
Two diagnoses 9 (45%) 2 (11.1%) –
Three or more diagnoses 8 (40%) 1 (5.6%) –
Emotion and Behavioral Regulation Measures
ERC Composite Score 2.62 (0.36) 3.03 (0.37) 3.55 (0.25) STO > ADHD > HC
BASC-2-PRS Hyperactivity Raw Score 20.60 (4.5) 14.56 (5.8) 8.61 (3.9) STO > ADHD > HC
BASC-2-PRS Attention Problems Raw Score 12.10 (4.0) 11.53 (2.6) 6.28 (3.2) STO = ADHD > HC
Open in a new tab

STO = severe temper outburst group; ADHD = attention-deficit/hyperactivity disorder group; HC = healthy comparisons group; BASC-2 PRS = Behavioral Assessment Scale for Children, Second Edition, Parent Report Scales; ERC = Emotion Regulation Checklist

Emotional and behavioral regulation

As shown in Table 1, group differences were observed for all emotion regulation and behavioral variables. First, overall group differences were observed in emotion regulation as measured by the ERC Composite score (F2,55 = 31.88, p < 0.001). As predicted, children in the STO group had poorer emotion regulation, evidenced by significantly lower ERC Composite scores than children in both comparison groups, ADHD (p = 0.002) and HC (p < 0.001). The ADHD group also had significantly poorer emotion regulation scores than the HC group (p < 0.001). The groups also differed significantly on BASC-2-PRS Hyperactivity raw scores (F2,55 = 29.97, p < 0.001). Post hoc tests showed greater scores in the STO group than the ADHD group (p = .001) and the HC group (p < 0.01). Children with ADHD were rated higher on BASC-2-PRS Hyperactivity than HC children (p = 0.001). Finally, group differences were also observed for BASC-2-PRS Attention Problems raw scores (F2,54 = 16.69, p < 0.001). Compared to the HC group, these scores were significantly higher in the STO (p < 0.001) and ADHD (p = 0.001) groups with no significant difference between the latter two groups.

Anterior midcingulate iFC analyses

Group analyses revealed significant differences, as shown in Figure 2. The STO group exhibited reduced positive aMCC iFC with a cluster in the ACC extending ventrally and rostrally from the aMCC seed (max = 3.83; MNI: 2, 22, 26) compared to the ADHD group. Conversely, the STO group exhibited significantly greater positive aMCC iFC with the precuneus (max = 3.58; MNI: –8, –68, 20) than the ADHD group.

Figure 2.

Figure 2

Open in a new tab

Normative comparisons

We then compared the iFC values extracted from these clusters (ACC and precuneus) for the STO and ADHD groups to those from the HC group. The STO group showed weaker positive iFC between the aMCC and local ACC regions (t36 = −2.661, p = .013) and stronger positive aMCC- precuneus iFC (t36 = 2.062, p = .047) than the HC group. No significant differences were found between the ADHD and HC groups on these measures. See Figure 2.

Dimensional analyses

As noted, psychopathology in the STO group might be more “severe” than in the ADHD group, and this difference, rather than differences in emotion dysregulation per se, may account for STO vs. ADHD differences in aMCC iFC. To address this issue, we performed multiple linear regression analyses across the STO and ADHD groups. The dependent variable was aMCC iFC, and the independent variables were ERC Composite scores and BASC-2-PRS Hyperactivity and Attention Problems raw scores. As shown in Table 2, analyses revealed that ERC Composite scores were not a significant predictor of aMCC iFC with surrounding cingulate regions (p = .96). However, BASC-2-PRS Hyperactivity scores significantly contributed to variability in this local ACC iFC (p = .01). There was also a trend for a relationship with BASC-2-PRS Attention Problems scores (p = .08). Conversely, ERC Composite scores contributed significantly to the variance of aMCC- precuneus iFC (p = .04), while BASC-2-PRS Hyperactivity and Attention Problems scores did not (p’s = .10 and .16, respectively).

Table 2.

Independent contributions of ADHD symptoms and emotion dysregulation to aMCC intrinsic functional connectivity

B (SE) t Significance
aMCC-ACC iFC
BASC-2-PRS Hyperactivity −.006 (.002) −2.75 p = .01
BASC-2-PRS Attention Problems .007 (.004) 1.797 p = .08
ERC Composite −.002 (.03) −.056 p = .96
aMCC- Precuneus iFC
BASC-2-PRS Hyperactivity .004 (.003) 1.695 p = .10
BASC-2-PRS Attention Problems −.006 (.004) −1.441 p = .16
ERC Composite −.072 (.034) −2.109 p = .04
Open in a new tab

aMCC = anterior midcingulate cortex; ACC = anterior cingulate cortex; iFC = intrinsic functional connectivity; BASC-2-PRS = Behavior Assessment System for Children Parent Rating Scale; ERC = Emotion Regulation Checklist

Discussion

The present study provides preliminary information about neural mechanisms underlying severe temper outbursts in children. Analyses focused on cingulate circuitry given the known involvement of this region in frustration and cognitive control of emotion. We predicted that children with severe temper outbursts would demonstrate altered iFC of aMCC circuits, as compared to children with ADHD free of severe temper outbursts, as well as healthy comparisons, with no differences expected between the two comparison groups. Compared to children with ADHD without outbursts, children with severe temper outbursts exhibited weaker positive aMCC iFC with local ACC regions and stronger positive iFC with the precuneus. These differences were also found between the STO group and healthy comparisons; in contrast, the ADHD and HC groups did not differ. Further, multiple regression analyses suggest unique associations between local ACC iFC and hyperactivity symptoms, and between aMCC- precuneus iFC and emotion regulation capacity. Thus, our findings suggest that disruptions in cingulate iFC may underlie both the behavioral and emotional dysregulation observed in children with severe temper outbursts.

Consistent with previous work (Kelly et al., 2008), all three groups showed positive iFC between the aMCC ROI and local ACC regions extending rostrally. However, children with severe temper outbursts exhibited a significant reduction in this iFC, compared to children with ADHD without outbursts, and healthy comparisons. Multiple regression analyses suggest that the group difference in local aMCC iFC reflects greater severity of ADHD symptoms in the STO group, rather than differences in emotion dysregulation. Previous work implicates the cingulate cortex in the overall pathophysiology of ADHD (Bush, Valera, & Seidman, 2005; Cortese et al., 2012). Compared to healthy youth, children with ADHD typically exhibit reduced task-related activation in the ACC (Dickstein et al., 2006) and reduced gray matter in the rostral ACC, which was identified in the current iFC analyses (Bonath, Tegelbeckers, Wilke, Flechtner, & Krauel, 2016). Disruption in local ACC iFC may also have implications for differential developmental outcomes between STO and ADHD youth. A recent study examined the associations between iFC of the executive function network in late adolescence (~ age 17) and changes in hyperactivity/impulsivity symptoms since age 11 (Francx et al., 2015). Greater symptom reduction was associated with stronger iFC of the ACC, suggesting that integration of this region into the executive control network may underlie symptom improvement. Similarly, the STO group in the present study evidenced a cluster of decreased local aMCC iFC that includes the same ACC region; this may reflect less integration of these regions into the executive control network, reflected behaviorally by increased severity of hyperactivity and impulsivity symptoms. As such, these children may be resistant to improvement in these symptoms with age. Longitudinal follow-up is needed to test this hypothesis further.

The STO group also exhibited significant positive iFC between the aMCC ROI and the precuneus that was not observed in either of the other two groups. Previous studies show no significant iFC between this cingulate region and precuneus in healthy children (Kelly et al., 2008) or adults (Margulies et al., 2007). Multiple regression analyses suggest that this hyperconnectivity is associated specifically with the emotion dysregulation demonstrated by children with STO. Similarly, greater global connectivity of a similar precuneus region has been shown in children and adults with bipolar disorder, a condition characterized by dysregulated emotion (Stoddard et al., 2016). The precuneus is a core region of the canonical default network (DN), which plays a role in social emotion regulation (Xie et al., 2016). Thus, the increased iFC between the aMCC and this region may reflect a greater reliance on others (i.e., parents, teachers) to help regulate strong emotions because children with STO do not possess sufficient skills to adequately self-regulate. Alternatively, hyperconnectivity with the precuneus may prevent the aMCC from effectively regulating negative affect. As discussed earlier, the aMCC and surrounding ACC regions are typically considered part of the executive control network and significant shifts occur in the iFC between and within this network and the DN across early development (Fair et al., 2007; Sato et al., 2014). Thus, the present finding may represent a deviation or delay in typical development that is mirrored by delayed acquisition of emotion regulation in children with STO. Of note, we failed to find differences in aMCC- precuneus iFC between the ADHD group without outbursts and healthy comparisons, which contrasts with previous findings in adults (Castellanos et al., 2008) and pediatric ADHD samples (Sun et al., 2012). It is possible that the abnormalities found in these earlier studies may have been due to the presence of emotion regulation impairments in ADHD, rather than to ADHD itself. This is an important hypothesis to consider; however, symptoms of emotion dysregulation were not included in these investigations so this cannot be directly tested.

The present findings need to be considered in the context of several study limitations. First, sample sizes were limited, reducing power to detect significant effects. It is challenging to obtain usable functional MRI data from children as young as five, particularly those with an ADHD diagnosis. Our overall success rate was good, allowing us to analyze more than 75% of our starting sample, but larger samples are needed to test complex interactions among ADHD and emotion regulation symptoms. A larger sample would also permit a more data-driven approach to examine emotion dysregulation and the functional connectivity of multiple regions of the ACC. Second, assessments of temper outbursts, ADHD symptoms, and emotion regulation all relied on parent reports. Thus, shared method variance may have contributed to elevated scores for both behavioral and emotional dysregulation in the STO group. However, inconsistent with this possibility, parents of children in the STO group did not endorse greater symptoms of inattention than parents of children in the ADHD group. Teacher reports were not available for all participants, and thus, could not be used as an additional measure of ADHD symptoms in this study. Inclusion of objective experimental or observational measures would further mitigate this concern. Third, study inclusion criteria did not allow examination of temper outbursts as a continuous measure. All children in the STO group had severe outbursts (limiting variability in severity) and to study clearly differentiated groups, children with ADHD with moderate tantrums were excluded (also providing little variance in outburst severity in the ADHD group). Such dimensional analyses might have provided greater power and increased the clinical relevance of the findings to a broader group of children with ADHD. Finally, to truly investigate the neural circuitry of severe temper outbursts without other confounding symptoms of dysregulation (i.e., hyperactivity and impulsivity), a 2 × 2 balanced design would be needed. While we have been able to examine three of these four groups in this study and previously, it has not been feasible to recruit a sufficient sample of children with STO without ADHD. Inclusion of a comparison group of children with ADHD without STO represents our best, albeit imperfect, effort to disentangle the associations of behavioral and emotional dysregulation with iFC measures. Thus, while we cannot definitively state that findings are specific to emotion dysregulation (STO), and not related to other symptoms of dysregulation (i.e., hyperactivity), they provide initial observations that such differentiation may exist, warranting further study.

In summary, findings provide initial evidence of disruptions in a core cingulate network in young children with severe temper outbursts. This represents a first attempt at disentangling iFC alterations associated with symptoms of hyperactivity/impulsivity from those of emotion dysregulation in a heterogeneous clinical group of children with severe temper outbursts. Replication of these results will be key to our understanding of neurobiological models of impairing severe temper outbursts in children, and ultimately, of the importance of investigating their longitudinal outcome.

Acknowledgments

Funding: We thank Aleta Angelosante, Ph.D., Vasco Lopes, Ph.D., and Sheina Godovich, as well as our participants and their families, for their contributions to this research. This work was supported by a Brain and Behavior Research Foundation (previously NARSAD) Young Investigator Award to A. K. Roy, a grant from the National Institute of Mental Health (R01MH091140-01; A.K. Roy, P.I.), a grant from the Seevak Family Foundation (R.G. Klein, P.I.), and an AACAP Pilot Research Award for Junior Faculty and Child and Adolescent Psychiatry Residents, supported by the Elaine Schlosser Lewis Fund awarded to L. Hulvershorn.

References

  1. Abler B, Walter H, Erk S. Neural correlates of frustration. Neuroreport. 2005;16(7):669–672. doi: 10.1097/00001756-200505120-00003. [DOI] [PubMed] [Google Scholar]
  2. Avants BB, Tustison NJ, Song G, Cook PA, Klein A, Gee JC. A reproducible evaluation of ANTs similarity metric performance in brain image registration. Neuroimage. 2011;54(3):2033–2044. doi: 10.1016/j.neuroimage.2010.09.025. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Bhatia MS, Nigam VR, Bohra N, Malik SC. Attention deficit disorder with hyperactivity among paediatric outpatients. Journal of Child Psychology and Psychiatry. 1991;32(2):297–306. doi: 10.1111/j.1469-7610.1991.tb00308.x. [DOI] [PubMed] [Google Scholar]
  4. Bonath B, Tegelbeckers J, Wilke M, Flechtner HH, Krauel K. Regional Gray Matter Volume Differences Between Adolescents With ADHD and Typically Developing Controls: Further Evidence for Anterior Cingulate Involvement. Journal of Attention Disorders. 2016 doi: 10.1177/1087054715619682. [DOI] [PubMed] [Google Scholar]
  5. Brotman MA, Schmajuk M, Rich BA, Dickstein DP, Guyer AE, Costello EJ, Leibenluft E. Prevalence, clinical correlates, and longitudinal course of severe mood dysregulation in children. Biological Psychiatry. 2006;60(9):991–997. doi: 10.1016/j.biopsych.2006.08.042. [DOI] [PubMed] [Google Scholar]
  6. Burke JD, Hipwell AE, Loeber R. Dimensions of oppositional defiant disorder as predictors of depression and conduct disorder in preadolescent girls. Journal of the American Academy of Child and Adolescent Psychiatry. 2010;49(5):484–492. doi: 10.1097/00004583-201005000-00009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Bush G, Valera EM, Seidman LJ. Functional neuroimaging of attention-deficit/hyperactivity disorder: a review and suggested future directions. Biological Psychiatry. 2005;57(11):1273–1284. doi: 10.1016/j.biopsych.2005.01.034. [DOI] [PubMed] [Google Scholar]
  8. Carlson GA, Potegal M, Margulies D, Gutkovich Z, Basile J. Rages–what are they and who has them? Journal of Child and Adolescent Psychopharmacology. 2009;19(3):281–288. doi: 10.1089/cap.2008.0108. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Castellanos FX, Margulies DS, Kelly C, Uddin LQ, Ghaffari M, Kirsch A, Milham MP. Cingulate-precuneus interactions: a new locus of dysfunction in adult attention-deficit/hyperactivity disorder. Biological Psychiatry. 2008;63(3):332–337. doi: 10.1016/j.biopsych.2007.06.025. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Cortese S, Kelly C, Chabernaud C, Proal E, Di Martino A, Milham MP, Castellanos FX. Toward systems neuroscience of ADHD: a meta-analysis of 55 fMRI studies. American Journal of Psychiatry. 2012;169(10):1038–1055. doi: 10.1176/appi.ajp.2012.11101521. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Deveney CM, Connolly ME, Haring CT, Bones BL, Reynolds RC, Kim P, Leibenluft E. Neural mechanisms of frustration in chronically irritable children. American Journal of Psychiatry. 2013;170(10):1186–1194. doi: 10.1176/appi.ajp.2013.12070917. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Dickstein SG, Bannon K, Castellanos FX, Milham MP. The neural correlates of attention deficit hyperactivity disorder: an ALE meta-analysis. Journal of Child Psychology and Psychiatry. 2006;47(10):1051–1062. doi: 10.1111/j.1469-7610.2006.01671.x. [DOI] [PubMed] [Google Scholar]
  13. Dominick KC, Davis NO, Lainhart J, Tager-Flusberg H, Folstein S. Atypical behaviors in children with autism and children with a history of language impairment. Research in Developmental Disabilities. 2007;28(2):145–162. doi: 10.1016/j.ridd.2006.02.003. [DOI] [PubMed] [Google Scholar]
  14. Fair DA, Dosenbach NU, Church JA, Cohen AL, Brahmbhatt S, Miezin FM, Schlaggar BL. Development of distinct control networks through segregation and integration. Proceedings of the National Academy of Sciences of the United States of America. 2007;104(33):13507–13512. doi: 10.1073/pnas.0705843104. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Francx W, Oldehinkel M, Oosterlaan J, Heslenfeld D, Hartman CA, Hoekstra PJ, Mennes M. The executive control network and symptomatic improvement in attention-deficit/hyperactivity disorder. Cortex. 2015;73:62–72. doi: 10.1016/j.cortex.2015.08.012. [DOI] [PubMed] [Google Scholar]
  16. Gatzke-Kopp LM, Greenberg M, Bierman K. Children’s parasympathetic reactivity to specific emotions moderates response to intervention for early-onset aggression. Journal of Clinical Child and Adolescent Psychology. 2015;44(2):291–304. doi: 10.1080/15374416.2013.862801. [DOI] [PubMed] [Google Scholar]
  17. Giesbrecht GF, Miller MR, Muller U. The anger-distress model of temper tantrums: Associations with emotional reactivity and emotional competence. Infant and Child Development. 2010;19:478–497. [Google Scholar]
  18. Greve DN, Fischl B. Accurate and robust brain image alignment using boundary-based registration. Neuroimage. 2009;48(1):63–72. doi: 10.1016/j.neuroimage.2009.06.060. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. He J, Jin X, Zhang M, Huang X, Shui R, Shen M. Anger and selective attention to reward and punishment in children. Journal of Experimental Child Psychology. 2013;115(3):389–404. doi: 10.1016/j.jecp.2013.03.004. [DOI] [PubMed] [Google Scholar]
  20. Johnco C, Salloum A, De Nadai AS, McBride N, Crawford EA, Lewin AB, Storch EA. Incidence, clinical correlates and treatment effect of rage in anxious children. Psychiatry Research. 2015;229(1–2):63–69. doi: 10.1016/j.psychres.2015.07.071. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Kanske P, Heissler J, Schonfelder S, Bongers A, Wessa M. How to regulate emotion? Neural networks for reappraisal and distraction. Cerebral Cortex. 2011;21(6):1379–1388. doi: 10.1093/cercor/bhq216. [DOI] [PubMed] [Google Scholar]
  22. Kaufman AS, Kaufman NL. Kaufman Brief Intelligence Test Second Edition (KBIT-2) Circle Pines, MN: American Guidance Service; 2004. [Google Scholar]
  23. Kaufman J, Birmaher B, Brent D, Rao U, Flynn C, Moreci P, Ryan N. Schedule for Affective Disorders and Schizophrenia for School-Age Children-Present and Lifetime Version (K-SADS-PL): initial reliability and validity data. J Am Acad Child Adolesc Psychiatry. 1997;36(7):980–988. doi: 10.1097/00004583-199707000-00021. [DOI] [PubMed] [Google Scholar]
  24. Kelly AM, Di Martino A, Uddin LQ, Shehzad Z, Gee DG, Reiss PT, Milham MP. Development of anterior cingulate functional connectivity from late childhood to early adulthood. Cerebral Cortex. 2008;19:640–657. doi: 10.1093/cercor/bhn117. [DOI] [PubMed] [Google Scholar]
  25. Leibenluft E, Cohen P, Gorrindo T, Brook JS, Pine DS. Chronic versus episodic irritability in youth: a community-based, longitudinal study of clinical and diagnostic associations. Journal of Child and Adolescent Psychopharmacology. 2006;16(4):456–466. doi: 10.1089/cap.2006.16.456. [DOI] [PubMed] [Google Scholar]
  26. Makris N, Buka SL, Biederman J, Papadimitriou GM, Hodge SM, Valera EM, Seidman LJ. Attention and executive systems abnormalities in adults with childhood ADHD: A DT-MRI study of connections. Cerebral Cortex. 2008;18(5):1210–1220. doi: 10.1093/cercor/bhm156. [DOI] [PubMed] [Google Scholar]
  27. Malone RP, Gratz SS, Delaney MA, Hyman SB. Advances in drug treatments for children and adolescents with autism and other pervasive developmental disorders. CNS Drugs. 2005;19(11):923–934. doi: 10.2165/00023210-200519110-00003. [DOI] [PubMed] [Google Scholar]
  28. Margulies DS, Kelly AM, Uddin LQ, Biswal BB, Castellanos FX, Milham MP. Mapping the functional connectivity of anterior cingulate cortex. Neuroimage. 2007;37(2):579–588. doi: 10.1016/j.neuroimage.2007.05.019. [DOI] [PubMed] [Google Scholar]
  29. McRae K, Hughes B, Chopra S, Gabrieli JD, Gross JJ, Ochsner KN. The neural bases of distraction and reappraisal. Journal of Cognitive Neuroscience. 2010;22(2):248–262. doi: 10.1162/jocn.2009.21243. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Pawliczek CM, Derntl B, Kellermann T, Gur RC, Schneider F, Habel U. Anger under control: neural correlates of frustration as a function of trait aggression. PLoS One. 2013;8(10):e78503. doi: 10.1371/journal.pone.0078503. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Perlman SB, Jones BM, Wakschlag LS, Axelson D, Birmaher B, Phillips ML. Neural substrates of child irritability in typically developing and psychiatric populations. Developmental Cognitive Neuroscience. 2015;14:71–80. doi: 10.1016/j.dcn.2015.07.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Potegal M, Davidson RJ. Temper tantrums in young children: 1. Behavioral composition. Journal of developmental and behavioral pediatrics : JDBP. 2003;24(3):140–147. doi: 10.1097/00004703-200306000-00002. [DOI] [PubMed] [Google Scholar]
  33. Reynolds CR, Kamphaus RW. Behavior Assessment System for Children. Second. Circle Pines, MN: 2004. [Google Scholar]
  34. Rich BA, Carver FW, Holroyd T, Rosen HR, Mendoza JK, Cornwell BR, Leibenluft E. Different neural pathways to negative affect in youth with pediatric bipolar disorder and severe mood dysregulation. Journal of Psychiatric Research. 2011;45(10):1283–1294. doi: 10.1016/j.jpsychires.2011.04.006. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Rich BA, Schmajuk M, Perez-Edgar KE, Fox NA, Pine DS, Leibenluft E. Different psychophysiological and behavioral responses elicited by frustration in pediatric bipolar disorder and severe mood dysregulation. American Journal of Psychiatry. 2007;164(2):309–317. doi: 10.1176/ajp.2007.164.2.309. [DOI] [PubMed] [Google Scholar]
  36. Rilling JK, Goldsmith DR, Glenn AL, Jairam MR, Elfenbein HA, Dagenais JE, Pagnoni G. The neural correlates of the affective response to unreciprocated cooperation. Neuropsychologia. 2008;46(5):1256–1266. doi: 10.1016/j.neuropsychologia.2007.11.033. [DOI] [PubMed] [Google Scholar]
  37. Rowe R, Costello EJ, Angold A, Copeland WE, Maughan B. Developmental pathways in oppositional defiant disorder and conduct disorder. Journal of Abnormal Psychology. 2010;119(4):726–738. doi: 10.1037/a0020798. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Roy AK, Klein RG, Angelosante A, Bar-Haim Y, Leibenluft E, Hulvershorn L, Spindel C. Clinical features of young children referred for impairing temper outbursts. Journal of Child and Adolescent Psychopharmacology. 2013;23(9):588–596. doi: 10.1089/cap.2013.0005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Sato JR, Salum GA, Gadelha A, Picon FA, Pan PM, Vieira G, Jackowski AP. Age effects on the default mode and control networks in typically developing children. Journal of Psychiatric Research. 2014;58:89–95. doi: 10.1016/j.jpsychires.2014.07.004. [DOI] [PubMed] [Google Scholar]
  40. Seidman LJ, Valera EM, Makris N, Monuteaux MC, Boriel DL, Kelkar K, Biederman J. Dorsolateral prefrontal and anterior cingulate cortex volumetric abnormalities in adults with attention-deficit/hyperactivity disorder identified by magnetic resonance imaging. Biological Psychiatry. 2006;60(10):1071–1080. doi: 10.1016/j.biopsych.2006.04.031. [DOI] [PubMed] [Google Scholar]
  41. Shackman AJ, Salomons TV, Slagter HA, Fox AS, Winter JJ, Davidson RJ. The integration of negative affect, pain and cognitive control in the cingulate cortex. Nature Reviews Neuroscience. 2011;12(3):154–167. doi: 10.1038/nrn2994. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Siegrist J, Menrath I, Stocker T, Klein M, Kellermann T, Shah NJ, Schneider F. Differential brain activation according to chronic social reward frustration. Neuroreport. 2005;16(17):1899–1903. doi: 10.1097/01.wnr.0000186601.50996.f7. [DOI] [PubMed] [Google Scholar]
  43. Spunt RP, Lieberman MD, Cohen JR, Eisenberger NI. The phenomenology of error processing: the dorsal ACC response to stop-signal errors tracks reports of negative affect. Journal of Cognitive Neuroscience. 2012;24(8):1753–1765. doi: 10.1162/jocn_a_00242. [DOI] [PubMed] [Google Scholar]
  44. Stoddard J, Gotts SJ, Brotman MA, Lever S, Hsu D, Zarate C, Leibenluft E. Aberrant intrinsic functional connectivity within and between corticostriatal and temporal-parietal networks in adults and youth with bipolar disorder. Psychological Medicine. 2016;46(7):1509–1522. doi: 10.1017/S0033291716000143. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Stringaris A, Cohen P, Pine DS, Leibenluft E. Adult outcomes of youth irritability: a 20-year prospective community-based study. American Journal of Psychiatry. 2009;166(9):1048–1054. doi: 10.1176/appi.ajp.2009.08121849. [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Stringaris A, Goodman R. Longitudinal outcome of youth oppositionality: irritable, headstrong, and hurtful behaviors have distinctive predictions. Journal of the American Academy of Child & Adolescent Psychiatry. 2009;48(4):404–412. doi: 10.1097/CHI.0b013e3181984f30. [DOI] [PubMed] [Google Scholar]
  47. Sun L, Cao Q, Long X, Sui M, Cao X, Zhu C, Wang Y. Abnormal functional connectivity between the anterior cingulate and the default mode network in drug-naive boys with attention deficit hyperactivity disorder. Psychiatry Research. 2012;201(2):120–127. doi: 10.1016/j.pscychresns.2011.07.001. [DOI] [PubMed] [Google Scholar]
  48. Swingler MM, Perry NB, Calkins SD. Neural plasticity and the development of attention: Intrinsic and extrinsic influences. Development and Psychopathology. 2015;27(2):443–457. doi: 10.1017/S0954579415000085. [DOI] [PubMed] [Google Scholar]
  49. Wakschlag LS, Choi SW, Carter AS, Hullsiek H, Burns J, McCarthy K, Briggs-Gowan MJ. Defining the developmental parameters of temper loss in early childhood: implications for developmental psychopathology. Journal of Child Psychology & Psychiatry. 2012;53(11):1099–1108. doi: 10.1111/j.1469-7610.2012.02595.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  50. Xie X, Mulej Bratec S, Schmid G, Meng C, Doll A, Wohlschlager A, Sorg C. How do you make me feel better? Social cognitive emotion regulation and the default mode network. Neuroimage. 2016;134:270–280. doi: 10.1016/j.neuroimage.2016.04.015. [DOI] [PubMed] [Google Scholar]
  51. Yu R, Mobbs D, Seymour B, Rowe JB, Calder AJ. The neural signature of escalating frustration in humans. Cortex. 2014;54:165–178. doi: 10.1016/j.cortex.2014.02.013. [DOI] [PubMed] [Google Scholar]

RESOURCES