Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2026 May 1.
Published in final edited form as: Psychol Assess. 2025 May;37(5):214–226. doi: 10.1037/pas0001368

Reproducible Structure with Measurement Invariance for the Parent-report Early Adolescent Temperament Questionnaire: Findings from three independent samples

Michael B Kozlowski 1, Hannah E Morton 1, Joel T Nigg 1, Sarah L Karalunas 2
PMCID: PMC12014144  NIHMSID: NIHMS2062943  PMID: 40257895

Abstract

Differences in adolescent temperament are associated with innumerable psychological outcomes in the developmental literature and can help link adult personality-based nosology to earlier development. The Early Adolescent Temperament Questionnaire, Revised (EATQ-R) is one important measure of adolescent temperament designed to capture constructs within the influential Rothbart temperament model. Yet conflicting factor structures and minimal evidence for measurement invariance across samples and clinical groups have limited its ability to further temperament-based understanding of psychopathology. The goal of the present study was to identify reproducible measurement structures for the parent-rated and self-rated EATQ-R in multiple large independent samples and to evaluate how that structure corresponded to their proposed theoretical structure. We also tested measurement invariance and compared temperament characteristics in youth with and without attention-deficit/hyperactivity disorder (ADHD). Findings support the lower-order theoretical structure using a reduced set of items in the parent-rated form, including evidence for measurement invariance across samples and clinical groups. Findings confirm important patterns of temperament variation associated with ADHD diagnosis, including lower Effortful Control and differences in expression of Negative Affect and Surgency. The self-rated form demonstrated poor structural validity and could not be reliably replicated in a confirmatory sample. Parent-reported temperament may help link personality-based models of psychopathology to earlier developmental periods where psychopathology often emerges.

Keywords: temperament, adolescence, factor analysis, measurement invariance, ADHD

Temperament broadly refers to early emerging individual differences in patterns of reactivity and self-regulation that differentiate across development to form the foundations of personality (Clark & Watson, 2021). Of many theoretical models, few are more influential than the work of Mary Rothbart and colleagues. They were among the first to theorize a complex temperament structure starting in early infancy that follows a developmental trajectory eventually leading to the emergence of adult personality traits (Evans & Rothbart, 2007; Rothbart, 1981; Rothbart et al., 2000; Rothbart & Mauro, 1990). Rothbart’s theory of temperament structure was built on earlier theoretical work (Chess et al., 1963; Cattell, 1973; Eysenck, 1967) and relied heavily on exploratory factor analysis to test its assumptions. Their theory suggests that temperament traits in middle childhood and adolescence at least partially align with adult personality models. This means that a clearer understanding of temperament may eventually help link child and adult models of psychopathology, which increasingly emphasize the importance of trait-level personality variation (Krueger et al., 2021).

Consistent with this suggestion, individual differences in temperament predict the severity of internalizing and externalizing psychopathology (Miola et al., 2021; Balle et al., 2022; Durdurak et al., 2022). In attention-deficit/hyperactivity disorder (ADHD, a neurodevelopmental disorder and the focus of our current report), emerging literature finds that temperament variation predicts both etiology and clinical course (Nigg et al., 2020a). Critically, temperament domains related to ADHD symptoms (e.g., impulsivity, attentional control) and those related to negative and positive emotionality all appear to have unique relevance to ADHD phenotype, genotype, and outcomes (Karalunas et al., 2014; Martel et al., 2022; Nigg et al., 2020b). Neurodevelopmental disorders have been particularly difficult to place within the emerging dimensional personality models in adult psychopathology, highlighting the importance of further study.

Despite temperament’s strong theoretical relevance and promising potential contributions to psychopathology research, measures of child and adolescent temperament often have insufficient psychometric properties (Shiner et al., 2021; Tackett & Durbin, 2017), potentially undermining studies that rely on them and hampering efforts to broaden dimensional adult models into earlier developmental stages when psychopathology emerges. Given both the theoretical importance of temperament assessment for psychopathology research and the psychometric limitations of existing measures, here we seek to clarify the structure of a widely-used measurement tool based on Rothbart’s temperament model: The Early Adolescent Temperament Questionnaire—Revised (EATQ-R; Ellis & Rothbart, 2001), including how the structure aligns or fails to align with the temperament model from which it was built.

Adolescence is a key period in which temperament solidifies into adult personality and during which there is divergence in clinical course related to psychopathology (Shiner & DeYoung, 2013; DiLorenzo et al., 2021) Clarifying the EATQ-R structure and its correspondence to theory can help inform future use of the measure for studies of both basic personality development and psychopathology. Unique to our current effort versus prior investigations of the EATQ-R (Latham et al., 2020; Lawson et al., 2021; Snyder et al., 2015), are: 1) independent sample replication in three large samples: the Michigan ADHD-1000 cohort (Nigg et al., 2023), the Oregon ADHD-1000 cohort (Nigg et al., 2023), and the Adolescent Brain & Cognitive Development (ABCD) cohort (Jernigan et al., 2018); 2) consideration of both self-report and parent-rated forms; and 3) examination of measurement invariance across ADHD and non-ADHD participants.

Rothbart’s Theoretical Structure of Temperament

Early work by Rothbart (1981) sought to clarify the structure of the Thomas and Chess (1977) infant temperament model and identified two facets of temperament related to positive (Surgency) and negative (Negative Affect) reactivity, as well as a third factor labeled Orienting/Regulatory Capacity that at this young age was indicated by items related to infant soothability, cuddliness, and reflexive attention (Gartstein & Rothbart, 2003). Additional studies extending temperament structure into middle childhood found consistency in the positive and negative reactivity traits. There was also support for a trait related to regulatory capacity but with a shift at the item level from behaviors that support infant-parent co-regulation to those reflecting the child’s regulatory control as would be expected in this age range. The regulatory factor was re-labeled Effortful Control to capture this shift in item-level representation (Karalunas, Nigg, & Figuracion, 2024).

Extending to adolescence, the same higher-order dimensions: Effortful Control, Negative Affect, and Surgency all remain core parts of the theoretical model. These higher-order dimensions are each indicated by a series of lower-order traits. At this age, Effortful Control is indicated by observable behaviors including control over attention (Attentional Focusing); the ability to manage one’s impulses (Activation Control); the ability to suppress inappropriate impulses (Inhibitory Control); and the detection of low-intensity stimulation within the environment (Perceptual Sensitivity). Negative Affect, as the name implies, is indicated by lower-order observable traits related to anticipation of distress (Fear), interruption of goal-oriented behavior (Frustration), and experiences with disappointment or suffering (Sadness). Finally, Surgency (sometimes termed Positive Emotionality) is indicated by lower-order traits such as sensation-seeking from novel or intense experiences (High-Intensity Pleasure), low levels of behavioral inhibition (low Shyness), and sensitivity to pleasurable experiences (Pleasure Sensitivity).

A fourth Affiliation factor sometimes emerges during adolescence and has been described as the desire for warmth or affection, related to sociability. However, it has been variously conceptualized as an independent higher-order factor indicated by scales designed to measure Affiliation, Perceptual Sensitivity, and Pleasure Sensitivity (Ellis & Rothbart, 2001; Latham et al., 2020) or as a lower-order factor subsumed under Surgency/Positive Emotionality (Snyder et al., 2015; Rothbart, 2007). The role and distinguishability of Affiliation from other aspects of positive emotionality remains a key theoretical question for the Rothbart temperament model (Lawson et al., 2021).

Empirical Status of the EATQ-R as a Temperament Measure

From a psychometric perspective, forms designed to capture the Rothbart temperament model have often required revision to adequately fit data in new samples (Kotelnikova et al., 2017; Kozlowski et al., 2022; Muris & Meesters, 2009) ranging from relatively minor model modifications, such as dropping unreliable or closely related items and allowing cross-loaded items (Kozlowski et al., 2022; Latham et al., 2020), to more substantial revision of the originally proposed factor structure (Kotelnikova et al., 2016). The primary tool for measuring Rothbart’s model of temperament traits during adolescence is the Early Adolescent Temperament Questionnaire, Revised (EATQ-R; Ellis & Rothbart, 2001). There are two current versions of the EATQ-R— one for parents to rate their children and another for adolescents to rate themselves. Both forms have multiple lower-order scales aligned with the theoretical model described above and intended to reflect the higher-order constructs of Effortful Control, Surgency, and Negative Affect. The parent-rated form has a single scale capturing Affiliation, while the self-report form includes additional scales that might form an Affiliation factor. Although both are widely used, the psychometric evidence for the theoretical structure remains limited (e.g., the parent-rated form was validated on less than 200 participants). Perhaps because of this, recent studies have provided mixed support for whether the form’s lower-order scales are reproducible and can be collapsed into a theorized three- (Effortful Control, Surgency, Negative Affect) or four-factor (Effortful Control, Surgency, Negative Affect, Affiliation) model.

A large study on the English self-rated form conducted by Snyder et al. (2015) found support for bifactor models with three general higher-order factors corresponding to Effortful Control, Negative Affect, and what they called a Positive Emotionality factor whose indicators were all related to Affiliation. A separate factor capturing the Surgency aspect of Positive Emotionality was distinct from the general Positive Emotionality/Affiliation factor. Fit statistics for these models were modest, but factors were ultimately retained given they broadly aligned with theory and there was evidence of validity based on correlations with symptom ratings. Lawson et al. (2021) attempted to replicate Snyder et al.’s (2015) structure with a community sample of Hispanic children and adolescents. They ultimately retained three separate correlated factors models with factors corresponding to Effortful Control, Affiliation/Surgency, and Negative Affect. However, the models did not fit the data according to traditional community standards and could not identify a consistent set of well-fitting models that accounted for the higher-order structure. Finally, Latham et al. (2020) conducted the largest study with the English self-rated form. They argued in support of a four-factor model with higher-order factors of Effortful Control, Surgency, Negative Affect, and Affiliation. Crucially, Latham et al.’s (2020) model, which unlike the other two studies provided adequate fit to the data, allowed for cross-loading of items and factors. Findings point to a more complex structure than what is typically modeled in strictly confirmatory approaches.

Importance of Accounting for Complex Structure

The Latham et al. (2020) findings highlight the point that strict confirmatory models where cross-loadings are specified to zero have rarely provided adequate fit to data for any temperament measure. Allowing cross-loadings can be interpreted as a sign of poor construct validity, but cross-loadings can also be interpreted as conceptually meaningful when assessing temperament and personality constructs where relationships among constructs are theoretically expected (Hopwood & Donnellan, 2010). For example, while high fear is expected to load primarily on a Negative Affect factor, low ratings for some fear items may also be associated with Surgency. Similarly, at the higher-order level, degree of Effortful Control would be expected to correlate moderately with expression of Surgency and Negative Affect. To this point, while the Latham et al. (2020) model allowed for cross-loadings, the cross-loaded items did not substantively change the interpretation of any of the factors.

Most confirmatory factor analytic approaches start by constraining cross-loadings to zero. However, in the case of complex structures that require such cross-loadings, this often leads to inflated correlations among factors (Marsh et al., 2014) and requires post-hoc modifications that tend to not be reproducible across samples, further limiting their utility for the field (Lawson et al., 2021; Kozlowski et al., 2022; Peterson et al., 2017). Exploratory Structural Equation Modeling (ESEM) provides an alternative approach that combines aspects of confirmatory and exploratory factor analytic approaches (Asparouhov & Muthén, 2009). Despite its name, in the context used here it is considered primarily a confirmatory approach that allows more flexibility than other confirmatory models. ESEM allows for the specification of a confirmatory model, while still allowing for minor cross-loading that would result in poor model fit in a traditional confirmatory factor analysis (CFA) framework. Prior studies of the EATQ-R have not applied ESEM, although its assumptions provide a strong fit to those of temperament theory (e.g., items on distinct lower-order factors that reflect the same higher-order construct would be expected to have cross-loadings). Further, although some studies have used sample splitting to test the reproducibility of identified best-fitting factor structures, studies have not generally included multiple independent samples, which will be a key step in assuring reproducibility.

Need to Address Multiple Reporters

Finally, current psychometric research on the EATQ-R has focused on the structure of the self-rated form to the virtual exclusion of studies examining the structure of the parent-rated form. Yet parent ratings are critical in adolescence given emotion and behavior ratings of multiple informants can yield important insights (King et al., 2013; van der Ende et al., 2012). For the EATQ-R in particular, the cross-informant agreement is generally low-to-moderate (r’s = 0.29–0.46; Ellis & Rothbart, 2001), and ample literature suggests the importance of considering multiple informants of behavior in this age range, with each adding valid information (Martel et al., 2017). The structure of the parent form is particularly uncertain because several temperament constructs (e.g., Sadness) are captured by scales described in some places as “behavioral scales” that are named to emphasize their relevance to clinical problems (e.g., Depressed Mood). However, these naming conventions reflect older understandings of psychopathology that emphasized categorical distinctions rather than dimensional variation, and there is minimal evidence that items on these scales reflect fundamentally different constructs than the related temperament scales. Examining the structure of the parent forms is an important goal for aligning existing research on adolescent psychopathology with temperament and personality risk perspectives.

Present Study

Temperament theory persists as an important area of research with direct implications for applied psychopathology research despite ongoing concerns about whether adequate measurement is available for dominant theoretical models of temperament development. Arriving at a reproducible solution to the EATQ-R and understanding how that structure aligns with the theory on which the measure was built would allow clinical research to move forward in areas where temperament is of specific interest. Here, we apply CFA and ESEM approaches to determine whether a reproducible EATQ-R structure that aligns with Rothbart’s temperament theory can be identified in three large, independent samples: 1) Michigan ADHD-1000 (a case-control sample spanning a wide age range from early through late adolescence); 2) Oregon ADHD-1000 (a case-control sample with a narrower age range in mid-adolescence); and 3) the Adolescent Brain & Cognitive Development cohort (ABCD; a population-based sample with a narrow age range in early adolescence). We further address gaps in the literature by examining the structure of both the child-report and parent-report forms, as well as addressing measurement invariance across clinical and non-clinical groups, focusing on ADHD as an exemplar clinical group where temperament variation is of specific interest. We anticipated that the more flexible ESEM model would provide a better fit to the data than the strict confirmatory model.

Method

Participants

To answer our research questions, we relied on three publicly available datasets that we examined sequentially.

Sample 1: Michigan ADHD-1000 Dataset

Participants in our discovery dataset were 705 children aged 8–17 (M = 11.19, SD = 2.03) recruited as part of a larger cross-sectional case-control study on ADHD in the Midwestern region of the United States (Nigg et al., 2023). 402 participants met the research criteria for ADHD and 303 met the criteria for community controls (See Nigg et al., 2023 and/or the supplemental information for more details). Table 1 contains demographic information for all participants. The data analysis for the current study was approved by the Oregon Health & Science University Institutional Review Board (protocol #7100).

Table 1.

Participant Demographics

Michigan-1000 Oregon-1000 ABCD Study®
N 705 439 10,414
M (SD) Age 11.46(2.27) 14.39(1.44) 12.00(0.66)
Sex assigned Male % 54.8% 58.5% 52.9%
Sex assigned Female % 45.2% 33.5% 47.1%
ADHD N 402 225 363
Race
 White % 71.6% 85% 75.8%
 Black % 9.4% 6.8% 19.8%
 Asian % 1.1% 5.7% 6.8%
 American Indian/Alaska Native % 0.9% 1.8% 3.5%
 Native Hawaiian/Pacific Islander % 0.1% 0.7% 0.4%
 Other and/or more than one race % 11.3% 30.7% 6.3%
 Unknown or unreported % 6.8% 0.7% 0.9%
Ethnicity
 Hispanic % 5.4% 5.9% 20%
Median Income $60,000 $50,000–75,000 $35,000–$49,999
Maternal report % 77% 78.5% 84.3%
Open in a new tab

Sample 2: Oregon ADHD-1000 Dataset

Participants in our test/confirmation sample were 439 children aged 12–18 (M = 14, SD = 1.45) recruited in the Pacific Northwest, participating in a large, longitudinal case-control study of ADHD (Nigg et al., 2023). Data for the present work came from the sixth wave of data collection. At that wave, 225 participants met the criteria for ADHD or had elevated ADHD symptom profiles and 214 were determined to be non-ADHD controls by best-estimate research diagnostic procedures. Comorbid psychopathology was free to vary in both groups (see Nigg et al., 2023 and/or the supplemental information for more details). The study was approved by the Oregon Health & Science University Institutional Review Board (protocol #4818 and #6258)

Sample 3: ABCD Study®

Participants in our generalizability sample came from the Adolescent Brain & Cognitive Development Study (ABCD Study®), a large, population-based, longitudinal study of child and adolescent development (Jernigan et al., 2018). The EATQ-R data was collected during the two-year follow-up visit of the study (ABCD Data Release 4.0) and contains parent ratings of 10,414 children aged 10–14 (M = 12, SD = 0.66). Of those, 553 parents elected to complete the Spanish version of the EATQ-R. These ratings were analyzed separately from the English form. To characterize ADHD, we relied on “ADHD tier 4” definitions developed by Cordova et al. (2022) in the baseline sample because it represents the most stringent classification of ADHD and is most comparable to procedures in Sample 1 and Sample 2. By these criteria, 363 participants met the criteria for ADHD and had a completed parent-report EATQ-R. The self-reported EATQ-R is not available in ABCD. The ABCD Study received their own central IRB approval supplemented by site-specific IRBs in some cases. The specific use of the current data is IRB exempt because it uses only deidentified data in secondary data analyses.

Measures

The Early Adolescent Temperament Questionnaire Revised (EATQ-R)

The EATQ-R was completed by both parents and children in Samples 1 and 2; only the parent-rated form was available in Sample 3. The parent-rated form is a 62-item form containing eight lower-order temperament subscales: Activation Control, Affiliation, Attention, Fear, Frustration, High-Intensity Pleasure, Inhibitory Control, and Shyness, and two additional “behavioral” scales: Depressed Mood and Aggression. The youth self-rated form contains 65 items and all of the subscales from the parent form, as well as scales measuring Pleasure Sensitivity and Perceptual Sensitivity, which are omitted from the parent-rated form at this age. All items from both forms were used in the present analysis. On both versions, items are rated using a five-point Likert-type scale.

Plan for Analysis

The Michigan ADHD-1000 sample served as a discovery cohort, the Oregon ADHD-1000 sample as the test cohort, and the ABCD sample as a generalizability cohort (for the parent form only). Despite ABCD being the largest cohort overall, this order allowed us to have a single test sample for both self- and parent-report forms and to prioritize tests of measurement invariance in samples with the best-characterized clinical groups. For all analyses, items were treated as continuous given the assumption of a normal latent distribution of temperament constructs (Robitzsch, 2020). Analyses for all factor analytic models were conducted in Mplus version 8.8 (Muthén & Muthén, 1998–2017) using the Maximum Likelihood Robust estimator with the COMPLEX command to account for family-level clustering among pairs of siblings and to handle item-level missingness. Items were evaluated for skewness and kurtosis using an absolute value cutoff of 3.00. All items met these criteria. MCAR tests conducted in R using the naniar package (Tierney & Cook, 2023) revealed data was missing completely at random (specific values for each sample are available in the supplemental material).

Discovery sample

Lower-order factor models

For both forms, we first tested the lower-order theoretical model using both ESEM and CFA models as confirmatory approaches. For the lower-order CFAs, items were specified as loading factors as outlined in the standard scoring procedures. Factors were allowed to correlate. Because ESEM models require a rotation, we employed a geomin rotation with 100 random starts. For CFA models, we also specified 100 starts. Standard combination rules were used as guidance to determine model fit: the model chi-square, CFI ≥ 0.90, TLI ≥ 0.90, RMSEA ≤ 0.08, and SRMR < 0.08.

When the lower-order theoretical model could not be retained due to poor fit, we proceeded with discovery analyses to identify the best-fitting model. This involved conducting exploratory factor analyses with parallel analysis over 1000 iterations in our discovery sample. The model retained factors with eigenvalues produced at the 95% confidence interval (Lim & Jahng, 2019). For trimming, we removed items that either had a low factor loading on every factor (λ ≤ 0.40) or a high cross-loading on a non-theorized factor (λ ≥ 0.20). We additionally evaluated items with highly correlated errors and removed similar items (Marsh et al., 2005) while also making efforts to preserve the lower-order scales. We included this second consideration because removing items solely based on their factor loadings could impair item coverage of the construct and may not necessarily be reproducible in other samples (Clark & Watson, 2019). Trimmed item sets were then interrogated by a second set of ESEM and CFA models (Swami et al., 2023). This allowed us to identify a best-fitting empirical model in our discovery sample.

Higher-order factor model

For the parent form, although the lower-order empirical model did not perfectly match the theoretical model, there were enough similarities in the lower-order factors to justify testing whether these factors aligned with the proposed 3-factor higher-order model as well. We tested higher-order models using ESEM (Morin & Asparouhov, 2018) given that the generally poorer fit of CFA models at the lower-order level would necessarily result in a poor fit of the higher-order models using this approach. As described in the introduction, although there is general consensus on many aspects of the theoretical model, there are also a number of variations in how the exact higher-order theoretical model could be described, particularly in adolescence (e.g., whether behavioral scales are included or not, how Affiliation is modeled). The theoretical model guiding our interpretation of ESEM is depicted in Figure 1. As described in detail below, based on lower-order model results, no higher-order models were tested for the self-report form.

Figure 1.

Figure 1

Open in a new tab

depicts A) the items loading on each lower-order scale in the final model for the parent-rated EATQ-R. Crossloadings are shown in lighter grey. B) the higher-order structure of parent-rated temperament operationalized in the confirmatory models. Some aspects of the theoretical model are not fully settled. As depicted, Affiliation is a single, lower-order scale but whether it should be subsumed under Surgency or modeled as a stand-alone factor remains an area of theoretical debate. The Aggression and Depressed Mood scales are sometimes described as “behavioral” rather than temperament scales. C) the lower-order retained factors in the best-fitting final model. Effortful Control emerged as a single factor at the lower-order level. For Surgency and Negative Affect, we provide the lower-order factor loadings for the Michigan, Oregon, and ABCD samples, respectively. The lower-order factor structure replicated across all three datasets and bears strong, if not identical, resemblance to the theorized lower-order structure. The higher-order model is less well-supported given variation in strength of lower-order factor loadings (captured by dotted rather than solid lines from indictors to higher order factors).

Test & Generalizability Samples

In the test and generalizability samples, we first tested the lower-order, theoretical models using both ESEM and CFA approaches. Because we retained an alternate empirical solution in the discovery sample that differed from the original theoretical model, we also tested that empirical model in the test and generalizability samples via both CFA and ESEM approaches.

Reliability

Composite reliability for lower- and higher-order factors was evaluated in R under ICM-CFA assumptions using the laavan and semTools packages (Bell et al., 2024; Fu et al., 2022).

Measurement invariance

For our final retained models, we tested for configural, metric, and scalar measurement invariance between ADHD and typically-developing participants using the Multiple Group ESEM approach (MG-ESEM) and retained models according to suggested cut-offs for invariance testing (Chen, 2007). These are ΔCFI ≤ 0.01, ΔRMSEA ≤ 0.015, and ΔSRMR ≤ 0.03 between the configural and metric model and ΔCFI ≤ 0.01, ΔRMSEA ≤ 0.015, and ΔSRMR ≤ 0.15 between the metric and scalar models. We also tested for sample invariance across all three datasets and tested for language invariance in the case of the parent-rated form in ABCD given a subset of the parent-rated form was administered in Spanish.

Sensitivity Analyses

In supplemental analyses, we identified alternate models based on theory, our empirically derived solutions, and/or prior literature (Lawson et al., 2021; Snyder et al., 2015). Fit statistics and descriptions for all supplemental model tests are available in the supplemental material.

Transparency and Openness

Raw data is publicly available at the NIH Data Archive. Analyses scripts and assembled data sets for re-analysis are available by emailing the corresponding author. Exemplar scripts are also available in the supplement. These particular analyses were not preregistered.

Results

Parent Form: Discovery Sample (Michigan ADHD-1000)

Confirmatory Models

Table 2 contains model fit information for all statistical models. The confirmatory ESEM model of the theoretical lower-order factors was an overall modest fit to the data (see Table 2, M1 for model fit and Table S1 for item loadings). Three items from High-Intensity Pleasure formed a “travel” factor where the remaining items from that particular scale loaded on a separate factor. The items from Inhibitory Control did not load on any discernable factor except a single item that cross-loaded on factors resembling the Activation Control and Attention subscales. A separate factor included items from the original Aggression and Frustration scales. Another factor was characterized only by a cross-loaded friendship item from Affiliation (‘Would like to be able to spend time with a good friend every day’) and one factor contained no primary loadings. Although the theoretical model was a sufficient fit to the data statistically, it was unable to be retained due to these issues. As expected due to restricted cross-loading, the CFA model corresponding to the theoretical lower-order factor structure was a poorer fit to our data than the ESEM theoretical model (See Table 2, M2). The omega reliability estimates for many factors were below thresholds for empirical utility (see Table 3). Although confirmatory models, particularly ESEM, provided a modest fit, neither approach provided strong evidence for the exact theoretical model so we moved on to exploratory models.

Table 2.

Fit statistics exploratory structure equation and confirmatory factor analysis models of the EATQ-R

Sample Rater Model # Description χ2 BIC CFI TLI RMSEA RMSEA (90% CI) SRMR
MI Parent M1 ESEM LO Th. 2011.34 76,147 0.93 0.89 0.03 0.031–0.037 0.02
MI Parent M2 CFA LO Th. 3557.27 64,729 0.79 0.78 0.05 0.049–0.053 0.07
MI Parent M3 ESEM LO Emp. 1108.91 58,226 0.94 0.92 0.03 0.029–0.037 0.02
MI Parent M4 CFA LO Emp. 1358.89 41,822 0.88 0.87 0.04 0.048–0.052 0.07
MI Parent M5 HO ESEM NA Emp. 109.35 19,416 0.97 0.95 0.04 0.028–0.053 0.02
MI Parent M6 HO ESEM SU Emp. 175.26 19,170 0.93 0.90 0.06 0.049–0.071 0.07
OR Parent M7 ESEM LO Th. 2134.92 72,193 0.93 0.90 0.03 0.035–0.041 0.02
OR Parent M8 CFA LO Th. 3684.98 52,821 0.78 0.77 0.05 0.055–0.060 0.07
OR Parent M9 ESEM LO Emp. 898.97 46,476 0.95 0.92 0.04 0.036–0.045 0.02
OR Parent M10 CFA LO Emp. 1256.2 33,941 0.90 0.89 0.04 0.044–0.053 0.06
OR Parent M11 HO ESEM NA Emp. 144.23 17,826 0.95 0.92 0.05 0.042–0.065 0.03
OR Parent M12 HO ESEM SU Emp. 175.29 17,664 0.95 0.91 0.06 0.053–0.075 0.06
ABCD Parent M13 ESEM LO Th. 2134.92 72,193 0.93 0.90 0.03 0.035–0.041 0.02
ABCD Parent M14 CFA LO Th. 46842.24 1,759,096 0.75 0.74 0.05 0.049–0.050 0.06
ABCD Parent M15 ESEM LO Emp. 898.97 46,476 0.95 0.92 0.04 0.036–0.045 0.02
ABCD Parent M16 CFA LO Emp. 21261.86 1,134,362 0.81 0.80 0.05 0.052–0.053 0.06
ABCD Parent M17 HO ESEM NA Emp. 144.23 17,826 0.95 0.92 0.05 0.042–0.065 0.03
ABCD Parent M18 HO ESEM SU Emp. 175.29 17,664 0.95 0.91 0.06 0.053–0.075 0.06
MI Child M19 ESEM LO Th. 1992.98 143,336 0.94 0.90 0.02 0.023–0.028 0.02
MI Child M20 CFA LO Th. 5344.46 143,533 0.67 0.65 0.05 0.048–0.051 0.13
MI Child M21 9F ESEM LO Emp. 2345.39 142,776 0.92 0.90 0.02 0.025–0.030 0.02
OR Child M22 ESEM LO Th. 2104.63 68,582 0.90 0.85 0.03 0.035–0.041 0.02
OR Child M23 CFA LO Th. 4540.15 68,036 0.67 0.65 0.05 0.057–0.061 0.20
OR Child M24 9F ESEM LO Emp. 2397.71 68,100 0.89 0.85 0.03 0.036–0.042 0.02
Open in a new tab

Note: All χ2 models are statistically significant at p < .001. ESEM = Exploratory Structure Equation Model, CFA= Confirmatory Factor Analysis, LO = Lower Order, HO = Higher Order, SU = Surgency, NA = Negative Affect, Emp. = Empirical, Th. = Theoretical.

Table 3.

Reliability estimates (ω) for Original and Revised Parent Rated Factors

Original Factor Label MI-1000 OR-1000 ABCD Rev. Factor Label MI-1000 OR-1000 ABCD MI-1000 OR-1000 ABCD
ω ω ω ω ω ω r r r
Activation Control 0.89 0.88 0.84 Effortful Control 0.92 0.92 0.89 0.95 0.96 0.96
Attentional Focusing 0.83 0.85 0.73 Effortful Control 0.92 0.92 0.89 0.95 0.97 0.97
Inhibitory Control 0.66 0.73 0.56 Effortful Control 0.92 0.92 0.89 0.85 0.85 0.90
Shyness 0.86 0.89 0.84 Shyness 0.86 0.89 0.83 0.97 0.98 0.99
Affiliation 0.69 0.76 0.74 Affiliation 0.68 0.76 0.76 0.91 0.93 0.92
Depression 0.75 0.76 0.70 Depression 0.75 0.78 0.72 0.97 0.98 0.98
Fear 0.68 0.71 0.62 Fear 0.71 0.73 0.63 0.89 0.92 0.92
High-Intensity Pleasure 0.66 0.82 0.72 High-Intensity Pleasure 0.74 0.80 0.65 0.74 0.79 0.81
Frustration 0.76 0.82 0.79 Anger 0.80 0.83 0.87 0.87 0.89 0.90
Aggression 0.77 0.82 0.78 Anger 0.80 0.83 0.87 0.91 0.89 0.89
Open in a new tab

Note. Pearson’s r represents the factor score correlation between the correlated factors CFA of the theoretical scales and the restructured scales using lower-order ESEM. For all samples, the empirically derived Effortful Control factor is correlated with the Activation Control, Attentional Focusing, and Inhibitory control factors from the theoretical model. The empirically derived Anger score is correlated with the theoretically derived Frustration and Anger factors from which it draws items. For all other factors, the empirically derived factor is correlated with its matching theoretical factor (e.g., the theoretically derived Shyness factor is correlated with the empirically derived Shyness factor, etc). All correlations were statistically significant at p < 0.01.

Exploratory Models

We subjected items to exploratory factor analysis with parallel analysis over 1000 iterations. Results supported a seven-factor structure. Eigenvalues are available in the supplemental material. The first factor contained strong loadings from items representing Activation Control, Attention, and Inhibitory Control though the latter was only represented by a single item. We interpreted it as a composite of Effortful Control. The second factor contained items from the originally proposed Aggression and Frustration scales on a single factor. We interpreted this as an Anger factor given the breadth of Aggression, Irritability, and Frustration captured in the item content. The remaining five factors closely matched the theoretical model with primary loadings from High-Intensity Pleasure, Affiliation, Shyness, Depression, and Fear.

Inspecting the loadings revealed 16 candidate items that met our trimming criteria. These included most of the items from Inhibitory Control, which did not load on any discernable factor; a travel item from High-Intensity Pleasure; and two items about completing homework on time. Two Shyness items (‘I am shy’ and ‘I am not shy’) contained strongly correlated errors but were retained as their removal had a substantive impact on reliability. We removed the identified items and then respecified the seven-factor EFA model using the same parameters as before. In this model, the two remaining travel items from High-Intensity Pleasure loaded with items measuring Affiliation. We removed these two items along with two Aggression items and two Fear items that loaded poorly on all factors, according to our trimming criteria. None of these modifications changed our interpretation of the empirically-derived factors.

Given that all seven factors still contained primary loadings and had a good match to theory, we specified a seven-factor ESEM as a confirmatory model, which was a good fit to the data (see Table 2, M3 for model fit statistics and Table 3 for factor reliability scores; Table S3 contains the item loadings). The correlations between the empirically-derived factor scores and the scores from the corresponding theoretical factors were high (mean r = .90, range .74 – .97; see Table 3). The final retained model for the lower-order structure is shown in Figure 1.

The Higher-order Structure

We next specified two ESEM models for the higher-order factors: Negative Affect (Fear, Depression, and Anger) and Surgency (High Intensity Pleasure, Affiliation, and Shyness) (see Figure 1). (Effortful Control was already represented by a single composite factor at the lower-order level.) Model fit was sufficient for both models (See Table 2, M5 and M6), although both were just-identified because they had only three indicators. For the Surgency model, the loadings for High-Intensity Pleasure (λ = 0.71) and Affiliation (λ = 0.78) onto the higher-order factor were strong. The loading for Shyness was relatively weak in comparison (λ = 0.18). Reliability for the higher-order factor was very weak (ωhigher-order = 0.01). In the Negative Affect model, the loadings for Anger (λ = 0.99) and Depression (λ = 0.53) were strong while the loading for Fear was much weaker (λ = 0.14) in comparison. Reliability for the Negative Affect factor was stronger than the Surgency factor (ωhigher-order = 0.53), but still below what is typically acceptable for a factor to be viewed as unidimensional; unidimensionality could not be be further interogated statistically because factors are just-identified.

Parent Form: Test sample (Oregon ADHD-1000)

Theoretical Lower-order Model

In the Oregon ADHD-1000 dataset, ESEM found that the theoretical model was a good fit for the data (Table 2, M7). However, as in the discovery sample, items did not load on their theorized factors (Table S4). The CFA of the theoretical model was a poor fit to the data (Table 2, M8).

Empirical Lower-order Model

ESEM found that the empirical model was a good fit to the data (Table 2, M9) and, importantly, items loaded on the same factors as was observed in the discovery sample (Table S5) indicating reproducibility. The empirical factors showed strong reliability and correspondence with the theoretically-derived subscales, as was also true in the discovery sample (Table 3). The CFA model of the empirically-derived lower-order structure was an adequate fit to the data as well (Table 2, M10).

Higher-order Structure

The higher-order models for Surgency and Negative Affect were a both good fit to the data (Table 2, M11 and M12) but patterns of lower-order factor loadings differed somewhat from the discovery sample (see Figure 1). The loadings of the first-order factors on the Surgency model were strongest for Affliation (λ = 0.88) and Shyness (λ = −0.55) followed by a comparatively weaker loading for High-Intensity Pleasure (λ = 0.36). The reliability for the Surgency higher-order factor was again very weak (ωhigher-order = 0.15). For Negative Affect, the loadings onto the higher-order factor were strongest for Anger (λ = 0.94), followed by Depression (λ = 0.57) and Fear (λ = 0.41). The reliability for the Negative Affect higher-order factor was again better than the reliability found in the Surgency factor below a typically accepted threshold for unidimensionality (ωhigher-order = 0.64).

Parent Form: Generalizabilty in ABCD

Theoretical Lower-order Model

Here, we utilized the sample of participants who completed the English language form (N = 9,776). The theoretical lower-order ESEM model was an acceptable fit to the data statistically (Table 2, M13), but we noted that it again suffered from the identification of a travel testlet as the primary loadings on one of the factors, non-specific loadings for Inhibitory Control, and factors mirroring primary loadings of an already existing factor (see Table S6 for item loadings). The CFA for the theoretical model was a poor fit to the data (Table 2, M14), as observed in the first two samples.

Empirical Lower-order Model

ESEM found that the empirical lower-order model was a good fit in the generalizability sample (Table 2, M15). Primary item loadings were on similar factors as in the two other datasets but with noticeably lower loadings for items on the Fear and Depression factors. This was somewhat expected given that the EATQ-R was revised to include items measuring psychopathology, which would have a lower base rate in a large normative sample than samples enriched for psychopathology. Table S7 contains the item loadings for this model. The CFA of the lower-order empirical model derived in our discovery sample was a much poorer fit to the data than the ESEM model (Table 2, M16).

Higher-order Models

The higher-order models for Negative Affect and Surgency were a good fit to the data (Table 2, M17 and M18). Similar to what was seen in the Oregon ADHD-1000 sample, the loadings of the first-order factors on the Surgency model were strongest for Affliation (λ = 0.98) and Shyness (λ = −0.49) followed by a weaker loading for High-Intensity Pleasure (λ = 0.14). The reliability for the Surgency higher-order factor was again very weak (ωhigher-order = 0.06). For Negative Affect, the loadings onto the higher-order factor were strongest for Depression (λ = 0.99), followed by Anger (λ = 0.47) and Fear (λ = 0.37). The reliability for the Negative Affect higher-order factor was, once again, higher than for the Surgency factor, but below a typically accepted threshold for unidimensionality (ωhigher-order = 0.64). Figure 1 depicts the factor loadings in all three samples.

Parent Form: Measurement Invariance

We tested the measurement invariance of our empirically-derived lower-order model across 1) samples, 2) ADHD and typically-developing groups, and 3) English and Spanish forms by testing the increasingly restrictive configural, metric, and scalar models. There was strong invariance across all three samples. Similarly, strong invariance was observed across ADHD and typically-developing youth in all three samples. Finally, strong invariance was observed across English and Spanish language versions of the parent-rated EATQ-R in ABCD. The results of this analysis are in Table 4.

Table 4.

Results of Invariance Testing for Revised Lower Order Structure Models Across ADHD and Samples

Sample Rater Factor Model Test χ2 BIC CFI TLI RMSEA SRMR χ2DIFF ΔCFI ΔRMSEA ΔSRMR
All Parent Sample Configural 1643.89 97,965 0.956 0.93 0.036 0.02 -- -- -- --
Metric 1840.27 96,687 0.959 0.94 0.032 0.03 227.20 0.003 −0.003 −0.01
Scalar 1973.81 96,604 0.951 0.94 0.034 0.03 134.07 −0.008 0.002 0.00
MI Parent ADHD Configural 2127.76 59,732 0.891 0.84 0.044 0.03 -- -- -- --
Metric 2290.70 58,403 0.909 0.89 0.037 0.05 235.07 0.018 −0.007 0.02
Scalar 2354.20 58,225 0.905 0.88 0.038 0.05 64.41 −0.004 0.001 0.00
OR Parent ADHD Configural 1618.18 47,691 0.909 0.86 0.050 0.03 -- -- -- --
Metric 1786.97 46,609 0.919 0.90 0.043 0.05 226.24 0.01 −0.007 0.02
Scalar 1816.35 46,435 0.920 0.90 0.042 0.05 28.18 0.001 −0.001 0.00
ABCD Parent ADHD Configural 8395.61 1,218,451 0.948 0.92 0.034 0.01 -- -- -- --
Metric 6527.08 1,215,092 0.948 0.93 0.030 0.02 1156.82 0 −0.004 0.01
Scalar 9395.29 1,214,904 0.945 0.93 0.031 0.02 424.94 −0.003 0.001 0.00
ABCD Parent Language Configural 6658.29 1,116,486 0.952 0.92 0.032 0.01 -- -- -- --
Metric 7329.51 1,112,167 0.948 0.93 0.030 0.02 1559.86 −0.004 −0.002 0.01
Scalar 8199.44 1,116,076 0.941 0.92 0.032 0.02 903.816 −0.007 0.002 0.00
Open in a new tab

Note: All χ2 models are statistically significant at p < .001.

Given that strong invariance was supported, we compared latent means between parent ratings of typically-developing children (N = 10,305) and children meeting criteria for ADHD (N = 726) in a combined dataset. Two-tailed tests of the latent means revealed significant differences between the two groups on Effortful Control (z = − 1.69, p < 0.001), Anger (z = − 0.16, p < 0.001), Depression (z = 0.79, p < 0.001), Fear (z = 0.71, p < 0.001), and Shyness (z = − 0.16, p < 0.001). No latent mean differences were observed in High-Intensity Pleasure (z = 0.01, p = 0.92) or Affiliation (z = 0.05, p = 0.29).

Self-Report Form: Discovery sample (Michigan-1000)

Confirmatory Models

We conducted our investigation of the self-report form in a similar fashion to the parent-rated form. A confirmatory ESEM resulted in a good statistical fit (Table 2, M19). However, as was the case for the parent-rated form, several factors in the lower-order theoretical model did not contain any discernable primary loadings, and items from Activation Control, Attention, and Inhibitory Control loaded onto a single factor. In addition, items from Fear and Depression factors cross-loaded complexly across all theorized Negative Affect factors interpreting these factors difficult at the lower-order level. Two items that denoted life in a big city had no discernable primary loadings. The CFA for the self-report form did not fit the data (Table 2, M20). The reliability estimates for many of the factors were below the suggested thresholds for empirical use (see Table S12). Thus, as with the parent-rated form, neither confirmatory approach provided strong evidence for the theoretical model and so we moved on to exploratory models.

Exploratory Models

Exploratory factor analysis and parallel analysis iterated 1000 times suggested a 9-factor structure. Similar to what was observed in the parent form, items from Activation Control, Attention, and Inhibitory Control loaded primarily on a single factor, which we termed Effortful Control (six items; λ = 0.53 – 0.67). The remaining nine factors contained primary loadings from the other theorized scales although many of the items in this model either failed to load meaningfully on a factor (λ ≤ 0.40) or had a substantive cross-loading on a non-target factor (λ ≥ 0. 20). Applying our trimming criteria would have removed about half the items from the original form due to these problems, resulting in potentially low construct coverage per factor. Given this, we did not attempt to further refine the exploratory models nor did we attempt higher-order models. Instead, we retained the nine-factor lower-order model using all of the theorized items. Confirmatory ESEM indicated this model was an acceptable fit to our data (Table 2, M21).

Self-Report Form: Test Sample (Oregon ADHD-1000).

Theoretical Lower-order Model

The lower-order theoretical model was an acceptable fit to the data using an ESEM approach (Table 2, M22). However, it contained the same issues as in the discovery sample, including items that did not load on their representative factors and one factor containing no discernable primary loadings. The CFA model of the theoretical structure was a poor fit to the data (Table 2, M23).

Empirical Lower-order Model

The nine-factor lower-order ESEM model was a poor fit to the test sample (Table 2, M24). Examining the item loadings revealed considerable differences between the empirical structure identified in the discovery dataset and the test dataset, although loadings on Effortful Control and Aggression appeared similar to one another (Table S9). Overall, we determined that there was no fully reproducible factor structure for the entire self-report form.

Discussion

Temperament models play a crucial role in understanding psychopathology risk and resilience. They provide a theoretical link between emerging personality-based nosologies of mental health problems and the earlier developmental periods when many of these problems emerge (Krueger et al., 2021). However, studies of temperament and psychopathology are currently limited by inadequate measurement tools, and it is unclear whether findings using existing tools can be interpreted in relation to the theoretical models and constructs they are intended to capture. Here, results identified a theoretically-interpretable, shortened version of the parent-rated EATQ-R that is reproducible across three independent samples and offers strong measurement invariance across ADHD and typically-developing adolescents. Replicable structure and strong invariance allow us to identify temperament characteristics of ADHD across three of the largest cohorts with adolescent psychopathology data available. Parent-reported temperament may provide an important tool to further study psychopathology in this developmental period.

The current work identified a seven-factor lower-order model for the parent-rated EATQ-R that replicated across three independent samples (see Figure 1). The lower-order factor structure could be reproduced using both Spanish-language and English-language forms in ABCD and was invariant across samples and clinical groups. Despite widespread criticism of the self-report version of the EATQ-R (which we turn to in more detail below), the current results suggest a robust, reproducible structure for the parent-report EATQ-R form. Overal, it largely aligns with the theoretical model on which it is built, suggesting it can be meaningfully applied to studies of the lower-order temperament facets, including their studies of their relationships to psychopathology.

Yet several important differences from the originally proposed theoretical structure are also notable. First, as compared to the original model, we found that items theorized to load on separate Aggression and Frustration scales were cross-loaded and formed a single reliable Anger factor. The Anger factor was replicated across all three datasets. The items notably contained descriptions where frustration is displayed as either observable emotional expression or behaviors but not internal experiences of frustration, likely due to reliance on parent report. Although these scales are sometimes discussed as reflecting conceptually distinct temperament (Frustration) and “behavioral” (Aggression) domains, the current work is in line with recent conceptualizations that emphasize the overlap of traits and clinical symptoms.

Second, the subscales related to Effortful Control were distilled into a single latent factor raising questions about the meaningful separation of these facets of the theoretical construct. In the retained model, Inhibitory Control was captured by only a single item. Findings here mirror those from a previous study of a similar measure intended for younger children (Kozlowski et al., 2022), suggesting that this construct is not well-captured in existing parent temperament ratings of their children. It may be that additional items are needed to best capture the construct of Inhibitory Control. Alternatively, while facets of attentional and inhibitory control can be at least partially differentiated at the level of cognitive task performance (e.g., Smith et al., 2020), it may be that they are not easily discernable in behavioral ratings no matter which items are used. Nonetheless, we show that the shorter Effortful Control scale identified here has strong correspondence to all of the original scales that contribute items (including Inhibitory Control). It is reliable as a unitary construct and to a degree suggests clinical utility. The shorter scale may reduce participant burden while capturing much of the same information.

The remainder of the original lower-order factors were generally well-preserved from the originally proposed factor structure. Each lower-order factor was represented by at least four items with strong primary loadings in the parent report. Critically, however, results showed that minor cross-loaded structures fit the data better than models that do not allow cross-loading, suggesting the temperament structure is indeed more complicated than the single item-per-factor approach that is often applied. Parsimonious models may not always require a strict absence of cross-loadings, particularly in the case of relatively complex behavioral indicators that reflect the outcomes of multiple psychological influences (e.g., Marsh et al., 2014). Loosening these restraints allowed us to reproduce discrete factors that demonstrated strong correspondence with their original counterparts with only minor item-level loadings allocated to other factors.

For the parent-rated form, we also found strong measurement invariance in all three samples. As such, we were able to make comparisons on temperament traits with ADHD in the combined sample while accounting for measurement error. Unsurprisingly, parent ratings of Effortful Control in children meeting research diagnostic procedures for ADHD were substantively lower than those of typically-developing children supporting theoretical conceptions of its role in ADHD where children’s self-regulatory capacity appears to be diminished. In addition, children with ADHD experienced lower Anger and Shyness, as well as higher ratings of Depressive Mood and Fear. These findings partially converge with earlier literature suggesting a temperamental profile to ADHD characterized by low Effortful Control, higher Negative Emotionality, and higher Surgency; however, the presence of lower and not higher Anger is at odds with prior suggestions that ADHD is characterized by anger dysregulation (Shaw et al., 2014; Karalunas et al., 2019). One possibility is that specific items retained on the EATQ capture aspects of the Anger construct that are different from those on other measures. For example, the retained items capture primarily externally-oriented anger versus internally-experienced anger or frustration given the latter may be harder for parents to observe. Alternatively, the emerging temperament literature on ADHD emphasizes the wide heterogeneity in negative and positive affect dysregulation within this diagnostic group (Karalunas & Nigg, 2020). It may simply be that the combined sample here was characterized by relatively fewer ADHD children with the anger dysregulation profile. Additional sub-phenotyping work using these refined measures will be needed.

Nevertheless, this work sets the stage for better integrating neurodevelopmental disorders, such as ADHD, with developing personality-based nosologies. While some features of these disorders may be already reflected in existing spectra, others likely require the addition of new indicators or regrouping of existing indicators. Equally important, the structure of personality-based nosology may differ across developmental stages. For example, high surgency/extraversion may have different relevance in early versus later development (Figuracion et al., 2024) and associations between Negative Affect and core ADHD symptoms may either weaken or strengthen over time. Optimizing the measurement of temperament features during adolescence is a key step on the way to testing these possibilities.

While the lower-order model was reasonably well reproduced, results for the higher-order model were less convincing and require more caution in interpretation. Although higher-order factors with a good fit to the data could be obtained in all samples, the factors are just-identified (each had only 3 indicators). This means that the fit statistics reflect the fit of items to the lower-order factors; the true fit of the higher-order factors cannot be assessed. The higher-order factor models may still provide some support for the model if factor loadings are strong and in the expected directions. However, the evidence here was also mixed. For Surgency, factor loadings for at least one of the indicators were quite low in all samples, contributing to low reliability of the higher-order factor. For Negative Affect, factor loadings were more aligned with theory but still with several moderate loadings contributing to modest reliability estimates for the higher-order factor. Figure 1 summarizes the factor loadings for both Surgency and Negative Affect. Overall, the higher-order factors for Surgency and Negative Affect are less well-supported and interpretable than their lower-order indicators and, for this reason, we suggest that the best use of our revisions to the parent-rated form is to interpret the lower-order factors.

In contrast to the promising results of the parent-rated form, for the self-rated form, the empirically-derived structure in our discovery dataset did not generalize to our test dataset, similar to prior studies (Lawson et al., 2021; Latham et al., 2020). This is the now third large-scale report on the non-reproducibility of the English language adolescent form. Items measuring Effortful Control and Aggression had high primary loadings and low residual variances in both datasets, suggesting some specific scales may be reliably recovered. However, the interpretation of the remaining factors warrants serious caution given that even more liberal methods of confirmatory modeling could not recover them reliably. It seems unclear how useful self-report temperament will be with this particular form.

Limitations and Constraints on Generalizability

Limitations of the present effort here include a lack of a child-rated form in the large ABCD sample. A third sample may have added further empirical support to our chief argument for researchers to exercise caution in using the self-rated form. However, given the existing body of work on the self-rated form, we do not suspect we would be able to retain an empirical model in a third sample. Also, while we tested invariance in language and clinical groups, invariance across other demographic features may be informative.

In our modifications to the parent form, it was necessary to apply an item trimming threshold to recover partial factors. Though factor score correlations supported strong correspondence between these and the scores for the factors as theorized, items with lower loadings in the entire sample may have important meaning for population subgroups. This possibility is most evident in the primary loadings for the Fear factor, which were weaker in ABCD than in the two clinical samples.

This possibility is related to a broader point about construct validity. While the EATQ-R seems to correspond to the temperament model it was designed to measure (at least at the lower-order level), we were limited in construct coverage to the specific items already included in the measure. Additional items may be needed to adequately cover all relevant aspects of temperament and/or to enhance the construct validity of the existing scales. Similarly, there is no single, definitive theoretical model against which the EATQ-R should be tested. The theory is developing and is described with minor variations throughout the literature. That said, there is general agreement on major pieces (e.g., all models would expect Effortful Control, Surgency, and Negative Affect factors and all models would identify mostly overlapping lower-order indicators of those factors). Here, we selected a logical and practicably testable model to guide our CFAs and also our interpreation of ESEMs. Tests of other possible theoretical models are provided in the supplement for completeness, but we note that none provided better fit to the data and none would change our overall conclusions. The broad correspondence of the identified lower-order model in all three samples with theory provides assurance that these measures may prove useful for study of psychopathology and temperament development.

Conclusion

Overall, the strength of the present research lies in its support of the utility of the parent-rated form as a measurement tool for applying Rothbart’s adolscent model to study of development and psychopathololgy. Based on our findings and the prior literature, we do not recommend using the adolescent self-report form. The Effortful Control and Aggression subscales may be salvageable but would require additional study and may provide limited information in the absence of other reliable factors from the self-rated form. Differences in parent-rated temperament may inform clinical screening procedures for ADHD and or help link neurodevelopmental disorders to emerging personality-based nosologies.

Supplementary Material

supplemental material

Public Impact Statement:

This study provides evidence for the reliability and validity of a modified version of the parent-rated Early Adolescence Temperament Questionnaire-Revised, a commonly used instrument in developmental and psychopathology research. It discourages the use of the self-rated form, which does not meet common standards for validity.

Acknowledgments

This work was supported by NIH Grants R01MH59015, NIH R01 MH124824, and NIH R01MH120109. EATQ data were collected using the REDCAP platform under NIH grant UL1TR002369. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH. Raw data are publicly available at the NIH Data Archive (NDA). Analyses scripts and assembled data sets for re-analysis are available by emailing the corresponding author. These particular analyses were not preregistered.

Reference:

  1. Asparouhov T, & Muthén B (2009). Exploratory structural equation modeling. Structural Equation Modeling: A Multidisciplinary Journal, 16(3), 397–438. doi: 10.1080/10705510903008204 [DOI] [Google Scholar]
  2. Balle M, Fiol-Veny A, de la Torre-Luque A, Llabres J, & Bornas X (2022). Temperamental Change in Adolescence and Its Predictive Role on Anxious Symptomatology. Behavioral Sciences, 12(6). 10.3390/bs12060194 [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Bell SM, Chalmers RP, & Flora DB (2024). The impact of measurement model misspecification on coefficient omega estimates of composite reliability. Educational and Psychological Measurement, 84(1), 5–39. 10.1177/00131644231155804 [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Cattell RB (1973). Personality and mood by questionnaire. Jossey-Bass. [Google Scholar]
  5. Clark LA, & Watson D (2019). Constructing validity: New developments in creating objective measuring instruments. Psychological Assessment, 31(12), 1412–1427. 10.1037/pas0000626 [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Clark LA, & Watson D (2021). Temperament: Theory and research. In John OP & Robins RW (Eds.), Handbook of personality: Theory and research (pp. 145–175). The Guilford Press. [Google Scholar]
  7. Chen FF (2007). The sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling, 14(3), 464–504. 10.1080/10705510701301834 [DOI] [Google Scholar]
  8. Chess S, Thomas A, Rutter M, & Birch HG (1963). Interaction of temperament and environment in the production of behavioral disturbances in children. The American Journal of Psychiatry, 120, 142–148. 10.1176/ajp.120.2.142 [DOI] [PubMed] [Google Scholar]
  9. Cordova MM, Antovich DM, Ryabinin P, Neighbor C, Mooney MA, Dieckmann N…. Nigg JT (2022). Attention-Deficit/Hyperactivity Disorder: Restricted Phenotypes Prevalence, Comorbidity, and Polygenic Risk Sensitivity in the ABCD Baseline Cohort. Journal of the American Academy of Child and Adolescent Psychiatry, 61(10), 1273–1284. 10.1016/j.jaac.2022.03.030 [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Durdurak BB, Altaweel N, Upthegrove R, & Marwaha S (2022). Understanding the development of bipolar disorder and borderline personality disorder in young people: a meta-review of systematic reviews. Psychological Medicine, 52(16), 1–14. Advanced online publication. 10.1017/S0033291722003002 [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Di Lorenzo R, Balducci J, Poppi C, Arcolin E, Cutino A, Ferri P, … & Filippini T (2021). Children and adolescents with ADHD followed up to adulthood: A systematic review of long-term outcomes. Acta Neuropsychiatrica, 33(6), 283–298. [DOI] [PubMed] [Google Scholar]
  12. Evans DE, & Rothbart MK (2007). Developing a model for adult temperament. Journal of Research in Personality, 41(4), 868–888. 10.1016/j.jrp.2006.11.002 [DOI] [Google Scholar]
  13. Ellis LK, & Rothbart M (2001). Early Adolescent Temperament Questionnaire--Revised (EATQ-R). APA PsycTests. 10.1037/t07624-000 [DOI] [Google Scholar]
  14. Eysenck HJ (1967). The biological basis of personality. Thomas: Spring-field, Ill. [Google Scholar]
  15. Figuracion MT, Kozlowski MB, Macknyk KS, Heise MB, Pieper SM, Alperin BR., Morton HE, Nigg JT, & Karalunas SL (2024). The relationship between emotion dysregulation and error-related negativity in adolescents diagnosed with ADHD. Research on Child and Adolescent Psychopathology. 54(4), 605–620. 10.1007/s10802-023-01127-z [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Fu Y, Wen Z, & Wang Y (2022). A Comparison of Reliability Estimation Based on Confirmatory Factor Analysis and Exploratory Structural Equation Models. Educational and Psychological Measurement, 82(2), 205–224. 10.1177/00131644211008953 [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Gartstein MA, & Rothbart MK (2003). Studying infant temperament via the revised infant behavior questionnaire. Infant Behavior & Development, 26(1), 64–86. 10.1016/S0163-6383(02)00169-8 [DOI] [Google Scholar]
  18. Hopwood CJ, & Donnellan MB (2010). How should the internal structure of personality inventories be evaluated? Personality and Social Psychology Review, 14(3), 332–346. 10.1177/1088868310361240 [DOI] [PubMed] [Google Scholar]
  19. Jernigan TL, Brown SA, & Dowling GJ (2018). The Adolescent Brain Cognitive Development Study. Journal of Research on Adolescence: The Official Journal of the Society for Research on Adolescence, 28(1), 154–156. 10.1111/jora.12374 [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Karalunas SL, Fair D, Musser ED, Aykes K, Iyer SP, & Nigg JT (2014). Subtyping attention-deficit/hyperactivity disorder using temperament dimensions: Toward biologically based nosologic criteria. JAMA Psychiatry, 71(9), 1015–1024. 10.1001/jamapsychiatry.2014.763 [DOI] [PMC free article] [PubMed] [Google Scholar] [Retracted]
  21. Karalunas SL, Gustafsson HC, Fair D, Musser ED, & Nigg JT (2019). Do we need an irritable subtype of ADHD? Replication and extension of a promising temperament profile approach to ADHD subtyping. Psychological Assessment, 31(2), 236–247. 10.1037/pas0000664 [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Karalunas SL, & Nigg JT (2020). Heterogeneity and subtyping in attention-deficit/hyperactivity disorder—considerations for emerging research using person-centered computational approaches. Biological psychiatry, 88(1), 103–110. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Karalunas SL, Nigg JT, & Figuracion MT (2024). The intersection of cognition and emotion in developmental psychopathology. In Bell MA (Ed.), Child development at the intersection of emotion and cognition (2nd ed., pp. 259–286). American Psychological Association. 10.1037/0000406-013 [DOI] [Google Scholar]
  24. King KM, Lengua LJ, & Monahan KC (2013). Individual differences in the development of self-regulation during pre-adolescence: connections to context and adjustment. Journal of Abnormal Child Psychology, 41(1), 57–69. 10.1007/s10802-012-9665-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Kotelnikova Y, Olino TM, Klein DN, Kryski KR, & Hayden EP (2016). Higher- and lower-order factor analyses of the Children’s Behavior Questionnaire in early and middle childhood. Psychological Assessment, 28(1), 92–108. 10.1037/pas0000153 [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Kotelnikova Y, Olino TM, Klein DN, Mackrell SVM, & Hayden EP (2017). Higher and Lower Order Factor Analyses of the Temperament in Middle Childhood Questionnaire. Assessment, 24(8), 1050–1061. 10.1177/1073191116639376 [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Kozlowski MB, Antovich D, Karalunas SL, & Nigg JT (2022). Temperament in middle childhood questionnaire: New data on factor structure and applicability in a child clinical sample. Psychological Assessment, 34(12), 1081–1092. 10.1037/pas0001180 [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Krueger RF, Hobbs KA, Conway CC, Dick DM, Dretsch MN, Eaton NR, Forbes MK, … HiTOP Utility Workgroup (2021). Validity and utility of Hierarchical Taxonomy of Psychopathology (HiTOP): II. Externalizing superspectrum. World Psychiatry: Official Journal of the World Psychiatric Association (WPA), 20(2), 171–193. 10.1002/wps.20844 [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Latham MD, Dudgeon P, Yap MBH, Simmons JG, Byrne ML, Schwartz OS,… Allen NB (2020). Factor structure of the Early Adolescent Temperament Questionnaire-Revised. Assessment, 27(7), 1547–1561. 10.1177/1073191119831789 [DOI] [PubMed] [Google Scholar]
  30. Lawson KM, Atherton OE, & Robins RW (2021). The structure of adolescent temperament and associations with psychological functioning: A replication and extension of Snyder et al. (2015). Journal of Personality and Social Psychology, 121(5), e19–e39. 10.1037/pspp0000380 [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Lim S, & Jahng S (2019). Determining the number of factors using parallel analysis and its recent variants. Psychological Methods, 24(4), 452– 467. 10.1037/met0000230 [DOI] [PubMed] [Google Scholar]
  32. Marsh HW, Ellis LA, Parada RH, Richards G, & Heubeck BG (2005). A Short Version of the Self Description Questionnaire II: Operationalizing Criteria for Short-Form Evaluation With New Applications of Confirmatory Factor Analyses. Psychological Assessment, 17(1), 81–102. 10.1037/1040-3590.17.1.81 [DOI] [PubMed] [Google Scholar]
  33. Marsh HW, Morin AJ, Parker PD, & Kaur G (2014). Exploratory structural equation modeling: an integration of the best features of exploratory and confirmatory factor analysis. Annual Review of Clinical Psychology, 10, 85–110. 10.1146/annurev-clinpsy-032813-153700 [DOI] [PubMed] [Google Scholar]
  34. Martel MM, Elkins AR, Eng AG, Goh PK, Bansal PS, Smith-Thomas TE, …Nigg JT (2022). Longitudinal Temperament Pathways to ADHD Between Childhood and Adolescence. Research on Child and Adolescent Psychopathology, 50(8), 1055–1066. 10.1007/s10802-022-00902-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Martel MM, Markon K, & Smith GT (2017). Research Review: Multi-informant integration in child and adolescent psychopathology diagnosis. Journal of Child Psychology and Psychiatry, and Allied Disciplines, 58(2), 116–128. 10.1111/jcpp.12611 [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Miola A, Baldessarini RJ, Pinna M, & Tondo L (2021). Relationships of affective temperament ratings to diagnosis and morbidity measures in major affective disorders. European psychiatry: The Journal of the Association of European Psychiatrists, 64(1), e74. 10.1192/j.eurpsy.2021.2252 [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Morin AJS, & Asparouhov T (2018). Estimation of a hierarchical Exploratory Structural Equation Model (ESEM) using ESEM-within-CFA. Montreal, QC: Substantive Methodological Synergy Research Laboratory. [Google Scholar]
  38. Muris P, & Meesters C (2009). Reactive and regulative temperament in youths: Psychometric evaluation of the Early Adolescent Temperament Questionnaire-Revised. Journal of Psychopathology and Behavioral Assessment, 31(1), 7–19. 10.1007/s10862-008-9089-x [DOI] [Google Scholar]
  39. Muthén LK, & Muthén BO (1998–2017). Mplus User’s Guide. (8th ed.). Los Angeles, CA: Muthén & Muthén. [Google Scholar]
  40. Nigg JT, Karalunas SL, Gustafsson HC, Bhatt P, Ryabinin P, Mooney MA, Faraone SV, Fair DA, & Wilmot B (2020a). Evaluating chronic emotional dysregulation and irritability in relation to ADHD and depression genetic risk in children with ADHD. Journal of Child Psychology and Psychiatry, and Allied Disciplines, 61(2), 205–214. 10.1111/jcpp.13132 [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Nigg JT, Sibley MH, Thapar A, & Karalunas SL (2020b). Development of ADHD: Etiology, Heterogeneity, and Early Life Course. Annual Review of Developmental Psychology, 2(1), 559–583. 10.1146/annurev-devpsych-060320-093413 [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Nigg JT, Karalunas SL, Mooney MA, Wilmot B, Nikolas MA, Martel MM,…Fair DA (2023). The Oregon ADHD-1000: A new longitudinal data resource enriched for clinical cases and multiple levels of analysis. Developmental Cognitive Neuroscience, 60, 101222. 10.1016/j.dcn.2023.101222 [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Peterson ER, Mohal J, Waldie KE, Reese E, Atatoa Carr PE, Grant CC, & Morton SMB (2017). A Cross-Cultural Analysis of the Infant Behavior Questionnaire Very Short Form: An Item Response Theory Analysis of Infant Temperament in New Zealand. Journal of Personality Assessment, 99(6), 574–584. 10.1080/00223891.2017.1288128 [DOI] [PubMed] [Google Scholar]
  44. Rothbart MK (1981). Measurement of temperament in infancy. Child Development, 569–578. 10.2307/1129176 [DOI] [Google Scholar]
  45. Rothbart MK (2007). Temperament, Development, and Personality. Current Directions in Psychological Science, 16(4), 207–212. 10.1111/j.1467-8721.2007.00505.x [DOI] [Google Scholar]
  46. Rothbart MK (2011). Becoming who we are: Temperament and personality in development. New York: Guilford Press. [Google Scholar]
  47. Rothbart MK, Ahadi SA, & Evans DE (2000). Temperament and personality: Origins and outcomes. Journal of Personality and Social Psychology, 78(1), 122–135. 10.1037/0022-3514.78.1.122 [DOI] [PubMed] [Google Scholar]
  48. Rothbart MK, & Mauro JA (1990). Questionnaire approaches to the study of infant temperament. In Colombo J & Fagen JW (Eds.), Individual differences in infancy: Reliability, stability, prediction (pp. 411–429). Lawrence Erlbaum Associates, Inc. [Google Scholar]
  49. Shaw P, Stringaris A, Nigg J, & Leibenluft E (2014). Emotion dysregulation in attention deficit hyperactivity disorder. The American Journal of Psychiatry, 171(3), 276–293. 10.1176/appi.ajp.2013.13070966 [DOI] [PMC free article] [PubMed] [Google Scholar]
  50. Shiner RL, Soto CJ, & De Fruyt F (2021). Personality assessment of children and adolescents. Annual Review of Developmental Psychology, 3(1), 113–137. 10.1146/annurev-devpsych-050620-114343 [DOI] [Google Scholar]
  51. Shiner RL, & DeYoung CG (2013). The structure of temperament and personality traits: A developmental perspective. The Oxford handbook of developmental psychology, 2, 113–141. [Google Scholar]
  52. Snyder HR, Gulley LD, Bijttebier P, Hartman CA, Oldehinkel AJ, Mezulis A…Hankin, B. L. (2015). Adolescent emotionality and effortful control: Core latent constructs and links to psychopathology and functioning. Journal of Personality and Social Psychology, 109(6), 1132–1149. 10.1037/pspp0000047 [DOI] [PMC free article] [PubMed] [Google Scholar]
  53. Swami V, Maïano C, & Morin AJ (2023). A guide to exploratory structural equation modeling (ESEM) and bifactor-ESEM in body image research. Body Image, 47, 101641. 10.1016/j.bodyim.2023.101641 [DOI] [PubMed] [Google Scholar]
  54. Tackett JL, & Durbin CE (2017). Advances in research on youth personality introduction to the special issue. Journal of Research in Personality, 67, 1–2. 10.1016/j.jrp.2017.03.004 [DOI] [Google Scholar]
  55. Tierney N & Cook D (2023). “Expanding Tidy Data Principles to Facilitate Missing Data Exploration, Visualization and Assessment of Imputations.” Journal of Statistical Software, 105(7), 1–31. doi: 10.18637/jss.v105.i07.36798141 [DOI] [Google Scholar]
  56. van der Ende J, Verhulst FC, & Tiemeier H (2012). Agreement of informants on emotional and behavioral problems from childhood to adulthood. Psychological Assessment, 24(2), 293–300. 10.1037/a0025500 [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

supplemental material
NIHMS2062943-supplement-supplemental_material.docx (243.3KB, docx)

RESOURCES