Evaluating the construct validity of adult ADHD and SCT among college students: A multitrait-multimethod analysis of convergent and discriminant validity

Abstract

Objective

To advance our understanding of adult attention-deficit/hyperactivity disorder (ADHD) and sluggish cognitive tempo (SCT), the present study investigates their construct validity by exploring the nature of trait- and method-related variance in self and parent ratings of ADHD and SCT.

Method

Using a multitrait-multimethod (MTMM) design, response variance in college undergraduates’ (n = 3,925) and a subset of their parents’ (n = 2,242) ratings was decomposed into method, trait, and error-specific variance.

Results

Global evidence for convergent and discriminant validity was supported, but parameter-level comparisons suggest that method effects, situational specificity, and ADHD’s core feature – inattention – are prominent.

Conclusion

This investigation offers two important conclusions: (a) SCT appears to be a related but separate factor from ADHD; and (b) self- and parent-ratings of emerging adult ADHD exhibit low to moderate correlations and support the situational specificity hypothesis, suggesting that multiple raters should be consulted when assessing adult ADHD. Implications of these findings and recommendations for the continued study of SCT are discussed.

Despite strong support for the validity of the overall diagnosis of ADHD (Willcutt et al., 2012), the best way to describe heterogeneity among individuals with ADHD remains unclear. Almost three decades ago, factor analytic work (Neeper & Lahey, 1986; see Becker, 2013 for review) established a 5-item sluggish cognitive tempo (SCT) factor that consistently distinguished between DSM-III attention deficit disorder (ADD) with and without hyperactivity (Lahey et al., 1988; Stanford & Hynd, 1994). Two SCT items – “often daydreams when should be attending” and “often is sluggish or drowsy” – were subsequently included in the initial DSM-IV ADHD field trials, but were removed due to poor negative predictive power despite demonstrating the highest positive predictive power among the inattention symptoms (Frick, Lahey, Applegate, & Kerdyck, 1994).

Validity of SCT in children and adolescents

Factor analyses

During the two decades since the DSM-IV was published, the number of studies that either mention or focus on SCT has increased dramatically (Becker, 2013). Over a dozen studies have examined the factor structure of ADHD and SCT in samples of children and adolescents. Although the number and wording of the specific SCT items varied across studies, the majority of studies conducted to date found a factor composed of putative SCT items that was separate from the inattention and hyperactivity-impulsivity symptoms described in DSM-IV (see Willcutt et al., 2012 for review; Barkley, 2012; Bauermeister, Barkley, Bauermeister, Martínez, & McBurnett, 2012; Becker, Fite, et al., 2013; Becker, Langberg, Luebbe, Dvorsky, & Flannery, 2013; Jacobson & Murphy-Bowman, 2012). SCT ratings were highly correlated with inattention but only moderately correlated with hyperactivity-impulsivity. Further, one study found that the relationship between SCT and hyperactivity became negative after controlling for the significant correlation between inattention and SCT (Penny, Waschbusch, Klein, Corkum, & Eskes, 2009). Taken together, these results suggest that SCT is separable from ADHD in children and adolescents, and is more strongly associated with inattention than hyperactivity-impulsivity.

External validity

After initial factor analyses demonstrated that SCT symptoms are separable from ADHD, several studies have begun to examine the external correlates of SCT. These studies indicate that symptoms of SCT are associated with multiple aspects of functional impairment and elevated rates of internalizing symptoms, even after controlling for symptoms of ADHD and other psychopathology and other relevant demographic characteristics (Bauermeister et al., 2012; Becker & Langberg, 2013a; Becker et al., 2013b; Willcutt et al., 2014), Therefore, although these findings are tempered by small and variable pools of SCT items and few studies that compare results across more than one rater, these studies provide strong support for the existence and separability of SCT in children and adolescents.

ADHD and SCT in adults

Adult ADHD accounts for roughly 76% of the disorder’s annual US economic burden of $143 to $266 billion (Doshi et al., 2012). A growing literature supports the validity of the inattention and hyperactivity-impulsivity symptom dimensions in adult samples (Willcutt et al., 2012), but only three studies have examined SCT in adults, and the measures of SCT in these studies were limited to self-report ratings (Barkley, 2012; Becker et al., 2013b; Flannery, Becker, & Luebbe, 2014). These initial studies indicate that SCT symptoms are separable from ADHD and are associated with significant self-organization, problem solving, occupational, academic, and social impairment when symptoms of ADHD are controlled (Barkley, 2012; Becker, et al., 2013b; Flannery et al., 2014), suggesting that further research is needed to clarify the relation between ADHD and SCT in adults.

The current study

The current study addresses these gaps in the literature by administering measures of ADHD symptoms and a large set of potential SCT items to a large sample of young adults. In addition to self-report ratings by the young adults, parent ratings were also obtained for a subset of participants, providing an estimate of inter-rater agreement and facilitating a direct comparison of the factor structure of the two sets of ratings. It is also important to note that the aforementioned correlations between SCT and ADHD reflect combined trait and method variance, and thus may overestimate the true relationship between these constructs. The main objectives were as follows:

1. To clarify the relation between SCT and ADHD in young adults, confirmatory factor analyses were first fit separately to parent and self-report ratings. We hypothesized that for both parent and self-report ratings the best-fitting solution would include separate SCT, inattention, and hyperactivity-impulsivity factors, consistent with earlier studies of children. We further hypothesized that the inter-factor correlations would replicate those of previous studies.

2. After testing the adequacy of a three-factor solution, the relations between parent and self-report ratings of ADHD and SCT were examined using a full MTMM framework. These analyses provided estimates of inter-rater agreement and a direct test of whether the factor structure can be equated across raters. We hypothesized that self and parent ratings would show moderate agreement for all symptom dimensions, providing key support for the internal validity of ADHD and SCT in young adults. However, we also anticipated that these analyses would reveal significant rater effects, suggesting that some variance is unique to each rater.

Given these objectives and hypotheses, ADHD and SCT symptom ratings were subjected to the following analytic plan: (a) separate comparisons of one-, two-, and three-factor solutions for self- and parent-ratings; (b) comparison of childhood to current ratings; (c) investigation of interrater agreement among all items and items belonging to the separate factors; (d) evaluation of matrix-level convergent and discriminant validity; and (e) evaluation of item-level convergent and discriminant validity.

Method

In order to diagnose adult ADHD, Barkley (2006) recommends the use of two raters, data from both childhood and present, and the DSM-IV’s 18-item checklist for ADHD. The current study extends these best practices into studying the construct validity (i.e., convergent and discriminant validity) of SCT and ADHD in college undergraduates using self and parent ratings of current and childhood behavior. Because ratings based on the past six months are likely to be more reliable than retrospective memory of ages 7 to 12, and because Barkley (2013) suggests that SCT symptoms may manifest in later adolescence, only current ratings (i.e., from the past six months) were used in the present study.

Participants

In exchange for course credit, an unselected sample of 3,925 undergraduates (35.6% male) ranging in age from 18.0 to 41.7 years old (M = 19.5, SD = 1.5, <1% sample over 25.0 years) completed a battery of self-report rating scales that included the Self-Report form of the ADHD Current and Childhood Symptoms Scales (Barkley & Murphy, 1998). Permission was also requested to send the Other Report version of the Current and Childhood Symptom Scales (Barkley & Murphy, 1998) to the participant’s parent or other primary caregiver during childhood, and approximately 72% of the participants agreed. Common reasons for refusal included privacy concerns, absence of contact with/availability of caregiver, and caregiver’s insufficient English proficiency. Parents (N = 2,142, approximately 76% return rate) voluntarily completed and returned the aforementioned questionnaire. The sample of 2,142 participants with both self-report and parent-report ratings was randomly split into two groups of equal size to provide a within-study replication of the MTMM model, and the remaining sample of 1,783 participants for whom parent ratings were not available was used as a second replication sample for self-ratings. The split-half groups did not differ in sex or age (p = 0.32), but the self-ratings replication sample differed from these groups in age (p < .05; d < .10) due to the large sample size; no other significant differences were noted.

Measures

DSM-IV ADHD symptoms

The ADHD Current and Childhood Symptom Scales (Barkley & Murphy, 1998) were used to obtain self-ratings of the 18 symptoms of DSM-IV ADHD. On the Current Symptom Scale the participant is asked to indicate how often in the last 6 months each of the 18 DSM-IV ADHD symptoms is true on a 4-point Likert scale (never or rarely, sometimes, often, and very often). Parent/caregiver ratings were obtained in a parallel format using the Other Report version of the Current Symptom Scales. Self and parent ratings of inattention and hyperactivity displayed excellent internal consistency (α_self = .95 and .91, respectively; α_parent = .93 and .95, respectively).

Sluggish cognitive tempo items

Ten potential SCT items (sluggish, slow to respond; seems not to hear, needs things repeated; seems to be “in a fog”; drowsy or sleepy; easily confused; daydreams, stares into space, or gets lost in his/her thoughts; absentminded, forgets things easily; seems to be unaware of her/his surroundings (for example, doesn’t notice wet paint or a dangerous situation); apathetic/seems to lack motivation; underactive, slow moving, or lacks energy) were developed based on theoretical models of SCT and items used in previous studies of ADHD and SCT (Carlson & Mann, 2002; McBurnett, Pfiffner, & Frick, 2001; Penny et al., 2009). Each item was added to the ADHD rating scale and administered in the same format, although the last two items were omitted from the self-report questionnaire. Self and parents SCT ratings displayed good to excellent internal consistency (α_self = .89; α_parent = .94).

Data analyses

Confirmatory factor analyses of parent and self-report ratings

Structural and measurement model analyses were conducted using EQS 6.1 (Bentler, 2008). The initial CFAs included estimation of each hypothesized latent construct predicting its respective indicators, with latent factor variances set to 1.0 in order to obtain an over-identified model. Noting that the exclusive reliance upon parent reports can overestimate the associations between psychopathology and other correlates (Garner, Marceaux, Mrug, Patterson, & Hodgens, 2010), as well as inter-trait correlations, data were analyzed separately for the emerging adults and their parents prior to estimation of the full MTMM model. This was done in order to compare the fit of one- vs. two- vs. three-factor solutions for self and parent ratings.

In order to investigate item-level interrater agreement for two raters, two-way mixed, absolute interclass correlations, which are equivalent to a weighted κ with quadratic weights, were calculated to take into account the ordinality of these data (Norman & Streiner, 2007).

MTMM design

One advantage of the MTMM design is that method variance is eliminated from latent trait variables, as opposed to manifest variables, thereby providing more accurate estimates of relationships among traits. In Campbell and Fiske’s (1959) conceptualization, this method allows the investigation and extraction of trait and method variance from observed scores, which are inherently a composite of trait variance, method variance, and measurement error. Thus, this type of analysis also permits the exploration of method effects – biases that can occur when the same method is used to assess different traits (e.g., halo effects, response bias, etc.) – but prominent source effects can also reflect situation-specific differences in the behavior being rated (Gomez, Burns, Walsh, & Hafetz, 2005). Convergent and discriminant validity are then supported if all trait variances are significant and greater than the method variance. If the questionnaire fails to exhibit these properties, this could mean that the rating scale is a poor measure of the ADHD and SCT traits. Alternatively, high levels of method variance could indicate that parent and self-report ratings are valid within situations, but reflect true differences in behavior across settings (Gomez et al., 2005; Gomez, 2007). The MTMM design can therefore advance the SCT and ADHD literature by modeling inter-trait correlations, the validity of these constructs, and the presence of method effects.

Extending Campbell and Fiske’s seminal work, Widaman (1985) developed a means by which convergent and discriminant validity can be investigated using nested model comparisons with the hypothesized correlated trait-correlated method model. Convergent validity reflects the extent to which different assessment methods (i.e., different raters or data collection methods) agree in their measurement of the same trait(s), whereas discriminant validity is the extent to which assessment methods diverge in their measurement of different traits.

Utilizing the MTMM design, the discriminant and convergent validity of SCT were evaluated via nested model comparisons. The hypothesized model (Model 1) allows the correlations between traits and raters (i.e., methods) to vary freely, whereas Model 2 omits the trait factors and only allows the interrater correlation to vary. A comparison of Model 1 and Model 2 will thus provide insight into the convergent validity of the proposed traits. Model 3 again allows the correlation between raters to vary, but the traits are included and set to be perfectly correlated with one another (i.e., a single trait model). Model 4 allows the correlations between traits to vary freely while the method factors are specified as uncorrelated. Comparison of Model 1 (multi-trait/multi-method) vs. Model 3 (mono-trait/multi-method) evaluates the discriminant validity of the specified traits, whereas Model 1 vs. Model 4 evaluates the presence of method effects due to systematic differences in how raters have completed the questionnaire.

Preliminary analyses revealed significant deviation from normality and thus robust maximum likelihood estimation was used (Satorra & Bentler, 1994). Given that ratings of a single item were only missing from four participants, the pairwise-present method had little effect on any parameter estimates. Lastly, because the items were rated on a 4-point Likert scale, the data was specified to be categorical during the estimation procedure (Lee, Poon, & Bentler, 1992).

Poor-fitting models are typically due to one of the following model misspecifications: 1) too few or too many factors; 2) poor item selection; 3) incorrect assignment of items to latent factors; and 4) correlated errors between items (Brown, 2006). Given these obstacles, estimation of a proper solution in CFA models of MTMM data can be regarded as a rare find (Kenny & Kashy, 1992; Marsh, 1989). Although some of the covariance between items is explained by the latent factor, remaining covariance is likely due to another common cause, such as similarly worded items or method effects (Brown, 2006). The MTMM design is unique in its ability to systematically control for method effects, suggesting that correlated errors, if necessary, be made between similarly worded or otherwise related items.

All study procedures were fully approved by the Institutional Review Boards of the University of Colorado Boulder. All participants read and agreed to the informed consent document prior to enrollment in the study.

Results

Individual CFAs for each of the constructs provided adequate to excellent fit to the data for both self and parent ratings (see Table 1). Furthermore, the three-factor solutions demonstrated smaller AIC values, and thereby superior fit, for both self and parent ratings. Having established the superiority of a three-factor solution, the full MTMM model was estimated (Table 2). The hypothesized model displayed a close fit to the present data (CFI = .99; RMSEA = .037; 90% CI = [.036, .039]).

Table 1.

Fit Indices for Measurement and One- vs. Two- vs. Three-factor Solutions of Self and Parent Ratings

Measure	SB χ2	df	CFI	RMSEA	90% CI	AIC	Mean Loadings (SD)
Self
ADHD-I	144.8	26	.97	.065	[.055, .076]	–	.64 (.11)
ADHD-HI	92.5	27	.99	.048	[.037, .058]	–	.72 (.06)
SCT	100.8	20	.99	.061	[.050, .073]	–	.70 (.13)
Parent
ADHD-I	122.0	26	.99	.066	[.057, .069]	–	.78 (.04)
ADHD-HI	116.4	27	.99	.056	[.045, .056]	–	.83 (.06)
SCT	195.9	35	.99	.066	[.057, .075]	–	.80 (.06)
Self
One-factor	1807.6	299	.97	.069	[.066, .072]	1209.6	–
Two-factor	1110.7	272	.98	.054	[.050, .057]	566.7	–
Three-factor	774.3	244	.99	.045	[.041, .049]	286.3	–
Parent
One-factor	3090.9	350	.97	.086	[.083, .088]	2390.9	–
Two-factor	1413.8	321	.99	.056	[.053, .059]	771.8	–
Three-factor	1009.6	291	.99	.048	[.045, .051]	427.6	–

Note. S-B χ²= Satorra-Bentler corrected chi-square statistic; CFI = comparative fit index; RMSEA = root mean square error of approximation; CI = confidence interval; AIC = Aikaike information criterion; ADHD-I = ADHD-predominantly inattentive type; ADHD-HI = ADHD-predominantly hyperactive/impulsive type; SCT = sluggish cognitive tempo.

Table 2.

Summary of Goodness-of-Fit Indexes for Full MTMM Models

Model		S-B χ2	df	CFI	RMSEA	90% CI
1	Freely correlated traits; freely correlated methods	3280.4	1319	0.99	0.037	[.036, .039]
2	No traits; freely correlated methods	6539.1	1376	0.96	0.059	[.058, .061]
3	Perfectly correlated traits; freely correlated methods	3776.6	1322	0.98	0.042	[.040, .043]
4	Freely correlated traits; uncorrelated methods	3212.7	1320	0.99	0.037	[.035, .038]

Note: MTMM = multitrait-multimethod; S-B χ² = Satorra-Bentler corrected chi-square statistic; CFI = comparative fit index; RMSEA = root-mean-square error of approximation; CI = confidence interval.

Childhood vs. Current Ratings

As noted above, only current ratings of ADHD and SCT symptoms were used for the MTMM analyses. To provide some insight into the relationship between ratings of current and childhood symptomatology, zero-order correlations of total inattention, hyperactivity-impulsivity, and SCT ratings were calculated. The correlations between ratings of current and childhood symptomatology, presented as self followed by parent ratings, were as follows: inattention (.68, .77); hyperactivity-impulsivity (.71, .76); and SCT (.62, .76), all p < .001.

Interrater Agreement

The absolute interclass correlations (i.e., weighted κ estimates) for the entire sample suggest fair to moderate agreement (M = .34, SD = .10). The zero-order correlations between summary factor scores (i.e., total of 0–3 ratings for each dimension) for self and parent ratings of inattention, hyperactivity, and SCT are as follows: r = .48, r = .41, and r = .33, respectively, all p < .001. Lastly, interrater correlations between the method factors in the full model were r = .49 and r = .49 in the replication sample, suggesting both shared and unique effects (Table 3).

Table 3.

Latent Trait Intercorrelations of Each Model

Model	Inatt-Hyp/Imp	Inatt-SCT	Hyp/Imp-SCT	Parent-Self
1	.16+	.66	.02b	.49
Replic.	.14b	.73	−.07b	.49
4	.83	.92	.74	–

Note. Significant at p < .0001, unless otherwise noted. n = 1,071 for Models 1 and 4; n = 1,783 for replication sample. Inatt = inattention; Hyp/Imp = hyperactivity/impulsivity; SCT = sluggish cognitive tempo; Replic. = replication sample.

^aNot significant.

⁺p = .053.

Global assessment of convergent validity

Using Widaman’s (1985) method, convergent validity is supported by a significant χ² difference test (in this case, a Satorra-Bentler scaled χ² difference test) between a model in which the traits (i.e., inattention, hyperactivity, SCT) are specified (Model 1) and one in which they are not (Model 2). As shown in Table 4, the ΔS-B χ² (change in Satorra-Bentler corrected chi-sqaure statistic) was highly significant and the difference in practical fit was larger than Cheung and Rensvold’s (2002) recommended threshold of .01, thereby arguing for global evidence of convergent validity. Self- and parent-raters generally agree in their ratings of these traits, at least at the correlation and covariance matrix level.

Table 4.

Differential Goodness-of-Fit Indexes for MTMM Nested Model Comparisons

Model Comparisons	Difference in
	S-B χ2	df	CFI
Test of Convergent Validity
Model 1 vs. Model 2 (traits)	903.0a	57	0.02
Test of Discriminant Validity
Model 1 vs. Model 3 (traits)	1166.9a	3	0.01
Model 1 vs. Model 4 (methods)	1.6b	1	0.001

Note: MTMM = multitrait-multimethod; S-B χ² = Satorra-Bentler corrected chi-square statistic; CFI = comparative fit index.

^ap < 0.001.

^bp = 0.21.

Global assessment of discriminant validity

As described above, Widaman’s method assesses the presence of discriminant validity using two nested model comparisons. The first comparison, which assesses the extent to which independent measures of different traits are correlated, is conducted by comparing Model 1 with one in which the traits are perfectly correlated (Model 3). Complete trait discriminant validity suggests the presence of negligible inter-trait correlations. Discriminant validity of trait ratings is supported in this case (see Table 4).

Because the same method was used to gather data from emerging adults and their parents, and because raters may approach the same questionnaire differently, the next test examines the presence of method or source effects (see MTMM design section above for further explanation). This is accomplished via a comparison of Model 1 with a model in which method factors are uncorrelated (Model 4). In this comparison, a large ΔS-B χ² or ΔCFI supports a lack of discriminant validity. As shown in Table 4, a nonsignificant ΔS-B χ² and small ΔCFI indicate the absence of method effects, at least at the matrix level. As noted above, the previous evidence for convergent and discriminant validity only represents a global assessment. To obtain more precise measures of trait- and method-related variance, we now discuss parameter-level estimates.

Parameter-level assessment of convergent validity

To investigate the variance explained by traits and methods, Widaman (1985) recommends squaring their respective factor loadings in order to assess the percentage of variance explained. The sum of these percentages and the squared error term equals 1.00 within rounding error. Variances explained by traits, methods, and error for the hypothesized model (Model 1) are presented in Table 5.

Table 5.

Variance Explained by Trait, Source, and Error for ADHD-I, ADHD-H, and SCT Items

Item	Self Ratings			Parent Ratings
	Trait	Source	Error	Trait	Source	Error
I 1	.00ns	.40	.59	.18	.55	.27
I 2	.00ns	.58	.42	.09	.70	.21
I 3	.01ns	.48	.51	.03	.58	.40
I 4	.07	.46	.47	.23	.56	.20
I 5	.08	.38	.54	.28	.51	.21
I 6	.01ns	.48	.52	.10	.56	.34
I 7	.07	.45	.48	.27	.43	.30
I 8	.00ns	.62	.38	.06	.76	.19
I 9	.05	.64	.31	.24	.51	.25
	(M = .03)	(M = .50)	(M = .47)	(M = .16)	(M = .57)	(M = .26)
H 1	.01	.40	.59	.02	.55	.43
H 2	.00ns	.43	.57	.02	.67	.31
H 3	.02	.51	.47	.02	.69	.30
H 4	.04	.37	.59	.09	.56	.35
H 5	.07	.10	.83	.07	.44	.49
H 6	.26	.18	.56	.31	.32	.37
H 7	.15	.25	.60	.24	.45	.31
H 8	.07	.50	.43	.16	.54	.30
H 9	.23	.34	.43	.28	.42	.30
	(M = .10)	(M = .34)	(M = .56)	(M = .13)	(M = .52)	(M = .35)
SCT 1	.05	.37	.57	.38	.41	.21
SCT 2	.01	.50	.48	.14	.58	.28
SCT 3	.03	.52	.45	.26	.56	.19
SCT 4	.04	.28	.69	.38	.32	.31
SCT 5	.01	.49	.50	.14	.54	.32
SCT 6	.00ns	.54	.46	.15	.50	.35
SCT 7	.04	.63	.33	.12	.57	.31
SCT 8	.02	.48	.51	.10	.43	.48
SCT 9	–	–	–	.26	.39	.35
SCT 10	–	–	–	.50	.29	.21
	(M = .03)	(M = .48)	(M = .50)	(M = .24)	(M = .46)	(M = .30)

Note. All standardized loadings significant at p < .05, unless bold (p < .001) or indicated as nonsignificant (ns). Trait, source, and error variances for each item sum to 1.0 within rounding error. SCT items 9 & 10 not rated by self. I = inattention; HI = hyperactivity-impulsivity; SCT = sluggish cognitive tempo.

Evidence of convergent validity is assessed via the magnitude of trait loadings, and in this case, the percentage of total variance explained by the trait. Except for self ratings of “talks excessively” (.26) and “interrupts or intrudes on others” (.23), the proportion of trait variance (i.e., for ADHD-I, ADHD-HI, and SCT) was small to minimal, and source variance was greater than trait variance for all items except “talks excessively”. The proportion of parent-rated trait variance, however, was larger in comparison that of self ratings, suggesting that parent ratings may be more valid than self ratings from college-aged adults. The mean variance explained by parent ratings compared to that of self ratings was as follows: ADHD-I (.16 versus .03), ADHD-HI (.13 versus .10), and SCT (.24 versus .03).

As indicated by Table 5, the self-rated trait loadings (i.e., the square root of each variance) for five of the ADHD-I, one of the ADHD-HI, and one of the SCT items were not significant. All of the source loadings, however, were highly significant. By comparing these trait and method loadings, we see that source factors account for a greater proportion of variance than traits for all but two questionnaire items (Table 5). Although global support for convergent validity was upheld via model comparison, the proportion of response variability due to method variance generally exceeds that of trait variance, lessening support for convergent validity.

Parameter-level assessment of discriminant validity

Discriminant validity at the parameter level is assessed using the factor correlation matrices. As can be seen in Figure 1, discriminant validity is clearly supported for the differentiation between SCT and ADHD-HI (r = .02, ns), and modestly so for the difference between ADHD-I and ADHD-HI (.16, ns). As suggested by these two nonsignificant inter-trait correlations (when controlling for ADHD-I and SCT, respectively), studies that use single raters will overestimate these correlations due to common source effects (e.g., Model 4 estimates in Table 3). Although the correlation between ADHD-I and SCT (.66) is large, it is consistent with previous studies of SCT in children and adults (see previous factor analyses section) and suggests that these are related but separable factors.

Figure 1.

Hypothesized model 1 with inter-factor correlations. ADHD-I = attention-deficit/hyperactivity disorder – predominantly inattentive type; HI = predominantly hyperactive/impulsive type; SCT = sluggish cognitive tempo. Measured variables are labeled by rater and ADHD-Rating Scale item number, such that S16 corresponds to the 16^th self-rated item, which belongs to the hyperactive/impulsive latent trait. ^***p < .0001; ⁺p = .053.

Summary of results

These results support the existence of correlated but separable SCT and ADHD factors, with parent ratings explaining more trait variance than self ratings. Self and parent ratings demonstrate low to moderate correlation, but a great deal of variance is still unique to each rater.

Discussion

Construct validity of adult ADHD and SCT

The current literature strongly supports a two-factor model with separate inattention and hyperactivity-impulsivity dimensions as the optimal model to describe the structure of symptoms of ADHD (Willcutt et al., 2012), but few groups have directly investigated the relations between the ADHD symptoms dimensions and SCT. Studies using single raters of children suggest that SCT is a related but separate factor from ADHD with unique neuropsychological, psychosocial, and impairment-related correlates. The current study adds to this literature as the fourth exploration of ADHD and SCT symptoms among young adults. Consistent with previous factor analytic studies of children and adults, we found support for a three-factor solution in which SCT items loaded on a factor separate from symptoms of inattention and hyperactivity-impulsivity. Replicating the majority of findings from child and adolescent studies, SCT was more closely related to features of inattention than hyperactivity.

The current study is the first study of adults to include both self-report and parent-report ratings of ADHD and SCT, an important strength that facilitates a direct test of inter-rater reliability and the extent to which parent and self-report ratings measure the same construct. Although convergent and discriminant validity for ADHD and SCT was supported at the matrix level, variance attributed to each source was consistently greater than trait variance.

Possible explanations for the modest correlation between raters (.49; .49 in the replication sample; Table 3) include similar impressions of the rated behaviors and almost identical wording of the questionnaire items (i.e., only minor changes in wording were necessary to tailor the rating scale to each rater – see Self versus Other Report forms of the ADHD Current and Childhood Symptom Scales (Barkley & Murphy, 1998)). This interrater correlation also indicates that parents and their children generally agree on their ratings of attention and behavior, yet significant and meaningful differences exist. As discussed below, the proportion of trait variance explained by parents’ ratings suggests that the parent-generated representation of their child’s behavior may be less subjectively biased than the self-generated ratings.

The finding that mean trait loadings are higher for parents and thus contain less error variance suggests that parent ratings may be more effective than self ratings when measuring ADHD symptoms among college undergraduates. It is important to note that functional impairment was not included in this model, and therefore parent ratings may only be superior for assessing trait-specific symptom frequency. This same preference for parent ratings was also supported for measuring SCT.

As described by Gomez et al. (2005), this modest evidence of convergent and discriminant validity for self and parent scales is based upon the important assumption that the young adults’ behavior is consistent with what their parents see or know of their daily conduct. Given the varying degrees of how often parents see their college-aged children or speak with them on a regular basis, it seems plausible that the young adults’ day-to-day experience could be vastly different from what their parents know. If parents aren’t familiar with the child’s daily behavior and academic functioning, their ratings may instead reflect the child’s earlier behavior and attention (i.e., from before the past six months). Follow-up investigations will test this hypothesis by exploring similarities and differences between current and childhood ratings.

Indeed, low to moderate rates of inter-rater agreement are a nearly ubiquitous finding across all measures of psychopathology, indicating that this is a central issue for the field, and not a unique problem for ADHD (Achenbach & Rescorla, 2001; Gadow & Sprafkin, 1998). In the case of ADHD, differences between parent and teacher ratings are at least partially explained by differences in children’s behavior in the home and school settings (Gomez et al., 2005; Hart, Lahey, Loeber, & Hanson, 1994). Furthermore, Hart et al. (1994) found that parent and teacher ratings each accounted for unique variance in measures of functional impairment, suggesting that ratings by both reporters are valid and provide unique, clinically relevant information.

Having explored these potential violations of the aforementioned assumption, it follows that ratings of ADHD and SCT are likely to demonstrate more source than trait variance across self and parent ratings. To further explore the possibility that source outweighs trait variance due to situational specificity, ratings could also be collected from friends and/or significant others.

Implications for identification and treatment of adult ADHD

The current study is one of the first to investigate the construct validity of adult SCT. Despite almost three decades since SCT was first described (Neeper & Lahey, 1986), surprisingly few investigations have directly compared SCT and ADHD in a study that included information from multiple raters. It is especially important to incorporate multiple raters in both research studies and clinical practice for disorders such as ADHD that depend so heavily upon rating scales. Whereas pediatric assessments have readily available caregivers and often teachers to provide collateral ratings of attentional difficulties and impairment, evaluations of adult ADHD frequently rely exclusively upon self ratings. Given the varying types of method biases and response styles noted above, especially the subjective bias of rating one’s self, it follows that multiple raters should always be sought when possible (Eid & Diener, 2006; Willcutt, Hartung, Lahey, Loney, & Pelham, 1999). For college-aged adults, it may also be helpful assess parent-child contact or to obtain information from close friends and/or significant others.

Although results in studies of children have been mixed (Willcutt et al., 2012), carefully refined SCT items may eventually prove useful in differentiating between ADHD subtypes, such as ADHD-I and ADHD-combined type. Future work is also needed in samples of adults to test whether SCT may moderate relations between ADHD and important clinical correlates such as functional impairment, and whether individuals with ADHD and high levels of SCT may respond differently to psychosocial therapy or pharmacotherapy than low-SCT individuals.

Limitations and future directions

The current study has some important limitations. Given the nature of this emerging adult population, these results may not generalize to older adults or similarly aged adults who are not enrolled in post-secondary education. However, previous findings regarding the relationship between SCT symptoms and academic impairment (Becker & Langberg, 2013b; Langberg, Becker, & Dvorsky, 2014; Willcutt et al., 2014) highlight the necessity of studying this college-age population. Although this sample was unselected for attentional difficulties, thereby providing stronger ecological validity, future analyses with comparison groups selected for SCT would help advance this area of research. The majority of studies exploring SCT have been conducted using clinical samples or school children with a prior diagnosis of ADHD. Efforts to recruit at a national or community level with a particular focus on young adults are needed.

Secondly, an important limitation of any structural equation modeling approach is the existence of numerous, equally well-fitting models. The current model only provides an acceptable description of the data. Though the hypothesized MTMM model estimated here was guided by decades of ADHD item analyses and theoretical models of the construct of SCT, alternative models are possible and should be explored in subsequent analyses.

This study adds to an extensive literature supporting the validity of SCT as a construct. Key next steps for further work include studies of the neuropsychological, behavioral, neurophysiological, and developmental correlates of SCT. Developing a cognitive model of SCT could help improve early identification and intervention, as well as proffer effective occupational and educational accommodations. As a first step, future studies of related constructs such as ADHD would be well served to include brief measures of SCT. The current study did not measure symptoms of other related disorders, which would have provided insight into additional correlates of SCT. Future investigations might examine the factor structure of ADHD and SCT by exploring the overlap of SCT symptoms with sleep disturbances or assessing the general content of mind wanderings among high-SCT populations. Current debate surrounding the adaptive vs. disabling nature of mind wandering (Baird et al., 2012) could also be informed by evaluating differences between healthy and high-SCT groups.

Finally, despite the developmental instability of the categorical ADHD subtype classifications described in DSM-IV (Willcutt et al., 2012), the DSM-IV symptom dimensions and the overall ADHD diagnosis appear to be persistent throughout the lifespan, with diagnostic symptomatology and impairment extending into adulthood for 50% to 80% of cases (Barkley, 2013; Wilens, Faraone, & Biederman, 2004). Initial results provide support for the stability and developmental influences of SCT (Becker, 2014), but additional studies of a longer developmental period are needed.

Biographies

Daniel R. Leopold, MA, is a clinical psychology PhD student in the dual-degree neuroscience track at the University of Colorado Boulder.

Angela Bryan, PhD, is a social psychology professor in the Department of Psychology and Neuroscience at the University of Colorado with a specialty in quantitative methods. Her work covers areas such as health psychology, behavioral change, substance use, and translational neuroscience. She is head of the social psychology program at the University of Colorado Boulder and teaches the department’s structural equation modeling course.

Bruce F. Pennington, PhD, is a developmental neuropsychologist who applies genetic and neuropsychological research methods to evaluate the comorbidity among dyslexia, ADHD, specific language impairment, and autism. He is a John Evans Professor of Psychology at the University of Denver, where he heads the Developmental Cognitive Neuroscience program.

Erik G. Willcutt, PhD, is a professor in the clinical psychology area of the Department of Psychology and Neuroscience at the University of Colorado. He also holds dual appointments in the Institute for Behavioral Genetics and the Center for Neuroscience. He completed his PhD in clinic child psychology in 1998 at the University of Denver.