Factor structure and construct validity of the parent-reported Inventory of Callous-Unemotional Traits among high-risk 9 year olds

Abstract

The presence of Callous-Unemotional (CU) traits delineates a subgroup of youth with severe antisocial behavior. However, debate surrounds the best method to assess CU traits. This study examined the factor structure of the parent-reported Inventory of Callous Unemotional Traits (ICU) among high risk 9-year olds (N = 540) and its predictive validity over one year. Confirmatory Factor Analysis showed support for a three-factor bifactor model and revised two-factor model using a shortened ICU. Within a three-factor bifactor framework the general callous-unemotional (CU) traits factor and specific uncaring factor scores were related to higher externalizing and lower internalizing behavior problems at ages 9.5 and 10.5. Findings were replicated using teacher-reported outcomes. However, results suggest the need for item refinement and highlight the utility of a two-factor solution using a shortened ICU. In particular, the meaning of the unemotional items is discussed in relation to the conceptualization of CU traits.

In the last 20 years, research has examined callous unemotional (CU) traits among antisocial youth as a theoretical downward extension of the affective features of adult psychopathy (Frick, O’Brien, Wootton, & McBurnett, 1994). Measures of CU traits assess behaviors, such as deficits in empathic concern, shallow affect, and lack of guilt. In recent years, several reviews have summarized the literature showing that the presence of CU traits is related to more severe antisocial behavior in childhood and adolescence and that these traits identify a homogenous subgroup of children with specific risk factors who may require tailored interventions (Frick & White, 2008; Frick, Ray, Thornton, & Kahn, 2014). In recognition of the growing body of research that has demonstrated the utility of assessing CU traits during middle childhood and adolescence, a specifier for the diagnosis of Conduct Disorder based on conceptualizations of CU traits was added to the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition and termed ‘with limited prosocial emotions’, (DSM-5; American Psychiatric Association, 2013). Measurement of CU traits thus remains a significant research focus with important clinical implications. However, gaps remain in our knowledge of the underlying construct of CU behavior¹ and how best to measure it, especially among school-aged children. In response to the shortcomings of previous measures, the Inventory of Callous-Unemotional Traits (ICU; Frick, 2004) was developed to comprehensively assess CU behavior, but remains at the center of debate, particularly in relation to its psychometric properties and the extent to which these properties inform conceptualizations of CU behavior (see Lahey, 2014). In the current study, we examined the factor structure and construct validity of the parent-reported ICU. In particular, we examined the widely-employed three-factor bifactor (3FBF) and newly proposed two-factor (2F) solutions, and tested the predictive validity of ICU scores in a large, high-risk sample of 9.5-year-olds followed longitudinally over a year.

Factor structure of the Inventory of Callous-Unemotional Traits (ICU)

The ICU comprises 12 positively- and 12 negatively-worded items, including four items included in the ‘limited prosocial emotions’ DSM-5 specifier that index CU behavior (‘I care about how well I do at school’, ‘I feel guilty when I do something wrong’, ‘I do not show emotions’, and ‘I am concerned about the feelings of others’). In terms of factor structure, the best fit for the 24-item ICU has typically been obtained by models specifying a 3FBF solution (Essau, Sasagawa, & Frick, 2006). Bifactor models represent an appealing way to model multidimensionality, specifying a general factor that captures shared variance across all items, while simultaneously modeling specific variance of separate dimensions within subsets of items. Bifactor models are common in the intelligence literature (e.g., Carroll, 1993), where conceptualizations of the structure of mental ability comprise both general and specific skills. Bifactor models have also been applied to antisocial behavior, including among studies of adult psychopathy (e.g., Patrick, Hicks, Nichol, & Krueger, 2007) and to model different forms of aggression among antisocial youth (e.g., Tackett, Daoud, De Bolle, & Burt, 2013). Generally, these studies have demonstrated an overarching general factor (e.g., psychopathy or aggression) but with underlying specific factors (e.g., interpersonal, affective, or lifestyle traits).

In the ICU 3FBF model, items load onto three specific factors (callousness, uncaring, and unemotional), while simultaneously loading onto a general CU behavior factor. A 3FBF solution has been replicated in studies of the self-reported ICU among forensic (N =248, ages 12–20, Kimonis et al., 2008) and community samples that represent different ages periods and countries (Table 1; e.g., N=347, ages 12–18, Fanti, Frick, & Georgiou, 2009; N=455, ages 14–20, Roose, Bijttebier, Decoene, Claes, & Frick, 2010; N=540, ages 10–14, Ciucci, Baroncelli, Franchi, Golmaryami, & Frick, 2014). Nevertheless, because much research on the ICU has focused on healthy community samples recruited from schools or on forensic/clinical samples, there is a need for studies that assess dimensional samples that include a full range of antisocial behavior and CU behavior.

Table 1.

Summary of sample descriptives and model fit statistics reported in previous studies examining ICU factor structure among youth

	N	Country	Sample type	Age range	% female	df	χ2	GFI	AGFI	CFI	RMSEA	SRMR	AIC	CAIC	χ2/df
Essau et al. (2006)	1443	Germany	Com.	13–18	46
1 factor						252	2475.366	.731	.680		.120		2571.366	2831.682
3 factor						249	2213.166	.783	.738		.109		2316.166	2596.808
3 bifactor						228	1824.942	.819	.762		.103		1968.942	2365.143
3 bifactor + MI						203	1034.464	.894	.844		.078		1228.464	1762.235
Kimonis et al. (2008)	248	USA	Foren.	12–20	24
1 factor						252	890.76			.50	.10				3.53
Original 4 factor hierarchical						249	800.13			.57	.10				3.21
3 bifactor						231	502.30			.79	.07				2.17
3 factor hierarchical (deleted 2 and 10)						206	471.25			.79	.07				2.29
3 bifactor (deleted 2 and 10)						187	343.52			.87	.06				1.84
Fanti et al. (2009)	347	Cyprus	Com.	12–18	49
1 factor						252	931.88			.572	.089	.090	20,833.12
3 factor						249	722.77			.702	.075	.082	20,630.01
3 bifactor						228	508.35			.824	.058	.061	20,455.59
3 bifactor + MI						212	372.12			.919	.047	.050	20,401.36
Roose et al. (2010)	455	Belgium	Com.	14–20	44
1 factor youth report							2133.43	.72	.66	.77	.13		2229.49	2475.21	8.47
1 factor parent report							1070.74	.63	.56	.76	.15		1166.74	1360.51	4.25
1 factor teacher report							1297.01	.52	.43	.78	.19		1393.01	1574.81	5.15
1 factor combined report							901.61	.61	.54	.84	.15		997.61	1179.41	3.58
3 factor youth report							998.95	.85	.81	.89	.08		1100.95	1362.09	4.01
3 factor parent report							722.49	.72	.66	.85	.11		824.49	1030.37	2.90
3 factor teacher report							816.34	.64	.56	.86	.14		4000.29	4091.19	3.28
3 factor combined report							574.55	.71	.65	.92	.10		676.55	869.71	2.31
3 bifactor youth report							674.53	.89	.86	.92	.07		818.53	1187.20	2.96
3 bifactor parent report							375.12	.83	.78	.93	.07		519.12	809.78	1.65
3 bifactor teacher report							534.03	.73	.64	.90	.11		678.03	950.73	2.34
3 bifactor combined report							348.31	.80	.74	.96	.07		492.31	765.01	1.53
Feilhauer et al. (2012)	383	Holland	Mixed	8–20	0
1 factor						252	1054.56	.777		.70	.105	.093	1411.140	1648.646	4.18
3 factor						249	846.639	.829		.78	.086	.087	1048.295	1300.644	3.40
3 factor hierarchical						249	846.639	.829		.78	.086	.087	1048.295	1300.644	3.40
3 bifactor						228	706.725	.861		.82	.077	.077	882.459	1238.718	3.09
Houghton et al. (2013)	268	Australia	Com.	7–13	43
3 factor						167	380.09			.85	.07
2 factor (unemotional + callous versus uncaring)						169	417.43			.82	.07
2 factor (callous versus uncaring)						103	280.32			.86	.08
2 factor + MI (callous versus uncaring)						95	221.63			.90	.07
Ezpeleta et al. 2013	620	Spain	Strat.	3–4	50
1 factor						1055	6706.30			.77	.093
3 factor						1041	3670.30			.89	.064
3 bifactor						1004	2864.20			.92	.055
Ciucci et al. 2014	540	Italy	Com.	10–14	53
1 factor						209	836.44	.86	.83	.67	.07	.07	924.44		4.00
3 factor						206	616.93	.89	.87	.78	.06	.06	710.93		2.99
3 bifactor						187	426.52	.93	.91	.87	.05	.05	558.52		2.28
3 factor hierarchical						198	442.06	.93	.91	.87	.05	.05	552.06		2.23
Hawes et al. 2014	250	USA	Clinic	6–12	0
1 factor						252	797.66			.78	.09
3 factor						249	629.352			.84	.08
3 bifactor						228	553.36			.87	.08
Revised 2 factor (callous versus uncaring)						53	100.21			.97	.06
Revised 2 bifactor (similar fit but selected 2 factor as it provided, ‘a more parsimonious model’ (p.7)						43	86.57			.97	.06

Note. Com =community (i.e., normative, healthy, or school-based sample); Foren =forensic; Strat =stratified. Degrees of freedom vary across previous studies depending on item set being used (e.g., 22 items instead of 24 items).

Limitations of 3FBF ICU models

Despite advances in research examining models, there are also a number of limitations associated with the 3FBF for the ICU, including poor-to-acceptable model fit indices (see Table 1), marginally acceptable internal consistency of the unemotional subscale, the need to remove items, and error terms being specified to correlate according to modification indices. Further, the callousness subfactor is largely comprised of negatively-worded items, whereas the uncaring factor is comprised of positively-worded items, suggesting that the 3FBF structure may be driven by method variance related to response styles. Finally, an increasing number of studies have not replicated a 3FBF (see Table 1). Across a range of samples using the youth-reported ICU, studies have reported solutions for models with five (N = 383; ages 8–18, Feilhauer, Cima, & Arntz, 2012), two (N = 268, ages 7–13, Houghton, Hunter, & Crow, 2013), and three factors (N = 620, ages 3–4, Ezpelata, Osa, Granero, Penelo, & Domenech, 2013), as well as a recently proposed two-factor solution using a 12-item version of the ICU (N = 250, ages 6–12, Hawes et al., 2014).

However, despite these alternative model solutions and the limitations outlined above, results of studies supporting a 3FBF model have been the justification for which subsequent studies have simply used a summed subscale or total ICU scores within analyses. Notably, studies are also using the parent-reported ICU to assess CU behavior in young children (e.g., Somech & Elizur, 2012), despite the fact that only two previous studies have examined the factor structure of the parent-reported ICU, and both reported inadequate fit for a 3FBF model (Hawes et al., 2014; Roose et al., 2010; Table 1). It is troublesome that the field is moving forward on less than solid psychometric grounds in the context of the new DSM-5 specifier, which highlights the need for reliable measurement of CU behavior for diagnosis and classification. In particular, questions surround the use of the ICU for diagnosis of limited prosocial emotions and, specifically, which version of the ICU (e.g., 24-item or 12-item; e.g., Hawes et al., 2014) best assesses the CU behavior construct, particularly using parent report.

Construct validity of the ICU – externalizing and internalizing behavior outcomes

Indeed, aside from factor structure, an important aspect of psychometric work relating to the ICU surrounds its construct validity. First, previous studies have supported the utility of total summed ICU scores, which typically exhibit high internal consistencies. Further, total self-reported ICU scores correlate positively with externalizing outcomes (e.g., Fanti et al., 2009; Kimonis et al., 2008). Total scores derived from the parent-reported ICU are also related to more behavior problems (e.g., at-risk adolescents, ages 13–17, N = 70, Berg et al., 2013; also see Roose et al., 2010) and criminogenic risk (detained adolescents, ages 12–18, N = 94; White, Cruise, & Frick, 2013). Second, callous and uncaring subscale scores are strongly related to externalizing and aggressive outcomes, in line with findings reported for total ICU scores. However, even though the construct is ‘callous-unemotional’, unemotional subscale scores typically fail to explain unique variance in outcomes and demonstrate inconsistent correlation with measures of externalizing behavior (e.g., Berg et al., 2013). Recently Hawes and colleagues proposed a revised 2F model for the ICU, comprised of only callous and uncaring subfactors (unemotional items were dropped, with the exception of item 6, ‘does not show emotions’), based on an examination of item-total correlations and item-response theory. Their 2F solution demonstrated good model fit, high internal consistency, acceptable test-retest reliability, convergent validity, and replication in an independent sample. This initial revision of the ICU highlights the need for further examination of the validity of ‘unemotional’ items within the context of youth CU behavior.

CU behavior has also been examined in relation to internalizing problem behaviors. Broadly, results mirror those from the adult psychopathy literature, with the theory being that CU behavior is related to fearlessness, low anxiety, and low internalizing (Lykken, 1995). For example, one study found that CU behavior was related to fewer internalizing symptoms over time (N = 1862, ages 5–8 at baseline, Pardini, Stepp, Hipwell, Stouthamer-Loeber, & Loeber, 2012). However, ICU scores have also been shown to predict higher levels of internalizing problems (e.g., Berg et al., 2013; Essau et al., 2006). One explanation for these discrepant findings may derive from the fact that externalizing behavior problems are often strongly associated with internalizing symptoms. Thus, the direction of association between CU behavior and anxiety may be positive, until concurrent externalizing behavior is accounted for, when it becomes negative (i.e., cooperative suppression; see Frick et al., 2014; Lilienfeld, 2003). However, this issue is yet to be addressed in longitudinal analyses using the parent-reported ICU and controlling for overlap between internalizing and externalizing.

Implications of a 3FBF model

Finally, studies have rarely considered the meaning of the 3FBF solution. In particular, the meaning of specific factors may be different when partialling out variance in a general CU behavior factor. Equally, it is unclear what a general factor means in terms of capturing correlation between items once unique variance relating to specific subfactors has been accounted for. Somewhat surprisingly, no studies to date have examined unique associations between specific factors and externalizing or internalizing outcomes, accounting for variance explained by a CU general factor. Rather, studies have solely examined associations between summed subscale scores based on the 3FBF model, even though the 3FBF model implies a need for general and specific factor scores (see Lahey, 2014). Indeed, when summed total ICU scores are used, potentially unique variance captured by specific factors is lost. However, harnessing a 3FBF may produce more precise associations with relevant behavioral outcomes. It is thus important to examine whether a 3FBF predicts criterion variables differently than summed scores.

Gaps in the literature

Taken together, a number of gaps thus emerge in the ICU literature. First, although the parent-reported ICU is already widely in use, there remains a need for its validation, particularly among high-risk samples of youth who exhibit a range of scores on measures of antisocial behavior. In addition, the majority of previous studies that have examined the ICU have assessed samples with wide age ranges (Table 1), making it difficult to draw conclusions about its factor structure during specific developmental periods. Further, no studies to date have examined the longitudinal and predictive validity of the parent-reported ICU across informants. From a prevention perspective this is an important question especially if we can clarify whether ICU scores identify children at risk of developing more entrenched behavior problems before they reach clinical levels, particularly in youth already at high risk for later psychopathology. Second, studies examining construct validity have typically been cross-sectional in design, making it difficult to draw conclusions about the longitudinal predictive value of the ICU. The few longitudinal studies have typically focused on treatment or forensic samples, among which ICU scores predicted risk of future poor outcomes (e.g., White, Frick, Lawing, & Bauer, 2013). Third, while the results from studies that have examined externalizing problems have generally been consistent, studies are needed that examine associations between the ICU and internalizing problems and take into account any overlap between internalizing and externalizing symptoms. Finally, no previous studies have considered the predictive validity of the 3FBF scales versus summed ICU scores.

Aims of current study

The current study seeks to clarify the factor structure of the ICU and its construct validity in a number of ways. First, we examined the parent-reported version of the ICU in a large sample (N = 540; 50% female) of 9-year-olds at risk for conduct problems. We used reports from both primary and alternative caregivers. For primary caregiver reports on the ICU, we compared four models reported in previous studies (one factor, three oblique correlated factors, 3FBF, and revised 2F model) using Confirmatory Factor Analysis (CFA) and replicated findings with alternative caregiver-reported ICU data (e.g., co-parent, father, grandmother). Within a 3FBF framework, we examined associations between ICU scores and primary caregiver versus teacher reported outcomes age 9.5 and predictions to age 10.5. We focused on narrow symptom outcomes for broad-band externalizing (aggressive vs. rule-breaking) and internalizing (anxious-depressed vs. withdrawn-depressed). We computed models for internalizing behaviors while controlling for the overlap between internalizing and externalizing symptoms. We also compared findings using summed scores based on the 3FBF versus 2F solutions specifically with externalizing behavior outcomes. The major goal of the current study was to comprehensively examine the factor structure of the ICU and its predictive validity over time in a way that informed psychometrics of the measure, the use of summed versus factor scores, proposed revisions to the measure, and our understanding of the CU behavior construct.

Methods

Participants

Participants included 731 mother–child dyads recruited between 2002 and 2003 from Women, Infants, and Children (WIC) Nutritional Supplement Program programs in the metropolitan areas of Pittsburgh, PA, and Eugene, OR, and in and outside of Charlottesville, VA (Dishion et al. 2008). Participants were originally recruited to be part of a randomized controlled trial of the Family Check-Up, a preventative intervention for use in high-risk environments to address normative challenges facing parents from toddlerhood onwards (see Dishion et al., 2008). Families were invited to participate if they had a son or daughter between age 2 years 0 months and 2 years 11 months. Recruitment risk criteria were defined as 1 SD above normative means or established clinical cut points on screening measures in at least two of the following three domains: (a) child behavior problems (e.g., conduct problems - Eyberg Child Behavior Inventory; Robinson, Eyberg, & Ross, 1980), (b) primary caregiver problems (e.g., maternal depression, daily parenting stress, or self-report of substance), and (3) sociodemographic risk (low education or low family income). Thus, children in the study selected as ‘high risk’ based on established risk factors for later conduct problems. Specifically, aside from socioeconomic or family risk, families qualified for the original study if children scored in the clinical range on the Intensity or Problem Scales of the Eyberg Behavior Inventory, which comprised 44% of the sample at recruitment, making the sample community-based but enriched/over-sampled for those with early conduct problems (see Dishon et al., 2008; Robinson et al., 1980). However, because the sample was a community (versus clinical) sample and not all children met inclusion criteria based on this definition of clinically-meaningful frequencies of conduct problems, there was variability in the frequency of child conduct problems.

Of the 1,666 families who had children of the appropriate age and who were contacted across study sites, 879 met the eligibility requirements (52% in Pittsburgh, 57% in Eugene, and 49% in Charlottesville), and 731 (83.2%) consented to participate. The children in the sample had a mean age of 29.9 months (SD = 3.2) at the age 2 assessment (approximately 2.5 years old). Across sites, primary caregivers self-identified as belonging to the following ethnic groups: 28% African American, 50% European American, 13% biracial, and 9% other groups. During screening, more than 66% of enrolled families had an annual income < $20,000, and the average number of family members per household was 4.5 (SD = 1.63). Forty-one percent of the sample had a high school or general education diploma. Following the baseline assessment, half the sample was randomly assigned to receive the Family Check-Up intervention (see Dishion et al., 2008); thus intervention status is used as a covariate in all analyses. Of 731 families who initially participated, we had ICU data for 540 (74%) at age 9.5. Of the 540 children with ICU data at age 9.5, we had alternative caregiver reports on the ICU for 401 (74%) children, primary caregiver-reported data for 404 children (75%) at age 10, teacher-reported data for 358 children at age 9.5 (66%), and teacher-reported data for 318 children at age 10.5 (59%). Selective attrition analyses conducted via Chi-squared tests or ANOVAs indicated that there were no differences between children for whom we did and did not have ICU data for according to intervention status (p = .40), race (p = .19), family income (p = .19), and initial, baseline levels of child problem behavior as reported by either primary (p > .70) or alternative (p > .90) caregivers. However, parent education was lower among those families for whom we did not obtain age 9.5 ICU data from - those lost to follow-up were less likely to have at least a high school education (p > .001).

Measures

Recruitment began when children were age 2 and annual assessments (with the exception of age 6) were conducted at family homes using a variety of questionnaires, interviews, assessor impressions, and videotaped observations. From age 7.5 onwards, we also collected data from teachers. The current study uses questionnaire data collected from homes (primary caregiver and alternative caregiver reports) and schools (teacher reports) at ages 9.5 and 10.5 years old. The majority of primary caregivers at age 9.5 were biological mothers (90%). Alternative caregivers were most typically a biological father (45%), the romantic partner of the child’s mother (11%), a grandparent (10%), or an aunt/uncle (4%).

Demographics questionnaire – covariates

Primary caregivers completed a demographics questionnaire at age 2 (Dishion et al., 2008). Consistent with past studies in this sample, child gender was coded as female=0 (n = 271; 50.2%); male=1 (n = 269; 49.8%). Child’s race was coded as ‘Caucasian/other’=0 (n = 304; 56.3%); ‘Black African-American/biracial’=1 (n = 236; 43.7%). Ethnicity was coded as ‘non-Hispanic’=0 (n = 474; 87.8%); ‘Hispanic’=1 (n = 64; 11.9%). Consistent with past studies in this sample, parent education was coded as ‘less than high school=0 (n = 113; 20.9%) and ‘high school and beyond’ =1 (n = 428; 79.1%. Gross annual family income was coded as ≤ $14,999=0 (n = 268; 49.6%); ≥ $15,000=1 (n = 272; 50.4%). Finally, as data were collected from multiple sites and sites differed with respect to the urbanicity and ethnic/racial composition of participants, location was included as a covariate to account for these potential differences. Note that the pattern of findings is unchanged if we include quasi-continuous parent education and family income variables. Further, the cut-points reported represent meaningful differences between groups within our relatively high-risk sample.

CU Traits (age 9.5)

We assessed CU traits at age 9.5 via primary and alternative caregiver reports on the 24-item ICU (Frick, 2004). Each item is rated on a 4-point scale (0 =not true; 1 =somewhat true; 2 =very true; 3 =definitely true). Self-reported ICU data were not collected.

Externalizing and Internalizing problem behavior (ages 9.5 and 10.5)

Primary caregivers completed the Child Behavior Checklist (CBCL; Achenbach, 1991a) and teachers completed the Teacher Report Form of the Child Behavior Checklist (TRF; Achenbach, 1991b). Both questionnaires consist of an externalizing (33 items for the CBCL and 34 for the TRF) and internalizing (31 items for CBCL and 35 for TRF) problem behavior scale. The externalizing scale consists of two subscales, measuring aggressive (e.g., defiant and talks back, disrupts class discipline) and rule-breaking (e.g., steals, fights) behaviors. The internalizing scale comprises withdrawn-depressed (e.g., likes to be alone, withdrawn), somatic complaints (e.g., over-tired, headaches) and anxious-depressed (e.g., fears mistakes, needs to be perfect) subscales. To examine more precise associations within broad-band externalizing and internalizing scales, models were examined focusing on aggressive versus rule-breaking subscale anxious-depressed versus withdrawn-depressed subscales.

Analytic strategy

Aim 1 – To examine the factor structure of the ICU

First, we computed inter-item polychoric correlations for the ICU using primary and alternative caregiver reports². We then used CFA in Mplus version 7.2 (Muthén & Muthén, 2014) to compare model fit for one-factor, correlated three-, 3FBF, and revised 2F solutions for the primary caregiver-reported ICU. We also tested the 3FBF and revised 2F model using alternative-caregiver reports enabling corroboration of model fit within our sample. Models were estimated with mean and variance adjusted weighted least squares estimation (WLSMV) for use with ordinal items (Flora & Curran, 2004). We considered model fit to be adequate if the Root Mean Square Error of Approximation (RMSEA) and Comparative Fit Index (CFI) values met guidelines (i.e., RMSEA<.06 and CFI >.95; Hu & Bentler, 1999). Because we used WLSMV estimation, we carried out corrected chi-square differences test with DIFFTEST in Mplus (Muthén & Muthén, 2014). We examined descriptive statistics and zero-order correlations between summed ICU total and subscale scores for the 3FBF and revised 2F solutions. Internal consistencies of summed ICU scores were assessed using Cronbach’s alpha.

Aim 2 – To test cross-sectional and longitudinal construct validity of the ICU

We computed cross-sectional and longitudinal zero-order correlations between age 9.5 summed total and subscale primary caregiver-reported ICU scores and both primary caregiver- and teacher-reported externalizing and internalizing scores at ages 9.5 and 10.5. Next, we computed a series of path models to examine the prediction of primary caregiver- and teacher-reported externalizing and internalizing problem behaviors by 3FBF general and specific factor scores. We computed separate models to examine prediction of aggressive versus rule-breaking behavior and anxious-depressed versus anxious-withdrawn behavior. We examined cross-sectional associations at age 9.5 and longitudinal associations with age 10.5 scores (controlling for autoregressive effects). We were thus able to examine the pattern of findings for a general CU factor when variance in specific factors was accounted for and vice versa. However, for purposes of enabling comparison with studies that have computed summed total and subscale scores and for practical translation, we also examined associations with aggressive versus rule-breaking behavior within regression models using summed scores. As before, to be transparent with the data, we computed separate models for cross-sectional and longitudinal associations and examined both within and across informant associations. Finally, we compared findings for regression models examining associations between ICU summed scores and aggressive versus rule-breaking behavior when summed scores were based on the 3FBF solution versus the revised 2F solution. In all models examining cross-sectional and longitudinal associations with externalizing and internalizing subscales, we controlled for intervention status, project location, child gender, race, and ethnicity, parent education, and family income.

Results

Factor structure of the ICU

We computed polychoric correlations among items of the primary caregiver-reported ICU (Table 2). There were modest-moderate correlations among items³. Consistent with previous studies, we dropped item 10, ‘does not let feelings control him/her’ (e.g., Ciucci et al., 2014) as higher endorsement of this item was not related to endorsement of other ICU items. It thus appeared that raters were interpreting item 10 as indexing a desirable behavior. In addition, items 15 (‘always tries his/her best’) and 23 (‘works hard on everything’), which have similar item content, were highly related (r =.75). We dropped item 23 as it caused difficulties in the model estimation stage, which appeared to be related to an issue of multicollinearity. We found similar associations among items using alternative-caregiver reports (not shown for brevity, but available on request). We thus computed all models using 22 of the original 24 ICU items.

Table 2.

Polychoric correlations between items (parent-reported version of ICU)

	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23
1. Expresses his/her feelings open
2. Does not know right from wrong.	.10
3. Is concerned about school work	.24	.19
4. Does not care who s/he hurts	.10	.52	.16
5. Feels bad or guilty	.36	.30	.37	.39
6. Does not show emotions	.39	.51	.05	.30	.27
7. Does not care about being on time	.09	.33	.15	.37	.07	.26
8. Concerned about feelings of others	.32	.43	.33	.56	.61	.29	.16
9. Does not care if s/he is in trouble	.19	.43	.34	.57	.39	.36	.31	.32
10. Does not let feelings control	−.21	−.07	−.06	−.06	−.19	−.05	−.003	−.23	−.001
11. Does not care about doing things well	.13	.39	.20	.44	.25	.31	.52	.22	.51	.05
12. Seems very cold and uncaring.	.22	.57	.18	.53	.39	.57	.33	.39	.55	−.005	.47
13. Easily admits to being wrong.	.42	.30	.30	.40	.45	.16	.24	.41	.31	−.20	.26	.39
14. It is easy to tell how s/he is feeling	.53	.23	.22	.24	.42	.41	.20	.41	.28	−.14	.21	.37	.45
15. Always tries his/her best.	.26	.35	.31	.42	.34	.23	.44	.37	.40	−.21	.55	.35	.38	.42
16. Apologizes (“says s/he is sorry”)	.35	.36	.31	.49	.59	.35	.26	.52	.43	−.19	.41	.39	.54	.51	.58
17. Tries not to hurt others feelings	.30	.51	.26	.50	.48	.31	.26	.58	.46	−.26	.39	.52	.48	.42	.55	.70
18. Shows no remorse	.25	.50	.23	.58	.45	.44	.24	.42	.57	−.08	.41	.58	.35	.28	.35	.49	.38
19. Is expressive and emotional	.33	.05	.15	.02	.30	.35	.15	.26	.07	−.02	.17	.14	.21	.45	.23	.26	.24	.04
20. Does not put time into things	.08	.40	.17	.38	.15	.24	.43	.18	.39	−.02	.68	.30	.31	.12	.54	.29	.29	.37	.001
21. Feelings of others are unimportant	.13	.51	.13	.51	.27	.33	.25	.38	.52	−.08	.46	.52	.24	.21	.29	.37	.37	.60	.01	.31
22. Hides his/her feelings	.34	.24	.14	.23	.10	.47	.19	.16	.19	−.03	.29	.50	.18	.31	.12	.14	.12	.25	.14	.18	.25
23. Works hard on everything	.23	.21	.27	.29	.33	.11	.37	.31	.35	−.15	.48	.17	.39	.30	.75	.51	.45	.27	.26	.54	.23	.05
24. Does things to make others feel good	.37	.34	.31	.40	.44	.35	.23	.53	.39	−.14	.37	.42	.43	.46	.49	.63	.65	.39	.30	.30	.30	.19	.53

Note. Item 10 inversely related to other ICU items and excluded from subsequent analyses. High overlap between items 23 and 15 caused problems in model estimation stage; item 23 excluded from subsequent analyses.

We examined one-factor, three-correlated factor, 3FBF, 3FBF with correlated residuals, and revised 2F models for the primary caregiver-reported ICU (Table 3 and Figure 1). The one-factor model showed poor fit to the data although the moderate loadings of all 22 items onto a general factor were notable and hinted at shared variance among items. The three-correlated factor model fit the data better than the one-factor model (Δχ²(3) = 254.76, p < .001); however, the fit was still poor. The 3FBF model fit the data better than a three-correlated factor model (Δχ²(19) = 224.24, p < .001), but the fit was only just acceptable and a number of items on the uncaring subscale loaded in the opposite direction and several were non-significant (Table 3). Because of this poor fit, we examined modification indices. We allowed error terms of five pairs of items to correlate, although we only specified correlations when there was overlap in item content (Figure 1). However, only one of five item pairs was the same as specified in a previous study (see Fanti et al., 2009 who specified 16 pairs of items to correlate, including one overlapping pair with us, items 15 and 20). It is noteworthy, however, that other studies incorporating modification indices have not consistently specified how many items (e.g., Essau et al., 2006) or which items were specified to correlate (e.g., Ciucci et al., 2014). Based on items having negative items loadings on their specific factors, we specified four items (8, 3, 5, & 13) to only have loadings on the general factor. This 3FBF model with modification indices showed good fit to the data (Table 3). We also examined the fit of this 3FBF model specifying the same correlated residuals for the alternative caregiver-reported ICU. Loadings were similar and the fit was good, demonstrating corroboration of our 3FBF model across informants (Table 3). We also compared model fit for males versus females. We conducted multi-group analyses comparing model fit when factor loadings and intercepts were fixed versus freed using the DIFFTEST procedure. We found that the fixed model showed significantly better fit, suggesting that loadings were similar across males and females. Finally, we examined a revised 2F model with 12 ICU items (Hawes et al., 2014). This newly-proposed two-factor solution showed good model fit for both primary caregiver (see Table 3) and alternative caregiver reports (χ²(53) = 126.98, p < .001; CFI = .98; RMSEA = .05; details available on request). However, because the 2F model is based on a different item set than the 3FBF, we could not compare model fit directly.

Table 3.

Factor loadings and model fit statistics for one factor, three-correlated-factor, 3FBF, and revised 2F solutions for parent-reported ICU (Frick, 2004)

	One factor	Three correlated factor			Three-factor bifactor				Three-factor bifactor with MI				Three-factor bifactor with MI (alternative caregiver report)				Two-factor
		Callous	Uncar	Unemo	G	Callous	Uncar	Unemo	G	Callous	Uncar	Unemo	G	Callous	Uncar	Unemo	Callous	Uncar
1. Expresses feelings openly	.47***			.63***	.42***			.52***	.43***			.51***	.51***			.65***
6. Does not show emotions.	.55***			.71***	.51***			.39***	.52***			.38***	.42***			.23***	.55***
14. Easy to tell how s/he is feeling	.60***			.81***	.56***			.54***	.58***			.52***	.50***			.36***
19. Expressive and emotional	.35***			.47***	.30***			.43***	.31***			.42***	.24***			.58***
22. Hides feelings from others	.35***			.45***	.30***			.35***	.29***			.37***	.28***			.33***
2. Does not know right from wrong	.60***	.65***			.54***	.37***			.53***	.42***			.37***	.49***
4. Does not care who is hurts	.68***	.73***			.61***	.38***			.60***	.44***			.77***	.36***			.78***
7. Does not care about being on time	.44***	.48***			.36***	.40***			.30***	.46***			.16***	.68***
8. Concerned about feelings of others	.68***	.73***			.73***	−.07ns			.73***	-			.73***	-				.73***
9. Does not care if in trouble	.64***	.69***			.56***	.43***			.55***	.47***			.52***	.49***			.74***
11. Does not care about doing well	.65***	.70***			.51***	.63***			.45***	.51***			.38***	.56***			.61***
12. Seems very cold and uncaring	.69***	.73***			.63***	.37***			.61***	.43***			.71***	.37***			.77***
18. Shows no remorse	.66***	.71***			.59***	.43***			.58***	.48***			.55***	.49***			.78***
20. Does not put the time into things	.56***	.60***			.42***	.55***			.33***	.44***			.31***	.43***
21. Feelings of others unimportant	.56***	.61***			.46***	.48***			.45***	.55***			.35***	.48***			.67***
3. Concerned about school work	.40***		.41***		.43***		−.17*		.43***		-		.40***		-
5. Feels bad or guilty	.65***		.68***		.70***		−.16*		.70***		-		.68***		-			.70***
13. Easily admits to being wrong	.60***		.62***		.63***		−.02ns		.64***		-		.69***		-
15. Always tries his/her best	.68***		.71***		.71***		.07ns		.61***		.29***		.59***		.38***
16. Apologizes to persons	.80***		.82***		.80***		.19***		.79***		.29***		.78***		.27***			.83***
17. Tries not to hurt others feelings	.78***		.81***		.77***		.42***		.77***		.35***		.79***		.27***			.82***
24. Does things to make others feel good	.71***		.74***		.71***		.24***		.71***		.26***		.68***		.26***			.74***
Χ2	1510.95***	1096.73***			891.29***				603.32***				428.75***				173.42***
df	df =209	df =206			df =187				df =186				df = 186				df =53
CFI; RMSEA	.83; 10	.89; .08			.91; .08				.95; .06				.95; 05				.98; .05

Note.

^*p <.05;

^**p<.01;

^***p<.001.

Items paraphrased for brevity. All parent-reported ICU except if stated. MI =modification indices. In the three-factor bifactor model with MI, error terms of the following items were allowed to correlate: 15 with 07; 15 with 11; 22 with 12; 20 with 11; 20 and 15. In addition, items 8, 3, 5, and 13 were specified to only have general CU factor variance (no specific factor variance). Note that a 2BF (i.e., general CU factor and two specific factors of callous and uncaring) ran into estimation problems as the uncaring items loaded strongly (all loadings > .70) onto the general factor. Thus, the two-correlated factor appears to offer a more parsimonious method to express the variance in these two factors.

Figure 1.

Final three-factor bifactor model for parent- and alternative caregiver-reported ICU

Note. Items 10 and 23 were excluded. Items 8, 3, 5, and 13 were specified to have general variance but no specific (i.e., subfactor) variance. The error terms of the following items were specified to correlate: (i) ‘always tries best’ (item 15) with ‘does not care about doing things well’ (item 11); (ii) ‘hides feelings from others’ (item 22) with ‘seems very cold and uncaring’ (item 12); (iii) ‘does not like to put the time into doing things well’ (item 20) with ‘does not care about doing things well’ (item 11); (iv) ‘does not like to put the time into doing things well’ (item 20) with ‘always tries best’ (item 15); and (v) ‘always tries best’ (item 15) with ‘does not care about being on time’ (item 7). The model was replicated using both parent and alternative caregiver reports on the ICU. For loadings see Table 3.

Total and subscale summed scores

Based on our 3FBF solution, we created a 22-item 3FBF total score, a 9-item callous subscale score, a 4-item uncaring score, and a 5-item unemotional subscale score. Based on the 2F model, we created a 12-item 2F total score, a 7-item callous subscale score, and a 5-item uncaring subscale score. There were moderate inter-correlations among unemotional, callous, and uncaring subscales within 3FBF summed scores (range, r = .31–.52, p < .001) and between callous and uncaring 2F scores (r = .49, p < .001) (Table 4). There was high internal consistency for total ICU scores (3FBF, α = .87; 2F, α = .84), and the callous (3FBF, α = .78; 2F, α = .76) and uncaring subscales (3FBF, α = .81; 2F, α = .84), which were similar across the two solutions, and acceptable internal consistency for the 3FBF unemotional subscale (α = .65). The total 2F score featured 12 of the 22 items included in the total 3FBF score, and not surprisingly, they were highly related (r = .94, p < .001). The 2F callous subscale score featured six of the 9 items included in the 3FBF callous subscale and an additional item that appeared in the 3FBF unemotional scale; these scales were strongly associated (r = .93, p < .001). Similarly, the 2F uncaring scale featured three of the four 3FBF uncaring items and two different items (one from 3FBF general factor with no specific variance, and one from 3FBF callous scale), and they were also strongly related (r= .90, p < .001). Based on bivariate associations, it thus appears that the refined ICU produces scale scores that are similar to those derived from the full version of the measure. Nevertheless, the utility of dropping the unemotional items (and having no unemotional subscale) required further investigation via an examination correlates. We focused our results on models computed within a 3FBF framework so we could examine associations with externalizing and internalizing outcomes for a general factor controlling for variance explained by specific factors and vice versa. However, we also computed total and subscale summed scores and examined associations with externalizing outcomes, which enabled comparability of our findings with previous studies and allowed us to compare the pattern of findings for total and subscale scores derived from the 3FBF versus 2F solutions.

Table 4.

Descriptives, inter-scale and inter-subscale bivariate correlations, and internal consistencies for summed scores derived from three-factor bifactor and two-factor solutions

	Three-factor bifactor solution (3FBF)				Revised two-factor solution (2F) (Hawes et al., 2014)
	Total	Callous	Uncaring	Unemotional	Total	Callous	Uncaring
Total ICU score (22 items)
Callous subscale score (9 items)	.77***
Uncaring subscale score (4 items)	.84***	.52***
Unemotional subscale score (5 items)	.69***	.31***	.45***
Total ICU score (12 items)	.94***	.78***	.84***	.51***
Callous subscale score (9 items)	.75***	.93***	.49***	.37***	.82***
Uncaring subscale score (5-items)	.86***	.48***	.90***	.49***	.90***	.49***
M	18.33	3.33	4.51	4.45	8.13	2.20	5.92
SD	8.83	3.51	2.84	2.49	5.36	2.70	3.50
α	.87	.78	.81	.65	.84	.76	.84

Note.

^*p <.05;

^**p<.01;

^***p<.001

Cross-sectional and longitudinal construct validity of the ICU

Cross-sectional and longitudinal bivariate correlations – summed scores

There were strong associations within primary caregiver and teacher reports of externalizing problem behavior from ages 9.5–10.5, and moderate associations between primary caregiver and teacher reports of scores (see Table 5), suggesting convergence across time and informants. There were modest-to-strong zero-order associations between primary caregiver-reported ICU summed scores for the 3FBF and 2F solutions and primary caregiver-reported externalizing symptoms, including the aggressive and rule-breaking subscales (range, r =.16–.60, p < .01). The magnitude of associations was greater for total, callous, and uncaring scores than for unemotional scores. There were modest-moderate zero-order associations between 3FBF and 2F primary caregiver-reported summed ICU total scores and callous and uncaring subscale scores and teacher-reported externalizing scores (range, r =.10–.25, p < .10). Associations with 3FBF unemotional scores were smaller in magnitude and less likely to be significant (range, r = .09, ns, −.13, p < .13). For internalizing symptoms, zero-order associations within primary caregiver reports were high over time (range, r = .41–.71, p < .001). However, associations between primary caregiver and teacher reports of internalizing problem behavior were lower in magnitude (range, r =. 05, ns, −.29, p < .001; Table 6). Finally, we found moderate positive zero-order associations between primary caregiver-reported ICU and internalizing scores (range, r = .05, ns, −.44, p < .001). Associations between primary caregiver-reported ICU scores and teacher-reported internalizing were smaller and less likely to be significant, although still tended to be positive in directionality.

Table 5.

Zero-order cross-sectional and longitudinal correlations between summed total and subscale scores for the primary caregiver-reported ICU (three bifactor and two-factor solutions) and parent- and teacher-reported externalizing behavior problems (aggression and rule-breaking) at ages 9.5 and 10.5

	Primary caregiver-reported				Teacher-reported
	age 9.5		age 10.5		age 9.5		age 10.5
	Agg	RuleB	Agg	RuleB	Agg	RuleB	Agg	RuleB
Three-factor bifactor
Total ICU (n =22) (PC)	.56***	.53***	.47***	.44***	.18**	.17**	.21***	.23***
Callous subscale (PC)	.57***	.55***	.45***	.43***	.14**	.10†	.15**	.14*
Uncaring subscale (PC)	.50***	.47***	.44***	.40***	.19***	.18**	.19**	.20***
Unemotional subscale (PC)	.18***	.19***	.18***	.16**	.09†	.11*	.09	.13**
Revised two-factor
Total ICU (n =12) (PC)	.61***	.57***	.50***	.47***	.20***	.18***	.23***	.25***
Callous subscale (PC)	.57***	.54***	.45***	.43***	.17**	.13*	.18**	.18**
Uncaring subscale (PC)	.49***	.45***	.42***	.39***	.16**	.18**	.20***	.23***

Note.

^†p < .10;

^*p <.05;

^**p<.01;

^***p<.001.

Ext =Externalizing; Agg =Aggression; RuleB =Rule-breaking; P = primary caregiver-reported; T =teacher-reported.

Table 6.

Zero-order cross-sectional and longitudinal correlations between summed total and subscale scores for the primary caregiver-reported ICU (three bifactor and two-factor solutions) and parent- and teacher-reported internalizing behavior problems (anxious and withdrawn) at ages 9.5 and 10.5

	Primary caregiver-reported				Teacher-reported
	age 9.5		age 10.5		age 9.5		age 10.5
	Anx	With	Anx	With	Anx	With	Anx	With
Three-factor bifactor
Total ICU (n =22) (PC)	.20***	.42***	.19***	.37***	.05	.13*	.05	.15**
Callous subscale (PC)	.29***	.45***	.23***	.36	.02	.10†	.08	.16**
Uncaring subscale (PC)	.14**	.30***	.17**	.30***	.08	.13*	.05	.09
Unemotional subscale (PC)	.05	.31***	.05	.25***	.02	.13*	−.04	.13*
Revised two-factor
Total ICU (n =12) (PC)	.20***	.40***	.21***	.37***	.07	.13*	.08	.14*
Callous subscale (PC)	.26***	.44***	.22***	.35***	.04	.09	.08	.15**
Uncaring subscale (PC)	.11*	.27***	.16**	.29***	.07	.13*	.05	.08

Note.

^†p < .10;

^*p <.05;

^**p<.01;

^***p<.001.

Anx=Anxious-depressed; With=Withdrawn-depressed; PC = primary caregiver-reported; T=teacher-reported.

Cross-sectional and longitudinal construct validity – 3FBF latent model (Tables 7 & 8)

Table 7.

Regression models showing 3FBF ICU model general and specific factor scores predicting primary caregiver- and teacher-reported age 9.5 and 10.5 externalizing behavior

	Within informant				Across informant
	Primary caregiver reports age 9.5		Primary caregiver reports age 10.5		Teacher reports age 9.5		Teacher reports age 10.5
	Aggressive	Rule-breaking	Aggressive	Rule-breaking	Aggressive	Rule-breaking	Aggressive	Rule-breaking
Primary-caregiver-reported 3FBF model of ICU age 9.5
‘General’ CU factor score	.51***	.46***	.09†	.16**	.11†	.11†	.16**	.22***
Callous specific factor	.41***	.43***	.06	.08	.10	.03	.01	.03
Uncaring specific factor	.15***	.15***	.09	.05	.14	.12	−.10	−.09
Unemotional specific factor	−.21***	−.13*	−.03	−.07	.01	.02	−.13†	−.10

Note.

^†p < .10;

^*p <.05;

^**p<.01;

^***p<.001.

All models controlled for intervention group, gender, race, ethnicity, and project site. Models predicting age 10.5 outcomes controlled for age 9.5 primary caregiver-reported aggressive or rule-breaking behavior as relevant. Controlling for concurrent internalizing disorder did not change the pattern of findings – results are not shown for brevity.

Table 8.

Regression models showing 3FBF ICU model general and specific factor scores predicting primary caregiver- and teacher-reported age 9.5 and 10.5 internalizing behavior

	Within informant				Across informant
	Primary caregiver reports age 9.5		Primary caregiver reports age 10.5		Teacher reports age 9.5		Teacher reports age 10.5
	Withdrawn- depressed	Anxious- depressed	Withdrawn- depressed	Anxious-depressed	Withdrawn- depressed	Anxious- depressed	Withdrawn- depressed	Anxious-depressed
Primary-caregiver-reported 3FBF model of ICU age 9.5
‘General’ CU factor score	.02 (.28***)	−.27*** (.08)	−.09† (.05)	−.12** (.05)	.07† (.14*)	.01 (.06)	−.06 (.06)	−.15* (.03)
Callous specific factor	.15** (.40***)	−.05 (.30***)	−.06 (.08)	−.10* (.06)	.001 (.03)	−.03 (.001)	.19** (.10†)	.08 (.02)
Uncaring specific factor	.02 (.12†)	.001 (.15†)	.04 (.18*)	−.01 (.13†)	−.05 (.12†)	−.02 (.07)	.01 (−.10)	.13 (−.10)
Unemotional specific factor	.38*** (.30***)	.16*** (.07)	.11 (.09)	.03 (.02)	.12 (.30***)	.003 (.03)	.13 (−.10)	−.06 (.06)

Note.

^†p < .10;

^*p <.05;

^**p<.01;

^***p<.001.

All models controlled for intervention group, gender, race, ethnicity, and project site. Models predicting age 10.5 outcomes controlled for age 9.5 primary caregiver-reported aggressive or rule-breaking behavior as relevant. Models predicting internalizing symptoms include controlling for externalizing disorder. However, estimates when concurrent externalizing behavior is not controlled for are shown in italics and parentheses. These results highlight cooperative suppression effects of externalizing behavior in relation to associations between ICU scores and internalizing symptoms.

In cross-sectional models at age 9.5, higher ICU general factor scores were associated with higher levels of both primary caregiver- and teacher-reported aggressive and rule-breaking behaviors (Table 7). Higher callous and uncaring specific scores were also related to higher primary caregiver-reported aggressive and rule-breaking behaviors. However, the unemotional specific factor predicted fewer primary caregiver-reported aggressive and rule-breaking behaviors, and was unrelated to teacher-reported outcomes in cross-sectional models. In longitudinal autoregressive models that controlled for earlier externalizing behavior problems, the ICU general factor significantly predicted increases in both teacher and primary caregiver-reported rule-breaking. General ICU factors scores also predicted increases in teacher-reported aggressive behavior at age 10.5 (prediction of primary caregiver-reported aggressive behavior was a trend). It is noteworthy that the effect of the ICU general factor was greater in magnitude for the prediction of rule-breaking (primary caregiver-reported, β = .16, p < .01; teacher-reported, β = .22, p < .001) compared with the prediction of aggression (primary caregiver-reported, β = .09, p < .10; teacher-reported, β = .16, p < .01). None of the specific factors accounted for unique variance in primary caregiver-reported aggressive or rule-breaking from ages 9.5–10.5.

When we examined associations with internalizing outcomes (anxious-depressed vs. withdrawn-depressed), we compared the pattern of effects when we did and did not control for concurrent externalizing behavior. In cross-sectional models, we found that a higher ICU general factor score was related to lower primary caregiver-reported anxious-depressed scores at age 9.5 (Table 8), but only after accounting for concurrent externalizing behavior. Accounting for variance explained by the general CU behavior factor however, we found that unemotional and callous specific factors were related to higher withdrawn-depressed scores, and unemotional scores were also related to higher anxious-depressed scores. There were no significant associations between the ICU general or specific factors scores and teacher-reported outcomes in cross-sectional models at age 9.5 (Table 8). In longitudinal autoregressive models (Table 8), we found that ICU general factor scores were related to decreases in primary caregiver- and teacher-reported anxious-depressed scores and primary caregiver-reported withdrawn-depressed scores from ages 9.5 to10.5, but again, only after controlling for concurrent externalizing behavior at age 10.5. Scores on the callous specific factor were also related to decreases in primary caregiver-reported anxious-depressed behavior. In line with cross-sectional models, we found that the unemotional specific factor was related to increases in primary caregiver- and teacher-reported withdrawn-depressed scores. Likewise, the callous specific factor predicted increases in teacher-reported withdrawn-depressed scores. We also report estimates when we did not control for concurrent externalizing (in italics and parentheses; Table 8), the results of which reinforce the importance of considering cooperative suppression effects between externalizing and internalizing behavior in relation to associations with CU behavior. Specifically, the direction of effects between the general ICU factor and primary caregiver-reported anxious-depressed score reversed when taking account of comorbid externalizing behavior symptoms (see Discussion).

Cross-sectional and longitudinal construct validity - summed scores

We re-examined associations with externalizing and internalizing outcomes using summed total and subscale scores within regression analyses. The pattern of findings broadly mirrored that obtained when associations were examined within a 3FBF framework. For brevity, we thus only present results from examining associations with externalizing outcomes (Table 9; results of models examining associations with internalizing outcomes available on request from authors). We also examined associations using summed scores based on the 2F model to test whether this more parsimonious set of items performed similarly to the 22-item set. Both the 3FBF and 2F produced summed total scores and callous and uncaring subscale scores that were cross-sectionally related to higher primary caregiver- and teacher-reported aggressive and rule-breaking behavior. In longitudinal autoregressive models, summed total scores of the 3FBF and 2F models predicted increases in primary caregiver and teacher reports of rule-breaking at age 10.5. In longitudinal autoregressive models, the uncaring subscale of both model solutions was related to increases in rule-breaking behavior across informant. As with the 3BF analyses however, we found that unemotional summed scores were cross-sectionally related to lower primary caregiver reported aggressive and rule-breaking behavior, accounting for overlap with other subscales⁴.

Table 9.

Regression models showing summed ICU total and subscale scores predicting primary caregiver- and teacher-reported age 9.5 and 10.5 externalizing behavior

		Within informant
		Primary caregiver reports age 9.5		Primary caregiver reports age 10.5
		Aggressive	Rule-breaking	Aggressive	Rule-breaking
3FBF	Model 1 – total ICU score
	Total ICU (n =22) (PC)	.55***	.52***	.03	.08*
	Model 2 – subscale scores
	Callous subscale (PC)	.43***	.44***	.02	.03
	Uncaring subscale (PC)	.32***	.26***	.03	.08†
	Unemotional subscale (PC)	−.11***	−.07†	−.001	−.02
2F	Model 3 – total revised ICU score
	Total ICU (n =12) (PC)	.59***	.56***	.03	.10*
	Model 4– total revised subscale scores
	Callous subscale (PC)	.43***	.43***	.003	.03
	Uncaring subscale (PC)	.27***	.23***	.03	.08†

		Across informant
		Teacher reports age 9.5		Teacher reports age 10.5
		Aggressive	Rule-breaking	Aggressive	Rule-breaking
3FBF	Model 5 – total ICU score
	Total ICU (n =22) (PC)	.15**	.13**	.08	.16**
	Model 6 – subscale scores
	Callous subscale (PC)	.10†	.04	−.01	.06
	Uncaring subscale (PC)	.09	.11†	.11†	.10
	Unemotional subscale (PC)	−.01	.02	−.03	−.01
2F	Model 7 – total revised ICU score
	Total ICU (n =12) (PC)	.17***	.15**	.09†	.17**
	Model 8– total revised subscale scores
	Callous subscale (PC)	.16**	.08	.01	.07
	Uncaring subscale (PC)	.06	.09†	.12*	.14*

Note.

^†p < .10;

^*p <.05;

^**p<.01;

^***p<.001.

Ext =Externalizing; Agg =Aggression; RuleB =Rule-breaking; PC = primary caregiver-reported; T =teacher-reported. All models controlled for intervention group, gender, race, ethnicity, and project site. Models predicting age 10.5 outcomes controlled for age 9.5 primary caregiver-reported aggressive or rule-breaking behavior as relevant. Controlling for concurrent internalizing disorder did not change the pattern of findings – results are not shown for brevity.

Discussion

In the current study, we addressed a number of questions surrounding the parent-reported ICU. We found acceptable model fit for a 3FBF model with correlated residuals for both primary and alternative caregiver reports, thus providing some corroboration of this structure among a sample of high-risk children aged 9.5. We also found good model fit for a revised 2F model using a reduced 12-item pool, providing support for the proposal by Hawes and colleagues (2014) to focus only on callous and uncaring, and to trim item content. Total and subscale scores from the 3FBF and 2F models showed acceptable-to-high internal consistencies. An examination of the cross-sectional and longitudinal construct validity of scores within a 3FBF framework suggested that the ICU provides predictive validity in relation to covert forms of antisocial behavior indexed via a measure of rule-breaking behavior, although effect sizes of this prediction were modest in magnitude within autoregressive models. Our results speak both to the assessment of CU behavior using this measure and to the construct itself.

Aim 1 – To examine the factor structure of the ICU

We found that a 3FBF model, with modification indices guiding correlation of five pairs of items, showed the best model fit for primary and alternative caregiver reports on the ICU. It is unclear how similar our use of modification indices is to those employed in previous studies, which have not always reported the correlations specified among item residuals (e.g., Ciucci et al., 2014; Essau et al., 2006). The need to employ modification indices is a limitation associated with the ICU that we had wanted to avoid. However, the necessity for this approach in both the current study and previous studies suggests overlapping item content within and between factors that may be compounded further by similar semantic item structure (e.g., 8 items begin with ‘does not’). Further, it is noteworthy that the factor loadings of the uncaring specific factor seemed to be the most affected within the bifactor model, suggesting that ‘uncaring’ may be most closely aligned with a general ‘CU’ factor (Pardini, Hawes, Burke, & Loeber, 2014). Thus, although we replicated the most commonly-reported factor structure (i.e., 3FBF), the psychometric properties of the ICU continue to appear far from robust. Importantly, from a modeling perspective, the 2F solution of Hawes and colleagues (2014) had good model fit and appears to offers a more parsimonious assessment of a central callous and uncaring construct than the 3FBF model. This 2F model could also prove to be more stable across samples as modification indices and other specific changes were not needed. The 2F solution is supported by another recent study that assessed children aged 7–12, where CFA showed that a 2F model comprising callous and uncaring dimensions fit the data best (Houghton et al., 2013).

Aim 2 – To test cross-sectional and longitudinal construct validity of the ICU

Externalizing problem behavior

Within a 3FBF framework, the ICU general factor was cross-sectionally related to higher aggressive and rule-breaking behavior. The general factor also predicted increases in rule-breaking behavior from ages 9.5–10.5 across informants and settings. This finding highlights that by leveraging shared variance among ICU items, and controlling for unique variance of specific factors, we tapped a construct with incremental validity in relation to the development of behavior problems, particularly covert antisocial behavior. Nevertheless, the practical utility of a ‘general bifactor’ model may be limited given the inability to model a meta-factor with individual client data. Interestingly however, summed total and uncaring subscale scores implied by both the 3FBF and 2F models also predicted increases in rule-breaking from ages 9.5–10.5, after accounting for earlier rule-breaking. It was surprising that despite the widespread use of the ICU and the predominance of the 3FBF, no previous studies had examined associations of ICU scores with relevant outcome variables within a 3FBF (see Lahey, 2014). The fact that we found a similar pattern of findings when analyses were carried out using summed scores or within the 3FBF framework is striking. Ultimately, assessment is a practical enterprise, and our results provide justification for future studies to use a summed total score using either 22 ICU items or the revised 12-item ICU to create a total score with predictive utility. Further, the results suggest that uncaring items may be particularly pertinent for identifying those youth at risk for displaying increases in rule-breaking behavior among high-risk children. At the same time, effect sizes were modest in magnitude, especially for longitudinal autoregressive models, which may be due to the high stability in externalizing behavior. Nevertheless, the incremental validity of the ICU scores suggests that knowing about CU behavior as indexed via the ICU may be helpful for targeting or tailoring specific intervention or treatment components to different groups of youth (e.g., Dadds et al., 2014).

Whereas total and uncaring scores were reliably related to more externalizing problems across informants, we found that unemotional subfactor scores were related to lower scores when we controlled for variance in general CU behavior within a 3FBF framework. This effect was particularly notable for cross-sectional parent-reported aggressive and rule-breaking symptoms. As few studies have examined the longitudinal predictive validity of ICU within a 3FBF framework, we interpret our findings with caution. One possibility is that once variance in general CU behavior and specific uncaring and callousness factors is partialled, the variance remaining in unemotionality actually relates to emotional resiliency or a lack of externalizing problem behavior, and thus represents a marker of positive mental health. We return to an evaluation of the unemotional part of the CU behavior construct later in this discussion.

Internalizing problem behavior

Within a 3FBF, we obtained expected negative associations between the general CU behavior factor and anxious-depressed scores. Notably, the general factor predicted lower anxious-depressed scores at age 10.5, controlling for autoregressive effects, and across primary caregiver- and teacher-reported outcomes. Thus, our results support the notion that the shared general variance within ICU items is related to lower anxiety, in line with the defining characteristics of affective aspects of psychopathy among adult samples (Frick et al., 2014; Lykken, 1995). Findings were replicated using summed scores for the total ICU and uncaring scores implied by both the 3FBF and 2F model for primary caregiver-reported anxious-depressed scores. However, this pattern of effects only emerged when we controlled for comorbid externalizing symptoms; otherwise ICU general scores were related to higher anxious-depressed scores. This reversal in the direction of associations is notable, and represents possible cooperative suppression (see Frick et al., 2014). By partialing the variance in this way, we appeared to be accounting for comorbidity between externalizing and internalizing, which may be underpinned by some higher-order dimension, such as or behavioral dysregulation or negative emotionality (for further discussion of this issue, see Lilienfeld, 2003; Hyde, Byrd, Votruba-Drzal, & Hariri, & Manuck, 2014). Only when this overlapping negative emotionality was accounted for were we able to obtain expected negative associations between CU behavior and anxiety (also see Loney, Frick, Clements, Ellis, & Kerlin, 2003). At the same time, our results highlight that while partialing variance is a useful approach to demonstrate construct validity within the context of statistical modeling, the reality may be more complex with many children presenting with problems across multiple domains of functioning.

In contrast to the negative association between ICU total scores and anxious-depressed symptoms, we found callousness to be related to higher withdrawn-depressed scores even after controlling for concurrent externalizing problems. It may be that this finding emerged as a statistical artifact with a 3FBF framework. However, a similar pattern of findings was found using summed scores in regression models. Further, Hawes and colleagues (2014) also reported a positive association between ICU callous scores and measures of internalizing behavior problems. Thus, when variance in other subscales or a general CU behavior factor is partialled, it appears that callousness may relate to parents or teachers endorsing children as being socially-withdrawn, isolated, or low in mood. This finding highlights the need for careful consideration of the wording of callous items. In particular, it is noteworthy that the uncaring subscale is comprised of positively-worded items, whereas the callous subscale is negatively-worded. Future studies are needed to examine whether the same pattern of associations is achieved when callousness is assessed with positive-worded items and uncaring with negatively-worded items. Moreover, it may indicate that the suppression effects discussed above do not generalize to all types of internalizing, only those focused on symptoms of anxiety (vs. depression).

The meaning of ‘unemotional’

We found that when callous, uncaring, or general factor scores were controlled for, unemotional behavior was related to more withdrawn-depressed symptoms. In conjunction with the finding that unemotional scores were related to lower aggressive and rule-breaking scores, our results provide thus support for the conclusions of Hawes and colleagues (2014) that consideration needs to be given to conceptualizations of ‘unemotional’ among children and adolescents. In particular, the ICU unemotional items may not be doing a good job of capturing ‘unemotionality’ as it relates to the nomological network of CU behavior. Interestingly, in a previous paper assessing this sample in the preschool years (Hyde et al., 2013a), we developed a ‘home-grown’ measure of CU-like behavior and found that traditional ‘unemotional’ items did not load with items indexing callousness and uncaring.

Alternatively, it may be that ‘unemotionality’ as indexed by the ICU is interpreted by informants as withdrawn, anhedonic, or shy behavior, which differs somewhat from conceptualizations of the unemotional component of CU behavior indexing reduced anxiety or fearlessness (cf., low fear in psychopathy; Lykken, 1995). In support of this notion, Ezpeleta and colleagues (2013) reported a positive association between teacher-reported unemotionality and anxiety among preschoolers. A review of individual cases suggested that teachers did appear to be rating ‘unemotional’ children as shy, socially phobic, and unable to express feelings (Ezpeleta et al., 2013, p. 102). At the same time, it is possible that once variance in callous and uncaring are partialled, unemotionality predicts covert aggression not captured in either our aggressive and rule-breaking subscales (e.g., lying that remains undetected; relational aggression).

Another explanation is that the general CU behavior factor acts as a suppressor variable, obscuring the relationship of the unemotional subfactor with criterion-related variables when considered in the 3FBF framework. Indeed, in zero-order correlations, unemotional scores were associated with higher externalizing, albeit with effect sizes that were smaller in magnitude than for those for callous or uncaring. However, this conclusion is difficult to reconcile with the positive associations we found between unemotional and internalizing problem behaviors. Further, while a lack of emotional responsivity has been documented at a neurobiological level for antisocial youth with CU behavior (i.e., reduced amygdala responsivity to others’ fear and reduced responsivity to punishment; see Hyde, Shaw, & Hariri, 2013b), it is less clear that children with high CU behavior are, in fact, less emotionally expressive, as implied by the ICU unemotional items (e.g., ‘hides feelings from others’). Indeed, previous studies have shown that adolescents with high CU behavior display significant negative emotionality, including anger, anxiety, and depression (e.g., Kimonis, Frick, Cauffman, Goldweber, & Skeem, 2012).

As such, there is a need for refinement of items to capture more accurately what is meant by ‘unemotional’. For example, some youth may be callous/uncaring but display fearlessness and low anxiety, whereas other may be callous/uncaring and show high levels of anxious or fearful behavior as a result of early environmental risk (Kimonis et al., 2012; Waller et al., 2013). Indeed, classic descriptions of psychopathy typically focus on low fear rather than a lack of emotionality in general (Lykken, 1995). Thus, revised items are needed to capture distinction in CU behavior in the context of negative emotional responses (including anger or anxiety) versus low levels of fear. Incorporating assessment of temperamental dimensions, such as negative emotionality or prosociality may help to differentiate between antisocial subgroups of youth with and without CU behavior (Lahey, 2014). At the same time, one item of the original unemotional scale (‘does not show emotions’) was retained in the shortened ICU, albeit as part of the callous factor (Hawes et al., 2014). Thus, it appears that continued consideration of the emotional displays of children (or lack thereof) is warranted to provide the affective context for any callous or uncaring behavior (see Rowe, 2014). However, our results relating to unemotionality highlight that the CU behavior construct, at least that assessed by the ICU, is relatively narrow and indexes only some of the personality traits linked to adult psychopathy. Thus the CU behavior construct should not be synonymously equated with psychopathy.

Implications for assessment of the CU construct

There are a number of implications for assessment of the CU construct. First, the results provide justification for practitioners to assess CU behavior using a sum of 22 ICU items, as implied by our 3FBF, or using the revised 12-item set proposed by Hawes and colleagues (2014) which both added unique variance in relation to the prediction of future behavior problems across informants and settings. Use of these ICU summed scores may help in the diagnosis of the DSM-5 ‘limited prosocial emotions’ specifier. Second, the 3FBF implies separable components of the CU behavior construct. However, we also found unexpected correlations with criterion-related variables that undermine conceptualizations of the subfactors, particularly the unemotional subfactor. The ‘unemotional’ items do not appear to operate as intended in the nomological basis of CU behavior and thus may not be useful clinically or conceptually. Taken together, these points suggest that in using the ICU for assessment, the most meaningful and reliable predictive validity is derived via use of a latent general or summed total score. However, our results also highlight the need for alternative self-, parent-, or teacher-reported measures of CU behavior with stronger psychometric properties. Further, the need for alternative methodologies is implied, especially in relation to how items are interpreted (i.e., callousness as withdrawal or anhedonia). One new assessment approach, the ‘Clinical Assessment of Prosocial Emotions’ (CAPE; Frick, 2013) is currently under development, and is designed to assess CU traits via semi-structured and self-reported interviews. Finally, our analyses were conducted within a dimensional framework, and as such, we cannot speak to the application of the ICU within person-centered analyses or for categorical “diagnosis”. However, evidence from neuroimaging studies has demonstrated divergent patterns of amygdala reactivity among youth with Conduct Disorder based on their level of CU behavior (e.g., amygdala hyporeactivity to threat for high CU behavior versus amygdala hyperreactivity to threat for low CU behavior; Viding et al., 2012), which represents a strong, objective test of the discriminant validity of ICU scores. In this example, a cut-off score was created using a median split on ICU total scores (Viding et al., 2012, p. 1110). However, future studies are needed to establish the validity of cutoff scores using the ICU or other measures of CU behavior, including the CAPE.

Strengths and limitations

There were a number of strengths to the study, including a fairly large sample size, having children all assessed at the same age, use of a prospective longitudinal design, use of multiple informants, and corroboration of findings across informants. However, there were several limitations. First, we focused on low-income children with risk factors across multiple domains, including sociodemographic risk and early child problem behavior. Thus, it is unclear whether our results would generalize to children from higher-income families with fewer risk factors. At the same time, our findings represent a useful complement to previous studies that have examined the ICU factor structure and that have focused on adolescents (e.g., Fanti et al., 2009), community or school samples (e.g., Ezpeleta et al., 2013), and incarcerated youth (Kimonis et al., 2008). Results from the current study likely bridge the distribution across these studies in having some children low and others very high on externalizing. Second, while we collected data from teachers for externalizing and internalizing outcomes, we only had primary and alternative caregiver reports for the ICU. Nevertheless, it remains unclear who the best informant is for assessing youth CU behavior and our results suggest a need to investigate alternative ways to find about these emerging aspects of personality, especially in relation to concerns regarding the validity of self-reported measures of psychopathy (e.g., Lilienfeld, 1994). Finally, there was some attrition in our sample by 10.5 years old. We used FIML to accommodate missing data for models testing prediction by ICU scores for the 540 families for whom we had ICU data at age 9.5, although it is unclear whether the prediction of outcomes by primary caregiver-reported ICU scores would have differed among those lost to follow-up.

Conclusions

We found support for use of a general CU behavior score based on 22-items of the ICU, which was related to future rule-breaking behavior and lower anxious-depressed behavior problems over a one year period across informants and settings. Thus, our study supports the use of the ICU among high-risk children to identify those at risk for developing covert forms of antisocial behavior. However, the effect size of predictions was not large, and future work is indicated for better assessment and identification of youth with particularly accelerating forms of externalizing behavior problems. In particular, our study presents a number of questions moving forward for this relatively young field. While there appears to be utility in using a total score that captures shared variance across this set of items, more psychometric work is needed to examine the item wording and specific correlates of subfactors, particularly the unemotional subfactor. Indeed, total summed ICU scores derived from a refined 12-item version of the measure showed comparable longitudinal predictive validity to the original item set. Further, the model fit for the 2F solution for these 12 items was very good, and this model appears to offer a parsimonious method of assessing callous and uncaring behaviors with more robust psychometric support.