Differential Risks of Covarying and Pure Components in Mother and Teacher Reports of Externalizing and Internalizing Behavior Across Ages 5 to 14

Abstract

In a sample of 585 children assessed in kindergarten through 8th grade, we fit a confirmatory factor model to both mother- and teacher-reported symptoms on the Achenbach checklists (CBCL, TRF) and determined that a covariation factor of externalizing and internalizing behaviors existed, in addition to the pure-form factors of externalizing and internalizing for each reporter. In 3 structural equation models, between 8 and 67% of the variance in these 6 latent factors was accounted for by a set of antecedent child, sociocultural, parenting, and peer risk variables. Each of the 6 latent factors, taken 2 at a time, was predicted by a unique set of risk variables; however, there were some patterns that held for both mother- and teacher-report symptom factors: Child temperamental unadaptability and female gender were predictors of higher internalizing symptoms; child temperamental resistance to control, parental harsh punishment, male gender, low SES, and peer rejection were related to higher externalizing symptoms whereas child temperamental unadaptability was related to lower externalizing symptoms; and peer rejection and family stress were also related to the covarying, externalizing-plus-internalizing component of both mother and teacher reports.

INTRODUCTION

Comorbidity refers to the coexistence of two or more distinct disorders in the same individual at the same point in time (Achenbach, 1991a; Caron & Rutter, 1991). Comorbidity is not only pervasive but, at times, occurs more frequently than single disorders and has worse developmental outcomes than single-form disorders (see review by Nottelman & Jensen, 1995) Research on this phenomenon is extremely important to the field of developmental psychopathology because it has implications for the validity of past and future classification systems, etiological theories, treatment outcome research, and treatment recommendations.

The study of comorbidity addresses three important questions (Rutter & Sroufe, 2000): (1) Is comorbidity a real phenomenon or merely a reflection of nosological and methodological problems? (2) Do comorbid disorders have different antecedents, correlates, developmental pathways, outcomes, and treatment responses than their pure form counterparts? (3) Do conclusions from previously published studies of risks and treatments that either neglected to assess for comorbid disorders or excluded comorbid subjects apply to more representative populations with multiple disorders? Ignoring the issue of comorbidity may lead to erroneous theories about developmental psychopathology and ineffective treatments. Although comorbidity has been documented in many epidemiological and longitudinal studies, what comorbidity actually means, whether it is a valid phenomenon worthy of future study, and what risks and mechanisms might be related to its etiology and development are not yet established (Caron & Rutter, 1991; Rutter & Sroufe, 2000). Clearly, the study of comorbidity and its development is still in an “embryonic state” (Loeber, Burke, Lahey, Winters, & Zera, 2000, p. 1475).

The current paper will (a) present conceptual and methodological frameworks for studying comorbidity or covariance and then use the frameworks to ascertain if the covariance of externalizing and internalizing behaviors is a unique factor, separate from the pure-form factors of externalizing and internalizing behaviors, and (b) examine risk variables related to the pure and covarying conditions of these syndromes using both mother and teacher reports.

FRAMEWORKS FOR STUDYING COOCCURRENCE

Three major conceptual issues exist. One of these is the fundamental distinction between true comorbidity and apparent comorbidity (Caron & Rutter, 1991). Although two or more truly independent disorders can cooccur (i.e., true comorbidity), apparent comorbidity can be produced artifactually by a number of confounds.

Few researchers have attempted to separate artifactual from true comorbidity. Artifactual comorbidity (or covariance) can be produced by sampling bias, nosological confusion, severity, and measurement variance. (a) Sampling bias is relevant to clinical samples, in which the rate of comorbidity always tends to be greater than in general population samples (Angold, Costello, & Erkanli, 1999), because the probability of multiple disorders is a function of the combined probabilities for referral for each of the disorders. Caron and Rutter (1991) recommended studies of general population samples. In this study, we used a nonreferred, community sample of adolescents randomly recruited from three cities, (b) Artifactual comorbidity may result from the type of classification system used in a study, because some systems do not distinguish pathological conditions from nonpathological ones or discriminate well between clinical diagnoses (Caron & Rutter, 1991). We hope to avoid such “nosological confusion” by using Achenbach’s empirically based taxonomy and related scales, the Child Behavior Checklist (CBCL; Achenbach, 1991b) and the Teacher Report Form (TRF; Achenbach, 1991c). Both have been empirically derived from representative populations, have been shown to discriminate well between referred and nonreferred children, and use nonoverlapping lists of symptoms of externalizing and internalizing behaviors. (c) Severity is another potential confound in assessing true comorbidity. For example, suppose children with cooccurring conduct disorder/depression have worse outcomes than those with pure conduct disorder and those with pure major depression. However, if those with cooccurring disorders had higher scores on the component disorders than either of the “pure” groups, then the differences between individuals with pure versus covarying disorders on outcomes could be ascribed to the cooccurring group having more severe psychopathology rather than having distinct pathological conditions. In our confirmatory factor model, we test whether the respondents’ factor scores for pure externalizing, pure internalizing, and covarying externalizing and internalizing symptoms are normally distributed with similar means across the sample, thus examining whether the confound of severity might be present or not. (d) Measurement variance may also give rise to artifactual comorbidity because of the possible correlation between the error components associated with two or more disorders (Lilienfeld, Waldman, & Israel, 1994) or with informant variance (Hinden, Compas, Howell, & Achenbach, 1997). Following Weiss and Catron (1994), we use a confirmatory factor model of mother- and teacher-reports of externalizing and internalizing symptoms to estimate the error components of the observed scores on the CBCL and the TRF. Estimation of these error components allows us to investigate the covariation of the “true” scores on these measures. By allowing the error components to correlate within informants, we control for informant effects (Hinden et al., 1997). Because we also analyze covariation in mother- and teacher-reports separately, we are able to tease apart which risk variables are related to covariation in these two separate domains of home and school, thus controlling for one more possible artifact. Therefore, we may be able to describe patterns of risks for the pure externalizing, pure internalizing, and covarying sets of symptoms; determine whether these patterns are different across disorders and across informants; and, achieve an empirical foundation for future construction of a developmental theory of covarying externalizing and internalizing syndromes.

A second issue is that, although the terms cooccurrence, comorbidity, and covariance have been used interchangeably, they actually refer to three separate phenomena (Hinden et al., 1997; Lilienfeld et al., 1994). Cooccurrence results from labeling an individual with two or more psychopathological conditions (diagnoses, symptoms, or syndromes). Cooccurrence does not necessarily mean that these conditions actually exist, in a valid and reliable manner, but just that the individual has been identified, whether correctly or for artifactual reasons, as having both of these conditions. In contrast, the term comorbidity is used to define the valid (i.e., not produced by artifacts) coexistence of two or more categorically defined and distinct disorders, such as in DSM IV (American Psychiatric Association, 1994) or International Classification of Diseases-10 (World Health Organization, 1992). Covariation is a term used more typically in the context of a dimensional approach to psychopathology and refers to the statistical degree to which one syndrome correlates with another syndrome. Like the term comorbidity, covariance should be reserved for a dimensionally defined pattern of cooccurrence that has been shown to be unaffected by artifacts. Although most research has focused on the categorical approach of comorbidity, we focus our study on the dimensional approach of covariation. Thus, true covariance may exist, but apparent covariance may also be produced artifactually by a number of confounds. These artifacts will be detailed in the following section.

The third issue is that, even after controlling for artifacts, if two specific multiple disorders are shown to be a valid case of comorbidity, true comorbidity exists only for that specific arrangement of disorders, not for other cooccurrences. Potentially many different comorbidities may exist. This issue also applies to the dimensional approach of covariance. Rather than referring to comorbidity or covariance as a factual and general entity, we must consider systematically the specific cooccurrence of categorical or dimensional disorders on a “case by case” basis. We examine the specific covariation of externalizing and internalizing behaviors and how this covariation might differ from pure forms of externalizing and internalizing behaviors.

POSSIBLE RISK VARIABLES

Numerous risks have been identified for externalizing and internalizing problems in childhood and adolescence. Deater-Deckard, Dodge, Bates, and Pettit (1998) organized measures of risks in the present longitudinal sample into four broad domains, representing aspects of the child, sociocultural environment, parenting, and peer-group experiences. Risk variables in each of these four domains provided unique contributions to the prediction of subsequent externalizing problems. We chose risk variables that represent the same four domains.

Child Predictors: Gender and Temperament

In general, males develop externalizing difficulties more often than females (e.g., Achenbach, Howell, Quay, & Conners, 1991; Keiley, Bates, Dodge, & Pettit, 2000). In contrast, being female is a risk for internalizing problems (e.g., Achenbach et al., 1991; Walden & Garber, 1994). Several studies have shown that cooccurring externalizing and internalizing behaviors may be more likely to occur in males than in females (Graham & Rutter, 1973; Somersalo, Solantau, & Almqvist, 1999). The temperament trait of difficultness, negative emotionality, or irritability has been related to both internalizing and externalizing behaviors, and so may be a factor in their cooccurrence (Bates, Bayles, Bennett, Ridge, & Brown, 1991; Guerin, Gottfried, Oliver, & Thomas, 1994; Keiley, Bates, Dodge, & Pettit, 2002; McClowry et al., 1994). The temperament trait of resistance to control or unmanageability often predicts the development of externalizing behavior problems, but does not predict as well the development of internalizing problems (Bates et al., 1991; Biederman et al., 1990; Caspi & Silva, 1995; Hagekull, 1994; Keenan & Shaw, 1994; Keiley et al., 2002). The temperamental trait of unadaptability, behavioral inhibition, or fearfulness, which appears to underlie a disposition to anxiety, typically predicts internalizing problems, if it predicts any behavior problem, and a tendency to be avoidant (Bates, 2000; Caspi, Henry, McGee, Moffitt, & Silva, 1995; Kagan, 1998; Keiley et al., 2002; Rubin, Coplan, Fox, & Calkins, 1995).

Sociocultural Predictors: SES, Stress, and Race

Numerous studies have found that children from lower socioeconomic levels tend to have more externalizing symptoms than children from higher levels of SES (Achenbach et al., 1991; Dodge, Pettit, Bates, & Valente, 1995; Nottelman & Jensen 1995). Low SES has also been related to internalizing problems in children and adolescents (e.g., Keiley et al., 2000; Nottelman & Jensen 1995).

In general, the literature suggests that African American children have higher levels of externalizing behaviors than their counterparts (Dodge, Pettit, & Bates, 1994; Nottelman & Jensen 1995), although the effects of SES and race might be confounded (Keiley et al., 2000). Some studies have found that African American children also show more internalizing behaviors (Garrison, Jackson, Marsteller, McKeown, & Addy, 1990; Velez, Johnson, & Cohen, 1989).

Stressful life events, which tend to be more frequent at lower SES levels (Conger et al., 1992), frequently are associated with the development of both externalizing and internalizing behaviors (Deater-Deckard et al., 1998; D’Imperio, Dubow, & Ippolito, 2000). Because stressful life events, lower SES, and being African American have been related to both externalizing and internalizing behaviors, they may also be risk factors for the covariation of these symptoms.

Parenting Predictor: Harsh Parenting

Physically harsh discipline has been linked to numerous negative developmental outcomes such as externalizing and internalizing behaviors (e.g., Keiley, Howe, Dodge, Bates, & Pettit, 2001). Few studies have examined harsh parenting and cooccurring externalizing and internalizing behaviors. Fergusson, Lynksey, and Horwood (1996) did find that lack of parental attachment and a family history of offending were related to the cooccurrence of conduct problems (conduct disorder/oppositional defiant disorder) and affective disorders (depression/dysthmia).

Peer Predictors: Rejection and Neglect

Children who are rejected by peers show more externalizing behavior than those who are not rejected (e.g., Coie, Lochman, Terry, & Hyman, 1992; Keiley et al., 2000). Similarly, children who are rejected by peers have more internalizing symptoms than their nonrejected peers (e.g., Coie et al., 1992; Panak & Garber, 1992). Some evidence exists that children who have been neglected by their peers are more likely to develop internalizing symptoms (e.g., Harrist, Zaia, Bates, Dodge, & Pettit, 1997). In terms of the relationship between peer problems and cooccurring externalizing and internalizing behaviors, Wright, Zakriski, and Drinkwater (1999) found that the cooccuring group (externalizing and internalizing) evidenced poor peer relations, and in ordinary peer conversations, showed elevated levels of both aggression and withdrawal. Rudolph, Hammen, and Burge (1994) found that children with cooccurring externalizing and internalizing problems had more ratings of peer rejection than did the normal and pure internalizing groups, but equivalent to the pure externalizing group.

This Study

Although a large number of studies exist that report various risks for externalizing and internalizing problems, most of them have focused on only one disorder, therefore the risks investigated and the etiological and developmental theories based on such data may not be specific to that disorder. To test the specificity of risk variables, they must be examined for different problem behaviors (e.g., externalizing and internalizing symptoms) within a single study. Another limitation is that most studies have failed to take into account cooccurring conditions, let alone whether those conditions are cases of true or apparent comorbidity/covariance. Once again, without an acknowledgment and careful examination of cooccurring conditions, erroneous conclusions and theories with poor internal validity can result. To test whether risk variables are related to specific psychopathological conditions, they must be examined for different pure-form and cooccurring problem behaviors (e.g., externalizing and internalizing symptoms), while controlling for as many artifacts as possible.

We attempt to reduce some of the more serious artifactual confounds of cooccurrence and investigate whether a factor of covarying externalizing and internalizing behaviors exists in mother reports of these symptoms (MC) over a period of 9 years, in addition to the expected pure internalizing (MI) and pure externalizing (ME) factors. For teacher reports, we examine the same three factors—two “pure” form factors (TI and TE) and a covarying factor (TC). We explore how child characteristics (gender, temperament), sociocultural environment (SES, race, stress), parenting (harsh discipline), and peer experiences (rejection, neglect) might be related to these factors of pure internalizing (MI, TI), pure externalizing (ME, TE), and shared or covarying internalizing/externalizing (MC, TC) for both mothers’ and teachers’ reports.

METHOD

Sample

The Child Development Project (CDP; Dodge et al., 1994) is a three-site longitudinal study of a community sample of 585 families who had 5-year-old children entering kindergarten in 1987 and 1988. Parents at a large number of schools at each site were approached randomly during preregistration for kindergarten by research staff inviting them to participate. About three fourths of the parents approached agreed to participate. Fifteen percent of the sample did not preregister. These participants were recruited during the 1st week of kindergarten through the schools, mail, or telephone. Data were collected each year from kindergarten to eighth grade. Eighty-three percent of the sample was European American and 17% were African American, 48% were female and 52% were male. The mean level of Hollingshead SES was 39.7. Approximately 25% of the sample did not participate in the eighth-grade assessment. We examined statistics for attrition effects on demographic variables, outcome and predictor variables at time one between respondents who were missing data and those who were not. No differences were found. The sample statistics are displayed in Table I.

Table I.

Sample Statistics of Variables for the Confirmatory Factor and Structural Equation Models (n = 585).

Variable	N	%	Mean (SD)	Range
Child variables
Gender
Female	281	48
Male	302	52
Temperament
Difficultness			3.30 (0.95)	1.11–6.38
Resistance to control			3.58 (1.12)	1.00–7.00
Unadaptability			2.99 (1.09)	1.00–6.25
Socioculural environment
SES			39.64 (14.03)	8.00–66.00
Race:
European American	475	81
African American	108	19
Stress			3.02 (0.98)	1.00–5.00
Parenting
Harshest punishment			1.69 (1.07)	0.00–4.00
Peer relationships
Rejected	67	12
Neglected	70	12

Measures

Outcome Variables

The outcome variables were mother- and teacher-reported externalizing and internalizing behavior scale scores from the 112-item Child Behavior Checklist (CBCL; Achenbach, 1991b) and Teacher Report Form (TRF; Achenbach, 1991c), with items rated 0 (rarely/never), 1 (sometimes), or 2 (often). Mothers completed the questionnaires in the summer or early fall of the school year; teachers did so in the winter or early spring of the same year. Externalizing behaviors are those considered to be aggressive, disruptive, or delinquent, whereas internalizing behaviors are those considered to be withdrawn, anxious, somatic, or depressed. The externalizing behavior scale is composed of 33 items in the CBCL and 34 items in the TRF The internalizing behavior scale is composed of 31 items on the CBCL and 35 items on the TRF. Because the mother- and teacher-report externalizing and internalizing scale scores were highly skewed toward the low end of the scales, we transformed the scores by adding one and taking the natural logarithm before further analysis.

Child Predictors

Gender was represented by a dummy variable, coded “ 1” for females and “0” for males. Temperament measures were from maternal report at age 5 on the Retrospective Infant Characteristics Questionnaire (RICQ; Bates, Pettit, Dodge, and Ridge, 1998), based on the Infant Characteristics Questionnaire (Bates & Bayles, 1984; ICQ; Bates, Freeland, & Lounsbury, 1979), which asked the mother to rate the child’s temperament traits during his/her infancy; Difficultness or irritability contained nine items pertaining to the tendency to fuss and cry frequently and intensely, expression of negative effect, and demandingness (α = .86). Resistance to control or unmanageability had three items pertaining to persistence in playing with an object or going someplace when told not to and upset when removed from something (α = .83). Unadaptability or behavioral inhibition had four items indexing negative responses to novel people, situations, foods, and eventual adaptation to novelty (α = .72). Bates et al. (1998) found a moderate degree of linkage from mother-reports in infancy to their retrospective reports at age 10 years.

Sociocultural Predictors

Family SES (Hollingshead, 1975) was calculated according to the Hollingshead Four-Factor Index, combining mother’s and father’s years of education and occupation. When no father lived at home, the mother’s responses were double-weighted. Race was represented by a dummy variable, coded “1” for European American and “0” for African American. A composite variable of stress was computed from variables assessed during the interview with the mother in which she was asked whether the family had experienced each of 10 life events during the years from birth to 4 years old and from 4 to 5 years old (Changes and Adjustments Scale; Dodge et al., 1994). These events included moving or major remodeling of the house, child-medical problems, close family medical problems, death of an important person, divorce/separation of parents, parent/target child separation, financial problems, legal problems, extended family stress, and addition/deletion of a sibling. The interviewer used a 5-point scale (1 = minimal challenge, 3 = moderate, and 5 =severe, frequent challenges) to rate the extent to which the child and the family faced these events during the two different eras. These ratings were then averaged across the two eras. An alpha of .64 and interrater agreement of .79 were reported (Dodge et al., 1995).

Parenting Predictor

Harsh punishment was assessed by the interviewer asking whether the parents spanked the child and what was the most intense physical punishment they had used during the year the child was 4–5 years old. The interviewer used a 5-point scale (0= no physical punishment, 1= hand on clothes, 2= hand on skin, 3 = object on clothes, and 4 = object on skin) to evaluate the level of harsh punishment.

Peer Predictors

During peer sociometric interviews in kindergarten, children were shown pictures of each classmate and asked to rate how much he/she liked that classmate on a 5-point scale. Next, the child was asked to nominate up to three peers as especially liked and three peers as especially disliked. A mean popularity rating was calculated as the mean rating received from all classmates (scale = 1–5). Frequencies of liking and disliking nominations were summed and standardized within classrooms. A social preference score was calculated by subtracting the disliking score from the liking score and a social impact score was computed by adding the liking and disliking scores and then standardizing within classrooms. Children who received social preference scores below −1, liking scores below 0, and disliking scores above 0 were classified as rejected by their peers. Peer-rejection in kindergarten was represented as a dummy variable, coded “1” for rejected and “0” for nonrejected. Children who received social impact scores below −1, liking scores below 0, and disliking scores below 0 were classified as neglected by their peers. Peer-neglect in kindergarten was represented as a dummy variable, coded “1” for neglected and “0” for non-neglected.

Analysis

Using the structural equation modeling program, MPlus (Muthen & Muthen, 1998), we fit a confirmatory factor model to represent how much of the variance in mother’s and teacher’s externalizing and internalizing ratings over 9 years could be considered covarying, purely externalizing, and purely internalizing for each of the two raters. In confirmatory factor analysis (CFA), a hypothesized model is fit to observed data. In our CFA model (Fig. 1), we hypothesized that the observed scores for mother-reported externalizing behaviors for 9 years (kindergarten through eighth grade) loaded on two factors, the pure externalizing factor (ME) and the covariation factor (MC) for mother-reported symptoms. In like manner, we hypothesized that the observed scores for mother-reported internalizing behaviors over these same 9 years loaded on two factors, the pure internalizing factor (MI) and the same covariation factor (MC) for mother-reported symptoms. We similarly hypothesized teacher scores loaded on three factors, a pure externalizing (TE), a pure internalizing (TI), and a covariation (TC) factor. The covariation factor for mother reports represents the amount of covariance of the mother’s internalizing and externalizing scores over the 9 years. The pure internalizing factor represents the part of internalizing scores over 9 years that does not covary with externalizing scores or is not error. The pure externalizing factor represents the part of externalizing scores over the 9 years that does not covary with internalizing or is not error. The teacher factors can be understood similarly.

Fig. 1.

Path diagram of the confirmatory factor model of mother- and teacher-reported externalizing and internalizing behaviors with six latent factors. Only the corresponding factors across rates are correlated with each other.

In CFA, covariation between factors can also be estimated. Because we were hypothesizing that for each rater three distinct factors existed, we allowed only the corresponding factors across raters to correlate with each other (see Fig. 1). That is, the ME factor was allowed to correlate only with the TE factor, the MI factor only with the TI factor, and the MC factor only with the TC factor, thus creating three sets of orthogonal factors—externalizing factors, internalizing factors, and covarying factors. The CFA model partials the variance in the observed variables (the 9 years of CBCL and TRF externalizing and internalizing scores, represented in Fig. 1 by squares) into latent factors (ME, MI, MC, TE, TI, TC—represented by circles) and random error terms (represented by small arrows pointing to the observed scores). By allowing the error terms of teacher reports of externalizing and internalizing to covary at each time point, and the errors of mother reports of externalizing and internalizing to covary at each time point, we control for informant effects (Hinden et al., 1997). The correlations between the errors at each time point for teacher reports of externalizing and internalizing scores have been omitted from Fig. 1 for ease of interpretation. Similarly, the correlations between the errors at each time point for mother reports have been omitted. These correlations represent informant effects. The hypothesized correlations between specified latent factors (ME and TE, MI and TI, MC and TC) are represented by two-headed arrows. This hypothesized CFA model was fit to the data. After assessing the model fit, we determined whether each of the six latent factors contained a significant amount of variance that could then be predicted by our four sets of risk variables. That is, we then determined how our chosen risk variables correlated with the six latent factors that represented both pure and covarying factors that underlie the mother- and teacher-reported externalizing and internalizing scores.

To determine the relationships between the selected risk variables and the six latent factors in the CFA model, two series of structural equation models (SEM) were fit. In the first series, each of the 10 risk variables was added to the CFA model by itself to determine if, separately, it was correlated with the six latent factors. For example, the risk variable, gender, was allowed to predict simultaneously all six latent factors (MI, ME, MC, TI, TE, TC). In effect, this SEM is a set of six simultaneous regression equations in which each equation represents the regression of one of the six latent factors on gender. After determining the effects of this predictor, gender, on the six latent factors, it was removed and the second predictor was entered and allowed to predict the six latent factors, and so on. In this first set of models, the six latent factors were no longer orthogonal and unrelated, only their residuals were orthogonal; that is, even though we still only allowed corresponding factors across raters to correlate with each other (e.g., ME with TE), through the paths representing the regression of the six latent factors on each predictor, the latent factors actually did covary with all of the other factors (e.g., ME with TC through the gender paths). From the examination of the unpartialed correlations from these simultaneous regressions of the six latent factors on each risk variable, we were able to determine which risk variables might be related to each of the six latent factors when these factors were not orthogonal.

Because our main interest was in the relationships between our predictors and sets of latent variables that were orthogonal or unrelated to each other, we then fit a second series of SEM. In this second series, we added the 10 risk variables, in groups of variables that denoted the child (gender, temperament—difficultness, resistance to control, unadaptability), sociocultural (SES, race, and stress), parenting (harsh punishment), and peer environments (rejection, neglect), to three separate SEM. In the first SEM, we allowed these 10 variables to predict only ME and TE, thus retaining the orthogonality of the remaining latent factors (MI, TI, MC, and TC); in this model only corresponding factors across raters correlated with each other as was hypothesized in Fig. 1.⁸ In the second SEM, the 10 variables predicted only MI and TI, while in the third SEM, they predicted only MC and TC with the continuing constraint that only corresponding factors across raters correlated with each other. Each of the three models fit represented the simultaneous regression of two related latent factors (e.g., ME and TE) on the 10 risk variables. See Fig. 2 for a representation of one of these SEM, the one in which MC and TC are predicted by the 10 variables.

Fig. 2.

Path diagram of the SEM model of mother- and teacher-reported externalizing and internalizing behaviors with six latent factors with only two of the factors (MC and TC) predicted by 10 risk variables from four domains (child: gender, temperament—unadaptability, resistance to control, difficultness; sociocultural environment: SES, race, stress; parenting: harsh punishment; peer relations: rejection, neglect). Only the corresponding factors across raters are correlated with each other.

MPLUS allowed us to include respondents with missing data by using maximum likelihood estimation (Little & Rubin, 1987). The coverage (amount of data that were not missing) for the outcome variables ranged from 56 to 99% and for the predictors from 66 to 100%.

RESULTS

The CFA model fit the data well (χ²/df= 2.33; CFI = .92; TLI = .92; RMSEA = .05, p = .86; SRMR = .07).⁹ Each of the six latent factors contained significant variance (mother reports: ME variance, σ² = .159, p < .001; MI variance, σ² = .132, p < .001; MC variance, σ² = .280, p < .001; teacher reports: TE variance, σ² = .638, p < .001; TI variance, σ² = .074, p < .01; TC variance, σ²= .153, p < .001). The standardized regression loadings of the latent factors on the observed variables measure the degree to which the observed measures converge on the factors; that is, these loadings are similar to convergent validity coefficients. In this CFA, the standardized loadings ranged from .48 to .51 for ME, .41 to .45 for MI, .60 to .67 for MC, .62 to .69 for TE, .28 to .29 for TI, and .32 to .42 for TC. The squared multiple correlations indicate how much of the variation in the observed variables has been included in the six latent factors. For the mother’s externalizing scores the squared multiple correlations ranged from .62 to .71, for mother’s internalizing scores, .53 to .65, for teacher’s externalizing scores, .48 to .60, and for teacher’s internalizing scores, .23 to .26. In addition, significant correlations existed among latent factors that were allowed to covary. MC and TC were moderately correlated (r = .36, p < .001), as were MI and TI (r = .31, p< .01). ME and TE were the most highly correlated latent factors (r = .75, p < .001). The six latent factors did contain significant variance, therefore we could build the two sets of SEMs to determine how the risk variables related to these six latent factors. In addition, we examined the factor scores from the CFA model for the respondents on each of the six latent constructs. The estimated means of these factor scores were all zero and the distributions were symmetric, indicating that severity was controlled in our fitted model.

In the first series of SEM, each of the 10 risk variables was added to the model by itself to determine if, separately, it was correlated with the six latent factors. Each of these 10 models fit the data (Ranges: χ²/df= 2.2–2.3; CFI = .92–.93; TLI = .92; RMSEA = .05, p = .92–.96; SRMR = .07). To determine if each risk predictor by itself was significantly predicting the variance in the latent factors, we fit a reduced model in which that predictor’s effects on the six latent factors were constrained to be zero and conducted the appropriate Δχ² test. The delta chi-square statistics (with their associated change in degrees of freedom) indicated that the six latent factors did depend separately upon each of the 10 risk variables. These unpartialed correlations representing the regression coefficients of the, now, nonorthogonal latent factors on each risk predictor are found in parentheses in Table II, which will be described below.

Table II.

Standardized Parameter Estimates From the Three Structural Equation Models of the six Variance Factors, Two Related Factors at a Time (ME with TE, MI with TI, MC with TC), Regressed on All Risk Predictors.

Predictor	Unique mother externalize	Unique teacher externalize	Unique mother internalize	Unique teacher internalize	Covarying mother	Covarying teacher
Female	−.17*** (−.08)	−.29*** (−.31***)	.13* (.20***)	.16* (.06)	.07 (− .05)	−.17** (−.02)
Difficult	.09 (.09)	−.01 (.02)	−.01 (.02)	−.04 (−.01)	.11*(.19***)	−.04 (.03)
Resistant	.29*** (.31***)	.14** (.17**)	−.08 (.06)	−.04 (.06)	.22*** (.20***)	.12 (.06)
Unadaptable	−.23*** (.07)	−.18*** (.03)	.21*** (.31***)	−16* (.49***)	−.01 (− .06)	−.06 (− .30**)
SES	−.25*** (−.40***)	−.22*** (.01)	.11 (−.12)	−.34*** (.45***)	−.06 (.08)	−.50*** (− .72***)
White	.06 (− .04)	−.12** (−.19**)	.05 (.10)	.04 (−.10)	.18*** (.03)	−.08 (−.10)
Stress	.13** (.11)	.05 (.08)	.08 (.01)	.10 (.07)	.20*** (.22***)	.13* (.18***)
Harsh punishment	.22*** (.18**)	.10* (.17***)	.01 (−.06)	−.07 (−.11)	.24*** (.24***)	.05 (.23**)
Peer neglect	−.01 (−.03)	−.05 (− .04)	.02 (.03)	.20** (.26**)	.01 (.01)	.11 (−. 06)
Peer reject	.17*** (.20***)	.24*** (.23***)	.07 (−.01)	.23** (− .05)	.13** (.15**)	.43*** (.38***)
R2%	43.0	7.9	26.4	35.0	27.3	67.1

Note. In parentheses, unpartialed standardized parameter estimates from separate structural equation models of the six variance factors regressed, simultaneously, on each predictor entered singly (n = 585).

^*p < .05.

^**p < .01.

^***p < .001.

In the second series of SEM, we added the 10 risk variables, in groups of variables that denoted the child, sociocultural, parenting, and peer environment variable, to three separate SEM as described earlier. In the first SEM, they predict only ME and TE, in the second they predict only MI and TI, and in the third, they predict only MC and TC. In each of these models only corresponding factors across raters were allowed to correlate with each other. For each of these three SEM, as we entered a group of variables, we fit a reduced model in which those predictors’ effects on the latent factors were constrained to be zero and conducted the appropriate Δχ² test. In each of the three SEM, the Δχ² tests indicated that the three pairs of latent factors did depend upon the child, sociocultural, parenting, and peer variables. The three final models fit the data well (Ranges: χ²/df= 2.0–2.2; CFI = .89–.91; TLI = .89–.91; RMSEA = .04–.05, p = .99–1.00; SMSR = .07–09). Table II presents the standardized parameter estimates for the six partially orthogonal latent factors predicted by the 10 risk variables from these three models in the second series of SEM. In addition, the unpartialed correlations from the first series of SEM of the nonorthogonal six latent factors regressed on the 10 risk factors entered separately are presented in parentheses.

In the first model in this second series of SEM, the pure externalizing problem factors, ME and TE, were predicted by the 10 risk variables. Both of these factors were predicted significantly by several child variables: gender (−.17, p < .001 for ME and −.29, p < .001 for TE),¹⁰ Unadaptable temperament (−.23, p < .001 for ME and − .18, p < .001 for TE), and Resistance to Control (.29, p < .001 for ME and .14, p < .01 for TE). The sociocultural variable, SES, significantly predicted both ME (−.25, p < .001) and TE (−.22, p < .001). The sociocultural variable, ethnicity, predicted TE (−.12, p < .01) and the last sociocultural variable, stress, predicted ME (.13, p < .01). Harsh punishment was a significant parenting variable for both ME (.22, p < .001) and TE (.10, p < .05), as was the peer variable, peer rejection, (.17, p < .001 for ME and .24, p < .001 for TE). In this first final fitted model, significant correlations still existed among the factors that were allowed to correlate. The ME and TE factors were still the most highly correlated (r = .42, p < .001), but at a lower level than in the baseline factor model (r = .75, p < .001), because by adding predictors we had partialed out, not only variance in the two factors that was related to the predictors, but also covariance between the these two factors. The MI and TI factors (r = .41, p < .001) and the MC and TC factors (r = .27, p < .01) were somewhat similarly correlated in this model as they were in the CFA model (r = .31, p < .01 and r = .36, p < .001, respectively).

In the second model, in which MI and TI were predicted, the child variables, gender (.13, p < .05 for MI and .16, p < .05 for TI) and Unadaptable temperament (.21, p < .001 for MI and .16, p < .05 for TI), had significant effects. Only TI was predicted significantly by any of the other variables, the sociocultural variable, SES (−.34, p < .001), and the two peer environment variables, peer neglect (.20, p < .01), and peer rejection (.23, p < .01). In this second final fitted model, the MI and TI factors were correlated (r = .31, p < .01) at the same level as in the baseline factor model; although we had partialed out variance in the two factors that was related to the predictors, we had not predicted much of that variance, thus their covariance was not reduced. The ME and TE factors (r = .72, p < .001) and the MC and TC factors (r = .35, p < .001) were similarly correlated in this model as they were in the CFA model (r = .75, p < .001 and r = .36, p < .001, respectively).

In the third model, in which MC and TC were predicted, the child variable, gender, only had an effect on TC (−.17, p < .01), while two of the child temperament variables, Difficult (.11, p < .05) and Resistance to Control (.22, p < .001) had an effect on the MC factor. The sociocultural variable, family stress, predicted both the MC (.20, p < .001) and the TC factors (.13, p < .05), whereas SES predicted TC (−.50, p < .001) and ethnicity predicted MC(.18, p < .001). Harsh parental punishment only predicted MC (.24, p < .001), but peer rejection predicted both MC (. 13, p < .01) and TC (.43, p < .001). In this third final fitted model, the MC and TC factors were no longer significantly correlated (r = .10, p > .05), while in the CFA model they were (r = .36, p < .001). We had partialed out so much of the variance in the two factors that the covariance between them became nonsignificant. The ME and TE factors (r = .72, p < .001) and the MI and TI factors (r = .45, p < .001) were similarly correlated in this model as they were in the CFA model (r = .75, p < .001 and r = .31, p < .01, respectively).

To determine how well the risk predictors explained the variance in each of the six factors, as we entered the four sets of risk variables, we examined the R² statistics for the six latent factors that are provided by Mplus (Table III). In SEM the R² statistics provided indicate the proportional reduction in variance in the dependent latent factor upon entering the predictors. These statistics are constructed by calculating the difference between the variance in a latent factor in the baseline CFA model (with no predictors) to the variance in that same factor after entering the predictor and dividing this difference by the original variance ( $[{\sigma }_{\text{baseline}}^2-{\sigma }_{\text{final}}^2]/{\sigma }_{\text{baseline}}^2$). For example, from Table III, child risk variables account for 21.7% of the variance in the ME factor. Upon adding the sociocultural variables, the total amount of variance accounted for rises to 34.1 %. Hence, the amount of variance accounted for by adding the sociocultural variables is 12.4% (34.1–21.7%). Similarly, the amount of variance accounted for upon the addition of each group of risk variables can be calculated for each of the six latent factors. In the final three fitted models represented in Table II and in the last row of Table III, these 10 risk predictors reduced the variance in the ME factor by 43.0% and in the TE factor by 35.0%, in the MI factor by 7.9% and in the TI factor by 27.3%, and in the MC factor by 26.4% and the TC factor by 67.1 %.

Table III.

Cumulative Proportional Reduction in Variance (In Percent) Attributable to the Four Groups of Predictors in the Regression of the six Variance Factors, Two Related Factors at a Time (ME with TE, MI with TI, MC with TC), Regressed on All Risk Predictors (n = 585)

Risk predictor group	Unique mother externalize	Unique teacher externalize	Unique mother internalize	Unique teacher internalize	Covarying mother	Covarying teacher
Group I: Child risk variables	21.7	15.7	6.0	12.5	11.4	3.6
Group II: Sociocultural risk variables	34.1	27.4	7.7	21.6	18.9	43.6
Group III: Parenting risk variable	39.6	28.7	7.9	22.0	24.5	45.4
Group IV: Peer risk variables	43.9	35.0	7.9	27.3	26.4	67.1

Finally, we briefly mention another approach to modeling prediction of the pure and covarying components of reports of child adjustment. We had originally fit a series of models in which the four predictor groups were added to the baseline model one group at a time, but were allowed to predict all of the latent factors simultaneously. After entering each group of predictors, we retained only the significant predictors, before entering the next group of predictors. The parameter estimates for this final fitted model were similar to the parameter estimates in parentheses in Table II, but a few discrepancies existed because of the nonorthogonal nature of the latent factors. The factors were now nonorthogonal through the predictors, because we allowed the predictors to predict all of the six latent constructs at once. In this alternative model, it turned out that the only factors that were not correlated were the externalizing and internalizing factors within raters; that is, no correlation existed between ME and MI or between TE and TI. In this alternative, nonorthogonal model, results were predominantly congruent with our main model presented in Table II; however, no effect existed for gender on ME, TI, or TC, unlike in the main model. The effect of unadaptability on both MC and TC was negative and significant, as in the main model, but none existed for ME or TE. The effect of socioeconomic status on TI was positive, not negative, as in the main model. Stress did not have an effect on ME and peer rejection did not have an effect on TI, unlike the main model.

DISCUSSION

Confirmatory factor analysis showed that a covariation factor of externalizing and internalizing behaviors existed for both mother- and teacher-reported symptoms on the Achenbach checklists (MC, TC). In addition, factors of pure externalizing (ME, TE) and pure internalizing (MI, TI) existed in the two domains of mother- and teacher-reported behaviors. In our three final fitted SEM models, between 8 and 67% of the variance in these six latent factors was accounted for by the child, sociocultural, parenting, and peer risk variables.

The relationships between the risk variables and the externalizing, internalizing, and covariation factors are controlled for all of the other variables and factors in the model. For example, the positive relationship between the temperament variable, Unadaptability, and the MI and TI factors indicates that children who were rated by their mothers and/or teachers as unadaptable had internalizing factor scores that were significantly higher than those for children who were not unadaptable. These relationships, however, do not preclude the possibility that unadaptability might also have relationships with other components of adjustment, because we did not allow the MI or TI factors to correlate with the ME, TE, MC, or TC factors. Thus, children who were high on one factor may be high or low on other factors. And, in fact, unadaptability also predicted lower levels of both mother- and teacher-rated externalizing behavior.

Effects of Child Risk Variables

Gender was a risk predictor for all but one factor, MC. Girls tend to develop more internalizing and fewer externalizing problems than do boys (e.g., Achenbach et al., 1991; Dodge et al., 1994; Graham & Rutter, 1973; Keiley et al., 2000; Somersalo et al., 1999). The teachers noted that girls showed less covarying externalizing and internalizing behavior than did the boys. The mothers’ ratings did not show this effect. Teachers may be aware that girls who are externalizing are also likely to have symptoms of an internalizing disorder, while mothers may not. Or, it is also likely that children behave differently in school than at home.

The child temperament variable of difficultness was related only to MC. Research has shown that difficultness is related to both externalizing and internalizing symptoms (Rothbart & Bates, 1998; Rubin et al., 1995), but no previous studies have shown that this temperament variable is related to actual covariance. Replication will be critical, but the findings may mean that negative emotionality and demandingness are a core part of the overlap between externalizing and internalizing symptoms as perceived by mothers. Temperamental resistance to control was highly related to both the ME and TE factors, as well as to MC factor. We would expect the former relationships from previous research studies (Bates et al., 1998; Deater-Deckard et al., 1998; Rubin et al., 1995). Children who are resistant to control tend to develop externalizing difficulties. In terms of the relationship of resistance to control and MC, perhaps mothers observe that their aggressive, misbehaving children also experience anxiety and sadness about the social consequences of their inability to control their behavior. Or perhaps temperamental resistance to control marks a tendency to dysregulation of both behaviors and emotions. The finding that unadaptability was positively related to both MI and TI factors and negatively related to both ME and TE factors supports what has already been shown in previous research (e.g., Bates, Pettit, & Dodge, 1995; Keiley et al., 2002; Rothbart & Bates, 1998; Rubin et al., 1995); that is, children who are fearful and inhibited tend to develop internalizing symptoms, and are less likely, over time, to show externalizing symptoms.

Effects of Sociocultural Risk Variables

SES was related to the ME and TE factors; children from lower SES families were rated by their mothers and teachers as being more externalizing than were the children from higher SES families. This finding supports previous research that lower SES is a risk factor for externalizing behaviors (Achenbach et al., 1991; Dodge et al., 1995). In addition, SES was related to the TI, but not the MI, factor; only teachers rated children from lower SES families as being more internalizing than children from higher SES families. Taken together with the finding that SES was also related to the TC factor, perhaps teachers are more aware of the effects of SES on child functioning; children who appear internalizing or both externalizing and internalizing are more likely to be from lower SES environments than children who are not evidencing these behaviors. This finding replicates several studies in which lower SES has been shown to be a risk factor for both externalizing and internalizing behaviors (see review by Nottelman & Jensen, 1995), Less is known about the relationship between SES and covarying conditions.

The relationship between race and the MC factor has not been found in previous research. The European American children were seen by mothers as having higher levels of covarying internalizing and externalizing behaviors than were the African American children. Perhaps, as previously suggested (Keiley et al., 2000), African American families provide some protective factors for their children against the development of the combination of externalizing and internalizing behavior, Alternatively, African American and European American mothers may ascribe different meanings to the symptoms listed in the CBCL. For example, perhaps European American mothers are more apt to see acting out behavior as reflecting an underlying emotional problem. The relationship between race and the TE factor has been found in previous studies (Achenbach et al., 1991; Dodge et al., 1994; Graham & Rutter, 1973; Keiley et al., 2000; Somersalo et al., 1999); European American children were rated by teachers as being less externalizing than the African American children.

The relationship between stress and both the MC and TC factors suggests that children who have experienced very stressful life events are showing more covarying externalizing and internalizing behaviors. A relationship between stress and externalizing and internalizing behaviors, although not necessarily concurrent behaviors, consistently has been found (Kaslow, Deerin, & Racusin, 1994; Weiss, Dodge, Bates, & Pettit, 1992). Perhaps stress simultaneously contributes to dysregulation of both behaviors and emotions. The relationship between the TC factor and stress was less pronounced than was the relationship for the mother reports. Perhaps this difference was due primarily to differing behaviors at home versus school. The relationship between stress and ME is similar to what has been reported in the literature (Deater-Deckard et al., 1998; D’Imperio et al., 2000).

Effects of Parenting Risk Variable

Replicating previous research, harsh parental punishment was related to both the ME and TE factors (Dodge, Pettit, & Bates, 1997; Keiley et al, 2001). Children who experience harsh punishment are more likely to develop externalizing difficulties than are those children who are not physically punished. And, alternatively, children whose behavior is challenging are more likely to be treated harshly by parents. In addition, the relationship between harsh punishment and the MC factor indicates that those children who have experienced this form of punishment are more likely to show both internalizing and externalizing behaviors. Although, the relationship between physical punishment and both internalizing and externalizing behaviors is common (Kaslow et al., 1994; Weiss et al., 1992), this may be the first finding of a relationship with actual covarying behavior.

Effects of Peer Risk Variables

A fascinating finding was the relationship between peer neglect and the TI factor. Only a few studies have reported evidence of this theoretically obvious relationship (e.g., Harrist et al., 1997). Children who are neglected by their peers appear to suffer from more internalizing symptoms than do their nonneglected peers, at least according to teachers. Perhaps our finding sheds new light on the meaning of peer neglect. This is another instance of disparity between teachers’ and mothers’ sensitivity to the consequences of peer relationships. Teachers, much more than mothers, see the peer relations that give rise to the sociometric ratings, and thus may be more likely to see the internalizing symptoms related to peer neglect.

However, peer rejection was related to the pure externalizing latent factors of both teachers and mothers, which is consistent with prior literature (e.g., Deater-Decker et al., 1998); that is, rejection is a function of aggressive behavior and high agreement exists between mothers and teachers on levels of externalizing behavior. Peer rejection also was related to both the MC and TC factors, although more strongly for the teachers. As previous research has noted, children who are rejected by their peers are more externalizing, but often experience depression, anxiety, and other internalizing symptoms because of their lack of social contact (Loeber & Keenan, 1994), Perhaps, at school, in relationships with their peers, rejected children not only act out more, but also evidence more internal symptoms as they struggle with the effects of their aggressive behavior on their relationships and their school work. Or, perhaps children’s tendencies to be anxious and withdrawn at the same time as disruptive and aggressive are of particular annoyance to peers. The relationship between peer rejection and the TI factor replicates previous research (e.g., Coie et al, 1992; Panak & Garber, 1992). This relationship seems more likely to reflect the consequences of rejection rather than the causes of it, but perhaps the pure TI factor also signifies some irritating negative emotionality. Internalizing as perceived by mothers was not associated with peer sociometrics, which may be a function of a home–school difference in children’s internalizing behavior.

Limitations and Future Research

This study has produced many findings that converge with previous studies. At the same time, the statistical approach used here supplements most previous studies by separating the covarying and pure syndromes and thus indicating risks that might be specific to the covarying versus pure syndromes. Nevertheless, the current study’s focus on two broad-band syndromes of externalizing and internalizing behaviors may have obscured more fine-grained patterns of true covariation. Previous research on narrow-band syndrome comorbidity (McConaughy & Skiba, 1993; Verhulst & van der Ende, 1993) suggests the need to look at more narrowly defined patterns of cooccurrence.

As was mentioned in our introduction, although our study focused on the dimensional approach of covariation, most research has focused on the categorical approach of comorbidity. Instead of looking at dimensional factors and related risks, we could examine risk variables in relation to groups of children who “have” and “do not have” diagnostic levels of pure-form and comorbid externalizing and internalizing problems. A future paper by the same authors will explore the specificity of risks, while reducing potential sources of artifact, via a categorical approach.

An additional limitation is that we examined externalizing and internalizing symptoms summarized across 9 years rather than at separate developmental periods (e.g., early childhood, middle childhood, early adolescence), thus we have not investigated possible differences in terms of symptoms and associated risks during these different eras. This is another topic for future research.

This study examined a very comprehensive pool of risk variables, but not an exhaustive one. The selection of risk factors may have been slanted toward externalizing problems because this study used data from the CDP, which itself was primarily a study of externalizing problems and may not have been sensitive to risk factors commonly associated with internalizing problems. In addition, we did not measure genetic variation. As previous research has suggested (Gjone & Stevenson, 1997; O’Connor, McGuire, Reiss, Hetherington, & Plomin, 1998), genetic components may be influential in increasing vulnerability to both externalizing and internalizing dimensions.

Many of this study’s measures of risk were based on mothers’ retrospective and concurrent reports assessed through an interview and questionnaires at entry to kindergarten. Because mothers were informed by the interviewers that any questionable conduct reported to them would be relayed to state and county authorities, and perhaps also wanting to present their families in a positive light, some mothers may have withheld or distorted information. It seems more likely, however, that this would cause our results to underestimate rather than overestimate relationships of risk variables with externalizing and internalizing outcome variables. Note that the outcome variables were assessed by both mothers and teachers, which reduces concerns about the overall pattern of predictor-outcome relations being simply a rejection of informant bias.

The respondents in this study were only at low to moderate risk for the development of externalizing and internalizing symptoms, hence our results may not extend to high-risk samples and severely disturbed children. Even though one of the main aims of this study was to identify risks associated with covarying and pure-form externalizing and internalizing problems, we did not examine the exact developmental processes that lead to these outcomes. Not all variables that have been associated with psychopathology are risks; some of them may be concomitants or even consequences of psychopathology. Although the current study’s pool of risk variables was assessed at age five and used to examine outcomes from ages 5–14, some of these risk variables may be consequences of psychopathology during or before kindergarten. This study did not examine the full breadth of potential combinations of risks across all four domains. As previous research has suggested (e.g., Bates et al., 1998), risks operate interactively in the development of psychopathology. Finally, because we only examined risk variables in kindergarten, we cannot determine, from our findings, whether these risks were related to later externalizing and internalizing problems because of their impact on early development or because of their association with other risks that persist throughout childhood.

CONCLUSIONS

The issue of cooccurrence has generated a great deal of research over the past decade, but few studies have examined associated risk factors nor the specificity of such risks, both of which are essential in generating internally valid theories of cooccurrence and pure-form problems behaviors. Even fewer studies have taken a dimensional approach to cooccurrence and rarely have researchers attempted to evaluate and reduce methodological artifacts within this area. This study therefore offers a significant contribution to the field because it represents the most comprehensive and detailed examination of specific risk factors for cooccurring and pure form externalizing and internalizing behaviors, employs an advanced statistical analysis of covarying behaviors, and attempts to reduce certain methodological artifacts.

Despite several limitations, we have been able to determine that within our sample a unique factor of covarying externalizing and internalizing behaviors existed in mother- and teacher-reports of these symptoms over a 9-year period. We were also able to reduce most of the serious artifactual confounds of covariance as we examined the risks that might be associated with the covarying, pure externalizing, and pure internalizing factors. Because our findings suggest that covarying externalizing/internalizing conditions can be thought of as more similar to pure externalizing behaviors than to pure internalizing, treatments targeting the risks associated with cooccurrence may most productively apply treatments found to be effective for externalizing problems. Future research should explore not only the efficacy of such treatments for children with covarying externalizing and internalizing problems, but the development of etiological theories for these behaviors. As was demonstrated in this study, research on both apparent and true covariance/comorbidity offers the field an excellent opportunity to uncover a more valid but also more complex picture of psychopathology than currently exists, and consequently the prospect of more effective prevention and intervention programs.