Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2013 May 1.
Published in final edited form as: Compr Psychiatry. 2011 Jul 14;53(4):323–332. doi: 10.1016/j.comppsych.2011.05.008

Examination of the structure of psychopathology using latent class analysis

Thomas M Olino 1, Daniel N Klein 2, Richard F Farmer 3, John R Seeley 4, Peter M Lewinsohn 5
PMCID: PMC3196784  NIHMSID: NIHMS300803  PMID: 21757192

Abstract

A number of recent studies using factor analytic methods find that the structure of psychopathology reflects broad internalizing and externalizing dimensions, with the internalizing dimension being further divided into fear and distress disorders. While these variable-centered studies have provided important insights into the structure of psychopathology, they provide limited information about the classification of individual cases. The present study examines patterns of lifetime internalizing and externalizing psychopathology in participants from the Oregon Adolescent Depression Project using latent class analysis, which classifies individuals rather than variables. A four class solution best fit the data. The largest class (62.5%) included individuals with relatively little psychopathology; one class (16.4%) was largely characterized by internalizing disorders; one class (16.9%) was largely characterized by externalizing disorders; and the final class (4.2%) was characterized by both internalizing and externalizing disorders. The validity of the classes was further examined using data on psychiatric morbidity, temperament, and family aggregation of psychopathology. Classes differed on indices of positive, negative, and disinhibited temperament in ways that were consistent with theoretical predictions. Patterns of familial aggregation of psychopathology demonstrated relative specificity of transmission of different disorders. Overall, the findings support conclusions from studies of dimensional models of internalizing and externalizing disorders, and extend them to person-centered approaches to classification.

Keywords: psychopathology, structure, personality, latent class analysis

Several studies on the structure of internalizing and externalizing psychopathology conducted over the last decade have largely relied on variable–centered methodologies such as factor analysis for investigating structural associations among various forms of psychopathology [1–6]. In many of these studies, the authors concluded that the best fitting model was a higher-order internalizing factor with two facets, reflecting distress and fear disorders, and a single externalizing factor [1–3, 5]. Exceptions to this general pattern, however, have been reported. In one instance, the best-fitting model included distress, fear, and externalizing factors, but not a higher-order internalizing factor [6], while another study found that the best fitting model included single factors for internalizing and externalizing disorders [4]. Similar findings have also been reported in studies of children and adolescents [7, 8], suggesting continuity in structure across development. In each of the studies that reported higher-order internalizing and externalizing factors, moderate associations between factors were noted, suggesting significant covariation among internalizing disorders, among externalizing disorders, and across internalizing and externalizing disorders.

Variable–centered studies, including those that use factor analytic methodologies, describe how disorders are organized with respect to one another. Findings from such studies are often cited by those who argue for a diagnostic system that emphasizes dimensions underlying psychopathology. The current instantiation of psychiatric diagnosis, however, parallels medical diagnosis whereby the identification of collections of symptoms that reflect an individual-specific etiology and course of illness are emphasized. Because diagnostic concepts are intended to classify the individual person, methods used to derive disorder classifications should ideally parallel this goal. One person–centered approach could be to categorize individuals based on internalizing and externalizing dimension severity. In practice, however, this approach would be problematic as it is presently unclear how many categories would be necessary to describe the individual or what levels of severity would warrant categorization. An alternative approach would be to use classification methods that are designed to identify relatively homogenous sub-samples from a heterogeneous larger population. One such person-oriented approach is latent class analysis (LCA). Results of these analyses provide more individual-specific profiles that may elucidate disease related processes by empirically identifying relatively homogenous subpopulations from the observed samples based on observed variables. There have been relatively few attempts to use LCA to inform the structure of psychopathology. Kessler et al. [9] examined a LCA of 19 disorders in the NCS-R and found that a seven class solution best fit the data. A more recent re-analysis of these and the original NCS data [10], using more recently developed fit indices, found that a five class solution best fit the data. In this solution, there was one class with few disorders, pathological classes with largely distress, fear, and externalizing disorders, and a multimorbid disorder class with both internalizing and externalizing. Thus, these results strongly paralleled the results found in factor analytic studies.

In addition to choosing a methodological approach that is theoretically consistent with the goals of classification, it is also important to consider the validity of the classification system. With few exceptions [e.g., 11, 12, 13], the validity of dimensional and latent class models of psychopathology derived from factor-analytic studies remains largely untested. Previous work has demonstrated that, relative to healthy controls, individuals with specific internalizing disorders report higher levels of negative emotionality [14–17] and individuals with externalizing disorders report higher levels of disinhibition [18–20]. Findings for positive emotionality, particularly with respect to the internalizing domain of psychopathology, are more equivocal. Relative to healthy controls, individuals with depressive disorders have lower levels of positive emotionality [14, 16, 17, 21]; however, individuals with some anxiety disorders (particularly social phobia) and those with schizophrenia also report lower levels of positive emotionality [15, 22]. Watson and Naragon-Gainey (2010) have recently argued for the relative, but not absolute, specificity of positive emotionality to depression compared to social phobia and schizophrenia.

Evidence for validity of models of psychopathology may also be informed by data on familial history of disorders. Some family studies find evidence for independent transmission of different types of internalizing disorders [23, 24], while others find evidence for shared transmission [25, 26]. Family studies of externalizing disorders are similarly characterized by conflicting data, with some studies supporting independent transmission of disorders [27–29] and others supporting shared transmission [30, 31]. Fewer studies have examined familial aggregation of disorders across the internalizing-externalizing dimensions. Those that have examined this issue have also produced equivocal results, with some finding independent [32], and others finding common [31], transmission of internalizing and externalizing disorders.

The present study examines the structure of psychopathology using LCA. This method clusters individuals based on probability profiles of observed variables. In the present research, these observed variables consist of specific DSM–defined internalizing and externalizing disorders. Our expectations for the number and description of classes are informed by evidence from factor analytic studies. We anticipate the emergence of two classes that broadly reflect internalizing disorders and externalizing domains and a third class of individuals with little psychopathology. While factor analytic studies often converge on distress and fear disorders as lower order facets of the internalizing dimensions, the higher order internalizing factor is usually almost perfectly correlated with a lower-order latent factor defined by distress disorders [1–3, 5, 6], and is thus redundant. Consequently, we do not anticipate the emergence of individual classes corresponding to distress and fear disorders. Further, as factor analytic studies identify moderate associations between internalizing and externalizing disorders, we also anticipate the emergence of a fourth class with high rates of disorders from both broadband dimensions.

Following the identification of the class solution that best fits the data, we then compare classes on rates of specific disorders, demographic features, psychiatric morbidity, temperament, and family history of depressive, anxiety, and substance use disorders. Based on the literature on the epidemiology of internalizing disorders, we anticipate that classes characterized by internalizing disorders will be comprised largely of women [33]. Likewise, based on the epidemiology of externalizing disorders, we anticipate that classes characterized by externalizing disorders will be comprised largely of men [34]. Based on data on relationships between temperament and psychopathology, we anticipate that, when compared with the class defined by low levels of psychopathology, the following will be observed: (a) classes characterized by internalizing and externalizing disorders will demonstrate higher levels of negative emotionality; (b) classes characterized by internalizing disorders will demonstrate lower levels of positive emotionality; and (c) classes characterized by externalizing disorders will demonstrate higher levels of disinhibition [18, 35, 36]. Lastly, we anticipate modest specificity of familial aggregation [25, 30, 31], with classes characterized by internalizing disorders having stronger associations with family history of internalizing disorders and classes characterized by externalizing disorders having stronger associations with family history of externalizing disorders.

Methods

Participants

The present study uses data from the Oregon Adolescent Depression Project (OADP) [37], a longitudinal study of a large cohort of high school students who were assessed twice during adolescence, a third time when the average age was 24, and a fourth time when the average age was 30. Participants were randomly selected from nine high schools in western Oregon. A total of 1,709 adolescents (ages 14–18; mean age 16.6, SD = 1.2) completed the initial (T1) assessments between 1987 and 1989. The participation rate at T1 was 61%. All youth provided informed consent before completing research procedures. Approximately one year later, 1,507 of the adolescents (88%) returned for a second evaluation (T2). Differences between the sample and the larger population from which it was selected, and between participants and those who declined to participate or dropped out of the study before T2, were small [37]. However, individuals with a history of disruptive behavior disorder at T1 were more likely to drop-out of the study (16.8% vs. 6.0%, χ2[1, N =1,709] =31.22, p < .001).

For the third assessment, all adolescents with a history of a depressive disorder by T2 (n = 360) or a history of non-mood disorders (n = 284), and a random sample of adolescents with no history of psychopathology by T2 (n = 457), were invited to participate in a third (T3) evaluation. All non-white T2 participants were retained in the T3 sample to maximize ethnic diversity. Of the 1,101 T2 participants selected for a T3 interview, 941 (85%) completed the age 24 evaluation. The T2 diagnostic groups did not differ on the rate of participation at T3. At age 30, all T3 participants were asked to complete another interview assessment (mean age = 30.45, SD = .70, range = 28–34). Of the 941 who participated in the T3 assessment, 816 (87%) completed the T4 assessment. Differences were few and small between those who participated in T3 but not T4 and who participated in T3 and T4 [38]. The reference sample for the current study includes participants who completed the T4 follow-up.

Measures

Proband diagnostic measures

At T1 and T2, probands were interviewed with a version of the Schedule for Affective Disorders and Schizophrenia for School-Age Children [K-SADS; 39], which combined features of the Epidemiologic and Present Episode versions, and included additional items to derive Diagnostic and Statistical Manual of Mental Disorders, 3rd edition revised (DSM-III-R) [40] diagnoses. Follow-up assessments at T2 and T3 were jointly administered with the Longitudinal Interval Follow-Up Evaluation [LIFE; 41]. The K-SADS/LIFE procedure provided information regarding the onset and course of disorders since the previous interview. The T4 interview consisted of a joint administration of the LIFE and the Structured Clinical Interview for DSM–IV [SCID; 42] to probe for new or continuing episodes since T3. Diagnoses were based on DSM-III-R criteria for T1 and T2 and Diagnostic and Statistical Manual of Mental Disorders, 4th edition (DSM-IV) [43] criteria for T3 and T4. Lifetime diagnoses were used as the indicators of psychopathology in this paper, such that the individual was considered to have the diagnosis if it was identified as a past or current disorder at any of the assessments.

A subset of interviews from each wave was rated from audio or videotapes by a second interviewer for reliability purposes: T1 = 263, T2 = 162, T3 = 190, and T4 = 124 interviews. Diagnostic agreement among raters was indexed by kappa. To avoid potential inflation, deflation, and/or unreliability of the kappa statistic, reliability was calculated only for categories diagnosed 10 or more times by both raters combined. Fleiss (1981) provides guidelines for the interpretation of kappa, whereby values ≥ 0.75 denote excellent agreement beyond chance, those between 0.75 and 0.40 are indicative of good to fair agreement, and coefficients < 0.40 reflect poor agreement. Across the four assessment waves, inter-rater diagnostic reliability was good to excellent for disorders that occurred with sufficient frequency. The reliability of one disorder, major depressive disorder (MDD), could be determined for each of the 4 waves (M kappa = .84; range: .81 – .86). Alcohol abuse/dependence (AUD) and cannabis abuse/dependence (CUD) were diagnosed with sufficient frequency among raters during 3 of the 4 waves, and the mean kappas were, respectively, .77 (range: .74 to .82) and .79 (range: .72 to .83). Hard drug abuse/dependence (DRG) was diagnosed with sufficient frequency among raters during 2 of the 4 waves, with the mean kappa for this disorder being .76 (range: .69 to .83). Kappa coefficients for dysthymia (.56), post-traumatic stress disorder (PTSD; .73), specific phobias (.66), panic disorder (.81), separation anxiety disorder (.83), attention-deficit/hyperactivity disorder (ADHD; .89) and oppositional defiant disorder (ODD; .77) could only be determined for 1 of the 4 assessment waves. Generalized anxiety disorder (GAD), social phobia, obsessive-compulsive disorder (OCD), and conduct disorder (CD) were not diagnosed with sufficient frequency during any assessment wave to allow an evaluation of diagnostic reliability.

At T3, adult antisocial behavior (AAB) was assessed using the Personality Disorder Examination (PDE) (Loranger, 1988) and at T4, AAB was assessed using the International Personality Disorder Examination (IPDE; Loranger et al., 1994). The IPDE is the updated version of the PDE and they share a number of features, including established interrater reliability, temporal stability, and minimal state effects (Loranger et al., 1991). The IPDE has an extensive scoring manual, which defines the scope and meaning of each item. The antisocial personality disorder module included prompts to assess symptoms of AAB that were present beginning at age 15 and AAB diagnoses did not require the presence of CD before age 18. Based on audiotaped interviews, the interrater reliability (indexed by kappa) for AAB assessed at T3 and T4 had a mean kappa of .61 (range: .49 –.74). AAB and CD were treated independently in the present paper.

First-degree family members of OADP participants were interviewed using the Structured Clinical Interview for DSM-IV, non-patient version [SCID-NP; 42] at the time of the T3 assessment. All family member participants provided written informed consent before completing research procedures. Of the 816 original probands who participated in the T4 assessment, information about at least one family member was available for 732 (89.7%). Data are available on a total of 2414 family members averaging 3.29 (SD = 1.15) family members per original proband. Interviews were conducted without knowledge of the offspring’s diagnoses. The interrater reliability of lifetime diagnoses (based on 184 randomly selected interviews) was excellent for depression (including both MDD and DYS; κ = .94), any anxiety disorder (κ = .90), AUD (κ = .86), and substance use disorders (SUD; inclusive of alcohol, cannabis, hard drug abuse/dependence) (κ = .89).

Family history data were collected from the original OADP participants and at least one other family member using a modified version of the Family Informant Schedule and Criteria (FISC; Mannuzza & Fyer, 1990), supplemented with items necessary to derive DSM-IV diagnoses. The interrater reliability of lifetime diagnoses (based on 242 randomly selected interviews) were good-excellent for MDD (κ = .90), any anxiety disorder (κ = .77), AUD (κ = . 90), and cannabis/hard drug abuse/dependence (κ = .82).

As multiple data sources were available for most parents, we derived lifetime best-estimate DSM-IV diagnoses from all available information [44]. Two diagnosticians, from a team of four senior clinicians, independently derived best-estimate diagnoses without knowledge of offspring diagnoses. Disagreements were resolved by consensus. Interrater reliability of the independently derived best-estimate diagnoses prior to the resolution of discrepancies was excellent for MDD (κ = .91), any anxiety disorder (κ = .94), AUD (κ = .97), and SUD (κ = .96).

Participant interviews at T3 and T4 were conducted by telephone, which generally yields comparable results to face-to-face interviews [45, 46]. Most interviewers had advanced degrees in a mental health field and several years of clinical experience.

Psychiatric morbidity and psychological constructs

As part of the diagnostic assessment, interviewers provided ratings based on the DSM Global Assessment of Functioning (GAF) scale and probed for the number of suicide attempts over the interval of time assessed by each interview. The total number of lifetime suicide attempts through T4 was examined in this paper. At T4, probands reported on their highest level of education completed (i.e., number of years of school completed), and number of weeks unemployed in past year (based on a six-point scale).

Proband temperament

Probands were asked to complete the General Temperament Survey [GTS; 47] for a sub-study approximately 2½ years before the T4 assessment. The GTS is a 90-item measure that assesses positive, negative, and disinhibited temperament, which include content related to extraversion, neuroticism, and (inversely) conscientiousness [48]. Data were available for 716 (87.7%) of probands.

Data analysis

Latent class analyses (LCA) were conducted using Mplus 6 [49]. Models were estimated for 2 through 10 classes. Lifetime diagnoses through T4 of dysthymia, MDD, social phobia, specific phobia, separation anxiety disorder, OCD, PTSD, GAD, panic disorder, ADHD, CD, ODD, AAB, AUD, cannabis use disorder, and hard drug disorders were entered as indicators of latent classes. Model fit was evaluated based on the Bayesian information criterion (BIC), with lower BIC indicating better fit, and the bootstrap likelihood ratio test (BLRT). The BLRT examines whether there is significant improvement in model fit when estimating k classes relative to the k − 1 class. Recent simulation work suggests that the BLRT is the most sensitive index for identifying the correct number of classes in models similar in number of indicators and sample sizes to those estimated here [50].

After identifying the best fitting solution, we extracted the most likely class membership for each participant and compared classes on demographic characteristics, disorders within each class, psychiatric morbidity, temperament, and familial aggregation of disorders. Post-hoc comparisons on demographic characteristics and disorders within each class were estimated using chi-square tests. Comparisons on psychiatric morbidity and temperament were estimated using analysis of variance, using appropriate distributional assumptions for specific outcomes (e.g., number of lifetime suicide attempts were estimated using Poisson distributions). Lastly, associations between class membership and familial aggregation of depressive, anxiety, and substance use disorders were estimated individually using survival analysis. Models were estimated with MPlus using the TYPE=COMPLEX option to accommodate the inclusion of multiple family members for probands.

Results

Identification of Best Fitting Class Solution

LCA were performed by estimating models for 2 through 10 classes. However, models are only presented for 2 through 5 classes because inadmissible solutions were found for models of 6 through 10 classes. The minimum BIC was found for the 3 class solution (Table 1), suggesting that this is the preferred solution. However, the BLRT found that the 4 class solution provided a significant improvement relative to the 3 class solution. Additionally, the BLRT found that the 5 class solution did not provide a better fit than the 4 class solution. Thus, based on the simulation work suggesting that the BLRT outperforms the BIC in identifying the optimal solution [50], we relied on the 4 class solution.

Table 1.

LCA Fit Information

Classes LL BIC AIC Parameters BLRT
2 −3341.10 6903.44 6748.19 33 479.54***
3 −3259.30 6853.82 6618.60 50 163.60***
4 −3227.41 6904.02 6588.83 67 63.77***
5 −3209.90 6982.98 6587.80 84 35.03
Open in a new tab
***

p < .001. LL = Log-likelihood value; BIC = Bayesian Information Criteria; AIC = Akaike Information Criteria; BLRT = bootstrap likelihood ratio test.

Class 1 accounted for 62.5% of the sample and demonstrated the lowest overall rates of psychopathology (Figure 1), although there were high rates of MDD (40.3%) and AUD (21.4%) in this class. Compared to other identified classes, individuals in this class had milder histories of MDD, with a lower proportion of cases having moderate/severe and recurrent episodes, and milder histories of AUD, with most cases having lifetime diagnoses of alcohol abuse rather than dependence (Table 2). Only 16.5% of cases had both lifetime internalizing and externalizing disorders.

Figure 1. Probabilities of specific forms of psychopathology across each latent class.

Figure 1

Open in a new tab

Class 1 = Healthy; Class 2 = Internalizing; Class 3 = Externalizing; Class 4 = Comorbid Internalizing and Externalizing. MDD = Major Depressive Disorder; OCD = Obsessive-Compulsive Disorder; PTSD = Post-Traumatic Stress Disorder; GAD = Generalized Anxiety Disorder; ADHD = Attention Deficit Hyperactivity Disorder; ODD = Oppositional Defiant Disorder; AAB = Adult Antisocial Behavior.

Table 2.

Psychiatric Morbidity Across Classes.

Class 1 (62.5%) Class 2 (16.9%) Class 3 (16.4%) Class 4 (4.2%) F/χ2
Female Sex1 288 (56.5)a 111 (80.4)b 52 (38.8)c 29 (85.3)b 59.78***
Number of Disorders2 .83 (.03)a 2.92 (.07)b 3.54 (.10)c 6.08 (.25)d 1571.30***
Number of Times in Treatment2 .42 (.04)a 1.17 (.11)b 1.68 (.18)c 3.12 (.48)d 292.13***
Number of Lifetime Suicide Attempts2 .36 (.20)a 1.24 (.72)a,b .68 (.22)a,b 3.00 (1.26)b 11.61**
T4 GAF3 82.29 (9.12)a 75.12 (11.77)b 73.44 (12.47)b 68.12 (15.60)c 47.35***
Unemployment3 .38 (.15)a .43 (.18)b .42 (.16)b .51 (.22)c 7.99***
Yrs. Of Education3 14.73 (1.86)a 14.19 (1.92)b 13.86 (1.95)b,c 13.29 (1.83)c 12.30***
MDD Characteristics
 Moderate/Severe1 165 (71.4)a 110 (80.3)b 69 (87.3)b,c 30 (93.4)c 9.62***
 Recurrence1 110 (47.6)a 83 (60.6)b 52 (65.8)b 25 (78.1)b 17.21**
Alcohol Dependence1 48 (43.2)a 25 (55.6)a 86 (72.3)b 26 (86.7)b 29.82***
Open in a new tab
***

p < .001;

**

p < .01.

1

Denotes that table entries include N(%) and χ2-statistic is computed;

2

Denotes that table entries include M (SE) and F-statistic is computed using a count distribution; and

3

Denotes that table entries include M (SD) and F-statistic is computed. Statistics for MDD Characteristics are based only on cases with a lifetime diagnosis of MDD. Statistics for Alcohol Dependence are based only on cases with a lifetime Alcohol Use Disorder Diagnosis. Differing superscripts reflect significant differences at p < .05.

Class 2 accounted for 16.9% of the sample and demonstrated elevated rates of dysthymia (19.2%), MDD (100.0%), separation anxiety (15.4%), PTSD (18.7%), panic disorder (18.0%), and AUD(34.2%). Taken together, this class largely reflects elevated rates of internalizing disorders (albeit with a number of cases of AUD). Approximately half of the individuals in this class (49.3%) had both lifetime internalizing and externalizing disorders.

Class 3 accounted for 16.4% of the sample and demonstrated elevated rates of MDD (58.8%), CD (27.6%), AAB (13.0%), and alcohol (84.2%), cannabis (82.0%), and hard drug use disorders (53.7%). Taken together, this class largely reflects elevated rates of externalizing disorders (albeit with a number of cases of MDD). More than half of the individuals in this class (62.7%) had both lifetime internalizing and externalizing disorders.

Lastly, Class 4 accounted for 4.2% of the sample and demonstrated elevated rates of dysthymia (18.0%), MDD (96.2%), social phobia (30.0%) and specific phobia (26.7%), separation anxiety (20.6%), OCD (15.0%), PTSD (43.2%), GAD (15.7%), panic disorder (45.2%), ODD (14.7%), and alcohol (85.4%), cannabis (76.2%), and hard drug use disorders (61.1%). All individuals in this class (100%) had both lifetime internalizing and externalizing disorders. Interestingly, the most prevalent externalizing disorders were substance use disorders and ODD, but rates of conduct disorder and AAB were extremely low.

Class Comparisons: Demographics, Psychiatric Morbidity, and Psychosocial Functioning

Chi-square analyses found no significant differences between classes on race (defined as Caucasian vs. non-Caucasian; Table 2). However, gender differences were evident, such that the percentage of women significantly increased from the highly externalizing class to the class with lower rates of psychopathology to the highly internalizing class to the comorbid class.

We next examined psychiatric morbidity, as indexed by the number of lifetime disorders, number of times in psychiatric treatment, and the number of lifetime suicide attempts (Table 2). The number of lifetime disorders and number of times in psychiatric treatment increased from the class with little psychopathology, to the predominantly internalizing class, to the predominantly externalizing class, to the comorbid class. Higher numbers of lifetime suicide attempts were found in the comorbid class than the class with little psychopathology; however, the number of lifetime suicide attempts in the predominantly internalizing and externalizing classes did not differ from each other or from the other classes.

Psychosocial functioning was indexed by the GAF, educational attainment, and recent unemployment (Table 2). The class with lower rates of psychopathology was functioning best, with significantly higher GAF scores and educational attainment and significantly lower unemployment than the three other classes. Individuals in the predominantly internalizing and externalizing classes did not differ from each other on these indices. Individuals in the comorbid class were performing most poorly, with lower GAF scores and higher levels of unemployment than individuals in the predominantly internalizing and externalizing classes. No significant differences were observed between individuals within the comorbid and predominantly externalizing classes on educational attainment.

Class Comparisons: Temperament

Completion of the GTS was not associated with class membership, χ2 (3) = 2.28, p = .55. Individuals in the predominantly internalizing class had lower levels of Positive Temperament than individuals in the class with lower rates of psychopathology and the predominantly externalizing class, but did not differ from those in the comorbid class. Individuals in the comorbid class had lower levels of Positive Temperament than individuals in the class with lower rates of psychopathology, but did not differ from individuals in the predominantly externalizing class. Individuals in the predominantly internalizing and comorbid classes had higher levels of Negative Temperament than individuals in the predominantly externalizing class, who, in turn, had higher levels of Negative Temperament than individuals in the class with lower rates of psychopathology. Individuals in the predominantly externalizing and comorbid classes had higher levels of Disinhibited Temperament than individuals in the class with lower rates of psychopathology and individuals in the predominantly internalizing class.

Class Comparisons: Family History of Psychopathology

Table 4 displays hazard ratios (and 95% confidence intervals) for using class membership to predict depressive, anxiety and substance use disorders in first-degree relatives. Compared to the class with lower rates of psychopathology, the predominantly internalizing, predominantly externalizing, and comorbid classes were all more likely to have family members with depressive disorders. The rate of depressive disorders was also significantly greater among relatives of individuals in the comorbid class than relatives of probands in the externalizing class, while the rate in relatives of members of the internalizing class was intermediate and did not differ from the other two classes. Compared to the classes with lower rates of psychopathology and predominantly externalizing disorders, the relatives of members of the predominantly internalizing and comorbid classes had significantly higher rates of anxiety disorders. Finally, relatives of individuals in the predominantly internalizing, predominantly externalizing, and comorbid classes had higher rates of substance use disorders than relatives of probands in the class with little psychopathology. Moreover, relatives of individuals in the comorbid class also had a higher rate of substance use disorders than the relatives of members of the predominantly internalizing and externalizing classes.

Table 4.

Latent Class Membership Predicting Family History of Psychopathology

1. Lower 2. Internal 3. External 4. Comorb 2 v. 1 3 v. 1 4 v. 1 3 v. 2 4 v. 2 4 v.3
n (%) n (%) n (%) n (%) HR (95% CI) HR (95% CI) HR (95% CI) HR (95% CI) HR (95% CI) HR (95% CI)
Familial 379 (24.8) 157 (39.0) 122 (31.3) 41 (42.3) 1.66 (1.37, 1.96)*** 1.33 (1.09, 1.58)** 1.94 (1.41, 2.55)*** .80 (.64, 1.01) 1.17 (.83, 1.64) 1.46 (1.03, 2.06)*
Depression Familial 166 (10.6) 65 (16.1) 41 (10.5) 19 (20.0) 1.51 (1.10, 2.07)* .98 (.67, 1.42) 1.89 (1.16, 3.07)* .65 (.42, . 99)* 1.25 (.74, 2.11) 1.93 (1.10, 3.40)*
Anxiety Familial SUD 460 (30.1) 154 (38.2) 171 (43.8) 55 (57.9) 1.32 (1.09, 1.60)** 1.58 (1.31, 1.90)*** 2.21 (1.69, 2.88)*** 1.19 (.96, 1.49) 1.67 (1.24, 2.24)** 1.40 (1.04, 1.87)*
Open in a new tab
*

p < .05;

**

p < .01;

***

p < .001. Lower = Class 1; Internal = Class 2; External = Class 3; Comorb = Class 4. SUD = Substance Use Disorder. Differing superscripts reflect significant differences at p < .05.

Discussion

Previous literature on the structure of psychopathology has largely relied on variable–centered methods such as factor analysis. Models derived from these methods describe the organization of psychopathology across individuals; however, disorder classification is intended to provide information about individual cases. To accomplish this latter goal, alternative methods such as LCA are required. The present study used LCA to identify classes of psychopathology, with the validity of the resultant classes indicated by dissimilar associations between class membership and psychiatric morbidity, psychological functioning, temperament, and family history of psychopathology.

Our analyses suggested that a four class solution best fit the data. The classes provided groupings of individuals with lower probabilities of psychopathology, higher probabilities of internalizing disorders, higher probabilities of externalizing disorders, and higher probabilities of both internalizing and externalizing disorders, respectively. These classes are consistent with the pattern of combinations that can be derived from the internalizing and externalizing dimensions reported in factor analytic studies. Additionally, the present results are largely consistent with those reported from the NCS and the NCS-R samples [10]. The main difference between results presented in these studies and those of the present study is that we did not find distinct classes based on the fear and distress disorders from the internalizing domain.

Inspection of each class revealed some notable patterns. In the class with relatively low levels of psychopathology, there were substantial rates of MDD and alcohol use disorders although these tended to be relatively mild conditions. In general, individuals in this class had lower rates of recurrence and lower episode severity than individuals with MDD in the remaining three classes. Similarly, the alcohol use disorders generally took the form of abuse rather than dependence; indeed, the rate of alcohol dependence was lower in this class than all other classes. As MDD and alcohol use disorders are ubiquitous across all classes, this suggests that there may be considerable heterogeneity in these syndromes. On the whole, this class had the lowest levels of psychiatric morbidity and highest psychosocial functioning, and lowest familial aggregation of psychopathology.

In the class with internalizing disorders, all individuals met criteria for MDD and there were increased rates of most other internalizing disorders as well as alcohol and hard drug use disorders relative to the class with lower levels of psychopathology. It is possible that the individuals in this class relied on alcohol and/or drugs to self-medicate. This class may therefore represent individuals with secondary substance abuse problems related to their internalizing disorders. Alternatively, typologies of alcoholism often identify a group with predominantly internalizing problems [e.g., Cloninger’s type 1 vs type 2 alcoholism; 51]. Individuals in this class were more likely to be women, have higher levels of psychiatric morbidity, and have lower psychosocial functioning than individuals in the class with relatively low levels of psychopathology. This class also had higher levels of negative and lower levels of positive temperament and elevated rates of depressive and anxiety disorders in their relatives compared to the class with low levels of psychopathology.

Within the class with externalizing disorders, most individuals met criteria for alcohol, cannabis, and hard drug use disorders and MDD, while a large minority also met criteria for CD. These results are consistent with previous data that demonstrated a prospective association between CD and depression [52]. This sequence has been described as a failure model [53] in which youth with CD fail to succeed in multiple domains of functioning due to their externalizing behaviors, with repeated failures eventually resulting in depression. Individuals in this class were more likely to be men, had more psychiatric disorders, and initiated treatment on more occasions than individuals in the class with relatively low levels of psychopathology. These classes, however, were, similar on suicide attempts and level of psychosocial functioning. This class also had higher levels of negative and disinhibited temperament and elevated rates of SUDs in their relatives compared to the class with low levels of psychopathology.

In the class with both internalizing and externalizing disorders, many if not most individuals met criteria for MDD, panic disorder, PTSD, and alcohol, cannabis, and hard drug use disorders. Although this was the smallest class based on prevalence, this class appeared to account for much of the observed comorbidity between internalizing and externalizing disorders in the entire sample. Similarly, all individuals in this class had both lifetime internalizing and externalizing disorders. Interestingly, the class with both internalizing and externalizing disorders had higher rates of ODD, but lower higher rates of CD and AAB than the externalizing class. This is consistent with data indicating that while CD and antisocial personality disorder fall squarely on the externalizing spectrum (Krueger, Markon, Patrick, Benning, & Kramer, 2007), ODD predicts the later development of both internalizing and externalizing disorders [54–57]. The class with both internalizing and externalizing disorders consisted predominantly of women and had the highest degree of psychiatric morbidity and impairment. Members of this class also had higher levels of negative and disinhibited temperament, and lower levels of positive temperament than the class with low levels of psychopathology. Finally, this class was characterized by the highest rates of depressive, anxiety and substance use disorders in relatives.

These data suggest that the primary dimensions of internalizing and externalizing disorders identified in factor analytic studies are largely replicated using person-oriented methods. However, our findings did not fully replicate the distinction between distress and fear disorders within the internalizing domain, perhaps because previous research has indicated that these lower-order factors have near-perfect correlations with a higher-order internalizing latent factor.

Several investigations have concluded that there are replicable patterns of associations between temperament dimensions and specific disorders or disorder classes. Specifically these studies report that most forms of psychopathology are characterized by high levels of negative emotionality, depressive disorders are characterized by low levels of positive emotionality, and externalizing disorders are characterized by high levels of disinhibited temperament [16–22, 36, 58, 59]. Our findings largely mirrored these results, support the continuity between temperament and psychopathology, and demonstrate coherence between factor analytic and person-oriented approaches.

Finally, this research also examined the familial aggregation of depressive, anxiety, and substance use disorders, and whether aggregation among family members was largely related to disorders within the same domain [25, 30, 31]. The data suggest relative specificity of the family transmission of internalizing and externalizing disorders. The rates of depressive disorders in relatives incrementally increase across the externalizing, internalizing, and comorbid classes. Similarly, the aggregation of familial SUD increases as one compares rates among the internalizing, externalizing, and comorbid classes, further suggesting some specificity of familial aggregation.

In summary, the present study complements the work on dimensional models of psychopathology by identifying person-oriented clusters of psychopathology. The results provide converging support for the broad dimensions of internalizing and externalizing disorders, but raise the possibility that the distinction between fear and distress disorders may be less salient than some of the factor analytic studies suggest. In addition to addressing the structure of psychopathology, we also examined the validity of the distinctions between classes using demographic, clinical, temperament, and family history data. These data supported the validity of our cluster solution by confirming most of the hypothesized differences between clusters. Lastly, our data on associations between class membership and familial psychopathology generally supports transmission within clusters.

Strengths of this study include careful diagnostic assessments on four occasions across 15 years and the inclusion of a large number of psychiatric disorders. We also examined a variety of external variables from multiple domains to validate the latent classes. However, the study also has a number of limitations. First, like other community-based studies of the structure of psychopathology, we omitted other important, but less common, disorders (e.g., bipolar, psychotic, and eating disorders); thus, it is unclear how to incorporate these other disorders into this framework. Second, not all participants had complete data on temperament or family history. However, no associations between missingness of temperament or family history data and class membership were found, suggesting minimal bias. Third, while the four class solution was identified as the preferred model, the precision in assigning class membership was reasonable (entropy = .74) but less than optimal. Simulations suggest that significance tests for associations between class membership and external variables may, therefore, be less reliable than desired (Clark & Muthén, under review). However, most statistical tests were highly significant. In addition, we repeated the analyses using a second method that weighted observations based on the posterior probability of class assignment and adjusted the analyses for imprecision of class assignment (Clark & Muthén, under review). These analyses produced identical results. Fourth, we reported findings from the LCA models that were not adjusted for the complex sampling procedures. We did this to accommodate the BLRT as a model selection tool, which is not available when using sampling weights. However, patterns of BIC were similar for the analyses relying on the BIC for analyses utilizing the sampling weights. [60]. These analyses produced identical results. Lastly, the current paper focused exclusively on LCA models, and did not examine confirmatory factor analysis (CFA) and factor mixture models (FMM). Comparisons between these modeling approaches are beyond the scope of this paper. Future work on the organization of psychopathology will likely benefit from further simulation studies to identify the most appropriate statistical indices for comparing model validity and fit between CFA, LCA, and FMM [61] methods.

Table 3.

Mean levels of temperament across classes.

Healthy Internalizing Externalizing Comorbid F
(62.5%) (16.9%) (16.4%) (4.2%)
Negative Temp. 7.76 (6.12)a 14.30 (7.86)b 10.88 (6.96)c 14.87 (8.14)b 39.47***
Positive Temp. 20.22 (4.95)a 17.57 (6.61)b 19.26 (5.69)a,c 17.34 (7.56)b,c 9.32***
Disinhibited Temp. 7.61 (4.85)a 8.38 (4.47)a 12.10 (5.46)b 11.81 (6.40)b 29.44***
Open in a new tab
***

p < .001; Temp. = Temperament. Healthy = Class 1; Internalizing = Class 2; Externalizing = Class 3; Comorbid = Class 4. Differing superscripts reflect significant differences at p < .05.

Acknowledgments

This work was supported by National Institute of Mental Health Grants R01MH66023 (Dr. Klein), R01 MH40501, R01 MH50522, R01 MH52858, and R01 DA012951 (Dr. Lewinsohn). Portions of these data were presented at the Twenty Fourth Annual Meeting of the Society for Research in Psychopathology, Seattle, WA; October, 2010.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Contributor Information

Thomas M. Olino, University of Pittsburgh School of Medicine

Daniel N. Klein, Stony Brook University

Richard F. Farmer, Oregon Research Institute

John R. Seeley, Oregon Research Institute

Peter M. Lewinsohn, Oregon Research Institute

References

  • 1.Slade T, Watson D. The structure of common DSM-IV and ICD-10 mental disorders in the Australian general population. Psychol Med. 2006;36(11):1593–600. doi: 10.1017/S0033291706008452. [DOI] [PubMed] [Google Scholar]
  • 2.Krueger RF. The structure of common mental disorders. Arch Gen Psychiatry. 1999 Oct;56(10):921–6. doi: 10.1001/archpsyc.56.10.921. [DOI] [PubMed] [Google Scholar]
  • 3.Krueger RF, Markon KE. Reinterpreting comorbidity: A model-based Approach to Understanding and Classifying Psychopathology. Annual Review of Clinical Psychology. 2006;2:111–33. doi: 10.1146/annurev.clinpsy.2.022305.095213. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Krueger RF, Caspi A, Moffitt TE, Silva PA. The structure and stability of common mental disorders (DSM-III-R): A longitudinal-epidemiological study. J Abnorm Psychol. 1998 May;107(2):216–27. doi: 10.1037//0021-843x.107.2.216. [DOI] [PubMed] [Google Scholar]
  • 5.Vollebergh WAM, Iedema J, Bijl RV, de Graaf R, Smit F, Ormel J. The structure and stability of common mental disorders: the NEMESIS study. Arch Gen Psychiatry. 2001;58(6):597–603. doi: 10.1001/archpsyc.58.6.597. [DOI] [PubMed] [Google Scholar]
  • 6.Beesdo-Baum K, Höfler M, Gloster AT, Klotsche J, Lieb R, Beauducel A, et al. The structure of common mental disorders: A replication study in a community sample of adolescents and young adults. International Journal of Methods in Psychiatric Research. 2009;18(4):204–20. doi: 10.1002/mpr.293. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Lahey BB, Rathouz PJ, Van Hulle C, Urbano RC, Krueger RF, Applegate B, et al. Testing structural models of DSM-IV symptoms of common forms of child and adolescent psychopathology. J Abnorm Child Psychol. 2008;36(2):187–206. doi: 10.1007/s10802-007-9169-5. [DOI] [PubMed] [Google Scholar]
  • 8.Lahey BB, Applegate B, Waldman ID, Loft JD, Hankin BL, Rick J. The structure of child and adolescent psychopathology: Generating new hypotheses. J Abnorm Psychol. 2004;113(3):358–85. doi: 10.1037/0021-843X.113.3.358. [DOI] [PubMed] [Google Scholar]
  • 9.Kessler RC, Chiu WT, Demler O, Walters EE. Prevalence, severity, and comorbidity of 12-month DSM-IV disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry. 2005;62(6):617–27. doi: 10.1001/archpsyc.62.6.617. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Vaidyanathan U, Patrick CJ, Iacono WG. Patterns of comorbidity among mental disorders: a person-centered approach. Compr Psychiatry. doi: 10.1016/j.comppsych.2010.10.006. in press. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Markon KE. How things fall apart: Understanding the nature of internalizing through its relationship with impairment. J Abnorm Psychol. 2010;119(3):447–58. doi: 10.1037/a0019707. [DOI] [PubMed] [Google Scholar]
  • 12.Farmer RF, Seeley JR, Kosty DB, Lewinsohn PM. Refinements in the hierarchical structure of externalizing psychiatric disorders: Patterns of lifetime liability from mid-adolescence through early adulthood. J Abnorm Psychol. 2009;118:699–710. doi: 10.1037/a0017205. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Krueger RF, Finger MS. Using item response theory to understand comorbidity among anxiety and unipolar mood disorders. Psychol Assess. 2001 Mar;13(1):140–51. [PubMed] [Google Scholar]
  • 14.Clark LA, Watson D. Tripartite model of anxiety and depression: Psychometric evidence and taxonomic implications. J Abnorm Psychol. 1991 Aug;100(3):316–36. doi: 10.1037//0021-843x.100.3.316. [DOI] [PubMed] [Google Scholar]
  • 15.Brown TA, Chorpita BF, Barlow DH. Structural relationships among dimensions of the DSM-IV anxiety and mood disorders and dimensions of negative affect, positive affect, and autonomic arousal. J Abnorm Psychol. 1998;107:179–92. doi: 10.1037//0021-843x.107.2.179. [DOI] [PubMed] [Google Scholar]
  • 16.Lonigan CJ, Hooe ES, David CF, Kistner JA. Positive and negative affectivity in children: Confirmatory factor analysis of a two-factor model and its relation to symptoms of anxiety and depression. J Consult Clin Psychol. 1999;67:374–86. doi: 10.1037//0022-006x.67.3.374. [DOI] [PubMed] [Google Scholar]
  • 17.Watson D, Weber K, Assenheimer JS, Clark LA, Strauss ME, McCormick RA. Testing a tripartite model: I. Evaluating the convergent and discriminant validity of anxiety and depression symptom scales. J Abnorm Psychol. 1995 Feb;104(1):3–14. doi: 10.1037//0021-843x.104.1.3. [DOI] [PubMed] [Google Scholar]
  • 18.Iacono WG, Carlson SR, Taylor J, Elkins IJ, McGue M. Behavioral disinhibition and the development of substance-use disorders: Findings from the Minnesota Twin Family Study. Dev Psychopathol. 1999;11(04):869–900. doi: 10.1017/s0954579499002369. [DOI] [PubMed] [Google Scholar]
  • 19.Iacono WG, Malone SM, McGue M. Behavioral disinhibition and the development of early-onset addiction: Common and specific influences. Annual Review of Clinical Psychology. 2008;4:325–48. doi: 10.1146/annurev.clinpsy.4.022007.141157. [DOI] [PubMed] [Google Scholar]
  • 20.Young SE, Friedman NP, Miyake A, Willcutt EG, Corley RP, Haberstick BC, et al. Behavioral disinhibition: Liability for externalizing spectrum disorders and its genetic and environmental relation to response inhibition across adolescence. J Abnorm Psychol. 2009;118(1):117–30. doi: 10.1037/a0014657. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Brown TA, Campbell LA, Lehman CL, Grisham JR, Mancill RB. Current and lifetime comorbidity of the DSM-IV anxiety and mood disorders in a large clinical sample. J Abnorm Psychol. 2001;110(4):585. doi: 10.1037//0021-843x.110.4.585. [DOI] [PubMed] [Google Scholar]
  • 22.Naragon-Gainey K, Watson D, Markon KE. Differential relations of depression and social anxiety symptoms to the facets of extraversion/positive emotionality. J Abnorm Psychol. 2009 May;118(2):299–310. doi: 10.1037/a0015637. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Klein DN, Lewinsohn P, Rohde P, Seeley J, Shankman S. Family study of co-morbidity between major depressive disorder and anxiety disorders. Psychol Med. 2003 May;33(4):703–14. doi: 10.1017/s0033291703007487. [DOI] [PubMed] [Google Scholar]
  • 24.Weissman MM, Wickramaratne P, Adams PB, Lish JD, Horwath E, Charney D, et al. The relationship between panic disorder and major depression: A new family study. Arch Gen Psychiatry. 1993;50(10):767–80. doi: 10.1001/archpsyc.1993.01820220017003. [DOI] [PubMed] [Google Scholar]
  • 25.Steinhausen HC, Foldager L, Perto G, Munk-Jørgensen P. Family aggregation of mental disorders in the nationwide Danish three generation study. Eur Arch Psychiatry Clin Neurosci. 2009;259(5):270–7. doi: 10.1007/s00406-008-0865-0. [DOI] [PubMed] [Google Scholar]
  • 26.Maier W, Minges J, Lichtermann D. The familial relationship between panic disorder and unipolar depression. J Psychiatr Res. 1995;29(5):375–88. doi: 10.1016/0022-3956(95)00024-y. [DOI] [PubMed] [Google Scholar]
  • 27.Faraone SV, Biederman J, Keenan K, Tsuang MT. Separation of DSM-III attention deficit disorder and conduct disorder: Evidence from a family-genetic study of American child psychiatric patients. Psychol Med. 2009;21(01):109–21. doi: 10.1017/s0033291700014707. [DOI] [PubMed] [Google Scholar]
  • 28.Merikangas KR, Stolar M, Stevens DE, Goulet J, Preisig MA, Fenton B, et al. Familial transmission of substance use disorders. Arch Gen Psychiatry. 1998;55(11):973. doi: 10.1001/archpsyc.55.11.973. [DOI] [PubMed] [Google Scholar]
  • 29.Petty CR, Monuteaux MC, Mick E, Hughes S, Small J, Faraone SV, et al. Parsing the familiality of oppositional defiant disorder from that of conduct disorder: A familial risk analysis. J Psychiatr Res. 2009;43(4):345–52. doi: 10.1016/j.jpsychires.2008.03.010. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Hicks BM, Krueger RF, lacono WG, McGue M, Patrick CJ. Family transmission and heritability of externalizing disorders: A twin-family study. Arch Gen Psychiatry. 2004 Sep;61(9):922–8. doi: 10.1001/archpsyc.61.9.922. [DOI] [PubMed] [Google Scholar]
  • 31.Nurnberger JI, Jr, Wiegand R, Bucholz K, O’Connor S, Meyer ET, Reich T, et al. A family study of alcohol dependence: Coaggregation of multiple disorders in relatives of alcohol-dependent probands. Arch Gen Psychiatry. 2004;61(12):1246–56. doi: 10.1001/archpsyc.61.12.1246. [DOI] [PubMed] [Google Scholar]
  • 32.Klein DN, Lewinsohn PM, Seeley JR, Rohde P. A family study of major depressive disorder in a community sample of adolescents. Arch Gen Psychiatry. 2001 Jan;58(1):13–20. doi: 10.1001/archpsyc.58.1.13. [DOI] [PubMed] [Google Scholar]
  • 33.Kessler RC, Berglund P, Demler O, Jin R, Koretz D, Merikangas KR, et al. The epidemiology of major depressive disorder results from the National Comorbidity Survey Replication (NCS-R) Journal of the American Medical Association. 2003;289(23):3095–105. doi: 10.1001/jama.289.23.3095. [DOI] [PubMed] [Google Scholar]
  • 34.Seedat S, Scott KM, Angermeyer MC, Berglund P, Bromet EJ, Brugha TS, et al. Cross-national associations between gender and mental disorders in the World Health Organization World Mental Health Surveys. Arch Gen Psychiatry. 2009;66(7):785–95. doi: 10.1001/archgenpsychiatry.2009.36. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35.Watson D, Clark LA, Weber K, Assenheimer JS, Strauss ME, McCormick RA. Testing a tripartite model: II. Exploring the symptom structure of anxiety and depression in student, adult, and patient samples. J Abnorm Psychol. 1995 Feb;104(1):15–25. doi: 10.1037//0021-843x.104.1.15. [DOI] [PubMed] [Google Scholar]
  • 36.Watson D, Naragon-Gainey K. On the specificity of positive emotional dysfunction in psychopathology: Evidence from the mood and anxiety disorders and schizophrenia/schizotypy. Clin Psychol Rev. 2010;30:839–48. doi: 10.1016/j.cpr.2009.11.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.Lewinsohn PM, Hops H, Roberts RE, Seeley JR, Andrews JA. Adolescent psychopathology: I. Prevalence and incidence of depression and other DSM-III--R disorders in high school students. J Abnorm Psychol. 1993 Feb;102(1):133–44. doi: 10.1037//0021-843x.102.1.133. [DOI] [PubMed] [Google Scholar]
  • 38.Olino TM, Klein DN, Lewinsohn PM, Rohde P, Seeley JR. Longitudinal associations between depressive and anxiety disorders: A comparison of two trait models. Psychol Med. 2008 Mar;38(3):353–63. doi: 10.1017/S0033291707001341. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39.Orvaschel H, Puig-Antich J, Chambers WJ, Tabrizi MA, Johnson R. Retrospective assessment of prepubertal major depression with the Kiddie-SADS-E. J Am Acad Child Adolesc Psychiatry. 1982;21:392–7. doi: 10.1016/s0002-7138(09)60944-4. [DOI] [PubMed] [Google Scholar]
  • 40.American Psychiatric Association. Diagnostic and statistical manual of mental disorders. 3 rev. Washington, DC: Author; 1987. [Google Scholar]
  • 41.Keller MB, Lavori PW, Friedman B, Nielsen E, Endicott J, McDonald-Scott P, et al. The Longitudinal Interval Follow-up Evaluation - a Comprehensive Method for Assessing Outcome in Prospective Longitudinal-Studies. Arch Gen Psychiatry. 1987 Jun;44(6):540–8. doi: 10.1001/archpsyc.1987.01800180050009. [DOI] [PubMed] [Google Scholar]
  • 42.First MB, Spitzer RL, Gibbon M, Williams JBW. The Structured Clinical Interview for DSM-IV Axis I Disorders – Non-patient edition. 1996. In press. [Google Scholar]
  • 43.American Psychiatric Association. Diagnostic and statistical manual of mental disorders. 4. Washington, DC: Author; 1994. rev. [Google Scholar]
  • 44.Leckman JF, Sholomskas D, Thompson WD, Belanger A, Weissman MM. Best Estimate of Lifetime Psychiatric-Diagnosis - a Methodological Study. Arch Gen Psychiatry. 1982;39(8):879–83. doi: 10.1001/archpsyc.1982.04290080001001. [DOI] [PubMed] [Google Scholar]
  • 45.Rohde P, Lewinsohn PM, Seeley JR. Comparability of telephone and face-to-face interviews in assessing axis I and II disorders. Am J Psychiatry. 1997 Nov;154(11):1593–8. doi: 10.1176/ajp.154.11.1593. [DOI] [PubMed] [Google Scholar]
  • 46.Sobin C, Weissman MM, Goldstein RB, Adams P, Wickramaratne P, Warner V, et al. Diagnostic Interviewing for Family Studies - Comparing Telephone and Face-to-Face Methods for the Diagnosis of Lifetime Psychiatric-Disorders. Psychiatr Genet. 1993 Win;3(4):227–33. [Google Scholar]
  • 47.Clark LA, Watson D. The General Temperament Survey (GTS) 1990. In press. [Google Scholar]
  • 48.Watson D, Clark LA. On traits and temperament: General and specific factors of emotional experience and their relation to the five-factor model. J Pers. 1992 Jun;60(2):441–76. doi: 10.1111/j.1467-6494.1992.tb00980.x. [DOI] [PubMed] [Google Scholar]
  • 49.Muthén LK, Muthén BO. Mplus User’s Guide. 6. 1998–2010. In press. [Google Scholar]
  • 50.Nylund KL, Asparouhov T, Muthén BO. Deciding on the number of classes in latent class analysis and growth mixture modeling: A Monte Carlo simulation study. Structural Equation Modeling. 2007;14(4):535–69. [Google Scholar]
  • 51.Johnson BA, Cloninger CR, Roache JD, Bordnick PS, Ruiz P. Age of onset as a discriminator between alcoholic subtypes in a treatment seeking outpatient population. Am J Addict. 2000;9(1):17–27. doi: 10.1080/10550490050172191. [DOI] [PubMed] [Google Scholar]
  • 52.Burke JD, Loeber R, Lahey BB, Rathouz PJ. Developmental transitions among affective and behavioral disorders in adolescent boys. Journal of Child Psychology and Psychiatry. 2005;46(11):1200–10. doi: 10.1111/j.1469-7610.2005.00422.x. [DOI] [PubMed] [Google Scholar]
  • 53.Capaldi DM. Co-occurrence of conduct problems and depressive symptoms in early adolescent boys: II. A 2-year follow-up at Grade 8. Dev Psychopathol. 1992;4(01):125–44. doi: 10.1017/s0954579499001959. [DOI] [PubMed] [Google Scholar]
  • 54.Kim-Cohen J, Caspi A, Moffitt TE, Harrington HL, Milne BJ, Poulton R. Prior juvenile diagnoses in adults with mental disorder developmental follow-back of a prospective-longitudinal cohort. Arch Gen Psychiatry. 2003;60(7):709–17. doi: 10.1001/archpsyc.60.7.709. [DOI] [PubMed] [Google Scholar]
  • 55.Lavigne JV, Cicchetti C, Gibbons RD, Binns HJ, Larsen L, DeVito C. Oppositional defiant disorder with onset in preschool years: Longitudinal stability and pathways to other disorders. Journal of American Academy of Child and Adolescent Psychiatry. 2001;40(12):1393–400. doi: 10.1097/00004583-200112000-00009. [DOI] [PubMed] [Google Scholar]
  • 56.Speltz ML, McClellan JON, DeKlyen M, Jones K. Preschool boys with oppositional defiant disorder: Clinical presentation and diagnostic change. Journal of American Academy of Child and Adolescent Psychiatry. 1999;38(7):838–45. doi: 10.1097/00004583-199907000-00013. [DOI] [PubMed] [Google Scholar]
  • 57.Stringaris A, Goodman R. Three dimensions of oppositionality in youth. Journal of Child Psychology and Psychiatry. 2009;50(3):216–23. doi: 10.1111/j.1469-7610.2008.01989.x. [DOI] [PubMed] [Google Scholar]
  • 58.Clark LA. Temperament as a unifying basis for personality and psychopathology. J Abnorm Psychol. 2005 Nov;114(4):505–21. doi: 10.1037/0021-843X.114.4.505. [DOI] [PubMed] [Google Scholar]
  • 59.Clark LA, Watson D. Temperament: A new paradigm for trait psychology. In: Pervin LA, John OP, editors. Handbook of personality: Theory and research. 2. New York, NY: Guilford Press; 1999. pp. 399–423. [Google Scholar]
  • 60.Clark S, Muthén BO. Relating latent class analysis results to variables not included in the analysis. under review. [Google Scholar]
  • 61.Hallquist MN, Pilkonis PA. Quantitative Methods in Psychiatric Classification: The Path Forward Is Clear but Complex: Commentary on Krueger and Eaton (2010) Personality Disorders: Theory, Research, and Treatment. 2010;1(2):131–4. doi: 10.1037/a0020201. [DOI] [PubMed] [Google Scholar]

RESOURCES