A Comparison and Integration of Structural Models of Depression and Anxiety in a Clinical Sample: Support for and Validation of the Tri-level Model

Abstract

Prominent structural models of depression and anxiety arise from two traditions: (1) the tri-partite/integrative hierarchical model based on symptom dimensions, and (2) the fear/anxious-misery model based on diagnostic comorbidity data. The tri-level model of depression and anxiety was developed to synthesize these structural models, postulating that narrow (disorder-specific), intermediate (fear and anxious-misery), and broad (general distress) structural factors are needed to most fully account for covaration among these symptoms. Although this model has received preliminary support (Prenoveau et al., 2010), the current study compares it to the above established models and seeks to validate the best-fitting structure. We evaluated the tri-level model and alternative structural models in a large clinical sample (N = 1,000) using bi-factor analysis. In exploratory and confirmatory subsamples, the tri-level model provided a good fit to the data and each of the three levels (narrow, intermediate, and broad) accounted for substantial variance; this model provided a superior fit relative to more parsimonious competing structural models. Furthermore, impairment was independently associated with all three levels of the tri-level model, comorbidity was most closely linked to the broad tri-level dimensions, and the factors generally showed the expected convergent/discriminant associations with diagnoses. Results suggested several revisions to prior research: (1) worry may be best modeled at the broadest structural level, rather than as an indicator of anxious-misery or fear; (2) social interaction anxiety may belong with anxious-misery, rather than fear; and (3) obsessive-compulsive disorder is generally associated with fear disorders, but hoarding is associated with both fear and anxious-misery.

General scientific summary

This study suggests that the structure of depression and anxiety disorders is best characterized by three hierarchical levels: narrow symptoms dimensions, broader fear and anxious-misery dimensions, and a general distress dimension that includes all symptoms. All three levels appear to be clinically meaningful, as each is associated with important outcomes such as comorbidity rates and impairment.

The Diagnostic and Statistical Manual of Mental Disorders (DSM; American Psychiatric Association, 2013) has long served as the primary framework for the taxonomy of psychopathology, wherein disorder groupings were formed atheoretically and descriptively (e.g., Watson, 2005). However, over the past several decades, researchers have increasingly focused on alternative taxonomies that are more closely grounded in empirical findings, with the hope that “carving nature at its joints” might lead to important insights regarding the etiology and treatment of psychological disorders. The within-diagnosis heterogeneity and high rates of diagnostic comorbidity of DSM diagnoses were viewed as particularly problematic (Watson, 2005), as (1) different individuals with the same disorder may have very different presentations and etiologies, and (2) high rates of comorbidity suggest that current diagnostic categories are “split” too narrowly, complicating research and treatment of single disorder entities.

Several approaches have been particularly influential in identifying empirically based phenotypic symptom dimensions in the internalizing disorders. Building upon Clark and Watson’s tripartitite model (1991) and Zinbarg and Barlow’s hierarchical model of the anxiety disorders (1996), Mineka and colleagues (Mineka, Watson, & Clark, 1998) put forth the integrative hierarchical model. Like prior models, the integrative hierarchical model posited that each disorder may be characterized by general component(s) (conceptualized as negative affectivity or general distress) that contribute to comorbidity among disorders, as well as specific or narrow components that distinguish them (i.e., anxious arousal for panic, low positive affectivity for depression). Mineka and colleagues further posited that the size of the general and specific components varies across disorders. For instance, depression and generalized anxiety disorder (GAD) are strongly linked to negative affectivity, whereas disorders such as social anxiety disorder and obsessive-compulsive disorder (OCD) show a substantially weaker relationship with negative affectivity (e.g., Brown, Chorpita, & Barlow, 1998; Watson, Gamez, & Simms, 2005). In addition, Mineka and colleagues suggested that symptom specificity should be viewed in relative, rather than absolute, terms. For instance, while low positive affectivity is relatively specific to depression, it is also associated with social anxiety disorder (e.g., Brown et al., 1998; Naragon-Gainey, Watson, & Markon, 2009). Taken together, each disorder may be characterized by a combination of general and narrow/specific components, wherein the magnitude of the association with each component can be specified. Of note, the integrative hierarchical model acknowledged the possible existence of intermediate structural factors that characterize some but not all disorders, but such intermediate factors were not specified or described in this model.

A second approach was developed by Krueger (1999), who directly analyzed disorder comorbidity data under the assumption that comorbidity rates are likely to reflect shared etiological processes and characteristics. Based primarily on diagnoses (although a few studies have analyzed symptom dimensions), these phenotypic and genetic structural models have converged on a two factor structure of internalizing disorders: (1) fear disorders, and (2) anxious-misery disorders (also called distress disorders) (e.g., Kendler, Prescott, Myers, & Neale, 2003; Krueger, 1999; Sellbom, Ben-Porath, & Bagby, 2008; Slade & Watson, 2006; Watson, 2005). In these models, the fear disorders consist of panic disorder, social anxiety disorder, agoraphobia, and specific phobia, whereas the anxious-misery disorders include depression, dysthymia, posttraumatic stress disorder, and GAD. However, the optimal placement in this structure for several disorders remains ambiguous. There are relatively few data addressing OCD’s location in this structure, with conflicting evidence regarding whether OCD is a fear disorder (Miller, Fogler, Wolf, Kaloupek, & Keane, 2008; Prenoveau et al., 2010; Slade & Watson, 2006), is associated with both fear and distress disorders (Raines, Allan, Oglesby, Short, & Schmidt, 2015; Wright et al., 2013), or whether it does not belong with either the distress or fear disorders (Cox, Clara, Hills, & Sareen, 2010; Markon, 2010; Sellbom et al., 2008). Although GAD is generally placed with the anxious-misery disorders, some have argued that worry should be grouped with both the anxious-misery and the fear disorders (for a review, see Mennin, Heimberg, Fresco, & Ritter, 2008). Similarly, social anxiety has typically served as an indicator of the fear disorders, but Prenoveau and colleagues (2010) reported that it loaded with both the fear and anxious-misery disorders, consistent with findings that social anxiety shares dispositional features with depression and GAD (e.g., Naragon-Gainey & Watson, 2011). Finally, recent studies have also directly identified or implied a general psychopathology factor (p-factor) that accounts for shared variance among internalizing, externalizing, and thought disorders and is substantially correlated with neuroticism or negative affectivity (Caspi et al., 2014; Lahey et al., 2012; Markon, 2010).

While both approaches have furthered knowledge of the structure of internalizing disorders, they have some complementary strengths and weaknesses. The integrative hierarchical model has primarily identified symptom-specific narrow factors and a general factor. In contrast, approaches analyzing disorder comorbidity have identified intermediate structural factors (i.e., fear and anxious-misery factors), but generally do not focus on more narrow factors because disorders, rather than symptoms, serve as latent variable indicators in these models. In such a case, each disorder is assessed with a single diagnosis, precluding the formation of a narrow, symptom- or disorder-specific factor since multiple indicators are required to form a latent variable.¹ Thus, neither approach has included the full breadth of latent variables that may be needed in a complete structural model of the internalizing disorders and their underlying symptoms.

The tri-level model (Prenoveau et al., 2010) attempts to bridge these two approaches by simultaneously modeling narrow symptom dimensions that are (relatively) disorder-specific, intermediate factors corresponding to fear and anxious-misery, and a single general distress factor that accounts for broad comorbidity. In the initial examination of this model, a bi-factor analysis was used in which all factors were specified as independent of one another to evaluate the unique association of indicators with each factor, as well as whether all factors are needed to account for covariances in the model. Based on the items from multiple common self-report inventories of mood and anxiety symptoms, the narrow symptom dimensions identified were depression, fears of specific stimuli, anxious arousal/somatic tension, social fears, and interoceptive/agoraphobic fears (a positive affect factor was also modeled, but was removed from the final model because it accounted for minimal variance). This model fit the data well in a sample of non-clinical adolescents (though high-risk adolescents were over-sampled) and the structure remained stable when re-assessed one year later. Importantly, the factors at all three levels were associated with disorder clinical severity ratings (CSRs) in a theoretically consistent manner, and the narrow factors showed specificity to the corresponding disorder CSRs.

The findings of Prenoveau et al. (2010) suggest that it may be necessary to consider all three hierarchical levels (narrow, intermediate, and broad) to fully model the structure of internalizing disorders, but this study was limited in several ways. First, it was conducted in a non-clinical adolescent sample, although the sample was likely more pathological than a standard community sample because high-risk individuals with elevated levels of neuroticism were over-sampled. Structural analyses of clinical samples generally show evidence of a particularly strong general distress or demoralization factor (e.g., Naragon-Gainey, Gallagher, & Brown, 2013; Noordhof, Sellbom, Eigenhuis, & Kamphuis, 2015), suggesting that narrow and/or intermediate factors may be less informative among those who are clinically distressed. Furthermore, the inevitable restriction of range with regard to psychopathology (particularly disorders with relatively low base rates) in a non-clinical sample may have weakened or distorted some associations. Thus, it is important to test this model in a clinical sample to directly assess its generalizability and utility among individuals with current psychopathology and higher levels of distress. Second, Prenoveau and colleagues did not include indicators of obsessive-compulsive symptoms or generalized anxiety/worry. Given that the results of any structural model are limited by and a function of the included indicators, it is necessary to include a broader set of relevant internalizing symptom dimensions before drawing strong conclusions regarding structure.

The aim of the current study is to replicate and extend the tri-level model in a large clinical outpatient sample with high rates of mood and anxiety disorders, as well as to compare it to other established or plausible structural models (e.g., a model in the tradition of the tripartite/integrative hierarchical model with narrow factors and general distress only; a model in the tradition of fear/anxious-misery comorbidity models with intermediate factors and general distress only; a single factor model) and to examine the validity and contribution of each level of the factors. The sample was split into two subsamples to first establish the model in a data-driven, exploratory manner and then confirm it in the second independent sample. After establishing a sound and replicable specification of the tri-level model, comparisons with competing models and validity analyses were conducted using the full sample, in order to maximize the precision of parameter estimates.

As in Prenoveau et al. (2010), we used bi-factor analysis to specify the tri-level model, which allows each item to load directly on multiple uncorrelated factors that reflect different levels of abstraction. Thus, all items load on general distress, as well as on the fear or distress factor and an appropriate narrow factor (see Figure 1 for an illustration of the model). The bi-factor model has been used in numerous recent structural studies of psychopathology due to several advantages over higher-order models (i.e., models wherein broad latent variables are indicated by multiple more narrow latent variables, rather than the observed indicators) (e.g., Caspi et al., 2014; Kim & Eaton, 2015; Lahey et al., 2012; Sharp et al., 2015; Zinbarg & Barlow, 1996). In bi-factor analysis, the narrow, intermediate, and broad factors are statistically independent of one another, which allows for a stringent test of whether each factor is structurally necessary after accounting for all of the other factors (e.g., Chen, West, & Sousa, 2006). For example, Caspi and colleagues (2014) found that internalizing and externalizing factors, in addition to general distress, were needed in order to explain covariation among mental disorders. In the current study, we test the need for each structural level of the tri-level model by comparing it to more parsimonious models with fewer factors (i.e., models implied by the integrative hierarchical model and the fear/anxious-misery model), which should be favored if they account for the data equally well. In addition, the magnitude of the relations between each factor and their indicators are directly quantified by their factor loadings in bi-factor models (e.g., Chen et al., 2006), and we use this desirable property to determine how much common variance is explained by narrow factors and by more general factors.

Figure 1.

Structural specification for the initial (baseline) tri-level model. Note that items are not labeled and not all items are shown for the sake of presentational clarity (see Table 1 for item indicators of each narrow factor). Item error terms are also not shown. GD = general distress, A–M = anxious-misery, Dep = depression, PA = positive affect, Soc Anx = social anxiety, Spec = specific fears, OC = obsessive-compulsive symptoms, AA/SS = anxious arousal/somatic sensations, Int/Agor = interoceptive/agoraphobic fears. All factors are uncorrelated.

Another important advantage of bi-factor models compared to higher-order models is that they enable examination of the incremental validity of each factor (over and above the other factors) in predicting external variables (e.g., Chen et al., 2006). The necessity of multiple levels of specificity in predicting meaningful external outcomes has been demonstrated in a number of studies using bi-factor models (e.g., Caspi et al., 2014; Lahey et al., 2012; Sharp et al., 2015). For example, the externalizing factor, but not the internalizing factor, was independently related to suicide attempts and psychiatric hospitalization even after accounting for the strong relation between general psychopathology and these variables (Caspi et al., 2014). In the current study, we examine the validity of the tri-level factors via their associations with impairment and with an index of comorbidity (i.e., number of diagnoses). We expect that, if there is substantial and meaningful variance at all three levels of the tri-level model, impairment should be independently associated with all three levels. However, comorbidity should more closely track the broad factors due to their non-specific nature, and it should be particularly closely linked with General Distress. We also examine the convergent and discriminant validity of each of the tri-level factors with diagnoses of relevant anxiety and depressive disorders, with the hypothesis that the narrow factors will be specifically related to their corresponding diagnosis (e.g., Anxious Arousal/Somatic Sensations should be specifically linked to panic with agoraphobia), the intermediate factors will be specifically related to their constituent disorders, and General Distress will be more broadly related to the disorders.

Method

Participants and Procedure

The sample consisted of 1,000 adults who presented for assessment and treatment at the Center for Anxiety and Related Disorders at Boston University² between July 2002 and March 2007; for our initial analyses involving the specification of the tri-level model, they were divided into two groups of 500 (based on order of presentation at the clinic) to provide independent exploratory and confirmatory samples. The majority of the sample was female (58.7%) and the average age was 32.66 years old (SD = 11.80, range = 18 to 79). Most participants identified as Caucasian (88.4%), and the remaining participants identified as African-American (3.0%), Asian (4.4%), Latino/Hispanic (3.9%), or Other/Multiple (0.3%). Participants completed self-report questionnaires, as well as a clinical diagnostic interview at the Center (see below). Rates of current clinical disorders occurring frequently in the sample were: social anxiety disorder (47.2%), major depressive disorder or dysthymic disorder (34.2%), GAD (28.3%), panic disorder with or without agoraphobia (24.6%), OCD (16.4%), and specific phobia (15.2%).

Measures

Diagnostic interview

Current and past diagnoses were established with the Anxiety Disorders Interview Schedule for DSM–IV: Lifetime version (ADIS-IV-L; Di Nardo, Brown, & Barlow, 1994) (though only current diagnoses are reported in this study). The ADIS-IV-L is a semi-structured interview designed to ascertain reliable diagnosis of the DSM–IV anxiety, mood, somatoform, and substance use disorders and to screen for the presence of other conditions (e.g., psychotic disorders). The ADIS-IV-L provides dimensional assessment of the key and associated features of disorders (0–8 ratings); such features are dimensionally rated regardless of whether a formal DSM–IV diagnosis is under consideration. A reliability study entailing two independent administrations of the ADIS-IV-L indicated good-to-excellent interrater agreement for current disorders (range of κs = .67 to .86) except dysthymia (κ = .31; Brown, Di Nardo, Lehman, & Campbell, 2001).

Self-report questionnaires

Individual items were selected to match those in the original tri-level model (Prenoveau et al., 2010). When identical or near-identical indicators of a particular construct were not available, other items that were known to be strong indicators of the construct were included instead. For example, the social anxiety factor in the original tri-level model primarily consisted of items assessing performance social anxiety, whereas the items available in this sample largely measured social interaction anxiety. In addition, to provide better coverage of the general distress construct, items from affective or trait measures were used to measure this content when necessary. Table 1 shows the specific items that were used as indicators of each factor.

Table 1.

Completely Standardized Factor Loadings on Narrow Symptom Factors in Sample 1 (Final Model)

	Standardized factor loadings
Indicators	Intero/Agor	AA/Tens	Social Fears	Spec Fears	Worry	OC	Depression	PA
APPQ11 (away from home alone)	.68
APPQ7 (getting gas at dentist)	.64
APPQ13 (alone in isolated area	.75
APPQ27 (going over bridge)	.63
APPQ10 (exercise alone)	.68
APPQ26 (effects of alcohol)	.63
APPQ6 (hiking on hot day)	.75
BAI1 (numbness/tingling)		.67
BAI11 (feeling of choking)		.55
BAI2 (felt hot)		.62
DASS7 (trembling)		.66
BAI6 (dizzy/lightheaded)		.84
DASS4 (difficulty breathing)		.68
BAI3 (wobbly legs)		.72
BAI8 (unsteady)		.82
BAI19 (faint)		.86
DASS19 (heart palpitations)		.66
DASS2 (dry mouth)		.58
BAI18 (indigestion)		.63
SIAS4 (mixing at work)			.77
SIAS6 (tense w/acquaintance)			.78
SIAS7 (uncomfortable mixing)			.89
SIAS10 (difficulty talking)			.89
SIAS12 (feel awkward)			.82
SIAS14 (talking w/attractive)			.71
SIAS15 (don’t know what to say)			.88
SIAS16 (nervous mixing)			.90
SIAS17 (say embarrassing)			.85
SIAS19 (tense in groups)			.93
APPQ1 (talking to people)			.89
APPQ12 (introducing to groups)			.83
APPQ21 (speaking on phone)			.73
APPQ22 (meeting strangers)			.87
APPQ24 (enter crowded room)			.80
SFQ1 (snakes)				.61
SFQ2 (insects)				.72
SFQ3 (dogs)				.65
SFQ4 (cats)				.72
SFQ5 (rodents)				.73
SFQ6 (other animal)				.52
SFQ7 (high places)				.53
SFQ8 (thunderstorms)				.64
SFQ9 (water)				.51
SFQ11 (blood in minor cut)				.67
SFQ12 (injection/shot)				.52
SFQ13 (blood drawn)				.54
PSWQ1 (worry about time)					.42
PSWQ2 (worries overwhelm)					.85
PSWQ3 (don’t tend to worry)					.70
PSWQ4 (worry in situations)					.76
PSWQ5 (can’t help worry)					.83
PSWQ6 (worry under pressure)					.73
PSWQ7 (always worry)					.90
PSWQ8 (easy to dismiss worry)					.64
PSWQ9 (worry after finish)					.70
PSWQ10 (never worry)					.69
PSWQ11 (don’t worry)					.50
PSWQ12 (worrier)					.67
PSWQ13 (notice worrying)					.81
PSWQ14 (can’t stop worrying)					.86
PSWQ15 (worry all the time)					.90
PSWQ16 (worry until done)					.66
OCI1 (saved up many things)						.50
OCI7 (collect things don’t need)						.49
OCI13 (avoid throwing away)						.64
OCI12 (unpleasant thoughts)						.68
OCI18 (nasty thoughts)						.56
OCI5 (hard to touch dirty objects)						.65
OCI11 (have to wash self)						.70
OCI17 (wash hands often)						.69
OCI3 (upset if not arranged)						.55
OCI9 (upset if others rearrange)						.59
OCI15 (need to arrange)						.58
OCI8 (repeatedly check locks)						.66
OCI14 (repeatedly check switches)						.72
OCI2 (check too often)						.71
OCI4 (compelled to count)						.59
OCI10 (repeat numbers)						.68
OCI16 (good/bad numbers)						.58
BDI1 (felt sad)							.77
BDI7 (low self-esteem)							.74
N41 (felt discouraged)							.72
BDI2 (lack of interest)							.81
BDI8 (blamed self)							.66
BDI17 (irritable)							.61
DASS16 (not enthusiastic)							.85
DASS10 (nothing look forward)							.87
BDI4 (lack of pleasure)							.69
BDI11 (restless/agitated)							.51
DASS21 (life meaningless)							.83
DASS5 (no initiative)							.77
BDI15 (lack of energy)							.78
BDI13 (difficulty w/decisions)							.67
N37 (cheerful)								.86
N47 (fast pace)								.46
PAN3 (excited)								.57
N42 (not optimist)								−.83
PAN10 (proud)								.65
N12 (not light-hearted)								−.59
PAN9 (enthusiastic)								.74
N7 (laugh easily)								.63
N32 (bursting w/energy)								.47

Note. N = 500. AA/ Sens = Anxious Arousal/Somatic Sensations; Intero/Agor = Interoceptive/Agoraphobic Fears; Spec Fears = Specific Fears; OC = Obsessive-Compulsive Symptoms; PA = Positive Affect; APPQ = Albany Panic and Phobia Questionnaire; BAI = Beck Anxiety Inventory; DASS = Depression Anxiety Stress Scales; SIAS = Social Interaction Anxiety Scale; SFQ = Specific Fears Questionnaire; PSWQ = Penn State Worry Questionnaire; OCI = Obsessive Compulsive Inventory; BDI = Beck Depression Inventory; N = NEO Five Factor Inventory; PAN = Positive and Negative Affect Scales.

Interoceptive/Agoraphobic Fears were assessed with items from the 8-item Interoceptive and 9-item Agoraphobia scales of the Albany Panic and Phobia Questionnaire (APPQ; Rapee, Craske, & Barlow, 1994/1995; cf. Brown, White, & Barlow, 2005). The Anxious Arousal/Somatic Sensations items were taken from the Beck Anxiety Inventory (BAI; Beck & Steer, 1993), a 21-item scale assessing somatic symptoms of anxiety, and the Anxiety Scale from the 21-item version of the Depression Anxiety Stress Scales (DASS; Lovibond & Lovibond, 1995; cf. Antony, Bieling, Cox, Enns, & Swinson, 1998). Social Fears items were drawn from the 20-item Social Interaction Anxiety Scale (SIAS; Mattick & Clarke, 1998; cf. E. J. Brown et al., 1997) and the 10-item APPQ Social Phobia scale (Rapee et al., 1994/1995). Specific Fears were assessed with items from the Specific Phobia Questionnaire (SPQ), which is a self-report version of the ADIS-IV-L specific phobia feared situations. The items assessing Worry consisted of the Penn State Worry Questionnaire items (PSWQ; Meyer, Miller, Metzger, & Borkovec, 1990; cf. Brown, Antony, & Barlow, 1992), a 16-item self-report measure of chronic worry. The Obsessive-Compulsive Symptoms items were from the Obsessive-Compulsive Inventory – Revised (OCI-R; Foa et al., 2002), a commonly-used 18-item measure of obsessions, compulsions, and hoarding symptoms. Depression items were drawn from the Beck Depression Inventory – II (BDI-II; Beck, Steer, & Brown, 1996), the Depression scale of the DASS (Lovibond & Lovibond, 1995), and the Neuroticism Scale of the NEO Five-Factor Inventory (NFFI; Costa & McCrae, 1992). Last, Positive Affect items were taken from the Positive Affect Scale of the PANAS (PANAS; Watson, Clark, & Tellegen, 1988) and the Extraversion scale of the NFFI (Costa & McCrae, 1992).

Self-reported impairment was also assessed to examine the validity of the tri-level factors, using the sum of the five items from the Subjective Symptoms Scale (SSS). The SSS is a modification of a scale introduced by Hafner and Marks (1976) (Work and Social Adjustment Scale [WSAS]; see also Mataix-Cols et al., 2005; Mundt, Marks, Shear, & Greist, 2002; Naragon-Gainey, Gallagher, & Brown, 2014), and respondents rate the extent (ranging from 0 = not at all to 8 = severe) that symptoms have interfered with five different areas of daily functioning (private leisure, work, household tasks, social leisure, family relationships) over the past week. The measure has shown strong internal consistency and evidence of good construct validity (Mataix-Cols et al., 2005; Mundt, Marks, Shear, & Greist, 2002; Naragon-Gainey et al., 2014).

Data Analysis

The raw data were analyzed using a latent variable software program (Mplus 7.4; Muthén & Muthén, 1998–2015), and robust weighted least squared estimation (WLSMV) was utilized because it is recommended for categorical outcomes (Muthén, du Toit, & Spisic, 1997). Goodness of fit of the models was evaluated using the root mean square error of approximation (RMSEA) and its 90% confidence interval, the Tucker–Lewis index (TLI), and the comparative fit index (CFI). Acceptable model fit was defined in part by the criteria described by Hu and Bentler (1999): RMSEA values close to 0.06 or below, and CFI and TLI values close to .95 or above. Because the χ² difference between models is not distributed as χ² with WLSMV estimators, the DIFFTEST feature of Mplus was used to compare competing models. The acceptability of the models was further evaluated by the presence/absence of salient localized areas of strains in the solutions (e.g., modification indices) and the strength and interpretability of the parameter estimates.

Results

Narrow Factor Structure

We first assessed the structure of the eight narrow factors (i.e., Interoceptive/ Agoraphobic Fears, Anxious Arousal/Somatic Sensations, Social Fears, Specific Fears, Worry, Obsessive-Compulsive Symptoms, Depression, and Positive Affect), excluding the intermediate and general level factors, in Sample 1 to determine whether the selected items were good indicators of their assigned factors (see the “Measures” section for a description of item selection).³ All factors were allowed to correlate in the confirmatory factor analysis. Several error covariances were specified a priori to account for sources of unique variance, based on prior empirical research and theory. First, based on previous research indicating strong covariances among the items of each OCI-R subscale (i.e., Washing, Obsessing, Checking, Ordering, Hoarding, and Neutralizing; e.g., Foa et al., 2002; Huppert et al., 2007), the error terms among the items within each OCI-R subscale were allowed to covary. Second, following the empirical findings of Brown (2003), we allowed the error terms of the PSWQ reverse keyed items to covary, as well as correlations between items 9 (worry after finishing) and 16 (worry until done), and items 7 (always worry) and 15 (worry all of the time) due to similar wording. Third, covariances among the error terms of the three PANAS items were permitted due to evidence of strong method effects for these items that use single adjectives (e.g., Chmielewski & Watson, 2009), rather than full sentences/phrases as found in all of the other questionnaires in this study.

The initial model was a borderline to acceptable fit to the data, χ² (5300) =8242.94, p < .001, CFI = .93, TLI = 0.93, RMSEA = 0.033 (90% CI = 0.032, 0.035), and all specified loadings and error covariances were significant. However, an examination of model parameter estimates and modification indices suggested several minor revisions to the model. Items with standardized factor loadings below |.40| were removed, resulting in the removal of BDI19 (can’t concentrate; loading = .16) and BDI20 (tired/fatigued; loading = .23) from the Depression factor. In addition, modification indices indicated that OCI28 (difficulty controlling thoughts) was not a specific indicator of OCD, as it had a large modification index on Depression, Positive Affect, and Worry. Therefore, this item was also removed. Note that these three items tap broad problems that are common across numerous symptom dimensions, providing theoretical support for their removal from the symptom-specific factors. Last, one post hoc error covariance was freed given a very large modification index between the two specific fear items assessing fear of needles (SFQ12 and SFQ13). This revised model provided a very good fit to the data: χ² (4993) = 7153.10, p < .001, CFI = .95, TLI = 0.95, RMSEA = 0.029 (90% CI = 0.028, 0.031), and standardized factor loadings from the CFA are shown in Table 1.

Tri-level Model Specification

The full tri-level model was then examined in Sample 1, including the eight narrow factors as specified in Table 1 as well as intermediate factors (i.e., Anxious-Misery and Fear) and a single General Distress factor (see Figure 1). Based on the literature reviewed earlier, we hypothesized that the items from the Depression, Positive Affect, and Worry factors would load on the intermediate Anxious-Misery factor, whereas the items from the Social Anxiety, Specific Fears, Obsessive-Compulsive Symptoms, Anxious Arousal/Somatic Sensations, and Interoceptive/Agoraphobic Fears factors would load on the intermediate Fear factor. However, we also tested the relative fit of a number of alternative nested models using χ² difference tests, as previous research was mixed regarding the best placement of several symptoms on the intermediate factors (see the Introduction). All items were allowed to load on the General Distress factor, and all factors were specified to be uncorrelated.

The fit indices and χ² difference tests for the baseline model and competing specifications of the tri-level model in Sample 1 are shown in the top portion of Table 2. The baseline model described above had a borderline to acceptable fit to the data: χ² (4817) = 7727.52, p < .001, CFI =.94, TLI = 0.93, RMSEA = 0.035 (90% CI = 0.033, 0.036). Three different specifications of the intermediate factors were assessed next in separate models: Worry items loading on Fear in addition to Anxious-Misery; Social Anxiety items loading on Anxious-Misery in addition to Fear; and OCD items loading on Anxious-Misery in addition to Fear. In each of these cases, omitting the additional loadings (that is, using the specification of the baseline tri-level model) resulted in a significant decrement in model fit relative to these three models (ps < .001; see Table 2 for full results).

Table 2.

Fit Indices for Structural Models

Model	Model χ2 (df)	χ2 diff test (df)	CFI	TLI	RMSEA (90% CI)
Competing specifications of the intermediate tri-level factors in Sample 1a
1. Baseline tri-level model	7,727.52(4,817)***	.--	.94	0.93	0.035 (0.033, 0.036)
2. Add Worry to Fear	7,033.37(4,801)***	814.43(16)***	.95	0.95	0.030 (0.029, 0.032)
3. Add Social Anxiety to Anxious-Misery	6,509.79(4,802)***	697.33(15)***	.96	0.96	0.027 (0.025, 0.028)
4. Add OC to Anxious-Misery	7,440.76(4,800)***	117.67(18)***	.94	0.94	0.033 (0.032, 0.035)
5. Final tri-level model	6,383.19(4,770)***	790.88(47)***	.96	0.96	0.026 (0.024, 0.028)
Competing structural models in the Combined Sampleb
1. Tri-level model	8,505.19(4,770)***	.--	.96	0.96	0.028 (0.027, 0.029)
2. Single factor model	42,485.40(5,021)***	12,328.04(252)***	.62	0.61	0.086 (0.086, 0.087)
3. Intermediate and General Distress factors only	18,752.88(4,871)***	5,043.66(102)***	.86	0.85	0.053 (0.053, 0.054)
4. Narrow and General Distress factors only	15,430.59(4,919)***	2,570.49(150)***	.89	0.89	0.046 (0.045, 0.047)
5. Narrow and intermediate factors only	10,776.59(4,871)***	974.20(102)***	.94	0.93	0.035 (0.034, 0.036)

N = 500 for Sample 1; N = 1000 for the Combined Sample.

^***p < .001. AA/Somatic Sens = Anxious Arousal/Somatic Sensations; OC = Obsessive-Compulsive Symptoms.

^aFor presented χ² difference tests of nested models, Model 1 is nested within Models 2–5.

^bModels 2–5 are nested within Model 1.

Based on these results, a final tri-level was evaluated that incorporated all of the changes that significantly improved model fit. In this model, Depression, Positive Affect, Worry, Social Anxiety, and Obsessive-Compulsive symptom items loaded on the Anxious-Misery factor, whereas the Social Anxiety, Specific Fears, Obsessive-Compulsive Symptoms, Anxious Arousal/Somatic Sensations, Interoceptive/Agoraphobic Fears, and Worry items loaded on the Fear Factor, and all items loaded on the General Distress Factor. This final tri-level model was an excellent fit to the data in Sample 1: χ² (4770)= 6383.19, p < .001, CFI = .96, TLI = 0.96, RMSEA = 0.026 (90% CI = 0.024, 0.028). It was then evaluated in Sample 2 to assess its generalizability to an independent sample. The model was an excellent fit to the data in Sample 2: χ² (4770) = 6311.09, p < .001, CFI = .97, TLI = 0.96, RMSEA = 0.025 (90% CI = 0.024, 0.027), as well as in the combined sample: χ² (4770) = 8595.19, p < .001, CFI = .96, TLI = 0.96, RMSEA = 0.028 (90% CI = 0.027, 0.029). Therefore, this model was retained, and all subsequent analyses use the combined sample (rather than separate subsamples) to maximize the precision of parameter estimates.

Standardized factor loadings in the combined sample for the narrow, intermediate (i.e., Anxious- Misery and Fear) factors, and General Distress factor are shown in Table 3. Of note, each level of the model accounted for substantial common variance: the narrow factors accounted for nearly half of the common variance (48%), the intermediate factors accounted for 32% of the common variance, and General Distress accounted for the remaining 20% (see the bottom of Table 3 for estimates for each narrow factor). Standardized loadings were generally strong and significant on all factors, with several notable systematic exceptions. Only one of the SIAS items assessing social anxiety loaded significantly on the Fear factor, whereas all of the APPQ social phobia items loaded significantly on Fear. Half of the Positive Affect items did not load significantly on the General Distress factor, and most of the OC items did not load significantly on Anxious-Misery (the only significant loadings were all three hoarding items, one cleaning item, and one ordering item). Last, the Specific Fears narrow factor was primarily defined by animal fears, and the three blood/injection/injury items failed to load significantly on this factor.

Table 3.

Completely Standardized Factor Loadings for the Final Tri-level Model in the Combined Sample

	Standardized factor loadings
Indicators	Dep.	PA	Soc.	Worry	OC	Spec.	AA/Som	Int/Agor	Fear	A-M	Gen. Dist.
BDI1 (felt sad)	.49									.43	.46
BDI7 (low self-esteem)	.35									.50	.42
N41 (felt discouraged)	.00									.52	.47
BDI2 (lack of interest)	.49									.51	.41
BDI8 (blamed self)	.17									.46	.45
BDI17 (irritable)	.29									.17	.53
DASS16 (not enthusiastic)	.51									.57	.42
DASS10 (nothing look forward)	.63									.51	.40
BDI4 (lack of pleasure)	.52									.44	.34
BDI11 (restless/agitated)	.37									.12	.45
DASS21 (life meaningless)	.54									.55	.36
DASS5 (no initiative)	.34									.49	.46
BDI15 (lack of energy)	.46									.39	.48
BDI13 (difficulty w/decisions)	.43									.29	.46
N37 (cheerful)		.53								−.71	−.14
N47 (fast pace)		−.02								−.56	.17
PAN3 (excited)		.23								−.65	.10
N42 (not optimist)		−.46								.59	.30
PAN10 (proud)		−.04								−.68	−.11
N12 (not light-hearted)		−.41								.38	.24
PAN9 (enthusiastic)		.21								−.71	−.09
N7 (laugh easily)		.35								−.51	−.14
N32 (bursting w/energy)		.27								−.43	.05
SIAS4 (mixing at work)			.57				.00			.51	.13
SIAS6 (tense w/acquaintance)			.64				.07			.40	.16
SIAS7 (uncomfortable mixing)			.69				.00			.57	.11
SIAS10 (difficulty talking)			.70				.00			.52	.13
SIAS12 (feel awkward)			.71				−.06			.39	.27
SIAS14 (talking w/attractive)			.52				.06			.44	.13
SIAS15 (don’t know what to say)			.74				−.02			.50	.17
SIAS16 (nervous mixing)			.71				.03			.52	.14
SIAS17 (say embarrassing)			.73				−.06			.34	.32
SIAS19 (tense in groups)			.73				.05			.57	.12
APPQ1 (talking to people)			.75				.09			.41	.20
APPQ12 (introducing to groups)			.70				.19			.37	.15
APPQ21 (speaking on phone)			.64				.16			.30	.12
APPQ22 (meeting strangers)			.72				.23			.44	.18
APPQ24 (enter crowded room)			.69				.28			.33	.22
PSWQ1 (worry about time)				.36			.11			.07	.19
PSWQ2 (worries overwhelm)				.48			.09			.19	.69
PSWQ3 (don’t tend to worry)				.61			.18			.09	.30
PSWQ4 (worry in situations)				.61			.20			.21	.46
PSWQ5 (can’t help worry)				.66			.12			.08	.51
PSWQ6 (worry under pressure)				.49			.13			.22	.45
PSWQ7 (always worry)				.63			.19			.17	.56
PSWQ8 (easy to dismiss worry)				.53			.11			.17	.33
PSWQ9 (worry after finish)				.44			.16			.17	.47
PSWQ10 (never worry)				.66			.21			.11	.33
PSWQ11 (don’t worry)				.39			.06			.14	.29
PSWQ12 (worrier)				.62			.18			.12	.38
PSWQ13 (notice worrying)				.63			.10			.05	.52
PSWQ14 (can’t stop worrying)				.57			.10			.11	.64
PSWQ15 (worry all the time)				.58			.14			.15	.65
PSWQ16 (worry until done)				.39			.12			.14	.52
OCI1 (saved up many things)					.34		.20			.19	.22
OCI7 (collect things don’t need)					.41		.16			.19	.23
OCI13 (avoid throwing away)					.41				.24	.22	.26
OCI12 (unpleasant thoughts)					.13				.17	−.08	.59
OCI18 (nasty thoughts)					.21				.12	.04	.44
OCI5 (hard to touch dirty objects)					.44				.38	.09	.26
OCI11 (have to wash self)					.50				.30	.03	.37
OCI17 (wash hands often)					.48				.31	−.03	.35
OCI3 (upset if not arranged)					.41				.27	.02	.32
OCI9 (upset if others rearrange)					.48				.28	.16	.29
OCI15 (need to arrange)					.51				.29	.01	.29
OCI8 (repeatedly check locks)					.54				.36	−.01	.29
OCI14 (repeatedly check switches)					.53				.37	−.02	.30
OCI2 (check too often)					.54				.26	.11	.41
OCI4 (compelled to count) .					.57				.26	−.06	.26
OCI10 (repeat numbers)					.54				.35	−.03	.31
OCI16 (good/bad numbers)					.44				.33	−.08	.29
SFQ1 (snakes)						.67			.44		.11
SFQ2 (insects)						.41			.44		.31
SFQ3 (dogs)						.34			.44		.21
SFQ4 (cats)						.37			.51		.24
SFQ5 (rodents)						.61			.50		.21
SFQ6 (other animal)						.35			.35		.14
SFQ7 (high places)						.13			.59		.05
SFQ8 (thunderstorms)						.17			.66		.20
SFQ9 (water)						.14			.57		.12
SFQ11 (blood in minor cut)						.01			.70		.16
SFQ12 (injection/shots)						−.03			.54		.13
SFQ13 (blood drawn)						−.08			.52		.17
BAI1 (numbness/tingling)							.53		.17		.31
BAI11 (feeling of choking)							.59		.38		.05
BAI2 (felt hot)							.35		.19		.32
DASS7 (trembling)							.46		.01		.43
BAI6 (dizzy/lightheaded)							.77		.24		.31
DASS4 (difficulty breathing)							.58		.19		.33
BAI3 (wobbly legs)							.67		.19		.27
BAI8 (unsteady)							.65		.21		.39
BAI19 (faint)							.80		.23		.24
DASS19 (heart palpitations)							.54		.12		.32
DASS2 (dry mouth)							.33		.08		.39
BAI18 (indigestion)							.37		.14		.37
APPQ11 (away from home alone)								.47	.45		.21
APPQ7 (getting gas at dentist)								.35	.54		.11
APPQ13 (alone in isolated area)								.29	.56		.24
APPQ27 (going over bridge)								.20	.70		.04
APPQ10 (exercise alone)								.69	.41		.21
APPQ26 (effects of alcohol)								.36	.44		.19
APPQ6 (hiking on hot day)								.68	.47		.17
Proportion of common variance explained by each factor	.05	.02	.13	.09	.07	.03	.07	.03	.14	.18	.20

N = 1000. APPQ = Albany Panic and Phobia Questionnaire; BAI = Beck Anxiety Inventory; DASS = Depression Anxiety Stress Scales; SIAS = Social Interaction Anxiety Scale; SFQ = Specific Fears Questionnaire; PSWQ = Penn State Worry Questionnaire; OCI = Obsessive Compulsive Inventory; BDI = Beck Depression Inventory; N = NEO Five Factor Inventory; PAN = Positive and Negative Affect Scales; AA = Anxious Arousal. Loadings in bold are not significant at p < .05.

Comparisons to Competing Structural Models

Next, the final tri-level model was compared with several more parsimonious competing structural models (see the bottom portion of Table 2 for fit indices and nested χ² difference tests).⁴ First, we examined whether a single factor could adequately reproduce the covariances among items, similar to one of the models tested in Capsi et al. (2014). This model fit poorly and significantly worse than the tri-level model (p < .001). Next, we examined two models that roughly correspond to the major structural approaches that the tri-level model attempts to integrate: (1) intermediate factors and General Distress only (similar to anxious-misery/fear models based on diagnoses), and (2) narrow factors and General Distress only (similar to the tri-partite/integrative hierarchical model). Both fit the data poorly and resulted in a significant decrement in fit relative to the tri-level model (ps < .001). Last, a model that only included the narrow and intermediate factors (no General Distress) yielded a borderline to acceptable fit to the data (χ² = 10,776.59(4871), p < .001, CFI = .94, TLI = 0.93, RMSEA = 0.035 (90% CI = 0.034, 0.036). This model also resulted in a significant decrement in fit when compared to the tri-level model (p < .001).

Associations of Diagnoses, Comorbidity, and Impairment with the Tri-level Factors

We examined correlations of the tri-level model factors with the total number of diagnoses assigned by the clinician (this includes anxiety, depressive, bipolar, somatic, and substance use disorders) and with self-reported impairment to test criterion validity with outcomes that mark general psychopathology severity. These correlations were computed directly within the tri-level structural model (rather than by extracting factor scores and correlating them with diagnoses and impairment). In addition, correlations of specific diagnoses were calculated to examine patterns of convergent and discriminant validity for the tri-level factors (see Table 4 for all correlations). Note that because all of the tri-level factors are orthogonal to one another, significant correlations indicate specific and incremental associations relative to the other factors. In the examination of convergent/discriminant validity, we included DSM-IV diagnoses of panic disorder with or without agoraphobia (PDA), social phobia (SOC), specific phobia (SPEC), GAD, OCD, and a combined variable for major depressive disorder and/or dysthymic disorder (DEP). Steiger’s z-test for dependent correlations (Meng, Rosenthal, & Rubin, 1992) was used to compare the magnitudes of correlations between each specific diagnosis and each tri-level factor, using a p value of .001 (z = 3.09) because of the large number of comparisons involved.

Table 4.

Correlations of Self-Reported Impairment and DSM-IV Diagnoses with the Tri-level Model Factors in the Combined Sample

	Diagnoses
Factors	PDA	SOC	SPEC	GAD	OCD	DEP	NUM DIS	IMP
Narrow factors
Interoceptive/Agoraphobic Fears	.62***	−.30***	.01	−.02	−.13*	−.07	.05	.04
Anxious Arousal/Somatic Sens	.57***	−.21***	−.04	−.05	−.29***	.07	.08*	.14***
Social Fears	−.17***	.64***	−.11*	−.10*	−.14**	−.06	.08**	.00
Specific Fears	−.23***	.15**	.00	.00	.11	.07	.02	−.07
Worry	.02	−.07	.00	.35***	.08	−.02	.11***	−.04
Obsessive-Compulsive Symptoms	−.13*	−.04	−.03	−.04	.60***	.05	.10**	.03
Depression	.10	−.22***	−.09	−.09	−.08	.42***	.05	.27***
Positive Affect	−.06	.05	−.05	.02	.07	−.06	−.03	−.11*
Intermediate Factors
Fear	.22***	−.09*	.23**	.10*	.21***	−.07	.18***	.15***
Anxious-Misery	−.22***	.47***	−.20**	* -.04	.02	.55***	.30***	.09*
Broad Factor
General Distress	−.05	.15**	−.12*	.39***	.27***†	.38***	.42***	.40***

Note. N = 1,000. PDA = panic disorder with or without agoraphobia; SOC = social phobia; SPEC = specific phobia; GAD = generalized anxiety disorder; OCD = obsessive-compulsive disorder; DEP = major depression or dysthymic disorder; NUM DIS = total number of mood, anxiety, somatic, and substance use diagnoses; IMP = self-reported impairment. For comparisons of specific diagnoses within each row (NUM DIS and IMP were not included), bold-faced and underlined values indicate that a correlation is significantly stronger (p < .001) than the non-bolded, non-underlined correlations in that row.

^*p < .05;

^**p < .01;

^***p < .001.

^†The correlation of OCD with General Distress was not significantly greater than that of SOC (p > .001), but OCD’s correlation with General Distress was stronger than those of PDA and SPEC (p < .001).

Impairment was significantly correlated with factors at each of the three hierarchical levels. At the narrow factor level, Depression, PA, and Anxious Arousal/Somatic Sensations were each significantly associated with impairment (rs = .27, -.11, and .14, respectively). Both of the intermediate factors and General Distress were also significantly associated with impairment (rs = .40 with General Distress, .15 with Fear, and .09 with Anxious-Misery). As expected, the number of diagnoses assigned was most strongly correlated with broad factors, particularly General Distress (rs = .42 with General Distress, .30 with Anxious-Misery, and .18 with Fear), whereas there were weakly significant associations with several of the narrow factors (rs = .08-.11).

For most of the narrow tri-level factors, the expected patterns of convergent and discriminant validity with specific diagnoses emerged. A diagnosis of PDA was more strongly correlated with the Interoceptive/Agoraphobic Fears (r = .62) and Anxious Arousal/Somatic Sensations (r = .57) factors than were any other diagnoses (range of z-values = 15.61–21.35 and 12.19–20.16, respectively). A diagnosis of SOC was most closely associated with the Social Fears factor (r = .64; zs = 18.15–24.17), a diagnosis of GAD was most closely (but relatively weakly) associated with the Worry factor (r = .35; zs = 6.44–10.39), a diagnosis of OCD was most closely associated with the Obsessive-Compulsive Symptoms factor (r = .60; zs = 15.72–17.58), and a diagnosis of DEP was most closely (but only moderately) associated with the Depression factor (r = .42; zs = 7.23–19.54). However, a diagnosis of SPEC was not significantly associated with Specific Fears (r = -.00), and none of the diagnoses were significantly associated with Positive Affect (rs = -.06-.07).

Turning to the intermediate tri-level factors (i.e., Anxious-Misery and Fear), there were no significant differences in correlational magnitude for the Fear factor with diagnoses of OCD, SPEC, PDA, and GAD (zs = 0.21–2.91). Although the magnitudes of these correlations were small (rs = .10-.23), all four of these diagnostic categories were more strongly correlated with Fear than were the remaining diagnoses (i.e., SOC and DEP; zs = 3.88–6.47). Conversely, SOC and DEP were most strongly associated with the Anxious-Misery factor (rs = .47 and .55, respectively) in comparison to the other diagnoses (zs = 10.61–19.17). Last, the correlations of DEP (r = .38), GAD (r = .39), and OCD (r = .27) with General Distress did not differ significantly from one another (zs = 0.26–2.98) and each was more strongly associated with General Distress than were the other diagnoses (zs = 5.98–12.06), although there was not a significant difference between the correlations of OCD and SOC (r = .15) with General Distress (z = 2.65).

Discussion

We found that the tri-level model, tested previously in a non-clinical sample only, was replicated in two large, independent clinical samples. Importantly, this model provided a better fit to the data than other simpler models that have been proposed for the internalizing disorders— most prominently, models in the tradition of the integrative hierarchical model and the fear/anxious-misery comorbidity model—and the factors showed evidence of validity with regard to relevant clinical outcomes that bolsters their clinical significance and meaningfulness. The tri-level model serves to synthesize the tri-partite/integrative hierarchical models with analyses of comorbidity data, and our results demonstrate the need for multiple non-specific factors (i.e., general distress, fear, and anxious-misery) in addition to narrow symptom factors when modeling internalizing symptoms. It is notable that even in a clinical treatment-seeking sample with high rates of comorbidity and elevated general distress, each level of the hierarchy uniquely accounted for substantial covariation among items. Perhaps surprisingly in such a sample, the narrow factors accounted for nearly half of the common variance, with the intermediate factors accounting for about one-third and General Distress about one-fifth.

Narrow Factors

The assumptions about symptom relationships at the disorder level inherent in the DSM-5 classification were generally supported, as all of the narrow-level factors were necessary in the full model and they were generally significantly and specifically related to corresponding diagnoses. As such, the fear and anxious-misery dimensions do not appear to fully capture the covariance among these symptoms, and meaningful distinctions remain at the disorder/symptom level.

However, some narrow factors were not related to diagnoses as expected. The narrow Specific Fears factor was not significantly correlated with a diagnosis of SPEC, nor was it positively associated with any other diagnoses. This may reflect the heterogeneous nature and relatively low covariances of specific fears items, as compared to other symptom items modeled here, or the fact that animal fears dominated this factor relative to other specific fears. Similarly and consistent with an analysis of the tri-level model in a non-clinical sample (Prenoveau et al., 2010), the narrow Positive Affect factor accounted for relatively little variance (although in post hoc analyses, fit decreased significantly if it was removed), was not significantly correlated with any diagnoses, and loaded very weakly on General Distress. An examination of the loadings of items from these two factors in the full model revealed that the specific fear items were among the strongest markers of the Fear factor, whereas the positive affect items were primary indicators for the Anxious-Misery factor, suggesting that these items best distinguish and characterize the intermediate level factors. As such, it appears that the intermediate factors absorbed most of their variance (consistent with the large correlation of a diagnosis of specific phobia with Fear), and the small amount of remaining variance in the narrow factors had little relevance to diagnoses.

Intermediate Factors

As hypothesized and in agreement with prior findings (e.g., Prenoveau et al., 2010; Watson, 2005), we found that the intermediate-level Fear factor was marked by items assessing social anxiety, specific fears, obsessive-compulsive symptoms, anxious arousal/somatic sensations, and interoceptive/agoraphobia fears. Likewise, positive affect, depression, and worry items all loaded on the Anxious-Misery factor; it is noteworthy that positive affect best characterized the shared variance across these symptoms, after accounting for the narrow factors and general distress. Also consistent with our expectations, the Fear factor was significantly associated with diagnoses of PDA, SPEC, and OCD, whereas the Anxious-Misery factor was significantly associated with DEP. However, testing of alternative models also suggested some modifications to this initially hypothesized model and some diagnoses were not associated as expected with the intermediate factors. We should note that most prior work has modeled fear and anxious-misery using a second-order model (i.e., indicators do not load directly on fear or anxious-misery; rather, these factors are formed from the shared variance among their constituent disorder factors), whereas we used a bi-factor model for reasons discussed previously. Thus, the different model specifications may have contributed to some of the discrepant findings.

Aligned with the arguments of Mennin et al. (2008), worry symptoms loaded significantly not only on Anxious-Misery but also on Fear in our data. We also found that GAD diagnoses were significantly but weakly associated with the Fear factor, but that they were unrelated to the Anxious-Misery factor. This is somewhat surprising given that GAD (and depression) are generally the core markers of Anxious-Misery (e.g., Watson, 2005), and these results conflict with those of the tri-level model in a non-clinical sample, where GAD clinical severity ratings were associated with Anxious-Misery but not Fear (Prenoveau et al., 2010). One possible reason for this discrepancy is that the 2010 examination of the tri-level model did not include sufficient worry items for a narrow worry factor to emerge, likely leading to somewhat different structural outcomes as compared to the current study. In addition, it is noteworthy that in our data the worry items were more strongly associated with the broad General Distress factor than with either of the intermediate factors, serving as the strongest indicators of General Distress. Consistent with the idea that GAD/worry symptoms are very closely related to neuroticism and may represent the “basic emotional disorder” (e.g., Brown et al., 1998), our results suggest that worry primarily resides at the highest level of abstraction in this structure.

A second modification that improved model fit was to add social anxiety symptoms to Anxious-Misery; furthermore, diagnoses of SOC were correlated with Anxious-Misery but not with Fear. Of note, all of the social anxiety items loaded significantly on Anxious-Misery, whereas only the items from the APPQ (not those from the SIAS) loaded on Fear. While this distinction could reflect measure variance, it may also reflect a substantive distinction, as the SIAS items are intended to specifically assess social interaction anxiety, whereas the APPQ is a more general measure that also includes performance/observation social anxiety. Prior work has shown that interaction anxiety (with or without performance anxiety) is more closely associated with depression and low positive affect, whereas performance anxiety (without interaction anxiety) is more closely linked with panic sensations and specific phobias (e.g., Carter & Wu, 2010; Hughes et al., 2006). Thus, it is possible that the alignment of social anxiety symptoms with Fear or Anxious-Misery may depend on the type of social anxiety; the 2010 test of the tri-level model emphasized performance/observation social anxiety and found somewhat different results. There has long been evidence of a close link between depression and social anxiety that is due in part to low levels of positive affect (e.g., Brown et al., 1998; Mineka et al., 1998; Naragon-Gainey et al., 2009), and these data suggest that social anxiety, and particularly interaction anxiety, may be better modeled as part of the anxious-misery spectrum, rather than as a fear disorder.

Although obsessive-compulsive items loaded on Fear as expected, we also tested whether they should be added to the Anxious-Misery factor, given conflicting prior evidence. We found that only the hoarding items from the OCI-R consistently loaded significantly on the Anxious-Misery factor, providing further evidence for DSM-5’s conceptualization of hoarding disorder as distinct from OCD. Furthermore, diagnoses of OCD were associated with Fear but not Anxious-Misery. These results extend the tri-level model beyond prior examinations that did not include OCD, and suggest that OCD symptoms are most closely associated with the Fear disorders. They also highlight a strength of the current approach: structural distinctions among symptom dimensions within the same disorder (i.e., social interaction anxiety and hoarding loading on Anxious-Misery, whereas other social anxiety and OCD symptoms did not) can only be revealed when relatively homogeneous symptoms, rather than disorders, are used as indicators.

General Distress

It is noteworthy that even in this investigation that did not include a very broad range of psychopathology (i.e., we focused on internalizing disorders and did not include externalizing or thought disorders), we still found that a single General Distress factor was needed, above and beyond the narrow and intermediate factors. The vast majority of items (representing content from all narrow factors) loaded significantly on General Distress, but depression and worry items were the strongest markers. In terms of diagnoses, this factor correlated most strongly with GAD and DEP (the correlation with OCD was smaller in magnitude, though not significantly different, and the correlation with SOC was significantly weaker). Thus, the General Distress factor seems to be most strongly characterized by disorders grouped in anxious-misery, rather than fear, consistent with a recent findings in an epidemiological sample that the general internalizing factor was nearly identical to anxious-misery but more weakly associated with fear (Kim & Eaton, 2015). In addition, depression and GAD typically have the strongest links to neuroticism/negative affect (e.g., Brown et al., 1998; Watson et al., 2005).

A model that excluded General Distress was a near-acceptable fit to our data, though it fit significantly more poorly than the tri-level model. Nonetheless, if the General Distress dimension were not clinically and substantively meaningful, some would argue that this more parsimonious model that fits reasonably well should be preferred. We turned to validity analyses to address this question, and our results highlighted the unique and incremental importance of the General Distress factor in picking up on broad indicators of psychopathology severity. Specifically, General Distress was more strongly associated with impairment and the number of diagnoses received than were any other factors in this model, demonstrating that the General Distress factor pertains to this broadest level of risk that reflects a propensity towards multiple disorders and general psychosocial impairment (consistent with the idea of a general p-factor; e.g., Caspi et al., 2014). Thus, if General Distress were omitted, the model would lose much of its ability to reflect and predict broad, non-specific liabilities that are important for etiological considerations and in clinical treatment settings.

What does the General Distress factor represent conceptually? Numerous researchers have speculated on the interpretation of a p-factor; though prior studies were done in the context of the inclusion of externalizing and/or thought disorders, they have relevance for our General Distress factor as well. Most substantive interpretations focus on general psychopathology factors as indicators of a broad, non-specific state that arises from a general liability towards psychopathology (Bohnke & Croudace, 2015; Caspi et al., 2014; Lahey et al., 2012). Consistent with our findings, a p-factor should be linked to other indicators of severity, such as impairment, comorbidity, duration, and family history of psychopathology (Caspi et al., 2014). Note that a general psychopathology factor is not necessarily indicative of a single cause, although the often observed strong association with neuroticism and negative affectivity suggest that it may be largely isomorphic with these traits (perhaps particularly when considering the internalizing disorders only) (Griffith et al., 2010; Lahey et al., 2012).

Other interpretations are methodological in nature: General Distress could be reflective of response styles, such as an individual’s tendency to select extreme response options (Bohnke & Croudace, 2015) or to describe themselves in socially-(un)desirable terms (e.g., reporting poor functioning and negative emotions) (Lahey et al., 2012), perhaps as a “cry for help” among clinical participants. The latter is related to demoralization, or a non-specific unpleasant affect state that clinical internalizing samples frequently endorse and can account in part for the associations among affectively-laden personality traits (e.g., Noordhof et al., 2015). Counter to this interpretation, however, a large General Distress factor was found when the tri-level model was assessed in a non-clinical adolescent sample and in an unselected college student sample (Prenoveau, Haimann, & Zinbarg, 2016; Prenoveau et al., 2010). Finally, current mood state could have affected individuals’ report of their recent symptoms (i.e., mood-state distortion), which could inflate the size of a general distress dimension. However, a study from this same treatment center (in a different sample of individuals) found that that mood-state distortion did not greatly impact associations of symptoms with affective personality traits (Naragon-Gainey et al., 2013). Overall, it is not possible to arrive at a single or definitive interpretation of General Distress, but its theoretically-consistent associations with clinical diagnoses (which would be less influenced than self-report measures by response biases and methodological effects, since they are clinician-rated) and with markers of severity bolster our confidence in an interpretation that is at least partially substantive.

Limitations and Conclusions

This study had numerous strengths, including robust results across two large clinical subsamples and validation of the structural factors with meaningful clinical outcomes. However, it is important to acknowledge several limitations. First, although we examined associations of structural results with clinical diagnoses derived from semi-structured interview, the structural model itself was formed solely from self-report measures. In addition, the use of multiple inventories increases the generalizability of the model, but it also introduces the possibility of method effects due to differences across measures in instructions, rating scales, time frames, and other format features. Second, we used temperament items for some indicators (primarily strong markers of Positive Affect and General Distress). While these items are generally closely related to symptoms when assessed cross-sectionally (and particularly in clinical samples), these indicators may have skewed structural results. We also did not include indicators of all internalizing symptoms (i.e., posttraumatic stress disorder, eating disorders). Third, although the sample presented with substantial variability across most internalizing disorders, social anxiety disorder was particularly prevalent in this sample (nearly half of the sample met criteria) and the specific distribution of disorders and symptoms may have impacted results.

Fourth, our modeling approach had several limitations. We could not include information criteria that penalize for model complexity (e.g., Bayesian Information Criteria) because they are not available when using WLSMV estimators, although we note that TLI does include a penalty for a lack of model parsimony. In addition, our fairly large sample means that χ² difference tests were highly powered to detect small differences, which some may argue leads to identifying trivial differences in model fit. However, we assert that in the context of comparing two theoretically driven models, the ability to detect even small differences is an advantage rather than a liability, assuming those differences are substantively meaningful. With regard to this latter point, we did not solely focus on the model goodness of fit statistics and χ² difference testing, but also examined and found evidence for their substantive value as illustrated by validity associations. We also note that, as in any model building approaches, results may vary depending on the extent and ordering of decisions for post hoc revisions. Error covariances have the potential for overfitting models, but we attempted to avoid this by focusing on a priori covariances and adding only one post hoc error covariance. Finally, this structure was examined cross-sectionally and therefore is not informative with regard to the development or temporal dynamics of symptoms.

We replicated and extended prior work on the tri-level model, demonstrating that factors at three levels of breadth are necessary in order to explain covariance among markers of a range of internalizing symptoms in a clinical sample, and that these factors appear to be clinically meaningful. The tri-level model brings together prior research that had largely focused only on symptoms or on diagnoses, and it is consistent with recent evidence for a general psychopathology factor. Future research should examine which levels of the hierarchy are most amenable to change in therapy, as well as how these phenotypic dimensions correspond to risk for psychopathology using genetic and biological indices.