Abstract
Much of our knowledge about the relationships among domains of psychopathology is built on the diagnostic categories described in the Diagnostic and Statistical Manual of Mental Disorders (DSM), with relatively little research examining the symptom-level structure of psychopathology. The aim of this study was to delineate a detailed hierarchical model of psychopathology—from individual symptoms up to a general factor of psychopathology—allowing both higher- and lower-order dimensions to depart from the structure of the DSM. We explored the hierarchical structure of hundreds of symptoms spanning 18 DSM disorders, in two large samples—one from the general population in Australia (n = 3175), and the other a treatment-seeking clinical sample from the USA (n = 1775). There was marked convergence between the two samples, offering new perspectives on higher-order dimensions of psychopathology. We also found several noteworthy departures from the structure of the DSM in the symptom-level data.
Keywords: Structure of psychopathology, symptom-level analyses, HiTOP, empirical classification
Research on the quantitative classification of mental disorders has gained momentum recently with the formation of the Hierarchical Taxonomy of Psychopathology (HiTOP) consortium and model (see Figure S1; Kotov et al., 2017). The HiTOP model synthesises 20 years of research on broad dimensions of mental disorders (i.e., psychopathology) and maladaptive personality. These dimensions align with social, environmental, genetic, neurophysiological, and biological risk factors, and have exciting potential to thereby accelerate research on the etiology of psychopathology (Conway et al., 2019; Kotov et al., 2017; Waszczuk et al., 2019; Zald & Lahey, 2017). However, most of the research on the empirical structure of psychopathology in adults has been based on traditional diagnostic categories—primarily those in the Diagnostic and Statistical Manual of Mental Disorders (DSM; American Psychiatric Association, 2013)—as the units of analysis (Kotov et al., 2017; Wright et al., 2013). This has both constrained our understanding of the structure of psychopathology to the disorder-level structure of the DSM and limited our ability to characterize the more detailed (i.e., symptom-level) structure of psychopathology, as described below.
Teasing apart reliable and detailed phenotypes of psychopathology is important for research, practice, and assessment; for example, to improve our understanding of specific causal mechanisms or risks (Hyman, 2007; Kozak & Cuthbert, 2016; Sonuga-Barke, 2016), to characterize the specific nature, scope, and severity of an individual’s presenting symptoms to match them with the most appropriate interventions available (Hopwood et al., 2019; Ruggero et al., 2019), and to facilitate efficient and accurate assessment in clinical and primary care settings (e.g., Batterham, Sunderland, Carragher, & Calear, 2016; Sunderland, Batterham, Carragher, Calear, & Slade, 2019; Sunderland et al., 2017). These aims are particularly well-served in a hierarchical model that provides researchers and clinicians control over the level of specificity or granularity of the constructs of interest (e.g., for characterizing detailed mechanisms that underlie a specific symptom or symptom cluster, or for understanding risks for psychopathology broadly; Krueger et al., 2018). The aim of the present study was thus to delineate a detailed hierarchical model of psychopathology—from individual symptoms up to a general factor of psychopathology—using data from clinical and community samples.
The Heavy Reliance on DSM Categories to Date
As mentioned above, the literature on the empirical structure of adult psychopathology has been dominated by analyses of the patterns of comorbidity or covariation among DSM diagnoses (Kotov et al., 2017; Wright et al., 2013). This approach has uncovered robust dimensions that account for the systematic patterns of co-occurrence among mental disorders (the pink sections of Figure S1), but has resulted in limited research characterizing the more detailed structure of psychopathology (the blue sections of Figure S1). The heavy reliance on DSM diagnoses to date is due to the fact that these diagnoses are almost invariably the focal constructs of large representative population surveys of mental health. The use of large representative samples improves the likelihood of uncovering robust and generalizable models of psychopathology. However, structured clinical interviews such as the Composite International Diagnostic Interview (CIDI; Kessler & Üstün, 2004) are used in these studies to keep average interview times to a reasonable minimum, following skip-out rules to efficiently determine whether a given diagnosis is “present or absent”. For example, if someone does not report depressed mood or anhedonia, they are not typically asked about the other seven Criterion A symptoms of major depressive disorder (MDD; e.g., fatigue, difficulty concentrating, or suicidality). This skip-structure can result in substantial bias and missingness in the symptom-level data, restricting analyses of these data to only the cardinal symptoms of each disorder, and thus limited coverage of psychopathology. When the categorical (present vs. absent) diagnoses are analyzed instead, we cannot characterize the detailed structure of psychopathology at all.
Our understanding of the structure of psychopathology is thus largely bound to the structure of DSM diagnostic categories, which is a problem for three reasons in particular. First, the heterogeneity within many mental disorders is lost. For example, the DSM symptom criteria for MDD can be arranged into 945 different symptom presentations that meet criteria for the diagnosis (Fried & Nesse, 2015). When these varied syndromes—and their subthreshold variants—are all collapsed into a single “present vs. absent” MDD diagnosis, we lose the opportunity to study variability within the category as well as valuable information regarding the severity of presenting symptoms. Second, overlapping criteria between diagnoses (i.e., symptoms that contribute to multiple disorders) are also unaccounted for when analysing categories, which may inflate the patterns of covariation that underlie the structure. For example, generalized anxiety disorder (GAD) and MDD share symptoms of fatigue, insomnia, and difficulty concentrating, making them more likely to covary due to shared phenomenology, rather than perhaps due to sharing an underlying internalizing liability (Borsboom, 2002). Finally, although uncommon, when analyses focus on data in which hierarchical exclusion rules have been applied to diagnoses (e.g., GAD not being diagnosed if it occurs during an episode of depression), the patterns of association among disorders and the resulting structural models can be distorted (see Conway & Brown, 2018; Kotov et al., 2018).
Overall, the predominance of research conducted using DSM diagnoses means we have a limited understanding of the detailed empirical structure of psychopathology. The heterogeneity within and homogeneity between diagnoses highlight potential patterns of cross-cutting symptom clusters that do not follow traditional diagnostic boundaries. By using symptoms instead of DSM diagnoses as the units of analysis, we can model this complexity and allow both higher-order dimensions (i.e., broad spectra of psychopathology that span multiple traditional diagnoses) and lower-order dimensions (i.e., empirically derived syndromes) of psychopathology to depart from the structure of the DSM, if warranted empirically.
Extant Symptom-Level Research
Given the benefits of analyzing symptom-level data, there have been several studies that have taken this approach with the aim of understanding the detailed empirical structure of adult psychopathology1. Analyses within the transdiagnostic internalizing (e.g., Dornbach-Bender et al., 2017; Grisanzio et al., 2018; Waszczuk et al., 2017; Zinbarg & Barlow, 1996), externalizing (Krueger et al., 2007), and thought disorder (Kotov et al., 2016) spectra have begun to elucidate the detailed structure of these dimensions. These studies have often had impressive detail in symptom-level information, albeit limited breadth. By only examining symptoms within a single spectrum, these studies have not been able to characterize convergence and divergence of symptoms between the spectra—for example, whether symptoms like insomnia or difficulty concentrating (which span diagnoses in multiple spectra) are better conceptualized under one spectrum over another, or as transdiagnostic indicators. Further, to handle the substantial complexity of analyzing numerous observed variables in multivariate models, some studies have constrained symptoms within their traditional diagnostic categories (e.g., symptoms of depression being parcelled together prior to analyses of the full data set; e.g., Waszczuk et al., 2017; Zinbarg & Barlow, 1996), limiting opportunities to identify any departures from the structure of the DSM.
Other studies have examined a smaller number of symptoms across a broader variety of psychopathology, which allows for the possibility that heterogeneity within traditional diagnoses may be accounted for by multiple spectra (e.g., obsessive-compulsive disorder [OCD] may have symptom components differentially related to internalizing and thought disorder dimensions; Faure & Forbes, 2020; Watson, Wu, & Cutshall, 2004). Markon (2010) included broad coverage of psychopathology (50 symptoms spanning 14 disorders), and personality pathology (73 symptoms spanning 10 personality disorders) in a large representative adult sample, finding four transdiagnostic spectra—internalizing, externalizing, thought disorder, and pathological introversion—that are reflected in the HiTOP model (the latter as detachment; see Figure S1). Wright et al. (2013) subsequently examined 33 symptom-level indicators spanning 11 disorders in a large representative sample of adults, finding five subfactors (distress, fear, OCD, alcohol use, and drug use) and three spectra (internalizing, psychosis, and externalizing) that also informed the structure of the HiTOP model. Both of these studies have been important in explicating our understanding of the structure of psychopathology. However, the narrow coverage of each diagnosis (i.e., only one to three symptoms were assessed for most of the disorders in each sample) limited characterization of the detailed structure of psychopathology. Much like analyses of disorder-level indicators, these studies thus largely informed our understanding of the higher-order structure of psychopathology.
In sum, most studies have either had comprehensive assessment of a narrow cross-section of psychopathology or scant measurement of a broader cross-section. None of the studies to date have had access to comprehensive data measuring adult psychopathology symptoms spanning common and uncommon mental disorders, which is needed to characterize a detailed structural model. Further, most research has been conducted in non-clinical samples, which may have lower representation and variability of psychopathology, and thus less detailed structure (Kotov et al., 2011).
The Present Study
The aim of the present study was to characterize the symptom-level structure of psychopathology using comprehensive coverage of common and uncommon mental disorders in two large clinical and community samples of adults. Between the two samples, nearly all of the HiTOP spectra and subfactors were represented in the analyses; the exceptions were sexual problems, and the two spectra that are predominantly related to personality disorders in the traditional DSM nomenclature (i.e., antagonistic externalizing and detachment; see Figure S1). The absence of personality pathology from these analyses is noteworthy, as dimensions of maladaptive personality often appear to act as a skeleton for joint structural models with other psychopathology (e.g., Forbes et al., 2017; Kotov et al., 2011; Markon, 2010; Wright & Simms, 2015). These were secondary analyses of existing data collected in two different studies. However, 97 overlapping symptoms were assessed in both samples, which also allowed us to characterize convergence in the structures between samples and propose an overarching hierarchical model.
Method
Participants and Procedure
Participants were drawn from two larger studies. First, the community sample was drawn from the Assessing Mental Health (AMH) study (Batterham et al., 2016), which tested large self-report item banks of a variety of mental health problems in a population-based Australian adult sample with the aim of developing new static and adaptive brief mental health screeners for social anxiety disorder, panic disorder, posttraumatic stress disorder (PTSD), OCD, attention-deficit hyperactivity disorder (ADHD), drug use, psychosis, and suicidality. A detailed description of the study methods is available elsewhere (Batterham et al., 2016). Briefly, participants were recruited through online social media advertising for a mental health study, with a target population of Australian adults aged 18 years or older. All individuals who completed the full form of the survey (n = 3175) were included in the present analyses. The study had a planned missingness design for participants who reported never having a drink containing alcohol (n = 705; 22.2%), no trauma exposure (n = 1296; 40.8%), or never using drugs other than alcohol (n = 2524; 79.5%) who did not respond to the remaining alcohol use, traumatic reactions, and substance use items, respectively. All participants’ responses were retained in analyses using pairwise complete information for the calculation of item-level correlations, and taking the mean of valid item responses in each symptom cluster prior to estimating cluster-level correlations based on pairwise complete information (see below). The sample characteristics are presented in Table S1 and demonstrate diversity in terms of age, level of education, and location of residence, but relative homogeneity in gender (79.6% female, versus 50.7% in the general population; Australian Bureau of Statistics, 2016a) and language spoken at home (93.3% English only, versus 72.7% in the general population; Australian Bureau of Statistics, 2016b). The sample also overrepresented individuals with psychopathology, relative to population prevalence rates (see Batterham et al., 2016), with 53% endorsing symptoms that reflected the full DSM-5 diagnostic criteria for a depressive, anxiety, or substance use disorder (see Table S1). For our purposes, this overrepresentation is ideal, as it corresponds with variability in the symptoms that is useful for modelling the patterns in their covariation.
The clinical sample was drawn from the Rhode Island Methods to Improve Diagnostic Assessment and Services (MIDAS) project (Zimmerman, 2016), which has run for over 20 years through Rhode Island Hospital’s Department of Psychiatry with a focus on integrating research assessments into routine clinical practice. Participants are individuals presenting for an intake evaluation at the community outpatient psychiatry practice who consent to participate in the MIDAS project. The present sample includes all individuals who completed the self-report battery of clinically useful outcome measures described below (n = 1775), which was included in the MIDAS assessment battery from 2004 to 2013. Participants had the option to skip questions in the survey, resulting in each item missing 0.2–7.8% of participants’ data (99.6% of items had < 5% missing data). As above, responses with missing data were retained and the correlations that formed the basis of the analyses at each step were estimated based on pairwise complete data. The sample characteristics are presented in Table S1, and demonstrate diversity in terms of age, gender, level of education, and marital status, but not in terms of race (87.7% white, which was slightly higher than the population in Rhode Island [81.4%]; United States Census Bureau, 2010). Nearly all participants (98.4%) met criteria for a DSM-IV mental disorder, with the most common primary diagnoses being depressive or anxiety disorders (see Table S1).
Measures
Both studies were designed to assess a variety of psychopathology. The AMH study used a large item pool derived from a systematic literature review for existing scales that assess social anxiety disorder, panic disorder, PTSD, OCD, adult ADHD, drug use, psychosis, and suicidality. Unique (non-redundant) and unambiguous items that were specific to the disorder of interest (n = 2002) were phrased in a standardized format (past tense, over the past month, assessed on a 5-point Likert scale) and sent for feedback from consumers, expert researchers and clinicians that was used to select the item pools assessed in the study (see Batterham et al., 2015 for a detailed description of the methods). All items that assessed psychopathology symptoms over the past month on a 5-point Likert scale (from 0 ‘never’ to 4 ‘always’) were analysed in the present study, which included 583 items in total: 463 from the items pools described above and 120 from the Patient-Reported Outcomes Measurement Information System (PROMIS) measures of depression, anxiety, alcohol use, and anger (Pilkonis et al., 2011, 2016). Notably, PROMIS depression focuses on affective and cognitive (versus somatic) symptoms; PROMIS anxiety assesses fear, anxious misery, hyperarousal, and somatic symptoms related to anxious arousal broadly; and PROMIS anger assesses affective, cognitive, and behavioural manifestations of anger.
The MIDAS project included a battery of self-report symptom scales between 2004 and 2013. These scales assessed symptoms of eating pathology (weight and shape concerns, binging, and purging), major depression (including suicidality), panic, agoraphobia, PTSD, OCD, generalized anxiety, social anxiety, alcohol use, substance use, pain/somatization, illness anxiety, psychosis, mania, irritability/anger, and self-injury behaviors. The 235 items were measured on a 5-point Likert scale (from 0 ‘not at all true’ to 4 ‘almost always true’) assessing experiences over the past week. Clinically useful depression, anxiety (i.e., somatic and psychic anxiety), and social anxiety scales have been published based on these data (Dalrymple et al., 2013; Zimmerman et al., 2008; Zimmerman et al., 2010); all of the symptom scales were developed based on DSM-IV symptom criteria, the content of the Psychiatric Diagnostic Screening Questionnaire (Zimmerman & Mattia, 2001), and structured clinical interview questions for DSM-IV diagnoses.
Data Analysis
Item processing.
All analyses were conducted in R version 3.6.2 (R Core Team, 2019). First, the items in each sample were pre-processed as follows. In both samples, items with 95% or more of the sample endorsing 0 (‘never’ or ‘not at all’) were removed, due to lack of variability, to reduce sparse cells in estimation of the correlation matrices. In AMH, 19 psychosis items and two suicide items were removed; in MIDAS, one alcohol use, one substance use, and four psychosis items were removed (see Table S2). On face value, these items reflected the most severe manifestations of the constructs. Next we coded items with overlapping content in both samples, with the aim of tracing similarities and differences in the placement of the items between the two samples. These matches in item content were coded by two fourth-year undergraduate interns, with the provisional list of matches and any disagreements in coding re-coded by the first author (see Table S3). This process resulted in 97 items—henceforth labelled trace items—treated as overlapping in content between samples. These items were used to characterize similarities and differences in the structure of the two samples.
Data reduction.
Following this item-level processing, our next step involved data reduction in the form of combining items into highly homogeneous symptom clusters. This approach was necessary to reduce the very large number of items assessed in each sample to make multivariate modelling approaches to the data tractable. The aim of this data reduction step was to retain the detail in the symptom-level data and only combine items that would normally be deemed psychometrically redundant (i.e., too highly correlated to be distinct). To this end, we used two clustering methods to identify highly homogeneous symptom clusters based on item-level Spearman correlations in each sample2. First, the iclust function from the psych package in R (Revelle, 2019), which forms clusters based on average and minimum split-half reliability (alpha and beta coefficients, respectively). Highly conservative settings were used to form clusters, such that items or item clusters were only combined into a larger cluster if alpha and beta both increased for both clusters, and if beta was ≥ .9 (i.e., 90% or more of the variance in the items was associated with a shared general factor). The ICLUST method has been found to outperform exploratory factor analyses in characterizing latent structure in large item pools (e.g., Revelle, 1979). Second, Ward’s (1963) hierarchical agglomerative clustering was used, which is based on a dissimilarity matrix of the items (1 – Spearman correlation matrix) that identifies and combines the two most similar items/clusters (i.e., merging the two items or clusters that result in the smallest increase in the sum of squared error) iteratively until a single cluster is formed. We cut the resulting hierarchy at the last unitary (single-item) cluster, and compared the symptom clusters with those derived using iclust. These two methods were used to ensure large clusters of items did not form based on method characteristics specific to a single clustering method (i.e., to err on the side of only merging items into truly homogeneous clusters). Items were merged into a symptom cluster, by taking their mean, if both methods included them in a single cluster—and, for the trace item pool, if they were also included in a single cluster in both samples. The resulting items and item clusters (see Tables S4 and S16) were used as the units of analysis and conceptualized as the first level of the hierarchical structure of psychopathology (cf. signs and symptoms in the HiTOP framework in Figure S1).
Hierarchical modelling.
Following data reduction, the hierarchical structure in both samples was elucidated using an extended bass-ackwards method (Forbes, 2020). This method builds on Goldberg’s (2006) bass-ackwards approach, extracting orthogonal principal components (1, 2, 3, …n; varimax rotation was used here) and examining the component correlations between sequential levels. Loehlin and Goldberg (2014) suggested that a component correlation |r| ≥ .9 between levels indicates the perpetuation of the same construct between levels of the hierarchy, and a component correlation .3 ≤ |r| < .9 indicates a higher-order component splitting into more specific lower-order components. The extension to this approach examines component correlations among all levels of the hierarchy after removing redundant components that perpetuate through multiple levels of the hierarchy (|r| ≥ .9 and Tucker’s congruence coefficient > .95) and apparent artefactual components that emerge as a consequence of forcing a specific number of components on a given level of the hierarchy. This extended approach aims to fully elucidate the hierarchical structure of the data—for example, examining how broad higher-order components that emerge early in the hierarchy relate to all of the lower-order components at the bottom of the hierarchy; this is not possible in the traditional bass-ackwards framework, which focuses exclusively on correlations between components on adjacent levels. To reduce confirmation bias in determining which components represented artefacts in the structure, we examined convergence with the hierarchical structure that emerged using Ward’s hierarchical agglomerative cluster analysis (cf. Forbes et al., 2017; Figure S2); components were only removed from interpretation of the structure if they were deemed artefactual on theoretical grounds and did not emerge in the cluster analysis hierarchy. Each of these decisions is described below, and the full traditional bass-ackwards results (including all components at each level of the hierarchy; Figures S3 and S4) are also interpreted for comparison.
The number of components to extract in each data set was based on the maximum number of meaningful factors indicated by parallel analysis and Velicier’s minimum average partial (MAP), calculated using the fa.parallel and vss functions of the psych package (Revelle, 2019), respectively. Given the very large number of variables being analyzed here, we focus on loadings ≥ .4 in interpreting the hierarchical models. Detailed information on the component loadings and correlations for all estimated components are reported in extensive supplementary materials (Tables S5–S15 and S17–S30). We report summaries of these results in-text.
Results
Assessing Mental Health Survey
Highly Homogeneous Symptom Clusters
The data reduction process reduced the full item pool of 562 items in AMH to 155 symptom clusters—including 74 individual items, and 81 clusters ranging from 2 to 32 items (see Table S4 for a full list of which items formed each cluster, and for the names of each cluster). For the most part, the symptom clusters were cleanly constructed from within the same parent item pools (i.e., intended to measure the same target disorder construct). There were six exceptions: (1) The racing or pounding heart cluster comprised items from the PROMIS anxiety (“I had a racing or pounding heart”) and panic (“I experienced palpitations, a pounding heart or a rapid heart rate”) inventories; (2) the avoidance of social situations cluster included a PROMIS anxiety item (“I avoided public places or activities”) along with seven social anxiety items assessing avoidance of social activities (e.g., “I came up with excuses to avoid social situations”); (3) the concerns about being observed/public speaking cluster included a PROMIS anxiety item (“I worried about other people’s reactions to me”) along with 21 other social anxiety items assessing concerns about being observed or public speaking (e.g., “I avoided speaking in front of groups of people”); (4) the agitated cluster comprised items from the PROMIS anxiety (“I felt fidgety”) and ADHD (“I had difficulty sitting still”) item banks; (5) the difficulty sleeping cluster comprised a PROMIS anxiety item (“I had difficulty sleeping”) and an ADHD item (“mental restlessness prevented me from sleeping”); and (6) the difficulty concentrating cluster comprised items from the PROMIS anxiety (“I felt indecisive” and “I had trouble paying attention”) and PROMIS depression (“I had trouble making decisions”) item banks along with 13 items from the ADHD item pool assessing cognitive difficulties (e.g., “I had difficulty maintaining focus”). Notably, on face value, the first three of these six exceptions represent appropriate clustering of items from the transdiagnostic PROMIS measure of anxiety with similar items from the disorder-specific item banks. By contrast, the latter three instances reflect symptom overlap between distinct diagnostic constructs (i.e., “hybrid” symptom clusters reflecting symptom components from multiple diagnoses).
Hierarchical Structure
Parallel analysis suggested a maximum of 13 components, and the MAP reached a minimum with 16 factors, so we extracted 1 to 16 components to estimate the initial bass-ackwards hierarchy. The levels with 11-16 components all had components with only one or two unique indicators, so 1 to 10 components were extracted in the final hierarchical model (i.e., one component on the first level, two components at the second level, and so on, down to ten components at the bottom of the hierarchy). The results for each level of the model and the correlations between the levels are given in Tables S5–S15, and the traditional bass-ackwards solution is shown in Figure S3. Four components in the bass-ackwards solution were identified as likely artefacts and were also absent in the agglomerative cluster analysis (Figure S2), so were removed from the hierarchy interpreted below: All three components on the third level of the hierarchy (C1-C3, see Table S7) where thought disorder (D2) indicators were split between internalizing (C1) and substance use (C2), and a slight rotation of the alcohol use component emerged (C3) that was virtually redundant with the lower-order alcohol use component (E5; r = .88, congruence coefficient = .93). The fourth artefact was a psychosis and suicide component (F4) where several suicidality symptom clusters loaded with psychosis (G4) indicators, but this pattern was not seen on any other level of the hierarchy—either through component correlations between levels, or symptom clusters cross/loadings within levels—and did not emerge in the agglomerative cluster hierarchy.
A summary of the hierarchical structure that emerged among the remaining components is shown in Figure 1A. The first unrotated principal component—labelled general psychopathology by convention—was dominated by anxiety (panic, generalized anxiety, and social anxiety symptoms) and negative affect (core depression symptoms). By contrast, symptoms from the substance use, psychosis, and alcohol use item pools tended to be weak indicators of this first unrotated principal component. The general psychopathology component split into three components: (1) substance and alcohol use, composed of the lower-order substance use and alcohol use components; (2) thought disorder, composed of the OCD and psychosis components; and (3) internalizing, composed of disinhibited negative affect and fear components. The disinhibited negative affect component was composed of disinhibition (anger and attentional dysregulation) and a secondary loading from distress (suicidality/hopelessness, with cross-loadings from social anxiety symptom clusters). The fear component was composed of anxiety, PTSD, and cross-loadings from social anxiety components, with weaker loadings from some of the suicidality/hopelessness symptom clusters.
Figure 1.
Extended bass-ackwards structure in the (A) Assessing Mental Health data and (B) Rhode Island Methods for Improving Diagnostic Assessment and Services data. Component names are presented in boxes—along with the alphanumeric labels used to identify them in the supplementary materials—and the solid lines represent the strongest component correlation for each lower-order component with the higher-order components. Line weights are proportional to the component correlation. Dashed lines are secondary component correlations .3 ≤ |r| < .9 that were not accounted for by tracing the hierarchical structure from the bottom up. The dotted line in Figure 1B from externalizing to general psychopathology is a primary component correlation |r| < .3. The red line in Figure 1B from anger to phobic avoidance is a negative correlation.
There were only two noteworthy differences comparing this hierarchy to the structure of the full bass-ackwards hierarchy (Figure S3). First, by examining correlations among components at all levels of the hierarchy, the association between PTSD and fear emerged; in Figure S3, the strongest correlation for PTSD with a component on the preceding level was with social anxiety (I8; r = .31), but it had a stronger association with fear (E1; r = .34) that helped to clarify the underlying hierarchy. Second, the association between social anxiety and fear was not evident when only examining correlations between sequential levels of the hierarchy; in Figure S3, the strongest correlation for social anxiety (I8) was with distress (H6; r = .46), but social anxiety also had a secondary correlations with the higher-order fear component (E1; r = .32), and its strongest association was with internalizing (D1; r = .49).
Symptom-Level Perspectives
Table 1 shows the primary loadings and cross-loadings of the symptom clusters on the ten components at the lowest level of the hierarchy. The ten components closely reflected the target constructs in the item pools. For example, components corresponding to substance use, alcohol use, psychosis, OCD, anger, attentional dysregulation, social anxiety, and PTSD emerged. There were two exceptions to this pattern: (1) the suicide and depression symptom clusters formed a single component of suicidality/hopelessness that also included the OCD single-item cluster intrusive thoughts about self-harm, and (2) the panic and non-hybrid PROMIS anxiety symptom clusters formed a single anxiety component that also included the social anxiety single-item cluster tension headaches before social situations and cross-loadings from the PTSD clusters anxious arousal due to trauma and re-experiencing trauma.
Table 1.
Primary loadings and cross-loadings of the symptom clusters on the ten components at the lowest level of the hierarchy in the Assessing Mental Health data
| Symptom cluster | j1 | j2 | j3 | j4 | j5 | j6 | j7 | j8 | j9 | j10 |
|---|---|---|---|---|---|---|---|---|---|---|
| Fear of fear | 0.76 | |||||||||
| Worry about panic (i) | 0.74 | |||||||||
| Afraid of physical symptoms | 0.73 | |||||||||
| Felt could not breathe (i) | 0.72 | |||||||||
| Panic episode | 0.71 | |||||||||
| Fear of leaving the house alone (i) | 0.70 | |||||||||
| Thought might be dying | 0.69 | |||||||||
| Racing or pounding heart | 0.66 | |||||||||
| Chest pain (i) | 0.65 | |||||||||
| Feared fainting (i) | 0.65 | |||||||||
| Dizzy or lightheaded (i) | 0.65 | |||||||||
| Rush of fear. and related impairment | 0.64 | |||||||||
| Frightened by nausea (i) | 0.63 | |||||||||
| Afraid would have a heart attack, stroke, suffocate, or die (i) | 0.62 | |||||||||
| Fear of specific situations | 0.62 | |||||||||
| Sought help for panic (i) | 0.58 | |||||||||
| Nervous or anxious (i) | 0.57 | |||||||||
| Fear | 0.55 | |||||||||
| Fear of loss of control | 0.55 | |||||||||
| Safety zone (i) | 0.54 | |||||||||
| Claustrophobia (i) | 0.54 | |||||||||
| Trembling (i) | 0.49 | |||||||||
| Feeling anxious | 0.49 | |||||||||
| Disconnected or detached | 0.48 | |||||||||
| Tension headaches before social situations (i) | 0.47 | 0.44 | ||||||||
| Perfectionism | 0.74 | |||||||||
| Compulsions | 0.73 | |||||||||
| Checking for mistakes | 0.71 | |||||||||
| Strict routine doing ordinary things (i) | 0.66 | |||||||||
| Checking so nothing terrible would happen (i) | 0.64 | |||||||||
| Need for order (i) | 0.64 | |||||||||
| Fear of mistakes | 0.63 | |||||||||
| Responsibility to ensure everything was in order (i) | 0.63 | |||||||||
| Very strict with self (i) | 0.62 | |||||||||
| Contamination concerns | 0.62 | |||||||||
| One right way to do things (i) | 0.61 | |||||||||
| Got stuck doing routine behaviours (i) | 0.60 | |||||||||
| Checking (i) | 0.59 | |||||||||
| Performed rituals (i) | 0.58 | |||||||||
| Do everything exactly right leaving home (i) | 0.58 | |||||||||
| Try to prevent harm (i) | 0.57 | |||||||||
| Obsessive thoughts (i) | 0.53 | |||||||||
| Compulsions (i) | 0.53 | |||||||||
| Rituals for protection (i) | 0.52 | |||||||||
| Difficult to touch rubbish (i) | 0.52 | |||||||||
| Fear of acting on compulsions | 0.52 | |||||||||
| Having upsetting thought made it more likely to happen (i) | 0.52 | |||||||||
| Upsetting thoughts | 0.49 | |||||||||
| Guilt about obsessions (i) | 0.45 | |||||||||
| Repeat words to stop obsessions (i) | 0.45 | |||||||||
| Thoughts would harm other people unintentionally (i) | 0.42 | |||||||||
| Heard voices | 0.68 | |||||||||
| Paranormal experiences | 0.66 | |||||||||
| Auditory hallucinations | 0.66 | |||||||||
| Sense of unreality | 0.63 | |||||||||
| Someone/something playing games with mind (i) | 0.58 | |||||||||
| Special powers | 0.57 | |||||||||
| Doubted dreams were the product of own mind (i) | 0.57 | |||||||||
| Thought insertion (i) | 0.56 | |||||||||
| Thought broadcasting | 0.56 | |||||||||
| Paranoia (being followed or observed) | 0.56 | |||||||||
| Mistook noises for voices (i) | 0.54 | |||||||||
| Lights or colours seemed brighter (i) | 0.53 | |||||||||
| Olfactory hallucination (i) | 0.51 | |||||||||
| Paranoia (people want to hurt them) | 0.50 | |||||||||
| Ideas of reference (i) | 0.49 | |||||||||
| Felt part of body did not belong (i) | 0.48 | |||||||||
| Felt was very special (i) | 0.48 | |||||||||
| Had a sixth sense (i) | 0.45 | |||||||||
| Paranoia (people are against them) | 0.45 | |||||||||
| Thoughts of people trying to upset deliberately (i) | 0.44 | |||||||||
| Sense of danger or dread (i) | 0.41 | |||||||||
| Something seriously wrong with body (i) | 0.39 | |||||||||
| Felt attacked (i) | 0.38 | |||||||||
| Prioritising drug use | 0.84 | |||||||||
| Urge to use drugs | 0.81 | |||||||||
| Role and relationship impairment from drug use | 0.80 | |||||||||
| More drugs than intended and despite psychological problems | 0.80 | |||||||||
| Dependence on drugs | 0.79 | |||||||||
| Used drugs after deciding not to | 0.78 | |||||||||
| Tolerance to drugs | 0.76 | |||||||||
| Money problems due to drug use | 0.76 | |||||||||
| Used drugs to get high | 0.75 | |||||||||
| Relationship friction around drug use | 0.74 | |||||||||
| Had to keep taking drugs once started (i) | 0.70 | |||||||||
| Withdrawal from drugs (i) | 0.68 | |||||||||
| Used drugs in hazardous situations (i) | 0.66 | |||||||||
| Time recovering from drugs (i) | 0.60 | |||||||||
| Using larger amounts of alcohol than intended | 0.82 | |||||||||
| Felt should cut down drinking (i) | 0.81 | |||||||||
| Loss of control of drinking | 0.80 | |||||||||
| Drinking large amounts | 0.80 | |||||||||
| Drank because nothing to do (i) | 0.74 | |||||||||
| Drank for negative affect | 0.74 | |||||||||
| Drank to unwind | 0.73 | |||||||||
| Drank because annoyed | 0.71 | |||||||||
| Drank because lonely (i) | 0.69 | |||||||||
| Drank because tense | 0.69 | |||||||||
| Fast drinking for quick effect (i) | 0.67 | |||||||||
| Drank because angry with self (i) | 0.65 | |||||||||
| Large amounts of time drinking | 0.65 | |||||||||
| Drank because deserved it (i) | 0.58 | |||||||||
| Drank because of physical pain (i) | 0.48 | |||||||||
| Interrupting | 0.68 | |||||||||
| Inattention | 0.68 | |||||||||
| Impulsivity | 0.67 | |||||||||
| Blurting | 0.66 | |||||||||
| Talkative (i) | 0.63 | |||||||||
| Difficulty concentrating | 0.63 | |||||||||
| Difficulty waiting turn (i) | 0.60 | |||||||||
| Restless | 0.60 | |||||||||
| Misjudged time (i) | 0.58 | |||||||||
| Difficulty delaying gratification (i) | 0.57 | |||||||||
| Needed deadlines (i) | 0.56 | |||||||||
| Bored | 0.53 | |||||||||
| Agitated | 0.52 | |||||||||
| Reckless behaviour (i) | 0.44 | |||||||||
| Difficulty sleeping | 0.32 | 0.34 | ||||||||
| Social interaction concerns | 0.75 | |||||||||
| Fear of being centre of attention | 0.74 | |||||||||
| Concerns about being observed/public speaking | 0.72 | |||||||||
| Avoidance of social situations | 0.72 | |||||||||
| Avoidance of social situations (i) | 0.70 | |||||||||
| Fear of negative evaluation | 0.68 | |||||||||
| Avoid disagreeing with others (i) | 0.63 | |||||||||
| Avoidance of crowded places (i) | 0.61 | |||||||||
| Guilt and low self-worth | 0.45 | 0.44 | ||||||||
| Angry | 0.78 | |||||||||
| Bad temper | 0.77 | |||||||||
| Irritable | 0.75 | |||||||||
| Anger fixation | 0.74 | |||||||||
| Resentful | 0.71 | |||||||||
| Hostile | 0.68 | |||||||||
| Guilt about anger (i) | 0.67 | |||||||||
| Stubborn (i) | 0.65 | |||||||||
| Felt like breaking things (i) | 0.58 | |||||||||
| Impatient | 0.50 | 0.51 | ||||||||
| Envy (i) | 0.41 | |||||||||
| Suicidal ideation | 0.80 | |||||||||
| Suicidal ideation and plans | 0.79 | |||||||||
| Suicidal thoughts (better off dead) | 0.75 | |||||||||
| Suicidal plans | 0.73 | |||||||||
| Hopelessness | 0.68 | |||||||||
| Told someone about suicidality (i) | 0.56 | |||||||||
| Depressed mood | 0.43 | 0.55 | ||||||||
| Intrusive thoughts about self-harm (i) | 0.54 | |||||||||
| Anhedonia | 0.53 | |||||||||
| Unafraid of dying (i) | 0.53 | |||||||||
| Felt like a failure (i) | 0.42 | 0.51 | ||||||||
| Dysphoria due to trauma | 0.77 | |||||||||
| Reexperiencing trauma | 0.40 | 0.75 | ||||||||
| Avoidance of cues and emotional detachment | 0.73 | |||||||||
| Anxious arousal due to trauma | 0.43 | 0.70 | ||||||||
| Self-blame for trauma | 0.64 | |||||||||
| Trauma amnesia | 0.53 |
Note. All primary loadings are shown, and cross-loadings > .4. Difficulty sleeping had a primary loading < .4, and a secondary loading of similar magnitude, so both are presented. Italicized symptom cluster names denote constructs (not including hybrid symptom clusters) ostensibly ‘out of place’ based on their primary or secondary component loading versus the target construct of the items. (i) denotes a single-item cluster. Loadings >.4 are bolded. j1 = anxiety; j2 = OCD; j3 = psychosis; j4 = substance use; j5 = alcohol use; j6 = attentional dysregulation; j7 = social anxiety; j8 = anger; j9 = suicidality/hopelessness; j10 = PTSD.
As shown in Table 1, there were several other examples of symptom clusters that did not coalesce with the other items from their original item pool at the lowest level of the hierarchy. For example, the guilt and low self-worth cluster from the PROMIS depression item pool loaded on the social anxiety component, and the depressed mood and felt like a failure symptom clusters—also from the PROMIS depression item pool—cross-loaded there too. Similarly, the impatient symptom cluster from the ADHD item pool loaded on the anger component. The hybrid clusters of agitated symptoms (ADHD and PROMIS anxiety items) and difficulty concentrating (ADHD, PROMIS anxiety, and PROMIS depression items) loaded only on attentional dysregulation, whereas the difficulty sleeping hybrid cluster (ADHD and PROMIS anxiety items) did not have a primary loading (>.4) on any component, but had weaker loadings on both attentional dysregulation and anxiety. There were only two other single-item clusters that did not have a primary loading (>.4) on any component at the bottom of the hierarchy—something seriously wrong with body and felt attacked—both of which had weaker primary loadings on the psychosis component.
At other levels of the hierarchy, there were some places where symptom clusters changed in terms of their component loading patterns (see Tables S5–S14 for full results). For example, reckless behaviour loaded on the higher-order substance and alcohol use component, which is consistent with the externalizing spectrum we might expect to emerge if maladaptive personality (e.g., antagonism) were included in the hierarchy. The feeling anxious and fear symptom clusters (comprising 81% of the PROMIS anxiety items) cross-loaded between the fear, distress, and disinhibited negative affect components. Finally, the OCD symptom clusters upsetting thoughts, intrusive thoughts about self-harm, and fear of mistakes, and the psychosis symptom cluster paranoia (people are against them), tended to have primary or cross-loadings on the internalizing, disinhibited negative affect, and/or fear components, rather than on the thought disorder component with other OCD and psychosis symptoms.
MIDAS
Highly Homogeneous Symptom Clusters
The data reduction process reduced the full item pool of 229 items in MIDAS to 92 symptom clusters—including 40 individual items, and 52 clusters ranging from 2-9 items (see Table S16 for a full list of which items formed each cluster, and for the names of each cluster). As for the AMH data, nearly all of the symptom clusters were cleanly constructed from items intended to assess the same diagnostic construct. There were only two symptom clusters composed of items intended to reflect different diagnostic constructs: (1) the difficulty sleeping cluster comprised a depression item (“I had difficulty sleeping”) and a generalized anxiety item (“I had problems sleeping because I worried about things”); and (2) the irritable cluster comprised a mania item (“I was much more irritable than usual”), a generalized anxiety item (“I was snappy or irritable because I felt stressed out”), and three anger items (“I yelled or argued”; “I let little things irritate me”; “I was rude to people from anger”). In both cases, these clusters represented symptom overlap between distinct diagnostic constructs and were labelled “hybrid” clusters.
Hierarchical Structure
Parallel analysis suggested 13 components and the MAP first reached a minimum with 14 factors, so we extracted 1 to 14 components. The level with 14 components had a component with only one unique identifier, so 1 to 13 components were extracted in the final hierarchical model. The results for each level of the model and the correlations between the levels are given in Tables S17 to S30, and the traditional bass-ackwards solution is shown in Figure S4, which presented some challenges in interpretation. For example, the narrower components of the internalizing spectrum emerged and recombined several times moving through the hierarchy: internalizing (C1) split into fear (D1) and distress (D2), which perpetuated for one level before reforming into internalizing (F1) and re-emerging as fear (H1) and distress (H2). To simplify the hierarchical structure for presentation, we removed the lower-order manifestations of recurring variables when Tucker’s congruence coefficient indicated that the components were equal (i.e., was greater than .95; Lorenzo-Seva & Ten Berge, 2006) although the component correlations fell under the threshold of r ≥ .9 (r = .83 to r= .89).
There were also several apparent artefacts among the 91 components in the full bass-ackwards solution: For example, a component of low mania symptoms with weak positive cross-loadings for suicidality symptoms emerged (H7). Conceptually redundant versions of the lower-order social anxiety, psychosis, and PTSD components emerged that had weak component loadings (i.e., L9 weak social anxiety; J9 weak psychosis; and E5 weak PTSD). There were also several instances where robust components that perpetuated through multiple levels of the hierarchy manifested as slight variations of those components with weak cross-loadings from other constructs: The OCD and psychosis component on the 8-component solution (H3) included weak cross-loadings from two agoraphobia symptom clusters; eating pathology symptoms loaded, often weakly, on the alcohol use (G7) and depression (F3) components; and several thought disorder indicators loaded on a fear (C2) component when there was no longer a coherent thought disorder component. None of these structures emerged in an agglomerative cluster hierarchy solution (Figure S2), so they were removed from interpretation of the larger hierarchical structure below. One component was initially considered a likely artefact, but ultimately retained in the hierarchy: the component initially labelled substance use with weak thought disorder (B2) mirrored a cluster that emerged in Figure S2 where the broad thought disorder indicators interleaved with substance and alcohol use in the structure. This led us to reconsider the component. On closer inspection, the thought disorder indicators with substantive loadings were mania symptoms related to impulsivity and hyperactivity (i.e., often related to externalizing psychopathology), so this component was retained and labelled externalizing.
A summary of the hierarchical structure that emerged among the remaining components is shown in Figure 1B. The first unrotated principal component, again labelled general psychopathology, was dominated by anxiety (i.e., panic, generalized anxiety, and social anxiety symptom clusters) with particularly weak loadings for substance and alcohol use symptom clusters. Correspondingly, the general psychopathology component was most strongly associated with the internalizing and fear components with a secondary correlation from the broad thought disorder component, and only a weak association with the externalizing component. Broad thought disorder was composed of core thought disorder (OCD, psychosis, and mania components) plus PTSD, and eating pathology. Internalizing was composed of distress and fear components. In turn, distress was composed of suicidality, depression, and anger components, with all of the generalized anxiety symptom clusters also loading >.4; and fear was composed of phobic avoidance (agoraphobia and social anxiety components) and somatic anxiety (panic, generalized anxiety, illness anxiety, and pain symptom clusters). As shown in Figure 1B, there were also several examples of secondary correlations for the lower-order components that spanned multiple broad spectra (e.g., eating pathology had a secondary correlation with distress, and OCD and psychosis had a secondary correlation with phobic avoidance).
The only substantive difference compared to the interpretation of the structure of the full bass-ackwards hierarchy (Figure S4) was that the association between eating pathology and broad thought disorder was not evident when exclusively focusing on associations between sequential levels of the hierarchy. Much of the complexity in the traditional bass-ackwards structure (e.g., many components with multiple secondary loadings) was related to the apparent artefacts in the structure described above.
Symptom-Level Perspectives
Table 2 shows the primary loadings and cross-loadings of the symptom clusters on the 13 components at the lowest level of the hierarchy. The 13 components again closely reflected the target constructs in the item pools. For example, components corresponding to substance use, alcohol use, psychosis, mania, OCD, PTSD, eating pathology, anger, agoraphobia, and social anxiety emerged. There were two exceptions to this pattern of items coalescing into the target constructs in the symptom measures: (1) the depression items split into separate depression and suicidality components, and (2) as mentioned above, a somatic anxiety cluster subsumed symptoms of panic, generalized anxiety, illness anxiety, and pain.
Table 2.
Primary loadings and cross-loadings of the symptom clusters on the thirteen components at the lowest level of the hierarchy in the Rhode Island Methods for Improving Diagnostic Assessment and Services data.
| Symptom cluster | m1 | m2 | m3 | m4 | m5 | m6 | m7 | m8 | m9 | m10 | m11 | m12 | m13 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Racing or pounding heart (i) | 0.82 | ||||||||||||
| Physical anxiety symptoms | 0.82 | ||||||||||||
| Trembling (i) | 0.80 | ||||||||||||
| Short of breath (i) | 0.79 | ||||||||||||
| Panic episode | 0.79 | ||||||||||||
| Dizzy or lightheaded (i) | 0.78 | ||||||||||||
| Fear of loss of control or death/choking/tingling | 0.75 | ||||||||||||
| Worry about panic (i) | 0.74 | ||||||||||||
| Generalized anxiety disorder core symptoms | 0.67 | ||||||||||||
| Nervous or anxious (i) | 0.65 | ||||||||||||
| Difficulty relaxing and concentrating | 0.64 | 0.41 | |||||||||||
| Feared fainting (i) | 0.63 | ||||||||||||
| Afraid having heart attack (i) | 0.61 | ||||||||||||
| Worried something bad might happen (i) | 0.54 | ||||||||||||
| Difficulty sleeping | 0.47 | ||||||||||||
| General somatic and pain symptoms | 0.45 | ||||||||||||
| Illness anxiety | 0.43 | ||||||||||||
| Stomach pain or bloating | 0.38 | ||||||||||||
| Fear of crowded places (i) | 0.75 | ||||||||||||
| Avoided leaving home (i) | 0.73 | ||||||||||||
| Fear of leaving the house (i) | 0.72 | ||||||||||||
| Avoidance of crowded places (i) | 0.67 | ||||||||||||
| Fear and avoidance of open spaces | 0.63 | ||||||||||||
| Fear and avoidance of queues | 0.63 | ||||||||||||
| Fear and avoidance of specific places | 0.61 | ||||||||||||
| Fear and avoidance of cars | 0.59 | ||||||||||||
| Fear and avoidance of travel | 0.58 | ||||||||||||
| Fear of being home alone (i) | 0.39 | ||||||||||||
| Checking and counting | 0.75 | ||||||||||||
| Performed rituals (i) | 0.70 | ||||||||||||
| Compulsions (i) | 0.69 | ||||||||||||
| Checking (i) | 0.67 | ||||||||||||
| Contamination concerns | 0.67 | ||||||||||||
| Need for order | 0.65 | ||||||||||||
| Repeat words to erase obsessions (i) | 0.59 | ||||||||||||
| Obsessive thoughts (i) | 0.52 | ||||||||||||
| Hoarding (i) | 0.52 | ||||||||||||
| Excessive drug use causing problems | 0.89 | ||||||||||||
| Impairment due to substance use, and attempts to cut down | 0.86 | ||||||||||||
| Used drugs to get high | 0.80 | ||||||||||||
| Used drugs in the morning (i) | 0.80 | ||||||||||||
| Urge to use drugs (i) | 0.79 | ||||||||||||
| Could not stop using drugs (i) | 0.79 | ||||||||||||
| Used drugs in hazardous situations (i) | 0.78 | ||||||||||||
| Dysphoria due to trauma | 0.87 | ||||||||||||
| Avoid internal cues (i) | 0.85 | ||||||||||||
| Reexperiencing trauma | 0.84 | ||||||||||||
| Guilt related to trauma (i) | 0.80 | ||||||||||||
| Avoid external cues | 0.79 | ||||||||||||
| Anxious arousal due to trauma | 0.77 | ||||||||||||
| Angry | 0.84 | ||||||||||||
| Lost temper (i) | 0.83 | ||||||||||||
| Grouchy (i) | 0.82 | ||||||||||||
| Irritable | 0.82 | ||||||||||||
| Felt like breaking things | 0.62 | ||||||||||||
| Difficulty concentrating | 0.65 | ||||||||||||
| Felt like a failure (i) | 0.62 | ||||||||||||
| Major depression core symptoms | 0.60 | ||||||||||||
| Guilt (i) | 0.58 | ||||||||||||
| Hypersomnia (i) | 0.45 | ||||||||||||
| Agitated (i) | 0.36 | 0.37 | |||||||||||
| Preoccupation with food | 0.81 | ||||||||||||
| Weight and shape concerns | 0.76 | ||||||||||||
| Drive for thinness (i) | 0.69 | ||||||||||||
| Thoughts about purging | 0.61 | ||||||||||||
| Purging behaviours | 0.58 | ||||||||||||
| Increased appetite when depressed (i) | 0.56 | ||||||||||||
| Hyperactivity | 0.80 | ||||||||||||
| High energy and decreased need for sleep | 0.76 | ||||||||||||
| Inflated self-esteem | 0.74 | ||||||||||||
| Euphoria | 0.74 | ||||||||||||
| Impulsivity | 0.61 | ||||||||||||
| Racing thoughts | 0.43 | ||||||||||||
| Anxious about social situations | 0.68 | ||||||||||||
| Avoidance of social situations (i) | 0.68 | ||||||||||||
| Fear of being centre of attention | 0.67 | ||||||||||||
| Fear of negative evaluation | 0.65 | ||||||||||||
| Fear and avoidance of eating/drinking/writing while observed | 0.44 | ||||||||||||
| Thought broadcasting (i) | 0.72 | ||||||||||||
| Thought insertion and broadcasting | 0.71 | ||||||||||||
| Paranoia (people want to hurt me) | 0.64 | ||||||||||||
| Visual hallucinations (i) | 0.64 | ||||||||||||
| Ideas of reference (i) | 0.61 | ||||||||||||
| Paranoia | 0.50 | ||||||||||||
| Impairment due to drinking, and attempts to cut down | 0.91 | ||||||||||||
| Thought about cutting down (i) | 0.89 | ||||||||||||
| Loss of control of drinking | 0.81 | ||||||||||||
| Drinking large amounts | 0.81 | ||||||||||||
| Suicidal ideation (i) | 0.79 | ||||||||||||
| Suicidal thoughts (better off dead) | 0.74 | ||||||||||||
| Physically hurt self | 0.64 | ||||||||||||
| Hopelessness (i) | 0.48 | 0.57 |
Note. All primary loadings are shown, and cross-loadings > .4. Agitated (i) had a primary loading < .4, and a secondary loading of similar magnitude, so both are presented. Italicized symptom cluster names denote constructs (not including hybrid symptom clusters) ostensibly ‘out of place’ based on their primary or secondary component loading versus the target construct of the items. (i) denotes a single-item cluster. Loadings >.4 are bolded. m1 = somatic anxiety; m2 = agoraphobia; m3 = OCD; m4 = substance use; m5 = PTSD; m6 = anger; m7 = depression; m8 = eating pathology; m9 = mania; m10 = social anxiety; m11 = psychosis; m12 = alcohol use; m13 = suicidality.
As shown in Table 2, there were also several examples of symptom clusters that did not coalesce with the other items from their original item pool at the lowest level of the hierarchy. For example, the single-item depression cluster increased appetite when depressed loaded on the eating pathology component, the anger symptom cluster physically hurt self loaded on the suicidality component, and the generalized anxiety cluster difficulty relaxing and concentrating cross-loaded on the depression component. The hybrid symptom cluster irritable (anger, mania, and generalized anxiety items) loaded on anger; and the hybrid cluster difficulty sleeping (depression and generalized anxiety items) loaded on somatic anxiety. Several symptom clusters did not have a primary loading (> .4) on any component, including stomach pain or bloating (primary loading on somatic anxiety), fear of being home alone (primary loading on agoraphobia), and agitated (similar weak loadings on depression and somatic anxiety).
At other levels of the hierarchy, there were some places where symptom clusters changed in terms of their component loading patterns (see Tables S17–S29 for full results). For example, hypersomnia did not load > .4 on any of the higher-order components. Obsessive thoughts (OCD) and racing thoughts (mania) symptom clusters loaded on the distress and internalizing components, but not on the core thought disorder or broad thought disorder components where other OCD and mania symptoms loaded. Further, all four generalized anxiety symptom clusters cross-loaded between the fear and distress components.
Combined Structure: Trace Items
The large majority (89%) of the trace items loaded on equivalent components at the lowest level of each hierarchy, with only a few exceptions (see Table S3). For example, avoidance of crowded places (trace item 67) was part of an agoraphobia component in MIDAS, but part of a social anxiety component in AMH where agoraphobia was not explicitly assessed. In AMH, feeling guilty (trace item 3) had a primary loading on the social anxiety component as part of the guilt and low self-worth symptom cluster, but also cross-loaded to the suicidality/hopelessness component, in line with its primary loading on the depression component in MIDAS. Similarly, difficulty sleeping (trace item 10) had weak loadings on the attentional dysregulation and anxiety components in AMH, and a primary loading on the somatic anxiety component in MIDAS, which showed some consistency in coalescing with generalized anxiety and panic symptoms in both samples. Mania was not assessed in AMH, so the trace items that loaded on the mania component in MIDAS split across two components in AMH: felt like a very special person (trace item 40) loaded on the psychosis component, and racing thoughts, restlessness, and reckless behaviour (trace items 20-24) loaded on the attentional dysregulation component. Similarly, ADHD was not assessed in MIDAS, so the trace items that loaded on the attentional dysregulation component in AMH split across two other components in MIDAS: difficulty concentrating and making decisions (trace items 17-18) loaded on the depression component, and feeling fidgety (trace item 19) had weak loadings on both the depression and somatic anxiety components.
All of these differences in the placement of trace items appeared to reflect differences in the constructs covered between the two studies, rather than substantive differences in structure. Further, the hierarchical structures had substantial overlap, with differences that could be accommodated by integrating the two hierarchies and splitting two components: (1) splitting somatic anxiety from MIDAS into separate illness anxiety and pain and anxiety syndromes to reflect the consistent syndrome of panic and generalized anxiety items coalescing seen in both samples; and (2) splitting suicidality/hopelessness from AMH into separate suicidality and depression/hopelessness syndromes to mirror their separation in MIDAS. If we take the similarity in higher-order and lower-order structures between the two samples as evidence supporting a similar hierarchical structure in both samples, we might expect an overarching hierarchical structure similar to Figure 2.
Figure 2.
An integrated summary of the two hierarchies. Symptom and symptom cluster labels are listed with all of their constituent symptoms in the supplementary materials, as are all component loadings. The order of symptom clusters listed under each syndrome is based on the strength of the component loadings in the Assessing Mental Health (AMH) data, which included more symptom clusters. If same cluster name was in both data sets, only unique cluster names were added into the list from the Rhode Island Methods for Improving Diagnostic Assessment and Services (MIDAS) data, based on a joint ordering of component loadings.
Bolded labels of symptom s/symptom clusters appear under multiple syndromes, or represent clusters with trace items that loaded under multiple components (denoted with a trace item number; e.g., t2; see Table S3).
Italicized labels denote symptoms and symptom clusters that are ostensibly ‘out of place’, based on the target construct of the items in at least one of the samples. Note there were other instances of symptoms and symptom clusters loading on different components at the higher-order level that are not included in the figure, described in-text.
Grey labels denote symptoms and symptom clusters that had a primary loading < .4 on the corresponding component.
a Pure depression/hopelessness and suicidality components did not emerge in the AMH data, so ordering of the clusters in these syndromes is based on MIDAS component loadings with unique AMH cluster names added based on joint ordering of component loadings.
b Denotes items that were split out of a larger component based on face value to reflect a finer-grained structure in the other data set. For example, illness anxiety and pain were part of a somatic anxiety component in MIDAS, which was split into two separate syndromes here (illness anxiety and pain versus anxiety) to include the consistent syndrome of panic and generalized anxiety items coalescing seen in both samples. This split was done based on the original item pools (i.e., the symptom clusters comprising items designed to measure illness anxiety and pain were moved into the separate syndrome). Similarly, suicidality items were part of a larger suicidality/hopelessness component in AMH, which was split into two separate syndromes here (suicidality and depression/hopelessness) to mirror their separation in MIDAS. This split included moving the six top-loading symptom clusters—comprising only items designed to measure suicidality AMH—to be part of the suicidality syndrome, and leaving the remaining symptom clusters on the depression syndrome.
Discussion
Most of our knowledge of the higher-order structure of psychopathology is constrained by the structure of the DSM, and research to date has been limited in its ability to characterize the symptom-level structure of psychopathology. The aim of this study was to allow the higher- and lower-order dimensions of psychopathology to depart from the structure of the DSM by delineating a detailed hierarchical model from individual symptoms up to a general factor of psychopathology. We analyzed data from two large samples—one population-based with an over-representation of psychopathology, and one clinical. Together the two samples had symptom-level assessment spanning nearly all of the spectra and subfactors in the current consensus model of the empirical structure of psychopathology (i.e., HiTOP; Kotov et al., 2017), although personality pathology was notably absent. Despite the differences between the two samples’ participants and measurement of psychopathology, there was substantial convergence between the higher-order dimensions that emerged in the two hierarchies. We therefore proposed an overarching hierarchical model to integrate them, which had some noteworthy differences compared with the higher-order dimensions in the current HiTOP model and provided new perspectives on the lower-order structure of psychopathology. We turn now to summarize the findings and interpret them in the context of extant research on the structure of psychopathology.
General Psychopathology
The first unrotated principal components had prominent panic, generalized anxiety, and social anxiety symptoms in both hierarchical models. If we compare this finding to the literature on a general factor of psychopathology, this prominence of fear symptoms is somewhat at odds with the literature that has often found general psychopathology to be dominated by distress or thought disorder indicators (e.g., Caspi et al., 2014; Lahey et al., 2012), although consistent with the finding that panic attacks represent a core indicator of current and future psychopathology (e.g., Baillie & Rapee, 2005). Recent hypotheses regarding the meaning of a general factor of psychopathology have often been oriented around the notion of an index of impairment (e.g., Caspi & Moffitt, 2018; Smith et al., 2020; Widiger & Oltmanns, 2016), which would be a logical outcome of characterizing the overlap among clinical diagnoses that almost invariably include associated impairment as a core criterion. As such, when categorical diagnoses are used as the observed variables, the prominence of disorders like GAD and schizophrenia may reflect the particularly high levels of impairment associated with meeting criteria for these diagnoses. By contrast, when examining the patterns of covariation among symptoms, the level of impairment is no longer embedded in the indicators (cf. Rapee & Spence, 2004). The prominence of anxiety symptoms here may thus be related to their prevalence and variability in the samples, corresponding to larger correlations with other symptom clusters (i.e., compared to symptoms with more restricted range) making them strong indicators of the shared variance captured in the first unrotated principal component.
Interestingly, a parallel set of hypotheses have emerged that the general factor represents disinhibited negative affect (e.g., Carver, Johnson, & Timpano, 2017; DeYoung & Krueger, 2019; Forbes, Rapee, & Krueger, 2019), and this novel component emerged in our AMH model and was indeed substantially related to general psychopathology (r = .56). As above, it seems likely that the general factor of psychopathology may be less prone to reflect impairment and distress embedded in diagnostic categories when symptoms are the units of analysis instead. Future research should examine the robustness of the nature of the general factors of psychopathology between samples and methods with the aim of clarifying whether the construct has a generalizable utility.
The general psychopathology component split into three broad spectra in both samples: substance and alcohol use (or a weak externalizing component), thought disorder, and internalizing. We will now discuss each of these branches of the hierarchical models in turn.
Substance and Alcohol Use
In both samples, substance and alcohol use indicators dominated the higher-order dimensions where we would typically see an externalizing spectrum comprising substance use together with disinhibition and antagonism (e.g., Krueger et al., 2007). This broad externalizing spectrum has robust meta-analytic support as well as substantial validity evidence (e.g., Krueger & South, 2009; Krueger & Markon, 2006; Soe-Agnie et al., 2017), but the absence of any antagonism indicators in the models examined here likely meant that externalizing psychopathology did not have adequate coverage to emerge. Some support for a weak externalizing component was evident in both samples—for example, reckless behavior loaded with substance and alcohol use in AMH, as did impulsivity and hyperactivity in MIDAS. However, it was interesting that the other indicators of aggression and disinhibition did not converge with substance and alcohol use (cf. Krueger et al., 2007), but instead tended to covary with indicators of negative affect and cognitive impairment in the internalizing disorders, as discussed below. The externalizing component was generally weakly associated with other domains of psychopathology in MIDAS, with the exception of broad thought disorder—likely due to the shared mania indicators between these components. By contrast, the substance and alcohol use component in AMH was more closely related to general psychopathology, corresponding to a much larger proportion of participants with symptoms corresponding to full threshold substance or alcohol use disorders (25% in AMH versus 2% in MIDAS).
Thought Disorder
A thought disorder component also emerged in both models, with OCD and psychosis as the core indicators, perhaps reflecting uncontrollable mental events. The close relationship between psychosis and OCD—and the corresponding primary location of OCD on a thought disorder spectrum in both models—is in contrast to the HiTOP model where OCD is an indicator of the fear subfactor under internalizing (see Figure S1). This finding adds to the growing literature that has included coverage of thought disorder indicators and subsequently found OCD to be part of the thought disorder spectrum in adults and adolescents (e.g., Caspi et al., 2014; Laceulle, Vollebergh, & Ormel, 2015). Taken together with the literature that has found that OCD symptom clusters are differentially related to internalizing and thought disorder spectra (e.g., Faure & Forbes, 2020; Watson et al., 2004), OCD should perhaps cross-load between fear and thought disorder spectra in the HiTOP model (cf. Kotov, Perlman, Gamez & Watson, 2015). This conclusion was also supported by finding OCD symptom clusters to have primary loadings on both the higher-order thought disorder and fear components, and by the secondary correlation between the OCD and psychosis component with phobic avoidance in MIDAS, as discussed below.
Similarly, while in the HiTOP model mania cross-loads between the thought disorder and internalizing spectra (Figure S1), we found mania to be a strong indicator of the broad thought disorder and core thought disorder components (r = .48 and r = .59, respectively) in MIDAS, and to have a weak negative association (r = −.18) with the internalizing component. Only the racing thoughts symptom cluster cross-loaded with internalizing, in line with other research suggesting that items assessing racing thoughts are transdiagnostic (i.e., shared with depression), rather than specific to mania (Stanton et al., 2019). Increasingly, it seems clear that symptoms of mania tend to co-occur with thought disorder symptoms when shorter recall periods are used (e.g., the past week here), and that comorbidity with internalizing symptoms is found when longer (e.g., lifetime) recall periods are used—likely reflecting the finding that individuals who experience manic episodes often also experience depressive episodes and anxiety disorders, albeit not simultaneously (Olfson et al., 2017). Similarly, despite the substantial representation of individuals with a bipolar disorder diagnosis in the MIDAS sample (9% prevalence), there was no indication here of a bipolar syndrome evident in coherence between current mania and depression symptoms. These results suggest that the provisional association of mania with both the internalizing and thought disorder spectra in the current working HiTOP model (Figure S1) could perhaps be revised to a specific association with thought disorder.
Beyond the core thought disorder component comprising psychosis, OCD, and mania, there was also a broad thought disorder component that emerged in MIDAS, including eating pathology and PTSD. Notably, eating pathology had similar associations with the core thought disorder (r = .30), broad thought disorder (r = .36) and distress (r = .34) components, somewhat consistent with evidence for associations between eating disorders and schizophrenia (e.g., Zhang et al., 2020), OCD (e.g., Forbes et al., 2017), and internalizing psychopathology (e.g., Forbush et al., 2010). However, we suggest that these results are interpreted with caution, as eating pathology had low representation (e.g., < 1% of the sample met criteria for any eating disorder diagnosis) and the symptoms tended to load inconsistently throughout the various levels of the hierarchy (e.g., with weak primary loadings on distress, depression, alcohol use, and thought disorder components; see Tables S19–S23). It will be important to examine symptom-level analyses in other samples with better representation of eating pathology to test and validate the different possible structural models.
As mentioned above, PTSD was also part of the broad thought disorder component in MIDAS. By contrast, PTSD was an indicator of the fear component under internalizing in AMH. This represented one of the few substantive differences between the two samples, which we accommodated in the overarching hierarchical model (Figure 2) by having PTSD span the thought disorder, fear, and internalizing spectra. Interestingly, the placement of PTSD in both samples was in contrast to the placement of PTSD under distress in the HiTOP model (Figure S1), but we did not find evidence to suggest that these discrepant findings could be accounted for by differential associations of PTSD symptom clusters with different spectra (cf. Gootzeti, Markon, & Watson, 2015; Steel, Fowler, & Holmes, 2005). PTSD was highly internally consistent in both samples with few substantial cross-loadings of PTSD symptom clusters once the PTSD component had emerged (none in MIDAS). Both samples had good representation of the fear and core thought disorder components, substantial coverage of PTSD symptoms, and substantial representation of traumatic experiences (e.g., 51.3% and 54.8% of AMH and MIDAS, respectively, reported at least one non-zero response to a PTSD symptom). This result, too, will be important to test in other samples and analyses.
Internalizing
Finally, the internalizing branches of the hierarchies were the most detailed in both samples. Familiar fear and distress components (cf. Kruger & Markon, 2006) emerged in both samples; the former characterized by prominent panic symptoms, and the latter by suicidality and depression symptoms. The anger component was also closely related to distress in both samples, in line with the characterization of experiences of anger as emotional distress (e.g., Pilkonis et al., 2011). In AMH where ADHD symptoms were also measured, a disinhibition component emerged comprising ADHD and anger symptoms. Together with distress, disinhibition indicated the novel disinhibited negative affect component discussed above, which in turn indicated internalizing together with fear. Other novel components found in the MIDAS data included phobic avoidance (i.e., capturing the prominent behavioral avoidance of feared situations shared by social anxiety and agoraphobia) and somatic anxiety (i.e., largely somatic symptoms captured in the panic, generalized anxiety, social anxiety, illness anxiety, and pain/somatization symptoms). The location of illness anxiety and pain/somatization symptom clusters in this somatic anxiety component could not be examined for convergence between samples, but their loading under a broad internalizing component mirrors several other analyses (e.g., Forbes et al., 2017; Krueger et al., 2003; Markon, 2010; Simms et al., 2012), so may provide further evidence to clarify the placement of the provisional somatoform spectrum in the HiTOP model (Figure S1).
The uncoupling of agoraphobia from panic seen in these results has also been examined previously at the diagnostic level, where agoraphobia covaried with social anxiety and specific phobias, similar to our findings here (Greene & Eaton, 2016). Greene and Eaton also found panic disorder without agoraphobia covaried with GAD, dysthymia, and major depression on a distress dimension. There was some indication of somatic anxiety (i.e., largely panic and GAD symptoms) relating to distress here also, but that seemed to be largely driven by the GAD symptoms, as discussed below. Overall, we tended to find panic symptoms to be the strongest indicator of fear, so further analyses of these relationships in symptom-level data would be an interesting avenue for future research.
Symptom-Level Perspectives
At the lowest levels of the hierarchy, the symptom-level approach allowed us to account for heterogeneity within and homogeneity between DSM diagnoses, providing new perspectives on the detailed structure of psychopathology. Notably, most components closely reflected the DSM constructs that the items were designed to assess. Both samples had anger, social anxiety, OCD, psychosis, PTSD, substance use, and alcohol use components that closely mirrored the structure of the original item pools. There were also eating pathology, mania, and agoraphobia components that emerged corresponding to the unique item pools included in MIDAS, and an attention dysregulation component largely mirroring the unique ADHD item pool in AMH.
The separate suicidality and PROMIS depression item pools converged in AMH, reflecting their coherence in DSM major depression, but the major depression item pool split into core depression symptoms versus suicidality in MIDAS where representation of depression symptoms was higher (e.g., 11% versus 40% of the sample reported symptoms meeting criteria for MDD in AMH and MIDAS, respectively). The illness anxiety and pain items were also not differentiated in MIDAS, likely due to their limited representation in the sample and model (i.e., 1-2 symptom clusters, and 2% prevalence). Similarly, GAD and panic symptoms were not differentiated in either sample—although this was despite their substantial representation in both samples (i.e., 15-26 symptom clusters and 4-20% prevalence). While GAD was not measured directly in AMH (i.e., the transdiagnostic PROMIS anxiety measure was used), the items measured in MIDAS corresponded closely to the DSM symptom criteria of GAD; a GAD syndrome did not emerge in either case. One reason for this may be that GAD and panic symptoms were assessed on the same time scale (i.e., over the past month for AMH and the past week for MIDAS), so the distinction between the chronicity of GAD symptoms versus the acute nature of panic symptoms was lost in many cases.
Despite the close convergence with panic symptoms at the lower-order component level, GAD symptoms did diverge in their consistent cross-loadings between the higher-order fear and distress (or disinhibited negative affect) components in both samples. The associations of GAD with panic and the fear dimension are in contrast to the consistent finding that GAD shares more in common with depressive versus anxiety disorders (e.g., Watson, 2005), and thus represents a robust indicator of the distress spectrum in the literature to date (e.g., Kotov et al., 2017; Krueger & Markon, 2006). This finding may be an indication that symptom overlap between MDD and GAD diagnoses inflates the rate of comorbidity observed between them, with implications for the corresponding structural models that rely on categorical diagnoses as units of analysis.
Disorder-level heterogeneity was also evident for several diagnoses wherein constituent symptom clusters loaded across multiple spectra. For example, as alluded to earlier, OCD symptoms of fear of mistakes and upsetting and obsessive thoughts loaded on the fear, distress, and internalizing components in both samples, diverging from other symptoms on the thought disorder components. Social anxiety symptoms also often cross-loaded between fear, distress, and disinhibited negative affect components—for example, fear of negative evaluation tended to load more strongly on distress and disinhibited negative affect in both samples (cf. Lovibond & Rapee, 1993). This heterogeneity may also have been driving the cross-loading observed for the social anxiety component in AMH, reinforcing the importance of examining symptom-level information where available in future research.
Major depression was a particularly heterogeneous diagnosis in our models, mirroring research that has highlighted substantial variation in MDD symptom profiles and called for symptom-level analyses (e.g., Fried & Nesse, 2015). Specifically, in MIDAS, increased appetite when depressed loaded on eating pathology, difficulty sleeping loaded on anxiety, being so fidgety it was hard to sit still cross-loaded on anxiety, and suicidality items formed a separate cluster. Similarly, in AMH, guilt and low self-worth, felt like a failure, and depressed mood cross-loaded on social anxiety, and difficulty making decisions was in the difficulty concentrating cluster loading on attentional dysregulation. This lack of coherence among depression symptoms was despite the good representation of depression items in both samples, and high prevalence (40%) of MDD in MIDAS in particular. These findings reinforce the notion that studying MDD as a single ‘present versus absent’ category is likely to lose important information and variation at the symptom level.
At the most detailed level of the models, there was also useful information about the utility of individual symptoms and symptom clusters for differential diagnosis—that is, teasing apart disorder-level syndromes based on symptoms that are robust and specific indicators of one syndrome, versus transdiagnostic indicators of multiple syndromes (symptoms and symptom clusters that loaded on multiple syndromes are bolded in Figure 2). This was particularly interesting to consider for symptoms that represented overlapping criteria between multiple diagnoses. For example, irritability was a symptom assessed in item pools aiming to measure anger, GAD, and mania, but irritability symptoms consistently loaded only with anger in both samples, suggesting it might be better conceptualized as an indicator of anger more so than GAD or mania. By contrast, other overlapping symptoms—such as restlessness and difficulty concentrating from the GAD, depression, and ADHD item pools—tended to demonstrate low specificity and thus appear to represent transdiagnostic symptoms that would not be useful for differentiating the different syndromes.
Limitations and Future Directions
These are secondary analyses of existing data from two studies that were not specifically designed for the purpose of understanding the symptom-level structure of psychopathology, which led to four particularly important limitations in the present study that should be kept in mind in interpreting these findings. First, the differences between the samples and methods led to some challenges in comparing the two sets of results. For example, these data were from two different countries and cultures (i.e., Australia and the USA), and were drawn from different populations (the general community, and a treatment-seeking clinical sample) within those contexts. Further, the two studies used different measures assessing different time frames (past month versus past week for AMH and MIDAS, respectively) and varied in their coverage of domains of psychopathology. Future research should examine the replicability of these findings in diverse samples where methodological differences do not introduce noise. However, these differences between the samples made the convergence in the results noteworthy: overall, the syndrome-level components that were measured in both samples were very similar, and all six of the higher-order dimensions that had disorder-level coverage in both samples emerged consistently.
Second, the measurement of psychopathology in both samples was geared towards uncovering syndromes in the DSM: the majority of the symptom-level items were from measures designed to assess a single internally consistent construct. The process of measure development usually involves dropping the interstitial and non-specific (transdiagnostic) symptoms (cf. Clark & Watson, 2019)—i.e., the ‘noise’ between the boundaries of disorder-level constructs that we are particularly interested in here. Several of the item pools in AMH were less refined, as they were based on a systematic review of multiple extant measures, but items were still eliminated if they were deemed unrelated or not specific to the disorder of interest (Batterham et al., 2015). Further, the items in both studies were administered in blocks corresponding to the domain of psychopathology that they were intended to measure, which likely further reinforced the structure of the DSM by priming participants to think about their symptoms in the context of the broader syndrome (i.e., demand characteristics that may introduce local dependence among items). This may be an explanation for why the overlapping symptoms (trace items) assessed in the ADHD and mania item pools did not converge in the hierarchical structure between the two samples. Future research should consider fully randomizing item pools. By contrast, it was a strength that the items were all assessed on consistent response scales and using consistent timeframes within each study, as this will have minimized the likelihood of bias due to differences on these measurement characteristics corresponding with the boundaries between traditional DSM diagnoses (cf. Markon 2010; Wright et al., 2013). Overall, the correspondence of the study methods with DSM constructs means that departures from the DSM structure found here (e.g., hybrid item clusters, cross-loadings, and symptoms and symptom clusters acting as indicators for a nontarget construct) are even more compelling.
Third, the granularity of some symptoms was not ideal for the purpose of delineating the symptom-level structure of psychopathology. For example, many of the items included a direct link between symptoms and their cause or context, which may have introduced artefactual structures into the hierarchical models (e.g., asking about dysphoria due to trauma or role impairment due to substance use). Similarly, items like “I feared social or work situations because I felt that people were judging me” might be better assessed as two separate items, giving us the opportunity to estimate empirically whether these experiences covary; fear of work or social situations may also be related to contamination concerns, and feeling judged by people may be related to paranoia, for example. Measuring the symptoms separately could allow the patterns of covariation to guide the placement of the symptoms. Since we cannot measure all permutations of causes, outcomes, and impairment related to a symptom, this approach would provide more opportunities to learn about the detailed structure of psychopathology by empirically estimating these relationships based on patterns of covariation.
Finally, these data did not cover all domains of psychopathology. There was good direct coverage of about 18 DSM-IV and DSM-5 diagnoses in total, reflecting some of the more burdensome and prevalent mental disorders, but this is a small proportion of the breadth of psychopathology described in the DSM alone. One noteworthy absence was the inclusion of personality pathology in these analyses, which as mentioned above often appears to act as a skeleton for joint structural models—particularly the core externalizing domains of antagonism and disinhibition (e.g., Forbes et al., 2017; Kotov et al., 2011; Markon, 2010; Wright & Simms, 2015). If future studies include personality pathology and broader coverage of other domains of psychopathology, different structures might emerge. Overall, we encourage future studies to collect data with the specific aim of understanding the symptom-level structure of psychopathology, assessing randomized items that comprehensively assess psychopathology at a fine level of granularity. Further, these analyses were exploratory and focused on characterizing the patterns of covariation among the symptoms; the results should be tested for replicability in other samples and using other analytic methods, as well as for criterion validity in predicting important correlates of psychopathology, to determine their utility in empirical classification efforts.
Conclusion
This was the first comprehensive and detailed analysis of the hierarchical structure of psychopathology that emerges when analyzing symptom-level data, representing an important step towards identifying reliable and detailed phenotypes of psychopathology to improve current methods in clinical research, practice, and assessment of mental illness. We used two large and varied samples that were ideal for these analyses, given the representation, variability, and breadth of measurement of psychopathology. A summary of the results is presented in Figure 2. There was marked convergence between the two samples, offering new perspectives on higher-order structures, including several differences compared to the current HiTOP model, and three novel higher-order dimensions that will require replication in other samples and methods. We also found several departures from the structure of the DSM in the symptom-level data that should be extended in future research specifically designed to quantify the symptom-level structure of psychopathology. We hope that these results assist in clarifying the way forward for quantitative classification efforts as the field moves beyond the confines of the structure of DSM disorders.
Supplementary Material
Acknowledgments
Funding: MKF is supported by a Macquarie University Research Fellowship, RMR is supported by Australian Research Council Laureate Fellowship FL150100096, ALC is supported by NHMRC Fellowships 1122544 and 1173146, and RFK is supported in part by the U.S. National Institute on Aging grants R01AG053217 and U19AG051426.
Footnotes
We note that research in children and adolescents has long taken this approach (e.g., Achenbach & Edelbrock, 1978), and has led the way in moving the literature towards focusing on symptom-level information (e.g., Afzali et al., 2018; Carragher et al., 2016; de la Cruz et al., 2018; Haltigan et al., 2018; Lahey et al., 2008). Similarly, research on the structure of normative and maladaptive personality has characterized a comprehensive dimensional model on the basis of item-level analyses (e.g., Cattell, 1943; Krueger, Derringer, Markon, Watson & Skodol, 2012). However, in adult samples the literature on the structure of psychopathology (traditional Axis I disorders) has maintained a strong focus on patterns of comorbidity or covariation among DSM disorders since the internalizing and externalizing spectra were first uncovered in adults (Krueger et al., 1998).
Spearman correlations were estimated based on pairwise complete data, and the 562*562 correlation matrix in the AMH data was not positive definite, which required smoothing by eigenvalue decompositions (Bock, Gibbons, &Muraki, 1988).
References
- Achenbach TM, & Edelbrock CS (1978). The classification of child psychopathology: a review and analysis of empirical efforts. Psychological bulletin, 55(6), 1275. [PubMed] [Google Scholar]
- Afzali MH, Sunderland M, Carragher N, & Conrod P (2018). The structure of psychopathology in early adolescence: study of a Canadian sample: la structure de la psychopathologie au debut de Tadolescence: etude d’un echantillon canadien. The Canadian Journal of Psychiatry, 63(4), 223–230. [DOI] [PMC free article] [PubMed] [Google Scholar]
- American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). 10.1176/appi.books.9780890425596 [DOI] [Google Scholar]
- Australian Bureau of Statistics. (2016a). Australia, 2016 Census QuickStats. Retrieved February 4th 2020 from https://quickstats.censusdata.abs.gov.au/census_services/getproduct/census/2016/quickstat/036?opendocument
- Australian Bureau of Statistics. (2016b). Media Release: Census reveals a fast changing, culturally diverse nation. Retrieved February 4th 2020 from https://www.abs.gov.au/ausstats/abs@.nsf/lookup/Media%20Release3
- Baillie AJ, & Rapee RM (2005). Panic attacks as risk markers for mental disorders. Social Psychiatry and Psychiatric Epidemiology, 40(3), 240–244. [DOI] [PubMed] [Google Scholar]
- Batterham PJ, Brewer JL, Tjhin A, Sunderland M, Carragher N, & Calear AL (2015). Systematic item selection process applied to developing item pools for assessing multiple mental health problems. Journal of clinical epidemiology, 65(8), 913–919. [DOI] [PubMed] [Google Scholar]
- Batterham PJ, Sunderland M, Carragher N, & Calear AL (2016). Development and community-based validation of eight item banks to assess mental health. Psychiatry research, 243, 453–462. [DOI] [PubMed] [Google Scholar]
- Bock RD, Gibbons R, & Muraki E (1988). Full-Information Item Factor Analysis. Applied Psychological Measurement, 12 (3), 261–280. [Google Scholar]
- Borsboom D (2002). The structure of the DSM. Archives of general psychiatry, 59(6), 569–570. [DOI] [PubMed] [Google Scholar]
- Carragher N, Teesson M, Sunderland M, Newton NC, Krueger RF, Conrod PJ, … & Slade T (2016). The structure of adolescent psychopathology: a symptom-level analysis. Psychological Medicine, 46(5), 981–994. [DOI] [PubMed] [Google Scholar]
- Carver CS, Johnson SL, & Timpano KR (2017). Toward a functional view of the p factor in psychopathology. Clinical Psychological Science, 5, 880–889 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Caspi A, Houts RM, Belsky DW, Goldman-Mellor SJ, Harrington H, Israel S, … Poulton R (2014). The p factor one general psychopathology factor in the structure of psychiatric disorders? Clinical Psychological Science, 2(2), 119–137. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Caspi A & Moffitt TE (2018). All for one and one for all: Mental disorders in one dimension. American Journal of Psychiatry, 175, 831–844 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Cattell RB (1943). The description of personality: basic traits resolved into clusters. The Journal of Abnormal and Social Psychology, 35(4), 476–506. 10.1037/h0054116 [DOI] [Google Scholar]
- Conway CC, & Brown TA (2018). Evaluating dimensional models of psychopathology in outpatients diagnosed with emotional disorders: A cautionary tale. Depression and anxiety, 35(9), 898–902. [DOI] [PubMed] [Google Scholar]
- Conway CC, Forbes MK, Forbush KT, Fried ET, Hallquist MN, Kotov R, … & Eaton NR (2019). A hierarchical taxonomy of psychopathology can transform mental health research. Perspectives on psychological science, 14(3), 419–436. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Dalrymple K, Martinez J, Tepe E, Young D, Chelminski I, Morgan T, & Zimmerman M (2013). A clinically useful social anxiety disorder outcome scale. Comprehensive psychiatry, 54(1), 758–765. [DOI] [PMC free article] [PubMed] [Google Scholar]
- de la Cruz LF, Vidal-Ribas P, Zahreddine N, Mathiassen B, Brondbo PH, Simonoff E, … & Stringaris A (2018). Should Clinicians Split or Lump Psychiatric Symptoms? The Structure of Psychopathology in Two Large Pediatric Clinical Samples from England and Norway. Child Psychiatry & Human Development, 49(4), 607–620. [DOI] [PMC free article] [PubMed] [Google Scholar]
- DeYoung CG & Krueger RF (2018). A cybernetic theory of psychopathology. Psychological Inquiry, 29, 117–138 [Google Scholar]
- Dornbach-Bender A, Ruggero CJ, Waszczuk MA, Gamez W, Watson D, & Kotov R (2017). Mapping internalizing psychopathology at the finest level: Convergent validity and structure based on alternative measures. Comprehensive Psychiatry, 79, 31–39. [DOI] [PubMed] [Google Scholar]
- Faure K, & Forbes MK (2020). Clarifying the Placement of Obsessive-Compulsive Disorder in the Empirical Structure of Psychopathology. Preprint on Open Science Framework. 10.31219/osf.io/gfxue [DOI] [Google Scholar]
- Forbes MK (2020). Extending Goldberg’s bass-ackwards method for developing hierarchical structural models. Preprint on Open Science Framework. 10.31219/osf.io/ksxwv [DOI] [Google Scholar]
- Forbes MK, Kotov R, Ruggero CJ, Watson D, Zimmerman M, & Krueger RF (2017). Delineating the joint hierarchical structure of clinical and personality disorders in an outpatient psychiatric sample. Comprehensive psychiatry, 79, 19–30. doi: 10.1016/j.comppsych.2017.04.006 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Forbes MK, Rapee RM, & Krueger RF (2019). Opportunities for the prevention of mental disorders by reducing general psychopathology in early childhood. Behaviour research and therapy, 119, 103411. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Forbush KT, South SC, Krueger RF, Iacono WG, Clark LA, Keel PK, … & Watson D (2010). Locating eating pathology within an empirical diagnostic taxonomy: evidence from a community-based sample. Journal of abnormal psychology, 119(2), 282. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Fried EI, & Nesse RM (2015). Depression is not a consistent syndrome: An investigation of unique symptom patterns in the STAR* D study. Journal of Affective Disorders, 172, 96–102. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Goldberg LR (2006). Doing it all Bass-Ackwards: The development of hierarchical factor structures from the top down. Journal of Research in Personality, 40(A), 347–358. 10.1016/j.jrp.2006.01.001 [DOI] [Google Scholar]
- Gootzeit J, Markon K, & Watson D (2015). Measuring dimensions of posttraumatic stress disorder: The Iowa traumatic response inventory. Assessment, 22(2), 152–166 [DOI] [PubMed] [Google Scholar]
- Greene AL, & Eaton NR (2016). Panic disorder and agoraphobia: A direct comparison of their multivariate comorbidity patterns. Journal of affective disorders, 190, 75–83. [DOI] [PubMed] [Google Scholar]
- Grisanzio KA, Goldstein-Piekarski AN, Wang MY, Ahmed APR, Samara Z, & Williams LM (2018). Transdiagnostic symptom clusters and associations with brain, behavior, and daily function in mood, anxiety, and trauma disorders. JAMA psychiatry, 75(2), 201–209. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Hopwood CJ, Bagby RM, Gralnick T, Ro E, Ruggero C, Mullins-Sweatt S, Kotov R, Bach B, Cicero DC, Krueger RF, Patrick CJ, Chmielewski M, DeYoung CG, Docherty AR, Eaton NR, Forbush KT, Ivanova MY, Latzman RD, Pincus AL, … Zimmermann J (2019). Integrating psychotherapy with the hierarchical taxonomy of psychopathology (HiTOP). Journal of Psychotherapy Integration. Advance online publication. 10.1037/int0000156 [DOI] [Google Scholar]
- Haltigan JD, Aitken M, Skilling T, Henderson J, Hawke L, Battaglia M, … & Andrade BF (2018). “P” and “DP:” examining symptom-level bifactor models of psychopathology and dysregulation in clinically referred children and adolescents. Journal of the American Academy of Child & Adolescent Psychiatry, 57(6), 384–396. [DOI] [PubMed] [Google Scholar]
- Hyman SE (2007). Can neuroscience be integrated into the DSM-V?. Nature Reviews Neuroscience, 8, 725–732. [DOI] [PubMed] [Google Scholar]
- Kessler RC, & Üstün TB (2004). The world mental health (WMH) survey initiative version of the world health organization (WHO) composite international diagnostic interview (CIDI). International journal of methods in psychiatric research, 73(2), 93–121. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kotov R, Foti D, Li K, Bromet EJ, Hajcak G, & Ruggero CJ (2016). Validating dimensions of psychosis symptomatology: Neural correlates and 20-year outcomes. Journal of abnormal psychology, 725(8), 1103. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kotov R, Krueger RF, Watson D, Achenbach TM, Althoff RR, Bagby RM, … & Eaton NR (2017). The Hierarchical Taxonomy of Psychopathology (HiTOP): a dimensional alternative to traditional nosologies. Journal of abnormal psychology, 126(A), 454. [DOI] [PubMed] [Google Scholar]
- Kotov R, Perlman G, Gámez W, & Watson D (2015). The structure and short-term stability of the emotional disorders: a dimensional approach. Psychological Medicine, 15(8), 1687–1698. [DOI] [PubMed] [Google Scholar]
- Kotov R, Ruggero CJ, Krueger RF, Watson D, Yuan Q, & Zimmerman M (2011). New dimensions in the quantitative classification of mental illness. JAMA Psychiatry, 68 (10), 1003–1011 [DOI] [PubMed] [Google Scholar]
- Kotov R, Ruggero CJ, Krueger RF, Watson D, & Zimmerman M (2018). The perils of hierarchical exclusion rules: A further word of caution. Depression and Anxiety, 35(9), 903–904. doi: 10.1002/da.22826 [DOI] [PubMed] [Google Scholar]
- Kozak MJ, & Cuthbert BN (2016). The NIMH research domain criteria initiative: Background, issues, and pragmatics. Psychophysiology, 53, 286–297 [DOI] [PubMed] [Google Scholar]
- Krueger RF, Caspi A, Moffitt TE, & Silva PA (1998). The structure and stability of common mental disorders (DSM-III-R): a longitudinal-epidemiological study. Journal of abnormal psychology, 107(2), 216. [DOI] [PubMed] [Google Scholar]
- Krueger RF, Chentsova-Dutton YE, Markon KE, Goldberg D, & Ormel J (2003). A cross-cultural study of the structure of comorbidity among common psychopathological syndromes in the general health care setting. Journal of abnormal psychology, 112(3), 437. [DOI] [PubMed] [Google Scholar]
- Krueger RF, Derringer J, Markon KE, Watson D, & Skodol AE (2012). Initial construction of a maladaptive personality trait model and inventory for DSM-5. Psychological medicine, 42(9), 1879–1890. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Krueger RF, Kotov R, Watson D, Forbes MK, Eaton NR, Ruggero CJ, … & Zimmerman J (2018). Progress in achieving quantitative classification of psychopathology. World Psychiatry, 17(3), 282–293. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Krueger RF, & Markon KE (2006). Reinterpreting comorbidity: A model-based approach to understanding and classifying psychopathology. Annu. Rev. Clin. Psychol, 2, 111–133 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Krueger RF, Markon KE, Patrick CJ, Benning SD, & Kramer MD (2007). Linking antisocial behavior, substance use, and personality: an integrative quantitative model of the adult externalizing spectrum. Journal of abnormal psychology, 116(4), 645. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Laceulle OM, Vollebergh WA, & Ormel J (2015). The structure of psychopathology in adolescence: Replication of a general psychopathology factor in the TRAILS study. Clinical Psychological Science, 5(6), 850–860 [Google Scholar]
- Lahey BB, Rathouz PJ, Van Hulle C, Urbano RC, Krueger RF, Applegate B, … & Waldman ID (2008). Testing structural models of DSM-IV symptoms of common forms of child and adolescent psychopathology. Journal of abnormal child psychology, 36(2), 187–206. [DOI] [PubMed] [Google Scholar]
- Lahey BB, Applegate B, Hakes JK, Zald DH, Hariri AR, & Rathouz PJ (2012). Is there a general factor of prevalent psychopathology during adulthood? Journal of abnormal psychology, 121(4), 971. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Loehlin JC, & Goldberg LR (2014). Do personality traits conform to lists or hierarchies?. Personality and individual differences, 70, 51–56. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Lovibond PF, & Rapee RM (1993). The representation of feared outcomes. Behaviour Research and Therapy, 31(6), 595–608. [DOI] [PubMed] [Google Scholar]
- Markon KE (2010). Modeling psychopathology structure: A symptom-level analysis of Axis I and II disorders. Psychological medicine, 40(2), 273–288. [DOI] [PubMed] [Google Scholar]
- Olfson M, Mojtabai R, Merikangas KR, Compton WM, Wang S, Grant BF, & Blanco C (2017). Reexamining associations between mania, depression, anxiety and substance use disorders: results from a prospective national cohort. Molecular psychiatry, 22(2), 235–241. [DOI] [PubMed] [Google Scholar]
- Pilkonis PA, Choi SW, Reise SP, Stover AM, Riley WT, Cella D, & PROMIS Cooperative Group. (2011). Item banks for measuring emotional distress from the Patient-Reported Outcomes Measurement Information System (PROMIS®): depression, anxiety, and anger. Assessment, 18(3), 263–283 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Pilkonis PA, Yu L, Dodds NE, Johnston KL, Lawrence SM, & Daley DC (2016). Validation of the alcohol use item banks from the Patient-Reported Outcomes Measurement Information System (PROMIS®). Drag and alcohol dependence, 161, 316–322. [DOI] [PMC free article] [PubMed] [Google Scholar]
- R Core Team (2019). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org/ [Google Scholar]
- Rapee RM, & Spence SH (2004). The etiology of social phobia: Empirical evidence and an initial model. Clinical Psychology Review, 24, 737–767. doi: 10.1016/j.cpr.2004.06.004 [DOI] [PubMed] [Google Scholar]
- Revelle W (1979). Hierarchical cluster analysis and the internal structure of tests. Multivariate Behavioral Research, 14(1), 57–74. [DOI] [PubMed] [Google Scholar]
- Revelle W (2019). psych: Procedures for Psychological, Psychometric, and Personality Research. Northwestern University, Evanston, Illinois. R package version 1.8.12, https://CRAN.R-project.org/package=psych. [Google Scholar]
- Ruggero CJ, Kotov R, Hopwood CJ, First M, Clark LA, Skodol AE, Mullins-Sweatt SN, … & Zimmerman J (2019). Integrating the Hierarchical Taxonomy of Psychopathology (HiTOP) into clinical practice. Journal of Consulting and Clinical Psychology, 87, 1069–1084. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Simms LJ, Prisciandaro JJ, Krueger RF, & Goldberg DP (2012). The structure of depression, anxiety and somatic symptoms in primary care. Psychological medicine, 42(1), 15–28 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Smith GT, Atkinson EA, Davis HA, Riley EN, & Oltmanns JR (2020). The General Factor of Psychopathology. Annual Review of Clinical Psychology, 16. [DOI] [PubMed] [Google Scholar]
- Sonuga-Barke EJ (2016). Editorial: Distinguishing between the challenges posed by surface and deep forms of heterogeneity to diagnostic systems: Do we need a new approach to subtyping of child and adolescent psychiatric disorders? Journal of Child Psychology and Psychiatry, 57, 1–3 [DOI] [PubMed] [Google Scholar]
- Stanton K, Khoo S, Watson D, Gruber J, Zimmerman M, & Weinstock LM (2019). Unique and transdiagnostic symptoms of hypomania/mania and unipolar depression. Clinical Psychological Science, 7(3), 471–487 [Google Scholar]
- Steel C, Fowler D, & Holmes EA (2005). Trauma-related intrusions and psychosis: an information processing account. Behavioural and Cognitive Psychotherapy, 33(2), 139–152. [Google Scholar]
- Sunderland M, Batterham P, Carragher N, Calear A, & Slade T (2019). Developing and validating a computerized adaptive test to measure broad and specific factors of internalizing in a community sample. Assessment, 26(6), 1030–1045. [DOI] [PubMed] [Google Scholar]
- Sunderland M, Slade T, Krueger RF, Markon KE, Patrick CJ, Kramer MD (2017). Efficiently measuring dimensions of the externalizing spectrum model: development of the Externalizing Spectrum Inventory – Computerized Adaptive Test. Psychological Assessment, 29(1), 868–880. [DOI] [PubMed] [Google Scholar]
- United States Census Bureau. (2010). Profile of General Population and Housing Characteristics: 2010. Retrieved February 6th 2020 from https://factfinder.census.gov/faces/tableservices/jsf/pages/productview.xhtml?pid=DEC_10_DP_DPDP1&src=pt
- Ward JH Jr (1963). Hierarchical grouping to optimize an objective function. Journal of the American statistical association, 55(301), 236–244. [Google Scholar]
- Waszczuk MA, Eaton NR, Krueger RF, Shackman AJ, Waldman ID, Zald DH, … & Kotov R (2019). Redefining phenotypes to advance psychiatric genetics: Implications from hierarchical taxonomy of psychopathology. Journal of Abnormal Psychology. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Waszczuk MA, Kotov R, Ruggero C, Gamez W, & Watson D (2017). Hierarchical structure of emotional disorders: From individual symptoms to the spectrum. Journal of Abnormal Psychology, 126(5), 613. [DOI] [PubMed] [Google Scholar]
- Watson D (2005). Rethinking the mood and anxiety disorders: A quantitative hierarchical model for DSM-V. Journal of Abnormal Psychology, 114(4), 522–536 [DOI] [PubMed] [Google Scholar]
- Watson D, Wu KD, & Cutshall C (2004). Symptom subtypes of obsessive-compulsive disorder and their relation to dissociation. Journal of Anxiety Disorders, 18(4), 435–458. [DOI] [PubMed] [Google Scholar]
- Widiger TA, & Oltmanns JR (2016). The general factor of psychopathology and personality. Clinical Psychological Science, 2167702616657042 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Wright AG, Krueger RF, Hobbs MJ, Markon KE, Eaton NR, & Slade T (2013). The structure of psychopathology: toward an expanded quantitative empirical model. Journal of abnormal psychology, 122(1), 281. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Wright AG, & Simms LJ (2015). A metastructural model of mental disorders and pathological personality traits. Psychological medicine, 45(11), 2309–2319. doi: 10.1017/S0033291715000252 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Zald DH, & Lahey BB (2017). Implications of the hierarchical structure of psychopathology for psychiatric neuroimaging. Biological Psychiatry: Cognitive Neuroscience and Neuroimaging, 2(4), 310–317. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Zhang R, Larsen JT, Kuja-Halkola R, Thornton L, Yao S, Larsson H, … & Bergen SE (2020). Familial co-aggregation of schizophrenia and eating disorders in Sweden and Denmark. Molecular Psychiatry, 1–9. [DOI] [PubMed] [Google Scholar]
- Zimmerman M (2016). A review of 20 years of research on overdiagnosis and underdiagnosis in the Rhode Island Methods to Improve Diagnostic Assessment and Services (MIDAS) Project. The Canadian Journal of Psychiatry, 61(2), 71–79. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Zimmerman M, Chelminski I, McGlinchey JB, & Postemak MA (2008). A clinically useful depression outcome scale. Comprehensive psychiatry, 49(2), 131–140. [DOI] [PubMed] [Google Scholar]
- Zimmerman M, Chelminski L, Young D, & Dalrymple K (2010). A clinically useful anxiety outcome scale. The Journal of clinical psychiatry. [DOI] [PubMed] [Google Scholar]
- Zimmerman M, & Mattia JI (2001). The Psychiatric Diagnostic Screening Questionnaire Development, reliability and validity. Comprehensive Psychiatry, 42(3), 175–189. 10.1053/comp.2001.23126 [DOI] [PubMed] [Google Scholar]
- Zinbarg RE, & Barlow DH (1996). Structure of anxiety and the anxiety disorders: a hierarchical model. Journal of Abnormal Psychology, 105(2), 181. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.