Appropriate Use of Bifactor Analysis in Psychopathology Research: Appreciating Benefits and Limitations

Abstract

Co-occurrence of psychiatric disorders is well documented. Recent quantitative efforts have moved toward an understanding of this phenomenon, with the general psychopathology or p-factor model emerging as the most prominent characterization. Over the past decade, bifactor model analysis has become increasingly popular as a statistical approach to describe common/shared and unique elements in psychopathology. However, recent work has highlighted potential problems with common approaches to evaluating and interpreting bifactor models. Here, we argue that bifactor models, when properly applied and interpreted, can be useful for answering some important questions in psychology and psychiatry research. We review problems with evaluating bifactor models based on global model fit statistics. We then describe more valid approaches to evaluating bifactor models and highlight 3 types of research questions for which bifactor models are well suited to answer. We also discuss the utility and limits of bifactor applications in genetic and neurobiological research. We close by comparing advantages and disadvantages of bifactor models with other analytic approaches and note that no statistical model is a panacea to rectify limitations of the research design used to gather data.

Comorbidity among heterotypic mental disorders is ubiquitous (1), leading some to suggest that mental disorders have more common/shared than unique processes. Psychology and psychiatry thus have increasingly used quantitative methods to model covariation among disorders and organize them into higher-order domains (2–4). Such models can separate psychopathology deficits shared by multiple disorders from those unique to specific disorders. One increasingly popular quantitative framework is the bifactor model (5). This model specifies that covariance among observed indicators can be accounted for by a latent general factor, reflecting common variance among all indicators, and one or more latent group factors,¹ reflecting additional common variance for subsets of indicators (5,6). Group factors are specified to be orthogonal (uncorrelated) to the general factor, so group factors reflect common variance among indicator subsets that is separable from the general factor.²

The bifactor model has become popular in psychology and psychiatry research as a method to 1) model commonality and uniqueness across mental disorders and 2) relate said common and unique factors to putative antecedents (e.g., treatments, genetic/environmental factors, neurobiological substrates, personality traits) and outcomes (e.g., cognitive development, academic performance, distress, self-harm, suicidality) (7–13). Many studies have applied bifactor models to document a general psychopathology or p factor reflecting commonality among all forms of psychopathology along with several narrower psychopathology group factors, most commonly internalizing (depression, anxiety),³ externalizing (antisocial and substance use disorders), and psychosis (7,10,14–18). Bifactor and related hierarchical models (19) are being incorporated into emerging frameworks for conceptualizing, studying, and diagnosing psychopathology (17,20). Bifactor models are also applied in other psychology subfields to describe constructs such as cognitive abilities (21), personality traits (22,23), and work interests (24).

However, recent conceptual, methodological, and empirical work has highlighted problems with common approaches to evaluating and interpreting bifactor models. These criticisms include a tendency to overfit, such that the bifactor model is inappropriately favored by fit indices; frequent anomalous results, including small factor loadings and zero or negative group factor variances; instability of the general factor, such that the nature of the general factor changes across samples or indicators; problems with identification; questions regarding interpretation of orthogonal latent factors; and concerns about reification in searches for genetic or biological substrates of the p factor (25–28). Here, we suggest that bifactor models, properly applied and interpreted, can be useful for answering some important questions in psychology and psychiatry research. We briefly review problems with the widespread practice of evaluating bifactor and other structural models based solely on global model fit statistics. We then describe better approaches to evaluating bifactor models and highlight three types of research questions that bifactor models are well suited to answer. Finally, we compare bifactor models with other analytic approaches, discuss applications in psychobiological research, and note that no statistical model is a panacea for limitations of the data collection design.

PROBLEMATIC INTERPRETATION OF BIFACTOR MODELS: RELIANCE ON GLOBAL MODEL FIT

The major criticism of the bifactor model is its potential for overfitting (29). A common approach to evaluating structural models is to compare several possible models and then retain the model showing the best overall (global) fit statistics such as χ², comparative fit index (CFI), Tucker–Lewis index (TLI), root mean square error of approximation (RMSEA), unbiased standardized root mean square residual (USRMR), and Akaike information criterion (AIC) (10,11). This approach is problematic because global fit statistics can favor the bifactor model even when it is a poor description of the data. The confirmatory bifactor model is extremely flexible. The only major constraint imposed on the data is the group factor to which each item belongs. The exact patterns of items’ loadings onto the general and group factors are typically permitted to freely vary. Essentially, the model absorbs as much item variance as possible into the general or group factors. Because of this flexibility, the bifactor model can exhibit good global fit even if the pattern of loadings does not resemble a bifactor structure in any meaningful sense. For example, consider Figure 1B. Here, items from only one subgroup show strong loadings onto the general factor, with negligible loadings onto their group factor. Items from the other two subgroups load weakly onto the general factor and load strongly onto their group factors. This pattern of loadings implies that the bifactor structure is a poor description of the data despite adequate global fit statistics. In this example, the general factor is not really a general factor at all but rather just a group factor that has been shuffled into a different part of the model [cf. Giordano and Waller (30)]. In this case, the appropriate conclusion is that the data represent three weakly correlated factors. A similar interpretation can be made in the common case that one of the group factor variances is near zero or negative (26). The bifactor model’s flexibility can be particularly problematic when comparing bifactor model results across samples (29). Because the model seeks to absorb as much variance as possible, the pattern of factor loadings defining the general or group factors can be unstable across samples [(9,31–33); but cf. (30)]. These divergent patterns make comparing results difficult because the nature and meaning of modeled latent factors are not consistent.

Figure 1.

Bifactor models fit to measures of cognitive ability (A) and social attitudes (B). Standardized factor loadings and residual variances are shown. (A) N = 213, χ²₁₈ = 24.331, Tucker–Lewis index (TLI) = .988, root mean square error of approximation (RMSEA) [90% confidence interval (CI)] = .041 [.000, .078], unbiased standardized root mean square residual (USRMR) [90% CI] = .029 [.015, .044]. This model shows clear bifactor structure, with good global model fit as well as strong general and group factor loadings. (B) N = 463, χ²₄₃ = 67.984, TLI = .978, RMSEA [90% CI] = .035 [.018, .051], USRMR [90% CI] = .042 [.024, .059]. Despite good global model fit, this model does not show clear bifactor structure. Factor loadings ≥ .40 are in bold. Models were fit using the lavaan package (version 0.6–3) (113) in R (version 3.5.3) (114). A–C, indicators (subtests or item parcels); F, group factors; G, general factor.

The bifactor model’s flexibility can also enable it to show superior global fit compared with alternatives even when the other models were themselves used to simulate data (29,34–42). For example, skewed item distributions and unmodeled cross-loadings or correlated residuals all can lead fit statistics to favor the bifactor model over a correlated factors model (with no general factor) even if the correlated factors model more accurately describes the true structure (42,43). The bifactor model’s flexibility can also result in good model–data fit even when used with very noisy data or nonsense response patterns (29,41). Thus, it is inappropriate to use global fit statistics to evaluate the bifactor model or favor it over alternative models (25,44).⁴ Instead, choosing to apply the bifactor model should be based on the specific research question. In applying the bifactor model, we suggest that researchers adopt the stance that all models are wrong but some are useful (45,46). Below, we describe 3 types of research questions that bifactor models are well suited to address.

USEFUL APPLICATIONS OF THE BIFACTOR MODEL

When a latent variable model is fit to psychopathology data [see (47,48) for discussions on choice of latent variable models versus alternatives such as network models], bifactor models are useful for their ability to separate indicator variance associated with a general factor from variance associated with narrower group factors or specific indicators. This separation of general variance from unique variance can inform several questions.

Presence, Strength, and Content of a General Factor

The most immediate question that bifactor modeling can address is as follows: If a general factor is present, how strong is it and what content characterizes it? An example of this type of question concerns the widely noted covariation between major depressive disorder and anxiety disorders. The tripartite model of anxiety–depression co-occurrence posits a common core of general distress, physiological hyperarousal, and anhedonia (49). Simms et al. (50) used a bifactor model and found that all indicators loaded similarly and strongly onto the general factor (along with group anxiety and depression factors), supporting that depression and anxiety share a common core interpreted as general distress.

Questions concerning strength and content of a general factor should focus on the pattern of factor loadings. What content makes up the general factor? Is it even across indicators or dominated by just a few? Are factor loadings strong or weak? For example, loadings of specific cognitive ability tests show very strong loadings (.50–.70) onto the general cognitive ability factor (21) (cf. Figure 1A), suggesting that it is a major factor that must be explained to understand cognitive test performance. Conversely, loadings of Big Five scale scores onto the general factor of self-reported normal-range personality are weaker and more variable (e.g., mean λ = .27, range = .12–.49) (22), suggesting that this factor is not really a Big One personality factor (51). At the extreme, the results in Figure 1B for a measure of social attitudes suggest no general factor at all. Here, the general factor only reflects indicators of one group factor, with negligible loadings for other indicators. This pattern suggests that a general factor should be rejected entirely for these scales.

Relatedly, bifactor analysis can help to elucidate the content of group factors. Are indicator loadings onto a group factor meaningfully large and in the theoretically expected direction (cf. Figure 1A)? Or, does the group factor mostly reflect idiosyncratic features of only a few indicators (cf. F_C in Figure 1B) or an uninterpretable pattern of positive and negative loadings? The latter patterns would imply that there is not a coherent group factor separable from the content contained in the general factor. [For discussions of best practices in interpreting patterns of factor loadings in bifactor models, see (30,52,53)]. To enhance comparability across samples, researchers should consider drawing on previous bifactor model results to add additional constraints to the model such as constraining the relative magnitudes of indicator loadings onto general versus group factors using informative Bayesian priors based on previous studies or even fixing factor loadings to specific previously estimated values.

Caveat: Indicator Selection Influences General Factor Meaning.

Like any model, results of the bifactor model are influenced by the indicators included. The exact meaning of the general factor can change depending on the measures included in the analyses. Results of psychopathology bifactor analyses have been inconsistent owing to variations in the indicators used. For example, the p factor has variously reflected general distress (7,8,32,54), psychosis (14), unipolar/bipolar depression (14), and self and interpersonal dysfunction (55–57).

If indicators from one subgroup are overrepresented, these may come to dominate the general factor. For example, in psychopathology bifactor analyses, internalizing indicators are often overrepresented (7,9), leading the p factor to primarily reflect these features. Specific uncommonly modeled indicators can also substantially alter patterns of factor loadings if included or excluded. For example, including borderline personality disorder can strengthen all loadings on the p factor owing to this disorder’s moderate correlations with most other DSM diagnoses (56). Including eating disorders or specific phobia can clarify the distinction between the p factor and the internalizing group factor (8).

Indicator level of specificity also influences bifactor model results. Bifactor models in psychopathology research have been fit to individual self-report or clinician-ascertained symptoms (50,58). self- or informant-reported scale sum scores (7,44,59), clinician-ascertained symptom counts (9,10,14), and DSM-based categorical diagnoses (18,32). For example, common criteria or symptom overlap across DSM-based disorders may inflate the strength of the general factor if modeled using symptom counts or diagnoses rather than individual symptoms unless steps are taken to correct for overlap (60). These concerns are also relevant if modeling relationships among scale sum scores because many scales include similar items related to general distress or emotional lability. Similarly, if skipout logic with zero imputation is used during assessment, this may also inflate correlations among diagnosis indicators and, accordingly, the strength of the general factor.

Caveat: Sampling Can Affect General Factor Strength and Content.

The sample used can also affect the strength and pattern of loadings onto the general and group factors—although this issue is not specific to bifactor models. For example, university students are less likely to report multimorbidity than inpatient clinical samples, leading to a weaker general factor. Some clinical instruments, such as the Hamilton Depression Rating Scale (61), may also have low precision in relatively healthy student or community samples, reducing reliability and general factor loadings. In an inpatient clinical or otherwise distressed/help-seeking sample, respondents may have high levels of acute distress and endorse a wide variety of negative symptoms. This will increase loadings onto the general factor and change its interpretation to more reflect current distress than persistent psychopathology (62). Comingling samples with large mean differences (e.g., college students and clinical sample) might produce a spurious latent taxon, especially when the indicators are truly present or absent rather than dimensional (e.g., hallucinations). Features of certain disorders, such as borderline personality disorder or depression, might similarly increase acquiescent endorsement of negative symptoms. This will increase p-factor strength if these disorders are frequently represented in a sample.

These more artifactual forms of a general factor can be addressed by using stronger research designs than cross-sectional self-ratings. For example, modeling a stable general factor in longitudinal data can help to disentangle persistent general psychopathology from current distress (63). Influences of distress-acquiescent responding can be reduced by using multirater or multimethod designs (63). In non-self-rating designs, there is little evidence for a general factor of normal-range personality (23); in informant ratings or across raters, only two weakly correlated higher-order metatraits–α/integration/stability and β/exploration/plasticity–are supported (64). This finding suggests that the general factor of normal-range personality is a self-rating-specific evaluative factor. In contrast, parent ratings and multirater ratings of psychopathology tend to show a similarly strong general p factor as self-ratings [but cf. Constantinou and Fonagy (65)].

Reliability Analysis to Guide Application and Interpretation of Scales

A second useful application of bifactor analyses is reliability analysis to guide interpretations of total versus subtest scores from multifaceted inventories. Using factor loadings from a bifactor analysis, researchers can compute ω_h (omega hierarchical) reliability, which denotes the proportion of variance in a total sum score attributable to the general factor (66–69). If ω_h is large, as in cognitive ability batteries (70), this supports computing a total score and interpreting it as reflecting primarily the general factor. If ω_h is small, as in self-rated normal-range personality (22,51), this indicates that a total score is better understood as a composite of narrower factors. Researchers should consider whether computing a total score or focusing on subtest scores is more meaningful. For example, several psychometric studies of the Anxiety Sensitivity Index (71) have examined the utility of its three subscales. These studies have found that the anxiety general factor accounted for more than 75% of the variance in the items, whereas the subscale group factors generally accounted for negligible amounts of variance (72,73). Even more dramatically, bifactor analyses of the Derogatis Symptom Checklist and Brief Symptom Inventory (74,75) have found that the general factor accounts for ≈95% of the variance in these items (76,77). These results suggest that these measures should primarily be interpreted using total scores. Depending on the sampling and research design (e.g., see above), we may even question whether the narrow constructs captured by the group factors are empirically distinguishable constructs.

Relatedly, for each subscale, researchers can compute ω_s (omega subscale), the reliability of the subscale after removing variance associated with the general factor. If ω_h is large but ω_s is small, the remaining variance in the specific factors might not be meaningfully interpretable, and any individual differences cannot be reliably captured in narrower facets separate from the general factor. For example, the Wechsler Adult Intelligence Scale subscales have weak reliability after controlling for general cognitive ability (ω_s = .13–.47) (70). Similarly, on average across psychopathology measures, only ~37% of the variance (43% of the reliable variance [general + group factors, excluding item-specific variance and error]) in subscales was attributable to the narrow group factor constructs (65). These results indicate that many psychopathology subscales should primarily be interpreted as reflecting the general factor; interpreting differential subscale profiles may be highly unreliable [see also Bulut et al. (78)].

Relations of General and Group Factors With External Variables

A third useful application of bifactor analysis is examining differential relations of general and narrow group factors with external variables such as correlations with putative antecedents, criterion variables, and changes in response to treatment. For example, in parent-rated child psychopathology, low executive functioning and family relationship risk factors were related to the general p factor but not to fear, distress, or externalizing group factors (9). In contrast, both the p factor and externalizing group factor prospectively predicted teacher-reported academic performance, behavioral problems, grade retention, and special education status (7). In adults, both the p factor and internalizing group factor uniquely predicted suicidality and nonsuicidal self-injury (11).

Because observed total scores or subscale scores may reflect a mixture of general and group factor variance, observed score correlations might reflect the influence of the general factor, the group factor, or both. Observed correlations may be inflated or attenuated as estimates of relations between the broad/narrow psychopathology constructs and other variables (79). For example, the perfectionism facet of conscientiousness was negatively related to university student physical and mental health after controlling for general conscientiousness; this relationship was obscured when both sources of variance were combined in the observed subscale score (80). By separating the predictive power of broad and narrow factors, bifactor modeling can provide a clearer picture of the nomological network of psychopathology.

Elaborating the stability and nomological network of general and group factors is also useful for discerning whether factors reflect substantive constructs or artifacts (81). For example, studies of childhood and adolescent psychopathology have found that both the p factor and group factors are stable and similar in factor strength over time, suggesting that the p factor reflects more than transient current distress (10,63,82,83). Similarly, if a general factor shows unique relationships with important antecedents, biological substrates, or outcomes, this supports its interpretation as a meaningful construct; the p factor shows genetic correlations with neuroticism (8) and reduced gray matter volume in prefrontal areas (15).

Practically, bifactor analyses aimed at assessing the reliability and external validity of the general and group factors can help to guide the level of focus in assessment and treatment planning (13,84,85). Can we predict treatment outcomes based on the general factor alone, or do we need an assessment of group factors? Does matching patients to treatment based on narrow group factors produce better outcomes than matching them on the general factor alone? Do broad-based treatments such as dialectical behavior therapy target only general psychopathology or do they also affect narrow personality disorder aspects as well?

COMPARISON WITH ALTERNATIVE MODELS

We describe several useful applications of bifactor models. This is not to suggest that they are a panacea or appropriate for all research questions. Below, we compare the bifactor model with several common alternatives and consider when these alternatives may be more useful.

Higher-Order Model

The most similar model to the bifactor is the higher-order factor model, where the items load onto their respective group factors, which then load onto the general factor. Researchers sometimes prefer to use a higher-order model if their theoretical model conceptualizes the narrow factors as components of the general factor rather than as distinct entities. For example, one common liabilities model posits that individuals first develop a general liability for psychopathology, which then differentiates into more specific pathologies in response to environmental experiences (86).

However, even though the higher-order model may on its face appear more appropriate for such conceptualizations, mathematically it is extremely similar to the bifactor model. Indeed, the higher-order model is simply a somewhat more constrained version of the bifactor model (19,38,43). Given their mathematical similarity, the two models make very similar predictions about indicator covariances and typically yield comparable results. Hypotheses that can be tested using the higher-order model can often be more easily evaluated using a constrained form of bifactor model. The Schmid–Leiman transformation represents a higher-order model in the format of a constrained bifactor model, making the relative strength of the different latent variables clearer (30). Thus, the bifactor model format can be useful, even when a higher-order model is theoretically preferred. The advantage of the bifactor model format for describing a general factor over the higher-order model is that it directly teases apart the unique contributions to the indicators of the general and group factors (21). In the bifactor model, unique aspects of the group factors are represented as distinct variables from the general factor. In the higher-order model, unique aspects of the group factors are represented as the residuals (disturbances) of the first-order latent variables, which together with variance from the general factor jointly influence the indicators. This arrangement makes it more difficult to use reliability analysis to assess the degree to which scores primarily reflect the general or group factors. Likewise, it becomes more difficult to assess the differential external validity of the general or group factors because the unique relations of the group factors must be drawn from the residuals (disturbances) rather than from the latent group factors themselves. In general, we argue that the bifactor model format makes evaluating theoretical hypotheses about general and group factors clearer and more interpretable, even if the conceptual model posits that the general factor directly contributes to the narrow group factors.

Correlated Traits Model

The correlated traits model includes correlated group factors but no overarching general factor—for example, a model specifying internalizing and externalizing factors with no general p factor (2). If correlations among first-order factors are small, the correlated traits model can provide a simpler and easier-to-interpret description of the structure of a measure. However, if correlations among factors are large, the bifactor model can be more useful, particularly for discerning whether factors’ common or unique variance is the primary source of measures’ predictive power [cf. the challenges of estimating relative importance for correlated predictors (87–89)].

Network Models

In a more recently developed psychometric approach, network models posit that indicators (e.g., symptoms) directly influence each other without any unobserved latent variables (90). These models are used to test dynamic mutualism and other network theories of psychopathology, which hypothesize that covariance among symptoms does not reflect a common latent variable, but rather reflects the effects of individual symptoms reinforcing each other (e.g., a cascading downward spiral) (91,92); the p factor is an effect of symptom covariance, not a cause (93). Although network versus common cause theories of psychopathology are importantly distinct, it is important to remember that the mathematics of common factor and network models are highly similar (47,48,94). Every network model can be expressed as an equivalent factor model and vice versa. Cross-sectional network models have the same limitations as factor models in terms of interpreting the meaning of symptom co-occurrence; network models themselves cannot provide insight into processes or development any more than common factor analysis can. Where network models can be useful compared with the bifactor model (or other common factor models) is if the research question concerns the nomological network of individual symptoms. For example, Fried et al. (95) used network analysis to examine unique relationships of individual depression symptoms with inflammation biomarkers [cf. (96,97)]. Because indicator-specific variance is typically regarded as error in bifactor and other common factor models, such symptom-specific relationships are easier to examine with network models. As with factor models, exploring processes with network models requires longitudinal data.

THE BIFACTOR MODEL AND BIOLOGICAL SUBSTRATES OF PSYCHOPATHOLOGY

A growing area of research examines biological substrates of psychological constructs such as neurobiological and genetic correlates of individual differences in personality, cognition, or psychopathology (98–101). For example, several studies have examined or proposed correlations of psychopathology general and group factors with genetic single nucleotide polymorphisms or neurobiological variables (e.g., gray matter volume, volume or activation of amygdala/prefrontal cortex circuits, hypothalamic-pituitary-adrenal axis, hippocampus, ventral striatum) (15,102–105). These questions may benefit from bifactor models’ utility for examining external variable relations. For example, if individual symptoms or disorders show stronger or more coherent associations with genetic, neurobiological, or biomarker variables than with latent factors, these associations might suggest that the latent factors do not reflect specific biological liabilities, but rather reflect measurement artifacts or common socioenvironmental factors. Conversely, if many symptoms are associated with common biological substrates, and extracted latent variables show even stronger relations with these substrates, this would support an interpretation of common biological liabilities for diverse psychopathology indicators. In this way, neurobiological or genetic data can serve as a constraint that can help to discriminate between alternative models that psychological data alone cannot differentiate [cf. Shanmugan et al. (106)].

Careful attention must be paid, however, to measurement fidelity of biological variables. For example, functional neuroimaging is notoriously plagued by artifacts and instability of extracted intrinsic connectivity networks (107). Similarly, factor analyses of allele co-occurrence have dubious interpretation. Because of humans’ small effective population size compared with other species, natural selection effects in humans are frequently smaller than genetic drift effects [cf. (108)]. Accordingly, co-occurrence of specific alleles in a population does not necessarily reflect a consequence of selection or otherwise indicate genetic substrates for a specific complex phenotype (such as psychopathology). Remedies such as removing the first principal component cannot completely remove these effects and may lose relevant genetic information. Likewise, fitting latent variable models—including bifactor models—to polygenic risk scores might suggest shared genetic correlates, but this does not imply that the p factor is a unitary biological entity or alone reveal biological mechanisms; genes code for proteins, not psychopathology (109). These issues can make it challenging to interpret studies that fit factor models to both psychopathology measures and genetic or neurobiological variables (110–113). In modeling neurobiological or genetic data, researchers must first ensure that their models of neural or genetic structure are consistent with underlying molecular processes.

Ultimately, an important point to remember is that studying biological substrates does not make phenomena being studied more real or valid. Psychopathology is phenomenologically defined and diagnosed at the level of affective, cognitive, and behavioral symptoms; biological substrates may be important, but they are not necessary to understand assessment, diagnosis, development, or treatment of psychological disorders (114,115). Psychological and biological explanations complement each other; they reflect different levels of specificity to explain the same phenomena, but there is no one-to-one mapping between psychological constructs and biological factors. Instead, psychological constructs emerge from interactions among dynamic processes over development (101).

MODELING CANNOT FIX INADEQUATE RESEARCH DESIGN

To close, we reiterate that statistical modeling cannot make fundamental limitations of data disappear. The questions that data can address are a function of the research design, not the model chosen to analyze them. Cross-sectional relationships among indicators cannot speak to developmental processes regardless of the type of model (bifactor or network) or indicator (behavioral symptoms or biological variables) used. The appropriate level of analysis for psychopathology (e.g., symptoms, disorders, spectra, p factor) is a question best addressed empirically—for example, by comparing genetic and neurobiological correlates of individual symptoms versus broader factors (27,105)—or pragmatically—for example, do components from one model yield stronger predictions of clinically relevant outcomes than components from another model? Paired with rigorous research designs that can rule out methodological confounds, such as longitudinal and multi-informant designs, bifactor modeling can be a useful tool for investigating the nature of psychopathology constructs (broad and narrow) and their mechanisms, development, and responsiveness to treatment.