Abstract
BACKGROUND:
It has not been established that suicide risk with mixed symptoms is any greater than the depressive component or if there is synergy between depressive and manic symptoms in conveying suicide risk.
METHODS:
The National Network of Depression Centers Mood Outcomes Program collected data from measurement-based care for 17,179 visits from 6,105 unique individuals with clinically diagnosed mood disorders (998 bipolar disorder, 5,117 major depression). The Patient Health Questionaire-8 (PHQ-8) captured depressive symptoms and the Altman Self-Rating Mania scale (ASRM) measured hypomanic/manic symptoms. Generalized linear mixed models assessed associations between depressive symptoms, manic symptoms, and their interaction (to test for synergistic effects of mixed symptoms) on the primary outcome of suicidal ideation or behavior (secondarily suicidal behavior only) from the Columbia-Suicide Severity Rating Scale (C-SSRS). Moderation was assessed.
RESULTS:
PHQ-8 scores were strongly associated with suicide-related outcomes across diagnoses. ASRM scores showed no association with suicidal ideation/behavior in bipolar disorder and an inverse association in major depression. There was no evidence of synergy between depressive and manic symptoms. There was no moderation by sex, race, or mood disorder polarity. Those over 55 years of age showed a protective effect of manic symptoms, which was lost when depressive symptoms were also present (mixed symptoms).
DISCUSSION:
Mixed depressive and manic symptoms convey no excess risk of suicidal ideation or behavior beyond the risk conveyed by the depressive symptoms alone. Depressive symptoms are strongly linked to suicidal ideation and suicidal behavior and represent an important and potentially modifiable risk factor for suicide.
Keywords: Bipolar disorder, Major Depressive Disorder, Mania, Depression, Mixed Symptoms, Suicidal ideation, Attempted Suicide
INTRODUCTION
The course of illness in mood disorders is frequently complicated by the presence of significant depressive symptoms during manic or hypomanic episodes and significant hypomanic or manic symptoms during episodes of major depression. Long-term prospective studies suggest these mixed states may be experienced by up to 38% of individuals with bipolar disorder [1, 2]. Mixed states have long been considered to be a high risk period for suicide; however, most of the evidence on this topic comes from cross-sectional associations between those who have had mixed states at some point and prior suicide attempts [3]. One recent systematic review of 16 cross-sectional studies of individuals with bipolar disorder found increased suicidal behavior in mixed state compared to mania/hypomania, and overall no difference in suicidal behavior in mixed state than in depression [4]. While there have been some limited prospective studies, the International Society of Bipolar Disorders Task Force on Suicide concluded “there was insufficient data to specifically analyze mixed episodes” as a risk factor due to low rates of mixed episodes in a limited number of studies [5]. In a sample of 120 individuals with bipolar I followed for 10 years, Gonzalez-Pinto et al. found that those with a history of mixed states had more mood episodes (~10 vs. ~5) and more suicide attempts (29.5% vs. 13%). The difference in suicide attempts was significant in only univariate models [1]. In a two year follow-up of 91 individuals with bipolar I, Johnson et al. performed a cross-sectional analysis using the month with the highest level of suicidality from a modified Hamilton Rating Scale for Depression and found depression, hopelessness, and, less strongly, mixed states were associated with suicidal ideation and attempts [6]. Two analyses that fully utilized the same prospective data from the Jorvi Bipolar Study indeed confirmed higher risk of suicide attempts with mixed states [7, 8]. In both of these analyses, risk estimates were quantitatively higher for categorically defined mixed states than pure depressive states although neither analysis assessed whether there was any risk beyond that attributable to the underlying depressive symptoms.
Understanding the interplay between depressive and manic symptoms in mixed states has important implications. If it is the case that depressive symptoms and manic symptoms exert only additive effects, then there is little reason to think of them as anything more than independent risk factors for suicide. As such, depressive and manic symptoms, whether separated in time or co-occurring, could be considered separately in risk assessment. If, however, depressive and manic symptoms interact to convey a greater-than-additive effect, this would be suggestive of synergy and would be of great interest. Kenneth Rothman classically described synergy as occurring when “the risk attributable to a combined exposure exceeds the sum of the risks attributable to each exposure separately [9].” Recent analyses of longitudinal data have found that mixed states are not associated with an elevated risk for suicide outcomes beyond that attributable to the component depressive symptoms [2, 10]. Persons et al. addressed this question in a cohort of 429 individuals with bipolar disorder who were followed for a mean of 18 years through the Collaborative Depression Study (CDS). In the CDS, individuals with bipolar mixed states experienced a greater burden of depressive symptoms, and the risk of suicidal behavior in mixed states was not increased beyond the risk attributable to the depressive component [2]. Fiedorowicz et al. replicated these findings in a cohort of 290 patients with bipolar disorder with more limited prospective follow-up in the National Network of Depression Centers (NNDC) Clinical Care Registry, a real-world sample of patients with mood disorders receiving treatment at sites across the United States. This study similarly found no evidence of increased suicide risk in mixed states beyond that explainable by the depressive symptoms [10]. Ostacher et al. similarly reported no association between manic (and therefore mixed symptoms) and suicidal ideation or behavior in the NIMH LiTMUS trial [11].
Several studies have found that mixed states are associated with more frequent or longer episodes of mood symptoms [3]. Gonzalez-Pinto et al. followed 120 participants with bipolar I disorder for a decade and found that those who experienced mixed states had roughly twice as many mood episodes over follow-up [1]. Several cross-sectional studies have also reported those with mixed symptoms had more prior or longer episodes or a more chronic course for bipolar disorder [12–15] and major depression [12]. The presence of mixed states is also associated with an earlier age of onset for both bipolar disorder [1] and major depression [12, 16]. An early age of onset is associated with a more chronic course of illness in mood disorders [17]. The aggregate of evidence therefore supports that the cross-sectional association between those who have had mixed states and prior suicide attempts is confounded by a more persistent course of depressive symptoms.
In the CDS cohort, we found some evidence of moderation by bipolar subtype such that weeks in which manic and depressed states occurred were associated with a greater suicide risk in bipolar I disorder but not in bipolar II disorder [2]. Moderation of the effects of mood symptoms on suicidality by bipolar subtype was, however, not replicated in the NNDC analysis, but there was evidence of moderation by sex such that women, but not men experienced an increased suicide risk during manic states [9]. While these previous analyses suggest that mood state related suicide risk may not be homogeneous across individuals with bipolar disorder, they were underpowered to test for moderation. There is a need to conduct adequately powered analyses on these putative moderators and to examine other factors that alter the effects of mood state on suicidality.
In this study, we sought to use a large, real-world sample, the NNDC Mood Outcomes Program (MOP), to assess whether mixed states elevate suicide risk beyond any risk attributable to the manic or depressive components of mood symptoms alone. These analyses will be extended to unipolar major depressive disorder and are better powered to assess pre-specified moderators: age, sex, race, polarity (bipolar/unipolar), and bipolar subtype (assessed only within the bipolar disorder subsample).
METHODS
This study utilized real-world data, defined here as data obtained from normal clinical practice [18], from the NNDC Mood Outcomes Program (MOP). The National Network of Depression Centers is a comprehensive, multidisciplinary collaboration of leading mood disorder experts across 26 academic medical centers of excellence, dedicated to reducing the burdens of morbidity and mortality in mood disorders. The Mood Outcomes Program is a clinical program created by the NNDC to collect standardized patient-reported outcomes at each clinical visit via a patient portal. This program simultaneously improves patient care by allowing clinicians to view patient data across visits in real time and increases the fund of knowledge by creating a large-scale longitudinal database. Each site meets their institutional review board requirements for the provision of clinical data. Because this data is provided as part of standard clinical care at each site, patient consent is not obtained. The data is centralized in collaboration with the Altarum Institute, a non-profit health informatics organization [19].
Mood was captured as a component of MOP using the Patient Health Questionnaire-9 (PHQ-9) for depressive symptoms and the Altman Self-Rating Mania (ASRM) scale for manic/hypomanic symptoms [21]. To avoid overlap with outcome, only the first 8 items of the PHQ-9 were used, also known as the PHQ-8 [20]. The PHQ-8 was treated as a continuous measure in primary analyses. For the categorical determination of clinically significant depressive symptoms, we used a PHQ-8 threshold of ≥10, which has been shown to have high sensitivity and specificity in various clinical and community samples [21, 22]. Manic/hypomanic symptoms were captured using continuous ASRM score in primary analyses and secondarily categorically using a cutpoint of >5, which has high sensitivity but quite variable specificity (33–87%) for mania, depending on the severity of symptoms [23, 24]. For our purposes, and in keeping with our previous publications, mixed symptoms were defined as concurrent depressive and manic/hypomanic symptoms [2, 10].
This analysis was conducted for MOP data from June 2015 to December 2019, at which time the ASRM was removed from the common assessment package due to its psychometric limitations. MOP data during this date range was collected from measurement-based care at the following ten member centers: Johns Hopkins University, Mayo Clinic, Medical University of South Carolina, Penn State University, University of Illinois at Chicago, University of Colorado Anschutz Medical Campus, University of Iowa, University of Louisville, University of Michigan, and University of Texas at Houston. For privacy reasons, sites are not named in the MOP. The MOP patient database for this date range includes more than 10,000 individuals with a mood disorder, within which we restricted to those with bipolar disorder or major depressive disorder. We included data from all patients with a diagnosis of bipolar I disorder (ICD10 codes F31.1-F31.78; F30.1-F30.9), bipolar II disorder (ICD10 code F31.81), bipolar disorder not otherwise specified (ICD10 codes F31.8; F31.89; F31.9), and major depressive disorder (ICD10 codes F06.3-F06.4; F32.0-F34.0; F34.1-F40.0). Per NNDC data protocol, the mood disorder diagnosis used across all analyses is the most recent chart diagnosis from the clinicians providing care at these NNDC centers.
Suicidal ideation and behavior was captured using the Columbia-Suicide Severity Rating Scale (C-SSRS), with suicidal ideation defined as any thoughts of killing oneself or wishing oneself dead and suicidal behavior defined as any suicide attempt or preparations for an attempt since last visit. The C-SSRS demonstrates 100% sensitivity and 96% specificity compared to evaluation by a panel of experts [25].
Our a priori power calculations conservatively assumed a dataset of 9,000 patients with only one visit per participant and a similar proportion with bipolar disorder, mixed states, and suicide outcomes as was observed in our prior analysis of the NNDC Clinical Care Registry, where 31% of visits included suicidal ideation or behaviors [10]. Using PS Power and Sample Size Calculations Version 3.0, under the assumptions of an uncorrected chi-square case to minimize assumptions, with 24% of visits being classified as mixed and using the aforementioned estimates from the Clinical Care Registry, we would have 84% power (at alpha=0.05) to detect a 10% difference in suicidal ideation (centered around estimate of 50%) between categorical mixed states and depression and a 6% difference in risk of suicidal behavior (centered around estimate of 9%). With any correlation between repeated observations in the actual analysis, these power calculations conservatively underestimate power.
Analyses were conducted using SAS 9.4 (SAS Institute, Inc., Cary, NC). Generalized linear mixed models (GLMM - binomial distribution, logit link (canonical)) with a random intercept term were used to account for the violation of the assumption of independent observations with repeat visits (the unit of analysis) for some patients. Linear mixed models did not include any variable for time to accommodate repeated measures data with highly irregular follow-up periods. Mixed symptoms were modeled as a depressive × manic symptom interaction term in primary analyses; for the purpose of describing visits, patients were considered to have had a mixed state if they met both the thresholds of PHQ-8 ≥10 and ASRM >5. In primary models, depressive symptoms were modeled as the continuous PHQ-8 score and manic symptoms as the continuous ASRM score with suicidal ideation or behavior as the dependent variable without centering of variables to assess moderation [26]. The Baron and Kenny approach was selected over the MacArthur approach as the data did not allow ascertainment of temporality [27]. Continuous variables were used to model the interaction to reduce the likelihood of biased results when grouping categorical variables which can falsely imply synergy [28]. A significant interaction would suggest significantly greater (or lower) suicide risk in mixed states beyond what can be additively attributed to the component depressive or manic symptoms. We secondarily assessed the narrower but more clinically relevant outcome of suicidal behavior only. Models also addressed the potential for moderation by sex, race (White/non-White due to observed frequencies), age (<55 or ≥55 years) bipolarity, bipolar subtype (within sub-sample with bipolar disorder). The arbitrary age threshold was determined a priori over more popular thresholds of 60 or 65 so as to have better balance between groups while still using a threshold that has been used elsewhere [29–31]. These putative moderators could all be assumed to precede the mood symptom target variable. Moderation was assessed for depressive, manic, and mixed symptoms as continuous variables adding an interaction term for each of the aforementioned putative moderators as indicator variables. Moderation of mixed symptoms subsequently involved a three-way interaction term (PHQ-8×ASRM×moderator). When present, moderation was illustrated with stratified models. Post hoc logistic regression models used a categorical threshold of PHQ-8 ≥ 10 for depressive symptoms and ASRM > 5 for manic/hypomanic symptoms with mixed symptoms modeled as their concurrence.
RESULTS
The sociodemographic and clinical characteristics of the 6,105 patients in the sample are reported in Table 1. Relative to those with a diagnosis of bipolar disorder, patients with major depressive disorder were also more likely to be diagnosed with a co-occurring anxiety disorder and were more likely to be female. They also tended to be younger. The differences between those were categorically classified as experiencing mixed symptoms, defined as a PHQ-8≥10 and an ASRM>5 at any visit, are reported in Supplemental Tables 1 for bipolar disorder and Supplemental Table 2 for major depressive disorder. The overall mean PHQ-8 score was 10.3 (SD 6.7) and the overall mean ASRM score was 2.7 (SD 3.0). Relative to patients with bipolar disorder, those with major depressive disorder had a higher mean PHQ-8 score (10.4 vs. 9.9, p<0.001) and a lower mean ASRM score (2.6 vs. 3.2, p<0.001). Those who categorically reported mixed symptoms at any visit reported significantly higher mean PHQ-8 and ASRM scores for both the bipolar disorder subsample (PHQ-8: 13.1 vs. 8.9, p<0.001; ASRM: 5.7 vs. 2.5, p< 0.001) and the major depressive disorder subsample (PHQ-8: 13.0 vs. 9.8, p<0.001; ASRM: 4.6 vs. 2.2, p<0.001).
TABLE 1. Sociodemographic and Clinical Characteristics of Sample.
This table highlights key demographic variables and co-occurring mental disorders in this sample stratified by diagnosis of a bipolar disorder or major depressive disorder. Significant differences by diagnostic grouping are highlighted in bold.
| Total (N=6105) | Bipolar Disorder (n=988) | Major Depression (n=5117) | ||
|---|---|---|---|---|
| n(%) | n(%) | n(%) | χ2 p-value | |
| Age Category | <0.001 | |||
| Less than 30 years old | 1889 (30.9%) | 257 (26.0%) | 1632 (31.9%) | |
| 30–39 years old | 1362 (22.3%) | 225 (22.8%) | 1137 (22.2%) | |
| 40–49 years old | 984 (16.1%) | 158 (16.0%) | 826 (16.2%) | |
| 50–59 years old | 963 (15.8%) | 190 (19.2%) | 773 (15.1%) | |
| 60 years old or older | 907 (14.9% | 158 (16.0%) | 749 (14.6%) | |
| Sex | (missing = 42) | (missing = 15) | (missing = 27) | 0.03 |
| Female | 3985 (65.7%) | 605 (62.2%) | 3380 (66.4%) | |
| Male | 2075 (34.2%) | 367 (37.7%) | 1708 (33.6%) | |
| Other | 3 (0.1%) | 1 (0.1%) | 2 (0.04%) | |
| Race | (missing = 40) | (missing = 7) | (missing = 33) | 0.15 |
| American India /Alaskan Native | 35 (0.6%) | 8 (0.8%) | 27 (0.5%) | |
| Asian/Asian American | 126 (2.1%) | 18 (1.8%) | 108 (2.1%) | |
| Black or African American | 451 (7.4%) | 68 (6.9%) | 383 (7.5%) | |
| Native Hawaiian/Pacific Islander | 7 (0.1%) | 3 (0.3%) | 4 (0.1%) | |
| White | 5160 (85.1%) | 846 (86.3%) | 4314 (84.9%) | |
| Other | 230 (3.8%) | 34 (3.5%) | 196 (3.9%) | |
| Unknown | 56 (0.9%) | 4 (0.4%) | 52 (1.0%) | |
| Married | (missing=230) | (missing = 29) | (missing = 201) | <0.001 |
| Single | 2583 (44.0%) | 354 (36.9%) | 2229 (45.3%) | |
| Committed Relationship | 418 (7.1%) | 81 (8.4%) | 337 (6.9%) | |
| Married | 2058 (35.0%) | 366 (38.2%) | 1692 (34.4%) | |
| Divorced | 572 (9.7%) | 110 (11.5%) | 462 (9.4%) | |
| Separated | 132 (2.2%) | 28 (2.9%) | 104 (2.1%) | |
| Widowed | 112 (1.9%) | 20 (2.1%) | 92 (1.9%) | |
| Comorbidities | ||||
| Generalized Anxiety Disorder | 1482 (24.3%) | 113 (11.4%) | 1369 (26.7%) | <0.001 |
| Obsessive-Compulsive Disorder | 127 (2.1%) | 12 (1.2%) | 115 (2.2%) | 0.04 |
| Panic Disorder | 214 (3.5%) | 19 (1.9%) | 195 (3.8%) | 0.003 |
| Phobic Disorder | 233 (3.8%) | 24 (2.4%) | 209 (4.1%) | 0.01 |
| Reaction to Severe Stress/Adjustment Disorders | 561 (9.2%) | 62 (6.3%) | 499 (9.7%) | <0.001 |
| Other Anxiety Disorder | 766 (12.5%) | 77 (7.8%) | 689 (13.5%) | <0.001 |
| Substance Abuse Disorder | 475 (7.8%) | 78 (7.9%) | 397 (7.7%) | 0.88 |
Analyses were performed for 17,179 visits in this sample (an average of 2.8 visits/patient with a range of 1 to 36). By categorically defined and mutually exclusive mood states, there were a total of 7,535 visits with depressive symptoms, 1,710 visits with manic/hypomanic symptoms, 1,028 visits with both (mixed symptoms), and 6,906 visits with neither clinically relevant depressive nor manic/hypomanic symptoms. Among the subsample with bipolar disorder, there were 816 visits with depressive symptoms, 259 visits with manic/hypomanic symptoms, 212 visits with mixed symptoms, and 891 visits with no clinically relevant depressive or manic/hypomanic symptoms. At a total of 5,839 visits (34.0% of total visits), suicidal ideation or behavior was reported with 1,068 of those visits reporting suicidal behavior (6.2% of total visits). The subsample with bipolar disorder, included 988 patients with 2,178 visits, at which 698 (32.0%) reported suicidal ideation or behavior and 129 (5.9%) reported suicidal behavior. The percentage of visits in which our primary outcome (suicidal ideation or behavior) was observed grouped by whether the patient categorically reported no clinically significant depressive or manic symptoms, depressive symptoms, manic symptoms, or both (PHQ-8≥10 and ASRM>5) are illustrated in Figure 1a with Figure 1b restricted to the secondary outcome of suicidal behavior. Supplemental Figures 1a–d present the same information stratified by diagnosis of bipolar disorder or major depressive disorder.
FIGURE 1: Frequency of Visits with Primary Outcome of Suicidal Ideation or Behavior by Mood State.
This figure illustrates the overall proportion of visits by mood state for which the primary outcome of suicidal ideation or behavior was reported within the combined patient sample. Visits with “no depression or mania” were determined by a self-reported PHQ-8<10 and an ASRM≤5. Visits with “depression” were defined by a PHQ-8≥10 and an ASRM≤5 and mania by a PHQ-8<10 and an ASRM>5. “Mixed” visits were those in which the PHQ-8≥10 and ASRM>5. This figure is provided for descriptive purposes as primary analyses utilized these mood measures as continuous variables.
In our primary GLMMs with suicidal ideation or behavior as the dependent variable, the PHQ-8×ASRM interaction terms (as continuous variables) were not significant (p’s 0.38–0.95). Thus, there was no evidence of more- or less-than-additive effect of the symptom categories. Depressive symptoms, as measured by the PHQ-8, were strongly related to suicidal ideation/behavior in all models. Manic/hypomanic symptoms, as measured by the ASRM, were not related to suicidality in the subsample with bipolar disorder and were inversely associated with the outcome in the subsample with major depression in models adjusting for sex, age, race, marital status, and the presence of co-occurring substance use or anxiety disorders (β= −0.42, 95% CI −0.067 to −0.016, p=0.001). The results of these models are detailed in Table 2. Similar results were observed in secondary GLMMs on the dependent variable of suicidal behavior, with the exception of an association of mania/hypomania with increased risk of suicidal behavior in the subsample with major depression (β 0.05, 95% CI 0.02–0.09, p=0.004). The results of these models are detailed in Table 3.
Table 2.
Generalized Linear Mixed Models: Relationship of Mood State to Suicidal Ideation or Behavior
| Unadjusted | ||||||||
|---|---|---|---|---|---|---|---|---|
| Bipolar | Major Depression | |||||||
| β | SE | 95% CI | p-value | β | SE | 95% CI | p-value | |
| Mixed symptoms2 | −0.0002 | −0.06 | −0.007–0.007 | 0.95 | 0.002 | 0.002 | −0.002–0.005 | 0.38 |
| Depressive Symptoms | 0.23 | 0.017 | 0.19–0.26 | <0.001 | 0.25 | 0.0073 | 0.24–0.27 | <0.001 |
| Manic Symptoms | 0.017 | 0.023 | −0.028–0.062 | 0.46 | −0.024 | 0.013 | −0.049–0.0006 | 0.056 |
| Multivariable-adjusted1 | ||||||||
| Bipolar | Major Depression | |||||||
| β | SE | 95% CI | p-value | β | SE | 95% CI | p-value | |
| Mixed symptoms2 | 0.0008 | 0.004 | −0.007–0.008 | 0.84 | 0.002 | 0.002 | −0.002–0.006 | 0.39 |
| Depressive Symptoms | 0.23 | 0.017 | 0.20–0.26 | <0.001 | 0.25 | 0.0076 | 0.24–0.27 | <0.001 |
| Manic Symptoms | 0.015 | 0.024 | −0.031–0.062 | 0.52 | −0.042 | 0.013 | −0.067- −0.016 | 0.001 |
Adjusted for sex, age, race, marital status, and presence of comorbid substance use or anxiety disorders
Modeled as the interaction of depressive symptoms (PHQ-8) and manic symptoms (ASRM)
Table 3.
Generalized Linear Mixed Model Analyses: Effects of Continuous Mood State on Suicidal Behavior
| Unadjusted | ||||||||
|---|---|---|---|---|---|---|---|---|
| Bipolar | Major Depression | |||||||
| β | SE | 95% CI | p-value | β | SE | 95% CI | p-value | |
| Mixed symptoms2 | 0.004 | 0.005 | −0.006–0.013 | 0.44 | 0.003 | 0.003 | −0.002–0.007 | 0.31 |
| Depressive Symptoms | 0.12 | 0.020 | 0.082–0.16 | <0.001 | 0.11 | 0.0088 | 0.098–0.13 | <0.001 |
| Manic Symptoms | 0.055 | 0.032 | −0.0081–0.12 | 0.087 | 0.069 | 0.018 | 0.034–0.10 | 0.001 |
| Multivariable-adjusted1 | ||||||||
| Bipolar | Major Depression | |||||||
| β | SE | 95% CI | p-value | β | SE | 95% CI | p-value | |
| Mixed symptoms2 | 0.005 | 0.005 | −0.0054–0.015 | 0.35 | 0.002 | 0.003 | −0.030–0.098 | 0.51 |
| Depressive Symptoms | 0.12 | 0.021 | 0.076–0.16 | <0.001 | 0.11 | 0.009 | 0.092–0.13 | <0.001 |
| Manic Symptoms | 0.023 | 0.035 | −0.045–0.092 | 0.51 | 0.051 | 0.018 | 0.017–0.086 | 0.004 |
Adjusted for sex, age, race, marital status, and presence of comorbid substance use or anxiety disorders
Modeled as the interaction of depressive symptoms (PHQ-8) and manic symptoms (ASRM)
Post hoc analyses looking at mood state categorically among patients with bipolar disorder demonstrated a higher odds of suicidal ideation or behavior with depressive symptoms (OR 7.93, 95% CI 5.67–11.09, p<0.001), but not manic/hypomanic symptoms (OR 1.04, 95% CI 0.72–1.49, p=0.85) or mixed symptoms (OR 0.94, 95% CI 0.46–1.90, p=0.86). Similar relationships were noted among patients with major depression for depressive symptoms (OR 9.71, 95% CI 8.35–11.28, p<0.001) and mixed symptoms (OR 1.03, 95% CI 0.73–1.47, p=0.86), with a protective effect noted for manic/hypomanic symptoms (OR 0.60, 95% CI 0.50–0.72, p<0.001).
Moderation analyses revealed no effects of sex, race, mood disorder polarity, or bipolar subtype (evaluated only within sub-sample with bipolar disorder) on the relationship between depressive symptoms, manic/hypomanic symptoms, or their interaction (mixed symptoms) on the primary outcome of suicidal ideation or behavior. There was significant moderation by age (age <55 years vs. age ≥ 55 years) for depressive symptoms (p<0.001), manic/hypomanic symptoms (p=0.001) and mixed symptoms (p=0.02). In subsequent stratification by age category, among those younger than 55 years old, depressive symptoms (β 0.26, SE 0.010, 95% CI 0.25–0.28, p<0.001), but not manic/hypomanic symptoms (β −0.006, SE 0.024, 95% CI −0.053–0.041, p=0.80) or the interaction of depressed and manic symptoms (β=−0.007, SE 0.002, 95% CI −0.004–0.003, p=0.74) were associated with suicidal ideation or behavior. Among those 55 years of age or older, however, depressive symptoms (β 0.20, SE 0.016, 95% CI 0.17–0.23, p<0.001) were associated with higher risk while manic/hypomanic symptoms with lower risk (β −0.17, SE 0.048, 95% CI −0.26- −0.076, p<0.001) of suicidal ideation or behavior, while the depression by manic symptom interaction (β 0.009, SE 0.004, 95% CI 0.001–0.017, p=0.03) suggested that protective effect was dampened by the co-occurrence of depressive symptoms. The correlations between the PHQ-8 and ASRM overall and stratified by the age categories for those with bipolar disorder and major depression are shown in Supplemental Table 3. These correlations do not appear to differ by age although the negative correlation between the ASRM and PHQ-8 in major depression (r= −0.15, p<0.001) is not seen in bipolar disorder (r=0.015, p=0.45)
DISCUSSION
In this large, real-world, multi-center, longitudinal, clinical sample, we confirmed findings that mixed symptoms (manic symptoms present during depression and vice versa) do not convey any increased risk for suicidal ideation or behavior beyond that attributable to depressive symptoms, which appear to be the primary driver of risk in mood disorders. Individuals who categorically reported mixed symptoms at any visit had an overall greater burden of depressive and manic symptoms, as evidenced by significantly higher mean PHQ-8 and ASRM scores across visits, relative to those without mixed symptoms. Depressive symptoms, as measured by the PHQ-8, were strongly associated with self-reported suicide outcome measures from the C-SSRS. Interestingly, scores on the ASRM were actually associated with lower risk in those with major depression, but not bipolar disorder. In this larger sample, for which we had adequate power to assess moderation, we failed to replicate findings on moderation by polarity or sex from prior moderation analyses in smaller samples. We identified moderation by age, such that those over 55 years of age demonstrate a protective effect of manic symptoms, and this deserved further investigation. This effect may be explained by older individuals having more lifetime experience with managing manic symptoms, and as such being less likely to be moved to suicidal thinking with them. The significant PHQ-8×ASRM interaction within this group likely represents loss of the protective effect of manic symptoms when depressive symptoms are also present.
The differences in moderation analyses notwithstanding, the primary models for this analysis successfully replicate the results of our prior analyses, producing a strikingly similar response to the question of whether there is synergy between depressive and hypomanic/manic symptoms and suicide risk [2, 10]. With this contribution, all three analyses on this topic have led to the same conclusion and failed to demonstrate such synergy. The current analysis is most similar to the prior analysis done with the NNDC predecessor to the Mood Outcomes Program, the Clinical Care Registry, which involved a smaller sample without adequate power to assess moderators [10]. A key limitation of both the NNDC Mood Outcomes Program and Clinical Care Registry analyses related to the outcome measure that was dominated by suicidal ideation, which is arguably a problematic surrogate outcome for suicide research. Both analyses secondarily looked at suicide attempts with similar results.
The first analysis of data from the Collaborative Depression Study used suicide attempts and suicide deaths as the primary outcome [2]. This suggests that the failure to demonstrate synergy cannot be ascribed to use of a surrogate outcome. All three studies directly testing the hypothesis of synergy between depressive and hypomanic/manic symptoms were not only negative, but suggested even less than additive effects. There are two important prospective studies that looked at mood symptoms as time varying covariates in a survival analysis that temper our conclusions of less-than-additive effects. The only prospective analyses to contrast risk between mood states, including mixed states, showed higher estimates of risk with mixed symptoms relative to depressive, both in the same sample [7, 8]. In an analysis of 176 participants of the Jorvi Bipolar Study followed for 18 months, Holma et al. observed an incidence of 878 (95% CI 421–1,615) suicide attempts per 1,000 patient-years during mixed states and 411 (95% CI 292–562) during major depressive states. The total number of suicide attempts in mixed states was not reported nor was patient-years of follow-up in mixed states to calculate this number [7]. In a prior analysis of the same sample, Valtonen et al. reported that there were only 10 suicide attempts during 11.42 years of time spent in mixed phases, corresponding to a RR of 37.7 (95% CI 11.8–120.3), higher than the estimate for major depression (RR 18.1 (95% CI 6.5–50.8) [8]. These findings are not necessarily inconsistent with our results if they are driven by more severe depressive symptoms during the mixed states, which was not assessed.
The inverse association between ASRM score and suicidal ideation or behavior in major depressive disorder was not seen with bipolar disorder. In patients with major depressive disorder, it seems plausible that the scale is instead inadvertently capturing improvements in depressive symptoms. If a similar effect is seen for some patients with bipolar disorder as well, this could obscure any mania-related associations. The low specificity of the ASRM in detecting mania is an important limitation of this analysis, however, [24] and any resultant misclassification could bias results toward the null hypothesis, which must be considered given the inability of this study to detect an association with mixed manic and depressive symptoms. The self-reported ASRM may also fail to accurately and adequately capture mania. Insight is likely to be more impaired for manic than depressive symptoms, introducing the potential for comparative difficulty in accurately cataloging and rating the severity of manic symptoms [32]. In addition, the ASRM only captures symptoms related to select facets, such as elevated mood and increased energy, and doesn’t capture other accompanying factors such as irritability, impulsivity, or psychosis. Our prior Collaborative Depression Study analysis did not rely on a self-report rating [2] and future studies should consider use of a comprehensive, clinician-administered mania assessment where able. Mania is a complex mood state, consisting of many possible combinations of symptoms. As such, aggregate measures of manic symptom may miss the influence of a more complex interplay between manic and depressive symptoms on suicide risk. It may be the case that only certain combinations of symptoms (or valence of symptoms) increase suicide risk, while other combinations do not. Relatedly, this current study does not account for the potential influence of personality traits such as neuroticism or categorical personality disorders like borderline personality disorder, which demonstrates considerable symptom overlap with bipolar disorder and a resultant potential for misdiagnosis [33, 34]. The current dataset did not have any systematic assessment of personality. In the prior Collaborative Depression Study analysis, models adjusting for neuroticism did not alter the findings although neuroticism itself was an independent predictor of risk [2].
The strength of this study is that a large sample allowed us to assess potential moderators of depressive, manic, and mixed mood symptoms on suicidal ideation or behavior with greater power than prior analyses. Given the multi-center design and racial and ethnic diversity of this sample, the results may be more generalizable than from those of samples from more traditional clinical research studies. However, the real-world data brings its own unique limitations. Patients had limited follow-up assessments at irregular time intervals. This precluded us from modeling time in the models that still allowed for repeat observations per patient, although we did not model changes over time between visits. Diagnoses were determined by clinicians at NNDC Centers, but there is potential for misclassification since formal, structured diagnostic interviews were not used. This may explain some of the unexpected findings in the sample, such as a lower frequency of reported anxiety disorders in bipolar, compared to unipolar major depression, which although unusual was also observed in an early analysis from the Collaborative Depression Study [35]. This unexpected lower frequency of anxiety disorders may be the result of diagnostic overshadowing, where clinicians are less likely to report a co-occurring anxiety disorder for bipolar disorder than major depression, or due to a more limited diagnosis in clinical care compared to research studies that utilize more comprehensive diagnostic instruments. Mood symptoms and outcome were assessed at each clinic visit and therefore represent overlapping, but not identical time periods.
Suicidal ideation is also a problematic surrogate outcome for suicidal behavior, which we assessed only secondarily to ensure adequate power for moderation analyses; as such, the relationships observed between mood symptoms and our composite outcome measure of suicidal ideation or behavior may not hold for attempted suicide or suicide death. There is some suggestion of this in our findings among the patient subgroup with major depressive disorder, in which we detected a protective effect of mania/hypomania in our composite primary outcome of suicidal ideation or behavior, but an increased risk associated with mania/hypomania in our secondary outcome of suicidal behavior alone. One potential explanation could be that manic symptoms increase the risk of attempting suicide without increasing suicidal thoughts, such as through increasing impulsivity, although this was not observed in the smaller, less powered subsample with bipolar disorder. It is also possible that there is greater misclassification of manic symptoms among the subgroup with major depressive disorder, such that the ASRM within this subgroup may also be capturing a component of recovery from depressive symptoms. Another limitation of this study is the use of self-assessment instruments to capture mood symptoms and suicidal ideation/behavior, which introduces the potential for measurement bias. Diagnoses were obtained through normal clinical practice and therefore did not strictly employ the Structured Clinical Interview for DSM-5 to derive psychiatric diagnoses or determine whether participants met DSM-5 criteria for the mixed symptom specifier. While this larger sample allowed us to assess previously unexplored moderation by race, minority groups were underrepresented and we were forced to use a crude (White/non-White) race category. More diverse samples would be more useful to assess moderation and may be even more generalizable.
Taken together, this and recent analyses do not suggest any additional risk of suicide-related outcomes with mixed depressive and manic symptoms beyond the contribution of depressive symptoms themselves. That is, there does is no evidence of synergy between depressive and manic symptoms on suicidal ideation/behavior at a given visit and the direction of the association was toward less than additive effects. Our results are consistent with depressive symptoms being the primary mood symptoms influencing risk of suicidal thoughts and behaviors. Clinicians should heed the importance of depressive symptoms in suicide risk assessment and aggressively target these symptoms as a potentially modifiable risk factor in bipolar disorder. While the presence of mixed depressive and manic symptoms can be an important correlate of a harsher course of illness, there is no compelling evidence to date that their presence within a given mood episode needs to be uniquely considered in the assessment of suicide risk.
Supplementary Material
FIGURE 2: Frequency of Visits with Suicidal Behavior by Mood State.
This figure illustrates the overall proportion of visits by mood state for which suicidal behavior was reported within the combined patient sample. Visits with “no depression or mania” were determined by a self-reported PHQ-8<10 and an ASRM≤5. Visits with “depression” were defined by a PHQ-8≥10 and an ASRM≤5 and mania by a PHQ-8<10 and an ASRM>5. “Mixed” visits were those in which the PHQ-8≥10 and ASRM>5. This figure is provided for descriptive purposes as primary analyses utilized these mood measures as continuous variables.
HIGHLIGHTS.
Depressive symptoms are strongly associated with concomitant suicidal ideation and behavior
Mixed symptoms convey no greater risk for suicidal ideation/behavior than depressive symptoms alone
Mood disorder polarity did not appear to moderate the risk associated with mood state
Mood disorder polarity, sex, and race did not appear to moderate the relationships
Older individuals were less likely to have suicidal ideation/behavior with manic symptoms
Any protective effect of manic symptoms in older adults disappears with concurrent depressive symptoms
ACKNOWLEDGEMENTS
This work was funded by a grant from the National Network of Depression Centers.
Role of Funding Sources
The National Network of Depression Centers (NNDC) provided feedback on the study design and provided access to the data. All authors are or have been at institutions affiliated with the network.
This work was funded by a National Network of Depression Centers 2019 Momentum Grant.
Jess Fiedorowicz received research support for a project with Myriad Genetics, Inc. Dr. Fiedorowicz was funded by the National Institute of Mental Health (R01MH111578) and Institute for Clinical and Translational Science at the University of Iowa (U54TR001356).
Shervin Assari is funded through the National Institutes of Mental Health.
Michael Ostacher is a full-time employee of the Department of Veterans Affairs, United States Government. He has had grants from National Institute on Drug Abuse (NIDA); personal fees from Alkermes, Janssen Pharmaceuticals (Johnson & Johnson), Sage Therapeutics, and grants from Otsuka Pharmaceutical and Palo Alto Health Sciences, outside the submitted work.
Fernando Goes, John Nurnberger, and William Coryell have received research support from Janssen Pharmaceuticals.
Footnotes
CONFLICT OF INTEREST
The authors have no other potential conflicts of interest to disclose.
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
REFERENCES
- [1].Gonzalez-Pinto A, Barbeito S, Alonso M, Alberich S, Haidar MK, Vieta E, Tabares-Seisdedos R, Zorrilla I, Gonzalez-Pinto MA, Lopez P, Poor long-term prognosis in mixed bipolar patients: 10-year outcomes in the Vitoria prospective naturalistic study in Spain, J Clin Psychiatry 72(5) (2011) 671–6. [DOI] [PubMed] [Google Scholar]
- [2].Persons JE, Coryell WH, Solomon DA, Keller MB, Endicott J, Fiedorowicz JG, Mixed state and suicide: Is the effect of mixed state on suicidal behavior more than the sum of its parts?, Bipolar Disord 20(1) (2018) 35–41. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [3].Swann AC, Lafer B, Perugi G, Frye MA, Bauer M, Bahk WM, Scott J, Ha K, Suppes T, Bipolar mixed states: an international society for bipolar disorders task force report of symptom structure, course of illness, and diagnosis, Am J Psychiatry 170(1) (2013) 31–42. [DOI] [PubMed] [Google Scholar]
- [4].Lage RR, Santana CMT, Nardi AE, Cheniaux E, Mixed states and suicidal behavior: a systematic review, Trends Psychiatry Psychother 41(2) (2019) 191–200. [DOI] [PubMed] [Google Scholar]
- [5].Schaffer A, Isometsa ET, Azorin JM, Cassidy F, Goldstein T, Rihmer Z, Sinyor M, Tondo L, Moreno DH, Turecki G, Reis C, Kessing LV, Ha K, Weizman A, Beautrais A, Chou YH, Diazgranados N, Levitt AJ, Zarate CA Jr., Yatham L, A review of factors associated with greater likelihood of suicide attempts and suicide deaths in bipolar disorder: Part II of a report of the International Society for Bipolar Disorders Task Force on Suicide in Bipolar Disorder, Aust N Z J Psychiatry 49(11) (2015) 1006–20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [6].Johnson SL, McMurrich SL, Yates M, Suicidality in bipolar I disorder, Suicide Life Threat Behav 35(6) (2005) 681–9. [DOI] [PubMed] [Google Scholar]
- [7].Holma KM, Haukka J, Suominen K, Valtonen HM, Mantere O, Melartin TK, Sokero TP, Oquendo MA, Isometsa ET, Differences in incidence of suicide attempts between bipolar I and II disorders and major depressive disorder, Bipolar Disord 16(6) (2014) 652–61. [DOI] [PubMed] [Google Scholar]
- [8].Valtonen HM, Suominen K, Haukka J, Mantere O, Leppamaki S, Arvilommi P, Isometsa ET, Differences in incidence of suicide attempts during phases of bipolar I and II disorders, Bipolar Disord 10(5) (2008) 588–96. [DOI] [PubMed] [Google Scholar]
- [9].Rothman KJ, Synergy and antagonism in cause-effect relationships, Am J Epidemiol 99(6) (1974) 85–8. [DOI] [PubMed] [Google Scholar]
- [10].Fiedorowicz JG, Persons JE, Assari S, Ostacher MJ, Zandi P, Wang PW, Thase ME, Frye MA, Coryell W, of G the National Network of Depression Centers Bipolar Disorders Interest, Depressive symptoms carry an increased risk for suicidal ideation and behavior in bipolar disorder without any additional contribution of mixed symptoms, J Affect Disord 246 (2019) 775–782. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [11].Ostacher MJ, Nierenberg AA, Rabideau D, Reilly-Harrington NA, Sylvia LG, Gold AK, Shesler LW, Ketter TA, Bowden CL, Calabrese JR, Friedman ES, Iosifescu DV, Thase ME, Leon AC, Trivedi MH, A clinical measure of suicidal ideation, suicidal behavior, and associated symptoms in bipolar disorder: Psychometric properties of the Concise Health Risk Tracking Self-Report (CHRT-SR), J Psychiatr Res 71 (2015) 126–33. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [12].Akiskal HS, Benazzi F, Family history validation of the bipolar nature of depressive mixed states, J Affect Disord 73(1–2) (2003) 113–22. [DOI] [PubMed] [Google Scholar]
- [13].Perugi G, Akiskal HS, Micheli C, Toni C, Madaro D, Clinical characterization of depressive mixed state in bipolar-I patients: Pisa-San Diego collaboration, J Affect Disord 67(1–3) (2001) 105–14. [DOI] [PubMed] [Google Scholar]
- [14].Keller MB, Lavori PW, Coryell W, Andreasen NC, Endicott J, Clayton PJ, Klerman GL, Hirschfeld RM, Differential outcome of pure manic, mixed/cycling, and pure depressive episodes in patients with bipolar illness, JAMA 255(22) (1986) 3138–42. [PubMed] [Google Scholar]
- [15].Maj M, Pirozzi R, Magliano L, Bartoli L, Agitated depression in bipolar I disorder: prevalence, phenomenology, and outcome, Am J Psychiatry 160(12) (2003) 2134–40. [DOI] [PubMed] [Google Scholar]
- [16].Sato T, Bottlender R, Schroter A, Moller HJ, Frequency of manic symptoms during a depressive episode and unipolar ‘depressive mixed state’ as bipolar spectrum, Acta Psychiatr Scand 107(4) (2003) 268–74. [DOI] [PubMed] [Google Scholar]
- [17].Estrada-Prat X, Van Meter AR, Camprodon-Rosanas E, Batlle-Vila S, Goldstein BI, Birmaher B, Childhood factors associated with increased risk for mood episode recurrences in bipolar disorder-A systematic review, Bipolar Disord 21(6) (2019) 483–502. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [18].Makady A, de Boer A, Hillege H, Klungel O, Goettsch W, What Is Real-World Data? A Review of Definitions Based on Literature and Stakeholder Interviews, Value Health 20(7) (2017) 858–865. [DOI] [PubMed] [Google Scholar]
- [19].Zandi PP, Wang YH, Patel PD, Katzelnick D, Turvey CL, Wright JH, Ajilore O, Coryell W, Schneck CD, Guille C, Saunders EFH, Lazarus SA, Cuellar VA, Selvaraj S, Dill Rinvelt P, Greden JF, DePaulo JR, Development of the National Network of Depression Centers Mood Outcomes Program: A Multisite Platform for Measurement-Based Care, Psychiatr Serv 71(5) (2020) 456–464. [DOI] [PubMed] [Google Scholar]
- [20].Kroenke K, Spitzer RL, Williams JB, The PHQ-9: validity of a brief depression severity measure, J Gen Intern Med 16(9) (2001) 606–13. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [21].Kroenke K, Strine TW, Spitzer RL, Williams JB, Berry JT, Mokdad AH, The PHQ-8 as a measure of current depression in the general population, J Affect Disord 114(1–3) (2009) 163–73. [DOI] [PubMed] [Google Scholar]
- [22].Wu Y, Levis B, Riehm KE, Saadat N, Levis AW, Azar M, Rice DB, Boruff J, Cuijpers P, Gilbody S, Ioannidis JPA, Kloda LA, McMillan D, Patten SB, Shrier I, Ziegelstein RC, Akena DH, Arroll B, Ayalon L, Baradaran HR, Baron M, Bombardier CH, Butterworth P, Carter G, Chagas MH, Chan JCN, Cholera R, Conwell Y, de Man-van Ginkel JM, Fann JR, Fischer FH, Fung D, Gelaye B, Goodyear-Smith F, Greeno CG, Hall BJ, Harrison PA, Harter M, Hegerl U, Hides L, Hobfoll SE, Hudson M, Hyphantis T, Inagaki MD, Jette N, Khamseh ME, Kiely KM, Kwan Y, Lamers F, Liu SI, Lotrakul M, Loureiro SR, Lowe B, McGuire A, Mohd-Sidik S, Munhoz TN, Muramatsu K, Osorio FL, Patel V, Pence BW, Persoons P, Picardi A, Reuter K, Rooney AG, Santos IS, Shaaban J, Sidebottom A, Simning A, Stafford MD, Sung S, Tan PLL, Turner A, van Weert HC, White J, Whooley MA, Winkley K, Yamada M, Benedetti A, Thombs BD, Equivalency of the diagnostic accuracy of the PHQ-8 and PHQ-9: a systematic review and individual participant data meta-analysis, Psychol Med (2019) 1–13. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [23].Altman EG, Hedeker D, Peterson JL, Davis JM, The Altman Self-Rating Mania Scale, Biol Psychiatry 42(10) (1997) 948–55. [DOI] [PubMed] [Google Scholar]
- [24].Altman E, Hedeker D, Peterson JL, Davis JM, A comparative evaluation of three self-rating scales for acute mania, Biol Psychiatry 50(6) (2001) 468–71. [DOI] [PubMed] [Google Scholar]
- [25].Posner K, Brown GK, Stanley B, Brent DA, Yershova KV, Oquendo MA, Currier GW, Melvin GA, Greenhill L, Shen S, Mann JJ, The Columbia-Suicide Severity Rating Scale: initial validity and internal consistency findings from three multisite studies with adolescents and adults, Am J Psychiatry 168(12) (2011) 1266–77. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [26].Baron RM, Kenny DA, The moderator-mediator variable distinction in social psychological research: conceptual, strategic, and statistical considerations, J Pers Soc Psychol 51(6) (1986) 1173–82. [DOI] [PubMed] [Google Scholar]
- [27].Kraemer HC, Kiernan M, Essex M, Kupfer DJ, How and why criteria defining moderators and mediators differ between the Baron & Kenny and MacArthur approaches, Health Psychol 27(2S) (2008) S101–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [28].Lodder P, Modeling synergy: How to assess a Type D personality effect, J Psychosom Res 132 (2020) 109990. [DOI] [PubMed] [Google Scholar]
- [29].Ylikoski A, Erkinjuntti T, Raininko R, Sarna S, Sulkava R, Tilvis R, White matter hyperintensities on MRI in the neurologically nondiseased elderly. Analysis of cohorts of consecutive subjects aged 55 to 85 years living at home, Stroke 26(7) (1995) 1171–7. [DOI] [PubMed] [Google Scholar]
- [30].Shimoni A, Kroger N, Zabelina T, Ayuk F, Hardan I, Yeshurun M, Shem-Tov N, Avigdor A, Ben-Bassat I, Zander AR, Nagler A, Hematopoietic stem-cell transplantation from unrelated donors in elderly patients (age >55 years) with hematologic malignancies: older age is no longer a contraindication when using reduced intensity conditioning, Leukemia 19(1) (2005) 7–12. [DOI] [PubMed] [Google Scholar]
- [31].Streiner DL, Cairney J, Veldhuizen S, The epidemiology of psychological problems in the elderly, Can J Psychiatry 51(3) (2006) 185–91. [DOI] [PubMed] [Google Scholar]
- [32].Gazalle FK, Frey BN, Hallal PC, Andreazza AC, Cunha AB, Santin A, Kapczinski F, Mismatch between self-reported quality of life and functional assessment in acute mania: a matter of unawareness of illness?, J Affect Disord 103(1–3) (2007) 247–52. [DOI] [PubMed] [Google Scholar]
- [33].Fiedorowicz JG, Black DW, Borderline, bipolar, or both?, Current Psychiatry 9(1) (2010) 20–24,29–32. [Google Scholar]
- [34].Eskander N, Emamy M, Saad-Omer SM, Khan F, Jahan N, The Impact of Impulsivity and Emotional Dysregulation on Comorbid Bipolar Disorder and Borderline Personality Disorder, Cureus 12(8) (2020) e9581. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [35].Andreasen NC, Grove WM, Endicott J, Coryell WH, Scheftner WA, Hirschfeld RMA, Keller MB, The phenomenology of depresion, Psychiatrie et Psychobiologie 3 (1988) 1–10. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.