Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2020 Mar 1.
Published in final edited form as: J Affect Disord. 2018 Dec 21;246:775–782. doi: 10.1016/j.jad.2018.12.057

Depressive symptoms carry an increased risk for suicidal ideation and behavior in bipolar disorder without any additional contribution of mixed symptoms

Jess G Fiedorowicz 1,3,4,*, Jane E Persons 2,*, Shervin Assari 4, Michael J Ostacher 5,6, Peter Zandi 7, Po W Wang 6, Michael E Thase 8, Mark A Frye 9, William Coryell, National Network of Depression Centers Bipolar Disorders Interest Group1
PMCID: PMC6914253  NIHMSID: NIHMS1544189  PMID: 30623823

Abstract

Objectives

To determine whether the risk of suicidal ideation or behavior during mixed states exceeds that attributable to the depressive components of these states alone in bipolar disorder.

Methods

We utilized real-world, longitudinal clinical data collected on 290 patients with bipolar disorders (bipolar I, bipolar II, and bipolar not otherwise specified (NOS)) from the National Network of Depression Centers (NNDC) Clinical Care Registry (CCR) seen for 891 visits over a mean of 27.5 weeks. Depressive symptoms were measured with the Patient Health Questionnaire-9 (PHQ-9), manic symptoms with the Altman Self-Rating Mania (ASRM), and suicidal ideation and behavior with the Columbia-Suicide Severity Rating Scale (C-SSRS), obtained as part of the routine, measurement-based care provided across the NNDC. The relations between depressive symptoms, manic symptoms, and the interaction thereof (mixed symptoms) on coinciding suicidal ideation and behavior were modeled in generalized linear mixed models.

Results

Depressive symptoms, as measured by the PHQ-9, were strongly associated with suicidal ideation and behavior (p<0.0001), while there was no significant association with manic symptoms as measured by the ASRM or the interaction between depressive and manic symptoms. Similar results were observed when the outcome was restricted to suicidal behavior and when mood was modeled categorically. There was evidence of a gender by ASRM interaction (p=0.011) and risk of suicidal ideation or behavior was significant for women, but not men with manic symptoms.

Limitations

Diagnoses were based on clinician assessment and not structured interview. Mood assessments were self-report rather than clinician-administered. Suicidal ideation was more frequently observed than suicidal behavior (23/272 visits where outcome positive).

Conclusions

Depression represents the primary mood state accounting for suicide risk in bipolar disorder. Co-occurring symptoms of mania (mixed symptoms) do not appear to convey an elevated risk for suicidal ideation or behavior beyond that explained by the depressive symptoms alone.

Keywords: Bipolar disorder, Mania, Depression, Suicidal ideation, Attempted Suicide

INTRODUCTION

Mixed states have been variably described as a combination of manic and depressive symptoms, often occurring during transitions between these mood states (Perlis, Ostacher et al. 2010, Swann, Lafer et al. 2013). Mixed states are experienced by an estimated 20% of individuals with bipolar disorder based on retrospective estimates from cross-sectional studies (Rihmer, Gonda et al. 2008, Swann, Lafer et al. 2013, Shim, Woo et al. 2015) and by 37–38% of individuals with bipolar disorder based on long-term prospective studies (Gonzalez-Pinto, Barbeito et al. 2011, Persons, Coryell et al. 2018). Mixed states are considered to be a high-risk state for suicide regardless of the specific constellation of symptoms or predominant polarity (Swann, Lafer et al. 2013). However, this has been largely assumed from cross-sectional studies demonstrating greater past suicidal behavior among those with a history of mixed symptoms compared to those without such history (Strakowski, McElroy et al. 1996, Cassidy, Murry et al. 1998, Balazs, Benazzi et al. 2006, Goldberg, Perlis et al. 2009, Swann, Steinberg et al. 2009, Pacchiarotti, Mazzarini et al. 2011).

Few prospective studies have investigated suicidal behavior across mood states. A prospective study that followed 120 persons with bipolar I disorder for up to 10 years found that those with a history of mixed states (37% of the sample prospectively) had a more chronic course of illness, with approximately 10 mood episodes compared to 5 mood episodes per patient in those without a history of mixed states. Individuals with a history of mixed states also had more suicide attempts (30% vs. 13%), a difference that was significant in univariate analysis, but not after adjustment for age of onset (Gonzalez-Pinto, Barbeito et al. 2011), a feature that has been consistently linked to a more chronic or persistent depressive course of illness (Coryell, Fiedorowicz et al. 2013). A two-year prospective cohort study followed 91 individuals with bipolar I disorder, looking at the month in which patients scored highest on the suicide item from a modified Hamilton Rating Scale for Depression, and found higher suicidality to be more strongly correlated with depression and hopelessness, but less robustly with mixed state (Johnson, McMurrich et al. 2005). Using time-varying variables over 18 months of follow-up, Valtonen et al. found an elevated risk of suicide attempts in mixed phases (HR 28.0, 95% C.I. 7.9–99.2) although did not directly test whether this was in excess of that attributable to depressive symptomatology. Persons JE et al. recently conducted a prospective analysis of 429 individuals with bipolar disorder followed for a mean of 18 years in the Collaborative Depression Study to prospectively assess suicidal behavior with mixed states in a time-varying manner and in relation to the component depressive or manic states (Persons, Coryell et al. 2018). This analysis found that those who experienced mixed symptoms had a much more persistent course of depressive symptomatology and it was this greater depressive symptom burden that explained 71% of the excess risk for suicidal behavior. There was no increased risk of suicidal behavior in mixed states beyond that attributable to the component depressive symptoms (Persons, Coryell et al. 2018).

We sought to replicate this analysis of the Collaborative Depression Study utilizing real-world evidence (Sherman, Anderson et al. 2016), consisting of validated mood and suicide assessment scales collected during routine clinical care, not clinical research, from patients enrolled in the National Network of Depression Centers (NNDC) registry. We hypothesized that suicidal ideation or behavior would be more common in those with current depression and not rendered more likely by the concurrent presence of manic or hypomanic symptomatology. While mania/hypomania is more common in bipolar I than bipolar II disorder (Judd, Schettler et al. 2003), the prevalence of mixed depressive symptoms with hypomania does not differ by bipolar subtype although is greater in women (Suppes, Mintz et al. 2005). We subsequently also tested whether gender and bipolar subtype moderated the relation between mood and suicidal ideation or behavior.

METHOD

Study Sample

The NNDC is an inter-dependent nonprofit consortium of 26 academic depression centers. We utilized real world evidence from the NDDC Clinical Care Registry (CCR). The registry includes patients with a mood disorder who were 18 years or older, could speak English, and were receiving treatment at one of the participating centers. During the initial phase of establishing the registry between 12/24/2011–11/29/2014, longitudinal data was collected through measurement-based care at 18 United States centers affiliated with the NNDC: Brigham and Women’s Hospital (Harvard Medical School), Duke University, Emory University, Johns Hopkins University, Massachusetts General Hospital (Harvard Medical School), Mayo Clinic, McLean Hospital (Harvard Medical School), Medical University of South Carolina, Stanford University, University of Illinois at Chicago, University of Cincinnati & Linder Center of HOPE, University of Iowa, University of Louisville, University of Massachusetts, University of Michigan, University of Pennsylvania, University of San Francisco and Weill Cornell Medical College. The Michigan Institute for Clinical Health Research manages the collection of the de-identified data and database creation. For protection of privacy, sites are identified by number and not name in the database. We included data from all visits for individuals in the CCR with a diagnosis of bipolar I disorder, bipolar II disorder, and bipolar disorder not otherwise specified (NOS), even including those with only one visit.

Mood Measures

Mood measures were selected as part of a common assessment package through expert consensus. Depressive symptoms were captured using the Patient Health Questionnaire-9 (PHQ-9), a nine-item depression scale based on DSM-IV diagnostic criteria for major depressive disorder (Kroenke, Spitzer et al. 2001). Scores for this measure range between 0 and 27 based on the persistence of depressive symptoms over the past two weeks. Primary analyses utilized PHQ-9 as a continuous measure, but for identifying depression categorically an established cut-off of ≥ 10 was used, which has a pooled specificity for major depression of 0.87 (Moriarty, Gilbody et al. 2015).

Manic/hypomanic symptoms were captured using the five-item Altman Self-Rating Mania (ASRM) scale (Altman, Hedeker et al. 1997). Scores range on this measure between 0 and 20 and a score of >5 has a high sensitivity but a variable specificity (33–87%) for mania, although the lower specificity was observed in samples with mild or hypomanic symptoms (Altman, Hedeker et al. 1997, Altman, Hedeker et al. 2001).

Mixed symptoms were defined as the simultaneous presence of depressive and manic symptoms. For primary models, mixed symptoms were modeled as an interaction term between the continuous PHQ-9 and ASRM measures described above. In the absence of an operationally-based definition of mixed states from patient screening scores, mixed states were categorically defined as the concurrence of a PHQ-9 ≥ 10 and an ASRM > 5.

Suicidal Behavior

The primary outcome under investigation is suicidal ideation or behavior as reported on the Columbia-Suicide Severity Rating Scale (C-SSRS) (Posner, Brown et al. 2011) at a coinciding visit to the mood ratings. Suicidal ideation was defined as any thoughts of killing oneself or wishing oneself dead, as both are associated with risk of suicide behavior (Szanto, Reynolds et al. 1996, Baca-Garcia, Perez-Rodriguez et al. 2011, Greist, Mundt et al. 2014). Suicidal behavior was defined as any suicide attempt (including interrupted or aborted attempts) or preparations for an attempt reported on the C-SSRS (Conway, Erlangsen et al. 2016). The C-SSRS has been shown to have very high sensitivity and specificity compared to expert consensus ratings for suicidal behavior and is sensitivity to change over time (Posner, Brown et al. 2011).

Statistical Analysis Plan

Analyses were conducted using SAS 9.4. In descriptive analyses, comparisons of those with and without observed mixed states at any visit with regard to demographic information, self-reported depressive and manic symptom burden, and any available comorbidities (e.g., panic disorder, generalized anxiety disorder, etc.) were made using chi-square for categorical measures and the Wilcoxon rank sum for continuous measures. As per the statistical analysis plan, these groupings were not used in subsequent analyses. Our a priori primary outcome variable was suicidal thoughts or behavior as reported on the Columbia-Suicide Severity Scale, modeled an indicator variable. Primary models assessed risk of suicidal thoughts or behavior as a function of depressive symptoms (Patient Health Questionnaire-9 (PHQ-9) score, continuous, linear effect, range 0–27) and manic/hypomanic symptoms (Altman Self-Rating Mania (ASRM) score, continuous, linear effect, range 0–20) using generalized linear mixed models (GLMM - binomial distribution, logit link (canonical)) with a random intercept term to account for repeated observations within patients. The variance-covariance matrix was unstructured and effects of time were not modeled. To model mixed symptomatology, a mania × depression interaction term was entered into the models as a covariate such that a significant interaction supports, a significantly greater (or lesser) risk than would be expected additively by the components attributable to manic or depressive symptoms alone. Multivariable-adjusted models adjusted for gender (indicator variable), age (continuous), race, marital status, and comorbidities (anxiety and substance use disorders). We assessed for moderation of any relation between depressive, manic, and mixed symptoms and our suicide outcome by bipolar diagnosis (bipolar I vs. bipolar II/NOS) and gender. A role for anxiety was explored by adding the seven-item Generalized Anxiety Disorder (GAD-7) score (Spitzer, Kroenke et al. 2006) as a covariate. Sensitivity analyses excluded those with bipolar disorder NOS, who were not included in the original statistical analysis plan but added prior to primary analyses to augment the sample size and improve generalizability. We secondarily modeled depressive and hypomanic/manic symptoms categorically using a PHQ-9 threshold of >=10 and an ASRM threshold of >5 in analogous GLMM models.

Results

There were 290 patients contributing a total of 891 visits included in the analysis. The majority of visits were outpatient encounters (94.6%), with a small percentage of visits (5.4%) occurring in an inpatient setting (642 outpatient vs. 37 inpatient, 212 missing); outpatient visits accounted for 87.1% of intake visits (176 outpatient, 26 inpatient, and 17 missing) and 97.7% of follow-up visits (466 outpatient, 11 inpatient, 195 missing). The mean number of visits per patent included in the CCR database was 3.1 (SD 3.5, range 1.0 – 21.0) and the mean length of follow-up was 27.5 weeks (SD 37.9, range 1.0 – 155.0). Suicidal ideation or behaviors were present at 272 visits (30.5%) with 23 of these 272 (8.5%) involving suicidal behaviors. Thus, 91.5% of outcomes observed involved suicidal ideation only without suicidal behaviors. Across visits, the overall mean ASRM score was 2.7 (median 1, SD 3.4) and mean PHQ-9 was 9.2 (median 8, SD 7.1).

The sociodemographic and clinical characteristics of this sample are detailed in Table 1 and stratified by those that meet our categorical threshold for a mixed state at some assessment. There were 44 patients (15.2%) observed to meet our operationalized definition of mixed state at any point in the follow-up period. Patients with mixed states were not appreciably different from those without apart from a marginally increased likelihood of having been diagnosed with anxiety disorder not otherwise specified. Individuals with a history of mixed states also had a significantly higher overall mean PHQ-9 score (13.4 vs. 8.2; p<0.0001) and a significantly higher mean ASRM score (5.3 vs. 2.1; p<0.0001) compared to those without. Among patients with depressive symptoms that exceed the PHQ-9≥10 threshold, the severity of depressive symptoms was similar for patients with and without mixed states (mean PHQ-9 score 15.0 vs. 16.4). Among patients with manic symptoms exceeding the ASRM>5 threshold, ASRM scores were similar for patients with and without mixed states (mean ASRM score 9.1 vs. 8.8).

Table 1.

Baseline sociodemographic and clinical characteristics of sample.

Total (n= 290) Mixed States (n= 44) No Mixed States (n= 246)
n(%) n(%) n(%) χ2 p-value
Diagnosis
Bipolar I Disorder 152 (52.4%) 23 (52.3%) 129 (52.4%) 0.98
Bipolar II Disorder 101 (34.8%) 13 (29.6%) 88 (35.8%) 0.42
Bipolar Disorder NOS 37 (12.8%) 8 (18.2%) 29 (11.8%) 0.24
Co-occurring Psychiatric Diagnoses
Generalized Anxiety Disorder 19 (6.6%) 2 (4.5%) 17 (6.9%) 0.56
Obsessive-Compulsive Disorder 7 (2.4%) 0 7 (2.8%) 0.26
Panic Disorder 10 (3.4%) 1 (2.3%) 9 (3.7%) 0.64
Phobic Disorder 5 (1.7%) 1 (2.3%) 4 (1.6%) 0.76
Anxiety Disorder NOS 16 (5.5%) 5 (11.4%) 11 (4.5%) 0.065
Substance Use Disorder 5 (1.7%) 1 (2.3%) 4 (1.6%) 0.76
Past Psychiatric Hospitalizations 0.39
Yes 186 (64.2%) 30 (68.2%) 156 (63.4%)
No 74 (25.5%) 12 (27.3%) 62 (25.2%)
Unknown 30 (10.3%) 2 (4.5%) 28 (11.4%)
Gender (missing = 1) 0.32
Female 171 (59.2%) 29 (65.9%) 142 (58.0%)
Male 118 (40.8%) 15 (34.1%) 103 (42.0%)
Race 0.99
Asian/Asian American 6 (2.1%) 1 (2.3%) 5 (2.0%)
Black or African American 17 (5.9%) 3 (6.8%) 14 (5.7%)
Native Hawaiian/Pacific Islander 1 (0.3%) 0 1 (0.4%)
White 255 (87.9%) 38 (86.4%) 217 (88.2%)
Other 6 (2.1%) 1 (2.3%) 5 (2.0%)
Unknown 5 (1.7%) 1 (2.3%) 4 (1.6%)
Married (missing = 4) 0.60
Single 98 (34.3%) 19 (43.2%) 79 (32.6%)
Married 107 (37.4%) 17 (38.6%) 90 (37.2%)
In a committed relationship 32 (11.2%) 3 (6.8%) 29 (12.0%)
Divorced 36 (12.6%) 3 (6.8%) 33 (13.6%)
Separated 5 (1.8%) 1 (2.3%) 4 (1.7%)
Widowed 8 (2.8%) 1 (2.3%) 7 (2.9%)
Education Level (missing = 12) 0.11
8th grade or less 3 (1.1%) 2 (4.5%) 1 (0.4%)
Some High school 9 (3.2%) 2 (4.5%) 7 (3.0%)
High School Diploma 24 (8.6%) 6 (13.6%) 18 (7.7%)
Some College 77 (27.7%) 14 (31.8%) 63 (26.9%)
Technical or Associate’s degree 15 (5.4%) 2 (4.5%) 13 (5.6%)
Bachelor’s degree 83 (29.9%) 8 (18.2%) 75 (32.1%)
Advanced degree 67 (24.1%) 10 (22.7%) 57 (24.4%)
Employment
Full-time 77 (26.5%) 9 (20.5%) 68 (27.6%) 0.32
Part-time 38 (13.1%) 8 (18.2%) 30 (12.2%) 0.28
Student 20 (6.9%) 2 (4.6%) 18 (7.3%) 0.50
Homemaker 13 (4.5%) 2 (4.6%) 11 (4.5%) 0.98
Retired 42 (14.5%) 7 (15.9%) 35 (14.2%) 0.77
Disability 75 (25.9%) 19 (43.2%) 56 (22.8%) 0.004
Leave of Absence 10 (3.5%) 2 (4.6%) 8 (3.3%) 0.67
Unemployed 41 (14.1%) 5 (11.4%) 36 (14.6%) 0.57
Age Category 0.89
Less than 30 years old 63 (21.7%) 9 (20.5%) 54 (22.0%)
30–39 years old 56 (19.3%) 7 (15.9%) 49 (19.9%)
40–49 years old 43 (14.8%) 8 (18.2%) 35 (14.2%)
50–59 years old 65 (22.4%) 9 (20.5%) 56 (22.8%)
60 years old or older 63 (21.7%) 11 (25.0%) 52 (21.1%)
Open in a new tab

In the primary GLMM models, suicidal ideation or behavior was significantly associated with depression (β 0.31, SE 0.036, 95% CI 0.24 – 0.38, p<0.0001), but not mania (β 0.048, SE 0.049, 95% CI −0.048 – 0.14, p=0.33) or mixed symptoms (β 0.0021, SE 0.0072, 95% CI −0.012 – 0.016, p=0.77), adjusting for age, race, education, marital status, and comorbid substance use or anxiety disorders as shown in Table 2. In analogous secondary models restricting the outcome definition to suicidal behavior, similar results were observed with suicidal behavior significantly associated with depression (β 0.29, SE 0.13, 95% CI 0.032 – 0.55, p=0.028), but not mania (β −0.091, SE 0.12, 95% CI −0.33 – 0.15, p=0.45) or mixed symptoms (β 0.026, SE 0.020, 95% CI −0.014 – 0.062, p=0.18) as shown in Table 3. There was no significant moderation by bipolar I subtype (bipolar subtype by PHQ-9 interaction p=0.50, bipolar subtype by ASRM interaction p=0.072). The marginally significant bipolar subtype by ASRM interaction was driven by a higher but not statistically significant association between ASRM score and suicidal ideation/behavior in those with bipolar NOS. Exclusion of the heterogeneous bipolar NOS group from the analysis produces a more clearly non-significant bipolar subtype by ASRM interaction p=0.13. There was evidence of moderation by gender (gender by PHQ-9 interaction p=0.49, gender by ASRM interaction p=0.011). In stratified analysis by gender, among men suicidal ideation or behavior was significantly associated with depression (β 0.34, SE 0.070, 95% CI 0.21 – 0.48, p<0.0001) but not mania (β −0.11, SE 0.083, 95% CI −0.27 – 0.058, p=0.20) or mixed state (β 0.0078, SE 0.012, 95% CI −0.016 – 0.032, p=0.52); among women, suicidal ideation or behavior was significantly associated with depression (β 0.30, SE 0.043, 95% CI 0.22 – 0.39, p<0.0001) and mania (β 0.16, SE 0.066, 95% CI 0.03 – 0.29, p=0.016), but not mixed state (β −0.0020, SE 0.0097, 95% CI −0.021 – 0.017, p=0.84). Results did not substantively differ after omitting individuals with a diagnosis of bipolar NOS (including with regard to moderation by gender) or after further adjustment for GAD-7 score, which was not significantly related to outcome (p=0.82).

Table 2.

Generalized Linear Mixed Model Analyses: Effects of Continuous Mood State on combined Suicidal Behavior or Ideation

Unadjusted
β SE 95% CI p-value
Mixed state 0.0070 0.0072 −0.0071–0.021 0.33
Depression 0.31 0.033 0.24–0.37 <0.0001
Mania 0.055 0.047 −0.037–0.15 0.24
Multivariable-adjusted1
β SE 95% CI p-value
Mixed state 0.0021 0.0072 −0.012–0.016 0.77
Depression 0.31 0.036 0.24–0.38 <0.0001
Mania 0.048 0.049 −0.048–0.14 0.33
Open in a new tab
1

Adjusted for gender, age, race, education, marital status, and presence of comorbid substance use or anxiety disorders

Table 3.

Generalized Linear Mixed Model Analyses: Effects of Continuous Mood State on Suicidal Behavior

Unadjusted
β SE 95% CI p-value
Mixed state 0.026 0.020 −0.012–0.065 0.18
Depression 0.29 0.11 0.064–0.51 0.012
Mania −0.053 0.11 −0.28–0.17 0.64
Multivariable-adjusted1
β SE 95% CI p-value
Mixed state 0.024 0.019 −0.014–0.062 0.22
Depression 0.29 0.13 0.032–0.55 0.028
Mania −0.091 0.12 −0.33–0.15 0.45
Open in a new tab
1

Adjusted for gender, age, race, education, marital status, and presence of comorbid substance use or anxiety disorders

Categorical assessment of mood state revealed that a total of 160 patients (55.2%) met PHQ-9≥10 criteria for depression and 97 patients (33.4%) met ASRM>5 criteria for mania at any point during follow-up, contributing to a total of 379 observations meeting criteria for depression and 147 observations meeting criteria for mania with 70 of each overlapping for mixed. Figure 1 presents the raw data for the percentages of all visits in each mood state wherein the primary outcome was observed: 56.3% of visits with depression (PHQ-9≥10 and ASRM≤5), 13.0% of visits with mania (ASRM>5 and PHQ-9<10), 47.1% of mixed visits (PHQ-9≥10 and ASRM>5), compared to only 12.6% of visits meeting neither threshold. In categorical GLMM mood models as shown in Table 4, suicidal behavior was significantly associated with depression (OR 15.12, 95% CI 7.85 – 29.12, p<0.0001), but not mania (OR 0.85, 95% CI 0.40 – 1.81, p=0.67) or mixed state (OR 0.90, 95% CI 0.20 – 3.97, p=0.89).

Figure 1. Percentage of visits with suicidal behaviors or ideation.

Figure 1.

Open in a new tab

This figure illustrates the percentage of visits in each categorically-defined and mutually-exclusive mood state wherein the outcome of suicidal behavior or ideation was observed. A given patient could provide multiple visits in this presentation of raw data.

Table 4.

Generalized Linear Mixed Model Analyses: Effects of Binomial Mood State on combined Suicidal Ideation or Behavior

Unadjusted
OR 95% CI p-value
Mixed state 1.32 0.30 – 5.89 0.72
Depression 15.81 8.33 – 30.00 <0.0001
Mania 0.89 0.42 – 1.90 0.77
Multivariable-adjusted1
OR 95% CI p-value
Mixed state 0.90 0.20 – 3.97 0.89
Depression 15.12 7.85 – 29.12 <0.0001
Mania 0.85 0.40 – 1.81 0.67
Open in a new tab
1

Adjusted for gender, age, race, education, marital status, and presence of comorbid substance use or anxiety disorders

Discussion

In this real world sample of patients with mood disorders receiving treatment at sites across the United States, we found a robust association between both PHQ-9 score and categorical presence of depressive symptoms (PHQ-9≥10) and suicidal ideation and behavior. Consistent with a recent analysis from the Collaborative Depression Study (Persons, Coryell et al. 2018) and from the Lithium Treatment Moderate dose Use Study (LiTMUS) study (Ostacher, Nierenberg et al. 2015), we failed to identify any additional risk associated with mixed states that couldn’t be explained by depressive symptoms alone in both primary models (assessing mood symptoms as a continuous variable) and in secondary categorical models. With an OR of 15.8 (95% C.I. 8.3–30.0), our estimates of elevated risk with depressive symptoms was comparable to that reported for major depressive phases by Valtonen et al. (RR 18.1, 95% C.I. 6.5–50.8), but lower than that reported by Persons et al. (HR 2.0, 95% C.I. 1.3–3.0) (Valtonen, Suominen et al. 2008, Persons, Coryell et al. 2018).

Interestingly, our analyses also uncovered significant gender differences, demonstrated by a gender by mania interaction, such that mania was associated with suicidal ideation or behavior in women, but not men with bipolar disorder in follow-up stratified analyses. This is not likely explained by women having higher rates of bipolar II disorder (Baldassano, Marangell et al. 2005) as the prior Collaborative Depression Study Analysis found a relationship between mania and suicidal behavior only in bipolar I disorder (Persons, Coryell et al. 2018), a result that was not replicated in the current analysis. Akin to our results, in the aforementioned LiTMUS study, a six-month randomized trial of 283 subjects with bipolar I or II disorder, a measure of suicide risk, the Concise Health Risk Tracking Self-Report, was closely correlated with measures of depression but had no association with manic symptoms (Ostacher, Nierenberg et al. 2015). Large prospective samples will be needed to more clearly discern and replicate gender and bipolar subtype as putative moderators of mood states on suicidal ideation and behavior as four times the sample size is required to detect an interaction as is needed to detect a main effect of the same magnitude (Leon and Heo 2009).

In the current study, the prevalence of mixed states, as operationally defined using PHQ-9 and ASRM cutoffs was 15%, which is lower than the 20% reported in previous cross-sectional studies (Rihmer, Gonda et al. 2008, Swann, Lafer et al. 2013, Shim, Woo et al. 2015) and the 37–38% from past prospective studies (Gonzalez-Pinto, Barbeito et al. 2011, Persons, Coryell et al. In Press). Given that prospective prevalence estimates come from studies with 10–18 years of follow-up, our estimates would have been expected to more closely approximate the 20% reported from cross-sectional studies. This slightly lower than expected observed prevalence estimate may due to the lack of a systematic assessment at baseline for any prior mixed states. While any resultant underestimation may have impacted our delineation of groups based on history of mixed vs. no history of mixed states, these groupings were not used in any of the prospective statistical models. The lower than expected prevalence estimate may also reflect differences in our sample, which consisted of actual patients being treated at NNDC centers, rather than participants in clinical trials or the application of PHQ-9 and ASRM-based definitions. As expected, Patients who demonstrated mixed states had marginally more frequent unspecified anxiety disorders although otherwise lacked the additional co-morbidity reported in samples with a history of mixed states (Valtonen, Suominen et al. 2007, Persons, Coryell et al. 2018). The prevalence of co-occurring anxiety and substance use disorders is well under that expected from systematic assessments of representative samples (Merikangas, Akiskal et al. 2007) and suggests that even in specialized mood disorders centers that co-occurring conditions continue to be underdiagnosed on clinical interview alone in the absence of structured evaluations (Zimmerman and Mattia 1999).

Patients who demonstrated mixed states also appeared to have higher mean PHQ-9 and ASRM scores at each visit. While this would be expected based on prior studies (Gonzalez-Pinto, Barbeito et al. 2011, Persons, Coryell et al. 2018), there is the potential for this to be tautological since mixed states were defined as meeting both thresholds at a given visit and each patient had a a mean of only just over three clinic visits. There were similar mean mood ratings between groups (mixed vs. not mixed) for visits exceeding either depressive or manic rating thresholds. Our finding of a primary role for depressive symptoms in the manifestation of suicidal ideation and behavior was independent of concurrent anxiety symptoms or the categorical presence of a diagnosed anxiety disorder, consistent with that previously reported in a large cross-sectional study of mixed symptoms and suicidal thoughts and behavior (Valtonen, Suominen et al. 2007).

Unlike the Collaborative Depression Study analysis, which found a higher risk of suicidal behavior with mania in bipolar I disorder (Persons, Coryell et al. 2018), the current analysis did not observe any risk of suicidal ideation or behavior with mania in bipolar I disorder, although this was observed in women. A stronger link between mania and poor sleep quality has been observed I women (Saunders, Fernandez-Mendoza et al. 2015), which could influence suicidal thoughts or behaviors (Perlis, Grandner et al. 2016). In analyses from the Stanley Foundation Bipolar Treatment Network, broadly-defined mixed symptoms have also been found to be more common in women and associated with a greater burden of depressive symptoms (Suppes, Mintz et al. 2005, Miller, Suppes et al. 2016). Variable findings may have been influenced by the lower specificity of the ASRM when used in samples with a spectrum of mood elevation symptoms (Altman, Hedeker et al. 2001) and more limited ability relative to clinician-administered mania scales of capturing impairments in insight and function, which are welldocumented in mania (Dell’Osso, Pini et al. 2002, Judd, Schettler et al. 2008). Although it is not established whether women have more insight than men into mood symptoms (Baldassano, Marangell et al. 2005), one study assessing insight in remitted patients with bipolar disorder reported a quantitatively but not significantly disproportionate number of women in the preserved insight group with cell counts translating to a marginally significant association (χ2=2.9, p=0.09) (Dias, Brissos et al. 2008). Any differential misclassification of mania on the self-report ASRM due to group differences in insight could result in apparent moderation as an artifact. Categorical mania (ASRM>5) was seen at 1/3 of visits, more than would be expected from prospective cohort data, wherein depression was present 59% of weeks and mania 2% of weeks over long-term follow-up in the Collaborative Depression Study (Judd, Schettler et al. 2003). This greater than expected frequency of mania is likely due to the low specificity of ASRM – that is, ASRM scores >5 often do not actually represent categorical mania in a clinic population – but may in part be due to selection bias with clinic visits more likely when ill (although the PHQ-9 threshold for depression was met at approximately 55% of visits as expected).

This study is notable for the partial replication of prior observational data in a real world clinical setting, although the sample has more limited and irregular follow-up assessments. Information on past suicidal behavior was not available. Diagnoses were made by clinicians in NNDC Centers of Excellence and not by structured interview. Mixed symptoms and states were operationalized by PHQ-9 and ASRM scales and the validity of such self-report measures to define mixed states is not established. While meant to approximate DSM-5 and with use of established thresholds, assessments could not be based directly on DSM-5 criteria. The DSM-5 mixed features specifier is applied during defined major depressive episodes with at least three manic/hypomanic symptoms or during defined manic/hypomanic episodes with at least three depressive symptoms (American Psychiatric Association 2013). Our models were not impacted by the irregular follow-up, and the repeated measurements available improved our ability to estimate risk of suicidal thoughts and behavior. Our outcome, which primarily involved suicidal thoughts, is a surrogate for the more clinically relevant suicidal behavior, and suicidal behavior, albeit infrequently observed, was secondarily assessed with similar results. Although potentially clinically relevant in its own right, even suicidal behavior is a questionable surrogate for completed suicide (Wortzel, Gutierrez et al. 2010). Our sample included patients in a clinical setting and may be more representative of the general population of those treated for mood disorders than those who have participated in clinical research studies, especially clinical trials. Patients had relatively low levels of mood symptoms and the results may not generalize to more severe presentations, although the majority of the Persons et al. sample were inpatients on intake. The full use of the longitudinal data with generalized linear mixed models that assessed the impact of mood symptoms on suicidal ideation and behavior at any point in time within patients was a notable strength of the study

This analysis does not suggest any additional suicide risk of mixed depressive and manic symptoms beyond the contribution of the depressive and manic symptoms themselves, with depressive symptoms standing out unequivocally as the primary syndromal influence on risk of suicidal thoughts of behavior. Our findings suggest women are at greater risk of suicidal ideation and behavior with mania and future study is indicated to better understand this potential moderator of suicide risk with mania. With continued growth of the network, the NNDC registry may be a useful resource to later confirm this moderator as well as to reassess the previously reported moderation of risk with mania by bipolar subtype. In the meantime, clinicians should heed the importance of depressive symptoms in assessments of suicide risk and aggressively target depressive symptoms as a potentially modifiable risk factor for suicide.

HIGHLIGHTS.

  • Depressive symptoms are a strong and robust predictor of concomitant suicidal ideation and behavior in bipolar disorder.

  • Mixed depressive and manic symptoms appear to convey no greater risk for suicidal ideation and behavior than that conveyed by the co-occurring depressive symptomatology.

  • Suicidal ideation and behavior was significantly more frequent in women with manic symptoms than in men with manic symptoms.

  • Clinicians should pay particular attention to depressive symptoms and syndromes in bipolar disorder for the purpose of suicide risk assessment and management.

Acknowledgements

This work was made possible by a research collaboration supported by the National Network of Depression Centers (NNDC), an inter-dependent nonprofit consortium of 26 academic depression centers. The CCR was also supported through CTSA grant award UL1TR000422 to the Michigan Institute for Clinical Health Research. Thanks to NNDC Executive Director Pat Rinvelt, MBA for her assistance with data sharing, coordination, and editing. Thanks to Caleb Adler and Wendy Marsh for feedback on the statistical analysis plan.

Conflict of Interest

Jess Fiedorowicz is supported by the National Heart Lung and Blood Institute (P01HL014388), the Institute for Clinical and Translational Science at the University of Iowa (U54TR001356), and Myriad Genetics, Inc. (grant and consulting). Shervin Assari is partially supported by the Heinz C. Prechter Bipolar Research Fund and the Richard Tam Foundation at the University of Michigan Depression Center. Dr. Ostacher is a full-time employee of the Department of Veterans Affairs. He reports being advisory/consultant to Acadia, Eli Lily, Johnson & Johnson, Lundbeck, Otsuka, Sunovion, and Supernus Pharmaceuticals, and has received grant support from Palo Alto Health Sciences and the National Institute on Drug Abuse in the past three years. Dr. Thase has no disclosures pertaining to this research. He does report the following other relationships during the past three years. He was an advisory/consultant to Acadia, Alkermes, Allergan (Forest, Naurex), AstraZeneca, Cerecor, Eli Lilly, Johnson & Johnson (Janssen, Ortho-McNeil), Lundbeck, MedAvante, Merck, Mocksha8, Nestlé (PamLab), Neuronetics, Novartis, Otsuka, Pfizer, Shire, Sunovion, and Takeda. He has received grant support from the National Institute of Mental Health, the Agency for Healthcare Research and Quality, the Patient Centered Outcomes Research Institute, Acadia, Alkermes, Assurex, Avanir, Forest Pharmaceuticals, Johnson & Johnson, Otsuka Pharmaceuticals, and Takeda. Dr. Thase received royalties from the American Psychiatric Press, Guilford Publications, Herald House and W.W. Norton & Company, Inc. Dr. Thase’s spouse, Dr. Diane Sloan, works for Peloton Advantage, which does business with Pfizer and AstraZeneca. Dr. Mark Frye also has no disclosures pertaining to this research. He has received grant support from AssureRx, Mayo Foundation, Myriad, National Institute of Alcohol Abuse and Alcoholism, National Institute of Mental Health, and Pfizer. He has served as a consultant (Mayo) to Janssen, Mitsubishi Tanabe Pharma Corporation, Myriad, Neuralstem, Inc., Otsuka American Pharmaceutical, Sunovion, and Teva Pharmaceuticals. All other authors have no potential conflicts of interest to disclose.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  1. Altman E, Hedeker D, Peterson JL and Davis JM (2001). “A comparative evaluation of three selfrating scales for acute mania.” Biol Psychiatry 50(6): 468–471. [DOI] [PubMed] [Google Scholar]
  2. Altman EG, Hedeker D, Peterson JL and Davis JM (1997). “The Altman Self-Rating Mania Scale.” Biol Psychiatry 42(10): 948–955. [DOI] [PubMed] [Google Scholar]
  3. American Psychiatric Association (2013). Diagnostic and statistical manual of mental disorders: DSM-5. Washington, D.C., American Psychiatric Association. [Google Scholar]
  4. Baca-Garcia E, Perez-Rodriguez MM, Oquendo MA, Keyes KM, Hasin DS, Grant BF and Blanco C (2011). “Estimating risk for suicide attempt: are we asking the right questions? Passive suicidal ideation as a marker for suicidal behavior.” J Affect Disord 134(1–3): 327–332. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Balazs J, Benazzi F, Rihmer Z, Rihmer A, Akiskal KK and Akiskal HS (2006). “The close link between suicide attempts and mixed (bipolar) depression: implications for suicide prevention.” J Affect Disord 91(2–3): 133–138. [DOI] [PubMed] [Google Scholar]
  6. Baldassano CF, Marangell LB, Gyulai L, Ghaemi SN, Joffe H, Kim DR, Sagduyu K, Truman CJ, Wisniewski SR, Sachs GS and Cohen LS (2005). “Gender differences in bipolar disorder: retrospective data from the first 500 STEP-BD participants.” Bipolar Disord 7(5): 465–470. [DOI] [PubMed] [Google Scholar]
  7. Cassidy F, Murry E, Forest K and Carroll BJ (1998). “Signs and symptoms of mania in pure and mixed episodes.” J Affect Disord 50(2–3): 187–201. [DOI] [PubMed] [Google Scholar]
  8. Conway PM, Erlangsen A, Teasdale TW, Jakobsen IS and Larsen KJ (2016). “Predictive Validity of the Columbia-Suicide Severity Rating Scale for Short-Term Suicidal Behavior: A Danish Study of Adolescents at a High Risk of Suicide.” Arch Suicide Res: 1–15. [DOI] [PubMed] [Google Scholar]
  9. Coryell W, Fiedorowicz J, Leon AC, Endicott J and Keller MB (2013). “Age of onset and the prospectively observed course of illness in bipolar disorder.” J Affect Disord 146(1): 34–38. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Dell’Osso L, Pini S, Cassano GB, Mastrocinque C, Seckinger RA, Saettoni M, Papasogli A, Yale SA and Amador XF (2002). “Insight into illness in patients with mania, mixed mania, bipolar depression and major depression with psychotic features.” Bipolar Disord 4(5): 315–322. [DOI] [PubMed] [Google Scholar]
  11. Dias VV, Brissos S, Frey BN and Kapczinski F (2008). “Insight, quality of life and cognitive functioning in euthymic patients with bipolar disorder.” J Affect Disord 110(1–2): 75–83. [DOI] [PubMed] [Google Scholar]
  12. Goldberg JF, Perlis RH, Bowden CL, Thase ME, Miklowitz DJ, Marangell LB, Calabrese JR, Nierenberg AA and Sachs GS (2009). “Manic symptoms during depressive episodes in 1,380 patients with bipolar disorder: findings from the STEP-BD.” Am J Psychiatry 166(2): 173–181. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Gonzalez-Pinto A, Barbeito S, Alonso M, Alberich S, Haidar MK, Vieta E, Tabares-Seisdedos R, Zorrilla I, Gonzalez-Pinto MA and Lopez P (2011). “Poor long-term prognosis in mixed bipolar patients: 10-year outcomes in the Vitoria prospective naturalistic study in Spain.” J Clin Psychiatry 72(5): 671–676. [DOI] [PubMed] [Google Scholar]
  14. Greist JH, Mundt JC, Gwaltney CJ, Jefferson JW and Posner K (2014). “Predictive Value of Baseline Electronic Columbia-Suicide Severity Rating Scale (eC-SSRS) Assessments for Identifying Risk of Prospective Reports of Suicidal Behavior During Research Participation.” Innov Clin Neurosci 11(9–10): 23–31. [PMC free article] [PubMed] [Google Scholar]
  15. Johnson SL, McMurrich SL and Yates M (2005). “Suicidality in bipolar I disorder.” Suicide Life Threat Behav 35(6): 681–689. [DOI] [PubMed] [Google Scholar]
  16. Judd LL, Schettler PJ, Akiskal HS, Maser J, Coryell W, Solomon D, Endicott J and Keller M (2003). “Long-term symptomatic status of bipolar I vs. bipolar II disorders.” Int J Neuropsychopharmacol 6(2): 127–137. [DOI] [PubMed] [Google Scholar]
  17. Judd LL, Schettler PJ, Solomon DA, Maser JD, Coryell W, Endicott J and Akiskal HS (2008). “Psychosocial disability and work role function compared across the long-term course of bipolar I, bipolar II and unipolar major depressive disorders.” J Affect Disord 108(1–2): 49–58. [DOI] [PubMed] [Google Scholar]
  18. Kroenke K, Spitzer RL and Williams JB (2001). “The PHQ-9: validity of a brief depression severity measure.” J Gen Intern Med 16(9): 606–613. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Leon AC and Heo M (2009). “Sample Sizes Required to Detect Interactions between Two Binary Fixed-Effects in a Mixed-Effects Linear Regression Model.” Comput Stat Data Anal 53(3): 603–608. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Merikangas KR, Akiskal HS, Angst J, Greenberg PE, Hirschfeld RM, Petukhova M and Kessler RC (2007). “Lifetime and 12-month prevalence of bipolar spectrum disorder in the National Comorbidity Survey replication.” Arch Gen Psychiatry 64(5): 543–552. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Miller S, Suppes T, Mintz J, Hellemann G, Frye MA, McElroy SL, Nolen WA, Kupka R, Leverich GS, Grunze H, Altshuler LL, Keck PE Jr. and Post RM (2016). “Mixed Depression in Bipolar Disorder: Prevalence Rate and Clinical Correlates During Naturalistic Follow-Up in the Stanley Bipolar Network.” Am J Psychiatry 173(10): 1015–1023. [DOI] [PubMed] [Google Scholar]
  22. Moriarty AS, Gilbody S, McMillan D and Manea L (2015). “Screening and case finding for major depressive disorder using the Patient Health Questionnaire (PHQ-9): a meta-analysis.” Gen Hosp Psychiatry 37(6): 567–576. [DOI] [PubMed] [Google Scholar]
  23. Ostacher MJ, Nierenberg AA, Rabideau D, Reilly-Harrington NA, Sylvia LG, Gold AK, Shesler LW, Ketter TA, Bowden CL, Calabrese JR, Friedman ES, Iosifescu DV, Thase ME, Leon AC and Trivedi MH (2015). “A clinical measure of suicidal ideation, suicidal behavior, and associated symptoms in bipolar disorder: Psychometric properties of the Concise Health Risk Tracking Self-Report (CHRT-SR).” J Psychiatr Res 71: 126–133. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Pacchiarotti I, Mazzarini L, Kotzalidis GD, Valenti M, Nivoli AM, Sani G, Torrent C, Murru A, Sanchez-Moreno J, Patrizi B, Girardi P, Vieta E and Colom F (2011). “Mania and depression. Mixed, not stirred.” J Affect Disord 133(1–2): 105–113. [DOI] [PubMed] [Google Scholar]
  25. Perlis ML, Grandner MA, Chakravorty S, Bernert RA, Brown GK and Thase ME (2016). “Suicide and sleep: Is it a bad thing to be awake when reason sleeps?” Sleep Med Rev 29: 101–107. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Perlis RH, Ostacher MJ, Goldberg JF, Miklowitz DJ, Friedman E, Calabrese J, Thase ME and Sachs GS (2010). “Transition to mania during treatment of bipolar depression.” Neuropsychopharmacology 35(13): 2545–2552. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Persons JE, Coryell WH, Solomon DA, Keller MB, Endicott J and Fiedorowicz JG (2018). “Mixed state and suicide: Is the effect of mixed state on suicidal behavior more than the sum of its parts?” Bipolar Disord 20(1): 35–41. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Persons JE, Coryell WH, Solomon DA, Keller MB, Endicott J and Fiedorowicz JG (In Press). “Mixed state and suicide: Is the effect of mixed sate on suicidal behavior more than teh sum of its parts?” Bipolar Disorders. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Posner K, Brown GK, Stanley B, Brent DA, Yershova KV, Oquendo MA, Currier GW, Melvin GA, Greenhill L, Shen S and Mann JJ (2011). “The Columbia-Suicide Severity Rating Scale: initial validity and internal consistency findings from three multisite studies with adolescents and adults.” Am J Psychiatry 168(12): 1266–1277. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Rihmer A, Gonda X, Balazs J and Faludi G (2008). “The importance of depressive mixed states in suicidal behaviour.” Neuropsychopharmacol Hung 10(1): 45–49. [PubMed] [Google Scholar]
  31. Saunders EF, Fernandez-Mendoza J, Kamali M, Assari S and McInnis MG (2015). “The effect of poor sleep quality on mood outcome differs between men and women: A longitudinal study of bipolar disorder.” J Affect Disord 180: 90–96. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Sherman RE, Anderson SA, Dal Pan GJ, Gray GW, Gross T, Hunter NL, LaVange L, Marinac-Dabic D, Marks PW, Robb MA, Shuren J, Temple R, Woodcock J, Yue LQ and Califf RM (2016). “Real-World Evidence - What Is It and What Can It Tell Us?” N Engl J Med 375(23): 2293–2297. [DOI] [PubMed] [Google Scholar]
  33. Shim IH, Woo YS and Bahk WM (2015). “Prevalence rates and clinical implications of bipolar disorder “with mixed features” as defined by DSM-5.” J Affect Disord 173: 120–125. [DOI] [PubMed] [Google Scholar]
  34. Spitzer RL, Kroenke K, Williams JB and Lowe B (2006). “A brief measure for assessing generalized anxiety disorder: the GAD-7.” Arch Intern Med 166(10): 1092–1097. [DOI] [PubMed] [Google Scholar]
  35. Strakowski SM, McElroy SL, Keck PE Jr. and West SA (1996). “Suicidality among patients with mixed and manic bipolar disorder.” Am J Psychiatry 153(5): 674–676. [DOI] [PubMed] [Google Scholar]
  36. Suppes T, Mintz J, McElroy SL, Altshuler LL, Kupka RW, Frye MA, Keck PE Jr., Nolen WA, Leverich GS, Grunze H, Rush AJ and Post RM (2005). “Mixed hypomania in 908 patients with bipolar disorder evaluated prospectively in the Stanley Foundation Bipolar Treatment Network: a sex-specific phenomenon.” Arch Gen Psychiatry 62(10): 1089–1096. [DOI] [PubMed] [Google Scholar]
  37. Swann AC, Lafer B, Perugi G, Frye MA, Bauer M, Bahk WM, Scott J, Ha K and Suppes T (2013). “Bipolar mixed states: an international society for bipolar disorders task force report of symptom structure, course of illness, and diagnosis.” Am J Psychiatry 170(1): 31–42. [DOI] [PubMed] [Google Scholar]
  38. Swann AC, Steinberg JL, Lijffijt M and Moeller GF (2009). “Continuum of depressive and manic mixed states in patients with bipolar disorder: quantitative measurement and clinical features.” World Psychiatry 8(3): 166–172. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Szanto K, Reynolds CF 3rd, Frank E, Stack J, Fasiczka AL, Miller M, Mulsant BH, Mazumdar S and Kupfer DJ (1996). “Suicide in Elderly Depressed Patients: Is Active vs. Passive Suicidal Ideation a Clinically Valid Distinction?” Am J Geriatr Psychiatry 4(3): 197–207. [DOI] [PubMed] [Google Scholar]
  40. Valtonen HM, Suominen K, Haukka J, Mantere O, Leppamaki S, Arvilommi P and Isometsa ET (2008). “Differences in incidence of suicide attempts during phases of bipolar I and II disorders.” Bipolar Disord 10(5): 588–596. [DOI] [PubMed] [Google Scholar]
  41. Valtonen HM, Suominen K, Mantere O, Leppamaki S, Arvilommi P and Isometsa E (2007). “Suicidal behaviour during different phases of bipolar disorder.” J Affect Disord 97(1–3): 101–107. [DOI] [PubMed] [Google Scholar]
  42. Wortzel HS, Gutierrez PM, Homaifar BY, Breshears RE and Harwood JE (2010). “Surrogate endpoints in suicide research.” Suicide Life Threat Behav 40(5): 500–505. [DOI] [PubMed] [Google Scholar]
  43. Zimmerman M and Mattia JI (1999). “Psychiatric diagnosis in clinical practice: is comorbidity being missed?” Compr Psychiatry 40(3): 182–191. [DOI] [PubMed] [Google Scholar]

RESOURCES