Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2017 Mar 15.
Published in final edited form as: J Affect Disord. 2016 Jan 1;193:109–116. doi: 10.1016/j.jad.2015.12.066

Identifying a clinical signature of suicidality among patients with mood disorders: a pilot study using a machine learning approach

Ives Cavalcante Passos 1,2,*, Benson Mwangi 1,*, Bo Cao 1, Jane E Hamilton 1, Mon-Ju Wu 1, Xiang Yang Zhang 1,3, Giovana B Zunta-Soares 1, Joao Quevedo 1, Marcia Kauer-Sant'Anna 2, Flávio Kapczinski 2, Jair C Soares 1
PMCID: PMC4744514  NIHMSID: NIHMS748871  PMID: 26773901

Abstract

Objective

A growing body of evidence has put forward clinical risk factors associated with patients with mood disorders that attempt suicide. However, what is not known is how to integrate clinical variables into a clinically useful tool in order to estimate the probability of an individual patient attempting suicide.

Method

A total of 144 patients with mood disorders were included. Clinical variables associated with suicide attempts among patients with mood disorders and demographic variables were used to ‘train’ a machine learning algorithm. The resulting algorithm was utilized in identifying novel or ‘unseen’ individual subjects as either suicide attempters or non-attempters. Three machine learning algorithms were implemented and evaluated.

Results

All algorithms distinguished individual suicide attempters from non-attempters with prediction accuracy ranging between 65%-72% (p<0.05). In particular, the relevance vector machine (RVM) algorithm correctly predicted 103 out of 144 subjects translating into 72% accuracy (72.1% sensitivity and 71.3% specificity) and an area under the curve of 0.77 (p<0.0001). The most relevant predictor variables in distinguishing attempters from non-attempters included previous hospitalizations for depression, a history of psychosis, cocaine dependence and post-traumatic stress disorder (PTSD) comorbidity.

Conclusion

Risk for suicide attempt among patients with mood disorders can be estimated at an individual subject level by incorporating both demographic and clinical variables. Future studies should examine the performance of this model in other populations and its subsequent utility in facilitating selection of interventions to prevent suicide.

Keywords: suicide, bipolar disorder, depression, machine learning, big data, personalized medicine

Introduction

When a patient with a depressive episode and no known previous history of suicide attempts first presents for treatment, what is the likelihood that this individual patient will attempt suicide? This question is a major unmet challenge in mood disorder treatment and also an important question for society at large, as suicide is a tragic but highly preventable event (Mann et al., 2005). Suicide accounted for 4.8% and 5.7% of total global deaths in female and male subjects aged 15-49 years respectively in 2010 (Lozano et al., 2012). In the same year, suicide was reported as the sixth leading cause of life years lost in North-America (Lozano et al., 2012). Furthermore, it has been reported that approximately 90% of individuals who die through suicide are diagnosed with a mental disorder prior to their death (Arsenault-Lapierre et al., 2004). However, there is little awareness in medical practice on objective stratification of suicide risk, which has led suicide to be referred to as “the quiet epidemic” (Turecki, 2014). The current situation is particularly worrisome in major depressive disorder (MDD) and bipolar disorder (BD), given the high prevalence of these disorders and the strong association between suicide and depressive symptoms.

Suicide is usually viewed as an extreme response to a catastrophic event, such as loss of a close relative (M A Oquendo et al., 2014). However, many individuals, including patients with mood disorders, go through these kinds of stressors and yet they do not attempt suicide (Maria A Oquendo et al., 2014). Consequently, a growing body of knowledge has put forward several risk factors associated with patients that attempt suicide (Brundin et al., 2015; Mann and Currier, 2010; Maria A Oquendo et al., 2014; Turecki, 2014). They are developmental and distal factors (as opposed to precipitating factors, such as loss of a close relative) associated to the suicidal behavior (Turecki, 2014; Turecki et al., 2012). These efforts have largely reported average group-level differences between suicide attempters and non-attempters. However, what is not known is how to integrate these variables to build a signature of suicide attempt and estimate the probability of an individual patient with mood disorder attempting suicide while faced with a stressful event.

In this study we set out to investigate whether a predictive clinical signature derived from easily accessible clinical and demographic variables can objectively identify individual patients likely to attempt suicide. Predictor variables were selected using a priori knowledge. Notably, the majority of clinical variables selected were related to psychiatric comorbidities (Almeida et al., 2012; Bhui et al., 2012; Blackmore et al., 2008; Blanco et al., 2012; Bostwick and Pankratz, 2000; Bottlender et al., 2000; de Araújo et al., 2015; Foley et al., 2006; Galfalvy et al., 2006; Goes et al., 2012; Goldstein et al., 2012; Gonzalez-Pinto et al., 2006; Holma et al., 2014; Isometsä et al., 2014; Johnson et al., 1991; Katz et al., 2011; Lenze et al., 2000; Morina et al., 2013; Neves et al., 2009; Oquendo et al., 2010; Schaffer et al., 2014a, 2014b; Simon et al., 2004; Soloff et al., 2000; Stein, 2006; Torres et al., 2011; Webb et al., 2014), given the recent findings that effects of mental disorders on the risk of suicide attempt were exerted almost exclusively through a general psychopathology factor representing the shared effect across all mental disorders (Hoertel et al., 2015). As a result, we implemented a set of algorithms that are able to integrate the information from multiple variables to subsequently identify an individual patient's probability or risk of being a suicide attempter.

Machine learning in psychiatric research is an emerging field with a great potential for innovation and paradigm shift as these algorithms facilitate integration of multiple measurements as well as allows objective predictions of previously ‘unseen’ observations. Typically, these algorithms are implemented in three key stages. First, subjects' data are separated into algorithm ‘training’ and ‘testing’ sets with the former being used to ‘train’ a machine learning model. Second, the predictive accuracy of the ‘learnt’ model is evaluated using the ‘testing’ set that consists of observations previously ‘unseen’ by the model, and algorithm accuracy, sensitivity and specificity reported. In this study, we implemented three machine learning algorithms namely; least absolute shrinkage and selection operator (LASSO) (Robert Tibshirani, 2011), support vector machines (SVM) (Vapnik, 1998) and relevance vector machine (RVM) (Tipping, 2001). The LASSO technique was implemented using a generalized linear model (GLM) with a LASSO regularization penalty. A detailed discussion of these algorithms in the context of psychiatric research is given elsewhere (Mwangi et al., 2012).

In summary, the main objective of the current study was to establish a clinically useful predictive signature able to accurately determine individual patients' likely to committ suicide. We used easily accessible clinical variables to achieve our aim. Of note, this is a proof-of-concept study due to a small sample size.

Methods

The Institutional Review Board of the University of Texas Health Science Center at Houston approved the study. All subjects signed informed consent before any study-related procedures with ample time for questions. Participants included in the current study were recruited from January 2006 to June 2010.

Participants

A total of 144 subjects were recruited from the community and psychiatric clinics through flyers, radio, and newspaper advertisements. Inclusion criteria were subjects with MDD or BD types I or II (DSM-IV), and aged between 18 and 65. Exclusion criteria were head trauma with residual effects, neurological disorder, and uncontrolled major medical conditions.

Assessments

Subjects were evaluated using a socio-demographic history form to assess age, gender, years of education, and occupational status. Axis-I diagnoses and clinical characteristics were assessed with the Structured Clinical Interview for DSM-IV axis-I Disorders (SCID-I), which was administered by fully trained staff. The criterion for suicidality was one or more documented actual suicide attempts assessed by the suicide history form from the Conte Center for the Neuroscience of Mental Disorders (New York State Psychiatric Institute, Columbia University). Current dimensional mood and anxiety symptoms were assessed with the Hamilton Depression Rating Scale (HDRS), the Young Mania Rating Scale (YMRS), and Hamilton Anxiety Rating Scale (HARS). Data from all instruments were collected regardless of treatment status.

Selection of predictor variables and machine learning

Selection of predictor variables to be utilized in ‘training’ an algorithm is a challenge in machine learning. However, a recommended method of selecting relevant predictor variables is the use of expert domain knowledge – largely from previously published literature (Mwangi et al., 2013; Perlis, 2013). We performed a structured search on Pubmed to find relevant studies that report association of clinical and demographic variables included in this study with suicide attempts (Table 1). Of note, we chose patients with mood disorders because a) the developmental variables associated with suicide in both BD and MDD are somewhat similar (see table 1); and b) our goal was to create a tool that could be used with the large number of patients with mood disorders treated in primary care settings (Table 1).

Table 1. Clinical and demographic variables selected to develop to ‘train’ the model.

Variable Type Study Results
Age Continuous variable. Gonzalez A et al., 2006 Suicidal risk was higher in young patients with BD.
Galfalvy H et al., 2006 Younger age predicted suicide attempts among patients with BD.
Perlis RH et al., 2007 Younger age was associated with suicide risk in patients with MDD.
Gender Categorical variable: 0 – male; 1 – female. Schaffer A et al., 2015 Female gender was associated with suicide attempts among patients with BD.
Schaffer A et al., 2014 Female gender was associated with suicide attempts among patients with BD.
Perlis RH et al., 2007 Female gender was associated with suicide risk among patients with MDD.
Race Categorical variable: 0 – Caucasian 1 – Non- Caucasian Bhui KS et al., 2012 Black African men have the highest rates of suicide compared to the white group.
Years of education Continuous variable De Araújo RM et al., 2015 Elementary school as highest education level completed was associated with suicide attempt.
Current employed Categorical variable: 0 – no; 1 – yes. Blackmore ER, 2008 Being unemployed was significantly associated with a suicidal act.
Mood Diagnosis Categorical variable: 0 – BD; 1 – MDD. Holma KM et al., 2014 Patients with BD had higher cumulative incidence of suicide attempts compared to those with MDD.
Bottlender R, 2000 Patients with BD had higher past suicide attempts compared to those with MDD.
Social Phobia Categorical variable: 0 – no; 1 – yes. Simon et al., 2004 Comorbid social phobia was associated with greater likelihood of suicide attempts in patients with BD.
Stein MB, 2006 Social anxiety disorder is associated with an increased risk for MDD and a more malignant course, characterized by increased likelihood of suicide attempts
Specific Phobia Categorical variable: 0 – no; 1 – yes. Goes FS et al., 2012 Subject with specific phobia was associated with increased suicide attempts among patients with BD.
Panic Disorder Categorical variable: 0 – no; 1 – yes. Simon et al., 2004 Comorbid panic disorder was associated with greater likelihood of suicide attempts in patients with BD.
Katz C et al., 2011 Comorbid panic disorder was associated with greater likelihood of suicide attempts in patients with MDD.
Generalized Anxiety Disorder Categorical variable: 0 – no; 1 – yes. Simon et al., 2004 Comorbid generalized anxiety disorder was associated with greater likelihood of suicide attempts in patients with BD.
Foley DL et al., 2006 Comorbid generalized anxiety disorder was associated with greater likelihood of suicide attempts in patients with MDD.
Lenze EJ et al., 2000 Comorbid generalized anxiety disorder was associated with greater likelihood of suicide attempts in depressed elderly patients.
Obsessive Compulsive Disorder Categorical variable: 0 – no; 1 – yes. Simon et al., 2004 Current obsessive compulsive disorder was associated with greate likelihood of suicide attempts in patients with BD.
Torres AR et al., 2011 MDD was associated with suicide attempts in patients with obsessive compulsive disorder.
Posttraumatic Stress Disorder Categorical variable: 0 – no; 1 – yes. Simon et al., 2004 Comorbid posttraumatic stress disorder was associated with greater likelihood of suicide attempts in patients with BD.
Morina N et al., 2013 Comorbid posttraumatic stress disorder was associated with greater likelihood of suicide attempts in patients with MDD.
Alcohol/Cocaine dependence Categorical variables: 0 – no; 1 – yes. Oquendo MA et al., 2010 Comorbid alcohol use disorders were associated with suicide risk among individuals with BD
Blanco C et al., 2012 Substance use disorder was associated with higher rates of suicide attempts in patients with MDD.
Webb RT et al., 2014 Comorbid alcohol or drug use disorders predicted suicide attempts among patients with BD.
Number of depressive episodes and priorhospitalization by depressive episodes Continuous variable. Goldstein TR et al., 2012 Severity and number of depressive symptoms was associated with suicide attempts among patients with BD
Isometsa et al., 2014 Patients with BD and prior hospitalization by depressive episode had increased suicide risk.
Gonzalez A et al., 2006 Number of depressive episodes and ospitalization was associated with suicidal risk in patients with BD.
Psychotic symptoms Categorical variable: 0 – no; 1 – yes. Johnson J et al 1991 Comorbid psychotic symptoms were associated with suicide attempts among patients with MDD.
Bottlender R et al., 200 Psychotic symptoms were associated with suicidality among patients with BD.
Open in a new tab

BD, bipolar disorder; MDD, major depressive disorder.

Three machine learning algorithms (LASSO, SVM, and RVM) were implemented in MATLAB (The Mathworks, Inc., Natick, Massachusetts, United States). Specifically, predictor variables (see Table 1) were normalized by z-scoring and together with corresponding categorical labels (1 – suicide attempt, 0 – non-attempt) input into the machine learning algorithm. Here, we briefly describe these machine learning algorithms but detailed and technical discussions are given elsewhere (Robert Tibshirani, 2011; Tipping, 2001; Vapnik, 1998). SVM and RVM are ‘kernel-learning’ algorithms which transform a set of input variables (feature vectors) into a ‘kernel’ or similarity matrix through a ‘kernel function’ and subsequently used in the algorithm ‘learning’ process. These algorithms can utilize multiple types of kernel functions (e.g. linear, polynomial, Gaussian) but in the current study a linear function was utilized in training both SVM and RVM algorithms as described elsewhere (Mwangi et al., 2012). In contrast, the LASSO utilizes a typical linear regression formulation but introduces a ‘penalization’ procedure by assigning some predictor variables zero coefficients or weighting factors - a process also referred to as regularization. The penalization or regularization process reportedly improves the linear regression algorithm prediction accuracy and generalization ability but entails identifying an optimal regularization parameter (Robert Tibshirani, 2011). In the current study, the regularization parameter was identified through a 10-fold cross-validation process using training data only.

Class imbalance problem

Class imbalance is a common problem in machine learning where observations (subjects) in one class (e.g. suicide non-attempters) exceed observations in the other class (e.g. suicide attempters). A typical machine learning algorithm trained using an imbalanced data set assigns new observations to the majority class (e.g. suicide non-attempters) (Dubey et al., 2014). In this study, the class imbalance problem was circumvented by ‘under-sampling’ the majority class (suicide non-attempters) followed by training an algorithm with a balanced sample – a process which was repeated until all observations in the majority class were selected at least once and predictions aggregated as shown in Figure 1.

Figure 1. Flow diagram showing algorithm training and testing process.

Figure 1

Open in a new tab

*The class imbalance problem was addressed through a resampling step, which entailed under-sampling the majority class (suicide non-attempters) followed by algorithm training. This process was repeated 1000 iterations and algorithm predicted probabilities averaged over the resampling iterations. Class imbalance is a recurrent challenge in machine learning as recently highlighted by Libbrecht and colleagues (53). A leave-one-out cross-validation approach was used to separate algorithm training and testing data.

Model testing and validation

A unique characteristic of machine learning algorithms is the ability to utilize a cross-validation approach to separate data into model ‘training’ and ‘testing’ sets. In this study, a leave-one-out cross-validation (LOOCV) approach was used. LOOCV involves training a machine learning algorithm with all subjects except one while the ‘left out’ subject is used for algorithm testing. This iterative process is repeated until all subjects are left out for algorithm testing at least once as shown in Figure 1 and described elsewhere (Johnston et al., 2014). Cross-validation is used in machine learning to establish the generalization ability of an algorithm to new or previously ‘unseen’ subjects. The validity of the algorithms in predicting individual suicide attempters from non-attempters was evaluated using prediction accuracy, sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV). In this study, sensitivity and specificity represented correctly predicted suicide attempters (true positives) and correctly predicted non-attempters (true negatives) respectively. A ‘classical’ prediction accuracy was calculated as the sum of true positives, true negatives and divided by the total sample. In contrast, to account for the class imbalance problem, a ‘balanced’ accuracy was calculated as the average of predicted sensitivity (true positives) and specificity (true negatives) as reported elsewhere (Feis et al., 2013). In addition, PPV was calculated as the proportion of subjects predicted as suicide attempters that were actual positives (attempters) and NPV as the proportion of subjects predicted as non-attempters but were actual non-attempters (true negatives). Receiver operating characteristic curves and the corresponding area under curve were computed and reported. Lastly, chi-square statistical test between actual and machine learning predicted labels were calculated and the results reported. The outcome of a chi-square test was considered significant at p<0.05.

Results

A total of 144 subjects with BD or MDD were included in the study. Table 2 summarizes subjects' clinical and demographic characteristics. All algorithms (RVM, SVM and LASSO) distinguished individual suicide attempters from non-attempters with prediction accuracy ranging (64.7%-72%) and all models were significant (chi-square p<0.05) as detailed in Table 3. In particular, the RVM algorithm correctly identified 103 out of 144 as either attempters or non-attempters. This translated into 72% accuracy (72.1% sensitivity and 71.3% specificity) and significant at χ2 p<0.0001. The RVM receiver operating characteristic (ROC) curve and the ‘confusion matrix’, which were used in calculating the sensitivity, specificity and area under ROC curve values, are shown in Figure 2. To calculate ‘concentration of risk’ – RVM predicted subjects' probabilities were divided into 20 groups of equal sizes (ventiles) and number of subjects correctly predicted within the top two ventiles calculated. In both top two ventiles, 5 out of 7 subjects were correctly predicted – translating into 71.4 % accuracy in both ventiles. Individual subjects' predicted probabilities are shown in Table S1 of supplementary materials.

Table 2. Demographic and clinical characteristics.

Attempters (n=43) Non-attempters (n= 101) p-value
Age (years) 35.26 ± 11.14 37.53 ± 12.96 0.317
Years of education 13.22 ± 3.56 14.55 ± 3.098 0.026
Gender
 Male 14 (32.5%) 39 (38.6%) 0.616
 Female 29 (67.5%) 62 (61.4%)
Currently employed 13 (30.2%) 50 (49.5%) 0.051
Race 0.037
 Caucasian 22 (51.2%) 72 (71.3%)
 African American 7 (16.2%) 6 (5.9%)
 Hispanic or Latino 14 (32.6%) 23 (22.8%)
Number of hospitalization 1.58 ± 1.8 0.29 ± 0.82 <0.001
Mood disorder diagnosis 0.014
 BD 40 (93%) 74 (73.2%)
 MDD 3 (7%) 27 (26.7%)
Comorbid anxiety disorders
 GAD 6 (13.9%) 14 (13.9%) 1.000
 OCD 4 (9.3%) 11 (10.9%) 1.000
 Social Phobia 7 (16.3%) 14 (13.9%) 0.905
 Panic Disorder 13 (30.2%) 18 (17.8%) 0.150
 PTSD 13 (30.2%) 16 (15.8%) 0.081
 Specific Phobia 5 (11.6%) 6 (5.9%) 0.404
Comorbid substance use disorder
 Cocaine 10 (23.3%) 9 (8.9%) 0.039
 Alcohol 23 (53.5%) 31 (30.7%) 0.016
Lifetime psychotic symptoms 18 (41.9%) 19 (18.8%) 0.007
Open in a new tab

Independent t-test was performed to numeric variables, while chi-square test with Yates correction was performed to categorical variables. BD, bipolar disorder; GAD, generalized anxiety disorder; MDD, major depressive disorder; OCD, obsessive-compulsive disorder; PTSD, posttraumatic stress disorder.

Table 3.

Algorithm performance in identifying individual suicide attempters from non-attempters among patients with mood disorders. The balanced accuracy (sensitivity + specificity/2) was used to identify the best performing algorithm due to the class imbalance problem.

Algorithm RVM SVM LASSO
Area Under Curve 0.77 0.65 0.73
Balanced accuracy (%) (Specificity+sensitivity/2) 72 64.7 68
Classical accuracy (%) 71.5 67.4 72.9
Specificity (%) 71.3 71.3 80.2
Sensitivity (%) 72.1 58.1 55.8
PPV (%) 51.7 46.3 54.6
NPV (%) 85.7 80.0 81.0
Open in a new tab

LASSO, least absolute selection shrinkage and selection operator; NPV, negative predictive value; PPV, positive predictive value; RVM, relevance vector machine; SVM, support vector machines.

Figure 2. ‘Confusion matrix’ and receiver operating characteristic (ROC) curve.

Figure 2

Open in a new tab

‘Confusion matrix’ and receiver operating characteristic (ROC) curve were used in computing the algorithm accuracy, sensitivity, specificity and area under the ROC curve. The RVM algorithm performed best as compared to SVM and LASSO.

The most relevant predictor variables in distinguishing suicide attempters from non-attempters that were assigned positive coefficients or weighting factors by RVM – indicating an increase in suicide attempters included; 1) a high number of previous hospitalizations for depression, 2) a history of psychosis, 3) cocaine dependence, and 4) PTSD. In contrast, patients’ age and mood diagnosis (0 - bipolar disorder or 1 - unipolar depression) were assigned negative weighting factors indicating a reduction of these variables was relevant in identifying suicide attempters. Figure 3 shows predictor variables with corresponding weighting factors.

Figure 3. Clinical signature.

Figure 3

Open in a new tab

Bar graph showing weighting factors assigned to eachclinical variable by RVM based on their relevance in distinguishing suicide attemptersfrom non-attempters. Clinical variables which increase the probability of an individual patient being a suicide attempter were assigned positive weighting factors whilst those that decrease the probability of a patient being a suicide attempt were assigned negative weighting factors. Variables related to anxiety and substance use disorder are lifetime comorbidities. PD, panic disorder; PTSD, posttraumatic stress disorder. Predictor variables weighting factors should be interpreted with caution as they are selected to collective maximize prediction accuracy as opposed to representing individual coefficient accuracy.

An additional RVM algorithm was evaluated by removing previous number of hospitalizations in the analysis and the algorithm was able to identify individual subjects with accuracy = 68.9%, sensitivity = 74%, specificity = 63%, p<0.05 and area under ROC curve =0.71. This is an indication that past hospitalization variable was not the only contributor to the algorithm performance.

Discussion

Although risk factors for suicide are widely known, synthesizing easily accessible clinical information to optimize suicide risk identification in subjects with mood disorders has been an elusive goal up to now. This is the first study to evaluate the feasibility of using a clinical tool developed with advanced machine learning algorithms to identifying an individual's risk of being a suicide attempter among patients with mood disorders. All algorithms achieved greater than chance (>50%) accuracy in distinguishing attempters from non-attempters. Most notably, the RVM algorithm achieved the best prediction accuracy 72% and an area under receiver operating characteristic curve of 0.77. Notably, the prediction accuracy, sensitivity and specificity of the machine learning algorithm in identifying previously ‘unseen’ suicide attempters was determined through a robust cross-validation approach. The main outcome of the machine learning algorithm was a probability score able to quantify the risk of a subject being a suicide attempter.

Most relevant predictor variables distinguishing suicide attempters from non-attempters by RVM included number of prior hospitalizations for depression, lifetime psychotic symptoms, and comorbidity with cocaine dependence or PTSD. Previous average group level studies showed that these variables are associated with suicide attempts in patients with BD. In this sense, a prospective study with 72 bipolar I disorder patients showed that suicidal risk was higher with more hospitalization (Gonzalez-Pinto et al., 2006), while two studies showed that psychotic symptoms were associated with suicidality in patients with MDD and BD (Bottlender et al., 2000; Johnson et al., 1991). In addition, a cross-sectional study with 500 patients with BD reported that patients with BD and comorbid PTSD presented more suicide attempts compared against patients with BD and without PTSD (Simon et al., 2004). Finally, a review suggested that there is cross-sensitization between cocaine use and number of mood episodes in patients with bipolar disorder leading to faster illness progression, which may potentially increase the number of suicide attempts (Post and Kalivas, 2013).

In contrast with other branches of medicine (e.g. cardiology, oncology) – the lack of risk stratification approaches in psychiatric clinical practice is notable (Kapczinski and Passos, 2015). Noteworthy risk stratification applications include, prediction of cardiac arrest risk, cancer screening, and prediction of mortality risk in intensive care units (Anothaisintawee et al., 2012; Lee et al., 1999). Indeed, in personalized medicine, risk stratification plays an important role in treatment planning from the time of diagnosis. For instance, in acute care of myocardial infarction efforts to develop treatment algorithms on the basis of clinical presentation are increasingly integrated in clinical practice (5).

One important question is how this suicide risk stratification tool can be applied in a clinical practice to optimize individual patient treatment? For example, it is possible that individual patients with a high risk of attempting suicide as identified by the algorithm would benefit from preventive strategies or antisuicidal drugs. For instance, a randomized clinical trial of suicidal patients reported that a selective serotonin-reuptake inhibitor led to a greater acute improvement in suicidal ideation compared to a noradrenergic-dopaminergic drug. Notably, this therapeutic advantage was greater in patients with the most severe suicidal ideation (Grunebaum et al., 2012). Furthermore, evidence from meta-analysis reported that lithium reduces suicide attempts (Cipriani et al., 2013). In addition, clozapine recently emerged as one option for treatment-resistant patients with BD and also showed a protective effect against suicidal behavior compared with olanzapine (Meltzer et al., 2003). These studies further highlight the need to develop novel suicide stratification algorithms as presented in this study and allow individualized treatments and care for patients predicted with a high risk of attempting suicide.

The current study has some potential limitations. The study sample is small and fromthe Texas region which may not fully represent the entire United States population. This mayexplain the reason why comorbid generalized anxiety disorder (GAD) and obsessive-compulsive disorder (OCD) findings were not fully in agreement with previous studies.However, multivariate machine learning techniques may have found unexpected interactionsamong predictive variables and therefore unexpected results on GAD and OCD. Therefore,future studies should confirm or reject these results. In this sense, the current study serves as aproof-of-concept and future longitudinal studies should collect larger samples from multiplecenters to replicate current findings. In line with our hypothesis, our findings are related topatients with mood disorder and may not be generalized to patient with other mentaldisorders. However, future studies may utilize a similar approach to accommodate othermental disorders. Of note, borderline personality disorder that is also shown to be associatedwith suicide attempts in patients with mood disorders (Schaffer et al., 2014a) was notincluded in the algorithm due to missing data in our database. Notwithstanding theselimitations, this study is the first to describe and implement a risk stratification tool usingmachine learning algorithms to identify an individual's probability of attempting suicideamong patients with mood disorders. This algorithm could be used in a primary care settingby general practitioners to identify the risk of a mood disorder patient attempting suicide.Future work will focus on validating the algorithm using a larger sample as well asdeveloping a web-based suicide risk calculator. Importantly, the strengths of our studyinclude the application and testing of a multivariate machine learning algorithm in suicidestratification leading to a high area under the ROC curve of 0.77. Notably, a recent studyreported that most breast cancer prediction algorithms, including the widely-studied ‘Gailscore’ report area under ROC curves below .70 (Anothaisintawee et al., 2012). Indeed, machine learning is a fascinating field that may change the traditional doctor-patient relationship in psychiatry. We hypothesize that highly accurate predictive models will support important clinical decisions such as selection of treatment options, preventive strategies, and prognosis orientations. For instance, several studies recently utilized machine learning techniques to predict treatment response using functional brain scans and neurocognitive data (Hahn et al., 2015; Johnston et al., 2015; van Waarde et al., 2015). Moreover, a recent study reported a prediction model able to estimate the risk of renal failure in patients with BD treated with lithium (Castro et al., 2015). Therefore, future studies should develop web-based calculators and medical devices based on validated models to allow translation of these predictive models into clinical practice.

In summary, we report a highly accurate algorithm able to identify suicide attempts in patients with mood disorders using clinical and demographic data. These results suggest that it is possible to utilize clinical measures in identifying individual patients at greater risk of attempting suicide. Lastly, we hypothesize that the combination of genetics, blood markers, neuroimaging, and the clinical signature together with a machine learning algorithm could generate an even more accurate multi-modal signature.

Supplementary Material

supplement

Highlights.

  • We estimated risk for suicide attempt among patients with mood disorders

  • Prior hospitalizations, history of psychosis, cocaine use and PTSD comorbidities were the most relevant predictor variables

  • Risk for suicide attempt can be estimated at an individual subject level

  • Risk for suicide attempt can be estimated using demographic and clinical variables

Acknowledgments

Our study was supported in part by grants NIH grant MH68766, NIMH grant R01 085667, NIH grant RR2057, and the John S. Dunn Foundation from United States. Dr Jair C. Soares received these grants. ICP was supported by scholarship from Coordenacao de Aperfeicoamento de Pessoal de Nivel Superior (CAPES) and JQ, MKS, and FK were supported by Conselho Nacional de Desenvolvimento (CNPq).

Role of Funding Source: The funders had no role in the design, data collection, data management, data analysis and data interpretation.

Footnotes

Contributors: Dr Passos participated in the study design, data collection, data analysis, interpretation of findings, literature search, writing, implementation and approval of final manuscript. Dr Mwangi participated in the study design, writing, figures, tables, and approval of final manuscript. Dr Cao participated in the study design, literature search, figures, tables, and approval of final manuscript. Dr Hamilton participated in the literature search, writing, and approval of final manuscript. Dr Wu participated in literature search, figures, tables, and approval of final manuscript. Dr Zhang participated in the literature search, writing and approval of final manuscript. Dr Zunta-Soares participated in literature search, writing, and approval of final manuscript. Dr Quevedo participated in the study design, interpretation of findings, writing, and approval of final manuscript. Dr Kauer-Sant'Anna participated in the study design, interpretation of findings, writing, and approval of final manuscript. Dr Kapczinski participated in the study design, data analysis, interpretation of findings, literature search, writing, and approval of final manuscript. Dr Soares participated in the study design, data analysis, interpretation of findings, literature search, writing, and approval of final manuscript.

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  1. Almeida OP, Draper B, Snowdon J, Lautenschlager NT, Pirkis J, Byrne G, Sim M, Stocks N, Flicker L, Pfaff JJ. Factors associated with suicidal thoughts in a large community study of older adults. Br J Psychiatry. 2012;201:466–72. doi: 10.1192/bjp.bp.112.110130. [DOI] [PubMed] [Google Scholar]
  2. Anothaisintawee T, Teerawattananon Y, Wiratkapun C, Kasamesup V, Thakkinstian A. Risk prediction models of breast cancer: a systematic review of model performances. Breast Cancer Res Treat. 2012;133:1–10. doi: 10.1007/s10549-011-1853-z. [DOI] [PubMed] [Google Scholar]
  3. Arsenault-Lapierre G, Kim C, Turecki G. Psychiatric diagnoses in 3275 suicides: a meta-analysis. BMC Psychiatry. 2004;4:37. doi: 10.1186/1471-244X-4-37. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Bhui KS, Dinos S, McKenzie K. Ethnicity and its influence on suicide rates and risk. Ethn Health. 2012;17:141–8. doi: 10.1080/13557858.2011.645151. [DOI] [PubMed] [Google Scholar]
  5. Blackmore ER, Munce S, Weller I, Zagorski B, Stansfeld SA, Stewart DE, Caine ED, Conwell Y. Psychosocial and clinical correlates of suicidal acts: results from a national population survey. Br J Psychiatry. 2008;192:279–84. doi: 10.1192/bjp.bp.107.037382. [DOI] [PubMed] [Google Scholar]
  6. Blanco C, Alegría AA, Liu SM, Secades-Villa R, Sugaya L, Davies C, Nunes EV. Differences among major depressive disorder with and without co-occurring substance use disorders and substance-induced depressive disorder: results from the National Epidemiologic Survey on Alcohol and Related Conditions. J Clin Psychiatry. 2012;73:865–73. doi: 10.4088/JCP.10m06673. [DOI] [PubMed] [Google Scholar]
  7. Bostwick JM, Pankratz VS. Affective disorders and suicide risk: a reexamination. Am J Psychiatry. 2000;157:1925–32. doi: 10.1176/appi.ajp.157.12.1925. [DOI] [PubMed] [Google Scholar]
  8. Bottlender R, Jäger M, Strauss A, Möller HJ. Suicidality in bipolar compared to unipolar depressed inpatients. Eur Arch Psychiatry Clin Neurosci. 2000;250:257–61. doi: 10.1007/s004060070016. [DOI] [PubMed] [Google Scholar]
  9. Brundin L, Erhardt S, Bryleva EY, Achtyes ED, Postolache TT. The role of inflammation in suicidal behaviour. Acta Psychiatr Scand. 2015;132:192–203. doi: 10.1111/acps.12458. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Castro VM, Roberson AM, McCoy TH, Wiste A, Cagan A, Smoller JW, Rosenbaum JF, Ostacher M, Perlis RH. Stratifying Risk for Renal Insufficiency Among Lithium-Treated Patients: An Electronic Health Record Study. Neuropsychopharmacology. 2015 doi: 10.1038/npp.2015.254. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Cipriani A, Hawton K, Stockton S, Geddes JR. Lithium in the prevention of suicide in mood disorders: updated systematic review and meta-analysis. BMJ. 2013;346:f3646. doi: 10.1136/bmj.f3646. [DOI] [PubMed] [Google Scholar]
  12. De Araújo RMF, Mazzochi L, Lara DR, Ottoni GL. Thinking about dying and trying and intending to die: results on suicidal behavior from a large Web-based sample. J Clin Psychiatry. 2015;76:e359–65. doi: 10.4088/JCP.13m08864. [DOI] [PubMed] [Google Scholar]
  13. Dubey R, Zhou J, Wang Y, Thompson PM, Ye J. Analysis of sampling techniques for imbalanced data: An n = 648 ADNI study. Neuroimage. 2014;87:220–41. doi: 10.1016/j.neuroimage.2013.10.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Feis DL, Schoene-Bake JC, Elger C, Wagner J, Tittgemeyer M, Weber B. Prediction of post-surgical seizure outcome in left mesial temporal lobe epilepsy. NeuroImage Clin. 2013;2:903–11. doi: 10.1016/j.nicl.2013.06.010. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Foley DL, Goldston DB, Costello EJ, Angold A. Proximal psychiatric risk factors for suicidality in youth: the Great Smoky Mountains Study. Arch Gen Psychiatry. 2006;63:1017–24. doi: 10.1001/archpsyc.63.9.1017. [DOI] [PubMed] [Google Scholar]
  16. Galfalvy H, Oquendo MA, Carballo JJ, Sher L, Grunebaum MF, Burke A, Mann JJ. Clinical predictors of suicidal acts after major depression in bipolar disorder: a prospective study. Bipolar Disord. 2006;8:586–95. doi: 10.1111/j.1399-5618.2006.00340.x. [DOI] [PubMed] [Google Scholar]
  17. Goes FS, McCusker MG, Bienvenu OJ, Mackinnon DF, Mondimore FM, Schweizer B, Depaulo JR, Potash JB. Co-morbid anxiety disorders in bipolar disorder and major depression: familial aggregation and clinical characteristics of co- morbid panic disorder, social phobia, specific phobia and obsessive-compulsive disorder. Psychol Med. 2012;42:1449–59. doi: 10.1017/S0033291711002637. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Goldstein TR, Ha W, Axelson DA, Goldstein BI, Liao F, Gill MK, Ryan ND, Yen S, Hunt J, Hower H, Keller M, Strober M, Birmaher B. Predictors of prospectively examined suicide attempts among youth with bipolar disorder. Arch Gen Psychiatry. 2012;69:1113–22. doi: 10.1001/archgenpsychiatry.2012.650. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Gonzalez-Pinto A, Mosquera F, Alonso M, López P, Ramírez F, Vieta E, Baldessarini RJ. Suicidal risk in bipolar I disorder patients and adherence to long-term lithium treatment. Bipolar Disord. 2006;8:618–24. doi: 10.1111/j.1399-5618.2006.00368.x. [DOI] [PubMed] [Google Scholar]
  20. Grunebaum MF, Ellis SP, Duan N, Burke AK, Oquendo MA, John Mann J. Pilot randomized clinical trial of an SSRI vs bupropion: effects on suicidal behavior, ideation, and mood in major depression. Neuropsychopharmacology. 2012;37:697–706. doi: 10.1038/npp.2011.247. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Hahn T, Kircher T, Straube B, Wittchen HU, Konrad C, Ströhle A, Wittmann A, Pfleiderer B, Reif A, Arolt V, Lueken U. Predicting Treatment Response to Cognitive Behavioral Therapy in Panic Disorder With Agoraphobia by Integrating Local Neural Information. JAMA Psychiatry. 2015;72:68. doi: 10.1001/jamapsychiatry.2014.1741. [DOI] [PubMed] [Google Scholar]
  22. Hoertel N, Franco S, Wall MM, Oquendo MA, Kerridge BT, Limosin F, Blanco C. Mental disorders and risk of suicide attempt: a national prospective study. Mol Psychiatry. 2015;20:718–26. doi: 10.1038/mp.2015.19. [DOI] [PubMed] [Google Scholar]
  23. Holma KM, Haukka J, Suominen K, Valtonen HM, Mantere O, Melartin TK, Sokero TP, Oquendo MA, Isometsä ET. Differences in incidence of suicide attempts between bipolar I and II disorders and major depressive disorder. Bipolar Disord. 2014;16:652–61. doi: 10.1111/bdi.12195. [DOI] [PubMed] [Google Scholar]
  24. Isometsä E, Sund R, Pirkola S. Post-discharge suicides of inpatients with bipolar disorder in Finland. Bipolar Disord. 2014;16:867–74. doi: 10.1111/bdi.12237. [DOI] [PubMed] [Google Scholar]
  25. Johnson J, Horwath E, Weissman MM. The validity of major depression with psychotic features based on a community study. Arch Gen Psychiatry. 1991;48:1075–81. doi: 10.1001/archpsyc.1991.01810360039006. [DOI] [PubMed] [Google Scholar]
  26. Johnston BA, Coghill D, Matthews K, Steele JD. Predicting methylphenidate response in attention deficit hyperactivity disorder: a preliminary study. J Psychopharmacol. 2015;29:24–30. doi: 10.1177/0269881114548438. [DOI] [PubMed] [Google Scholar]
  27. Johnston BA, Mwangi B, Matthews K, Coghill D, Konrad K, Steele JD. Brainstem abnormalities in attention deficit hyperactivity disorder support high accuracy individual diagnostic classification. Hum Brain Mapp. 2014;35:5179–89. doi: 10.1002/hbm.22542. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Kapczinski F, Passos IC. Wet minds, dry minds, and the future of psychiatry as a science. Acta Psychiatr Scand. 2015;131:7–8. doi: 10.1111/acps.12357. [DOI] [PubMed] [Google Scholar]
  29. Katz C, Yaseen ZS, Mojtabai R, Cohen LJ, Galynker II. Panic as an independent risk factor for suicide attempt in depressive illness: findings from the National Epidemiological Survey on Alcohol and Related Conditions (NESARC) J Clin Psychiatry. 2011;72:1628–35. doi: 10.4088/JCP.10m06186blu. [DOI] [PubMed] [Google Scholar]
  30. Lee TH, Marcantonio ER, Mangione CM, Thomas EJ, Polanczyk CA, Cook EF, Sugarbaker DJ, Donaldson MC, Poss R, Ho KK, Ludwig LE, Pedan A, Goldman L. Derivation and prospective validation of a simple index for prediction of cardiac risk of major noncardiac surgery. Circulation. 1999;100:1043–9. doi: 10.1161/01.cir.100.10.1043. [DOI] [PubMed] [Google Scholar]
  31. Lenze EJ, Mulsant BH, Shear MK, Schulberg HC, Dew MA, Begley AE, Pollock BG, Reynolds CF. Comorbid anxiety disorders in depressed elderly patients. Am J Psychiatry. 2000;157:722–8. doi: 10.1176/appi.ajp.157.5.722. [DOI] [PubMed] [Google Scholar]
  32. Lozano R, Naghavi M, Foreman K, Lim S, Shibuya K, Aboyans V, Abraham J, Adair T, Aggarwal R, Ahn SY, Alvarado M, Anderson HR, Anderson LM, Andrews KG, Atkinson C, Baddour LM, Barker-Collo S, Bartels DH, Bell ML, Benjamin EJ, Bennett D, Bhalla K, Bikbov B, Bin Abdulhak A, Birbeck G, Blyth F, Bolliger I, Boufous S, Bucello C, Burch M, Burney P, Carapetis J, Chen H, Chou D, Chugh SS, Coffeng LE, Colan SD, Colquhoun S, Colson KE, Condon J, Connor MD, Cooper LT, Corriere M, Cortinovis M, de Vaccaro KC, Couser W, Cowie BC, Criqui MH, Cross M, Dabhadkar KC, Dahodwala N, De Leo D, Degenhardt L, Delossantos A, Denenberg J, Des Jarlais DC, Dharmaratne SD, Dorsey ER, Driscoll T, Duber H, Ebel B, Erwin PJ, Espindola P, Ezzati M, Feigin V, Flaxman AD, Forouzanfar MH, Fowkes FGR, Franklin R, Fransen M, Freeman MK, Gabriel SE, Gakidou E, Gaspari F, Gillum RF, Gonzalez-Medina D, Halasa YA, Haring D, Harrison JE, Havmoeller R, Hay RJ, Hoen B, Hotez PJ, Hoy D, Jacobsen KH, James SL, Jasrasaria R, Jayaraman S, Johns N, Karthikeyan G, Kassebaum N, Keren A, Khoo JP, Knowlton LM, Kobusingye O, Koranteng A, Krishnamurthi R, Lipnick M, Lipshultz SE, Ohno SL, Mabweijano J, MacIntyre MF, Mallinger L, March L, Marks GB, Marks R, Matsumori A, Matzopoulos R, Mayosi BM, McAnulty JH, McDermott MM, McGrath J, Mensah GA, Merriman TR, Michaud C, Miller M, Miller TR, Mock C, Mocumbi AO, Mokdad AA, Moran A, Mulholland K, Nair MN, Naldi L, Narayan KMV, Nasseri K, Norman P, O'Donnell M, Omer SB, Ortblad K, Osborne R, Ozgediz D, Pahari B, Pandian JD, Rivero AP, Padilla RP, Perez-Ruiz F, Perico N, Phillips D, Pierce K, Pope CA, Porrini E, Pourmalek F, Raju M, Ranganathan D, Rehm JT, Rein DB, Remuzzi G, Rivara FP, Roberts T, De León FR, Rosenfeld LC, Rushton L, Sacco RL, Salomon JA, Sampson U, Sanman E, Schwebel DC, Segui-Gomez M, Shepard DS, Singh D, Singleton J, Sliwa K, Smith E, Steer A, Taylor JA, Thomas B, Tleyjeh IM, Towbin JA, Truelsen T, Undurraga EA, Venketasubramanian N, Vijayakumar L, Vos T, Wagner GR, Wang M, Wang W, Watt K, Weinstock MA, Weintraub R, Wilkinson JD, Woolf AD, Wulf S, Yeh PH, Yip P, Zabetian A, Zheng ZJ, Lopez AD, Murray CJL, AlMazroa MA, Memish ZA. Global and regional mortality from 235 causes of death for 20 age groups in 1990 and 2010: a systematic analysis for the Global Burden of Disease Study 2010. Lancet. 2012;380:2095–128. doi: 10.1016/S0140-6736(12)61728-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Mann JJ, Apter A, Bertolote J, Beautrais A, Currier D, Haas A, Hegerl U, Lonnqvist J, Malone K, Marusic A, Mehlum L, Patton G, Phillips M, Rutz W, Rihmer Z, Schmidtke A, Shaffer D, Silverman M, Takahashi Y, Varnik A, Wasserman D, Yip P, Hendin H. Suicide prevention strategies: a systematic review. JAMA. 2005;294:2064–74. doi: 10.1001/jama.294.16.2064. [DOI] [PubMed] [Google Scholar]
  34. Mann JJ, Currier DM. Stress, genetics and epigenetic effects on the neurobiology of suicidal behavior and depression. Eur Psychiatry. 2010;25:268–71. doi: 10.1016/j.eurpsy.2010.01.009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Meltzer HY, Alphs L, Green AI, Altamura AC, Anand R, Bertoldi A, Bourgeois M, Chouinard G, Islam MZ, Kane J, Krishnan R, Lindenmayer JP, Potkin S. Clozapine treatment for suicidality in schizophrenia: International Suicide Prevention Trial (InterSePT) Arch Gen Psychiatry. 2003;60:82–91. doi: 10.1001/archpsyc.60.1.82. [DOI] [PubMed] [Google Scholar]
  36. Morina N, Ajdukovic D, Bogic M, Franciskovic T, Kucukalic A, Lecic-Tosevski D, Morina L, Popovski M, Priebe S. Co-occurrence of major depressive episode and posttraumatic stress disorder among survivors of war: how is it different from either condition alone? J Clin Psychiatry. 2013;74:e212–8. doi: 10.4088/JCP.12m07844. [DOI] [PubMed] [Google Scholar]
  37. Mwangi B, Ebmeier KP, Matthews K, Steele JD. Multi-centre diagnostic classification of individual structural neuroimaging scans from patients with major depressive disorder. Brain. 2012;135:1508–21. doi: 10.1093/brain/aws084. [DOI] [PubMed] [Google Scholar]
  38. Mwangi B, Tian TS, Soares JC. A Review of Feature Reduction Techniques in Neuroimaging. Neuroinformatics. 2013;12:229–244. doi: 10.1007/s12021-013-9204-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Neves FS, Malloy-Diniz LF, Corrêa H. Suicidal behavior in bipolar disorder: what is the influence of psychiatric comorbidities? J Clin Psychiatry. 2009;70:13–8. doi: 10.4088/jcp.08m04037. [DOI] [PubMed] [Google Scholar]
  40. Oquendo MA, Currier D, Liu SM, Hasin DS, Grant BF, Blanco C. Increased risk for suicidal behavior in comorbid bipolar disorder and alcohol use disorders: results from the National Epidemiologic Survey on Alcohol and Related Conditions (NESARC) J Clin Psychiatry. 2010;71:902–9. doi: 10.4088/JCP.09m05198gry. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Oquendo MA, Perez-Rodriguez MM, Poh E, Sullivan G, Burke AK, Sublette ME, Mann JJ, Galfalvy H. Life events: a complex role in the timing of suicidal behavior among depressed patients. Mol Psychiatry. 2014;19:902–9. doi: 10.1038/mp.2013.128. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Oquendo MA, Sullivan GM, Sudol K, Baca-Garcia E, Stanley BH, Sublette ME, Mann JJ. Toward a biosignature for suicide. Am J Psychiatry. 2014;171:1259–77. doi: 10.1176/appi.ajp.2014.14020194. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Perlis RH. A clinical risk stratification tool for predicting treatment resistance in major depressive disorder. Biol Psychiatry. 2013;74:7–14. doi: 10.1016/j.biopsych.2012.12.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Post RM, Kalivas P. Bipolar disorder and substance misuse: pathological and therapeutic implications of their comorbidity and cross-sensitisation. Br J Psychiatry. 2013;202:172–6. doi: 10.1192/bjp.bp.112.116855. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Robert Tibshirani. Regression shrinkage and selection via the lasso: a retrospective. J R Stat Soc. 2011;73:273–282. [Google Scholar]
  46. Schaffer A, Isometsä ET, Tondo L, H Moreno D, Turecki G, Reis C, Cassidy F, Sinyor M, Azorin JM, Kessing LV, Ha K, Goldstein T, Weizman A, Beautrais A, Chou YH, Diazgranados N, Levitt AJ, Zarate CA, Rihmer Z, Yatham LN. International Society for Bipolar Disorders Task Force on Suicide: meta-analyses and meta-regression of correlates of suicide attempts and suicide deaths in bipolar disorder. Bipolar Disord. 2014a doi: 10.1111/bdi.12271. [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Schaffer A, Sinyor M, Reis C, Goldstein BI, Levitt AJ. Suicide in bipolar disorder: characteristics and subgroups. Bipolar Disord. 2014b;16:732–40. doi: 10.1111/bdi.12219. [DOI] [PubMed] [Google Scholar]
  48. Simon NM, Otto MW, Wisniewski SR, Fossey M, Sagduyu K, Frank E, Sachs GS, Nierenberg AA, Thase ME, Pollack MH. Anxiety disorder comorbidity in bipolar disorder patients: data from the first 500 participants in the Systematic Treatment Enhancement Program for Bipolar Disorder (STEP-BD) Am J Psychiatry. 2004;161:2222–9. doi: 10.1176/appi.ajp.161.12.2222. [DOI] [PubMed] [Google Scholar]
  49. Soloff PH, Lynch KG, Kelly TM, Malone KM, Mann JJ. Characteristics of suicide attempts of patients with major depressive episode and borderline personality disorder: a comparative study. Am J Psychiatry. 2000;157:601–8. doi: 10.1176/appi.ajp.157.4.601. [DOI] [PubMed] [Google Scholar]
  50. Stein MB. An epidemiologic perspective on social anxiety disorder. J Clin Psychiatry. 2006;67(Suppl 1):3–8. [PubMed] [Google Scholar]
  51. Tipping ME. Sparse Bayesian Learning and the Relevance Vector Machine. J Mach Learn Res. 2001:211–244. [Google Scholar]
  52. Torres AR, Ramos-Cerqueira ATA, Ferrão YA, Fontenelle LF, do Rosário MC, Miguel EC. Suicidality in obsessive-compulsive disorder: prevalence and relation to symptom dimensions and comorbid conditions. J Clin Psychiatry. 2011;72:17–26. doi: 10.4088/JCP.09m05651blu. quiz 119–20. [DOI] [PubMed] [Google Scholar]
  53. Turecki G. The molecular bases of the suicidal brain. Nat Rev Neurosci. 2014;15:802–16. doi: 10.1038/nrn3839. [DOI] [PMC free article] [PubMed] [Google Scholar]
  54. Turecki G, Ernst C, Jollant F, Labonté B, Mechawar N. The neurodevelopmental origins of suicidal behavior. Trends Neurosci. 2012;35:14–23. doi: 10.1016/j.tins.2011.11.008. [DOI] [PubMed] [Google Scholar]
  55. Van Waarde JA, Scholte HS, van Oudheusden LJB, Verwey B, Denys D, van Wingen GA. A functional MRI marker may predict the outcome of electroconvulsive therapy in severe and treatment-resistant depression. Mol Psychiatry. 2015;20:609–14. doi: 10.1038/mp.2014.78. [DOI] [PubMed] [Google Scholar]
  56. Vapnik VN. Wiley Statistical Learning Theory. wiley; 1998. p. 768. [Google Scholar]
  57. Webb RT, Lichtenstein P, Larsson H, Geddes JR, Fazel S. Suicide, hospital-presenting suicide attempts, and criminality in bipolar disorder: examination of risk for multiple adverse outcomes. J Clin Psychiatry. 2014;75:e809–16. doi: 10.4088/JCP.13m08899. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

supplement
NIHMS748871-supplement.docx (58.2KB, docx)

RESOURCES