Using Machine Learning to Predict Suicide Attempts in Military Personnel

Abstract

Identifying predictors of suicide attempts is critical in intervention and prevention efforts, yet finding predictors has proven difficult due to the low base rate and underpowered statistical approaches. The objective of the current study was to use machine learning to examine predictors of suicidal behaviors among high-risk suicidal Soldiers who received outpatient mental health services in a randomized controlled trial of Brief Cognitive Behavioral Therapy for Suicide Prevention (BCBT) compared to treatment as usual (TAU). Self-report measures of clinical and demographic variables, administered prior to the start of outpatient treatment to 152 participants with recent suicidal thoughts and/or behaviors were analyzed using machine learning software to identify the best combination of variables for predicting suicide attempts during or after treatment. Worst-point suicidal ideation, history of multiple suicide attempts, treatment group (i.e., BCBT or TAU), suicidogenic cognitions, and male sex were found, in combination, correctly classified 30.8% of patients who attempted suicide during the two-year follow-up period. This combination has higher sensitivity than many models that have previously been used to predict suicidal behavior. Overall, this study provides a combination of variables that can be assessed clinical to help identify high-risk suicidal individuals.

1. Introduction

Suicide ranks among the top ten leading causes of death in the United States, with increasing rates over the past two decades (Stone et al., 2018). Rates among military personnel have also risen during this timeframe and have surpassed the gender- and age- adjusted general population rate for the first time in known history (e.g., Pruitt et al., 2016; Reger, Luxton, Skopp, Lee, & Gahm, 2009; Schoenbaum et al., 2014). Efforts to improve the identification of high-risk suicidal individuals have therefore received increased attention. Unfortunately, the dynamic nature of suicide risk makes it difficult to predict both who will and who will not attempt suicide (Bryan et al., 2020; Bryan & Rudd, 2016). Nonetheless, the fluid vulnerability theory of suicide (FVT; Bryan et al., 2020; Rudd, 2006) suggests that suicide risk also possesses stable properties that differ between individuals, referred to as baseline risk. Baseline risk includes risk and protective factors that are fixed and/or historical in nature (e.g., trauma exposure, history of suicide attempt).

Previous research indicates an individual’s most intense instance of suicidal ideation, referred to as worst-point suicidal ideation, is an especially strong predictor of suicide attempts and death by suicide (Beck et al., 1999; Law, Jin, Anestis, 2018; Nam, Hilimire, Jahn, Lehmann, & DeVylder, 2018; Nock et al., 2018). Larger deviations in suicidal ideation over the course of outpatient mental health treatment have also been observed among patients with a history of multiple suicide attempts (Bryan & Rudd, 2016), a clinical subgroup with deficits in self-regulatory capacity. Worst-point suicidal ideation may therefore be an indicator of self-regulation. The ability to self-regulate in response to stressful situations theoretically explains why different situations and stressors lead to suicidal crises in some individuals but not others (e.g., Beck & Haigh, 2014; Rudd, 2006).

Despite the strong association of worst-point suicidal ideation with recent and future suicidal behaviors, current suicidal ideation is often used in clinical settings as a primary measure of a patient’s suicide risk state. The presence and severity of suicide risk factors like psychiatric symptom severity and demographic factors are also frequently used by clinicians to assess a patient’s level of suicide risk (Bryan & Rudd, 2016). In a recent meta-analysis of 50 years of studies, the majority of suicide risk factors were only weakly correlated with suicidal behaviors when considered in isolation (Franklin et al., 2017). Additionally, various mental health disorders and symptoms have been used as predictors of both suicidal ideation and behavior (e.g., May & Klonsky, 2016; Ribeiro et al., 2018; Stein et al., 2010). Common mental health symptoms implicated in suicide risk include but are not limited to depression, anxiety, hopelessness, insomnia, exposure to traumatic events, and substance use. Clinically, these demographic and symptom measures are often used independently to inform a clinical judgement of risk level. Using novel approaches that are capable of combining multiple risk factors (e.g., demographics and mental health symptoms) in meaningful ways, such as machine learning, have been proposed to improve clinically useful suicide risk assessment methods (Nock, Kessler, & Franklin, 2016; Walsh et al., 2017). Machine learning methods differ from traditional analyses used to predict suicidal behaviors in that machine learning models generate an ideal algorithm from a set of predictors. By using multiple clinical risk factors, the machine can iteratively determine the best set of factors to predict a suicide attempt. This process provides a mathematically-driven approach to determining the complex combination of risk factors that predict suicide attempts. To date, machine learning methods have been used primarily by researchers and large healthcare systems, leaving out the possibility of using these methods in standard settings by clinicians.

Given the current low levels of being able to predict suicide attempts, the purpose of the current study was to use a machine learning clinical tool to identify the optimal configuration to predict both who attempts suicide and who does not attempt suicide using previously identified risk factors in combination rather than isolation. The current study uses data collected as part of a clinical trial for brief cognitive behavioral therapy for suicide prevention in a high-risk suicidal population (BCBT; Bryan & Rudd, 2018). Consistent with previous research, we hypothesized that optimal predictive models will include worst-point suicidal ideation rather that current suicidal ideation in combination with other commonly-used measures of suicide risk factors would create clinically useful profiles.

2. Methods

A full description of the sample and study procedures, with CONSORT diagram, are described in Rudd et al. (2015).

2.1. Participants and Procedures

2.1.1. Participants

The present study is a secondary data analysis of a randomized clinical trial testing the efficacy of BCBT (n = 76) as compared to treatment as usual (TAU; n = 76) for the treatment of suicidal active duty Army personnel. Participants all consented to the study and included 152 U.S. Army Soldiers (87.5% male) who reporting suicidal ideation during the past week and/or a suicide attempt in the past month. Participants ranged in age from 19 to 44 years old (M = 27.40, SD = 6.20) and self-identified as 72.4% Caucasian, 13.2% African American, 4.6% Native American, 2.0% Asian, 2.0% Pacific Islander, and 7.9% “other.” Inclusion criteria included active suicide ideation with intent to die during the past week and/or a suicide attempt during the past month. The majority of the sample had a history of suicide attempts with ~38% reporting a history of 2 or more suicide attempts, ~38% reporting a history of 1 suicide attempt, and ~24% denying a history of suicide attempt. Participants had a range of DSM-IV diagnoses with the two most common including major depressive disorder (~78%) and post-traumatic stress disorder (~39%). The only exclusion criterion was an inability to complete informed consent (e.g., active psychosis, intoxication). As reported in Rudd et al. (2015), 26 (17.1%) participants attempted suicide during the two-year follow-up period. Participants in TAU and BCBT did not significantly differ from each other on any demographic or clinical variable.

2.1.2. Procedure & Treatment

Eligible soldiers were randomized to either BCBT (Bryan & Rudd, 2018) or TAU. Brief cognitive behavioral therapy consisted of on average 12 sessions across three phases of treatment (For in depth review see Bryan & Rudd, 2018). Phase 1 consisted of a thorough narrative assessment, crisis response planning, and general emotion regulation skills training. Phase 2 consisted of cognitive interventions used to target suicide related cognitions through cognitive restructuring. In Phase 3, a relapse prevention task was conducted that focused on imaginal rehearsal of a crisis using their skills learned in treatment. Treatment as usual included standard care that was provided within the Department of Defense at the time of the study. This included individual and group therapy, support groups, psychiatric medication, and other adjunctive treatments. All therapists were credentialed as clinical providers in the military hospitals and treatment was provided at no cost.

2.2. Measures

The measures below represent common symptoms and constructs that are used both clinically and in research as indicators of risk factors for suicide. Although this list is not exhaustive of every risk factor, they include those that have previous research and were available from the larger clinical trial.

2.2.1. Suicide Attempt Self-Injury Interview (SASII)

The SASII (Linehan, Comtois, Brown, Heard & Wagner, 2006) was used to assess history of suicide attempts at baseline and the occurrence of suicide attempts during follow-up. The SASII is a structured interview that assesses several aspects of self-directed violence including intent, method and lethality. Suicide attempt was defined as behavior that is self-directed and deliberately results in injury or the potential for injury for which there is evidence of suicidal intent (Crosby et al., 2011).

2.2.2. Beck Scale for Suicidal Ideation, Current (BSSI-C) and Worst-Point (BSSI-W)

The BSSI is a 19-item (Beck & Steer, 1991) interviewer-based measure used to evaluate the intensity of thoughts, attitudes, and behaviors surrounding suicide. Although the items on the BSSI-C and BSSI-W are identical, the instructions differ by asking participants to consider each item during the past two weeks (BSSI-C) and at the worst point during their lives (BSSI-W).

2.2.3. Insomnia Severity Index (ISI)

The ISI (Morin et al., 2011) is a self-report questionnaire used to assess the nature, severity, and impact of sleep disturbance.

2.2.4. Beck Anxiety Inventory (BAI)

The Beck Anxiety Inventory (Beck, Steer & Brown, 1993) is a self-report questionnaire assessing the intensity of anxiety symptoms in the past month.

2.2.5. Interpersonal Needs Questionnaire (INQ)

The INQ-12 (Van Orden, 2008) assesses two components of Joiner’s (2005) interpersonal-psychological theory of suicide: thwarted belongingness and perceived burdensomeness.

2.2.6. Life Events Checklist (LEC)

The LEC (Gray Litz, Hsu, & Lombardo, 2004) is list of potentially traumatic experiences. Items can be summed to assess the extent of lifetime trauma exposure.

2.2.7. Beck Hopelessness Scale (BHS)

The BHS (Beck & Steer, 1993) is a self-report measure used to assess negative thinking about the future.

2.2.8. Beck Depression Inventory, Second Edition (BDI-II)

The BDI-II (Beck, Steer, &Brow 1997) is a self-report measure of recent depressive symptoms.

2.2.9. Post-traumatic Stress Disorder Checklist (PCL-5)

The PCL-5 (PCL-5; Weathers et al., 2013) is a self-report measure that assesses severity of post-traumatic stress disorder symptoms.

2.2.10. Alcohol Use Disorders Identification Test (AUDIT-C)

The AUDIT-C (Bush et al., 1998) is a screening tool of hazardous drinking asking about frequency and amount of consumption of alcohol.

2.2.11. Suicide Cognitions Scale (SCS)

The SCS (Bryan et al., 2014) is a self-report instrument that assesses a range of suicidogenic cognitions including entrapment, unbearability, and unlovability.

2.3. Data Analytic Strategy

Analyses were conducted using the MondoBrain Augmented Intelligence® System, an artificial intelligence data mining platform that uses an algebraic geometry algorithm to identify the optimal configuration of variables that predicts a given outcome. MondoBrain’s explanatory power is rooted in machine learning methods based on an agglomerative supervised learning system that assumes no orthogonality between dimensions. The system also does not make any assumption about the underlying statistical distributions of each variable. MondoBrain’s algorithm works in multiple steps. First, topological subspaces are extracted through dataset sampling. Second, the most locally influential features within that sampled subspace are identified. Third, the signal of that subspace is locally optimized by stabilizing boundary conditions. This process is continued until the signal of the topological subspace is greater than the theoretical maximum signal that could be found from any remaining contiguous hypervolume not bounded by a locally optimized subspace. The locally optimized subspace is represented by a combination of conditions along the most important features. This combination of conditions is represented as a rule describing the optimal combination of variable ranges that yield the most dominant statistical signal. For the present study, predictors included demographic variables (i.e., age, gender), treatment group, and all of the clinical variables listed in the Measures section. The outcome variable was occurrence of suicide attempt during the two-year follow-up period. Results provided by the MondoBrain system are reported as maximized z-scores for predicting suicide attempts, with larger absolute values indicating stronger predictive utility.

3. Results

Mean symptom scores at baseline were elevated (see Table 1), as would be expected in a clinical sample with recent suicidal thoughts and behaviors. Results of the machine learning analysis from MondoBrain are summarized in Table 2. The optimal combination of predictor variables assessed at baseline included, in descending order of importance (indicated by z-score) with defined optimal ranges per measure, BSSI-W score (optimal range: 22–28), a history of multiple suicide attempts, assignment to TAU group (indicated as a 0 in the model), SCS score (optimal range: 18–57), and male sex. This combination of variables correctly classified 8 of 26 participants who attempted suicide (30.8%) and misclassified only 1 of 126 participants who did not attempt suicide (0.8%), yielding 30.8% sensitivity, 99.2% specificity, 88.9% positive predictive value, and 87.4% negative predictive value.

Table 1.

Mean symptom scores at baseline among 152 treatment-seeking Soldiers with recent suicidal thoughts and/or behaviors.

Variable	Mean	SD	Min	Max
BSSI-W	19.1	9.0	0	36
BSSI-C	11.0	8.5	0	33
ISI	17.1	6.2	0	28
BAI	29.3	14.3	0	63
INQ-PB	27.9	13.0	0	42
INQ-TB	24.9	7.1	0	42
LEC	5.6	2.9	0	15
BHS	12.8	6.1	0	20
BDI-II	32.7	13.8	0	60
PCL	56.3	16.9	17	85
AUDIT	9.8	8.0	1	40
SCS	51.9	17.3	18	89

BSSI-W = Beck Scale for Suicide Ideation, Worst Point; BSSI-C = Beck Scale for Suicide Ideation, Current; ISI = Insomnia Severity Index; BAI = Beck Anxiety Inventory; INQ-PB = Interpersonal Needs Questionnaire, Perceived Burdensomeness subscale; INQ-TB = Interpersonal Needs Questionnaire, Thwarted Belongingness subscale; LEC = Life Events Checklist; BHS = Beck Hopelessness Scale; BDI-II = Beck Depression Inventory, 2^nd Edition; PCL = Posttraumatic Stress Disorder Checklist; AUDIT = Alcohol Use Disorders Identification Test; SCS = Suicide Cognitions Scale.

Table 2.

Optimal combination of predictors of suicide attempt during the two-year follow-up period among 152 treatment-seeking Soldiers with recent suicidal thoughts and/or behaviors

		Optimal Score Range
Predictor	z-score	Minimum	Maximum
BSSI-W	2.01	22	28
Prior Attemptsa	1.99		2
Treatment Groupb	0.82	0	0
SCS	0.44	18	57
Sexc	0.44	1	1

^aPrior Attempts were coded as 0=no prior attempts, 1=one prior attempt, 2=two or more prior attempts;

^bTreatment Group was coded as 0=treatment as usual and 1=brief cognitive behavioral therapy for suicide prevention;

^cSex was coded as 1=male, 2=female. Optimal range indicates range in which higher rates of suicide attempt occur.

Comparing MondoBrain, which is a proprietary algorithm, to more standard analyses may provide additional context for evaluation. Several logistic regression models are of often used including a forced entry with all variables, forward conditional stepwise entry, and backward conditional stepwise removal. The current study compared MondoBrain’s analyses using logistic regressions using maximum likelihood estimating and examining accuracy as the classification metric. When a standard logistic regression with forced entry of all predictor variables, the model correctly predicted 15.4% of individuals who attempted suicide. In a second commonly used analysis. Using forward conditional stepwise entry with p<.05 used for entry, p>.10 for removal, and a .5 classification cutoff, the model with the highest classification rate included both BSSI-C and INQ-TB as significant predictors. This model predicted 3.8% of individuals who attempted suicide. Using backward conditional stepwise removal with p>.10 for removal, and a .5 classification cutoff, the model with the highest classification rate included both BSSI-C and INQ-TB as significant predictors although this model. Two models in this analyses predicted 15.4% of individuals who attempted suicide although the final model was less predictive. Overall, all models were better at classifying those who did not attempt suicide than participants who did attempt suicide, which is a common trend in suicide research given the relative low base rate of suicide attempts. All models were comparable at classifying non-attempters. These comparison highlight the strengths of the Mondobrain algorithm compared to more standard analyses (See Table 3 for comparison).

Table 3.

Classification statistics from each statistical model.

Analysis	Sensitivity	Specificity	PPV (Precision)	NPV	Accuracy	F1
MondoBrain	0.308	0.992	0.889	0.874	0.873	0.457
LR: Forced Entry	0.154	1.000	1.000	0.849	0.853	0.267
LR: Forward	0.038	1.000	1.000	0.832	0.833	0.074
LR: Backward	0.000	1.000	---	0.826	0.826	---

LR = Logistic Regression; PPV = Positive predictive value; NPV = Negative predictive value; --- = value could not be computed due to no cases in related cells.

4. Discussion

In the present study, the participants most likely to attempt suicide during follow-up were characterized by the following combination of variables at baseline: male gender, assignment to TAU, history of multiple suicide attempts, severe worst-point suicide ideation, and low to moderate suicidal beliefs. This combination of variables correctly classified almost one-third of participants who attempted suicide in the subsequent two years while yielding only one false positive, resulting in good positive predictive value (88.9%) and negative predictive value (87.4%). This positive predictive value was much larger than the typical values observed among suicide risk assessment scales, which are typically less than 10% and occasionally reach 40% (Runeson et al., 2017). It is important to note that the sample in the current study was at high-risk for suicide and enrolled in a clinical trial for treatment related to suicide. However, the present results provide preliminary support for the utility of suicide risk assessment tools using machine learning methods within clinical practice settings with high-risk patients. Future research with machine learning is warranted to examine if this will replicate in other samples.

As hypothesized, worst-point suicidal ideation was a significant contributor to the optimal combination of predictors of suicide attempts, a finding that converges with previous research (Law, Jin, Anestis, 2018; Nam, Hilimire, Jahn, Lehmann, & DeVylder, 2018; Nock et al., 2018) and provides further support for the importance of assessing this particular risk factor. Of note, current suicidal ideation was not a key predictor of future suicide attempt in the present study. This may be due to the inclusion criteria of having current suicidal ideation, although previous studies suggest that worst-point suicidal ideation is a better predictor of suicidal behavior (e.g., Nock et al., 2018). Current suicidal ideation is often used by clinicians to determine risk, a more thorough assessment that includes a profile of risk factors will likely help improve risk assessment and clinical decision making. Worst-point suicidal ideation may be an especially valuable indicator of suicide risk because it reflects the point of greatest deviation from the individual’s point of homeostatic balance. Using a measure of worst-point suicidal ideation and understanding how it relates to self-regulatory processes could provide insight in future research into both predicting and preventing suicide risk.

Our results also suggest that several other variables that can be easily assessed at the start of treatment—history of suicide attempts, male gender, and suicidogenic cognitions (e.g., unbearability, entrapment, self-hatred)—also provide useful information about the risk of subsequent suicidal behaviors when combined with worst-point suicidal assessment. Our results also implicate the value of treatment type. Although the association of treatment group with subsequent suicide attempts was not as strong as worst-point suicidal ideation and previous suicide attempts, treatment group nonetheless outperformed a large number of variables known to be associated with suicidal behaviors, suggesting treatment type is more important than severity of psychological symptoms. Of note, several variables that are commonly used in assessing and predicting suicidal behaviors were not part of the predictive and different combinations of constructs should be pursued when examining suicidal risk factors. profile found from the machine learning analyses. This gives additional support that new methodological Future research should examine different methodologies using these constructs in order to determine how to best utilize all available data to help predict suicide attempts and target treatments to reduce overall risk.

Overall, the present results highlight the potential value of novel technologies, notably machine learning applications, to create predictive data-driven models that are practical and easy to use by clinicians. By using patient data and on a system that is easily used, clinicians could use complex machine learning programs in a simple dashboard in order to make clinical decisions. Another clinical implication is the potential value of supplementing the assessment of current or recent suicidal ideation with the assessment of worst-point suicidal ideation and suicidogenic cognitions. Although promising, the present results require replication in more diverse clinical samples. Related to this, several limitations warrant discussion. First, the study sample was relatively small with few suicide attempts during the follow-up period. Second, although a large number of clinical and demographic variables were included, other unmeasured variables may provide even greater clinical utility. Third, we used suicide attempt as our outcome instead of death by suicide. Fourth, MondoBrain is a proprietary system and needs further validation. The current study is the first to test MondoBrain. As noted, this analysis was more accurate at classifying suicide attempts compared to previous literature as well as traditional analyses such as logistic regressions. However, future research is needed to further validate MondoBrain against other machine learning algorithms and other statistical approaches as well as across different populations and risk levels. Adding new methods of analyses to suicide research can help move research forward in order to address an important topic. Overall, studies should apply this approach to larger samples, over a greater time period, and to varying levels of risk in order to capture subsequent suicide attempts as well as death by suicide. Nonetheless, this study provides an important contribution to the literature through the application of machine learning in the prediction of suicide attempts.

Highlights.

• Previous models used to predict suicide have very low sensitivity.

• Current study used machine learning approach to predict suicide attempts using a clinical trial dataset.

• Worst-point suicidal ideation, history of multiple suicide attempts, treatment group, suicidogenic cognitions, and male sex were found, in combination, correctly classified 30.8% of patients who attempted suicide during the two-year follow-up period.

• This sensitivity is higher than most suicide prediction models.