Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2022 Jul 20.
Published in final edited form as: Mol Psychiatry. 2022 Jan 20;27(3):1631–1639. doi: 10.1038/s41380-021-01423-4

Predicting Suicide Attempts Among U.S. Army Soldiers After Leaving Active Duty Using Information Available Before Leaving Active Duty: Results from the Study to Assess Risk and Resilience in Servicemembers-Longitudinal Study (STARRS-LS)

Ian H Stanley 1,2, Carol Chu 3,4, Sarah M Gildea 5, Irving H Hwang 5, Andrew J King 5, Chris J Kennedy 6, Alex Luedtke 7,8, Brian P Marx 1,2, Robert O’Brien 9, Maria V Petukhova 5, Nancy A Sampson 5, Dawne Vogt 1,2, Murray B Stein 10,11,12, Robert J Ursano 13, Ronald C Kessler 5
PMCID: PMC9106812  NIHMSID: NIHMS1763497  PMID: 35058567

Abstract

Suicide risk is elevated among military service members who recently transitioned to civilian life. Identifying high-risk service members before this transition could facilitate provision of targeted preventive interventions. We investigated the feasibility of doing this by attempting to develop a prediction model for self-reported suicide attempts (SAs) after leaving or being released from active duty in the Study to Assess Risk and Resilience in Servicemembers-Longitudinal Study (STARRS-LS). This study included two self-report panel surveys (LS1: 2016–2018, LS2: 2018–2019) administered to respondents who previously participated while on active duty in one of three Army STARRS 2011–2014 baseline self-report surveys. We focus on respondents who left active duty >12 months before their LS survey (n=8899). An ensemble machine learning model using predictors available prior to leaving active duty was developed in a 70% training sample and validated in a 30% test sample. The 12-month self-reported SA prevalence (SE) was 1.0% (0.1). Test sample AUC (SE) was .74 (.06). The 15% of respondents with highest predicted risk included nearly two-thirds of 12-month SAs and over 80% of medically serious 12-month SAs. These results show that it is possible to identify soldiers at high post-transition self-report SA risk before the transition. Future model development is needed to examine prediction of SAs assessed by administrative data and using surveys administered closer to the time of leaving active duty.

Introduction

Suicide prevention is a critical operational priority of the U.S. Departments of Defense (DoD)1 and Veterans Affairs (VA).2 Each year, nearly 200000 individuals transition from active duty military service to civilian life.3 For some, this transition is characterized by such psychosocial stressors as disruptions in support networks, housing instability, lack of employment, and financial strain,37 which are associated with increased risk of suicide-related behavior (SRB).8,9 Indeed, suicide risk in the year following the transition to civilian life is approximately 2.5 times as high as the rate among active duty personnel.1012

Recognizing the challenges associated with this transition, a 2018 Presidential Executive Order called for greater coordination between DoD and VA to enhance transitional services.13,14 Several important initiatives followed this Executive Order, including a Congressional bill requiring greater care coordination for transitioning service members.15 Prediction models for post-transition difficulties, such as SRBs, based on information available before the transition could help target individuals most in need of this coordination.16 Although several studies have successfully leveraged predictive analytics to identify individuals at elevated risk of SRB, most studies focused either on military personnel during active duty1719 or veterans regardless of time since transition to civilian life.2022

To address this gap, we present results from an analysis of data from the Study to Assess Risk and Resilience in Servicemembers-Longitudinal Study (STARRS-LS) to predict suicide attempts (SAs) reported by survey respondents as having occurred after leaving or being released from active duty. The prediction model incorporates survey, administrative, and geospatial data available prior to the time of leaving active duty. We focused on predicting SAs rather than suicide deaths because the latter were too rare to study in our survey sample. SAs are of interest because they have serious physical23 and psychiatric24 sequelae and are among the top predictors of subsequent suicides.25,26 However, as the predictors of non-fatal SAs differ from the predictors of suicides,27 we also examined model accuracy in predicting medically serious self-reported SAs, which may better approximate fatal SAs than do other SAs.28,29

Materials and Methods

Sample

Baseline surveys:

The baseline surveys were part of Army STARRS, a multi-component prospective epidemiological-neurobiological study with both baseline and longitudinal surveys designed to examine risk and protective factors for Army SRBs30 (Figure 1). There were 3 baseline surveys: 1) a 2011–2013 representative survey of all soldiers; 2) a separate 2011=2012 survey of new soldiers; and 3) a 2012–2014 survey of several Brigade Combat Teams before and after deployments. Field procedures have been reported elsewhere3033 and are reviewed in the methodology appendix. Written informed consent was obtained from participants. The Human Subjects Committees of the University of Michigan and USUHS (and for the Kuwait component, the Army Medical Research and Materiel Command) approved all recruitment, consent and field procedures.

Figure 1.

Figure 1

Broad Overview of the STARRS Surveys Used in this Study

Longitudinal surveys:

Data for the current report come from the two STARRS-LS surveys, which followed probability samples of baseline Army STARRS survey respondents September 2016-April 2018 (LS1) and April 2018-July 2019 (LS2) using a multi-stage sample design (Figure 1). We focus only on the LS respondents who were in the Regular Army at the time of their Army STARRS survey and no longer on active duty at the time of the LS survey (see methodology appendix for more details). We excluded LS2 respondents who reported a SA in the 12 months before LS1 to avoid double-counting any single respondent in the pooled LS1-LS2 analysis. This means that, by construction, none of the n=4,044 respondents considered here who were both in the LS1 and the LS2 samples reported a SA in the 12 months before LS1. The full analysis sample included n=8899 observations, composed of n=4,230 at LS1 (n=3694 separated from active duty and n=536 no longer on orders or activated in a Reserve or National Guard Component) and n=4669 at LS2 (n=4044 separated and n=625 no longer on orders or activated). Detailed information about recruitment into the baseline and longitudinal surveys is presented in Supplementary Figures 1 and 2.

Measures

Self-reported suicide attempts:

LS1 and LS2 included a section on suicidality adopted from the Columbia-Suicide Severity Rating Scale.34 One of these questions asked respondents Did you ever make a suicide attempt (i.e., purposefully hurt yourself with at least some intention to die) at any time since your last survey? Respondents who said yes were then asked about the number of such attempts and recency of (that attempt/their most recent attempt). We focus on SAs reported within 12 months of the survey. Medically serious SAs were distinguished from other self-reported SAs by asking respondents to describe the most serious injuries incurred from their SA (see the methodology appendix), as past research suggests that predictors of medically serious SAs are more like those of suicides than are the predictors of other SAs.28,29 Although it would have been ideal also to include SAs recorded in electronic health records (EHRs), we did not have access to EHRs for LS respondents no longer on active duty. Previous studies found that self-reports capture about 2/3rds of the SAs detected either by self-reports or medical records.8,35

Predictors:

A review of the literature identified 9 categories of predictors of SAs.27,3638 These included socio-demographics, Army career variables, personality characteristics, adverse childhood experiences, other lifetime traumatic events, chronic stressors, self-injurious thoughts and behaviors, physical health problems, and mental disorders (Supplementary Table 1). Factors associated with low SA risk, such as financial stabiliy and strong social networks, were defined in the inverse along with measures of stressors. As described in more detail in the methodology appendix, we identified 137 baseline Army STARRS survey individual questions or scales as indicators of these categories. Information was also taken from the STARRS Historical Administrative Data Study (HADS) database, which includes administrative data from 50 Army/ DoD administrative data systems (Supplementary Table 2). A review of these data systems led to the selection of 576 variables as indicators of previously known predictors of SAs (Supplementary Tables 37). In addition to individual-level predictors, we included 1,702 variables describing characteristics of the Census Block Groups and Counties where respondents resided that might predict SAs, again including not only variables expected to be risk factors (e.g., high neighborhood crime rate) but also protective (e.g., high neighborhood social capital)3941 (Supplementary Table 8).

Analysis methods

Analysis was carried out November 2020-April 2021. Most machine learning studies to predict SRB either use a single algorithm or try several different algorithms and choose the one with the best prediction accuracy.42 We instead used the Super Learner ensemble machine learning method, which allows results to be pooled across multiple algorithms by stacked generalization. This approach makes use of a weight generated via cross-validation in a user-specified collection (“ensemble”) to combine predicted outcome scores across all algorithms in a way guaranteed in expectation to perform at least as well as the best component algorithm according to a pre-specified criterion (in our case, minimizing MSE).43,44 Consistent with recommendations,45 we used a diverse set of algorithms in the ensemble to capture nonlinearities and interactions and reduce risk of misspecification (Supplementary Table 9).46,47 Model results were validated in a 30% test sample. We examined predictor importance using the model-agnostic kernel SHAP method, which estimates the marginal contribution to overall model accuracy of each variable in a predictor set.48 As discussed in more detail in the methodology appendix, we used a case-control sampling scheme in the training sample to deal with the problem of class imbalance caused by the rarity of SAs.49 Predicted probabilities were calibrated using isotonic regression50 in the training sample. In addition to estimating area under the ROC curve (AUC) to evaluate model accuracy in the test sample, we evaluated calibration accuracy to determine how well the model’s predicted probability approximates the actual event probability. This was done by estimating both the conventional expected calibration error (ECE) based on decile binning51 and the recently-developed integrated calibration index (ICI),52 which is based on the loess curve (smoothing span=0.75) and does not require binning.53 We also evaluated model fairness54 by calculating ICI and ECE in subsamples of the test sample to evaluate whether the association (relative-risk based on a robust Poisson model) between calibrated predicted probability of SA and observed SA in the test sample differed significantly across subsamples defined by sex and race-ethnicity. We then divided the test sample into 20 risk categories based on ventiles of predicted risk defined in the training sample and calculated both conditional and cumulative sensitivity (SN; the proportion of self-reported SAs within and across ventiles of predicted risk) and positive predictive value (PPV; prevalence of self-reported SAs within and across ventiles of predicted risk).

Data management and calculation of prevalence AUCs, ICI, and ECE were carried out in SAS version 9.4.55 SHAP values were estimated in Python.56 The Super Learner models were estimated in R version 3.6.3.57 The R packages used for each algorithm are listed in Supplementary Table 9.

Results

Sample composition

As noted above, the LS surveys were administered to individuals who participated initially in baseline Army STARRS surveys. Weighting was used to correct for STARRS survey nonresponse and loss to follow-up in the LS surveys. Samples were combined across all baseline Army STARRS surveys and pooled over LS1 and LS2 for purposes of building our prediction model. Median respondent age at the time of ending active duty was 26 (Table 1). The great majority of respondents were male (84.3%), Non-Hispanic White (67.6%), heterosexual (93.3%). Most had a high school education (70.7%) and were either currently (56.1%) or never (38.8%) married at the time of ending active duty. In terms of Army career, most either had 0 (43.5%) or exactly 1 (32.8%) combat deployment, were of junior enlisted rank (62.9%), and separated (88.8%; i.e., terminated their relationship with the Army) rather than deactivated (11.2%; i.e., continued in service as a member of the Army National Guard or Army Reserve) at the time of ending active duty.

Table 1.

Socio-demographic characteristics of the analytic sample (n=8899)

Total Medically serious self-reported suicide attempt Non-medically serious self-reported suicide attempt No self-reported suicide attempt
% (SE) % (SE) % (SE) % (SE) χ2,1 df
I. Demographics
 Age 2.5 8
  18–21 13.8 (0.9) 29.8 (18.5) 23.1 (6.3) 13.7 (0.9)
  22–24 22.2 (1.2) 35.8 (15.7) 20.7 (4.9) 22.2 (1.2)
  25–27 18.7 (1.0) 8.3 (5.3) 16.0 (6.2) 18.7 (1.0)
  28–33 22.1 (0.8) 13.5 (8.8) 30.5 (5.1) 22.0 (0.8)
  34+ 23.2 (1.0) 12.7 (8.9) 9.7 (3.0) 23.3 (1.0)
 Gender 0.6 3
  Female (vs. Male) 15.7 (0.8) 24.8 (12.1) 19.4 (4.7) 15.7 (0.8)
 Race 0.2 6
  Non-Hispanic white 67.6 (1.2) 66.2 (14.3) 69.3 (6.0) 67.6 (1.2)
  Non-Hispanic black 15.7 (0.9) 24.5 (12.8) 12.5 (3.5) 15.7 (0.9)
  Hispanic 10.6 (0.6) 9.3 (8.5) 10.9 (4.1) 10.6 (0.6)
  Other 6.1 (0.7) 0.0 (0.0) 7.3 (3.7) 6.1 (0.7)
 Sexuality 2.5 3
  Non-heterosexual (vs. heterosexual) 6.7 (0.5) 14.6 (8.3) 17.7 (5.3) 6.6 (0.5)
 Lifetime max education 4.9 6
  GED or equivalent 8.5 (0.6) 28.5 (18.4) 15.8 (5.3) 8.4 (0.6)
  HS diploma 70.7 (0.9) 52.8 (16.4) 79.8 (5.3) 70.6 (0.9)
  Some college 4.5 (0.4) 18.6 (11.6) 1.2 (0.7) 4.5 (0.4)
  College or more 16.4 (0.9) 0.0 (0.0) 3.3 (1.6) 16.5 (0.9)
 Lifetime marital history 0.6 4
  Currently 56.1 (1.4) 47.1 (15.7) 58.1 (6.5) 56.1 (1.5)
  Previously 5.1 (0.5) 0.0 (0.0) 2.7 (1.8) 5.2 (0.5)
  Never 38.8 (1.5) 52.9 (15.7) 39.2 (6.5) 38.8 (1.5)
II. Army career characteristics
 Lifetime combat deployment 2.5 4
  None 43.5 (1.4) 55.5 (16.8) 38.8 (6.4) 43.5 (1.4)
  Exactly 1 32.8 (1.1) 29.4 (12.8) 49.8 (6.5) 32.7 (1.1)
  2+ 23.7 (1.0) 15.1 (9.6) 11.4 (4.3) 23.8 (1.0)
 Rank 8.7 4
  Junior enlisted 62.9 (1.2) 88.5 (7.0) 75.2 (6.4) 62.7 (1.2)
  Senior enlisted 29.1 (1.0) 11.5 (7.0) 24.1 (6.4) 29.2 (1.0)
  Officer 8.0 (0.7) 0.0 (0.0) 0.7 (0.5) 8.1 (0.7)
 Leaving the Army 0.3 2
  Deactivated (vs separated) 11.2 (0.6) 20.3 (11.4) 11.6 (4.0) 11.2 (0.6)
 Total years of Army enlistment 2.0 8
  1–2 20.6 (1.1) 45.3 (17.7) 21.0 (5.5) 20.6 (1.1)
  3–4 30.4 (1.0) 25.0 (13.9) 34.4 (6.1) 30.4 (1.0)
  5–6 11.8 (0.8) 14.4 (7.4) 7.3 (3.6) 11.8 (0.8)
  7–8 11.8 (0.9) 0.7 (0.7) 21.4 (5.3) 11.8 (1.0)
  9+ 25.3 (1.1) 14.6 (9.6) 15.9 (4.7) 25.4 (1.1)
 Years since separation 1.5 8
  1 20.5 (0.8) 38.0 (14.7) 19.6 (4.6) 20.5 (0.8)
  2 20.2 (0.5) 12.5 (9.1) 23.0 (3.9) 20.2 (0.5)
  3 21.4 (0.7) 25.5 (11.9) 16.9 (4.2) 21.4 (0.7)
  4 18.4 (0.5) 23.5 (18.9) 16.6 (6.2) 18.4 (0.5)
  5+ 19.5 (0.9) 0.6 (0.6) 23.9 (6.6) 19.5 (0.9)
(n) (8899) (17) (102) (8780)

Note. Model estimates reflect weighted data.

1

Wald chi-square for comparing the distribution of demographic variables across three categories: no attempt, non-serious attempt, and medically serious attempt. For variables with zero cell counts in the medically serious category (i.e., race/ethnicity, lifetime education, marital history, and rank) a single observation with weight equal to 0.5 was inserted into the missing cell for purposes of calculating the χ2 statistic.

Sample response bias

Population variable distributions available for all soldiers on active duty at the time of the baseline Army STARRS surveys were compared to the weighted distributions in the LS samples to assess LS sample representativeness. Generally good consistency was found with socio-demographic distributions among NSS respondents compared to all soldiers enlisting in the same years as that survey. In addition, generally good consistency was found with both socio-demographic and Army career characteristic distributions among AAS/PPDS respondents compared to all soldiers on active duty during the same years as those surveys (Supplementary Tables 1011). However, the LS sample respondents somewhat over-represent soldiers who identified as non-Hispanic White and those with higher educations. In addition, the weighted LS subsamples originally surveyed as part of the AAS/PPDS surveys are consistent with population distributions of Army career characteristics as of the time of baseline survey recruitment.

SA Prevalence

n=119 LS respondents no longer on active duty for at least 12 months prior to their LS survey reported a SA in the past 12 months. Prevalence (SE) was 1.0% (0.1) in the total sample (Table 2), higher in LS1 (1.3% [0.2]) than LS2 (0.7% [0.2]), and comparable among members no longer on orders or activated in a Reserve or National Guard Component (1.1% [0.3]) than those separated from active duty (1.0% [0.1]). n=17 of the 119 were medically serious SAs.

Table 2.

12-month prevalence of suicide attempt in the pooled weighted sample of STARRS-LS respondents who left or were released from active duty 12 months or more before their LS survey (n=8899)1

Prevalence of self-reported suicide attempt2 Unweighted self-reported suicide attempt frequency (n)
Any Medically serious
% (SE) % (SE) Any Serious Total
LS1
 Separated 1.3 (0.2) 0.1 (0.0) (62) (11) (3694)
 Reserve/National Guard3 1.7 (0.7) 0.3 (0.3) (8) (2) (536)
  All 1.3 (0.2) 0.1 (0.0) (70) (13) (4230)
LS24
 Separated 0.7 (0.2) 0.1 (0.1) (41) (2) (4044)
 Reserve/National Guard3 0.6 (0.3) 0.1 (0.1) (8) (2) (625)
  All 0.7 (0.2) 0.1 (0.0) (49) (4) (4669)
LS1/LS2 combined
 Separated 1.0 (0.1) 0.1 (0.0) (103) (13) (7738)
 Reserve/National Guard3 1.1 (0.3) 0.2 (0.1) (16) (4) (1161)
  All 1.0 (0.1) 0.1 (0.0) (119) (17) (8899)

Abbreviations: STARRS-LS, Study to Assess Risk & Resilience in Servicemembers-Longitudinal Study; SE, standard error; LS1, STARRS-LS Wave 1; LS2, STARRS-LS Wave 2.

1

See the text for a description of weighting.

2

Weighted to correct for nonresponse bias.

3

No longer on orders or activated.

4

LS2 respondents who reported a suicide attempt at any time in the 12 months before the LS1 survey were excluded from the LS2 sample for purposes of this analysis.

Model results

Model fitting:

As detailed in the methodology appendix, preliminary analyses were carried out to investigate the implications of expanding or reducing the feature selection methods (Supplementary Table 12) and the number of features used in the ensembles (Supplementary Table 13) to reduce over-fitting. Optimal restrictions were used in building the model. Four algorithms had nonzero Super Learner importance weights in that model: stratified means, a penalized logistic regression, a random forest, and an extreme gradient boosting model (Supplementary Table 14).

Overall model fit:

The AUC (SE) in the full test sample was .74 (.06) (Figure 2). We also examined whether equally strong prediction could be achieved by limiting the predictors to those available in administrative records, thereby sparing soldiers the burden of completing a questionnaire. The AUC (SE) of the best-fitting model limited to administrative predictors was substantially lower (.63 [.06]), indicating that some self-report questions are needed to optimize prediction.

Figure 2. Receiver operating characteristic curves in the test sample and subsamples (n=2671).

Figure 2

Abbreviations: AUC, area under the receiver operating characteristic curve; SE, standard error.

Inspection of the predicted risk ventiles based on the best-fitting model applied to the test sample showed that respondents in the top 3 predicted risk ventiles had elevated conditional sensitivity, whereas the remaining ventiles generally had conditional sensitivities either close to or below expected values (Table 3). Respondents in these top predicted risk ventiles included 64.6% of SAs. The proportion of medically serious SAs among respondents in the top predicted risk ventiles (81.5%) was higher than the proportion of other self-reported SAs (60.4%), although this difference was not statistically significant (χ21=0.7, p=.42). SA prevalence was 3.5% in the top risk ventile and 2.6% across the top 3 predicted risk ventiles compared to 0.5% in the remainder of the sample.

Table 3.

Reported 12-month suicide attempt by ventiles of predicted risk in the test sample (n=2671)1

Distribution2 Sensitivity (SN) Positive Predictive Value (PPV)
Within-Ventile Cumulative Within-Ventile Cumulative
Risk Ventile3 % (SE) SN (SE) SN (SE) PPV (SE) PPV (SE)
 1 8.8 (0.9) 30.4 (10.7) 30.4 (10.3) 3.5 (1.4) 3.5 (1.4)
 2 3.2 (0.5) 6.9 (3.0) 37.3 (11.1) 2.2 (0.9) 3.1 (1.1)
 3 13.1 (0.8) 27.4 (9.8) 64.6 (12.7) 2.1 (0.8) 2.6 (0.7)
 4 4.7 (0.5) 6.9 (4.3) 71.5 (13.0) 1.5 (0.9) 2.4 (0.6)
 5 5.0 (1.0) 1.0 (0.7) 72.5 (13.1) 0.2 (0.1) 2.1 (0.5)
 6–20 65.1 (1.5) 27.5 (13.1) 100.0 (0.0) 0.4 (0.3) 1.0 (0.1)

Abbreviations: SE, standard error.

1

The n=2671 respondents in the test sample represent roughly 30% of the n=8899 in the total sample, including n=35 of the n=119 total-sample respondents who reported attempting suicide in the 12 months before their STARRS-LS survey. The remaining 70% of the total sample were in the training sample.

2

As the thresholds defining ventiles of predicted risk were based on the training sample, the proportions of test sample respondents in each ventile do not equal 5%.

3

Defined in terms of thresholds in the calibrated training sample to separate the sample into 20 subsamples of equal size rank ordered in terms predicted risk.

Subgroup analyses:

Subgroup analysis showed that the AUC of the best model was substantially lower among respondents who left or were released from active duty 3+ years before the LS survey (AUC=.64 [.10]; n=1439) than those most recently active less than 3 years ago (AUC=.83 [.04]; n=1232). This suggests indirectly that a survey administered before leaving or being released from active duty might yield even stronger results than those found here. Consistent with this speculation, AUC was also higher among respondents whose most recent baseline (i.e., prior to leaving active duty) survey was less than 3 years before their LS survey; (AUC=.95 [.05]; n=565) than 4+ years (AUC=.74 [.06]; n=2106).

Other subgroup analyses showed that AUC was higher among respondents who were no longer on orders or activated in a Reserve or National Guard Component at the time of the LS survey (AUC=.93 [.04]; n=340) than respondents completely separated (AUC=.70 [.06]; n=2331). AUC was also higher among women (AUC=.87 [.08]; n=373) than men (AUC=.72 [.06]; n=2298). The number of medically serious self-reported SAs was too small to be used as an outcome in a separate Super Learner model, but we were able to disaggregate AUC in the overall model by this distinction and found a higher AUC for medically serious (AUC=.93 [.04]) than other (AUC=.69 [.06]) self-reported SAs.

Calibration and fairness:

Model calibration was found to be excellent not only in the total test sample (ICI=.005; ECE=.003) but also in subsamples defined by sex and race-ethnicity (ICI=.006–.010; ECE=.003–.007). Consistent with the latter result, the fairness analysis found that the association between predicted probability of SA and observed SA did not differ significantly either between men and women (F1=3.4, p=.07) or between soldiers identifying as non-Hispanic White versus others (F1=0.1, p=.83; Supplementary Table 15).

Predictor importance:

A total of 807 variables in the final predictor set had significant (p<0.10, two-sided test) zero-order associations with SA in the training sample, including 54 survey variables, 199 administrative variables, and 554 geospatial variables. 37 of these 807 predictors were selected in the two-part lasso feature selection procedure, including 14 survey, 12 administrative, and 11 geospatial variables (Supplementary Table 13). A rank order of mean absolute SHAP values by individual predictors across these 37 showed that self-reported lifetime suicide plan in the baseline survey was the strongest predictor and that 5 of the top 10 came from the baseline survey (Figure 3). The 37 predictors included several each from the domains of self-injurious thoughts and behaviors, lifetime traumatic events, and socio-demographics (Supplementary Table 16). Fewer measures of mental or physical disorders or Army career variables were in this set. It is important to recognize, though, that the small number of predictors in our final model should not be interpreted as having special importance in and of themselves but should instead be seen as the best marker items for the 807 individually significant zero-order predictors in the full predictor set.

Figure 3. Predictor importance based on kernel SHAP values in the test sample (n=2671)1.

Figure 3

Abbreviations: SHAP, SHapley Additive exPlanations; LT, lifetime; (s), survey predictor; LTT, lifetime trauma; CBG, Census Block Group; (g), geospatial predictor; UL, unit-level; 12m, 12-month; perp, perpetrator; prop, property; (a), administrative predictor; txt, treatment; in/out, inpatient admission/outpatient visit; 4yr, 4 year; out, outpatient visit; pers, personality; stab, stabilized scale; meds, medications; MDE, major depressive episode; CL, county level; diff, difference; GDP, gross domestic product; psych, psychiatrists; alc, alcohol; prob, problems; endo/met, endocrine, nutritional/metabolic; CT, Census Tract; 2yr, 2 years; depr, depression; BD, bipolar disorder; in, inpatient admission; adv, adversities; stnd, standardized scale; vict, victim; comp, computers; CGB-CL, Census Block Group to County level; SSI, supplemental security income; dep, dependents.

1See Supplementary Table 13 for a description of the predictor variables. Survey predictors were measured retrospectively in the time period prior to leaving or being released from active duty. Administrative predictors were defined as the earlier of the two times of leaving/being released from active duty or December 31, 2016, given that our access to administrative data was only up to the end of 2016. Geospatial predictors were based on the Census Block Group or County of residence at the time of the LS1/LS2 survey.

2The SHAP value for an individual is the extent to which the predicted probability of suicide changes when a single variable is deleted from the prediction model averaged across all logically possible combinations of the 37 predictors. The model-agnostic kernel SHAP method was used to estimate SHAP values.40 As these values can be either positive or negative at the individual level, we report here the mean of the absolute SHAP value across all respondents in the test sample.

Discussion

Military leaders have heretofore been limited in their ability to predict which service members will attempt suicide during the high-risk time shortly after leaving or being released from active duty.1012,58 We found that a parsimonious model to predict self-reported 12-month SA can be developed using a short battery of survey questions along with administrative variables and variables about the characteristics of the geographic area where the soldier lives. This model has good prediction accuracy and calibration. About two-thirds of the soldiers who reported SAs and more than 80% who reported medically serious SAs were in the top 3 predicted risk ventiles. These results are likely conservative, as prediction accuracy was inversely proportional to time between the baseline survey and the LS survey. This means the model would be expected to perform better if based on a survey administered within a year of leaving or being released from active duty and the model was used to target preventive interventions over the first year after leaving. As the number of survey predictors in our model was small (n=14), it would be feasible to add this battery to the pre-existing annual post-separation survey currently administered to all soldiers scheduled to leave or be released from active duty over the next year.

The variables assessing self-injurious thoughts and behaviors ascertained via self-report, emerged as the most important predictors. Although this is consistent with previous research,37 evidence from studies of patients being screened for suicidality during routine outpatient visits who went on to die by suicide shortly thereafter,59,60 show that the great majority of such patients deny any suicidality. This occurs for a variety of reasons, including routine variations in suicidal ideation61 and concerns among suicidal patients that the health care system might restrict their activities.62 This concern was apparently less prominent in completing the STARRS surveys, but it is unclear whether the same would be true if the same questions were included in an Army-administered pre-separation survey given the importance of context to survey response.63 Future methodological research is needed to investigate this issue.

In addition, as noted in the subsection on variable importance in the results section, the predictors that came out as important in the small set selected by our final model should not be interpreted as causal risk factors64,65 but rather as best marker items representing the joint associations of the 807 individually significant zero-order predictors in the full predictor set. This means the predictor importance results are useful primarily in helping guide which self-report questions to ask in future pre-separation survey rather than as guides for intervention content.

Several suicide prevention initiatives for transitioning service members are currently being implemented, including the VA Solid Start program, which involves three contacts from the VA with all service members in the year following departure from the military to offer mental health and related resources.66 Referral to community-based universal interventions based on a public health perspective can be valuable components of such communications.67 However, soldiers identified as high-risk by our model may also benefit from additional enhanced case management or higher-intensity interventions5,6870 in the transitionary period depending on acuity and cost-effectiveness considerations.16,71 Guidance in this regard might be provided by subsequent analyses of the experiences of high-risk soldiers after leaving or being released from active duty that more proximally predict SA. In so doing, it would be important to examine causal mechanisms within the context of theoretical frameworks (e.g., the three-step theory of suicide)5,72 and to contextualize results within the VA’s broad public health strategy to suicide prevention.2,73

Our study has several noteworthy limitations. First, the sample was restricted to soldiers who participated in Army STARRS surveys in 2011–2014 and could be traced and resurveyed in 2016–2019, raising the possibility of sample bias. Second, the Army STARRS and STARRS-LS surveys were explicitly advertised as independent academic surveys in which identified respondent reports would not be made available to military leaders. It is unclear whether the same results would be found in surveys carried out by the military or VA. Third, substantial variation existed in the time lag between baseline surveys and LS surveys, leading to underestimation of overall model prediction accuracy and possibly to selection of suboptimal predictors. Fourth, variation in time between when respondents left active duty and the LS surveys introducing instability in model results that would be resolved if future baseline surveys were carried out shortly before soldiers left or were released from active duty. Fifth, SAs were assessed exclusively with self-reports; we did not additionally review administrative records to determine SA status at LS2 (post-active duty). Self-reports under-represent true SAs.74 It is not clear whether prediction accuracy would be different for SAs assessed only by administrative data.

Within the context of these limitations, the study demonstrated that data available prior to a service member leaving active duty can be used to predict self-reported suicide attempts following active duty service. This represents a crucial step forward as the VA and DoD seek to enhance their provision of suicide prevention resources for transitioning service members.1315,75 Future work is needed to examine prediction of suicide attempts assessed by administrative data and using surveys carried out under military auspices administered closer to the time of leaving active duty.

Supplementary Material

Supplemental Material

Acknowledgments

Funding:

Dr. Stanley was supported in part by a grant from the National Institute of Mental Health (T32MH019836). Army STARRS was sponsored by the Department of the Army and funded under cooperative agreement number U01MH087981 with the U.S. Department of Health and Human Services, National Institutes of Health, National Institute of Mental Health (NIH/NIMH). Subsequently, STARRS-LS was sponsored and funded by the Department of Defense (USUHS grant numbers HU00011520004 and HU0001202003). The grants were administered by the Henry M. Jackson Foundation for the Advancement of Military Medicine Inc. (HJF). The contents are solely the responsibility of the authors and do not necessarily represent the views of the Department of Health and Human Services, NIMH, the Department of the Army, Department of Defense or HJF.

The Army STARRS Team consists of Co-Principal Investigators: Robert J. Ursano, MD (Uniformed Services University) and Murray B. Stein, MD, MPH (University of California San Diego and VA San Diego Healthcare System)

Site Principal Investigators: James Wagner, PhD (University of Michigan) and Ronald C. Kessler, PhD (Harvard Medical School)

Army scientific consultant/liaison: Kenneth Cox, MD, MPH (Office of the Assistant Secretary of the Army (Manpower and Reserve Affairs))

Other team members: Pablo A. Aliaga, MA (Uniformed Services University); David M. Benedek, MD (Uniformed Services University); Laura Campbell-Sills, PhD (University of California San Diego); Carol S. Fullerton, PhD (Uniformed Services University); Nancy Gebler, MA (University of Michigan); Meredith House, BA (University of Michigan); Paul E. Hurwitz, MPH (Uniformed Services University); Sonia Jain, PhD (University of California San Diego); Tzu-Cheg Kao, PhD (Uniformed Services University); Lisa Lewandowski-Romps, PhD (University of Michigan); Alex Luedtke, PhD (University of Washington and Fred Hutchinson Cancer Research Center); Holly Herberman Mash, PhD (Uniformed Services University); James A. Naifeh, PhD (Uniformed Services University); Matthew K. Nock, PhD (Harvard University); Victor Puac-Polanco, MD, DrPH (Harvard Medical School); Nancy A. Sampson, BA (Harvard Medical School); and Alan M. Zaslavsky, PhD (Harvard Medical School).

As a cooperative agreement, scientists employed by the National Institute of Mental Health and U.S. Army liaisons and consultants collaborated to develop the study protocol and data collection instruments, supervise data collection, interpret results, and prepare reports. Although a draft of the manuscript was submitted to the U.S. Army and National Institute of Mental Health for review and comment before submission for publication, this was done with the understanding that comments would be no more than advisory.

Footnotes

Conflict of Interest Disclosures

In the past 3 years, RCK was a consultant for Datastat, Inc., Holmusk, RallyPoint Networks, Inc., and Sage Therapeutics. He has stock options in Mirah, PYM, and Roga Sciences.

In the past 3 years MBS received consulting income from Actelion, Acadia Pharmaceuticals, Aptinyx, atai Life Sciences, Boehringer Ingelheim, Bionomics, BioXcel Therapeutics, Clexio, EmpowerPharm, Engrail Therapeutics, GW Pharmaceuticals, Janssen, Jazz Pharmaceuticals, and Roche/Genentech. MBS has stock options in Oxeia Biopharmaceuticals and EpiVario. He is paid for his editorial work on Depression and Anxiety (Editor-in-Chief), Biological Psychiatry (Deputy Editor), and UpToDate (Co-Editor-in-Chief for Psychiatry). The other authors declare that they have no conflict of interest.

Supplementary Information: Supplementary information is available at MP’s website.

References

  • 1.Department of Defense. Department of Defense Strategy for Suicide Prevention. 2015. https://www.dspo.mil/Portals/113/Documents/TAB%20B%20-%20dssp_final%20USD%20PR%20signed.pdf. Accessed 14 July 2021.
  • 2.U.S. Department of Veterans Affairs. National Strategy for Preventing Veteran Suicide: 2018–2028. 2018. https://www.mentalhealth.va.gov/suicide_prevention/docs/Office-of-Mental-Health-and-Suicide-Prevention-National-Strategy-for-Preventing-Veterans-Suicide.pdf. Accessed 16 July 2021.
  • 3.U.S. Department of Veterans Affairs. The Military to Civilian Transition 2018: A Review of Historical, Current, and Future Trends. 2018. https://benefits.va.gov/TRANSITION/docs/mct-report-2018.pdf. Accessed 16 July 2021.
  • 4.Pease JL, Billera M, Gerard G. Military Culture and the Transition to Civilian Life: Suicide Risk and Other Considerations. Soc Work 2016; 61(1):83–86. [DOI] [PubMed] [Google Scholar]
  • 5.Sokol Y, Gromatsky M, Edwards ER, Greene AL, Geraci JC, Harris RE et al. The deadly gap: Understanding suicide among veterans transitioning out of the military. Psychiatry Res 2021; 300:113875. [DOI] [PubMed] [Google Scholar]
  • 6.Vogt D, King MW, Borowski S, Finley EP, Perkins DF, Copeland LA. Identifying factors that contribute to military veterans’ post-military well-being. Appl Psychol Health Well Being 2021; 13(2):341–356. [DOI] [PubMed] [Google Scholar]
  • 7.Geraci JC, Mobbs M, Edwards ER, Doerries B, Armstrong N, Porcarelli R et al. Expanded Roles and Recommendations for Stakeholders to Successfully Reintegrate Modern Warriors and Mitigate Suicide Risk. Front Psychol 2020; 11(1907). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Chu C, Zuromski KL, Bernecker SL, Gutierrez PM, Joiner TE, Liu H et al. A test of the interpersonal theory of suicide in a large, representative, retrospective and prospective study: Results from the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). Behav Res Ther 2020; 132:103688. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Blosnich JR, Montgomery AE, Dichter ME, Gordon AJ, Kavalieratos D, Taylor L et al. Social Determinants and Military Veterans’ Suicide Ideation and Attempt: a Cross-sectional Analysis of Electronic Health Record Data. J Gen Intern Med 2020; 35(6):1759–1767. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Shen YC, Cunha JM, Williams TV. Time-varying associations of suicide with deployments, mental health conditions, and stressful life events among current and former US military personnel: a retrospective multivariate analysis. Lancet Psychiatry 2016; 3(11):1039–1048. [DOI] [PubMed] [Google Scholar]
  • 11.Reger MA, Smolenski DJ, Skopp NA, Metzger-Abamukang MJ, Kang HK, Bullman TA et al. Risk of Suicide Among US Military Service Members Following Operation Enduring Freedom or Operation Iraqi Freedom Deployment and Separation From the US Military. JAMA Psychiatry 2015; 72(6):561–569. [DOI] [PubMed] [Google Scholar]
  • 12.Ravindran C, Morley SW, Stephens BM, Stanley IH, Reger MA. Association of Suicide Risk With Transition to Civilian Life Among US Military Service Members. JAMA Netw Open 2020; 3(9):e2016261–e2016261. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Executive Office of the President. Supporting Our Veterans During Their Transition From Uniformed Service to Civilian Life. 2018. https://www.federalregister.gov/documents/2018/01/12/2018-00630/supporting-our-veterans-during-their-transition-from-uniformed-service-to-civilian-life. Accessed 13 July 2021.
  • 14.U.S. Secretary of Defense, U.S. Secretary of Veterans Affairs, U.S. Secretary of Homeland Security. Joint Action Plan for Supporting Veterans During Their Transition from Uniformed Service to Civilian Life. 2018. https://www.va.gov/opa/docs/joint-action-plan-05-03-18.pdf. Accessed 14 July 2021.
  • 15.116th Congress (2019–2020). Commander John Scott Hannon Veterans Mental Health Care Improvement Act of 2019. 2020. https://www.congress.gov/bill/116th-congress/senate-bill/785. Accessed 24 September 2021.
  • 16.Ross EL, Zuromski KL, Reis BY, Nock MK, Kessler RC, Smoller JW. Accuracy Requirements for Cost-effective Suicide Risk Prediction Among Primary Care Patients in the US. JAMA Psychiatry 2021; 78(6):642–650. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Zuromski KL, Bernecker SL, Gutierrez PM, Joiner TE, King AJ, Liu H et al. Assessment of a Risk Index for Suicide Attempts Among US Army Soldiers With Suicide Ideation: Analysis of Data From the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). JAMA Netw Open 2019; 2(3):e190766–e190766. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Kessler RC, Warner CH, Ivany C, Petukhova MV, Rose S, Bromet EJ et al. Predicting Suicides After Psychiatric Hospitalization in US Army Soldiers: The Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). JAMA Psychiatry 2015; 72(1):49–57. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Kessler RC, Stein MB, Petukhova MV, Bliese P, Bossarte RM, Bromet EJ et al. Predicting suicides after outpatient mental health visits in the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). Mol Psychiatry 2017; 22(4):544–551. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Kessler RC, Bauer MS, Bishop TM, Demler OV, Dobscha SK, Gildea SM et al. Using Administrative Data to Predict Suicide After Psychiatric Hospitalization in the Veterans Health Administration System. Front Psychiatry 2020; 11:390. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Gradus JL, King MW, Galatzer-Levy I, Street AE. Gender Differences in Machine Learning Models of Trauma and Suicidal Ideation in Veterans of the Iraq and Afghanistan Wars. J Trauma Stress 2017; 30(4):362–371. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Kessler RC, Hwang I, Hoffmire CA, McCarthy JF, Petukhova MV, Rosellini AJ et al. Developing a practical suicide risk prediction model for targeting high-risk patients in the Veterans health Administration. Int J Methods Psychiatr Res 2017; 26(3). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Nock MK, Borges G, Bromet EJ, Cha CB, Kessler RC, Lee S. Suicide and Suicidal Behavior. Epidemiol Rev 2008; 30(1):133–154. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Stanley IH, Hom MA, Boffa JW, Stage DL, Joiner TE. PTSD from a suicide attempt: An empirical investigation among suicide attempt survivors. J Clin Psychol 2019; 75(10):1879–1895. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Ribeiro JD, Franklin JC, Fox KR, Bentley KH, Kleiman EM, Chang BP et al. Self-injurious thoughts and behaviors as risk factors for future suicide ideation, attempts, and death: a meta-analysis of longitudinal studies. Psychol Med 2016; 46(2):225–236. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Bostwick JM, Pabbati C, Geske JR, McKean AJ. Suicide Attempt as a Risk Factor for Completed Suicide: Even More Lethal Than We Knew. Am J Psychiatry 2016; 173(11):1094–1100. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Klonsky ED, May AM, Saffer BY. Suicide, Suicide Attempts, and Suicidal Ideation. Annu Rev Clin Psychol 2016; 12(1):307–330. [DOI] [PubMed] [Google Scholar]
  • 28.Beautrais AL. Suicides and serious suicide attempts: two populations or one? Psychol Med 2001; 31(5):837–845. [DOI] [PubMed] [Google Scholar]
  • 29.Suokas J, Lönnqvist J. Outcome of attempted suicide and psychiatric consultation: risk factors and suicide mortality during a five-year follow-up. Acta Psychiatr Scand 1991; 84(6):545–549. [DOI] [PubMed] [Google Scholar]
  • 30.Ursano RJ, Colpe LJ, Heeringa SG, Kessler RC, Schoenbaum M, Stein MB. The Army study to assess risk and resilience in servicemembers (Army STARRS). Psychiatry 2014; 77(2):107–119. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Heeringa SG, Gebler N, Colpe LJ, Fullerton CS, Hwang I, Kessler RC et al. Field procedures in the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). Int J Methods Psychiatr Res 2013; 22(4):276–287. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Kessler RC, Heeringa SG, Colpe LJ, Fullerton CS, Gebler N, Hwang I et al. Response bias, weighting adjustments, and design effects in the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). Int J Methods Psychiatr Res 2013; 22(4):288–302. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33.Kessler RC, Colpe LJ, Fullerton CS, Gebler N, Naifeh JA, Nock MK et al. Design of the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). Int J Methods Psychiatr Res 2013; 22(4):267–275. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Posner K, Brown GK, Stanley B, Brent DA, Yershova KV, Oquendo MA et al. The Columbia-Suicide Severity Rating Scale: initial validity and internal consistency findings from three multisite studies with adolescents and adults. Am J Psychiatry 2011; 168(12):1266–1277. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35.Lee DJ, Kearns JC, Wisco BE, Green JD, Gradus JL, Sloan DM et al. A longitudinal study of risk factors for suicide attempts among Operation Enduring Freedom and Operation Iraqi Freedom veterans. Depress Anxiety 2018; 35(7):609–618. [DOI] [PubMed] [Google Scholar]
  • 36.Nock MK, Deming CA, Fullerton CS, Gilman SE, Goldenberg M, Kessler RC et al. Suicide among soldiers: a review of psychosocial risk and protective factors. Psychiatry 2013; 76(2):97–125. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.Franklin JC, Ribeiro JD, Fox KR, Bentley KH, Kleiman EM, Huang X et al. Risk factors for suicidal thoughts and behaviors: A meta-analysis of 50 years of research. Psychol Bull 2017; 143(2):187–232. [DOI] [PubMed] [Google Scholar]
  • 38.Holliday R, Borges LM, Stearns-Yoder KA, Hoffberg AS, Brenner LA, Monteith LL. Posttraumatic Stress Disorder, Suicidal Ideation, and Suicidal Self-Directed Violence Among U.S. Military Personnel and Veterans: A Systematic Review of the Literature From 2010 to 2018. Front Psychol 2020; 11:1998. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39.Steelesmith DL, Fontanella CA, Campo JV, Bridge JA, Warren KL, Root ED. Contextual Factors Associated With County-Level Suicide Rates in the United States, 1999 to 2016. JAMA Netw Open 2019; 2(9):e1910936–e1910936. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40.Rehkopf DH, Buka SL. The association between suicide and the socio-economic characteristics of geographical areas: a systematic review. Psychol Med 2006; 36(2):145–157. [DOI] [PubMed] [Google Scholar]
  • 41.Fontanella CA, Saman DM, Campo JV, Hiance-Steelesmith DL, Bridge JA, Sweeney HA et al. Mapping suicide mortality in Ohio: A spatial epidemiological analysis of suicide clusters and area level correlates. Prev Med 2018; 106:177–184. [DOI] [PubMed] [Google Scholar]
  • 42.Kessler RC, Bernecker SL, Bossarte RM, Luedtke AR, McCarthy JF, Nock MK et al. The Role of Big Data Analytics in Predicting Suicide. In: Passos IMB, Kapczinski F. (eds). Personalized Psychiatry. Springer International Publishing: Cham, 2019, pp 77–98. [Google Scholar]
  • 43.Polley EC, Rose S, van der Laan MJ. Super Learning. In: Rose S, van der Laan MJ (eds). Targeted Learning: Causal Inference for Observational and Experimental Data. Springer New York: New York, NY, 2011, pp 43–66. [Google Scholar]
  • 44.Polley EC, LeDell E, Kennedy C, Lendle S, van der Laan MJ. SuperLearner: Super Learner Prediction [R package]. 2.0–24. 2018.
  • 45.LeDell E, van der Laan MJ, Petersen M. AUC-Maximizing Ensembles through Metalearning. Int J Biostat 2016; 12(1):203–218. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46.Kabir MF, Ludwig SA. Enhancing the Performance of Classification Using Super Learning. Data-enabled Discov Appl 2019; 3(1):1–13. [Google Scholar]
  • 47.Kennedy CJ. Guide to SuperLearner. 2017. https://cran.r-project.org/web/packages/SuperLearner/vignettes/Guide-to-SuperLearner.html. Accessed 21 July 2021.
  • 48.Lundberg S, Lee S-I. A Unified Approach to Interpreting Model Predictions. 2017. https://arxiv.org/abs/1705.07874. Accessed 14 July 2021.
  • 49.Greenland S. Model-based estimation of relative risks and other epidemiologic measures in studies of common outcomes and in case-control studies. Am J Epidemiol 2004; 160(4):301–305. [DOI] [PubMed] [Google Scholar]
  • 50.Böken B. On the appropriateness of Platt scaling in classifier calibration. Info Syst 2021; 95:101641. [Google Scholar]
  • 51.Naeini MP, Cooper GF, Hauskrecht M. Obtaining Well Calibrated Probabilities Using Bayesian Binning. Proc Conf AAAI Artif Intell 2015. Jan; 2015:2901–2907. [PMC free article] [PubMed] [Google Scholar]
  • 52.Austin PC, Steyerberg EW. The Integrated Calibration Index (ICI) and related metrics for quantifying the calibration of logistic regression models. Stat Med 2019; 38(21):4051–4065. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 53.Austin PC, Steyerberg EW. Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers. Stat Med 2014; 33(3):517–535. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 54.Yuan M, Kumar V, Ahmad MA, Teredesai A. Assessing Fairness in Classification Parity of Machine Learning Models in Healthcare. 2021. Available at http://arxiv.org/abs/2102.03717. Accessed 20 July 2021.
  • 55.SAS Institute Inc. SAS ®Software 9.4 edn. Cary, NC, 2013. [Google Scholar]
  • 56.Lundberg S. Welcome to the SHAP documentation. 2018. https://shap.readthedocs.io/en/latest/index.html. Accessed 21 July 2021.
  • 57.R Core Team. R: A language and environment for statistical computing. 2021. https://www.r-project.org/. Accessed 20 July 2021.
  • 58.Bullman T, Hoffmire C, Schneiderman A, Bossarte R. Time dependent gender differences in suicide risk among Operation Enduring Freedom and Operation Iraqi Freedom veterans. Ann Epidemiol 2015; 25(12):964–965. [DOI] [PubMed] [Google Scholar]
  • 59.Louzon SA, Bossarte R, McCarthy JF, Katz IR. Does Suicidal Ideation as Measured by the PHQ-9 Predict Suicide Among VA Patients? Psychiatr Serv 2016; 67(5):517–522. [DOI] [PubMed] [Google Scholar]
  • 60.Simon GE, Coleman KJ, Rossom RC, Beck A, Oliver M, Johnson E et al. Risk of suicide attempt and suicide death following completion of the Patient Health Questionnaire depression module in community practice. J Clin Psychiatry 2016; 77(2):221–227. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 61.Rudd MD. Recognizing flawed assumptions in suicide risk assessment research and clinical practice. Psychol Med 2021; e-pub ahead of print 7 July 2021; doi: 10.1017/S0033291721002750. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 62.Richards JE, Whiteside U, Ludman EJ, Pabiniak C, Kirlin B, Hidalgo R et al. Understanding Why Patients May Not Report Suicidal Ideation at a Health Care Visit Prior to a Suicide Attempt: A Qualitative Study. Psychiatr Serv 2019; 70(1):40–45. [DOI] [PubMed] [Google Scholar]
  • 63.Tourangeau R, Yan T. Sensitive Questions in Surveys. Psychol Bull 2007; 133(5):859–883. [DOI] [PubMed] [Google Scholar]
  • 64.Kraemer HC, Kazdin AE, Offord DR, Kessler RC, Jensen PS, Kupfer DJ. Coming to terms with the terms of risk. Arch Gen Psychiatry 1997; 54(4):337–343. [DOI] [PubMed] [Google Scholar]
  • 65.Hubbard AE, Kennedy CJ, van der Laan MJ. Data-Adaptive Target Parameters. Targeted Learning in Data Science: Causal Inference for Complex Longitudinal Studies. Springer International Publishing: Cham, 2018, pp 125–142. [Google Scholar]
  • 66.U.S. Department of Veterans Affairs. VA Solid Start. 2020. https://www.benefits.va.gov/transition/solid-start.asp. Accessed 14 July 2021. [Google Scholar]
  • 67.Geraci J, Goodman M, Seim R. ETS Sponsorship Program Helps Communities Fulfill Their Responsibility to Help Service Members Successfully Transition to Civilian Life. VAntage. Washington, DC: VA Office of Mental Health and Suicide Prevention, 2019. [Google Scholar]
  • 68.Miller IW, Gaudiano BA, Weinstock LM. The Coping Long Term with Active Suicide Program: Description and Pilot Data. Suicide Life Threat Behav 2016; 46(6):752–761. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 69.Rudd MD, Bryan CJ, Wertenberger EG, Peterson AL, Young-McCaughan S, Mintz J et al. Brief cognitive-behavioral therapy effects on post-treatment suicide attempts in a military sample: results of a randomized clinical trial with 2-year follow-up. Am J Psychiatry 2015; 172(5):441–449. [DOI] [PubMed] [Google Scholar]
  • 70.Mann JJ, Michel CA, Auerbach RP. Improving Suicide Prevention Through Evidence-Based Strategies: A Systematic Review. Am J Psychiatry 2021; 178(7):611–624. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 71.Bernecker SL, Zuromski KL, Curry JC, Kim JJ, Gutierrez PM, Joiner TE et al. Economic Evaluation of Brief Cognitive Behavioral Therapy vs Treatment as Usual for Suicidal US Army Soldiers. JAMA Psychiatry 2020; 77(3):256–264. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 72.Klonsky ED, May AM. The Three-Step Theory (3ST): A New Theory of Suicide Rooted in the “Ideation-to-Action” Framework. Int J Cogn Ther 2015; 8(2):114–129. [Google Scholar]
  • 73.Carroll D, Kearney LK, Miller MA. Addressing Suicide in the Veteran Population: Engaging a Public Health Approach. Front Psychiatry 2020; 11:569069. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 74.Millner AJ, Lee MD, Nock MK. Single-Item Measurement of Suicidal Behaviors: Validity and Consequences of Misclassification. PLoS One 2015; 10(10). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 75.Department of Defense Office of Inspector General. Evaluation of the Department of Defense’s Implementation of Suicide Prevention Resources for Transitioning Uniformed Service Members (DODIG-2022–030). 2021. https://www.dodig.mil/reports.html/article/2841764/evaluation-of-the-department-of-defenses-implementation-of-suicide-prevention-r/. Accessed 15 November 2021.

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplemental Material

RESOURCES