Predicting suicide attempts among soldiers who deny suicidal ideation in the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS)

Abstract

Most nonfatal suicide attempts and suicide deaths occur among patients who deny suicidal ideation (SI) during suicide risk screenings. Little is known about risk factors for suicidal behaviors among such patients. We investigated this in a representative sample of U.S. Army soldiers who denied lifetime SI in a survey and were then followed through administrative records for up to 45 months to learn of administratively-recorded suicide attempts (SA). A novel two-stage risk assessment approach was used that combined first-stage prediction from administrative records to find the subsample of SI deniers with highest subsequent SA risk and then used survey reports to estimate a second-stage model identified the subset of individuals in the high-risk subsample at highest SA risk. 70% of survey respondents denied lifetime SI. Administrative data identified 30% of this 70% who accounted for 81.2% of subsequent administratively-recorded SAs. A relatively small number of self-report survey variables were then used to create a prediction model that identified 10% of the first-stage high-risk sample (i.e., 3% of all soldiers) at highest SA risk (accounting for 45% of SAs in the total sample). We close by discussing potential applications of this approach for identifying future SI deniers at highest SA risk.

Most identified risk factors for suicide in the general population are actually predictors of suicide ideation (SI; Kessler, Borges, & Walters, 1999; Nock, Kessler, & Franklin, 2016). Knowledge of such risk factors is of limited value in predicting suicidal behaviors in clinical settings, where the focus is on the small proportion of patients with SI who go on to engage in suicidal behavior (ten Have et al., 2009). Little is known about the predictors of these behaviors among patients with SI. Accordingly, researchers have begun to investigate predictors of suicide attempts among ideators (e.g., Nock et al., 2018). However, these investigations do not address the much larger segment of the at-risk population made up of individuals who engage in suicidal behaviors despite denying SI. Some recent studies have reported that the majority of suicide attempts and suicide deaths occur among patients who denied SI at previous healthcare appointments or surveys (Louzon, Bossarte, McCarthy, & Katz, 2016; Simon et al., 2013). For the sake of conciseness, we refer to these individuals as “non-ideators,” but it is important to recognize that such individuals are non-ideators by self-report at a single point in time. Some may truly have no history of SI, but others may be concealing their SI. In addition, some may have no SI history as of the time of their healthcare encounter but subsequently develop SI.

The high proportion of SAs made by non-ideators poses a major challenge for healthcare providers who wish to prevent suicide, most of whom assess suicide risk using a gateway question that inquires about SI. Administering an in-depth assessment to all non-ideators is clearly not an option because risk of suicidal behavior is very small in this group, even though the majority of suicide attempts occur among them. Consequently, any action taken to screen the many non-ideators would be cost-ineffective; it could also have a negative impact on rapport if a provider continued to insist upon asking suicide-related questions of a patient who had explicitly stated that suicidality was not a concern.

In this paper we present the results of an innovative approach to identifying non-ideators at high risk of suicidal behaviors among US Army soldiers. The approach involves a two-stage assessment process (Kessler et al., in press). The first stage takes advantage of the extensive amount of administrative data available for soldiers to develop a prediction model that can be implemented passively to identify the small proportion of non-ideators who have elevated SA risk. Although predictions based on this first-stage model are incapable of identifying the small proportion of non-ideators who have sufficiently high SA risk to warrant preventive intervention, previous research suggests that such a model can identify a large proportion of soldiers who are at low SA risk for exclusion from further assessment (Ursano et al., 2016). The remaining non-ideators are then the focus of further analysis. These soldiers are administered a series of self-report questions that are used to estimate a second-stage model to identify non-ideators at highest SA risk. This two-stage approach limits the number of non-ideators required to receive this self-report assessment in an effort to keep patient burden to a minimum.

We apply this two-step approach here to predict nonfatal SAs among non-ideators in a large sample of U.S. Army soldiers who participated in the main survey component of the Army Study to Assess Risk and Resilience in Servicemembers (STARRS; Ursano et al., 2014). The likelihood of denying SI is high in this population given that reporting SI could have negative career implications. Yet the need to implement suicide prevention programs in this population is particularly urgent given that the Army suicide rate climbed steeply in recent years (Schoenbaum et al., 2014) and preventing suicide represents a major priority and area of investment for the Army (Department of the Army, 2012).

Method

Participants

Participants were drawn from the STARRS Consolidated All-Army Survey (AAS). We focus on the 27,501 Regular Army soldiers who participated in that survey, agreed to have their survey data linked to their Army administrative data, and whose survey dates were non-missing. The consolidated AAS is made up of three component surveys. The largest of the three was a representative survey of non-deployed soldiers no longer in Initial Military Training carried out in 2011-2012 that contributed 17,462 respondents to the consolidated AAS (95.0% survey consent rate, 97.3% survey completion rate among consenters, 61.3% administrative data linkage consent rate among survey completers). The second was a survey of soldiers from three Brigade Combat Teams preparing to deploy to Afghanistan in 2013 that contributed 8,558 respondents to the consolidated AAS (98.7% survey consent rate, 99.2% survey completion rate among consenters, 90.9% administrative data linkage consent rate among survey completers). This second survey was added because soon-to-deploy soldiers were under-represented for logistical reasons in the first survey. The third was a survey of soldiers in transit to and from mid-deployment leave in Kuwait in 2013 that contributed 3,987 respondents to the consolidated AAS (80.9% survey consent rate, 86.5% survey completion rate among consenters, 55.6% administrative data linkage consent rate among survey completers). This third survey was added because none of the respondents in the first two surveys were currently deployed. The three surveys are described in detail elsewhere, along with a description of the methods used for data collection and for weighting the combined data to adjust for differential probabilities of selection and discrepancies between the sample and the population on a range of administrative variables (Kessler, et al., 2013; Millner et al., 2018). Procedures for obtaining informed consent and collecting data in all three surveys were approved by the Human Subjects Committees of the Uniformed Services University of the Health Sciences, the Institute for Social Research at the University of Michigan, and all other collaborating organizations. The analyses presented here focus only on soldiers who responded in the negative to two survey questions about active (“Did you ever in your life have thoughts of killing yourself?”) and passive (“Did you ever wish you were dead or would go to sleep and never wake up?”) SI.

Measures

The outcome measure of post-survey suicide attempts.

The outcome variable, presence of a suicide attempt (SA) subsequent to AAS survey completion, was assessed prospectively using administrative data available through December 2014 from the Army STARRS Historical Administrative Data System (HADS). The HADS integrates records from numerous data sources, four of which include information on SA: (i) the Department of Defense Suicide Event Report (DoDSER; Gahm et al., 2012) and DoDSER’s predecessor (i.e., the Army Suicide Event Report; ASER; Gahm et al., 2012); (ii) the Military Health System Data Repository; (iii) the Theater Medical Data Store; and (iv) the TRANSCOM (Transportation Command) Regulating and Command and Control Evacuating System. These databases together contain comprehensive healthcare encounter information from military and civilian treatment facilities, deployed operations, and aeromedical evacuations. The data used to define SA (described in more detail elsewhere; Ursano et al., 2015) included attempts documented in DoDSER records and ICD-9-CM codes for self-inflicted poisoning or injury with suicidal intent (codes E950-E958) from any of the other databases. Information was recorded on the date of the first SA of every AAS respondent subsequent to the date of survey.

Administrative predictors of suicide attempts.

We included in our prediction models a number of basic socio-demographic variables (age, sex, race/ethnicity, education, marital status, number and ages of children, religious denomination, country of origin) and Army career variables (years since enlistment, rank, current deployment status, number of prior combat deployments, major command) that are available for all soldiers in Army administrative records that have shown associations with suicidal behavior in some literature. We also included in the models a composite score representing the predicted probability of suicide fatality (not attempt) developed from a comprehensive historical analysis of HADS data for the entire Army over the years 2004-2009. In other words, we assessed whether the predicted probability of death by suicide was a predictor of SA, as no comparable historical model exists for SA. That score combined predictions from separate models reported in earlier publications for suicides among soldiers with a history of psychiatric hospitalization (Kessler et al., 2015) and other soldiers with a history of outpatient treatment for mental health problems (Kessler et al., 2017) along with an unpublished model developed using similar methods for soldiers with no history of inpatient or outpatient treatment of mental health problems. Close to 500 administrative variables were considered for inclusion in these models and between 12 and 20 were selected as final predictors after adjusting for the possibility of false positives. The coefficients were then applied to HADS data available for each AAS respondents as of the month prior to their completion of their AAS survey in 2011-2013 to generate a predicted probability of subsequent suicide. The predicted probabilities were standardized to a mean of 0 and variance of 1 in the subsample of respondents who reported no history of SI in the survey to facilitate interpretation of the regression coefficient associated with that predictor. By adding this composite historical variable into the model, we were able to aggregate a great deal of risk information from the HADS in a single variable, thereby preserving statistical power. However, it remained necessary to include additional predictors (i.e., the socio-demographic and career variables mentioned above) because suicide fatality and suicide attempt are distinct events with differing risk factors.

AAS self-reported predictors of suicide attempts.

AAS self-reports were used as predictors of SA in a number of previous STARRS reports that used retrospective self-reports of lifetime SA as the outcome variable (Millner, et al., 2017; Millner et al., 2018; Nock et al., 2018). Seven categories of AAS self-reports that were found in these studies to be significant predictors of SA are considered in the prospective model we develop here: 8 predictors for lifetime mental disorders, 4 for severity of 30-day mental disorder symptoms (i.e., in the 30 days prior to the survey), 27 for lifetime exposure to traumatic events (e.g., sexual assault, combat exposure), 29 for 12-month exposure to more common stressful events (e.g., divorce, death of a loved one, being jailed), 5 for severity of chronic stressors at the time of survey (e.g., in such areas as finances and romantic relationships), 3 for traumatic brain injury (both lifetime exposure and severity of 30-day post-concussive symptoms), and 6 for social support-related factors.

Data Analysis

Figure 1 depicts the stages of the analytic process, including which subset of the AAS sample was analyzed at each stage.

Figure 1.

Data analytic steps. White boxes represent the sample to which each stage of the analysis was applied. Gray boxes represent the portion of the sample from the previous stage that was excluded.

First-stage analysis.

The goal of the first stage of data analysis was to predict SA from administrative data available at the time of the survey. As noted above in the section on measures, we used for this purpose a composite suicide prediction score based on an earlier historical analysis of the HADS data for the years 2004-2009 in addition to basic socio-demographic and Army career variables available for each AAS respondent as of the month of their survey. Discrete-time survival analysis with person-month the unit of analysis (Singer & Willett, 1993) was used to estimate the model. To increase computational efficiency, we created a case-control sample consisting of all person-months with SA plus twenty randomly-selected control person-months for each attempt. The control person-months were weighted by the inverse of their probability of selection to reconstruct their population proportions. We began model-building by estimating two preliminary models, one for the administratively-recorded socio-demographic variables described above in the section on measures and the other for the Army career variables. Both models controlled for time since survey and season (to adjust for the seasonal pattern of SAs for which evidence was found in previous STARRS studies). The significant predictors in those models along with the standardized HADS composite suicide prediction score described above in the section on measures were then included as the predictors in a final first-stage model. An ROC curve for the predicted values based on that model was then plotted to inspect the sensitivity (SN; the proportion of all observed SAs that occurred among soldiers in a given range of the risk distribution) and positive predictive value (PPV; number of observed SAs per 100,000 person-years among soldiers in that range of the risk distribution) in predicting administratively-recorded SAs.

Second-stage analysis.

Inspection of the ROC curve for the first-stage model led us to focus additional analysis on the subset of AAS respondents who had high predicted probabilities of SA based on administrative data. We used the AAS survey data as the predictors in this second-stage analysis. We again used discrete-time survival analysis with person-month as the unit of analysis, beginning with separate models for each of the seven categories of predictors, controlling for seasonality and number of months since the survey. Predictors with significant univariate associations with SA were combined to generate within-category multivariate models, which were then trimmed to exclude nonsignificant predictors. The predictors in each of these within-category multivariate models were then combined into a final second-stage model.

To evaluate the possibility of over-fitting, we used 20 replicates of 10-fold cross-validation (10F-CV; James, Witten, Hastie, & Tibshirani, 2013) to generate estimates of out-of-sample performance based on the final second-stage model. 10F-CV is an approach in which model prediction accuracy is assessed in a sample of people separate from the sample in which the model was estimated by dividing the sample into 10 subgroups of equal size, estimating the model 10 separate times leaving out cases in one subsample each time, and using the coefficients from the model estimated when a given subsample was excluded to make predictions only for that omitted subsample. A pooled ROC curve is then estimated based on these combined predicted values to estimate how well the model would predict if it was applied to a new sample. Prior simulations have shown that this method yields fairly accurate estimates of out-of-sample performance of prediction models when model results are evaluated in the same sample as the one in which the model was estimated, but that performance can be improved in samples with sparse outcomes, which is the type of sample we have here, if the 10F-CV is repeated 20 separate times with independent designation of subsamples (i.e., 20 replications of the sample being divided into 10 random subsamples) and results averaged across these replicates (Smith, Seaman, Wood, Royston, & White, 2014). This is what we did here.

This second-stage discrete-time survival model assumed that the joint predictive effects of the risk factors were linear and additive. However, most theories of suicidal behaviors hypothesize that non-additivities exist among risk factors (Selby, Joiner, & Ribeiro, 2014). To address this issue, we used the super learner (SL) ensemble machine learning algorithm (van der Laan, Polley, & Hubbard, 2007) to apply a series of interactive models to the predictors in the final stage-two model to find the best-fitting specification. In addition to a penalized additive model (elastic net; Zou & Hastie, 2005), the non-additive models in the SL ensemble included adaptive splines (Friedman, 1991), random forests (Breiman, 2001), support vector machines with radial and polynomial kernels (Steinwart & Christmann, 2008), Bayesian additive regression trees (Chipman, George, & McCulloch, 2010), and gradient boosting (Freund & Schapire, 1999). 5F-CV was used to generate an ROC curve based on the SL ensemble. We used 5F-CV (i.e., cross-validation dividing the sample into five separate subsamples) rather than the 20 replicates of 10F-CV used to estimate the ROC curve for the second-stage model because SL already uses internal 10F-CV both to estimate individual models and to develop the weights to combine results across component models.

Results

Prevalence of Suicide Attempts Separately Among AAS Ideators and Non-Ideators

Lifetime SI was reported by 13.2% (n = 3,647 of 27,501) of AAS respondents. Subsequent suicide attempts were administratively recorded for 243 of the AAS respondents, a weighted 32.5% of whom (n = 75) reported SI in the survey and 67.5% (n = 168) of whom did not. The association between survey reports of SI and subsequent administratively-recorded SA was statistically significant (OR 3.0, 95% CI 1.8-5.0); the SA rate among those who denied SI was 124.8 per 100,000 person-years compared to 378.0 per 100,000 person-years among those who reported SI.

First-Stage Model: Predicting Suicide Attempts from Administrative Data

We focused on the 86.8% of AAS respondents (n = 23,854) that reported no lifetime SI in the survey. These respondents were followed administratively for a total of 718,055 person-months until either the first SA after the survey was recorded (subsequent SAs were ignored in the analysis), they separated from service, they died, or it was December 2014, whichever came first. (Suicide fatalities, n = 11, were considered deaths rather than suicide attempts. This was done based on evidence that the predictors of nonfatal SA are quite different from the predictors of suicide death, the most striking example being the much higher suicide death rate among men than women but much higher nonfatal SA rate among women than men. A case might be made that suicide deaths were considered SAs and included in the analysis, although results are unlikely to have been affected strongly due to the small number of suicide deaths in the sample.) There were 168 person-months in which an attempt took place among non-ideators; therefore, sampling of 20 control person-months for each attempt resulted in a total analytic sample of 3,528 (i.e., 168*20 + 168) person-months.

The final first-stage model appears in Table 1. (Detailed results of the two preliminary models are available on request.) Odds of administratively-recorded SA were inversely associated with age and positively associated with racial/ethnic minority status, having less than college education, being a first-generation American, enlisted rank, number of prior deployments, and the standardized HADS composite suicide prediction score.

Table 1.

Final first-stage model of administrative characteristics predicting subsequent administratively-recorded suicide attempts among Regular Army soldiers who denied lifetime suicide ideation in the Consolidated AAS Survey (n ₌ 3,251)

	Distribution		Univariate		Multivariate
	Est.a	SE	OR	95% CI	OR	95% CI
Demographics
Age (in decades, mean)	2.9	0.0	0.5*	[0.3-0.8]	0.4*	[0.2-0.9]
Racial/ethnic minority (vs. non-Hispanic White)	37.6	1.7	2.8*	[1.7-4.6]	2.8*	[1.5-5.2]
Not a college graduate (vs. college or more)	74.3	2.3	17.7*	[3.9-79.4]	2.8	[0.4-19.8]
First generation (vs. others)b	14.6	1.2	3.2*	[1.5-6.6]	2.5*	[1.1-5.7]
Army career characteristics
Enlisted rank (vs. officer)	79.4	2.2	42.8*	[5.8-315.0]	5.1	[0.4-75.4]
Number of prior deploymentsc	1.3	0.1	1.4*	[1.1-1.8]	1.5*	[1.2-2.0]
HADS composite predicted suicide score (mean)d	0.0	1.0	2.0*	[1.5-2.7]	2.0*	[1.5-2.6]

Note. Based on a discrete-time person-month survival model that subsampled a probability sample of 20 controls for each suicide attempt and weighted the control person-months by the inverse of their probability of selection to reconstruct their population proportions, resulting in a total of 3,528 person-months. Model OR's control for seasonality and months since survey. Predictor values were defined as of the month of survey.

^*p < .05

^aAll estimates are percentages unless labeled otherwise.

^bSoldiers are defined as being first generation either if they were born outside the U.S. or both their parents were born outside the U.S.

^cThis variable was coded in the range 0-4. Soldiers with more than 4 deployments were coded 4 to stabilize the regression coefficient.

^dThis is a standardized transformation of a composite measure to generate a predicted-probability-of-suicide from HADS administrative data based on an earlier analysis of the HADS database for the years 2004-2009. See text for further detail.

The AUC of the model was 0.82. An inspection of PPV by decile of predicted risk showed a clear discontinuity between the three highest-risk deciles, where the SA rate was in the range of 458.2-1024.8 per 100,000 person-years, and the lower risk deciles, where the SA rate was in the range of 0.0-180.0 per 100,000 person-years, as shown in Table 2. The cumulative PPVs for the lower-risk deciles were all below the rate in the general U.S. population (Piscopo, Lipari, Cooney, & Glasheen, 2016). Half of all soldiers with SA were in the highest risk decile (SN = 50.5%) and 81.2% in the three highest risk deciles. Based on these results, we focused second-stage model-building on the 30% of observations with highest predicted risk.

Table 2.

The operating characteristics of predictions based on the model in Table 1 applied to all Regular Army soldiers who denied lifetime suicide ideation in the Consolidated AAS Survey (n = 3,251)

	Positive predictive valuea		Sensitivityb
Risk decile	Attempts/ 100K PY	SE	%	SE
91-100%	1024.8	166.8	50.5	7.3
81-90%	324.0	79.8	16.8	5.4
71-80%	458.4	140.4	13.9	5.5
61-70%	89.2	37.8	2.3	1.1
51-60%	8.0	42.8	2.4	1.1
41-50%	180.0	100.5	7.9	4.2
31-40%	172.8	73.2	4.3	2.0
21-30%	93.8	48.8	1.9	1.4
11-20%	34.6	27.3	0.1	0.1
0-10%	0.0	0.0	0.0	-
Total	232.8	2.5	100.0	-

Note. As explained in the text, the 20:1 sampling of control person-months to attempt person-months, followed by selection of only those person-months of participants who denied lifetime SI, resulted in a total of 3,528 person-months. Values reflect weighted data.

^aPositive predictive value refers to the number of attempts per 100,000 person-years in each decile.

^bSensitivity refers to the weighted percentage of observed attempts that fell in each decile.

Second-Stage Model: Predicting Suicide Attempts from AAS Survey Data

We began model-building in this 30% high-risk segment of the case-control sample by estimating a series of seven preliminary models, one for each of the seven categories of AAS predictors described above in the section on measures. All these models controlled for seasonality, months since survey, and number of prior deployments (the variables from the first model that remained significant in the 30% high-risk subsample). (Detailed results of the seven preliminary second-stage models are available on request.) We then estimated a final second-stage model, shown in Table 3, that combined all the significant predictors from these seven preliminary models. Odds of administratively-recorded SA were positively associated with having any lifetime mental disorder, ever being bullied while in the Army, ever being responsible for the death of an enemy, and having been in jail or custody in the past year (“spent time in jail, stockade, correctional custody, or brig” on the survey). The AUC of the model was 0.62, computed by applying the coefficients from 20 replicates of 10F-CV model based on the matched case-control subsample to the entire population of all person-years observed among the 30% of the sample defined in the first-stage model as being at high risk (n = 718,055). As noted in the section on analysis methods, replicated 10F-CV was used to estimate the likely performance of the model if applied in a different sample.

Table 3.

Final second-stage model of AAS survey variables predicting subsequent administratively-recorded suicide attempts among the high-risk (based on the first-stage model) Regular Army soldiers who denied lifetime suicide ideation in the Consolidated AAS Survey (n = 1,141)

	Distribution		Univariate		Multivariate
	Est.a	SE	OR	95% CI	OR	95% CI
DSM-IV symptoms and disorders
Any lifetime mental disorder	47.6	3.6	4.2*	[1.7-10.1]	3.5*	[1.0-12.0]
Current mental disorder symptoms (mean)b	0.0	1.0	1.4*	[1.1-1.8]	0.9	[0.6-1.4]
Traumatic and stressful experiences
Ever bullied by unit	15.9	2.2	2.3*	[1.1-4.7]	2.2*	[1.1-4.6]
Ever responsible for death of an enemy	6.6	1.1	3.1*	[1.6-6.0]	3.1*	[1.2-7.7]
Recent interpersonal problems (mean)c	0.3	1.0	1.8*	[1.2-2.6]	1.5	[0.9-2.5]
Spent time in jail	2.0	0.6	4.6*	[1.1-18.7]	6.8*	[1.7-27.4]
Recent general stressors (mean)d	0.0	1.0	1.5*	[1.1-2.0]	1.1	[0.7-1.8]
Traumatic brain injury (TBI)
More-than-mild TBI in past 5 years	6.5	1.2	3.7*	[1.3-10.7]	1.3	[0.4-4.4]
Any other TBIe	26.7	3.1	1.0	[0.4-2.4]	0.5	[0.2-1.3]
HADS composite predicted suicide score (mean)f	0.0	1.0	1.5	[1.0-2.1]	1.4	[0.9-2.0]

Note. The sample for this model represents the 30% high-risk subsample of the full sample shown in Table 1 and has a total of 1,248 person-months. See the text for a description of the method used to define the high-risk subsample. Model estimates reflect weighted data. Model OR's control for seasonality and months since survey. Predictor values reflect information at the month of survey.

^*p < .05

^aAll estimates are percentages unless labeled otherwise.

^bCurrent mental disorder symptoms is the standardized sum of the 4 scores for severity of depression, anxiety, irritability, and PTSD. Higher values reflect more severe symptoms.

^cRecent interpersonal problems is a summary measure of four individual events: separation/divorce from spouse/partner, spouse/partner cheated, betrayal by someone close, and serious ongoing arguments/break-up with friend/relative. Values ranges from 0-3, with 0 indicating no recent interpersonal problems (75.1% of sample), 1 indicating exactly 1 (13.0% of sample), 2 indicating exactly 2 or 3 (9.9% of sample), and 3 indicating exactly 4 (2.0% of sample).

^dRecent general stressors is the standardized sum of the maximum scores in each of five life areas: financial situation/career, health, love life, relationship with family and health of loved ones, and problems with unit. Higher values reflect more severe life stress.

^eAny other TBI included more-than-mild TBI that occurred 6 or more years ago and any mild or any less-than-mild TBI.

^fThis is a standardized transformation of a composite measure to generate a predicted-probability-of-suicide from HADS administrative data based on an earlier analysis of the HADS database for the years 2004-2009. See text for further detail.

We also used all variables included in this final model (not just those that were significant when entered together) to estimate an SL ensemble machine learning model that allowed for nonlinearities and interactions among the predictors to see if the performance of the final logistic model could be improved on by relaxing the linearity and additivity assumptions implicit in that model. This entire series of procedures was applied five separate times in calculating the 5F-CV SL AUC, which was 0.83.

We inspected the ROC curves for the cross-validated second-stage and SL models, shown in Figure 2, and used these to guide the creation of relatively homogeneous risk strata, which are shown in Table 4. The SL model had substantially better out-of-sample performance than the original second-stage model. The highest risk stratum in the SL model, which consisted of the 5% of high-risk soldiers (i.e., 5% of 30% = 1.5% of all non-ideators) with highest cross-validated predicted SA risk, had a PPV of 3,273.6 per 100,000 person-years and a SN of 48.1% (compared to PPV = 702.0/100,000 person-years and SN = 10.1% in the cross-validated second-stage model). The second highest risk stratum in the SL model consisted of another 5% of high-risk soldiers, who had a PPV of 514.1 per 100,000 person-years and a SN of 7.4%, resulting in 55.5% of all SAs occurring among the 10% of soldiers at highest risk (within this 30% high-risk sample already selected by the administrative model, i.e., 3% of all non-ideators). It is striking that the 50% of soldiers with lowest predicted risk in the high-risk sample had PPV (61.6/100,000 person-years) considerably lower than in the general U.S. population (Piscopo et al., 2016), indicating that despite their classification as higher-risk based on the first-stage administrative predictors, the AAS survey data subsequently showed that they were actually lower-risk.

Fig. 2.

Receiver operating characteristics (ROC) curve for 20 replicates of ten-fold cross-validated logistic regression versus five-fold cross-validated SuperLearner in predicting prospective suicide attempts among high-risk subsample of non-ideators.

Table 4.

Operating characteristics of final logistic and super learner models predicting suicide attempt among high-risk and full sample of lifetime non-ideators (n = 1,189)

	Sensitivitya
	High-risk subsample				Full sampleb				Positive predictive valuec
	Logistic regression		SuperLearner		Logistic regression		SuperLearner		Logistic regression		SuperLearner
Risk stratumd	%	SE	%	SE	%	SE	%	SE	Attempts	SE	Attempts	SE
95-100%	6.0	2.0	48.1	7.0	3.8	1.6	39.1	6.1	702.0	250.8	3273.6	848.8
90-95%	8.7	3.0	7.4	4.0	7.6	3.2	6.0	3.3	1024.8	418.8	514.1	304.7
90-100%	14.7	4.3	55.5	7.1	11.4	3.8	45.1	6.2	862.8	304.8	1909.8	423.0
71-90%	35.1	4.4	26.3	6.5	25.5	4.5	21.4	5.5	1028.4	158.4	457.8	145.9
50-70%	18.2	2.2	9.5	4.1	13.1	3.8	7.7	3.3	534.0	80.4	156.1	73.3
0-50%	32.0	6.1	8.7	3.8	22.1	5.7	7.0	3.1	376.8	92.4	61.6	28.1
0-30%	15.6	5.1	2.4	1.4	9.5	4.6	1.9	1.1	304.8	112.8	27.4	16.2

Note. Selection of the person-months in the top 30% of risk based on the administrative model (see Tables 1 and 2) resulted in a total of 1,246 person-months. Values reflect weighted data.

^aSensitivity refers to the weighted percentage of observed attempts that fell in each quantile.

^bSensitivity for the full sample is the percentage of suicide attempts made by respondents in each quantile as a proportion of all suicide attempts in the full non-ideator sample, as opposed to the attempts made by the 30% of non-ideators classified as "high-risk" (who made 81.7% of all attempts among non-ideators). Thus, the 95-100% row shows the sensitivity among all non-ideators for the 0-1.5% (0.05*0.3 = 0.015) highest-risk of all non-ideators.

^cPositive predictive value refers to the number of attempts per 100,000 person-years in each stratum.

^dRisk strata were created by collapsing cross-validated ventiles with comparable likelihood ratios of SA (Deeks & Altman, 2004).

Discussion

The goal of this study was to determine the feasibility of developing a two-step procedure for identifying non-ideators at risk of suicide attempt among U.S. Army soldiers who denied suicidal ideation. The study had three key findings. First, consistent with recent prospective studies in other settings (Louzon et al., 2016; Simon et al., 2013), the preponderance of attempts (67.5%) occurred among soldiers who denied any lifetime suicidal ideation. This is especially notable given that soldiers who completed the survey were assured their responses would be confidential, which should have reduced motivation to conceal information. It may, then, be a robust phenomenon that most suicide attempts are made by non-ideators, at least among Army soldiers.

Second, the two-stage procedure showed the potential for strong predictive performance while limiting burden on patients and providers. The first-stage model was able to classify 70% of non-ideators as having minimal risk of suicide attempt based solely on administrative data. The second-stage model, which was based on responses to a limited number of self-report survey questions, was able to identify an additional 15% of non-ideators as minimal-risk (i.e., 50% of 30%) and classify the remainder as high-risk, with almost half of all attempts among non-ideators made by the stratum containing the 3% (i.e., 10% of 30%) at highest risk.

This model performance is particularly impressive given that it might be expected that predicting attempts among soldiers who denied SI would be more difficult than predicting attempts among ideators for several reasons. First, individuals who deny suicidal ideation may be less willing or able to report risk factors accurately. Second, the characteristics of ideation and other suicidal behaviors are strong risk factors for transitioning from ideation to attempt (Nock et al., 2018), whereas such predictors do not exist for non-ideators. Finally, it is possible that attempts among non-ideators are more frequently driven by situational variables that unfold quickly, whereas the drivers of attempts among ideators may be more stable or chronic (Millner, Lee, & Nock, 2017). It is striking that despite these challenges we were able to predict attempts among non-ideators with good accuracy.

A third key finding is that risk factors for suicide attempts among non-ideators in the Army are largely comparable to predictors found in other studies, although additional research is needed to clarify this by directly comparing ideators and non-ideators. Several of the risk factors found here correspond to predictors found among all Army soldiers in previous STARRS studies, including age, education, rank, deployment history, and mental illness history (e.g., Millner et al., 2017; Nock et al., 2014; Ursano et al., 2015; Ursano et al., 2016). The elevated risk of suicide attempt among racial/ethnic minority non-ideators contrasts with the finding in some previous STARRS studies (Millner et al., 2017; Ursano et al., 2015) that identifying as a racial/ethnic minority is protective, but the civilian literature is mixed, finding complex and inconsistent associations between racial/ethnic minority status and risk for suicidal behaviors (Mościcki, 2014). The finding that elevated life stress and interpersonal stressors, particularly bullying victimization, are risk factors is congruent with the civilian literature (e.g., Liu & Miller, 2014; McFeeters, Boyda, & O’Neill, 2015; Nielsen, Nielsen, Notelaers, & Einarsen, 2015; van Geel, Vedder, & Tanilon, 2014) and with a previous STARRS study finding that childhood bullying victimization is a risk factor for suicide attempts (Campbell-Sills et al., 2017). A handful of studies in veterans found associations between killing enemy combatants and suicide attempts and ideation (Bryan et al., 2015). Finally, regarding the association between recent jail time and suicide attempt, there is a body of research showing increased risk for suicide among recently-released prisoners (Zlodre & Fazel, 2012), and that elevated risk may be related to a stable set of traits or may persist (Kariminia et al., 2007). Most of the past studies cited here have not differentiated between ideators and non-ideators. Consequently, it is unclear whether overlap between the risk factors found in this study and those in previous studies reflects the fact that most individuals are non-ideators, so they comprise a large part of previous studies’ samples, or whether the risk factors for ideators and non-ideators are truly similar. Future studies would need to directly compare ideators and non-ideators to address this question.

Limitations and Future Directions

The circumstances in which the survey was administered might limit the generalizability of these findings. As noted previously, participants in this study were guaranteed confidentiality when asked about both suicidal ideation and risk factors. This would not be the case in a healthcare setting nor in any other setting in which self-reported information might be used to determine the need for further screening or intervention. And in the Army, such information could have negative career consequences. Higher rates of concealing ideation might consequently be expected when applying these findings in clinical practice in the military and possibly in other settings. There might also be some distortions in self-reports about other risk factors and in the predictive model based on such differences. As a result, any efforts on the part of the Army to develop a system to collect self-report data of the sort we used to develop our model would need to experiment with in order to determine the best way to collect such information and the strength of the resulting prediction model.

A related limitation in applying our results to a civilian population is that the rich administrative data available for all soldiers are either not available or not relevant for the general population. Given this limitation, a promising direction for this line of work in future research on civilians might be to develop a first-stage model that combines predictor variable data from electronic medical records with publicly available electronic administrative data systems available from commercial sources that might approximate the kinds of administrative data available for soldiers (e.g., www.accurint.com).

Another limitation is that the model was used to predict only the subset of SAs that come to medical attention. We know from comparisons of STARRS survey data with administrative data linked to survey reports that a substantial proportion of the SAs reported by soldiers in their surveys are not recorded in administrative records. And there are presumably yet other SAs that are both denied in surveys and not recorded in administrative records. Access to comprehensive data on all SAs would presumably allow a model to be created that would improve on the accuracy of our model in projecting future SA risk. Future research that uses panel survey data could address this limitation by using information reported in the baseline survey to predict the subsequent occurrence of SAs either reported in subsequent surveys and/or recorded in administrative records.

An additional limitation of this study is that, although strong, the predictive performance of the model is far from perfect. Enriching the feature set of previously-existing administrative data in the first-stage model could shrink the proportion of individuals who would require second-stage assessments. For example, the first-stage model could use text scraped from electronic medical record notes or social media postings (Ben-Ari & Hammond, 2015; Bryan et al 2017). Performance of the second-stage model might also be improved, since it is unlikely that the questions asked in the AAS happened to include the perfectly optimal items for predicting suicidal behavior. Future investigations could add other self-report measures or performance-based measures such as the Suicide Implicit Association Test, which in one study predicted suicide attempts beyond patients’ report of ideation and likelihood of reattempt (Nock et al., 2010). Further, the present study did not differentiate between those who attempted suicide soon after survey administration and those who attempted suicide several years later. The ideal model may differ for individuals at imminent versus long-term risk. Developing models with a variety of time horizons could improve accuracy and could inform the nature of the intervention offered (e.g., crisis intervention versus longer-term skills training or psychotherapy).

Despite this study’s limitations, we believe the strategy presented here is unique in suggesting a practical way to assess SA risk among non-ideators. The optimal way to do so is yet to be determined, though, because the variables available to us are unlikely to be the best ones in building models and we do not know either the ideal setting or the ideal frequency for this assessment. As one possible implementation in the Army, though, one could imagine a scenario in which all soldiers who report for routine healthcare visits are asked about suicidal ideation, perhaps on a tablet or computer while in the waiting room. For those who deny ideation, the tablet could compute the risk score based on the first-stage algorithm by pulling the soldier’s administrative data (i.e., with no additional input). Soldiers deemed at minimal risk based on the first-stage administrative model would receive no further assessment, whereas soldiers at higher risk would be administered an additional battery of self-report questions and a second-stage algorithm would be applied to those responses to determine the need for a more in-depth clinical assessment.

The question remains, then, what steps a clinician should take when faced with an at-risk individual who denies ideation after an in-depth clinical assessment. Appropriate interventions for non-ideators have yet to be identified; the best-supported treatments for reducing suicide attempt risk among servicemembers (e.g., Rudd et al., 2015) involve directly treating suicidal thoughts and behaviors, but such treatments are obviously not applicable for individuals who deny ideation. However, the risk factors for attempts found in this study do hint at possible targets for intervention among U.S. Army soldiers who deny ideation but who are predicted to be at high risk. If these risk factors play a causal role in suicide attempts (which would need to be confirmed in future studies), then treating those risk factors should be a viable strategy for reducing risk. Notably, we found that meeting criteria for one or more common mental disorders is a risk factor for SA. Being bullied was another significant predictor. Interventions for treating common mental disorders exist, as do training programs also exist for coping with interpersonal stressors. Such interventions might reduce SAs among high-risk soldiers who deny ideation. Given the observational nature of this study and the lack of evidence for the effects of such interventions on suicide attempts, though, these suggestions remain speculative.

Once way of advancing our understanding of potentially effective preventive interventions for high-risk soldiers who deny SI would be to carry out in-depth suicide-focused assessments of need for services with such soldiers and use the information obtained in such assessments both to guide treatment planning and to develop third-stage models that might be used for clinical decision support in selecting optimal treatments keyed to the varying risk factors of high-risk soldiers (Kessler et al., in press). Future research is also needed to replicate our findings regarding optimal first-stage and second-stage predictors and to investigate the ideal frequency and setting for assessment in order to pinpoint soldiers at high risk of suicidal behaviors for in-depth assessments and targeted preventive interventions.