Predicting U.S. Army suicides after hospitalizations with psychiatric diagnoses in the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS)

Abstract

IMPORTANCE

The U.S. Army experienced a sharp rise in suicides beginning in 2004. Administrative data show that among those at highest risk are soldiers in the 12 months after inpatient treatment of a psychiatric disorder.

OBJECTIVE

To develop an actuarial risk algorithm predicting suicide in the 12 months after US Army soldier inpatient treatment of a psychiatric disorder to target expanded post-hospital care.

DESIGN, SETTING, AND PARTICIPANTS

There were 53,769 hospitalizations of active duty soldiers in 2004–2009 with ICD-9-CM psychiatric admission diagnoses. Administrative data available prior to hospital discharge abstracted from a wide range of data systems (socio81 demographic, Army career, criminal justice, medical/pharmacy) were used to predict suicides in the subsequent 12 months using machine learning methods (regression trees, penalized regressions) designed to evaluate cross-validated linear, nonlinear, and interactive predictive associations.

MAIN OUTCOME

Suicides of soldiers hospitalized with psychiatric disorders in the 12 months after hospital discharge.

RESULTS

68 soldiers died by suicide within 12 months of hospital discharge (12.0% of all Army suicides), equivalent to 263.9 suicides/100,000 person-years compared to 18.5 suicides/100,000 person-years in the total Army. Strongest predictors included socio-demographics (male, late age of enlistment), criminal offenses (verbal violence, weapons possession), prior suicidality, aspects of prior psychiatric inpatient and outpatient treatment, and disorders diagnosed during the focal hospitalizations. 52.9% of post-hospital suicides occurred after the 5% of hospitalizations with highest predicted suicide risk (3,824.1 suicides/100,000 person years). These highest-risk hospitalizations also accounted for significantly elevated proportions of several other adverse post-hospital outcomes (unintentional injury deaths, suicide attempts, re-hospitalizations).

CONCLUSIONS AND RELEVANCE

The high concentration of risk of suicides and other adverse outcomes might justify targeting expanded post-hospital interventions to soldiers classified as having highest post-hospital suicide risk, although final determination requires careful consideration of intervention costs, comparative effectiveness, and possible adverse effects.

The U.S. Army suicide rate, although historically below the civilian rate, has climbed since 2004¹ to exceed the civilian rate.² Despite numerous efforts to address this problem, including universal interventions (e.g., Ask/Care/Escort prevention education and depression/PTSD/suicide screening in all primary care encounters) and high-risk interventions (e.g., post-deployment screening),³ the Army suicide rate has continued to climb. One potentially important group for targeted interventions is soldiers recently discharged from inpatient psychiatric treatment. Such patients have long been known to have high suicide risk.⁴ U.S. military administrative data document an 8-fold elevated suicide risk in the 3 months after psychiatric hospitalization and 5-fold elevated risk over the remainder of the 12 months post-hospitalization.⁵ A recent report pointing to similar patterns among civilians called for expansion of post-hospital suicide preventive interventions,⁶ noting that such interventions in the UK (e.g., required out-patient visits within one week of hospital discharge, assertive outreach for missed outpatient appointments, 24-hour community crisis teams, intensive community support for patients difficult to engage in traditional services) were associated with significant before-after reductions in post-hospital suicides.⁷

Despite potential benefits of post-hospital interventions, it is important to realize that suicide is a rare outcome even among recent-discharged psychiatric inpatients.⁸ This means that the benefits of delivering intensive post-hospital suicide prevention interventions to all recently-discharged inpatients would be low. A more rational allocation of treatment resources would be to combine relatively inexpensive universal interventions (e.g.,⁹) with more intensive targeted high-risk interventions.⁴ However, this tiered approach would require developing a reliable risk stratification scheme. The Departments of Veterans Affairs and Defense (VA/DoD) recently called for this kind of differentiation in their Clinical Practice Guideline (CPG) for the Assessment and Management of Patients at Risk for Suicide.¹⁰ However, the CPG provided little concrete guidance on how these assessments should be implemented. Research has shown consistently that clinicians are not very accurate in making such assessments.^11–14

One potentially promising approach to assessing post-hospital suicide risk would be to use administrative data available during hospitalization to generate an actuarial post-hospital suicide risk algorithm. Previous research has shown that actuarial suicide prediction is much more accurate than prediction based on clinical judgment.^11–14 An increasing number of computerized risk algorithms are being used as clinical decision support tools in other areas of medicine and have been shown to improve clinical processes.^15,16 Skepticism exists about developing such an algorithm for post-hospital suicide interventions based on the relatively weak associations found in previous studies between in-hospital predictors and subsequent suicides.¹⁷ However, a stronger risk algorithm might be developed in the Army due to the availability of integrated administrative data for all Army personnel. Absence of such data in the general population is widely-recognized as an impediment to big data healthcare solutions.¹⁸ A number of empirical studies have documented strong predictive associations between integrated Army and Department of Defense (DoD) administrative data and subsequent Army suicides,^19–23 although none attempted to develop a risk algorithm for post-hospital suicides. The objective of this study was to develop such an algorithm using administrative data from the Historical Administrative Data System (HADS) of the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS).²⁴

METHODS

Sample

There were 53,769 Regular Army hospitalizations in 2004–2009 with any ICD-9-CM psychiatric admission diagnosis exclusive of tobacco use disorders (See eTable 1 at http://www.armystarrs.org/publications). These hospitalizations involved 40,820 soldiers (30,763 with one hospitalization, 6,929 two, 3,128 more than two), representing 0.8% of all Regular Army soldiers in any 12-month period. We excluded the 13,936 additional hospitalizations where nicotine dependence was the only psychiatric diagnosis, as these were invariably for physical disorders and nicotine dependence was noted based on withdrawal during hospitalization. There was no elevated post-hospital suicide risk among these cases. We also excluded the 406 additional hospitalizations occurring through emergency departments due to a suicide attempt without an accompanying ICD-9-CM psychiatric diagnosis. Four of these 406 died in hospital, whereas none of the others died by suicide in the next 12 months. Based on evidence from other studies that predictors of post-hospital suicide vary with time since discharge and elevated risk persists 12 months post-discharge,²⁵ a discrete-time person-month survival file was created to examine suicides in the 12 months after hospital discharge, censoring all person-months at the beginning of new hospitalizations or terminations of active duty, and allowing interactions between substantive predictors and time since hospital discharge. All person-months with suicide were coded 1 on the outcome and all others coded 0. This file contained 334,936 person-months, an average of 6.2 months (334,936/53,760) after hospital discharge. This low average reflects high rates of termination of service and re-hospitalization within 12 months of each hospitalization.

Measures

The HADS includes data from 38 Army/DoD administrative data systems.²⁶ (See eTable 2 at http://www.armystarrs.org/publications) Troister et al.,²⁷ in a comprehensive review of 8 published studies of predictors of civilian post-hospital suicides, found five replicated classes of predictors: (i) socio-demographics (the most consistent being male gender and recent job loss); (ii) history of prior suicidal behaviors; (iii) quality of care (e.g., low continuity of care); (iv) time since hospital discharge (inversely related to suicide risk); and (v) other psychopathological risk factors (the most consistent being non-affective psychosis, mood disorders, and multiple comorbid psychiatric disorders). More recent studies found similar predictors.^17,28,29 We extracted HADS variables operationalizing these predictors and added Army career variables found to predict military suicides,^19–22 unit variables, criminal justice variables (violent crime victimization-perpetration), and measures of registered weapons. Importantly, all predictors other than those involving the hospitalization were defined as of the month before hospitalization, while predicted suicides were in the 12 months after hospital discharge.

We cast a wide net in extracting HADS measures of the predictor constructs. For example, we distinguished 23 categories of psychiatric diagnoses defined largely by aggregated ICD-9-CM codes (e.g., ADHD/learning disorders [ICD-9-CM 314.0-315.9]), 8 additional categories of behavioral stressors (e.g., marital problems, other stressors/adversities, suicidal ideation and self-damaging behavior), and summary measures of any prior admission diagnoses, admission count variables, and parallel outpatient variables (eTable 1 at http://www.armystarrs.org/publications).We also included NDC psychotropic medication codes collapsed into 15 categories (e.g., antianxiety antidepressant, antipsychotic) and 25 sub-categories (e.g., SSRI, SNRI, TCA) based on the First Databank (FDB) Enhanced Therapeutic Classification System™ (http://www.fdbhealth.com) (eTable 3 at http://www.armystarrs.org/publications). A total of 421 individual variables were constructed (eTable 4 at http://www.armystarrs.org/publications).

As the HADS data systems were not developed for research, there was more missing-inconsistent data in some (e.g., socio-demographic) component datasets than in research datasets. However, as HADS datasets are updated monthly, missing values typically appeared in earlier and/or later months, allowing nearest neighbor imputations. Remaining missing values were resolved using randomly selected multiple imputations.³⁰ Inconsistencies were reconciled using rational imputations (e.g., a soldier classified female one month but male others was recoded male).

Analysis methods

Discrete-time (person-month) survival analysis³¹ was used to predict suicides in the 12 months after hospitalization in three steps. First, functional forms of bivariate associations were examined and predictors transformed (usually sets of nested dichotomies but some collapsed-truncated continuous variables) to explore nonlinear multivariate associations. Second, all predictors were discretized and analyzed with 100 regression trees in distinct bootstrap pseudo-samples using R-package rpart³² to prevent over-fitting³³ and allow detecting interactions among predictors.^25,28 Third, predictors having significant bivariate associations and interactions emerging in 10%+ of regression trees were included as predictors in multivariate survival models.

A central challenge in the third step was multicollinearity among the 421 predictors. The classic way to address this problem is with stepwise analysis³⁴ but this over-fits.³⁵ Machine learning methods reduce over-fitting.^36,37 The machine learning method we used was the elastic net,³⁸ a penalized regression method that provides stable and sparse estimates of model parameters by explicitly penalizing over-fitting with a composite penalty λ{MPP × P_lasso + (1 − MPP) × P_ridge where MPP is a mixing parameter penalty with values between 0 and 1 that controls relative weighting between two types of penalties, the lasso penalty (P_lasso) and the ridge penalty (P_ridge). The parameter λ controls the total amount of penalization.³⁹ The ridge penalty handles multicollinearity by shrinking all coefficients smoothly towards 0 but retains all variables in the model.⁴⁰ The lasso penalty allows simultaneous coefficient shrinkage and variable selection, tending to select at most one predictor in each strongly correlated set, but at the expense of giving unstable estimates in the presence of high multicollinearity.⁴¹ The elastic net approach of combining the ridge and lasso penalties has the advantage of yielding more stable and accurate estimates than either ridge or lasso alone while maintaining model parsimony.³⁸

The three-step approach of combining regression trees with penalized regression for variable selection enabled us to incorporate possible interactions and non-linearities in a clinically meaningful way while controlling for possible over-fitting. R-package glmnet⁴² was used to estimate penalized models with MPPs of 0.1, 0.4, 0.7, and 1.0. (MPP=0.0 was not used due to multicollinearity in the full predictor set.) Internal 10-fold cross-validation selected the coefficient in front of the penalty. Comparative fit across the 20 specifications (i.e., 4 values of MPP for each of 5 constraints on number of predictors) was evaluated by inspecting area under the receiver operating characteristic curve (AUC) and Concentration of Risk (CR). CR is the proportion of observed suicides after hospitalizations in each ventile (i.e., 20 groups of hospitalizations of equal frequency) ordered by predicted suicide risk. Suicide risk of each hospitalization was calculated using coefficients to project risk as of 12 months after hospital discharge regardless of observed hospitalization data and censoring and standardized by time of hospitalization to adjust for temporal variation in suicide risk. Given that the number of hospitalizations per ventile was much larger than the number of suicides, we focused on CR in the highest-risk ventile in selecting the best penalized model.

Once a best penalized model was selected, a conventional discrete-time survival model with a logistic link function was estimated using the same predictors as the best penalized model to examine how much the penalty reduced model fit. As variance inflation factor (VIF) of coefficients in this model showed estimates to be unstable, we also used forward stepwise analysis with a .05-level entry criterion to select a stable subset of predictors for a reduced version of the logistic model. Coefficients in this reduced logistic model were then exponentiated to create odds-ratios (ORs) for ease of interpretation. Ventiles from the best penalized model were then collapsed into risk strata using the logic of stratum-specific likelihood ratios.⁴³ CR, AUC, and the standardized (for amount of uncensored time observed after each hospitalization) suicide rates per 100,000 person years were calculated for these risk strata. Finally, parallel rates of risk were calculated for unintentional injury deaths, attempted suicides, and re-hospitalizations in the same ventiles to evaluate other adverse outcomes associated with post-hospital suicide risk.

RESULTS

Patterns of post-hospital suicide

Sixty-eight hospitalized soldiers died by suicide within 12 months of hospital discharge (263.9 suicides/100,000/person-years versus 18.5/100,000 in the total Army),²³ representing 12.0% of all Army suicides. An additional 157 hospitalized soldiers died in other ways and 22,010 others terminated active duty for other reasons (e.g., administrative separation, retirement) within 12 months of hospital discharge.

Bivariate associations of predictors with suicide

No interactions emerged in more than 10% of regression trees: However, nearly one-third (31.0%) of the 421 bivariate associations between individual predictors and suicides were significant at the .05 level. (eTables 5–9 and 11–15 at http://www.armystarrs.org/publications). All these variables were used in the penalized multivariate models.

Selecting a best penalized survival model

Ten-fold cross-validation showed that AUC was maximized across the 20 penalized survival models for MPP=1.0 (lasso) with 73 predictors and MPP=0.1–0.7 with 72–122 predictors. (Figure 1) As the lasso model yielded the best cross-validated CR in the highest-risk ventile (52.9%), (Table 1) we estimated a conventional discrete-time survival model with a logistic link function using the same 73 predictors. This model had much higher AUC (0.89) and CR in the highest-risk ventile (61.8%) than the lasso model with the same predictors, but this was because of over-fitting (VIF > 5 for 6 coefficients). Forward stepwise analysis selected a more stable set of predictors in a reduced logistic model and this model, which contained 20 predictors, had slightly lower AUC (0.84) and CR in the highest-risk ventile (50.0%) than the lasso model.

Figure 1. Receiver Operating Characteristic (ROC) curves for discrete-time (person-month) elastic net penalized survival models with different mixing parameter penalties (MPPs) and for a conventional discrete-time survival model predicting post-hospital suicide.

¹Elastic net penalized survival models were estimated with different MPPs and allowing up to 421 predictors. The best cross-validated model was MPP=1.0 with 73 predictors. A conventional discrete-time survival model containing the same 73 predictors was unstable (VIF > 5.0 for 6 predictors). As a result, we used forward stepwise analysis with a .05 level entry criterion to select a more stable subset of the 73 predictors. Twenty predictors entered that model. The ROC curve shown here for the conventional model is based on those 20 predictors.

Table 1.

Concentration of risk (CR) in the 5% of hospitalizations (n=2,689) with highest predicted suicide risk, area under the receiver operating characteristic curve (AUC), and number of selected predictors (n_p) for elastic net discrete-time survival models varying in mixing parameter penalty (MPP) and approximate number of allowed predictors in the total sample of hospitalizations (n=53,769)¹

	Mixing Parameter Penalty (MPP)
	0.1	0.4	0.7	1.0
I. Allowed predictors = 25
CR	26.5	29.4	35.3	36.8
AUC	0.71	0.75	0.77	0.79
(np)	(30)	(27)	(26)	(30)
II. Allowed predictors = 50
CR	29.4	41.2	42.6	50.0
AUC	0.74	0.80	0.82	0.84
(np)	(53)	(51)	(53)	(56)
III. Allowed predictors = 100
CR	45.6	51.5	51.5	52.9
AUC	0.82	0.85	0.85	0.85
(np)	(109)	(89)	(72)	(73)
IV. Allowed predictors = 200
CR	48.5	51.5	51.5	52.9
AUC	0.84	0.85	0.85	0.85
(np)	(122)	(89)	(72)	(73)
V. Allowed predictors = 421
CR	48.5	51.5	51.5	52.9
AUC	0.84	0.85	0.85	0.85
(np)	(122)	(89)	(72)	(73)

¹Concentration of Risk (CR) is the proportion of all observed post-hospital suicides occurring in the 12 months after hospital discharge (or less than 12 months if the Soldier terminated services prior to 12 months after hospital discharge) that occurred after the 5% of hospitalizations classified by the model as having highest risk of suicide. See the section on analysis methods for a discussion of elastic net models and mixing parameter penalties.

Caution is needed in interpreting predictors in the reduced logistic model, as the variable selection algorithm maximized overall prediction accuracy rather than individual coefficient accuracy. It is nonetheless noteworthy that the model included variables in all predictor classes: (Table 2) 3 socio-demographics (male, enlistment at age 27+, Armed Forces Qualification Test score above the 50^th percentile; OR=1.9–7.9), access to firearms (number of registered pistols; OR=1.3), crime perpetration (weapons possession, verbal assault; OR=2.2–5.6), prior suicidality (OR=1.6–2.9), prior psychiatric treatment (0.3–5.6), and characteristics of the focal hospitalization (OR=0.4–6.0). The two ORs less than 1.0 were: (i) being above the 50^th percentile on the ratio of number of psychiatric hospitalizations to time in service; and (ii) PTSD during current hospitalization.

Table 2.

Coefficients (odds-ratios) in the discrete-time (person month) logistic survival model using forward stepwise selection of predictors and a .05 level entry criterion (n=53,769)¹

	OR	(95% CI)	VIF2
I. Socio-demographics
Male (Yes/no)	7.9*	(1.9–32.6)	1.0
Age of Enlistment 27+ (Yes/no)	1.9*	(1.0–3.5)	1.0
AFQT score above 50th percentile (Yes/no)	3.3*	(1.7–10.0)	1.0
II. Access to firearms
Number of registered pistols	1.3*	(1.0–1.6)	1.0
III. Crime perpetration
Number of verbal assault offenses in past 12 months	2.2*	(1.2–4.0)	1.0
Any non-violent weapons offense in past 24 months (Yes/no)	5.6*	(1.7–18.3)	1.0
IV. Suicidal behavior
Any prior suicide attempt since enlistment (Yes/no)	2.9*	(1.7–4.9)	1.0
Number of outpatient visits with suicidal ideation in past 12 months	1.6*	(1.1–2.5)	1.1
V. Other prior treatment
Six or more outpatient visits with a mental health specialty provider in past 12 months (Yes/no)	1.9*	(1.0–3.6)	1.4
Number of antidepressant prescriptions filled in past 12 months	1.3*	(1.1–1.7)	1.1
Number of psychiatric hospitalizations/time in service above the 50% percentile (Yes/No)	0.3*	(0.2–0.6)	1.2
Any prior inpatient psychiatric treatment in past 12 months (Yes/no)	1.8	(0.8–3.7)	1.8
Number of inpatient days in past 12 months with a diagnoses of …
Major depression	2.2*	(1.1–4.4)	1.4
Somatoform/dissociative disorder	5.6*	(1.8–17.7)	1.0
VI. Characteristics of focal hospitalization
Hospitalized in a civilian psychiatric hospital or civilian facility with a psychiatric unit (Yes/No)	1.6*	(1.0–2.7)	1.0
Disorders diagnosed during current hospitalization (Yes/no)
PTSD	0.4*	(0.2–0.7)	1.1
Suicidal ideation	2.4*	(1.3–4.7)	1.0
Non-affective psychosis	2.9*	(1.2–7.0)	1.0
Somatoform/dissociative disorder	3.6*	(1.2–10.8)	1.0
Hearing loss	6.0*	(2.1–17.4)	1.0

^*Significant at the .05 level, two- sided test. But note that the predictors were selected using stepwise analysis and the current p values are consequently inexact

¹The best penalized survival model was a lasso model with 73 predictors from the total of 421 predictors considered. A conventional discrete-time survival model using containing those same 73 predictors was unstable (VIF > 5.0 for 6 predictors). As a result, we used forward stepwise analysis with a .05 level entry criterion to select a more stable subset of the 73 predictors. The coefficients for the 20 predictors that entered are presented here.

²VIF=Variance Inflation Factor. VIF for the coefficient associated with predictor X_i in the above equation equals 1/(1-R²_i), where R²_i is the coefficient of determination of a regression equation in which X_i is the dependent variable and all the other 19 predictors of suicide are included as predictors of X_i. A VIF GT 5.0 is typically considered an indicator of high multicollinearity.⁴⁸

Concentration of risk and conditional risk distributions

Inspection of CR across predicted risk ventiles led to creation of four risk strata. The majority of suicides occurred in the highest-risk stratum (which was made up of the 5% of hospitalizations in the highest-risk ventile; CR=52.9%). (Figure 2) CR was dramatically lower (8.8%) in the second stratum (made up of the 5% of hospitalizations in the second-highest ventile), lower still (4.2%) in a third stratum (made up of the 35% of hospitalizations in the next 7 ventiles), and lowest (0.8%) in the fourth stratum (made up of the 55% of suicides in the lowest 11 ventiles).

Figure 2. Concentration of risk of post-hospital suicides by ventile of predicted risk based on the discrete-time MPP=1.0 penalized survival model1.

¹Ventiles are 20 groups of hospitalizations of equal frequency (2688 or 2689) dividing the total sample of 53,769 hospitalizations into groups defined by level of predicted suicide risk.

Suicide risk ranged between 1338.8/100,000 hospitalizations in the highest- and 20.3/100,000 in the lowest-risk strata. (Table 3) However, as average time-in-service after hospital discharge was considerably less than 12 months, suicide risk per 100,000 person-years was considerably higher than per 100,000 hospitalizations: between 3,824.1/100,000 person-years in the highest- and 40.9/100,000 in the lowest-risk strata.

Table 3.

Concentration of risk (CR) and conditional risk of post-hospital suicides by risk strata across all hospitalizations

	Strata of predicted suicide risk based on the lasso model2
	Highest risk stratum (1st ventile)	2nd ventile	3rd–9th ventiles	Lowest risk stratum (10th–20th ventiles)	Total
Observed number of suicides	36	6	20	6	68
CR1	52.9%	8.8%	4.2%	0.9%	--
n/100,000 hospitalizations	1338.8	223.3	106.3	20.3	126.5
n/100,000 person-years	3824.1	538.7	221.1	40.9	263.9
(n)	(2,689)	(2,687)	(18,820)	(29,573)	(53,769)

¹Concentration of Risk (CR), which is defined as the proportion of all the observed outcomes of the type occurring in the 12 months after hospital discharge (or less than 12 months if the Soldier terminated services prior to 12 months after hospital discharge) that occurred in the risk ventile represented by the column heading. CR is defined separately for each of the two highest risk ventiles and then as a per-ventile average for the next seven ventiles treated as a single risk stratum and then final 11 ventiles treated as a separate risk stratum.

²Ventiles of suicide risk = 20 groups of hospitalizations of equal frequency (n=2,688–2,689) dividing the total of 53,769 hospitalizations into groups defined by level of predicted suicide risk. The 3^rd–9^th ventiles were collapsed into a single risk stratum based on the fact that observed suicide risk was comparable in these seven ventiles. The 10^th–20^th ventiles were collapsed into a final risk stratum based on similar evidence.

Stability of estimates

CR in the highest-risk stratum did not differ significantly depending on whether: (i) hospitalization was in a facility with a mental health inpatient unit versus a general medical facility without such a unit (48.2% vs. 66.7; λ²₁=1.7, p=.19); (ii) the suicide occurred before versus after September 1, 2008 (median date of suicides during the study period) (38.7% vs. 70.3%; λ²₁=2.4, p=.12); or (iii) the suicide did versus did not occur within three months of hospital discharge (median time to post-discharge suicide) (52.6% vs. 56.7%; λ²₁=0.0, p=.99).

Associations of suicide risk with other adverse outcomes

Soldiers in the highest-risk stratum also had elevated risks of other adverse outcomes in the year after hospital discharge, including unintentional injury deaths (CR=10.1%; λ²₁=7.1, p=.008), suicide attempts (9.1%; λ²₁=332.7, p<.001), and re-hospitalizations (7.5%; λ²₁=893.4, p<.001). Soldiers in the highest predicted suicide risk stratum had 7 unintentional injury deaths, 830 suicide attempts, and 3,765 re-hospitalizations within 12 months of hospital discharge (492,666.2/100,000 person-years). At least one of these outcomes occurred after 46.3% of the highest-risk hospitalizations.

DISCUSSION

Although risk factors for suicide are widely-known, synthesizing this information to optimize suicide prediction has been an elusive goal up to now. This study addressed this problem by using machine learning to generate an actuarial suicide risk algorithm from Army/DoD administrative data, finding that 52.9% of suicides occurred after the 5% of hospitalizations with highest predicted risk. While interventions in this high-risk stratum would not solve the entire Army suicide problem given that post-hospital suicides account for only 12% of all Army suicides, the algorithm would presumably help target preventive interventions. Prior to clinical implementation, though, several key issues must be addressed.

The first question is whether the risk algorithm is sufficiently stable to predict future suicides given that it is based on only 68 prior suicides. It is noteworthy that the machine learning methods used to create the algorithm were designed explicitly to maximize stability of predictions/ Within-sample stability analyses found that CR did not vary significantly by type of inpatient facility, year of hospitalization, or number of months since hospital discharge. This does not guarantee future stability, though. algorithm stability will consequently be tested again in the 2010–2013 Army suicide data in a future study to address this question.

A second question is whether the risk algorithm improves on clinical judgment. The study was unable to examine this issue empirically because the Army electronic medical record does not include a structured field where clinicians must record suicide risk assessments. Nor was documentation of suicide risk assessment in clinical notes consistent during the study period. However, with improved documentation following the VA/DoD Clinical Practice Guideline, comparison of actuarial to clinical prediction may be possible in the future. As noted in the introduction, though, previous research has shown that actuarial suicide prediction is much more accurate than prediction based on clinical judgment.^11–14 This evidence is consistent with a large literature showing that actuarial methods are superior to expert judgments in many areas of prediction.^44,45 At the same time, the comprehensive suicide risk assessments required by the new VA/DoD CPG¹⁰ will generate information not included in administrative records. As a result, our algorithm should be seen as a component of this comprehensive clinical assessment rather than a substitute for this assessment.

A third question is whether suicide is sufficiently common in the highest-risk stratum and available interventions sufficiently powerful to make targeted post-hospital interventions efficient compared to alternative ways of deploying the same clinical resources. Our results shed no light on this question. The potential for harm also has to be taken into consideration, as intensive post-hospital interventions might lead to undue scrutiny by non-medical leaders that adversely affect soldier careers. This concern is all the more important given that the vast majority of soldiers identified as being high-risk do not commit suicide. While a formal analysis of comparative risks and benefits is beyond the scope of this report, it is noteworthy that the highest-risk stratum had significantly elevated risks of other adverse outcomes and that prevalence of at least one such outcome was present after 46.3% of highest-risk hospitalizations. Ameliorative effects of expanded high-risk interventions on these outcomes (i.e., unintentional injury deaths, suicide attempts, re-hospitalizations) are plausible, as numerous risk factors for suicide (e.g., depression, substance abuse) are also risk factors for these other outcomes^2,46,47 and most suicide prevention interventions recommended for high-risk patients are likely to affect these outcomes as well.^7,10 These presumed benefits would have to be considered in a broad361 based evaluation of risks and benefits of any future targeted high-risk post-hospital preventive interventions.

The major limitations of our analysis involve errors in the administrative data used as predictors (missing and inconsistent values, errors in ICD-9-CM diagnoses). In addition, the algorithm could almost certainly be improved if more nuanced risk factor data were available. As the new VA/DoD CPG contains a checklist of risk factors clinicians are urged to assess in evaluating suicide risk, creation of a system to record these assessments in the EMR along with the clinician’s clinical global impression of patient suicide risk might both increase the completeness of these assessments and provide a rich source of information for future risk algorithm refinement.