Predicting suicides after outpatient mental health visits in the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS)

Abstract

The 2013 U.S. Veterans Administration/Department of Defense Clinical Practice Guidelines (VA/DoD CPG) require comprehensive suicide risk assessments for VA/DoD patients with mental disorders but provide minimal guidance on how to carry out these assessments. Given that clinician-based assessments are known not to be strong predictors of suicide, we investigated whether a precision medicine model using administrative data after outpatient mental health specialty visits could be developed to predict suicides among outpatients. We focused on male non-deployed Regular U.S. Army soldiers because they account for the vast majority of such suicides. Four machine learning classifiers (naïve Bayes, random forests, support vector regression, elastic net penalized regression) were explored. 41.5% of Army suicides in 2004-2009 occurred among the 12.0% of soldiers seen as outpatient by mental health specialists, with risk especially high within 26 weeks of visits. An elastic net classifier with 10-14 predictors optimized sensitivity (45.6% of suicide deaths occurring after the 15% of visits with highest predicted risk). Good model stability was found for a model using 2004-2007 data to predict 2008-2009 suicides, although stability decreased in a model using 2008-2009 data to predict 2010-2012 suicides. The 5% of visits with highest risk included only 0.1% of soldiers (1047.1 suicides/100,000 person-years in the 5 weeks after the visit). This is a high enough concentration of risk to have implications for targeting preventive interventions. An even better model might be developed in the future by including the enriched information on clinician-evaluated suicide risk mandated by the VA/DoD CPG to be recorded.

Introduction

The historically low U.S. Army suicide rate climbed beginning in 2004¹ to exceed the civilian rate since 2009.^2-3 Preventive interventions exist to reduce Army suicides,⁴ including a protocol for outpatients treated by mental health specialists based on the 2013 VA/DoD Clinical Practice Guidelines (CPG) on Assessment and Management of Patients at Risk for Suicide for comprehensive suicide risk assessments of all patients in treatment for mental disorders followed by interventions for high-risk patients.⁵ Although the CPG includes recommendations for risk assessment and stratification, no precision medicine prediction scheme was provided. This is an important gap, as previous research shows clinicians are not good at predicting suicide and that statistical risk models produce better predictions.^6,7 The Army maintains electronic administrative systems that might be used to develop a risk model of this sort for soldier suicides. Two recent epidemiological studies demonstrated that such models can be developed. ^8-9 The current report presents a similar precision medicine model to predict suicides among soldiers in outpatient treatment with mental health specialists.

Materials and Methods

Sample

Analysis was based on the Historical Administrative Data System (HADS) of the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS),¹⁰ an integrated de-identified dataset of Army/Department of Defense administrative data systems (Appendix Table 1) for each month in service during the years 2004-2009 of all 975,057 Regular U.S. Army soldiers serving at any time during that time-period (32 million person-months), 569 of whom died by suicide. HADS construction and composition are discussed elsewhere.¹¹ We focused initially on soldiers with any outpatient visit having a diagnosis of a mental disorders (ICD-9-CM Codes 290.0-319) or V code indicative of life difficulties often associated with mental disorders (V15.81; V61-62.9; V71.01-71.09), as risk of suicide death was substantially elevated in this segment of the force. Models were built to predict suicide deaths subsequent to these visits using a wide range of HADS predictors. Over 8,000 such visits occurred for each suicide death. As it would have been computationally intensive to include all these control visits in the analysis, we selected a probability sample of control visits equal to roughly 100 times the number of suicide deaths and compared values on predictors available at the times of those visits to the values of the same predictors available at the times of visits that occurred before suicide deaths. Control visits were weighted to adjust for their under-sampling so that the weighted sum of control visits equaled the population distribution (i.e., somewhat more than 8,000 times the number of visits followed by suicide deaths). This kind of subsampling and weighing of controls improves the efficiency of estimation without introducing bias into estimates compared to an analysis that included all control visits.¹²

Predictors

Numerous epidemiological studies have examined predictors of suicide among outpatients^13-18 and military personnel.^1,19-26 HADS variables operationalized as many of these predictors as possible organized into six broad categories: socio-demographics, Army career (e.g., age-at-enlistment, occupation, deployment history), characteristics of the index visit, prior clinical factors (e.g., inpatient and outpatient mental and physical disorders, prescriptions, suicide attempts), crime codes (victimization and perpetration), and contextual factors (e.g., unit-level characteristics, registered weapons). We controlled year, season, and time until next visit to adjust for secular trends in the Army suicide rate and time-at-risk.

Given that the administrative data were collected for other purposes, we cast a wide net in extracting indicators of target constructs. For example, we examined 23 different categories of psychiatric diagnoses and 15 categories of NDC psychotropic medication codes based on the First Databank (FDB) Enhanced Therapeutic Classification System™^,27 (Appendix Tables 2-3). Nearly 1,000 variables were constructed (Appendix Tables 4). Missing socio-demographic and Army career data were corrected when possible with nearest neighbor temporal imputations. Remaining missing values and inconsistencies were resolved using rational imputation. (e.g., a soldier classified female one month but male all other months was recoded male). Details about missing data patterns are available in Appendix Table 5.

Analysis methods

De-identified HADS analysis was approved by the Human Subjects Committees of the Uniformed Services University of the Health Sciences for the Henry M. Jackson Foundation (the primary grantee), the University of Michigan, and Harvard Medical School. Analysis began with cross-tabulations examining suicide risk in the 12 months after each outpatient visit, distinguishing visits in the general medical and mental health specialty sectors by prior psychiatric hospitalization, gender and deployment status. Model-building began by estimating univariate associations of predictor with suicide using discrete-time survival analysis of suicide death (coded 1) compared to all other outcomes (i.e., some other death, a subsequent mental health specialty visit, separation from service, end of the follow-up period, all coded 0). A logistic link function was used to estimate coefficients with proc logistic in SAS 9.3.²⁸ Functional forms of significant non-dichotomous predictors were transformed to capture interpretable nonlinearities.

As multivariable associations were unstable, machine learning methods were used to generate stable estimates comparing four different classifiers: naïve Bayes²⁹ using the R-package e1071 naiveBayes;³⁰ random forest³¹ using the R-package RandomForest (RF);³² support vector regression³³ using the R-package e1071 svm; and elastic net penalized regression³⁴ using the R-package glmnet.³⁵ Hyperparameters were selected to maximize cross-validated sensitivity (i.e., the proportion of observed suicide deaths among predicted positives) in the 5% of visits with highest predicted suicide risk. Selection of the optimal classifier was based on the same criterion.

Once the best classifier was selected, operating characteristics were examined by comparing predicted probability of suicide death for each sampled person-visit to observed suicide death in the entire sample by calculating area under the receiver operating characteristic curve (AUC) and graphing proportional suicide deaths after visits in each ventile (i.e., 5%) of visits grouped from highest to lowest predicted probabilities. We then calculated sensitivity (, as noted above, the proportion of observed suicides after visits predicted to have high suicide risk) and positive predictive value (suicide rate after visits predicted to have high risk expressed as number of suicides /100,000 person-years) in high-risk ventiles along with specificity (the proportion of visits not followed by observed suicides after visits predicted not to have high suicide risk), negative predictive value (the non-suicide rate/100,000 person-years after such visits), and AUC (which, in the case of dichotomous predictors, is the mean of sensitivity and specificity). Given the rarity of suicide deaths, we report 1-negative predictive value (i.e., suicides/100,000 person-years) rather than negative predictive value. Visit-level estimates were then projected to the person-level by aggregating results for selected contiguous 12-month time periods for a probability sample of 100,000 soldiers. Model predictive validity was evaluated by using coefficients estimated in earlier years to predict suicides in later years.

Results

Outpatient visits and suicide by treatment sector, gender, deployment status and time

Sixty-eight (12%) of the 569 suicide deaths of Regular Army soldiers during 2004-2009 occurred among the 0.9% of soldiers with psychiatric hospitalizations in the prior 12 months (252.3/100,000 person-years). (Table 1) Another 240 (42.2%) suicides occurred among the 24.5% of soldiers without 12-month psychiatric hospitalization who were outpatients with target diagnoses or V codes (31.7/100,000 person-years). The remaining 261 (45.9%) suicides occurred among the other 74.6% of soldiers (11.3/100,000 person-years). Among the 0.9% with hospitalization, the suicide rate was highest among those seen outpatient after hospital discharge by both mental health and general medical treatment providers (0.65% of all soldiers; 312.2/100,000 person years), lowest among those seen only by mental health providers (0.11%; 85.7/100,000 person-years), and intermediate among those seen only by general medical providers (0.06%; 107.0/100,000 person-years). Among the 24.5% having outpatient visits without hospitalizations, the suicide rate was highest among those seen both by mental health and general medical providers (5.1% of all soldiers; 63.9/100,000 person years), intermediate among those seen only by mental health providers (6.0%; 36.1/100,000 person-years), and lowest among those seen only by general medical providers (13.4%; 17.4/100,000 person-years).

Table 1. The proportions of all soldiers and soldiers who died by suicide who had psychiatric hospitalization and outpatient treatment for mental disorders in the prior 12 months among Regular Army soldiers over the years 2004-2009 (n = 975,057).

	% of all soldiers	(se)	% of all suicides	Count of suicides	Person-months1	Suicides/ 100,000 PY	(se)
I. Psychiatric hospitalization
With subsequent outpatient visits to …
Mental health specialist (MHS) only	0.11	(0.01)	0.5	3	42,010	85.7	(50.4)
General medical (GM) only	0.06	(0.01)	0.4	2	22,423	107.0	(77.5)
Both MHS and GM	0.65	(0.03)	11.1	63	242,178	312.2	(41.3)
Without subsequent outpatient visits	0.04	(0.01)	0.0	0	16,416	0.0	(0.0)
Total	0.9	(0.03)	12.0	68	323,427	252.3	(31.8)
II. Outpatient visits with mental disorder diagnoses without hospitalization
Mental health specialist (MHS) only	6.0	(0.1)	11.8	67	2,227,928	36.1	(4.4)
General medical (GM) only	13.4	(0.1)	12.7	72	4,952,222	17.4	(2.1)
Both MHS and GM	5.1	(0.1)	17.8	101	1,897,210	63.9	(6.4)
Total	24.5	(0.1)	42.2	240	9,077,360	31.7	(2.1)
III. All other soldiers
Total	74.6	(0.1)	45.9	261	27,606,798	11.3	(0.7)
IV. All Regular Army soldiers
Total	100.0	--	100.0	569	37,007,585	18.5	(0.8)

Abbreviations: SE, standard error; PY, person-years.

¹We report number of person-months rather than numbers of soldiers because soldiers differ in how long they remain in service and analysis at the level of the person-month is more accurate at the level of the person.

Given the much higher suicide rate among outpatients seen by mental health providers than exclusively by general medical providers, we focused analysis on the former and distinguished between the 66 suicides with prior 12-month psychiatric hospitalization and the 168 suicides without such hospitalization. The population-at-risk consisted of 316,686 Regular Army soldiers making 2,950,967 outpatient mental health specialist visits in 2004-2009. 95.8% of these visits were made when patients were not deployed (173 suicides; 65.6/100,000 person-years) and the suicide rate after these visits was substantially higher among men than women (75.3/ versus 19.6/100,000 person-years), with 94.8% (164 of 173) of suicide deaths after these visits occurring among men. Based on these patterns, we focused analysis on non-deployed men. The majority (61.6%; 101/164) of suicide deaths in this group occurred within 5 weeks of mental health specialist outpatient visits (145.2, 96.3, 123.6, 116.5, and 115.1 suicides/100,000 person years, respectively, in those weeks), with a 57.4/100,000 person-years rate during the remainder of the first 6 months (28.7% [47/164] of suicide deaths over the 12 months after the index visit) and 31.3/100,000 person-year over the subsequent 6 months. Based on these results, we limited model-building to the 26 weeks after the index visit (148 suicides).

Selecting the optimal classifier

Roughly one-third of HADS variables for prior clinical characteristics (244/782 among soldiers with and 178/536 among soldiers without psychiatric hospitalizations) were significant univariate predictors of subsequent suicide. Much smaller proportions of variables characterizing the index outpatient visit (2/46), involvement in crime (2/67), and contextual factors (0/39) were significant. The significant univariate predictors plus 20 socio-demographic and 27 Army career variables were included in multivariable model-building. Based on many predictors about psychiatric hospitalization being significant, all analyses were carried out separately among soldiers who had (50 suicides) versus had not (97 suicides) psychiatric hospitalizations in the prior 12 months. The elastic net classifier out-performed the others in terms of higher cross-validated sensitivity in the weighted 5% of observations with highest predicted risk among both soldiers with and without prior 12-month psychiatric hospitalizations. Subsequent phases of analysis consequently focused on the elastic net models. Fourteen predictors were included in this model for soldiers with and 10 for soldiers without prior psychiatric hospitalizations.

Operating characteristics of model-based predictions

The model AUCs for the continuous distributions of predicted probabilities over 26 weeks were .72 among soldiers with prior psychiatric hospitalizations, .61 among soldiers without prior hospitalizations, and .66 among both combined. When the same models were applied to suicide deaths in the 5 weeks after the index visits, AUCs increased to .75 (prior hospitalization), .65 (no prior hospitalization), and .69 (both combined). Sensitivity was more than twice the expected value of 5% after visits in the three highest risk ventiles for both 26 weeks and 5 weeks (Figure 1) and either below or only slightly above their expected values in the remaining 17 ventiles, leading us to evaluate operating characteristics of two dichotomous classifications: between the top 1 and other 19 risk ventiles; and between and top 3 and other 17 risk ventiles.

Figure 1.

Proportion of suicide deaths that occurred within 5 and 26 weeks of most recent specialty mental health outpatient visits within ventiles¹ of visits ranked by predicted suicide risk based on the optimal elastic net penalized logistic regression model, male non-deployed Regular U.S. Army soldiers 2004-2009.

¹The bars show the observed proportions of suicide deaths within 5 weeks of each ventile (5% grouping) of specialty outpatient visits ranked by predicted suicide risk based on the optimal prediction model out of the population of all such visits made by male non-deployed Regular U.S. Army soldiers in 2004-2009.

All calculations of operating characteristics combined soldiers with and without prior hospitalizations. (Table 2) Sensitivity in the top ventile was 22.4-24.0% (26-5 weeks after visits). Comparable sensitivities were 45.6-48.0% in the top 3 ventiles. Specificity was 94.9-94.9% in the lowest 19 ventiles and 84.0-84.0% in the 17 lowest ventiles. Positive predictive value was 1076.8-1047.6/100,000 person-years in the top ventile and 602.3-605.9/100,000 person-years in the top 3 ventiles compared to 52.9-71.5/100,000 person-years in the remaining 17 ventiles (i.e., 1–negative predictive value). AUC was .59-.66.

Table 2. Operating characteristics of dichotomous classifications distinguishing soldiers in the top 1 and 3 ventiles of predicted risk of suicide death after mental health specialty outpatient visits made by male non-deployed soldiers based on the optimal logistic models combining soldiers with and without prior 12-month psychiatric hospitalizations¹.

	SENS2	(se)	SPEC2	(se)	PPV2	1–NPV2	AUC2
I. Suicides in the 26 weeks after the index outpatient visit
Highest 1 risk ventile	22.4	(3.1)	94.9	(0.1)	1076.8	71.6	.59
Highest 3 risk ventiles	45.6	(4.5)	85.0	(0.2)	602.3	52.9	.65
II. Suicides in the 5 weeks after the index outpatient visit
Highest 1 risk ventile	24.0	(4.5)	94.9	(0.1)	1047.6	96.9	.59
Highest 3 risk ventiles	48.0	(6.0)	85.0	(0.2)	605.9	71.5	.66

¹Predicted probabilities of suicide death in the 26 weeks and first 5 of those 26 weeks after outpatient specialty mental health visits in 2004-2009 based on separate models for soldiers with and without psychiatric hospitalizations in the prior 12-months were combined and the overall population of such visits was dichotomized either to distinguish the 5% (Highest 1 risk ventile) or 15% (Highest 3 risk ventiles) of visits with highest predicted risk from other visits .SENS and SPEC were estimated based on visit-level cross-tabulations with subsequent suicide deaths, while PPV and 1-NPV were based on calculations that took into consideration number of days at risk between the day of each visit and the end of the follow-up period (defined as either the occurrence of a subsequent visit, suicide death, some other termination of service, on end of the follow-up period, whichever came first).

²SENS = Sensitivity; SPEC = Specificity; PPV = Positive Predictive Value expressed as number of suicide deaths per 100,000 person-years; 1-NPV -= 1-Negative Predictive Value expressed as number of suicide deaths per 100,000 person-years; AUC = Area under the receiver operating characteristic curve for the dichotomous classification. Note that the AUC values reported here are lower than those reported in the text for the continuous prediction scores, as those reported here evaluate the accuracy of dichotomous classifications at particular points on the continuous scores rather than accuracy across the full ranges of those scores.

Person-level projections of visit-level results

As person-level inferences cannot be drawn from visit-level results, we drew a representative sample of 100,000 soldiers in service over the study period who did not die by suicide, combined them with all soldiers who died by suicide, and generated predicted suicide risk scores based on the coefficients in our best model for each mental health specialty outpatient visit of each soldier in this dataset. These visit-level scores were then aggregated to the person-level. The non-deployed men with 12-month mental health specialty outpatient visits had an average of 6.1 such visits. Extrapolating to an Army of 500,574 (the average number of non-deployed male soldiers on active duty in the Army over the study period), this would be 60,654 non-deployed men making 368,233 mental health specialty outpatient visits over a typical 12 months (17,629-55,286 visits in the 1-3 highest-risk ventiles). 4.2% of soldiers who made 12-month mental health specialty outpatient visits had visits in the top risk ventile, with a mean of 7.0 such visits and a mean of 10.3 weeks in the highest-risk time interval after such visits. This means that only 573 (i.e., .042 × .121 × 500,574 × 10.3/52) male non-deployed soldiers in an Army of 500,574 non-deployed men would be in the highest-risk group in a typical week. This number increased to 1,103 for patients in the highest-risk ventile over 26 weeks and to 3,657 for patients in the 3 highest-risk ventiles over 26 weeks.

Validation

Models were re-estimated in the 2004-2007 HADS data using the same predictors but allowing the coefficients to differ from the 2004-2009 model. Results were used to predict 2008-2009 suicides. AUC combining soldiers with and without prior psychiatric hospitalizations was .67-.72 predicting suicides within 26-5 weeks of most recent visit. Twenty-six week sensitivity was 26.7-41.3% for visits in the highest-1-3 risk ventiles. Five-week sensitivity was 29.8-47.4% for visits in the highest-1-3 risk ventiles. Replication of this validation exercise using coefficients estimated in 2008-2009 to predict suicides in 2010-2012 yielded much weaker results: sensitivities of 13.3-18.1% for 26-5 weeks in the 1 highest ventile and 36.1-27.4% in the 3 highest ventiles.

Model coefficients

The 14 predictors in the model for patients with prior hospitalization included 6 indicators of prior suicidality, 6 of prior inpatient-outpatient depression treatment, and 2 of non-affective psychosis and bipolar disorder treatment, all associated with elevated suicide risk. (Table 3, Model 1) Odds-ratios were all relatively modest (OR=1.01-1.32) due to elastic net penalties. Extreme coefficient instability (indicated by high variance inflation factors) occurred, in comparison, when a logistic regression model (Model 2) was estimated with the same predictors, although re-specification allowed this problem to be addressed in a less complex logistic model (Model 3) that retained essentially the same level of overall prediction accuracy (AUC=.72).

Table 3. Coefficients in the optimal elastic net and comparable conventional logistic regression models predicting suicide deaths within 26 weeks of mental health specialty outpatient visits made by non-deployed male soldiers with prior 12-month psychiatric hospitalizations.

	Elastic net1	Logistic 12			Logistic 23
	OR	OR	(95% CI)	VIF4	OR	(95% CI)	VIF4
Suicidality
Any suicide attempt since enlistment	1.0	>0.001	--	16.2	2.3*	(1.3-4.4)	1.1
Any suicide attempt past 12 months	1.1	<999.9	--	16.2
Suicidality, frequency outpatient, past 12 months	1.0	>0.001	--	82.4
Suicide ideation, frequency outpatient, past 12 months	1.1	<999.9	--	80.8
Suicidality, any outpatient, past 3 months	1.0	<999.9	--	48.4	2.6*	(1.5-4.5)	1.0
Suicide ideation, frequency outpatient, 3 months	1.0	>0.001	--	47.0
Depression
Any outpatient, past 12 months	1.0	1.9	(0.8-4.6)	2.2
Any inpatient, past 12 months	1.0	1.9	(0.5-7.0)	6.5	3.0*	(1.5-5.6)	1.0
Frequency inpatient, past 12 months	1.1	1.2	(0.8-1.7)	6.5
Any outpatient, past 3 months	1.0	1.2	(0.4-3.0)	2.6
Any outpatient, past 2-3 months	--				1.9	(1.0-4.0)	1.0
Any outpatient, past 1 month	1.3	1.2	(0.3-4.6)	5.3	3.8*	(1.9-7.4)	1.1
Frequency outpatient, past 1 month	1.1	1.3	(0.6-2.5)	4.9
Non-affective psychosis, any outpatient, past 3 months	1.0	2.5	(1.2-5.5)	1.1	2.7*	(1.3-5.8)	1.0
Bipolar, any outpatient, past 12 months	1.0	2.8	(1.4-5.6)	1.1	2.9*	(1.5-5.6)	1.0
AUC	0.72		0.73			0.72

Abbreviations: OR, odds ratio, CI, confidence interval; VIF, variance inflation factor; MDD, major depressive disorder; AUC, area under the receiver operating characteristic curve.

^*Significant at the .05 level, two-sided test.

¹The optimal elastic net model had a mixing parameter of α=0.3. This is weighted more in the direction of the ridge penalty (α=0.0), which favors the inclusion of all predictors even when they are highly correlated and adjustment for multicollinearity by coefficient shrinkage, than the lasso penalty (α=1.0), which favors deletion of all but one predictor in each highly correlated predictor set, suggesting that correlations among predictors are substantial. Inspection of the correlation matrix among predictors supports this intuition, as the condition number (the square root of the ratio of the eigenvalue of the first and last principal components of the correlation matrix among predictors) of 37.7 is above the value of 10.0 typically considered to indicate severe multicollinearity.⁴⁸ Pearson correlations among predictors have a range of -.04--.97, an inter-quartile range of .03-.14, and a median of .07. See also fn 4 on VIF.

²Logistic model 1 was a conventional logistic regression model with the same predictors as the optimal elastic net model. The effects of high correlations among predictors can be seen in the extreme instability of the coefficients as well as in high VIFs (See fn 4).

³Logistic model 2 was a revised logistic regression model using a reduced and transformed set of the original predictors designed to correct the problem of instability by removing redundant predictors. The success of this effort can be seen more in the lower values of VIF (See fn 4) than in Logistic model 1 and the comparable AUC despite the smaller number of predictors.

⁴VIF=variance inflation factor, the inverse of 1-R²_x, where R²_x is the coefficient of multiple determination of predictor x regressed on all other predictors in the model. A VIF value above 5/10 is typically considered to indicate a multicollinearity problem with predictor x. ⁴⁸

The 10 predictors in the model for patients with no prior hospitalization included one feature of the index visit -- whether with a psychiatrist (associated with elevated suicide risk) – along with 3 measures of treatment in the past month (frequency of visits for depression and ill-defined conditions; any inpatient treatment for a physical disorder), 3 measures of treatment in the past 3 months (any for either non-affective psychosis or personality disorder; number of anticonvulsant prescriptions), 2 measures of treatment in the past 12 months (frequency of outpatient visits for anxiety disorders; any prescription of an alcohol-narcotic abuse treatment agent), and a final measure for whether the soldier was an alleged perpetrator of multiple crimes in the 3 months before the index visit. (Table 4) The odds-ratios of these predictors were much more diverse than in the model for soldiers with prior hospitalizations (OR=1.2-8.8), reflecting the weaker associations among predictors (as indicated by the low variance inflation factors in the parallel logistic model).

Table 4. Coefficients in the optimal elastic net and conventional logistic regression models predicting suicide deaths within 26 weeks of mental health specialty outpatient visits made by non-deployed male soldiers without prior 12-month psychiatric hospitalizations.

	Elastic net1	Logistic
	OR	OR	(95% CI)	VIF2
Focal visit is with a psychiatrist	1.0	1.6*	(1.1-2.5)	1.0
Depression, frequency outpatient, past 1 month	1.1	1.5	(1.0-2.4)	1.0
Ill-defined conditions, frequency outpatient, past 1 month	1.6	1.9*	(1.1-3.2)	1.0
Any physical disorder, any inpatient, past 1 month	2.8	8.8*	(2.0-38.2)	1.0
Non-affective psychosis, any outpatient, past 3 months	1.2	3.5	(0.8-15.0)	1.0
Personality disorder, any outpatient, past 3 months	1.4	2.6*	(1.1-6.6)	1.0
Anticonvulsant prescription, frequency, past 3 months	1.3	1.8*	(1.1-2.9)	1.0
Anxiety, frequency outpatient, past 12 months	1.1	1.2*	(1.0-1.4)	1.1
Alcohol or narcotic treatment agent, any, past 12 months	1.6	4.4*	(1.6-12.5)	1.0
Multiple crime perpetrations in past 1 month	1.4	1.9*	(1.5-2.5)	1.0
AUC	0.61		0.62

Abbreviations: OR, odds ratio, CI, confidence interval; VIF, variance inflation factor; AUC, area under the receiver operating characteristic curve.

^*Significant at the .05 level, two-sided test.

¹The optimal elastic net model had a mixing parameter of α =0.5, which is an equal weighting between the ridge penalty (α =0.0) and the lasso penalty (α =1.0). (See Table 2, fn 1) This suggests that correlations among predictors are less substantial than in the model for visits made by soldiers with prior hospitalizations, where the optimal mixing parameter was α =0.3. Inspection of the correlation matrix among predictors supports this intuition, as the condition number of 3.1 was lower than in Table 2 and the Pearson correlations among predictors (range of -.10--.15, inter-quartile range .00-.05, median.02) were lower than in Table 2. See Table 2, fn 1.

²VIF=variance inflation factor, the inverse of 1-R²_x, where R²_x is the coefficient of multiple determination of predictor x regressed on all other predictors in the model. A VIF value above 5/10 is typically considered to indicate a multicollinearity problem with predictor x.⁴⁸

Discussion

Despite the elevated suicide risk of soldiers with mental health specialty outpatient visits, which is consistent with civilian research,³⁶ and the strong performance of our models, suicide was a rare outcome even among high-risk soldiers. This raises the question whether existing interventions are sufficiently powerful to make targeted preventive interventions cost-effective. There is controversy about this question.^37-39 Empirical adjudication would require analyses beyond the scope of this report on competing needs, costs, and cost-effectiveness of intervention options.^{40, 41} Our aim was to address a prior question : whether a useful precision medicine model can be developed. We showed that it can. The 5% of visits with highest predicted risk include only 0.1% of soldiers with very high suicide risk (1047.1/100,000 person-years in the 5 weeks after the visit). This is a small enough proportion of individuals accounting for a large enough proportion of suicides to have intervention implications.

Interpretation of model predictors should only be undertaken with caution because machine learning methods maximize model performance at the expense of individual coefficient accuracy. Nonetheless, four observations are noteworthy. First, the vast majority of predictors measured mental disorders found to be important in prior studies of soldier suicides.^19,21,23,42 The crime perpetration variable in the model for soldiers without prior hospitalization is consistent with evidence that a high proportion of soldiers who die by suicide had legal problems at the time of death.²¹

Second, we found that hospitalization for any physical health problem was an important predictor of soldier suicide. Although traumatic brain injury (TBI), a widely recognized suicide risk factor,^43-44 was included as a potential predictor, the fact that this composite variable was selected over TBI in the predictor set underscores the need for future investigation to focus clinical attention on broader hospitalized physical conditions linked to suicide.

Third, despite previous research consistently finding suicide predicted by socio-demographic characteristics indicating disadvantaged social status (e.g., young age, non-married status) and Army career characteristics indicating low status (e.g., low rank, demotion) predicting soldier suicide,^{19,20,22,25,42,45} no such predictors emerged in our optimal models. No attempt was made to determine whether this was because the clinical variables in our models mediated the effects of socio-demographic and Army career variables, but future investigation of this possibility might provide insights into modifiable targets of preventive interventions.

Fourth, important differences were found between patients with versus without prior psychiatric hospitalization. AUC and concentration of risk were higher in models for those with (AUC=.72-.75 for 26-5 weeks, with 28-36% of suicides occurring among the 5% of patients with highest predicted risk) than without (AUC=.61-.65, with 22-24% of suicides occurring among the 5% of patients with highest predicted risk) prior hospitalizations. All but one predictor in the model for patients with hospitalization involved characteristics of outpatient visits prior to the hospitalization rather than of the hospitalization, with a focus on suicidality, depression, bipolar disorder, and non-affective psychosis. The model for patients without hospitalization, in comparison, included a much wider array of diagnoses, the one with the highest odds-ratio being alcohol/drug treatment. Recent inpatient treatment for a physical disorder was also a very powerful predictor in that model. These differences suggest that the causal processes underlying suicide are different for patients with and without psychiatric hospitalization. Further investigation of these differences might provide insights to help customize preventive interventions.

Our analysis was limited by considering a large number of predictors of a small number of suicides, introducing risk of over-fitting. We addressed this problem by using cross-validation to select the number of predictors in final models and using penalized regression to select predictors, but residual over-fitting might have occurred. We evaluated this by predicting 2008-2009 suicides based on 2004-2007 models and 2010-2012 suicides based on 2008-2009 models. Model stability was very good between 2004-2007 and 2008-2009 but much lower between 2008-2009 and 2010-2012, possibly reflecting changes in Army policies-practices for managing suicide risk as awareness of the rising Army suicide rate increased. The only way to guard against such a possibility going forward would be to update prediction models regularly (e.g., annually) and carry out sensitivity analyses of the extent to which predictors change depending on the number of years of prior data used in developing the models.

Another set of limitations involves the administrative data used in our models, which had more missing, inconsistent, and possibly erroneous values than in data collected for research purposes and lacked indicators of some suicide risk factors documented in the literature. These limitations presumably resulted to reduced model performance. Yet the models nonetheless had good prediction accuracy that would presumably be improved by increasing data quality (e.g., adding predictors based on the checklist the VA/DoD CPG now urges clinicians to use to evaluate suicide risk). A final noteworthy limitation is that we were unable to follow soldiers out of service to predict suicides that occurred after separation. This right censoring is an important limitation for long-term prediction given that soldiers with mental disorders are more likely than others to terminate service.⁴⁵

It is unclear from the results reported here how much clinical judgment could be enhanced by having access to results of our models, as clinical assessments of suicide risk were not systematically recorded in Army medical records over the years we studied. As noted in the introduction, though, previous studies find that statistical models are much more accurate than clinical judgment of suicide risk,^2,46-48 consistent with a larger literature showing statistical methods outperform expert judgment in many areas of prediction,^6,7 suggesting that access to predictions based on our models could be of value to clinicians as one element in their evaluation of patient suicide risk.