Abstract
Objectives. The Veterans Health Administration (VHA) evaluated the use of predictive modeling to identify patients at risk for suicide and to supplement ongoing care with risk-stratified interventions.
Methods. Suicide data came from the National Death Index. Predictors were measures from VHA clinical records incorporating patient-months from October 1, 2008, to September 30, 2011, for all suicide decedents and 1% of living patients, divided randomly into development and validation samples. We used data on all patients alive on September 30, 2010, to evaluate predictions of suicide risk over 1 year.
Results. Modeling demonstrated that suicide rates were 82 and 60 times greater than the rate in the overall sample in the highest 0.01% stratum for calculated risk for the development and validation samples, respectively; 39 and 30 times greater in the highest 0.10%; 14 and 12 times greater in the highest 1.00%; and 6.3 and 5.7 times greater in the highest 5.00%.
Conclusions. Predictive modeling can identify high-risk patients who were not identified on clinical grounds. VHA is developing modeling to enhance clinical care and to guide the delivery of preventive interventions.
Over the past 8 years, the Veterans Health Administration (VHA), the health system of the Department of Veterans Affairs, strengthened its mental health services and supplemented them with specific programs for suicide prevention.1,2 However, suicide rates in VHA have been stable, without decreases that can be attributed to these enhancements.3 The stable rates stand in contrast to increased rates in other US populations, especially middle-aged men,4,5 and in veterans who do not use VHA3,6; VHA programs may have mitigated population-wide increases. Nevertheless, the finding that suicide rates in VHA remain high represents a strong call for action.
Although epidemiological research has identified an array of risk factors for suicide, effect sizes are, in general, small to moderate.7,8 Despite considerable research on how risk factors combine or interact to affect risk, few reports offer information from multivariate models that clinicians could use in decision-making.9–11 Two recent reports demonstrated that predictive modeling that uses information from medical and administrative records can identify patients at risk for suicide,12,13 and predictive modeling may be more accurate than clinical evaluations.13
There is general agreement about domains that clinicians should consider in evaluating patients’ risk of suicide.14,15 However, obtaining the information needed requires high levels of clinical skill, including the ability to instill a sense of trust.16 Accordingly, additional training has been recommended to ensure that a broad range of clinicians can conduct accurate assessments,2 and research is needed to enhance the sensitivity of evaluations, improve clinical assessments, and develop psychological and biological markers.17–20 Improvements in assessments are necessary, for example, to enable accurate identification of patients at imminent risk in the emergency department. However, improvements are not necessary prerequisites for use of predictive modeling to target preventive interventions.
In general, discussions of prevention in the field of mental health,21 including the 2012 National Strategy for Suicide Prevention,2 consider 3 levels of intervention: indicated clinical services for those with symptoms or warning signs associated with high risk, selective clinical and community preventive services for groups of individuals at increased risk, and universal public health strategies directed toward entire populations. The Department of Veterans Affairs’ suicide prevention strategy has focused on indicated strategies, for example, facilitating access to mental health services and related services, such as pain management, and on providing resources specifically for suicide prevention, including a crisis line integrated with clinical services.
To extend its indicated strategies, the Department of Veterans Affairs is implementing a Clinical Practice Guideline for the Assessment and Management of Patients at Risk for Suicide.22 In addition, it is working to develop selective strategies. Consistent with recent calls for research to develop a taxonomy of high-risk subgroups,23 VHA’s initial approach used decision-tree analyses, considering categories derived from the electronic medical record for demographics, mental health and medical diagnoses, and service utilization. Although it was possible to identify classes of patients at specific levels of increased risk, these were distributed across many small and complex subgroups. Findings did not support use of decision-tree analyses to guide system-wide policies. Accordingly, the focus shifted to evaluating predictive modeling of clinical and administrative data from the electronic medical record for estimating levels of risk for individual patients. If this proved feasible, the next steps would be for the health care system to develop methods for informing providers about which of their patients are at high risk and for enhancing care.
We developed and validated an actuarial model of data from electronic medical records and its use for predicting risk of suicide over periods ranging from 1 month to 1 year. Because the goal of modeling was to provide information about the feasibility of selective, risk-stratified preventive interventions, we focused on the extent to which suicide risk was concentrated in specific strata of patients. To provide additional information on the potential benefits of these interventions, we included information about nonsuicide mortality, from external causes and overall mortality. To guide implementation of selective interventions, we included descriptive information about patients in 2 specific strata, the top 0.10% and 5.00% of calculated risk.
METHODS
Clinical data came from the VHA National Patient Care Database, an archival data set that contains comprehensive patient-by-patient, encounter-by-encounter clinical and administrative data derived from the Department of Veterans Affairs’ electronic medical records. Data regarding vital status and cause of death were from the National Death Index, a centralized database maintained by the National Center for Health Statistics of death record information on file in state vital statistics offices. We conducted searches with established procedures.24 We constructed 3 samples: (1) model development, (2) model validation, and (3) prediction. The development and validation samples contained data for patient-months from all patients who died from suicide from October 1, 2008, to September 30, 2011, and had received VHA services in the previous 2 years (case patients) and a random 1% of patients who survived the month and received VHA services in the previous 2 years (control patients). We randomly assigned half of the case patients and half of the control patients to the development sample and half to the validation sample. Each contained patient-months from 3180 case patients and 1 056 004 control patients. Individuals could be selected in multiple months; 83.5% were sampled only once; 13.2%, twice; and 1.5%, 3 times or more. The prediction sample comprised all 5 969 662 individuals who were alive as of September 30, 2010, and had had VHA inpatient or outpatient encounters in the previous 2 years.
Measures
We considered previous VHA analyses25–34 and findings from other populations7,8 in selecting measures, and indexed them according to month of selection. Demographic measures were age, gender, race/ethnicity, marital status, urban or rural residence, geographic region, and period of military service. Contextual factors were military service–connected disability, homelessness, reports of military sexual trauma, and previous self-directed violence. Mental health measures were receipt of any mental health or substance abuse diagnoses and specific diagnoses. Medical measures were 34 specific diagnoses, including common conditions and those for which the literature suggests associations with depression, as well as visual and hearing impairment. We used a definition for pain-related diagnoses that is described elsewhere.28
Measures of VHA service utilization were receipt of any inpatient or outpatient services, their intensity, and how recent they were, for both mental health services and overall care. Measures for medications were receipt of antipsychotics, antidepressants, mood stabilizers, sedatives–anxiolytics, analgesic pain medications, opioid pain medications, anticonvulsant pain medications, tricyclic antidepressant pain medications, and muscle relaxants as classes, and several specific agents. We assessed most variables for specific periods, typically the previous 12 or 24 months. We included indicators of VHA service use, inpatient psychiatric discharges, and emergency department visits for each of the previous 24 months or at months 1, 2, 3, 6, 12, 18, and 24 as lag variables; we used similar methods to consider the time since the onset of specific conditions. We also assessed interactions between specific mental health measures, between gender and marital status, and between marital status and psychiatric diagnoses. In total, our analyses had 381 measures, including 31 interaction terms.
Models
Model development used multivariable logistic regression with unweighted data in the development sample to calculate parameter estimates for each variable. Analyses with weights to represent the underlying population (2 for case patients, 200 for control patients) gave a lower c statistic, yet findings were similar. We applied parameter estimates from the unweighted analyses to calculate the predicted probability of suicide for each individual. We ranked the patients according to their probabilities and used the rankings to define risk strata. We retained variables in the model regardless of statistical significance, because the analytic focus was on overall prediction of risk rather than interpretation of particular parameter estimates.
For each stratum, we calculated the number of case patients, the annualized suicide rate, and risk concentration; we defined risk concentration as the ratio of observed (unadjusted) case patients (of death from suicide, from nonsuicide external-cause mortality, or nonsuicide all-cause mortality) to the case patients that would be expected if the distribution were uniform across strata. We repeated these procedures for the validation sample with parameter estimates derived from the development sample. For the prediction sample, we used parameters from the development sample to estimate suicide risk concentration, by strata of predicted risk, in the subsequent 1 to 12 months.
Other analyses estimated risk concentration and related variables for nonsuicide external-cause mortality and for all nonsuicide mortality. Also for the prediction sample, analyses evaluated associations between calculated risk and patients’ presence at any point from October 2009 onward on VHA’s listing of those flagged as being at high risk for suicide, primarily through clinical identification of suicide attempts or severe ideation leading to changes in treatment. Finally, we obtained descriptive statistics for the strata of patients at the top 0.10% and 5.00% of predicted risks. Statistical analyses used SAS version 9.3 (SAS Institute, Cary, NC).
RESULTS
Suicide decedents were more likely than nonsuicide patients to be young, male, and unmarried; to live in a rural area; to have a history of or be at risk for homelessness; to have no service-connected disabilities; to have diagnosed mental health conditions, pain, sleep disorders, and traumatic brain injury; to have used VHA mental health services; to have had psychiatric hospitalizations; to have received mental health residential care and emergency department or urgent care; to have used psychotropic medication; and to have previously attempted suicide (Table 1). Characteristics of individuals in the development and validation samples differed only in urban residence (62% vs 60%, respectively; χ2 = 7.63; P = .022); diabetes prevalence (50% vs 49%; χ2 = 10.69; P = .001), traumatic brain injury (2% vs 3%; χ2 = 4.33; P = .037), and mental health residential stays (0.7% vs 1.2%; χ2 = 4.49; P = .034).
TABLE 1—
Prediction Sample, Descriptive Characteristics, Overall, Among Suicide Decedents Within 12 Months: Veterans Health Administration, United States, 2008–2011
| Characteristic | Unique Patients, No. (%) | Suicide Decedents Within 12 Mo, No. (%) | Suicides/100 000, No. | In Top 5.00% of Predicted Probability, No. (%) | In Top 0.10% of Predicted Probability, No. (%) |
| All | 5 969 662 | 2138 (0.04) | 35.8 | 298 483 | 5 969 |
| Age,*** y | |||||
| 18–29 | 359 361 (6.0) | 147 (6.9) | 40.9 | 19 594 (6.6) | 558 (9.4) |
| 30–39 | 398 643 (6.7) | 145 (6.8) | 36.4 | 19 887 (6.7) | 589 (9.9) |
| 40–49 | 619 383 (10.4) | 271 (12.7) | 43.8 | 37 337 (12.5) | 1 140 (19.1) |
| 50–59 | 1 039 251 (17.4) | 422 (19.7) | 40.6 | 76 483 (25.6) | 2 113 (35.4) |
| 60–69 | 1 703 881 (28.5) | 507 (23.7) | 29.8 | 68 843 (23.1) | 1 035 (17.3) |
| 70–79 | 983 879 (16.5) | 305 (14.3) | 31.0 | 31 568 (10.6) | 302 (5.1) |
| ≥ 80 | 865 264 (14.5) | 341 (16.0) | 39.4 | 44 771 (15.0) | 232 (3.9) |
| Gender*** | |||||
| Male | 5 368 026 (89.9) | 2055 (96.1) | 38.3 | 291 062 (97.5) | 5 799 (97.2) |
| Female | 601 636 (10.1) | 83 (3.9) | 13.8 | 7 421 (2.5) | 170 (2.9) |
| Marital status** | |||||
| Divorced | 1 153 431 (19.3) | 615 (28.8) | 53.3 | 118 741 (39.8) | 2 782 (46.6) |
| Married | 3 196 869 (53.6) | 867 (40.6) | 27.1 | 84 239 (28.2) | 1 282 (21.5) |
| Never married | 715 417 (12.0) | 347 (16.2) | 48.5 | 54 517 (18.3) | 1 352 (22.7) |
| Separated | 157 888 (2.6) | 65 (3.0) | 41.2 | 9 111 (3.1) | 220 (3.7) |
| Unknown | 390 061 (6.5) | 65 (3.0) | 16.7 | 1 466 (0.5) | 45 (0.8) |
| Widowed | 355 996 (6.0) | 179 (8.3) | 50.3 | 30 409 (10.2) | 288 (4.8) |
| Residence*** | |||||
| Urban | 3 830 628 (64.2) | 1277 (59.7) | 33.3 | 180 322 (60.4) | 3 914 (65.6) |
| Rural | 2 120 056 (35.5) | 857 (40.1) | 40.4 | 116 907 (39.2) | 2 022 (33.9) |
| Missing | 18 978 (0.3) | 4 (0.2) | 21.1 | 1 254 (0.4) | 33 (0.6) |
| Service connection,*** % | |||||
| None | 3 499 357 (58.6) | 1388 (64.9) | 39.7 | 208 197 (69.8) | 4 084 (68.4) |
| < 30 | 652 173 (10.9) | 244 (11.4) | 37.4 | 29 449 (9.9) | 613 (10.3) |
| 30–70 | 1 085 766 (18.2) | 311 (14.6) | 28.6 | 37 974 (12.7) | 685 (11.5) |
| > 70 | 732 366 (12.3) | 195 (9.1) | 26.6 | 22 863 (7.7) | 587 (9.8) |
| Psychiatric diagnosis in past 24 mo | |||||
| Any*** | 2 245 554 (37.6) | 1250 (58.5) | 55.7 | 271 939 (91.1) | 5 961 (99.9) |
| Depression*** | 1 216 754 (20.4) | 876 (40.1) | 72.0 | 221 475 (74.2) | 5 471 (91.7) |
| Schizophrenia*** | 105 664 (1.8) | 81 (3.8) | 76.7 | 18 720 (6.3) | 809 (13.6) |
| Bipolar disorder*** | 148 357 (2.5) | 190 (8.9) | 128.1 | 45 554 (15.3) | 2 370 (39.7) |
| PTSD*** | 646 892 (10.8) | 328 (15.3) | 50.7 | 58 891 (19.7) | 1 793 (30.0) |
| Personality disorder*** | 69 806 (1.2) | 90 (4.2) | 128.2 | 22 905 (7.7) | 1 650 (27.6) |
| Substance use disorder*** | 619 136 (10.4) | 503 (23.5) | 81.2 | 123 509 (41.4) | 4 845 (81.2) |
| Other anxiety disorder*** | 529 949 (8.9) | 372 (17.4) | 70.2 | 105 684 (35.4) | 2 900 (48.6) |
| Other disorder*** | 78 706 (1.3) | 69 (3.2) | 87.7 | 19 205 (6.4) | 960 (16.1) |
| Medications in past 24 mo | |||||
| Antidepressant*** | 1 448 072 (24.3) | 905 (42.3) | 62.5 | 221 005 (74.0) | 5 367 (89.9) |
| Antipsychotic*** | 495 886 (8.3) | 410 (19.2) | 82.7 | 105 398 (35.3) | 4 371 (73.2) |
| Mood stabilizer*** | 747 261 (12.5) | 538 (25.2) | 72.0 | 128 443 (43.0) | 4 153 (69.6) |
| Sedative/anxiolytic*** | 897 777 (15.0) | 733 (34.3) | 81.6 | 201 134 (67.4) | 5 133 (86.0) |
| Medical diagnoses in past 24 moa | |||||
| Cancer | 1 197 370 (20.1) | 439 (20.5) | 36.7 | 69 954 (23.4) | 1 216 (20.4) |
| Cardiovascular disease | 1 494 932 (25.0) | 523 (24.5) | 35.0 | 81 378 (27.3) | 1 654 (27.7) |
| COPD*** | 897 565 (15.0) | 416 (19.5) | 46.3 | 81 356 (27.3) | 1 896 (31.8) |
| Coronary artery disease** | 1 072 256 (18.0) | 329 (15.4) | 30.7 | 45 690 (15.3) | 650 (10.9) |
| Diabetes mellitus*** | 1 346 003 (22.6) | 413 (19.3) | 30.7 | 37 174 (12.5) | 593 (9.9) |
| Hearing impairment | 1 011 269 (16.9) | 362 (16.9) | 35.8 | 40 070 (13.4) | 613 (10.3) |
| Hypertension | 3 177 261 (53.2) | 1144 (53.5) | 36.0 | 160 618 (53.8) | 3 141 (52.6) |
| Any pain diagnosis *** | 312 585 (5.2) | 241 (11.3) | 77.1 | 56 710 (19.0) | 3 488 (58.4) |
| Sleep disorder*** | 708 777 (11.9) | 355 (16.6) | 50.1 | 80 378 (26.9) | 2 033 (34.1) |
| Traumatic brain injury*** | 61 962 (1.0) | 53 (2.5) | 85.5 | 10 730 (3.6) | 391 (6.6) |
| Any mental health services | |||||
| In past 2 y*** | 1 667 136 (27.9) | 933 (46.5) | 56.0 | 230 609 (77.3) | 5 903 (98.9) |
| In past year*** | 1 332 440 (22.3) | 853 (39.9) | 64.0 | 207 410 (69.5) | 5 801 (97.2) |
| Seen ≥ 4 times*** | 687 882 (11.5) | 527 (24.7) | 76.6 | 132 077 (44.3) | 5 378 (90.1) |
| Seen ≥12 times | 276 682 (4.6) | 260 (12.2) | 94.0 | 59 946 (20.1) | 4 159 (69.7) |
| Inpatient mental health discharge | |||||
| In past 12 mo*** | 65 382 (1.1) | 138 (6.5) | 211.1 | 36 210 (12.1) | 4 466 (74.8) |
| In past 1 mo*** | 9 563 (0.2) | 29 (1.4) | 303.3 | 6 981 (2.3) | 2 009 (33.7) |
| Any suicide attempt | |||||
| In past 12 mo*** | 8 879 (0.2) | 43 (2.0) | 484.3 | 5 818 (2.0) | 1 197 (20.1) |
| In past 1 mo*** | 1 343 (0.0) | 9 (0.4) | 670.1 | 1 101 (0.4) | 459 (7.7) |
| Any emergency department use in past 12 mo*** | 925 060 (15.5) | 488 (22.8) | 52.8 | 114 188 (38.3) | 4 845 (81.2) |
| Any emergency department or urgent care use in past 12 mo*** | 1 100 758 (18.4) | 584 (27.3) | 53.1 | 128 855 (43.2) | 5 169 (86.6) |
| Any mental health residential care use in past 12 mo*** | 22 727 (0.4) | 25 (1.2) | 110.0 | 3 620 (1.2) | 233 (3.9) |
| Any homelessness in past 12 mo*** | 135 871 (2.3) | 110 (5.1) | 81.0 | 19 236 (6.4) | 1 195 (20.0) |
Note. COPD = chronic obstructive pulmonary disease; PTSD = posttraumatic stress disorder.
Medical conditions present in 10% or more of the population.
*P < .05; **P < .01; ***P < .001, for difference between suicide and nonsuicide patients, derived from the χ2 or Fisher exact test.
We used logistic regression parameters derived from the development sample to stratify patients by their predicted risk for suicide (Table 2); the c statistic was 0.761 (95% confidence interval = 0.751, 0.771), suggesting that the model was acceptable.35,36 A comparison of findings from the development and validation samples indicated overfitting in the development sample. Nevertheless, findings from the validation sample documented strong risk concentration and high suicide rates in the top risk strata (e.g., 59.7 for the top 0.01%, 29.9 for the top 0.10%, 11.6 for the top 1.00%, and 5.7 for the top 5.00%). For the top 0.10% stratum and those at lower risk, findings regarding risk concentration in the development and validation samples were comparable, within 25%, and with greater agreement at lower tiers of predicted risk (e.g., for patients in the top 0.10%, risk concentrations in the development and validation samples were 39.0 and 29.9, respectively; for the top 1.00%, they were 14.0 and 11.6; for the top 5.00%, 6.3 and 5.7; and for the top 10.00%, 4.5 and 4.1). We used regression parameters from the development sample to stratify patients and predict suicide over time (Table 3, Figure 1). The observed suicide rate increased with greater calculated risk, peaking in the initial month and declining fairly monotonically over the subsequent year. For the highest-risk strata, suicide risks remained substantially elevated throughout the year. Nonsuicide external-cause and overall nonsuicide mortality were elevated, to a lower degree, in the highest strata of predicted risk for suicide, with increases that, in general, paralleled the predicted risk for suicide.
TABLE 2—
Suicide Risk Concentration, by Sample and Tier of Predicted Suicide Probability: Veterans Health Administration, United States, 2008–2011
| Prediction, Subsequent Mo |
Development, Index Mo |
Validation, Index Mo |
|||||||
| Tier of Predicted Probability, % | Suicide Decedents, % | Ratio of Observed Suicides to Expected | Annualized Suicide Rate | Suicide Decedents, % | Ratio of Observed Suicides to Expected | Annualized Suicide Rate | Suicide Decedents, % | Ratio of Observed Suicides to Expected | Annualized Suicide Rate |
| 0.01 | 1.4 | 142.2 | 5926.6 | 0.8 | 81.8 | 2954.5 | 0.6 | 59.7 | 2159.1 |
| 0.05 | 3.3 | 66.4 | 2762.0 | 2.5 | 50.9 | 1840.8 | 1.5 | 30.8 | 1113.6 |
| 0.10 | 4.3 | 42.7 | 1775.3 | 3.9 | 39.0 | 1409.0 | 3.0 | 29.9 | 1079.5 |
| 0.50 | 7.6 | 15.2 | 631.2 | 9.6 | 19.1 | 690.9 | 7.6 | 15.2 | 547.7 |
| 1.00 | 10.4 | 10.4 | 433.9 | 14.0 | 14.0 | 504.5 | 11.6 | 11.6 | 419.3 |
| 5.00 | 23.2 | 4.6 | 193.3 | 31.3 | 6.3 | 225.9 | 28.3 | 5.7 | 204.3 |
| 10.00 | 38.4 | 3.8 | 159.8 | 44.7 | 4.5 | 161.6 | 41.2 | 4.1 | 148.9 |
| 20.00 | 53.1 | 2.7 | 110.5 | 60.8 | 3.0 | 109.9 | 57.5 | 2.9 | 103.8 |
| 50.00 | 83.9 | 1.7 | 69.8 | 84.7 | 1.7 | 61.2 | 83.1 | 1.7 | 60.1 |
| 100.00 | 100.0 | 1.0 | 41.6 | 100.0 | 1.0 | 36.1 | 100.0 | 1.0 | 36.1 |
Note. Deaths from suicide were assessed in the month following calculation of predicted risk for the prediction sample and in the month of assessment for the development and validation samples.
TABLE 3—
Suicide, Nonsuicide External-Cause, and All Nonsuicide Mortality in Prediction Sample, by Tier of Predicted Suicide Probability: Veterans Health Administration, United States, 2008–2011
| Deaths In Subsequent Mo, No. |
All Deaths Observed During Study Period, % |
Mortality/100 000 Person-Years of Risk Time |
Mortality Risk Concentration Observed During Study Period |
||||||||||||||||||
| Tier of Predicted Suicide Probability, % | Patients, No. | 1 | 3 | 6 | 9 | 12 | 1 | 3 | 6 | 9 | 12 | 1 | 3 | 6 | 9 | 12 | 1 | 3 | 6 | 9 | 12 |
| Suicide mortality | |||||||||||||||||||||
| 0.01 | 596 | 3 | 5 | 5 | 7 | 7 | 1.4 | 0.9 | 0.5 | 0.4 | 0.3 | 5 927 | 3 328 | 1 682 | 1 570 | 1 174 | 142 | 92 | 48 | 45 | 33 |
| 0.10 | 5 969 | 9 | 16 | 21 | 27 | 35 | 4.3 | 2.9 | 2.0 | 1.7 | 1.6 | 1 775 | 1 063 | 706 | 605 | 586 | 43 | 29 | 20 | 17 | 16 |
| 1.00 | 59 696 | 22 | 51 | 94 | 128 | 176 | 10.4 | 9.4 | 9.0 | 8.1 | 8.2 | 434 | 339 | 316 | 287 | 295 | 10 | 9.4 | 9.0 | 8.1 | 8.2 |
| 5.00 | 298 493 | 49 | 130 | 262 | 371 | 507 | 23.2 | 23.9 | 25.0 | 23.6 | 23.7 | 193 | 173 | 176 | 166 | 170 | 4.6 | 4.8 | 5.0 | 4.7 | 4.7 |
| 10.00 | 596 966 | 81 | 208 | 390 | 563 | 760 | 38.4 | 38.2 | 37.1 | 35.8 | 35.5 | 160 | 138 | 131 | 126 | 127 | 3.8 | 3.8 | 3.7 | 3.6 | 3.6 |
| 50.00 | 2 984 831 | 177 | 455 | 856 | 1 255 | 1 725 | 83.9 | 83.5 | 81.5 | 79.9 | 80.7 | 70 | 60 | 58 | 56 | 58 | 1.7 | 1.7 | 1.6 | 1.6 | 1.6 |
| 100.00 | 5 969 662 | 211 | 545 | 1 050 | 1 571 | 2 138 | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 | 42 | 36 | 35 | 35 | 36 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 |
| Nonsuicide external-cause mortality | |||||||||||||||||||||
| 0.01 | 596 | 0 | 0 | 2 | 5 | 5 | 0.0 | 0.0 | 0.1 | 0.1 | 0.1 | 0 | 0 | 673 | 1 122 | 839 | 0 | 0 | 5.4 | 9.0 | 6.7 |
| 0.10 | 5 969 | 6 | 19 | 36 | 54 | 64 | 0.9 | 1.0 | 1.0 | 1.0 | 0.9 | 1 184 | 1 263 | 1 210 | 1 210 | 1 072 | 9.4 | 10 | 9.7 | 9.7 | 8.6 |
| 1.00 | 59 696 | 37 | 101 | 199 | 278 | 355 | 5.8 | 5.3 | 5.4 | 5.0 | 4.8 | 730 | 671 | 669 | 623 | 595 | 5.8 | 5.3 | 5.4 | 5.0 | 4.8 |
| 5.00 | 298 493 | 110 | 315 | 583 | 835 | 1 096 | 17.2 | 16.5 | 15.8 | 15.0 | 14.7 | 434 | 419 | 392 | 374 | 367 | 3.4 | 3.3 | 3.2 | 3.0 | 2.9 |
| 10.00 | 596 966 | 178 | 490 | 952 | 1 391 | 1 855 | 27.9 | 25.7 | 25.7 | 25.0 | 24.8 | 351 | 326 | 320 | 312 | 311 | 2.8 | 2.6 | 2.6 | 2.5 | 2.5 |
| 50.00 | 2 984 831 | 470 | 1 357 | 2 619 | 3 908 | 5 203 | 73.7 | 71.3 | 70.8 | 70.2 | 69.7 | 185 | 180 | 176 | 175 | 174 | 1.5 | 1.4 | 1.4 | 1.4 | 1.4 |
| 100.00 | 5 969 662 | 638 | 1 904 | 3 699 | 5 568 | 7 470 | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 | 126 | 127 | 124 | 125 | 125 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 |
| All nonsuicide mortality | |||||||||||||||||||||
| 0.01 | 5 96 | 4 | 9 | 20 | 30 | 39 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 7 902 | 5 991 | 6 730 | 6 730 | 6 544 | 2.5 | 1.8 | 2.0 | 2.0 | 2.0 |
| 0.10 | 5969 | 45 | 108 | 196 | 268 | 348 | 0.3 | 0.2 | 0.2 | 0.2 | 0.2 | 8 877 | 7 178 | 6 585 | 6 003 | 5 830 | 2.8 | 2.2 | 2.0 | 1.8 | 1.8 |
| 1.00 | 59 696 | 459 | 1 160 | 2 099 | 2 934 | 3 699 | 2.9 | 2.4 | 2.1 | 2.0 | 1.9 | 9 053 | 7 709 | 7 052 | 6 571 | 6 196 | 2.9 | 2.4 | 2.1 | 2.0 | 1.9 |
| 5.00 | 298 493 | 1 881 | 5 158 | 9 902 | 14 007 | 17 814 | 11.9 | 10.5 | 9.9 | 9.6 | 9.3 | 7 420 | 6 856 | 6 653 | 6 274 | 5 968 | 2.4 | 2.1 | 2.0 | 1.9 | 1.9 |
| 10.00 | 596 966 | 3 427 | 9 700 | 18 944 | 26 944 | 34 389 | 21.6 | 19.8 | 18.9 | 18.4 | 18.0 | 6 759 | 6 447 | 6 364 | 6 035 | 5 761 | 2.2 | 2.0 | 1.9 | 1.8 | 1.8 |
| 50.00 | 2 984 831 | 10 978 | 33 195 | 67 179 | 97 272 | 126 298 | 69.3 | 67.7 | 67.1 | 66.4 | 66.0 | 4 330 | 4 412 | 4 514 | 4 357 | 4 231 | 1.4 | 1.4 | 1.3 | 1.3 | 1.3 |
| 100.00 | 5 969 662 | 15 850 | 49 017 | 100 134 | 146 389 | 191 257 | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 | 31 26 | 3 258 | 3 364 | 3 279 | 3 204 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 |
FIGURE 1—
Prediction sample over 12 months, with top tiers of predicted probability for (a) suicide risk concentration and (b) suicide rate: Veterans Health Administration, United States, 2008–2011.
VHA’s program for flagging medical records for patients determined on clinical grounds to be at high risk allowed comparisons between calculated risks and clinical evaluations. Analyses that did not consider the timing of flags demonstrated that for the top 0.01% stratum, 184 of the 596 patients (31%) were flagged; these accounted for 3 of the 7 suicides observed for this stratum over 1 year. For the top 0.10%, 1253 of 5969 (21%) were flagged, accounting for 16 of 35 deaths; for the top 1.00%, 5225 of 59 696 (8.8%) were flagged, accounting for 38 of 176 suicides. If analyses had only considered associations that occurred within a specific window of time, the clinical flags might have been associated with higher degrees of predicted risk, but the likelihood that patients at high predicted risk would have been flagged would be lower. Regardless, the findings demonstrated that the majority of patients in the highest strata of predicted risk and the majority of deaths from suicide were not identified on clinical grounds.
Descriptive statistics on the patients in selected strata (Table 1) demonstrated greater proportions of younger (χ2 = 2949.6; P < .001), divorced (χ2 = 4360.4; P < .001), and urban patients in the top 0.10% risk stratum (χ2 = 16.7; P < .001). Both the 5.00% and the 0.10% strata had substantially greater prevalence rates for known risk factors. However, from a clinical perspective, these groups could be best distinguished, not by the proportion of patients with specific risk factors, but by their complexity and the extent of their comorbidities. For example, 81.0% of those in the top 5.00% and 98.50% in the top 0.10% had at least 1 of 8 mental health diagnoses: depression, schizophrenia, bipolar disorder, posttraumatic stress disorder, personality disorder, substance use disorder, other anxiety disorder, or other psychiatric disorder in the previous 12 months. The average numbers of conditions were 2.08 (SD = 1.08) and 3.15 (SD = 1.25), respectively. Furthermore, when we also considered pain, sleep, and traumatic brain injury, 84.0% in the top 5.00% and 98.8% in the top 0.10% had at least 1 of these 11 conditions, with average numbers of 2.41 (SD = 1.32), and 3.92 (SD = 1.58), respectively.
DISCUSSION
Our analyses demonstrated that predictive models with variables extracted from electronic medical records can identify strata of patients with substantial increases in their risk for suicide. We derived the model from case–control data for deaths from suicide and living patients. However, its clinical value lies in its success in predicting death from suicide over time.
We detected potential limitations to these analyses in differences in risk concentration between the development and validation samples that suggested that the analytic strategy led to overfitting and overestimation of the magnitude of risk concentration in the highest strata in the development sample. However, except for those in the tails of the distribution, representing the highest degree of risk, those beyond the top 0.10%, findings from the 2 samples were comparable and within 25%. Although differences between the samples raised questions about the magnitude of the predicted effects, findings from the development and validation samples indicated that risk concentration was likely to be between 60 and 82 for the top 0.01% stratum, 30 to 39 for the top 0.10%, 12 to 14 for the top 1.00%, and 5.7 to 6.3 for the top 5.00%; the higher apparent rate for the top 0.01% of the prediction sample likely reflected the low numbers of deaths and, therefore, the instability of the estimates.
Overall, despite the limitations, the basic conclusion must be that it is feasible to identify patients at defined strata of elevated risk for suicide by using measures derived from electronic medical records. Evaluation of the associations between patients clinically flagged as being at high risk and those identified through predictive modeling support the validity of the model; they also confirm the previous suggestion that predictive modeling can provide new information about who is at risk.13 The observation that the patients in the top 0.10% and 5.00% of predicted risk were characterized by extensive behavioral health comorbidities is important for designing and implementing interventions. However, VHA’s electronic medical records do not systematically include quantitative data on the severity of mental health conditions. Therefore, the findings that risk was related to comorbidity and complexity may, in part, reflect the nature of the information included in the model.
The model developed to calculate the risk of death from suicide predicted other external causes, and, to a lesser extent, overall nonsuicide mortality. The findings on nonsuicide external causes were probably attributable to shared risks between suicide and such causes of death as accidents, overdoses, and injuries. The findings on overall nonsuicide mortality may reflect the risk of suicide associated with serious medical illness, as well as shared risks for both suicide and natural deaths, perhaps related to nonadherence and unhealthy behaviors. It may be related to the overall increases in mortality that are observed, to a lower extent in VHA than other settings, in patients with serious mental illness.37 These findings suggest that selective clinical preventive strategies that use the predictive model to target potential recipients may lead to outcomes that go beyond suicide prevention.
The most direct clinical application of predictive modeling would be to allow targeting of selective clinical and preventive services. One strategy would enhance clinical care, possibly through a program where each patient’s provider(s), probably with support from a care manager, would reevaluate care plans, identify and address barriers to delivering evidence-based care, implement and monitor the outcomes of any needed changes in treatment, and repeat this process as necessary. Another approach would be to implement interventions derived from evidence-based strategies for suicide prevention,22,23 such as patient education, expressions of caring, regularly scheduled contact, outreach in response to missed appointments or gaps in care, facilitated access to services when patients experience the need, and training in coping skills such as problem solving.
The most obvious need may be to do everything possible to address the needs of patients who are at highest risk: for example, the patients in the highest 0.10% stratum. However, even if a strategy were fully effective in this stratum, it would prevent only a small proportion of the total number of suicides that occur in VHA each year. To achieve substantial reductions in the burden of suicide, it will be necessary to target larger strata of patients at lower—but still elevated—risk; for example, the 5.00% of patients who account for 24% of suicides in VHA patients over 1 year (Table 3). Because so many patients fall into this stratum, and because of the magnitude of the resources that would be required for a comprehensive approach for these patients, demonstration projects and research are needed to develop and validate an array of risk-stratified interventions that can be realistically delivered across a health care system.
Acknowledgments
We gratefully acknowledge contributions from Karen Austin, Stacey Collins, Rosalinda Ignacio, and Samantha Louzon.
Human Participant Protection
This project was conducted in support of clinical operations and approved by the Department of Veterans Affairs’ Mental Health Services and Office of Mental Health Operations. The study followed institutional guidelines for approval and protection of human participants.
References
- 1.Katz I. Lessons learned from mental health enhancement and suicide prevention activities in the Veterans Health Administration. Am J Public Health. 2012;102(suppl 1):S14–S16. doi: 10.2105/AJPH.2011.300582. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Office of the Surgeon General and National Action Alliance for Suicide Prevention. 2012 National Strategy for Suicide Prevention: Goals and Objectives for Action. Washington, DC: US Dept of Health and Human Services; 2012. [PubMed] [Google Scholar]
- 3.Kemp J. Suicide rates in VHA patients with comparisons with other Americans and other veterans through 2010. Available at: http://www.mentalhealth.va.gov/docs/suicide_data_report_update_january_2014.pdf. Accessed November 3, 2014.
- 4.Centers for Disease Control and Prevention. Suicide among adults aged 35–64 years—United States, 1999–2010. MMWR Morb Mortal Wkly Rep. 2013;62(17):321–325. [PMC free article] [PubMed] [Google Scholar]
- 5.Xu JQ, Kochanek KD, Murphy SL, Arias E. Mortality in the United States, 2012. Hyattsville, MD: National Center for Health Statistics; 2014. NCHS data brief 168. [Google Scholar]
- 6.Katz IR, McCarthy JF, Ignacio RV, Kemp J. Suicide among veterans in 16 states, 2005 to 2008: comparisons between utilizers and nonutilizers of Veterans Health Administration (VHA) services based on data from the National Death Index, the National Violent Death Reporting System, and VHA administrative records. Am J Public Health. 2012;102(suppl 1):S105–S110. doi: 10.2105/AJPH.2011.300503. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Oquendo MA, Halberstam B, Mann JJ. Risk factors for suicidal behaviors: the utility and limitations of research instruments. In: First MB, editor. Standardized Evaluation in Clinical Practice. Arlington, VA: American Psychiatric Press; 2003. pp. 103–130. [Google Scholar]
- 8.Mann JJ, Apter A, Bertolote J et al. Suicide prevention strategies: a systematic review. JAMA. 2005;294(16):2064–2074. doi: 10.1001/jama.294.16.2064. [DOI] [PubMed] [Google Scholar]
- 9.Galfalvy HC, Oquendo MA, Mann JJ. Evaluation of clinical prognostic models for suicide attempts after a major depressive episode. Acta Psychiatr Scand. 2008;117(4):244–252. doi: 10.1111/j.1600-0447.2008.01162.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Cooper J, Kapur N, Dunning J, Guthrie E, Appleby L, Mackway-Jones K. A clinical tool for assessing risk after self-harm. Ann Emerg Med. 2006;48(4):459–466. doi: 10.1016/j.annemergmed.2006.07.944. [DOI] [PubMed] [Google Scholar]
- 11.Tiet QQ, Ilgen MA, Byrnes HF, Moos RH. Suicide attempts among substance use disorder patients: an initial step toward a decision tree for suicide management. Alcohol Clin Exp Res. 2006;30(6):998–1005. doi: 10.1111/j.1530-0277.2006.00114.x. [DOI] [PubMed] [Google Scholar]
- 12.Kessler RC, Warner CH, Ivany C et al. Predicting suicides after psychiatric hospitalization in US Army soldiers: the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS) JAMA Psychiatry. 2015;72(1):49–57. doi: 10.1001/jamapsychiatry.2014.1754. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Tran T, Luo W, Phung D et al. Risk stratification using data from electronic medical records better predicts suicide risks than clinical assessments. BMC Psychiatry. 2014;14(1):76. doi: 10.1186/1471-244X-14-76. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Suicide Prevention Resource Center. Suicide prevention basics. Available at: http://www.sprc.org/basics/about-suicide. Accessed November 3, 2014.
- 15.Morriss R, Kapur N, Byng R. Assessing risk of suicide or self harm in adults. BMJ. 2013;347:f4572. doi: 10.1136/bmj.f4572. [DOI] [PubMed] [Google Scholar]
- 16.Ganzini L, Denneson LM, Press N et al. Trust is the basis for effective suicide risk screening and assessment in veterans. J Gen Intern Med. 2013;28(9):1215–1221. doi: 10.1007/s11606-013-2412-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Boudreaux ED, Horowitz LM. Suicide risk screening and assessment: designing instruments with dissemination in mind. Am J Prev Med. 2014;47(3 suppl 2):S163–S169. doi: 10.1016/j.amepre.2014.06.005. [DOI] [PubMed] [Google Scholar]
- 18.Glenn CR, Nock MK. Improving the short-term prediction of suicidal behavior. Am J Prev Med. 2014;47(3 suppl 2):S176–S180. doi: 10.1016/j.amepre.2014.06.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Claassen CA, Harvilchuck-Laurenson JD, Fawcett J. Prognostic models to detect and monitor the near-term risk of suicide: state of the science. Am J Prev Med. 2014;47(3 suppl 2):S181–S185. doi: 10.1016/j.amepre.2014.06.003. [DOI] [PubMed] [Google Scholar]
- 20.Research Prioritization Task Force. A Prioritized Research Agenda for Suicide Prevention: An Action Plan to Save Lives. Rockville, MD: National Action Alliance for Suicide Prevention; 2014. [Google Scholar]
- 21.Institute of Medicine. Reducing risks for mental disorders: frontiers for preventive intervention research. In: Mrazek PJ, Haggerty RJ, editors. Committee on Prevention of Mental Disorders, Division of Biobehavorial Sciences and Mental Disorders. Washington, DC: National Academies Press; 1994. [PubMed] [Google Scholar]
- 22.The Assessment and Management of Risk for Suicide Working Group. VA/DoD Clinical Practice Guideline for Assessment and Management of Patients at Risk for Suicide. Version 1.0. Washington, DC: Veterans Health Administration and Department of Defense; 2013. [Google Scholar]
- 23.Pringle B, Colpe LJ, Heinssen RK et al. A strategic approach for prioritizing research and action to prevent suicide. Psychiatr Serv. 2013;64(1):71–75. doi: 10.1176/appi.ps.201100512. [DOI] [PubMed] [Google Scholar]
- 24.McCarthy JF, Valenstein M, Kim HM, Ilgen M, Zivin K, Blow FC. Suicide mortality among patients receiving care in the Veterans Health Administration health system. Am J Epidemiol. 2009;169(8):1033–1038. doi: 10.1093/aje/kwp010. [DOI] [PubMed] [Google Scholar]
- 25.Ilgen MA, Bohnert ASB, Ignacio R et al. Psychiatric diagnoses and risk of suicide in veterans. Arch Gen Psychiatry. 2010;67(11):1152–1158. doi: 10.1001/archgenpsychiatry.2010.129. [DOI] [PubMed] [Google Scholar]
- 26.McCarthy JF, Blow FC, Ignacio RV, Ilgen MA, Austin KL, Valenstein M. Suicide among patients in the Veterans Affairs health system: rural–urban differences in rates, risks, and methods. Am J Public Health. 2012;102(suppl 1):S111–S117. doi: 10.2105/AJPH.2011.300463. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Valenstein M, Kim HM, Ganoczy D et al. Higher-risk periods for suicide among VA patients receiving depression treatment: prioritizing suicide prevention efforts. J Affect Disord. 2009;112(1–3):50–58. doi: 10.1016/j.jad.2008.08.020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Ilgen MA, Kleinberg F, Ignacio RV et al. Noncancer pain conditions and risk of suicide. JAMA Psychiatry. 2013;70(7):692–697. doi: 10.1001/jamapsychiatry.2013.908. [DOI] [PubMed] [Google Scholar]
- 29.Kim HM, Smith EG, Ganoczy D et al. Predictors of suicide in patient charts among patients with depression in the Veterans Health Administration health system: importance of prescription drug and alcohol abuse. J Clin Psychiatry. 2012;73(10):e1269–e1275. doi: 10.4088/JCP.12m07658. [DOI] [PubMed] [Google Scholar]
- 30.Ilgen MA, McCarthy JF, Ignacio RV et al. Psychopathology, Iraq and Afghanistan service, and suicide among Veterans Health Administration patients. J Consult Clin Psychol. 2012;80(3):323–330. doi: 10.1037/a0028266. [DOI] [PubMed] [Google Scholar]
- 31.Brenner LA, Ignacio RV, Blow FC. Suicide and traumatic brain injury among individuals seeking Veterans Health Administration services. J Head Trauma Rehabil. 2011;26(4):257–264. doi: 10.1097/HTR.0b013e31821fdb6e. 2100. [DOI] [PubMed] [Google Scholar]
- 32.Basham C, Denneson LM, Millet L, Shen X, Duckart J, Dobscha SK. Characteristics and VA health care utilization of U.S. veterans who completed suicide in Oregon between 2000 and 2005. Suicide Life Threat Behav. 2011;41(3):287–296. doi: 10.1111/j.1943-278X.2011.00028.x. [DOI] [PubMed] [Google Scholar]
- 33.Ilgen MA, Conner KR, Valenstein M, Austin K, Blow FC. Violent and nonviolent suicide in veterans with substance-use disorders. J Stud Alcohol Drugs. 2010;71(4):473–479. doi: 10.15288/jsad.2010.71.473. [DOI] [PubMed] [Google Scholar]
- 34.Pfeiffer PN, Ganoczy D, Ilgen M, Zivin K, Valenstein M. Comorbid anxiety as a suicide risk factor among depressed veterans. Depress Anxiety. 2009;26(8):752–757. doi: 10.1002/da.20583. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143(1):29–36. doi: 10.1148/radiology.143.1.7063747. [DOI] [PubMed] [Google Scholar]
- 36.Hosmer DW, Lemeshow S, Sturdivant RX. Applied Logistic Regression. 3rd ed. Hoboken, NJ: Wiley; 2013. [Google Scholar]
- 37.Kilbourne AM, Ignacio RV, Kim HM, Blow FC. Datapoints: are VA patients with serious mental illness dying younger? Psychiatr Serv. 2009;60(5):589. doi: 10.1176/ps.2009.60.5.589. [DOI] [PubMed] [Google Scholar]