Assessment of a Risk Index for Suicide Attempts Among US Army Soldiers With Suicide Ideation: Analysis of Data From the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS)

Kelly L Zuromski; Samantha L Bernecker; Peter M Gutierrez; Thomas E Joiner; Andrew J King; Howard Liu; James A Naifeh; Matthew K Nock; Nancy A Sampson; Alan M Zaslavsky; Murray B Stein; Robert J Ursano; Ronald C Kessler

doi:10.1001/jamanetworkopen.2019.0766

. 2019 Mar 15;2(3):e190766. doi: 10.1001/jamanetworkopen.2019.0766

Assessment of a Risk Index for Suicide Attempts Among US Army Soldiers With Suicide Ideation

Analysis of Data From the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS)

Kelly L Zuromski ^1,², Samantha L Bernecker ^1,², Peter M Gutierrez ^3,⁴, Thomas E Joiner ⁵, Andrew J King ¹, Howard Liu ¹, James A Naifeh ⁶, Matthew K Nock ², Nancy A Sampson ¹, Alan M Zaslavsky ¹, Murray B Stein ^7,⁸, Robert J Ursano ⁶, Ronald C Kessler ^1,^✉

¹Department of Health Care Policy, Harvard Medical School, Boston, Massachusetts

²Department of Psychology, Harvard University, Cambridge, Massachusetts

³Department of Psychiatry, University of Colorado School of Medicine, Aurora

⁴Rocky Mountain Mental Illness Research, Education, and Clinical Center, Rocky Mountain Regional Veterans Affairs Medical Center, Aurora, Colorado

⁵Department of Psychology, Florida State University, Tallahassee

⁶Center for the Study of Traumatic Stress, Department of Psychiatry, Uniformed Services University School of Medicine, Bethesda, Maryland

⁷Department of Psychiatry, University of California, San Diego, La Jolla

⁸Department of Family Medicine and Public Health, University of California, San Diego, La Jolla

Accepted for Publication: January 29, 2019.

Published: March 15, 2019. doi:10.1001/jamanetworkopen.2019.0766

^✉

Corresponding Author: Ronald C. Kessler, PhD, Department of Health Care Policy, Harvard Medical School, 180 Longwood Ave, Boston, MA 02115 (kessler@hcp.med.harvard.edu).

Author Contributions: Dr Kessler had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Zuromski, Bernecker, Joiner, Stein, Ursano, Kessler

Acquisition, analysis, or interpretation of data: Zuromski, Bernecker, Gutierrez, King, Liu, Naifeh, Nock, Sampson, Zaslavsky, Stein, Ursano, Kessler

Drafting of the manuscript: Zuromski, King, Kessler

Critical revision of the manuscript for important intellectual content: Zuromski, Bernecker, Gutierrez, Joiner, Liu, Naifeh, Nock, Sampson, Zaslavsky, Stein, Ursano, Kessler

Statistical analysis: King, Liu, Zaslavsky, Kessler

Obtained funding: Stein, Ursano, Kessler

Administrative, technical, or material support: Gutierrez, Joiner, Sampson

Supervision: Zuromski, Bernecker, Naifeh, Nock, Sampson, Stein, Ursano, Kessler

Conflict of Interest Disclosures: Dr Zuromski reported grants from the Military Suicide Research Consortium during the conduct of the study. Dr Bernecker reported grants from the Military Suicide Research Consortium during the conduct of the study. Dr Gutierrez reported grants from the US Department of Defense (DoD) during the conduct of the study. Dr Joiner reported grants from Florida State University during the conduct of the study. Dr Liu reported grants from the US Department of Health and Human Services (HHS), the DoD, and the Military Suicide Research Consortium during the conduct of the study. Dr Naifeh reported grants from the HHS and DoD during the conduct of the study. Dr Nock reported grants from the HHS, DoD, and Military Suicide Research Consortium during the conduct of the study. Dr Zaslavsky reported grants from HHS, the DoD, and the Military Suicide Research Consortium during the conduct of the study. Dr Stein reported grants from National Institute of Mental Health, the DoD, and the Military Suicide Research Consortium during the conduct of the study and serving as a consultant for Actelion, Dart Neuroscience, Healthcare Management Technologies, Janssen, Oxeia Biopharmaceuticals, Pfizer, Resilience Therapeutics, and Tonix Pharmaceuticals outside the submitted work. Dr Kessler reported grants from the HHS, the DoD, and the Military Suicide Research Consortium during the conduct of the study and reported receiving support for his epidemiological studies from Sanofi Aventis; serving as a consultant for Johnson & Johnson Wellness and Prevention, Sage, Shire, and Takeda; and serving on an advisory board for the Johnson & Johnson Services Inc Lake Nona Life Project outside the submitted work. Dr Kessler is a co-owner of DataStat, Inc, a market research firm that carries out health care research. No other disclosures were reported.

Funding/Support: The Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS) was sponsored by the Department of the Army and funded under cooperative agreement U01MH087981 (2009-2015) with the US Department of Health and Human Services, National Institutes of Health, and National Institute of Mental Health. Subsequently, STARRS-LS was sponsored and funded by the Department of Defense (USUHS grant HU0001-15-2-0004). This work was in part supported by the Military Suicide Research Consortium, an effort supported by the Office of the Assistant Secretary of Defense for Health Affairs (Award W81XWH-16-2-0003, W81XWH-16-2-0004).

Role of the Funder/Sponsor: As a cooperative agreement, scientists employed by the National Institute of Mental Health and US Army liaisons and consultants collaborated in the design and conduct of the study (including development of the Army STARRS survey study protocol and data collection instruments and supervision of data collection); management, analysis, and interpretation of the data; preparation, review, and approval of the manuscript; and decision to submit the manuscript for publication. Although a draft of the manuscript was submitted to the US Army and National Institute of Mental Health for review and comment before submission for publication, this was done with the understanding that comments would be no more than advisory.

Disclaimer: The contents are solely the responsibility of the authors and do not necessarily represent the views of the US Department of Health and Human Services, the National Institute of Mental Health, the US Department of the Army, the US Department of Defense, the Department of Veterans Affairs, or the Military Suicide Research Consortium.

Additional Contributions: Charles Hoge, MD (Center for Psychiatry and Neuroscience, Walter Reed Army Institute of Research, Silver Spring, Maryland, and Office of the Army Surgeon General, Falls Church, Virginia), provided comments on an earlier version of the article. He was not compensated for his contribution.

Additional Information: The Army STARRS Team consists of co–principal investigators Robert J. Ursano, MD (Uniformed Services University of the Health Sciences) and Murray B. Stein, MD, MPH (University of California San Diego and VA San Diego Healthcare System); Site Principal Investigators Steven Heeringa, PhD (University of Michigan), James Wagner, PhD (University of Michigan), and Ronald C. Kessler, PhD (Harvard Medical School); and Army liaison/consultant Kenneth Cox, MD, MPH (US Army Public Health Center [provisional]). Other team members are Pablo A. Aliaga, MS (Uniformed Services University of the Health Sciences); COL David M. Benedek, MD (Uniformed Services University of the Health Sciences); Laura Campbell-Sills, PhD (University of California San Diego); Chia-Yen Chen DSc (Harvard Medical School); Carol S. Fullerton, PhD (Uniformed Services University of the Health Sciences); Nancy Gebler, MA (University of Michigan); Joel Gelernter (Yale University); Robert K. Gifford, PhD (Uniformed Services University of the Health Sciences); Paul E. Hurwitz, MPH (Uniformed Services University of the Health Sciences); Sonia Jain, PhD (University of California San Diego); Tzu-Cheg Kao, PhD (Uniformed Services University of the Health Sciences); Lisa Lewandowski-Romps, PhD (University of Michigan); Holly Herberman Mash, PhD (Uniformed Services University of the Health Sciences); James E. McCarroll, PhD, MPH (Uniformed Services University of the Health Sciences); James A. Naifeh, PhD (Uniformed Services University of the Health Sciences); Tsz Hin Hinz Ng, MPH (Uniformed Services University of the Health Sciences); Matthew K. Nock, PhD (Harvard University); Nancy A. Sampson, BA (Harvard Medical School); CDR Patcho Santiago, MD, MPH (Uniformed Services University of the Health Sciences); Jordan W. Smoller, MD, ScD (Harvard Medical School); and Alan M. Zaslavsky, PhD (Harvard Medical School).

^✉

Corresponding author.

PMCID: PMC6484656 PMID: 30874786

Key Points

Question

Is a short self-report battery associated with improved assessment of risk for suicide attempt among soldiers with suicide ideation?

Findings

This cohort study of 3649 soldiers participating in the Army Study to Assess Risk and Resilience in Servicemembers survey found that a cross-validated model including self-reported history and severity of suicidal thoughts and behaviors, positive screens for mental disorders, and Army career characteristics was associated with administratively reported suicide attempts 18 to 45 months following baseline among respondents with lifetime suicide ideation at baseline. The 10% of those with suicide ideation who had the highest estimated risk accounted for 39.2% of subsequent suicide attempts.

Meaning

It may be feasible to develop a clinical risk index for suicide attempt given suicide ideation from a small number of self-report questions.

This cohort study investigates whether a brief self-report battery from a survey of US Army soldiers is associated with identification of those with suicide ideation who are at the highest risk of suicide attempt.

Abstract

Importance

The Department of Veterans Affairs recently began requiring annual suicide ideation (SI) screening of all patients and additional structured questions for patients reporting SI. Related changes are under consideration at the Department of Defense. These changes will presumably lead to higher SI detection, which will require hiring additional clinical staff and/or developing a clinical decision support system to focus in-depth suicide risk assessments on patients considered high risk.

Objective

To carry out a proof-of-concept study for whether a brief structured question battery from a survey of US Army soldiers can help target in-depth suicide risk assessments by identifying soldiers with self-reported lifetime SI who are at highest risk of subsequent administratively recorded nonfatal suicide attempts (SAs).

Design, Setting, and Participants

Cohort study with prospective observational design. Data were collected from May 2011 to February 2013. Participants were followed up through December 2014. Analyses were conducted from March to November 2018. A logistic regression model was used to assess risk for subsequent administratively recorded nonfatal SAs. A total of 3649 Regular Army soldiers in 3 Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS) surveys who reported lifetime SI were followed up for 18 to 45 months from baseline to assess administratively reported nonfatal SAs.

Main Outcomes and Measures

Outcome was administratively recorded nonfatal SAs between survey response and December 2014. Predictors were survey variables.

Results

The 3649 survey respondents were 80.5% male and had a median (interquartile range) age of 29 (25-36) years (range, 18-55 years); 69.4% were white non-Hispanic, 14.6% were black, 9.0% were Hispanic, 7.0% were another racial/ethnic group. Sixty-five respondents had administratively recorded nonfatal SAs between survey response and December 2014. One additional respondent died by suicide without making a nonfatal SA but was excluded from analysis based on previous evidence that predictors are different for suicide death and nonfatal SAs. Significant risk factors were SI recency (odds ratio [OR], 7.2; 95% CI, 2.9-18.0) and persistence (OR, 2.6; 95% CI, 1.0-6.8), positive screens for mental disorders (OR, 26.2; 95% CI, 6.1-112.0), and Army career characteristics (OR for junior enlisted rank, 30.0; 95% CI, 3.3-272.5 and OR for senior enlisted rank, 6.7; 95% CI, 0.8-54.9). Cross-validated area under the curve was 0.78. The 10% of respondents with highest estimated risk accounted for 39.2% of subsequent SAs.

Conclusions and Relevance

Results suggest the feasibility of developing a clinically useful risk index for SA among soldiers with SI using a small number of self-report questions. If implemented, a continuous quality improvement approach should be taken to refine the structured question series.

Introduction

The Department of Veterans Affairs/Department of Defense (VA/DoD) Clinical Practice Guidelines for the Assessment and Management of Patients at Risk of Suicide are currently being revised for release in 2019. In addition, the VA recently began implementing a new Suicide Risk Identification Strategy that includes an annual 3-stage suicide risk screening process for all VA health care patients.¹ The DoD is considering related changes. This could lead to a substantial increase in the number of patients who are required to receive an in-depth suicide risk assessment. Such an assessment can be resource intensive, taking up to 2 hours per patient to complete.²

Two options exist to address possible increased demand for in-depth suicide risk assessments: hire additional clinical staff and/or develop a clinical decision support system to improve targeting of these assessments. With regard to the second option, although risk of suicidal behaviors is significantly elevated among people with suicide ideation (SI), suicidal behaviors are rare even in this high-risk segment of the population.^3,4 Structured risk factor studies for suicidal behaviors among patients with SI typically conclude that prediction accuracy, while significantly greater than chance, is insufficient to guide clinical decision making because of the low proportion of high-risk patients who experience this rare outcome.⁵ However, it might be feasible to use a multistep decision-making process in which a brief initial battery of self-report questions administered to all patients reporting SI is used to develop a risk assessment model that isolates only a relatively small proportion of patients who have elevated risk of subsequent suicidal behaviors, allowing information about future risk to be used to help determine which patients would benefit most from in-depth suicide risk assessments added to usual care.⁶

Consistent with this possibility, a recent Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS) study⁷ of self-reported suicide attempts (SAs) among soldiers with SI in a STARRS survey found that the 10% of soldiers with highest SA risk based on a composite risk index with a relatively small number of self-report survey risk factors accounted for more than 80% of all SAs among respondents with SI. However, the validity of this index is uncertain because it was based on retrospective reports in a cross-sectional survey, a common limitation in SA risk factor studies.⁸

The objective of the current report is to address this limitation by carrying out a prospective proof-of-concept study where information from a baseline survey of Army STARRS respondents is used to assess subsequent risk of administratively reported SAs. Although the survey was not designed with this purpose in mind, results could be valuable in providing a lower-bound estimate of the extent to which a brief structured question battery might be used to target in-depth suicide risk assessments by identifying patients with SI who are at high risk of subsequent SAs. The analysis was carried out in the same sample as the one used to create the retrospective risk index mentioned above, but with a focus on prospective administratively recorded SAs rather than retrospectively self-reported SAs.

Methods

Sample

The study followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline.⁹ Analysis was carried out in the Army STARRS Consolidated All Army Survey (AAS), an epidemiological study that combined data across 3 surveys:

The core AAS, a survey administered from 2011 to 2012 in a probability sample of 17 462 active Regular Army, National Guard, and Army Reserve units worldwide, excluding soldiers deployed or in basic training;
A 2012 to 2013 AAS expansion that surveyed 3987 deployed soldiers stationed in Afghanistan while in Kuwait awaiting transit to or from middeployment leave; and
The baseline STARRS Pre-Post Deployment Survey, which surveyed 8558 soldiers in 3 Brigade Combat Teams shortly before deploying to Afghanistan in 2012.

Recruitment, informed consent, and data collection procedures, described in more detail elsewhere,¹⁰ were approved by the Human Subjects Committees of the Uniformed Services University of the Health Sciences (Bethesda, Maryland) for the Henry M. Jackson Foundation (the primary grantee), the Institute for Social Research at the University of Michigan (Ann Arbor, the organization collecting the data), Harvard Medical School (Boston, Massachusetts), and University of California, San Diego (La Jolla). All study participants gave written informed consent. Analyses for the current article were conducted from March through November 2018.

Information on handling bias, including sample weighting, response rate, and sample exclusions, is reported elsewhere.¹¹ We focus on the 27 501 Regular Army respondents with known survey dates. We excluded survey respondents who were in the activated Reserve Command because the administrative records needed to learn of SAs were missing for these individuals once they deactivated.

Outcome

Prospective information about SA through December 2014 (18-45 months after survey participation) was assessed in Army/DoD administrative systems that together contain comprehensive health care encounter information. One of these, the DoD Suicide Event Report, is a standardized reporting system that defines SAs rigorously. However, this system underreports because it relies on active surveillance by local mental health professionals that sometimes does not occur, leading us to also include SAs defined by International Classification of Diseases, Ninth Revision, Clinical Modification codes E950 to E958.¹² Combining the 2 definitions is justified by analyses reported elsewhere showing that they have very similar correlates.¹² Soldiers who died by suicide but without a prior nonfatal SA were excluded from analysis based on evidence in previous research that the risk factors for suicide death are different from the risk factors for nonfatal SA.⁸

Risk Factors

We focused on AAS respondents who reported either active (Did you ever in your life have thoughts of killing yourself?) or passive (Did you ever wish you were dead or would go to sleep and never wake up?) lifetime SI in the Columbia Suicidal Severity Rating Scale.¹³ Three broad classes of risk factors were considered based on results of our previous retrospective study.⁷

History and Severity of Self-Injurious Thoughts and Behaviors

Respondents reporting lifetime SI were asked about age of onset, years since onset, past 30-day SI, persistence during the week when SI was worst, and controllability during that week (How easy was it for you to control these thoughts or push them out of your mind when you wanted to?). Respondents were also asked about lifetime suicide plans and attempts, history of nonsuicidal self-injury, and history of “tempting fate” by engaging in dangerous behaviors that might have led to death.

Mental Disorders

The survey screened for 8 Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, mental disorders: lifetime major depressive episode, generalized anxiety disorder, posttraumatic stress disorder, bipolar disorder, panic disorder, intermittent explosive disorder, substance use disorders, and 6-month attention-deficit/hyperactivity disorder. The Composite International Diagnostic Interview¹⁴ screening scales^15,16 were used to assess bipolar disorder, panic disorder, intermittent explosive disorder, and attention-deficit/hyperactivity disorder. A modified self-report Family History Screen¹⁷ was used to screen for the other disorders. Diagnoses based on the Composite International Diagnostic Interview and Family History Screen have demonstrated good concordance with diagnoses based on blinded clinical interviews.^15,16,17

Sociodemographic and Army Career Characteristics

To consider demographic differences associated with SA, we also considered several sociodemographic variables (age, sex, race/ethnicity [categories defined by Army STARRS team and self-identified by participants], and marital status) and Army career characteristics (years in service, rank [junior enlisted E1-E4, senior enlisted E5-E9, officers including Warrant officers], deployment history, and military occupational specialty [using broad categories of combat arms, combat support, and combat service support]). More details on military occupational specialty are reported elsewhere.⁷

Statistical Analysis

Prior to conducting primary analyses, 20 multiple imputation (MI) person-month samples were constructed, each containing all 810 181 person-months across all respondents between the month of survey and either the month of first subsequent administratively recorded SA, month of separation from service, month of death, or December 2014, whichever came first. Cross-tabulations were used in this MI data set to examine associations of lifetime SI as of the time of survey with subsequent SA.

To build a model assessing risk of SA given lifetime SI, we used discrete-time person-month survival analysis with a logistic link function¹⁸ in the subsample of the 20 MI samples made up of respondents who reported lifetime SI in the survey (102 231 person-months). All survival analyses were conducted in SAS statistical software (SAS Institute Inc). Case-control subsampling was used for computational efficiency including all person-months with an SA and a probability sample of 20 times as many control person-months weighted by the inverse of their probability of selection. We considered all of the categories of risk factor variables listed above (ie, history and severity of self-injurious thoughts and behaviors, mental disorders, and sociodemographic and Army career characteristics) in analyses.

Univariate risk factors that were significantly associated with SA in this MI data set were combined to generate multivariate models within each of these 3 broad risk factor categories (eTables 1-4 in the Supplement). These models were then trimmed of nonsignificant risk factors, and remaining within-category risk factors were included in a final combined model. Logits ± 2 times MI standard errors taking the weighting and clustering of the samples into consideration using the Taylor series method¹⁹ were exponentiated to produce odds ratios (ORs) and 95% confidence intervals. We adjusted for overfitting in estimating out-of-sample performance by using 10-fold cross-validation (10F-CV)²⁰ in each of the 20 MI data sets to generate a pooled receiver operating characteristic (ROC) curve.²¹ Area under the ROC curve (AUC) was calculated to evaluate model fit.

Given that some theories of risk factors for suicide hypothesize the existence of interactions among some of the risk factors we considered,²² we also used the Super Learner ensemble machine learning algorithm²³ in R statistical software (R Project for Statistical Computing) to investigate whether stable interactions existed among risk factors included in our SA risk index. Super Learner creates optimal weights to combine associations based on a library of different machine learning classifiers through model averaging. Thirty-two classifers were used in our library (eTable 5 in the Supplement). Twenty replicates of 5F-CV were used to generate an ROC curve based on the Super Learner ensemble. We used 5F-CV rather than the 10F-CV used in the logistic model because Super Learner already uses internal 10F-CV to estimate models and develop the weights combining results across classifiers. Risk strata based on inspection of the cross-validated ROC curves were created²⁴ for purposes of calculating positive predictive value (expressed as the number of observed SAs per 100 000 person-years) and sensitivity (proportion of observed SAs among people in each risk stratum).

For all statistical tests conducted, we used 2-sided hypothesis tests and an a priori .05 level of significance. Missing data, which were very uncommon, were recoded to medians for all variables other than several aspects of self-injurious behaviors (missing in the range of 4%-12% across predictors). The latter were imputed using the method of MI²⁵ with 20 MI replicates generated using SAS proc MI.²⁶

Results

The Army STARRS Consolidated AAS sample consisted of 27 501 Regular Army soldiers (86.6% male; median [interquartile range] age as of December 2014, 27 [22-34] years [range, 18-68 years]; 64.1% white non-Hispanic, 17.3% black, 11.5% Hispanic, 7.1% other; 34.4% currently married, 59.1% never married, 6.5% previously married). Most of our analyses focused on the subset of Consolidated AAS respondents who endorsed lifetime SI (14.3% [3649 respondents]; 80.5% male; median [interquartile range] age as of December 2014, 29 [25-36] years [range, 18-55 years]; 69.4% white non-Hispanic, 14.6% black, 9.0% Hispanic, 7.0% other; 65.7% currently married, 25.1% never married, 9.2% previously married).

Distribution of Prospective SAs by Baseline SI History

From the full AAS sample, a weighted 14.3% of survey respondents reported lifetime SI, 7.5% without a lifetime plan or SA, 3.8% with a lifetime plan but no SA, 0.9% with a history of unplanned SA (ie, those who endorsed a prior SA but denied prior suicide plans), and 2.1% with a history of planned SA (ie, those who endorsed both prior SA and prior suicide plans) (Table 1). Subsequent administratively recorded SAs occurred from 18 to 45 months after baseline among a weighted 0.7% of AAS respondents (220 participants). This baseline suicidality gradient was significantly associated with subsequent SAs (χ²₄ = 9.4; P < .001). Prospective SA risk was lowest among soldiers with no lifetime SI in the AAS (236.4/100 000 person-years), nonsignificantly higher among soldiers with lifetime SI but no prior plan or SA (360.0/100 000 person-years; OR, 1.4; 95% CI, 0.7-3.1), and significantly higher among soldiers with a lifetime plan but no prior SA (529.2/100 000 person-years; OR, 3.1; 95% CI, 1.2-7.8), a prior unplanned SA (616.8/100 000 person-years; OR, 4.2; 95% CI, 1.1-15.4), and a prior planned SA (1202.4/100 000 person-years; OR, 8.8; 95% CI, 4.2-18.7).

Table 1. Distribution and Associations of Survey Self-Reported Lifetime Suicidality With Subsequent SAs in 27 501 Participants^a.

Survey-Reported Lifetime Suicidality	Distribution of Survey-Reported Lifetime Suicidality (n = 27 501)		Prospective SAs
			Distribution of Prospective SAs (n = 220)		SAs/100 000 Person-Years^c	OR (95% CI)
	% (SE)	No.^b	% (SE)	No.^b	SAs/100 000 Person-Years^c	OR (95% CI)
No lifetime suicidality	85.7 (0.3)	23 852	73.9 (4.2)	155	236.4 (25.2)	1 [Reference]
Any lifetime suicidality	14.3 (0.3)	3649	26.1 (4.2)	65	536.4 (98.4)	3.0 (1.8-5.0)^d
Ideation only	7.5 (0.3)	2118	9.6 (3.0)	29	360.0 (111.6)	1.4 (0.7-3.1)
Ideation with a plan but no attempt	3.8 (0.2)	787	6.7 (2.4)	11	529.2 (204.0)	3.1 (1.2-7.8)^d
Ideation with an unplanned attempt^e	0.9 (0.1)	219	1.6 (0.8)	6	616.8 (342.0)	4.2 (1.1-15.4)^d
Ideation with a planned attempt^f	2.1 (0.2)	525	8.3 (2.3)	19	1202.4 (369.6)	8.8 (4.2-18.7)^d
F₄^g						9.4^d

Open in a new tab

Abbreviations: OR, odds ratio; SA, suicide attempt.

^{^a}

Results for all Regular Army soldiers in the Army Study to Assess Risk and Resilience in Servicemembers Consolidated All Army Survey Results reflect weighted and multiply imputed data.

^{^b}

Percentage is based on weighted data but number is unweighted.

^{^c}

Respondents with SAs were censored at the month of their first SA.

^{^d}

Significant at the .05 level, 2-sided multiply imputed adjusted test.

^{^e}

Respondents reported a lifetime SA in the survey but denied ever having a suicide plan.

^{^f}

Respondents reported a lifetime SA in the survey with a suicide plan.

^{^g}

F test to evaluate the joint significance of the association between survey lifetime suicidality and subsequent administratively recorded nonfatal SAs.

Risk Factors Associated With Administratively Recorded SAs Among Soldiers With Prior Lifetime SI

Further analysis focused on the 14.3% of Regular Army AAS respondents with lifetime SI (3649 participants), who accounted for 26.1% of subsequent administratively recorded definite or probable SAs (65 respondents made 77 SAs consisting of 22 poisonings, 16 alcohol or drug related, 12 blunt or sharp objects, 8 firearm related, 12 other methods, and 7 missing method). Four other aspects of baseline suicidality (eTable 1 and eTable 2 in the Supplement) were significantly associated with SA in univariate survival models: SI recency in the 30 days before survey, persistence of worst-week SI, lifetime plan, and 2 or more prior lifetime SAs. Other risk factors significant at the univariate level were positive screens for 2 lifetime mental disorders (posttraumatic stress disorder and attention-deficit/hyperactivity disorder), counts for number of lifetime internalizing, externalizing, and total disorder screens (eTable 3 in the Supplement), and 3 highly intercorrelated measures of sociodemographic and career characteristics (young age, short time in service, and low enlisted rank) (eTable 4 in the Supplement). Four of these 8 significant risk factors remained significant in a final multivariate logistic model: 30-day SI (OR, 7.2; 95% CI, 2.9-18.0), worst-week SI persistence (OR, 2.6; 95% CI, 1.0-6.8), positive screens for 2 or more lifetime mental disorders (OR, 26.2; 95% CI, 6.1-112.0), and enlisted rank (OR for junior enlisted, 30.0; 95% CI, 3.3-272.5 and OR for senior enlisted, 6.7; 95% CI, 0.8-54.9) (Table 2). The cross-validated AUC of the final logistic model assessing subsequent risk of SA was 0.78. The cross-validated AUC of the Super Learner ensemble model using the same risk factors was 0.73, indicating that allowing for interactions did not improve assessment of SA risk compared with the main effects model.

Table 2. Associations of Survey Variables With Subsequent Suicide Attempts Given Survey-Reported Lifetime Suicide Ideation in 3649 Participants^a.

Variable	Distribution of Survey Variables^b		Best-Fitting Multivariate Logistic Model, OR (95% CI)^c
Variable	% (SE)	No.	Best-Fitting Multivariate Logistic Model, OR (95% CI)^c
History of suicidal thoughts and behaviors
Ideation onset at age 15-17 y vs ≤14 y	20.2 (1.0)	721	1.8 (0.6-5.5)
Ideation onset at age ≥18 y vs ≤14 y	46.2 (1.3)	1632	2.4 (0.6-10.2)
F₂^d			0.9
F₁^e			0.2
Time since onset of ideation, mean (SD), y^f	7.6 (3.0)	NA	1.1 (0.9-1.3)
Active ideation vs passive	79.0 (1.1)	2927	1.1 (0.2-4.8)
Ideation within past 30 d	10.5 (0.8)	426	7.2 (2.9-18.0)^g
Mental disorders
Count ≥2 vs 0 or 1	78.6 (1.4)	2675	26.2 (6.1-112.0)^g
Worst-week ideation persistence
7 d and/or ≥9 h/d vs neither	29.0 (1.0)	1054	2.6 (1.0-6.8)^g
Sociodemographic and Army career variables
Rank: junior vs officer	45.7 (1.5)	1894	30.0 (3.3-272.5)^g
Rank: senior vs officer	38.1 (1.6)	1335	6.7 (0.8-54.9)
F₂^d			7.0^g
F₁^e			9.2^g

Open in a new tab

Abbreviation: OR, odds ratio.

^{^a}

Results for Regular Army soldiers in the Army Study to Assess Risk and Resilience in Servicemembers Consolidated All Army Survey who self-reported lifetime suicide ideation. Results reflect weighted and multiply imputed data.

^{^b}

Percentage is based on weighted data but number is unweighted.

^{^c}

The best-fitting multivariate model dropped nonsignificant survey variables from prior models for each category of such variables (see eTables 1-4 in the Supplement) other than controls for ideation age at onset, years since ideation age at onset, and active (vs passive) ideation.

^{^d}

F test to evaluate the joint significance of categorical variable levels.

^{^e}

F test to evaluate whether the ORs for this categorical variable are significantly different from each other.

^{^f}

Time since onset of suicide ideation ranged from 1 to 11 years, with values top coded at 11 (ie, if a participant reported ideation onset 15 years ago, they were set to 11).

^{^g}

Significant at the .05 level, 2-sided multiply imputed adjusted test.

Model Performance

Risk strata were collapsed based on inspection of cross-validated ROC curves (Figure). The proportion of observed SAs among the 5% and 10% of AAS respondents in the highest-risk strata (sensitivity) was much higher than the 5% to 10% expected by chance (33.5%-39.2%) (Table 3). Suicide attempts occurred at a rate of 3962.4/100 000 person-years in the highest-risk ventile (positive predictive value). The 30% of soldiers with lowest risk accounted for 7.3% of SAs.

Table 3. Associations of 10-Fold Cross-Validated Risk Strata With Subsequent Suicide Attempts Given Survey-Reported Lifetime Suicidality in 3649 Participants^a.

Percentile	Sensitivity, % (SE)^b	PPV, No. (SE)^c	Alternative Contrasts Among Risk Strata, OR (95% CI)
Percentile	Sensitivity, % (SE)^b	PPV, No. (SE)^c	5 Strata	3 Strata
Top 10%	39.2 (8.2)	2244.4 (618.8)		9.4 (4.5-19.4)^d
Top 5%	33.5 (7.8)	3.692.4 (1071.3)	32.9 (10.4-103.8)^d
6%-10%	5.7 (4.4)	633.6 (514.6)	5.11 (0.7-37.9)
11%-30%	28.7 (8.9)	824.5 (306.1)	6.8 (1.8-26.6)^d	3.4 (1.3-8.8)^d
Bottom 70%	32.1 (8.0)	239.7 (67.3)		1 [Reference]
31%-70%	24.8 (7.6)	338.7 (107.7)	2.8 (0.8-10.3)
71%-100%	7.3 (4.1)	120.3 (70.1)	1.0 [Reference]
F_4/2^e			16.0^d	18.0^d

Open in a new tab

Abbreviations: OR, odds ratio; PPV, positive predictive value.

^{^a}

Results for Regular Army soldiers in the Army Study to Assess Risk and Resilience in Servicemembers Consolidated All Army Survey who self-reported lifetime suicide ideation. Estimates are pooled across 20 multiply imputed replicate data sets, each using 10-fold cross-validation.

^{^b}

Sensitivity is the proportion of all suicide attempts occurring among respondents in the risk stratum represented in the row.

^{^c}

Positive predictive value is the number of suicide attempts occurring per 100 000 person-years among respondents in the risk stratum represented in the row.

^{^d}

Significant at the .05 level, 2-sided multiply imputed adjusted test.

^{^e}

F test to evaluate the joint significance of categorical variable levels for alternative contrasts. The 4 df F test evaluates the significance of the 5-strata classification; the 2 df F test evaluates the significance of the 3-strata classification.

Discussion

Consistent with previous research,^3,4,27 we found that although SI is significantly associated with SA, only a minority (26.1%) of subsequent SAs occurred among the 14.3% of soldiers who reported a lifetime history of SI. Nonetheless, a risk classification system for soldiers with lifetime SI could be useful given their elevated SA risk as long as a parallel system is developed for soldiers who deny SI.²⁸

Our cross-validated logistic regression model isolated 5% to 10% of respondents who accounted for substantially higher proportions of subsequent administratively recorded SAs given SI than expected by chance (33.5%-39.2%). The risk factors in this model were broadly consistent with those found in previous studies.^7,29,30,31 In contrast to other recent Army STARRS work,²⁸ we found no advantage to using a machine learning model that allowed for interactions over a simple logistic regression model, although meaningful interactions might have existed if our risk factor set was more comprehensive. That possibility needs to be considered if this line of investigation is pursued in the future.

A risk classification scheme with the strength of the one developed here would presumably not be of value in guiding decisions about care in the absence of a precision treatment model that found the classification scheme useful in predicting heterogeneity of treatment response because few patients screened as high risk will, in fact, make an SA, and most of those who made an SA were not classified as high risk.⁵ However, such a scheme, if improved (discussed below), could have considerable value in providing decision support for decisions regarding allocation of clinical resources. In addition, if the value of a true-positive relative to the cost of a false-positive can be specified precisely (including the cost of administering an in-depth suicide risk assessment to a patient who would not otherwise make an SA relative to the investment of that clinical effort in providing more intensive treatment for patients at high SA risk), an optimal decision threshold could be specified for such a scheme rather than using the arbitrary 5%, 10%, 30%, and 70% thresholds we used here.³² The development of this type of risk classification scheme aligns with decision tools used in other areas of medicine to weigh the relative costs and benefits of alternative clinical decisions (eg, Gage et al³³).

It is noteworthy that the out-of-sample prediction accuracy of a model like the one developed here could be considerably higher than our cross-validated AUC because some known risk factors for SA were not included in our survey, such as frequency of 30-day SI, 30-day suicide plans, and intent to act on such plans.^4,34,35 Nor did we include variables from electronic medical records, which have been shown to be associated with suicidal behaviors over and above self-report data.³⁶ Furthermore, prediction accuracy would almost certainly have increased if we had focused on a shorter and more clinically useful time horizon than the 18 to 45 months we used here. We were forced to use this long time horizon because of the small sample size. Based on these considerations, it is reasonable to assume that prediction accuracy could be improved, perhaps substantially so, if a revised version of our data collection, assessment, and prediction scheme was implemented in a continuous quality improvement framework whereby exploratory data capture methods and questions were revised across successive revisions.³⁷

Limitations

Our results should be interpreted in the context of several limitations in addition to those involving question selection and time horizon. First, a population sample rather than a clinical sample was studied. Second, respondents were given assurance that their survey responses would be confidential, a guarantee not made by the military medical system. Third, we focused only on SAs recorded in administrative records. Methodological research suggests that the vast majority of such cases are, in fact, true SAs but that medical records miss other true SAs that are incorrectly coded as accidents in addition to those that never come to medical attention.^38,39,40 These other SAs should be included if this line of investigation is pursued in the future.

Conclusions

The high cross-validated concentration of risk of our best model (ie, 39.2% of all administratively recorded SAs occurring among the 10% of soldiers at highest predicted risk) suggests that a useful risk index could be developed that assigns a predicted SA risk score to all patients who report prior SI. In making successive refinements to create an optimal index, it would be useful to consider other self-report measures, emerging biomarkers, other novel measures (eg, natural language analysis of electronic clinical notes⁴¹ or social media posts⁴²), and shorter time horizons for SA risk.

Supplement.

eTable 1. Self-Reported Risk Factors for Subsequent Administratively Recorded Suicide Attempts Involving History of Self-Injurious Thoughts and Behaviors Among Regular Army Soldiers Who Reported Lifetime Suicide Ideation in the STARRS Consolidated All Army Survey (n = 3,649)

eTable 2. Self-Reported Risk Factors of Subsequent Administratively Recorded Suicide Attempts Involving Severity of Self-Injurious Thoughts and Behaviors Among Regular Army Soldiers Who Reported Lifetime Suicide Ideation in the STARRS Consolidated All Army Survey (n = 3,649)

eTable 3. Self-Reported Risk Factors of Subsequent Administratively Recorded Suicide Attempts Involving History of Mental Disorders Among Regular Army Soldiers Who Reported Lifetime Suicide Ideation in the STARRS Consolidated All Army Survey (n = 3,649)

eTable 4. Self-Reported Risk Factors of Subsequent Administratively Recorded Suicide Attempts Involving Socio-Demographics and Army Career Characteristics Among Regular Army Soldiers Who Reported Lifetime Suicide Ideation in the STARRS Consolidated All Army Survey (n = 3,649)

eTable 5. Hyperparameter Settings for Super Learner Ensemble

Click here for additional data file.^{(154.1KB, pdf)}

References

1.Deputy Under Secretary for Health for Operations and Management Suicide Risk Screening and Assessment Requirements. Washington, DC: Department of Veterans Affairs; 2018. [Google Scholar]
2.Rudd MD. Core competencies, warning signs, and a framework for suicide risk assessment in clinical practice In: Nock MK, ed. The Oxford Handbook of Suicide and Self-injury. New York, NY: Oxford University Press; 2014:-. [Google Scholar]
3.Louzon SA, Bossarte R, McCarthy JF, Katz IR. Does suicidal ideation as measured by the PHQ-9 predict suicide among VA patients? Psychiatr Serv. 2016;67(5):517-522. doi: 10.1176/appi.ps.201500149 [DOI] [PubMed] [Google Scholar]
4.Simon GE, Coleman KJ, Rossom RC, et al. . Risk of suicide attempt and suicide death following completion of the Patient Health Questionnaire depression module in community practice. J Clin Psychiatry. 2016;77(2):221-227. doi: 10.4088/JCP.15m09776 [DOI] [PMC free article] [PubMed] [Google Scholar]
5.Carter G, Milner A, McGill K, Pirkis J, Kapur N, Spittal MJ. Predicting suicidal behaviours using clinical instruments: systematic review and meta-analysis of positive predictive values for risk scales. Br J Psychiatry. 2017;210(6):387-395. doi: 10.1192/bjp.bp.116.182717 [DOI] [PubMed] [Google Scholar]
6.Kessler RC, Bernecker SL, Bossarte RM, et al. . The role of big data analytics in predicting suicide In: Passos I, Mwangi B, Kapczinski F, eds. Personalized and Predictive Psychiatry—Big Data Analytics in Mental Health. New York, NY: Springer Nature; in press. [Google Scholar]
7.Nock MK, Millner AJ, Joiner TE, et al. . Risk factors for the transition from suicide ideation to suicide attempt: results from the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). J Abnorm Psychol. 2018;127(2):139-149. doi: 10.1037/abn0000317 [DOI] [PMC free article] [PubMed] [Google Scholar]
8.Franklin JC, Ribeiro JD, Fox KR, et al. . Risk factors for suicidal thoughts and behaviors: a meta-analysis of 50 years of research. Psychol Bull. 2017;143(2):187-232. doi: 10.1037/bul0000084 [DOI] [PubMed] [Google Scholar]
9.von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP; STROBE Initiative . The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. PLoS Med. 2007;4(10):e296. doi: 10.1371/journal.pmed.0040296 [DOI] [PMC free article] [PubMed] [Google Scholar]
10.Kessler RC, Colpe LJ, Fullerton CS, et al. . Design of the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). Int J Methods Psychiatr Res. 2013;22(4):267-275. doi: 10.1002/mpr.1401 [DOI] [PMC free article] [PubMed] [Google Scholar]
11.Kessler RC, Heeringa SG, Colpe LJ, et al. . Response bias, weighting adjustments, and design effects in the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). Int J Methods Psychiatr Res. 2013;22(4):288-302. doi: 10.1002/mpr.1399 [DOI] [PMC free article] [PubMed] [Google Scholar]
12.Ursano RJ, Kessler RC, Heeringa SG, et al. ; Army STARRS Collaborators . Nonfatal suicidal behaviors in U.S. Army administrative records, 2004-2009: results from the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). Psychiatry. 2015;78(1):1-21. [DOI] [PMC free article] [PubMed] [Google Scholar]
13.Posner K, Brown GK, Stanley B, et al. . The Columbia-Suicide Severity Rating Scale: initial validity and internal consistency findings from three multisite studies with adolescents and adults. Am J Psychiatry. 2011;168(12):1266-1277. doi: 10.1176/appi.ajp.2011.10111704 [DOI] [PMC free article] [PubMed] [Google Scholar]
14.Kessler RC, Ustün TB. The World Mental Health (WMH) Survey Initiative Version of the World Health Organization (WHO) Composite International Diagnostic Interview (CIDI). Int J Methods Psychiatr Res. 2004;13(2):93-121. doi: 10.1002/mpr.168 [DOI] [PMC free article] [PubMed] [Google Scholar]
15.Kessler RC, Santiago PN, Colpe LJ, et al. . Clinical reappraisal of the Composite International Diagnostic Interview Screening Scales (CIDI-SC) in the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). Int J Methods Psychiatr Res. 2013;22(4):303-321. doi: 10.1002/mpr.1398 [DOI] [PMC free article] [PubMed] [Google Scholar]
16.Kessler RC, Calabrese JR, Farley PA, et al. . Composite International Diagnostic Interview screening scales for DSM-IV anxiety and mood disorders. Psychol Med. 2013;43(8):1625-1637. doi: 10.1017/S0033291712002334 [DOI] [PubMed] [Google Scholar]
17.Weissman MM, Wickramaratne P, Adams P, Wolk S, Verdeli H, Olfson M. Brief screening for family psychiatric history: the family history screen. Arch Gen Psychiatry. 2000;57(7):675-682. doi: 10.1001/archpsyc.57.7.675 [DOI] [PubMed] [Google Scholar]
18.Efron B. Logistic regression, survival analysis, and the Kaplan-Meier curve. J Am Stat Assoc. 1988;83(402):414-425. doi: 10.1080/01621459.1988.10478612 [DOI] [Google Scholar]
19.Wolter KM. Introduction to Variance Estimation. New York, NY: Springer; 2007. [Google Scholar]
20.James G, Witten D, Hastie T, Tibshirani R. An Introduction to Statistical Learning: With Applications in R. Vol 103 New York, NY: Springer Texts in Statistics; 2013. doi: 10.1007/978-1-4614-7138-7 [DOI] [Google Scholar]
21.Smith GC, Seaman SR, Wood AM, Royston P, White IR. Correcting for optimistic prediction in small data sets. Am J Epidemiol. 2014;180(3):318-324. doi: 10.1093/aje/kwu140 [DOI] [PMC free article] [PubMed] [Google Scholar]
22.O’Connor RC, Nock MK. The psychology of suicidal behaviour. Lancet Psychiatry. 2014;1(1):73-85. doi: 10.1016/S2215-0366(14)70222-6 [DOI] [PubMed] [Google Scholar]
23.van der Laan MJ, Polley EC, Hubbard AE. Super learner. Stat Appl Genet Mol Biol. 2007;6(1):e25. doi: 10.2202/1544-6115.1309 [DOI] [PubMed] [Google Scholar]
24.Deeks JJ, Altman DG. Diagnostic tests 4: likelihood ratios. BMJ. 2004;329(7458):168-169. doi: 10.1136/bmj.329.7458.168 [DOI] [PMC free article] [PubMed] [Google Scholar]
25.Little RJA, Rubin DB. Statistical Analysis With Missing Data. 2nd ed New York, NY: John Wiley & Sons; 2002. doi: 10.1002/9781119013563 [DOI] [Google Scholar]
26.SAS/STATR Software [computer program]. Version 9.3 for Unix. Cary, NC: SAS Institute Inc; 2010.
27.Nock MK, Kessler RC, Franklin JC. Risk factors for suicide ideation differ from those for the transition to suicide attempt: the importance of creativity, rigor, and urgency in suicide research. Clin Psychol. 2016;23(1):31-34. doi: 10.1111/cpsp.12133 [DOI] [Google Scholar]
28.Bernecker SL, Zuromski KL, Gutierrez PM, et al. . Predicting suicide attempts among soldiers who deny suicidal ideation in the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS) [published online December 6, 2018]. Behav Res Ther. doi: 10.1016/j.brat.2018.11.018 [DOI] [PMC free article] [PubMed] [Google Scholar]
29.Nock MK, Borges G, Bromet EJ, et al. . Cross-national prevalence and risk factors for suicidal ideation, plans and attempts. Br J Psychiatry. 2008;192(2):98-105. doi: 10.1192/bjp.bp.107.040113 [DOI] [PMC free article] [PubMed] [Google Scholar]
30.Joiner TE Jr, Steer RA, Brown G, Beck AT, Pettit JW, Rudd MD. Worst-point suicidal plans: a dimension of suicidality predictive of past suicide attempts and eventual death by suicide. Behav Res Ther. 2003;41(12):1469-1480. doi: 10.1016/S0005-7967(03)00070-6 [DOI] [PubMed] [Google Scholar]
31.Beck AT, Brown GK, Steer RA, Dahlsgaard KK, Grisham JR. Suicide ideation at its worst point: a predictor of eventual suicide in psychiatric outpatients. Suicide Life Threat Behav. 1999;29(1):1-9. [PubMed] [Google Scholar]
32.Vickers AJ, Cronin AM. Traditional statistical methods for evaluating prediction models are uninformative as to clinical value: towards a decision analytic framework. Semin Oncol. 2010;37(1):31-38. doi: 10.1053/j.seminoncol.2009.12.004 [DOI] [PMC free article] [PubMed] [Google Scholar]
33.Gage BF, Waterman AD, Shannon W, Boechler M, Rich MW, Radford MJ. Validation of clinical classification schemes for predicting stroke: results from the National Registry of Atrial Fibrillation. JAMA. 2001;285(22):2864-2870. doi: 10.1001/jama.285.22.2864 [DOI] [PubMed] [Google Scholar]
34.Katz IR, Kemp JE, Blow FC, McCarthy JF, Bossarte RM. Changes in suicide rates and in mental health staffing in the Veterans Health Administration, 2005-2009. Psychiatr Serv. 2013;64(7):620-625. doi: 10.1176/appi.ps.201200253 [DOI] [PubMed] [Google Scholar]
35.Simon GE, Rutter CM, Peterson D, et al. . Does response on the PHQ-9 Depression Questionnaire predict subsequent suicide attempt or suicide death? Psychiatr Serv. 2013;64(12):1195-1202. doi: 10.1176/appi.ps.201200587 [DOI] [PMC free article] [PubMed] [Google Scholar]
36.Simon GE, Johnson E, Lawrence JM, et al. . Predicting suicide attempts and suicide deaths following outpatient visits using electronic health records. Am J Psychiatry. 2018;175(10):951-960. doi: 10.1176/appi.ajp.2018.17101167 [DOI] [PMC free article] [PubMed] [Google Scholar]
37.National Learning Consortium Continuous quality improvement (CQI) strategies to optimize your practice. https://www.healthit.gov/sites/default/files/tools/nlc_continuousqualityimprovementprimer.pdf. Published April 30, 2013. Accessed August 19, 2018.
38.Randall JR, Roos LL, Lix LM, Katz LY, Bolton JM. Emergency department and inpatient coding for self-harm and suicide attempts: validation using clinician assessment data. Int J Methods Psychiatr Res. 2017;26(3). doi: 10.1002/mpr.1559 [DOI] [PMC free article] [PubMed] [Google Scholar]
39.Kim HM, Smith EG, Stano CM, et al. . Validation of key behaviourally based mental health diagnoses in administrative data: suicide attempt, alcohol abuse, illicit drug abuse and tobacco use. BMC Health Serv Res. 2012;12:18. doi: 10.1186/1472-6963-12-18 [DOI] [PMC free article] [PubMed] [Google Scholar]
40.Barak-Corren Y, Castro VM, Javitt S, et al. . Predicting suicidal behavior from longitudinal electronic health records. Am J Psychiatry. 2017;174(2):154-162. doi: 10.1176/appi.ajp.2016.16010077 [DOI] [PubMed] [Google Scholar]
41.McCoy TH Jr, Castro VM, Roberson AM, Snapper LA, Perlis RH. Improving prediction of suicide and accidental death after discharge from general hospitals with natural language processing. JAMA Psychiatry. 2016;73(10):1064-1071. doi: 10.1001/jamapsychiatry.2016.2172 [DOI] [PMC free article] [PubMed] [Google Scholar]
42.Bryan CJ, Butner JE, Sinclair S, Bryan ABO, Hesse CM, Rose AE. Predictors of emerging suicide death among military personnel on social media networks. Suicide Life Threat Behav. 2018;48(4):413-430. doi: 10.1111/sltb.12370 [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplement.

eTable 5. Hyperparameter Settings for Super Learner Ensemble

Click here for additional data file.^{(154.1KB, pdf)}

[zoi190047r1] 1.Deputy Under Secretary for Health for Operations and Management Suicide Risk Screening and Assessment Requirements. Washington, DC: Department of Veterans Affairs; 2018. [Google Scholar]

[zoi190047r2] 2.Rudd MD. Core competencies, warning signs, and a framework for suicide risk assessment in clinical practice In: Nock MK, ed. The Oxford Handbook of Suicide and Self-injury. New York, NY: Oxford University Press; 2014:-. [Google Scholar]

[zoi190047r3] 3.Louzon SA, Bossarte R, McCarthy JF, Katz IR. Does suicidal ideation as measured by the PHQ-9 predict suicide among VA patients? Psychiatr Serv. 2016;67(5):517-522. doi: 10.1176/appi.ps.201500149 [DOI] [PubMed] [Google Scholar]

[zoi190047r4] 4.Simon GE, Coleman KJ, Rossom RC, et al. . Risk of suicide attempt and suicide death following completion of the Patient Health Questionnaire depression module in community practice. J Clin Psychiatry. 2016;77(2):221-227. doi: 10.4088/JCP.15m09776 [DOI] [PMC free article] [PubMed] [Google Scholar]

[zoi190047r5] 5.Carter G, Milner A, McGill K, Pirkis J, Kapur N, Spittal MJ. Predicting suicidal behaviours using clinical instruments: systematic review and meta-analysis of positive predictive values for risk scales. Br J Psychiatry. 2017;210(6):387-395. doi: 10.1192/bjp.bp.116.182717 [DOI] [PubMed] [Google Scholar]

[zoi190047r6] 6.Kessler RC, Bernecker SL, Bossarte RM, et al. . The role of big data analytics in predicting suicide In: Passos I, Mwangi B, Kapczinski F, eds. Personalized and Predictive Psychiatry—Big Data Analytics in Mental Health. New York, NY: Springer Nature; in press. [Google Scholar]

[zoi190047r7] 7.Nock MK, Millner AJ, Joiner TE, et al. . Risk factors for the transition from suicide ideation to suicide attempt: results from the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). J Abnorm Psychol. 2018;127(2):139-149. doi: 10.1037/abn0000317 [DOI] [PMC free article] [PubMed] [Google Scholar]

[zoi190047r8] 8.Franklin JC, Ribeiro JD, Fox KR, et al. . Risk factors for suicidal thoughts and behaviors: a meta-analysis of 50 years of research. Psychol Bull. 2017;143(2):187-232. doi: 10.1037/bul0000084 [DOI] [PubMed] [Google Scholar]

[zoi190047r9] 9.von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP; STROBE Initiative . The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. PLoS Med. 2007;4(10):e296. doi: 10.1371/journal.pmed.0040296 [DOI] [PMC free article] [PubMed] [Google Scholar]

[zoi190047r10] 10.Kessler RC, Colpe LJ, Fullerton CS, et al. . Design of the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). Int J Methods Psychiatr Res. 2013;22(4):267-275. doi: 10.1002/mpr.1401 [DOI] [PMC free article] [PubMed] [Google Scholar]

[zoi190047r11] 11.Kessler RC, Heeringa SG, Colpe LJ, et al. . Response bias, weighting adjustments, and design effects in the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). Int J Methods Psychiatr Res. 2013;22(4):288-302. doi: 10.1002/mpr.1399 [DOI] [PMC free article] [PubMed] [Google Scholar]

[zoi190047r12] 12.Ursano RJ, Kessler RC, Heeringa SG, et al. ; Army STARRS Collaborators . Nonfatal suicidal behaviors in U.S. Army administrative records, 2004-2009: results from the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). Psychiatry. 2015;78(1):1-21. [DOI] [PMC free article] [PubMed] [Google Scholar]

[zoi190047r13] 13.Posner K, Brown GK, Stanley B, et al. . The Columbia-Suicide Severity Rating Scale: initial validity and internal consistency findings from three multisite studies with adolescents and adults. Am J Psychiatry. 2011;168(12):1266-1277. doi: 10.1176/appi.ajp.2011.10111704 [DOI] [PMC free article] [PubMed] [Google Scholar]

[zoi190047r14] 14.Kessler RC, Ustün TB. The World Mental Health (WMH) Survey Initiative Version of the World Health Organization (WHO) Composite International Diagnostic Interview (CIDI). Int J Methods Psychiatr Res. 2004;13(2):93-121. doi: 10.1002/mpr.168 [DOI] [PMC free article] [PubMed] [Google Scholar]

[zoi190047r15] 15.Kessler RC, Santiago PN, Colpe LJ, et al. . Clinical reappraisal of the Composite International Diagnostic Interview Screening Scales (CIDI-SC) in the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). Int J Methods Psychiatr Res. 2013;22(4):303-321. doi: 10.1002/mpr.1398 [DOI] [PMC free article] [PubMed] [Google Scholar]

[zoi190047r16] 16.Kessler RC, Calabrese JR, Farley PA, et al. . Composite International Diagnostic Interview screening scales for DSM-IV anxiety and mood disorders. Psychol Med. 2013;43(8):1625-1637. doi: 10.1017/S0033291712002334 [DOI] [PubMed] [Google Scholar]

[zoi190047r17] 17.Weissman MM, Wickramaratne P, Adams P, Wolk S, Verdeli H, Olfson M. Brief screening for family psychiatric history: the family history screen. Arch Gen Psychiatry. 2000;57(7):675-682. doi: 10.1001/archpsyc.57.7.675 [DOI] [PubMed] [Google Scholar]

[zoi190047r18] 18.Efron B. Logistic regression, survival analysis, and the Kaplan-Meier curve. J Am Stat Assoc. 1988;83(402):414-425. doi: 10.1080/01621459.1988.10478612 [DOI] [Google Scholar]

[zoi190047r19] 19.Wolter KM. Introduction to Variance Estimation. New York, NY: Springer; 2007. [Google Scholar]

[zoi190047r20] 20.James G, Witten D, Hastie T, Tibshirani R. An Introduction to Statistical Learning: With Applications in R. Vol 103 New York, NY: Springer Texts in Statistics; 2013. doi: 10.1007/978-1-4614-7138-7 [DOI] [Google Scholar]

[zoi190047r21] 21.Smith GC, Seaman SR, Wood AM, Royston P, White IR. Correcting for optimistic prediction in small data sets. Am J Epidemiol. 2014;180(3):318-324. doi: 10.1093/aje/kwu140 [DOI] [PMC free article] [PubMed] [Google Scholar]

[zoi190047r22] 22.O’Connor RC, Nock MK. The psychology of suicidal behaviour. Lancet Psychiatry. 2014;1(1):73-85. doi: 10.1016/S2215-0366(14)70222-6 [DOI] [PubMed] [Google Scholar]

[zoi190047r23] 23.van der Laan MJ, Polley EC, Hubbard AE. Super learner. Stat Appl Genet Mol Biol. 2007;6(1):e25. doi: 10.2202/1544-6115.1309 [DOI] [PubMed] [Google Scholar]

[zoi190047r24] 24.Deeks JJ, Altman DG. Diagnostic tests 4: likelihood ratios. BMJ. 2004;329(7458):168-169. doi: 10.1136/bmj.329.7458.168 [DOI] [PMC free article] [PubMed] [Google Scholar]

[zoi190047r25] 25.Little RJA, Rubin DB. Statistical Analysis With Missing Data. 2nd ed New York, NY: John Wiley & Sons; 2002. doi: 10.1002/9781119013563 [DOI] [Google Scholar]

[zoi190047r26] 26.SAS/STATR Software [computer program]. Version 9.3 for Unix. Cary, NC: SAS Institute Inc; 2010.

[zoi190047r27] 27.Nock MK, Kessler RC, Franklin JC. Risk factors for suicide ideation differ from those for the transition to suicide attempt: the importance of creativity, rigor, and urgency in suicide research. Clin Psychol. 2016;23(1):31-34. doi: 10.1111/cpsp.12133 [DOI] [Google Scholar]

[zoi190047r28] 28.Bernecker SL, Zuromski KL, Gutierrez PM, et al. . Predicting suicide attempts among soldiers who deny suicidal ideation in the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS) [published online December 6, 2018]. Behav Res Ther. doi: 10.1016/j.brat.2018.11.018 [DOI] [PMC free article] [PubMed] [Google Scholar]

[zoi190047r29] 29.Nock MK, Borges G, Bromet EJ, et al. . Cross-national prevalence and risk factors for suicidal ideation, plans and attempts. Br J Psychiatry. 2008;192(2):98-105. doi: 10.1192/bjp.bp.107.040113 [DOI] [PMC free article] [PubMed] [Google Scholar]

[zoi190047r30] 30.Joiner TE Jr, Steer RA, Brown G, Beck AT, Pettit JW, Rudd MD. Worst-point suicidal plans: a dimension of suicidality predictive of past suicide attempts and eventual death by suicide. Behav Res Ther. 2003;41(12):1469-1480. doi: 10.1016/S0005-7967(03)00070-6 [DOI] [PubMed] [Google Scholar]

[zoi190047r31] 31.Beck AT, Brown GK, Steer RA, Dahlsgaard KK, Grisham JR. Suicide ideation at its worst point: a predictor of eventual suicide in psychiatric outpatients. Suicide Life Threat Behav. 1999;29(1):1-9. [PubMed] [Google Scholar]

[zoi190047r32] 32.Vickers AJ, Cronin AM. Traditional statistical methods for evaluating prediction models are uninformative as to clinical value: towards a decision analytic framework. Semin Oncol. 2010;37(1):31-38. doi: 10.1053/j.seminoncol.2009.12.004 [DOI] [PMC free article] [PubMed] [Google Scholar]

[zoi190047r33] 33.Gage BF, Waterman AD, Shannon W, Boechler M, Rich MW, Radford MJ. Validation of clinical classification schemes for predicting stroke: results from the National Registry of Atrial Fibrillation. JAMA. 2001;285(22):2864-2870. doi: 10.1001/jama.285.22.2864 [DOI] [PubMed] [Google Scholar]

[zoi190047r34] 34.Katz IR, Kemp JE, Blow FC, McCarthy JF, Bossarte RM. Changes in suicide rates and in mental health staffing in the Veterans Health Administration, 2005-2009. Psychiatr Serv. 2013;64(7):620-625. doi: 10.1176/appi.ps.201200253 [DOI] [PubMed] [Google Scholar]

[zoi190047r35] 35.Simon GE, Rutter CM, Peterson D, et al. . Does response on the PHQ-9 Depression Questionnaire predict subsequent suicide attempt or suicide death? Psychiatr Serv. 2013;64(12):1195-1202. doi: 10.1176/appi.ps.201200587 [DOI] [PMC free article] [PubMed] [Google Scholar]

[zoi190047r36] 36.Simon GE, Johnson E, Lawrence JM, et al. . Predicting suicide attempts and suicide deaths following outpatient visits using electronic health records. Am J Psychiatry. 2018;175(10):951-960. doi: 10.1176/appi.ajp.2018.17101167 [DOI] [PMC free article] [PubMed] [Google Scholar]

[zoi190047r37] 37.National Learning Consortium Continuous quality improvement (CQI) strategies to optimize your practice. https://www.healthit.gov/sites/default/files/tools/nlc_continuousqualityimprovementprimer.pdf. Published April 30, 2013. Accessed August 19, 2018.

[zoi190047r38] 38.Randall JR, Roos LL, Lix LM, Katz LY, Bolton JM. Emergency department and inpatient coding for self-harm and suicide attempts: validation using clinician assessment data. Int J Methods Psychiatr Res. 2017;26(3). doi: 10.1002/mpr.1559 [DOI] [PMC free article] [PubMed] [Google Scholar]

[zoi190047r39] 39.Kim HM, Smith EG, Stano CM, et al. . Validation of key behaviourally based mental health diagnoses in administrative data: suicide attempt, alcohol abuse, illicit drug abuse and tobacco use. BMC Health Serv Res. 2012;12:18. doi: 10.1186/1472-6963-12-18 [DOI] [PMC free article] [PubMed] [Google Scholar]

[zoi190047r40] 40.Barak-Corren Y, Castro VM, Javitt S, et al. . Predicting suicidal behavior from longitudinal electronic health records. Am J Psychiatry. 2017;174(2):154-162. doi: 10.1176/appi.ajp.2016.16010077 [DOI] [PubMed] [Google Scholar]

[zoi190047r41] 41.McCoy TH Jr, Castro VM, Roberson AM, Snapper LA, Perlis RH. Improving prediction of suicide and accidental death after discharge from general hospitals with natural language processing. JAMA Psychiatry. 2016;73(10):1064-1071. doi: 10.1001/jamapsychiatry.2016.2172 [DOI] [PMC free article] [PubMed] [Google Scholar]

[zoi190047r42] 42.Bryan CJ, Butner JE, Sinclair S, Bryan ABO, Hesse CM, Rose AE. Predictors of emerging suicide death among military personnel on social media networks. Suicide Life Threat Behav. 2018;48(4):413-430. doi: 10.1111/sltb.12370 [DOI] [PubMed] [Google Scholar]

PERMALINK

Assessment of a Risk Index for Suicide Attempts Among US Army Soldiers With Suicide Ideation

Kelly L Zuromski, PhD

Samantha L Bernecker, PhD

Peter M Gutierrez, PhD

Thomas E Joiner, PhD

Andrew J King, MS

Howard Liu, SD

James A Naifeh, PhD

Matthew K Nock, PhD

Nancy A Sampson, BA

Alan M Zaslavsky, PhD

Murray B Stein, MD, MPH

Robert J Ursano, MD

Ronald C Kessler, PhD

Key Points

Question

Findings

Meaning

Abstract

Importance

Objective

Design, Setting, and Participants

Main Outcomes and Measures

Results

Conclusions and Relevance

Introduction

Methods

Sample

Outcome

Risk Factors

History and Severity of Self-Injurious Thoughts and Behaviors

Mental Disorders

Sociodemographic and Army Career Characteristics

Statistical Analysis

Results

Distribution of Prospective SAs by Baseline SI History

Table 1. Distribution and Associations of Survey Self-Reported Lifetime Suicidality With Subsequent SAs in 27 501 Participantsa.

Risk Factors Associated With Administratively Recorded SAs Among Soldiers With Prior Lifetime SI

Table 2. Associations of Survey Variables With Subsequent Suicide Attempts Given Survey-Reported Lifetime Suicide Ideation in 3649 Participantsa.

Model Performance

Figure. Receiver Operating Characteristic Curve of 10-Fold Cross-Validated Logistic Model Predicting Subsequent Suicide Attempts Given Survey-Reported Lifetime Suicide Ideation in 3649 Participants.

Table 3. Associations of 10-Fold Cross-Validated Risk Strata With Subsequent Suicide Attempts Given Survey-Reported Lifetime Suicidality in 3649 Participantsa.

Discussion

Limitations

Conclusions

References

Associated Data

Supplementary Materials

ACTIONS

PERMALINK

RESOURCES

Similar articles

Cited by other articles

Links to NCBI Databases

Table 1. Distribution and Associations of Survey Self-Reported Lifetime Suicidality With Subsequent SAs in 27 501 Participants^a.

Table 2. Associations of Survey Variables With Subsequent Suicide Attempts Given Survey-Reported Lifetime Suicide Ideation in 3649 Participants^a.

Table 3. Associations of 10-Fold Cross-Validated Risk Strata With Subsequent Suicide Attempts Given Survey-Reported Lifetime Suicidality in 3649 Participants^a.