Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2019 Aug 1.
Published in final edited form as: Psychol Serv. 2018 Aug;15(3):270–278. doi: 10.1037/ser0000236

Predictive Utility of an Emergency Department Decision Support Tool in Patients with Active Suicidal Ideation

Edwin D Boudreaux 1, Celine Larkin 2, Nisha Kini 3, Lisa Capoccia 4, Michael H Allen 5, Julie Goldstein Grumet 6, Morton M Silverman 7, Richard McKeon 8, Bruce Barton 9, Ivan Miller 10, Scott W Formica 11, Carlos A Camargo Jr 12
PMCID: PMC6082179  NIHMSID: NIHMS968489  PMID: 30080084

Abstract

Emergency department (ED) clinicians routinely decide the disposition of patients with suicidal ideation, with potential consequences for patient safety, liability, system costs and resources. An expert consensus panel recently created a six-item decision support tool for patients with passive or active suicidal ideation. Individuals scoring a zero (exhibiting none of the tool’s six items) are considered “lower risk” and suitable for discharge, while those with non-zero scores are considered “elevated risk” and should receive further evaluation. The current study tested the predictive utility of this tool using existing data from the Emergency Department Safety Assessment and Follow-up Evaluation. ED patients with active suicide ideation (n=1368) were followed for 12 months after an index visit using telephone assessment and medical chart review. About one-in-five patients had attempted suicide during follow-up. Because of the frequency of serious warning signs and risk factors in this population, only three patients met tool criteria for “lower risk” at baseline. The tool had perfect sensitivity, but exceptionally low specificity, in predicting suicidal behavior within six weeks and 12 months. In logistic regression analyses, several tool items were significantly associated with suicidal behavior within six weeks (suicide plan, past attempt) and 12 months (suicide plan, past attempt, suicide intent, significant mental health condition, irritability/agitation/aggression). Although the tool did not perform well as a binary instrument among those with active suicidal ideation, having a suicide plan identified almost all attempters while suicide plan and past attempt identified over four-fifths of near-term attempts.

Keywords: Emergency department, suicide, decision support, prediction


The prevalence of suicidal ideation is about one in ten non-psychiatric emergency department (ED) patients (Allen et al., 2013; Claassen & Larkin, 2005; Ilgen et al., 2009). Recent studies suggest that implementing universal screening can improve detection of this hidden risk (Boudreaux et al., 2016). However, there is significant concern about the clinical workload associated with screening larger populations.

In 2015, the Substance Abuse and Mental Health Services Administration (SAMHSA) and the Suicide Prevention Resource Center (SPRC) used a RAND consensus methodology to create a guide on treating suicide risk in EDs (SPRC, 2015a). The guideline contained a decision support tool intended to be used with patients who already had screening or clinical results suggesting the presence of passive (thoughts of being better off dead) or active (thoughts of killing oneself) suicidal ideation. It was intended to help stratify those who could be considered “lower risk” and prioritized for discharge from those at “elevated risk” and needing further evaluation by a mental health clinician. The decision support tool was created by locating 14 existing suicide risk assessment tools and guides, generating a list of potential items, and using iterative expert review to refine the list based on clinical usefulness, acuity, feasibility, objectivity and applicability (SPRC, 2015b). The final decision support tool consists of six binary risk criteria: having a suicide plan, suicidal intent, a past suicide attempt, a significant mental health condition, a substance use disorder, and irritability, agitation, or aggression. The expert panel defined a “low-risk patient” as someone who has passive or active suicidal ideation but has none of the six risk criteria. This was based on the consensus that suicidal ideation and any of the six major warning signs or risk factors required at least further assessment whereas their absence would be consistent with a lower risk of a subsequent attempt. The goal was negative prediction in an effort to ameliorate any clinical burden associated with universal screening. However, the decision support tool has never been studied or used with a clinical population. Therefore, its utility in classifying individuals to assist with decision making or in predicting suicide-related outcomes is not known.

The current study is a secondary analysis of existing data that tested the predictive utility of the six criteria comprising the tool in patients recruited as part of the Emergency Department Safety Assessment and Follow-up Evaluation (ED-SAFE) (Arias et al., 2016; Boudreaux et al., 2016; Boudreaux et al., 2013), a study examining suicide risk screening and prevention in the ED setting. ED-SAFE did not include the actual SPRC/SAMHSA Decision Support Tool prospectively because the ED-SAFE pre-dated the tool’s development. However, the ED-SAFE did collect data that closely aligned with the six SPRC/SAMHSA risk criteria in a large sample of ED patients with active suicidal ideation and followed them for 12 months to track their suicidal behavior. Consequently, ED-SAFE data provide an opportunity to retrospectively study the utility of risk criteria similar to the SPRC/SAMHSA Decision Support Tool. The aim of the current analyses was to examine the relationship between the tool’s criteria and future suicidal behavior.

Methods

Detailed descriptions of the ED-SAFE study design, setting, participants, procedures, data collection, human subjects’ protections, and adverse event reporting have been published previously (Boudreaux et al., 2016; Boudreaux et al., 2013). Although the SPRC/SAMHSA Decision Support Tool was designed for use with patients with passive or active suicidal ideation, and particularly with EDs using universal screening (hence detecting incidental or hidden suicidal risk), the current analyses focus on a sample of ED patients with active ideation only, because this was an inclusion criterion for the ED-SAFE study. ED SAFE’s primary outcome was suicide attempt or completed suicide: the sample was selected for risk of attempt and in the usual care phase, 23% of participants made a suicide attempt.

Setting

Participants in the ED-SAFE were recruited in eight general EDs across seven U.S. states. Participating EDs ranged from small community hospitals to large academic centers. Annual census ranged from 27,145 to 54,075, and all but one of the sites were teaching hospitals. Data were collected from August 2, 2010 through November 8, 2013.

Design

The ED-SAFE study, described in detail by Boudreaux et al. (Boudreaux et al., 2016; 2013; Miller et al., 2017), used a three-phase, interrupted time series design to assess the feasibility and effectiveness of screening all adult ED patients for suicide risk and to test several interventions to reduce suicidal behavior among people who screen positive for suicide risk. Several elements of optimized screening and counselling were put in place over the three study phases. Phase 1 formed a baseline period with treatment-as-usual, wherein patients were treated according to the usual and customary care at each site, serving as the control for the subsequent study phases. In Phase 2, the Patient Safety Screener-3 (PSS-3) was used for all ED patients, regardless of their presenting complaint. Phase 3 involved providing outpatient suicide prevention resources at discharge and a series of post-discharge telephone advising calls to eligible patients, namely those who endorsed thoughts of killing oneself in the past week or an actual, aborted, or interrupted attempt to kill oneself in the past week including the current visit.

All participants enrolled into ED-SAFE were followed for one year after their index ED visit using telephone assessments and medical chart reviews. Trained blinded interviewers (distinct from those delivering the post-discharge advising phone-calls) at a centralized call center conducted outcome assessments at 6, 12, 24, 36, and 52 weeks. If acute suicide risk was detected, the participant was immediately connected to the Boy’s Town Suicide Prevention Hotline. Additionally, using a standardized form, trained chart abstractors at each site conducted chart reviews six and 12 months after enrollment. Institutional review boards at each site approved the study. All participants gave informed consent. A Data Safety Monitoring Board organized by the NIMH oversaw and monitored the study.

Participants

Throughout the study phases, adult ED patients with any level of self-harm behavior or ideation were identified via “real time” medical record review by research staff and approached for an eligibility interview. If the patient confirmed either active suicidal ideation or a suicide attempt within the past week and agreed to complete follow-up assessments, the patient was considered for enrollment. Patients who endorsed passive ideation only were not eligible. Exclusion criteria included 1) being medically or cognitively unable to participate in the assessment or counseling, 2) currently dwelling in a non-community setting, 3) currently in state custody or with pending legal action, 4) having no permanent residence or reliable telephone service, 5) having an insurmountable language barrier, and 6) previous ED-SAFE enrollment. A total of 1636 patients met inclusion criteria, of whom 1376 participants were enrolled and consented to take part. Of the enrolled participants, 1089 patients completed at least one telephone interview during the 52-week study period. Medical charts were reviewed for all participants.

Measures

The SPRC/SAMHSA Decision Support Tool consists of six dichotomous (present/absent) items, namely thoughts of carrying out a suicide plan, suicide intent, past suicide attempt, significant mental health condition, substance use problem, and irritability/agitation/aggression. Because the ED-SAFE did not use the actual decision support tool, the six risk indicators comprising the tool were carefully mapped to risk criteria that were collected as part of the ED-SAFE’s baseline assessment and which were conceptually similar or identical. Table 1 contains a map from the SPRC/SAMHSA Decision Support Tool criteria to the specific variables from the ED-SAFE. All variables that were relevant from the baseline assessment and medical record review were used. The primary outcome was fatal or non-fatal suicide attempt. An event was defined as a suicide attempt if it was a potentially self-injurious behavior with intent to die. Fatal and non-fatal suicide attempts were detected using data from the telephone interviews, medical record reviews, and vital statistics registry reviews over the 12 months after the index ED visit during which the patient was enrolled.

Table 1.

Map of Operationalizing SPRC/SAMHSA Decision Tool Items to ED-SAFE Data Items

SPRC/SAMHSA item ED-SAFE data item(s)
1. Thoughts of carrying out a plan
Recently, have you been thinking about how you might kill yourself? Yes to one of:
- ‘At any time during the past week, including today, have you thought about HOW you might do this [kill yourself]?’
- ‘At any time during the past week, including today, have you started to work out, or actually worked out, the details of how to kill yourself?’

2. Suicide intent
Do you have any intention of killing yourself? Yes to:
- ‘At any time during the past week, including today, have you had any intention of acting on these thoughts of killing yourself?

3. Past suicide attempt?
Have you ever tried to kill yourself? Yes to:
‘At any time in your life, including today, have you made a suicide attempt?

4. Significant mental health conditions
Have you had treatment for emotional problems?
Do you have a mental health issue that affects your ability to do things in life?
Yes to any one of:
- Diagnosed with (List: Depression; Bipolar disorder; An anxiety disorder; Attention Deficit Disorder (ADD/ADHD); An eating disorder, like anorexia or bulimia; Schizophrenia or schizoaffective disorder; Any other psychiatric disorder
- ‘Are you currently taking ANY medication(s) for an emotional or psychological problem?’
- ‘Have you ever been hospitalized for a psychological or emotional problem?’
- ‘Did you stay overnight in the hospital because of psychiatric or other mental health problems, including suicidal thoughts or actions?’

5. Substance use problems
Has drinking or drug use ever been a problem for you?
Or, administer CAGE or another standardized substance use disorder screener
Positive response to any one of:
- ‘Which of these drugs, if any, have you used (not for medical purposes? (List Marijuana, Painkillers, Cocaine, Tranquilizers or sedatives, Hallucinogens, Stimulants, Ecstasy, Heroin, Cold or cough medicines)
- Diagnosed with ‘An alcohol use disorder, like alcohol abuse or dependence?’
- Diagnosed with ‘Any drug use disorder, like drug abuse or dependence?’
- A score of 8 or over for patients under 65 years of age or a score 7 or more for patients over 65 years on the first three items of the AUDIT (Babor et al., 2001):
(1) “How often do you have a drink containing alcohol?” Never=0; Monthly or less=1; 2 to 4 times a month=2; 2 to 3 times a week=3; 4 or more times a week=4 PLUS
(2) “How many drinks containing alcohol do you have on a typical day when you are drinking?” 1 or 2= 0; 3 or 4= 1; 5 or 6=2; 7, 8, or 9= 3; 10 or more= 4 PLUS
(3) “How often do you have four or more drinks on one occasion?” Never= 0; Less than monthly= 1; Monthly=2; Weekly= 3; Daily or almost daily=4

6. Irritability/agitation/aggression
(Recently, have you felt so anxious or agitated that you could just jump out of your skin? Have you been having conflicts or getting into fights with other people?) Yes to one of:
- ‘Feeling so restless you couldn’t sit still’
- Diagnosed with ‘An anxiety disorder?’

Any individual who screened positive in any one of the questions above will be defined as meeting the “elevated risk” criteria.

Analysis

Baseline assessment data from all three phases of ED-SAFE project were used to calculate the proportion of patients who screened positive for active suicidal ideation that met the “lower risk” criteria proposed by SPRC/SAMHSA, defined as being negative on all six indicators. The proportion of patients with active suicidal ideation who exhibited suicidal behavior, defined as a suicide attempt or death by suicide in the 12 months following the ED visit, were categorized by whether or not they met the “lower risk” criteria. Analyses included separate examination of the predictive utility of the individual risk indicators. The individual tool items that predicted suicidal behavior outcome at p<0.10 in chi square analyses were entered into a multivariable logistic regression model, along with age (continuous), sex, phase, and site. We tested individual items for correlation and multicollinearity before adding them to the logistic regression model. The six predictors of suicidal behavior outcome showed low correlation (Phi correlation coefficient <0.28), and tests for multicollinearity yielded low condition index (<13.5), low variance inflation (<1.2), and high tolerance (>0.85) indicating low evidence for multicollinearity. These analyses were conducted using SAS software (SAS Institute Inc.).

In addition, a decision tree analysis was completed using R software to help guide the development of decision rules using the fewest number of indicators possible. Individual tool items with the highest predictive value for outcome as determined by the odds ratio and p-value were included in the decision tree until no significant splits remained or the sample size was too small to justify a split. The results are summarized into a tree-like structure to illustrate the significant branching based on the subgrouping of the indicators. Probability for suicidal behavior outcome for each node are reported.

Results

The ED-SAFE study enrolled 1,376 individuals. Of these, 1,368 individuals (99.4%) presented with active suicidal ideation within one week of the visit, including the day of presentation, and are included in these analyses. Table 2 presents a summary of the descriptive characteristics of the sample, the study distribution of the “low” versus “elevated” risk according to the SPRC/SAMHSA tool, as well as the six individual risk indicators.

Table 2.

SPRC/SAMHSA Decision Support Tool, Descriptive Statistics

Sample Characteristics n (%)
Age (Mean, SD) 37.5 (13.2)
Female 764 (55.9%)
Met decision tool criteria for “low risk” 3 (0.2%)
Met criteria for suicidal behavior outcome at 6 weeks 109 (8.0%)
Met criteria for suicidal behavior outcome at 12 months 286 (20.9%)
Thoughts of carrying out a suicide plan 1171 (85.8%)
Suicide intent 947 (69.5%)
Past suicide attempt 979 (71.6%)
Presenting with current suicide attempt 347 (25.4%)
Significant mental health condition 1242 (90.8%)
Substance use problem 895 (65.4%)
Irritability/agitation/aggression 1231 (90.0%)
Phase 1: Treatment as Usual 491 (35.9%)
Phase 2: Screening Only 377 (27.6%)
Phase 3: Intervention 500 (36.6%)

Categorizing patients by risk level

Only three (<1%) participants met criteria for “lower risk” according to the SPRC/SAMHSA tool (namely, had none of the six indicators). Averaged across all study phases, 1,171 (86%) had thoughts of carrying out a suicide plan, 947 (69%) had suicidal intent, 979 (72%) had a past suicide attempt, 1,224 (89%) had a significant mental health condition, 895 (65%) had a substance abuse problem, and 1,120 (82%) exhibited irritation/agitation/aggression. When the six criteria were summed as a scale, the mean was 4.65 (SD =1.19) across all participants. The proportion of patients who met criteria for “lower risk” and “elevated risk” did not differ across different phases of the study (see supplemental table A1).

The tool’s test operating characteristics are summarized in Table 3. In predicting the suicidal behavior outcome within six weeks and 12 months, the individual risk factors had a high sensitivity and negative predictive value but low specificity and positive predictive value. When patients are divided into low-risk and elevated categories based on the decision tool criteria, the same trend continues; the specificity, however, drops to <1% at both six-week and 12-month follow-up.

Table 3.

Operating Characteristics of the “Elevated Risk” and Individual Risk Criteria on SPRC/SAMHSA Decision Support Tool

Sensitivity
(%)
Specificity
(%)
PPV
(%)
NPV
(%)
Suicidal behavior outcome at 6 weeks
 Thoughts of carrying out a suicide plan 94.5 15.0 8.8 96.9
 Suicide intent 81.7 31.5 9.4 95.2
 Past suicide attempt 86.2 29.7 9.6 96.1
 Significant mental health condition 96.3 9.7 8.5 96.8
 Substance use problem 71.6 35.1 8.7 93.5
 Irritability/agitation/aggression 93.6 10.3 8.3 94.9
 Met decision tool criteria for “elevated risk” 100.0 0.2 8.0 100.0
Suicidal behavior outcome at 12 months
 Thoughts of carrying out a suicide plan 94.1 16.4 23.0 91.2
 Suicide intent 78.9 32.9 23.7 85.5
 Past suicide attempt 81.5 31.1 23.8 86.4
 Significant mental health condition 96.5 10.7 22.2 92.1
 Substance use problem 69.2 35.6 22.1 81.4
 Irritability/agitation/aggression 94.8 11.3 22.0 89.1
 Met decision tool criteria for “elevated risk” 100.0 0.3 21.0 100.0

PPV = Positive predictive value; NPV = Negative predictive value;

Note: “Elevated risk” classification was based on having at least one of the six individual risk factors

Individual Decision Support Tool items and Suicidal Outcomes

Table 4 summarizes the bivariate relations between each individual risk factor and the suicidal behavior outcome within six weeks and 12 months of the index presentation. Having a suicidal behavior outcome within six weeks was significantly associated with having a suicide plan, suicide intent, past suicide attempt, significant mental health condition, and being female. Having a suicidal behavior outcome within 12 months was associated with having a suicide plan, suicide intent, past suicide attempt, significant mental health condition, and irritability/agitation/aggression at baseline.

Table 4.

Association between Individual SPRC/SAMHSA Decision Support Tool Criteria and Suicidal Behavior Outcome at 6 Weeks and 12 Months

Meeting criteria for suicidal behavior outcome at 6 weeks Meeting criteria for suicidal behavior outcome at 12 months

No Yes p (chi-sq) No Yes p (chi-sq)
Thoughts of carrying out a suicide plan n 1068 103 0.007 902 269 <.001
% 85.0 94.5 83.6 94.1
Suicide intent n 858 89 0.004 723 224 <.001
% 68.5 81.7 67.1 78.9
Past suicide attempt n 885 94 <.001 746 233 <.001
% 70.3 86.2 69.0 81.5
Significant mental health condition n 1137 105 0.037 966 276 <.001
% 90.3 96.3 89.3 96.5
Substance use problem n 817 78 0.160 697 198 0.128
% 64.9 71.6 64.4 69.2
Irritability/agitation/aggression n 1129 102 0.193 960 271 0.003
% 89.7 93.6 88.7 94.8
Female n 691 73 0.015 597 167 0.330
% 54.9 67.0 55.2 58.4
Age Mean 37.4 36.5 0.495 37.2 37.8 0.539
SD 13.2 12.8 13.4 12.3

p-value from t-test

Table 5 reports the full logistic regression models. Having a suicide plan (OR=2.8; 95% CI: 1.2–6.5) and having any past suicide attempt (OR=2.3; 95%CI: 1.3–4.2) were independent predictors of the suicidal behavior outcome within six weeks of presentation. The Hosmer-Lemeshow Goodness of Fit shows that our model fits the data (p-value=0.49). Significant mental health condition (OR=2.9; 95% CI:1.4–5.8), having a suicide plan (OR=2.4; 95% CI:1.4–4.2), irritability/agitation/aggression (OR=1.9; 95% CI:1.1–3.3), past suicide attempt (OR=1.6; 95%CI: 1.1–2.3;), and suicide intent (OR=1.4; 95% CI:1.0–2.0) were significant predictors of the suicidal behavior outcome at any time during the 12-month follow-up period, after adjusting for age, sex, phase, and site. The Hosmer-Lemeshow Goodness of Fit shows that our model fits the data (p-value=0.77).

Table 5.

Odds Ratios for Individual SPRC/SAMHSA Decision Support Tool Criteria Predicting Suicidal Behavior Outcome at 6 Weeks and 12 Months

Adjusted OR 95% CI p
Suicidal behavior outcome at 6 weeks
 Thoughts of carrying out a suicide plan 2.8 1.2 6.5 0.019
 Past suicide attempt 2.3 1.3 4.2 0.003
Suicidal behavior outcome at 12 months
 Thoughts of carrying out a suicide plan 2.4 1.4 4.2 0.001
 Suicide intent 1.4 1.0 2.0 0.039
 Past suicide attempt 1.6 1.1 2.3 0.009
 Significant mental health condition 2.9 1.4 5.8 0.004
 Irritability/agitation/aggression 1.9 1.1 3.3 0.030

Note: Odds Ratios are adjusted for age, sex, study site, and phase

CI= Confidence interval; OR= odds ratios

Decision Tree Analyses

Decision tree analyses for suicidal behavior outcome in the first six weeks after presentation (Figure 1) showed that the most important indicator of a suicidal behavior outcome is having a suicide plan (8.8% vs. 3.1%). Based on this subgrouping of the patient population, the next significant factor was any past suicide attempt: among individuals with a suicide plan, individuals with a past suicide attempt had a risk probability of 10.5% compared to 4.2% among those with no past suicide attempt. No other subgrouping of patients produced a significant OR for the six-week suicidal behavior outcome in the regression model to be included in the decision tree. Of the 109 participants making a suicide attempt within 6 weeks, suicide plan was endorsed by 103 (94.5%); only 6 of 109 made an attempt without endorsing a plan at the ED visit. Within the group with a plan, 90 had a prior attempt (82.6%) while 19 attempters (17.4%) were negative for both planning and prior attempt.

Figure 1.

Figure 1

Decision Tree for Suicide Risk during the first 6 weeks

Decision tree analyses for suicidal behavior outcome within 12 months of presentation (Figure 2) showed that the most important indicator is having a suicide plan (23% vs. 9%). Within those having a plan, having a past suicide attempt further predicted suicidal behavior (26% vs. 15%). Within the subgroup of patients who had both a suicide plan and a past suicide attempt, individuals with a significant mental health condition had a higher probability of suicidal behavior outcome during the 12-month follow-up (27.1% vs. 10%). Adding suicide intent to this high risk group increased the probability of suicidal behavior to 28.2%, versus 21.2% if the individual had none of these risk factors.

Figure 2.

Figure 2

Decision Tree for Suicide Risk any time during the study

Based on the decision tree analysis, the logic flow of identifying patients at the highest risk for the early (six-week) suicidal behavior outcome would be: (1) having a suicide plan; if (1) is yes, then (2) past suicide attempt. The logic flow of identifying patients at the highest risk for the 12-month suicidal behavior outcome would be: (1) having a suicide plan; if (1) is yes, then (2) past suicide attempt; if (2) is yes, then (3) significant mental health condition; if (3) is yes, then (4) suicide intent.

Discussion

Many individuals presenting to ED have some level of suicide risk (Claassen & Larkin, 2005) and a significant proportion of those who die by suicide present to an ED in the month before their death (Ahmedani et al., 2014). Implementing universal screening improves detection of suicide risk (Boudreaux et al., 2016). The potential to increase detection without the resources for further assessment and disposition has become an impediment to increased screening, and not all of these individuals are severe enough to warrant inpatient hospitalization, even among those who present with active suicidal ideation. ED clinicians, therefore, face an important predicament in deciding who may be discharged without further mental health evaluation and who should be admitted or receive some other intervention prior to discharge. These decisions have obvious personal consequences for patients (Hume & Platt, 2007) and families (Jankovic et al., 2011). It also has significant economic repercussions, considering that direct medical costs in the US for non-fatal suicidal behavior alone were estimated at over $1.5 billion in 2013 (Shepard, Gurewich, Lwin, Reed, & Silverman, 2016).

The SPRC/SAMHSA Decision Support Tool was developed to provide a simple way to assist this decision-making process by suggesting factors that could prioritize a patient for discharge from the ED, assuming there are no other clinical factors that suggest that a mental health evaluation is warranted (SPRC, 2015a). The decision tool, as originally designed, with “lower risk” being defined as any individual with active or passive ideation but none of the six tool items, did not perform well as a putative decision tool in the ED-SAFE sample of over 1300 patients with active ideation. Less than one percent of the patients with active suicidal ideation met criteria for being considered “lower risk”: nearly every single actively suicidal patient had at least one of the six indicators, thereby placing them at “elevated risk”. Most patients exhibited several indicators. Because of the high frequency of the tool’s warning signs and risk factors within this population of actively suicidal individuals, the tool could not identify a substantial portion of patients as “lower risk” in this sample of ED patients with active suicidal ideation. Therefore, the utility of the original binary scoring strategy for guiding decisions with patients endorsing active suicidal ideation appears questionable. Nevertheless, we acknowledge that the tool merits further validation, both prospectively and in samples with more heterogeneous risk (e.g., patients with passive ideation). However, one tool item identified almost 95% of attempters and two items identified 83% of attempters in the six weeks after an index presentation, while missing 17% of attempters and no deaths. These two tool items revealed much of the risk of a proximal attempt while other tool items added little. Over the course of a year, a less relevant time frame for EDs, other factors appear to be associated with distal risk. Substance use did not appear to be a useful element in either time frame.

In addition to testing the tool as a whole, we examined each of the six risk indicators independently to determine their potential value in predicting suicidal behavior. After controlling for possible confounding variables, two of the tool items predicted suicidal behavior within six weeks of the index visit, namely thoughts of carrying out a suicide plan and having a past suicide attempt. These two factors have previously been shown to be associated with increased risk of future suicidal behavior among those presenting with suicidal behavior (Larkin, Di Blasi, & Arensman, 2014). Severe anxiety and agitation (Busch, Fawcett, & Jacobs, 2003) has been related to imminent suicide in a psychiatric inpatient sample but was not predictive of six-week outcome in the current analyses. It should be noted however, that our operationalization was related to restlessness and anxiety rather than aggression or irritability, which may account for its lack of association with short-term outcomes. Identifying short-term risk factors for suicidal behavior is especially relevant for disposition decisions in healthcare settings, but such efforts are hampered by methodological issues such as long assessment windows and retrospective self-report (Glenn & Nock, 2014), as well as the difficulty in developing study designs that can control for health service intervention between baseline measurement and outcome in real-world settings. When extending the timeframe to 12 months after the ED visit, the current analyses identified three additional indicators that were significantly associated with a suicidal behavior outcome. Suicidal intent and a history of a significant mental health condition are well-established risk factors from previous research on those presenting with suicidal behavior, while the presence of irritability, agitation, or aggression has been less often examined (Larkin et al., 2014). Substance use frequently complicates the assessment of suicidal ideation in EDs so it is noteworthy that substance use did not overlap with other variables nor it was useful as a predictor, although it is correlated with a number of adverse outcomes and is an important focus of attention by itself.

This is the first empirical examination of the SPRC/SAMHSA Decision Support Tool. The results suggested that the decision tool as a whole did not perform well in predicting suicidal behavior in the short- or medium- term with this sample of ED patients with active suicidal ideation. These results require contextual interpretation. Our analysis is the first to examine this tool empirically, but we did not do so prospectively and the results should be interpreted cautiously for several reasons. First, it is possible that there was some misalignment between the data collected as part of the ED-SAFE and the decision support tool’s indicators, particularly the partial operationalization of irritability/agitation/aggression and substance use problems in the current analyses. In addition, the sample enrolled into the ED-SAFE clinical trial endorsed active suicidal ideation and could therefore have represented a higher risk group of patients than the general population of ED patients with active or passive suicidal ideation. Indeed, three-quarters of the participants in the current study reported a prior suicide attempt, an important risk factor for subsequent suicide (Carroll et al., 2014), and one-quarter of the participants were presenting because of a current suicide attempt. Further study is required to examine the utility of the tool in lower acuity patients. In addition, while every effort was made to recruit a sample representative of ED patients with active suicidal ideation, selection bias could have influenced the sample composition and had unpredictable effects on the performance of the decision support tool. For example, admitted patients may have been over-represented among those enrolled in ED-SAFE. The data used in the current analyses were collected as part of an intervention trial, so there may be some differences in characteristics or outcomes among patients recruited across the various study phases. For example, the association between baseline risk and future suicidal behavior may have been somewhat attenuated in the participants from Phase 3. However, our supplemental analyses verified that the prevalence of each of the six tool criteria did not differ significantly across study phases. Finally, our study exclusion criteria, such as dwelling in a non-community setting, state custody or with no permanent residence, may have resulted in the exclusion of some patients with a higher risk of suicide (Hwang et al., 2009; Webb et al., 2012).

Further analyses revealed important information on alternative strategies for using the decision tool to inform assessment, because several of the individual tool items did possess modest predictive utility. The decision tool would benefit from prospective research that incorporates the existing criteria and adds other risk factors in order to validate the best combination of risk factors useful for classifying risk.

In addition to the SPRC/SAMHSA Decision Support Tool, ED physicians have been adopting the Columbia-Suicide Severity Rating Scale (C-SSRS; Posner et al., 2011) as a means of stratifying patients at risk of suicide. In contrast to broader risk factors assessed by the SPRC/SAMHSA Decision Support Tool, the C-SSRS focusses on the fine-grained details of patients’ suicidal ideation and behavior. Posner and colleagues showed that adolescents with the two highest levels of ideation severity, namely intent or intent with plan, on the C-SSRS were at increased risk of a prospective suicide attempt. Another recent study (Horwitz, Czyz, & King, 2015) examined the outcomes of psychiatric emergency services patients aged 15 to 24 years: they found that the C-SSRS item on suicidal ideation improved prediction of 18-month suicide attempts, above and beyond a past suicide attempt. In the current analyses, having a suicide plan was a stronger factor than previous suicide attempt in predicting future suicidal behavior in our sample of patients with active suicidal ideation.

Conclusion

The current findings join a growing body of research suggesting that no one tool will be robust enough to accurately predict suicidal outcomes among ED patients (Quinlivan et al., 2016). Rather, tools and empirically supported risk factors can be used to complement skilled clinical evaluation as part of a comprehensive, patient-centered approach.

Supplementary Material

Supplemental Table A1

Acknowledgments

In addition to the listed authors, the Emergency Department Safety Assessment and Follow-up Evaluation (ED-SAFE) Investigators included the following: Marian E. Betz, MD, MPH (University of Colorado Hospital, Aurora); Jeffrey M. Caterino, MD (The Ohio State University Medical Center, Columbus); Brandon Gaudiano, PhD (Butler Hospital and Brown University, Providence, Rhode Island); Talmage Holmes, PhD, MPH (University of Arkansas Medical Center, Little Rock); Maura Kennedy, MD, MPH (Beth Israel Deaconess Medical Center, Boston, Massachusetts); Frank LoVecchio, DO, MPH (Maricopa Medical Center, Phoenix, Arizona); Lisa A. Uebelacker, PhD (Memorial Hospital of Rhode Island and Brown University, Providence); Lauren Weinstock, PhD (Brown University, Providence, Rhode Island); and Wesley Zeger, DO (University of Nebraska Medical Center, Omaha).

Original grant: The project was supported by Award Number U01MH088278 from the National Institute on Mental Health. Supplemental funding for current analyses were paid for by a grant from SAMHSA

Contributor Information

Edwin D. Boudreaux, University of Massachusetts Medical School, Worcester, MA

Celine Larkin, University of Massachusetts Medical School, Worcester, MA.

Nisha Kini, University of Massachusetts Medical School, Worcester, MA.

Lisa Capoccia, Education Development Center, Waltham, MA.

Michael H. Allen, University of Colorado School of Medicine, Aurora, CO

Julie Goldstein Grumet, Education Development Center, Waltham, MA.

Morton M. Silverman, Education Development Center, Waltham, MA

Richard McKeon, Substance Abuse and Mental Health Services Administration, Rockville, MD.

Bruce Barton, University of Massachusetts Medical School, Worcester, MA.

Ivan Miller, Department of Psychiatry and Human Behavior, Brown University, Butler Hospital, Providence, RI.

Scott W. Formica, Social Science Research and Evaluation, Inc., Burlington, MA

Carlos A Camargo, Jr, Massachusetts General Hospital and Harvard Medical School, Boston, MA.

References

  1. Ahmedani BK, Simon GE, Stewart C, Beck A, Waitzfelder BE, Rossom R, Solberg LI. Health care contacts in the year before suicide death. J Gen Intern Med. 2014;29(6):870–877. doi: 10.1007/s11606-014-2767-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Allen MH, Abar BW, McCormick M, Barnes DH, Haukoos J, Garmel GM, Boudreaux ED. Screening for suicidal ideation and attempts among emergency department medical patients: Instrument and results from the Psychiatric Emergency Research Collaboration. Suicide and Life-Threatening Behavior. 2013;43(3):313–323. doi: 10.1111/sltb.12018. [DOI] [PubMed] [Google Scholar]
  3. Arias SA, Miller I, Camargo CA, Jr, Sullivan AF, Goldstein AB, Allen MH, Boudreaux ED. Factors associated with suicide outcomes 12 months after screening positive for suicide risk in the emergency department. Psychiatric Services. 2016;67(2):206–213. doi: 10.1176/appi.ps.201400513. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Babor TF, Higgins-Biddle JC, Saunders JB, Monteiro MG, World Health Organization . AUDIT: The alcohol use disorders identification test: Guidelines for use in primary health care. Geneva: WHO; 2001. [Google Scholar]
  5. Boudreaux ED, Camargo CA, Jr, Arias SA, Sullivan AF, Allen MH, Goldstein AB, Miller IW. Improving suicide risk screening and detection in the emergency department. American Journal of Preventive Medicine. 2016;50(4):445–453. doi: 10.1016/j.amepre.2015.09.029. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Boudreaux ED, Miller I, Goldstein AB, Sullivan AF, Allen MH, Manton AP, Camargo CA., Jr The Emergency Department Safety Assessment and Follow-up Evaluation (ED-SAFE): Method and design considerations. Contemporary Clinical Trials. 2013;36(1):14–24. doi: 10.1016/j.cct.2013.05.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Busch KA, Fawcett J, Jacobs DG. Clinical correlates of inpatient suicide. The Journal of Clinical Psychiatry. 2003;64(1):14–19. doi: 10.4088/JCP.v64n0105. [DOI] [PubMed] [Google Scholar]
  8. Carroll R, Metcalfe C, Gunnell D. Hospital presenting self-harm and risk of fatal and nonfatal repetition: systematic review and meta-analysis. PLoS One. 2014;9(2):e89944. doi: 10.1371/journal.pone.0089944. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Claassen CA, Larkin GL. Occult suicidality in an emergency department population. British Journal of Psychiatry. 2005;186:352–353. doi: 10.1192/bjp.186.4.352. [DOI] [PubMed] [Google Scholar]
  10. Glenn CR, Nock MK. Improving the short-term prediction of suicidal behavior. American Journal of Preventive Medicine. 2014;47(3 Suppl 2):S176–180. doi: 10.1016/j.amepre.2014.06.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Horwitz AG, Czyz EK, King CA. Predicting future suicide attempts among adolescent and emerging adult psychiatric emergency patients. Journal of Clinical Child and Adolescent Psychology. 2015;44(5):751–761. doi: 10.1080/15374416.2014.910789. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Hume M, Platt S. Appropriate interventions for the prevention and management of self-harm: a qualitative exploration of service-users’ views. BMC Public Health. 2007;7(1):9. doi: 10.1186/1471-2458-7-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Hwang SW, Wilkins R, Tjepkema M, O’Campo PJ, Dunn JR. Mortality among residents of shelters, rooming houses, and hotels in Canada: 11 year follow-up study. BMJ. 2009;339:b4036. doi: 10.1136/bmj.b4036. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Ilgen MA, Walton MA, Cunningham RM, Barry KL, Chermack ST, De Chavez P, Blow FC. Recent suicidal ideation among patients in an inner city emergency department. Suicide and Life-Threatening Behavior. 2009;39(5):508–517. doi: 10.1521/suli.2009.39.5.508. [DOI] [PubMed] [Google Scholar]
  15. Jankovic J, Yeeles K, Katsakou C, Amos T, Morriss R, Rose D, Priebe S. Family caregivers’ experiences of involuntary psychiatric hospital admissions of their relatives–a qualitative study. PLoS One. 2011;6(10):e25425. doi: 10.1371/journal.pone.0025425. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Larkin C, Di Blasi Z, Arensman E. Risk factors for repetition of self-harm: a systematic review of prospective hospital-based studies. PLoS One. 2014;9(1):e84282. doi: 10.1371/journal.pone.0084282. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Miller IW, Camargo CA, Jr, Arias SA, Sullivan AF, Allen MH, Goldstein AB, Boudreaux ED. Suicide prevention in an emergency department population: The ED-SAFE study. JAMA Psychiatry. 2017 doi: 10.1001/jamapsychiatry.2017.0678. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Posner K, Brown GK, Stanley B, Brent DA, Yershova KV, Oquendo MA, Mann JJ. The Columbia–Suicide Severity Rating Scale: Initial validity and internal consistency findings from three multisite studies with adolescents and adults. The American Journal of Psychiatry. 2011;168(12):1266–1277. doi: 10.1176/appi.ajp.2011.10111704. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Quinlivan L, Cooper J, Davies L, Hawton K, Gunnell D, Kapur N. Which are the most useful scales for predicting repeat self-harm? A systematic review evaluating risk scales using measures of diagnostic accuracy. BMJ Open. 2016;6(2) doi: 10.1136/bmjopen-2015-009297. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Shepard DS, Gurewich D, Lwin AK, Reed GA, Jr, Silverman MM. Suicide and suicidal attempts in the United States: Costs and policy implications. Suicide and Life-Threatening Behavior. 2016;46(3):352–362. doi: 10.1111/sltb.12225. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. SPRC. Caring for Adult Patients with Suicide Risk: A Consensus Guide for Emergency Departments. Waltham, MA: Suicide Prevention Resource Center; 2015a. [Google Scholar]
  22. SPRC. Development of Caring for Adult Patients with Suicide Risk: A Consensus Guide for Emergency Departments– Technical report. Waltham, MA: Suicide Prevention Resource Center; 2015b. [Google Scholar]
  23. Webb RT, Shaw J, Stevens H, Mortensen PB, Appleby L, Qin P. Suicide risk among violent and sexual criminal offenders. Journal of Interpersonal Violence. 2012;27(17):3405–3424. doi: 10.1177/0886260512445387. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplemental Table A1

RESOURCES