Abstract
Background:
Validated secondary screeners are needed to stratify suicide risk among those with non-negligible risk. This study tested the predictive utility of the Emergency Department Safety Assessment and Follow-up Evaluation (ED-SAFE) Secondary Screener (ESS), one of the screeners listed by The Joint Commission’s Patient Safety Goal 15 resources as a potential secondary screener for acute care settings.
Methods:
We performed secondary analyses of data collected for the ED-SAFE study. Data were collected during an ED visit for 1,376 patients who endorsed active suicide ideation or a suicide attempt in the past week. Participants were followed for 12 months using telephone based assessments, review of health care records, and National Death Index query. We examined the predictive validity of the individual items, total score, and a scoring algorithm using the total score and critical items. Bivariable analyses, multivariable logistic regression, and test operating characteristics were calculated.
Results:
Of the 1,376 patients enrolled, most were positive for at least one indicator. Four of the indicators were significantly associated with several outcomes. Based on score and critical items, the patients were trichotomized: the three strata were associated with significantly different rates of prospective suicidal behavior, with 52% of the high-risk group engaging in suicidal behavior within 12 months.
Conclusion:
The ESS possesses adequate operating characteristics for triage purposes. We recommend validation in new samples to confirm its operating characteristics and potentially reduce its length by removing the substance and agitation items, which offered little predictive utility in this study.
Emergency departments (ED) treat two general categories of patients with suicide risk. The first and most obvious category comprises patients who present with a primary psychiatric chief complaint, for which suicidal ideation or behavior is often present and may even be the primary reason for the presentation. Approximately 4% of all ED visits are related to a psychiatric chief complaint, with 1% of all visits due specifically to suicidal ideation or a suicide attempt.1–3 The second category of ED patients with suicide risk is composed of those presenting with a non-psychiatric chief complaint but who nevertheless have recent or current suicidal ideation or a recent suicide attempt. Several ED studies enrolling patients with non-psychiatric chief complaints have found that active suicidal ideation (defined as having thoughts of killing oneself) in the week or two before the ED visit is present in about one in ten patients,4–6 a rate far exceeding general community-based point prevalence rates of approximately 4%.7 Viewed from a different angle, 36% of suicide decedents make non-psychiatric ED visits in the year prior to death.8
Historically, this suicide risk among individuals presenting with non-psychiatric complaints—referred to variously as incidental, latent, or occult risk—has rarely been detected during routine clinical care, because ED workflow tends to focus on addressing the primary presenting complaint. To improve detection of this incidental risk, suicide prevention advocates have promoted universal screening of all patients who present to the ED, regardless of chief complaint.9,10 Studies funded by the National Institute for Mental Health and others demonstrate the feasibility and effectiveness of universal screening in ED and other acute care settings to improve detection of non-negligible risk.11–12 When combined with evidence-based interventions, screening can help significantly reduce future suicide attempts.13
By design, universal screening approaches for suicide are typically highly sensitive, identify a broad spectrum of suicide risk, and seek to minimize false negatives.14 Ideally, non-negligible suicide risk is identified rapidly by a small number of primary screening questions, making it feasible in the busy ED, and those identified as at risk are further triaged or stratified with secondary screening. Not all patients with non-negligible suicide risk, as identified by primary screening, need the same level of evaluation and treatment.14 Managing a medical patient who endorses recent active suicidal ideation with no intent, plan, or behavior in the same way as a patient presenting with a current suicide attempt may result in overly restrictive care for the medical patient and misallocate scarce ED resources, such as constant observation personnel. Overtreatment can result in workflow delays that have repercussions for the entire ED, especially if the default protocol entails consulting psychiatry for any patient with any level of suicide risk. On the other hand, the medical patient’s suicide risk cannot be ignored and should receive some kind of clinical intervention. The Joint Commission9,15 and others10,14,16 have emphasized that suicidal patients should be further stratified in an efficient way using a structured, evidence based procedure, and suicide mitigation strategies should be applied accordingly, with more intense interventions, such as one-to-one constant observation, being reserved for those with high risk. While stratification has been discouraged on the basis that suicide risk occurs on a continuum and because of poor performance of existing scales,17 triage stratification is required by accrediting bodies9 and is indispensable for medical providers to inform their necessarily categorical choices, such as whether to request a full psychiatric evaluation or implement full safety precautions.
One secondary screener included in The Joint Commission’s list of evidence-informed tools is the six-item Emergency Department Safety Assessment and Follow-up Evaluation (ED-SAFE) Secondary Screener (ESS) (Figure 1). The ESS was created by the ED-SAFE team using a review of the literature and vetting with national experts in suicide risk at the time the ED-SAFE began in 2009. The screener consists of six indicators related to suicidal ideation and behavior, psychiatric history, substance use, and agitation. The indicators can be ascertained by the clinician through patient interview, health record review, collateral information (such as from family, friends, EMS, or law enforcement), or clinical observation. The ESS is designed to be used in conjunction with the Patient Safety Screener-3 (PSS-3),18 a primary suicide risk screener used to detect non-negligible risk. Those who screen positive on the PSS-3 should receive secondary screening with the ESS to further stratify risk. Despite being informed by suicide risk prediction studies, there are no published predictive validity studies specifically examining the ESS. To address this knowledge gap, the current study examined the relation between the ESS and future suicidal behavior using a secondary analysis of existing data collected as part of the ED-SAFE.11,19,20
Figure 1:
Shown are the six indicators of the Emergency Department Safety Assessment and Follow-up Evaluation Secondary Screener (ESS).
METHODS
Overview
Detailed descriptions of the ED-SAFE study design, setting, participants, procedures, data collection, human subjects’ protections, adverse event reporting, and outcomes have been published previously.11,13,19,20 Briefly, it was a hybrid type 2 implementation trial that used a three phase, interrupted time series design to (1) assess the feasibility and effectiveness of universal suicide risk screening in adult ED patients and (2) test the efficacy of a multi-component intervention to reduce suicidal behavior among people who screen positive for suicide risk. All participants enrolled into the longitudinal part of the study were followed for one year after their index ED visit using multi-method ascertainment of suicide-related ideation and behavior. Trained, supervised, blinded interviewers at a centralized research call center conducted outcome assessments at 6, 12, 24, 36, and 52 weeks. Additionally, trained chart abstractors at each site conducted health care chart reviews for the 12-month observation period, and the National Death Index was queried for deaths, including cause of death and probability of suicide. The current analyses use data from the longitudinal, clinical trial part of the study. It included all data from the baseline assessment and chart review completed for the index ED visit to ascertain the indicators making up the ESS and all sources of follow-up data to ascertain the presence of suicidal behavior longitudinally. Institutional review boards at each site approved the study. All participants gave informed consent. A NIMH-appointed Data Safety Monitoring Board oversaw and monitored the ED-SAFE.
Setting
Participants in the ED-SAFE were recruited in eight general EDs ranging from small community hospitals to large academic centers and spanning seven U.S. states. Annual patient census ranged from 27,145 to 54,075 and all but one of the sites were teaching hospitals. Data were collected from August 2010 through November 2013.
Participants
Throughout the study phases, adult ED patients with any level of self-harm ideation or behavior noted as part of their ED assessment were identified in real time by embedded research staff and further assessed to confirm the presence of either active suicidal ideation or a suicide attempt within the past week. These non-negligible risk patients were further interviewed for eligibility into the longitudinal, clinical trial portion of the study. Exclusion criteria included (1) being medically or cognitively unable to participate in the assessment or counseling, (2) currently dwelling in a non-community setting, (3) currently in state custody or with pending legal action, (4) having no permanent residence or reliable telephone service, (5) having an insurmountable language barrier, and (6) previous ED-SAFE enrollment.
Measures
Demographics.
We collected data on age, sex, ethnicity, race, education, employment, marital status, and living circumstances.
ESS.
The indicators we ascertained were (1) having both active ideation in the past week and a lifetime suicide attempt, (2) having begun a suicide plan, (3) recent or current intent to act on ideation, (4) a lifetime history of a psychiatric hospitalization, (5) a pattern of excessive substance abuse, and (6) current irritability, agitation, or aggressiveness. Data for the ESS indicators were collected by the research staff using patient interview and clinical chart review. There were two reasons we chose to use the ESS data collected by research assistants for our analyses, rather than ESS screeners documented by ED clinicians during clinical care. First, the research assistants collected data on the ESS indicators for all three phases of the study, while the ESS was only introduced clinically in the last phase. Since all participants in the third phase of the study received the intervention, the clinician-administered ESS screener data was confounded by intervention effects. Second, we had much more consistency and stronger quality control measures for research staff ascertainment of the indicators as compared with clinical staff ascertainment, and therefore have greater confidence in the quality and completeness of the data collected by research staff. Appendix 1 shows how each of the ESS indicators was operationalized for the current study.
Giving one point per indicator endorsed, the ESS can be summed to create a total score ranging from 0 to 6. The ESS contains critical items as well, including having begun a suicide plan and having recent or current suicidal intent, which can be used in conjunction with the sum score to help define risk stratification. This approach of focusing on suicide plan and intent follows a long tradition in the field of suicide risk assessment, is supported by several studies,21 and has recently become the definition of high risk, also known as serious risk, used by The Joint Commission.15 Consequently, in addition to examining the raw sum score alone, we created a stratification procedure using the raw sum score and presence of critical items as follows: “mild” means a patient obtained a sum score of 0 to 2 and has no intent or plan, “moderate” means a patient obtained a sum score of 3 to 4 or has intent or plan (not both), and “high” means a patient obtained a sum score of 5 to 6 or has both intent and plan. In the current analyses, any patient who presented to the ED with an actual suicide attempt was automatically considered as high risk.
Suicidal Behavior.
The presence of suicidal behavior across the 12-month follow-up was assigned to each participant using all sources of data available, including telephone assessments, medical records, and the National Death Index (NDI). The telephone assessment used the full version of the Columbia Suicide Severity Rating Scale (CSSRS)21 to ascertain suicide-related behaviors. We created two composite outcomes:13 “suicide attempts” was defined as fatal or non-fatal suicide attempts, and “any suicidal behavior” was defined as any suicidal behavior, including preparatory, aborted, and interrupted attempts, in addition to fatal or non-fatal attempts. We followed standard accepted definitions for these constructs.22
Analysis
We calculated descriptive statistics for all data. We created variables for each of the two outcome variables for the first 6 weeks and the entire 12 months after the index visit, resulting in a short-term (6 weeks) and long-term (12 months) event horizon for both “suicide attempt” and “any suicidal behavior” outcomes. Analyses included reliability of the ESS scale (Cronbach’s alpha), separate examination of the predictive utility of the individual indicators with a focus on teasing apart the relative importance of suicidal plan and intent, the raw sum, and the stratification using the raw sum in conjunction with critical items. Chi-square analyses were used for bivariable analyses and logistic regression for multivariable analyses. Finally, we calculated test operating characteristics, including sensitivity, specificity, positive predictive value, and negative predictive value for each of the strata, and created receiver operating characteristic (ROC) curves and calculated the area under the curve (AUC) for each outcome. All analyses were completed using SPSS, version 24.0.23
RESULTS
Description of the Sample
We enrolled 1,376 individuals. Table 1 presents the sample’s descriptive characteristics. Most patients were positive for at least one ESS indicator. The least prevalent indicator was previous psychiatric hospitalization (n = 898, 65%), and the most prevalent indicator was having begun a suicide plan (n = 1,185, 86%). The internal consistency was low: Cronbach’s alpha was 0.42.
Table 1.
Description of Sample
| n | % | ||
|---|---|---|---|
| Age (years) | 18–24 | 321 | 23.3% |
| 25–−34 | 321 | 23.3% | |
| 35–−44 | 332 | 24.1% | |
| 45–−54 | 262 | 19.0% | |
| 55–−64 | 117 | 8.5% | |
| ≥65 | 23 | 1.7% | |
| Sex | Male | 607 | 44% |
| Female | 769 | 56% | |
| Ethnicity | Hispanic | 171 | 12% |
| Non-Hispanic | 1205 | 88% | |
| Race | White | 1057 | 77% |
| Black/African American | 234 | 17% | |
| Asian | 17 | 1% | |
| Education | Less than high school | 279 | 20% |
| High school graduate | 394 | 29% | |
| Post-high school | 703 | 51% | |
| Employment | Employed full-time | 259 | 19% |
| Not employed full-time | 1117 | 81% | |
| Marital status | Single (never married) | 701 | 51% |
| Married | 259 | 19% | |
| Divorced/widowed/other | 416 | 30% | |
| Living circumstances | Living alone | 361 | 26% |
| Not living alone | 1015 | 74% | |
| Secondary screener items | Item 1: Active ideation and lifetime attempt | 986 | 72% |
| Item 2: Plan | 1185 | 86% | |
| Item 3: Intent | 971 | 71% | |
| Intent AND plan | 907 | 66% | |
| Item 4: Psych hospitalization | 898 | 65% | |
| Item 5: Excessive substance use | 977 | 71% | |
| Item 6: Aggressive/irritable | 1144 | 83% | |
| Secondary screener sum score (median) | 5.0 | IQR (4–6) | |
| Simple strata | Mild (score 0–−2) | 122 | 9% |
| Moderate (score 3–4) | 485 | 35% | |
| High (score 5–6) | 769 | 56% | |
| Final strata | Mild (score 0–2) | 59 | 4% |
| Moderate (score 3–4 or Intent or Plan) | 328 | 24% | |
| High (5–6 or “Intent AND plan”) | 989 | 72% | |
| Suicide outcomes | Suicide attempt at 6 weeks | 109 | 8% |
| Suicide attempt at 12 months | 287 | 22% | |
| Any suicidal behavior at 6 weeks | 364 | 27% | |
| Any suicidal behavior at 12 months | 638 | 46% |
Unadjusted Analyses
Tables 2 and 3 show the association between each demographic characteristic, ESS indicators, ESS sum score, various ESS stratifications, and each of the suicide behavior outcomes. Items 1 to 4 were each significantly associated with outcomes at each time point, as was endorsing intent and plan (Items 2 and 3). Item 5, substance use, was not associated with any outcome except for “any suicidal behavior” at 12 months. Item 6, irritability/aggression, was not significantly associated with outcomes at any point.
Table 2.
Unadjusted Associations Between Demographic Characteristics and Outcomes
| Suicide attempt 6 weeks (n = 109; 8%) | Suicide attempt 12 months (n = 287; 22%) | Any suicidal behavior 6 weeks (n = 364; 27%) | Any suicidal behavior 12 months (n = 638; 46%) | ||||||
|---|---|---|---|---|---|---|---|---|---|
| n (%) | Chi | n (%) | Chi | n (%) | Chi | n (%) | Chi | ||
| Age group (years) | 18–24 (n = 321) | 31 (10%) | 0.88 (n.s.) | 55 (17%) | 4.94 (n.s.) | 70 (22%) | 6.71† | 113 (35%) | 22.87‡ |
| 25–34 (n = 321) | 21 (7%) | 73 (23%) | 81 (25%) | 141 (44%) | |||||
| 35–44 (n = 332) | 28 (8%) | 73 (22%) | 91 (27%) | 166 (50%) | |||||
| 45–54 (n = 262) | 20 (8%) | 59 (23%) | 76 (29%) | 143 (55%) | |||||
| 55–64 (n = 117) | 8 (7%) | 24 (21%) | 42 (36%) | 68 (58%) | |||||
| ≥ 65 (n = 23) | 1 (4%) | 3 (13%) | 4 (17%) | 7 (30%) | |||||
| Gender | Male (n = 607) | 36 (6%) | 5.90* | 119 (20%) | 1.03 (n.s.) | 133 (22%) | 11.52† | 258 (43%) | 6.52* |
| Female (n = 769) | 73 (10%) | 168 (22%) | 231 (30%) | 380 (49%) | |||||
| Ethnicity | Hispanic (n = 171) | 12 (7%) | 0.22 (n.s.) | 23 (14%) | 6.49* | 44 (26%) | 0.05 (n.s.) | 68 (40%) | 3.42 (n.s.) |
| Non-Hispanic (n = 1205) | 97 (8%) | 264 (22%) | 320 (27%) | 570 (47%) | |||||
| Race | White (n = 1057) | 89 (8%) | 1.55 (n.s.) | 236 (22%) | 5.97* | 276 (26%) | 0.27 (n.s.) | 503 (47%) | 2.73 (n.s.) |
| Black/African American (n=234) | 20 (9%) | 0.15 (n.s.) | 45 (19%) | 0.45 (n.s.) | 76 (33%) | 5.26* | 112 (48%) | 0.25 (n.s.) | |
| Asian (n = 17) | 0 (0%) | 1.48 (n.s.) | 2 (12%) | 0.86 (n.s.) | 1 (6%) | 3.74 (n.s.) | 3 (18%) | 5.71* | |
| Education | Less than high school (n = 279) | 30 (11%) | 6.23* | 72 (26%) | 6.65* | 89 (32%) | 6.13* | 146 (52%) | 7.41* |
| High school graduate (n = 394) | 35 (9%) | 85 (22%) | 105 (27%) | 189 (48%) | |||||
| Post-high school (n = 703) | 44 (6%) | 130 (19%) | 170 (24%) | 303 (43%) | |||||
| Employment | Employed full-time (n = 259) | 10 (4%) | 7.21† | 32 (12%) | 13.97‡ | 40 (15%) | 19.88‡ | 87 (34%) | 20.94‡ |
| Not employed full-time (n = 1117) | 99 (9%) | 255 (23%) | 324 (29%) | 551 (49%) | |||||
| Marital status | Single (never married) (n = 701) | 66 (9%) | 6.11* | 149 (21%) | 0.29 (n.s.) | 177 (25%) | 1.52 (n.s.) | 300 (43%) | 7.67* |
| Married (n = 259) | 21 (8%) | 51 (20%) | 68 (26%) | 126 (49%) | |||||
| Divorced/ widowed/ other (n = 416) | 22 (5%) | 87 (21%) | 119 (29%) | 212 (51%) | |||||
| Living circumstances | Living alone (n = 361) | 26 (7%) | 0.65 (n.s.) | 80 (22%) | 0.50 (n.s.) | 103 (29%) | 1.09 (n.s.) | 182 (50%) | 3.23 (n.s.) |
| Not living alone (n = 1015) | 83 (8%) | 207 (20%) | 261 (26%) | 456 (45%) | |||||
p < 0.05;
p < 0.01;
p < 0.001;
n.s., not significant.
Table 3.
Unadjusted Associations Between Screening Characteristics and Outcomes
| Suicide attempt 6 weeks (n =109; 8%) | Suicide attempt 12 months (n = 287; 22%) | Any suicidal behavior 6 weeks (n = 364; 27%) | Any suicidal behavior 12 months (n = 638; 46%) | ||||||
|---|---|---|---|---|---|---|---|---|---|
| n (%) | Chi | n (%) | Chi | n (%) | Chi | n (%) | Chi | ||
| Secondary screener items | Item 1: Active ideation and lifetime attempt (n = 986) | 94 (10%) | 12.39‡ | 233 (24%) | 16.21‡ | 299 (30.3%) | 26.79‡ | 502 (50.9%) | 28.92‡ |
| Item 2: Plan (n = 1185) | 104 (9%) | 8.55† | 269 (23%) | 17.56‡ | 335 (28%) | 14.48‡ | 585 (49%) | 30.91‡ | |
| Item 3: Intent (n = 971) | 90 (9%) | 8.21† | 226 (23%) | 11.68† | 288 (30%) | 17.44‡ | 494 (51%) | 26.97‡ | |
| Intent AND plan (n = 907) | 88 (10%) | 11.37† | 219 (24%) | 16.98‡ | 277 (31%) | 22.24‡ | 474 (52%) | 35.97‡ | |
| Item 4: Psych hospitalization (n = 898) | 87 (10%) | 11.06‡ | 232 (26%) | 38.80‡ | 283 (32%) | 34.03‡ | 482 (54%) | 55.53‡ | |
| Item 5: Excessive substance use (n = 977) | 80 (8%) | 0.33 (n.s.) | 219 (22%) | 4.96* | 256 (26%) | 0.11 (n.s.) | 459 (47%) | 0.51 (n.s.) | |
| Item 6: Aggressive/irritable (n =1144) | 97 (9%) | 2.89 (n.s.) | 245 (21%) | 1.28 (n.s.) | 310 (27%) | 1.45 (n.s.) | 535 (47%) | 0.44 (n.s.) | |
| Sum score | 0 (n = 2) | 0 (0%) | 23.75‡ | 0 (0%) | 49.55‡ | 0 (0%) | 43.40‡ | 0 (0%) | 68.38‡ |
| 1 (n = 28) | 1 (4%) | 2 (7%) | 1 (4%) | 4 (14%) | |||||
| 2 (n = 92) | 1 (1%) | 7 (8%) | 10 (11%) | 20 (22%) | |||||
| 3 (n = 190) | 6 (3%) | 23 (12%) | 36 (19%) | 67 (35%) | |||||
| 4 (n = 295) | 19 (6%) | 47 (16%) | 68 (23%) | 128 (43%) | |||||
| 5 (n = 415) | 37 (9%) | 97 (23%) | 124 (30%) | 214 (52%) | |||||
| 6 (n = 354) | 45 (13%) | 111 (31%) | 125 (35%) | 205 (58%) | |||||
p < 0.05;
p < 0.01;
p < 0.001;
n.s., not significant.
Multivariable Analyses
Tests for multicollinearity among the six indicators yielded low condition index (< 12.14), low variance inflation (< 1.21), and high tolerance (> 0.996), indicating low collinearity. Table 4 summarizes the logistic regression results. Item 1, ideation AND lifetime attempt; Item 2, plan; and Item 4, psychiatric hospitalization, were independently associated with suicide attempt at 6 weeks. Item 2, plan, and Item 4, psychiatric hospitalization, were independently associated with the suicide attempt at 12 months. Similar patterns were noted for the “any suicidal behavior” outcome (Table 4). Appendix 2 shows that the utility of the first four items held when the regressions were limited to patients without a current attempt.
Table 4.
Logistic Regression of Secondary Screener Predictors and Suicide Outcomes
| Suicide attempt 6 weeks (n = 109; 8%) | Suicide attempt 12 months (n = 287; 22%) | Any suicidal behavior 6 weeks (n = 364; 27%) | Any suicidal behavior 12 months (n = 638; 46%) | |||||
|---|---|---|---|---|---|---|---|---|
| OR | 95% CI | OR | 95% CI | OR | 95% CI | OR | 95% CI | |
| Item 1 | 1.92 | 1.07–3.46 | 1.36 | 0.96–1.93 | 1.63 | 1.19–2.25 | 1.38 | 1.05–1.80 |
| Item 2 | 2.80 | 1.09–7.18 | 2.49 | 1.47–4.23 | 1.81 | 1.16–2.80 | 2.16 | 1.51–3.10 |
| Item 3 | 1.34 | 0.78–2.32 | 1.16 | 0.82–1.63 | 1.31 | 0.95–1.79 | 1.32 | 1.01–1.70 |
| Item 4 | 1.84 | 1.12–3.01 | 2.41 | 1.73–3.34 | 1.97 | 1.48–2.63 | 2.16 | 1.70–2.76 |
| Item 5 | 1.00 | 0.64–1.57 | 1.28 | 0.94–1.75 | 0.85 | 0.65–1.11 | 0.97 | 0.76–1.24 |
| Item 6 | 1.60 | 0.86–2.99 | 1.13 | 0.78–1.63 | 1.17 | 0.83–1.64 | 1.03 | 0.76–1.38 |
| Model | Chi: 33.08* | Chi = 69.67* | Chi = 68.03* | Chi = 100.15* | ||||
Item 1, Active ideation and lifetime attempt; Item 2, Plan; Item 3, Intent; Item 4, Psychiatric hospitalization; Item 5, Excessive substance use; Item 6, Aggressive/irritable.
p < 0.001
ROC
The ROCs are depicted in Figure 2. The AUC for each outcome was as follows: suicide attempt at 6 weeks: 0.64 (95% CI: 0.59–0.69; p < 0.001); suicide attempt at 12 months: 0.63 (95% CI: 0.60–0.67; p < 0.001); any suicidal behavior at 6 weeks: 0.61 (95% CI: 0.58–0.64; p < 0.001); and any suicidal behavior at 12 months: 0.62 (95% CI: 0.59–0.65; p < 0.001). Chi squares for trend showed significant associations between the ESS sum score and each of the suicide outcomes (Table 3).
Figure 2:
ROC curve of associations between secondary screener score and outcomes: (a) association between score and suicide attempt at 6 weeks; (b) association between score and suicide attempt at 12 months; (c) association between score and any suicidal behavior at 6 weeks; and (d) association between score and any suicidal behavior at 12 months.
Three Group Stratification: Mild, Moderate, High
Table 5 and Appendix 3 show a statistically significant step increase across mild, moderate, and high strata for both suicide outcomes at 6 weeks and 12 months. The test operating characteristics of three strata are shown in Table 6a (strata based on simple sum) and Table 6b (strata based on sum and critical items). A moderate cut-off led to high sensitivity but poor specificity, while a high cut-off led to moderate sensitivity and specificity.
Table 5.
Strata and Associations with Suicidal Behavior
| Number of patients in stratum | Suicide attempt at 6 weeks | Suicide attempt at 12 months | Any suicidal behavior at 6 weeks | Any suicidal behavior at 12 months | |
|---|---|---|---|---|---|
| Simple strata | |||||
| Mild (score 0–2) | n = 122 | 2 (2%) | 9 (7%) | 11 (9%) | 24 (20%) |
| Moderate (score 3–4) | n = 485 | 25 (5%) | 70 (14%) | 104 (21%) | 195 (40%) |
| High (score 5–6) | n = 769 | 82 (11%) | 208 (27%) | 249 (32%) | 419 (55%) |
| Chi test for trend across strata | 19.30* | 42.33* | 39.11* | 61.82* | |
| Final strata | |||||
| Mild (score 0–2) | n = 59 | 1 (2%) | 5 (9%) | 4 (7%) | 10 (17%) |
| Moderate (score 3–4 or Intent OR Plan) | n = 328 | 13 (4%) | 43 (13%) | 61 (19%) | 116 (35%) |
| High (score 5–6 or Intent AND Plan) | n = 989 | 95 (10%) | 239 (24%) | 299 (30%) | 512 (52%) |
| Chi test for trend across strata | 13.50* | 23.12* | 29.38* | 48.01* | |
p<0.001
Table 6a.
Diagnostic Values of Simple Secondary Screener Strata (Mild 0–2; Moderate 3–4; High 5–6) From Emergency Department Safety Assessment and Follow-Up Evaluation Data in Determining Prospective Suicidal Behavior Split by Cut-Off of Moderate and Cut-Off OF High
| SS High/moderate versus mild | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Sensitivity | 95% CI | Specificity | 95% CI | PPV | 95% CI | NPV | 95% CI | LR | 95%CI | |
| Suicide attempt at 6 weeks | 98% | 94–99% | 10% | 8–11% | 9% | 7–10% | 98% | 94–99% | 1.08 | 1.08–1.09 |
| Suicide attempt at 12 months | 99% | 94–98% | 10% | 9–12% | 22% | 20–25% | 93% | 87–96% | 1.08 | 1.08–1.08 |
| Any suicidal behavior at 6 weeks | 97% | 95–98% | 11% | 9–13% | 28% | 26–31% | 91% | 85–95% | 1.09 | 1.09–1.09 |
| Any suicidal behavior at 12 months | 96% | 95–98% | 13% | 11–16% | 49% | 46–52% | 80% | 72–86% | 1.11 | 1.11–1.11 |
| SS High versus mild/moderate | ||||||||||
| Sensitivity | 95% CI | Specificity | 95% CI | PPV | 95% CI | NPV | 95% CI | LR | 95%CI | |
| Suicide attempt at 6 weeks | 75% | 66–82% | 46% | 43–49% | 11% | 9–13% | 96% | 94–97 | 1.39 | 1.37–1.40 |
| Suicide attempt at 12 months | 73% | 67–77% | 49% | 46–52% | 27% | 24–30% | 87% | 84–89% | 1.41 | 1.40–1.42 |
| Any suicidal behavior at 6 weeks | 68% | 65–73% | 49% | 46–52% | 32% | 29–36% | 81% | 78–84% | 1.33 | 1.32–1.34 |
| Any suicidal behavior at 12 months | 66% | 62–69% | 53% | 49–56% | 55% | 51–58% | 64% | 60–68% | 1.39 | 1.37–1.40 |
PPV, positive predictive value; NPV, negative predictive value; LR, likelihood ratio.
Table 6b.
Diagnostic Values of Final Secondary Screener Strata (Scores Plus Critical Items) from Emergency Department Safety Assessment and Follow-Up Evaluation Data in Determining Prospective Suicidal Behavior Split by Cut-Off of Moderate and Cut-Off of High
| SS High/moderate versus mild | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Sensitivity | 95% CI | Specificity | 95% CI | PPV | 95% CI | NPV | 95% CI | LR | 95%CI | |
| Suicide attempt at 6 weeks | 99% | 95–100% | 5% | 4–6% | 8% | 8–8% | 98% | 89–100% | 1.04 | 1.02–1.06 |
| Suicide attempt at 12 months | 98% | 96–99% | 5% | 4–6% | 21% | 21–22% | 92% | 81–96% | 1.03 | 1.01–1.06 |
| Any suicidal behavior at 6 weeks | 99% | 97–99% | 5% | 4–7% | 27% | 27–28% | 93% | 83–97% | 1.05 | 1.03–1.07 |
| Any suicidal behavior at 12 months | 98% | 97–99% | 7% | 5–9% | 48% | 47–48% | 83% | 72–91% | 1.05 | 1.03–1.08 |
| SS High versus mild/moderate | ||||||||||
| Sensitivity | 95% CI | Specificity | 95% CI | PPV | 95% CI | NPV | 95% CI | LR | 95%CI | |
| Suicide attempt at 6 weeks | 87% | 79–93% | 29% | 27–32% | 10% | 9–10% | 96% | 94–98% | 1.24 | 1.14–1.34 |
| Suicide attempt at 12 months | 83% | 78–87% | 31% | 28–34% | 24% | 23–25% | 88% | 84–90% | 1.21 | 1.13–1.29 |
| Any suicidal behavior at 6 weeks | 82% | 78–86% | 32% | 29–35% | 30% | 29–32% | 83% | 80–86% | 1.20 | 1.13–1.28 |
| Any suicidal behavior at 12 months | 80% | 77–83% | 35% | 32–39% | 52% | 50–53% | 67% | 63–71% | 1.24 | 1.16–1.33 |
PPV, positive predictive value; NPV, negative predictive value; LR, likelihood ratio.
DISCUSSION
While implementing universal screening improves suicide risk detection,11,18 many ED clinicians fear it will impede workflow if mitigation plans and resources are not tailored to risk stratification. Indeed, this fear might come true if we treat all patients with some identified suicide risk with identical, conservative protocols, like being disrobed and searched, placed under constant observation, and receiving a full psychiatric evaluation. However, evidence-informed methods for triaging or stratifying patients and guiding decision making has been lacking. Using the ESS sum score and critical items to define three strata (mild, moderate, high), < 2% of patients in the mild stratum had a suicide attempt in the six weeks after the index visit, compared to > 9% of the high stratum. Similarly, in the long term, approximately 17% of the mild stratum had some type of suicidal behavior in the 12 months after their index visit compared to over 50% of high stratum patients. Interestingly, our results examining specific indicators suggest that the current emphasis on presence of plan and intent for defining high risk should be further evaluated. While plan alone, intent alone, and the combination of plan and intent all were associated with prospective suicidal behavior, two other indicators (the presence of active ideation with lifetime attempt and a past psychiatric hospitalization) were at least as strongly associated with suicidal behavior in unadjusted analyses. In the multivariable regression that included all six indicators, having a suicide plan rose to the strongest predictor across all outcomes and time points, while suicidal intent tended to be non-significant, suggesting the presence of a plan may be more important in defining high risk than having intent. Finally, the operating characteristics revealed patterns consistent with expectations for a screener with a primary purpose of early triage, namely strong sensitivity but relatively weak specificity. We would recommend initial completion of the secondary screener by the nurse followed by further suicide risk assessment by the ED physician for all three strata, with those in the high risk stratum being prioritized for full safety precautions and evaluation by a mental health clinician. For additional suggestions, one can refer to the American College of Emergency Physicians’ iCARE2 tool that elaborates on suicide risk mitigation strategies.16,24
There are other screeners that can be used for secondary screening and stratification. In recent years, the Columbia-Suicide Severity Rating Scale (C-SSRS)21 Triage version has gained attention. It contains six items that assess passive suicidal ideation, active suicidal ideation, suicide method, suicidal intent, a detailed suicide plan, and lifetime and three-month presence of any suicidal behavior, including preparatory behavior, abortive attempts, interrupted attempts, or a suicide attempt. Similar to the ESS, it can be used to create three risk strata. Mild risk is defined as having active ideation but without method, intent, plan or lifetime suicidal behavior. Moderate risk is defined as having active ideation and suicidal method but without intent or plan, or having lifetime suicidal behavior but not in the past three months. High risk is defined as patients with suicidal intent or plan or as having suicidal behavior in the past three months. Posner and colleagues21 showed that adolescents with the two highest levels of ideation severity, namely intent or intent with plan, on the full version of the C-SSRS were at increased risk of a prospective suicide attempt. However, there are no published psychometric data on the C-SSRS Triage version and the proposed three-group stratification approach. It is notable that the C-SSRS Triage focuses only on suicidal ideation and behavior; although suicidal behavior is necessarily preceded at some point by ideation, Berman25 cautions against focusing too heavily on ideation in assessing suicide risk. The ESS includes other non-suicide-specific risk factors, such as a history of past psychiatric hospitalizations, which has substantive empirical support in the literature and was a confirmed independent predictor in our analyses.
Another secondary screening tool is the SAMHSA ED decision support tool created using a rigorous RAND (Research and Development Corporation) expert consensus methodology. It too has recently been validated with ED-SAFE data using methods similar to those reported here.26 The primary differences between the ESS and the SAMHSA tool rest in the specific indicators and in the stratification algorithm. The ESS uses the presence of both active ideation and a lifetime attempt as an indicator, while the SAMHSA tool uses lifetime attempt as a separate indicator, regardless of whether active ideation is currently present. While the ESS uses a psychiatric hospitalization as an indicator, the SAMHSA tool uses any significant mental health history as an indicator. Finally, the SAMHSA tool does not use critical items, like plan and intent, as part of stratification logic in contrast with the current analyses of the ESS.
Although the current study has several strengths, including prospective outcomes utilizing multiple data sources, it has several potential limitations. The sample consisted of patients with active suicidal ideation or a suicide attempt in the week before an index ED visit. Such patients represent a more severe subgroup of ED patients detected as being at risk of suicide. Second, the validation criterion was suicidal behavior. Although it is usually beneficial to assess prospective behavior as an outcome, ED-SAFE interventions received during and after the ED encounter may have confounded the association between predictors and outcome in a way that an alternative validation criterion, such as an independent clinician risk assessment during the ED visit, would not. Third, the original ESS refers to past two weeks’ ideation but in the current analyses, we only had access to responses based on the past week because that was the timeframe of items used in the baseline interview. Finally, for the current study, the ESS indicators were operationalized based on self-report and chart review, but not clinician assessment. When the ESS is applied in the clinical setting as intended, self-report and chart review would be augmented by collateral report and clinical impression to ascertain each indicator.
CONCLUSION
This study represents an evidence-informed effort to stratify patients with suicide risk and guide ED physicians’ decision-making. The ESS possesses adequate operating characteristics when considering the tool’s initial triage purpose. It goes beyond suicidal ideation to assess well-recognized risk factors for suicide, including history of psychiatric hospitalization, prior attempt, and substance use, as well as state factors, namely intoxication and agitation. We recommend prospective validation in new samples to confirm its operating characteristics and potentially reduce its length by removing the substance abuse and agitation items, which appeared to offer little predictive utility in the current study. Although the ESS cannot replace a full mental health assessment, it can help to direct limited mental health resources to patients who need them most.
Acknowledgements:
The authors gratefully acknowledge the ED-SAFE site investigators and research staff.
Funding and Support: Research reported in this publication was supported by the National Institute on Mental Health of the National Institutes of Health under award number U01MH088278. The project was fully financed with federal funds ($11,709,446).
Appendix 1:
How Each of the Secondary Screener Indicators Was Operationalized
| Construct | Item wording | Source |
|---|---|---|
| 1. Positive on recent ideation and lifetime attempt | Yes to “At any time during the past week, including today, have you actually had any thoughts of killing yourself?” AND Yes to “At any time in your life, including today, have you made a suicide attempt?” OR Yes to “Is your current visit due to a suicide attempt?” |
Baseline assessment |
| 2. Suicide “plan” (Method) | Yes to “At any time during the past week, including today, have you thought about HOW you might do this?” OR Yes to “Is your current visit due to a suicide attempt?” |
Baseline assessment |
| 3. Intent | Yes to “At any time during the past week, including today, have you had any intention of acting on these thoughts of killing yourself?” OR Yes to “Is your current visit due to a suicide attempt?” |
Baseline assessment |
| 4. Lifetime psychiatric hospitalization | “Have you ever been hospitalized for a psychological or emotional problem?” OR “Psychiatric inpatient hospitalizations” within past 6 months |
Baseline assessment Chart review |
| 5. Pattern of excessive substance use | ‘How often do you have a drink containing alcohol? ‘How many drinks containing alcohol do you have on a typical day when you are drinking? ‘How often do you have four or more drinks on one occasion?’ ‘An alcohol use disorder, like alcohol abuse or dependence?’ ‘Any drug use disorder, like drug abuse or dependence?’ ‘Alcohol abuse (current intoxication or evidence of any problem use) ‘Blood alcohol level indicates intoxication’ ‘Intentional illegal or prescription drug misuse’ ‘Any positive urine tox screen’ Substance abuse inpatient hospitalizations |
Baseline assessment Chart review |
| 6. Irritable, agitated, or aggressive | “Feeling so restless you couldn’t sit still” OR “An anxiety disorder?” OR Thoughts or threats of harm toward other people |
Baseline assessment Chart review |
Appendix 2:
Logistic regression of secondary screener-6 predictors and suicide outcomes only for those presenting to the ED with active suicidal ideation but no suicidal behavior
| Suicide attempt 6 weeks (n = 109; 8%) | Suicide attempt 12 months (n = 287; 22%) | Any suicidal behavior 6 weeks (n = 364; 27%) | Any suicidal behavior 12 months (n = 638; 46%) | |||||
|---|---|---|---|---|---|---|---|---|
| OR | 95% CI | OR | 95% CI | OR | 95% CI | OR | 95% CI | |
| Item 1 | 1.92 | 1.04–3.56 | 1.37 | 0.95–1.99 | 1.78 | 1.27–2.50 | 1.51 | 1.14–2.01 |
| Item 2 | 2.76 | 1.07–7.09 | 2.41 | 1.42–4.11 | 1.81 | 1.16–2.82 | 2.16 | 1.50–3.11 |
| Item 3 | 1.32 | 0.75–2.32 | 1.14 | 0.80–1.62 | 1.40 | 1.02–1.94 | 1.41 | 1.06–1.86 |
| Item 4 | 1.87 | 0.98–3.58 | 2.19 | 1.46–3.30 | 1.77 | 1.24–2.51 | 2.00 | 1.49–2.69 |
| Item 5 | 1.09 | 0.64–1.86 | 1.50 | 1.04–2.16 | 1.03 | 0.75–1.41 | 1.15 | 0.87–1.52 |
| Item 6 | 1.13 | 0.56–2.28 | 1.22 | 0.77–1.94 | 1.03 | 0.69–1.53 | 1.00 | 0.71–1.43 |
| Model | Chi: 24.94*** | Chi= 52.92*** | Chi= 57.65*** | Chi=87.30*** | ||||
Item 1= Active ideation and lifetime attempt; Item 2= Plan; Item 3= Intent; Item 4= Psychiatric hospitalization; Item 5=Excessive substance use; Item 6= Aggressive/irritable
p<0.001
Appendix 3: Association between final strata and prospective outcomes at (a) 6 weeks and (b) 12 months.
This figure shows the step change in prospective suicidal behavior associated with strata based on the secondary screener
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Publisher's Disclaimer: Disclaimer: The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. NIMH was represented on the ED-SAFE Steering Committee by Amy Goldstein, PhD. She collaborated with the other committee members and investigators to oversee the conduct of the study, data collection, analysis, and interpretation. NIMH provided DSMB oversight of the study. The NIMH DSMB liaison was Adam Haim, PhD.
Trial Registration: NCT01150994
Contributor Information
Edwin D. Boudreaux, Departments of Emergency Medicine, Psychiatry, and Quantitative Health Sciences, University of Massachusetts Medical School, Worcester, MA.
Celine Larkin, Department of Emergency Medicine, University of Massachusetts Medical School, Worcester, MA.
Carlos A. Camargo, Jr., Department of Emergency Medicine, Massachusetts General Hospital, Harvard Medical School, Boston, MA.
Ivan W. Miller, Department of Psychiatry and Human Behavior, Alpert Medical School of Brown University, Butler Hospital, Providence, RI..
REFERENCES
- 1.Barratt H, et al. Epidemiology of mental health attendances at emergency departments: Systematic review and metal-analysis. PLoS One. 2016;11(4): e0154449. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Moore BJ, et al. Trends in emergency department visits, 2006–2014 Statistical Brief #227. Rockville, MD: Agency for Healthcare Research and Quality, 2017. Accessed May 30, 2019 https://www.hcup-us.ahrq.gov/reports/statbriefs/sb227-Emergency-Department-Visit-Trends.pdf [Google Scholar]
- 3.Owens PL, et al. Emergency department visits related to suicidal ideation, 2006–2013 Statistical Brief #220. Rockville, MD: Agency for Healthcare Research and Quality, 2017. Accessed May 30, 2019 https://www.ncbi.nlm.nih.gov/books/NBK442036/ [Google Scholar]
- 4.Allen MH, et al. Screening for suicidal ideation and attempts among emergency department medical patients: Instrument and results from the Psychiatric Emergency Research Collaboration. Suicide Life Threat Behav. 2013;43(3):313–323. [DOI] [PubMed] [Google Scholar]
- 5.Claassen CA, et al. Occult suicidality in an emergency department population. Br J Psychiatry. 2005;186:352–353. [DOI] [PubMed] [Google Scholar]
- 6.Ilgen MA, et al. Recent suicidal ideation among patients in an inner city emergency department. Suicide Life Threat Behav. 2009;39(5):508–517. [DOI] [PubMed] [Google Scholar]
- 7.Han B, et al. Suicidal ideation among community-dwelling adults in the United States. Am J Public Health. 2014;104(3):488–497. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Ahmedani BK, et al. Health care contacts in the year before suicide death. J Gen Intern Med. 2014;29(6):870–877. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.The Joint Commission, 2018. Accessed September 20, 2019 https://www.jointcommission.org/assets/1/18/Suicide_Prevention_Resources_to_support_NPSG150101_Nov201821.PDF
- 10.Suicide Prevention Resource Center. Caring for adult patients with suicide risk: A consensus guide for emergency departments. Waltham, MA: Suicide Prevention Resource Center, 2015. Accessed September 20, 2019 https://www.sprc.org/edguide [Google Scholar]
- 11.Boudreaux ED, et al. Improving suicide reisk screening and detection in the emergency department. Am J Prev Med. 2016;50(4):445–453. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Roaten K, et al. Development and implementation of a universal suicide risk screening program in a safety-net hospital system. Jt Comm J Qual Patient Saf. 2018;44(1):4–11. [DOI] [PubMed] [Google Scholar]
- 13.Miller IW, et al. Suicide prevention in an emergency department population: The ED-SAFE study. JAMA Psychiatry. 2017;74(6):563–570. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Boudreaux ED, et al. Suicide risk screening and assessment: designing instruments with dissemination in mind. Am J Prev Med. 2014;47(3):163–169. [DOI] [PubMed] [Google Scholar]
- 15.The Joint Commission. The Joint Commission Perspectives: FAQs: Suicide risk recommendations. Vol. 38 No. 7. Oak Brook, IL: Joint Commission Resources, 2018. Accessed September 20, 2019 https://www.jointcommission.org/assets/1/6/Suicide_Risk_Recommendations_FAQs_July_Perspectives.pdf [Google Scholar]
- 16.Wilson MP, et al. (in press). Emergency department recommendations for suicide prevention in adults: The ICAR2E mnemonic and a systematic review of the literature. Am J Emerg Med. DOI: 10.1016/j.ajem.2019.06.031 [DOI] [PubMed] [Google Scholar]
- 17.Large MM, et al. Can we usefully stratify patients according to suicide risk?. BMJ. 2017;359:j4627. [DOI] [PubMed] [Google Scholar]
- 18.Boudreaux ED, et al. The patient safety screener: Validation of a brief suicide risk screener for emergency department settings. Arch Suicide Res. 2015;19(2):151–160. [DOI] [PubMed] [Google Scholar]
- 19.Arias SA, et al. Factors associated with suicide outcomes 12 months after screening positive for suicide risk in the emergency department. Psychiatr Serv. 2016;67(2):206–213. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Boudreaux ED, et al. The Emergency Department Safety Assessment and Follow-up Evaluation (ED-SAFE): Method and design considerations. Contemp Clin Trials. 2013;36(1):14–24. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Posner K, et al. The Columbia–Suicide Severity Rating Scale: Initial validity and internal consistency findings from three multisite studies with adolescents and adults. Am J Psychiatry. 2011;168(12):1266–1277. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Posner K, et al. Columbia Classification Algorithm of Suicide Assessment (C-CASA): classification of suicidal events in the FDA’s pediatric suicidal risk analysis of antidepressants. Am J Psychiatry. 2007;164:1035–1043. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.IBM Corp. Released 2016. IBM SPSS Statistics for Windows, Version 24.0. Armonk, NY: IBM Corp. [Google Scholar]
- 24.American College of Emergency Physicians. iCARE2: A tool for managing suicidal patients in the ED. Copyright 2019. Accessed May 30, 2019 https://www.acep.org/patientcare/iCar2e/
- 25.Berman AL, et al. Risk factors proximate to suicide and suicide risk assessment in the context of denied suicide ideation. Suicide Life Threat Behav. 2018;48(3):340–352. [DOI] [PubMed] [Google Scholar]
- 26.Boudreaux ED, et al. Predictive utility of an emergency department decision support tool in pateints with active suicideal ideation. Psychol Serv. 2018;15(3):270. [DOI] [PMC free article] [PubMed] [Google Scholar]