Comparing patterns of recent mental health service use for predicting suicidal events following emergency department mental health visits in the United States: A national cohort study

Abstract

Objective

To examine patterns of recent service use to predict non-fatal suicidal events shortly following emergency department (ED) visits for mental health.

Methods

For this retrospective cohort study, we used Optum electronic health record data from 2,445,597 ED mental health episodes (2015–2022) for persons aged ≥ 10 years. We then constructed a series of logistic regression models to evaluate how six permutations of characterizing prior 180-day mental health service use predicted acute non-fatal suicidal events within 180 days of ED discharge beyond demographic characteristics and ED mental health diagnoses. Model performance was assessed by area under the receiver operating curve (AUC).

Results

Overall, 7.2% (n = 176,000) of episodes resulted in an acute suicidal event within 180 days. Model performance improved from demographic characteristics and ED mental health diagnoses (AUC = 0.76) when past 180-day service use variables were added, but with minimal differences between a binary variable of any mental health service use compared to monthly counts, weighted slopes, or interactions (AUC’s 0.78–0.82). The final model containing demographics, ED mental diagnosis, and past 180-day service use yielded an AUC of 0.83. The most predictive past service use variables were any inpatient or ED event for self-harm or suicide ideation (OR = 5.45, 95% CI = 5.37, 5.54) and any ED mental health visit (OR = 1.87, 95% CI = 1.84, 1.90).

Conclusions

As part of evaluating suicide risk in ED settings, information about recent acute care for mental health, suicidal ideation, or self-harm use significantly contributes to the short-term prediction of non-fatal suicidal events.

Introduction

Suicide risk is critically high in the months following emergency department (ED) visits for mental health problems, especially among ED patients who were treated for suicidal ideation or self-harm [1, 2]. However, accurately identifying high risk ED patients remains a vexing challenge as screening by itself does not reduce risk of future suicidal behavior[3] and 33%−50% of patients denied suicidal ideation prior to their non-fatal or fatal suicide attempt [4]. In an effort to improve identification of high suicide risk patients, various modelling techniques using health care data have been recently applied, including machine learning approaches[5], trajectory-based modelling of service use[6], and latent class models [7].

Deeper examination of patterns of prior health care encounters may provide valuable insights as growing research suggests that suicide risk is time-varying and may be associated with health care use. For example, one study found that a concentration of mental health service use in a 3-month period prior to an index visit was significantly more predictive than longer lookback periods of non-fatal and fatal self-harm within 90 days [8]. Other research suggests that repeated or escalating inpatient or ED mental health episodes may indicate unresolved crises, whereas individuals with erratic, declining, or missed outpatient mental health visits may be particularly at-risk [9–13].

Advanced modelling techniques using health care data to predict suicide-related outcomes are superior to traditional clinical approaches [14]. Most research using health records has focused on identifying at-risk patients following outpatient visits or hospitalizations [15] or among relatively narrow ED populations. For example, prior prediction models focused on suicide-related outcomes following ED encounters include patients treated for self-harm[16], aged ≤18 years[17], or who completed a suicide ideation rating scale during the ED encounter [18]. In a recent study using health records to predict suicide mortality within 90 days of an ED encounter for any mental health condition, the model yielded an area under the receiver operating curve (AUC) of 0.82 (95% CI = 0.81–0.84)[19]. In this study, multiple approaches were used to characterize prior service use, including the total number of mental health encounters in outpatient, ED, and inpatient settings as well as 52 possible time patterns reflecting differing permutations (e.g., first onset, most recent occurrence, and increase or decrease over time) [19].

In most suicide prediction models, static characteristics including demographic (i.e., male sex, race, and age) and diagnostic features (e.g., major depression and psychosis) are salient suicide risk factors with prior treatment for deliberate self-harm being the most robust clinical predictor [15]. Recent developments suggest that adding dynamic, time-sensitive changes in healthcare utilization patterns (e.g., visit frequency and timing) can improve the predictive performance above and beyond demographic and clinical characteristics [20, 21]. However, it is unclear which indicators of prior service use are most effective at capturing complex temporal relationships (e.g., slope-related trends and other non-linear patterns of service use preceding an index mental health visit) to enhance the prediction of suicide-related outcomes. In this brief report, we evaluate the incremental performance of different ways of characterizing recent mental health care use, beyond demographic and diagnostic characteristics, in predicting acute suicidal events within 180 days of an ED mental health visit as part of the development of a clinical decision support tool for ED physicians to assess suicide risk. Based on emerging research that incorporating temporality of recent service use improves model performance[20, 21], we hypothesized that more granular approaches (e.g., increased clustering of escalating encounters more proximal to the time of a non-fatal suicidal event) would provide more accurate predictions of subsequent suicide risk.

Methods

Data sources

This retrospective cohort study used de-identified, structured electronic health record (EHR) data from the Optum Labs Data Warehouse (OLDW). The Optum EHR database contains longitudinal health information for over 200 million commercial, Medicaid, and Medicare Advantage enrollees representing a range of ages and geographical regions. The database includes diagnoses, procedures, medications, laboratory results, hospitalizations, and outpatient visits from integrated delivery networks including > 140,000 providers from > 700 hospitals and > 7,000 clinics. For the current study, EHR data were limited to ED visits between 9/30/2015-9/30/2022, which coincides with the adoption of ICD-10 in U.S. healthcare settings. In the U.S., ICD diagnostic codes are typically entered for billing and clinical documentation purposes. Because codes are often required to justify payment for services rendered, diagnoses that are connected to billable encounters tend to be more consistently and accurately coded. Research in the U.S. demonstrated that coding for self-harm in ED settings are well-captured (i.e., 88%) and improved with the transition in 2015 from ICD-9 to ICD-10[22, 23].

The data for this study, which are deidentified and previously collected, were deemed exempt from human participants review by the University Institutional Review Board. According to U.S. federal regulations governing human subjects research, secondary research utilizing data that is recorded anonymously and cannot be linked to subjects through identifiers is classified as exempt under 45 CFR 46.

Cohort assembly

We identified all ED visits persons aged > 10 years containing an EHR diagnosis for mental disorder (ICD-9 codes 290–319; ICD-10 codes F01-F99), intentional self-harm (ICD-9 codes E950-959; ICD-10 codes X71–X83), poisoning by drugs, medications or biological substances coded as intentional self-harm (ICD-10 codes T36 – T50), toxic effects of nonmedical substances coded as intentional self-harm (ICD-10 codes T51 – T65) asphyxiation, suffocation, or hanging codes as intentional self-harm (ICD-10 code T71), suicide attempt (ICD-10 code T14.91) or suicidal ideation (ICD-9 code V62.84; ICD-10 code R45.851).

We excluded ED visits for which the only mental health diagnosis was nicotine dependence (ICD-9 305.1; ICD-10 F17). Study visits were also excluded for contiguous observation episode prior to the index ED visit, length of ED stay of >4 days, and no evidence of healthcare activity within the participating health networks in the 180 days prior to the index visit. We selected a length of stay of less than 4 days to focus on patients receiving acute ED care that reflect individuals who undergo assessment, stabilization, and disposition (either discharge or admission) within the standard ED workflow, rather than patients with extended boarding in the ED. Individuals with no evidence of healthcare activity in the 180 days prior to their index ED episode or in the 180 post-index period were excluded because the absence of data compromised our ability to discern whether they truly had no care or were treated in a different health system not included in the Optum database. Episodes occurring within 2 days of discharge were combined as they were considered a continuation of care from the initial event based on prior research [24]. All ED episodes meeting these criteria were included in analyses, thus individual patients could contribute more than one ED mental health encounter. The final sample size was 2,445,597 ED episodes, representing 1,341,253 individuals.

Outcome

Acute non-fatal suicidal events were defined as any ED or inpatient claim within 180 days of ED discharge with a diagnosis of intentional self-harm (ICD-9 codes E950-959; ICD-10 codes X71–X83), poisoning by drugs, medications or biological substances coded as intentional self-harm (ICD-10 codes T36 – T50), toxic effects of nonmedical substances coded as intentional self-harm (ICD-10 codes T51 – T65) asphyxiation, suffocation, or hanging codes as intentional self-harm (ICD-10 code T71), suicide attempt (ICD-10 code T14.91), or suicidal ideation (ICD-9 code V62.84; ICD-10 code R45.851) [25].

Predictors

We selected patient demographic characteristics (sex, race/ethnicity, and age) and mental health diagnoses during the ED encounter as well as three binary characterizations of past 180-day service use: (1) any prior ED visit for any mental health disorder, self-harm, or suicide ideation; (2) any prior inpatient or ED event for self-harm or suicide ideation; and (3) any prior service use for self-harm or suicide ideation in any setting (i.e., ED, inpatient, outpatient). In addition to this predictor group, six characterizations of prior service use were selected to represent progressively more granular approaches that account for temporality and intensity of healthcare use as research suggests that this approach enhances predictive modelling of subsequent suicide risk [20, 21].

Each of the six modelling strategies was designed to progressively test how temporal patterns of past 180-day service use predicted non-fatal suicidal events, beginning with aggregate measures and moving toward increasingly detailed temporal configurations. The first modelling approach included a single binary indicator for any mental health-related encounter in any setting (i.e., ED, inpatient, outpatient) 180 days prior to the index ED event. The second model included a single independent variable containing the aggregate count of mental health encounters in any setting in the prior 180 days. The third approach was a set of six predictors representing the monthly count of mental health encounters in any setting for each of the six months in the prior 180 days. The fourth modelling strategy was a weighted slope based on the sum of the six monthly counts with each count multiplied by a monthly exponential decay function (see footnote in Table 2). The fifth approach included a set of six binary indicators of any mental health event in any setting in each month during the prior 180 days. The sixth modelling approach included a set of factors containing the interaction of the six monthly binary service use indicators from the prior model (64 combinations) to provide all permutations of monthly service use patterns. This operationalization accounted for fine-grained temporal variations in past service use (e.g., consistent, intermittent, or front- or back-loaded service engagement) to assess whether these distinct trajectories were differentially associated with the outcome.

Table 2.

Comparison of areas of under the curve (AUCs) for 180-day prior service use variables predictive of acute non-fatal suicidal events within 180 days following emergency department mental health encounter

	Prior ED visit for mental health, self-harm, or suicide ideation	Prior inpatient or ED event for self-harm or suicide ideation	Any prior service use for self-harm or suicide ideation
Characterizing ED visit only
Demographics only	0.634	0.634	0.634
Demographics and diagnosis from ED visit	0.760	0.760	0.760
Characterizing prior 180 days service use *
1. Any event in prior 6 months (1 predictor)	0.795	0.817	0.783
2. Count of events in prior 6 months (1 predictor)	0.801	0.815	0.794
3. Monthly counts of events in each of prior 6 months (6 predictors)	0.802	0.815	0.795
4. Weighted slope of each person’s monthly use** (6 predictors)	0.802	0.815	0.795
5. Monthly presence of any event in prior 6 months (6 predictors)	0.806	0.819	0.797
6. Interactions of monthly binary (32 predictors)	0.806	0.819	0.798

* Models included demographic characteristics and diagnosis from ED visit

** Weighted slope based on monthly exponential decay function of time prior to ED episode multiplied by monthly event count (1 month = 1.00, 2 months = 0.50, 3 months = 0.250, 4 months = 0.125, 5 months = 0.063, 6 months = 0.031)

Analysis

The analysis was conducted in 3 stages. First, we characterized the ED cohort and the sample who experienced an acute non-fatal suicidal event following ED discharge based on demographic, diagnostic, and prior mental health service use characteristics using unadjusted logistic regression. Second, we constructed a series of logistic regression models with different predictor set combinations to evaluate how different methods of characterizing mental health service use patterns in the prior 180 days influenced the AUC above and beyond characteristics from the ED encounter. Our base model utilized only demographic characteristics. The second model included demographic and mental health diagnosis variables from the ED encounter. Each subsequent model included demographic and diagnosis features from the ED encounter as well as 1 of the 6 permutations to characterizing monthly mental health service use patterns in the 180 days prior to the ED episodes described in the previous section. Using this approach, separate logistic regression analyses evaluated the relative strength of three characterizations of prior 180-day mental health service use: (1) prior ED visit for any mental health disorder, self-harm, or suicide ideation; (2) prior inpatient or ED event for self-harm or suicide ideation; and (3) any prior service use for self-harm or suicide ideation. To ensure the robustness of our findings, we used a 2/3 training set and 1/3 test set split to generate and evaluate all models and ensure that the performance metrics were reflective of the model’s generalizability. Each model was assessed by examining the out-of-sample test AUC, a measure of model performance, to evaluate how well the model distinguished between individuals who did and did not experience acute non-fatal suicidal events within 180 days of an ED visit.

Results

Cohort characteristics

Among the 2,445,597 ED episodes, most were by patients who were women (58.1%), white non-Hispanic (70.5%), either 18–34 or 45–64 years old (29.9% and 30.9%, respectively), and diagnosed with either depressive (31.9%) or anxiety (40.4%) disorders (Table 1). Overall, 7.2% (n = 176,000) resulted in an acute non-fatal suicidal event within 180 days. Although female and white, non-Hispanic patients made up most of the ED episodes, in unadjusted models the demographic characteristics associated with significantly higher odds of an acute suicidal event following ED discharge were being male (OR = 1.82) or Black non-Hispanic (OR = 1.25). Lower likelihood of an acute suicidal event within 180 days were associated with being aged ≥ 65 years and diagnosed with anxiety disorders during the ED visit. Clinical characteristics associated with increased likelihood of an acute suicidal event included an ED diagnosis of suicidal ideation or self-harm (OR = 6.22), personality disorder (OR = 4.49), psychotic disorder (OR = 3.86), or bipolar disorder (OR = 2.21).

Table 1.

Characteristics of emergency department visits with mental health diagnoses

	% of total sample N = 2,445,597	% with acute suicidal event within 180 days N = 176,000	Unadj OR
Sex	%	%
Male	41.9	9.6	1.82(1.80,1.84)
Female	58.1	5.5	ref
Race/Ethnicity
Hispanic	6.7	6.7	0.96(0.94,0.98)
Black, non-Hispanic	18.3	8.5	1.25(1.23,1.26)
White, non-Hispanic	70.5	6.9	ref
Asian, Non-Hispanic	0.6	4.6	0.65(0.60,0.71)
Other	3.9	7.1	1.03(1.00,1.06)
Age (years)
10–17 years	5.7	9.8	1.16(1.14,1.19)
18–34 years	29.9	8.5	ref
35–44 years	17.8	8.7	1.03(1.02,1.04)
45–64 years	30.9	7.4	0.86(0.85,0.87)
65 + years	15.8	1.6	0.18(0.17,0.18)
ED Mental Health Conditions
Suicidal ideation or self-harm
Yes	8.0	26.6	6.22(6.14,6.29)
No	92.0	5.5	ref
Psychotic disorders
Yes	7.4	20.2	3.86(3.81,3.91)
No	92.6	6.2	ref
Depressive disorders
Yes	31.9	8.4	1.28(1.27,1.29)
No	68.1	6.7	ref
Bipolar disorders
Yes	9.8	13.4	2.21(2.18,2.24)
No	90.2	6.5	ref
Anxiety disorders
Yes	40.4	5.4	0.63(0.62,0.64)
No	59.6	8.4	ref
Trauma- and stress-related disorders
Yes	6.3	12.2	1.89(1.86,1.92)
No	93.7	6.9	ref
Disruptive behavior disorders
Yes	1.1	17.3	2.75(2.67,2.84)
No	98.9	7.1	ref
Personality disorders
Yes	1.4	25.1	4.49(4.38,4.60)
No	98.6	6.9	ref
Nicotine-related disorders *
Yes	29.3	9.8	1.66(1.64,1.67)
No	70.7	6.1	ref
Alcohol-related disorders
Yes	14.7	11.2	1.80(1.78,1.82)
No	85.3	6.5	ref
Other substance related disorders
Yes	14.6	10.4	1.63(1.61,1.65)
No	85.4	6.7	Ref
180-Day Prior Service Use
ED visit for mental health, self-harm, or suicide ideation
Yes	40.0	12.9	4.24(4.20,4.29)
No	60.0	3.4	ref
Prior inpatient or ED event for self-harm of suicide ideation
Yes	8.1	39.6	14.5(14.3,14.6)
No	91.9	4.3	ref
Any prior service use for self-harm or suicide ideation
Yes	62.4	9.7	3.37(3.32,3.41)
No	37.6	3.1	ref
Summary of 3 prior service use variables
Yes	69.8	9.2	3.91(3.85,3.97)
No	30.2	2.5	ref

* Nicotine-related disorders were not a qualifying diagnosis and included only if comorbid with another mental health condition

Examining different predictor models of acute suicidal events

The base logistic regression models containing only demographic characteristics had an AUC of 0.63, which increased after adding mental health diagnoses from the ED encounters to an AUC to 0.76 (Table 2). For all three mental health service variables, the addition of prior 180-day mental health service use permutations to the model only modestly increased AUC values, however, there was little improvement beyond the binary service use variable for any event in the prior 180 days (e.g., AUC values ranging from 0.78 to 0.82) with more nuanced permutations of prior service use (e.g., monthly counts, weighted slopes, interactions) failing to yield meaningful increases in model performance (Table 2).

When demographic characteristics, ED mental health diagnosis, and three characterizations of prior 180-day service use variables were used together in a logistic model to predict acute suicidal events within 180 days of the ED encounter, the model achieved an AUC value of 0.83. The strongest predictor was prior inpatient or ED event for self-harm or suicide ideation (OR = 5.45, 95% CI = 5.37, 5.54) followed by prior ED visit for any mental health disorder, self-harm, or suicide ideation (OR = 1.87, 95% CI = 1.84, 1.90) and any prior service use for self-harm or suicide ideation (OR = 1.56, 95% CI = 1.53, 1.59).

Discussion

This retrospective cohort study examined factors associated with suicidal and self-harm events within 180 days of an ED mental health visit with a focus on comparing the incremental predictive performance of different permutations of prior service use. Key findings include that suicidal ideation or self-harm, personality disorder, psychotic disorder, or bipolar disorder diagnosed during the ED encounter were clinically meaningful predictors of non-fatal suicidal events within 180 days. Our final prediction model using demographic, ED diagnoses, and prior 180-day service variables had an overall AUC value of 0.83 for non-fatal suicidal events. Contrary to our hypothesis, more granular modelling approaches of prior service use that accounted for temporality and intensity of healthcare encounters (e.g., monthly counts, weighted slopes, interactions) did not meaningfully improve predictive performance beyond a simple binary indicator for any mental health-related encounter in the prior 180 days.

The classification performance of our final model compares favorably with previous studies using structured EHR data to predict non-fatal and fatal suicide-related events shortly following ED mental health visits, which observed AUC values ranging from 0.65–0.82[17–19]. Whereas some studies that examined different temporal patterns of prior service use found that binary indicators to be as predictive of suicide-related outcomes as more sophisticated representations[19, 26], other studies found temporally-informed service use predictors to be superior [20, 27]. Similarly, one study observed health care encounters in the 3 months prior to a mental health visit were significantly more predictive than more distant encounters of non-fatal and fatal self-harm within 90 days [28]. One potential reason for the inconsistent results may include differences in patient populations as some have focused on ED and inpatient patients[19, 26], whereas others included outpatients or all patients in a healthcare system [20, 27, 28]. Additional research is needed to inform how patients with elevated suicide risk can be identified using predictive modelling approaches.

Implications

From a clinical perspective, these findings can assist ED clinicians in screening patients to identify those who carry the highest short-term risk of subsequent non-fatal suicidal events and most in need of more personalized, stepped-up care. Single encounter, in-person suicide prevention interventions significantly improve linkage with follow-up mental health care and reduce subsequent suicide attempts [29]. Oher brief interventions show promise in reducing suicide risk, such as caring contacts (e.g., texts, post cards), follow-up case management and supportive counseling, and safety planning [3, 30, 31].

In the U.S., the Joint Commission requires suicide screening of all patients who are being evaluated or treated for behavioral health conditions as their primary reason for care using a validated screening tool, such as the Columbia Suicide Severity Rating Scale (C-SSRS) [32]. Our final model (AUC = 0.83) performed substantially better than the reported value for screening with the C-SSRS for predicting suicide attempt (AUC = 0.59) following an ED episode [33]. Given that many patients deny suicidal ideation prior to self-harm [4] and barriers to disclosure during ED assessment of suicide risk have been identified[34, 35], inquiries about prior mental health service use may be a viable clinical strategy to reduce stigma and discomfort to better inform risk assessment. Especially in healthcare settings that do not routinely screen, EHR information about prior acute mental health care can assist clinicians with stratifying risk and allocating limited clinical resources based on patient need. Additionally, 53.7% of U.S. community hospitals lack psychiatric services for ED consultation[36], requiring frontline clinicians to make difficult decisions regarding risk assessment and appropriate care that could be informed by reviewing a patient’s EHR for prior acute psychiatric care. Because of its predictive value, information about past psychiatric hospitalization has been incorporated into secondary screening tools to stratify suicide risk in ED settings [37, 38].

Limitations

This study has several limitations. First, our cohort consisted of ED patients treated for mental health disorders and is not representative of the general population. Although this dataset included a diverse range of publicly insured, commercially insured, and uninsured patients, roughly 20% of adults who attempt suicide do not have mental disorders [39] and thus may not be represented in this ED mental health sample. Second, we relied on ICD diagnosis codes and EHR diagnoses, which were based on routine clinician assessments and were not subjected to independent expert validation. Some concerns exist over the validity and completeness of ICD codes for suicidal ideation and behaviors[40, 41], however recent U.S. research demonstrated significant improvements in completeness and accuracy of coding of intentional self-injury and self-poisoning with the transition from ICD-9 to ICD-10 in the U.S. on October 1, 2015[22, 23]. Additionally, our findings are limited to episodes of acute suicidal ideation or self-harm that resulted in an ED encounter and may not generalize suicide-related events that resolved in the community or were treated in outpatient settings. Third, because of our reliance on ICD codes, we were unable to distinguish suicide attempts from non-suicidal self-injury which can represent distinct clinical entities and confer different risk of subsequent self-harm [42, 43]. Additionally, ICD codes do not provide details about the clinical severity or context of deliberate self-harm.

Conclusions

ED clinicians commonly face clinical uncertainties in evaluating suicide risk, even when using standardized screening measures. In the current study, a prediction model using structured EHR data demonstrated relatively accurate prediction of acute non-fatal suicidal events within 180 days following ED mental health encounters. As part of evaluating suicide risk in ED settings, these findings can help to inform clinical determinations for which patients warrant higher levels of follow-up care.

Author contributions

TS: Conceptualization, Methodology, Writing – Original DraftSCM: Conceptualization, Methodology, Investigation, Writing – Original Draft, Supervision, Funding acquisitionMX: Software, Formal analysis, Data CurationSWC: Project administration; Writing - Review & EditingTL: Formal analysis, Writing-Review & EditingLHU: Methodology, Writing-Review & Editing; NC: Methodology, Writing-Review & EditingNJW: Conceptualization, Methodology, Writing-Review & Editing MO: Conceptualization, Methodology, Investigation, Writing-Original Draft, Supervision, Funding acquisition.

Funding

This research was supported by grant 5R01MH126895-02 from the National Institute for Mental Health.

Data availability

No datasets were generated or analysed during the current study.

Declarations

Role of the sponsor

The sponsors had no role in the design, analysis, interpretation, or publication of this study.

Conflict of interest

The authors have no conflicts of interest to report.