Predicting suicide death after emergency department visits with mental health or self-harm diagnoses

Abstract

Objective –

Use health records data to predict suicide death following emergency department visits.

Methods –

Electronic health records and insurance claims from seven health systems were used to: identify emergency department visits with mental health or self-harm diagnoses by members aged 11 or older; extract approximately 2500 potential predictors including demographic, historical, and baseline clinical characteristics; and ascertain subsequent deaths by self-harm. Logistic regression with lasso and random forest models predicted self-harm death over 90 days after each visit.

Results –

Records identified 2,069,170 eligible visits, 899 followed by suicide death within 90 days. The best-fitting logistic regression with lasso model yielded an area under the receiver operating curve of 0.823 (95% CI 0.810 - 0.836). Visits above the 95^th percentile of predicted risk included 34.8% (95% CI 31.1 – 38.7) of subsequent suicide deaths and had a 0.303% (95% CI 0.261 - 0.346) suicide death rate over the following 90 days. Model performance was similar across subgroups defined by age, sex, race, and ethnicity.

Conclusions –

Machine learning models using coded data from health records have moderate performance in predicting suicide death following emergency department visits for mental health or self-harm diagnosis and could be used to identify patients needing more systematic follow-up.

People seen in emergency departments for mental health problems or self-harm are at high risk for subsequent death by suicide. Among visitors to California emergency departments, those presenting with self-harm experienced one-year suicide mortality 60 times that of the general population, and those presenting with suicidal ideation experienced suicide mortality 30 times as high¹. A similar pattern, but with smaller increases in risk, was seen among Medicaid beneficiaries visiting emergency rooms for suicidal ideation or self-harm². Among US health plan members who died by suicide, nearly half had at least one emergency department visit in the prior year with approximately one quarter visiting the emergency department for a mental health or substance use condition³.

Accurate identification of emergency department visitors at highest risk for suicidal behavior remains a challenge. While the Joint Commission recommends systematic screening of people receiving mental health care in emergency departments⁴, standard self-report questionnaires have limited sensitivity and only moderate predictive accuracy when evaluated in emergency department settings^5–9.

A large literature describes the accuracy of machine learning models based on health records data to identify people at risk for suicidal behavior after outpatient visits or hospitalizations¹⁰. Some of these models achieve overall classification performance (as measured by area under the receiver operating curve or c-statistic) exceeding 85%^10–12. Rather than identifying previously unknown predictors or categories of risk, these statistical models outperform screening questionnaires or simple risk checklists by simultaneously considering large numbers of risk factors recorded in health records¹³.

Less previous research has evaluated classification performance of prediction models using health records to identify emergency department visitors at increased risk for suicide. Among patients seen for self-harm in the Canadian province of Alberta, machine learning models using linked health care and social service databases predicted suicide mortality over the following 90 days with overall classification performance or AUC of approximately 0.88¹⁴. In a sample of patients seen by a Boston psychiatric emergency service, a machine learning model using health records data and patients’ self-reports of suicidal ideation and other risk factors predicted subsequent suicide attempt with an AUC of 0.79⁸. In a pediatric emergency department population, a prediction model using records data predicted subsequent suicide-related visit with an AUC of 0.78⁹. In a sample of California emergency department visits, machine learning models limited to data from that single emergency department visit predicted subsequent suicide death with AUCs ranging from 0.69 to 0.77 across different racial and ethnic groups¹⁵. No previous reports including US samples describe prediction of suicide death following emergency department care using coded or discrete data extracted from longitudinal health records data.

We describe here the development and validation of a model predicting death by suicide following an emergency department visit involving a mental health or self-harm diagnosis using records data from seven large integrated health systems.

METHODS

Data were collected from records of seven health systems (HealthPartners, Henry Ford Health, and the Colorado, Hawaii, Northwest, Southern California, and Washington regions of Kaiser Permanente) serving a combined population of approximately 6 million members or patients in the states of California, Colorado, Hawaii, Michigan, Minnesota, Oregon, and Washington. Members are insured or enrolled via employer sponsored insurance, individual insurance, Medicare, Medicaid, and subsidized insurance exchange plans. Patients served by these systems are generally representative of service area populations in terms of age, race, and ethnicity¹⁶. Responsible institutional review boards for each health system approved the use of de-identified records data for this research.

Research centers in each participating health system maintain harmonized research data warehouses integrating data from electronic health records (for services provided by the health system) and insurance claims (for external services covered by the health insurance plan)¹⁷. Health system data are linked to state and/or national vital statistics data to ascertain date and cause of death.

The study sample included all encounters by health system members to emergency departments or urgent care centers for which any diagnosis of a mental health condition, a substance use disorder, or a self-harm injury or poisoning was recorded. A list of included diagnoses can be found at https://mhresearchnetwork.org/resources/mhrn-data-resources/useful-tools/ . Eligible visits were limited to those for which the patient was enrolled in the participating health system on the visit date, but no duration of prior enrollment was required. An individual with more than one eligible encounter during the study period would contribute multiple encounters to the sample. Eligible encounters were excluded from analyses in cases of death by cause other than self-harm within 90 days of the index encounter.

For each eligible encounter, potential predictors of subsequent suicide death recorded during the prior 60 months were extracted from research data warehouses. Predictors included age, sex recorded in the electronic health record (most often indicating sex assigned at birth), self-reported race and ethnicity recorded in the electronic health record (typically reported at an outpatient visit in response to standard categories), source of insurance coverage, mental health and substance use disorder diagnoses (in 14 categories), chronic pain diagnoses, traumatic brain injury diagnoses, dispensed prescriptions for mental health medications (in 8 categories), prior injury or poisoning diagnoses (in 4 categories), and prior emergency department or inpatient encounters with mental health diagnoses, prior outpatient mental health specialty visits, responses to PHQ-9 depression questionnaires (both total scores and response to the 9^th item regarding thoughts of death or self-harm), 17 indicators of chronic medical illness following the Charlson Comorbidity score, and neighborhood-level indicators of household income and educational attainment. To improve generalizability across health systems with potentially different prescribing patterns, diagnosis coding systems, and coding practices, representation of diagnoses and prescriptions collapsed large numbers of potential predictors into relatively broad clinical categories (e.g. all depression diagnoses combined into a single category, all benzodiazepines combined into a single category). For each category of mental health or substance use diagnosis or service use, count data for each of the prior 60 months (e.g., number of days on which diagnosis was recorded in each of the 60 months, number of days’ supply of medication dispensed during each of the 60 months) were transformed into 52 possible time patterns reflecting various permutations of first onset, most recent occurrence, and increase or decrease over time. Prescription dispensing data were transformed into 24 possible time patterns. Time categories for diagnosis, utilization, and medication predictors all included potential adjustments for duration of prior enrollment in the participating health system. A data dictionary describing potential predictors and time patterns is included in Appendix A.

Suicide deaths and deaths from other causes were ascertained by linking health system records to state mortality records. These linkages ascertain date and cause of death for all members or patients regardless of duration of health plan enrollment. Deaths with any ICD-10 cause of death code in the range X60 through X84 or Y87.0 (indicating intentional self-harm) or the range Y10 through Y34 or Y87.2 (indicating undetermined intent) were considered suicide deaths in these analyses. Inclusion of those deaths coded as having undetermined intent increased the count of suicide deaths by approximately 5 percent.

Logistic regression with lasso models¹⁸ were developed in RStudio¹⁹ using the CRAN package keras²⁰. Five-fold cross-validation was used to find the optimal degree of penalization by maximizing the area under the receiver operating curve (AUC) on the left-out-folds. Confidence intervals (CIs) for AUC and other performance measures were constructed using 10,000 bootstrap replicates²¹. Logistic models considered a large number of potential interaction terms between the predictors listed in Appendix A, yielding 12,066 candidate predictors for logistic models. Random forest models were estimating using the CRAN package ranger²². R¹⁹ was used to fit models, using a variation of the predictors in Appendix A that allowed the model to assign missing values to either side of a split in a node. Again, five-fold cross-validation and the AUC was used to establish optimal values for the minimal size for splitting a node, as well as the number of predictors selected to consider splitting on at each node. All potential predictors were standardized to have a range from 0 to 1 prior to model fitting.

Model performance characteristics were evaluated using the out-of-sample cross-validation predictions generated during the cross-validation process. Previous research indicates that this approach provides adequate protection against optimism or over-fitting when predicting infrequent events²³ and specifically for predicting suicide death using health records data²⁴. We report the AUC of the models with the optimal parameter values for the out-of-sample predictions as well as positive predictive value and sensitivity using a variety of cut-points and calibration tables. Performance was evaluated overall as well as by age, type of mental health diagnosis at the index emergency department visit (with or without self-harm event), sex (male, female), and race (Asian, Black/African American, Native Hawaiian/Pacific Islander, American Indian/Alaskan Native, White, more than one race selected, Other, unknown) and ethnicity (Hispanic, non-Hispanic).

RESULTS

The criteria described above identified 2,070,638 eligible visits, of which 1,468 were excluded due to death from cause other than self-harm within 90 days. Of the remaining 2,069,170 eligible visits by 989,306 unique patients, 899 were followed by a self-harm death within 90 days. The 899 visits followed by self-harm death represented 807 unique self-harm deaths as some self-harm deaths were linked with more than one prior visit. Characteristics of the final visit sample, including those followed and not followed by self-harm death, are shown in Table 1.

Table 1 –

Characteristics of emergency department visits with mental health or self-harm diagnoses

	All eligible visits n=2,069,170		Fatal self-harm within 90 days (n=899)
Diagnoses at index encounter
Includes injury or poisoning	380,677	18.40%	210	23.36%
No injury or poisoning	1,688,493	81.60%	689	76.64%
Female	1,241,034	59.98%	428	47.61%
Race
Asian	83,138	4.02%	39	4.34%
Black	208,644	10.08%	33	3.67%
Native Hawaiian/Pacific Islander	9,316	0.45%	1	0.11%
American Indian/Alaskan Native	11,801	0.57%	5	0.56%
White	1,427,362	68.98%	683	75.97%
More than one	76,146	3.68%	21	2.34%
Other	10,702	0.52%	7	0.78%
Unknown	242,061	11.70%	110	12.24%
Ethnicity
Hispanic	568,799	27.49%	88	9.79%
Not Hispanic	1,255,115	60.66%	590	65.63%
Unknown	245,256	11.85%	221	24.58%
Diagnoses in prior 12 months
Depressive disorder	931,002	44.99%	559	62.18%
Anxiety Disorder	895,934	43.30%	540	60.07%
Bipolar Disorder	161,495	7.80%	168	18.69%
Schizophrenia Spectrum Disorder	65,865	3.18%	60	6.67%
Personality Disorder	148,850	7.19%	163	18.13%
Alcohol Use Disorder	223,389	10.80%	261	29.03%
Other Substance Use Disorder	270,069	13.05%	261	29.03%
Maximum Item 9 score in past 12 mos
None	1,591,003	76.89%	586	65.18%
Not at all	315,940	15.27%	165	18.35%
Several days	82,597	3.99%	68	7.56%
More than half the days	37,760	1.82%	21	2.34%
Nearly every day	41,870	2.02%	59	6.56%
Self-harm in prior 12 months	53,727	2.60%	138	15.35%
MH ED visit in prior 12 months	850,802	41.12%	515	57.29%
MH Inpatient stay in prior 12 months	488,900	23.63%	461	51.28%
MH Specialty visit in prior 12 months	683,664	33.04%	585	65.07%

The best-fitting logistic regression with lasso model included 496 predictors and an overall classification performance or AUC in out-of-sample folds of 0.823 (95% CI 0.810 to 0.836). AUC was similar across the 5 out-of-sample folds (Appendix Figure 1). The best-fitting random forest model (including 200 trees with a minimum node size of 100,000) had poorer classification performance, with an out-of-sample AUC of 0.772 (95% CI 0.755 to 0.788). More detailed performance metrics for the best-fitting lasso model are described below. Parallel metrics for the best-fitting random forest model are presented in Appendix Tables 2 and 3.

Table 2 displays classification performance (sensitivity and positive predictive value or PPV) at different cut-points for the best-fitting logistic regression with lasso model at cut-points of the risk score distribution selected a priori. A cut-point at the 95^th percentile of the risk distribution (i.e., top 5% of visits) had a sensitivity of 34.8% (95% CI 31.1% to 38.7%) and yielded a sample of visits with a risk of suicide death over the following 90 days of 0.303% (95% CI 0.261% to 0.345%). Table 3 displays calibration performance (predicted and observed risk) for strata of the risk distribution selected a priori. Visits in the top 0.5% of the risk distribution had an observed risk of approximately 1 in 200 or approximately 80 times as high as visits in the bottom 50% (0.567% vs 0.007%). Calibration performance for ten equally-sized decile strata is shown in Appendix Table 4 and Figure 2.

Table 2 –

Classification performance of the best-fitting lasso logistic model at cut-points defined a priori

Percentile Cut-Point	Sensitivity	PPV
50%	92.3%	0.080%
75%	73.7%	0.128%
90%	47.2%	0.205%
95%	34.8%	0.303%
99%	11.6%	0.503%
99.5%	6.7%	0.580%

Table 3 –

Calibration of the best-fitting lasso logistic model across risk strata selected a priori

Percentile Stratum	% of Events in this stratum	Predicted Risk	Observed Risk
0% to 50%	7.7%	0.009%	0.007%
50% to 75%	19.2%	0.031%	0.033%
75% to 90%	25.8%	0.068%	0.075%
90% to 95%	13.0%	0.123%	0.113%
95% to 99%	22.9%	0.229%	0.250%
99% to 99.5%	4.7%	0.450%	0.406%
99.5% to 100%	6.6%	0.911%	0.567%

The 20 most influential positive and negative predictors in the best-fitting logistic regression with lasso models, determined by the magnitude of the appropriate coefficient, are shown in Table 4. The most influential predictors included many expected predictors of suicidal behavior (prior self-harm or injury and poisoning diagnoses, prior mental health inpatient care, mental health diagnoses), often selected in interaction terms.

Table 4 –

Most influential predictors selected by best-fitting lasso logistic model. Asterisks indicate interaction terms.

MOST INFLUENTIAL POSITIVE (HIGHER RISK) PREDICTORS	Coefficient	MOST INFLUENTIAL NEGATIVE (LOWER RISK) PREDICTORS	Coefficient
Age	1.069965	Dementia Dx at visit	−0.84173
Depression Dx at visit	0.578624	Days with ADD Rx in last 2 years * race=black	−0.6589
Insurance not through ACA exchange	0.447669	Months with ADD Dx in past 2 years * Hispanic=yes	−0.56789
Any injury/poisoning in last month * male sex	0.369343	Months since last month with max number of self-harm Dx	−0.54792
# of PHQ9 Item 9 scores = 0 in last 12 mos * male sex	0.285185	Months since 2nd-generation antipsychotic Rx * Hispanic=yes	−0.4357
# mos in last 3 mos with mental health inpatient care	0.251476	Number of months in last year with antidepressant Rx	−0.40999
# of last 3 mos with 2nd-gen antipsychotic Rx * Item 9 score = 0	0.225704	No 2nd-generation antipsychotic Rx in past 5 years	−0.4044
Days supply anticonvulsant dispensed in last 3 mos * male sex	0.225379	No benzodiazepine Rx in past 5 years	−0.40408
# of last 3 mos with 2nd-gen antipsychotic Rx * Item 9 score missing at visit	0.211783	No self-harm Dx in last 5 years	−0.39867
# mos since initial 2nd-gen antipsychotic Rx * male sex	0.21109	PHQ8 score never greater than 20 in past 5 years	−0.36773
Most recent month with 2nd-gen antipsychotic Rx * Hispanic=unknown	0.21049	Pain Dx at visit	−0.36515
% of outpatient visits in last month with depression Dx	0.199566	# Months with autism Dx last 5 yrs * Age 11-17	−0.36312
Proportion of outpatient visits in last 3 mos with depression Dx	0.181717	Days in last 5 yrs with non-laceration self-harm violent injury * age 11-17	−0.3616
No Dx of bipolar disorder in last 5 yrs * male sex	0.163581	No lithium Rx in last 5 yrs	−0.33591
# of Item 9 scores = 3 in last 3 mos * Item 9 missing at visit	0.154509	No anticonvulsant Rx in last 5 yrs	−0.33564
# mos in last 24 mos with antidepressant Rx * male sex	0.149195	No non-laceration self-harm violent injury in last 5 yrs	−0.33327
# mos in last 3 mos with anxiety Dx * male sex	0.148294	# months with 2nd-gen antipsychotic Rx in last 12 mos	−0.31127
Alcohol use disorder Dx at index visit	0.148202	% of last 60 mos with high specialty mental health visit rate	−0.2685
# mos in last 3 mos with 1st-gen antipsychotic Rx * male sex	0.128839	# mos in last 12 mos with alcohol use disorder Dx * male sex	−0.25132
# of Item 9 scores = 3 in last month * male sex	0.115935	Mos since this person’s highest specialty mental health visit rate	−0.23629

Additional analyses examined consistency of results across patient subgroups defined by sex, age, race, ethnicity, and type of diagnosis at the index emergency department visit are found in Table 5. Numbers of suicide deaths in some subgroups were too small for stable estimates of model performance. Otherwise, overall classification performance, as measured by AUC, as well as sensitivity and PPV at the 95^th percentile were similar across all subgroups examined. Classification performance was also similar across participating health systems and across years in the study period (Appendix Table 5).

Table 5 –

Classification performance of the best-fitting lasso model across patient subgroups

	# of visits	# (%) of visits followed by suicide death	AUC	95% CI for AUC	Sensitivity at 95th %ile	PPV at 95th %ile
Total	2,069,170	899 (0.043%)	0.823	(0.810, 0.836)	34.8%	0.303%
Diagnoses at index encounter
Includes injury or poisoning	380,677	210 (0.055%)	0.840	(0.814, 0.865)	34.0%	0.289%
No injury or poisoning	1,688,493	689 (0.041%)	0.819	(0.803, 0.836)	35.0%	0.306%
Recorded Sex
Female	1,241,034	428 (0.035%)	0.824	(0.805, 0.843)	35.0%	0.311%
Male	828,074	348 (0.042%)	0.821	(0.796, 0.845)	34.2%	0.287%
Age
11 thru 21	240,462	51 (0.021%)	0.826	(0.786, 0.863)	34.4%	0.266%
22 thru 35	375,615	157 (0.042%)	0.839	(0.811, 0.864)	31.6%	0.330%
36 thru 50	412,426	211 (0.051%)	0.805	(0.775, 0.838)	27.1%	0.247%
51 thru 65	431,794	293 (0.068%)	0.823	(0.786, 0.858)	39.6%	0.371%
66 and older	608,873	187 (0.031%)	0.823	(0.790, 0.854)	39.5%	0.286%
Race/Ethnicity
Asian	83,138	39 (0.047%)	0.844	(0.770, 0.907)	40.0%	0.289%
Black	208,644	33 (0.016%)	0.822	(0.767, 0.873)	35.9%	0.316%
Native Hawaiian/Pacific Islander	9,316	1 (0.011%)	n/a	n/a	n/a	n/a
American Indian/Alaskan Native	11,801	5 (0.042%)	n/a	n/a	n/a	n/a
White	1,427,362	683 (0.048%)	0.819	(0.801, 0.837)	34.3%	0.291%
More than one	76,146	21 (0.028%)	0.815	(0.754, 0.865)	26.8%	0.289%
Other	10,702	7 (0.065%)	n/a	n/a	n/a	n/a
Unknown	242,061	110 (0.045%)	0.829	(0.789, 0.871)	34.2%	0.322%
Ethnicity
Hispanic	568,799	88 (0.016%)	0.839	(0.811, 0.866)	42.4%	0.355%
Not Hispanic	1,255,115	590 (0.047%)	0.818	(0.798, 0.837)	32.6%	0.279%
Unknown	245,256	221 (0.901%)	0.809	(0.768, 0.852)	27.4%	0.277%
Site
1	123,388	90 (0.073%)	0.772	(0.728, 0.816)	29.2%	0.421%
2	178,007	142 (0.080%)	0.766	(0.716, 0.812)	20.8%	0.180%
4	1,429,368	445 (0.031%)	0.838	(0.822, 0.853)	39.3%	0.323%
5	30,043	18 (0.060%)	0.774	(0.656, 0.874)	16.7%	0.133%
6	136,482	105 (0.077%)	0.813	(0.771, 0.853)	30.4%	0.322%
7	161,472	96 (0.059%)	0.815	(0.767, 0.860)	23.6%	0.161%
10	10,410	3 (0.029%)	n/a	n/a	n/a	n/a
Year
2009-2010	327,152	187 (0.057%)	0.824	(0.794, 0.852)	31.3%	0.275%
2011-2012	407,202	184 (0.045%)	0.796	(0.762, 0.828)	33.1%	0.280%
2013-2014	468,128	193 (0.041%)	0.813	(0.786, 0.838)	34.8%	0.333%
2015-2016	606,493	241 (0.040%)	0.845	(0.821, 0.866)	39.6%	0.333%
2017	260,195	94 (0.036%)	0.837	(0.801, 0.870)	30.8%	0.246%

n/a - number of events is too small for reliable estimate

DISCUSSION

We find that a prediction model using only discrete or coded data extracted from health system records was able to predict suicide death after an emergency department visit with overall classification performance or AUC of approximately 0.82. Overall classification performance was similar to that of previous efforts to predict suicide death following emergency department visits from records data^{8, 9, 14}. Performance far exceeded that previously reported for screening using the Columbia Suicide Severity Rating Scale, where reported AUCs were 0.667 for prediction of suicide death⁵ and 0.589 for prediction of suicide attempt⁶ following emergency department visit.

This encounter-based prediction model is designed to inform interventions that might occur during or soon after an emergency department encounter, such as more intensive assessment, safety planning, or more intensive outpatient follow-up. This encounter-based approach contrasts with population-based approaches, such as the Veterans Affairs REACH-VET model²⁵, intended to inform outreach independent of any encounter.

As shown in Table 1, visits followed by self-harm events more often exhibited expected risk factors for suicidal behavior, including current or prior mental health diagnoses, prior self-reports of suicidal ideation, prior self-harm events, prior use of specialty mental health outpatient care, and prior emergency department or inpatient care with mental health diagnoses. As expected²⁶, none of these individual risk factors were strongly enough associated with subsequent suicide death to guide decisions by emergency department clinicians. Combination of these and other risk indicators into a high-dimensional prediction score yielded much more accurate classification of risk than any individual risk factor or simple combination of risk factors.

Overall classification performance was generally similar across patient subgroups defined by presenting problems (with or without self-harm event), sex, age, race, and ethnicity, with the caveat that numbers of suicide deaths in some racial groups were too small to support stable estimates of performance. As expected, PPV at any risk percentile cut-point was greater in subgroups with higher overall suicide mortality rate. Given that PPV varies with average risk, users of this or any prediction model might choose different cut-points or thresholds for groups with higher and lower overall risk of self-harm. Selection of the optimal threshold for action for any group depends on the relative priority of avoiding false negative errors (sensitivity) and avoiding false positive errors (PPV). The consequences of errors (such as “welfare checks” by armed law enforcement) certainly could vary across racial and ethnic groups.

Overall classification performance as measured by AUC was modestly lower than parallel models developed in these health systems to predict suicide death following mental health specialty visits or general medical visits with mental health diagnoses, for which AUC statistics for prediction of fatal self-harm were 0.861 and 0.833 respectively¹¹. This lower performance of the same method among emergency department visitors may reflect several possible differences between prediction tasks. The number of fatal self-harm events included in this sample was smaller than the numbers of events in earlier samples of outpatient visits. Patients in this sample were less likely to have prior outpatient mental health visits where risk factors for suicide would have been identified and recorded. Suicide following an emergency department visit may simply be less predictable, even if typical risk factors were identified and recorded.

Fortunately, suicide death was infrequent even among those with high predicted risk. While risk among visits above the 99^th percentile was 80 times higher for visits below the 50^th percentile, the observed rate for visits above the 99^th percentile was still only slightly higher than 1 in 200. That absolute risk or positive predictive value is certainly not high enough to determine clinical decisions regarding intensive or aggressive interventions such as immediate hospitalization or involuntary treatment. Statistical predictions could, however, identify patients in need of additional assessment, safety planning, or more active follow-up care²⁷. For comparison, a mortality rate of 1 in 200 is similar to head injury mortality rates among adult²⁸ and pediatric²⁹ emergency department patients identified by widely accepted decision rules recommending additional imaging assessment after minor head injury. National quality metrics have long recommended timely follow-up after an emergency department visit for a mental health condition³⁰, and failure to attend follow-up is associated with higher risk of subsequent suicide death³¹. Nevertheless , fewer than half of patients in US commercial health plans (and fewer than 40% insured by Medicaid or Medicare) complete a follow-up visit within 7 days³⁰. Prediction models could identify patients for whom more vigorous efforts to arrange and assure follow-up care are needed. We focus on predicting risk over 90 days following an emergency department visit, because that is the period during which interventions such as safety planning and assurance of follow-up care might occur or act to reduce risk.

The relatively rarity of suicide death guarantees than even a highly accurate prediction model would yield a high rate of false positives. Consequently, any implementation should focus on interventions with low potential for harm or infringement of autonomy.

The most influential predictors of suicide death (Table 5) included many expected risk factors, such as prior depression diagnoses, prior injury and poisoning events, and prior responses to item 9 of the PHQ-9 regarding suicidal ideation. Those expected risk factors were sometime selected through interactions with sex, age, or race and ethnicity. We should caution against over-emphasizing or over-interpreting any individual predictors, since models include large numbers of overlapping or correlated predictors. We should especially caution against any causal interpretation of individual risk factors. For example, the finding that absence of any prior lithium use was associated with lower risk likely reflects confounding or selection (people at low risk are less likely to be prescribed lithium) than a causal effect (use of lithium increases risk of suicide death).

Overall classification performance of this and similar prediction models^{8, 9, 14} appears superior to reported performance of self-report questionnaires for prediction of suicide attempt^{6, 7} or suicide death⁵ following emergency department care. In any clinical implementation, however, prediction models would supplement rather than replace or override self-report questionnaires or clinician assessments^{8, 13}. Empirical comparison of screening questionnaire results and statistical models for predicting risk of self-harm finds that the combination of the two methods is superior to either alone^7–9.

A model or algorithm including approximately 500 predictors extracted from health records could be challenging to implement in routine care. We have previously reported that, for prediction of suicidal behavior following mental health outpatient visits, simpler models considering only a few hundred potential predictors performed approximately as well as more complex models considering several thousand predictors³². Simpler models might perform as well in the emergency department setting, but that question must be addressed empirically.

This prediction model does not consider other sources or types of data that might contribute to prediction of suicide death. Text of clinical notes, during or before the index encounter, might include additional information regarding suicidal ideation, history of suicidal behavior, early life experiences, or recent negative events – all of which might improve prediction of suicidal behavior^{33, 34}. Medical records would not capture information regarding risk factors not reported to treating clinicians or events occurring between the emergency department encounter and a subsequent self-harm event.

The performance or potential utility of this prediction model depends on the availability of comprehensive records data for people seen in emergency departments. Records data in these health systems capture both care provided in health system facilities (via EHR databases) and care provided at outside facilities (via insurance claims databases). We expect that prediction models would be less accurate in settings where complete records of care prior to an emergency department encounter are not available. Reeves and colleagues¹⁵ reported somewhat poorer overall performance of a prediction model using data from a single emergency department visit.

We cannot speculate how the performance of this specific prediction model would generalize to other health systems, other patient populations, or later time periods. While prediction performance was generally similar across various subgroups in this sample of visits, prediction performance in other settings could differ due to differences in patients served, differences in patterns of health service use, or differences in health records systems. Models developed in these health systems to predict suicide attempts after outpatient visits have performed similarly in other settings^{35, 36} and in subsequent time periods³⁷, but similar testing for generalizability or portability would be necessary for models predicting suicide mortality after emergency department visits. Approximately 0.1% of emergency department encounters were excluded due to death by cause other than suicide withing 90 days, and our findings may not apply to that group.

CONCLUSIONS

Among visits to emergency departments with mental health, substance use disorder, or self-harm diagnoses, discrete data elements extracted from health system records can accurately classify risk of suicide death over the following 90 days. While these predictions are not accurate enough to direct control decisions regarding need for hospitalization or involuntary treatment, they can assist clinicians in identifying patients needing higher levels of follow-up care.