Effect of an Emergency Department Process Improvement Package on Suicide Prevention

Key Points

Question

Can a process improvement package in the emergency department targeting suicide prevention using evidence-based practices reduce subsequent suicidal behaviors among patients at risk of suicide?

Findings

In this randomized clinical trial including 2761 patients, after a 12-month implementation period, using an interrupted time series model, a significant decrease in incidence of suicidal attempts, ideation, and behavior was observed across the 3 phases of the study (baseline, implementation, and maintenance).

Meaning

Department-wide interventions aiming to improve clinic workflow can reduce likelihood of suicide, suicidal ideation, and suicide-related behaviors in the 12 months following the patient’s initial ED visit.

This randomized clinical trial determines if an emergency department process improvement package, with a subfocus on improving the implementation of collaborative safety planning, reduces subsequent suicide-related behaviors.

Abstract

Importance

Suicide is a leading cause of deaths in the US. Although the emergency department (ED) is an opportune setting, ED-initiated interventions remain underdeveloped and understudied.

Objective

To determine if an ED process improvement package, with a subfocus on improving the implementation of collaborative safety planning, reduces subsequent suicide-related behaviors.

Design, Setting, and Participants

The Emergency Department Safety Assessment and Follow-up Evaluation 2 (ED-SAFE 2) trial, a stepped-wedge cluster randomized clinical trial conducted in 8 EDs across the US, used an interrupted time series design with three 12-month sequential phases: baseline, implementation, and maintenance. A random sample of 25 patients per month per site 18 years and older who screened positive on the Patient Safety Screener, a validated suicide risk screener, were included. The primary analyses focused on those who were discharged from the ED, while secondary analyses focused on all patients who screened positive, regardless of disposition. Data were collected on patients who presented for care from January 2014 to April 2018, and data were analyzed from April to December 2022.

Interventions

Each site received lean training and built a continuous quality improvement (CQI) team to evaluate the current suicide-related workflow in the ED, identify areas of improvement, and implement efforts to improve. Each site was expected to increase their universal suicide risk screening and implement collaborative safety planning for patients at risk of suicide who were discharged home from the ED. Site teams were centrally coached by engineers experienced in lean CQI and suicide prevention specialists.

Main Outcomes and Measures

The primary outcome was a composite comprising death by suicide or suicide-related acute health care visits, measured over a 6-month follow-up window.

Results

Across 3 phases, 2761 patient encounters were included in the analyses. Of these, 1391 (50.4%) were male, and the mean (SD) age was 37.4 (14.5) years. A total of 546 patients (19.8%) exhibited the suicide composite during the 6-month follow-up (9 [0.3%] died by suicide and 538 [19.5%] of a suicide-related acute health care visit). A significant difference was observed for the suicide composite outcome between the 3 phases (baseline, 216 of 1030 [21%]; implementation, 213 of 967 [22%]; maintenance, 117 of 764 [15.3%]; P = .001). The adjusted odds ratios of risk of the suicide composite during the maintenance phase was 0.57 (95% CI, 0.43-0.74) compared with baseline and 0.61 (0.46-0.79) compared with the implementation phase, which reflect a 43% and 39% reduction, respectively.

Conclusions and Relevance

In this multisite randomized clinical trial, using CQI methods to implement a department-wide change in suicide-related practices, including the implementation of a safety plan intervention, yielded a significant decrease in suicide behaviors in the maintenance period of the study.

Trial Registration

ClinicalTrials.gov Identifier: NCT02453243

Introduction

With 44 834 suicides in 2020, or 123 per day, suicide remains the 10th leading cause of death in the US.^1,2,3 Suicide-related emergency department (ED) visits have been steadily increasing for nearly 2 decades, resulting in close to 1.5 million such visits.⁴ In addition to the 1.1% of all ED visits directly related to suicidal ideation or behavior,^4,5,6 3% to 8% of medical ED patients endorse suicidal ideation or behavior when screened,^7,8,9,10,11 making the ED a suicide risk–enriched health care setting that offers a unique opportunity to prevent suicide.

While suicide prevention organizations, such as the National Action Alliance for Suicide Prevention¹²and The Joint Commission,¹³ view EDs as essential settings for suicide prevention, care provided in the ED often neglects to fully incorporate suicide-related best practices. In a study of more than 10 000 patients presenting to the ED for intentional self-injury, one-half of Medicaid patients and approximately one-third of patients with private insurance who were discharged home were not evaluated by a mental health professional during their visit.¹⁴ Furthermore patients presenting with other confounding diagnoses may affect the quality of suicide-related care they receive. For example, individuals presenting with acute alcohol intoxication are less likely to receive a thorough risk evaluation when presenting to the ED.¹⁵ Notably, when patients are not evaluated, they demonstrate an increased probability of repeat visits for suicidal behavior following their discharge.^16,17 Surveys of ED physicians and nurses suggest that only a minority feel confident in their ability to provide best practices for suicide prevention, such as collaborative safety planning, lethal means safety counseling, and referral to appropriate outpatient behavioral health care.^2,3 Reinforcing this survey data, the Emergency Department Safety Assessment and Follow-up Evaluation (ED-SAFE) study^7,18,19 a large, National Institute of Mental Health–funded pragmatic suicide prevention trial, found profound deficiencies in usual care among 8 geographically diverse EDs. Of the 497 suicidal patients enrolled who received treatment as usual (TAU), none reported developing a safety plan with a clinician, only 40% had any documentation in their ED record of having been asked about access to lethal means, and less than 10% of those discharged had an appointment scheduled with a behavioral health specialist prior to discharge.²⁰

The deficits in suicide-related evidence-based practice delivery are difficult to remediate because the ED is a complex treatment setting characterized by high patient volume, patient heterogeneity, and a need for both fast and efficient care. Furthermore, EDs often face a range of staffing issues, including high turnover, inadequate training in suicide-related care, and insufficient access to behavioral health specialists.^3,21 Simply training ED clinicians to implement a new intervention is unlikely to lead to clinician behavior change toward suicide risk management. Instead, a continuous quality improvement (CQI) strategy is needed to ensure consistency, effectiveness, and routine use of new procedures and interventions.^22,23,24,25 A variety of CQI strategies have been used in health care. They generally share several key components, including iterative cycles of improvement; building multidisciplinary teams; engaging stakeholders from the target setting (clinicians and patients); use of quality tools, such as process mapping where clinical workflow is described and depicted visually; recognition of the importance of leadership commitment; and active engagement of frontline staff throughout the process. Standardization of roles, tools, and process has played a significant role in CQI by reducing unwanted variation and sharing best practices.^26,27 The Zero Suicide framework,²⁸ the leading model for improving suicide care in health care settings, includes CQI as a fundamental performance element. The results of using CQI to improve suicide-related care in the Henry Ford Health System²⁹ and others^30,31 have found reductions in suicide by up to 75%.

The original ED-SAFE study used a CQI approach to implement universal suicide risk screening and a multicomponent ED-initiated suicide prevention intervention for adults. The study concluded that (1) it was feasible to implement universal suicide screening for adults across 8 geographically and sociodemographically diverse EDs; (2) implementing universal screening improved suicide risk detection from 2.9% to 5.7%⁷; and (3) the multicomponent intervention resulted in a 30% relative reduction in suicidal behavior compared with treatment as usual.¹⁸ The present study, ED-SAFE 2,²⁶ aims to evaluate whether a CQI approach to improving suicide-related care would result in improved patient outcomes across the same 8 EDs as the original study.

Methods

ED-SAFE 2 was a pragmatic clinical trial with aims pertaining to further improving suicide risk screening and detection, as well as improving post-ED suicide-related outcomes through delivery of best-practice suicide-related care, with a primary focus on collaborative safety planning. A detailed description of the ED-SAFE 2 study methodology has been published elsewhere,²⁶ and the trial protocol can be found in Supplement 1. This article focuses on analyzing the primary post-ED suicide-related outcomes (ie, the main effect); other articles will report outcomes related to suicide risk screening and detection, as well as the impact on delivery of suicide-related best-practices (ie, mediators).

Study Design and Settings

A stepped-wedge design³² was used, wherein each of the 8 EDs was randomly assigned to one of 4 cohorts, and each cohort was randomized to a start date separated by 2 months. The 8 EDs were diverse, ranging in size and geographic location (eTable 1 in Supplement 2). They were selected based on their participation in the original ED-SAFE study and their willingness to continue introducing further suicide-related performance improvements. The study was approved by the Institutional Review Board at UMass Chan Medical School and each site’s institutional review board. Data for individual patients were collected retrospectively by medical record review, which allowed for written informed consent to be waived. Data were collected across three 12-month phases: baseline, implementation, and maintenance. The earliest data were collected on patients who presented for care in January 2014, and the last data were collected on patients who presented in April 2018. Data were analyzed from April to December 2022. During each of the 3 phases, each month, 25 patients with positive findings on the suicide risk screener (ie, yes to active ideation in the past 2 weeks or suicide attempt in the past 6 months) were randomly selected from each site to be included in the study sample, yielding 2400 reviewed medical records per phase. The hospital associated with one of the sites closed during the maintenance phase. Consequently, we have 8 sites’ data for the baseline and implementation phases but only 7 for the Maintenance phase (2100 reviewed medical records).

Study Phases

The baseline phase acted as the TAU control condition. It occurred after the final phase of the original ED-SAFE study was completed, thereby reflecting the state of the ED after the support and attention provided by the original study was completed. This fallow phase had no external grant support or special attention to suicide risk. However, care was already improved, so the baseline phase should be considered an enhanced TAU relative to EDs that had not already gone through a CQI process to implement universal screening and improve ED care.

During the implementation phase, lean, a CQI approach described below, was used to sustain improvements observed during the original ED-SAFE study, such as universal screening, and to introduce additional improvements. During the maintenance phase, sites focused on consolidating and sustaining gains observed during the implementation phase and on continued improvement of care processes. This involved evaluating the effectiveness of the training and the compliance to the intervention as well as adjusting or abandoning any countermeasures established during implementation.

Implementation Strategy and Intervention

Lean

Lean is a CQI strategy that has been adopted widely in health care. Each ED chose a site lead that oversaw the effort and convened a multidisciplinary lean team. The site team’s members had representation from frontline staff, management, information technology/information support, patient safety, and quality assurance. At the start of the implementation phase, each team received 1-day, on-site training in lean principles from a doctoral-trained industrial engineer with lean expertise. During the training and monthly follow-up expert coaching calls, the team evaluated their ED’s suicide-related workflow, identified gaps between current and optimal care, designed solutions to close gaps, and oversaw the implementation using an iterative approach guided by metrics on key performance indicators they collected. Site-level CQI activities were reported to the coordinating center monthly. Consistent with CQI principles that emphasize solutions addressing root causes associated with process problems, sites could identify as many domains of improvement as necessary. Each had to implement 2 intervention elements, described below.

Screening

Each site had to sustain and improve the quality of universal suicide risk screening that was implemented during the original ED-SAFE. The outcomes of this effort will be reported in a separate article.

Collaborative Safety Planning

Each site had to implement a version of collaborative safety planning (the Safety Planning Intervention [SPI])^33,34,35 for patients who screened positive for suicide risk but who were discharged from the ED. The SPI involves a trained clinician creating a 6-step safety plan collaboratively with the patient to help the individual manage suicidal crises. An SPI codeveloper and master trainer (Barbara H. Stanley, PhD, or G. K. B.) visited each ED in person and provided training to clinicians and at least 1 site trainer. The training used didactics, demonstration, and participatory role playing. Each site trainer trained other clinicians and helped oversee SPI implementation and quality assurance. The SPI trainers met with the sites monthly and, as needed, reviewed deidentified safety plans to assess fidelity and provided feedback to the sites. Site trainers had to establish competence in SPI through role playing using a standardized patient that was independently rated by the site’s SPI master trainer using the Safety Plan Intervention Rating Scale.³⁶ Cycles of training and review of role plays occurred until the site trainer scored above the satisfactory threshold on the Safety Plan Intervention Rating Scale.

Participants

For the baseline phase, 2400 patient encounters (for patients 18 years and older) were selected and reviewed, with 1030 (42.9%) being discharged from the ED. For the implementation, 2400 encounters were selected and reviewed, with 967 (40.3%) being discharged. For the maintenance phase, because 1 site closed, only 2100 encounters were selected and reviewed, with 764 (36.4%) being discharged. Race and ethnicity data were self-reported and documented by clinicians in the medical record. See Figure 1 for a CONSORT diagram depicting the cohort sample sizes.

Figure 1. Randomized Interrupted Time Series CONSORT Diagram.

Only patients discharged from the emergency department (ED) to home or community and without another ED visit within 6 months were included.

Outcomes

A range of system, clinician, and patient outcomes were assessed, as described by Boudreaux et al.²⁶ The current article presents data on the primary suicide-related composite outcome (the main effect) defined as (1) suicide-related acute care, including ED visits and hospitalizations related to suicidal ideation or suicidal attempts, or (2) death by suicide within 6 months of the index visit. This was ascertained through a structured review of the medical records associated with the participating health system by trained research staff as well as a review of the National Death Index. The US Centers for Disease Control’s uniform definitions³⁷ for self-directed violence guided the coding manual. The primary analytic sample included patients who screened positive for suicide risk and were discharged from the ED, since this group was safety planning targeted.³⁵ The primary outcome was represented as a binary outcome: having a suicide composite outcome over the 6 months after the ED visit (yes or no). A secondary outcome comprising the total number of episodes of suicide-related acute care was calculated (continuous).

Statistical Analysis

Our analytic strategy outlined in Boudreaux et al²⁶ described evaluating the effect of the care process improvements on suicide outcomes and evaluating care processes, especially safety planning, as moderators and/or mediators. Our analyses reported herein evaluate the effect on the suicide composite outcome across and within the 3 phases, ie, the main effect. Future articles will evaluate processes of care as moderators or mediators of the effect on suicidal behavior.

Our primary analyses tested our hypothesis that suicide composite outcomes would be reduced and sustained after implementation of improved suicide-related evidence-based practices. It occurred in 5 steps: (1) descriptive statistics; (2) comparing patient and encounter characteristics across phases; (3) comparing suicide composite and constituent elements across phases; (4) comparing incident rates across phases; and (5) generalized estimating equation (GEE) models for suicide composite across phases. We used a multimethod approach to both provide a validation of the trends and present different perspectives on the results, such as evaluating aggregate phase results vs within phase changes. Logistic regression models adjusting for covariates were used to confirm results from the χ² tests.

To examine change of suicide composite outcome over time, particularly postintervention, an interrupted time series analysis was applied. The model was constructed using an interrupted time series paradigm to investigate the change in odds ratios (ORs) across phases in the suicide composite outcome and the within-phase slope changes, both unadjusted and adjusted. The model fit the suicide-related composite outcome (yes or no) as a dependent variable; independent variables include time (month for each phase) and a time indicator for the original and new phase. Two separate models were conducted: one included participants in the baseline and implementation phases to evaluate the early effect of the CQI intervention compared with baseline, and the second model included participants in the implementation and maintenance phases to evaluate the sustainability of the early intervention effects.

We completed all analyses using only index encounters of patients identified as having suicide risk who were discharged from the ED, the primary target population. Because patients often re-present to the ED, it was possible for them to have been randomly selected multiple times. This means that analyses could be conceptualized at the encounter level or at the patient level. We used a blackout window of 6 months after the patient’s first index visit such that any repeated visit that might have been randomly selected during the 6-month outcome window was no longer included. To address this, we used κ statistics to test the probability that a patient had a second index visit 6 months after their first index visit compared with the probability of a new patient having an initial index visit, with the expectation that if the agreement was low, then encounters could be viewed as independent. Results indicated that agreement between suicide composite values based on the encounter-level cohort and the patient-level cohort were low (κ of approximately 0.4; eTable 2 in Supplement 2). For simplicity, we present the encounter-level analyses. Patient-level analyses are available by request.

P values were calculated using χ², t, and log-rank tests as well as GEE logic analysis and negative binomial regression tests, as appropriate. Significance was set at P < .05, and all P values were 2-tailed. Analyses were performed using SAS version 9.4 (SAS Institute) and Stata version 17 (StataCorp).

Results

Descriptive Statistics

Across 3 phases, 2761 patient encounters were included in the analyses. Of these, 1391 (50.4%) were male, and the mean (SD) age was 37.4 (14.5) years. Table 1 shows the descriptive statistics for the target sample. Most patients were insured (2344 [84.9%]), had a depression diagnosis (2467 [89.4%]), and endorsed suicidal ideation in the past 2 weeks prior to the index visit (2314 [83.8%]). A total of 1115 (40.4%) had a lifetime suicide attempt, and 176 (6.4%) had a suicide attempt in the past 24 hours prior to the index visit.

Table 1. Descriptive Statistics and Comparison Across Phase.

Characteristic	Patient encounters, No. (%)				P value
	Overall (N = 2761)	Baseline phase (n = 1030)	Implementation phase (n = 967)	Maintenance phase (n = 764)
Study site
Maricopa	198 (7.2)	70 (6.8)	62 (6.4)	66 (8.6)	<.001
Rhode Island	196 (7.1)	107 (10.4)	89 (9.2)	0
Ohio	533 (19.3)	198 (19.2)	224 (23.2)	111 (14.5)
Arkansas	232 (8.4)	84 (8.2)	77 (8.8)	71 (9.3)
Beth Israel	325 (11.8)	101 (9.8)	106 (11)	118 (15.4)
Nebraska	379 (13.7)	121 (11.7)	139 (14.4)	119 (15.6)
Colorado	544 (19.7)	195 (18.9)	182 (18.8)	167 (21.9)
Marlborough	354 (12.8)	154 (15)	88 (9.1)	112 (14.7)
Demographic characteristics
Sex, No.	2761	1030	967	764
Male	1391 (50.4)	510 (49.5)	495 (51.2)	386 (50.5)	.51
Female	1370 (49.6)	520 (50.5)	472 (48.8)	378 (49.5)
Age, mean (SD), y	37.4 (14.5)	38.8 (14.4)	36.7 (14.5)	36.5 (14.5)	.001
Race, No./total No. (%)a
African American	537/2706 (19.8)	215/1010 (21.3)	181/950 (19.1)	141/746 (18.9)	.01
American Indian or Alaska Native	21/2706 (0.8)	3/1010 (0.3)	4/950 (0.4)	14/746 (1.9)
Asian	41/2706 (1.5)	13/1010 (1.3)	14/950 (1.5)	14/746 (1.9)
Native Hawaiian or Other Pacific Islander	2/2706 (0.1)	1/1010 (0.1)	0/950 (0)	1/746 (0.1)
White	1934/2706 (71.5)	721/1010 (71.4)	692/950 (72.8)	521/746 (69.8)
Other race	171/2706 (6.3)	57/1010 (5.6)	59/950 (6.2)	55/746 (7.4)
Ethnicity
Hispanic	308 (11.2)	106 (10.3)	108 (11.2)	94 (12.3)	<.001
Non-Hispanic	2352 (85.2)	852 (82.7)	839 (86.8)	661 (86.5)
Not documented	101 (3.7)	72 (7)	20 (2.1)	9 (1.2)
Type of insurance
Insured	2344 (84.9)	856 (83.1)	841 (87)	647 (84.7)	.05
Uninsured/not documented	417 (15.1)	174 (16.9)	126 (13)	117 (15.3)
Psychiatric disorder, No.
Depression	2467 (89.4)	890 (86.4)	874 (90.4)	703 (92)	<.001
Bipolar	124 (4.5)	21 (2)	61 (6.3)	42 (5.5)	<.001
Anxiety	240 (8.7)	56 (5.4)	102 (10.5)	82 (10.7)	<.001
Schizophrenia	92 (3.3)	48 (4.7)	22 (2.3)	22 (2.9)	.009
Suicide ideation and attempt history at baseline
Suicide ideation in past 2 wk	2314 (83.8)	822 (79.8)	823 (85.1)	669 (87.6)	<.001
Lifetime suicide attempt	1115 (40.4)	379 (36.8)	439 (45.4)	297 (38.9)	<.001
Attempt in past 24 h	176 (6.4)	46 (4.5)	72 (7.4)	58 (7.6)	.002
Attempt in past mo	176 (6.4)	54 (5.2)	76 (7.9)	46 (6)	.009
Attempt in past 6 mo	315 (11.4)	114 (11.1)	135 (14)	66 (8.6)	<.001
Attempt longer than 6 mo	448 (16.2)	165 (16)	156 (16.1)	127 (16.6)	.48

^a Race and ethnicity data were self-reported and documented by clinicians in the medical record.

Table 1 also shows that patients were not evenly distributed among sites, with some sites having much smaller representation. This was due to large variability in discharge rates across sites. In addition, participants were slightly older, slightly less likely to be insured, and slightly less likely to be Hispanic in the baseline phase. Slight differences in the presence of psychiatric disorders were noted across phases. These baseline differences were adjusted for in the final GEE models, Cox models, and negative binomial regression model.

Suicide Composite and Elements

Table 2 and Figure 2 summarize the suicide composite outcome and constituent variables for the entire sample and across the 3 phases. Of the 2761 encounters, 546 (19.8%) had 1 suicide-related event (ED visit for ideation or attempt or suicide death) during the 6-month follow-up period, including 9 patients (0.3%) who died by suicide and 538 (19.5%) with an ED visit due to suicidal ideation or an attempt. The events occurred in the following time intervals related to the index visit: between 1 and 4 weeks (306 [56.0%]), 4 and 6 weeks (72 [13.2%]), 6 and 12 weeks (52 [9.5%]), and 12 and 26 weeks (116 [21.2%]).

Table 2. Suicide Composite Outcome and Comparison Across Phases.

Outcome	Patient encounters, No. (%)				χ2	P value
	Overall (N = 2761)	Baseline phase (n = 1030)a	Implementation phase (n = 967)	Maintenance phase (n = 764)
Suicide compositeb
Yes	546 (19.8)	216 (21)	213 (22)	117 (15.3)	χ22 = 13.60	.001
No	2215 (80.2)	814 (79)	754 (78)	647 (84.7)
Component of suicide composite
Suicide death (most conservative)c
Yes	4 (0.1)	1 (0.1)	2 (0.2)	1 (0.1)	NA	.84
No	2757 (99.9)	1029 (99.9)	965 (99.8)	763 (99.9)
Suicide death (less conservative)d
Yes	9 (0.3)	2 (0.2)	4 (0.4)	3 (0.4)	NA	.70
No	2752 (99.7)	1028 (99.8)	963 (99.6)	761 (99.6)
ED visit related to suicide attempt
Yes	131 (4.7)	52 (5.0)	54 (5.6)	25 (3.3)	χ22 = 5.38	.07
No	2630 (95.3)	978 (95.0)	913 (94.4)	739 (96.7)
ED visit related to suicidal ideation alone
Yes	407 (14.7)	163 (15.8)	155 (16.0)	89 (11.6)	χ22 = 8.05	.02
No	2354 (85.3)	867 (84.2)	812 (84.0)	675 (88.4)
ED visit related to ideation or attempt
Yes	538 (19.5)	215 (20.9)	209 (21.6)	114 (14.9)	χ22 = 14.20	.001
No	2223 (80.5)	815 (79.1)	758 (78.4)	650 (85.1)

Abbreviations: ED, emergency department; NA, not applicable.

^a This table includes only participants discharged home from the ED, the primary target sample.

^b The suicide composite outcome was defined as suicide death within 6 months after the index date, any ED or inpatient visit with an International Classification of Diseases code reflecting suicidal ideation or attempt, or research assistant abstraction noting that suicidal ideation or suicidal behavior was present during the visit.

^c Death defined by International Classification of Diseases code of intentionally killing self.

^d Death defined by suicide International Classification of Diseases code or International Classification of Diseases code for self-inflicted injury, undetermined/accident poisoning, or other events of undetermined intent.

Figure 2. Change of Site-Aggregate Suicide Composite Rate Postintervention for Patients With Suicide Risk Encounters Discharged From the Emergency Department.

The blue lines indicate the change in site-aggregated suicide composite rate postintervention.

The suicide composite outcome differed significantly between the 3 phases (baseline, 216 of 1030 [21%]; implementation, 213 of 967 [22%]; maintenance, 117 of 764 [15.3%]; P = .001). This appeared to be driven largely by the differences in suicide-related ED visits, as deaths across the phases were very small. Pairwise tests of the phases indicated composite rates from the maintenance phase were significantly lower than rates from both the implementation phase (χ²₂ = 12.5; P < .001) and baseline phase (χ²₂ = 9.3; P = .002), but the implementation phase did not differ from the baseline phase (χ²₂ = 0.33; P = .57). Similar results were observed from multivariable logistic analysis. The adjusted ORs of risk of the suicide composite during the maintenance phase was 0.57 (95% CI, 0.43-0.74) compared with baseline and 0.61 (0.46-0.79) compared with the implementation phase, which reflect a 43% and 39% reduction, respectively (eTable 3 in Supplement 2). The results of the interrupted time series were plotted for a visual inspection; Figure 1 illustrates the change of average site aggregate rate over time (month) after patient enrollment and parameters postintervention. Figure 2 illustrates that the suicide composite reduced at 0.49% per month over time after implementation of the safety measures, which translates to approximately a 6% suicide composite rate reduction over a 12-month period between the implementation and maintenance phases. This reduction was comparable with rates in Table 2, where suicide composite rates changed from 22% at the implementation phase to 15.3% at the maintenance phase.

Interrupted Time Series GEE Models

Table 3 presents interrupted time series analysis results. The results of model 1 indicate that there was no month-to-month change of OR within the baseline phase (OR, 0.99; 95% CI, 0.95-1.03; P = .64) but a level increase after transition from the baseline to implementation phase (OR, 2.81; 95% CI, 1.18-6.70; P = .02). Although the suicide composite rate gradually decreased over the implementation phase (Figure 1), this change did not reach statistical significance (OR, 0.96; 95% CI, 0.90-1.02; P = .21).

Table 3. Odds Ratio (OR) Change Between Phases and the Within-Phase Slope Change for the Suicide Composite.

Measure	OR (95% CI)	P value
Unadjusted
Model 1: baseline vs implementation phase
Baseline phase month-to-month change	0.99 (0.95-1.03)	.64
Level change after transition to intervention	2.81 (1.18-6.70)	.02
Implementation phase month-to-month change	0.96 (0.90-1.02)	.21
Model 2: implementation vs maintenance phase
Implementation phase month-to-month change	0.95 (0.91-0.99)	.03
Level change after transition to maintenance	0.44 (0.14-1.40)	.17
Maintenance phase month-to-month change	1.09 (1.01-1.17)	.02
Adjusted a
Model 1: baseline vs implementation phase
Baseline phase month-to-month change	0.97 (0.93-1.02)	.24
Level change after transition to intervention	3.05 (1.22-7.62)	.02
Implementation phase month-to-month change	0.97 (0.91-1.04)	.40
Model 2: implementation vs maintenance phase
Implementation phase month-to-month change	0.95 (0.90-0.995)	.03
Level change after transition to maintenance	0.33 (0.10-1.13)	.08
Maintenance phase month-to-month change	1.11 (1.03-1.20)	.009

^a Adjusted for age, race and ethnicity, site, suicide ideation in the past 2 weeks, lifetime suicide attempt, depression, bipolar, anxiety, and schizophrenia.

The results of model 2 showed a month-to-month OR decrease within the implementation phase (OR, 0.95; 95% CI, 0.91-0.99; P = .03), no level change after transition from the implementation to maintenance phase (OR, 0.44; 95% CI, 0.14-1.40; P = .17), and a slight month-to-month increase within the maintenance phase (OR, 1.09; 95% CI, 1.01-1.17; P = .02). The results are not altered after adjusting for covariates.

Discussion

To our knowledge, the ED-SAFE 2 study is the largest pragmatic clinical trial to study implementation of suicide-related evidence-based practices in the ED, identifying 2761 encounters with a suicide-related index visit that led to discharge. It differs from most other clinical trials on brief ED-based interventions by virtue of our delivery of the interventions by routine clinical staff, not research clinicians, creating strong external validity. Our aggregated analysis comparing participants treated during each of the 3 phases revealed that the probability of having a suicide composite outcome in the 6 months after an index visit were significantly lower during the maintenance phase compared with the baseline phase (117 of 764 [15.3%] vs 216 of 1030 [21.0%]; P = .001). The improvement in the suicide composite, while relatively small in absolute terms, was confirmed using Cox proportional hazards and survival analyses (eResults, eTable 4, and the eFigure in Supplement 2). Interestingly, the implementation phase exhibited an overall steady decrease in suicide composite rates over time. This finding did not reach statistical significance in the early stage of the implementation phase; however, later in the implementation phase through the early stage of the maintenance phase, the decreased ORs became statistically significant. This is important because it suggests the impact of the CQI intervention may be delayed, and the effect may be incremental and extended over a long timeline. This is consistent with the fundamental nature of CQI approaches. One does not simply go live with the full intervention effect realized immediately; rather, the process involves implementation, measurement, and iterative improvement. Consequently, we would expect the effect to phase in over time, which is an important methodologic consideration when planning future trials and argues for a long period of time to study the transition to full implementation. Similarly, implementation can wane over time due to a variety of factors, such as staff turnover, fidelity drift, or fatigue. It is equally important to factor in sufficient time to evaluate maintenance of intervention effects. Our results suggested a ramp up to the intervention effect followed by a potential weakening of effect at the end of the maintenance phase, complicating the overall intervention effect size. Future analyses will evaluate the processes associated with both effects, including examining if gradual improvements in processes of care account for the escalating intervention effect.

Between the baseline and implementation phase, we report an absolute reduction in incidence of the suicide composite of 5%, although it was not significant. This finding is in line with previously published research on therapeutic interventions, such as a 2015 meta-analysis focusing on suicide attempts and self-harm in adolescents³⁸; they report a 4% absolute risk reduction among the 19 studies synthesized. Our number needed to treat to prevent future suicidal behavior comparing composite rates in the baseline phase with the maintenance phase was 25.

When comparing the maintenance phase with the baseline, we observed a relative risk reduction of approximately 30%. The magnitude of this change is consistent with the original ED-SAFE study, which found a relative risk reduction of 20%. The effect size is larger than the magnitude of change observed in other studies of brief ED interventions (eg, caring contacts, safety planning).^39,40 The relative risk reduction is particularly notable because it was attained even when considering that the baseline phase was an enhanced TAU phase, as all EDs had already progressed through the original ED-SAFE. This creates greater confidence that the observed effect was not due to chance or other nonspecific experimental factors. Further, this effect size is respectable in the face of the known voltage drop that occurs when implementing evidence-based practices into real-world settings that were originally derived in highly controlled randomized clinical trials.⁴¹

Limitations

Our study has several limitations. We did not randomize by patient or by cluster, so historical or secular changes may have contributed to our outcomes. To address this, we controlled for potential differences in samples and time by using multiple covariates, but it remains possible that residual confounding occurred. Additionally, by using a stepped-wedge design, we further reduced potential bias. Notably, national suicide rates and suicide-related ED visits remained constant or increased over the study period.¹ Second, we had insufficient power to detect differences in deaths by suicide or attempts, an unavoidable problem in most suicide research because of the rarity of suicide death. Third, having a very diverse sample is a strength because our sample has adequate representation from historically underrepresented racial and ethnic minority groups. However, it can be viewed as a weakness because the final sample does not reflect the racial and ethnic composition of the US in general. Fourth, we relied on electronic medical records to identify eligible suicide composite outcomes, thereby potentially underestimating rates.

Conclusions

The current study demonstrates the effectiveness of CQI strategies to implement suicide prevention practices in the ED. Numerous evidence-based practices for suicide prevention have been developed by researchers but have not been widely or sustainably implemented. CQI represents an implementation approach that is congruent with the culture of the US health care setting and allows EDs to own suicide-related practice changes and sustain them over time.

Supplement 1.

Trial Protocol

Supplement 2.

eResults.

eTable 1. Site Characteristics

eTable 2. Kappa Statistic for the Agreement Between Encounters and Patients Who Had a Suicide-Related ED Visit and Were Discharged From the ED

eTable 3. Logistic Regression Analysis of the Suicide Composite Across Study Phases

eTable 4. Hazard Ratio of the Suicide Composite Outcome by Study Phase for Patients With Suicide Risk Encounters Discharged From the ED

eFigure. Kaplan-Meier for Time to the Suicide Composite Event for Encounters Discharged From the ED Stratified by Study Phase

Supplement 3.

Group Information. ED-SAFE 2 Investigators

Supplement 4.

Data Sharing Statement