Implementation of Passive Deterioration Index Alerts in an Intermediate Care Unit: A Failed Early Warning System Strategy

Abstract

Objectives

Traditional early warning systems (EWS) have shown uncertain efficacy in real-world settings. More recently, machine learning models like the Epic Deterioration Index (DTI) have been developed, promising greater accuracy. Recognizing the potential of DTI, but also the pervasive issue of alert fatigue with interruptive (i.e., pop-up) EWS alerts, our institution implemented a DTI-enabled EWS with passive alerts (colored icons visible in prespecified locations within the electronic health record). We hypothesized that our intervention would reduce the time to treatment for deteriorating patients.

Methods

We piloted a DTI-enabled EWS in a 30-bed intermediate care unit at a large academic medical center. DTI scores, alert icons, and vital signs appeared on a custom Patient List interface. In the event of an alert, charge nurses were expected to conduct a bedside assessment and escalate care as necessary. We compared the 111-day pre- and postimplementation periods, with alert-to-action time as the primary outcome. Secondary outcomes included mortality, length of stay, ICU transfer, documentation rate, and provider acceptance.

Results

Among 301 patients with an elevated-risk score (156 pre- and 145 postimplementation), we found no significant differences in alert-to-action time (469 vs. 359 minutes before alert; p  = 0.96), with provider actions typically occurring several hours before the alert in both periods. There were no significant differences in mortality (10.3% vs. 13.1%; p  = 0.56), length of stay (15.7 vs. 12.8 days; p  = 0.23), or ICU transfer (8.3% vs. 6.2%; p  = 0.63). Charge nurses documented acknowledgment of the alert in 18.6% of cases, and acceptance was poor. Most nurses expressed a preference for interruptive alerts and more prominent DTI display locations.

Conclusion

In this single-unit pilot, passive DTI-enabled EWS alerts did not improve time to intervention or clinical outcomes. High-risk DTI scores often occurred after clinical deterioration had already been recognized.

Background and Significance

Early warning systems (EWS) aim to detect and treat clinical deterioration in hospitalized patients, but their effectiveness is uncertain. A 2021 Cochrane review found that EWS relying on traditional risk scores had minimal effect on patient outcomes. 1 Newer machine learning models, such as the Epic Deterioration Index (DTI), adopted by hundreds of hospitals, 2 3 4 are thought to improve predictive accuracy, but evidence of improved patient outcomes is scarce. 5 6

Enthusiasm for DTI from other hospitals and informal forums prompted our institution to use it to enhance patient safety. We recognized the challenge of alert fatigue, a common issue with traditional EWS that often leads clinicians to ignore interruptive (i.e., pop-up) alerts, reducing system effectiveness. 7 8 9 To address this, our multidisciplinary workgroup (see Methods section) designed a DTI-enabled EWS with passive alerts—colored icons displayed in specific electronic health record (EHR) locations—integrated into the charge nurses' workflow. We chose this approach for two reasons: passive alerts provide critical awareness without contributing to alert fatigue, and charge nurses, with their unit-wide oversight, can facilitate timely team-based assessments for high-risk patients.

The objective of this study was to evaluate the effect of a DTI-enabled EWS with passive alerts, targeting charge nurse workflow, on the timeliness of clinical interventions for patients at high risk of deterioration in an intermediate care unit. We hypothesized that this intervention would reduce the alert-to-action time (the time between an elevated-risk alert and clinical intervention) compared to the preimplementation period.

Methods

Study Setting and Context

This research project was performed within a large, Midwestern academic medical center. Driven by the institutional goal to improve early recognition of patient deterioration while minimizing alert fatigue, and following the rationale outlined above, we selected a 30-bed intermediate care unit for our DTI-enabled EWS pilot study after obtaining buy-in from the unit's Nurse Manager and Medical Director. A multidisciplinary workgroup of nearly forty stakeholders and sponsors was convened in February 2023 to guide the implementation, co-led by a hospitalist and the Rapid Response Team (RRT) System Liaison ( Supplementary Table S1 , available in the online version only). This study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects and was approved by the University of Minnesota Institutional Review Board (STUDY00017419, “Predicting and preventing patient deterioration”) on December 2, 2022.

Model Configuration and Threshold Selection

DTI is a proprietary ordinal regression model that outputs risk scores every 15 minutes, ranging from 0 to 100. 2 Higher scores indicate a higher probability of ICU transfer, RRT, or code team activation within 12 hours, or death within 38 hours. Specific details of the variables and their weights used to predict deterioration are not available publicly, but broadly include vital signs, nursing assessments, and lab results.

Risk score thresholds at which to prompt clinical action were determined based on retrospective DTI performance in the pilot unit, using our previously described methods. 10 Positive predictive value (PPV) was further enhanced by incorporating the “delta score,” defined as the difference between the current score and the lowest score over the preceding 6 hours. Others have demonstrated that changes in the DTI score over time are prognostic of clinical deterioration. 11

We presented DTI performance metrics at varying risk score thresholds ( Supplementary Fig. S1 , available in the online version only) to the stakeholder group. Stakeholders decided which risk score threshold to use by weighing sensitivity versus PPV against implementation feasibility and workflow integration considerations. The following risk score thresholds were chosen: low (green) = absolute score less than 60 AND delta score less than 20; medium (yellow) = absolute score between 60 and 69 OR delta score 20 or more; high (red) = absolute score 70 or more OR absolute score 60 or more with delta score 10 or more.

Intervention Design and Implementation

EWS typically alert care teams to potential patient deterioration through interruptive notifications in the EHR, such as pop-up Best Practice Advisory messages. 12 However, several studies have found that interruptive deterioration alerts are frequently ignored, contributing to alert fatigue and poor adherence. 13 14 Instead, we aimed to create a workflow that utilized passive alerts (icons visible in prespecified locations within the EHR) to stimulate interdisciplinary communication and critical thinking for patients with elevated DTI risk scores. 15 16 Based on stakeholder input, we designated the unit charge nurse responsible for monitoring and acting upon elevated-risk alerts. Charge nurses have an established role and interest in overseeing unit-wide acuity to coordinate resources, adjust staffing, and support bedside nurses with assessments and care escalation when necessary.

We created a custom Patient List for the pilot unit with prespecified columns showing all patients' current DTI score, an alert icon indicating whether they were medium- or high-risk, the delta score, and vital signs ( Supplementary Fig. S2 , available in the online version only). When high-risk criteria were met, a red icon was displayed, signaling that the charge nurse should assess the patient with the bedside nurse, discuss with the primary team, and activate the RRT (if concerned) within 30 minutes. When medium-risk criteria were met, a yellow icon was displayed, signaling that the charge nurse should complete this same process within 2 hours. Charge nurses were asked to document their assessments in a flowsheet, accessed by double-clicking the alert icon. A swimlane diagram of the workflow, created through multiple iterations with our stakeholder group, is shown in Supplementary Fig. S3 (available in the online version only).

The nursing practice and education leader distributed a training tipsheet 1 week prior to go-live ( Supplementary Fig. S2 , available in the online version only). We communicated the pilot through institutional channels, including the Vice President of Medical Affairs' monthly email and the Division of Hospital Medicine's monthly meeting. After a 5-day technical acclimatization period, we launched the pilot on November 20, 2023. The project team decided to terminate the pilot after 100 days due to low acceptance. Low acceptance was determined through several indicators: (1) regular check-in meetings with unit leadership revealed persistent concerns about suboptimal workflow integration (i.e., challenging for charge nurses to remember to view DTI scores), (2) documentation rates of alert acknowledgment remained consistently low (<20%) throughout the implementation period, and (3) survey responses suggested limited perceived clinical value. Rather than continue to push a poorly accepted EWS, we decided to conclude the pilot to preserve stakeholder relationships and maintain opportunities for future EWS implementations using different approaches.

Outcome Measures

Our planned primary outcome was the time between an elevated-risk alert (medium- or high-risk) and provider action, referred to as the alert-to-action time. We define a provider as a physician or an APP. The time of an elevated-risk alert was denoted as TIME1 and included the first instance of either a medium- or high-risk alert. Provider action was defined as an appropriate order or treatment for deterioration, as determined by the team's clinical experts ( Table 1 ), within a 24-hour window (±1,440 minutes) around TIME1.

Table 1. Timing of provider actions relative to elevated-risk alerts.

	Preimplementation		Postimplementation
Provider action	n	Minutes between alert and action (median, IQR)	n	Minutes between alert and action (median, IQR)
Lactic acid ordered	71	−108 (−534, 72)	64	−116 (−541, 166)
Arterial blood gas ordered	19	−160 (−546, 158)	6	74 (−74, 210)
Venous blood gas ordered	67	−115 (−391, 621)	51	−142 (−502, 330)
Telemetry ordered	71	−127 (−434, 162)	47	−151 (−469, 274)
STAT chest X-ray ordered	59	−182 (−561, 28)	41	−299 (−590, −46)
STAT abdominal X-ray ordered	9	65 (−228, 254)	10	162 (−430, 334)
STAT CT head ordered	18	−684 (−925, 358)	16	−360 (−857, −36)
STAT CT chest ordered	7	−263 (−602, 136)	8	−667 (−1,288, −292)
STAT CT abdomen/pelvis ordered	11	90 (−610, 316)	7	−372 (−722, −285)
Fluid bolus administered	22	−116 (−225, 153)	8	156 (−377, 757)
IV antibiotic ordered	48	−180 (−488, 108)	25	−232 (−395, 74)
STAT blood product ordered	7	881 (−182, 1,057)	7	−253 (−510, 304)
CPAP/BiPAP initiated	28	442 (186, 721)	17	432 (106, 743)

Note: Negative values indicate that provider action occurred before the alert, while positive values indicate that provider action occurred after the alert.

Planned secondary outcomes included (1) mortality, length of stay, and ICU transfer within 24 hours after alert, (2) time between elevated-risk alert and charge nurse documentation, and (3) nursing acceptance. Nursing acceptance was measured via a survey among the pilot unit's charge nurses and RRT nurses. The survey contained five quantitative Likert-scale items derived from technology acceptance theories 17 18 19 and several open-ended items that elicited participants' experiences ( Supplementary Fig. S4 , available in the online version only). The survey was emailed on February 19, 2024 (3 months after go-live) and was open for 3 weeks.

Statistical Analysis

To measure our primary outcome, we defined equal pre- and postimplementation periods centered around the go-live date. Each period lasted 111 days before and after go-live. During the preimplementation period, the DTI model was running silently in the background, generating risk scores and timestamps that were not visible to providers, which allowed us to compare provider actions relative to theoretical alerts in both periods using identical methodology. We used the Mann-Whitney U test for nonnormally distributed data to assess differences in alert-to-action time and length of stay between implementation periods. We assessed differences in mortality and ICU transfer rates using chi-square tests.

Results

The pilot ran from November 20, 2023, to March 10, 2024. New alerts (medium- or high-risk) were triggered a median of 1.0 times per day preimplementation ( n  = 156, IQR: 1.0–2.0) and 1.0 postimplementation ( n  = 145, IQR: 0.0–2.0; Supplementary Fig. S5 , available in the online version only). In the preimplementation period, the first provider action occurred a median of 469 minutes (IQR: 787–121 minutes) before the alert compared to 359 minutes (IQR: 860–40 minutes) postimplementation ( p  = 0.96). The timing of individual provider actions relative to alerts is listed in Table 1 and shown collectively in Fig. 1 .

Fig. 1.

Temporal distribution of alert-to-action times before and after implementation. Alert-to-action times are measured as the first occurrence of any provider action. Histograms are offset slightly on the x-axis to enable visualization of overlapping data between the two phases. Blue bars and kernel density estimation curve represent preimplementation; orange bars and kernel density estimation curve represent postimplementation. Vertical dashed lines mark the median alert-to-action times for each phase. Negative values indicate that provider action occurred before the alert, while positive values indicate that provider action occurred after the alert.

In-hospital mortality occurred in 16 of 156 patients (10.3%) preimplementation and 19 of 145 patients (13.1%) postimplementation ( p  = 0.56). Median length of stay was 15.7 days (IQR: 7.3–32.9) preimplementation and 12.8 days (IQR: 7.0–30.6) postimplementation ( p  = 0.23). ICU transfer within 24 hours of an alert occurred in 13 of 156 patients (8.3%) preimplementation and 9 of 145 patients (6.2%) postimplementation ( p  = 0.63). Of the alerts that fired postimplementation, 27 (18.6%) were associated with any documentation, meaning that charge nurses had double-clicked the alert icon and recorded their assessment. The median time between alert and documentation was 17.0 hours (IQR: 3.16–57.6 hours).

We received 14 responses (seven charge nurses; seven RRT nurses) from 72 survey invitations (19.4% response rate). Likert-scale survey results are shown in Fig. 2 and provided numerically in Supplementary Table S2 (available in the online version only). All survey respondents indicated they would prefer predictions displayed on a different screen: in the Nurse Brain, Unit Manager, or Summary Page. Additionally, 12 of 14 respondents (86%) said they would prefer to have an interruptive alert generated in response to an elevated-risk score. Nurses pointed out that high scores commonly reflected chronic conditions rather than true worsening and that interventions were often already underway when they saw an alert. We share several illustrative free-text responses below.

Fig. 2.

Nurses' acceptance of the early warning system.

“We have lots of high o2 needs patients and other problem areas that elevated a lot of our patient scores without having them truly be deteriorating.”—Participant 2 (charge nurse)

“I enjoyed the concept of this idea, however, I did not feel the delivery was useful […] I had noted several times that patient's were declining or an RRT had been called on the pt prior to noting the change in the index score.”—Participant 3 (charge nurse)

“…usually if a pt was actively deteriorating charge & bedside RN were already aware and dealing with the changes so later, when I would remember to look at the DI dashboard having a formal 'discussion' wasn't necessary. On a med surg floor where they may not be able to communicate as much, the formal discussion would probably be helpful”—Participant 7 (charge nurse)

“So much of nursing is critical thinking, a computer generating a score (that is usually not accurate) is frankly insulting […] Nursing generates this deterioration index score fluently all day everyday, its how we prioritize patient care, its reporting critical results, its calling a rapid response, its alerting the physician to concerning symptoms or a change in patient status, we are the deterioration index.”—Participant 11 (charge nurse)

Discussion

This pilot study aimed to improve deterioration recognition and management in an intermediate care unit by integrating Epic DTI scores into charge nurses' workflows using a passive alert strategy. Our analysis revealed three critical insights: (1) alert-to-action time, mortality, length of stay, and ICU transfer rates did not change after implementation; (2) providers had already initiated treatment for deterioration before most alerts were generated; and (3) nurses reported low acceptance and found little additive clinical value in the DTI scores as presented.

A key finding of our study was that clinical interventions frequently preceded DTI alerts. The median time of first provider action was several hours before an elevated-risk alert in both pre- and postimplementation periods. This suggests that DTI was often confirming a clinical picture of deterioration that providers had already recognized and were addressing, rather than flagging previously unrecognized deterioration. Concordant results were noted in qualitative feedback from nurses who shared that high DTI scores often reflected chronic conditions or situations where interventions were already underway, a perception that has been echoed by nurses in similar work. 7 Consequently, DTI's PPV for ICU transfer in our pilot (7.3%), already at the lower end of the 6 to 17% range reported for other EWS, 20 may offer little practical value: even for the minority of predictions that correctly identify patients who deteriorate, most of these patients are likely already known to be critically ill.

Evaluating model performance relative to clinical intervention, as suggested by Winter et al, 21 who found that clinicians had acted before alerts in 90% of pediatric deterioration cases, provides a more realistic gauge of a deterioration model's clinical utility than PPV alone. 22 Given that several DTI variables represent provider-initiated interventions (e.g., supplemental oxygen) or result from provider-initiated monitoring orders (e.g., scheduled neurological assessments), DTI may function more as a diagnostic severity tool—reflecting current acuity—rather than a forward-looking predictor of future, unrecognized decline. 23 Reframing DTI as a severity score could inform more appropriate workflow integration, such as using scores to prioritize rounding or intensify monitoring for patients whose scores remain high despite interventions.

Our strategy utilized passive alerts, displaying DTI scores and icons on a custom Patient List, to reduce alert fatigue and promote team-based critical thinking. However, this approach was ineffective. Although the Patient List consolidated DTI scores and vital signs, accessing it required additional clicks, likely contributing to poor adoption. This was evident in the low documentation rate (18.6%) and unanimous nurse survey feedback requesting DTI displays in more accessible EHR interfaces, such as the Nurse Brain, Unit Manager, or Summary Page. Retrospectively, we recognized that the display location hindered usability, but modifying it during the pilot was challenging due to the project's fixed IT development cycle, which limited opportunities for real-time adjustments.

Contrary to our expectations, the majority of nurses (86%) expressed a desire for interruptive alerts, adding to ongoing uncertainty in the literature about how to optimally deliver real-time alerts. 24 While our findings support the concept that passively displayed information may be insufficient to prompt timely clinical action, 25 26 27 28 interruptive alerts are not a panacea, especially if the underlying predictive model lacks clinical utility. For instance, Bedoya et al found that despite a technically successful implementation of interruptive pop-up alerts for their National Early Warning Score-enabled EWS, poor model performance led to widespread alert dismissal (86% of alerts ignored) and no measurable clinical effect. 13 This highlights a critical trade-off: passive alerts may minimize alert fatigue but fail to change behavior if not highly visible or valued, whereas interruptive alerts risk contributing to alert fatigue unless they provide timely, accurate, and actionable information.

Our experience contrasts with that of Gallo et al, who reported reduced escalations in care associated with their DTI-enabled EWS in which a threshold score of 65 triggered interruptive alerts that prompted structured huddles to identify causes of deterioration. 29 They noted that while the DTI prediction itself might not always be informative, its value lay in aligning clinical teams around standard downstream actions for high-risk patients. 15 This suggests DTI was used more as an acuity-stratification tool to prompt consistent team responses, rather than purely as a prediction tool for unrecognized deterioration, potentially explaining its more favorable reception in that setting.

The optimal assignment of responsibility for EWS alert response remains an area of uncertainty. 24 While we designated charge nurses as primarily responsible, their adoption was low. Our survey indicated varied preferences, with some favoring bedside nurses, others charge nurses, and some suggesting shared responsibility. Previous EWS implementations designating charge nurses as primary alert recipients have yielded mixed results, which appear heavily influenced by the underlying model's performance. For example, Evans et al 30 reported an association with decreased mortality when charge nurses received alerts from an EWS model with an impressive PPV of 91%. Conversely, Bailey et al 31 found no effect on clinical outcomes when charge nurses received alerts from an EWS model with a PPV of only 15%. This disparity suggests that the clinical performance of a prediction model is likely more important to engagement and success than the specific role assigned to monitor alerts. If alerts are frequently false positives or lag behind clinical judgment, any designated responder may disengage.

Implications for Practice and Research

Our findings offer several implications for the implementation and future development of EWS. For clinical practice, simply lowering DTI thresholds to trigger alerts earlier is unlikely to be a viable solution, as this would predictably increase false positives and exacerbate alert burden without necessarily identifying unrecognized deterioration. Our attempt to minimize alert fatigue through passive alerting failed, primarily because our DTI-enabled EWS, as implemented, did not offer sufficient actionable, preemptive information to incentivize the extra effort required by charge nurses to monitor its alerts. This created a negative feedback loop of low adoption and dissatisfaction. We contend that for an EWS to be successful, it must be underpinned by a robustly performing model that provides new, clinically relevant information in a timely manner.

For future research, several avenues warrant exploration. First, the definition of clinical deterioration used to train and evaluate EWS models needs to be revisited. Instead of training models to predict the traditional, late-stage definition of deterioration (ICU transfer, cardiac arrest, or death), we should train models to predict earlier indicators of clinically relevant decline, including worsening vital signs and new treatment orders. 32 33 34 Our group, for instance, has explored combining such actions into a revised digital signature of deterioration that identifies deterioration over 2.5 hours earlier than ICU transfer alone. 35 The next step is to build or retrain algorithms to predict this or similar upstream (early and potentially reversible) timepoints in the deterioration continuum. Furthermore, integrating qualitative indicators, such as documented clinician worry or nuanced changes in nursing assessments captured via natural language processing, 36 has shown promise as a strategy for enhancing deterioration prediction models' clinical performance and adoption. 37

Second, EWS evaluation metrics must evolve beyond traditional measures of sensitivity and PPV. Companion metrics should quantify the proportion of “novel” alerts that genuinely precede clinical recognition and action, 22 thereby measuring the EWS's additive value beyond existing clinical judgment and workflows.

Third, once a model demonstrates strong predictive performance and clinical utility in real-time (e.g., through prospective validation where clinical experts vet alerts, as in Bailey et al 31 ), then rigorous testing of different implementation strategies can be more meaningfully undertaken. This includes comparing passive versus interruptive alerts (or hybrid approaches) and systematically evaluating different models of alert ownership (e.g., bedside nurse, charge nurse, physician, dedicated on-site, or virtual monitoring teams 38 39 ). Co-design methodologies involving multidisciplinary stakeholders, including clinicians and IT specialists, remain essential for tailoring implementations to local contexts and workflows. 40 The early and sustained involvement of IT, with the flexibility to iteratively update the system based on user feedback, is crucial for optimizing both model delivery and usability. 41 Incorporating “guerrilla” in-situ usability sessions (i.e., brief 10-minute simulations with practicing clinicians directly in the clinical setting where the system will be used), as described by Kandasamy et al, 42 could proactively identify and mitigate workflow barriers before widespread implementation is attempted.

This study has several limitations. First, our pilot was confined to a single intermediate care unit within one large academic medical center using specific DTI thresholds, potentially limiting the generalizability of these results to other clinical settings or patient populations. Second, the modest sample size and relatively short observation period reduced our statistical power to detect changes in clinical outcomes. Third, our survey response rate was low (19.4%), and while adequate for exploratory insights, the findings may be subject to nonresponse bias and may not represent the full spectrum of nursing perspectives. While our survey captured valuable qualitative insights about workflow challenges, we did not conduct formal timing studies or direct workflow observations that could have provided a more granular understanding of how charge nurses attempted to integrate DTI monitoring with their other responsibilities. Finally, while we explored nurses' perceptions of the EWS, the study was terminated before we could formally survey physicians and APPs, whose experiences and expectations regarding EWS may differ.

Conclusion

This pilot study of a DTI-enabled EWS implemented with passive alerts in an intermediate care unit was unsuccessful. Despite efforts to mitigate alert fatigue and promote interdisciplinary communication by tasking charge nurses with monitoring a noninterruptive Patient List display of DTI scores, we observed poor acceptance and no improvement in clinically meaningful outcomes or timeliness of intervention. DTI alerts often lagged behind clinicians' recognition of deterioration, suggesting the tool may function more as an indicator of current severity rather than a predictor of deterioration. Future efforts should prioritize developing more clinically actionable deterioration prediction models and rigorously testing workflow integration strategies co-designed with end users.

Clinical Relevance Statement

This study of an AI-enabled early warning system revealed that passively displaying deterioration alerts to charge nurses in a dashboard did not improve the timeliness of clinical response, as providers had often already started treatment before alerts were seen. Epic Deterioration Index scores may be more valuable as tools for assessing current patient severity rather than as early warning indicators of unrecognized clinical deterioration.

Multiple-Choice Questions

1. Which workflow integration strategy was deliberately avoided in this implementation?

   1. Custom patient lists

   2. Interruptive alerts

   3. Score trending displays

   4. Vital sign integration

   Correct Answer: The correct answer is option b. The implementation specifically avoided interruptive alerts to reduce alert fatigue and promote more thoughtful, team-based care. This decision was deliberate but ultimately proved problematic as nurses expressed a preference for interruptive alerts.

2. What pattern emerged regarding the timing of clinical interventions relative to Epic Deterioration Index alerts?

   1. Interventions occurred randomly throughout the day

   2. Interventions typically followed alerts by several hours

   3. Interventions usually preceded the system alerts

   4. Interventions coincided exactly with alerts

   Correct Answer: The correct answer is option c. One of the study's key findings was that clinical teams had typically already recognized deterioration and initiated appropriate interventions before the system generated alerts. The median time of first provider action was several hours before alerts in both pre- and postimplementation periods, indicating that the system was often confirming clinical decisions rather than prompting new interventions.

Funding Statement

Funding This study was supported by grant P30HS029744 from the Agency for Healthcare Research and Quality (AHRQ). The University of Minnesota Office of Academic Clinical Affairs, Clinical Translational Science Institute, and Center for Learning Health System Sciences offered additional support through the Minnesota Learning Health System (MN-LHS) Mentored Career Development Program. The content is solely the responsibility of the authors and does not necessarily represent the official views of AHRQ or MN-LHS.