Minimal Impact of Implemented Early Warning Score and Best Practice Alert for Patient Deterioration

Abstract

Objective:

Prior studies have looked at NEWS performance in predicting in-hospital deterioration and death, but data are lacking with respect to patient outcomes following implementation of National Early Warning Score (NEWS). We sought to determine the effectiveness of NEWS implementation on predicting and preventing patient deterioration in a clinical setting.

Design:

Retrospective cohort study

Setting:

Tertiary care academic facility and a community hospital.

Patients:

Patients 18 years of age or older hospitalized from March 1, 2014 to February 28, 2015 during pre-implementation of NEWS to August 1, 2015 to July 31, 2016 after NEWS was implemented.

Intervention(s):

Implementation of NEWS within the electronic health record (EHR) and associated best practice alert.

Measurements and Main Results:

In this study of 85,322 patients (42,402 patients pre-NEWS and 42,920 patients post-NEWS implementation) the primary outcome of rate of ICU transfer or death did not change after NEWS implementation, with adjusted HRs of 0.94 (0.84, 1.05) and 0.90 (0.77, 1.05) at our academic and community hospital respectively. In total, 175,357 BPAs fired during the study period, with the BPA performing better at the community hospital than the academic and predicting an event within 12 hours 7.4% versus 2.2% of the time, respectively. Re-training NEWS with newly generated hospital-specific coefficients improved model performance.

Conclusions:

At both our academic and community hospital, NEWS had poor performance characteristics and was generally ignored by frontline nursing staff. As a result, NEWS implementation had no appreciable impact on defined clinical outcomes. Refitting of the model using site specific data improved performance and supports validating predictive models on local data.

INTRODUCTION

Despite the abundance of electronic health record (EHR) data, the early identification of deteriorating patients in the hospital remains a challenge. Delays in care for such patients can have a detrimental impact on clinical outcomes, (1, 2) as patients transferred unexpectedly to the intensive care unit often have worse outcomes and increased mortality (3–6). Furthermore, survival to discharge following in-hospital cardiac arrest is estimated to be only 22%. (4, 7) Unfortunately, predicting clinical deterioration remains intrinsically difficult because critical information such as preceding abnormal vital signs (8–11) is often missed by providers. (1, 12) While EHR data allows for real-time clinical prediction models with good performance,(13) the practicality and usefulness of such models is not yet fully understood.

In response to such challenges, hospitals now use vital signs, structured EHR data, and early warning scores (EWS) to help identify decompensating patients. (14) Recently, the National Early Warning Score (NEWS), developed and implemented by the Royal College of Physicians in the United Kingdom, has been shown to outperform other EWS as a predictor for in-hospital death and cardiac arrest within 24 hours, with area under the receiver operating characteristic (AUROC) curves of 0.894 (95% CI 0.887–0.902) and 0.857 (955 CI 0.847–0.868), respectively. (15) The composite NEWS is derived from a point system based on abnormal physiological parameters, where a higher score reflects greater risk of clinical deterioration. (16) While many studies have looked at how well NEWS performs in predicting in-hospital deterioration and death, (17–20) data are limited regarding patient outcomes following the actual implementation of NEWS in any clinical setting. (21) Indeed, it is unclear whether action based EWS can prevent adverse events. In this study, we examine the performance and impact on patient outcomes of real-world implementation of NEWS at both a tertiary care academic hospital and a community hospital.

METHODS

From March to June 2015, NEWS was integrated into the EHR at two of our facilities: a large academic hospital with 957 acute care beds and over 41,000 admissions per year and a community hospital with 369 acute care beds and over 15,000 admissions per year. NEWS was selected by our health system due to its strong predictive performance, widespread adoption, and feasibility for integration into our clinical workflow and Epic-based EHR system. For patients hospitalized on intermediate/non-ICU wards, a best practice advisory (BPA) was triggered to a patient’s care nurse if the NEWS reached a threshold of 7 or higher, which was at the recommendation of the Royal College of Physicians. In response to the trigger, nurses were prompted to acknowledge the BPA by clicking a button to either indicate that a rapid response team (RRT) had been called or to identify the reason why an RRT was not called (e.g. comfort care, known sepsis, febrile neutropenia on antibiotics, trauma or surgery less than 24 hours prior). The BPA was not implemented as a “hard stop,” meaning nurses were also given the option to cancel or accept the BPA without indicating a specific reason. The NEWS score was calculated continuously and updated whenever a recorded vital sign changed. The BPA would fire in real time. In our evaluation we used the real time values to emulate the historic performance.

Prior to implementation, six months of validation and user testing was performed by our information technology department, which we refer to as the “pre-NEWS” period. This included two months in which the BPA was activated in silent mode to assess BPA frequency. Care nurses and nursing leadership were trained on the model parameters and BPA-triggered workflows during educational conferences, online learning modules and live demonstrations. After roll-out of the BPA, regular meetings were held with nursing staff to reinforce appropriate BPA response.

To evaluate the impact of NEWS implementation, we considered four components. First, we assessed changes in mortality and ICU transfer rates before and after implementation of the BPA. Second, we assessed nurse response to the BPA. Third, we determined the predictive performance of the NEWS. Fourth, we calculated the association between NEWS variables and outcomes.

Cohort Definition

We defined the “pre-NEWS” period as March 1, 2014 – February 28, 2015. NEWS implementation was completed in June 2015, after which we allowed a 2-month acclimation period; thus, we defined the “post-NEWS” period as August 1, 2015 – July 31, 2016. We abstracted data on all adult (age greater than or equal to 18 years) inpatient admissions during these time periods. The cohort included both patients on surgical and medical wards. However, we did not include events which occurred during surgery or planned transfer to the surgical ICU post operatively.

Outcome Definition

Our primary outcome of interest was inpatient mortality or unanticipated transfer to the ICU from an intermediate medical ward. Death during surgery and planned transfer to the ICU immediately after surgery were not considered events of interest because perioperative patients were not subject to the BPA. When calculating person-time for event rates, we removed patients from the risk-set when they were not on a medical ward.

Covariate Definition

The NEWS is a simple score comprised of 7 vital signs: systolic blood pressure (SBP), respiratory rate (RR), heart rate (HR), temperature, oxygen saturation (O2), supplemental oxygen, and level of consciousness. We extracted vital signs each time they were updated in a patient’s medical record. In addition, we extracted information on patient’s age, sex, race, and comorbidities (diabetes, cancer, chronic kidney disease, chronic obstructive pulmonary disease, myocardial infarction, stroke, HIV, transplant). Comorbidities were defined via the Clinical Classification Software Groupers, which are managed by the Agency for Healthcare Research and Quality. (22) Finally, we extracted information on the amount of time the ICU was on diversion and unable to accept patients. Our organization has a standard operating report indicating daily ICU diversion.

Statistical Analysis

All analyses were stratified on hospital facility (academic versus community). We calculated the differences in patient characteristics during the pre/post periods, using chi-square tests and Wilcoxon Rank Sum tests as appropriate. After establishing equipoise between the two time periods we calculated unadjusted incident rates, both relative and absolute for each period per 100 patient hospital days. Next, we calculated the adjusted hazard ratios (aHR) for deterioration, adjusting for individual patient characteristics as well as monthly ICU divert time. All demographic factors listed in Table 1 were used in calculating the aHR. Since we considered discharge to be a competing event, we used the Fine and Gray method to estimate the sub-distribution hazard ratio. (23) To assess the impact of the BPA on nursing staff, we assessed whether nurses accepted or ignored the BPA. To avoid over-counting BPAs that fired multiple times, we coarsened the alerts into 30-minute intervals. Then to assess the accuracy of nursing response to the BPA, we calculated how often the alert was followed by an event within 12 hours.

Table 1.

Baseline Pre and Post NEWS- Academic Facility/Community Hospital

Characteristic	Pre-Academic	Post-Academic	Pre-Community	Post-Community
Demographics
Age (yrs)
n	27928	28622	14474	14298
(Median, 25th-75th)	61 (49– 71)	62 (49– 71)	57 (38– 71)	57 (37– 71)
Percent Female	(47.4%)	(47.5%)	(64.9%)	(64.5%)
Race
Black	7840/27928 (28.1%)	8422/28622 (29.4%)	5593/14474 (38.6%)	5540/14298 (38.7%)
Other	1587/27928 (5.7%)	1617/28622 (5.6%)	994/14474 (6.9%)	1010/14298 (7.1%)
White	18501/27928 (66.2%)	18583/28622 (64.9%)	7887/14474 (54.5%)	7748/14298 (54.2%)
Total Length of Stay (Days)
(Median, 25th-75th)	3.95 (2.24– 7.08)	4.11 (2.33– 7.51)	3.03 (2.03– 4.99)	2.95 (2.01– 4.95)
Groupers
Diabetes	29.6%	31.1%	31.3%	31.5%
Malignancy	30.6%	30.9%	13.1%	13.2%
CKD	19.0%	20.6%	18.0%	18.9%
COPD	10.9%	12.3%	11.4%	12.7%
Myocardial Infarction	9.6%	9.8%	7.7%	8.0%
Stroke	5.2%	6.4%	4.1%	3.8%
HIV	1.2%	1.2%	0.9%	1.0%
DNAR	9.8%	10.7%	11.5%	10.1%
Transplant	6.6%	6.1%	0.7%	0.6%
Event Rates (Per 100 Person Days)
Event Rate	0.711	0.665	0.620	0.568
Death Rate	0.175	0.182	0.262	0.251
ICU Rate	0.537	0.483	0.429	0.361

We estimated the predictive accuracy of the NEWS score via the c-statistic. In lieu of calculating a single c-statistic appropriate for time varying data, which would not have allowed us to assess whether NEWS performed better at different time periods, we calculated the c-statistic at discrete time periods and over distinct time horizons. Specifically, we considered a patient’s NEWS over 12-hour increments starting from admission to 7 days into the admission. We then assessed, over discrete 4-hour increments, the presence of an event over the next 48 hours.

Finally, we compared the NEWS coefficients to the optimal coefficients for our data. Using a time varying Cox Model, we utilized NEWS variables and cut-points to re-estimate the association with the outcome. We estimated a more optimal score, using only the same seven variables. Instead of creating cut-points, we kept the vitals continuous and used regression splines to allow for non-linearity. We re-assessed the c-statistics for this model using the same cut-points as above. All analyses were performed in R version 3.2.1.

Role of Funding Source

Research reported in this publication was supported by the National Institute of Diabetes and Digestive and Kidney Disease (K25DK097279 to BAG) National Institute of Allergy and Infectious Diseases (T32-AI007392 to MEC) and Duke Center for Integrative Health (to BAG, RCS and CO). The funding bodies had no role in the study’s design, conduct, review, or reporting or the decision to submit the manuscript for publication.

IRB

This study was approved by our institution’s institutional review boards (Pro00065513).

RESULTS

During the study period, data were collected for a total of 85,322 patients, including 42,402 patients pre-NEWS and 42,920 patients post-NEWS implementation. The academic facility accounted for two-thirds of the study population. Table 1 presents the baseline characteristics of the pre/post cohorts at each facility. Age, gender and length of stay did not differ meaningfully between the two time periods. The population at the academic facility showed a high burden of comorbid conditions (e.g. diabetes, malignancy, and chronic kidney disease) as well a large population of transplant patients. The community hospital also had a high burden of comorbid conditions including diabetes and chronic kidney disease, however, there were fewer patients with malignancies or transplants in comparison to the academic facility.

Across both facilities, ICU transfer was more common than mortality, typically occurring within the first 2 days of a patient’s hospitalization (Figure 1). Our primary outcome of ICU transfer or death showed no change after implementation of NEWS across both facilities (Table 2), with adjusted HRs of 0.94 (0.84, 1.05) and 0.90 (0.77, 1.05) at our academic facility and community hospital respectively.

Figure 1:

Cumulative Incidence plot for time to each of the three possible outcomes: discharge, ICU transfer and death for the tertiary care (a) and referral hospital (b) respectively. The right side is a zoomed in version of the y-axis. Most adverse events happen within the first 2 days of admission. There were more adverse outcomes at the academic hospital.

Table 2.

ICU transfer or death showed no change after implementation of NEWS across both facilities

	ICU Transfers/ Death
	Academic Facility	Community Hospital
Total Events
Pre-NEWS	1092	345
Post-NEWS	1051	309
Event Rate (events per 100-patient days)
Pre-NEWS	0.71	0.62
Post-NEWS	0.67	0.57
Relative Change (95% CI)	−6.7% (−15.0%,1.7%)	−9% (−24.2%, 6.2%)
Absolute Change (95% CI)	−0.05 (−0.1,0.1)	−0.05 (−0.14, 0.04)
Adjusted HR (95% CI)	0.94 (0.84, 1.05)	0.90 (0.77, 1.05)

In total, 175,357 BPAs fired during the study period (117,352 at the academic facility and 58,005 at the community hospital). The BPA performed better at the community hospital than the academic, predicting an event within 12 hours 7.4% versus 2.2% of the time. At both hospitals, the BPAs were ignored 86% of the time. At our academic facility, when a nurse accepted the BPA, patients were more likely to have an event within 12-hours compared to when they were ignored with an odds ratio of 1.23 (1.11, 1.36). There was no difference at our community hospital (an odds ratio of 0.98, CI 0.90, 1.06, see supplemental Table 1).

During the first 48 hours, NEWS showed moderate performance (based on c-statistic) in predicting outcomes at both facilities over the next 24 hours, ranging from 0.72 – 0.80 and 0.74 and 0.90 for the academic and community facilities respectively (Figure 2, Supplemental Table 2). We note that the first 48 hours of hospitalization was when most events occurred. However, as length-of-stay increased for patients, the predictive performance of NEWS improved. When we considered the same NEWS variables, but generated new hospital-specific coefficients, the new models performed better, particularly at our academic facility (Figure 3, Supplemental Table 3). Supplemental Figures 1 and 2 illustrate NEWS and the calibrated NEWS coefficients performance characteristics across varying time horizons. We assessed each hospital-specific model in the other hospital and noted decreased performance (Supplemental Figure 3, Supplemental Table 4). Facility specific associations of the re-weighted Cox Model can be found in the Supplemental Tables 5–7.

Figure 2:

Number of events (a) and NEWS performance (b) over the next 24 hours within the tertiary and community hospitals over time. Most events happen within the first 2 days of an admission. Conversely, the NEWS performs better later in an admission. The community hospital shows better performance than the tertiary hospital.

Figure 3:

Performance of internally estimated model parameters for the tertiary and community hospital respectively over the next 24 hours. The model performs overall better than the NEWS score and similarly performs better later in the admission.

DISCUSSION

EWSs are rapidly being integrated into clinical workflows with the goal of early identification of decompensating patients. Considering a significant percentage of decompensation events occur within the initial hours of an acute admission, it is crucial that EWS perform accurately within this time-frame. NEWS has been shown in multiple studies to predict clinical deterioration with reasonable sensitivity and specificity (24–26) and has been widely adopted in the United Kingdom. However, to the best of our knowledge, the clinical impact of NEWS implementation had not previously been studied in a real-world setting. In the literature, EWSs have been published with varying performance; however, compared to other popular EWSs such as VIEWS (AUROC of 0.888) as a predictor of death with 24 hours (27) and MEWS (AUROC of 0.77) as a predictor for cardiac arrest (28), the performance of NEWS within our health system was substantially worse. Furthermore, NEWS underperformed at both our academic facility as well as our community hospital, although there was a general trend towards better performance at our community hospital. Interestingly, NEWS performed at its worst during the time period when most events occurred. In our cohort, the majority of ICU transfers and cardiac arrests happened within the first 48 hours of admission. NEWS’s operating characteristics were poorest during this time. As patients progressed through admission, the number of events dropped off significantly, while NEWS’s operating characteristics improved significantly. Essentially, when there were numerous events, NEWS performed poorly, and when there were very few events, NEWS performed well.

The poor operating characteristics of NEWS at our institution is reflected in nursing response to the BPAs. The majority of BPAs were ignored by care nurses. Furthermore, because nurses were ignoring the BPA, the logic in the background would cause the BPA to repeatedly fire on the same patient. This in turn created a large quantity of alerts that required no intervention by clinicians and led to alert fatigue in frontline nursing staff. Anecdotal feedback from nurses confirmed the constant burden of alerts repeatedly firing on individual patients. Some patients had BPAs fire over 100 times per 24 hours. Furthermore, there were several other nursing workflows that relied on BPAs, totaling an additional 10,000 BPAs per day firing to nurses thus likely contributing to the overall load of alarms. Although alert fatigue predates the EHR, (29) the widespread adoption of EHRs into clinical workflow has intensified the issue. More than 6% of all computerized provider order entry systems can generate an alert (30) of which 49–96% of alerts are overridden. (31) Furthermore, alert fatigue begets more alert fatigue (32) and the downstream consequences of alert fatigue can include missed alerts, delay in treatment or diagnosis, or impaired decision-making when responding to future alerts. This suggests that perhaps a BPA is not the right approach to notify end users of deterioration. Despite the poor performance of the BPA, when the BPA was “accepted” patients were more likely to have an event, suggesting that nurses were appropriately interpreting the alert.

While the granular data collected within EHRs is optimal for near-term risk assessment, (33) it should be noted that lack of BPA acknowledgement is not a unique failure of NEWS. Even a high performing early warning system will have a high rate of false positives because of the rarity of events in a short time horizon. Future implementation of predictive models warrants training of front line staff that no predictive model is perfect and to expect a certain degree of false alarms. Instead of treating alerts as a go/no-go decision tool, it may be more appropriate to consider them an “awareness” tool that needs to be integrated with subjective clinical judgment.

In summary, the NEWS performed poorly across both facilities within our health system. Since the NEWS was neither sensitive nor specific for our populations, alerts were often sent to the care nurse in situations requiring no clinical intervention, or alerts were never sent to the care nurse for patients who did warrant intervention. This discordance led to alert fatigue and a general mistrust of the entire NEWS workflow.

Our refit of NEWS was not meant as an attempt to propose a better score, but rather to illustrate that risk scores ought to be customized for the clinical environment. Our analysis suggests the NEWS variables were statistically and clinically significant for the intended events of ICU transfer and death within our cohort. However, the coefficients assigned to each category did not reflect our population. Large academic health systems such as ours have a high case mix index, representing greater disease severity and variety in patients treated at these hospitals. (34) The population at our academic facility also contained a large cohort of transplant patients. This greater patient acuity could have led to a higher baseline level of abnormal physiologic parameters and therefore higher baseline NEWS, thus causing a loss in specificity. Although the populations between our academic facility and community facility were different, NEWS performed poorly across both institutions. The mismatch of weighted coefficients supports validating a predictive model on local data prior to implementation. Furthermore, the addition of other clinically important factors, such as laboratory values, to risk models has been shown to improve performance characteristics. (35)

While EHRs have become a primary resource for development of prediction models, external validation is oftentimes not performed when implementing a predictive model. (13) A study performed by Siontis et.al. noted that out of 127 new prediction models, only 32 (25.2%) had subsequent external validation studies. Furthermore, predictive models had significant drops in their operating characteristics when validated on external datasets compared to their derived datasets. (36) Our implementation of NEWS illustrates the need for external validation. (37) Furthermore, our study illustrates the need to validate predictive models to the granularity of each hospital.

Our study has limitations common in quasi-experimental study design. Although our analysis suggests temporally the implementation of NEWS had no impact on important clinical outcomes, a pre-post design would not allow for us to control elements that were changing at the same time as the NEWS was implemented. Thus, changes in outcomes during the study period could not be fully attributed to NEWS implementation. We also analyzed the hospital populations in aggregate without a thorough analysis of a subgroup population. Thus, we made a broad assumption that patients have similar deterioration patterns across demographics, diagnoses, and objective data. It is entirely possible that NEWS had better operating characteristics for a specific subgroup. Lastly, we did not have reliable laboratory or medication orders across both facilities to assess the escalation of care beyond the BPA.

CONCLUSIONS

The implementation of NEWS at our institution provides an important lesson on EWS that have been developed at an outside organization. An EWS performing well during validation at one center does not definitively translate into an optimized EWS at another institution, especially one with a different case mix index. An automated alert triggered by an underperforming EWS may lead to alert fatigue and breakdown of the clinical response pathway. Institutions should assess all predictive models planning to be implemented into clinical workflow by using locally generally data and educate clinical staff on the anticipated strengths and weakness of the model prior to rollout. Our example should serve as an “early warning” to those considering EWS implementation at their center.