Performance and Improvement of Track and Trigger Scores Before Rescue and Transfer in the General Care Setting

Abstract

Objectives:

This work aims to understand the performance of early warning scores calculated using intermittent vital signs to identify general deterioration in the medical/surgical setting and explore the feasibility of implementing near real-time scores using continuous monitoring as part of a systems approach to inpatient assessment.

Methods:

This retrospective study used comparative trends and performance analyses to compare the ability of 4 early warning scores to identify patients requiring rescue and/or transfer to a higher level of care. Simulation was used to explore improvement in deterioration recognition with estimated scores calculated every 5 minutes as compared with those calculated using available intermittent vital signs to understand the potential impact of score calculation where continuous monitoring is available.

Results:

The National Early Warning Score performed better than other scores in identifying patients needing rescue and/or transfer to higher levels of care, with a sensitivity versus specificity analysis area under the curve value of 0.82. The National Early Warning Score also produced clinically acceptable misclassification ratios of 1:1 at scores of 6-7 or above. Simulation using this score, estimated every 5 minutes, improved detection of rescue and transfer events by over 5 hours on average.

Conclusions:

Early warning scores can be used for the detection of general deterioration events and, if calculated frequently using parameters from continuous monitoring systems, can augment alarm-based continuous monitoring to create a system to further reduce unwitnessed arrests and deaths.

Hospitalized patients who require major resuscitative interventions due to late recognition of deterioration suffer worse outcomes, including increased mortality, morbidity, and length of stay.¹ Interventions such as the deployment of rapid response teams have been developed to support early response to deterioration. An integrated systems approach to improve deterioration recognition and mitigate death related to treatable complications has also been described by McGrath et al,^2,3 with a special focus on general care units (GCU), where resources for patient assessment are often limited. Two of the primary tactics described in this approach were early warning scores (EWS) that use weighted individual physiological measurements to provide clinicians with an aggregate patient-state score,^4–6 and surveillance monitoring using continuous monitoring and alarm-based thresholds to detect severe and sustained deteriorations.^7,8 Recent studies demonstrated that higher EWS values are associated with in-hospital cardiac arrest,⁹ ICU admission, and 30-day mortality,¹⁰ while studies of surveillance monitoring have shown positive effects by reducing transfers to higher levels of care,^11,12 mortality,¹³ and rescue activity.¹⁴

Despite the success of these individual approaches, each has limitations leading to potential gaps in performance and patient safety. In hospitals where EWSs have been implemented, scores are typically calculated once a day, with more frequent observation recommended for patients with higher scores.¹⁵ Several studies^16–19 have noted that severe decompensation and cardiac arrest are often preceded by more gradual and possibly treatable decline, including a study evaluating adult patients from the Get With the Guidelines—Resuscitation registry, which found that abnormal vital signs were present 1 to 4 hours before in-hospital cardiac arrest.²⁰ This highlights the fact that there is often a lag between the onset of signs of deterioration and clinician situational awareness based on EWS. While alarms from surveillance monitoring can provide real-time notification of changes in physiology, implementation challenges such as identifying clinically relevant events for detection and defining actionable alarm settings without creating “alarm fatigue”^21,22 have necessitated focus on detection of emergent events, with the potential to miss the opportunities for earlier intervention.

Evolving technology and a continued move toward systems approaches to clinical decision support systems (CDSS) provide possibilities for improvement of deterioration detection.^23,24 A multilevel approach including active notification of scores calculated at the bedside using continuous monitoring with a cadence that would provide patient state change information over minutes to hours, complimented by continuous surveillance monitoring with alarms to notify of acute deterioration that occurs in seconds to minutes, could be used as part of an integrated system to address a range of situations and clinical conditions with different physiological presentations and development times.²

In this study, we explore one aspect of such a configuration by characterizing performance and potential improvement in EWS in the general care setting. First, we compare the performance of several EWS using methods applied previously for sepsis identification²⁵ to identify rescue events and transfers to a higher level of care, endpoints selected for their likelihood to allow good management of serious but treatable complications, and avert failure-to-rescue deaths. We then explore current limitations of the EWS application and consider how continuous monitoring capabilities associated with surveillance monitoring could be leveraged to provide near-real-time state information to support earlier recognition of patient deterioration.

METHODS

Setting

With Institutional Review Board approval, the present study was conducted at a level 1 trauma center located in rural New England. Patient acuity is reported in the top 5% in the United States.²⁶ The hospital GCU setting has over 220 beds with continuous pulse oximetry–based monitoring in place for all patients during their stay. Institutional policies and procedures define tiered rescue activations ranging from critical care nurse consultation (Life Safety Consult, LSC), Rapid Response Team (RRT), STAT airway, and code blue activations (see Appendix A, Supplemental Digital Content 1, http://links.lww.com/JPS/A706, for activation information). The nominal standard for patients in this setting is for bedside vital signs to be collected every 4 hours, with more frequent observation based upon clinician assessment and specific orders.

General Cohort Characteristics and Score Information

The case-control study includes adult patients (age ≥18 y) admitted between November 2015 and December 2021, with at least 1 segment of their inpatient visit in a GCU. Cases included patients who had recognized deterioration, referred to here as an event and defined as a rescue activation during their GCU stay, with or without transfer to a higher level of care (HLOC), or a HLOC transfer that occurs independent of rescue activation. The control population includes patients with no recorded deterioration events in a ratio of 3 controls to every 1 case. The total time from GCU admission to the event for each case was used to match the total time for respective controls beginning with GCU admission.

Electronic medical record (EMR) data for each patient visit was obtained for cases and controls including demographics (age and sex), primary medical diagnosis codes (MDCs), diagnosed hypercapnic respiratory failure [required for some score calculations using International Classification of Disease (ICD) 10 codes J96.02, J96.10, J96.22, J96.92, and J96.12], calculated Charlson Comorbidity Index²⁷ (CCI), length of stay (LOS), discharge status, and rescue event activity. Discrete vital signs data with timestamps were obtained from the EMR for the GCU stay segment, including pulse rate (PR), oxygen saturation (SpO2), temperature, noninvasive blood pressure (NiBP), and respiratory rate (RR). Supplemental oxygen administered and level of consciousness via the Glasgow Coma scale (GCS)²⁸ or Alert, Verbal, Pain, Unresponsive (AVPU) scale²⁹ were also collected.

The National Early Warning Score (NEWS), National Early Warning Score 2 (NEWS2), Modified Early Warning Score (MEWS), and quick Sequential Organ Failure Assessment (qSOFA) (see Appendix B, Supplemental Digital Content 1, http://links.lww.com/JPS/A706, for score definitions) scores were calculated for the GCU stay period for each case and control using available vital signs and application of methods for calculating scores with missing data as described elsewhere.²⁵ Score times were calculated relative to the end of the period of interest, which is event time for cases and end of the case-matched stay time for controls. Patients without scores were eliminated from further analysis. Matlab (Mathworks, version 2022b) and R (Foundation for Statistical Computing, 2024) were employed to perform all data analyses.

Trend Analysis

Case and control score trends were compared using a linear mixed effects model with smoothing by quadratic regression splines for the 24-hour period preceding the events.

Score Performance

Sensitivity and specificity were calculated as a function of both score values and hours before the event to create receiver operating curves (ROC) for each of the scores. Area under the receiver operating curve (AUROC) was evaluated for various time periods preceding the event with a maximum of 24 hours. To assess performance for higher acuity patients, HLOC transfers were further categorized as urgent or emergent using criteria related to delivery of fluid boluses, administration of vasopressor and/or inotropic drugs, or intubation within the hour before the transfer.³⁰ Ratios of false positives to false negatives were computed for each score threshold over various time periods with a maximum of 24 hours. This analysis supports the identification of score thresholds appropriate for the activation of resources when deterioration is detected, while also minimizing nuisance notifications.

Score Availability Analysis and Modeling

The distribution of available scores was used to understand the practical implications of current EWS calculation methods based on intermittent vital signs collection. This analysis was limited to NEWS scores based on results from the score performance analysis. Measures of score availability for the patient visit and for various intervals of the patient stay were calculated both with and without scores calculated using missing data rules. Score distributions were created to show variation over time based on the availability of vital signs for each group. Simulation was then used to explore improvements in recognition of patient deterioration that could be obtained with more frequent EWS score calculation using continuous monitoring. For this evaluation, a NEWS score of 6 or greater was used as the deterioration recognition threshold based upon upper end activation criteria for urgent response that has been broadly deployed by the National Health Service across the United Kingdom.³¹ In addition, a NEWS of 6 aligns with study institution criteria for requesting an LSC, prompting an assessment by a clinician with core competencies in the care of critical care patients. Each patient with an event included in the simulation had at least 1 NEWS score below 6 before a score of 6 or more was reached. NEWS scores were randomly interpolated assuming an autoregressive correlation, AR (1), of 0.6 for measurements 4 hours apart; this is equivalent to 0.989 between measurements 5 minutes apart. The standard deviation of NEWS scores within 24 hours based on patients with events was calculated as 2.2. Score values were randomly interpolated with this level of within-patient variation. Time saved was calculated as the difference in time from when the 5-minute scores reach the threshold, as compared to the time with the observed scores using intermittent vital signs.

RESULTS

General Cohort Characteristics and Score Information

Table 1 provides a summary of case and control demographics and visit characteristics. Patients in the case group were older compared with control patients (P<0.0001), more likely to be male (P<0.0001), and had higher CCI (P<0.0001). LOS for cases was significantly longer than LOS for controls (P<0.0001), and cases had a higher rate of death at discharge (14.86% versus 0.5%, P=0). For events associated with cases, 61.54% were LSC, 7.82% RRT, 1.49% STAT Airway, 0.95% code blue activations, and 28.20% were non-rescue–related transfers. The transfer rate among all the rescue events was 21.18%.

TABLE 1.

Comparison of Cohort Characteristics and Composition by Type of Event of Cases

	Cases (n=5780)	Controls (n=17340)	P
Characteristic
Age (mean years±SD)	64.82±15.87	60.73±17.13	<0.0001
Sex (% male of population)	54.65	51.04	<0.0001
CCI (mean±SD)	8.42±4.55	6.27±4.25	<0.0001
Case Mix Index (CMI)	3.14±2.67	2.04±1.46	0
LOS of visit (mean days±SD)	17.63±24.35	5.87±10.72	0
Transfers to HLOC (%)	49.38	0	NA
Death at discharge (%*)	14.86	0.50	0
Death after rescue event (%†)	0.52	NA	NA
Case composition by type of event	Events [n (%)]	Transfers to HLOC [n (%)]
LSC	3557 (61.54)	985 (17.04)
RRT	452 (7.82)	134 (2.32)
STAT Airway	86 (1.49)	72 (1.25)
Code Blue	55 (0.95)	33 (0.57)
Non-rescue–related transfers	1630 (28.2)	1630 (28.2)

t tests and chi-square tests were used to evaluate the differences between cases and controls

^*% of the entire population,

^†% of the rescue population

As expected, qSOFA produced the most scores as it requires the least number of parameters for calculation (n=57,175 for cases and n=90,223 for controls), and NEWS/NEWS2 the least (n=55,149 for cases and n=88,257 for controls). The proportion of scores for which calculation rules were applied for qSOFA was 31.24% for cases and 45.57% for controls, and for NEWS, it was 31.13% for cases and 45.05% for controls.

Trend Analysis

A comparison of scores calculated over the 24-hour period before the event is depicted in Figure 1. In general, control group scores decline slightly as time from admission increases, while scores in the control group increase markedly as event time approaches, with a notable increase at ~5 hours before the event. NEWS and NEWS2 produced the same scores in 89% of cases and 98.09% of controls, resulting in nearly identical trend plots. MEWS and qSOFA score trends follow the general pattern set by NEWS, although their score ranges differ.

FIGURE 1.

Score trajectories for cases and controls. Mean score values over time (continuous lines) and the associated standard error (dotted lines) are shown for each of the 4 EWS calculated. The x-axis represents the time from the event (0) backwards to 24 hours before the event (−24).

Score Performance

ROC curves illustrating score performance for the 5-hour period before the event are shown in Figure 2. NEWS/NEWS2 scores produced the highest AUROC values (0.84), compared with MEWS and qSOFA, which produced values of 0.77 and 0.72, respectively. AUROC analyses for a range of other time periods revealed a gradual increase in AUROC values for intervals closer to the event. For example, NEWS AUROC for the period from 15, 5, and 1 hours before the event were 0.82, 0.84, and 0.86, respectively. The ROC analyses of cases by type of event for the entire 24-hour period preceding an event produced AUROC values between 0.66 (RRT) and 0.85 (emergent transfers) for NEWS scores, and 0.62 (RRT) and 0.78 (emergent transfers) for qSOFA. The presence of an HLOC transfer following a rescue event impacted score values as compared with those for rescues without an HLOC transfer. For example, NEWS AUROC was 0.85 for LSC events, followed by a transfer to HLOC, versus 0.79 for LSC events without transfers.

FIGURE 2.

Receiver operating characteristic (ROC) curve for NEWS, NEWS2, MEWS, and qSOFA. The ROC reveals the tradeoff between sensitivity (proportion of cases whose score exceeds or equals cutoff points that are labeled on the curve) and specificity (proportion of controls whose score is less than cutoff points) for each score. Here, performance is shown for the situation when the deterioration event occurred within 5 hours of the score exceeding the threshold.

Figure 3 displays the proportion of false positives to false negatives across score thresholds over the 24 hours before the event, highlighting the tradeoff between misclassification of different types and clinical importance. The figure shows that a NEWS/NEWS2 score of 4 produces a ratio of 10 false positives to 1 false negative, whereas a NEWS /NEWS2 score between 6 and 7 generates a 1:1 ratio. For MEWS and qSOFA, the 1:1 ratio is achieved with scores of ~5 and 2, respectively.

FIGURE 3.

False-positive versus false-negative percentage for NEWS, NEWS2, MEWS, and qSOFA scoring systems within 24 hours leading up to the event. Diagonal dashed lines refer to ratios (1:1, 10:1, and 25:1) of false positives to false negatives. Prevalence is 0.0404, and cutoff score values for thresholds are labeled along the curves.

Score Availability Analysis and Simulation

The calculation of available scores in relation to hours of GCU stay showed that there were 2.11 scores for every 4 hours of GCU stay for cases and 1.12 for controls when score calculation included application of missing data rules. Results without these rules applied were 1.16 scores for every 4 hours of GCU stay for cases and 0.77 for controls. As shown in Figure 4, cases have a greater proportion of higher value scores throughout the 24-hour period leading to the event as compared with controls. For example, in the hour preceding the event, there are 26.01 times more scores of NEWS greater than 6 for cases as opposed to controls. The mean ratio of scores greater than 6 between cases and controls across all 24 hours is 11.64 (SD: 4.02). For the 5-6 score value range, there are 4.43 times more NEWS scores for cases than controls. The ratio of the mean of score values in the 5-6 range for cases and controls across all 24 hours is 2.94 (SD: 0.39).

FIGURE 4.

NEWS score distributions. The number of scores for each hour-long interval is normalized by the number of patients admitted during that hour (all patients who were in a GCU unit and could have had a score calculated based on measured vital signs). Normalized score counts for each hour are stratified by score threshold ranges and shown for cases and controls.

The simulation of NEWS scores based on a 5-minute calculation period showed that a score of 6 or more was reached at an average of 5.02 hours (median: 3.28 h, IQR: 1.44, 7.17) before the scores were calculated every 4 hours, as depicted in Figure 5. As expected, the figure also shows a gradual decline in the time gained for recognition from more frequent score calculations, especially after the standard intermittent vital signs interval of 4 hours is reached. Although no tradeoff analysis is shown here, it is evident that a score calculation frequency of 5 minutes versus 4 hours offers significant improvement, given that the lead time for recognition in this simulation is 30 minutes or less for over 50% of patients with events.

FIGURE 5.

Additional recognition time using estimated scores calculated in 5-minute increments versus recognition time with scores calculated with intermittent vital signs. Interquartile range (IQR) and median are given in addition to the average time of the NEWS score reaching 6 or above using estimated scores calculated every 5 minutes, as compared with the time for the same criteria to be met using scores calculated with intermittent vital signs. The proportion of patients with events is shown for each 5-minute increment of time saved.

DISCUSSION

This study demonstrates that NEWS and NEWS2 scores outperform qSOFA and MEWS scores in the identification of general deterioration events defined as rescue events or non-rescue–related transfers. Score discrimination between cases and controls was evident in trend analysis, as NEWS scores for event patients were nearly double that of controls 24 hours before the event (3.5 and 1.8, respectively), and at 5 hours prior the trended scores showed a significant increase.

NEWS/NEWS2 had good event predictive performance as measured by AUROC (0.84) and clinically acceptable rates of misclassification with response activation at scores of 6-7 or higher. More importantly, simulation demonstrated that a semi-continuous NEWS score updated every 5 minutes would identify patients likely to suffer deterioration episodes requiring rescue and/or transfer to an HLOC more than 5 hours earlier than is currently possible with standard care that depends on intermittent vital signs data.

These results rely on data gathered in a setting where continuous monitoring is used to collect intermittent vital signs via EMR integration. This situation is likely to have resulted in more scores and more accurate scores^32,33 being available than in hospitals without continuous monitoring, especially for patients with events where alarms are likely to draw attention to the bedside as deterioration progresses. In addition, the patient complexity at the study institution is in the top 5% CMI among all tertiary hospitals nationally, which reflects patient admission types, risk of complication, and the EWS patterns observed. These factors may impact the generalizability of the study results at other institutions.

Many studies have explored the traditional use of EWS to aid clinicians in prioritizing patients who should be monitored more closely, typically with increased frequency of vital signs checks.^34–36 Most often, these implementations provide an EWS before morning rounds or at most each shift to allow nursing and physician teams to identify patients deemed to be at higher risk of complications.³⁷ Some hand-over protocols explicitly seek to identify “watcher” patients who deserve extra vigilance and attention.^38–40 In contrast, this study explores using NEWS as a composite patient-state indicator that might extend/improve the performance of existing continuous surveillance systems, such as those based on pulse oximetry that have been shown to reduce the need for rescue and/or transfer to HLOC. Such a system would need to calculate an EWS score in near real time. The empiric analyses performed here, including over 70,000 patients being treated in a high acuity tertiary hospital and based on a methodology developed for assessing EWSs performance, identified NEWS as a good candidate for creating a continuous score.

The directly measured and derived parameters generated with continuous pulse oximetry monitoring map well to the parameters needed to calculate a NEWS score (Appendix A, Supplemental Digital Content 1, http://links.lww.com/JPS/A706) every 5 minutes. Specifically, SpO₂, PR, and RR are available continuously with pulse oximetry. Newer general care monitoring platforms now include continuous temperature, electrocardiogram (EKG) rhythm analysis, and automated intermittent blood pressure as well. While supplemental oxygen and mental status are lacking, many current multiparameter systems for use in general care allow EMR integration and 2-way communication such that discrete fields like supplemental oxygen and mental status could be polled for every 5 minutes.

Early warning scores have a long history of use and validation.^36,41–43 Multiple studies have shown NEWS to perform better than other publicly available EWS schemes^25,44,45 and better than the proprietary EPIC Deterioration Index and the Rothman Indexes.⁴⁶ Given that these scores have been validated to predict transfer and/or death in the next 24 hours, they are an improvement over prior tools that purport to predict in-hospital mortality and risk of death within 1 year after hospitalization.⁴⁷ However, this is a limited conceptualization of the utility of EWS in detecting deterioration and not the focus of the present study. Here, we focus on score performance related to the identification of patients who will deteriorate in minutes to hours, with the goal of real-time response at the bedside.

Given that early detection of deterioration remains an important goal in inpatient patient safety, these data support the assertion that EWSs can be used continuously along with alarms as part of a system to prompt actionable bedside interventions before the need for cardiopulmonary resuscitation, rapid response team intervention, or emergent transfer. Robust multiparameter scores calculated at the bedside, calculated frequently enough to provide earlier recognition of deterioration, could be used as part of a system that includes multiple clinical decision support tools such as alarms and other attention redirection methods.^2,3,7 Furthermore, activation criteria could be established for tiered responses at the level of a critical care consultation, rapid response team, or resuscitation requiring cardiopulmonary resuscitation, intubation, and advanced cardiac life support. The current reality is that dependence on intermittent vital signs limits applicability and our ability to achieve this goal. Increasing use of continuous monitoring will be foundational to exploring and validating continuous EWSs to create a more effective system for deterioration detection.^48–50