Validation of Early Warning Scores at Two Long-Term Acute Care Hospitals

Abstract

OBJECTIVE:

Early warning scores (EWSs) were developed to identify high-risk patients on the hospital wards. Research on EWSs has focused on patients in short-term acute care hospitals, but there are other settings, such as long-term acute care hospitals (LTACHs), where these tools could be useful. However, the accuracy of EWSs in LTACHs is unknown.

DESIGN:

Observational cohort study

SETTING:

Two LTACHs in Illinois from January 2002 until September 2017

PATIENTS:

Admitted adult LTACH patients

INTERVENTIONS:

None

MEASUREMENTS AND MAIN RESULTS:

Demographic characteristics, vital signs, laboratory values, nursing flowsheet data, and outcomes data were collected from the electronic health record. The accuracy of individual variables, the Modified Early Warning Score (MEWS), the National Early Warning Score version 2 (NEWS2), and our previously developed eCART score were compared for predicting the need for acute hospital transfer or death using the area under the receiver operating characteristic curve (AUC). A total of 12,497 patient admissions were included, with 3,550 experiencing the composite outcome. The median age was 65 (IQR 54–74), 46% were female, and the median length of stay in the LTACH was 27 days (IQR 17–40), with an 8% in-hospital mortality. Laboratory values were the best predictors, with blood urea nitrogen being the most accurate (AUC 0.63) followed by albumin, bilirubin, and white blood cell count (AUCs of 0.61). Systolic blood pressure was the most accurate vital sign (AUC 0.60). eCART (AUC 0.72) was significantly more accurate than NEWS2 (AUC 0.66) and MEWS (AUC 0.65; p<0.01 for all pairwise comparisons).

CONCLUSIONS:

In this retrospective cohort study, we found that eCART was significantly more accurate than MEWS and NEWS2 for predicting acute hospital transfer and mortality. Because laboratory values were more predictive than vital signs and the average length of stay in an LTACH is much longer than short-term acute hospitals, developing a score specific to the LTACH population would likely further improve accuracy, thus allowing earlier identification of high-risk patients for potentially life-saving interventions.

INTRODUCTION

Early warning scores have been developed and implemented across the United States and other countries around the world in order to improve the identification and treatment of clinical deterioration¹. These scores include the commonly used Modified Early Warning Score (MEWS)^1–2, which was developed using clinical opinion and only utilizes vital signs, as well as more complex scores developed using machine learning³. For example, our group developed the electronic Cardiac Arrest Risk Triage (eCART) score³ using a large, multicenter dataset that included vital signs, demographics, and laboratory results in a machine learning model. Although the literature is rapidly evolving, studies suggest that more complex scores outperform simpler ones, and pairing early warning scores to clinical actions in the acute hospital setting, such as the deployment of rapid response teams, may improve outcomes^4–5.

To date, studies investigating the accuracy of early warning scores have focused on patients hospitalized in acute care hospitals. However, an increasing number of patients with chronic illnesses are transferred to long-term acute care hospitals (LTACHs) following their discharge from acute care settings^6–7. These patients have high morbidity and mortality⁶, and some of them develop new acute conditions that cause clinical deterioration and the need for transfer back to the acute hospital setting^8–9. Because early intervention can improve outcomes in many critical care conditions, including sepsis and cardiogenic shock^10–12, it is possible that early warning scores could play a role in the LTACH setting as well in order to prompt potentially life-saving interventions. However, the accuracy of early warning scores in patients admitted to an LTACH is unknown. Therefore, we aimed to compare the accuracy of early warning scores, individual demographic characteristics, vital signs, and laboratory values for predicting clinical deterioration in a multicenter LTACH database.

MATERIALS AND METHODS

This study was deemed exempt by The University of Chicago’s Institutional Review Board (IRB#19–0974).

Study setting

All patients admitted to two RML Specialty Hospitals in Illinois (Chicago, with approximately 70 beds, and Hinsdale, with approximately 90 beds) from January 2002 until September 2017 were included in this retrospective study of prospectively collected data. Based on administrative data, up to two-thirds of patients are ventilator dependent at some point during their stay, while the remainder are admitted for complex medical issues or wound care. Demographic characteristics (age and sex), vital signs (temperature, blood pressure, heart rate, respiratory rate, oxygen saturation, mental status), laboratory values (electrolytes, renal function, complete blood count, liver function tests), nursing flowsheet data (Braden Scale and amount of oxygen delivered), and outcomes data were collected from the electronic health record (Meditech; Westwood, MA). ICD-9 and ICD-10 billing codes were collected from administrative data and used to determine the number of Elixhauser comorbidities for each patient admission.

Outcomes

The primary outcome of the study was the composite outcome of either death (unplanned or comfort care) or transfer to an acute care hospital. Both of these individual outcomes were investigated as secondary outcomes.

Statistical analysis

The MEWS, National Early Warning Score version 2 (NEWS2)¹³, and the eCART score were calculated for each observation in the dataset during a patient’s admission. The MEWS is based on vital signs, with abnormalities scored on a 0–3 scale and then summed to calculate a total MEWS score. The NEWS2 is a recently updated version of the NEWS, which is used throughout the UK, and is similar in format to the MEWS but with modified weightings. The eCART score was previously developed using logistic regression with linear splines and includes demographics, vital signs, and laboratory values as predictors. All scores were calculated based on prior publications and not modified for this study with the exception that the “prior intensive care unit stay” and “time on the general wards” variables were omitted from the eCART calculation as they are not relevant to this population. As per prior work in this area, if a variable was missing during an individual time point, then the most recent prior value for that variable was used, and if no prior value was available, then the median value for that variable was imputed.

Accuracy was calculated and compared between the individual variables, MEWS, NEWS2, and eCART for predicting death or acute hospital transfer within 24 hours using the area under the receiver operating characteristic curve (AUC) for all observations in the dataset. Analyses were performed using Stata version 15.1 (StataCorps; College Station, Texas), and a two-tailed p-value <0.05 was used to denote statistical significance.

RESULTS

A total of 12,497 admissions were included in the study, with 3,550 (28%) experiencing an acute hospital transfer (n=2,544; 20%) or death (n=1,006; 8%). The median (IQR) Braden Scale of the cohort was 13 (IQR 11–15) and they had a median of three (IQR 2–5) Elixhauser comorbidities. Patients who experienced the composite outcome were older (66 vs. 64 years; p<0.001), more likely to be male (55% vs. 53%; p=0.03), and had a shorter length of stay (median 18 (IQR 8–34) vs. 29 (IQR 20–41) days, p<0.001) than those who did not experience an event.

For the individual predictors of the composite outcome, laboratory values were the most accurate, with blood urea nitrogen being the most accurate (AUC 0.63) followed by albumin, bilirubin, and white blood cell count (AUCs of 0.61); Figure 1. Systolic blood pressure was the most accurate vital sign for predicting the composite outcome (AUC 0.60). Findings were similar for the individual outcomes, with blood urea nitrogen being the most accurate predictor of both death and acute hospital transfer (Table 1).

Figure 1.

Areas under the receiver operating characteristic curves (AUCs) for eCART, the National Early Warning Score version 2 (NEWS2), the Modified Early Warning Score (MEWS), and the twenty most accurate variables for predicting whether an event occurred within 24 hours of an observation.

Abbreviations: eCART = electronic Cardiac Arrest Risk Triage score; NEWS2 = National Early Warning Score version 2; MEWS = Modified Early Warning Score; BUN = blood urea nitrogen; WBC = white blood cell count; SBP = systolic blood pressure; CO2 = carbon dioxide; DBP = diastolic blood pressure; O2 Saturation = oxygen saturation; FiO2 = fraction of inspired oxygen; AST = aspartate aminotransferase

Table 1.

Early warning score and variable accuracy for acute hospital transfer and death in the next 24 hours.

Variable	AUC (95% CI)
	Hospital Transfer (n=2,544)	Death (n=1,006)
eCART	0.67 (0.67, 0.68)	0.83 (0.82, 0.83)
NEWS2	0.62 (0.62, 0.63)	0.75 (0.75, 0.76)
MEWS	0.62 (0.62, 0.63)	0.72 (0.71, 0.73)
BUN	0.61 (0.60, 0.61)	0.69 (0.69, 0.70)
Bilirubin	0.60 (0.59, 0.60)	0.65 (0.64, 0.65)
Albumin	0.59 (0.58, 0.59)	0.65 (0.65, 0.66)
WBC	0.58 (0.58, 0.59)	0.67 (0.66, 0.67)
SBP	0.57 (0.57, 0.57)	0.67 (0.67, 0.68)
Anion gap	0.58 (0.57, 0.58)	0.63 (0.63, 0.64)
Creatinine	0.58 (0.57, 0.58)	0.62 (0.62, 0.63)
Respiratory rate	0.57 (0.56, 0.57)	0.64 (0.63, 0.65)
CO2	0.57 (0.57, 0.58)	0.61 (0.61, 0.62)
Total protein	0.57 (0.57, 0.58)	0.61 (0.61, 0.62)
DBP	0.55 (0.54, 0.55)	0.66 (0.65, 0.66)
Heart rate	0.57 (0.57, 0.58)	0.58 (0.58, 0.59)
Calcium	0.56 (0.56, 0.57)	0.59 (0.58, 0.59)
O2 saturation	0.54 (0.54, 0.54)	0.64 (0.64, 0.65)
Platelet count	0.56 (0.56, 0.56)	0.59 (0.58, 0.59)
Hemoglobin	0.57 (0.56, 0.57)	0.57 (0.56, 0.58)
FiO2	0.55 (0.55, 0.56)	0.60 (0.60, 0.61)
Glucose	0.56 (0.55, 0.56)	0.56 (0.55, 0.56)
AST	0.54 (0.54, 0.55)	0.59 (0.58, 0.59)
Braden scale	0.53 (0.53, 0.53)	0.61 (0.60, 0.61)

Abbreviations: eCART = electronic Cardiac Arrest Risk Triage; NEWS2 = National Early Warning Score version 2; MEWS = Modified Early Warning Score; BUN = blood urea nitrogen; WBC = white blood cell count; SBP = systolic blood pressure; CO2 = carbon dioxide; DBP = diastolic blood pressure; O2 Saturation = oxygen saturation; FiO2 = fraction of inspired oxygen; AST = aspartate aminotransferase

eCART (AUC 0.72) was significantly more accurate than NEWS2 (AUC 0.66) and MEWS (AUC 0.65) for predicting the composite outcome (p<0.01 for all pairwise comparisons). Findings were similar for the secondary outcome of need for acute hospital transfer (AUC 0.67 for eCART vs. 0.62 for both NEWS2 and MEWS; p<0.01) and death (AUC 0.83 for eCART vs. 0.75 for NEWS2 and 0.72 for MEWS; p<0.01).

CONCLUSION

In this multicenter study, we found that the eCART score was significantly more accurate than MEWS and NEWS2 for predicting the combined outcome of need for acute hospital transfer or death as well as for each individual outcome. This relative improvement in accuracy was similar to the accuracy difference in the original development study published in a multicenter acute hospital database⁴. In addition, we found that laboratory results were the best predictors of the composite outcome, with systolic blood pressure being the most accurate vital sign predictor. These results suggest that early warning scores may have a role in identifying critically ill patients in the LTACH setting.

Our finding that laboratory values were the most accurate predictors of clinical deterioration in the LTACH setting differs from prior work on the acute hospital wards where vital signs, especially respiratory rate, have been shown to be the most accurate variables^{4, 14}. Interestingly, this finding mirrors our prior work investigating predictors of intensive care unit readmission, where we also found that blood urea nitrogen was the most accurate predictor and laboratory results generally outperformed vital signs¹⁴. This suggests that in the LTACH setting, measures of chronic illness and comorbidity, such as albumin and blood urea nitrogen, are better predictors than traditional vital signs. The fact that respiratory rate and oxygen saturation were less predictive in this population than on the general wards may also be due to the fact that many LTACH patients have some component of compromise at baseline⁶, with two-thirds of the patient population in this study requiring mechanical ventilation, making it harder to determine whether abnormalities are new. This suggests that future, LTACH-specific models should incorporate trends in order to account for baseline abnormalities in these vitals, similar to what we have previously found in ward models where the addition of trends significantly improved accuracy¹⁵. As suggested by prior studies, the inclusion of additional laboratory biomarkers may also further improve accuracy^16–17.

Similar to the acute hospital setting, we found that eCART was a significantly better predictor of clinical deterioration than MEWS in the LTACH setting⁴. Although the AUCs were lower in this study (0.72 vs. 0.77 for eCART and 0.65 vs. 0.70 for MEWS), the improvement in accuracy for the more complex eCART score was similar⁴. This suggests that eCART would demonstrate increased detection and lower false alarms if implemented in real-time to alert clinicians of high-risk patients. Such alerts could be used to direct attention and resources to the highest risk patients, which may potentially improve their outcomes (e.g., through earlier interventions in the LTACH or swifter transfer to an acute hospital if needed). This hypothesis deserves further study in future interventional trials.

The current study has several limitations. First, the study cohort was from two LTACHs in the same system in Illinois, and the results may not be generalizable to other centers. In addition, mental status documentation was available in less than 5% of patients. However, MEWS, NEWS2, and eCART include mental status as a predictor variable, so it is unlikely that this altered their comparative accuracies. Furthermore, manually collected variables, such as respiratory rate, are more prone to human error as compared to laboratory biomarkers, which could have affected the results of this study. Finally, we only compared MEWS, NEWS2, and eCART in this study; there are over 100 different early warning scores in the literature, and investigating the accuracy of all of these scores was beyond the scope of this work.

In conclusion, we found that eCART was significantly more accurate than MEWS and NEWS2 in this multicenter study of early warning scores in the LTACH setting. In addition, laboratory results were the best predictors of clinical deterioration, and most vital signs had fair to poor accuracy. These results suggest that early warning scores may have a role in the LTACH setting, thus allowing for earlier identification of high-risk patients and potentially life-saving interventions.