Using machine learning to improve the accuracy of patient deterioration predictions: Mayo Clinic Early Warning Score (MC-EWS)

Abstract

Objective

We aimed to develop a model for accurate prediction of general care inpatient deterioration.

Materials and Methods

Training and internal validation datasets were built using 2-year data from a quaternary hospital in the Midwest. Model training used gradient boosting and feature engineering (clinically relevant interactions, time-series information) to predict general care inpatient deterioration (resuscitation call, intensive care unit transfer, or rapid response team call) in 24 hours. Data from a tertiary care hospital in the Southwest were used for external validation. C-statistic, sensitivity, positive predictive value, and alert rate were calculated for different cutoffs and compared with the National Early Warning Score. Sensitivity analysis evaluated prediction of intensive care unit transfer or resuscitation call.

Results

Training, internal validation, and external validation datasets included 24 500, 25 784 and 53 956 hospitalizations, respectively. The Mayo Clinic Early Warning Score (MC-EWS) demonstrated excellent discrimination in both the internal and external validation datasets (C-statistic = 0.913, 0.937, respectively), and results were consistent in the sensitivity analysis (C-statistic = 0.932 in external validation). At a sensitivity of 73%, MC-EWS would generate 0.7 alerts per day per 10 patients, 45% less than the National Early Warning Score.

Discussion

Low alert rates are important for implementation of an alert system. Other early warning scores developed for the general care ward have achieved lower discrimination overall compared with MC-EWS, likely because MC-EWS includes both nursing assessments and extensive feature engineering.

Conclusions

MC-EWS achieved superior prediction of general care inpatient deterioration using sophisticated feature engineering and a machine learning approach, reducing alert rate.

INTRODUCTION

Background and significance

Physiological deterioration in the hospital is often unrecognized and can generally be defined as “any significant worsening in the condition of a hospitalized patient that can result in patient morbidity and/or mortality.”¹ It is commonly caused by sepsis or acute respiratory failure, with sepsis alone involved in 34% to 52% of in-hospital deaths and estimated to have an in-hospital mortality of 10%, and acute respiratory failure estimated to have an in-hospital mortality of 20%.^2,3 Physiological deterioration can lead to profound clinical instability or cardiac arrest, requiring care escalation.⁴ Research has shown that up to 85% of such deteriorations are preceded by abnormal patient vital signs,^5–8 suggesting that many of these outcomes may be preventable.⁹

Early intervention during patient deterioration may improve patient outcomes. For example, delayed transfer of critically ill patients to the intensive care unit (ICU) is associated with increased mortality^10,11 and mortality increases approximately 8% for every hour that antibiotic treatment is delayed for patients with septic shock.¹²

To address the crucial need for quick interventions during clinical deterioration, hospitals created rapid response systems responsible for responding to patients’ acute needs.¹³ A typical rapid response system has 3 limbs: an afferent (to detect deteriorating patients), an efferent (to respond to those deteriorations), and an analytic administrative portion to continually assess and improve the system.^14,15 In an effort to support the afferent limb (ie, the detection of physiological deterioration), modeling methods and patient data have been utilized to develop scores that aid in assessing and predicting patient deterioration early in a hospital visit.

To date, over 70 early warning scores (EWSs) have been developed, each using different combinations of vital sign components.¹⁶ However, when used without the other 2 limbs, EWSs fall short in 2 ways: they fail to show quantitative improvements in key clinical outcomes and they unfortunately generate a significant number of false alerts, contributing to alert fatigue for providers.^17–19 In fact, even EWSs that perform well in standard comparative analyses (eg, area under the receiver-operating characteristic curve [AUROC] ) can generate a surprisingly high number of false alerts.²⁰

To improve the predictive accuracy of EWSs, some data scientists are using very large volumes of routinely collected patient measurements (eg, vital signs and laboratory results) in development and calibration of predictive models.^21–23 Additional improvements in EWSs have also been made through the utilization of sophisticated modeling techniques, such as machine learning. These techniques improve predictive accuracy for clinical deterioration on general care wards.^24–27 Missing from these are other, less routinely used nursing assessments of patient functional status and mobility.

Nursing assessments can be independently predictive of patient deterioration,^28,29 and also maintain significant value in a machine learning model for pediatric patients.³⁰ Incorporation of these specific predictors into a machine learning model for adult patients is not sufficiently explored. Additionally, clinically relevant interactions between patient vital signs and laboratory results have been explored in the emergency department³¹ and pediatric ICU³² settings but not in general care settings.

OBJECTIVE

The objective of this study was to develop and validate the predictive accuracy of the Mayo Clinic Early Warning Score (MC-EWS), a machine learning model for predicting acute deterioration in general care wards that utilizes not only patient vital signs and laboratory tests as predictors, but also nursing assessments and variables defining known physiologic interactions.

MATERIALS AND METHODS

Population and datasets

Following approval from Mayo Clinic’s Institutional Review Board, we retrospectively collected data for consecutive hospitalizations to general care beds in 2 quaternary care hospitals in Minnesota, and a tertiary care hospital in Arizona. All adult (>18 years of age) patients that spent any time in a general care or telemetry bed were included. Hospitalizations were excluded if they were entirely ICU stays or if they were primarily for research, rehabilitation, or psychiatric purposes.

We defined “encounter” as a hospitalization and “episode” as any uninterrupted period of time spent in the general care or telemetry wards. An encounter consists of 1 or more episodes, interrupted by ICU transfers or procedures. Data from time in the ICU or procedure settings were excluded. In our datasets, each row represented a time point: a moment in time in each patient’s hospitalization. When a new value was entered for a variable, a row was added, with the latest values carried forward for the remaining variables in that row.

A schematic description of the datasets used can be seen in Figure 1. Admissions between January 2010 and December 2011 to 2 referral hospitals in southeastern Minnesota (the 2 hospitals were consolidated in 2014) were split as follows: one-third was used for a validation dataset and the remaining two-thirds was used to build the training dataset by selecting only 1 time point per hospitalization, as follows. In this two-thirds dataset, we first defined cases as hospitalizations that had an outcome of interest (see Outcomes) and controls as hospitalizations without one. Then we selected specific time points from the case and control hospitalizations to build our final training set. From the case hospitalizations, we selected the time point of the first outcome. From the remaining hospitalizations, we selected a random time point for each, and randomly subset so the final training dataset had a 1:10 case-to-control ratio. The idea was to produce a slimmed-down and faster-running dataset that allowed the model to distinguish, “is this a deteriorating patient, or a nondeteriorating patient at a random time point in their hospitalization?” Admissions between January 2011 and June 2015 to a referral hospital in Arizona were used as an external validation dataset.

Figure 1.

Datasets and sample sizes.

Predictors

Predictors included demographics, vital signs, laboratory test results, and nursing assessments. The data were preprocessed to screen for extreme outliers that were not clinically plausible in order to correct for any values that had been entered into the medical record incorrectly. Supplementary Appendix 1 includes a list of the valid ranges for vital signs. Feature engineering was used to capture physiologically relevant interaction terms of the primary variables (ie, shock index as heart rate divided by systolic blood pressure). Other calculated variables included measurements of a variable through time (maxima, minima, and ranges over the previous 24 or 48 hours). The full list of predictor candidates used can be found in Supplementary Appendix 2. At each time point, we used the latest value obtained in a general care ward. For newly admitted patients, we used results from laboratory studies obtained immediately preceding transfer to general care wards. Values obtained when the patient was in the ICU or operating room were not included.

Patient location

Patients were excluded from the deterioration prediction model during the periods of time spent in the ICU, operating room, or undergoing a procedure. To determine patient location at the time of variable measurement, we used 2 different sources: the bed registry and the location provided by the patient monitoring system. Bed registration data is always present and represents the patient bed “reserved” for their overnight stay. The monitoring system location data is automatically recorded when a patient is attached to a monitoring machine containing a physical address (patient’s room). All highly monitored areas (ICU, operating room, and procedure room) have the same patient monitoring system, and every time a patient is in those areas a location is recorded. Combining these 2 data sources, we could accurately determine a patient’s location in the hospital throughout their hospitalization.

Standard of care in the study hospitals

In the general care beds, the standard of nursing care during the period of the study was to obtain a set of vitals every 8 to 12 hours, with flexibility to increase the frequency at nurses’ discretion. Vital signs are hand entered by nursing staff into the electronic medical record, or automatically collected and hand-confirmed by a nurse. Vital signs on medical and surgical floors with telemetric capability are collected more frequently with electronic capture and recording into the electronic health record. Laboratory studies in both floor settings are ordered at the physician discretion at any time of day. The rapid response team (RRT) responds at the request of nurses, physicians, or advanced allied health providers in both the general care and telemetry floors of the hospitals 24 hours per day, every day. A list of the criteria for calling RRT for each hospital is displayed in Appendix 3.

Outcomes

We used widely accepted surrogates that are representative of the conceptual outcome “acute physiological deterioration.” The primary outcome was a composite of resuscitation call for cardiorespiratory arrest (code), call to the RRT, or unplanned transfer to the ICU. Immediate postoperative transfers to the postanesthesia care unit or ICU immediately after surgery were considered planned and were not included as outcomes in the analyses. Our secondary outcome for sensitivity analysis was the composite outcome of unplanned ICU transfer or code.

Model building

We built the model on the training set, which included 1 time point per hospitalization. This time point included some information about previous time points. We selected a gradient boosting machine³³ as the machine learning method for its ability to deal with missing values, interactions between predictors, and nonlinear relationships.^33,34 A series of gradient boosting machine models were built on the training dataset using different shrinkage, depth, and minimum observations per node. The final model, which we call the MC-EWS, was selected based on its performance in the cross-validated training dataset.

Model accuracy analysis

The MC-EWS was evaluated in both the internal and external validation datasets. These datasets have multiple time points per hospitalization. First, a score was calculated at each patient time point. Then a score cutoff was defined as a proposed threshold for alerting clinicians to possible deterioration. Every time a patient’s score was over that defined cutoff, it was considered an alert. To prevent double-counting of repeated triggers, any alerts in the 24 hours after an initial trigger were not considered for the purposes of calculating the model’s accuracy. The alert was considered a true positive if a primary outcome occurred during that 24-hour window. No alert in the following 24 hours indicated a false positive. False negatives were outcomes that had not been detected by an alert with the aforementioned analysis. True negatives were 24-hour windows in which there were neither alerts nor outcomes. That analysis was repeated for different alert thresholds for both the MC-EWS and, as a comparator, different cutoff values of the National Early Warning Score (NEWS).³⁵ Then, sensitivity, specificity, positive predictive value (PPV), and alerts for day per 10 patients were calculated for each score cutoff. Sensitivity and specificity were used to build the receiver-operator characteristics (ROC) curve, and to calculate the area under it (AUROC).

Because RRT calls are dependent on RRT calling criteria at each of the 2 sites as well as on individual provider or nurses’ judgement, we performed a sensitivity analysis to test the model’s robustness. This was done by removing RRT calls and repeating the accuracy analysis for the secondary outcome of only ICU transfers and resuscitation calls. This was then compared with the analysis for the primary outcome.

R version 3.1.3 “Smooth Sidewalk” (R Foundation for Statistical Computing, Vienna, Austria) was used for statistical analysis and the R package “gbm” was used for model building.

RESULTS

Datasets

Figure 1 shows the relationships between each of the datasets used in this study and presents their sample sizes. Our initial dataset was first divided by geographical location (ie Rochester, Arizona); we then used a 67%/33% split to further divide the internal data into 2 sets for training and validation. This resulted in 3 distinct datasets for our analysis: a training dataset (the slimmed-down subset of 24 500 hospitalizations, with only 1 time point per hospitalization), a dataset for internal validation of the model (25 784 hospitalizations, all time points included), and a dataset for external validation of the model (53 956 hospitalizations, all time points included).

Descriptive statistics

Table 1 presents the patient and hospitalization characteristics in our datasets for training (and parent pretraining dataset), internal validation, and external validation. The results show that population characteristics were comparable across all datasets (with the exception of the training dataset, purposefully enriched to a 1:10 case-to-control ratio). Event rates for all 3 portions of the primary outcome were similar in the pretraining and internal validation datasets but higher in the training and external validation datasets.

Table 1.

Patient characteristics across datasets

	Internal			External
Variable	Pretraining (n = 51 826)	Training (n = 24 500)	Validation (n = 25 784)	Validation (n = 53 956)
Total patientsa	39 425	21 165	19 580	32 761
Female patients	20 067 (50.9)	10 453 (49.4)	10 058 (51.4)	15 198 (46.4)
Mean age, y	58.5	60.3	58.4	63.8
Total episodesb	76 503	37 925	38 043	57 156
Total time points	11 697 727	34 923	5 955 389	5 037 866
Length of stay, d
LOS of hospitalization, d
25th percentile	2	3	2	2
50th percentile	3	5	3	3
75th percentile	6	9	6	5
Events
Total events	3185	3185	1547	6909
RRT calls	1519 (47.7)	1519 (47.7)	749 (48.4)	4353 (63)
Code 45	190 (6.0)	190 (6.0)	98 (6.3)	207 (3)
Unscheduled transfer to ICU	1476 (46.3)	1476 (46.3)	700 (45.2)	2349 (34)
Event rate per 100 episodes
Total event rate	4.2	11.4	4.1	12.1
RRT call rate	2.0	5.4	2.0	7.6
Code 45 rate	0.2	0.7	0.3	0.4
Unscheduled transfers to ICU rate	1.9	5.3	1.8	4.1

Values are n (%).

ICU: intensive care unit; LOS: length of stay; MC-EWS: Mayo Clinic Early Warning Score; NEWS: National Early Warning Score; RRT: rapid response team.

^aTotal unique patients. Some patients had more than 1 hospitalization in our dataset.

^bAn episode is an uninterrupted period of time when a patient is in a general care ward.

Score performance in validation datasets

Figure 2 shows a comparison of the ROC curves (top section) and alert rate vs sensitivity curves (bottom section) for both the MC-EWS and NEWS in the internal validation dataset and external validation dataset, using our primary composite outcome (RRT, ICU transfer, or resuscitation call). The ROC curves show that the MC-EWS outperforms the NEWS in both datasets. The sensitivity vs alert rate curves also show that the MC-EWS generates fewer alerts than the NEWS at a fixed sensitivity level.

Figure 2.

Mayo Clinic Early Warning Score (MC-EWS) and National Early Warning Score (NEWS) in the validation sets for primary outcome. AUROC: area under the receiver-operating characteristic curve; ROC: receiver-operating characteristic.

Table 2 compares the performance of the 2 scores (MC-EWS and NEWS) in the external validation dataset. Each set of measurements is reported at equal sensitivity between the MC-EWS and NEWS to allow for a direct comparison of the other measures. The MC-EWS demonstrates superior performance across all metrics at every level of sensitivity. For example, Table 2 shows that at a sensitivity of 73%, the MC-EWS has a PPV of 0.12 and generates 0.70 alerts per day per 10 patients. This represents a 45% lower alert rate compared with what would be generated by NEWS, with a PPV of 0.07 generating 1.27 alerts per day per 10 patients at the same sensitivity of 0.73.

Table 2.

Performance of MC-EWS and NEWS in external validation dataset

Sensitivity	MC-EWS			NEWSa			Difference in Alert Rate (%)
	Specificity	PPV	Alerts per Day per 10 Patients	Specificity	PPV	Alerts per Day per 10 Patients
0.73	0.94	0.12	0.70	0.88	0.07	1.27	−45
0.81	0.90	0.08	1.09	0.84	0.06	1.71	−36
0.89	0.83	0.06	1.78	0.76	0.04	2.47	−28
0.94	0.73	0.04	2.81	0.70	0.03	3.11	−10

MC-EWS: Mayo Clinic Early Warning Score; NEWS: National Early Warning Score; PPV: positive predictive value.

^aLinear interpolation was used to calculate the PPV, specificity and alert rate at each sensitivity value for NEWS.

Sensitivity analysis

Figure 3 shows the ROC curves (left) and alert rate vs sensitivity curves (right) for both the MC-EWS and NEWS. The curves show little change after removing RRT calls as an outcome, with MC-EWS again demonstrating consistent improvement over the NEWS. There is only a minimal loss in the performance of the MC-EWS in this sensitivity analysis (AUROC of 0.932 in sensitivity analysis vs 0.937 with primary outcome).

Figure 3.

Mayo Clinic Early Warning Score (MC-EWS) and National Early Warning Score (NEWS) in the external validation set for secondary outcome. AUROC: area under the receiver-operating characteristic curve; ROC: receiver-operating characteristic.

Relative influence of model predictors

Figure 4 presents the top 20 variables in the final model, selected by the machine learning algorithm from the more than 120 variables that were included. The top 4 variables (which account for 40% of the relative importance) include 2 secondary variables (Kirkland probability, and respiratory index in the presence of oxygen supplementation)³⁶ and 2 primary variables (systolic blood pressure and heart rate).

Figure 4.

Relative influence of top 20 variables. BUN: blood urea nitrogen; DBP: diastolic blood pressure; HCO₃: bicarbonate; INR: international normalized ratio; pCO₂: partial pressure of carbon dioxide; RASS: Richmond Agitation Sedation Scale; RR: respiratory rate; SBP: systolic blood pressure; SpO₂: oxygen saturation.

DISCUSSION

We have developed the MC-EWS, a machine learning model that accurately predicts acute deterioration in the general care wards, represented by the primary composite outcome of RRT call, resuscitation call, or unplanned transfer to the ICU. Over 120 variables were used as predictors, including: demographics, vital signs (including trend over time), laboratory results, nursing assessments, and clinically relevant interactions. This model demonstrated excellent discrimination (AUROC = 0.937 in the external validation set) and could be implemented to alert providers to act early in the deterioration process, with the goal of reducing preventable mortality and morbidity. Compared with the NEWS, our model achieved a 45% reduction in alert rate for a comparable sensitivity of 72%.

There have been other attempts to incorporate machine learning approaches to the prediction of in-hospital mortality or acute deterioration in the general care wards, but they have achieved lower accuracy overall. Arguably, this is because no other approach has incorporated all of the elements described: a high number of predictor variables including nursing assessments, and feature engineering. The models are not publicly available, so we could not calculate and directly compare them within our dataset, but we provide a brief comparison subsequently.

The APPROVE score predicts acute respiratory failure and mortality.²⁴ It did not, however, include feature engineering, and its discrimination (AUROC = 0.87) was lower than our MC-EWS. Giannini et al²⁵ developed a random forest classifier to predict severe sepsis and septic shock using 587 variables, including time series data. However, to keep alerts at 10 per day for implementation, sensitivity was only 26%, meaning that most deteriorating patients did not trigger an alert. The MEWS++ random forest model, developed by Kia et al³⁷ to predict death or escalation of care within 6 hours, also used an extensive range of predictors, including the time series of frequently measured variables. It compares favorably to our MC-EWS, with a sensitivity of 78.9% and PPV of 11.5% at their chosen cutoff, but it would likely benefit from capturing relevant physiologic interactions between variables. Other attempts have used deep learning, which has the advantage of automatically learning features, and has been applied to predicting in-hospital cardiac arrest.³⁸ However, for that model, the alarm rate of 0.3 per patient per hour at a sensitivity of 75% was still too high. The most likely explanation for why this model was not able to achieve a higher accuracy is that it used only 4 vital signs as predictors missing the lift obtained from nursing assessments and laboratory results.

We chose to model our outcome as a composite of clinical responses (ICU transfer, RRT call, or code), rather than mortality, to capture instances in which escalation of care was considered necessary, rather than including events of expected death in which de-escalation was the appropriate clinical response. Because of the subjectivity of RRT calling criteria, a sensitivity analysis was done to evaluate the performance of MC-EWS with RRT calls removed as an outcome. Results showed little difference when compared with the accuracy analysis for the primary outcome, corroborating that RRT calls are a reliable surrogate for physiological deterioration. The slight reduction in performance for both MC-EWS and NEWS was likely due to the lower number of total outcomes after eliminating RRT calls as an outcome. We compared the performance in the internal and external validation sets (see Supplementary Appendix 4). We noted a slight increase in predictive performance in our external validation dataset compared with the internal validation set. This could be due to a higher frequency of vital sign recordings, or more accurate data entry.

Examining the most influential predictors in the MC-EWS yields some interesting insights. The top predictor, Kirkland score,³⁶ was a composite derived from the shock index, Braden score,³⁹ respiratory rate, and oxygen saturation. The inclusion of the Braden score, which consists of several nursing assessments such as nutrition, mobility, and sensory perception, supports our emphasis on including nursing assessments in the model. Looking at other top predictors, we found that 7 of the top 10 were either secondary variables derived from others or time-dependent variables. In an application in which every improvement in discrimination is vital, this demonstrates the value feature engineering can provide.

Strengths of our study include the selection of a large number of variables for creation of the model, particularly nursing assessments, the range of values over time, and clinically relevant interactions. The low-bias gradient boosted machine learning method used was able to capture nonlinear relationships and incorporate “missing” as a possible value to be evaluated. Missing variable values provide important clinical context regarding provider medical decision making (ordering of a laboratory study) or realities of clinical operations at that moment (no time for nurse to chart a vital sign). Because a missing value does not necessarily reduce our expectation of risk the same way a normal level of a variable would, it is important to consider a missing variable value as possible valuable.

Our study is not without its limitations. The MC-EWS was developed and validated in 3 tertiary and quaternary care hospitals operated by Mayo Clinic and may perform differently if evaluated in smaller community hospitals or those with different rapid response systems. Future research could be done to improve upon the performance of the MC-EWS, for example, by using continuous monitoring devices or wearable data. Deep learning models have been developed using several vital signs as predictors of deterioration with promising results.^38,40,41 Future studies could focus on applying these methodologies to multi-institution datasets. In addition, the development data were older (2010-2011). However, we do not expect, with the possible exception of COVID-19 (coronavirus disease 2019), that changes over time in the practice or patient parameters would invalidate our findings.

Additionally, implementation of MC-EWS should be piloted to measure whether this machine learning automatic alert system can trigger consistent reactions (efferent limb of the rapid response system) resulting in earlier diagnostic and therapeutic interventions. Other machine learning scores have been implemented as afferent limbs of rapid response systems. While they have resulted in improved standardization of care,⁴² they are often not perceived as helpful by providers.⁴³ It will be important to explore how to design a rapid response system that integrates cleanly into existing workflows, is trusted by caregivers, and assesses counterbalance measures to ensure that it does not adversely affect the volumes of RRT calls and ICU transfers.

CONCLUSION

We developed the Mayo Clinic Early Warning Score, a machine learning model that accurately predicts acute deterioration in the general care wards and is ready for an implementation study as the afferent limb of a rapid response system.

FUNDING

This study was funded by Mayo Clinic’s Department of Medicine internal research funds.