Less is More: Detecting Clinical Deterioration in the Hospital with Machine Learning Using Only Age, Heart Rate, and Respiratory Rate

Abstract

Aim:

We sought to develop a machine learning analytic (eCART Lite) for predicting clinical deterioration using only age, heart rate, and respiratory data, which can be pulled in real-time from patient monitors and updated continuously without need for additional inputs or cumbersome electronic health record integrations.

Methods:

We utilized a multicenter dataset of adult admissions from five hospitals. We trained a gradient boosted machine model using only current and 24-hour trended heart rate, respiratory rate, and patient age to predict the probability of intensive care unit (ICU) transfer, death, or the combined outcome of ICU transfer or death. The area under the receiver operating characteristic curve (AUC) was calculated in the validation cohort and compared to those for the Modified Early Warning Score (MEWS), National Early Warning Score (NEWS), and eCARTv2, a previously-described, 27-variable, cubic spline, logistic regression model without trends.

Results:

Of the 556,848 included admissions, 19,509 (3.5%) were transferred to an ICU and 5,764 (1.0%) died within 24 hours of a ward observation. eCART Lite significantly outperformed the MEWS, NEWS, and eCART v2 for predicting ICU transfer (0.792 vs 0.711, 0.743, and 0.775, respectively; p<0.01) and the combined outcome (0.795 vs 0.722, 0.755, 0.786, respectively; p<0.01). Two of the strongest predictors were respiratory rate and heart rate.

Conclusion:

Using only three inputs, we developed a tool for predicting clinical deterioration that is similarly or more accurate than commonly-used algorithms, with potential for use in inpatient settings with limited resources or in scenarios where low-cost tools are needed.

Introduction

Traditional early warning scores such as the Modified Early Warning Score (MEWS) rely on a full set of vital signs to risk stratify hospitalized patients.^1–2 Machine learning algorithms, which add laboratory results and demographics as inputs, have proven to be more accurate, resulting in fewer false alarms and better detection but generally require complex electronic health record (EHR) integrations to pull all the required variables in real time. In prior work, our team has developed the electronic Cardiac Arrest Risk Triage (eCART) Score to accurately quantify a patient’s risk of clinical deterioration, using vital signs, laboratory data, and demographic information.³ A 27-variable cubic spline logistic regression version of the tool (eCARTv2) is currently running live in production at numerous acute care hospitals, integrated with various electronic health records to extract the required inputs in real-time and drive clinical workflows. Meanwhile, our research team has continued to iterate on the model, using increasingly larger data sets, advanced analytics and additional inputs with progressive increases in accuracy.³

We have repeatedly demonstrated that variables such as respiratory rate, heart rate, and age are the most important predictor variables, even in more complex prediction algorithms that utilize a wide-range of physiologic variables.^4–5 These three variables are globally utilized and collected in a variety of patient care settings, including those with limited electronic health record technologies and resources. We have also demonstrated that adding vital sign trends to early detection models, in addition to current vital sign values, improves accuracy.⁶ Limiting the number of inputs and removing the requirement for an electronic health record integration could lower the bar for use of these tools and allow wider applicability across not only traditional and established inpatient hospitals but also low-resource inpatient and acute care settings and potentially even post-acute environments. These various low-resource environments are often not equipped for or accustomed to routine collection of labs or other clinical elements at the same frequency as large inpatient hospital settings. Further, many healthcare settings may not have access to complex electronic health record systems needed to run existing risk prediction algorithms which have to be fully integrated with these technologies. The development of an abbreviated eCART model that requires fewer and easily obtainable variables could provide the answer to early detection of critical illness across a wide variety of healthcare settings.

In this study, we sought to develop a machine learning analytic (eCART Lite) for predicting clinical deterioration using only age, heart rate, and respiratory data, which can be pulled in real-time from patient monitors and updated continuously without need for additional inputs or expansive electronic health record (EHR) systems.

Methods

We utilized a previously described multicenter dataset of 556,848 adult medical-surgical admissions from the University of Chicago and from four NorthShore University HealthSystem hospitals in Illinois (Evanston, Glenbrook, Highland Park, and Skokie hospitals). The dataset was split prospectively by site into a 70% derivation set and a 30% validation set, consistent with our prior studies. We trained a gradient boosted machine (GBM) model, which is a tree-based ensemble that grows decision trees sequentially to better predict difficult cases, in the derivation cohort using only current and 24-hour trended heart rate and respiratory rate along with patient age to predict the probability of ICU transfer, death, or the combined outcome of ICU transfer or death within 24 hours. Hyperparameters were tuned using ten-fold cross-validation in the derivation dataset. AUC, sensitivity, and specificity of the new model were compared to eCART v2 developed in our previous research,³ the MEWS,² and the National Early Warning Score version (NEWS)⁷ using whether the outcome occurred within the next 24 hours of each variable observation. This study was approved by the University of Chicago Biological Sciences Division Institutional Review Board (IRB#17-1342). Analyses were performed using Stata version 15.1 (StataCorps; College Station, Texas) and R version 3.6.1 (The R Foundation for Statistical Computing; Vienna, Austria).

Results

A total of 556,848 adult medical-surgical admissions from the University of Chicago and from four NorthShore University HealthSystem hospitals in Illinois (Evanston, Glenbrook, Highland Park, and Skokie hospitals) were included in the study. Of those, 167,122 unique encounters were included in the validation dataset used for analysis. Of those, 4,841 (2.90%) were transferred to an ICU, 927 (0.55%) died, and 5,768 (3.45%) experienced either ICU transfer or death within 24 hours of a ward observation (Table 1). Patients who had an outcome were older (68 vs 62, p<0.01), more likely to be male (52% vs 41%, p<0.01), more likely to be Black (33% vs 29%, p<0.01) and had a longer median length of stay (10.1 days vs 2.8 days, p<0.01). eCART Lite significantly outperformed the MEWS, NEWS and eCARTv2 in predicting ICU transfer (0.792 vs 0.711, 0.743, and 0.775, respectively; p<0.01) and the combined outcome of ICU transfer or death (0.795 vs 0.722, 0.755, 0.786, respectively; p<0.01) (Table 2). However, eCART Lite had the lowest AUC for predicting mortality alone, with eCARTv2 outperforming all the others, with an AUC of 0.916. The three strongest weighted variables in the model included, in decreasing order of importance, maximum respiratory rate in the preceding 24 hours, current respiratory rate, and current heart rate (Figure 1). The Receiver Operating Characteristic curve for predicting the combined outcome of death or ward-to-ICU transfer for each score is shown in Figure 2.

Table 1.

Characteristics by cohort

Characteristic	Derivation (n=389,726)	Validation (n=167,122)	p-value
Age, years (median, IQR)	62 (46, 76)	62 (45, 76)	0.28
Female (n, %)	226,874 (58)	98,747 (59)	<0.01
Black (n, %)	109,692 (28)	52,884 (32)	<0.01
Hispanic (n, %)	16,788 (4)	7,958 (5)	<0.01
Length of stay, days (median, IQR)	3 (2, 5)	3 (2, 5)	<0.01
Ward-to-ICU transfer ever during stay (n, %)	14,606 (4)	4,903 (3)	<0.01
In-hospital mortality ever during stay (n, %)	4,437 (1)	1,327 (1)	<0.01
Ward-to-ICU transfer within first 24 hrs	14,254 (4)	4,841 (3)	<0.01
In-hospital mortality within first 24 hrs	2,681 (1)	927 (1)	<0.01
Hospital Distribution a
NorthShore HealthSystem Evanston Hospital (n, %)	79,896 (21)	31,617 (19)	-
NorthShore HealthSystem Glenbrook Hospital (n, %)	49,570 (13)	20,895 (13)	-
NorthShore HealthSystem Highland Park Hospital (n, %)	41,547 (11)	17,930 (11)	-
NorthShore HealthSystem Skokie Hospital (n, %)	37,994 (10)	19,515 (12)	-
University of Chicago Medicine (n, %)	178,504 (46)	76,642 (46)	-

Abbreviations: ICU = Intensive Care Unit; IQR = interquartile range (25,75); hrs = hours

^a2,738 encounters from the NorthShore HealthSystem did not have location data and thus could not be traced back to a specific hospital

Table 2.

Comparing Area Under the Receiver Operating Curve (AUC) of eCART Lite to commonly used early warning scores

Outcome	Area Under Curve (AUC)
	MEWS	NEWS	eCART v2	eCART Lite
Ward to ICU transfer (n = 4,841)	0.711 (0.709, 0.713)	0.743 (0.741, 0.745)	0.775 (0.773, 0.776)	0.792 (0.791, 0.794)
Death (n = 927)	0.854 (0.849, 0.859)	0.892 (0.888, 0.896)	0.916 (0.913, 0.919)	0.822 (0.817, 0.827)
Combined outcome (n = 5,598)	0.722 (0.720, 0.724)	0.755 (0.753, 0.756)	0.786 (0.784, 0.787)	0.795 (0.793, 0.796)

Abbreviations: ICU = Intensive Care Unit; eCART = electronic Cardiac Arrest Risk Triage; MEWS = Modified Early Warning Score; NEWS = National Early Warning Score

Figure 1. Variable Importance Plot.

The relative importance of each variable to the GBM model, scaled to a maximum of 100. RR = respiratory rate; HR = heart rate; max = highest value in last 24 hours; min = lowest value in last 24 hours; median = median of values in last 24 hours; mean = mean of values in last 24 hours; sd = standard deviation of values in last 24 hours; delta = difference between most recent and previous value; slope = difference between latest and earliest values in last 24 hours.

Figure 2. Area Under the Receiver Operating Curve Plots for Each Early Warning Score.

Risk score specificity and sensitivity for predicting the combined outcome of death or ward-to-ICU transfer for each early warning score. ICU = Intensive Care Unit; eCART = electronic Cardiac Arrest Risk Triage; MEWS = Modified Early Warning Score; NEWS = National Early Warning Score

Discussion

In this multicenter study, we found that the eCART Lite score was significantly more accurate than the MEWS, NEWS, and eCART v2 for predicting transfer to the intensive care unit (ICU) or the combined outcome of ICU transfer or death. In addition, we found that respiratory-related vital signs were the strongest predictors of these outcomes, with maximum respiratory rate in the preceding 24 hours being the most accurate predictor. This reflects similar findings in acute hospital wards where respiratory rate was also a highly predictive variable in identifying deterioration.^3,8 Further, models should incorporate trends to account for baseline abnormalities in these vitals, similar to what we have previously found in ward models where the addition of trends significantly improved accuracy.⁶ This suggests that vital signs, such as respiratory rate, that are present in almost all care settings are critical to the accuracy of risk prediction algorithms and that an abbreviated tool, such as eCART Lite, would be highly practicable across a variety of clinical care settings due to its simplicity, ease of use, and cost-effectiveness.

We found that eCART Lite significantly better predictor of some deterioration outcomes than eCART v2, the MEWS, and the NEWS. Thus, eCART Lite may increase detection of high-risk patients if implemented in real-time and could be widely applicable across acute hospital settings with limited access to electronic health record variables (i.e. new health centers or those with limited funding or technological resources) or potentially non-acute care facilities, such as skilled nursing facilities, long term care settings, or even home health care. This kind of early detection could direct attention and resources to the highest risk patients and improve patient outcomes.

The current study has several limitations. First, this study cohort is from five hospital systems in Illinois, and the results may not be generalizable to health centers in other geographical regions. Further, manually collected variables, such as respiratory rate, are more prone to human error which could have affected the results of this study. Additionally, there is potential that the high predictive value of age, as shown in Figure 1, may be confounded by the limited number of variables used in the eCART Lite model; further investigation is needed to determine the true predictive accuracy of age in both limited variable models and models that use a larger set of physiologic and clinical variables, as the current literature is mixed.^4,9 Finally, eCART Lite performed more poorly when predicting mortality alone; thus, further investigation of limited variable models is needed to improve the accuracy of mortality prediction.

Conclusions

Using machine learning and only three readily-available patient parameters, we developed a tool for predicting impending clinical deterioration that is significantly more accurate than many widely used tools that require more inputs. Such a model could be used in a variety of healthcare settings, especially low-resource acute care facilities and potentially post-acute care or home health care settings if paired with telemetry or other sensors which measure heart rate and respiratory rate, to improve the identification of high-risk patients in real time when additional data and expansive EHR resources are not available.