Derivation of a cardiac arrest prediction model using ward vital signs

Abstract

Objective

Rapid response team (RRT) activation criteria were created using expert opinion and have demonstrated variable accuracy in previous studies. We developed a cardiac arrest risk triage (CART) score to predict cardiac arrest (CA) and compared it to the Modified Early Warning Score (MEWS), a commonly cited RRT activation criterion.

Design

A retrospective cohort study.

Setting

An academic medical center in the United States.

Patients

All patients hospitalized from November 2008 to January 2011 who had documented ward vital signs were included in the study. These patients were divided into three cohorts: patients who suffered a CA on the wards, patients who had a ward to intensive care unit (ICU) transfer, and patients who had neither of these outcomes (controls).

Interventions

None.

Measurements and Main Results

Ward vital signs from admission until discharge, ICU transfer, or ward CA were extracted from the medical record. Multivariate logistic regression was used to predict CA, and the CART score was calculated using the regression coefficients. The model was validated by comparing its accuracy for detecting ICU transfer to the MEWS. Each patient’s maximum score prior to CA, ICU transfer, or discharge was used to compare the areas under the receiver operating characteristic curves (AUC) between the two models. Eighty-eight CA patients, 2820 ICU transfers, and 44519 controls were included in the study. The CART score more accurately predicted CA than the MEWS (AUC 0.84 vs. 0.76;P=0.001). At a specificity of 89.9%, the CART score had a sensitivity of 53.4% compared to 47.7% for the MEWS. The CART score also predicted ICU transfer better than the MEWS (AUC 0.71 vs. 0.67;P<0.001).

Conclusions

The CART score is simpler and more accurately detected CA and ICU transfer than the MEWS. Implementation of this tool may decrease RRT resource utilization and provide a better opportunity to improve patient outcomes than the MEWS.

INTRODUCTION

Approximately 200,000 in-hospital cardiac arrests (CAs) occur in the United States each year, and only 20% of these patients survive to discharge.(1, 2) Despite decades of research, this dismal survival rate has changed little.(1, 3) There is evidence that many CAs may be preventable and that warning signs such as abnormal vital signs occur hours before the event.(4–8) This evidence led to the development of rapid response teams (RRTs), a multidisciplinary group of trained caregivers who bring critical care resources to deteriorating patients on the hospital wards.(9) Despite the common sense nature of this intervention, clinical trials have failed to demonstrate a consistent improvement in hospital-wide CA rates or mortality.(10, 11)

There are over 50 different published criteria designed to activate the RRT and direct them to critically ill ward patients.(12, 13) Importantly, these criteria were created using expert opinion and were not statistically derived using ward vital signs.(14, 15) In the Medical Early Response Intervention and Therapy (MERIT) study, the only multicenter randomized trial of RRTs, the criteria used to activate the RRT had a sensitivity and specificity of less than 50% for CA, ICU transfer, or death.(10, 15) In addition, when present, the activation criteria triggered the RRT less than 15 minutes before the adverse event in most cases.(10, 15) Furthermore, studies investigating the accuracy of published activation criteria found a wide range of sensitivities and specificities.(13, 14) The implementation of RRT activation systems with poor accuracy results in critically ill patients remaining on the wards without needed interventions and an overburdened system due to a high rate of false alarms. Development of an accurate prediction tool to detect critically ill patients on the wards would improve identification of at-risk patients and decrease false-positives that lead to alarm-fatigue and increase healthcare costs.

No study to date has derived a ward CA prediction model using vital signs. The aim of this study was to derive the CA risk triage (CART) score using ward vital signs and then compare it to the Modified Early Warning Score (MEWS), a commonly cited RRT activation criterion (Figure 1).(16–19)

Figure 1.

Modified Early Warning Score (MEWS)

Abbreviations: Unresp, unresponsive; BP, blood pressure

MATERIALS AND METHODS

OVERVIEW

We performed a retrospective cohort study to derive the CART score to identify CA patients on the wards and compared its accuracy to the MEWS. Although the primary goal of this study was to develop a CA risk prediction tool, we also validated the model by comparing its ability to identify patients transferred to the intensive care unit (ICU) to the MEWS. This patient population was chosen for validation because they usually demonstrate evidence of physiologic deterioration and would likely suffer a CA if not detected.

The study protocol, consent, and data collection mechanisms were approved by the University of Chicago Institutional Review Board. A waiver of consent was granted based on minimal harm and general impracticability. Collection of patient information was designed to comply with the Health Insurance Portability and Accountability Act of 1996 (HIPAA) regulations.

STUDY SETTING AND POPULATION

The study was conducted at an academic, tertiary care hospital with approximately 500 inpatient beds. All patients hospitalized from November 1, 2008 until January 31, 2011 who had vital signs documented on the wards (including telemetry units outside the ICU) were included in the study. Patients were separated into three groups: those who suffered a ward CA, those transferred from the ward to the ICU who did not suffer a ward CA, and those who were discharged without having either event (controls).

Patients who suffered a CA, defined as the loss of a palpable pulse with attempted resuscitation, on the ward were identified using a prospectively collected and verified CA quality improvement database that has been previously described.(20–22) If a patient had more than one CA, only data prior to the first arrest were used. Those who had both a ward CA and a ward to ICU transfer were only counted as CA patients. ICU transfer patients were identified using the hospital’s admission, transfer, and discharge administrative database. If a patient had more than one ward to ICU transfer, only data before the first event were included.

Our hospital has had an RRT in place since 2008 that is led by a critical care nurse and respiratory therapist with consultation from a hospitalist attending physician and/or pharmacist upon request. The RRT activation criteria include the general descriptors of “tachypnea,” “tachycardia,” “hypotension,” and “staff worry,” but specific vital sign thresholds are not stated. This team is separate from the team that responds to a CA.

DATA COLLECTION

Demographic data for all study patients were obtained from administrative databases. Time and location stamped vital signs, including temperature, blood pressure, heart rate, oxygen saturation, respiratory rate, and mental status were obtained from the hospital’s electronic medical record (EPIC, Verona, Wisconsin). Pulse pressure index (pulse pressure divided by systolic blood pressure) was also calculated. Mental status was collapsed from four drop-down menu fields in the electronic medical record (orientation, level of consciousness, motor response, and responsiveness) into one score (alert, responsive to voice, responsive to pain, and unresponsive (AVPU)).(23)

Only ward vital signs from admission until discharge (controls), first ICU transfer (ICU patients) or first ward CA were included in the study. If a CA patient also had a previous ICU transfer, only vital signs following the patient’s last ICU transfer until CA were included. Vital signs within 30 minutes of CA were excluded because the goal was to predict the event with enough time to potentially intervene.

MODEL DERIVATION AND VALIDATION

Each patient’s maximum and minimum value of each vital sign documented on the ward from admission until discharge (controls) or CA was used for model derivation because patients have varying numbers of vital signs collected on the ward and may have abnormalities at different time points before CA. All vital signs and patient age were investigated as potential predictors of CA. Vital sign and age cut-off thresholds were chosen using inflection points from locally weighted least squares regression (LOWESS) smoother curves and refined using univariate logistic regression by combining categories with similar odds ratios. Stepwise multivariate logistic regression with backwards elimination was performed to derive the final model using the Akaike information criterion (AIC). This measure of model fit penalizes models with large numbers of variables, which is consistent with our goal of developing a simple, parsimonious model. To create the CART score, the beta coefficients from the final multivariate model were multiplied by a factor to create a scoring system with cut-off scores with the same sensitivity and specificity as the MEWS at the threshold often cited in the literature (>4) to allow direct comparison between the scoring systems.(16, 24)

After model derivation, every simultaneous vital sign set for CA and control patients was scored using the MEWS and the CART score. If any variable was missing for score calculation, the most recent value was imputed, similar to what would be done in clinical practice. If a patient had no previous values of the missing variable then a normal value was imputed.(25, 26) Each patient’s highest MEWS and CART score was used to create receiver operating characteristic (ROC) curves for detecting CA. The area under the ROC curve (AUC) for each model was calculated by the trapezoidal rule, and the ROC curves were compared using the method of DeLong.(27) This analysis was repeated during model validation by scoring every vital sign set for ICU transfer and control patients and then comparing the ROC curves for the MEWS and CART score.

CART SCORE CHANGE OVER TIME

The mean CART scores for CA patients, controls, and ICU transfer patients were compared every eight hours in the 48-hour time period prior to the event, using vital sign sets measured closest to but before each eight-hour time point. A randomly selected 48-hour period was used for each control patient for score calculation.

All tests of significance used a 2-sided P<0.05. Statistical analyses were completed using R (R Foundation for Statistical Computing, Vienna, Austria) and Stata version 11.2 (StataCorp, College Station, Texas).

RESULTS

PATIENT CHARACTERISTICS

A total of 47427 patients were included in the study (88 CA patients, 2820 ICU transfers, and 44519 controls). One additional CA occurred on the ward during the study period, but this patient was not included in the study because there were no documented ward vital signs before the event. Patient demographic data are shown in Table 1. Compared to controls, CA patients were older (mean age 64±16 vs. 54±18;P<0.001), had a longer length of stay (median 11 (IQR 5–26) vs. 3 (IQR 1–5) days;P<0.001), and had a lower survival to discharge rate (31% vs. 99.7%;P<0.001). CA patients were more likely to have a prior ICU stay (41% vs. 9%;P<0.001) and RRT call during the study period (7% vs. 0.3%;P<0.001) than control patients. Compared to controls, ICU transfer patients were older (mean age 60±16 vs. 54±18;P<0.001), had a longer length of stay (median 11 (IQR 7–19) vs. 3 (IQR 1–5) days;P<0.001), and lower survival to discharge rate (85% vs. 99.7%;P<0.001).

Table 1.

Patient characteristics

Characteristic	Cardiac arrest patients (n=88)	ICU transfer patients (n=2820)	Controls (n=44519)
Age, mean (SD), years	64 (16)*	60 (16)*	54 (18)
Female sex	50 (57)	1364 (48)*	25444 (57)
Admitting service
Medical	65 (73)*	1560 (55)*	27804 (62)
Surgical	23 (26)*	1223 (43)*	13962 (31)
Unknown	0 (0)*	37 (1)*	2753 (6)
Length of stay, median (IQR)	11 (5–26)*	11 (7–19)*	3 (1–5)
Hours of ward data, median (IQR)	51 (22–166)	40 (13–103)*	51 (26–108)
Prior ICU stay	36 (41)*	423 (15)*	3998 (9)
RRT call during study period	6 (7)*	274 (10)*	116 (0.3)
Survived to discharge	27 (31)*	2410 (85)*	44399 (99.7)

Abbreviations: IQR, interquartile range; RRT, rapid response team; ICU, intensive care unit.

^*Denotes statistically different than controls at P<0.05

Data are shown as number (percentage) unless otherwise specified.

CARDIAC ARREST RISK SCORE DERIVATION

Stepwise regression resulted in a final model that contained respiratory rate, heart rate, diastolic blood pressure, pulse pressure index, and age (Table 2). Minimum respiratory rate and minimum heart rate were not investigated in the multivariable model because they were not significant predictors of CA in univariate analysis. Pulse pressure index was dropped from the final model for simplicity because it must be calculated and is less intuitive than traditional vital signs, and its removal did not change the AUC of the model (0.84 for both models). The predictor cut-offs, beta coefficients, and the CART score are shown in Table 3.

Table 2.

Model derivation results for candidate models in stepwise logistic regression.

Model variables*	Variable removed	P-value for variable removal	AIC
RR, HR, DBP, Age, PPI, O2Sat, SBP, Temp, MS	Full model	-	1145
RR, HR, DBP, Age, PPI, O2Sat, SBP, Max Temp, MS	Min Temp	0.96	1143
RR, HR, DBP, Age, PPI, O2Sat, Max SBP, Max Temp, MS	Min SBP	0.72	1139
RR, HR, DBP, Age, Min PPI, O2Sat, Max SBP, Max Temp, MS	Max PPI	0.66	1135
RR, HR, DBP, Age, Min PPI, O2Sat, Max SBP, Max Temp	Mental status	0.36	1134
RR, HR, DBP, Age, Min PPI, O2Sat, Max SBP	Max Temp	0.37	1133
RR, HR, DBP, Age, Min PPI, O2Sat	Max SBP	0.29	1132
RR, HR, DBP, Age, Min PPI	O2Sat	0.26	1131
RR, HR, Min DBP, Age, Min PPI	Max DBP	0.10	1131

Abbreviations: AIC, Akaike information criteria; AUC, area under the receiver operating characteristic curve; RR, respiratory rate; HR, heart rate; DBP, diastolic blood pressure; PPI, pulse pressure index; O2sat, oxygen saturation; SBP, systolic blood pressure; Temp, temperature; MS, mental status; Max, maximum; Min, minimum

^*Variables are both maximum and minimum vital signs unless otherwise noted except oxygen saturation (minimum only), heart rate (maximum only), and respiratory rate (maximum only).

Table 3.

Derived cardiac arrest prediction model

Vital Sign	Cardiac arrests, n (%)a [n=88]	Controls, n (%)a [n=44519]	Beta coefficient	Score
Respiratory rate
<21	21 (24)	29997 (67)	Reference	0
21–23	19 (22)	8118 (18)	0.9	8
24–25	17 (19)	3688 (8)	1.4	12
26–29	12 (14)	1732 (4)	1.7	15
>29	19 (22)	984 (2)	2.4	22
Heart rate
<110	41 (47)	33710 (76)	Reference	0
110–139	32 (36)	9911 (22)	0.5	4
>139	15 (17)	898 (2)	1.4	13
Diastolic BP
>49	42 (48)	33783 (76)	Reference	0
40–49	28 (32)	8869 (20)	0.5	4
35–39	6 (7)	1007 (2)	0.6	6
<35	12 (14)	860 (2)	1.5	13
Age
<55	22 (25)	21025 (47)	Reference	0
55–69	27 (31)	13962 (31)	0.5	4
>69	39 (44)	9532 (21)	1.0	9

^aResults reported are number (percent) of cardiac arrest and control patients with maximum (respiratory rate and heart rate) and minimum (diastolic blood pressure) vital sign values in each category.

Abbreviations: BP, blood pressure

COMPARISON TO MEWS

To create the CART score, beta coefficients were multiplied by a factor of nine, as shown in Table 3, because this resulted in a model containing cut-points with the same sensitivity and specificity as the MEWS at cut-off >4. The CART score was a better predictor of CA than MEWS (AUC 0.84 vs. 0.76;P=0.001, Figure 2). At a specificity of 89.9%, the CART score (cut-off >17) had a sensitivity of 53.4% compared to the MEWS (cut-off >4) sensitivity of 47.7%. For those CA patients detected by both systems at these thresholds, the CART score detected CA earlier than the MEWS (median 48 hours vs. 42 hours prior to the event;P=0.85), but this difference was not statistically significant. Compared to the MEWS at cut-off >4 (specificity 89.9%), the CART score at cut-off >20 had a specificity of 91.9% with the same sensitivity (47.7%). This would have resulted in 890 less patient calls over the study period (3648 vs. 4538 calls) while detecting the same number of CAs. In addition, the CART score predicted ICU transfer better than the MEWS (AUC 0.71 vs. 0.67;P<0.001). Both the CART score (AUC 0.84 vs. 0.71; P<0.001) and MEWS (AUC 0.76 vs. 0.67; P<0.001) predicted CA better than ICU transfer.

Figure 2.

Receiver operating characteristic curves of the CART score and MEWS.

Abbreviations: AUC, area under the receiver operating characteristic curve; CART, cardiac arrest risk triage; MEWS, modified early warning score

Figure 3 illustrates the cumulative percent of both CA patients and the entire hospital population on the wards as the CART score cut-off threshold decreases. Drawing a vertical line up from a specific CART score denotes the percent of the ward population with a score of that value or higher and the percent of CA patients that were identified at that cut-off threshold.

Figure 3.

Cumulative percentage of cardiac arrest patients and percentage of the total population on the hospital wards for different maximum CART score cut-offs. For example, at a cut-off of 23 approximately 35% of cardiac arrest patients and 5% of the total hospitalized population are identified.

Abbreviations: CART, cardiac arrest risk triage

CART CHANGE OVER TIME

The change in mean CART over time for CA, ICU transfer, and control patients is shown in Figure 4. The mean CART scores were statistically different between CA patients and controls (8±6 vs. 4±4;P<0.001) and between ICU transfer patients and controls (6±6 vs. 4±4;P<0.001) at 48 hours prior to the event, and the differences increased leading up to the event. Mean CART scores were significantly higher for CA patients compared to ICU transfers at 48 hours and 24 hours but not at 30 minutes before the event (9±8 vs. 10±10;P=0.08).

Figure 4.

Change in CART over time prior to cardiac arrest, ICU transfer, and discharge.*

*Time −0.5h= 30 minutes before cardiac arrest, time of last ICU patient vital sign set, or time of last vital sign set in random 48 hours (controls)

Abbreviations: CART, cardiac arrest risk triage; CA, cardiac arrest; ICU, intensive care unit; TF, transfer

DISCUSSION

In this retrospective cohort study of over 47,000 patients, we derived a CA prediction tool using ward vital signs that contains respiratory rate, heart rate, diastolic blood pressure, and age. Our model has fewer variables than the MEWS and more accurately identified patients who had a CA or ICU transfer. In addition, it identified CA patients a median of 48 hours before the event at a cut-off with 90% specificity. By deriving a model using ward vital signs, our study helps clarify the best predictors of CA and provides a simple system that, once externally validated, could be implemented in other hospitals to activate their RRT.

There are several possible reasons why the CART score outperformed the MEWS in this study. First, the MEWS utilizes poor predictors of adverse outcomes such as low respiratory rate and bradycardia, while excluding significant predictors such as diastolic blood pressure and age. We have previously shown that diastolic blood pressure is a significant predictor of ward CA.(28) Since most clinicians focus on systolic blood pressure when initiating therapy for shock, the addition of diastolic blood pressure into our prediction model instead of systolic blood pressure is especially useful. The importance of including age in a predictive model for hospitalized patients has been demonstrated in other studies. For example, in a study by Duckett and colleagues, including age in a scoring system that used admission vital signs to predict in-hospital mortality increased their model’s AUC from 0.74 to 0.81.(29) Also, Smith et al. studied the relationship between admission vital signs and hospital mortality and found that age was a significant independent predictor of mortality.(30) Inclusion of age in our model means that a 70 year-old patient will need less vital sign abnormalities than a 30-year old to get the same CART score, potentially reflecting the lower physiologic reserve and higher prevalence of CA in older patients. In addition, our model weighs vital signs based on their predictive ability. For example, respiratory rate has previously been shown to be the most predictive vital sign for adverse events on the wards.(31–33) The weighting in the CART score reflects this, as higher respiratory rates greatly increase a patient’s score and to a higher degree than other vital signs. This contrasts with MEWS, which scores all vital signs from zero to three. It is important to note that our model has more cut-points than the MEWS, providing institutions more options to choose from with different specificity levels based on their individual resources.

Previous investigators have created prediction tools to detect adverse events in hospitalized patients other than CA. For example, Cuthbertson and colleagues created a model to predict ICU transfer in a case-control study of surgical patients.(33) They derived discriminant functions that included respiratory rate, heart rate, and oxygen saturation that could detect ICU transfer with a high degree of accuracy. However, this outcome has limited generalizability as ICU transfer criteria vary among institutions. Cretikos and colleagues performed a case-control study evaluating the accuracy of different cut-off thresholds of the vital signs used in the MERIT trial RRT activation criteria.(34) Their final model had a sensitivity of 59.6% and specificity of 93.7% for the combined endpoint of ICU transfer, CA, or death. Recently, Prytherch and colleagues developed the VitalPAC™ Early Warning Score (ViEWS) utilizing existing knowledge of the relationship between physiologic data and adverse outcomes and “trial and error.” Their model had an AUC of 0.89 for detecting in-hospital mortality in medical patients within 24 hours. Despite these efforts, no study to date has derived a prediction model to detect ward CA using data from the entire hospitalized population.

In clinical practice, the CART score could be used to trigger increased frequency of vital sign collection and monitoring, consultation by a critical care physician, or even automatic triggering of an RRT. The CART score is simple enough to be calculated at the bedside using a calculator or hand-held electronic device. It could also be included in the electronic medical record as a clinical decision support tool. Implementation of a CA risk score into the medical record would provide a summary measure of how likely a patient is to suffer physiologic deterioration. It could inform both ward staff and a critical care consult team regarding which patients are at highest risk of adverse events, thus potentially avoiding some ward CAs. This knowledge is especially important in light of the increasing number of patient hand-offs that occur in hospitals today. It is important to note that identification of patients likely to suffer a CA is just one part of the “chain of prevention” as discussed by Smith.(35) Additional work needs to be done to improve other aspects of the hospital safety net, including the triage decisions and interventions provided for these critically ill patients once their deterioration is identified.

Our study has several strengths compared to other published vital sign investigations. First, our study encompasses a cohort of over 47,000 patients in a 27-month period. Previous studies predominantly used a case-control design that does not provide information regarding how a system would perform if implemented in the entire hospital.(32–34, 36, 37) Second, we focused on ward CA as the primary outcome as opposed to ICU transfer or hospital mortality. The goal of our study was to develop a clinical prediction tool that can assist in the identification of patients currently missed by physicians and staff on the ward. CA patients are failures of the system as most have abnormal vital signs and yet are inappropriately left on the ward. Our prediction model is optimized for these high-risk patients who are most likely to benefit from a clinical support tool. Some investigators have used ICU transfer as the outcome of interest, but transfer criteria vary in different hospitals and among different physicians making it a less generalizable outcome. Also, these patients are already identified by current systems and then transferred to the ICU and thus would be less likely to benefit from a clinical prediction tool than CA patients. In addition, other studies have investigated vital signs for their ability to predict hospital mortality.(16, 38, 39) Although important, hospital mortality is a complex variable that is influenced by factors such as do-not-resuscitate status and can occur both in the ICU and on the ward. By using vital signs prior to CA, we created a predictive model that focuses on the optimal target for RRT intervention: those patients missed by the current hospital safety net.

Our study has several limitations. First, this is a single-center study at an academic institution, and our findings may not be generalizable to some hospitals. The survival to discharge rate for CA patients in this study (31%) is higher than the average reported from previous studies.(3, 40) However, this survival rate is within the range reported in the literature.(41–43) In addition, we only tested the CART score against the MEWS, so future studies are needed to validate our model in other clinical settings and against other systems, such as the ViEWS. Also, our sample of CA patients was not large enough to split into a derivation and validation cohort. However, we validated the CART score on a separate at-risk population: patients transferred to the ICU. Finally, when using this large patient database we cannot be sure that all ICU transfers were “unexpected,” which may in part explain why the AUCs for both models decreased for predicting this event.

CONCLUSIONS

We derived a CA prediction tool using ward vital signs and validated it as a model for physiologic decline by confirming its ability to predict ICU transfer. Our model performed better than the MEWS, contains fewer variables, and detected CA a median of 48 hours prior to the event with a high specificity. Implementation of this system would allow identification of those patients at greatest risk for suffering adverse events on the wards while minimizing unnecessary false positives that can overburden the system.