Abstract
Background:
Early identification of children at risk for cardiac arrest would allow for skill training associated with improved outcomes and provides a prevention opportunity.
Objective:
Develop and assess a predictive model for cardiopulmonary arrest using data available in the first 4 hours.
Methods:
Data from PICU patients from 8 institutions included descriptive, severity of illness, cardiac arrest, and outcomes.
Results:
Of the 10074 patients, 120 satisfying inclusion criteria sustained a cardiac arrest and 67 (55.9%) died. In univariate analysis, patients with cardiac arrest prior to admission were over 6 times and those with cardiac arrests during the first 4 hours were over 50 times more likely to have a subsequent arrest. The multivariate logistic regression model performance was excellent (area under the ROC curve = 0.85 and Hosmer-Lemeshow statistic, p=0.35). The variables with the highest odds ratio’s for sustaining a cardiac arrest in the multivariable model were admission from an inpatient unit (8.23 (CI: 4.35 - 15.54)), and cardiac arrest in the first 4 hours (6.48 (CI: 2.07 - 20.36). The average risk predicted by the model was highest (11.6%) among children sustaining an arrest during hours >4 to 12 and continued to be high even for days after the risk assessment period; the average predicted risk was 9.5% for arrests that occurred after 8 PICU days.
Conclusions:
Patients at high risk of cardiac arrest can be identified with routinely available data after 4 hours. The cardiac arrest may occur relatively close to the risk assessment period or days later.
Keywords: Pediatric Intensive Care Unit, cardiac arrest, cardiopulmonary resuscitation, critical care, severity of illness, predictive analytics
Introduction
Cardiac arrests occur in approximately six thousands hospitalized children yearly.1–5 Increasingly, these cardiac arrests predominantly occur in the pediatric intensive care unit (PICU)6 as a result of physiological decompensation in spite of critical care monitoring and therapies.2,6,7 Notably, hospital survival rates following cardiac arrest are improving from rates of about 15% in the 1990’s to current hospital survival rates of approximately 50%.1,3,4,6,8–10
Improvements in cardiopulmonary resuscitation outcomes have been associated with both better organization and training of resuscitation teams, and improved resuscitation techniques.4,11,12 Recently, clinical identification of high risk patients and point-of-care, just-in-time bedside skill training focused on these high-risk patients demonstrated improved outcomes. Thus, identification of patients at risk for cardiac arrest has the practical importance of improved outcomes through focused skill training and situational awareness allowing pro-active team preparedness.13–15
Despite this potentially important clinical impact, there are few objective, early warning methods to assess cardiac arrest risk for children admitted to the PICU. Early warning scores have been used with variable success to identify patients at risk for clinical deterioration16,17 and a checklist has been developed in a single PICU to identify daily those patients at increased proximate risk.7,13 High risk events antecedent to cardiac arrests have also been identified in adults.18–21 Our aims were to identify clinical data available within the first 4 hours of PICU admission that are associated with pediatric cardiac arrest and to use these data to develop and assess a predictive model for assigning objective risks to suffer a cardiopulmonary arrest.
Methods
Data and Patients
The data for this analysis originated in the Trichotomous Outcome Prediction in Critical Care (TOPICC) study conducted by the Collaborative Pediatric Critical Care Research Network (CPCCRN) of the Eunice Kennedy Shriver National Institute of Child Health and Human Development. Data collection methods and the institutional characteristics have been previously described.22 There were seven sites and one was composed of two institutions. In brief, patients from newborn to less than 18 years were randomly selected and stratified by hospital from December 4, 2011 to April 7, 2013. Patients from both general/medical and cardiac/cardiovascular PICUs were included. Moribund patients (vital signs incompatible with life for the first two hours after PICU admission) were excluded. Only the first PICU admission during a hospitalization was included. In this analysis, we excluded all patients dying within the first 4 hours of PICU stay because we focused on predicting cardiac arrest after the first 4 hours (see below). The protocol was approved by all participating Institutional Review Boards. Other analyses utilizing this database have been published.1,22–26 In particular, a previous publication detailed the descriptive characteristics of those patients with cardiac arrest as well as the characteristics of those arrests and outcomes.1
Data included descriptive and demographic information (Table 1 and Table 2). A cardiac arrest was defined as chest compression for at least 1 minute and/or defibrillation.1 Admission source was classified as emergency department, inpatient unit, post intervention unit, or admission from another institution. Diagnosis was classified by system of primary dysfunction based on the reason for PICU admission; cardiovascular conditions were classified as congenital or acquired. Interventions included both surgery and interventional catheterization. Pre-ICU cardiac arrest included closed chest massage within 24 hours prior to hospitalization or after hospital admission, but prior to PICU admission. The Functional Status Scale (FSS) was used to describe baseline (pre-hospital admission) functional status as good (FSS 6, 7), mild dysfunction (FSS 8, 9), moderate dysfunction (FSS 10 – 15), severe dysfunction (FSS 16-21) and very severe dysfunction (FSS >21).27
Table 1.
Descriptive Data for the Total Population and Cardiac Arrest Patients After 4 Hours with Univariate Odds Ratios for Developing Cardiac Arrest.
| Variable | N (%) or Median (IQR) | Cardiac Arrest After 4 Hours (n (%)) | Univariate Odds Ratio (95% CI) for CA > 4 Hours | Significance Level (1) |
|---|---|---|---|---|
| Sample Size Surviving > 4 Hours (2) | 10074 | 120 (1.2%) | ||
| Median Age in Years | 3.7 (0.8,10.8) | |||
| 0 - <1 week | 317 (3.1%) | 11 (3.5%) | reference | <0.001 |
| 1 week - < 2 weeks | 175 (1.7%) | 9 (5.1%) | 1.51 (0.61-3.71) | |
| 2 weeks - <1 month | 137 (1.4%) | 3 (2.2%) | 0.62 (0.17-2.27) | |
| > 1 month to <12 months | 2162 (21.5%) | 47 (2.2%) | 0.62 (0.32-1.21) | |
| 12 months - <144 months | 5110 (50.7%) | 41 (0.8%) | 0.23 (0.11-0.44) | |
| >144 months | 2173 (21.6%) | 9 (0.4%) | 0.12 (0.05-0.28) | |
| Insurance (3,4) | 0.09 | |||
| Commercial | 4167 (41.4%) | 41 (1.0%) | reference | |
| Government | 5417 (53.8%) | 69 (1.3%) | 1.30 (0.88-1.92) | |
| Other/Unknown | 490 (4.9%) | 10 (2.0%) | 2.10 (1.04-4.21) | |
| Race (3) | (4) | |||
| Black | 2294 (22.8%) | 30 (1.3%) | Reference | |
| Caucasian | 5162 (51.2%) | 48 (0.9%) | 0.71 (0.45-1.12) | |
| Other/Unknown | 2618 (26.0%) | 42 (1.6%) | 1.23 (0.77-1.97) | |
| Baseline Functional Status(5) | ||||
| Good | 7230 (71.8%) | 77 (1.1%) | Reference | 0.09 |
| Mild | 1023 (10.2%) | 19 (1.9%) | 1.76 (1.06-2.92) | |
| Moderate | 1269 (12.6%) | 20 (1.8%) | 1.49 (0.91-2.44) | |
| Severe | 417 (4.1%) | 2 (0.5%) | 0.45 (0.11-1.83) | |
| Very Severe | 135 (1.3%) | 2 (1.5%) | 1.40 (0.34-5.75) | |
| Cardiac Arrest Prior to PICU Admission | <0.001 | |||
| Yes | 141 (1.4%) | 10 (7.1%) | 6.82 (3.49-13.32) | |
| No | 9933 (98.6%) | 110 (1.1%) | reference | |
| ICU Type | <0.001 | |||
| General | 8140 (80.8%) | 70 (0.9%) | reference | |
| Cardiac | 1929 (19.2%) | 50 (2.6%) | 3.07 (2.13-4.43) | |
| Intervention (6) and Non-Interventions | <0.001 | |||
| Post-Intervention Admission | 3795 (37.7%) | |||
| Post-intervention - Cardiac | 1406 (14.0%) | 30 (2.1%) | 1.57 (1.03-2.39) | |
| Post-intervention - Non-Cardiac | 2389 (23.7%) | 4 (0.2%) | 0.12 (0.04-0.33) | |
| Medical Admission (Non-Intervention) | 6279 (62.3%) | 86 (1.4%) | reference | |
| PICU Admission Status | 0.03 | |||
| Elective | 3666 (36.4%) | 32 (0.9%) | reference | |
| Emergency | 6408 (63.6%) | 88 (1.4%) | 1.58 (1.05-2.38) | |
| Admission Source | <0.001 | |||
| Emergency Department | 3248 (32.2%) | 27 (0.8%) | 0.93 (0.56-1.54) | |
| Routine Care Area | 1096 (10.9%) | 30 (2.7%) | 3.11 (1.90-5.11) | |
| Referral Hospital | 1935 (19.2%) | 29 (1.5%) | 1.68 (1.02-2.77) | |
| Post-Intervention Areas | 3795 (37.7%) | 34 (0.9%) | reference | |
| Arrest During First 4 Hours in PICU | <0.001 | |||
| Yes | 19 (0.2%) | 7 (36.8%) | 51.32 (19.84-132.76) | |
| No | 10055 (99.8%) | 113 (1.1%) | reference | |
Abbreviations: IQR = Interquartile Range; CA = Cardiac Arrest; FSS = Functional Status Scale
Wald Test
Sample includes those surviving > 4 hours in the PICU. Four patients died within the first 4 hours.
Other includes unknowns
Not included in univariate statistics and in multivariate modeling due to the high number of unknowns.
Good = FSS of 6,7; Mild Dysfunction = FSS 8,9; Moderate = FSS 10 - 15; Severe = FSS 16 - 21; Very Severe = FSS >21
Interventions include surgeries and interventional catheterizations
Table 2.
Diagnostic Information and Severity of Illness for the Total Population and Cardiac Arrest (CA) After 4 Hours with Univariate Odds Ratios for Developing Cardiac Arrest.
| Variable | N (%) or Median (IQR) (1) | Cardiac Arrest After 4 Hours (n (%)) | Univariate Odds Ratio (95% CI) for CA > 4 Hours | Significanc e Level (2) |
|---|---|---|---|---|
| Primary System of Dysfunction | <0.001 | |||
| Respiratory | 3375 (33.5%) | 36 (1.1%) | reference | |
| Cardiovascular Disease | 2427 (24.1%) | 63 (2.6%) | ||
| Congenital | 1755 (17.4%) | 43 (2.5%) | 2.33 (1.49-3.64) | |
| Acquired | 672 (6.7%) | 20 (3.0%) | 2.85 (1.64-4.95) | |
| Neurologic | 2022 (20.1%) | 6 (0.3%) | 0.28 (0.12-0.66) | |
| Miscellaneous/Other | 2250 (22.3%) | 15 (0.7%) | 0.62 (0.34-1.14) | |
| Cardiovascular Disease Diagnosis | <0.001 | |||
| None | 8009 (79.5%) | 60 (0.7%) | reference | |
| Acquired | 145 (1.4%) | 6 (4.1%) | 5.72 (2.43-13.46) | |
| Congenital | 1920 (19.1%) | 54 (2.8%) | 3.83 (2.65-5.56) | |
| Acute or Chronic Cancer Diagnosis | 0.11 | |||
| Yes | 730 (7.2%) | 4 (0.5%) | reference | |
| No | 9344 (92.8) | 116 (1.2%) | 2.28 (0.84-6.20) | |
| Trauma Diagnosis | 0.15 | |||
| Neurological | 472 (4.7%) | 2 (0.4%) | 0.35 (0.09-1.41) | |
| Non-Neurological | 178 (1.8%) | 4 (2.2%) | 1.88 (0.69-5.15) | |
| No Trauma | 9424 (93.5%) | 114 (1.2%) | reference | |
| Sepsis or Shock Diagnosis | <0.001 | |||
| Yes | 696 (6.9%) | 19 (2.7%) | 2.58 (1.57-4.23) | |
| No | 9378 (93.1%) | 101 (1.1%) | reference | |
| Respiratory Failure Diagnosis | 0.23 | |||
| Yes | 2780 (27.6%) | 39 (1.4%) | 1.27 (0.86-1.86) | |
| No | 7294 (72.4%) | 81 (1.1%) | reference | |
| PRISM Score | 2 (0, 5) | CA: 8 (3, 15) No CA: 2 (0, 5) | 1.13 (1.11-1.15) | <0.001 |
| Cardiac PRISM | 0 (0, 0) | CA: 0 (0, 3) No CA: 2 (0, 0) | 1.35 (1.27-1.45) | <0.001 |
| Cardiac PRISM for Patients with Cardiac Intervention (n=1408) | 0 (0, 3) | CA: 3 (0, 3) No CA: 0 (0, 3) | 1.48 (1.28-1.71) | <0.001 |
| Cardiac PRISM for Other Patients (n=8666) | 0 (0, 0) | CA: 0 (0, 3) No CA: 0 (0, 0) | 1.32 (1.22-1.43) | <0.001 |
| Respiratory PRISM | 0 (0, 1) | CA: 2 (0, 5.5) No CA: 0 (0, 1) | 1.29 (1.23-1.36) | <0.001 |
| Neurological PRISM | 0 (0, 0) | CA: 0 (0, 0) No CA: 0 (0, 0) | 1.10 (1.05-1.16) | <0.001 |
| Chemistry PRISM | 0 (0, 0) | CA: 2 (0, 3) No CA: 0 (0, 0) | 1.34 (1.24-1.45) | <0.001 |
| Hematological PRISM | 0 (0, 0) | CA: 0 (0, 2) No CA: 0 (0, 0) | 1.27 (1.19-1.36) | <0.001 |
| Median (IQR) Hospital Length of Stay (3) | 4.9 (2.5,11.0) | CA: 25.5 (8.2, 57.0) No CA: 4.8 (2.4, 10.8) | <0.001 | |
| Outcome at Hospital Discharge (3) | <0.001 | |||
| New Morbidity (4,5) | 463 (4.6%) | 16 (13.3%) | ||
| Survival with No New Morbidity | 9340 (92.7%) | 37 (30.8%) | ||
| Death | 271 (2.7%) | 67 (55.9%) | ||
Abbreviations: IQR = Interquartile Range; CA = Cardiac Arrest; FSS = Functional Status Scale; PRISM = Pediatric Risk of Mortality;
Sample includes those surviving > 4 hours in the PICU. Four patients died within the first 4 hours.
Wald Test
Outcome percentages are shown for each column.
Not used in the prediction model.
Morbidity is defined as a FSS increase of 3 or more
Physiological profiles were measured using the Pediatric Risk of Mortality (PRISM) score28 with a shortened time interval (2 hours prior to PICU admission to 4 hours after admission for laboratory data and the first 4 hours of PICU care for other physiological variables). For this analysis, we also partitioned the PRISM score into cardiovascular (heart rate, systolic blood pressure, temperature), neurological (pupillary reactivity, mental status), respiratory (arterial PO2, pH, PCO2, total bicarbonate), chemical (glucose, potassium, blood urea nitrogen, creatinine), and hematological (white blood cell count, platelet count, prothrombin and partial thromboplastin time) components. We computed the risk of mortality with a previously developed algorithm26 and assessed the association of mortality risk with the risk of cardiac arrest (see below) with the Pearson correlation coefficient.
The timing interval for assigning the admission time and assessing data was modified for cardiac patients under 91 days of age because some institutions admit infants to the PICU prior to a cardiac intervention to “optimize” the clinical status but not for intensive care; in these cases, the post-intervention period more accurately reflects intensive care. However, in other infants for whom the cardiac intervention is delayed after PICU admission or the intervention is a therapy required due to failed medical management of the acute condition, the first vital sign is the most appropriate initial time for admission and PRISM data collection time interval since it is the start of the ICU critical illness.22
Model Development
Our approach for modeling focused on the data available in the first 4 hours of care to estimate the subsequent risk of cardiac arrest. We did not attempt to predict cardiac arrests in the first 4 hours of PICU care because these events are generally associated with pre-ICU factors. We did include cardiac arrests occurring in the first 4 hours of care as a predictor variable for risk of subsequent arrests. The four hour modeling period was chosen because physiological profiles are important predictors of death, and this data set utilized physiological profiles measured with the PRISM score obtained from the first 4 hours of admission.24,26
Statistical analyses utilized SAS 9.4® (SAS Institute Inc. Cary, NC 27513-2414, USA) for descriptive statistics, model development, and fit assessment, and R 3.1.1 (R Foundation for Statistical Computing, https://www.r-project.org/) for evaluation of predictive ability. Patient characteristics were descriptively compared and evaluated across sites using the Kruskal-Wallis test for continuous variables, and the Pearson chi-squared test for categorical variables. The statistical analysis was under the direction of R.H.
Our primary outcome was the first cardiac arrest occurring after the first 4 hours of PICU care. Since this outcome was relatively rare, we did not split the sample into development and validation sets, in order to maximize the sample size available for model development. Univariate mortality odds ratios were computed and variables with a significance level <0.1 were considered candidate predictors for the final model. Variables were not included if there was a large percentage of unknowns/missing. A non-automated (examined by biostatistician and clinician at each step) backward stepwise selection approach was used to select factors. Multi-categorical factors (e.g., categorized age) had levels or factors combined when appropriate per statistical and clinical criteria. We also tested the total PRISM score and its partitioned scores, and the association of these partitioned scores with their “natural” diagnostic group (e.g. cardiac PRISM with patients with cardiac conditions versus non-cardiac conditions). Construction of a clinically relevant, sufficiently predictive model using information readily available to the clinician took precedence over inclusion based solely on statistical significance. Final candidate models were evaluated based on receiver operating characteristic (ROC) (discrimination), and the Hosmer-Lemeshow goodness of fit (calibration). To augment the potential utility of the ROC curve, we also assessed the relationship between the true positive rate (sensitivity, recall) and positive predictive value (precision, PPV) with the precision-recall curve and a plot illustrating the relationship between the number needed to evaluate (NNTE, 1/PPV) and the true positive rate.29,30 The NNTE is the number of patients classified as “high risk” using a particular cutpoint to identify one actual cardiac arrest.
Results
Overall, there were 10,078 patients in the TOPICC database. Four patients died in the first 4 hours after PICU admission, resulting in 10074 patients used for this analysis. A total of 132 patients sustained cardiac arrests after PICU admission. Nineteen arrests occurring during the first 4 hours were not included as outcomes. Of the 10074 patients in the analysis, 120 sustained a cardiac arrest after the first 4 hours of PICU care including seven patients who had also sustained a cardiac arrest during the first 4 hours of their PICU stay (Figure 1). Of these 120 patients, 67 (55.9%) died, 16 (13.3%) sustained a significant new morbidity, and 37 (30.8%) were discharged without a significant new morbidity (Figure 1).
Figure 1.
Cardiac Arrests Occurring in the PICU After the First Four Hours (n = 120). Only the first cardiac arrest occurring after the first 4 hours is included. Among the 10,074 children in the original sample surviving for four hours after admission, 19 had a cardiac arrest during the first four hours of admission. Of these 19, seven also had a cardiac arrest >=4 hours post-admission and are included in this figure, along with 113 whose first arrest occurred >=4 hours post admission. Of the 120 patients this figure, 67 (55.9%) died, 16 (13.3%) sustained a significant new morbidity, and 37 (30.8%) were discharged without a significant new morbidity.
Univariate comparisons of the descriptive data revealed differences between the total population and those with cardiac arrest after the first 4 hours of PICU care (Table 1). Overall, the population median age was 3.7 years and cardiac arrests were associated with younger ages (p<.001). Baseline functional status was similar. Cardiac arrest prior to admission or during the first 4 hours of care was strongly associated with a subsequent ICU cardiac arrest. Patients with cardiac arrest prior to admission were over 6 times more likely (p<.001) and those with cardiac arrests during the first 4 hours were over 50 times more likely (p<.001) to have another PICU cardiac arrest after 4 hours of PICU care. PICU cardiac arrest was also associated (p<.001) with receiving a cardiac surgical or catheterization intervention compared to those who did not, care in the cardiac PICU compared to the medical/surgical PICU, and admission from another hospital or the same hospital inpatient area compared to the post-anesthesia care unit.
Diagnostic and severity of illness data are displayed in Table 2. Compared to respiratory dysfunction, the primary system of dysfunction most associated with increased risk of cardiac arrest was cardiac while neurological dysfunction had a significantly lower risk (p<.001). Several specific diagnoses were assessed for their association with cardiac arrest. Cardiac diagnoses and septic shock were associated with cardiac arrest (p<.001) while cancer, trauma, and respiratory failure were not. The total admission PRISM score and the subcategories of cardiac PRISM, respiratory PRISM, neurological PRISM, chemistry PRISM, and hematological PRISM were associated with cardiac arrest (all p<.001). The cardiac PRISM had a significantly different association with cardiac arrest for those patients following a cardiac intervention than for non-cardiac patients while the other component PRISM scores did not have different associations with their related system dysfunctions and to non-related dysfunctions (p<.001).
The variables significant in multivariate logistic regression were age, PICU admission source, two diagnoses (cardiac disease and non-neurological trauma), the PRISM score (separated into cardiac and non-cardiac), cardiac interventions (surgery and interventional catheterization) and a good baseline functional status (Table 3). The most powerful categorical variables were referral from an inpatient unit, cardiac arrest in the first 4 hours of ICU care, non-neurological trauma and the presence of cardiac disease. While the non-cardiac PRISM score was included for all patients, modelling indicated substantially improved performance when the cardiac PRISM was assigned separate coefficients for patients admitted to the PICU following cardiac interventions.
Table 3.
Multivariate Model for Predicting ICU Cardiac Arrest
| Variable | Coefficient (Standard Error) | Odds Ratio (95% CI) |
|---|---|---|
| Intercept | -6.63 (0.35) | |
| Age at PICU Admission | ||
| <1 month | 0.67 (0.32) | 1.96 (1.04-3.69) |
| 1 month to 12 months | 0.91 (0.22) | 2.49 (1.61 - 3.84) |
| >12 months | reference | reference |
| ICU Admission Source | ||
| Post-procedure | Reference | Reference |
| Referral From Outside Hospital | 1.31 (0.32) | 3.70 (1.96 - 6.98) |
| Inpatient Unit (Same Hospital) | 2.11 (0.32) | 8.23 (4.35 - 15.54) |
| Emergency Department (Same Hospital) | 1.22 (0.34) | 3.40 (1.73 - 6.66) |
| Cardiac Disease Diagnosis (Congenital or Acquired) | 1.49 (0.25) | 4.44 (2.72 - 7.26) |
| Baseline FSS = Good (1) | -0.63 (0.21) | 0.53 (0.35 - 0.81) |
| Noncardiac PRISM (for all patients, per PRISM point) (2) | 0.094 (0.014) | 1.10 (1.07 - 1.13) |
| Cardiac PRISM (for patients with Cardiac Intervention) (per PRISM point) (3,4) | 0.357 (0.075) | 1.43 (1.23 - 1.66) |
| Cardiac PRISM (for patients without Cardiac Intervention) (per PRISM point) (3) | 0.142 (0.050) | 1.15 (1.05 - 1.27) |
| Non-Neurological Trauma (Primary Injury) | 1.61 (0.55) | 4.99 (1.71 - 14.58) |
| Cardiac Arrest First 4 Hours ICU (5) | 1.87 (0.58) | 6.48 (2.07 - 20.36) |
Abbreviations: CI = Confidence Interval; FSS = Functional Status Scale; PRISM = Pediatric Risk of Mortality
Good = FSS 6, 7
Non-cardiac PRISM = PRISM Score without heart rate, systolic blood pressure, temperature.
Cardiac interventions include surgery and interventional catheterization.
Cardiac PRISM = the following variables: heart rate, systolic blood pressure, temperature
Cardiac Arrest = chest compressions for at least 1 minute.
The predictive performance of the model was excellent with an area under the ROC curve of 0.85 (Figure 2A). The Hosmer-Lemeshow statistic (p=0.35, Supplemental Table 1) was consistent with adequate fit of the model to the outcome data. Figure 2A shows the ROC curve labelled with 5 selected “risk cutpoints” corresponding to true positive rates of 20%, 40%, 75%, 88% and 95% of the 120 patients with cardiac arrest after 4 hours in the PICU. The ROC curve indicates that these cutpoints correspond to false positive rates ranging from less than 5% to nearly 60% as the true positive rate increases. The chance that a patient classified as “high risk” based on these cutpoints truly has a cardiac arrest is the positive predictive value (PPV) or precision. The PPV is plotted versus the true positive rate in Figure 2B and the corresponding decision matrix is shown in Supplemental Table 2. The PPV ranges from over 20% with the lowest true positive rate to less than 5% with true positive rates of 75% or greater. An alternate representation of the relationship in Figure 2B is shown in Figure 2C, which displays the true positive rate versus the NNTE to identify one actual arrest. The NNTE increases from less than 10 to over 60 as criteria to designate a patient as “high risk” are made progressively less stringent, allowing the identification of more patients as the true positive rate increases.
Figure 2 A – C.
Ⓐ,Ⓑ,Ⓒ,Ⓓ and Ⓔ correspond to risk cutpoints yielding 20%, 40%, 75%, 88%, and 95% true positive rates, respectively.
Figure 2A. The Receiver Operating Characteristic (ROC) Curve. The area under the curves (AUC) is 0.85. The false positive rates corresponding to the risk cutpoints range from less than 10% to greater than 60%.
Figure 2B. The Precision-Recall (P-R) curve. The positive predictive value corresponding to the risk cutpoints range over 20% with the lowest true positive rate to less than 10% with the highest true positive rate.
Figure 2C. The Number Needed to Evaluate (NNTE). The NNTE increases from less than 10 to over 60 as the criteria designating patients as “high risk” (true positive rates) is made progressively more stringent.
Figure 3 illustrates the relationship between the timing of the first cardiac arrest after the first 4 hours to the predicted risk of cardiac arrest from Table 3. The average risk predicted by the model was highest (11.6%) among the 20 children who sustained an arrest between 4 hours and 12 hours after PICU admission (the first 8 hours following the risk prediction time period). Importantly, the risk does not substantially decrease over time. Predicted risk continues to be high among those having a cardiac arrest later in the PICU stay and the average predicted risk was 9.5% among the 39 children with arrests that occurred after 8 days in the PICU. The average risk for all patients not having a PICU cardiac arrest was 1.1%. Overall, the mortality risk for those patients sustaining a cardiac arrest after the first 4 hours was 13.6% while the mortality risk for those not sustaining a cardiac arrest was 2.6%. Mortality risk was highly correlated with cardiac arrest risk (r = 0.596, p<.001).
Figure 3.
Risk and Timing of the First ICU Cardiac Arrests After 4 Hours. The risks are indicated by columns and the line indicates the number of patients. The average risk for all patients not having a cardiac arrest was 1.1% (not shown).
Discussion
Cardiac arrests in hospitalized children predominantly occur in the PICU as the result of physiological decompensation in spite of critical care monitoring and therapies.2,6,7 Progressive shock, worsening respiratory failure and other pathophysiological processes may eventually progress to cardiac arrest. The proposed prediction model is based on readily available variables including physiological derangements measured by the PRISM score and categorical variables including diagnoses, referral source, age, functional status, and cardiac arrest in the first 4 hours. Our model was not only successful at identifying patients at high risk of cardiac arrest in early hours following data collection but also for many days after admission.
Since cardiac arrests usually occur as a consequence of physiological decompensation in spite of critical care therapies, it is likely that our prediction model also anticipates the development of severe or worsening critical illness. Recently, Niles et al demonstrated the effectiveness of an expert-derived checklist to anticipate PICU cardiac arrests and “code bell activations” within 24 hours. The checklist was primarily based on identifying patients receiving the “extremes” of therapeutic support and included some severe deviations in physiological variables.7 The excellent near-term performance of the checklist demonstrates that predicting risk for cardiac arrest is most frequently focused on identifying those children with severe critical illness and who “push the limits” of our ability to provide effective therapies.
The advent of point-of-care and just-in-time training for caregivers of patients at high risk of cardiac arrest is a needed intervention for patients identified as high risk. While cardiac arrest outcomes are improving, perhaps we could do better. When effective CPR is given, outcomes improve.31,32 For example, when rescuers achieve American Heart Association guideline recommendations for CPR, patients are almost twice as likely to have excellent blood pressure during CPR 11 and 10 times as likely to survive for at least 24 hours after the event.31 Early identification of patients at high risk for cardiac arrest could help assure that the cardiac arrest is prevented or treated with suitable expertise.33,34 The team can review physiological considerations and be prepared to use available monitoring techniques and be prepared to start CPR promptly, perhaps minimizing potentially life-threatening delays in initiating chest compressions. For patients arresting soon after admission, ongoing skill training will continue to be important.
The ROC and PPV curves demonstrate the amount of effort needed to identify and follow patients who are at significant risk for cardiac arrest. For example, in order to identify 95% of patients who would sustain a cardiac arrest, approximately 65 patients would require identification and follow-up for each identified arrest. In the clinical context, this would require “labelling” a patient at high risk and ensuring the staff is appropriately trained in CPR techniques. Identifying the “sickest” patients is already routine and insuring appropriate CPR skill is becoming more commonplace. It is also possible that identifying patients at risk could help prevent these events from occurring. The care of patients not “labelled” as high risk could re-assessed to reflect there low risk status.
A limitation of this analysis is that the number of patients in this cohort with a cardiac arrest was insufficient to both develop a robust model and validate it in a sufficient sample, even with a total sample size of over 10,000. Model precision is not as important if the goal is to identify all or most of the high risk patients. This type of model building and testing is suitable for routine data collection via the medical record. Unfortunately, it would not have been possible to conduct this analysis without a dedicated data collection effort because the outcome, cardiac arrest, does not have a standard definition that has been routinely incorporated into electronic health records.
Currently, predictive analytics such as our model can help us identify critically ill children at high and low risk for physiological deterioration needing cardiopulmonary resuscitation. Predictor variables used in this effort and other potential independent variables are readily recorded in real time in the PICU. We explored the practical impact of having such systems available in PICUs. Identifying “the sickest” patients and taking special precautions or making special efforts to insure optimal care is routine in PICUs. In this case, identifying these patients, insuring that the care team has appropriate CPR skills, and proactive planning could improve their outcomes if they do suffer a cardiac arrest. Perhaps more importantly, this predictive analytical approach coupled with appropriate interventions may be able to prevent cardiac arrest.
Conclusions
Using data from the first 4 hours of PICU stay, patients at high risk of cardiac arrest can be identified with routinely available data. The cardiac arrest may occur relatively close to the risk assessment period or days later. The identification of these high-risk patients may afford opportunities for the focused point-of-care, just-in-time bedside skill training that has recently been found to improve outcomes.
Supplementary Material
Acknowledgments
Funding Source: Supported, in part, by the following cooperative agreements from the Eunice Kennedy Shriver National Institute of Child Health and Human Development, National Institutes of Health, Department of Health and Human Services: U10HD050096, U10HD049981, U10HD049983, U10HD050012, U10HD063108, U10HD063106, U10HD063114 and U01HD049934. This content is solely the responsibility of the authors and does not necessarily represent the views of the National Institutes of Health.
Individuals Acknowledged and Roles
Teresa Liu, MPH, CCRP; University of Utah (project management, Data Coordinating Center)
Jean Reardon, MA, BSN, RN; Children’s National Medical Center (institutional project management, data collection)
Elyse Tomanio, BSN, RN; Children’s National Medical Center (institutional project management, data collection)
Morelia Menicucci, MD, CCRP; Children’s National Medical Center (data collection)
Fidel Ramos, BA; Children’s National Medical Center (institutional project management, data collection)
Aimee Labell, MS, RN; Phoenix Children’s Hospital (institutional project management, data collection)
Courtney Bliss, BS, DTR; Phoenix Children’s Hospital (data collection)
Jeffrey Terry, MBA; Children’s Hospital Los Angeles (data collection)
Margaret Villa, RN; Children’s Hospital Los Angeles and Mattel Children’s Hospital UCLA (institutional project management, data collection)
Jeni Kwok, JD; Children’s Hospital Los Angeles and Mattel Children’s Hospital (institutional project management, data collection)
Amy Yamakawa, BS; Children’s Hospital Los Angeles and Mattel Children’s Hospital UCLA (data collection)
Ann Pawluszka, BSN, RN; Children’s Hospital of Michigan (institutional project management)
Symone Coleman, BS, MPH; Children’s Hospital of Michigan (data collection)
Melanie Lulic, BS; Children’s Hospital of Michigan (data collection)
Mary Ann DiLiberto, BS, RN, CCRC; Children’s Hospital of Philadelphia (institutional project management, data collection)
Carolann Twelves, BSN, RN; Children’s Hospital of Philadelphia (data collection)
Monica S. Weber, RN, BSN, CCRP; University of Michigan (institutional project management, data collection)
Lauren Conlin, BSN, RN, CCRP; University of Michigan (data collection)
Alan C. Abraham, BA, CCRC; Children’s Hospital of Pittsburgh of University of Pittsburgh Medical Center (institutional project management, data collection)
Jennifer Jones, RN; Children’s Hospital of Pittsburgh of University of Pittsburgh Medical Center (data collection)
Jeri Burr, MS, RN-BC, CCRC; University of Utah (project management, Data Coordinating Center)
Nichol Nunn, BS, MBA; University of Utah (project management, Data Coordinating Center)
Alecia Peterson, BS, CMC; University of Utah (project management, Data Coordinating Center)
Carol Nicholson, MD (former Project Officer, Eunice Kennedy Shriver National Institute of Child Health and Human Development, National Institutes of Health, Department of Health and Human Services, for part of the study period)
Tammara L. Jenkins, MSN, RN; Pediatric Trauma and Critical Illness Branch, Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD), the National Institutes of Health (NIH), Bethesda, MD
Robert Tamburro, MD, MSc; Pediatric Trauma and Critical Illness Branch, Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD), the National Institutes of Health (NIH), Bethesda, MD
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Contributor Information
Murray M. Pollack, Department of Pediatrics, Children’s National Health System and the George Washington University School of Medicine and Health Sciences, Washington DC.
Richard Holubkov, Department of Pediatrics, University of Utah School of Medicine, Salt Lake City, UT.
Robert A. Berg, Department of Pediatrics, Children’s Hospital of Philadelphia, Philadelphia, PA.
Christopher J. L. Newth, Department of Anesthesiology and Critical Care Medicine, Children’s Hospital Los Angeles, University of Southern California Keck School of Medicine, Los Angeles, CA.
Kathleen L. Meert, Department of Pediatrics, Children’s Hospital of Michigan, Detroit, MI.
Rick E. Harrison, Department of Pediatrics, University of California at Los Angeles, Los Angeles, CA.
Joseph Carcillo, Department of Critical Care Medicine, Children’s Hospital of Pittsburgh, Pittsburgh, PA.
Heidi Dalton, Department of Child Health, Phoenix Children’s Hospital and University of Arizona College of Medicine-Phoenix, Phoenix, AZ (currently INOVA Fairfax Medical Center and George Washington University).
David L. Wessel, Department of Pediatrics, Children’s National Medical Center, Washington DC.
J. Michael Dean, Department of Pediatrics, University of Utah School of Medicine, Salt Lake City, UT.
References
- 1.Berg RA, Nadkarni VM, Clark AE, et al. Incidence and Outcomes of Cardiopulmonary Resuscitation in PICUs. Crit Care Med. April 2016;44(4):798–808. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Knudson JD, Neish SR, Cabrera AG, et al. Prevalence and outcomes of pediatric in-hospital cardiopulmonary resuscitation in the United States: an analysis of the Kids’ Inpatient Database*. Crit Care Med. November 2012;40(11):2940–2944. [DOI] [PubMed] [Google Scholar]
- 3.Girotra S, Nallamothu BK, Spertus JA, et al. Trends in survival after in-hospital cardiac arrest. N Engl J Med. November 15 2012;367(20):1912–1920. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Girotra S, Spertus JA, Li Y, et al. Survival trends in pediatric in-hospital cardiac arrests: an analysis from Get With the Guidelines-Resuscitation. Circ Cardiovasc Qual Outcomes. January 1 2013;6(1):42–49. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Topjian AA, Berg RA, Nadkarni VM. Pediatric cardiopulmonary resuscitation: advances in science, techniques, and outcomes. Pediatrics. November 2008;122(5):1086–1098. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Berg RA, Sutton RM, Holubkov R, et al. Ratio of PICU versus ward cardiopulmonary resuscitation events is increasing. Crit Care Med. October 2013;41(10):2292–2297. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Niles DE, Dewan M, Zebuhr C, et al. A pragmatic checklist to identify pediatric ICU patients at risk for cardiac arrest or code bell activation. Resuscitation. February 2016;99:33–37. [DOI] [PubMed] [Google Scholar]
- 8.Slonim AD, Patel KM, Ruttimann UE, Pollack MM. Cardiopulmonary resuscitation in pediatric intensive care units. Crit Care Med. December 1997;25(12):1951–1955. [DOI] [PubMed] [Google Scholar]
- 9.Alten JA, Klugman D, Raymond TT, et al. Epidemiology and Outcomes of Cardiac Arrest in Pediatric Cardiac ICUs. Pediatr Crit Care Med. October 2017;18(10):935–943. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Berg RA, Sutton RM, Reeder RW, et al. Association Between Diastolic Blood Pressure During Pediatric In-Hospital Cardiopulmonary Resuscitation and Survival. Circulation. December 26 2017. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Knight LJ, Gabhart JM, Earnest KS, Leong KM, Anglemyer A, Franzon D. Improving code team performance and survival outcomes: implementation of pediatric resuscitation team training. Crit Care Med. February 2014;42(2):243–251. [DOI] [PubMed] [Google Scholar]
- 12.Topjian AA, Berg RA, Nadkarni VM. Advances in recognition, resuscitation, and stabilization of the critically ill child. Pediatr Clin North Am. June 2013;60(3):605–620. [DOI] [PubMed] [Google Scholar]
- 13.Wolfe H, Zebuhr C, Topjian AA, et al. Interdisciplinary ICU cardiac arrest debriefing improves survival outcomes*. Crit Care Med. July 2014;42(7):1688–1695. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Cheng A, Overly F, Kessler D, et al. Perception of CPR quality: Influence of CPR feedback, Just-in-Time CPR training and provider role. Resuscitation. February 2015;87:44–50. [DOI] [PubMed] [Google Scholar]
- 15.Zebuhr C, Sutton RM, Morrison W, et al. Evaluation of quantitative debriefing after pediatric cardiac arrest. Resuscitation. September 2012;83(9):1124–1128. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Akre M, Finkelstein M, Erickson M, Liu M, Vanderbilt L, Billman G. Sensitivity of the pediatric early warning score to identify patient deterioration. Pediatrics. April 2010;125(4):e763–769. [DOI] [PubMed] [Google Scholar]
- 17.Panesar R, Polikoff LA, Harris D, Mills B, Messina C, Parker MM. Characteristics and outcomes of pediatric rapid response teams before and after mandatory triggering by an elevated Pediatric Early Warning System (PEWS) score. Hosp Pediatr. May 2014;4(3):135–140. [DOI] [PubMed] [Google Scholar]
- 18.Schein RM, Hazday N, Pena M, Ruben BH, Sprung CL. Clinical antecedents to in-hospital cardiopulmonary arrest. Chest. December 1990;98(6):1388–1392. [DOI] [PubMed] [Google Scholar]
- 19.Hodgetts TJ, Kenward G, Vlachonikolis IG, Payne S, Castle N. The identification of risk factors for cardiac arrest and formulation of activation criteria to alert a medical emergency team. Resuscitation. August 2002;54(2):125–131. [DOI] [PubMed] [Google Scholar]
- 20.Alvarez CA, Clark CA, Zhang S, et al. Predicting out of intensive care unit cardiopulmonary arrest or death using electronic medical record data. BMC medical informatics and decision making. 2013;13:28. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Ong ME, Lee Ng CH, Goh K, et al. Prediction of cardiac arrest in critically ill patients presenting to the emergency department using a machine learning score incorporating heart rate variability compared with the modified early warning score. Crit Care. 2012;16(3):R108. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Pollack MM, Holubkov R, Funai T, et al. Simultaneous Prediction of New Morbidity, Mortality, and Survival Without New Morbidity From Pediatric Intensive Care: A New Paradigm for Outcomes Assessment. Crit Care Med. August 2015;43(8):1699–1709. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Pollack MM HR, Funai T, Clark A, Berger JT, Meert K, Newth CJ, Shanley T, Moler F, Carcillo J, Berg RA, Dalton H, Wessel DL, Harrison RE, Doctor A, Dean JM, Jenkins TL; Eunice Kennedy Shriver National Institute of Child Health and Human Development Collaborative Pediatric Critical Care Research Network. Pediatric Intensive Care Outcomes: Development of New Morbidities During Pediatric Critical Care. Pediatr Crit Care Med. 2014;15(9):821–827. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Pollack MM, Dean JM, Butler J, et al. The ideal time interval for critical care severity-of-illness assessment. Pediatr Crit Care Med. June 2013;14(5):448–453. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Pollack MM, Holubkov R, Funai T, et al. Relationship between the functional status scale and the pediatric overall performance category and pediatric cerebral performance category scales. JAMA Pediatrics. July 2014;168(7):671–676. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Pollack MM, Holubkov R, Funai T, et al. The Pediatric Risk of Mortality Score: Update 2015. Pediatr Crit Care Med. January 2016;17(1):2–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Pollack MM, Holubkov R, Glass P, et al. Functional Status Scale: new pediatric outcome measure. Pediatrics. July 2009;124(1):e18–28. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Pollack MM, Patel KM, Ruttimann UE. PRISM III: an updated Pediatric Risk of Mortality score. Critical Care Medicine. 1996;24(5):743–752. [DOI] [PubMed] [Google Scholar]
- 29.Romero-Brufau S, Huddleston JM, Escobar GJ, Liebow M. Why the C-statistic is not informative to evaluate early warning scores and what metrics to use. Crit Care. August 13 2015;19:285. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Davis JGM. The Relationship Between Precision-Recall and ROC Curves. Proceedings of the 23rd International Conference on Machine Learning (ICML) 2006. [Google Scholar]
- 31.Sutton RM, French B, Niles DE, et al. 2010 American Heart Association recommended compression depths during pediatric in-hospital resuscitations are associated with survival. Resuscitation. September 2014;85(9):1179–1184. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Sutton RM, French B, Nishisaki A, et al. American Heart Association cardiopulmonary resuscitation quality targets are associated with improved arterial blood pressure during pediatric cardiac arrest. Resuscitation. February 2013;84(2):168–172. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Kleinman ME, de Caen AR, Chameides L, et al. Part 10: Pediatric basic and advanced life support: 2010 International Consensus on Cardiopulmonary Resuscitation and Emergency Cardiovascular Care Science With Treatment Recommendations. Circulation. October 19 2010;122(16 Suppl 2):S466–515. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.de Caen AR, Berg MD, Chameides L, et al. Part 12: Pediatric Advanced Life Support: 2015 American Heart Association Guidelines Update for Cardiopulmonary Resuscitation and Emergency Cardiovascular Care. Circulation. November 3 2015;132(18 Suppl 2):S526–542. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.