Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2021 Sep 1.
Published in final edited form as: Pediatr Crit Care Med. 2020 Sep;21(9):820–826. doi: 10.1097/PCC.0000000000002414

A Vital Sign-Based Model to Predict Clinical Deterioration in Hospitalized Children

Anoop Mayampurath 1,2, Priti Jani 1, Yangyang Dai 2, Robert Gibbons 3, Dana Edelson 3, Matthew M Churpek 3,†
PMCID: PMC7483876  NIHMSID: NIHMS1583697  PMID: 32511200

Abstract

Objective:

Clinical deterioration in hospitalized children is associated with increased risk of mortality and morbidity. A prediction model capable of accurate and early identification of pediatric patients at risk of deterioration can facilitate timely assessment and intervention, potentially improving survival and long-term outcomes. The objective of this study was to develop a model utilizing vital signs from electronic health record data for predicting clinical deterioration in pediatric ward patients.

Design:

Observational cohort study

Setting:

An urban, tertiary-care medical center.

Patients:

Patients less than 18 years admitted to the general ward during years 2009–2018.

Interventions:

None

Measurements and Main Results:

The primary outcome of clinical deterioration was defined as a direct ward to intensive care unit (ICU) transfer. A discrete-time logistic regression model utilizing six vital signs along with patient characteristics was developed to predict ICU transfers several hours in advance. Among 31,899 pediatric admissions, 1,375 (3.7%) experienced the outcome. Data were split into independent derivation (years 2009–2014) and prospective validation (years 2015–2018) cohorts. In the prospective validation cohort, the vital sign model significantly outperformed a modified version of the Bedside Pediatric Early Warning System (BedsidePEWS) score in predicting ICU transfers 12 hours prior to the event (c-statistic 0.78 vs. 0.72, P < 0.01).

Conclusions:

We developed a model utilizing six commonly used vital signs to predict risk of deterioration in hospitalized children. Our model demonstrated greater accuracy in predicting ICU transfers than the modified BedsidePEWS. Our model may promote opportunities for timelier intervention and risk mitigation, thereby decreasing preventable death and improving long-term health.

Keywords: pediatrics, clinical deterioration, risk assessment, electronic-health records, vital signs, decision support techniques

INTRODUCTION

Hospitalized children who experience clinical deterioration are at increased risk for mortality.1–3 Pediatric cardiac arrests are associated with poor long-term neurological and functional outcomes.4–7 To effectively prevent morbidity and mortality, care team personnel must identify deteriorating pediatric patients as early and accurately as possible.3,8,9

Several early warning scores have been developed to detect pediatric patients at risk for deterioration. Common examples include the Brighton Pediatric Early Warning Score (Brighton PEWS), the pediatric early warning system (PEWS) score, and the BedsidePEWS, which is a simpler version of the PEWS. These scoring systems have been validated in single and multicenter studies.7,10–13 However, their reliance on subjectively derived elements make them susceptible to different interpretations, increased burden of screening, and variable performance across centers.14,15,16Additionally, a recently concluded randomized controlled trial demonstrated that implementation of BedsidePEWS did not decrease in-hospital mortality, challenging its utility as a tool aimed at reduction of preventable death.17 Within adults, prediction models that utilize electronic health record (EHR) data have demonstrated superior performance in early detection of clinical deterioration compared to clinical standards.18 However, the applicability of this approach to predicting deterioration events in pediatric patients is unclear. In addition, because clinical deterioration is relatively rare in children in comparison to adults, detecting these events accurately while minimizing burden of screening is more challenging.

The aim of this study was to develop a model utilizing six objective and commonly used vital signs to predict deterioration in hospitalized children. We define pediatric clinical deterioration as a direct transfer from ward to the intensive care unit (ICU). To compare the performance of our model at various time points in advance of the event, we utilized a modified version of BedsidePEWS that includes physiological variables after removal of subjective scoring elements. We further investigated the accuracy of our model in detecting critically deteriorating patients in comparison to modified BedsidePEWS. We hypothesize that our statistically-derived model would detect pediatric clinical and critical deterioration earlier and more accurately than the modified BedsidePEWS. Our model could be used as the basis for an alert system for care personnel and rapid response units to intervene for a patient at risk of deterioration.

MATERIALS AND METHODS

Setting and Study Population

We conducted an observational cohort study of all consecutive pediatric (age < 18 years) ward admissions to the University of Chicago Medicine Comer Children’s Hospital from April 2009 to May 2018. Comer Children’s Hospital is a tertiary care center averaging approximately 5,000 admissions per year. All birth encounters for newborn patients were excluded from this population. Brighton PEWS, a scoring scheme that includes several subjective components and is designed to measure changes from baseline, was implemented in the hospital in 2013 and contributed to decision-making regarding transfer to the ICU. Rapid response teams were implemented in our hospital in 2008. The study was approved by the University of Chicago Institutional Review Board (IRB# 18–0645).

Data Sources

All clinical variables were collected from electronic health record (EHR; Epic, Verona, WI) data, whereas patient demographics and discharge disposition were collected from administrative data. Data elements were extracted from the Clinical Research Data Warehouse maintained by the Center for Research Informatics (CRI) at the University of Chicago.

Outcome and Predictor Variables

The primary outcome of pediatric clinical deterioration was defined by a direct ward-to-ICU transfer. The secondary outcome was a critical deterioration event, which is defined by Bonafide et al. as the need for mechanical ventilation, administration of vasoactive drugs, or mortality within 12 hours of transfer to the ICU.8 Transfer information from the hospital’s admission-discharge-transfer (ADT) database was used to determine the primary outcome. Patients who were transferred to a procedural area for care services or for surgery and then transferred to ICU were not considered to have the primary outcome of clinical deterioration. Predictor variables included four patient characteristics (age, sex, race, and ethnicity) and observations of six commonly-used vital signs (temperature, respiratory rate, heart rate, oxygen saturation, systolic blood pressure, and diastolic blood pressure) measured on the ward. Patient age was categorized into four age groups (infants [< 2 years], preschool [2–5 years], school-age [6–11 years], and adolescent [> 11 years]) based on prior work.19 Patient race was categorized into black, white, or other, and patient ethnicity was categorized into Hispanic and non-Hispanic groups. Patients who experienced clinical deterioration were compared to patients who remained on the ward through descriptive statistics (t-test for age, Wilcoxon rank sum test for hospital length of stay, and chi-squared tests for other categorical variables).

Prediction Model

Data were split based on admission year into derivation (years 2009–2014) and prospective validation (years 2015–2018) cohorts to build the prediction model. We utilized a discrete-time logistic regression framework to develop our model predicting pediatric ward to ICU transfer.20 Vital sign data in the derivation cohort were blocked into 12-hour intervals with the last recorded vital sign chosen as representative for that period. Missing values for vital signs were handled by carrying forward last-known observations. If no prior values were available, we imputed medians that were calculated based on all patients within the same ward. A logistic regression model was derived to predict ward-to-ICU transfer in the next interval. The model utilized a linear parameterization of time. Through five-fold cross-validation on the derivation cohort using area under the receiver operating characteristic curve (AUC) as the metric, a window of 12 hours was chosen as the optimal time interval over which to train the model. This duration of 12 hours has also been used previously to test PEWS accuracy in pediatric patients.7 Predicted probabilities returned by the model for the test dataset were used to calculate test AUC. These probabilities could be utilized in an alert system as they indicate risk of deterioration for each patient. The modified BedsidePEWS scoring system was back-calculated for all patients in the cohort as follows. First, we extracted temperature, heart rate, systolic blood pressure, respiratory rate, and oxygen saturation from the EHR. We then assigned BedsidePEWS risk scores based on published age-based cutoffs for each variable, and summed the risk scores into a cumulative modified BedsidePEWS score indicating risk of deterioration.13

Our model performance was compared to modified BedsidePEWS in the validation cohort using AUC measures to predict ICU transfers in 6 hour increments up to 36 hours in advance of the event. Additionally, a secondary analysis was also performed by comparing the performance of our model with modified BedsidePEWS in predicting critical deterioration events. We did not compare our model performance with Brighton PEWS, which was implemented in our hospital in 2013 and used in decision-making regarding transfer to the ICU, since this would lead to confounding during model validation. However, to test the possibility of care policy changes impacting validation accuracy, we compared our modified BedsidePEWS accuracy for predicting ICU transfers before and after Brighton PEWS implementation. Analyses were performed using R version 3.3 (R Project for Statistical Computing), with two-sided P < .05 denoting statistical significance. AUC differences between models were compared for using the non-parametric approach illustrated in DeLong et al.21

RESULTS

Study Population

A total of 1,375 (3.6%) patients experienced a ward-to-ICU transfer in the cohort of 38,199 patients over the nine-year period from 2009–2018. Patients transferred to the ICU were similar to patients who stayed on the wards in terms of age, sex, race, ethnicity, and admission location (see Table 1). As expected, ICU transfer patients were more likely to die than patients who remained on the wards throughout their hospitalization (3% vs 0.07%, P < 0.001), and had longer hospital lengths of stay (median 9 days vs. 2 days, P < 0.001).

Table 1:

Comparison of characteristics between pediatric patients who experienced an ICU transfer event and those who remained on the ward throughout hospitalization.

Characteristic Patient admissions with ICU transfer (n=1,375) Patients admissions without ICU transfer (n=36,824)
Age yrs, mean (sd) 6 (6)* 7(6)
Female, n (%) 632 (46) 16,699 (45)
Race, n (%)
 Black 821 (60) 21,756 (59)
 White 387 (28) 10,623 (29)
 Other 167 (12) 4, 445 (12)
Hispanic, n (%) 183 (13) 4,308 (12)
Mortality, n (%) 44 (3)* 24 (0.07)
Hospital length of stay days, median (IQR) 9 (5, 18)* 2 (1, 4)
Initial Hospital Location, n (%)
 Ward 244 (18)* 4,893 (13)
 ED 664 (48) 18,916 (51)
 ICU 61 (4) 1326 (4)
 OR 49 (4) 2782 (8)
 Other 357 (26) 8907 (24)
Open in a new tab
*

P < .001 compared to patients who remained on the ward.

IQR: Inter-quantile range

ED: Emergency Department

ICU: Intensive Care Unit

OR: Operating Room

Model performance

In the prospective validation cohort of 9,822 patients, our vital sign model significantly outperformed the modified BedsidePEWS in predicting ICU transfers at 12 hours (AUC 0.78 vs 0.72, P < 0.001) and 24 hours (AUC 0.76 vs. 0.69, P < 0.001). Figure 1 compares the AUCs (with 95% confidence intervals) of the vital sign-based model versus modified BedsidePEWS in predicting ICU transfers across several time intervals prior to the event. As seen, the vital sign-based model demonstrates improved prediction performance over the modified BedsidePEWS at all time points. Model performance was equivalent when tested on neonates (336 patient admissions, AUC 0.78 [95%CI: 0.74–0.83]) and non-neonates (9,486 patient admissions AUC 0.78 [95%CI: 0.77–0.79]).

Figure 1: Model discrimination for predicting ICU transfers.

Figure 1:

Open in a new tab

AUC comparisons between our derived vital sign-based model and the modified BedsidePEWS for predicting ICU transfers at 6–36 hours prior to the transfer. Bars indicate AUC 95% confidence intervals at each time point.

Figure 2 depicts the performance of the vital sign-based model in predicting critical deterioration events (i.e., ICU transfers followed by mechanical ventilation, vasopressors, or mortality within 12 hours) in the prospective validation cohort as compared to modified BedsidePEWS. Our model shows similar improvements in performance over modified BedsidePEWS at 6 hours (AUC 0.77 vs. 0.73, P=0.003), 12 hours (AUC 0.76 vs. 0.71, P<0.001), 18 hours (AUC 0.75 vs. 0.70, P<0.001), and 24 hours (AUC 0.72 vs. 0.67, P < 0.001) before the event.

Figure 2: Model discrimination for predicting critical deterioration events.

Figure 2:

Open in a new tab

AUC comparisons between our derived of vital sign-based model and the modified BedsidePEWS for predicting critical deterioration events (ICU transfer followed by mechanical ventilation, vasopressor administration, or mortality within 12 hours) at 6–36 prior to event. Bars indicate AUC 95% confidence intervals at each time point

We further compared the performance of the vital sign model with modified BedsidePEWS at various scoring thresholds (Table 2).22 At a specificity of 90%, the vital sign-based model had a sensitivity of 53%, compared with 36% for the modified BedsidePEWS model. The number of patients needed to evaluate at these thresholds to detect one event for the vital-sign model and the modified BedsidePEWS were 13 and 15 respectively. At the above-mentioned specificity, the median time from the first alert from our model to ICU transfer in patients experiencing clinical deterioration was 11 hours (IQR: 5–34 hours). The median length of total hospital stay for these patients was 7 days (IQR: 4–15 days) and the mortality rate was 2%.

Table 2:

Sensitivity and specificity of different cut-offs for our vital sign model and modified BedsidePEWS for patients suffering a clinical deterioration event compared with those not experiencing any event in the test dataset. Vital sign model cut-offs are represented as predicted probabilities x 1000.

Model Cutoff Sensitivity (%, 95%CI) Specificity (%, 95% CI)
Vital Sign Model (Predicted Probability x1000)
≥ 6 82 (80–83) 54 (53–54)
≥ 9 73 (72–75) 70 (70–70)
≥ 13 64 (63–66) 82 (81–82)
≥ 17 57 (55–59) 88 (87–88)
≥ 20 53 (51–55) 90 (90–91)
≥ 23 49 (47–51) 93 (93–93)
≥ 27 43 (42–45) 94 (94–94)
≥ 39 33 (31–34) 97 (97–97)
≥ 48 27 (26–29) 98 (98–98)
≥ 67 19 (18–21) 99 (99–99)
Modified BedsidePEWS
≥ 2 80 (78–80) 48 (48–49)
≥ 3 56 (55–58) 77 (77–77)
≥ 4 36 (35–38) 90 (90–91)
≥ 6 14 (13–15) 98 (98–98)
Open in a new tab

At a similar sensitivity (57% for the vital sign-based model and 56% for Modified BedsidePEWS), the vital-sign model had a specificity of 88% and a number needed to evaluate of 15 patients to detect one event whereas the modified BedsidePEWS had a specificity of 77% and a number needed to evaluate of 20 patients to detect one event. The median time from first alert from the vital-sign model to time of ICU transfer was 11 hours (IQR: 6–42 hours). Patients who experienced clinical deterioration and had an alert based on this threshold had a median length of hospital stay of 8 days (IQR: 4–15 days) and a mortality rate of 2%.

Figure 3 illustrates the discrimination of each individual variable for predicting ICU transfer within 12 hours. Respiratory rate (AUC 0.74 [95%CI: 0.73–0.75]) and heart rate (AUC 0.73 [95%CI: 0.73–0.74]) were the most accurate individual variables. Temperature was the next most accurate vital sign variable, while systolic blood pressure, diastolic blood pressure and oxygen saturation were moderately predictive of clinical deterioration.

Figure 3: Predictive performance of each variable in the vital sign-based model.

Figure 3:

Open in a new tab

AUC (with 95% confidence interval) comparison of the overall vital sign-based model and individual features for predicting ICU transfers. Respiratory rate and heart rate were highly predictive of deterioration, as compared to other vital signs (temperature, systolic and diastolic blood pressure, and oxygen saturation). Time in the hospital was also predictive of deterioration. Apart from patient age (continuous variable) other patient characteristics such as sex, race (black/white/other), and ethnicity (Hispanic/Not-Hispanic) showed poor predictive accuracy.

Supplementary Table 1 depicts the odds-ratios, along with 95% confidence intervals, for the variables in the vital sign-based model for predicting ICU transfer within 12 hours. Replacing categorical groupings of age with a continuous variable did not impact overall model performance (AUC 0.78 vs. 0.78 at 12 hours in advance of the event). Inclusion of interaction terms between age and heart rate and age and respiratory rate did not impact performance (AUC without interaction terms: 0.78 [95% CI: 0.77–0.79], AUC with interaction terms: 0.79 [95% CI: 0.78–0.80]). Sensitivity analyses using time by vital sign interactions and also by testing a quadratic parameterization of time yielded similar accuracies to the main model. Supplementary Figure 1 depicts the predictive performance of the modified BedsidePEWS scoring model in our dataset. If the implementation of a pediatric early warning system in 2013 impacted the results of the vital sign-based model, the performance of the modified Bedside PEWS would likely have changed significantly over time. As shown, the AUC post-2013 only increased slightly from AUC pre-2013. This implies that there would likely be no meaningful change to the studied scores related to the implementation of the early warning system.

DISCUSSION

We developed a model that utilizes six commonly collected vital signs to predict ward to ICU transfer events in hospitalized children. Our model outperformed a modified version of the BedsidePEWS at multiple time points in advance of the event. In addition, our model provided more accurate and earlier prediction of critical deterioration events, defined as ICU transfers followed by mechanical ventilation, vasopressor administration, or mortality, than the modified BedsidePEWS. These data reveal the potential for our model to facilitate early identification and prevention of clinical and critical deterioration in pediatric patients.

Current pediatric early warning scores have several limitations including poor recording of all scoring variables, lack of broad validation, and methodological issues.16 In addition, risk-stratification algorithms that are prone to subjective interpretation could lead to poor sensitivity or reduced specificity, thereby increasing the burden to screen.16 Furthermore, for true positives, it has been observed that clinician identification of deterioration occurred before or simultaneous to alerts from current early warning scores, thereby rendering these scores potentially ineffective at early recognition.23 These system vulnerabilities highlight the potential advantage of using an objective, statistically-derived model to identify pediatric deterioration more accurately.

Recently, a cluster-randomized controlled trial (the EPOCH trial) sought to determine if the use of BedsidePEWS would decrease mortality or significant clinical deterioration.17 The trial was conducted across 7 countries and compared the outcomes of 10 hospitals without BedsidePEWS to 11 hospitals with BedsidePEWS and involved extensive training and emphasis on situational awareness. Results from this study concluded that BedsidePEWS did not decrease all-cause mortality, thereby not supporting the use of this interventional system in reducing pediatric patient mortality. The inability of BedsidePEWS systems to reliably prevent mortality is likely due to multiple factors. A recent study demonstrated that 54% of heart rate and respiratory rate readings in pediatric patients are classified as abnormal by textbook reference ranges, while up to 38% would have resulted in increase in pediatric early warning scores.24 This illustrates the need for further optimization of current early warning systems for children. Additionally, in comparison to adults, the natural limitations to subjective assessment are compounded by variation in age-related thresholds for normal versus abnormal vital signs and physiology. Finally, pediatric patients are often unable to contribute to the subjective assessment due to their age and development-related ability to communicate clinical signs such as dyspnea or location of pain.3 Overall, current pediatric early warning scores have a high reliance on the provider’s clinical acumen and knowledge base, which could vary across hospital settings. Although pediatric mortality is a rare event in comparison to adult mortality, predictive models that identify deterioration earlier have the potential to vastly improve care by preventing morbidity as well as decreasing the number of patients needed to evaluate.

Studies aimed at developing EHR-derived prediction models have shown promise but are limited in their clinical application. For example, Zhai et al. developed an EHR-based model for predicting ICU transfer within the first 24 hours of admission and showed improved performance over a PEWS system. However, this model was derived using data from the first day of hospitalization and therefore is not applicable for the duration of a patient’s hospital stay.25 Wellner et al. recently demonstrated success in predicting unplanned ICU transfers by using a matched case-control cohort study design and assigning control patients that are similar in age and discharge diagnoses to the case patients.26 In an another study, Da Silva et al. also used age and sex matched case-control study design to demonstrate how abnormal vital signs in combination with pediatric early warning scores can predict clinical deterioration within 24 hours.27 However, these models are restrictive in that the matching done during derivation may result in a model not applicable to a general pediatric population. In addition, some features such as discharge diagnoses are not available throughout hospitalization. Furthermore, both models incorporated pediatric early warning scores (PEWS and pediatric Rothman Index respectively) as one of the predictor variables making them vulnerable to issues regarding subjective assessments. In a recent study, Rubin et al. developed an ensemble boosting model predicting ICU transfer in pediatric patients utilizing vital signs recorded 2 hours to 8 hours prior to the event.28 However, this model is challenging to interpret and automate. Additionally, it does not consider the entire hospital stay for patients who remained on the ward, thereby failing to adjust for a patient’s entire length of care.

Our model distinguishes itself from this prior work by utilizing six objective and commonly used vital sign readings along with patient characteristics to predict clinical deterioration throughout a patient’s hospitalization. Our model is interpretable, simple to automate, and can be used to continuously score patients as it is independent of subject input. The scores returned by the model indicate the likelihood of patient having clinical deterioration in the next 12 hours, which can be used to alert care personnel and rapid response teams. Thus, the model can be used to reduce the burden of patient screening as well as reduce unnecessary ICU transfers. Further, our model could facilitate timely intervention and clinical assessments for deteriorating patients.

Our work also highlights the critical importance of using respiratory rate and heart rate to track clinical deterioration, as emphasized by previous studies.24,29 In addition, our work also indicates the importance of temperature in predicting deterioration as it may indicate infection, immune, or inflammatory conditions that could lead to critical illness.30,31

There are several limitations to our study. First, this study was performed at a single academic medical center. Therefore, generalizability to other centers may be limited. In particular, the criteria for transferring to the ICU may vary across hospital settings. However, in our secondary analysis, we demonstrated improved performance in predicting critical deterioration events, which is a more generalizable definition. The accuracy of our model in other hospitals will be further investigated in future multicenter studies. Second, because we could not compare our model against the full version of BedsidePEWS, we cannot reliably conclude that our objective model improves performance over the complete expert-derived scoring system. However, our modified BedsidePEWS score removed the subjective scoring components that are susceptible to misinterpretations and retained risk scores pertaining to physiological variables, thereby potentially leading to more consistent scoring. Third, the implementation of Brighton PEWS in 2013 could influence our prediction accuracy. However, our study of a modified BedsidePEWS scoring system showed that the accuracy largely remained the same from 2009 to 2013 and is marginally higher in the latter years post implementation of BrightonPEWS at our hospital, thereby validating the improved performance of our vital sign-based model. Further, our cohort includes neonates (patient admissions within 28 days of life) among the 0–2 years group. Abnormal vital signs are different between neonates and 1- or 2-year old children. However, our model performance in predicting clinical deterioration in advance did not change when tested on neonates or non-neonates. Our model is dependent on monitoring and documentation of vital signs, which are recorded less frequently in non-ICU patients, for timely recognition of deterioration. We have not explored other potential clinical indicators of deterioration, such as laboratory results and use of emergency medications. Incorporating these additional variables as well as adjusting for complex interactions through advanced machine-learning methods may increase accuracy.32 Finally, while our model can identify and flag patients at risk for clinical deterioration in advance, the clinical implications around these alerts deserve future study.23,33 Because rapid response team data were unavailable, we were unable to determine which patients were assessed by these teams in our dataset. We further could not account for patients assessed by the rapid response team but remained on the ward. It is important to note that clinical expertise and judgement are needed to recognize the cause of deterioration and to decide among courses of action such as increased monitoring, transfer to the ICU, or other specific interventions. The importance of bedside evaluation and clinical judgment limit the automation of care after the initial identification. Further work is needed to investigate these avenues and to determine the impact of our model on outcomes.

CONCLUSION

In conclusion, we developed and validated an EHR-derived, automated, vital sign-based statistical model that demonstrated improved accuracy over current risk scores in predicting clinical deterioration in pediatric patients hours in advance of the event. Once implemented into clinical practice, utilization of our model for early recognition of deterioration could provide an opportunity for risk mitigation and improved outcomes in hospitalized children.

Supplementary Material

Supplemental Data File (.doc, .tif, pdf, etc.)

ACKNOWLEDGMENTS

We would like to thank Timothy Holper MS, MA, Julie Johnson PhD, and Stacie Landron MS, RN from the Center for Research Informatics for assistance with data extraction and Mary Akel MPH for administrative support.

Copyright form disclosure:

Dr. Mayampurath’s institution received funding from National Institutes of Health (NIH) National Heart, Lung, and Blood Institute (K01HL148390), and he received funding from Litmus Health. Drs. Mayampurath and Churpek received support for article research from the NIH. Dr. Gibbons received funding from Glaxo Smith Kline, and he disclosed that he is the founder of Adaptive Testing Technologies. Dr. Edelson received funding from Quant HC (ownership interest), EarlySense Inc. (research support), Philips Healthcare (research support and honoraria), the American Heart Association (research support), and Laerdal Medical (research support). Drs. Edelson and Churpek disclosed that they have a patent pending (ARCD. P0535US.P2) for risk stratification algorithms for hospitalized patients. Dr. Churpek received funding from R01 from the National Institute of General Medical Sciences (NIGMS) (NIGMS R01 GM123193) and EarlySense (research support). The remaining authors have disclosed that they do not have any potential conflicts of interest.

Disclosures and Conflicts of Interest:

Dr. Mayampurath is supported by a career development award from the National Heart, Lung, and Blood Institute (K01HL148390). Dr. Churpek was supported by an R01 from NIGMS (R01 GM123193). Dr. Mayampurath has performed consulting services for Litmus Health (Austin, TX). Drs. Churpek and Edelson have a patent pending (ARCD. P0535US.P2) for risk stratification algorithms for hospitalized patients and have received research support from EarlySense (Tel Aviv, Israel). Dr. Edelson has received research support and honoraria from Philips Healthcare (Andover, MA). Dr. Edelson has ownership interest in Quant HC (Chicago, IL), which licenses eCART, a patient risk analytic, to hospitals through Philips Healthcare, EarlySense and AgileMD.

REFERENCES

  • 1.Nadkarni VM, Larkin G, Peberdy M, et al. First documented rhythm and clinical outcome from in-hospital cardiac arrest among children and adults. JAMA. 2006;295(1):50–57. doi: 10.1001/jama.295.1.50 [DOI] [PubMed] [Google Scholar]
  • 2.Tress EE, Kochanek PM, Saladino RA, Manole MD. Cardiac arrest in children. Journal of Emergencies, Trauma and Shock. 2010;3(3):267–272. doi: 10.4103/0974-2700.66528 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Lambert V, Matthews A, MacDonell R, Fitzsimons J. Paediatric early warning systems for detecting and responding to clinical deterioration in children: a systematic review. BMJ Open. 2017;7(3). doi: 10.1136/bmjopen-2016-014497 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Stremler R, Haddad S, Pullenayegum E, Parshuram C. Psychological Outcomes in Parents of Critically Ill Hospitalized Children. Journal of Pediatric Nursing: Nursing Care of Children and Families. 2017;34:36–43. doi: 10.1016/j.pedn.2017.01.012 [DOI] [PubMed] [Google Scholar]
  • 5.Berg RA, Nadkarni VM, Clark AE, et al. Incidence and Outcomes of Cardiopulmonary Resuscitation in Pediatric Intensive Care Units. Critical care medicine. 2016;44(4):798–808. doi: 10.1097/CCM.0000000000001484 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Suominen P, Olkkola KT, Voipio V, Korpela R, Palo R, Räsänen J. Utstein style reporting of in-hospital paediatric cardiopulmonary resuscitation. Resuscitation. 2000;45(1):17–25. doi: 10.1016/S0300-9572(00)00167-2 [DOI] [PubMed] [Google Scholar]
  • 7.Parshuram CS, Duncan HP, Joffe AR, et al. Multicentre validation of the bedside paediatric early warning system score: a severity of illness score to detect evolving critical illness in hospitalised children. Critical Care. 2011;15(4):R184. doi: 10.1186/cc10337 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Bonafide CP, Localio A, Roberts KE, Nadkarni VM, Weirich CM, Keren R. Impact of rapid response system implementation on critical deterioration events in children. JAMA Pediatrics. 2014;168(1):25–33. doi: 10.1001/jamapediatrics.2013.3266 [DOI] [PubMed] [Google Scholar]
  • 9.Sharek PJ, Parast LM, Leong K, et al. Effect of a rapid response team on hospital-wide mortality and code rates outside the icu in a children’s hospital. JAMA. 2007;298(19):2267–2274. doi: 10.1001/jama.298.19.2267 [DOI] [PubMed] [Google Scholar]
  • 10.Murray JS, Williams LA, Pignataro S, Volpe D. An Integrative Review of Pediatric Early Warning System Scores. Pediatr Nurs. 2015;41(4):165–174. [PubMed] [Google Scholar]
  • 11.Monaghan A Detecting and managing deterioration in children. Paediatr Nurs. 2005;17(1):32–35. doi: 10.7748/paed2005.02.17.1.32.c964 [DOI] [PubMed] [Google Scholar]
  • 12.Duncan H, Hutchison J, Parshuram CS. The pediatric early warning system score: A severity of illness score to predict urgent medical need in hospitalized children. Journal of Critical Care. 2006;21(3):271–278. doi: 10.1016/j.jcrc.2006.06.007 [DOI] [PubMed] [Google Scholar]
  • 13.Parshuram CS, Hutchison J, Middaugh K. Development and initial validation of the Bedside Paediatric Early Warning System score. Critical Care. 2009;13(4):R135. doi: 10.1186/cc7998 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Chapman SM, Maconochie IK. Early warning scores in paediatrics: an overview. Archives of Disease in Childhood. 2019;104(4):395–399. doi: 10.1136/archdischild-2018-314807 [DOI] [PubMed] [Google Scholar]
  • 15.Chapman SM, Wray J, Oulton K, Pagel C, Ray S, Peters MJ. ‘The Score Matters’: wide variations in predictive performance of 18 paediatric track and trigger systems. Archives of Disease in Childhood. 2017;102(6):487–495. doi: 10.1136/archdischild-2016-311088 [DOI] [PubMed] [Google Scholar]
  • 16.Trubey R, Huang C, Lugg-Widger FV, et al. Validity and effectiveness of paediatric early warning systems and track and trigger tools for identifying and reducing clinical deterioration in hospitalised children: a systematic review. BMJ Open. 2019;9(5):e022105. doi: 10.1136/bmjopen-2018-022105 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Parshuram CS, Dryden-Palmer K, Farrell C, et al. Effect of a pediatric early warning system on all-cause mortality in hospitalized pediatric patients: The epoch randomized clinical trial. JAMA. 2018;319(10):1002–1012. doi: 10.1001/jama.2018.0948 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Churpek MM, Yuen TC, Winslow C, et al. Multicenter Development and Validation of a Risk Stratification Tool for Ward Patients. American Journal of Respiratory and Critical Care Medicine. 2014;190(6):649–655. doi: 10.1164/rccm.201406-1022OC [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Mayampurath A, Volchenboum SL, Sanchez-Pinto LN. Using photoplethysmography data to estimate heart rate variability and its association with organ dysfunction in pediatric oncology patients. npj Digital Medicine. 2018;1(1):29. doi: 10.1038/s41746-018-0038-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Singer JD, Willett JB. It’s About Time: Using Discrete-Time Survival Analysis to Study Duration and the Timing of Events. Journal of Educational Statistics. 1993;18(2):155–195. doi: 10.3102/10769986018002155 [DOI] [Google Scholar]
  • 21.DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44(3):837–845. [PubMed] [Google Scholar]
  • 22.Romero-Brufau S, Huddleston JM, Escobar GJ, Liebow M. Why the C-statistic is not informative to evaluate early warning scores and what metrics to use. Crit Care. 2015;19:285. doi: 10.1186/s13054-015-0999-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Winter MC, Kubis S, Bonafide CP. Beyond Reporting Early Warning Score Sensitivity: The Temporal Relationship and Clinical Relevance of “True Positive” Alerts that Precede Critical Deterioration. Journal of Hospital Medicine. 2019;14(3). doi: 10.12788/jhm.3066 [DOI] [PubMed] [Google Scholar]
  • 24.Bonafide CP, Brady PW, Keren R, Conway PH, Marsolo K, Daymont C. Development of Heart and Respiratory Rate Percentile Curves for Hospitalized Children. Pediatrics. 2013;131(4):e1150–e1157. doi: 10.1542/peds.2012-2443 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Zhai H, Brady P, Li Q, et al. Developing and Evaluating a Machine Learning Based Algorithm to Predict the Need of Pediatric Intensive Care Unit Transfer for Newly Hospitalized Children. Resuscitation. 2014;85(8):1065–1071. doi: 10.1016/j.resuscitation.2014.04.009 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Wellner B, Grand J, Canzone E, et al. Predicting Unplanned Transfers to the Intensive Care Unit: A Machine Learning Approach Leveraging Diverse Clinical Elements. JMIR Med Inform. 2017;5(4):e45. doi: 10.2196/medinform.8680 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.da Silva YS, Fiedor Hamilton M, Horvat C, et al. Evaluation of Electronic Medical Record Vital Sign Data Versus a Commercially Available Acuity Score in Predicting Need for Critical Intervention at a Tertiary Children’s Hospital. Pediatric Critical Care Medicine. 2015;16(7):644. doi: 10.1097/PCC.0000000000000444 [DOI] [PubMed] [Google Scholar]
  • 28.Rubin J, Potes C, Xu-Wilson M, et al. An ensemble boosting model for predicting transfer to the pediatric intensive care unit. International Journal of Medical Informatics. 2018;112:15–20. doi: 10.1016/j.ijmedinf.2018.01.001 [DOI] [PubMed] [Google Scholar]
  • 29.Fleming S, Thompson M, Stevens R, et al. Normal ranges of heart rate and respiratory rate in children from birth to 18 years: a systematic review of observational studies. Lancet. 2011;377(9770):1011–1018. doi: 10.1016/S0140-6736(10)62226-X [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Barbi E, Marzuillo P, Neri E, Naviglio S, Krauss BS. Fever in Children: Pearls and Pitfalls. Children (Basel). 2017;4(9). doi: 10.3390/children4090081 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Lockwood J, Reese J, Wathen B, et al. The Association Between Fever and Subsequent Deterioration Among Hospitalized Children With Elevated PEWS. Hospital Pediatrics. February 2019:hpeds.2018–0187. doi: 10.1542/hpeds.2018-0187 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Churpek MM, Yuen TC, Winslow C, Meltzer DO, Kattan MW, Edelson DP. Multicenter Comparison of Machine Learning Methods and Conventional Regression for Predicting Clinical Deterioration on the Wards. Critical care medicine. 2016;44(2):368–374. doi: 10.1097/CCM.0000000000001571 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33.Ginestra JC, Giannini HM, Schweickert WD, et al. Clinician Perception of a Machine Learning–Based Early Warning System Designed to Predict Severe Sepsis and Septic Shock. Critical Care Medicine. 2019;Online First. doi: 10.1097/CCM.0000000000003803 [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplemental Data File (.doc, .tif, pdf, etc.)
NIHMS1583697-supplement-Supplemental_Data_File___doc___tif__pdf__etc__.docx (2.4MB, docx)

RESOURCES