Dynamic Mortality Risk Predictions for Children in Intensive Care Units: Development and Validation of Machine Learning Models

Abstract

Objective:

Assess a machine learning method of serially updated mortality risk.

Design:

Retrospective analysis of a national database (Health Facts®, Cerner Corporation, Kansas City, MO).

Setting:

Hospitals caring for children in intensive care units (ICUs).

Patients:

A total of 27,354 admissions cared for in ICUs from 2009 to 2018.

Interventions:

None

Main Outcome:

Hospital mortality risk estimates determined at 6-hour time periods during care in the ICU. Models were truncated at hour 180 due to decreased sample size secondary to discharges and deaths.

Measurements and Main Test Results.

The Criticality Index, based on physiology, therapy, and care intensity, was computed for each admission for each time period and calibrated to hospital mortality risk (Criticality Index - Mortality (CI-M)) at each of 29 time periods (initial assessment: 6 hours; last assessment: 180 hours). Performance metrics and clinical validity were determined from the held-out test sample (n = 3,453, 13%). Discrimination assessed with the area under the receiver operating characteristic curve was 0.852 (95% CI: 0.843, 0.861) overall and ≥ 0.80 for all individual time periods. Calibration assessed by the Hosmer-Lemeshow goodness-of-fit test showed good fit overall (p = 0.196) and was statistically not significant for 28 of the 29 time periods. Calibration plots for all models revealed the intercept ranged from −0.002 to 0.009, the slope ranged from 0.867 to 1.415, and the R² ranged from 0.862 to 0.989. Clinical validity assessed using population trajectories and changes in the risk status of admissions (clinical volatility) revealed clinical trajectories consistent with clinical expectations and greater clinical volatility in deaths than survivors (p < 0.001).

Conclusion and Relevance:

Machine learning models incorporating physiology, therapy, and care intensity can track changes in hospital mortality risk during intensive care. The CI-M’s framework and modeling method are potentially applicable to monitoring clinical improvement and deterioration in real-time.

Introduction

Mortality risk is the traditional measure of severity of illness for children in intensive care units (ICUs).. Static mortality risk measures, including the Pediatric Risk of Mortality (PRISM) score and the Pediatric Index of Mortality (PIM), were developed with a focus on quality assessment and, therefore, focus on the early portion of an ICU stay (1, 2). These were not designed for nor widely adopted for use in individual patients, at least in part because rapid changes in severity and therapies and the frequent acquisition of data made risk assessments obsolete soon after they were obtained. The Pediatric Logistic Organ Failure (PELOD) score can be collected serially, but PELOD is updated only daily and the relatively simplistic framework limits the potential to enhance clinician assessment (3).

Early identification of ICU patients at increasing or high risk of in-hospital mortality might improve clinical and operational decision-making and potentially improve outcomes. Dynamic mortality risk assessment, objectively tracking changes in mortality risk, could support the clinical decision making of health care providers, especially those with less experience or expertise. Machine learning methods applicable to the continuous flow of ICU data could assess improvement or deterioration of individual patients. A recent “proof of concept” report showed potential for this approach (4). Recently, we developed the Criticality using machine learning, a new pediatric severity index for pediatric inpatients based on physiology, therapies, and intensity of care (5–7). This framework follows the prominent threads of critical care severity research including early qualitative assessments, identification and quantification of therapies, and physiological profiles (8–12). The Criticality Index was computed every 6 hours and was calibrated to the probability of ICU care. As the Criticality Index increased for ICU patients, the intensity and complexity of care increased. Therefore, we hypothesized that we could use the Criticality Index to estimate probability of hospital death in ICU patients (Criticality Index – Mortality (CI-M)) using 6 hour time periods to measure dynamic changes in mortality risk. Prior to use in individuals, predictive models developed from populations that purport to measure changing clinical status should maximize predictive performance, especially calibration at the different times used to assess change because, if they are not well-calibrated, change cannot be reliably assessed. Additionally, other measures of construct validity should be used to assess “real-life” scenarios.

The overall aim of this study was the assessment of a machine learning method of serially updating mortality risk for children in ICUs. In this analysis, we estimated mortality risk using the Criticality Index and used serial models over time to maximize performance for individual admissions. We hypothesized that this method would perform well in terms of discrimination, calibration and other performance metrics and would reflect serial risk changes for survivors and deaths. A priori, we anticipated trajectories of mortality risk as follows: the risks of high-risk deaths would stay high or increase over time; the risks of high-risk survivors would decrease; the risks of low-risk deaths would increase; and the risks of low-risk survivors would stay low. We also anticipated that deaths would, in general, have a more volatile clinical course than survivors when assessed with serial mortality risks.

Methods

Sample

The dataset was derived from Health Facts® (Cerner Corporation, Kansas City, MO), a voluntary, comprehensive de-identified clinical database on admissions from U.S. hospitals with a Cerner data-use agreement. Data are obtained from the electronic health record (EHR), are date- and time-stamped, and include admission and demographic information, laboratory results, medications information, diagnostic and procedure codes, vital signs, respiratory data, and hospital outcome. Not all data are available for all admissions. Health Facts® is representative of the United States (13) and used in previous care assessments, including the APACHE score (14), the Criticality Index (5–7), and pediatric medication use (15, 16).

Details on data cleaning and definitions, medication classification, and diagnostic classifications have been published (5, 6, 17). For this analysis, we emphasized the multi-system nature of ICU disease by categorizing all systems of dysfunction based on the discharge ICD-9 and ICD-10 classifications. Inclusion criteria included age <22 years (18) and care in the ICU from 2009 to 2018. Exclusion criteria included hospital length of stay (LOS) >100 days, ICU LOS >30 days, or neonatal ICU care. There were 88 hospitals with an average of 311 admissions/hospital. The dataset included hospitals contributing both small and large samples to expand the generalizability of the methodology. The study was approved by the Children’s National institutional review board (protocol #9282). The information and methodology conform to the “Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD)” guidelines and are included in the Supplemental Digital Data (SDD) 1.

Variables

The Criticality Index independent variables consist of 6 vital signs, 30 routinely measured laboratory variables, and medications classified into 143 categories, and also included the number of measurements/time period (5). The variables, definitions, and statistics have been reported in detail (5–7). Detailed data on the independent variables and imputed values are shown in SDD 1. The primary diagnosis was not used for modeling because it was determined at discharge. In the initial studies, positive pressure ventilation was used only to classify high-intensity ICU care. For this study, we used positive pressure ventilation as an independent variable and categorized an admission as “Yes” for ventilation for all time periods following its implementation. Criteria for positive pressure were continuous positive airway pressure, positive end expiratory pressure, and/or peak inspiratory pressure Consistent with other machine learning models, we imputed laboratory results and vital signs using the last known result(4, 19–21). These imputed values have been reported (5, 6) and were identified by setting the count equal to zero (see below).

The outcome for all CI-M models (see below) was hospital outcome as survival or death. Time to death was not included in the model since a majority of deaths in many pediatric ICUs are associated with brain death or withdrawal and limitations of care (22).

Machine Learning Methodology and Statistical Analysis

The hospital course was discretized into consecutive 6-hour time periods. Models were independently developed for each time period (n=29) from the second (hour 6) to the 30th (hour 180). Modeling was truncated at 180 hours to ensure an appropriate sample for goodness-of-fit (GOF) testing since the test sample size was reduced to <750 admissions and 30 deaths due to discharges and deaths. For each time period, 87% of admissions were randomly used for model development and 13% for testing. We used individual machine leaning models for each time interval to maximize predictive performance because the eventual intent is to apply these methods to individual admissions.

We computed each admission’s Criticality Index for each time period using the previously published machine learning methodology and detailed in SDD 1. (5, 6, 7). Previously, we demonstrated that as the Criticality Index increases, the intensity and complexity of care increases. For each time period, we added a final step that calibrated the Criticality Index and positive pressure ventilation variables to hospital outcome of survival or death using generalized thin plate splines for binary outcomes (23, 24). The performance of the individual CI-M models and the composite of all models was assessed using a) discrimination (area under the receiver operating characteristic curve [AUROC]) and calibration and b) specificity, precision, F1 score, Mathew’s correlation coefficient (MCC), and negative predictive value at sensitivities of 0.85, 0.90, and 0.95. Calibration was assessed using two approaches. First, we computed the Hosmer-Lemeshow GOF test for each time period using risk interval with at least 250 admissions. Second, we assessed calibration plots of the observed and expected proportion of deaths with linear regression using at least 10 risk intervals with equal numbers of admissions. Performance metrics for the calibration plots included the regression line slope, intercept, and the coefficient of determination (R²). We also compared the observed to predicted number of outcomes for each of the calibration plot risk intervals and reported the percentage of intervals with no statistical evidence of difference (p > .05) (25–28). For perfect calibration, the intersect would be zero, the slope would be 1, and the R² would be 1. We expected approximately 5% of the observed versus expected outcome intervals in the calibration plots would be statistically different. Since real-time application would be dependent, in part, on the worst performance at any time period, we examined the worst performing models for each of the calibration methods using the AUROCs, the GOF tests, the calibration plots, and observed versus expected proportion of deaths, and compared these data to the first and the last time period models.

Construct validity was assessed using population trajectories and mortality risk changes in consecutive time periods (clinical volatility) for individuals. First, we plotted the clinical trajectories (6) in the following groups based on the mortality risk determined from the first time period: Deaths in the highest risk decile of deaths (high-risk deaths), survivors in the highest risk decile of survivors (high-risk survivors), deaths in the lower risk deciles of deaths (low-risk deaths), and survivors in the lower risk deciles of survivors (low-risk survivors). A priori, we expected that the mortality risk of high-risk deaths would remain high risk or increase over time, the risk of high-risk survivors would decrease, the risk of low-risk deaths would increase, and the risk of low-risk survivors would remain low. Second, we computed the change in mortality risk for individuals over consecutive time intervals and evaluated all changes, and the maximum positive (clinical deterioration) and maximum negative (clinical improvement) change per admission. A priori, we expected that deaths would have larger changes representing more clinical volatility (instability). These comparisons used the Mann–Whitney U test.

Results

A total of 27,354 admissions were included (Table 1). The mortality rate was 1.8% and the median ICU length of stay was 53 hours (25^th, 75^th percentiles: 24, 117 hours). Respiratory, cardiovascular, and neurological dysfunction occurred in 34.8% (n=9515), 31.2% (n=8541), and 25.7% (n=7024) of admissions, respectively. The test sample started with 3453 admissions and 97 deaths with a decrease due to discharges and deaths of 6–7% per time period during the first 48 hours in the ICU and 3–6% per time period after 48 hours. A total of 47.5% (n=46) of the deaths occurred in the first 48 hours.

Table 1.

Population characteristics of children in intensive care units.

Characteristic	Data
N	27,354
Femaleb	12,477 (45.6%)
Raceb
Black	7,516 (27.45%)
Caucasian	14,445 (52.8%)
Other-Unknown	5,392 (19.7%)
Age (Months)a	155.5 (26.6, 224.8)
Hospital LOS (hours)a	86 (45, 171)
ICU LOS (hours)a	53 (24, 117)
Hospital Mortalityb	501(1.8%)
Positive Pressure Ventilationb,c	3,881 (14.2%)
Systems of Dysfunctionb,d
Respiratory System	9,515 (34.78%)
Nervous System	7,024 (25.68%)
Cardiovascular System	8,541 (31.22%)
Infectious and Parasitic Diseases	3,047 (11.14%)
Gastrointestinal System	6,464 (23.63%)
Hematology	5,790 (21.17%)
Musculoskeletal System	2,832 (10.35%)
Endocrine, Nutritional, Metabolic Diseases	9,281 (33.93%)
Neoplasms	1,099 (4.02%)
Injury and Poisonings	9,035 (33.03%)
Mental Disorders	7,379 (26.98%)

^aMedian (25^th percentile, 75^th percentile)

^bN (%)

^cCriteria for positive pressure were continuous positive airway pressure, positive end expiratory pressure, and/or peak inspiratory pressure.

^dCategorization based on all discharge ICD-9 and ICD-10 data.

The AUROC assessing discrimination ranged from 0.797 to 0.894 for individual CI-M models (Figure 1a) and 0.852 (95% CI: 0.843, 0.861) for all time periods (Figure 1b). Calibration assessed by GOF testing revealed that all models except one at 72 hours of ICU care had p-values of >.05 (see below). The calibration plots for all models for all time periods (Figure 2a) had intercepts ranging from −0.002 (hour 60) to 0.009 (hour 126), slopes ranging from 0.867 (hour 108) to 1.415 (hour 60), and R²’s ranging from 0.862 (hour 72) to 0.989 (hour 96). For all models combined (Figure 2b), the GOF significance level was 0.195, the intercept was 0.010, the slope was 0.903, and the R² was 0.862. Comparison of the observed versus expected proportion of deaths in all calibration plot risk interval found that 290 of 294 risk intervals (98.6%) were not statistically different. The overall and individual performance metrics at sensitivities of 0.85, 0.90, and 0.95 are shown in SDD 2. Overall, at a sensitivity of 0.90, specificity = 0.630 (95% CI: 0.625, 0.634), precision = 0.069 (95% CI: 0.065, 0.072), negative predictive value = 0.995 (95% CI: 0.995, 0.996), MCC = 0.184 (95% CI: 0.177, 0.191), and F1 = 0.127 (95% CI: 0.123, 0.132). Individual models were similar.

Figure 1.

Area under the receiver operating characteristic curves (AUROC) for all Criticality Index – Mortality models and for all models combined.

a. Separate models were developed for 29 6-hour time periods from hour 6 to hour 180. The dots are AUROCs for the individual models and the shaded areas are the 95% confidence intervals

b. Composite AUROC. All models have been combined.

Figure 2.

Calibration plot performance metrics for all Criticality Index – Mortality models and for all models combined

a. Separate models were developed for 29 6-hour time periods from hour 6 to hour 180. Calibration plot metrics include the regression line intercepts (black), slopes (tan) and R²’s (blue). The shaded areas are the 95% confidence intervals.

b. Composite calibration plot. The dashed line is the line of identity and the solid line is the regression line. The plot includes 294 risk intervals from all models. The comparison of observed proportions to expected proportions of outcome for each interval were not statistically different (p > 0.05) for 290 (98.64%) of the risk intervals. Each interval had ≥ 170 admissions. The Hosmer-Lemeshow goodness-of-fit test for the overall calibration was 0.195.

The two worst performing time periods were hour 72, the only time period when the GOF was <0.05, and hour 60 when the calibration plot regression line slope = 1.415 and R² = 0.862. The AUROCs, calibration plots, and observed versus expected comparisons for the risk intervals of these time periods and comparison data from the first and last time periods are shown in SDD 2, . Neither hour 72 nor hour 60 were associated with other indicators of poor performance.

Construct validity for potential clinical application used population trajectories and clinical volatility. The mortality risk trajectories (Figure 3) demonstrated the a priori expectations were correct. The high-risk death cohort had the highest mortality risks, that remained high throughout the ICU course. The high-risk survivor cohort and the low-risk death cohort had similar mortality risks for the first 48 hours but the survivor cohort risks improved and reached the level of the low-risk survivor group at approximately 5 days while the low-risk deaths cohort had mortality risks that rose slightly over the ICU course. The low-risk survivors’ mortality risks remained very low. Deaths demonstrated more clinical volatility than survivors (Figure 4a). The average increase in mortality risk in consecutive time periods was 0.021 for deaths and 0.006 for survivors (p<0.001) and the average decrease was 0.022 for deaths and 0.008 for survivors (p<0.001). Figures 4b and 4c show the maximum deterioration and maximum improvement for survivors and deaths, illustrating the increased higher volatility of deaths compared to survivors. The average maximum deterioration for deaths was 0.050 and 0.015 for survivors (p<0.001) and the average maximum improvement was 0.063 for deaths and 0.022 for survivors (p<0.001).

Figure 3.

Trajectories for high risk deaths and survivors and low risk deaths and survivors. Risk was computed with the Criticality Index – Mortality. High risk indicates the highest mortality risk decile determined from the initial time period. The shaded areas are the 95% confidence intervals. The mortality risks cutpoints for the high risk cohorts were 15.1% for the deaths and 11.3% for the survivors. The trajectories were constructed from the total sample with the following sample sizes: low risk survivors: 24,168, admissions, high risk survivors: 2,685 admissions, low risk deaths: 451 admissions, high risk deaths: 50 patients.

Figure 4.

Change in mortality risk for individual admissions in consecutive time periods. Risk was computed with the Criticality Index – Mortality. Data are shown for survivors (tan) and deaths (black). The frequencies (vertical axis) are standardized such that the added areas under the bars are 1 for each outcome group in each figure.

a. Change in mortality risk for all consecutive time intervals. Deaths demonstrate increased clinical volatility compared to survivors. The average increase in mortality was 0.021 for deaths and 0.006 for survivors (p<0.001) and the average decrease was 0.022 for deaths and 0.008 for survivors (p<0.001).

b. Maximum mortality risk increase (clinical deterioration). The average maximum deterioration for deaths was 0.050 and 0.015 for survivors (p<0.001)

c. Maximum mortality risk decrease (clinical improvement). The average maximum decrease was 0.063 for deaths and 0.022 for survivors (p<0.001).

Discussion

This study demonstrated the applicability of the CI-M for assessments of serial changes in mortality risk for individuals. The Criticality Index was initially calibrated to probability of ICU care and has been applied to determining future care needs for hospitalized children (7). This study expands its use by recalibrating it to mortality risk. We used individual machine leaning models for 6-hour time intervals from 6 hours to 180 hours of ICU care to maximize predictive performance. Overall, model performance metrics were very good. The composite AUROC was 0.852 and, perhaps more important for potential individual applications, calibration was excellent with 28 of 29 Hosmer-Lemeshow GOF tests p > .05, observed versus expected outcomes in 290 of 294 (98.6%) risk intervals of the individual models not statistically different, and calibration plot metrics were very good for the overall performance and the individual models. We evaluated in detail the two time periods with the worst calibration metrics and found that, while single metrics were notable, the assessment with multiple metrics indicated that the calibration metrics were not consistently reduced. Thus, the methodology consistently performed well over the first seven and a half days of ICU care. Clinical validity including clinical trajectories and clinical volatility supported the potential for use in individuals by capturing the anticipated patterns of illness in survivors and deaths. Eventually, however, the methodology must be implemented and assessed in real-world use including comparison to other severity assessment methods.

Experienced intensivists are excellent at assessing patients using clinical snapshots (29). The flow and amount of ICU data for patient assessment is substantial. For example, over 200 variables have been estimated as useful for care of ventilated patients. (30). The ability to successfully integrate this large amount of changing information on a continuous basis lies beyond the capabilities of most knowledgeable, and perceptive care givers. (29). Less experienced and less skilled providers will integrate this information less well. Therefore, the addition of continuous or frequently updated risk assessments for children in ICUs could result in detection of clinical deterioration or improvement that might have been unappreciated, providing an opportunity for earlier interventions and the potential for improved outcomes.

The CI-M has a strong conceptual framework based in physiology, therapeutics, and therapeutic intensity. It is currently calibrated to six-hour time intervals but could be calibrated for continuous data. Single time period machine learning models for ICU mortality prediction have also performed very well (21, 31, 32). Our results are consistent with the recent single site “proof of concept” analysis demonstrating the potential for a machine learning approach to follow changes in clinical status (4). However, there are notable differences between the studies. First, our analysis is based on a multi-centered dataset, demonstrating the methodology can be applied widely. Second, our method uses a transparent set of variables (5, 6). Since these methodologies should be expected to supplement physician judgement, we excluded variables that might have included clinician’s prognosis. Third, the neural networks were different. Since both approaches have positive attributes, future studies will be needed to assess the best approach. Fourth, we took special effort to assess its potential for use in individuals including both GOF and calibration plots at 6 hour intervals, comparison of observed versus expected outcomes in 294 risk intervals, and assessments of mortality risk changes over time. Notably, we detected substantial differences in clinical volatility between deaths and survivors that could be useful for bedside use.

At this time, it is not clear if machine learning models developed on a multi-institutional dataset should be applied to individual sites, or if individual sites should apply the experience and machine learning approach from multi-institutional research to their sites. It is likely that models developed in individual units using a conceptual framework such as the Criticality Index or that of Aczon et al. supplemented with site-specific data and local decision making cutpoints will have improved performance and clinical applicability (4, 33). Optimizing performance is important if the application is intended for individual patients.

There are limitations to this analysis. First, we used a retrospective EHR dataset. While we have utilized this dataset for multiple pediatric analyses, (5–7, 15, 16) prospective data collection could add additional data elements. Second, a more extensive exploration of machine learning methods might have uncovered better performing models. Our methodology was primarily designed to evaluate our overall aims by assessing the potential for eventual clinical use, but not necessarily optimizing it for clinical use. Third, we did not evaluate the relative importance of individual data elements. Previously, we analyzed the global factors associated with prediction in our models and found that a relatively limited dataset composed primarily of physiological data and medication classes may be sufficient (34). Finally, prior to use as a patient-level assessment method, this methodology or any other will need real-world validation, including background or silent use and correlations of changes in clinical status with changes in the mortality risk computed by the models, and analyses of their usefulness vis-a-vis providers of different experience and expertise.

Conclusion

Changing mortality risks for pediatric ICU patients can be measured with machine learning models based on the Criticality Index. Discrimination and calibration for all CI-M models was very good, and clinical validity was demonstrated using clinical trajectories and clinical volatility. The CI-M framework and modeling method are potentially applicable to monitoring patient improvement and deterioration in real-time.