Abstract
Objectives:
To assess severity of illness trajectories described by the Criticality Index for survivors and deaths in five patient groups defined by the sequence of patient care in ICU and routine patient care locations.
Design:
The Criticality Index developed using a calibrated, deep neural network, measures severity of illness using physiology, therapies and therapeutic intensity. Criticality Index values in sequential 6-hour time periods described severity trajectories.
Setting:
Hospitals with pediatric inpatient and ICU care.
Patients:
Pediatric patients never cared for in an ICU (n=20,091), patients only cared for in the ICU (n=2096) and patients cared for in both ICU and non-ICU care locations (n=17,023) from the 2009–2016 Health Facts® database (Cerner Corporation, Kansas City, MO).
Interventions:
None.
Measurements and Main Results:
Criticality Index values were consistent with clinical experience. The median (25th-75th percentile) ICU Criticality Index values (0.878 (0.696, 0.966)) were more than 80-fold higher than the non-ICU values (0.010 (0.002, 0.099)). Non-ICU Criticality Index values for patients transferred to the ICU were 40-fold higher than those never transferred to the ICU (0.164 vs. 0.004). The median for ICU deaths was higher than ICU survivors (0.983 vs 0.875) (p<.001). The severity trajectories for the 5 groups met expectations based on clinical experience. Survivors had increasing Criticality Index values in non-ICU locations prior to ICU admission, decreasing Criticality Index values in the ICU, and decreasing Criticality Index values until hospital discharge. Deaths had higher Criticality Index values than survivors, steeper increases prior to the ICU, and worsening values in the ICU. Deaths had a variable course, especially those who died in non-ICU care locations, consistent with deaths associated with both active therapies and withdrawals/limitations of care.
Conclusion and Relevance:
Severity trajectories measured by the Criticality Index showed strong validity, reflecting the expected clinical course for 5 diverse patient groups.
Keywords: pediatrics, severity of illness, intensive care, machine learning, pediatric intensive care unit, dynamic modeling
Article tweet:
A new severity index for hospitalized children including those in the ICU is applicable to dynamic assessments of clinical status.
Introduction
Accurate and reliable assessments of clinical status in routine and intensive care patients are important to initiate therapies and anticipate future care needs. Risk models have been developed for assessing patients, frequently intensive care patients, although few are specifically designed to provide continuous or on-demand assessment of severity of illness. The most prominent pediatric and adult methods estimate mortality risk for intensive care unit (ICU) patients based on static, time-limited time periods such the first one, four or 24 hours after admission.1–6 Other methods, especially those targeted for routine care patients, reflect the likelihood of adverse events that might be prevented by early intervention or early transfer to the ICU.7,8
Objective assessments of clinical trajectories may improve patient care by recognizing past and predicting future patterns indicative of clinical deterioration or rapid improvement.9,10 Risk-based clinical trajectories based on objective assessments may be especially helpful for care providers with limited experience or training, and well-performing models might outperform even experienced health care providers in the early detection of subtle physiologic changes.11
We developed and validated a novel severity index for routine and intensive care pediatric patients, the Criticality Index, that estimates severity in finite time periods throughout the hospital course. The Criticality Index is based on variables that reflect physiology, therapy, and therapeutic intensity and was developed with machine learning methodology. The development and initial validation data are detailed in a companion paper.12 The primary aim of this analysis was to assess if the Criticality Index reliably reflected the expected severity trajectory determined for sequential 6-hour time periods for the whole hospital course. The expected severity trajectories were based on clinical experience, the sequence of care locations and hospital outcome. For example, patients admitted to routine care and then transferred to the ICU would be expected, on average, to become sicker prior to transfer to the ICU and survivors should show improvement prior to discharge from the ICU and hospital. These analyses were done in the context of developing and validating a method to predict severity trajectories.
Methods
Database
Database details including the source and data preparation are contained in the companion manuscript. In brief, the dataset was derived from the Health Facts® database (Cerner Corporation, Kansas City, MO) that collects comprehensive clinical data on patient encounters from hospitals in the United States with a Cerner data use agreement. Health Facts® data is date and time-stamped data from affiliated patient care locations including admission and demographic data, care-setting characteristics, laboratory results, medication data derived from pharmacy records, diagnostic and procedure codes, vital signs, respiratory data, and hospital outcome. Cerner Corporation has established Health Insurance Portability and Accountability Act compliance operating policies to establish de-identification of Health Facts®. The data has been assessed as representative of the United States,13 and used in previous care assessments, including the APACHE score14 and medication assessments for children in ICUs.15,16 Details of data cleaning, data definitions, medications and medication classification, laboratory data, vital signs and respiratory data have been published.12
Sample
This patient sample is a subsample of the development and validation sets used in the companion paper.12 Inclusion criteria included hospitalization in routine or intensive care units (ICUs) between January 2009 and June 2016, pediatric age defined as <22 years17 and laboratory, vital signs, and medication data. Exclusion criteria included length of stay outliers (hospital length of stay >100 days, ICU length of stay > 30 days) and care in the neonatal ICU. Patients with multiple ICU admissions during their hospitalization were also excluded because the sample was small and an expected clinical course could not be reliably anticipated. We included all patients with one ICU admission and a randomly selected sample of patients with only routine care approximately equal to the ICU sample in size. Other data included age, race, hospital and ICU lengths of stay, and hospital outcome as survival or death. For survivors discharged from the hospital directly from the ICU, we classified the discharge care location as home or chronic care. Chronic care was defined by discharge to another care facility (another hospital or skilled nursing, rehabilitation, or hospital facilities).
Independent Variables
Predictor variables are detailed in the companion manuscript and included the values and number of measurements of common laboratory test results, vital signs and coma scores, and medications. Medications included 1113 medications categorized into 143 medication categories.18
Criticality Index
The Criticality Index model estimates the probability of ICU care for 6-hour time period using a calibrated, deep neural network. The methodology and development and validation data are detailed in the companion manuscript. In brief, a deep neural network was fully-connected and designed to maximize accuracy for the binary task of classifying patients as being in the ICU or not. The modelling software used the R-package keras.19,20 Since deep neural networks have a tendency to be over-confident in classifications, we added an additional calibration step. Calibration was accomplished by using the neural network output for B-spline polynomials21 as covariates in a linear logistic regression with the outcome of ICU care. This calibration method is similar to the Platt scaling method for support vector machines.22 The output of this logistic regression is the Criticality Index. Each 6-hour time period has a single Criticality Index.
Severity Trajectories
Five trajectory groups based on the sequence of care locations are detailed in Table 1. Care locations were characterized as ICU or non-ICU. Because the time in care locations is variable, the time in each care location was normalized from 0 to 100%. The initial time period in a care location was 0% and the last was 100%. For example, if a patient cared for in a non-ICU care area for 24 hours has 4 time periods, the first Criticality Index was noted as time 0%, the last as 100%, the second as 33.3% and the third as 66.7%. The Criticality Index values between the known Criticality Index values were interpolated using linear functions. The severity trajectory for each care area was composed of 100 time points. These trajectories use all patients with 2 or more time periods in a care location.
Table 1.
Trajectory Groups
| Trajectory Characteristics | General Expectations | Survivors vs. Deaths | |
|---|---|---|---|
| Group 1. Non-ICU Care Only | Hospital admission and discharge from non-ICU care. | Criticality Index values will be low and remain low. | Deaths will have higher Criticality Index values than survivors. |
| Group 2. Non-ICU → ICU | Hospital admission to a non-ICU care area, transferred to the ICU, and discharged from the hospital from the ICU. | Admissions will have higher Criticality Index values than Group 1 during non-ICU care. Criticality Index values will increase prior to ICU admission. Survivor values will improve in the ICU prior to discharge. Hospital discharge Criticality Index values will be higher than Group 1. | Deaths will have higher Criticality Index values than survivors during their entire hospital stay. |
| Group 3. Non-ICU → ICU → non-ICU | Hospital admission to a non-ICU care area, transferred to the ICU, discharged from the ICU to a non-ICU care area, and discharged from the hospital from the non-ICU care area. | Admissions will have higher Criticality Index values than Group 1 during non-ICU care. Criticality Index values will increase prior to ICU admission. Survivor values will improve in the ICU prior to discharge to non-ICU care. Criticality Index values for survivors will continue to decrease in the non-ICU care areas until discharge. Discharge Criticality Index values will be lower than Group 2 and similar to Group 1. | Deaths will have higher Criticality Index values than survivors during their entire hospital stay. |
| Group 4. ICU only | Hospital admission to and hospital discharge from the ICU. | Criticality Index values will be high on admission to the ICU and improve for survivors, but not for deaths. | Deaths will have higher Criticality Index values than survivors during their ICU stay. |
| Group 5. ICU → non-ICU | Hospital admission to the ICU, transfer to non-ICU care areas, and hospital discharge from the non-ICU care areas. | Criticality Index values will be high and decrease in the ICU prior to discharge to non-ICU care for survivors. Criticality for survivors will continue to decrease in routine care until discharge. | Deaths will have higher Criticality Index values than survivors during their entire hospital stay. |
We also assessed Criticality Index changes in real time surrounding transitions (e.g. ICU admission and discharge, hospital admission and discharge) and compared these to expectations of increasing and decreasing severity. For example, patients admitted to the ICU from non-ICU care areas were expected to having increasing severity prior to ICU admission, and those recovering to be discharged from the ICU to non-ICU areas or out of the hospital were expected to have decreasing severity. These data are displayed in 24 hour real-time periods without interpolation. Only patients with complete data for the display periods are included.
Statistics
The Wilcoxon signed-rank test was used for paired comparisons. For multiple group comparisons, we used the Kruskal Wallis one-way analysis of variance and chi-square test. For Criticality Index pairwise comparisons between patient groups, we used the Mann–Whitney U test.
Data are presented as median (25th and 75th percentiles).
Results
This analysis includes 20,091 patients never admitted to the ICU, 2096 patients cared for only the ICU, and 17,023 patients cared for in both the ICU and non-ICU care locations. Descriptive characteristic of the trajectory groups (Table 1) are shown in Table 2. Overall, the median age was 97 months. Patients admitted directly to the ICU were younger than other groups and those never admitted to the ICU were oldest. The overall hospital mortality rate was 1.56%, the mortality rate for those cared for only in non-ICU care areas was 0.13% and the mortality rates for the groups cared for in the ICU ranged from 1.26% to 9.92%. There were different diagnostic distributions for each of the trajectory groups. The median number of 6-hour time periods was 16/patient. The most 6-hour time periods were in the Group 3 (non-ICU → ICU → non-ICU) (median 23/patient) and the least were in Group 4 (ICU only) (median 10/patient).
Table 2. Population Characteristics By Trajectory Groups.
See Table 1 for description of the trajectory groups.
| Groups (n) | |||||||
|---|---|---|---|---|---|---|---|
| Descriptor | All (39,210) | Group 1. Non-ICU (20,091) | Group 2. Non-ICU→ICU (554) | Group 3. Non-ICU → ICU→Non-ICU (4039) | Group 4. ICU (2096) | Group 5. ICU→Non-ICU (12,430) | P value |
| Age (months) (median) | 97 | 132 | 128 | 86 | 58 | 9 | < 0.01 |
| Age Group (n (%)) | <0.01 | ||||||
| < 2 years | 11436 (29.2) | 2049 (10.2) | 151 (27.3) | 1464 (36.3) | 942 (44.9) | 6830 (55.0) | < 0.01 |
| 2 years – 6 years | 5923 (15.1) | 4386 (21.8) | 60 (10.8) | 442 (10.9) | 142 (6.8) | 893 (7.2) | < 0.01 |
| 6 years – 13 years | 6991 (17.8) | 4924 (24.5) | 103 (18.6) | 564 (14.0) | 255 (12.2) | 1145 (9.2) | < 0.01 |
| 13 years – 22 years | 14860 (37.9) | 8732 (43.5) | 240 (43.3) | 1569 (38.9) | 757 (36.1) | 3562 (28.7) | < 0.01 |
| Female (n (%)) | 19,156 (48.9) | 10,663 (53.1) | 256 (46.2) | 1798 (44.5) | 921 (43.9) | 5518 (44.4) | < 0.01 |
| Race (n (%)) | |||||||
| African American | 9081 (23.2) | 4089 (20.4) | 143 (25.8) | 1032 (25.6) | 571 (27.2) | 3246 (26.) | <0.01 |
| Caucasian | 18609 (47.5) | 8691 (43.3) | 328 (59.2) | 2141 (53.0) | 1085 (51.8) | 6364 (51.2) | < 0.01 |
| Hospital LOS (hours) (1) | 102 (60,204) | 90 (60,162) | 96 (48,186) | 150 (84,294) | 72 (42,138) | 132 (72,282) | < 0.01 |
| ICU LOS (hours) (1) | 74 (33, 167) | NA | 66 (20, 142) | 53 (18,119) | 91 (46,165) | 81 (37, 188) | < 0.01 |
| 6-hour Periods/Pt (n) (1) | 16 (9, 31) | 14 (9, 24) | 13 (6, 26) | 23 (12, 42) | 10 (6, 22) | 20 (11, 43) | < 0.01 |
| Hospital Mortality (n (%)) | 612 (1.56) | 26 (0.13) | 13 (2.35) | 51 (1.26%) | 208 (9.92) | 314 (2.53) | < 0.01 |
| Diag Cat (n (%)) | 25,642 | 15,214 | 261 | 2098 | 1,138 | 6931 | |
| Heme | 1740 (6.8) | 1432 (9.4) | 15 (5.8) | 81 (3.9) | 23 (2.0) | 189 (2.7) | <0.01 |
| S and SC | 423 (1.7) | 184 (1.2) | 6 (2.) | 40 (1.9) | 15 (1.3) | 178 (2.5) | <0.01 |
| Circulatory | 1144 (4.5) | 460 (3.0) | 26 (10.0) | 163 (7.8) | 88 (7.7) | 407 (5.9) | <0.01 |
| GI | 2525 (9.9) | 2053 (13.5) | 18 (7.0) | 149 (7.1) | 34 (3.0) | 271 (3.9) | <0.01 |
| GU | 965 (3.8) | 631 (4.2) | 7 (2.7) | 45 (2.1) | 37 (3.3) | 245 (3.5) | <0.01 |
| MSC | 758 (3.0) | 521 (3.4) | 9 (3.5) | 78 (3.7) | 12 (1.1) | 138 (2.0) | <0.01 |
| Neurology | 1804 (7.0) | 1068 (7.0) | 26 (10.0) | 162 (7.7) | 71 (6.2) | 477 (6.9) | 0.18 |
| Respiratory | 3544 (13.8) | 2047 (13.5) | 37 (14.2) | 361 (17.2) | 177 (15.6) | 922 (13.) | <0.01 |
| NOS | 2655 (10.4) | 995 (6.5) | 17 (6.5) | 214 (10.2%) | 175 (15.4) | 1254 (18.1) | <0.01 |
| ENM&I | 3114 (12.1) | 1532 (10.0) | 18 (7.0) | 176 (8.4) | 234 (20.6) | 1154 (16.7) | <0.01 |
| Infectious | 2319 (9.0) | 1412 (9.3) | 28 (10.7) | 211 (10.1) | 85 (7.4) | 583 (8.4) | 0.02 |
| I & P | 2194 (8.6) | 870 (5.7) | 37 (14.2) | 273 (13.0) | 150 (13.2) | 864 (12.5) | <0.01 |
| Mental | 855 (3.3) | 612 (4.0) | 7 (2.7) | 61 (2.9) | 20 (1.8) | 155 (2.2) | <0.01 |
| Neoplasm | 1602 (6.3) | 1397 (9.2) | 10 (3.8) | 84 (4.0%) | 17 (1.5) | 94 (1.4) | <0.01 |
Abbreviations: NA: Not Applicable; LOS = Length of Stay; Pt = Patient; Diag Cat = Diagnostic Categories; S and SC = Skin and Subcutaneous Tissue; GU = Genital Urinary; MSC = Musculoskeletal and Connective Tissue; GI = Gastrointestinal; NOS = not otherwise specified; ENM&I = Endocrine, Nutritional, Metabolic and Immunity; I & P = Injury and Poisoning; heme = blood forming organs
Median (25th, 75th percentile )
Summary Criticality Index values were consistent with clinical expectations. The median ICU Criticality Index for all ICU time periods in trajectory groups 2 – 5 (0.878 (0.696, 0.966)) was more than 80-fold higher (P<.0001) than the non-ICU time periods (0.010 (0.002, 0.099)). Patients admitted to non-ICU care areas and transferred to the ICU had more than a 40-fold higher (P<.0001) Criticality Index (0.164 (0.029, 0.523)) than those admitted to non-ICU care areas but never admitted to the ICU (0.004 (0.001, 0.025)). Gradations of criticality for the first, last, maximum, and average for survivors, deaths, and those with and without positive pressure ventilation were also seen in the ICU patients (Supplemental Digital Content 1). The Criticality Index for all ICU death time periods (median 0.971 (0.911, 0.991)) was significantly (p < .0001) higher than ICU survivors (median 0.783 (0.537, 0.906)) and this was also observed for the first ICU time period (deaths: (0.983 (0.938, 0.999)); survivors: 0.875 (0.628, 0.974)), P<.0001). The Criticality Index for all time periods of patients receiving or who would receive positive pressure ventilation during the ICU admission was significantly higher than for those who did not (ventilated: 0.903 (0.799, 0.961)); not ventilated: 0.737 (0.486, 0.884), P<.0001).
The observed severity trajectories for survivors and deaths in the trajectory groups (Table 1) are displayed in Figure 1. Figures 2 and 3 show the real-time Criticality Index values for survivors and deaths for the 24-hour periods around care location transitions (hospital admission and discharge, ICU admission and discharge). The median and 25th-75th percentile data for survivors and deaths for initial or last time period for hospital admission, hospital discharge, ICU admission and ICU discharge are shown in Table 3. Panel 1 shows the trajectories for Group 1 patients hospitalized in non-ICU care areas for their entire hospital course. The Criticality Index values for survivors and deaths were both very low throughout their hospital course. Although there are only 26 deaths, the 75th percentile data demonstrate that deaths had a more variable course.
Figure 1.
Severity Trajectories Described by Criticality Index Values for Survivors and Deaths for 5 Trajectory Groups. The Trajectory Groups are described in the vertical panel on the right. ICU time periods are shaded. Time in each care location is normalized from 0% to 100% (see Methods). The sample size for each group is shown in the inset. Only patients with at least 2 time periods in each care area are included.
Figure 2.
Severity Trajectories Described by Criticality Index Values for Survivors for 5 Trajectory Groups. The Trajectory Groups are described in the vertical panel on the right. ICU time periods are shaded. Data are in real time (hours) around admission (A) and discharge (D) from the hospital and ICU. The sample size for each group is shown in the inset. Only patients with data for the indicated time intervals in each care area are included.
Figure 3.
Severity Trajectories Described by Criticality Index Values for Deaths for 5 Trajectory Groups. The Trajectory Groups are described in the vertical panel on the right. ICU time periods are shaded. Data are in real time (hours) around admission (A) and discharge (D) from the hospital and ICU. The sample size for each group is shown in the inset. Only patients with data for the indicated time intervals in each care area are included.
Table 3.
Criticality Indices for Initial Hospital and ICU Time Periods, Last ICU Discharge and Hospital Discharge
| Hospital Admission To Non-ICU (1) | Hospital/ICU Admission (1) | ICU Admission From Non-ICU (2) | ICU Discharge To Non-ICU (3) | ICU/Hospital Discharge (3) | Hospital Discharge From Non-ICU (4) | |
|---|---|---|---|---|---|---|
| Survivors | ||||||
| Group 1. Routine Care Only | 0.025 (0.005, 0.104) | 0.006 (0.001, 0.026) | ||||
| Group 2. Non-ICU → ICU | 0.419 (0.163, 0.695) | 0.655 (0.345, 0.860) | 0.371 (0.136, 0.673) | |||
| Group 3. Non-ICU → ICU → non-ICU | 0.427 (0.158, 0.697) | 0.675 (0.335, 0.865) | 0.380 (0.141, 0.658) | 0.141 (0.045, 0.373) | ||
| Group 4. ICU only | 0.920 (0.740, 0.982) | 0.609 (0.265, 0.832) | ||||
| Group 5. ICU → non-ICU | 0.914 (0.707, 0.986) | 0.603 (0.261, 0.810) | 0.278 (0.098, 0.614) | |||
| Deaths | ||||||
| Group 1. Routine Care Only | 0.005 (0.001, 0.077) | 0.002 (0.000, 0.066) | ||||
| Group 2. Non-ICU → ICU | 0.451 (0.254, 0.596) | 0.903 (0.843, 0.977) | 0.922 (0.414, 0.975) | |||
| Group 3. Non-ICU → ICU → non-ICU | 0.614 (0.369, 0.891) | 0.922 (0.761, 0.974) | 0.843 (0.386, 0.947) | 0.456 (0.156, 0.863) | ||
| Group 4. ICU only | 0.987 (0.954, 0.998) | 0.948 (0.796, 0.990) | ||||
| Group 5. ICU → non-ICU | 0.987 (0.948, 0.997) | 0.933 (0.728, 0.983) | 0.826 (0.295, 0.959) | |||
Initial admission time period to the non-ICU or ICU care areas.
Initial time period in the ICU.
Last ICU time period prior to non-ICU care.
Last hospital time period for discharges from non-ICU or ICU care areas.
The observed severity trajectories for Group 2 (Non-ICU→ICU) are shown in Figure 1, Panels 2. The hospital admission Criticality Index values are more than 15-fold higher than Group 1 patients (Survivors: 0.419 (0.163, 0.695); Deaths: 0.451 (0.254, 0.596). Survivors’ Criticality Index values gradually increased to ICU admission (0.655 (0.345, 0.860)) and showed wide variability, with some patients having a relatively low admission Criticality Index (25th percentile: 0.345) consistent with patients admitted primarily for monitoring and others having a high Criticality Index (75th percentile: 0.860) indicating more severe illness. Survivors’ Criticality Index values improved to ICU discharge (0.371 (0.136, 0.673)). The 105 (19.4%) survivors discharged from the ICU to chronic care environments had higher Criticality Index values (0.508 (0.208, 0.800)) than 436 (80.6%) discharged home (0.337 (0.127, 0.634)) (p <.001). Although there are only 13 deaths, they demonstrated a much steeper increase and a much tighter distribution of values on ICU admission (0.903 (0.843, 0.977)). While the increase in discharge Criticality Index values for deaths was relatively small (0.922 (0.414, 0.975)), the discharge 25th percentile decreased substantially consistent with brain death or withdrawal/limitations of care.
The observed severity trajectories for Group 3 (Non-ICU→ICU→non-ICU) are shown in Figure 1, Panel 3. Survivors in this group had very similar hospital admission Criticality Index values (0.427 (0.158, 0.697)) as Group 2 with a substantial increase to ICU admission (0.675 (0.335, 0.865), a substantial decrease to ICU discharge (0.380 (0.141, 0.658)) and a further decrease to hospital discharge (0.141 (0.045, 0.373)). The 51 deaths had higher hospital admission Criticality Index values (0.614 (0.369, 0.891)) than survivors, very high ICU admission Criticality Index values (0.922 (0.761, 0.974), and relatively high Criticality Index values at both ICU discharge (0.843 (0.386, 0.947)) and hospital discharge (0.456 (0.156, 0.863)).
Figure 1, Panels 4 and 5 illustrate the clinical course for patients initially admitted to the ICU (Group 4: ICU only; Group 5: ICU → non-ICU). The ICU admission Criticality Index values were very similar for survivors and deaths for both Group 4 (Survivors: 0.920 (0.740, 0.982); Deaths: 0.987 (0.954, 0.998)) and Group 5 (Survivors: 0.914 (0.707, 0.986); deaths: 0.987 (0.948, 0.997)). The Criticality Index values of Group 4 survivors decreased substantially at discharge (0.609 (0.265, 0.832)). A total of 17.0% of survivors (n=316/1858) were discharged to a chronic care environment. The Criticality Index values for these patients were higher (0.722 (0.341, 0.920) than those discharged to home (0.586 (0.257, 0.809) (p <0.0001). Group 4 deaths were discharged with very high Criticality Index values (0.948, (0.796, 0.990)). For those discharged to non-ICU care areas in Group 5, the Criticality Index values of survivors decreased substantially from admission to ICU discharge (0.603 (0.261, 0.810) and then Criticality Index values at hospital discharge were low (0.278, (0.098, 0.614). The deaths in Group 5 were discharged with high Criticality Index values from the ICU (0.933 (0.728, 0.983) and the hospital (0.826 (0.295, 0.959) implying that many of the deaths were discharged for comfort care outside of the ICU.
The 24 hour real-time Criticality Index values surrounding hospital and ICU admission and ICU and hospital discharge for survivors and deaths are shown in Figures 2 and 3, respectively. Survivors demonstrated substantial increases in the 24 hours prior to ICU admission and substantial decreases in the 24 hours prior to ICU discharge (all p<.001). Deaths demonstrated steeper trajectories and greater increases than survivors prior to ICU admission (all p<.001). While the median Criticality Index for deaths in the ICU were high, the 25th percent quartile was consistently low, indicating that a substantial portion of deaths were associated with withdrawal or limitation of care or brain death.
There were similarities in the Criticality Index values between the trajectory groups (Table 3). Group 2 and Group 3 survivors had similar in hospital admission to non-ICU areas, ICU admission from non-ICU care, and ICU discharge. Group 4 and Group 5 survivors had similar hospital admission to the ICU and ICU discharge. Group 3 and 5 survivors had similarities in hospital discharge from non-ICU care areas. Deaths had similar Criticality Index values from all trajectory groups for ICU discharge, and ICU admission from non-ICU care areas.
Discussion
Severity trajectories measured by the Criticality Index showed strong validity, reflecting the expected clinical course for 5 diverse patient groups. In this analysis, the severity trajectories of disease progression and resolution, the magnitude of illness severity associated with ICU and non-ICU care locations, the changes in severity associated with changes in care locations, and the severity differences associated with survival and death add to the validity of the Criticality Index. The Criticality Index values for those in non-ICU care areas for their entire hospital course were 40-fold lower than ICU Criticality Index values and remained so during the entire hospital stay. The severity of those initially hospitalized in non-ICU care areas increased very substantially until ICU illness. Those cared for in the ICU who survived had improving severity during the ICU course. Those discharged from the ICU to non-ICU areas showed continued decreasing severity. The Criticality Index values of those discharged from the hospital directly from the ICU were higher than those discharged from non-ICU care areas and for the patients discharged from the hospital directly from the ICU, those discharged to chronic care facilities had higher Criticality Index values than those discharged home. Deaths had higher Criticality Index values than survivors, more severe increases prior to ICU care, and higher values in the ICU. Notably, the variability of Criticality Index values for deaths was quite large, consistent with some patients having physiological decompensation and receiving high therapeutic intensity, and large amounts and therapy and others receiving less therapy, and lesser intensity of care which is consistent, with brain death and/or withdrawals and limitations of care.23–25 While the trajectories were analyzed with normalized time periods, the real-time data for the 24 hours surrounding care location transitions confirmed these data.
We performed these analyses for three reasons. First, we believe that this analysis was important prior to accepting the Criticality Index as a severity measure that integrates physiology, therapy, and therapeutic intensity. Other measures of severity calibrated to mortality1–5 that are based primarily on physiologic abnormalities would not be expected to demonstrate severity trajectories that are consistent with clinical experience because therapy and care intensity alter the physiologic profiles, changing the relationship between physiology and outcome. Second, we wanted to build on the previous analyses that demonstrated excellent separation of Criticality Index distributions in a severity hierarchy of patient groups, statistical metrics for the separation of ICU and non-ICU patients including an AUC for the ROC curve of 0.95 95% CI: (0.95,0.95) and accuracy of 0.866, and performance in classifying severity within individual patients cared for in both ICU and non-ICU care areas.12 Third,, future severity of illness research, if focused on predicting events or future clinical trajectories, should reliably describe the clinical course.10,26–28 Unfortunately, most models of severity use data obtained at a single time interval. The Criticality Index is structured for measurement in sequential time periods throughout a hospital course, making it applicable to predicting clinical states and describing the past trajectory for individual patients.
The appropriate use and interpretation of risks scores includes understanding how the risk model may be influenced by the clinical context. Risk scores measured early in the ICU course are used predominantly to assess quality of care and for case mix adjustment in research because physiology measured prior to or early during therapy is an assessment of initial severity.1,6 Pediatric early warning scores focus on associating a future physiology-based adverse event with a current physiological profile.7 The pediatric Rothman Index using vital signs, nursing assessments and cardiac measures proposes a snapshot of patient “acuity.”29 Other models focus on the longitudinal course of individual diseases and will have limited general applicability.30,31 The Criticality Index uses elements of physiology, therapy, and therapeutic intensity, the core elements of clinical status, because measurements of clinical status using these core elements are expected to better model predictions of future clinical status. This methodology is also applicable to more specific clinical populations such as cardiac intensive care or respiratory illness patients.
There are limitations to the current model and further research may correct these limitations and improve both the performance and relevance of the Criticality Index. First, the 6 hour time period was initially chosen because non-ICU care patients have a relative paucity of data and we believed that this was an appropriate time period to gather sufficient data in this patient group to explore the conceptual foundation of the Criticality Index in routine care as well as ICU patients. However, with a sound conceptual foundation, the Criticality Index model can use shorter time intervals or can be updated dynamically as new data become available. Second, the data elements were selected, in part, because of their availability in a research data set. As databases improve, more data elements could be added. Indeed, the conceptual framework was developed with this in mind and the machine learning approach was chosen to be able to accommodate the potentially expanding independent variables.32,33 For example, we did not include diagnostic information because only discharge data was available in this data set. Computing risk is improved when diagnostic information is included.34,35 Third, we did not fully explore all possible machine learning methods to maximize model performance. Alternate methods are expected to enhance performance.36–39
Conclusion
Accurate clinical assessment of patient status is important to anticipate future clinical needs. The Criticality Index models severity trajectories that are consistent with clinical expectations concerning disease progression and recovering, deterioration and improvement surrounding transitioning between non-ICU and ICU care locations, and differences between survivors and deaths.
Supplementary Material
Acknowledgments
Research Support
Supported by philanthropic support from Mallinckrodt LLC, and award numbers UL1TR001876 from the NIH National Center for Advancing Translational Sciences, and KL2TR001877 from the NIH National Center for Advancing Translational Sciences (Anita Patel). Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the National Center for Advancing Translational Sciences or the National Institutes of Health.
Copyright form disclosure: Drs. Rivera, Patel, Zeng-Treitlier, Chamberlain, Bost, Morizono, and Pollack’s institutions received funding from Mallinckrodt LLC. Drs. Patel, Morizono and Pollack received support for article research from the National Institutes of Health (NIH). Dr. Patel’s institution received funding from Awards Ul1TR001876 and KL2TR001877 from the NIH, National Center for Advancing Translational Sciences (NCATS). Dr. Morizono’s institution received funding from the NIH NCATS; he received funding from Cogthera LLC; and he received support for article research from Mallinckrodt. Dr. Pollack’s institution received funding from the NIH.
Footnotes
Dr. Heneghan disclosed that she does not have any potential conflicts of interest.
Contributor Information
Eduardo A. Trujillo Rivera, George Washington University School of Medicine and Health Sciences.
Anita K. Patel, Department of Pediatrics, Division of Critical Care Medicine, Children’s National Hospital and George Washington University School of Medicine and Health Sciences.
Qing Zeng-Treitler, George Washington University School of Medicine and Health Sciences.
James M. Chamberlain, Department of Pediatrics, Division of Emergency Medicine, Children’s National Hospital and George Washington University School of Medicine and Health Sciences.
James E. Bost, Children’s National Hospital and George Washington University School of Medicine and Health Sciences.
Julia A. Heneghan, Department of Pediatrics, Division of Critical Care Medicine, Children’s National Hospital and George Washington University School of Medicine and Health Sciences. Current affiliation Department of Pediatrics, Division of Critical Care Medicine, University of Minnesota Masonic Children’s Hospital..
Hiroki Morizono, Children’s National Research Institute, Associate Research Professor of Genomics and Precision Medicine, GWU School of Medicine and Health Sciences
Murray M. Pollack, Department of Pediatrics, Division of Critical Care Medicine, Children’s National Hospital and George Washington University School of Medicine and Health Sciences.
References
- 1.Pollack MM, Holubkov R, Funai T, et al. The Pediatric Risk of Mortality Score: Update 2015. Pediatr Crit Care Med 2016;17(1):2–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Leteurtre S, Duhamel A, Salleron J, Grandbastien B, Lacroix J, Leclerc F. PELOD-2: an update of the PEdiatric logistic organ dysfunction score. Crit Care Med 2013;41(7):1761–1773. [DOI] [PubMed] [Google Scholar]
- 3.Zimmerman JE, Kramer AA, McNair DS, Malila FM. Acute Physiology and Chronic Health Evaluation (APACHE) IV: hospital mortality assessment for today’s critically ill patients. Crit Care Med 2006;34(5):1297–1310. [DOI] [PubMed] [Google Scholar]
- 4.Vincent JL, de Mendonca A, Cantraine F, et al. Use of the SOFA score to assess the incidence of organ dysfunction/failure in intensive care units: results of a multicenter, prospective study. Working group on “sepsis-related problems” of the European Society of Intensive Care Medicine. Crit Care Med 1998;26(11):1793–1800. [DOI] [PubMed] [Google Scholar]
- 5.Richardson DK, Gray JE, McCormick MC, Workman K, Goldmann DA. Score for Neonatal Acute Physiology: a physiologic severity index for neonatal intensive care. Pediatrics 1993;91(3):617–623. [PubMed] [Google Scholar]
- 6.Straney L, Clements A, Parslow RC, et al. Paediatric index of mortality 3: an updated model for predicting mortality in pediatric intensive care*. Pediatr Crit Care Med 2013;14(7):673–681. [DOI] [PubMed] [Google Scholar]
- 7.Parshuram CS, Hutchison J, Middaugh K. Development and initial validation of the Bedside Paediatric Early Warning System score. Critical care (London, England). 2009;13(4):R135. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Smith GB, Prytherch DR, Meredith P, Schmidt PE, Featherstone PI. The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death. Resuscitation. 2013;84(4):465–470. [DOI] [PubMed] [Google Scholar]
- 9.Jha RM, Elmer J, Zusman BE, et al. Intracranial Pressure Trajectories: A Novel Approach to Informing Severe Traumatic Brain Injury Phenotypes. Crit Care Med 2018;46(11):1792–1802. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Duan H, Sun Z, Dong W, He K, Huang Z. On Clinical Event Prediction in Patient Treatment Trajectory Using Longitudinal Electronic Health Records. IEEE J Biomed Health Inform 2019. [DOI] [PubMed] [Google Scholar]
- 11.Badawi O, Liu X, Hassan E, Amelung PJ, Swami S. Evaluation of ICU Risk Models Adapted for Use as Continuous Markers of Severity of Illness Throughout the ICU Stay. Crit Care Med 2018;46(3):361–367. [DOI] [PubMed] [Google Scholar]
- 12.Rivera EAT, Patel AK, Chamberlain JM, et al. Criticality: A New Concept of Severity of Illness for Hospitalized Children. Pediatric Critical Care Medicine. 2020;submitted. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.DeShazo JP, Hoffman MA. A comparison of a multistate inpatient EHR database to the HCUP Nationwide Inpatient Sample. BMC Health Serv Res 2015;15:384. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Bryant C, Johnson A, Henson K, Freeseman-Freeman Stark M, Higgins T. Apache Outcomes Acriss Venues Predicing Inpatient Mortality Using Electronic Medical Record Data. Critical Care Medicine. 2018;46:8. [DOI] [PubMed] [Google Scholar]
- 15.Heneghan JAT-RE, Zeng-Treitler Q, Faruqe F, Morizona H, Bost JE, Pollack MM, Patel AK. Medications for Children Receiving Intensive Care: A National Sample”. Pediatric Critical Care Medicine. 2020(in press). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Patel AK, Trujillo-Rivera EA, Heneghan J, et al. Sedation, Analgesia and Neuromuscular Blockade: Current Practice in 66,443 Pediatric Patients Cared for in the Intensive Care Unit. Pediatr Crit Care Med 2020;in press. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Hardin AP, Hackell JM, Committee On P, Ambulatory M. Age Limit of Pediatrics. Pediatrics 2017;140(3). [DOI] [PubMed] [Google Scholar]
- 18.Fung KW, Kapusnik-Uner J, Cunningham J, Higby-Baker S, Bodenreider O. Comparison of three commercial knowledge bases for detection of drug-drug interactions in clinical decision support. J Am Med Inform Assoc 2017;24(4):806–812. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Allaire JJ, Chollet F. Keras: R Interface to ‘Keras’. R package version 2.2.5.0. 2019; https://CRAN.R-project.org/package=keras [cran.r-project.org] Accessed March 12, 2020. [Google Scholar]
- 20.Chollet F, J.J. A. Deep Learning with R Manning Publications Co.; 2018. [Google Scholar]
- 21.Eilers PH, Marx BD. Flexible smoothing with B-splines and penalties. Statistical science. 1996:89–102. [Google Scholar]
- 22.Platt J Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers. 1999;10(3):61–74. [Google Scholar]
- 23.Suttle ML, Jenkins TL, Tamburro RF. End-of-Life and Bereavement Care in Pediatric Intensive Care Units. Pediatr Clin North Am 2017;64(5):1167–1183. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Burns JP, Sellers DE, Meyer EC, Lewis-Newby M, Truog RD. Epidemiology of death in the PICU at five U.S. teaching hospitals*. Crit Care Med 2014;42(9):2101–2108. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Meert KL, Keele L, Morrison W, et al. End-of-Life Practices Among Tertiary Care PICUs in the United States: A Multicenter Study. Pediatr Crit Care Med 2015;16(7):e231–238. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Catling FJR, Wolff AH. Temporal convolutional networks allow early prediction of events in critical care. J Am Med Inform Assoc 2020;27(3):355–365. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Christie SA, Conroy AS, Callcut RA, Hubbard AE, Cohen MJ. Dynamic multi-outcome prediction after injury: Applying adaptive machine learning for precision medicine in trauma. PLoS One. 2019;14(4):e0213836. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Iezzoni LI, Restuccia JD, Shwartz M, et al. The utility of severity of illness information in assessing the quality of hospital care. The role of the clinical trajectory. Med Care. 1992;30(5):428–444. [DOI] [PubMed] [Google Scholar]
- 29.Rothman MJ, Tepas JJ 3rd, Nowalk AJ, et al. Development and validation of a continuously age-adjusted measure of patient condition for hospitalized children using the electronic medical record. J Biomed Inform 2017;66:180–193. [DOI] [PubMed] [Google Scholar]
- 30.Klein Klouwenberg PMC, Spitoni C, van der Poll T, Bonten MJ, Cremer OL, consortium M. Correction to: Predicting the clinical trajectory in critically ill patients with sepsis: a cohort study. Critical care (London, England). 2020;24(1):41. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Dumenci L, Perera RA, Keefe FJ, et al. Model-based pain and function outcome trajectory types for patients undergoing knee arthroplasty: a secondary analysis from a randomized clinical trial. Osteoarthritis Cartilage. 2019;27(6):878–884. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Shah ND, Steyerberg EW, Kent DM. Big Data and Predictive Analytics: Recalibrating Expectations. JAMA 2018;320(1):27–28. [DOI] [PubMed] [Google Scholar]
- 33.Miller DD, Brown EW. Artificial Intelligence in Medical Practice: The Question to the Answer? Am J Med 2018;131(2):129–133. [DOI] [PubMed] [Google Scholar]
- 34.Pollack MM, Holubkov R, Funai T, et al. Simultaneous Prediction of New Morbidity, Mortality, and Survival Without New Morbidity From Pediatric Intensive Care: A New Paradigm for Outcomes Assessment. Crit Care Med 2015;43(8):1699–1709. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Kramer AA, Higgins TL, Zimmerman JE. Comparing observed and predicted mortality among ICUs using different prognostic systems: why do performance assessments differ? Crit Care Med 2015;43(2):261–269. [DOI] [PubMed] [Google Scholar]
- 36.Che Z, Purushotham S, Cho K, Sontag D, Liu Y. Recurrent Neural Networks for Multivariate Time Series with Missing Values. Sci Rep 2018;8(1):6085. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Rajkomar A, Dean J, Kohane I. Machine Learning in Medicine. N Engl J Med 2019;380(14):1347–1358. [DOI] [PubMed] [Google Scholar]
- 38.Chen JH, Asch SM. Machine Learning and Prediction in Medicine - Beyond the Peak of Inflated Expectations. N Engl J Med 2017;376(26):2507–2509. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Bang S-J, Wang Y, Yang Y. Phased-LSTM Based Predictive Model for longitudinal EHR Data with Missing Values. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.