Predicting Duration of Invasive Mechanical Ventilation in the Pediatric ICU

Abstract

BACKGROUND:

Timely ventilator liberation can prevent morbidities associated with invasive mechanical ventilation in the pediatric ICU (PICU). There currently exists no standard benchmark for duration of invasive mechanical ventilation in the PICU. This study sought to develop and validate a multi-center prediction model of invasive mechanical ventilation duration to determine a standardized duration of invasive mechanical ventilation ratio.

METHODS:

This was a retrospective cohort study using registry data from 157 institutions in the Virtual Pediatric Systems database. The study population included encounters in the PICU between 2012–2021 involving endotracheal intubation and invasive mechanical ventilation in the first day of PICU admission who received invasive mechanical ventilation for > 24 h. Subjects were stratified into a training cohort (2012–2017) and 2 validation cohorts (2018–2019/2020–2021). Four models to predict the duration of invasive mechanical ventilation were trained using data from the first 24 h, validated, and compared.

RESULTS:

The study included 112,353 unique encounters. All models had observed-to-expected (O/E) ratios close to one but low mean squared error and R² values. The random forest model was the best performing model and achieved an O/E ratio of 1.043 (95% CI 1.030–1.056) and 1.004 (95% CI 0.990–1.019) in the validation cohorts and 1.009 (95% CI 1.004–1.016) in the full cohort. There was a high degree of institutional variation, with single-unit O/E ratios ranging between 0.49–1.91. When stratified by time period, there were observable changes in O/E ratios at the individual PICU level over time.

CONCLUSIONS:

We derived and validated a model to predict the duration of invasive mechanical ventilation that performed well in aggregated predictions at the PICU and the cohort level. This model could be beneficial in quality improvement and institutional benchmarking initiatives for use at the PICU level and for tracking of performance over time.

Introduction

Ventilator liberation has been recognized as a priority in the care of critically ill children with respiratory failure in order to reduce the morbidity associated with intubation and invasive mechanical ventilation in the pediatric ICU (PICU).^1,2 Early extubation minimizes respiratory muscle atrophy, exposure to sedatives, and improves mobilization. However, extubating too early can increase the risk for extubation failure and need for re-intubation.^3-5 Thus, it is imperative for providers in the PICU to determine the optimal timing for extubation to balance the risks of prolonged invasive mechanical ventilation duration with those of extubation failure.

The optimal timing for extubation depends on patient-specific factors, with marked differences between institutions with regard to duration of invasive mechanical ventilation and rates of extubation failure.^6-8 Some of these institutional differences may relate to patient-specific factors, whereas others may relate to variation in institutional practices related to ventilator liberation.^9,10 To reduce this variation, a set of international clinical practice guidelines were recently published describing best practices for ventilator liberation in pediatrics.¹¹ However, as we seek to improve ventilation outcomes through quality improvement interventions based on best practices, it is imperative that we have methods to benchmark duration of invasive mechanical ventilation. This could be accomplished similar to the way severity of illness scores are used to benchmark PICU mortality through standardized mortality ratios (SMRs).¹² There are currently no standard metrics to predict duration of invasive mechanical ventilation in pediatrics. Machine learning models have been developed to predict hospital and ICU stay, mortality, and extubation failure in adults, but these methods have had limited applications in children.^13-16

In this study, we sought to develop and validate a prediction model of duration of invasive mechanical ventilation in the PICU using a large, multi-institutional database and machine learning models. Such a model is needed for duration of invasive mechanical ventilation benchmarking in pediatric ventilator liberation initiatives.

QUICK LOOK.

Current knowledge

Optimal timing of extubation minimizes the risks of extubation failure, sedative side effects, and muscle atrophy. Standardized practices can decrease variability in duration of invasive mechanical ventilation across institutions.

What this paper contributes to our knowledge

Using the Virtual Pediatric Systems database, a machine learning model was able to accurately predict the duration of invasive mechanical ventilation on a population level. This model could be used as a benchmarking metric to facilitate quality improvement initiatives aimed at safely reducing the duration of invasive mechanical ventilation. Further research using more granular variables is needed to accurately predict duration of invasive mechanical ventilation at a patient level.

Methods

This was a retrospective observational study using the Virtual Pediatric Systems (VPS) database (Virtual Pediatric Systems, Los Angeles, California). VPS is a clinical database dedicated to standardized data sharing among PICUs and is used to track outcomes, measure quality, and conduct research.¹⁷ VPS neither endorsed nor restricted our interpretation of these data. This study was reviewed by the Indiana University Institutional Review Board under study title Creation of a Standard for Duration of Mechanical Ventilation in the Pediatric ICU and determined not human subjects research on January 3, 2022 (IRB number 13801).

The study population included subjects from 157 institutions contributing to the VPS database from 0–18 y old who were admitted to the PICU between January 1, 2012–December 31, 2021, with procedure codes for both endotracheal intubation and invasive mechanical ventilation. For inclusion, invasive mechanical ventilation had to be initiated either prior to PICU admission or within the first 24 h of admission. The primary study outcome was the duration of invasive mechanical ventilation and was calculated to the minute using the precise timing of initiation and discontinuation to the day and time. Only the initial intubation and episode of invasive mechanical ventilation were measured and used in the analysis. Extubation failures and reintubations were not included. Subjects with a procedure code for tracheostomy tube present on admission, those without data regarding the duration of invasive mechanical ventilation due to missingness, or who were intubated for < 24 h were excluded. Although some measurements appear to truncate the duration of invasive mechanical ventilation at day 14 for visualization purposes, these values were not truncated in the statistical analysis. Subjects who received a tracheostomy tube during their admission were included in the model, as were subjects who died during the PICU stay. This has potential to impair model performance as these subjects are likely to have higher disease burden and will be predicted to have longer duration of invasive mechanical ventilation that will be truncated by death. However, our goal was to create the model using only information available in the first 24 h of hospital admission. Therefore, these subjects were not excluded from the population, but a sensitivity analysis was done using the best-performing model type to compare performance when not including these subjects.

Subjects were stratified into 3 cohorts. The training cohort included all subjects admitted between 2012–2017. The first validation cohort, termed validation A, included all subjects admitted between 2018–2019. The second validation cohort, termed validation B, included all subjects admitted between 2020–2021. We chose to include 2 validation cohorts in order to assess the possible impact of the COVID-19 pandemic on the outcome of interest. Data for this study included subject demographics (age, sex, race, ethnicity), primary diagnostic category (STAR codes), severity of illness scores including the Pediatric Index of Mortality version 2¹⁸ (PIM-2) and the Pediatric Risk of Mortality version 3¹⁹ (PRISM 3), vital signs and laboratory data collected in the first 24 h, origin of admission, procedures, and clinical outcomes (PICU length of stay, duration of invasive mechanical ventilation, and mortality). International Classification of Disease codes versions 9 and 10 were mapped to previously identified terms to determine the number of complex chronic conditions for each subject using the method developed by Feudtner.²⁰ Only data collected in the first 24 h of admission were used in the model creation. Whereas VPS is quality controlled to prevent missing data elements, not all elements are mandatory. Missing values were imputed with age-based medians.

We tested 4 separate models to predict the duration of invasive mechanical ventilation for each subject encounter: a linear regression model with Least Absolute Shrinkage and Selection Operator (LASSO) regularization, a random forest model, a gradient boosted machine model, and a simple linear regression model using only the PRISM 3 score. The PRISM 3 score–based model was created as a baseline model to compare the more sophisticated models to. The random forest model was tuned sampling 200 trees (nTree = 200) and one random variable used in each tree (mtry = 1). The gradient boosted machine model was tuned using an nTree value of 1,000, a shrinkage value of 0.01, 10-fold cross-validation, and an interaction depth of 4. These models were compared using observed-to-expected (O/E) ratios, mean squared error (MSE), and R². Values are reported with 95% CI. The O/E values were calculated by dividing the average observed duration of invasive mechanical ventilation in the cohort by the average duration of invasive mechanical ventilation predicted by the model for the cohort. A random-effects model was created to compare the optimal model’s performance across institutions. The 9 institutions with the highest and lowest O/E ratios were compared longitudinally across the time frame to evaluate for changes in institutional performance over time. These institutions represented the top and bottom 10% of institutions that contributed consistently across the study period. The duration of invasive mechanical ventilation was log transformed in the random-effects model for normalization and estimating 95% CIs. In comparing the cohorts, categorical comparisons were made using chi-square tests, and continuous variable comparisons were made using Kruskal-Wallis tests. Statistical significance was determined using an alpha level of 0.05. All statistical tests, data manipulation, and figure generation were done using R-4.3 (R Foundation for Statistical Computing, Vienna, Austria) and reported using the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis reporting guidelines.^21,22

Results

Our total study population included 112,353 unique encounters from 157 reporting institutions. The training cohort from 2012–2017 included 69,306 unique encounters; the validation A cohort from 2018–2019 included 23,861 unique encounters, and the validation B cohort from 2020–2021 included 19,186 unique encounters. The subject characteristics are illustrated in Table 1. Overall, the cohorts had comparable distributions of demographics and clinical characteristics.

Table 1.

Subject Characteristics

The distribution of duration of invasive mechanical ventilation for the entire population is displayed in Figure 1. A prominent left shift was noted reflecting a high number of extubations within the first 48 h of PICU admission, which tapered off rapidly over the next several days.

Fig. 1.

Histogram of duration of invasive mechanical ventilation in days with subjects intubated > 14 d (truncated at 14).

Each of the 4 models had similar results, with O/E ratios close to 1 but low MSE and R² values. The model results and comparisons are presented in Table 2. The best-performing model was the random forest model. This model achieved O/E ratios of 1.004 (0.990–1.019) and 1.043 (1.030–1.056), MSE values between 103 (102–105) and 149 (147–151), and R² values of 0.032 (0.032–0.033) and 0.028 (0.028–0.029) in the validation cohorts. Using the full cohort, this model achieved an O/E ratio of 1.009 (1.004–1.016), an MSE of 108 (107–108), and R² value of 0.084 (0.084–0.085). A comprehensive list of model features is included in Supplemental Table 1 (See related supplementary materials at http://www.rcjournal.com), and the feature importance metrics for this model are available in Supplemental Figure 1 (See related supplementary materials at http://www.rcjournal.com). A sensitivity analysis was done after removing the 11,843 mortalities and 104 new tracheostomies from the data set. This model showed improved metrics when applied to the entire cohort: MSE 92 (91–92), R² 0.094 (0.094–0.095), and O/E ratio 1.006 (0.999–1.012). Supplemental Table 2 (See related supplementary materials at http://www.rcjournal.com) displays the range of missing values for each variable type.

Table 2.

Model Validation Measurements

Based on the model performance metrics, we used the random forest model to develop a random-effects model to compare the standardized duration of invasive mechanical ventilation ratio across institutions. Figure 2 displays the results of each individual institution’s O/E ratios for the entire study cohort. The institutional ratios center around 1 but display a large degree of variation, with individual institution’s O/E ratios ranging from 0.49–1.91. The 9 institutions with the highest and lowest O/E ratios were compared longitudinally across the time frame. These results are illustrated in Figure 3. The institutions with the lowest O/E ratios had a mean O/E ratio of 0.74 (0.72–0.75), whereas the institutions with the highest O/E ratios had a mean O/E ratio of 1.33 (1.31–1.35). The institutions with the highest O/E ratios displayed larger variability across the time frame than the institutions with the lowest O/E ratios. Of the 18 institutions included in Figure 3, four had overall improvement in their O/E ratios; 4 had overall worsening in their O/E ratios; 2 had large variation in both directions, and 8 had essentially consistent O/E ratios across the study period.

Fig. 2.

Caterpillar plot of observed-to-expected ratio by institution with 95% CI.

Fig. 3.

Institutional observed-to-expected ratios for the top and bottom deciles over time.

Discussion

This study sought to develop a standardized measure to benchmark duration of invasive mechanical ventilation in the PICU based on a prediction model using data collected in the first 24 h of admission. The data and various modeling methods used in this study may be helpful to compare duration of invasive mechanical ventilation for aggregated institutional data based on the random forest model performance in validation sets with O/E ratios close to 1. However, they are inadequate to predict duration of invasive mechanical ventilation at the subject level given the low R² values, which suggest that the model only explains a small portion of the variability across individual subjects. This information, for example, could be used to identify high-performing institutions with lower duration of invasive mechanical ventilation O/E ratios and assess whether those institutions have higher compliance with specific best practices related to ventilator liberation or track an individual institution’s performance over time after changes in ventilator liberation practices.²³

Decisions regarding the timing of extubation can be difficult for PICU providers. The risks of extubation failure and need for re-intubation must be weighed carefully with the risks of sedation and muscle atrophy from prolonged intubation. New research is being done in this area to provide PICU practitioners with up-to-date best-practice guidelines for limiting the side effects of intubation and mechanical ventilation and optimizing extubation success.¹¹ One of the objectives of this line of research is to decrease variability in the timing of extubation and duration of invasive mechanical ventilation both within institutions and across institutions. This study identified wide variability in the predicted versus observed duration of invasive mechanical ventilation between institutions. There are many potential reasons for this observation. Institutions with high volumes of intubated children are likely to have more experience with mechanical ventilation and may be more comfortable with earlier extubations, whereas institutions with low volumes of intubated children may be more hesitant to extubate early. Some institutions have lower rates of nighttime extubations than others due to concerns for increased risk of extubation failure at night, which may unnecessarily prolong the duration of invasive mechanical ventilation.²⁴ Variations in care provided that are unrelated to changes in patient condition are called unwarranted variations and have been implicated in decreasing the value of health care provided.²⁵ Thus, widely accepted standard metrics are needed to facilitate the process of decreasing variation between institutions.

Whereas the models developed in this study are not ideal for patient-level predictions, they achieved good performance for aggregated, population-level predictions. The O/E ratios, when aggregated across units in the validation sets, were close to one, indicating that the models can accurately predict the average aggregated duration of invasive mechanical ventilation. This could be useful for benchmarking the duration of invasive mechanical ventilation across institutions. The VPS database has previously incorporated the predicted mortality scores (PIM-2 and PRISM 3) to create an SMR for each participating center.¹⁷ This metric allows institutions to compare their SMR to other similar institutions and facilitate quality improvement work to improve the care provided. As mortality in the PICU is generally low, the duration of invasive mechanical ventilation may be a valuable outcome metric for future quality improvement studies in pediatric critical care medicine. The model developed in this study could be used to create a standard metric like the SMR, using multi-center collaboration to identify units that are consistently high performers with lower-than-expected O/E ratios. Our result shows how the O/E ratios varied by unit over the 10-y time period of this study, and several institutions consistently demonstrated O/E ratios below 1 with little variation. These institutions could be assessed to determine practices that may be associated with decreased duration of invasive mechanical ventilation.

Care bundles are emerging entities in pediatric critical care medicine that encompass a protocoled list of procedures, with the goal of decreasing variability in care and following best-practice guidelines, and have been established in the realms of early mobility and delirium prevention, with promising results.^26,27 Reviewing clinical practices of high-performing institutions could be a valuable approach to creating useable care bundles for early extubation and ventilator liberation. As further best-practice guidelines are developed, these initiatives could be incorporated into emerging care bundles and tracked using these models to determine the impact they have on duration of invasive mechanical ventilation at the unit level.

Creating prediction models for duration of invasive mechanical ventilation in pediatrics that perform well at the patient-level remains an important but elusive goal. In this study, we used data from VPS, which would seem ideal in this setting, as they contain a large number of intubated pediatric subjects with quality-controlled data that have been used previously to facilitate benchmarking across institutions.^12,17 However, the results show that models only explained a small portion of the variability across individual subjects and thus were unable to accurately predict duration of invasive mechanical ventilation at the subject level, which would be particularly helpful to enhance clinical decision making at the bedside as well as the use of risk-stratified pathways for ventilator liberation. Whereas we used data that are likely to contribute to the duration of invasive mechanical ventilation in PICU patients such as age, primary diagnosis, complex chronic conditions, and clinical variables associated with severity of illness on admission, our data set lacks other information that is typically used at the bedside to make clinical decisions regarding ventilator liberation, such as ventilator settings, most recent laboratory data, disease trajectories, and operational-level variables (eg, availability of personnel, time of day, concurrent unit-level severity of illness, etc). It is possible that more granular data obtained from electronic health records that contain these types of data may enable better-performing models that can more accurately predict the duration of invasive mechanical ventilation in individual subjects. The models were initially created as prediction models using only data from the first 24 h, and thus mortalities and those who underwent tracheostomies were included. Removing these subjects improved the model performance but likely not enough to overcome the lack of important clinical variables not included in the VPS database for this purpose.

The strengths of this study lie in the use of a large-scale, multi-institutional database to study intubated critically ill children. However, it has several limitations. This was an observational study, and thus many of the variables included are subject to selection bias and only associations can be made. The study cohorts were divided based on year of admission and were found to have multiple differences between them. Only the initial episode of invasive mechanical ventilation was measured in the study and used for analysis, thus missing potentially relevant data on extubation failure and need for re-intubation with further invasive mechanical ventilation.

The variables used in the prediction models were limited to those collected in the VPS database and do not include potentially influential data elements such as medications (eg, sedatives, neuromuscular-blocking agents), endotracheal tube details (eg, size, route), net fluid balance, etc. Finally, large database sets inherently have the potential for data entry error and are subject to issues with inter-rater reliability, although VPS has measures in place to ensure quality data.¹⁷

Conclusions

In this study, we used data obtained in the first 24 h of PICU stay in a large, multi-center data set to predict duration of invasive mechanical ventilation. The model performed poorly in patient-level predictions but had good performance at a population level and could be used for benchmarking purposes across institutions.