Multi-horizon predictive models for guiding extracorporeal resource allocation in critically ill COVID-19 patients

Abstract

Objective

Extracorporeal membrane oxygenation (ECMO) resource allocation tools are currently lacking. We developed machine learning (ML) models for predicting COVID-19 patients at risk of receiving ECMO to guide patient triage and resource allocation.

Material and Methods

We included COVID-19 patients admitted to intensive care units for >24 h from March 2020 to October 2021, divided into training and testing development and testing-only holdout cohorts. We developed ECMO deployment timely prediction model ForecastECMO using Gradient Boosting Tree (GBT), with pre-ECMO prediction horizons from 0 to 48 h, compared to PaO₂/FiO₂ ratio, Sequential Organ Failure Assessment score, PREdiction of Survival on ECMO Therapy score, logistic regression, and 30 pre-selected clinical variables GBT Clinical GBT models, with area under the receiver operator curve (AUROC) and precision recall curve (AUPRC) metrics.

Results

ECMO prevalence was 2.89% and 1.73% in development and holdout cohorts. ForecastECMO had the best performance in both cohorts. At the 18-h prediction horizon, a potentially clinically actionable pre-ECMO window, ForecastECMO, had the highest AUROC (0.94 and 0.95) and AUPRC (0.54 and 0.37) in development and holdout cohorts in identifying ECMO patients without data 18 h prior to ECMO.

Discussion and Conclusions

We developed a multi-horizon model, ForecastECMO, with high performance in identifying patients receiving ECMO at various prediction horizons. This model has potential to be used as early alert tool to guide ECMO resource allocation for COVID-19 patients. Future prospective multicenter validation would provide evidence for generalizability and real-world application of such models to improve patient outcomes.

BACKGROUND AND SIGNIFICANCE

Since the start of the severe acute respiratory syndrome coronavirus-2 SARS-CoV-2 (COVID-19) pandemic, extracorporeal membrane oxygenation (ECMO), a complex and resource-intensive therapy provided in intensive care units (ICU),^1–3 has been used for patients refractory to conventional therapies.^4,5 The use of ECMO for COVID-19 was subject to considerable debate^6–12 owing to the uncertain outcomes^8,13–31 and the ethical dilemma in the deployment of such resource-intensive therapies.^3,15,32–39 Despite these concerns, there was a consistent increase in COVID-19-related ECMO use worldwide.^26,40–43 International recommendations stated that ECMO should be initiated only at experienced centers,^27,35,44 with recent literature showing that new ECMO centers required supervision of regional experts.^45,46 Recommendations were often limited to guiding ECMO initiation using markers of refractory respiratory failure such as PaO₂/FiO₂ (PF) ratio,^44,47 or previously developed ECMO mortality prediction scores, recognizing that such scores were not developed to guide patient-level decision-making⁴⁸ and their inconsistency in predicting outcomes for COVID-19-related ECMO.^11,49

The substantial burden on the global healthcare system during the pandemic⁸ highlighted the lack of tools to guide resource allocation for high-risk resource-intensive therapies like ECMO. Currently, there are no objective tools to identify patients at the highest risk of receiving ECMO support with enough clinically actionable lead time prior to ECMO cannulation, potentially hindering optimal resource allocation within an institution or the ability to safely transport patients to ECMO-experienced centers. The need for such tools is further highlighted by the evolving evidence of the importance of early referral to ECMO-capable centers and the impact of early ECMO deployment on patient outcomes.^50–59

In this study, we had 2 objectives: first, to develop predictive models for early identification of patients at the highest risk of receiving ECMO support, at individual patient levels, and second, to evaluate the viability of these models after multiple time horizons with clinically meaningful lead time to allow patient triage and efficient resource allocation.

MATERIALS AND METHODS

Setting and data sources

Data were extracted from the institutional electronic health record (EHR) data registry of all COVID-19 patients, spanning 15 hospitals of a quaternary healthcare system in a bi-state region covering rural and urban areas with a catchment area spanning 5 states, for 19 months (March 3, 2020–October 1, 2021). This study was approved by the institutional review board of Washington University in St. Louis (IRB# 20201104) with waiver of consent. The Transparent Reporting of Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) guidelines⁶⁰ were used for reporting (Supplementary Material Digital Content 1).

Study population and outcomes

We included all COVID-19 patients confirmed by SARS-CoV-2 viral polymerase chain reaction admitted to an ICU for at least 24 h. ECMO indication was confirmed to be directly related to COVID-19 using chart review (performed by author, ASS). Patients were excluded if they were less than 3 years of age or met our institutional ineligibility criteria for ECMO at the time (age >70 years and body mass index >45 kg/m²). Data were divided into 2 cohorts to validate the model performance: data from the first 10 months (March 2020–December 2021) were grouped into a development cohort for model training and evaluation and data from the subsequent 9 months (January 2021–October 2021) were sequestered as a holdout cohort and only used for model testing. This approach was selected to evaluate the model performance and validity through the evolving waves of the pandemic. The primary prediction outcome was the provision and timing of COVID-19-related ECMO support utilizing the data 0–48 h prior to ECMO initiation.

Variables and data processing

All 23 765 available features in the data registry were reviewed and processed including merging of similar laboratory and flowsheet features, and exclusion of self-reporting static features. After sorting for feature availability, features available in less than 200 patients were excluded to better reflect the general ICU population. A total of 212 variables were utilized in the model development including (1) static variables: demographics, comorbidities (Charlson Comorbidity Index^61,62) and (2) continuous (time series) variables; laboratory, medications, and therapeutic variables (Supplementary Material Digital Content 2). Normal ranges and direction of abnormality were identified for time series variables and the worst values in the data collection period prior to the prediction horizons were concatenated with the static variables as model inputs. For ECMO patients, the worst values from admission to the start of the prediction horizons before ECMO initiation were included, whereas for non-ECMO patients we utilized the worst values from admission to the start of the prediction horizon prior to ICU discharge (Figure 1). Predictions for ECMO use were made at 2-h intervals from ICU admission to the start of the prediction horizon.

Figure 1.

Machine learning prediction model, prediction horizons. We developed a set multi-horizon machine learning prediction models. The models explored the prediction of ECMO use at certain data-free hours in advance termed prediction horizons. The prediction horizons were fixed for all patients and varied from 0 to 48 h. For predictions at each prediction horizon, we included measurements in the time interval from ICU admission to either discharge (for non-ECMO patients) or ECMO initiation (for ECMO patients) minus prediction horizon hours. ECMO: extracorporeal membrane oxygenation; ICU: intensive care unit.

Multi-horizon prediction models

We developed models capable of incorporating both static patient characteristics and time series variables for early prediction of ECMO utilization allowing for appropriate clinical interventions and preparation. As such, we developed a multi-horizon approach^63,64 to predict the risk of ECMO use every 2 h, with pre-determined prediction horizons ranging from 0 to 48 h. No data during the prediction horizons were included as the input. As shown in Figure 1, input variables were collected from admission to the start of prediction horizon prior to either: (1) ECMO initiation for patients supported on ECMO or (2) ICU discharge for non-ECMO patients. This approach was selected to evaluate the performance at various clinically actionable pre-ECMO prediction horizons. At each prediction horizon, patients who had already received the outcome (ECMO or discharge) were excluded from the model building.

Two multi-horizon prediction Gradient Boosting Tree (GBT) models were developed; ForecastECMO utilizing all included features and a model limited a set of 30 a priori selected clinically relevant variables deemed to be most influential in ECMO decision-making (by authors ASS and NS, including variables previously incorporated in ECMO outcome scores^49,65,66) labeled Clinical GBT. Prediction was optimized by developing a collection of GBT models at the different prediction horizons (Supplementary Material Digital Content 2).

Comparison models included (1) logistic regression (LR) models of all considered variables, where variable missingness was handled by mean imputation, (2) PF ratio, the only currently recommended marker for consideration of ECMO initiation,^{44,47,67–69} (3) ICU severity of illness scores, to reflect the complex resource allocation or patient transfer decision-making faced at the bedside. Sequential Organ Failure Assessment (SOFA) score, a widely used ICU severity of illness score was chosen for comparison,^70,71 which has been used to assess the severity of illness in COVID-19 patients,^{45,50,52,68,72} and (4) PREdiction of Survival on ECMO Therapy (PRESET) score, a widely available score to predict survival on ECMO for patients with severe acute respiratory distress syndrome^65,73 that has been evaluated in the context of COVID-19.²⁵

Model performance evaluation

Model evaluation consisted of 2 steps; first, we performed 10 repeat random shuffles of 10-fold cross-validation (100 results sets in total) in the development cohort and then evaluated the model on the holdout dataset. Each cross-validation iteration used a different stratified fold for model evaluation, and the remaining folds were used for model training. Two performance measures were recorded, area under receiver operating characteristic curve (AUROC) and area under precision recall curve (AUPRC) given the expected low positive ECMO rate and class imbalance. To help provide an assessment of clinical feasibility and explainability, we utilized Shapley Additive Explanations (SHAP),⁷⁴ to identify important predictive features.

Models were compared using the averaged AUROCs and AUPRCs from the 10 × 10 iterations. Analysis, model development, testing, and validation were performed in Python v3.6 with Sklearn,⁷⁵ XGBoost,⁷⁶ and Imblearn⁷⁷ packages, available on Github (Supplementary Material Digital Content 2). Data are presented as median and interquartile range (IQR) for quantitative variables and number and percentage for qualitative variables, unless otherwise specified. Mann-Whitney U or Chi square tests were used for group comparisons. A 2-sided P-value equal to or less than .05 was considered statistically significant.

RESULTS

Between March 3, 2020, and October 1, 2021, we identified 61 179 COVID-19-positive patients (33 017 in the development and 28 162 in the holdout cohorts respectively). After applying exclusion and inclusion criteria, the final cohort included 6247 patients, of whom 135 received ECMO support (67, 2.89%, in the development and 68, 1.73%, in the holdout datasets, respectively) (Figure 2 and Supplementary Material Digital Content 3).

Figure 2.

Patient selection CONSORT diagram. CONSORT diagram of patient selection process. Screening the institutional COVID-19 data registry, 61 179 patients were identified between March 3, 2020, and October 1, 2021. Patients not admitted to an ICU and those admitted for less than 24 h were excluded. Patients less than 3 years of age were excluded as their normal vital signs ranges are significantly different than adult patients, who constitute the majority of patients supported on ECMO, and because there have not been any local patients of this age group supported on ECMO for a primary COVID-19 pathology. Finally, following the institutional ECMO exclusion criteria, patients with BMI above 45 kg/m² and those above 70 years of age were excluded. The final included cohort was divided in 2, the development cohort from March 2020 to January 2021 for model development, training and testing, and the holdout cohort from January 2021 to October 2021 for model testing alone. BMI: body mass index; ECMO: extracorporeal membrane oxygenation; ICU: intensive care unit.

Cohort characteristics

The median ECMO duration was 425 h, IQR: [211, 704], with shorter ECMO duration in the development cohort as compared to the holdout cohort (median [IQR]: 369 [198, 554] vs 548 [219, 946] h, respectively, P = .01). The ECMO patients had significantly higher hospital mortality rates than the non-ECMO patients (48% vs 17%, P < .001 and 38% vs 16%, P < .001, in both the development and holdout cohorts, respectively). Regarding severity of illness, compared to the non-ECMO patients, ECMO patients had higher SOFA scores in the development cohort (12 [10, 13] vs 11 [7, 14], P = .001) but lower SOFA scores in the holdout cohort (9 [6, 12] vs 12 [9, 14], P = .001). In addition, patients supported on ECMO had longer durations of mechanical ventilation (10 [2, 22] vs 3 [1, 10] days, P < .001 and 21 [6, 37] vs 4 [1, 15], P < .001 in both the development and holdout cohorts) and hospital length of stay (24 [13, 42] vs 8 [4, 17] days, P < .001, 38 [27, 53] vs 8 [4, 18], P < .001, in the development and holdout cohorts, respectively). ECMO patients in both cohorts had significantly higher use of ICU resources including renal replacement therapy, neuromuscular blockade, pulmonary vasodilatory therapies, and varying doses of vasoactive medication infusions with no significant difference between the ECMO patients in both cohorts. Cohort demographics, characteristics, therapies, and outcomes are shown in Table 1.

Table 1.

Cohort demographics, characteristics, therapies and outcomes

Features	Total cohort, n = 6247	Development cohort		Holdout cohort
		ECMO, n = 67	Non-ECMO, n = 2251	ECMO, n = 68	Non-ECMO, n = 3861
Age (years)	54 [26, 64]	54 [44, 59]*	58 [45, 65]*	55 [43, 61]*	48 [13, 63]*
Male sex, n (%)	3550 (57)	46 (69)	1255 (56)*	45 (66)	2204 (57)
Caucasian, n (%)	3965 (64)	38 (57)*	1348 (60)*	42 (62)	2537 (66)
Height (cm)	168 [157, 178]	170 [163, 180]	170 [163, 178]	175 [168, 183]*	165 [145, 176]*
Weight (kg)	76 [56, 95]	85 [79, 105]*	84 [67, 100]*	88 [77, 109]*	71 [44, 91]*
BMI (kg/m2)	26 [20, 32]	30 [26, 35]*	28 [24, 34]*	29 [26, 34]*	25 [19, 31]*
Tobacco use, n (%)	1207 (19)	5 (7)*	500 (22)*	5 (7)*	697 (18)*
SOFAa	9 [6, 13]	12 [10, 13]*,**	11 [7, 14]*	9 [6, 12]*,**	12 [9, 14]*
Lowest PF ratioa	112 [66, 204]	56 [48, 69]*	107 [65, 201]*	55 [50, 63]*	126 [71, 218]*
Hospital mortality, n (%)	1079 (17)	32 (48)*	391 (17)*	26 (38)*	630 (16)*
CCI	4 [1, 7]	2 [1, 4.5]*	4 [2, 8]*	3 [1, 4]	3 [1, 7]
Chronic pulmonary disease, n (%)	2305 (37)	18 (27)*	899 (40)*	13 (19)*	1375 (36)*
Diabetes, n (%)	2994 (48)	36 (54)	1388 (62)	30 (44)	1540 (40)
Malignancy, n (%)	1537 (25)	6 (9)*	594 (26)*	10 (15)	927 (24)
Renal disease, n (%)	1369 (22)	13 (19)	568 (25)	9 (13)	779 (20)
Hospital length of stay (days)	8 [4, 18]	24 [13, 42]*,**	8 [4, 17]*	38 [27, 53]*,**	8 [4, 18]*
Mechanical ventilation (days)a	2 [0, 7]	10 [2, 22]*,**	3 [1, 10]*	21 [6, 37]*,**	4 [1, 15]*
CRRT, n (%)	386 (6)	16 (24)*	145 (6)*	14 (21)*	211 (5)*
Remdesivir, n (%)	764 (12)	27 (40)*,**	372 (17)*	26 (38)*,**	339 (9)*
Neuromuscular blockade, n (%)	631 (10)	45 (67)*	188 (8)*	56 (83)*	342 (9)*
Nitric oxide/Iloprost, n (%)	511 (8)	41 (61)*	196 (9)*	45 (67)*	229 (6)*
Dopa. <5 μg/kg/min, Dobu., milrinone or levosimendan, n (%)a	592 (10)	15 (22)*	145 (6)*	11 (16)*	421 (11)*
Dopa. 5–15 μg/kg/min, Epi/NorEpi <0.1 μg/kg/min, Vaso, Phenyl, n (%)a	3219 (52)	67 (100)*	1138 (51)*	68 (100)*	1946 (49)*
Dopa >15 μg/kg/min, Epi/NorEpi >0.1 μg/kg/min, n (%)a	2154 (35)	63 (94)*	726 (32)*	65 (96)*	1300 (33)*

Note: Data are presented as median and interquartile range unless otherwise specified.

Abbreviations: BMI: body mass index; CCI: Charlson Comorbidity Index; CRRT: continuous renal replacement therapy; Dobu: dobutamine; Dopa: dopamine; ECMO: extracorporeal membrane oxygenation; Epi: epinephrine; NorEpi: norepinephrine; NPPV: noninvasive positive pressure ventilation; Phenyl: phenylephrine; SOFA: Sequential Organ Failure Assessment; Vaso: vasopressin.

^aPrior to ECMO initiation for ECMO patients or prior to discharge for non-ECMO patients.

^* P-value <.05 by Mann-Whitney U or Chi square tests between ECMO and non-ECMO groups in each cohort.

^** P-value <.05 by Mann-Whitney U or Chi square tests between ECMO patients in both cohorts.

Model performance

In the development cohort, the ForecastECMO model outperformed the PF ratio, SOFA score, PRESET score, and Clinical GBT models, at all prediction horizons, in both performance metrics, AUROC and AUPRC (Figure 3A and B, respectively). Compared to the ForecastECMO model, the LR model of all included variables demonstrated comparable AUROC across the different thresholds (Figure 3A) but had consistently lower AUPRC performance (Figure 3B). Similarly, in the holdout cohort, the ForecastECMO model outperformed all compared models at all the studied prediction horizons in both AUROC and AUPRC (Figure 3C and D). Of interest, the LR model did not perform as well, with lower AUROC and AUPRC at all studied prediction horizons in comparison to the ForecastECMO model.

Figure 3.

Prediction models performance. (A) Development cohort: area under ROC at prediction horizons of 0–48 h prior to ECMO initiation or hospital discharge for ForecastECMO, Clinical GBT, PF ratio LR, LR model of all included variables, SOFA score, and PRESET score LR models. The ForecastECMO model outperformed the Clinical GBT, PF, SOFA, and PRESET score models at all prediction horizons. The linear LR model performed similar to ForecastECMO at many of the studied prediction horizons. During the 12–24-h prediction horizons (light dotted line), ForecastECMO had the highest performance at the 18-h prediction horizons (dark dotted line) compared to the remaining models. (B) Development cohort: area under PRC curve at prediction horizons of 0 to 48 hours prior to ECMO initiation or hospital discharge for ForecastECMO, Clinical GBT, PF ratio LR, LR model of all included variables, SOFA score, and PRESET score LR models. The ForecastECMO model outperformed all the compared models at all the studied prediction horizons with high performance at the 18-h prediction horizon (dark dotted lines) in the 12-24-h prediction horizon range (light dotted lines). (C) Holdout cohort: area under ROC at prediction horizons of 0–48 h prior to ECMO initiation or hospital discharge for ForecastECMO, Clinical GBT, PF ratio LR, LR model of all included variables, SOFA score, and PRESET score LR models. The ForecastECMO model outperformed all the comparable models including the linear LR at all prediction horizons. Of note, compared to the development cohort, the PRESET score LR model outperformed the SOFA score LR model at all the studied prediction horizons. The ForecastECMO maximal performance in the 12–24-h prediction horizons range (light dotted lines) was at 18 h (dark dotted line). (D) Holdout cohort: area under PRC curve at prediction horizons of 0–48 h prior to ECMO initiation or hospital discharge for ForecastECMO, Clinical GBT, PF ratio LR, LR model of all included variables, SOFA score, and PRESET score LR models. The ForecastECMO model outperformed all the compared models at all the studied prediction horizons. In the 12–24-h prediction horizons (light dotted lines), ForecastECMO continued to have high performance at 18 h (dark dotted line) compared to the other models. ECMO: extracorporeal membrane oxygenation; GBT: Gradient Boosting Tree; ICU: intensive care unit; LR: logistic regression; PF: PaO₂/FiO₂; PRC: precision recall curve; PRESET: PREdiction of Survival on ECMO Therapy; ROC: receiver operator curve; SOFA: Sequential Organ Failure Assessment.

Multi-horizon prediction

To identify clinically actionable time frames for patient triage or resource allocation, we focused on the prediction horizons of high performance between 12 and 24 h prior to ECMO initiation. We identified the 18-h prediction horizon to be of high performance for the ForecastECMO model in both the development and holdout cohorts. Compared to PF, SOFA, PRESET scores, LR, and Clinical GBT models ForecastECMO had an AUROC of 0.94 [CI: 0.93–0.95] versus 0.52, 0.56, 0.66, 0.92 [CI: 0.91–0.93], and 0.82 [CI: 0.8–0.83], respectively, in the development cohort (Figure 4A) and 0.952 [CI: 0.951–0.953] versus 0.564, 0.59, 0.726, 0.908 [CI: 0.908–0.908], and 0.889 [CI: 0.887–0.89], respectively, in the holdout cohort (Figure 4C). When compared to PF, SOFA, PRESET scores, LR, and Clinical GBT models, ForecastECMO had an AUPRC of 0.546 [CI: 0.51–0.582] versus 0.032, 0.032, 0.077, 0.474 [CI: 0.435–0.512], and 0.248 [CI: 0.218–0.277], respectively, in the development cohort (Figure 4B) and 0.376 [CI: 0.37–0.382] versus 0.011, 0.015, 0.039, 0.262 [CI: 0.261–0.262], and 0.345 [CI: 0.341–0.382], respectively, in the holdout cohort (Figure 4D). We further assessed the performance of both GBT models after calibration.⁷⁸ Again, the ForecastECMO model outperformed all compared models including the Clinical GBT model in both studied cohorts (Supplementary Material Digital Content 4).

Figure 4.

Models’ performance at select prediction horizons during 24 prior to ECMO initiation. (A) Development cohort: area under ROC for ForecastECMO, PF ratio LR, SOFA score, PRESET score LR, LR of all included variables, and Clinical GBT models at the 18-h prediction horizon. ForecastECMO outperformed all other models: PF ratio LR, SOFA score, PRESET score LR, LR of all included variables, and Clinical GBT models with AUROC of 0.94 [CI: 0.93–0.95] versus 0.52, 0.56, 0.66, 0.92 [CI: 0.91–0.93], and 0.82 [CI: 0.8–0.83], respectively. (B) Development cohort: area under PRC for ForecastECMO, PF ratio LR, SOFA score, PRESET score LR, LR of all included variables, and Clinical GBT models at the 18-h prediction horizon. ForecastECMO outperformed all other models: PF ratio LR, SOFA score, PRESET score LR, LR of all included variables, and Clinical GBT models with AUPRC of 0.546 [CI: 0.51–0.582] versus 0.032, 0.032, 0.077, 0.474 [CI: 0.435–0.512] and 0.248 [CI: 0.218–0.277], respectively. (C) Holdout cohort: area under ROC for ForecastECMO, PF ratio LR, SOFA score, PRESET score LR, LR of all included variables, and Clinical GBT models at the 18-h prediction horizon. ForecastECMO outperformed all other models: PF ratio LR, SOFA score, PRESET score LR, LR of all included variables, and Clinical GBT models with AUROC of 0.952 [CI: 0.951–0.953] versus 0.564, 0.59, 0.726, 0.908 [CI: 0.908–0.908], and 0.889 [CI: 0.887–0.89], respectively. (D) Holdout cohort: area under PRC for ForecastECMO, PF ratio LR, SOFA score, PRESET score LR, LR of all included variables, and Clinical GBT models at the 18-h prediction horizon. ForecastECMO outperformed all other models: PF ratio LR, SOFA score, PRESET score LR, LR of all included variables, and Clinical GBT models AUPRC of 0.376 [CI: 0.37–0.382] versus 0.011, 0.015, 0.039, 0.262 [CI: 0.261–0.262], and 0.345 [CI: 0.341–0.382], respectively. AUPRC: area under precision recall curve; AUROC: area under receiver operator curve; ECMO: extracorporeal membrane oxygenation; GBT: Gradient Boosting Tree; LR: logistic regression; PF: PaO₂/FiO₂; PRESET: PREdiction of Survival on ECMO Therapy; ROC: receiver operator curve; SOFA: Sequential Organ Failure Assessment.

Feature importance

On SHAP variable analysis for the overall cohort, the most salient features were the need for oxygen therapy, use of neuromuscular blockade, hemoglobin level, patient position,⁵² use of positive end-expiratory pressure (a marker of severity of hypoxemia),⁵² and D-dimer (commonly used throughout the pandemic as a marker of COVID-19 disease severity⁷⁹) all features consistent with progressive COVID-19-related respiratory failure (Figure 5A). Similarly, SHAP analysis for the ECMO patients showed oxygen therapy, patient position, hemoglobin, use of continuous neuromuscular blockade infusions, use of beta-agonist agents (commonly used for obstructive airway disease), and exhaled tidal volumes on mechanical ventilatory support (another marker of worsening pulmonary compliance and refractory respiratory failure) as the top features to contribute to the model prediction of the risk of ECMO utilization (Figure 5B) (Supplementary Material Digital Content 5). Similar variables were identified as the most important in model performance in predicting the risk of ECMO use in both the development and holdout cohorts on individual patient levels (Figure 6). For the false-positive patients (identified by the model to have a high likelihood of receiving ECMO but did not), the most salient features included oxygen therapy, crystalloid fluid resuscitation and hemoglobin levels (Supplementary Material Digital Content 6).

Figure 5.

ForecastECMO salient features by SHAP approach. (A) Analysis of the 20 most relevant features in the ForecastECMO model performance, of the 212 included features, in the total cohort using SHAP approach, with absolute SHAP values on y-axis. The features included supplemental oxygen delivery, use of neuromuscular blockade infusion, hemoglobin measurements, patient position, level of end-expiratory positive pressure ventilation level (CPAP/PEEP), D-dimer measurement, blood glucose level measurement, PLR, antibiotic use, and measurement of minute ventilation as the highest 10 contributing variables. (B) Analysis of the 20 most relevant features in the ForecastECMO model performance, of the 212 included features, in the ECMO cohort using SHAP approach, with SHAP values on y-axis. The 10 highest contributing features included supplemental oxygen delivery, patient position, hemoglobin measurements, use of neuromuscular blockade infusion, use of B agonist agents (usually used for patients with obstructive pulmonary disease), measurement of exhaled tidal volumes on invasive mechanical ventilation, measurement of ETCO2 (a measurement obtained on patients requiring invasive mechanical ventilation), measurement of plateau pressure on invasive mechanical ventilation, measurement of minute ventilation, and use of antibiotics. CPAP: continuous positive airway pressure; BP: blood pressure; ECMO: extracorporeal membrane oxygenation; ETCO2: end tidal carbon dioxide; MAP: mean arterial blood pressure; O₂: oxygen; PEEP: positive end-expiratory pressure; PLR: platelet-to-lymphocyte ratio; Resp: respiratory rate; SHAP: SHapley Additive exPlanations; TV: tidal volume.

Figure 6.

ForecastECMO prediction performance by force plots. (A) Force plot for prediction of ECMO utilization in a patient who did not receive ECMO in the development cohort. Features contributing most to the negative prediction by SHAP analysis included oxygen therapy, measurement of blood sodium level, patient age, and measurement of blood hematocrit concentration. (B) Force plot for the prediction of ECMO utilization in a patient who received ECMO support in the development cohort. Features contributing most to the positive prediction by SHAP analysis included the use of neuromuscular blockade infusion, supplemental oxygen therapy, plateau pressure measurement on invasive mechanical ventilation, and measurement of hemoglobin level. (C) Force plot for prediction of ECMO utilization in a patient who did not receive ECMO in the holdout cohort. Features contributing most to the negative prediction included the patient’s height, measurement of hemoglobin level, measurement of hematocrit concentration, and use of systemic corticosteroids. ECMO: extracorporeal membrane oxygenation; CPAP: continuous positive airway pressure; GCS: Glasgow coma scale; O₂: oxygen; PEEP: positive end-expiratory pressure; PLR: platelet-to-lymphocyte ratio; SHAP: SHapley Additive exPlanations.

DISCUSSION

Using data from 15 hospitals with a 5-state-wide catchment area and over 6000 patients admitted to ICUs, we developed a prediction model, ForecastECMO, with prediction horizons of 0–48 h ahead of ECMO deployment, that had high performance in both assessed metrics, AUROC and AUPRC, in predicting the risk of ECMO use. We showed the novel use of a multi-horizon prediction model approach to a clinical problem that requires early prediction for its translational use. With a clinically actionable time window in excess of 12 h, this model has the potential to alert clinicians to the possible need to escalate support in this select patient cohort, at an individual patient level, with enough lead time to allow for the necessary resource allocations or patient transport to either higher resourced centers or centers with more established ECMO experience, before the degree of critical illness precludes safe transport.

To date, ECMO-related scoring systems have focused on mortality prediction for patients already supported on ECMO.^48,73,80,81 With the evolution of the pandemic and the concerns regarding the resource-intense nature of ECMO and initially unclear outcomes, several studies attempted to validate such mortality prediction scores to the COVID-19 population with varied results.^25,49,82 Although these scoring systems provide important prognostic information, the performance of such scores in predicting patients at the highest risk of receiving ECMO with enough lead time has not been evaluated. Currently available scores also do not consider the evolving trajectories of this complex patient population. The approach presented in our study incorporates the patient data from ICU admission to ECMO initiation, reflecting the data-rich nature of the ICU environment and providing predictions on a patient level. The higher performance of the ForecastECMO model compared to Clinical GBT highlights the contribution of the added features in identifying the patient cohort at highest risk of receiving ECMO support, despite lower feature availability as has been previously reported in other machine learning (ML) models to predict patient outcomes.⁸³ Our model has the potential to fill a previously well-recognized gap in the available clinical tools, by providing clinicians at the bedside with early alert tool identifying patients at risk of receiving ECMO support with enough lead time to adjust patient triage or the necessary resource allocations.

The aim of this study was not to develop a model to replace clinical decision-making as to which patients should receive ECMO support, as these decisions are highly individualized and patient specific. Developing predictive models to identify the patients who benefit most from ECMO, truly assessing the impact of ECMO support on patient outcomes, requires evaluating both ECMO and propensity-matched non-ECMO cohorts. Rather, the aim of this work was the novel application of ML approaches to a real-life clinical problem: to identify which patients had the highest probability of receiving ECMO support based off clinician selection, while providing a potentially clinically relevant lead time prior to ECMO initiation. Such approaches aim at augmenting and not substituting clinical decision-making as outlined in the fundamental theories of biomedical informatics.⁸⁴

The ability to identify such patients, well in advance of the actual timing of ECMO initiation, provides clinicians for the first time with an objective tool to guide patient triage. Such tools add to those used at the bedside in deciding which patients would benefit most from transfer to more ECMO-capable centers before their trajectories progress and they become too sick to tolerate safe transfer. In addition, even in ECMO-experienced centers in times of stress and resource limitation, an early alert tool capable of identifying such patients provides the necessary data to guide allocation of scarce ECMO resources.

Studies throughout the pandemic have shown an evolution in the characteristics of patients supported on ECMO in the different pandemic waves.^42,85–87 These findings emphasize the need for early warning tools capable of identifying the patients at the highest risk of receiving this level of support regardless of the pandemic phase. To assess this aspect, we separated our study population into 2 cohorts with a comparable number of ECMO patients, development, and holdout cohorts. Consistent with previous reports, the ECMO patients in both cohorts had different characteristics. Despite those differences, our model showed similar performance in the holdout cohort, highlighting its internal validity. While further research and multicenter validation are needed, the high performance in the holdout cohort shows the transferability of the multi-horizon prediction approach to develop early warning tools in different waves in the pandemic and potentially to similar clinical problems. While the global healthcare restraints associated with the COVID-19 pandemic highlighted the need for tools to aid resource allocation for resource-intensive therapies, the need for such tools stretches beyond the pandemic. Our approach has the potential to be utilized in patient triage and resource allocation for other high-risk resource-intensive therapies.

Resource allocation and patient triage are often based on the severity of illness. As such, it was important to compare ForecastECMO to models based on general severity of illness (SOFA score), models based on the severity of respiratory failure (PF ratio) and models based on the commonly available ECMO mortality scoring systems (PRESET score). None of these metrics had high predictive performance in either cohort. This is particularly significant for PF ratio, a marker of severity of respiratory failure, which while it is recommended for immediate ECMO decision-making³⁵ has poor performance as an early prediction tool for patient triage.

When developing early alert models to identify patients at risk of receiving high-risk therapies such as ECMO, it is important to develop tools capable of identifying this “unseen” population with high precision and recall, considering the class imbalance, as less than 3% of patients in our cohort would eventually receive ECMO support.^88–90 The application of ML models to such real-world clinical problems, provides a level of precision approach that is currently missing from the available clinical tools. Although there is a potential risk of unnecessary patient transfers (false alarm) if they do not eventually receive ECMO support, that risk may be outweighed by the potential benefit to both the patient and the healthcare system with early risk identification. A recent analysis of a large COVID-19 ECMO hub demonstrated that even with expertise and robust mobile ECMO and transport programs, 12.5% of patients deemed to not yet need ECMO at the time of evaluation, progressed to secondarily receive ECMO. The authors also concluded that earlier transfer may be beneficial.⁹¹

The ForecastECMO model’s performance compared to a more limited clinical variable-based model, Clinical GBT, lends value to its potential as a useful early alert tool to aid in resource allocation or patient triage even in settings lacking the necessary ECMO expertise. While generalizability to such settings cannot be assumed without future external validation, ForecastECMO’s superior performance compared to the clinical variable-based comparator provides a promising initial signal to this approach.

To aid in clinical explainability, we utilized a SHAP variable importance analysis to determine variables’ contribution to the model performance. As expected, the most important variables included the method of and degree of respiratory support as expressed by supplemental oxygen delivery, the degree of mechanical ventilatory support (positive end-expiratory pressure, minute ventilation, and exhaled tidal volume), patient position given the recognition of the impact of prone positioning that evolved during the pandemic and the use of neuromuscular blockade for refractory respiratory failure. The magnitude of importance of these variables was also displayed on the model performance at individual patient levels for those predicted to receive or not receive ECMO support, lending credence to the model’s clinical relevance as it selected variables known to be clinically relevant without human input.

Our data showed a higher incidence of ECMO initiation within the first 6 h after admission to the ICU, highlighting the fact that many patients presented in physiological extremis, posing significant stress on the healthcare system with need for timely mobilization of ECMO resources. This further emphasizes the clinical potential for tools capable of identifying patients early in their course to allow for safe transport before physiological extremis, a fact emphasized by the growing evidence that early provision of ECMO support in this patient population, before worsening metabolic derangements, can be associated with improved outcomes.⁶⁸

Future directions include exploring the clinical explainability and plausibility of the model, how model results could be delivered in a clinically applicable way, validating this tool on multicenter data, with real-time clinical deployment and further refinement and development of similar tools for other therapies. Incorporation of these models into the EHR would provide an opportunity to prospectively evaluate their performance in the real clinical environment. In addition, while there have been advances in managing COVID-19-associated disease, the need for such tools extends beyond the pandemic and has the potential to be adapted for other high-risk therapies.

Limitations

It is important to view our results considering several limitations. First, our model was created on only retrospective data from 1 large healthcare system across different phases of the pandemic. During this time, there likely was heterogeneity in treatment modalities and outcomes. Second, this is a single-center study, despite spanning several hospitals across a large healthcare system. The current model relies on a large number of variables. Multicenter studies are necessary to further fine tune the model and necessary features and to test the generalizability of the model. Although we addressed this risk of over-fitting with subsampling by repeat cross-fold validation and random shuffling and validating on a holdout cohort, there continues to be a need for future external validation before any generalizable conclusions can be made. Third, to guarantee the availability of sufficient data for model building, we excluded patients admitted to an ICU for less than 24 h, thus adding the inability to assess the model performance on that patient cohort. Future prospective studies will be necessary to further test the model performance regardless of the patients’ admission duration. In addition, the proposed models were developed using a case-control retrospective approach, where the patient outcome was already known, and the model predicted the likelihood of treatment assignment at various time horizons. How the model performs in a prospective approach where the patients’ outcome is not known, will need to be further assessed in future studies. The goal of this work was to identify patients who ultimately receive ECMO support with enough lead time to allow the necessary resource allocation, thus cannot be used to identify the patient cohort who benefits most from ECMO support. Future work is thus needed for treatment effects predictive modeling to identify the cohorts who benefit most from ECMO support and potentially impact patient outcomes.

CONCLUSION

In this study, we developed a predictive model for ECMO use in COVID-19 ICU patients, with prediction horizons prior to ECMO initiation. The model has high prediction performance at potentially clinically actionable windows prior to cannulation and across prediction horizons. This tool has the potential to impact resource allocation decisions and impact the stressors posed on the healthcare system during the time of resource limitation. This pilot study opens the potential for the future development of similar tools for other high-risk, resource-intensive therapies.

FUNDING

This work was supported by the Big Ideas 2020 COVID Grant through the Healthcare Innovations Lab at the BJC Healthcare and Washington University in St. Louis School of Medicine. ASS has received research support from the Children’s Discovery Institute Faculty Development Award at Washington University in St. Louis. For the remaining authors, none were declared.

AUTHOR CONTRIBUTIONS

BX and HY performed the data cleaning, analysis, model building, and validation; ASS and NS provided conceptualization, clinical interpretation, and manuscript writing; TK, CL, and PROP provided guidance on data analysis, model building, and validation. All authors participated in manuscript revision and editing.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

CONFLICT OF INTEREST STATEMENT

None declared.

DATA AVAILABILITY

The data underlying this article will be shared on reasonable request to the corresponding author.