To the Editor:
During the coronavirus disease (COVID-19) pandemic, the number of patients eligible to receive extracorporeal membrane oxygenation (ECMO) has exceeded the availability of this resource-intensive therapy (1). To maximize benefit, many centers have incorporated survival predictions into allocation processes (2). However, existing tools to predict mortality among patients receiving ECMO were generated from patient cohorts with non–COVID-19–related acute respiratory distress syndrome (3, 4) and may perform poorly when applied to patient cohorts with COVID-19 (5). Whether a tool created from a cohort of patients with COVID-19 would demonstrate improved performance is unknown. We derived and validated a simple bedside tool for predicting hospital mortality for patients with COVID-19 being considered for ECMO and compared its performance to the performance of the Respiratory ECMO Survival Prediction (RESP) score, a commonly used prognostic model for survival during ECMO developed before the COVID-19 pandemic (3).
Methods
Study population
This analysis, approved by the Extracorporeal Lifesupport Organization (ELSO) Registry Scientific Oversight Committee, used deidentified, individual patient-level data extracted from the International ELSO Registry (6) at two time points: April 2021 (used to derive and internally validate the model) and October 2021 (obtained after the model was generated and used for external validation). Registry procedures have been previously described (7). Demographics, clinical data captured in the 6 hours before ECMO initiation, year, patient location (United States vs. other), and vital status at hospital discharge were extracted for adult patients who received venovenous ECMO with a confirmed diagnosis of COVID-19. Patients who received a mode other than venovenous ECMO and patients transferred to a different hospital on ECMO for whom outcomes were unavailable were excluded. The reporting of this work adheres to the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis criteria (8).
Model derivation and internal validation
Models were derived and internally validated based on data available in the ELSO registry at the time of initial data pull in April 2021. The primary outcome was in-hospital mortality. The candidate variables listed in Table 1 were selected a priori based on prior research (3, 7, 9) and investigator experience. We split the data into a training dataset (80% of observations) and a testing dataset (20% of observations). Then, we used five different approaches to identify the variables of most importance in prognostication: logistic regression, least absolute shrinkage and selection operator (LASSO), random forest, XGBoost, and classification and regression tree (CART). The CART was developed based on the information gain and was pruned at the smallest cross-validation error from a 20-fold cross-validation. Model performance was assessed using the area under the receiver operating characteristic curve (AUC) in the internal validation dataset. The 95% confidence interval (CI) of the AUC was computed with 2,000 stratified bootstrap replicates.
Table 1.
Candidate Variables, Including Demographics and Patient Characteristics before Cannulation
| Derivation Cohort |
Validation Cohort |
|||||
|---|---|---|---|---|---|---|
| Whole Cohort (N = 4,553) | Survivors (n = 2,166) | Nonsurvivors (n = 2,387) | Whole Cohort (n = 2,582) | Survivors (n = 1,231) | Nonsurvivors (n = 1,351) | |
| Characteristics | ||||||
| Year ECMO initiated | ||||||
| 2020 | 3,961 (87.0) | 1,917 (88.5) | 2,044 (85.6) | 490 (19.0) | 235 (19.1) | 255 (18.9) |
| 2021 | 592 (13.0) | 249 (11.5) | 343 (14.4) | 2,092 (81.0) | 996 (80.9) | 1,096 (81.1) |
| Location of cannulation | ||||||
| United States | 2,936 (64.5) | 1,349 (62.3) | 1,587 (66.5) | 1,681 (65.1) | 764 (62.1) | 917 (67.9) |
| Outside the United States | 1,617 (35.5) | 817 (37.7) | 800 (33.5) | 901 (34.9) | 467 (37.9) | 434 (32.1) |
| Age, yr | 50.0 (41.5–57.8) | 46.8 (38.1–54.4) | 53.2 (45.1–59.8) | 47.8 (38.3–55.5) | 44.5 (35.8–52.4) | 50.0 (41.7–57.6) |
| Sex | ||||||
| Female | 1,224 (26.9) | 649 (30.0) | 575 (24.1) | 837 (32.4) | 445 (36.1) | 392 (29.0) |
| Male | 3,328 (73.1) | 1516 (70.0) | 1,812 (75.9) | 1,745 (67.6) | 786 (63.9) | 959 (71.0) |
| Body mass index, kg/m2 | 32.4 (15.0–73.7) | 33.0 (28.0–38.5) | 32.0 (27.8–37.1) | 32.9 (8.4–72.3) | 33.4 (8.4–72.3) | 32.5 (15.2–69.8) |
| Race and ethnicity | ||||||
| Black | 528 (11.6) | 273 (12.6) | 255 (10.7) | 257 (10.0) | 119 (9.7) | 138 (10.2) |
| White (non-Hispanic) | 1,734 (38.1) | 794 (36.7) | 940 (39.4) | 1,173 (45.4) | 544 (44.2) | 629 (46.6) |
| Asian | 471 (10.3) | 239 (11.0) | 232 (9.7) | 199 (7.7) | 95 (7.7) | 104 (7.7) |
| Middle Eastern or North African | 178 (3.9) | 88 (4.1) | 90 (3.8) | 74 (2.9) | 37 (3.0) | 37 (2.7) |
| Other | 78 (1.7) | 33 (1.5) | 45 (1.9) | 53 (2.1) | 20 (1.6) | 33 (2.4) |
| Unknown | 142 (3.1) | 72 (3.3) | 70 (2.9) | 125 (4.8) | 64 (5.2) | 61 (4.5) |
| Multiple | 197 (4.3) | 98 (4.5) | 99 (4.1) | 98 (3.8) | 53 (4.3) | 45 (3.3) |
| Hispanic | 1,127 (24.8) | 529 (24.4) | 598 (25.1) | 576 (22.3) | 284 (23.1) | 292 (21.6) |
| Native American | 85 (1.9) | 32 (1.5) | 53 (2.2) | 22 (0.9) | 13 (1.1) | 9 (0.7) |
| Native Pacific Islander | 13 (0.3) | 8 (0.4) | 5 (0.2) | 5 (0.2) | 2 (0.2) | 3 (0.2) |
| Comorbidities | ||||||
| Diabetes | 1,364 (30.0) | 603 (27.8) | 761 (31.9) | 625 (24.2) | 274 (22.3) | 351 (26.0) |
| Hypertension | 1,632 (35.8) | 668 (30.8) | 964 (40.4) | 899 (34.8) | 374 (30.4) | 525 (38.9) |
| Preexisting cardiac disease | 164 (3.6) | 45 (2.1) | 119 (5.0) | 64 (2.5) | 12 (1.0) | 52 (3.8) |
| Preexisting respiratory disease | 180 (4.0) | 65 (3.0) | 115 (4.8) | 98 (3.8) | 31 (2.5) | 67 (5.0) |
| Preexisting renal insufficiency | 132 (2.9) | 52 (2.4) | 80 (3.4) | 60 (2.4) | 22 (1.8) | 38 (2.8) |
| Central nervous system dysfunction | 435 (9.6) | 187 (8.6) | 248 (10.4) | 264 (10.2) | 111 (9.0) | 153 (11.3) |
| Immunocompromised | 180 (4.0) | 76 (3.5) | 104 (4.4) | 75 (2.9) | 32 (2.6) | 43 (3.2) |
| Cancer | 82 (1.8) | 25 (1.2) | 57 (2.4) | 27 (1.0) | 10 (0.8) | 17 (1.3) |
| Pneumothorax | 581 (12.8) | 189 (8.7) | 392 (16.4) | 406 (15.7) | 146 (11.9) | 260 (19.2) |
| Acute associated nonpulmonary infections | 310 (6.8) | 139 (6.4) | 171 (7.2) | 203 (7.9) | 83 (6.7) | 120 (6.9) |
| Cardiac arrest | 150 (3.3) | 48 (2.2) | 102 (4.3) | 79 (3.1) | 26 (2.1) | 53 (3.9) |
| Acute renal failure | 1,279 (28.1) | 452 (20.9) | 827 (34.6) | 611 (23.7) | 183 (14.9) | 428 (31.7) |
| Renal replacement therapy | 299 (6.6) | 121 (5.6) | 178 (7.5) | 95 (3.7) | 36 (2.9) | 59 (4.4) |
| Presence of vasopressors | 2,641 (58.0) | 1,154 (53.3) | 1,487 (62.3) | 1,377 (53.3) | 611 (49.6) | 766 (56.7) |
| Presence of bicarbonate infusion | 319 (7.0) | 123 (5.7) | 196 (8.2) | 179 (6.9) | 59 (4.8) | 120 (8.9) |
| Intubation days | 3.3 (1.0–6.2) | 3.1 (1.0–5.9 | 3.6 (1.2–6.5) | 3.0 (0.9–5.9) | 2.6 (0.7–5.6) | 3.2 (1.0–6.2) |
| Respiratory rate | 26.0 (22.0–30.0) | 26.0 (20.0–30.0) | 27.0 (22.0–31.0) | 26.0 (22.0–30.0) | 26.0 (22.0–30.0) | 28.0 (23.8–30.0) |
| Positive end-expiratory pressure, cm H2O | 14.0 (10.0–16.0) | 14.0 (12.0–16.0) | 14.0 (10.0–16.0) | 14.0 (10.0–16.0) | 14.0 (10.0–16.0) | 14.0 (10.0–16.0) |
| Peak inspiratory pressure, cm H2O | 34.0 (30.0–38.0) | 33.0 (30.0–37.0) | 35.0 (30.0–39.0) | 33.0 (30.0–38.0) | 33.0 (29.0–37.0) | 34.0 (30.0–38.0) |
| pH | 7.29 (7.21–7.37) | 7.30 (7.23–7.38) | 7.28 (7.19–7.36) | 7.30 (7.22–7.37) | 7.31 (7.23–7.38) | 7.29 (7.21–7.36) |
| PaCO2, mm Hg | 61.0 (51.0–75.0) | 59.0 (50.0–72.0) | 63.0 (52.0–78.0) | 61.0 (50.0–75.0) | 59.0 (49.0–72.0) | 64.0 (52.8–78.0) |
| PaO2:FiO2, mm Hg | 70.0 (58.0–90.0) | 72.0 (59.0–92.3) | 69.0 (56.7–88.0) | 70.0 (56.4–87.8) | 71.0 (58.0–91.0) | 68.0 (55.6–85.0) |
| Bicarbonate, mEq/L | 28.0 (24.0–33.0) | 28.0 (24.0–33.0) | 28.0 (24.0–33.0) | 29.0 (25.0–33.0) | 28.0 (24.0–33.0) | 29.0 (25.0–34.0) |
| Lactate, mmol/L | 2.0 (1.0–2.0) | 2.0 (1.0–2.0) | 2.0 (1.0–3.0) | 2.0 (1.0–2.0) | 2.0 (1.0–2.0) | 2.0 (1.0–2.0) |
| Neuromuscular blocking agent | 3,390 (74.5) | 1,626 (75.1) | 1,764 (73.9) | 1,822 (70.6) | 844 (68.6) | 978 (72.4) |
| Prone positioning | 2,748 (60.4) | 1,294 (59.7) | 1,454 (60.9) | 1,454 (56.3) | 622 (53.6) | 792 (58.6) |
| Inhaled pulmonary vasodilators | 1,417 (31.1) | 649 (30.0) | 768 (32.2) | 811 (31.4) | 381 (31.0) | 430 (31.8) |
| Outcomes | ||||||
| ECMO days | 17.4 (9.4–30.0) | 15.8 (8.8–28.9) | 18.8 (10.0–30.9) | 20.2 (10.3–36.5) | 18.8 (9.8–36.9) | 21.6 (11.2–35.9) |
| In-hospital mortality | 2,387 (52.4) | — | — | 1,351 (52.3) | — | — |
Definition of abbreviation: ECMO = extracorporeal membrane oxygenation.
Data are expressed as median with interquartile range or frequency (%) unless otherwise indicated.
After reviewing the variable importance and model performance from the five models, we selected the eight most important predictor variables for the final prediction model. An educated CART model was selected because it allows nonlinear relationships, permits complex interactions, and is straightforward to interpret. It was developed with a maximum depth of 30 for any node in the final tree and a minimum number of 5% (183) observations in a terminal node.
External model validation
After the final predictive model was developed, additional records were obtained from ELSO reflecting patients whose data were entered between April 2021 and October 2021. The performance of the model in this external validation cohort was examined using AUCs.
RESP score testing
All variables included in the RESP score were available from the ELSO registry and included in the analysis. Immunocompromised status and central nervous system dysfunction were identified using International Classification of Disease Codes 9 and 10 as defined in prior work (10). The presence of COVID-19 was considered a viral pneumonia. The RESP score was calculated for each patient based on the work by Schmidt and colleagues (3) and used as the sole predictor of in-hospital mortality in the model. The performance of the model was examined in the derivation and internal and external validation cohorts using AUCs.
Computing environment
Statistical analyses used R (version 4.03). The rpart package (version 4.1-15) was used to run the CART modeling. The pROC package was used to calculate AUC (11).
Results
Derivation cohort
Of 4,966 patients with a diagnosis of COVID-19 who received ECMO, 298 (5.8%) were excluded for an ECMO configuration other than venovenous, and 78 (1.6%) were excluded for representing data from sites that initiated patients on ECMO and then transferred patients to an ECMO referral center. Baseline characteristics and outcomes were similar between the derivation cohort (n = 4,553) and the external validation cohort (n = 2,582) (Table 1).
After initial modeling, seven variables available before ECMO initiation were selected: age, acute kidney injury, pneumothorax, PaO2:FiO2, arterial pH, PaCO2, and serum lactate. The AUCs were 0.66 (95% CI, 0.64–0.68), 0.69 (95% CI, 0.67–0.71), 0.71 (95% CI, 0.69–0.73), and 0.66 (95% CI, 0.64–0.68) for the LASSO, random forest, logistic regression, and CART models, respectively.
The optimally pruned CART (Figure 1) included five variables and six splits that discriminated patients with the lowest probability of mortality (0.30 for a patient <41.5 years old without renal failure) from those with the highest probability of mortality (0.73 for a patient >41.5 years old with a pre-ECMO PaCO2 > 95 mm Hg).
Figure 1.
Classification and regression tree to predict in-hospital mortality. The classification tree consists of six decision rules with seven terminal daughter nodes. The tree incorporates five baseline and patient characteristics before cannulation: age, acute renal failure, PaCO2, PaO2:FiO2, and presence of a pneumothorax. Each node denotes the number of subjects in the node, the cutoff for each variable determining the branch point, and both the total number and accompanying rate of mortality. There were two low-risk nodes (<40% mortality; terminal nodes [TNs] 3 and 7), two moderate-risk nodes (40–60% mortality; TNs 2 and 6), and three high-risk nodes (>60% mortality; TNs 1, 4, and 5), with TN 4 being the highest-risk node. The area under the curve for this model was 0.66.
Model validation
Among 888 patients in the internal validation cohort and 2,582 patients in the external validation cohort, the model fit of the optimal tree was consistent with an AUC of 0.63 (95% CI, 0.62–0.65) and 0.62 (95% CI, 0.60–0.64), respectively.
RESP score performance
The AUCs for the model generated from the RESP score among the derivation, internal validation, and external validation cohorts were 0.61 (95% CI, 0.59–0.63), 0.63 (95% CI, 0.59–0.67), and 0.59 (95% CI, 0.57–0.62), respectively.
Discussion
In this analysis of data from more than 7,000 patients receiving venovenous ECMO, we found that the RESP score, a commonly used model for predicting mortality among patients receiving ECMO, performs poorly for patients with COVID-19. We evaluated multiple approaches to derive a new model for estimating risk of mortality among patients receiving ECMO for COVID-19, identifying the CART approach as having the best combination of accuracy and convenience for bedside use.
The newly derived model is the first to be validated for estimating the risk of death among patients receiving ECMO for COVID-19, providing estimates for a patient’s risk of death, ranging from a probability as low as 0.30 to as high as 0.73. Our results confirm prior findings that age is the single most important predictor of death; patients >53 years of age had a 0.66 probably of death. Although it performed at least as well as the more complicated RESP score, the CART model demonstrated only modest performance in identifying which patients would survive if provided with ECMO (AUC of 0.63 in the internal validation cohort and 0.62 in the external validation cohort).
Strengths of this study included the large sample size, inclusion of data from 353 centers in 44 countries, use of commonly available pre-ECMO clinical and physiologic variables, concordant results among five different prognostic modeling approaches, and use of a robust external validation cohort. Our choice of CART for the final model ensured that nonlinear relationships and complex interactions would be considered. The study also has limitations. All patients in the cohort received ECMO. Thus, the probability of death without ECMO and the expected treatment benefit attributable to ECMO cannot be estimated from this cohort. The ELSO registry includes data from self-selected centers with the resources to participate, which may limit generalizability. The fact that the newly developed risk model performed worse among patients with COVID-19 than models predicting survival with ECMO for non-COVID acute respiratory distress syndrome suggests that important predictors of mortality unique to the diagnosis COVID-19 remain unaccounted for in these data. These findings suggest that decision-making should not hinge solely on available risk models. Rather, the tool should be used with caution and as an adjunct to clinical judgment, expertise, and patient selection guidelines.
Footnotes
Supported by NHLBI grants K23HL153584 (J.D.C.) and K23HL143053 (M.W.S.).
Author Contributions: Study concept and design: W.D.G., J.W.S., W.G., C.J.L., and J.D.C.; acquisition of data: W.D.G. and J.E.T.; analysis and interpretation of data: W.D.G., W.G., C.J.L., and J.D.C.; drafting of the manuscript: W.D.G. and J.D.C.; critical revision of the manuscript for important intellectual content: W.D.G., J.W.S., W.G., C.J.L., J.E.T., M.B., T.W.R., M.W.S., and J.D.C.
Originally Published in Press as DOI: 10.1164/rccm.202202-0237LE on May 13, 2022
Author disclosures are available with the text of this letter at www.atsjournals.org.
References
- 1.Farmer B. Across the COVID-ravaged south, high-level life support is difficult to find [transcript]. Morning Edition. National Public Radio. September 6, 2021
- 2. MacLaren G, Fisher D, Brodie D. Treating the most critically ill patients with COVID-19: the evolving role of extracorporeal membrane oxygenation. JAMA . 2022;327:31–32. doi: 10.1001/jama.2021.22580. [DOI] [PubMed] [Google Scholar]
- 3. Schmidt M, Bailey M, Sheldrake J, Hodgson C, Aubron C, Rycus PT, et al. Predicting survival after extracorporeal membrane oxygenation for severe acute respiratory failure: the Respiratory Extracorporeal Membrane Oxygenation Survival Prediction (RESP) score. Am J Respir Crit Care Med . 2014;189:1374–1382. doi: 10.1164/rccm.201311-2023OC. [DOI] [PubMed] [Google Scholar]
- 4. Schmidt M, Zogheib E, Rozé H, Repesse X, Lebreton G, Luyt CE, et al. The PRESERVE mortality risk score and analysis of long-term outcomes after extracorporeal membrane oxygenation for severe acute respiratory distress syndrome. Intensive Care Med . 2013;39:1704–1713. doi: 10.1007/s00134-013-3037-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Joshi H, Flanagan M, Subramanian R, Drouin M. Respiratory ECMO Survival Prediction (RESP) score for COVID-19 patients treated with ECMO. ASAIO J . 2022;68:486–491. doi: 10.1097/MAT.0000000000001640. [DOI] [PubMed] [Google Scholar]
- 6.Extracorporeal Life Support Organization. SARS-CoV-2 registry addendum database definitions. 2020. [accessed 2021 May 2]. Available from: https://www.elso.org/Portals/0/Files/PDF/ELSO%20SARS-CoV-2%20Data%20Definitions%2004_03_2022.pdf
- 7. Barbaro RP, MacLaren G, Boonstra PS, Iwashyna TJ, Slutsky AS, Fan E, et al. Extracorporeal Life Support Organization Extracorporeal membrane oxygenation support in COVID-19: an international cohort study of the Extracorporeal Life Support Organization registry. Lancet . 2020;396:1071–1078. doi: 10.1016/S0140-6736(20)32008-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Moons KG, Altman DG, Reitsma JB, Ioannidis JP, Macaskill P, Steyerberg EW, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med . 2015;162:W1-73. doi: 10.7326/M14-0698. [DOI] [PubMed] [Google Scholar]
- 9. Lebreton G, Schmidt M, Ponnaiah M, Folliguet T, Para M, Guihaire J, et al. Paris ECMO-COVID-19 investigators Extracorporeal membrane oxygenation network organisation and clinical outcomes during the COVID-19 pandemic in Greater Paris, France: a multicentre cohort study. Lancet Respir Med . 2021;9:851–862. doi: 10.1016/S2213-2600(21)00096-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. O’Neil ER, Lin H, Li M, Shekerdemian L, Tonna JE, Barbaro RP, et al. Extracorporeal membrane oxygenation support for influenza a: retrospective review of the extracorporeal life support organization registry comparing H1N1 with other subtypes. Crit Care Explor . 2021;3:e0598. doi: 10.1097/CCE.0000000000000598. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Robin X, Turck N, Hainard A, Tiberti N, Lisacek F, Sanchez JC, et al. pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics . 2011;12:77. doi: 10.1186/1471-2105-12-77. [DOI] [PMC free article] [PubMed] [Google Scholar]