Supervised machine learning including environmental factors to predict in-hospital outcomes in acute heart failure patients

Benjamin Sibilia; Solenn Toupin; Nabil Bouali; Jean-Baptiste Brette; Arthur Ramonatxo; Guillaume Schurtz; Kenza Hamzi; Antonin Trimaille; Emmanuel Gall; Nicolas Piliero; Alexandre Unger; Stéphane Andrieu; Trecy Gonçalves; Fabien Picard; Vincent Roule; François Roubille; Sonia Houssany-Pissot; Océane Bouchot; Victor Aboyans; Reza Rossanaly Vasram; Thomas Bochaton; Damien Logeart; Alain Cohen Solal; Jérôme Cartailler; Alexandre Mebazaa; Jean-Guillaume Dillinger; Patrick Henry; Théo Pezel

doi:10.1093/ehjdh/ztae094

. 2024 Dec 16;6(2):190–199. doi: 10.1093/ehjdh/ztae094

Supervised machine learning including environmental factors to predict in-hospital outcomes in acute heart failure patients

Benjamin Sibilia ^1,^2,³, Solenn Toupin ^4,^5,⁶, Nabil Bouali ^7,⁸, Jean-Baptiste Brette ⁹, Arthur Ramonatxo ¹⁰, Guillaume Schurtz ¹¹, Kenza Hamzi ^12,^13,¹⁴, Antonin Trimaille ¹⁵, Emmanuel Gall ^16,^17,¹⁸, Nicolas Piliero ¹⁹, Alexandre Unger ^20,^21,^22,²³, Stéphane Andrieu ²⁴, Trecy Gonçalves ^25,^26,²⁷, Fabien Picard ²⁸, Vincent Roule ²⁹, François Roubille ³⁰, Sonia Houssany-Pissot ³¹, Océane Bouchot ³², Victor Aboyans ³³, Reza Rossanaly Vasram ³⁴, Thomas Bochaton ³⁵, Damien Logeart ^36,³⁷, Alain Cohen Solal ^38,³⁹, Jérôme Cartailler ^40,⁴¹, Alexandre Mebazaa ⁴², Jean-Guillaume Dillinger ^43,^44,⁴⁵, Patrick Henry ^46,^47,⁴⁸, Théo Pezel, on behalf of the Investigators of ADDICT-ICCU study^49,^50,^51,^✉,²

¹ Service de Cardiologie, Université Paris Cité, Hôpital Lariboisière, Assistance Publique-Hôpitaux de Paris, 2 rue Ambroise Paré, Paris 75010, France

² Inserm MASCOT—UMRS 942, University Hospital of Lariboisiere, 2 rue Ambroise Paré, Paris 75010, France

³ MIRACL.ai Laboratory, Multimodality Imaging for Research and Analysis Core Laboratory and Artificial Intelligence, University Hospital of Lariboisiere (AP-HP), 2 rue Ambroise Paré, Paris 75010, France

⁴ Service de Cardiologie, Université Paris Cité, Hôpital Lariboisière, Assistance Publique-Hôpitaux de Paris, 2 rue Ambroise Paré, Paris 75010, France

⁵ Inserm MASCOT—UMRS 942, University Hospital of Lariboisiere, 2 rue Ambroise Paré, Paris 75010, France

⁶ MIRACL.ai Laboratory, Multimodality Imaging for Research and Analysis Core Laboratory and Artificial Intelligence, University Hospital of Lariboisiere (AP-HP), 2 rue Ambroise Paré, Paris 75010, France

⁷ Department of Cardiology, University Hospital of Poitiers, Poitiers 86000, France

⁸ Service de Cardiologie, Centre Hospitalier de Saintonge, 11, Boulevard Ambroise-Paré, Saintes 17100, France

⁹ Cardiology Department, Rangueil University Hospital, Toulouse, France

¹⁰ Department of Cardiology, University Hospital of Poitiers, Poitiers 86000, France

¹¹ Department of Cardiology, University Hospital of Lille, Lille, France

¹² Service de Cardiologie, Université Paris Cité, Hôpital Lariboisière, Assistance Publique-Hôpitaux de Paris, 2 rue Ambroise Paré, Paris 75010, France

¹³ Inserm MASCOT—UMRS 942, University Hospital of Lariboisiere, 2 rue Ambroise Paré, Paris 75010, France

¹⁴ MIRACL.ai Laboratory, Multimodality Imaging for Research and Analysis Core Laboratory and Artificial Intelligence, University Hospital of Lariboisiere (AP-HP), 2 rue Ambroise Paré, Paris 75010, France

¹⁵ Department of Cardiovascular Medicine, Nouvel Hôpital Civil, Strasbourg University Hospital, Strasbourg 67000, France

¹⁶ Service de Cardiologie, Université Paris Cité, Hôpital Lariboisière, Assistance Publique-Hôpitaux de Paris, 2 rue Ambroise Paré, Paris 75010, France

¹⁷ Inserm MASCOT—UMRS 942, University Hospital of Lariboisiere, 2 rue Ambroise Paré, Paris 75010, France

¹⁸ MIRACL.ai Laboratory, Multimodality Imaging for Research and Analysis Core Laboratory and Artificial Intelligence, University Hospital of Lariboisiere (AP-HP), 2 rue Ambroise Paré, Paris 75010, France

¹⁹ Service de Cardiologie, Univ. Grenoble Alpes, CHU Grenoble Alpes, Grenoble 38000, France

²⁰ Service de Cardiologie, Université Paris Cité, Hôpital Lariboisière, Assistance Publique-Hôpitaux de Paris, 2 rue Ambroise Paré, Paris 75010, France

²¹ Inserm MASCOT—UMRS 942, University Hospital of Lariboisiere, 2 rue Ambroise Paré, Paris 75010, France

²² MIRACL.ai Laboratory, Multimodality Imaging for Research and Analysis Core Laboratory and Artificial Intelligence, University Hospital of Lariboisiere (AP-HP), 2 rue Ambroise Paré, Paris 75010, France

²³ Department of Cardiology, Hôpital Universitaire de Bruxelles, Hôpital Erasme, Université Libre de Bruxelles, Brussels, Belgium

²⁴ Service de Cardiologie, Hôpital Henri Duffaut, Avignon 84902, France

²⁵ Service de Cardiologie, Université Paris Cité, Hôpital Lariboisière, Assistance Publique-Hôpitaux de Paris, 2 rue Ambroise Paré, Paris 75010, France

²⁶ Inserm MASCOT—UMRS 942, University Hospital of Lariboisiere, 2 rue Ambroise Paré, Paris 75010, France

²⁷ MIRACL.ai Laboratory, Multimodality Imaging for Research and Analysis Core Laboratory and Artificial Intelligence, University Hospital of Lariboisiere (AP-HP), 2 rue Ambroise Paré, Paris 75010, France

²⁸ Service de Cardiologie, Université Paris Cité, Hôpital Cochin, Paris, France

²⁹ Department of Cardiology, Caen University Hospital, Caen, France

³⁰ Cardiology Department, University Hospital, PhyMedExp, University of Montpellier, INSERM, CNRS, CHRU, INI-CRT, Montpellier, France

³¹ Service de Cardiologie et Médecine Aéronautique, Hôpital d'Instruction des Armées Percy, Clamart 92140, France

³² Service de Cardiologie, Centre Hospitalier Annecy Genevois, Epagny Metz-Tessy 74370, France

³³ Department of Cardiology, University Hospital of Limoges, Limoges, France

³⁴ Department of Cardiology, Felix-Guyon University Hospital, Saint-Denis-de-La-Reunion, France

³⁵ Intensive Cardiological Care Division, Louis Pradel Hospital, Hospices Civils de Lyon, Bron, France

³⁶ Service de Cardiologie, Université Paris Cité, Hôpital Lariboisière, Assistance Publique-Hôpitaux de Paris, 2 rue Ambroise Paré, Paris 75010, France

³⁷ Inserm MASCOT—UMRS 942, University Hospital of Lariboisiere, 2 rue Ambroise Paré, Paris 75010, France

³⁸ Service de Cardiologie, Université Paris Cité, Hôpital Lariboisière, Assistance Publique-Hôpitaux de Paris, 2 rue Ambroise Paré, Paris 75010, France

³⁹ Inserm MASCOT—UMRS 942, University Hospital of Lariboisiere, 2 rue Ambroise Paré, Paris 75010, France

⁴⁰ Inserm MASCOT—UMRS 942, University Hospital of Lariboisiere, 2 rue Ambroise Paré, Paris 75010, France

⁴¹ MIRACL.ai Laboratory, Multimodality Imaging for Research and Analysis Core Laboratory and Artificial Intelligence, University Hospital of Lariboisiere (AP-HP), 2 rue Ambroise Paré, Paris 75010, France

⁴² Inserm MASCOT—UMRS 942, University Hospital of Lariboisiere, 2 rue Ambroise Paré, Paris 75010, France

⁴³ Service de Cardiologie, Université Paris Cité, Hôpital Lariboisière, Assistance Publique-Hôpitaux de Paris, 2 rue Ambroise Paré, Paris 75010, France

⁴⁴ Inserm MASCOT—UMRS 942, University Hospital of Lariboisiere, 2 rue Ambroise Paré, Paris 75010, France

⁴⁵ MIRACL.ai Laboratory, Multimodality Imaging for Research and Analysis Core Laboratory and Artificial Intelligence, University Hospital of Lariboisiere (AP-HP), 2 rue Ambroise Paré, Paris 75010, France

⁴⁶ Service de Cardiologie, Université Paris Cité, Hôpital Lariboisière, Assistance Publique-Hôpitaux de Paris, 2 rue Ambroise Paré, Paris 75010, France

⁴⁷ Inserm MASCOT—UMRS 942, University Hospital of Lariboisiere, 2 rue Ambroise Paré, Paris 75010, France

⁴⁸ MIRACL.ai Laboratory, Multimodality Imaging for Research and Analysis Core Laboratory and Artificial Intelligence, University Hospital of Lariboisiere (AP-HP), 2 rue Ambroise Paré, Paris 75010, France

⁴⁹ Service de Cardiologie, Université Paris Cité, Hôpital Lariboisière, Assistance Publique-Hôpitaux de Paris, 2 rue Ambroise Paré, Paris 75010, France

⁵⁰ Inserm MASCOT—UMRS 942, University Hospital of Lariboisiere, 2 rue Ambroise Paré, Paris 75010, France

⁵¹ MIRACL.ai Laboratory, Multimodality Imaging for Research and Analysis Core Laboratory and Artificial Intelligence, University Hospital of Lariboisiere (AP-HP), 2 rue Ambroise Paré, Paris 75010, France

^✉

Corresponding author. Tel: +33 149 95 82 24, Fax: +33 1 49 95 62 99, Email: theo.pezel@aphp.fr

Conflict of interest: None declared.

PMCID: PMC11914725 PMID: 40110211

Abstract

Aims

While few traditional scores are available for risk stratification of patients hospitalized for acute heart failure (AHF), the potential benefit of machine learning (ML) is not well established. We aimed to assess the feasibility and accuracy of a supervised ML model including environmental factors to predict in-hospital major adverse events (MAEs) in patients hospitalized for AHF.

Methods and results

In April 2021, a French national prospective multicentre study included all consecutive patients hospitalized in intensive cardiac care unit. Patients admitted for AHF were included in the analyses. A ML model involving automated feature selection by least absolute shrinkage and selection operator (LASSO) and model building with a random forest (RF) algorithm was developed. The primary composite outcome was in-hospital MAE defined by death, resuscitated cardiac arrest, or cardiogenic shock requiring assistance. Among 459 patients included (age 68 ± 14 years, 68% male), 47 experienced in-hospital MAE (10.2%). Seven variables were selected by LASSO for predicting MAE in the training data set (n = 322): mean arterial pressure, ischaemic aetiology, sub-aortic velocity time integral, E/e′, tricuspid annular plane systolic excursion, recreational drug use, and exhaled carbon monoxide level. The RF model showed the best performance compared with other evaluated models [area under the receiver operating curve (AUROC) = 0.82, 95% confidence interval (CI) (0.78–0.86); precision-recall area under the curve = 0.48, 95% CI (0.42–0.5), F1 score = 0.56). Our ML model exhibited a higher AUROC compared with an existing score for the prediction of MAE (AUROC for our ML model: 0.82 vs. ACUTE HF score: 0.57; P < 0.001).

Conclusion

Our ML model including in particular environmental variables exhibited a better performance than traditional statistical methods to predict in-hospital outcomes in patients admitted for AHF.

Study registration

ClinicalTrials.gov identifier: NCT05063097.

Keywords: Acute heart failure, Environmental factors, Carbon monoxide, Recreational drugs, Machine learning, Intensive cardiac care unit

Graphical Abstract

Introduction

Acute heart failure (AHF) continues to be the main cause for hospital admissions among patients over the age of 65 years and is responsible for a mortality exceeding 25%,¹ including an 8% mortality rate within hospitals at the 30-day mark.² It is therefore crucial to offer a high-performance prognostic stratification tool to identify these patients at risk of in-hospital poor outcomes. Beyond the clinical, biological, and echocardiographic parameters well established as prognostic factors,^3,4 several recent studies have shown a role for environmental factors in the in-hospital prognosis of these AHF patients. Indeed, recent studies reported that recreational drug use could be a strong prognosticator of in-hospital outcomes due to multifactorial acute cardiotoxicity.^5–7 Indeed, our working group has recently shown that recent consumption of recreational drugs in AHF patients was independently associated with a risk of in-hospital major adverse event (MAE) multiplied by more than seven compared with non-users.⁶ Another recent subanalysis showed that the level of exhaled CO, as a biomarker of air pollution in the patient's lungs, was independently associated with a higher risk of in-hospital mortality in AHF patients.⁸ Furthermore, a large meta-analysis emphasized the impact of air pollution, including carbon monoxide (CO) due to active and passive smoking, as an aggravating factor of AHF.⁹ However, until now, no tool is available to take into account all of these variables simultaneously.

Machine learning (ML) methods can recognize patterns in large data sets with a multiplicity of variables and can be used for improving risk stratification. Recently, ML techniques, using supervised or unsupervised approaches, have emerged as highly effective methods for prognostic prediction and decision-making in several cardiovascular diseases.^10,11 Compared with unsupervised ML, a supervised ML method overcomes the main limitations of traditional methods with the problems of collinearity, inadequate model complexity, and violation of rigid assumptions. In addition, the strength of supervised ML is its ability to approximate non-linear relationships between hundreds of variables. Although of potential interest, supervised ML techniques using all this spectrum of variables have not been evaluated for risk stratification in AHF patients.

Therefore, the aim of this study was to investigate the feasibility and accuracy of a supervised ML model using clinical, biological, environmental, and echocardiographic data to predict in-hospital MAE in a consecutive cohort of patients hospitalized for AHF.

Methods

Study population

All details of the Addiction in Intensive Cardiac Care Units (ADDICT-ICCU) study design have been published previously.¹² Briefly, this is a multicentre, prospective, observational study of all consecutive patients aged ≥18 years admitted to intensive cardiac care units (ICCUs) over 2 weeks in April 2021 at 39 centres across France (including all diagnoses). The main exclusion criterion was hospitalization for either a planned interventional procedure or more than 24 h at any hospital facility before admission to the ICCU. This was to prevent the risk of obtaining a negative urine drug assay in patients with recreational drug consumption more than 24 h prior to admission. The methodology of the baseline characteristics collection is detailed in Supplementary material online, Supplement S1. In this study, we decided to focus the analysis on patients admitted for AHF defined by symptoms and/or signs of heart failure with evidence of diastolic or systolic dysfunction by echocardiography and elevated levels of natriuretic peptide [B-type natriuretic peptide (BNP) > 35 pg/mL and/or N-terminal prohormone of BNP (NT-proBNP) > 125 pg/mL]³ (Figure 1). The main admission diagnosis was adjudicated by two independent experts at the end of the hospitalization following the current guidelines (see Supplementary material online, Supplement S2). The investigation conforms with the principles outlined in the Declaration of Helsinki. All patients had a written informed consent obtained at enrolment and were informed of the urine drug assay and exhaled CO measurement. This study has been approved by the Ethics Committee (Committee for the Protection of Human Subjects, Ile de France-7, France) and registered on the ClinicalTrials.gov website under the number NCT05063097.

Primary outcome

The primary outcome was in-hospital MAE defined as a composite of all-cause mortality, cardiogenic shock (requiring medical or mechanical haemodynamic support) and resuscitated cardiac arrest (severe ventricular arrhythmia requiring defibrillation or anti-arrhythmic agents). An independent committee of experts was in charge of reviewing anonymized medical documents according to standardized definitions.¹³

Baseline characteristics collection

The database records of the AHF population comprise baseline data that include clinical, demographic, reason for hospitalization, list of medications especially cardiovascular medications at admission, and history of cardiovascular disease. Blood pressure was measured non-invasively. Biological data were collected systematically upon admission, including haemoglobin, serum potassium, creatinine, the peak of troponin (hsTNI), the NT-proBNP or BNP. Standardized transthoracic echocardiography was performed systematically within the first 24 h of admission for all patients by a cardiologist and reviewed by a senior.

Regarding environmental factors, the presence of recreational drugs was determined through urine analysis using a dedicated drug assay (NarcoCheck®, Kappa City Biotech SAS, Montluçon, France) within 2 h of admission to the ICCU with excellent performance (the sensitivity and specificity of the urine drug assay were 91.7% and 97.7%, respectively).⁶ The following recreational drugs will be screened for all patients: cannabinoids (tetrahydrocannabinol), including cannabis and hashish; cocaine and metabolites, including cocaine and crack; amphetamines; 3,4-methylenedioxy-methylamphetamine (or ecstasy); and heroin and other opioids. Finally, a standardized exhaled CO measurement was systematically performed with a CO-Check Pro device (Micro Direct Diagnostics Ltd, UK) immediately on arrival in ICCU.

Machine learning

Fourteen clinical, four biological, three environmental, and seven echocardiographic parameters were available (detailed list provided in Supplementary material online, Supplement S3). Machine learning involved feature selection by least absolute shrinkage and selection operator (LASSO) algorithm and model building with random forest algorithm following guidelines for ML.¹⁴

Feature selection

After the imputation of missing values by the K-nearest neighbours with a value of K = 5, feature selection was performed using LASSO (Figure 2), which is a penalized logistic regression that reduces the number of variables (glmnet R package).¹⁵ The regularization parameter λ controls the strength of the shrinkage to get the best compromise between prediction performance and interpretability. The best λ was found using 10-fold cross-validation and the ‘1 standard error rule’, for which the area under the receiver operating curve (AUROC) of the most parsimonious model is no more than 1 standard error above the AUROC of the best model.¹⁶

Feature importance selected by least absolute shrinkage and selection operator. Least absolute shrinkage and selection operator was used to evaluate the worth of each variable by measuring the log-rank–based variable importance with respect to the outcome and then to rank the attributes by their individual evaluations (top to bottom). Only attributes resulting in log-rank based variable importance > 30 (above the vertical line) were subsequently used for the model building. CO, carbon monoxide; HF, heart failure; LASSO, least absolute shrinkage and selection operator; LVOT VTI, left ventricular outflow tract velocity–time integral; MAP, mean arterial; TAPSE, tricuspid annular plane systolic excursion.

Model building

To build the prediction model based on the training cohort, we compared four modelling strategies with logistic regression, random forest, extreme gradient boosting (XGBoost) model, and LASSO. To assess the performance of these algorithms, we used several metrics: AUROC, precision-recall area under the curve (PRAUC), F1 score, sensitivity, specificity, positive and negative prediction value, accuracy, balanced accuracy, Cohen’s kappa, Brier score, and McNemar test P-value. Due to the low occurrence of events and the imbalanced nature of the data, the F1 score and the precision-recall curve were preferred to determine the best model to minimize the overall misclassification cost and mitigate the problem of imbalanced data. For the prediction of in-hospital MAE, the best model was the ML model using random forest (Figure 3). Random forest corresponds to a method based on decision trees, where each tree is trained on a random sample from the data set. All the results presented are shown using only the testing cohort for validation.

Performance of machine learning models for prediction of in-hospital major adverse event. Receiver operating characteristic (area under the receiver operating curve, A) and precision-recall curves (precision-recall area under the curve, B) for prediction of in-hospital MAE were used to compare least absolute shrinkage and selection operator, random forest, logistic regression, and extreme gradient boosting, developed using the set of variables selected by least absolute shrinkage and selection operator. AUC, area under the curve; LASSO, least absolute shrinkage and selection operator; ROC, receiver operating curve; XGBoost, extreme gradient boosting.

For model development, patients hospitalized for AHF in the ADDICT-ICCU study were randomly partitioned into training (70%) and validation (30%) cohorts (see Supplementary material online, Supplement S4). We used a 10-fold cross-validation process to tune the model hyperparameters using grid search.

Statistical analysis

Continuous data will be reported as mean ± standard deviation for normally distributed data or as medians and interquartile range for non-normally distributed data. Categorical data will be reported as counts and percentages. Between-group comparisons will be performed using the Student’s t-test or Mann–Whitney test for continuous variables and the χ² or Fisher’s exact test for categorical variables, as appropriate.

Model discrimination was measured using the AUROC and its 95% confidence intervals (CIs). In addition, the positive predictive value (or precision) and the sensitivity (recall) across all possible risk thresholds for predicting in-hospital MAE were plotted using precision-recall curves and their area under the curve. The PRAUC, unlike the AUROC, is not affected by the number of true-negative results. In data sets with small event rates and therefore a large expected true-negative rate, such as the current study, the PRAUC is better for comparing different models. For both the AUROC and PRAUC, values closer to 1 correspond to more accurate models. DeLong's test was used to compare the AUROC of each model. We also calculated the F1 score, sensitivity, specificity, positive predictive value, and negative predictive value. In addition, we calculated a Brier score for each model as a measure of model accuracy.

Model calibration was measured using (i) the Hosmer–Lemeshow goodness-of-fit test; (ii) the reliability component of the Brier score; and (iii) graphically with plots showing the observed and predicted proportion of events, grouped by level of risk.

A two-tailed P-value < 0.05 will be considered statistically significant. All data will be analysed using R software, version 4.0.3 (R Project for Statistical Computing, R Foundation, Vienna, Austria).