Machine Learning Based Prediction of Myocardial Recovery in Patients with Left Ventricular Assist Device Support

Abstract

Background:

Prospective studies demonstrate that aggressive pharmacological therapy combined with pump speed optimization may result in myocardial recovery in larger numbers of patients supported with left ventricular assist device (LVAD). This study sought to determine whether use of machine learning based models predict LVAD patients with myocardial recovery resulting in pump explant

Methods:

20,270 adult patients with a durable continuous-flow LVAD in the INTERMACS registry were included in the study. 98 raw clinical variables were screened using least absolute shrinkage and selection operator (LASSO) for selection of features associated with LVAD-induced myocardial recovery. Machine learning models were developed in the training dataset (70%) and were assessed in the validation dataset (30%) by receiver operating curve (ROC) and Kaplan-Meier analysis.

Results:

LASSO identified 28 unique clinical features associated with LVAD-induced myocardial recovery including, age, etiology of HF, psychosocial risk factors, laboratory values, cardiac rate and rhythm, and echocardiographic indices. Machine learning models achieved areas under the ROC curve (AUCs) of 0.813 – 0.824 in the validation dataset outperforming logistic regression-based new INTERMACS recovery risk score (AUC of 0.796) and previously established LVAD recovery risk scores (I-CARS and I-TOPS) with AUCs of 0.744 and 0.748 (p< 0.05). Patients who were predicted to recover by machine learning model demonstrated significantly higher incidence of myocardial recovery resulting in LVAD explant in the validation cohort compared to those who were not predicted to recover (18.8% vs 2.6% at 4 years of pump support).

Conclusion:

Machine learning can be a valuable tool to identify subsets of LVAD patients who may be more likely to respond to myocardial recovery protocols.

Introduction

Mechanical unloading with left ventricular assist devices (LVADs) is associated with improvements in the structure and function of the failing myocardium, occasionally allowing for device explant in a small proportion of patients, also termed as bridge-to-recovery.¹ While the clinical, biological and genetic determinants of myocardial recovery on LVAD are poorly understood, prospective studies have suggested that a program-based approach incorporating guideline-directed neurohormonal inhibition, pump speed optimization, and serial assessment of native heart function with turn-down echocardiograms may help achieve myocardial recovery in larger number of LVAD patients.^{2, 3} Since achieving myocardial recovery requires active engagement of the LVAD care team as well as utilization of resources, proper patient selection remains key for successful patient and programmatic outcomes.

Using traditional statistical regression-based models, we and other investigators identified young age, non-ischemic etiology, and shorter duration of heart failure as significant predictors of myocardial recovery on LVAD support.^4–6 However, clinical data are exceedingly complex and contain non-linear interactions which may not be fully captured by traditional statistical methods. Machine learning has emerged as a powerful tool for learning complex data patterns and identifying non-linear interactions within large clinical datasets to formulate disease classifications as well as to predict patient outcomes.⁷ As a result, prediction models derived from machine learning algorithms may significantly outperform traditional regression-based models.⁸ Furthermore many machine learning models retain a high degree of explainability allowing for improved prediction as well as insights that can guide future innovation in clinical care. Using the multicenter INTERMACS Registry, we hypothesized that machine learning based prediction models may improve selection of candidates who are likely to recover on LVAD support compared to traditional regression-based models and uncover novel clinical risk factors that are positively or negatively associated with myocardial recovery on LVAD support.

Material and Methods

Patient Population

The INTERMACS registry was queried to include adult patients (≥ 18 years old) who received durable continuous-flow mechanical circulatory support from 2008 through 2017. Patients who were implanted with a right ventricular mechanical circulatory support device or total artificial heart were excluded from the study. Patients with complex congenital heart disease, hypertrophic or restrictive cardiomyopathy, or prior heart transplantation were also excluded from the analysis. The primary outcome of the analysis was LVAD explant specifically for myocardial recovery indication. This study has been approved by the Columbia University IRB. The data used in the study are available to other researchers for purposes of reproducing the results or replicating the procedure via data request from the National Heart, Lung, and Blood Institute Biological Specimen and Data Repository Information Coordination Center.

Data Processing

A total of 98 raw clinical variables in the INTERMACS dataset (Supplemental Table I) were screened and included for feature selection. Patients without pre-implant and early post-implant (1-week, 1-month, or 3-month) echocardiography data were excluded from the study. Median/mode imputation was used for the remainder of missing variables (Supplemental Figure I). Categorical data were pre-processed using one-hot encoding.⁹ Continuous data were normalized by subtracting the mean and scaling to unit variance. Data were randomly split into training and validation cohorts with a ratio of 70% while maintaining similar proportion of myocardial recovery end point in each cohort (Figure 1). 30% of the dataset was split into a test set which was used to validate performance of the model. We modeled the recovery prediction task as a binary classification problem. To eliminate irrelevant or redundant features, we performed feature selection by identifying the most predictive myocardial recovery variables using least absolute shrinkage and selection operator (LASSO). LASSO is a form of L1 regularization that adds the absolute magnitude of feature coefficients as the penalty term rather than the squared magnitude that is typically used. The effect is more significant regularization that decreases overfitting and creates a more generalizable, and often simpler to interpret, model better aligned with use in clinical care. Features with non-zero coefficients were selected for model training. Features with zero coefficient were considered redundant and eliminated from downstream modeling.

Figure 1. Study Design.

Derivation and Validation of Machine Learning Based Cardiac Recovery Prediction Model in the INTERMACS Registry

Model Training and Evaluation

We trained the machine learning models to predict risk of myocardial recovery using 28 unique features. Machine learning algorithms used in this study include Bayesian logistic regression (B-LR), linear support vector machine (SVM), gradient boosted decision tree (GBDT), neural network (NN), and random forest (RF) with 10-fold cross validation to fine-tune model parameters. Since only a small proportion of patients achieve myocardial recovery in the study population, area under curve (AUC) has been chosen as an evaluation metric as opposed to accuracy. Feature selection was utilized for all ML models due to improved performance observed in ablation studies (Supplemental Table II). To mitigate the issue of imbalanced classes, we applied class-weighting, down-sampling and synthetic minority over-sampling (smote) techniques to each training algorithm, and eventually used the technique with highest discriminatory performance for each model (Supplemental Table III).¹⁰ The predictive performance of the models was evaluated by ROC curve, Kaplan–Meier curve, and evaluation metrics including area under the ROC curve (AUC), sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and balanced accuracy. Performance of machine learning models were compared to established regression-based cardiac recovery prediction models from the INTERMACS including Cardiac Recovery Score (I-CARS)⁶ by the Utah group and Complete Recovery Score by the Columbia Group (I-TOPS)⁴ (Supplemental Table IV). A new multivariable logistic regression based INTERMACS LVAD recovery risk score was also developed from the ML training dataset (n= 6,885) using the same 98 raw clinical variables listed in Supplemental Table I. Discriminatory ability of this regression-based model was tested on the validation dataset using ROC analysis.

While the primary outcome of the study was LVAD explantation for myocardial recovery, we investigated predictive ability of ML model on echocardiographic LV recovery as an additional study endpoint in the validation dataset. This analysis was restricted to patients with who had echocardiographic data available before and after LVAD support. Echocardiographic LV recovery was defined as LVEF >40% on LVAD support at any time point during follow-up. Since the decision of device explantation for recovery could be confounded by transplant candidacy, we analyzed myocardial recovery with the competing events of heart transplantation and mortality on LVAD support.

Statistical Analysis

Statistical analysis was performed in R (version 3.6.2). For descriptive analysis, median (IQR) and frequencies (%) were assessed for continuous and categorical variables, respectively. The ROC curve and AUC analysis were conducted with pROC package. LASSO feature selection has been performed using R glmnet package. Sensitivity, specificity, PPV, NPV, Kappa, F1 score, and balanced accuracy were calculated with R caret packages. The B-LR, SVM, KNN, RF, GBDT, and NN models were called using method bayesglm, svmLinear, gbm, avNNet, and rf with default settings, respectively. Survival curves were developed by Kaplan–Meier method with log-rank test and plotted with R-packages survival and survminer. Competing event analysis of mortality, transplantation, or recovery on LVAD support was performed using R package cmprsk.

Results

Feature Selection

98 raw features extracted from the INTERMACS were included for feature selection by LASSO (Figure 2A). Of those, 28 unique features with non-zero coefficients were chosen for machine learning model training (Figure 2B and Supplemental Figure II). 14 features had a positive association with LVAD-associated myocardial recovery including bridge-to-recovery (BTR) implant strategy, current tobacco abuse, post-partum cardiomyopathy, recent cardiac diagnosis (1 month – 1 year), history of ETOH use, limited social support, non-compliance, higher hemoglobin, recent cardiac diagnosis (<1 month), sinus rhythm, elevated heart rate, elevated serum sodium, INTERMACS Profile 3, and other psychiatric disease (Figure 2B). 14 features had a negative association with LVAD-induced myocardial recovery including presence of an ICD, post-implant LVEF (0–20%), RVAD implantation with LVAD, post-implant LVEF (20–29%), old age, higher blood urea nitrogen, ischemic cardiomyopathy, use of centrifugal pump, higher post-implant LVEDD, bridging with IABP, pre-implant warfarin use, pre-implant amiodarone use, BTT indication, and concomitant surgical tricuspid procedure.

Figure 2. Feature Selection by LASSO.

A) LASSO variable trace profiles of 98 unique features, the vertical dashed line demonstrates the lambda value of 0.0039 providing the most regularized model B) Feature coefficients obtained by LASSO at the lambda value of 0.0039. Features positively associated with recovery are colored in blue, features negatively associated with recovery are colored in red.

Model Development and Performance

Five machine learning models including B-LR, linear SVM, GBDT, NN, and RF were developed from the training dataset. Tuning parameters for ML models have been summarized in Supplemental Table V. All ML models displayed promising performances to predict LVAD induced myocardial recovery in the validation cohort with AUCs > 0.810 (Table 1). Discriminatory ability of machine models was significantly better than the established regression-based INTERMACS recovery scores including I-CARS and I-TOPS which had AUCs < 0.750 (all p<0.001) (Figure 3A). Since I-CARS and I-TOPS scores were derived from earlier versions of INTERMACS dataset, we performed a new multivariable logistic regression analysis in the training dataset and derived a new INTERMACS LVAD Recovery Risk score (Supplemental Table VI). Logistic regression identified 16 clinical predictors of LVAD explantation for recovery. Importantly, 15 out of 16 predictors identified by logistic regression analysis were also captured by LASSO feature selection, except for frailty variable which was associated with lower chances of recovery (Supplemental Figure III). While the discriminatory ability of the new INTERMACS LVAD recovery risk score (AUC 0.796) was superior compared to I-CARS and I-TOPS, it remained inferior to top performing B-LR machine learning model (AUC 0.824) in the validation dataset (p=0.046) (Supplemental Figure IV). Next, patients in the validation dataset were classified into “predicted” versus “not predicted” to recover based on their B-LR derived recovery risk score being above or below the optimal cut-off as determined by the Youden’s index. Sensitivity, specificity, PPV, and NPV of the B-LR model in the validation dataset using the Youden’s index were 72.7%, 79.0%, 8.5%, and 99.0% (Table 1). Using this strategy, 659 out of 2290 (28.8%) patients in the validation cohort were classified as “predicted to recover” by B-LR model. Importantly, cumulative incidence of LVAD explantation for myocardial recovery was significantly higher in patients who were predicted to recover with ML compared to those who were not (5.1%, 11.5%, 15.8%, and 18.8% versus 0.2%, 1.4%, 1.9%, 2.6% at 1, 2, 3, and 4 years of LVAD support, log-rank p<0.001) (Figure 3B).

Table 1.

Performance of recovery risk prediction models in the validation cohort

ML Model	AUC	Sensitivity	Specificity	PPV	NPV	F1	Kappa	Balanced Accuracy
Bayesian Logistic Regression	0.824	72.7%	79.0%	8.5%	99.1%	0.152	0.111	0.759
Linear Support Vector Machine	0.816	81.8%	72.1%	7.3%	99.3%	0.134	0.090	0.770
Gradient Boosted Decision Tree	0.814	81.8%	68.6%	6.5%	99.3%	0.121	0.076	0.752
Neural network	0.813	81.8%	70.4%	6.9%	99.3%	0.127	0.127	0.761
Random Forest	0.818	85.7%	69.5%	7.0%	99.5%	0.129	0.085	0.725

Figure 3. Predictive Performance of Machine Learning Models in the Validation Cohort.

(A) ROC curve demonstrating AUC of machine learning (B-LR, linear SVM, GBDT, NN, and RF) and regression-based (I-CARS and I-TOPS) risk prediction models (B) Cumulative Incidence of LVAD Explant for Myocardial Recovery in Patients who were predicted or NOT predicted to recover by B-LR machine learning algorithm

When competing events of mortality and heart transplantation were analyzed, patients who were predicted to recover by B-LR machine learning model had significantly higher incidence of LVAD explantation for recovery (11.7% vs. 1.3%, p<0.001) and lower mortality on LVAD support (24.3% vs. 35.5%, p<0.001), while the incidence of heart transplantation was comparable (34.9% vs. 37.4%, p=0.147) (Figure 4A). Patients who were predicted to recover by B-LR machine were also significantly more likely to exhibit echocardiographic LV recovery on LVAD support (48.0% vs. 13.2%, log-rank p <0.001) (Figure 4B).

Figure 4. Predictive Performance of B-LR Machine Learning Model in the Validation Cohort.

(A) Competing event analysis of recovery, heart transplantation, and mortality on LVAD support (B) Cumulative Incidence of Echocardiographic Recovery in LVAD Patients who were predicted or NOT predicted to recover by B-LR machine learning algorithm

Discussion

This cohort study using a large national registry of LVAD patients investigated potential utility of machine learning algorithms in prediction of myocardial recovery on LVAD support. Our major findings are as follows: (1) Patients with psychosocial risk factors including history non-compliance, tobacco/ETOH use, and limited social support at the time of device implantation demonstrated higher likelihood of myocardial recovery on LVAD support; (2) early post-implant echocardiographic indices were more predictive of LVAD-induced myocardial recovery compared to pre-implant echocardiographic indices; (3) Machine learning algorithms outperformed regression-based traditional cardiac recovery scores including I-CARS and I-TOPS; (4) Nearly one in every 5 patients who were classified to recover by ML algorithm underwent LVAD explant for myocardial recovery by 4 years of device support. Taken together, these findings suggest that machine learning has the potential to improve clinical decision-making in selection of LVAD recovery candidates and improve patient outcomes.

Machine learning models confirmed several clinical risk factors that were previously shown to be associated with LVAD-induced myocardial recovery including presence/absence of ICD, age at implant, shorter duration of heart failure, renal function (BUN), and etiology of heart failure (post-partum and non-ischemic). Furthermore, we identified several clinical risk factors that were previously underappreciated with regards to myocardial recovery in LVAD patients. Specifically, several psychosocial risk factors including history of non-compliance, current tobacco use, history of ETOH use, limited social support, and other psychiatric disease were associated with higher likelihood of LVAD-induced myocardial recovery. One explanation for this finding is that, patients with psychosocial risk factors have the potential to become compliant with pharmacological therapy and eliminate toxic habits associated with heart failure following LVAD implantation. Another possibility is that patients with increased psychosocial risk are less likely to be considered for transplantation allowing for longer times on LVAD support and subsequent recovery. However, ML models incorporating psychosocial risk factors continued to predict recovery on LVAD even after adjustment for competing events of transplantation and mortality on device support. These findings also suggest that presence of psychosocial risk factors should not be an absolute contraindication for LVAD therapy and that select patients with psychosocial risk factors may achieve favorable outcome on LVAD support resulting in LVAD explant. While previous studies, including from INTERMACS, have suggested pre-implant LVEDD as potential predictor of myocardial recovery, machine learning models demonstrated that post-implant echocardiographic indices in the early post-implant period were significant predictors of recovery. This observation suggests that the interaction between native cardiac function and competing LVAD flow could be more informative than pre-implant left ventricular structure and function. Echocardiograms should be routinely obtained in the early post-implantation period, not only to optimize pump speed but also to assess for early signs of myocardial recovery (reduction in LVEDD and improvement in LVEF) in all patients.

Machine learning models demonstrated excellent discrimination for prediction of myocardial recovery on LVAD support with AUCs greater than 0.810 with > 99.0% negative predictive values (NPVs). Low positive predictive value (PPV) of machine learning models is not unexpected given the low incidence of myocardial recovery in LVAD patients. Inclusion of additional features that are not available in the INTERMACS such as pharmacological therapy, doses of medications used, optimization of pump speed, and turn-down echocardiographic or hemodynamic indices could improve discriminative ability of the models. A combinatorial machine learning based approach including analysis of EKG waveforms, radiographic and echocardiographic images, and LVAD core histopathology may help development of recovery prediction models with superior accuracy and precision. Similarly, machine learning based analysis of transcriptional profiling for coding and non-coding RNAs at the time of LVAD implantation may have the potential to improve patient selection for myocardial recovery.

Growing lines of evidence suggest limited incremental of machine learning algorithms over traditional statistical techniques in less complex datasets such as electronic medical records.^{11, 12} In addition, use and interpretation of complex ML models could be challenging for the clinician compared to simple logistic regression models. On the other hand, ML algorithms were shown to be superior to traditional risk models with regards to risk prediction of heart failure readmission and mortality.^{13, 14} Similarly, current analysis suggested superior discriminatory ability of ML models with regards to prediction of recovery in LVAD supported patients. Superior performance of ML models could be in part attributed to improved feature selection process, successful parameter tuning, and non-linear relationship within the dataset.

The current study has several limitations. First, our analysis was restricted to clinical variables present in the INTERMACS registry and was subject to missingness. Echocardiographic or radiographic image data which could substantially improve model performance were not available. Machine learning models were validated internally using random splitting of INTERMACS data however was not externally validated in part due to small number of recovery patients at any given center. Myocardial recovery has been modeled as a binary variable as opposed to time-dependent, however performance of model classification was assessed by Kaplan-Meier analysis in the validation dataset. Finally, the majority of patients included in the INTERMACS Registry were implanted with Heartmate 2 or Heartware LVAD while patients implanted with newer devices such as Heartmate 3 LVAD is not well represented in this cohort limiting generalizability of the findings.

In conclusion, in a large national registry, machine learning algorithms outperformed regression-based models with regards to prediction of myocardial recovery on LVAD support and identified several recovery-related risk factors that were previously underappreciated. Machine learning has the potential to improve decision-making and delivery of care to LVAD patients.

WHAT IS NEW?

• Machine learning can be successfully applied to large clinical datasets to identify patients who are likely to recover on LVAD support

• Machine learning based models have greater discriminatory capacity than conventional regression-based models in prediction of patients with high vs. low likelihood of recovery on LVAD support

• Psychosocial risk factors (tobacco use, ETOH use, limited social support, and non-compliance) are associated with higher likelihood of recovery on LVAD support based on machine learning based feature selection

WHAT ARE THE CLINICAL IMPLICATIONS?

• Machine learning tools can help the care team to better identify patients who are likely to recover on LVAD support so that the recovery efforts could be maximized on these individuals

• Future machine learning models incorporating echocardiographic, radiological, and pathological image data combined with clinical information could provide superior prediction of recovery candidates on LVAD support

Non-Standard Abbreviations

AUC

Area under curve

B-LR

Bayesian logistic regression

GBDT

Gradient boosted decision tree

LASSO

Least absolute shrinkage and selection operator

LVAD

Left ventricular Assist Device

NN

Neural network

RF

Random forest

ROC

Receiver operating characteristic

SVM

Support vector machine