Machine-Learning-Based In-Hospital Mortality Prediction for Transcatheter Mitral Valve Repair in the United States

Abstract

Background

Transcatheter mitral valve repair utilization has increased significantly in the United States over the last years. Yet, a risk-prediction tool for adverse events has not been developed. We aimed to generate a machine-learning-based algorithm to predict in-hospital mortality after TMVR.

Methods

Patients who underwent TMVR from 2012 through 2015 were identified using the National Inpatient Sample database. The study population was randomly divided into a training set (n = 636) and a testing set (n = 213). Prediction models for in-hospital mortality were obtained using five supervised machine-learning classifiers.

Results

A total of 849 TMVRs were analyzed in our study. The overall in-hospital mortality was 3.1%. A naïve Bayes (NB) model had the best discrimination for fifteen variables, with an area under the receiver-operating curve (AUC) of 0.83 (95% CI, 0.80–0.87), compared to 0.77 for logistic regression (95% CI, 0.58–0.95), 0.73 for an artificial neural network (95% CI, 0.55–0.91), and 0.67 for both a random forest and a support-vector machine (95% CI, 0.47–0.87). History of coronary artery disease, of chronic kidney disease, and smoking were the three most significant predictors of in-hospital mortality.

Conclusions

We developed a robust machine-learning-derived model to predict in-hospital mortality in patients undergoing TMVR. This model is promising for decision-making and deserves further clinical validation.

INTRODUCTION

Mitral regurgitation (MR) is the most common valve disease in the developed world, with a reported prevalence of 1.7% in the United States (US) adult population [1]. Although surgical replacement and/or repair have been the gold standard treatment for decades, around half of the patients with severe MR are not surgical candidates due to their increased perioperative risk owing to high comorbidity burden and advanced age, among other factors [2, 3]. In recent years, percutaneous edge-to-edge transcatheter mitral valve repair (TMVR) has emerged as a novel treatment option for these patients, with the MitraClip (Abbott Vascular, Menlo Park, California) being the only therapy that has so far been approved by the US Food and Drug Administration (FDA) [4–6]. However, as TMVR becomes widely available and its use continues to expand, concerns related to periprocedural complications—particularly in-hospital mortality, durability, and long-term outcomes—remain.

Due to the extreme clinical scenario, the complex nature of mitral valve structure, hemodynamics, and the infancy of transcatheter therapies, appropriate candidate selection is of paramount importance to achieve a favorable outcome. Yet, a risk-prediction tool for adverse outcomes after TMVR for severe MR has not been developed. To address this critical scientific gap and given recent reports highlighting the advantages of using artificial intelligence to predict adverse outcomes after transcatheter aortic valve replacement [7, 8], we sought to generate a machine-learning (ML)-based model to predict in-hospital mortality after TMVR, using a large national database. ML is a powerful tool that is increasingly utilized in clinical research in an effort to improve predictive modeling and uncover novel determinants of a specific outcome [7–10].

METHODS

2.1. Data source

Data were obtained from the National Inpatient Sample (NIS) database files. The NIS is part of the Healthcare Cost and Utilization Project (HCUP), which is sponsored by the Agency for Healthcare Research and Quality (AHRQ) and represents the largest publicly available all-payer inpatient care database in the US [11]. Further descriptions of the NIS database have been published previously [12, 13]. Because the NIS is a limited dataset with de-identified patient information, permission from the institutional review board was not required.

International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes were used to identify all patients aged 18 years and older who underwent TMVR (code 35.97) from January 1, 2012, to September 30, 2015. This last cutoff date was selected due to the implementation of the ICD-10-CM, which occurred on October 1, 2015. In a large database such as the NIS, having missing values in more than 1 variable can be challenging in the developmental process of a prediction model. To handle this issue, patients with missing information were excluded from the study (n = 107) (Figure 1). Missing data were distributed across variables such as race, primary payer, income, and elective admission. The final study population (n = 849) was allocated into two groups: the development and validation cohorts. The main endpoint of our study was all-cause in-hospital mortality. A list of ICD-9-CM codes used for the selection of the study population is provided in Table 1S in the Data Supplement. Baseline characteristics were obtained using either the Elixhauser comorbidities [14] or the corresponding ICD-9-CM codes described in Table 1S of the Data Supplement.

Figure 1.

Overview of the methods used for data extraction, training, and testing.

RF, random forest; NB, naïve Bayes; ANN, artificial neural network; LR, logistic regression; SVM, support-vector machine.

2.2. Study Design

The procedures and methods included in this study were a combination of data wrangling, sampling techniques, feature ranking, and classification model development and evaluation (Figure 1). A script in the programming language Python was developed to automatize the process [15]. Five ML classification algorithms were initially trained with 75% of the data (training or development cohort) and finally evaluated with the remaining 25% of the unseen data (testing or validation cohort). Patient data were randomly selected to create these two initial cohorts. The training (n = 636) and testing (n = 213) sets were allocated into two classes: 1) patients that survived the TMVR hospitalization (classed as “alive”, with n equaling 617 and 206, respectively]) and 2) patients who died (classed as “deceased”, with n equaling 19 and 7, respectively]). At first, the sets were imbalanced in regard to the number of surviving vs. nonsurviving patients. Different oversampling techniques (section 2.3) were applied to obtain a balanced representation of both classes. A feature ranking was performed with our dataset to identify the most relevant features related to the in-hospital-mortality outcome (section 2.4). Feature ranking is the process of finding a subset of relevant features that can be used to develop efficient learning models. Finally, five different ML classification algorithms (section 2.5) were trained and validated with the training and testing sets, respectively.

2.3. Preprocessing and sampling

Using functions also developed in Python, the raw NIS data were preprocessed to conform to the defined format, enabling their input into our script. A function to delete patients with missing values was included, along with a 1-hot-encoding function to transform categorical variables into numeric inputs.

Data imbalance is commonly encountered in healthcare datasets. In this study, the most frequent class (alive) represented 97% (n = 823) of the sample size, while the least frequent class (deceased) was 3% (n = 26). To tackle this problem, two different sampling methods, oversampling and undersampling, were applied to the training set. The oversampling technique SMOTE (Synthetic Minority Over-sampling Technique) [16] was used on the deceased class to increase the number of cases to 88, while a random undersampling (RUS) technique [17] was applied to the alive class to reduce the number of patients to 126. The final obtained training set was better balanced, with 58% of the patients in the alive class and 42% in the deceased class.

2.4. Feature ranking

Feature ranking is commonly employed in prediction modeling to reduce dimensionality and problem complexity [18]. In this study, feature ranking was performed with the aim of deploying a model with the most relevant variables that would be useful in a clinical scenario. We used ridge regression analysis to rank variables according to their computed weights [19]. After sorting the variables by weights, six training subsets were generated including the top 5, 10, 15, 20, 25, and 30 most relevant variables, respectively.

2.5. Model training and evaluation

Five ML algorithms were implemented in our study: random forest (RF) [20], logistic regression (LR) [21], support-vector machine (SVM) [22], naïve Bayes (NB) [23], and artificial neural network (ANN) [24]. The class to predict was deceased, representing in-hospital mortality after TMVR. The algorithms were iteratively trained with the six training subsets described above. Their performances were compared using the testing set to compute the metrics accuracy, sensitivity, specificity, f1-score, and area under the receiver-operating curve (AUC). AUC was the metric used to determine the best algorithm and generate the final model.

2.6. Statistical analysis

Continuous variables were compared using the 2-tailed Student’s t-test, whereas the chi-square or Fisher’s exact test was used for categorical data, whichever was appropriate. The performance of each ML prediction algorithm was assessed by measuring each algorithm’s AUC. A non-parametric DeLong test was used to compare the AUCs between models [25]. Statistical analyses were performed with the use of R version 3.5.1 GUI 1.70 (R Core Team, 2018), and values of p that were less than 0.05 were considered statistically significant.

RESULTS

Baseline characteristics

Out of the 849 patients included in our study, 26 (3.1%) did not survive hospitalization. Overall, no significant differences in baseline characteristics were observed between the development and validation cohorts (Table 1). The mean age of the overall cohort was 74.8 (± 12.5) years, and 44.3% were females. Caucasians were the most prevalent ethnicity (77.5%), with Hispanics and African Americans accounting for 8.0% and 6.2%, respectively. Medicare beneficiaries (80.3%) comprised the majority of the patients, with equal distribution across each quartile of household income. Prevalence of traditional cardiovascular risk factors such as hyperlipidemia, known coronary artery disease (CAD), prior myocardial infarction, being a smoker, diabetes, and hypertension were comparable between survivors and non-survivors.

Table 1.

Baseline characteristics of TMVR patients.

Variables	Overall n = 849	Development cohortn = 636	Validation cohortn = 213	P-value
Age	74.8 ± 12.5	74.6 ± 12.6	75.2 ± 12.2	.598
Female	376 (44.3)	276 (43.4)	100 (47.0)	.410
Race
Caucasian	658 (77.5)	485 (76.3)	173 (81.2)	.160
African American	53 (6.2)	39 (6.1)	14 (6.6)	.947
Hispanic	68 (8.0)	56 (8.8)	12 (5.6)	.184
Underweight	§	§	§	.435
Smoking	263 (31.0)	209 (32.9)	54 (25.4)	.049
Hyperlipidemia	466 (54.9)	337 (53.0)	129 (60.6)	.065
Known CAD	524 (61.7)	381 (60.0)	143 (67.1)	.070
Prior MI	124 (14.6)	102 (16.0)	22 (10.3)	.054
Prior PCI	132 (15.6)	103 (16.2)	29 (13.6)	.430
Prior CABG	208 (24.5)	149 (23.4)	59 (27.7)	.245
Prior TIA/Stroke	130 (15.3)	95 (14.9)	35 (16.4)	.679
Atrial fibrillation	488 (57.5)	364 (57.2)	124 (58.2)	.864
Acute heart failure	14 (2.0)	11 (2.0)	§	.994
Carotid artery disease	17 (2.0)	§	§	.206
Prior ICD	91 (10.7)	65 (10.2)	26 (12.2)	.495
Prior PPM	100 (11.8)	72 (11.3)	28 (13.2)	.828
Pulmonary hypertension	281 (33.1)	202 (31.8)	79 (37.1)	.178
ESRD	30 (3.5)	21 (3.3)	§	.676
Elective admission	623 (73.4)			.299
Primary payer
Medicare	682 (80.3)			.750
Medicaid	36 (4.2)	25 (3.9)	11 (5.2)	.564
Private insurance	106 (12.5)	79 (12.4)	27 (12.7)	.999
Income
Quartile 1	185 (21.8)	144 (22.6)	41 (19.3)	.346
Quartile 2	188 (22.1)	136 (21.4)	52 (24.4)	.409
Quartile 3	243 (28.6)	176 (27.7)	67 (31.5)	.332
Quartile 4	233 (27.4)	180 (28.3)	53 (24.9)	.379
Elixhauser comorbidities
Hypertension	608 (71.6)	454 (71.4)	154 (72.3)	.866
Obesity	73 (8.6)	54 (8.5)	19 (8.9)	.958
Peripheral vascular disease	103 (12.1)	71 (11.2)	32 (15.0)	.170
Chronic kidney disease	294 (34.6)	216 (34.0)	78 (36.6)	.534
Chronic pulmonary disease	230 (27.1)	168 (26.4)	62 (29.1)	.499
Liver disease	18 (2.1)	§	§	.268
Alcohol abuse	§	§	§	.460
Anemia	§	§	§	.678
Diabetes mellitus	213 (25.1)	151 (23.7)	62 (29.1)	.141

Data are presented as mean ± SD or n (%). Abbreviations: CAD, coronary artery disease; MI, myocardial infarction; PCI, percutaneous coronary intervention; CABG, coronary artery bypass graft; TIA, transient ischemic attack; ICD, implantable cardioverter-defibrillator; PPM, permanent pacemaker; ESRD, end-stage renal disease. ^§cells with tabulated data ≤ 10. The AHRQ confidentiality statute prohibits the use of HCUP data to identify any person (including but not limited to patients and physicians as well as other healthcare providers). The use of HCUP databases to identify any person constitutes a violation of this agreement and may constitute a violation of the AHRQ confidentiality statute and the HIPAA Privacy Rule. This agreement prohibits data recipients from releasing, disclosing, publishing, or presenting any individually identifying information obtained under its terms. The AHRQ omits from the dataset all direct identifiers that are required to be excluded from limited datasets as consistent with the HIPAA Privacy Rule. The AHRQ and the data recipient(s) acknowledge that it may be possible for a data recipient, through deliberate technical analysis of the datasets and with outside information, to attempt to ascertain the identity of particular persons. The risk of individual identification of persons is increased when an observation (i.e., individual discharge records) in any given cell of tabulated data is ≤ 10. This agreement expressly prohibits any attempt to identify individuals, including by the use of vulnerability analysis or penetration testing [11].

Predictor model

Of the five different ML methods (RF, NB, ANN, LR, and SVM), the NB model for 15 variables had the highest performance in terms of discriminating between survival and in-hospital mortality, with an AUC of 0.83 (95% CI, 0.80–0.87) (Figure 2). Also performing well with 15 variables were LR [AUC: 0.77 (95% CI, 0.58–0.95)], ANN [AUC: 0.73 (95% CI, 0.55–0.91)], and both RF and SVM [AUCs: 0.67 (95% CIs, 0.47–0.87)]. Interestingly, the RF and LR models obtained for 10 variables were as good as the best NB model obtained in the study, both of which former had AUCs of 0.82 (95% CI, 0.79–0.86; p = 0.34) (Table 2).

Figure 2.

Operating characteristic area under the curve (AUC) by machine-learning model.

RF, random forest; NB, naïve Bayes; ANN, artificial neural network; LR, logistic regression; SVM, support-vector machine.

Table 2.

Predictive performance of TMVR in-hospital mortality models by number of variables.^*

Number of variables	Random Forest	Naїve Bayes	Artificial Neural Network	Logistic Regression	Support-Vector Machine
5	0.50 (0.49–0.50)	0.57 (0.53–0.59)	0.50 (0.49–0.51)	0.57 (0.43–0.71)	0.57 (0.43–0.71)
10	0.82 (0.79–0.86)	0.78 (0.75–0.81)	0.81 (0.77–0.84)	0.82 (0.79–0.86)	0.81 (0.77–0.84)
15	0.67 (0.47–0.87)	0.83 (0.80–0.86)	0.73 (0.55–0.91)	0.77 (0.58–0.95)	0.67 (0.47–0.87)
20	0.61 (0.41–0.81)	0.74 (0.60–0.88)	0.74 (0.56–0.92)	0.71 (0.51–0.91)	0.61 (0.41–0.81)
25	0.47 (0.33–0.62)	0.72 (0.58–0.87)	0.71 (0.51–0.91)	0.71 (0.51–0.91)	0.63 (0.43–0.83)
30	0.65 (0.45–0.85)	0. 68 (0.49–0.86)	0.72 (0.52–0.92)	0.71 (0.51–0.91)	0.65 (0.45–0.85)

^*Data are presented as AUC (95% Confidence Interval)

Means of the variable rankings were obtained for the best three models (NB for 15 variables and both RF and LR for 10 variables) (based on performance and with a higher rank attributable to a stronger relationship to the prediction of in-hospital mortality). CAD, chronic kidney disease, being smoker, prior percutaneous coronary intervention, prior coronary artery bypass grafting, heart failure, peripheral vascular disease, obesity, alcohol abuse, and liver disease were the top 10 ranked variables for these models, in descending order of relative importance (Figure 3).

Figure 3.

Feature importance according to the mean ranking for the three best models. *

*The variable’s importance is inversely proportional to the rank. The X-axis depicts the top 10 variables, ordered according to mean ranking.

DISCUSSION

To the best of our knowledge, this the first study to use ML to predict outcomes in patients undergoing TMVR. The key findings from our study are that a) predictive models based on ML algorithms can aid in the prediction of in-hospital mortality with a good performance (AUC = 0.83) b) CAD, chronic kidney disease, and being smoker are the top three contributors to the accurate prediction of in-hospital mortality.

Our results are compelling, particularly for the field of structural valve intervention. We have generated the only available prediction model for in-hospital mortality after TMVR. According to recent reports, ML algorithms have a superior predictive value when it comes to detecting the potential for adverse outcomes after transcatheter valve replacement surgery [7, 8], but its application to TMVR outcomes have not been previously assessed. With the recent expansion of TMVR indications to include patients with heart failure and moderate to severe or severe secondary MR [4], it is expected that TMVR utilization will undoubtedly become a therapeutic option for many other patients. In fact, the use of TMVR was already increasing even before the addition of this latter indication [26]. It is for this reason that the need to generate predictive models for both in-hospital mortality and heart failure rehospitalization has become critical. Since the NIS database lacks heart failure readmission data, we focused our study on in-hospital mortality. The best predictive model in our study was generated by NB (AUC = 0.83). Interestingly, LR yielded the best NIS-TAVR score, which our group obtained using data from the NIS database [7]. Even though both studies used the same data source and primary outcome, changes in the study design as well as data preprocessing and sampling led to the NB model becoming the best classifier for our dataset. This could be explained by the fact that ML relies heavily on data structure and patterns. Additional insights into the application of ML in predicting adverse outcomes for structural valve interventions have been previously reported by our group [7, 27].

A real-world data analysis from the Society of Thoracic Surgery/American College of Cardiology transcatheter valve therapy registry reporting the outcomes of 2,952 patients undergoing TMVR showed an in-hospital mortality of 2.7%, similar to the 3.1% found in our study [28]. Remarkably, some similarities were noted between the two studies. Being a smoker was a significant predictor of mortality in our study, and, likewise, moderate to severe lung disease was similarly predictive of mortality in the STS/ACC TVT registry, after adjusting for confounding variables. Moreover, CAD was an important predictor of in-hospital mortality in our study. In the STS/ACC TVT Registry, lower baseline left ventricular ejection fraction was associated with mortality or rehospitalization for heart failure [28]. Since ischemic cardiomyopathy is the most common cause of left ventricular systolic dysfunction, the inclusion of CAD in our model may be considered an indirect representation of this variable.

Multiple prior studies have reported that baseline renal dysfunction is a predictor of adverse outcomes in patients undergoing TMVR [28, 29]. Though only 3.3% of the patients in the initial EVEREST II trial were found to have suffered from renal dysfunction, other US and European registries have reported higher prevalence (~30–65%) of patients with renal dysfunction [30–32]. Even in our study, 34.5% of the patients had renal disease, overall. This contrast in trial studies compared to real-world populations has been notable and has significantly affected outcomes. According to the STS/ACC TVT Registry, dialysis patients had the highest hazard ratio (2.19; 95% CI, 1.28–3.74) in terms of predicting 1-year mortality [28]. A prior comparison of TMVR vs. surgical mitral valve repair/replacement in patients with advanced renal disease showed that TMVR is a feasible option in such patients, though the former (TMVR) was performed on only a small group of patients [33]. An additional analysis from the STS/ACC TVT Registry showed a proportionally steep increase in in-hospital mortality, with increasing renal dysfunction [29]. Thus, it is not surprising, that the presence of chronic kidney disease was the second most predictive variable in our study, given this condition’s previously established impact on TMVR outcomes.

Unlike the Society for Thoracic Surgery Predicted Risk of Mortality score for surgical mitral valve repair/replacement, no current predictive model exists for TMVR. With the expanding indications for TMVR, it is imperative to have a predictive model of risk to stratify patients and identify those who would not be likely to benefit from the procedure. Thus, our early model will inform decision-making and will set the stage for further complex predictive models, which would include variables pertaining to valve anatomy and hemodynamics to improve accuracy.

Although our results are encouraging, clinical applications of ML in structural valve intervention are still premature. Data acquisition and interpretation, time and resources, and high error susceptibility have been highlighted as potential caveats for this novel field. Nonetheless, no matter how the data are processed or analyzed, decision regarding whether to consider a patient as appropriate for TMVR would still be made by the multidisciplinary heart team. Moreover, before fully implementing ML for TMVR, better granular data (with either VARC-2 or MVARC outcomes) is vital if we are to obtain more clinically reliable models. As interventionalists continue to gather significant amounts of patient data through imaging interpretation and electronic medical records, artificial intelligence (AI) is likely to become an essential tool for clinical decision-making, in both the medical office and catheterization laboratory. The unique nature of interventional cardiology makes it the perfect platform to further drive AI to the cutting edge of new innovations and transformations for the future interventionalist.

While the strength of this project lies in the real-world population-based information and the novel application of ML methods, some limitations need to be acknowledged. First, the patients that did not survive hospitalization were underrepresented, which may have introduced some bias. However, an over/undersampling techniques were applied to obtain a balanced dataset and make more reliable predictions. Second, the analyzed data correspond to the early stages of TMVR implementation in the US (2012–2015). Increased operator’s experience inherent to a learning curve along with higher institutional patient volume may contribute to bettering current procedural success and reducing complications. Moreover, our study period paralleled the approval of TMVR for primary MR in 2013 by the US FDA and is predominantly reflective of outcomes in patients with primary MR and prohibitive surgical risk. Additionally, information on STS scores, cause of death, the severity and etiology of MR (degenerative vs. functional), and other key prognostic data that are relevant to the success of the procedure, such as echocardiographic descriptions of the mitral apparatus (e.g., mitral leaflet calcification, leaflet tethering, mitral annular calcification, and mitral gradient), were not available in the NIS database. Thus, we could not determine the impact of these variables on in-hospital mortality after TMVR. On the other hand, the NIS is an observational cohort that uses ICD-9-CM and procedure codes, which raises the risk of coding errors. However, a low error rate within this dataset has been previously reported [34]. Finally, due to the retrospective nature of the analysis, the type of data-collection, lack of homogeneous definitions for the included variables as well as pending external validation, the generated model cannot be proposed for clinical use to date.

CONCLUSION

Using a data mining approach from a nationwide sample, we developed a ML-derived model to predict in-hospital mortality in patients undergoing TMVR. This model is promising and deserves rigorous clinical validation. Additional studies with contemporary and more granular data are warranted to improve the model’s discriminatory performance and applicability. In a digital era in which risk stratification is critical for decision-making, automated prediction algorithms could become the perfect approach to developing more precise and generalizable risk-assessment tools for structural valve interventions.

HIGHLIGHTS.

• The first available model for predicting in-hospital mortality after transcatheter mitral valve repair was developed using a machine learning approach.

• Coronary artery disease, chronic kidney disease, and being smoker are important predictors of in-hospital mortality after transcatheter mitral valve repair.

• The machine learning–based prediction model might be helpful in identifying patients at high risk of mortality after transcatheter mitral valve repair.

• Machine learning-based algorithms should be integrated into the shared decision-making process for mitral valve interventions.