Machine learning-based risk stratification for mortality in patients with severe aortic regurgitation

Abstract

Aims

The current guidelines recommend aortic valve intervention in patients with severe aortic regurgitation (AR) with the onset of symptoms, left ventricular enlargement, or systolic dysfunction. Recent studies have suggested that we might be missing the window of early intervention in a significant number of patients by following the guidelines.

Methods and results

The overarching goal was to determine if machine learning (ML)-based algorithms could be trained to identify patients at risk for death from AR independent of aortic valve replacement (AVR). Models were trained with five-fold cross-validation on a dataset of 1035 patients, and performance was reported on an independent dataset of 207 patients. Optimal predictive performance was observed with a conditional random survival forest model. A subset of 19/41 variables was selected for inclusion in the final model. Variable selection was performed with 10-fold cross-validation using random survival forest model. The top variables included were age, body surface area, body mass index, diastolic blood pressure, New York Heart Association class, AVR, comorbidities, ejection fraction, end-diastolic volume, and end-systolic dimension, and the relative variable importance averaged across five splits of cross-validation in each repeat were evaluated. The concordance index for predicting survival of the best-performing model was 0.84 at 1 year, 0.86 at 2 years, and 0.87 overall, respectively.

Conclusion

Using common echocardiographic parameters and patient characteristics, we successfully trained multiple ML models to predict survival in patients with severe AR. This technique could be applied to identify high-risk patients who would benefit from early intervention, thereby improving patient outcomes.

Graphical Abstract

Graphical Abstract.

Overview of the data collection, model processing, and validation [conditional survival forest plot shown for validation in test sample (20%)].

Introduction

Aortic regurgitation (AR) is a common valvular lesion associated with pressure and volume overload.¹ It can be well tolerated for years the before development of symptoms.^2–4 The current guidelines recommend intervention (repair or replacement) with onset of symptoms, significant left ventricular (LV) enlargement, or LV systolic dysfunction (ejection fraction <55%).⁵ The guidelines are based on small old studies conducted in the 1980s–90s.^6–12 There has been substantial advancement in surgical technique and the quality and durability of artificial valves since then; as a result, operative mortality and post-operative complications are significantly reduced.^13,14 Recent studies have suggested that we might be missing the window of early intervention in a significant number of patients by following the guidelines;^15–17 as a result, many patients have continued systolic dysfunction and higher adjusted mortality even after surgery.^18,19 We evaluated a set of machine learning (ML)-based algorithms to predict mortality in patients undergoing echocardiographic evaluation for moderate-to-severe and severe AR.

Methods

This study was approved by the institutional review board with a waiver of informed consent. Additionally, patients who had declined authorization to participate in research in Minnesota were also excluded. A total of 1100 patients with chronic moderate-to-severe and severe AR who underwent echocardiography at Mayo Clinic between 2004 and 2019 was included. The exclusion criteria were acute AR, acute infective endocarditis, prior valve repair, other valve lesions, or cardiomyopathy associated with LV enlargement or dysfunction. Of 1100 patients, 65 patients who had greater than 20% of echocardiographic variables missing were excluded, leaving 1035 in the final analysis.

Patient characteristics included in the dataset included demographics, New York Heart Association (NYHA) functional class, symptoms at baseline, systolic and diastolic blood pressure (BP), medical history (coronary artery disease, prior myocardial infarction, and prior coronary artery bypass graft procedure), congestive heart failure (CHF), history of prior malignancy, lung disease, hypertension, hyperlipidaemia, endocarditis, diabetes mellitus, Charlson comorbidity index, aortic valve replacement (AVR) surgery, and mortality. The relevant echocardiographic variables included the following: measures of LV size [end-diastolic dimension (LVEDD), end-systolic dimensions (LVESD), end-diastolic volume (LVEDV), and end-systolic volume (LVESV), including those indexed to body surface area (BSA)], LV ejection fraction, aortic valve morphology, sinus of Valsalva and ascending aorta linear dimensions, variables associated with AR assessment (effective regurgitant orifice area, regurgitant volume, and vena contracta), and variables included in diastolic function assessment [mitral annulus early diastolic tissue Doppler velocity (e′), ratio of mitral early diastolic inflow and tissue Doppler velocity (E/e′), and pulmonary artery systolic pressure].

Outcomes and analysis

Our hypothesis was that ML-based models can predict outcomes in patients with chronic severe AR with good discrimination as measured by the concordance index. The primary outcome was the time to all-cause mortality. For the primary analysis, patients were censored at the last follow-up record or 31 December 2019, whichever came first. For a secondary analysis of mortality under medical management, patients were censored at the time of AVR.

Data pre-processing

We randomly split 80% of the dataset to be training and validation and set aside 20% as the test dataset which was used to report results after model selection and hyperparameter tuning (results are reported only on the test dataset, which was not seen during the training process). No two patients had data in both the training or test datasets. Missing variables were imputed using multiple imputations by chained equations (MICE)²⁰ implemented by the Python library statsmodels allowing for 20 iterations to achieve stable convergence.²¹ Training and testing datasets were imputed separately to prevent information leakage. Categorical variables were encoded as binary, except for NYHA functional class, which was encoded as 1, 2, 3, or 4. Normalization was not used as a pre-processing step for any of the variables.

Since future valve replacement is unknown at baseline, in our training dataset, AVR was set to zero unless AVR occurred within 100 days after the index echocardiogram, with the assumption that an immediate AVR was likely known at baseline.

Feature selection

Feature selection was utilized as an additional pre-processing step to minimize the impact of overfitting. A random survival forest (RSF) model was utilized to reduce the number of candidate features due to the ease of generating variable importance metrics from tree-based metrics. Features were pre-selected with 10-fold cross-validation with 500 trees. From each fold, we selected the top 20 features based on feature representation in the 500 trees. Features represented in the top 20 in at least 6- of 10-folds were included in the final predictive dataset.

Modelling

Five-fold cross-validation training scheme was used where each model was trained on 80% of the training dataset (after the test split had been removed) and validated on the remaining 20%. A total of 12 model architectures were evaluated, including several Cox proportional hazard regression models, support vector machines (SVMs), linear multi-task logistic regression (LMTLR), neural multi-task logistic regression (NMTLR), and several forest-based models. Modelling was performed with open-source Python framework lifelines v0.26.0²² and PySurvival 0.1.2.²³ Tree-based gradient boosting methods were implemented with scikit-survival v0.15.0²⁴; hyperparameter tuning was used with the PySurvival package to optimize the top-performing example of each ML model in each family. Cox proportional hazard models were optimized for step size, penalty term, and, in the case of ElasticNet, the L1 penalizer value. Survival forests were optimized for maximum features at each node and minimum samples per leaf node. LMTLR was optimized for learning rate and bin count. NMTLR was optimized for one of the several architecture designs as well as learning rate and presence of dropout. Linear SVMs were optimized for the learning rate and L2 penalization value. Tree-based gradient boosting was optimized for the learning rate, sample rate, and presence of dropout. All models were optimized using loss Cox partial likelihood function as discrimination performance was measured by the concordance index (c-index).

Statistical analysis

Model performance was evaluated on the test dataset using the concordance index by averaging performance of the five independently trained models (trained independently on five cross-validation splits). For the top-performing models, the receiver operating characteristic (ROC) curve was computed at 1 year, 2 years, and death at any time between model predictions and actual survival. In a survival forest model, feature importance was evaluated in a permutation-based fashion. For a feature included in the model, the out-of-bag error was computed such that when this feature was encountered, a daughter node was assigned randomly. The feature importance was simply the difference between the original out-of-bag error and the error as calculated above.²⁵ On the other hand, the feature importance in a Cox proportional hazard model was measured by the negative binary logarithm of the P-values of each variable.

Results

A total of 1035 patients were included in the final analyses. The mean age of the patients was 60 ± 17 years; 187 (18%) were females. The patient and echocardiographic characteristics are presented in Table 1. During a median follow-up of 5.1 (IQR: 2.0–9.9) years, 208 patients died, and 518 underwent AVR.

Table 1.

Baseline clinical and echocardiographic characteristics (n = 1035)

Variable	Value
Age, years	60 ± 17
Gender, female	187 (18)
BMI, kg/m2	28 ± 5
BSA, m2	2.0 ± 0.2
SBP, mm Hg (n = 1034)	131 ± 20
DBP, mm Hg (n = 1034)	64 ± 13
Diabetes	109 (11)
Hypertension	494 (48)
Chronic lung disease	99 (10)
Hyperlipidaemia	392 (38)
Charlson comorbidity index (n = 1033)	1.6 ± 2.2
NYHA functional class (n = 1016)
ȃI	637 (63)
ȃII	265 (26)
ȃIII	106 (10)
ȃIV	8 (1)
Bicuspid aortic valve morphology (n = 1029)	366 (36)
LV EF, %	58 ± 9
LVEDD, mm (n = 1033)	60 ± 7
Indexed LVEDD, mm/m2 (n = 1033)	30 ± 4
LVESD, mm (n = 1018)	40 ± 7
Indexed LVESD, mm/m2 (n = 1018)	20 ± 4
LVEDV, mL	209 ± 67
Indexed LVEDV, mL/m2	104 ± 30
LVESV, mL	91 ± 41
Indexed LVESV, mL/m2	45 ± 20
Degree of AR
ȃModerate to severe	436 (42)
ȃSevere	599 (58)
Regurgitant volume, mL (n = 885)	71 ± 25
Effective regurgitant orifice area, mm2 (n = 832)	26 ± 14
Vena contracta, mm (n = 709)	6 ± 3
Medial E/e′ ratio (n = 962)	12 ± 24
RVSP, mm Hg (n = 785)	32 ± 10
Mid-ascending aorta, mm (n = 941)	41 ± 7
Mid-ascending aorta ≥ 45 mm	233 (25)
Mid-ascending aorta ≥ 50 mm	95 (10)
Mid-ascending aorta ≥ 55 mm	35 (4)
Sinus of Valsalva, mm (n = 971)	41 ± 6
Sinus of Valsalva ≥ 45 mm	229 (24)
Sinus of Valsalva ≥ 50 mm	53 (5)

Data are expressed as mean ± SD or number (percentage).

Abbreviations: BMI, body mass index; BSA, body surface area; SBP, systolic blood pressure; DBP, diastolic blood pressure; LVEF, left ventricular ejection fraction; LVEDD, left ventricular end-diastolic dimension; LVESD, left ventricular end-systolic dimension; LVEDV, left ventricular end-diastolic volume; LVESV, left ventricular end-systolic volume; AR, aortic regurgitation; TR, tricuspid regurgitation; E/e′, early mitral inflow/tissue Doppler velocity; RVSP, right ventricular systolic pressure.

The following features were selected for further modelling based on the feature selection pre-processing step using RSF: age, AVR, indexed LVESV, indexed LVEDV, BSA, EF, NYHA functional class, body mass index (BMI), diastolic BP, bicuspid aortic valve, aortic valve regurgitant volume, e′ velocity, echocardiographic correlate of LV filling pressures (E/e′), right ventricular systolic pressure (RVSP), mid-ascending aortic diameter, diagnosis of CHF, prior malignancy, chronic lung disease, and hypertension (Figure 1) (Graphical abstract).

Figure 1.

Features were pre-selected with 10-fold cross-validation with 500 trees. The features shown were represented in the top 20 in at least 6 of 10 folds and included in the final predictive dataset. Abbreviations: AVR: aortic valve replacement, LVEDV: left ventricular end-diastolic volume, LVESV: left ventricular end-systolic volume, BSA: body surface area, LVEF: left ventricular ejection fraction, BMI: body mass index, DBP; diastolic blood pressure, AR: aortic regurgitation, e′: mitral annulus early diastolic tissue Doppler velocity, E/e′: ratio of mitral early diastolic inflow and tissue Doppler velocity representing filling pressures, RVSP: right ventricular systolic pressure, CHF: congestive heart failure.

All-cause mortality

The mean concordance index of different models for primary outcome of all-cause mortality on both the validation and test datasets is presented in Table 2 and Figure 2. ElasticNet Cox regression ranked the highest among the Cox regression family, while Conditional RSF performed the best on the validation set among the Survival Forest family (concordance index: 0.79) and was selected for inclusion in an ensemble model. Overall, there were no appreciable or clinically relevant differences in the performance between models.

Table 2.

Mean c-indices of Cox and Survival Forest all-cause mortality models on validation and test sets

Model	Val/Ts	Mean c-index (standard error)
ElasticNet Cox regression	Test	0.81 (0.026)
	Validation	0.79 (0.008)
L1 regularized Cox regression	Test	0.77 (0.040)
	Validation	0.79 (0.012)
L2 regularized Cox regression	Test	0.79 (0.030)
	Validation	0.79 (0.008)
Standard Cox regression	Test	0.79 (0.034)
	Validation	0.79 (0.008)
Conditional survival forest	Test	0.81 (0.030)
	Validation	0.79 (0.007)
Extra survival trees	Test	0.80 (0.033)
	Validation	0.80 (0.008)
Random survival forest	Test	0.81 (0.025)
	Validation	0.79 (0.009)

Figure 2.

Concordance indices of Cox and survival forest all-cause mortality models on validation and test sets. The results represent 10 repeats of five-fold cross-validation.

The ROC curves and feature importance evaluated on the test dataset by the top-performing all-cause mortality ensemble models (ElasticNet Cox regression and conditional survival forest) are presented in Figure 3A and B. An ensemble model predicts the test risks by taking the average of predicted test risks on each of the five models in the five-fold cross-validation sets, which were used to compute the ROC curves. Feature importance on each of the five models was normalized and averaged, before being divided by the importance of age. The following variables were significant in both ElasticNet Cox regression and conditional survival forest models with slight differences in the relative importance: age, RVSP, prior malignancy, NYHA functional class, diagnosis of CHF, LVEF, regurgitant volume, AVR, BMI, mid-ascending aorta diameter, indexed LVEDV, BSA, chronic lung disease, diastolic BP, e′, E/e′, bicuspid aortic valve, hypertension, and indexed LVESV (Figure 4A and B and Supplementary material online, Figure S4). Age had the strongest association with mortality in both models; other top variables were RVSP, prior malignancy, NYHA functional class, diagnosis of CHF, E/e′, and LVEF.

Figure 3.

Receiver operating characteristic curves of top-performing all-cause mortality ensemble models for ElasticNet Cox regression (A) and conditional survival forest (B) on test set.

Figure 4.

Relative feature importance of top-performing all-cause mortality ensemble models for ElasticNet Cox regression (A) and conditional survival forest (B) on test set. The variables are presented in the order of importance with the highest on the top. Abbreviations: same as Figure 1.

All-cause mortality censored at AVR

All models had similar performance (mean concordance index: 0.76–0.79) (see Supplementary material online, Table S1, and Supplementary material online, Figure S1) for association with all-cause mortality censored at AVR. The AUC for L2 regularized Cox regression model and random survival forest model was similar (see Supplementary material online, Figure S2). The features of importance were similar to the ones associated with all-cause mortality with slight differences in relative importance between the two models: age, prior malignancy, RVSP, CHF, diastolic BP, NYHA functional class, and E/e′ (see Supplementary material online, Figure S3).

Discussion

Our study has several important findings: (1) we report that ML-based algorithms are able to predict mortality in patients with moderate-to-severe and severe AR; (2) the top variables included in our model associated with outcomes of mortality were age, RVSP, prior malignancy, NYHA functional class, diagnosis of CHF, E/e′, and LVEF; (3) we found Cox ElasticNet, which is a regular Cox model with both L1 (LASSO, or variable selection) and L2 (ridge, or general shrinkage of estimated coefficients) regularization applied while estimating the parameters, and RSF models, which are tree-based methods that do not estimate a parametric form for variables, to have strong overall performance. These complementary approaches provided consistently high discrimination, with a C-statistic of 0.81 on the best-performing model, although all models demonstrated similar performance.

AR is a common valvular lesion with an estimated prevalence of mild or higher grade of 12% in the men and women included in the Framingham Heart Study.²⁶ It is associated with volume and pressure load on the left ventricle which leads to enlargement and systolic dysfunction, followed by onset of symptoms.^1,4,7 The mortality rises with onset of symptoms, LV systolic dysfunction (LVEF <55%), and LV enlargement above the predefined thresholds (LVEDD > 65 mm, LVESD > 50 mm, and indexed LVESD > 25 mm/m²). Previous studies have shown an increase in mortality at a smaller threshold of indexed LV linear dimensions (iLVESD > 20 mm/m²) and better association of LV volumes than dimensions with symptoms and mortality.^17,27 The cut-offs defined in the guidelines are based on small old studies^7,9–12 when operative mortality was as high as 10%.²⁸ In recent years, with advancement of surgical techniques and newer generation artificial valves, operative mortality and post-operative complications have decreased, thereby necessitating newer ways to risk-stratify patients and identify those who would benefit from early intervention.^13,14,29

Many recent studies in cardiovascular medicine have shown excellent performance of ML-based algorithms to identify patients with heart failure and subclinical atrial fibrillation.^30–32 In a prospective study, an ML algorithm incorporating clinical and imaging-based variables such as computed tomography (CT) coronary calcium score significantly improved prediction of cardiovascular events compared to standard clinical risk assessment.³³ Similarly, an ML risk calculator in MESA cohort outperformed the ACC/AHA Risk Calculator by recommending less drug therapy and yet missing fewer events.³⁴ There are currently no studies using ML-based risk assessment in patients with valvular heart disease. Therefore, we sought to develop an ML-based algorithm that could predict mortality in patients with chronic severe AR. Our model identified the important clinical (age, BSA, NYHA class, prior malignancy, diagnosis of CHF, chronic lung disease, diastolic BP, hypertension, and aortic valve replacement) and echocardiographic variables (RVSP, indexed LVESV and LVEDV, LVEF, bicuspid valve, regurgitant volume, filling pressure, mid-ascending aorta diameter, mitral tissue early relaxation velocity, and filling pressures) with a predictive AUC value exceeding 0.84 on the test dataset. These factors have been shown to be associated with mortality in AR and other valvular lesions in retrospective analyses.^5,35–37 Age, NYHA functional class, diastolic BP, ejection fraction, and degree of regurgitation have been shown to be associated with mortality in recent studies including contemporary cohort of patients with AR.^15,27,38 Another large study including 1417 patients found higher RVSP to be significantly associated with long-term mortality; other significant factors were age, chronic kidney disease, prior cardiac surgery, symptoms, and LV size.¹⁴

We report ML-based algorithms for time-to-event outcomes and evaluated the performance of several ML models, observing the best performance with Cox ElasticNet and random survival forest models. It is critical to test multiple models to analyse heterogeneous high-dimensional data including clinical and imaging variables. Several models performed very well on both validation and test datasets, which reinforces the need for testing multiple models. Other studies have shown similar results particularly when feature selection is performed before performing Cox analysis.³⁹ Some advantages of Cox models include ease of performance and relative resistance to overfitting.

The limitation of our study includes retrospective single-centre analyses, and our results suggest the need for larger multicentre prospective studies to finalize and validate a single model (or ensemble of models) to predict patients at high risk of mortality with AR. Many of the important features identified through our modelling efforts may not be readily modifiable through interventions and in fact represent risk factors for near-term mortality in general. Nonetheless, models may be helpful in discussing the risks and benefits of further care and interventions such as AVR. In terms of the modelling, our primary objective was to narrow down the modelling space and bring our understanding of the risk factors associated with mortality in this population forward. Additional work leveraging both prospective data and data from other institutions will be needed to better define the operating characteristics of the candidate algorithms. Thus, our results provide proof-of-concept support to establish the feasibility of an ML approach and serve as the basis for future work.

In conclusion, our study reports the role of ML-based models including both clinical and echocardiographic variables in predicting all-cause mortality in patients with chronic severe AR. These findings suggest the future role of ML-based algorithms to identify high-risk patients after validation in future larger prospective studies.

Supplementary material

Supplementary material is available at European Heart Journal – Digital Health online.

Funding

This research was not funded by any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Data availability

The original patient data may be made available upon request to the corresponding author after review and discussion with the local IRB.