A Machine Learning Risk Score Predicts Mortality Across the Spectrum of Left Ventricular Ejection Fraction

Abstract

Background:

Heart failure (HF) guideline recommendations categorize patients according to left ventricular ejection (LVEF). Mortality risk, however, varies considerably within each category and the likelihood of death in an individual patient is often uncertain. Accurate assessment of mortality risk is an important component in the decision-making process for many therapies. In this report, we assess the accuracy of MARKER-HF, a recently described machine learning based risk score, in predicting mortality of patients in the three guideline defined HF categories and its ability to distinguish risk of death for patients within each category.

Methods and Results:

MARKER-HF was used to calculate mortality risk in a hospital based cohort of 4064 patients categorized into groups with reduced LVEF (HFrEF), mid-range LVEF (HFmrEF) or preserved LVEF (HFpEF).

MARKER-HF was substantially more accurate than LVEF in predicting mortality and was highly accurate in all three HF categories, with c-statistics ranging between 0.83 to 0.89. Moreover, MARKER-HF accurately discriminated between patients at high, intermediate and low levels of mortality risk within each of the three categories of HF used by guidelines.

Conclusions:

MARKER-HF accurately predicts mortality in patients within the three categories of HF used in guidelines for management recommendations and it discriminates between magnitude of risk of patients in each category. MARKER-HF mortality risk prediction should be helpful to providers in making recommendations regarding the advisability of therapies designed to mitigate this risk, particularly when they are costly or associated with adverse events, and for patients and their families in making future plans.

Graphical Abstract

Background

Guideline recommendations for heart failure (HF) categorize patients according to whether their left ventricular ejection fraction (LVEF) is reduced, mid-range or preserved (HFrEF, HFmrEF and HFpEF, respectively)^1,2. This categorization is based largely on concepts of pathophysiology that were incorporated into entry criteria for the pivotal HF trials upon which guideline recommendations are based. However, LVEF is an unreliable predictor of a patients clinical course and mortality risk varies widely within each category^3–5. An individual patient’s risk of dying is vitally important information for making therapeutic decisions, particularly for treatments that improve survival but are costly or expose patients to serious complications. An accurate means of predicting mortality risk for patients within each of the HF categories would of considerable value to providers and patients, alike. We have reported that a machine learning based risk score, Machine learning Assessment of RisK and EaRly mortality in Heart Failure (MARKER-HF), predicts mortality across the full spectrum of risk in patients with HF⁶. The predictive accuracy of MARKER-HF in patients with HFrEF, HFmrEF and HFpEF and its ability to discriminate between levels of risk within these categories, however, has not previously been assessed.

Aims

We assessed the accuracy of MARKER-HF⁶ to predict mortality and distinguish varying levels of risk for patients with HFrEF, HFmrEF and HFpEF.

Methods

This study conforms with principles outlined in the “Declaration of Helsinki”⁷ and was approved by the University of California, San Diego (UCSD) Health Institutional Review Board.

Details of the derivation and validation of MARKER-HF have been published⁶. The score is comprised of eight readily accessible variables (diastolic blood pressure, creatinine, blood urea nitrogen, hemoglobin, white blood cell count, platelets, albumin, and red blood cell distribution width [RDW]). It is available online at https://marker-hf.ucsd.edu.

Patients with a diagnosis of HF based on ICD codes (shown in the Supplement) were identified from their UCSD electronic health record during the period between 2006 and 2017. The first ICD code mention of HF from either in- or out-patient contact defined index time entry in the data base. Patients ≥81 years at the time of index event or who had an internal cardiac defibrillator (ICD) or mechanical circulatory support (MCS) device in place, evidence of sepsis, died within 7 days of the index event, obvious data base errors, or were missing one of the eight MARKER-HF variables were excluded. Echocardiographic measurement of LVEF obtained 30 +/− days from the index time was required for entry. Further detail regarding derivation of the population is included in the Supplement.

Statistical analysis

Patients were categorized as having HFpEF when LVEF was ≥50%, HFmrEF when LVEF was ≥40% and <50 and HFrEF when LVEF was <40%. MARKER-HF values for low, intermediate and high risk were defined as <−0.2, ≥−0.2 to <+0.2 and ≥0.2, respectively. Kaplan-Meier (KM) survival curves were compared by Log-Rank test. Receiver Operating Curves (ROCs) were compared using the Venkatraman test. Confidence interval for Areas Under the Curve (AUCs) were obtained by the De Long method. Analyses were performed using Python lifelines V0.24.3 and R pROC V1.14.0 packages. A two-tailed p-value of ≤ 0.05 was considered statistically significant.

Results

Characteristics of the 4064 patients studied are summarized in Table 1. Figure 1, panel A shows the distribution of their LVEF values. Figure 1, Panel B demonstrates a similar distribution of MARKER-HF scores in patients with HFrEF (n=782; 19.2%), HFmrEF (n=484; 11.9%) and HFpEF (n=2798; 68.8%) and Panel C shows similar a association of the scores with probability of survival in the three categories. As depicted in Figure 1, Panel D no significant differences between the KM mortality curves for patients in each of the EF categories were detected. The c-statistic for LVEF prediction of mortality was 0.52 [95% CI; 0.48–0.56].

Table 1:

Baseline patient characteristics

Cohort size:	(n=4064)
Demographic variables and LVEF
Age, years	59 (13)
Gender, % female	58
Ethnicity, %	55-White, 11-African-American, 24-Hispanic, 7-Asian, 3-Other
LVEF, %	55 (17)
MARKER-HF covariates
Diastolic Blood Pressure, mm Hg	70 (14)
BUN, mg/dL	23 (14)
Creatinine, mg/dL	1.2 (0.7)
Hemoglobin, g/dL	11.3 (2.6)
White Blood Cell Count, × 103/L	9.6 (6.0)
Platelet Count, ×103/L	211 (108)
Albumin, g/dL	3.6 (0.7)
RDW, %	15.4 (2.5)

Mean and standard deviation of variables is given. Abbreviations: left ventricular ejection fraction, LVEF; Machine learning Assessment of RisK and EaRly mortality in Heart Failure, MARKER-HF; blood urea nitrogen, BUN, red blood cell distribution width, RDW.

Figure 1, Panel A. Distribution of LVEF values. Panel B. MARKER-HF scores in the study population for each HF category, Panel C. MARKER-HF and one year survival for each HF category and Panel D. Kaplan Meier survival curves for each HF category.

There were no statistically significant differences in MARKER-HF distributions or in the survival curves for the three groups. In Panels A B, and C horizontal bars depict range of scores, vertical bars indicate 1 SD. Abbreviations: Left ventricular ejection fraction, LVEF; Heart failure with reduced ejection fraction, HFrEF; Heart failure with mid-range ejection fraction, HFmrEF; Heart failure with preserved ejection fraction, HFpEF; RMS, root mean square.

As shown in Figure 2, Panel A, MARKER-HF demonstrated a high degree of accuracy in predicting mortality in each HF category. The c-statistics for the ROCs were 0.88 [95% CI; 0.83–0.93], 0.83 [95% CI; 0.76–0.91] and 0.85 [95% CI; 0.82–0.88] respectively, for mortality prediction in patients with HFrEF, HFmrEF and HFpEF. Pairwise comparison showed no significant differences between the curves.

Figure 2. MARKER-HF prediction of mortality risk within HF categories.

Panel A depicts the ROC for MARKER-HF in the three groups. Panels B, C, and D depict Kaplan-Meier curves of survival probability for three ranges of MARKER-HF low (L), high (H), and medium (M) risk levels for HFrEF, HFmrEF, and HFpEF patients. The bands surrounding each curve represent 95% confidence intervals. Abbreviations: Same as in Figure 1.

As shown in Figures 2, Panels B, C and D, the MARKER-HF distinguishes between patients at high, intermediate and low mortality risk within each HF guideline category. Although the number of patients in the high-risk MARKER-HF category is not very large, particularly in the HFrEF and HFmrEF cohorts, as can be inferred from the widths of the 95% CI bands, a clear separation between the curves is seen.

Conclusions

Guideline recommendations for HF management categorize patients according to their LVEF^1,2. Broad application of therapies to all patients within categories, however, is not optimal as the risk to benefit ratio of a specific therapy for an individual may be uncertain. For treatments that reduce mortality, accurate assessment of a patients risk of death provides an important context for making therapeutic decisions. We have previously reported on the derivation and validation of MARKER-HF⁶. The risk score was shown to accurately predict mortality across gender, race, in- or out-patient setting and HF severity and its performance has been validated in independent populations⁶. We now demonstrate that MARKER-HF predicts mortality and clearly discriminates between levels of risk within each of the HF categories used for guideline recommendations.

Accurate prediction of mortality risk provides important information for making therapeutic decisions for treatments designed to improve survival, particularly when they are costly or associated with serious side effects. Thus, MARKER-HF should be of considerable value to healthcare providers and patients alike in deciding whether or not to pursue such therapies. The variables used to calculate MARKER-HF are inexpensive and readily available. An online website has made this risk score accessible to providers and MARKER-HF results could easily be incorporated into the decision making process in a variety of settings. For instance, the most compelling rationale for therapies such as cardiac transplantation and MCS is improved survival of patients with end-stage disease^8,9. Both treatments, however, are costly and involve an arduous clinical course for patients and their families. Knowledge of a patient’s mortality risk offers an important perspective for both providers and patients when considering these therapies. Another potential use is with decisions regarding ICD placement which is discouraged by guidelines when survival is anticipated to be less than a year¹⁰. MARKER-HF could be helpful when there is uncertainty about whether this is the case for an individual patient.

Risk prediction, no matter how accurate, does not replace clinical judgement. Rather, it adds information that can be integrated into the decision-making process. Although MARKER-HF does not discriminate between cardiovascular and non-cardiovascular causes of death, when used in concert with clinical and other data, accurate prediction of mortality with this risk score could help providers determine the advisability of various therapies and patients and their families prepare for the future.

Abbreviations:

HF

Heart failure

LVEF

Left ventricular ejection fraction

HFrEF

Heart failure with reduced ejection fraction

HFmrEF

Heart failure with mid-range ejection fraction

HFpEF

Heart failure with preserved ejection fraction

MARKER-HF

Machine learning Assessment of RisK and EaRly mortality in Heart Failure

EHR

Electronic health record

ICD

Internal cardioverter-defibrillator

KM

Kaplan-Meier

AUC

Area under the curve