Skip to main content
NCBI home page
Journal of the American Heart Association: Cardiovascular and Cerebrovascular Disease logoLink to Journal of the American Heart Association: Cardiovascular and Cerebrovascular Disease
. 2023 Sep 30;12(19):e029736. doi: 10.1161/JAHA.122.029736

Developing Clinical Risk Prediction Models for Worsening Heart Failure Events and Death by Left Ventricular Ejection Fraction

Rishi V Parikh 1,2, Alan S Go 1,3,4,5, Ankeet S Bhatt 1,6, Thida C Tan 1, Amanda R Allen 1, Kent Y Feng 6, Steven A Hamilton 6, Andrew S Tai 6, Jesse K Fitzpatrick 7, Keane K Lee 7, Sirtaz Adatya 7, Harshith R Avula 8, Dana R Sax 9, Xian Shen 10, Joaquim Cristino 10, Alexander T Sandhu 11,12, Paul A Heidenreich 11,12, Andrew P Ambrosy 1,3,6,✉
PMCID: PMC10727243  PMID: 37776209

Abstract

Background

There is a need to develop electronic health record–based predictive models for worsening heart failure (WHF) events across clinical settings and across the spectrum of left ventricular ejection fraction (LVEF).

Methods and Results

We studied adults with heart failure (HF) from 2011 to 2019 within an integrated health care delivery system. WHF encounters were ascertained using natural language processing and structured data. We conducted boosted decision tree ensemble models to predict 1‐year hospitalizations, emergency department visits/observation stays, and outpatient encounters for WHF and all‐cause death within each LVEF category: HF with reduced ejection fraction (EF) (LVEF <40%), HF with mildly reduced EF (LVEF 40%–49%), and HF with preserved EF (LVEF ≥50%). Model discrimination was evaluated using area under the curve and calibration using mean squared error. We identified 338 426 adults with HF: 61 045 (18.0%) had HF with reduced EF, 49 618 (14.7%) had HF with mildly reduced EF, and 227 763 (67.3%) had HF with preserved EF. The 1‐year risks of any WHF event and death were, respectively, 22.3% and 13.0% for HF with reduced EF, 17.0% and 10.1% for HF with mildly reduced EF, and 16.3% and 10.3% for HF with preserved EF. The WHF model displayed an area under the curve of 0.76 and mean squared error of 0.13, whereas the model for death displayed an area under the curve of 0.83 and mean squared error of 0.076. Performance and predictors were similar across WHF encounter types and LVEF categories.

Conclusions

We developed risk prediction models for 1‐year WHF events and death across the LVEF spectrum using structured and unstructured electronic health record data and observed no substantial differences in model performance or predictors except for death, despite differences in underlying HF cause.

Keywords: heart failure, left ventricular ejection fraction, mortality, prediction, risk models, worsening heart failure

Subject Categories: Heart Failure

Nonstandard Abbreviations and Acronyms

HFmrEF

heart failure with mildly reduced ejection fraction

HFpEF

heart failure with preserved ejection fraction

HFrEF

heart failure with reduced ejection fraction

WHF

worsening heart failure

Clinical Perspective.

What Is New?

  • Risk prediction models for 1‐year worsening heart failure events and death can robustly predict outcomes across clinical settings (ie, inpatient, emergency department, and outpatient) and categories of left ventricular ejection fraction (ie, reduced, midrange, preserved) using electronic health record–derived structured and unstructured data.

  • The most important predictors varied by outcome but not by left ventricular ejection fraction category and included surrogates of volume overload and congestion for worsening heart failure and age and comorbidity burden for death.

What Are the Clinical Implications?

  • It is feasible to develop accurate electronic health record–based risk models, using largely structured data elements, for worsening heart failure events across the care continuum to provide more personalized care management strategies.

Prior studies have shown that it is feasible and highly accurate to apply natural language process (NLP) algorithms, based on a standardized consensus definition, to electronic health record (EHR) data to systematically identify worsening heart failure (WHF) events. 1 , 2 The perceived population burden of hospitalizations for WHF may be >2‐fold higher using validated NLP‐based algorithms compared with diagnostic codes alone. 1 Notably, the disparity between reported rates of hospitalizations for WHF using diagnostic coding and NLP‐based algorithms is greater among those with heart failure (HF) with preserved ejection fraction (HFpEF) compared with either HF with reduced ejection fraction (HFrEF) or HF with mildly reduced ejection fraction (HFmrEF). This NLP‐based approach has also been leveraged to adjudicate hospitalizations, emergency department (ED) visits/observation stays, and outpatient encounters; that work demonstrated that nonhospitalized encounters account for ≈50% of total WHF events and portend a similarly poor prognosis as hospitalizations, with >10% to 15% of episodes leading to a subsequent 30‐day WHF event requiring a comparable or higher level of care. 2

Although numerous individual risk factors have been identified from published risk models for WHF, 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 there are significant limitations of existing predictive analytic efforts both in terms of the internal validity (ie, discrimination and calibration) and external generalizability (ie, data availability and ease of implementation). Several previously published risk models are based on highly phenotyped prospective cohort studies 8 or selected clinical trial participants, 11 , 12 , 13 , 14 which often include cardiac biomarkers, advanced diagnostic imaging, or functional testing that are not routinely collected as part of standard of care. In contrast, EHR data incorporate a wide range of structured, semistructured, and unstructured data elements collected as part of routine clinical care. 10 In addition, most prior attempts at developing predictive models have broadly addressed HF regardless of left ventricular ejection fraction (LVEF), 8 , 10 , 13 , 14 have focused on specific LVEF categories (ie, HFrEF alone), 11 , 12 and have not examined subgroup‐specific risk factors or developed risk models stratified by LVEF using a common source population. Finally, there is an unmet need to derive and validate predictive models for the full range of WHF events from hospitalizations to outpatient encounters given the similarly poor short‐term prognosis. 1 , 2

In this study, we aimed to examine predictors of WHF events and death by LVEF categories and develop clinical risk prediction models for events across the practice spectrum by leveraging available EHR data using NLP and machine learning methods.

Methods

Data, Materials, and Code Disclosure Statement

Due to the sensitive nature of protected health information and risks of reidentification, data used in this study are not publicly available. However, all code necessary to perform the analysis has been made publicly available at the GitHub repository (https://github.com/kpncstargroup/UTILIZE‐WHF_prediction_lvef/releases/tag/JAHA), along with the diagnosis and procedure codes used to identify comorbid conditions.

Setting and Source Population

Kaiser Permanente Northern California (KPNC) is a large, integrated health care delivery system with 21 hospitals and >260 freestanding clinics, where >4.5 million members receive comprehensive care (ie, inpatient, ED, and ambulatory encounters). Membership is highly representative of the local and statewide population with respect to age, sex, race and ethnicity, and socioeconomic status. 15 , 16 , 17 This study was approved by the KPNC Institutional Review Board, and a waiver of informed consent was obtained, because this is a retrospective data‐only study.

Study Overview and Cohort Assembly

We have previously reported the details of the study design. 1 , 2 Briefly, we created 9 calendar year cohorts from 2011 through 2019 including all active KPNC members aged ≥18 years on January 1 of each year (ie, the index date for each calendar year cohort) with previously diagnosed (ie, prevalent) HF. We defined prevalent HF as having a prior hospitalization for HF or ≥3 ambulatory visits for HF based on International Classification of Diseases, Ninth Revision and Tenth Revision (ICD‐9 and ICD‐10) codes (Table S1). We excluded patients with end‐stage kidney disease (ie, defined as receipt of chronic dialysis or kidney transplant), patients with stage D HF (ie, defined as receipt of a left ventricular assistive device or heart transplant), and patients who had <6 months of continuous health plan membership before the index date to ensure sufficient capture of baseline characteristics. We stratified patients into the following categories depending on their most recent LVEF before each index date: HFrEF (LVEF <40%), HFmrEF (LVEF 40%–49%), or HFpEF (LVEF ≥50%). Thus, patients who appear in multiple calendar year cohorts may be included in different LVEF categories in subsequent years if their LVEF changes across the study period. The overall design of the study and data flow are depicted in Figure 1.

Figure 1. Study design and data flow schematic.

Figure 1

Open in a new tab

ED indicates emergency department; HF, heart failure; HFmrEF, heart failure with mildly reduced ejection fraction; HFpEF, heart failure with preserved ejection fraction; HFrEF, heart failure with reduced ejection fraction; and LVEF, left ventricular ejection fraction.

Data Sources and Covariates

The KPNC Epic‐based EHR system was the primary source for hospitalization data, patient progress notes, and cardiac imaging reports (ie, echocardiograms and chest radiographs) used to define WHF events. The KPNC Virtual Data Warehouse was used to ascertain individual‐level demographics (ie, age, sex, self‐reported race and ethnicity), social history (self‐reported tobacco, alcohol, and drug use), linked census tract‐level demographics, concurrent comorbidities, prior procedures, laboratory values, vital signs, outpatient medications, and ECG measures as previously described and validated. 18 , 19 Data on LVEF and other echocardiographic parameters were extracted from semistructured echocardiogram reports using previously described NLP algorithms (described below). 20 Comorbid conditions and echocardiograms were ascertained within 5 years before each index date; laboratory results, vital signs, ECG results, and signs/symptoms of HF were ascertained within 1 year before each index date; baseline medication use was based on outpatient dispensed prescriptions within 120 days before each index date. Covariates were ascertained as the most recent value before the index date for continuous variables, or the presence of a condition at the index date. Additionally, for modeling purposes, we created covariates using longitudinal data by calculating the number of measurements or encounters (ie, with the corresponding diagnosis code for the condition) as well as the slopes and variances of continuous variables (ie, laboratory, vitals, echocardiogram, ECG) within the year before the index date.

Ascertainment of WHF Events

Qualifying clinical encounters for WHF included hospitalizations (ie, admissions lasting >24 hours), ED visits including observation stays, and outpatient encounters in primary care or cardiology clinics with a diagnosis code for HF. Episodes of WHF were defined as including ≥1 symptom, ≥2 objective findings (including ≥1 sign), and ≥1 change in HF‐related therapy (ie, new administration of intravenous loop diuretics or acute hemodialysis/continuous renal replacement therapy). For outpatient encounters, we also defined a change in HF‐related therapy as new initiation or augmentation of oral diuretics determined through pharmacy dispensing data and/or written provider documentation. These diagnostic criteria are based on a standardized definition of inpatient and outpatient WHF previously developed and validated by a consensus panel of trialists with expertise in clinical end point classification in collaboration with the US Food and Drug Administration. 21 This multidimensional definition is specific for WHF and has been shown to be accurate and reproducible in multiple pivotal trials of investigational drugs and devices seeking regulatory approval. 22 , 23 , 24 , 25 , 26

To ascertain WHF, we used NLP to parse relevant unstructured notes in the EHR (ie, provider notes, discharge summaries, and imaging reports) within 72 hours of a qualifying encounter. For outpatient encounters only, chest radiograph reports and laboratory values were assessed within the preceding 30 days or since the last hospital encounter and up to 1 week after the index clinical encounter. We developed rule‐based NLP queries using I2E software (Linguamatics [IQVIA], version 6.4.1), and included custom ontologies and clinical negations for each diagnostic criterion. 27 , 28 The specific NLP approach and queries have been described previously 1 , 2 and displayed a sensitivity, specificity, positive predictive value, negative predictive value, and accuracy of >90% to 95% for overall WHF for each encounter type in a validation set of >500 hospitalizations, ED visits/observation stays, and outpatient encounters compared with the gold standard of manual record review.

Follow‐Up and Death

Patients in each calendar year cohort were followed from January 1 (ie, the index date) through December 31 of each year or censored at health plan disenrollment, if earlier. Death was ascertained using comprehensive information from health plan administrative and clinical databases, member proxy reporting, Social Security Administration vital status files, and state death certificate information. 29

Statistical Analysis

We considered the unit of analysis to be the person‐year, and eligible unique individuals with prevalent HF were included across multiple calendar years with separate baseline ascertainment and prediction windows. We first describe all characteristics stratified by LVEF category. Next, we developed classification models for each 1‐year WHF encounter type (ie, hospitalization, ED visit/observation stay, and outpatient visit), separately and as a composite, and all‐cause mortality using gradient boosted decision tree algorithms (ie, the eXtreme Gradient Boosting [XGBoost] model) in the overall HF population and within each LVEF category (ie, 20 models). We trained models using data from 2011 to 2018 (ie, the training set) and measured model performance using data from 2019 (ie, the test set) overall and within each LVEF category. Model hyperparameters were tuned using a random grid search and selected using 5‐fold cross‐validation on the training sets (Table S2). Missing data for continuous and categorical variables were treated as unique values in decision tree development and were not imputed. The decision tree algorithm handles missing data natively, and places missing values in the downstream leaf that minimizes the loss function. All variables were included in their raw form, because the model does not require assumptions about linearity. Model discrimination was measured in each test set using the area under the curve (AUC), and calibration was measured using the mean squared error (MSE; ie, the Brier score 30 ). Discrimination refers to the model's ability to assign relatively higher risks to patients with the outcome than patients without the outcome (higher AUC is better), and calibration refers to the model's ability to accurately predict the absolute risks of the outcome (lower MSE is better). The area under the precision‐recall curve, maximum F1 score, and precision and recall at the threshold of the maximum F1 score are also reported on the test sets. The 95% CIs for all performance metrics were generated using percentiles from 1000 bootstrapped samples of the test set.

We then examined the importance of specific variables for the prediction of WHF and all‐cause mortality by LVEF categories. Variable importance was determined using the gain in accuracy at decision tree branching points across all decision trees during model development. Using the final values, we ranked the top 30 predictors of each outcome and compared their ranking across models for each LVEF category. We also measured variable importance using Shapley values by outcome and LVEF category. 31

Finally, we examined the marginal value of EHR data domains in predicting WHF and all‐cause mortality. Variables were categorized into the following domains: (1) individual‐level demographics, (2) census tract–level demographics, (3) social history, (4) diagnoses and disease‐related use, (5) cardiac procedures, (6) laboratory values, (7) vital signs, (8) medications, (9) ECG results, (10) NLP‐derived echocardiographic parameters, and (11) NLP‐derived signs and symptoms of HF. To rank the predictive usefulness of each domain, we conducted a forward selection procedure on top of a base model containing only individual‐level demographics (ie, age, sex, race and ethnicity). Data domains resulting in the highest increase in AUC in the test set at each step were sequentially added to the base model.

We conducted all analyses using SAS version 9.4 (SAS Institute, Cary, NC) and R version 4.1.1 (R Core Team, 2021). Models were built and evaluated using the H2O package version 3.36.1.1 in R (H2O.ai, Mountain View, CA).

Results

Cohort Assembly and Clinical Characteristics

We identified 338 426 adults with HF between January 1, 2011 and December 31, 2019: 61 045 (18.0%) with HFrEF, 49 618 (14.7%) with HFmrEF, and 227 763 (67.3%) with HFpEF (Table 1). The mean±SD age was 74.7±12.8 years, and 47.3% were women. Self‐reported race and ethnicity included 0.5% American Indian or Alaska Native, 10.8% Asian or Pacific Islander, 10.1% Black, 8.2% multiracial, 62.7% White, and 12.3% of Hispanic ethnicity. Subjects with HFpEF tended to be older, were more likely to be women, and had a higher prevalence of medical comorbidities compared with HFmrEF and HFrEF. Missing data (percent) for all covariates are shown in Table S3. The crude 1‐year risks of any WHF event (ie, hospitalization, ED visits/observation stay, and outpatient encounter) and all‐cause mortality were, respectively, 22.3% (95% CI, 21.9%–22.6%) and 13.0% (95% CI, 12.8%–13.3%) for HFrEF, 17.0% (95% CI, 16.6%–17.2%) and 10.1% (95% CI, 9.7%–10.3%) for HFmrEF, and 16.3% (95% CI, 16.1%–16.4%) and 10.3% (95% CI, 10.2%–10.4%) for HFpEF. The proportion of patients who were censored due to disenrollment from KPNC within 1 year of the index date was 3.3%.

Table 1.

Baseline Characteristics Overall and Stratified by Left Ventricular Ejection Fraction Category

Characteristic Overall HFrEF HFmrEF HFpEF
N=338 426 N=61 045 N=49 618 N=227 763
Individual demographics
Age, y 74.7 (12.8) 71.9 (13.4) 73.1 (12.8) 75.8 (12.4)
Female sex, n (%) 159 948 (47.3) 19 512 (32.0) 17 262 (34.8) 123 174 (54.1)
Self‐reported race, n (%)
American Indian or Alaska Native 1717 (0.5) 341 (0.6) 217 (0.4) 1159 (0.5)
Asian or Pacific Islander 36 398 (10.8) 6922 (11.3) 5344 (10.8) 24 132 (10.6)
Black 34 124 (10.1) 7847 (12.9) 4891 (9.9) 21 386 (9.4)
Multiracial 27 646 (8.2) 4694 (7.7) 3654 (7.4) 19 298 (8.5)
White 212 245 (62.7) 36 057 (59.1) 31 580 (63.6) 144 608 (63.5)
Unknown 26 296 (7.8) 5184 (8.5) 3932 (7.9) 17 180 (7.5)
Hispanic ethnicity, n (%) 41 788 (12.3) 8018 (13.1) 6200 (12.5) 27 570 (12.1)
Social history
Tobacco use, n (%) 19 090 (5.6) 4827 (7.9) 3163 (6.4) 11 100 (4.9)
Alcohol use, n (%) 112 672 (33.3) 21 068 (34.5) 18 169 (36.6) 73 435 (32.2)
Illicit drug use, n (%) 8599 (2.5) 2398 (3.9) 1492 (3.0) 4709 (2.1)
Census demographics, mean (SD) proportion within census tract
Education level (of individuals within patient's census tract)
<9th grade 0.07 (0.07) 0.07 (0.07) 0.07 (0.06) 0.07 (0.07)
9th–12th grade 0.06 (0.04) 0.07 (0.05) 0.06 (0.04) 0.06 (0.04)
High school graduate 0.20 (0.08) 0.21 (0.08) 0.20 (0.08) 0.20 (0.08)
Some college, no degree 0.23 (0.07) 0.23 (0.07) 0.23 (0.07) 0.23 (0.07)
Associate degree 0.08 (0.03) 0.08 (0.03) 0.09 (0.03) 0.08 (0.03)
Bachelor's degree 0.22 (0.10) 0.22 (0.10) 0.22 (0.10) 0.22 (0.10)
Graduate or professional degree 0.12 (0.09) 0.11 (0.09) 0.12 (0.09) 0.12 (0.09)
Doctorate degree 0.02 (0.02) 0.02 (0.02) 0.02 (0.02) 0.02 (0.02)
Median family income, $ 94 128.14 (41 148.77) 90 581.35 (39 775.67) 95 244.41 (41 335.45) 94 835.57 (41 418.82)
Median household income, $ 82 099.09 (35 161.85) 79 388.49 (34 150.15) 83 189.62 (35 314.86) 82 588.13 (35 360.25)
Born in the United States 0.74 (0.13) 0.74 (0.13) 0.74 (0.13) 0.74 (0.13)
Living with disability (aged >18 y) 0.10 (0.05) 0.10 (0.05) 0.10 (0.05) 0.10 (0.05)
Unemployed, civilian, noninstitutionalized aged between 18 y and 64 y 0.05 (0.03) 0.06 (0.03) 0.05 (0.03) 0.05 (0.03)
Covered by Medicare 0.17 (0.10) 0.16 (0.10) 0.17 (0.10) 0.17 (0.10)
Covered by Medicaid 0.19 (0.12) 0.20 (0.13) 0.19 (0.12) 0.19 (0.12)
Median value of home, $ 511 078.79 (314 842.37) 486 650.80 (298 221.80) 519 415.99 (316 604.98) 515 809.52 (318 461.08)
Population aged >65 y 0.16 (0.10) 0.15 (0.10) 0.16 (0.10) 0.16 (0.10)
People in same residence since y 2005 0.88 (0.12) 0.87 (0.12) 0.88 (0.11) 0.88 (0.12)
Neighborhood deprivation index −0.2 (0.9) −0.1 (0.9) −0.2 (0.9) −0.2 (0.9)
Medical history, n (%)
Duration of heart failure, y 5.5 (4.7) 5.6 (5.0) 5.2 (4.7) 5.5 (4.7)
Atrial fibrillation or flutter 154 533 (45.7) 24 930 (40.8) 22 269 (44.9) 107 334 (47.1)
Ventricular fibrillation or tachycardia 10 332 (3.1) 4108 (6.7) 2092 (4.2) 4132 (1.8)
Ischemic stroke or transient ischemic attack 26 521 (7.8) 4513 (7.4) 3576 (7.2) 18 432 (8.1)
Acute myocardial infarction 33 749 (10.0) 8379 (13.7) 7437 (15.0) 17 933 (7.9)
Unstable angina 5044 (1.5) 984 (1.6) 894 (1.8) 3166 (1.4)
Mitral or aortic valvular disease 78 776 (23.3) 12 507 (20.5) 10 772 (21.7) 55 497 (24.4)
Rheumatic heart disease 5600 (1.7) 944 (1.5) 748 (1.5) 3908 (1.7)
Peripheral artery disease 23 001 (6.8) 4170 (6.8) 3668 (7.4) 15 163 (6.7)
Venous thromboembolism 23 627 (7.0) 3795 (6.2) 2888 (5.8) 16 944 (7.4)
Other thromboembolic events 4707 (1.4) 760 (1.2) 685 (1.4) 3262 (1.4)
Diabetes 119 835 (35.4) 21 206 (34.7) 16 990 (34.2) 81 639 (35.8)
Hypertension 290 142 (85.7) 48 108 (78.8) 40 631 (81.9) 201 403 (88.4)
Dyslipidemia 297 223 (87.8) 54 612 (89.5) 44 380 (89.4) 198 231 (87.0)
Hyperthyroidism 18 584 (5.5) 2921 (4.8) 2451 (4.9) 13 212 (5.8)
Hypothyroidism 74 772 (22.1) 11 104 (18.2) 9533 (19.2) 54 135 (23.8)
Chronic liver disease 18 704 (5.5) 2715 (4.4) 2413 (4.9) 13 576 (6.0)
Chronic lung disease 156 911 (46.4) 23 885 (39.1) 20 177 (40.7) 112 849 (49.5)
Diagnosed dementia 29 908 (8.8) 4506 (7.4) 3646 (7.3) 21 756 (9.6)
Diagnosed depression 76 637 (22.6) 11 386 (18.7) 9710 (19.6) 55 541 (24.4)
Hospitalized bleed 22 405 (6.6) 3496 (5.7) 3081 (6.2) 15 828 (6.9)
Arthritis 69 206 (20.4) 9530 (15.6) 9312 (18.8) 50 364 (22.1)
Frailty 2797 (0.8) 428 (0.7) 393 (0.8) 1976 (0.9)
Hearing impairment 109 410 (32.3) 16 480 (27.0) 15 033 (30.3) 77 897 (34.2)
Visual impairment 286 475 (84.6) 47 691 (78.1) 40 387 (81.4) 198 397 (87.1)
Osteoporosis 53 860 (15.9) 6244 (10.2) 5811 (11.7) 41 805 (18.4)
Cardiac procedure history, n (%)
Coronary artery bypass graft 13 212 (3.9) 2523 (4.1) 2562 (5.2) 8127 (3.6)
Percutaneous coronary intervention 36 392 (10.8) 8588 (14.1) 7841 (15.8) 19 963 (8.8)
Implantable cardioverter defibrillator 18 141 (5.4) 11 177 (18.3) 2872 (5.8) 4092 (1.8)
Right heart catheterization 31 083 (9.2) 7909 (13.0) 5143 (10.4) 18 031 (7.9)
Coronary angiography 84 989 (25.1) 20 683 (33.9) 16 524 (33.3) 47 782 (21.0)
Catheter ablation 1738 (0.5) 199 (0.3) 257 (0.5) 1282 (0.6)
Cardiac resynchronization therapy 600 (0.2) 242 (0.4) 98 (0.2) 260 (0.1)
Pacemaker 5970 (1.8) 3372 (5.5) 825 (1.7) 1773 (0.8)
Organ transplant 33 267 (9.8) 7257 (11.9) 5357 (10.8) 20 653 (9.1)
Medications, n (%)
Angiotensin‐converting enzyme inhibitor 136 007 (40.2) 31 041 (50.8) 23 106 (46.6) 81 860 (35.9)
Angiotensin II receptor blocker 83 616 (24.7) 15 572 (25.5) 12 668 (25.5) 55 376 (24.3)
Angiotensin‐neprilysin inhibitor 674 (0.2) 471 (0.8) 136 (0.3) 67 (0.0)
Aldosterone receptor antagonist 34 505 (10.2) 13 836 (22.7) 6330 (12.8) 14 339 (6.3)
Diuretic 216 061 (63.8) 41 945 (68.7) 29 724 (59.9) 144 392 (63.4)
β‐Blocker 250 893 (74.1) 52 343 (85.7) 40 860 (82.3) 157 690 (69.2)
Calcium channel blocker 86 997 (25.7) 6414 (10.5) 8505 (17.1) 72 078 (31.6)
α‐Blocker 26 904 (7.9) 3641 (6.0) 3972 (8.0) 19 291 (8.5)
Any antihypertensive drug 305 348 (90.2) 56 863 (93.1) 45 854 (92.4) 202 631 (89.0)
Central α‐2 adrenergic agonist 9829 (2.9) 555 (0.9) 772 (1.6) 8502 (3.7)
Antiarrhythmic drug 28 983 (8.6) 7093 (11.6) 4329 (8.7) 17 561 (7.7)
Anticoagulant 110 396 (32.6) 19 703 (32.3) 16 637 (33.5) 74 056 (32.5)
Antiplatelet drug 34 219 (10.1) 7673 (12.6) 6484 (13.1) 20 062 (8.8)
Nonsteroidal anti‐inflammatory drug 19 544 (5.8) 2602 (4.3) 2600 (5.2) 14 342 (6.3)
Aspirin 24 381 (7.2) 5647 (9.3) 4357 (8.8) 14 377 (6.3)
Statin 234 073 (69.2) 43 821 (71.8) 35 870 (72.3) 154 382 (67.8)
Other lipid‐lowering drug 12 577 (3.7) 2285 (3.7) 1837 (3.7) 8455 (3.7)
Diabetic therapy 101 591 (30.0) 17 624 (28.9) 14 611 (29.4) 69 356 (30.5)
SGLT‐2 inhibitor 34 (0.0) 6 (0.0) 3 (0.0) 25 (0.0)
Hydralazine 37 484 (11.1) 8414 (13.8) 5101 (10.3) 23 969 (10.5)
Vasodilator 76 098 (22.5) 15 957 (26.1) 11 639 (23.5) 48 502 (21.3)
Nitrate 57 615 (17.0) 13 620 (22.3) 9895 (19.9) 34 100 (15.0)
Vital signs, mean (SD)
Body mass index, kg/m2 30.0 (7.8) 28.6 (6.8) 29.3 (7.0) 30.5 (8.1)
Systolic blood pressure, mm Hg 128.6 (20.5) 121.6 (19.8) 126.3 (19.9) 131.0 (20.3)
Diastolic blood pressure, mm Hg 69.3 (12.6) 68.4 (12.7) 69.6 (12.8) 69.5 (12.6)
Heart rate, bpm 75.6 (15.1) 76.0 (15.0) 75.5 (15.1) 75.4 (15.2)
Respiratory rate, breaths/min 19.1 (4.3) 19.1 (4.5) 19.0 (4.3) 19.1 (4.3)
Laboratory values, mean (SD)
Hemoglobin, g/dL 12.6 (1.9) 12.8 (1.9) 12.8 (1.9) 12.5 (1.9)
Hematocrit, % 38.8 (5.3) 39.3 (5.3) 39.2 (5.3) 38.5 (5.3)
Hemoglobin A1C, % 6.8 (1.4) 6.9 (1.5) 6.8 (1.4) 6.7 (1.4)
Glucose, mg/dL, median (q1, q3) 106.0 (93.0, 132.0) 106.0 (94.0, 134.0) 106.0 (94.0, 132.0) 105.0 (93.0, 131.0)
Low‐density lipoprotein, mg/dL 82.7 (32.5) 81.6 (32.7) 81.1 (32.0) 83.4 (32.6)
High‐density lipoprotein, mg/dL 47.2 (14.2) 44.8 (13.3) 45.9 (13.6) 48.1 (14.5)
Total cholesterol, mg/dL 155.2 (41.0) 150.9 (41.1) 152.0 (40.5) 157.1 (40.9)
Triglycerides, mg/dL 133.2 (99.8) 129.9 (91.6) 132.4 (127.0) 134.4 (94.8)
Brain‐type natriuretic peptide, ng/L, median (q1, q3) 269.0 (124.0, 539.0) 437.0 (194.0, 927.0) 299.0 (134.0, 608.0) 231.0 (109.0, 438.0)
Troponin I, ng/mL 0.4 (3.5) 0.7 (5.2) 0.7 (5.0) 0.2 (2.3)
Prothrombin international normalized ratio 1.8 (0.8) 1.8 (0.8) 1.9 (0.8) 1.8 (0.8)
Platelets, 1000/μL 221.1 (77.6) 212.1 (74.5) 216.2 (73.9) 224.5 (78.9)
Thyroid‐stimulating hormone, mg/dL 2.5 (4.7) 2.6 (4.9) 2.5 (6.0) 2.5 (4.4)
Thyroxine (T4), μg/dL 1.2 (0.3) 1.2 (0.3) 1.2 (0.3) 1.2 (0.3)
Alanine aminotransferase, U/L 22.2 (27.4) 24.2 (42.4) 22.8 (22.0) 21.5 (22.9)
Total bilirubin, mg/dL 0.8 (0.6) 0.9 (0.6) 0.8 (0.6) 0.7 (0.6)
Serum albumin, mg/dL 3.9 (28.0) 3.6 (0.6) 3.7 (0.6) 3.9 (33.8)
Serum creatinine, mg/dL 1.2 (0.5) 1.2 (0.5) 1.2 (0.5) 1.2 (0.5)
Estimated glomerular filtration rate, mL/min per 1.73 m2 64.8 (22.6) 65.1 (22.7) 66.7 (22.5) 64.3 (22.6)
Blood urea nitrogen, mg/dL 25.5 (14.1) 26.4 (14.4) 25.1 (13.9) 25.3 (14.1)
Urine albumin‐to‐creatinine ratio, mg/g, median (q1, q3) 27.2 (8.9, 135.7) 23.6 (8.0, 114.2) 25.2 (8.1, 116.9) 28.6 (9.4, 146.2)
Urine protein‐to‐creatinine ratio, mg/mg, median (q1, q3) 0.5 (0.2, 1.3) 0.5 (0.2, 1.1) 0.5 (0.2, 1.2) 0.5 (0.2, 1.3)
Urine dipstick proteinuria
Negative 140 651 (41.6) 24 398 (40.0) 19 629 (39.6) 96 624 (42.4)
Trace 46 923 (13.9) 8268 (13.5) 6500 (13.1) 32 155 (14.1)
1+ 47 497 (14.0) 8111 (13.3) 6641 (13.4) 32 745 (14.4)
2+ 25 553 (7.6) 4341 (7.1) 3564 (7.2) 17 648 (7.7)
3+ 7993 (2.4) 1122 (1.8) 972 (2.0) 5899 (2.6)
Unknown 69 809 (20.6) 14 805 (24.3) 12 312 (24.8) 42 692 (18.7)
White blood cells, 1000/µL 7.7 (4.0) 7.5 (3.6) 7.6 (3.9) 7.8 (4.1)
C‐reactive protein, mg/L 2.9 (5.2) 3.3 (5.6) 2.9 (5.0) 2.9 (5.2)
Serum sodium, mmol/L 139.4 (3.4) 139.3 (3.3) 139.4 (3.3) 139.4 (3.4)
Serum calcium, mmol/L 9.1 (0.6) 9.1 (0.6) 9.1 (0.6) 9.1 (0.6)
Serum phosphate, mmol/L 3.6 (0.8) 3.7 (0.8) 3.6 (0.8) 3.6 (0.8)
Serum potassium, mmol/L 4.3 (0.4) 4.4 (0.4) 4.4 (0.4) 4.3 (0.4)
Echocardiographic parameters
Left ventricular ejection fraction, %, mean (SD) 52.3 (13.3) 29.4 (6.1) 43.6 (2.6) 60.3 (5.5)
Aortic stenosis, n (%) 0.1 (0.3) 0.1 (0.3) 0.1 (0.3) 0.1 (0.3)
Aortic stenosis severity, n (%)
None 276 200 (81.6) 53 418 (87.5) 42 045 (84.7) 180 737 (79.4)
Mild 18 632 (5.5) 2078 (3.4) 2169 (4.4) 14 385 (6.3)
Moderate 6304 (1.9) 724 (1.2) 679 (1.4) 4901 (2.2)
Severe 3746 (1.1) 547 (0.9) 432 (0.9) 2767 (1.2)
Unknown 29 198 (8.6) 3789 (6.2) 3758 (7.6) 21 651 (9.5)
Aortic valve area, cm2, mean (SD) 2.1 (0.8) 2.1 (0.8) 2.1 (0.8) 2.0 (0.8)
Aortic valve maximum velocity, m/s, mean (SD) 1.7 (0.7) 1.5 (0.6) 1.6 (0.6) 1.8 (0.7)
Aortic valve velocity time integral, cm, mean (SD) 36.8 (19.1) 29.9 (15.7) 33.1 (16.3) 39.4 (19.9)
Mean aortic valve gradient, mm Hg, mean (SD) 8.2 (8.4) 6.1 (6.3) 6.9 (7.1) 9.0 (9.0)
Peak aortic valve gradient, mm Hg, mean (SD) 13.9 (13.8) 10.2 (10.1) 11.6 (11.3) 15.4 (14.9)
Bicuspid aortic valve, n (%) 2170 (0.6) 399 (0.7) 373 (0.8) 1398 (0.6)
Prosthetic valve, n (%) 14 325 (4.2) 1603 (2.6) 1802 (3.6) 10 920 (4.8)
Left ventricular hypertrophy, n (%)
None 97 802 (28.9) 21 010 (34.4) 15 098 (30.4) 61 694 (27.1)
Mild 98 092 (29.0) 14 219 (23.3) 14 112 (28.4) 69 761 (30.6)
Moderate 28 872 (8.5) 3866 (6.3) 3983 (8.0) 21 023 (9.2)
Severe 5581 (1.6) 1055 (1.7) 661 (1.3) 3865 (1.7)
Unknown 96 650 (28.6) 19 492 (31.9) 14 239 (28.7) 62 919 (27.6)
End‐diastolic diameter, cm, mean (SD) 5.0 (0.9) 5.8 (0.9) 5.3 (0.8) 4.7 (0.8)
End‐systolic diameter, cm, mean (SD) 3.6 (1.0) 4.8 (1.0) 4.0 (0.8) 3.2 (0.7)
End‐diastolic volume, mL, mean (SD) 107.0 (51.5) 153.0 (62.5) 119.8 (45.3) 88.6 (36.7)
End‐systolic volume, mL, mean (SD) 55.6 (41.4) 103.7 (52.5) 66.4 (29.4) 36.9 (21.6)
Left ventricular outflow tract diameter, cm, mean (SD) 2.1 (0.2) 2.1 (0.2) 2.1 (0.2) 2.1 (0.2)
Left ventricular outflow tract velocity time integral, cm, mean (SD) 20.4 (6.3) 16.2 (5.1) 18.2 (5.0) 22.0 (6.3)
Electrocardiographic parameters, mean (SD)
Cardiac rate, bpm 77.3 (19.9) 79.4 (19.8) 77.9 (20.0) 76.6 (19.8)
PR interval, s 179.7 (48.1) 179.1 (48.6) 179.9 (48.9) 179.8 (47.9)
QRS interval, s 113.4 (29.6) 128.0 (33.4) 119.2 (30.5) 108.1 (26.6)
Corrected QT Interval, s 461.8 (44.8) 476.3 (46.8) 466.4 (48.6) 456.9 (42.4)
QT interval, s 415.4 (54.8) 422.6 (56.5) 417.6 (56.7) 413.0 (53.7)
Heart failure signs and symptoms, n (%)
Shortness of breath 139 283 (41.2) 26 838 (44.0) 21 332 (43.0) 91 113 (40.0)
Paroxysmal nocturnal dyspnea 6294 (1.9) 1712 (2.8) 928 (1.9) 3654 (1.6)
Rales 35 352 (10.4) 6056 (9.9) 4539 (9.1) 24 757 (10.9)
S3 gallop 775 (0.2) 354 (0.6) 125 (0.3) 296 (0.1)
Lower extremity edema 75 029 (22.2) 12 650 (20.7) 10 003 (20.2) 52 376 (23.0)
Jugular venous distension 9286 (2.7) 2397 (3.9) 1450 (2.9) 5439 (2.4)
Hepatomegaly 424 (0.1) 85 (0.1) 64 (0.1) 275 (0.1)
Orthopnea 15 767 (4.7) 3926 (6.4) 2460 (5.0) 9381 (4.1)
Weight gain 12 713 (3.8) 3220 (5.3) 1994 (4.0) 7499 (3.3)
Abdominal swelling 6368 (1.9) 1165 (1.9) 776 (1.6) 4427 (1.9)
Tachypnea 11 795 (3.5) 2013 (3.3) 1493 (3.0) 8289 (3.6)
Fatigue 27 683 (8.2) 5758 (9.4) 4168 (8.4) 17 757 (7.8)
Pulmonary edema (on chest radiograph) 19 752 (5.8) 3576 (5.9) 2429 (4.9) 13 747 (6.0)
Cardiomegaly (on chest radiograph) 23 224 (6.9) 4998 (8.2) 3030 (6.1) 15 196 (6.7)
Pleural effusion (on chest radiograph) 25 024 (7.4) 4426 (7.3) 3227 (6.5) 17 371 (7.6)
Hypoxemia (SpO2 <92%) 21 166 (6.3) 2866 (4.7) 2347 (4.7) 15 953 (7.0)
Tachycardia (heart rate >100 bpm) 27 395 (8.1) 4813 (7.9) 3687 (7.4) 18 895 (8.3)
Open in a new tab

bpm indicates beats per minute; HFmrEF, heart failure with mildly reduced ejection fraction; HFpEF, heart failure with preserved ejection fraction; HFrEF, heart failure with reduced ejection fraction; SGLT‐2, sodium‐glucose cotransporter‐2; and SpO2, oxygen saturation.

Model Performance and Predictors for WHF Events

The overall model for any WHF event (ie, hospitalization, ED visits/observation stay, and/or outpatient encounter) among all patients with HF displayed an AUC of 0.764 (95% CI, 0.759–0.769) and MSE of 0.127 (95% CI, 0.125–0.129) (Table 2). Full model performance metrics and 95% CI, including F1 scores, precision, and recall are shown in Table S4. Model performance metrics were relatively consistent across hospitalizations, ED visits/observation stays, and outpatient encounters for WHF, with slightly lower discrimination for ED visits/observation stays (AUC, 0.760 [95% CI, 0.752–0.769]) than hospitalized (AUC, 0.774 [95% CI, 0.768–781]) and outpatient WHF (AUC, 0.772 [95% CI, 0.764–0.780]). Discrimination and calibration did not differ substantially by LVEF category (HFrEF versus HFmrEF versus HFpEF) for the composite of all WHF events or any individual WHF encounter type. The top 30 risk factors for WHF events by LVEF category are shown in Figure 2 and Table S5. Brain‐type natriuretic peptide, number of prior encounters with specific signs for WHF, and diuretic use were the most prognostically important risk factors for WHF irrespective of LVEF.

Table 2.

Performance Metrics of Predictive Models Overall and Within Each Left Ventricular Ejection Fraction Category

Category Outcome
Any encounter for WHF Hospitalizations for WHF ED visits/observation stays for WHF Outpatient encounters for WHF All‐cause mortality
AUC MSE AUC MSE AUC MSE AUC MSE AUC MSE
Any HF 0.764 0.127 0.774 0.087 0.760 0.047 0.772 0.057 0.827 0.076
HFrEF 0.756 0.153 0.775 0.108 0.731 0.059 0.759 0.077 0.824 0.086
HFmrEF 0.773 0.121 0.784 0.079 0.756 0.044 0.777 0.059 0.824 0.074
HFpEF 0.767 0.122 0.775 0.083 0.770 0.045 0.776 0.052 0.827 0.074
Open in a new tab

AUC indicates area under the curve; ED, emergency department; HF, heart failure; HFmrEF, heart failure with mildly reduced ejection fraction; HFpEF, heart failure with preserved ejection fraction; HFrEF, heart failure with reduced ejection fraction; MSE, mean squared error; and WHF, worsening heart failure.

Figure 2. Relative variable importance for worsening heart failure events by LVEF category.

Figure 2

Open in a new tab

BMI indicates body mass index; BNP, brain‐type natriuretic peptide; HFmrEF, heart failure with mildly reduced ejection fraction; HFpEF, heart failure with preserved ejection fraction; HFrEF, heart failure with reduced ejection fraction; and LVEF, left ventricular ejection fraction.

Model Performance and Predictors for All‐Cause Mortality

The model for all‐cause mortality displayed an AUC of 0.827 (95% CI, 0.821–0.832) and MSE of 0.076 (95% CI, 0.074–0.078) (Table 2). Model performance metrics for all‐cause mortality were comparable for HFrEF versus HFmrEF versus HFpEF. The top 30 risk factors for all‐cause mortality are shown in Figure 3 and Table S6. Age, brain‐type natriuretic peptide, other laboratory values (ie, hemoglobin, blood urea nitrogen, and serum albumin), and medical comorbidities were prognostically important risk factors for all‐cause mortality across LVEF categories.

Figure 3. Relative variable importance for all‐cause mortality by left ventricular ejection fraction category.

Figure 3

Open in a new tab

BMI indicates body mass index; BUN, blood urea nitrogen; HFmrEF, heart failure with mildly reduced ejection fraction; HFpEF, heart failure with preserved ejection fraction; and HFrEF, heart failure with reduced ejection fraction.

Incremental Model Performance by EHR Data Domain

Using a process of forward selection with sequential addition of EHR domains, we found that laboratory values, diagnostic codes, and pharmacy prescriptions provided the greatest incremental value in terms of model discrimination (ie, AUC) and calibration (ie, MSE) for any WHF event at 1 year (Figure 4). Laboratory values, diagnostic codes, and vital signs provided the greatest value for predicting 1‐year all‐cause mortality. Other EHR domains provided diminishing returns in terms of performance for both WHF and death, although optimal prediction of WHF events was additionally dependent on echocardiogram parameters, vital signs, and signs/symptoms of HF.

Figure 4. Performance metrics of predictive models for worsening heart failure and all‐cause mortality by sequential addition of EHR variable domains.

Figure 4

Open in a new tab

AUC indicates area under the curve; Dx, diagnoses; EHR, electronic health record; MSE, mean square error; Px, procedures; Rx, prescriptions; and WHF, worsening heart failure.

Discussion

We developed prediction models for WHF events and death at 1 year within a population‐based cohort of ambulatory patients with HF, leveraging structured and unstructured data from a longitudinal EHR system. Overall, model performance metrics were similar for WHF events across clinical settings and LVEF categories. Risk factors for WHF events were similar among different LVEF subgroups and predominantly included surrogates of volume overload/congestion. Notably, model discrimination and calibration were consistently better for all‐cause mortality versus WHF. In addition, risk factors for all‐cause mortality were more closely related to age and burden of medical commodities. Finally, structured EHR data, including laboratory values, diagnostic codes, pharmacy prescriptions, and vital signs, provided the most incremental value to model performance with diminishing returns from additional data elements.

We have previously reported that ED visits/observation stays and outpatient encounters account for ≈50% of total WHF events and are associated with a similarly poor short‐term prognosis when compared with hospitalizations for WHF. 1 , 2 The present analysis provides proof‐of‐concept that it is feasible to risk stratify for WHF events across the care continuum from ambulatory encounters to hospitalizations. This provides further evidence to support the validity of outpatient WHF as a clinical entity and raises the hypothesis that this end point may be a potential target for evaluating HF therapies. 32 , 33 , 34 , 35 , 36 , 37 Interestingly, although HF is a heterogeneous condition and the traditional paradigm has underscored the diagnostic, prognostic, and therapeutic implications of an LVEF‐based classification system, we found that prediction models had similar performance with shared risk factors for WHF events across the LVEF spectrum. 38 , 39 The most important risk factors were all markers of volume status/fluid management highlighting the central role congestion plays in the pathophysiology of WHF, regardless of LVEF. 40 , 41 , 42 , 43

We also found that predictive models for all‐cause mortality had better discrimination and more precise calibration compared with models for WHF events. This finding is consistent with the existing literature 3 , 4 , 5 , 6 and extends the implications to different clinical settings for WHF and LVEF categories. This is not unexpected, because death is a more objective outcome, and prior research has documented prominent geographic variation in the rate of hospitalizations for WHF even after controlling for baseline differences. 44 , 45 , 46 , 47 , 48 , 49 In addition to brain‐type natriuretic peptide, the most important risk factors for death were age, and diagnostic codes and laboratory values reflective of overall comorbidity. This is consistent with the concept that adults with HF are at high competing risk of noncardiovascular death, and the relative contribution of noncardiovascular factors increases with LVEF. The underlying cause of death may be noncardiovascular in more than half the individuals with HFpEF in the community. 50 , 51 The difference in risk factors for predicting WHF events and death suggest that these end points should be viewed as distinct from a public health perspective.

Finally, our EHR‐based models displayed similar performance and risk factors as published risk models based on prospective cohort studies and clinical trial databases. 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 Notably, 2 of the strongest predictors of WHF in our study were prior encounters with lower extremity edema and rales. Measures of symptomatic congestion and health status represent an important source of variability in the risk of future WHF events; many existing models incorporate patient‐reported measures of physical limitation, functional status, and symptom burden such as the New York Heart Association class or Kansas City Cardiomyopathy Questionnaire responses. 7 , 8 , 11 , 52 Unfortunately, the Kansas City Cardiomyopathy Questionnaire is primarily a research instrument not widely used in clinical care, and New York Heart Association class is often unavailable in structured EHR data sets and is not consistently noted in HF‐related encounters. In contrast, the presence of edema and rales is documented in nearly every encounter as a part of the routine physical examination. The relative importance of these specific EHR‐derived measures in our models suggests that they may capture similar variability in risk and confirms that symptom burden remains a strong predictor of WHF events in a population‐based setting. In the context of EHR data domains, signs/symptoms of HF were less prognostically valuable than laboratory values, comorbidities, medications, echocardiographic data, and vital signs. However, this may partially be due to the nature of the variables included (ie, laboratory values), which are continuous measures and are naturally able to capture more variation in risks.

Our study has several limitations. First, we did not externally validate our models, which may have overestimated performance and limited the generalizability to other populations. The objective of this study was to examine the differences in performance and predictors between HF subtypes and explore the potential of complex EHR data; further studies considering external validation using other EHR‐based data may provide incremental value on the generalizability of this approach. Second, we did not directly compare the performance of our approach to existing risk models, partially due to the limited availability of data elements included in those models that are outside typical clinical care. Third, we did not include patients with an unknown LVEF, because our primary aim was to derive and validate risk models across different LVEF categories. In our population, patients with an unknown LVEF represented ≈10% of all prevalent HF in any given calendar year, are relatively lower risk, and may not be proactively managed by providers. 1 , 2 Finally, relative variable importance may be sensitive to modeling technique and sampling and should be viewed as hypothesis‐generating, because identifying causal factors was not the main objective of our approach.

In conclusion, EHR‐based clinical risk prediction models for WHF events and death can robustly predict clinical outcomes across clinical settings and LVEF categories. Model performance was better for death, suggesting a residual opportunity to identify novel risk factors and innovative approaches to risk prediction for WHF. Variables indicating congestion appear to play a more prominent role in WHF, whereas age‐related comorbidity is more closely linked with the risk of death. Future research is needed to further delineate causal relationships with outcomes and establish the role of real‐time implementation of EHR‐based clinical risk prediction models for early intervention to prevent and/or delay WHF.

Sources of Funding

The study was supported by research grants from Novartis AG (East Hanover, NJ) and the Kaiser Permanente Northern California Community Benefit Program.

Disclosures

A.P.A. is supported by a Mentored Patient‐Oriented Research Career Development Award (K23HL150159) through the National Heart, Lung, and Blood Institute and has received relevant research support through grants to his institution from Abbott, Amarin Pharma, Edwards Lifesciences, Esperion, Lexicon, and Novartis. X.S., and J.C. are employees of Novartis AG (East Hanover, NJ). A.S.G. has received relevant research support through grants to his institution from the National Heart, Lung, and Blood Institute; National Institute of Diabetes, Digestive and Kidney Diseases; National Institute on Aging; Amarin Pharma, Inc.; Novartis; Janssen Research & Development; and CSL Behring. A.T.S. is supported by a grant from the National Heart, Lung, and Blood Institute (1K23HL151672–01). The remaining authors have no disclosures to report.

Supporting information

Tables S1–S6

Click here for additional data file. (170.9KB, pdf)

This article was sent to Sula Mazimba, MD, MPH, Associate Editor, for review by expert referees, editorial decision, and final disposition.

For Sources of Funding and Disclosures, see page 14.

References

  • 1. Ambrosy AP, Parikh RV, Sung SH, Narayanan A, Masson R, Lam PQ, Kheder K, Iwahashi A, Hardwick AB, Fitzpatrick JK, et al. A natural language processing‐based approach for identifying hospitalizations for worsening heart failure within an integrated health care delivery system. JAMA Netw Open. 2021;4:e2135152. doi: 10.1001/jamanetworkopen.2021.35152 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2. Ambrosy AP, Parikh RV, Sung SH, Tan TC, Narayanan A, Masson R, Lam PQ, Kheder K, Iwahashi A, Hardwick AB, et al. Analysis of worsening heart failure events in an integrated health care system. J Am Coll Cardiol. 2022;80:111–122. doi: 10.1016/j.jacc.2022.04.045 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3. Alba AC, Agoritsas T, Jankowski M, Courvoisier D, Walter SD, Guyatt GH, Ross HJ. Risk prediction models for mortality in ambulatory patients with heart failure: a systematic review. Circ Heart Fail. 2013;6:881–889. doi: 10.1161/CIRCHEARTFAILURE.112.000043 [DOI] [PubMed] [Google Scholar]
  • 4. Rahimi K, Bennett D, Conrad N, Williams TM, Basu J, Dwight J, Woodward M, Patel A, McMurray J, MacMahon S. Risk prediction in patients with heart failure: a systematic review and analysis. JACC Heart Fail. 2014;2:440–446. doi: 10.1016/j.jchf.2014.04.008 [DOI] [PubMed] [Google Scholar]
  • 5. Di Tanna GL, Wirtz H, Burrows KL, Globe G. Evaluating risk prediction models for adults with heart failure: a systematic literature review. PLoS One. 2020;15:e0224135. doi: 10.1371/journal.pone.0224135 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6. Mpanya D, Celik T, Klug E, Ntsinjana H. Predicting mortality and hospitalization in heart failure using machine learning: a systematic literature review. Int J Cardiol Heart Vasc. 2021;34:100773. doi: 10.1016/j.ijcha.2021.100773 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7. Levy WC. Seattle heart failure model. Am J Cardiol. 2013;111:1235. doi: 10.1016/j.amjcard.2013.01.286 [DOI] [PubMed] [Google Scholar]
  • 8. Pocock SJ, Ariti CA, McMurray JJ, Maggioni A, Kober L, Squire IB, Swedberg K, Dobson J, Poppe KK, Whalley GA, et al. Predicting survival in heart failure: a risk score based on 39 372 patients from 30 studies. Eur Heart J. 2013;34:1404–1413. doi: 10.1093/eurheartj/ehs337 [DOI] [PubMed] [Google Scholar]
  • 9. Lee DS, Austin PC, Rouleau JL, Liu PP, Naimark D, Tu JV. Predicting mortality among patients hospitalized for heart failure derivation and validation of a clinical model. JAMA. 2003;290:2581–2587. doi: 10.1001/jama.290.19.2581 [DOI] [PubMed] [Google Scholar]
  • 10. Peterson PN, Rumsfeld JS, Liang L, Albert NM, Hernandez AF, Peterson ED, Fonarow GC, Masoudi FA; American Heart Association Get With the Guidelines‐Heart Failure Program . A validated risk score for in‐hospital mortality in patients with heart failure from the American Heart Association get with the guidelines program. Circ Cardiovasc Qual Outcomes. 2010;3:25–32. doi: 10.1161/CIRCOUTCOMES.109.854877 [DOI] [PubMed] [Google Scholar]
  • 11. Angraal S, Mortazavi BJ, Gupta A, Khera R, Ahmad T, Desai NR, Jacoby DL, Masoudi FA, Spertus JA, Krumholz HM. Machine learning prediction of mortality and hospitalization in heart failure with preserved ejection fraction. JACC Heart Fail. 2020;8:12–21. doi: 10.1016/j.jchf.2019.06.013 [DOI] [PubMed] [Google Scholar]
  • 12. Collier TJ, Pocock SJ, McMurray JJ, Zannad F, Krum H, van Veldhuisen DJ, Swedberg K, Shi H, Vincent J, Pitt B. The impact of eplerenone at different levels of risk in patients with systolic heart failure and mild symptoms: insight from a novel risk score for prognosis derived from the EMPHASIS‐HF trial. Eur Heart J. 2013;34:2823–2829. doi: 10.1093/eurheartj/eht247 [DOI] [PubMed] [Google Scholar]
  • 13. Khazanie P, Heizer GM, Hasselblad V, Armstrong PW, Califf RM, Ezekowitz J, Dickstein K, Levy WC, McMurray JJ, Metra M, et al. Predictors of clinical outcomes in acute decompensated heart failure: acute study of clinical effectiveness of nesiritide in decompensated heart failure outcome models. Am Heart J. 2015;170:290–297. doi: 10.1016/j.ahj.2015.04.006 [DOI] [PubMed] [Google Scholar]
  • 14. O'Connor CM, Mentz RJ, Cotter G, Metra M, Cleland JG, Davison BA, Givertz MM, Mansoor GA, Ponikowski P, Teerlink JR, et al. The PROTECT in‐hospital risk model: 7‐day outcome in patients hospitalized with acute heart failure and renal dysfunction. Eur J Heart Fail. 2012;14:605–612. doi: 10.1093/eurjhf/hfs029 [DOI] [PubMed] [Google Scholar]
  • 15. Krieger N. Overcoming the absence of socioeconomic data in medical records: validation and application of a census‐based methodology. Am J Public Health. 1992;82:703–710. doi: 10.2105/AJPH.82.5.703 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16. Gordon NP. Characteristics of Adult Health Plan Members in the Northern California Region Membership, as Estimated from the 2011 Member Health Survey. Division of Research, Kaiser Permanente Medical Care Program; 2013. Accessed September 18, 2023. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5101088/ [Google Scholar]
  • 17. Koebnick C, Langer‐Gould AM, Gould MK, Chao CR, Iyer RL, Smith N, Chen W, Jacobsen SJ. Sociodemographic characteristics of members of a large, integrated health care system: comparison with US Census Bureau data. Perm J. 2012;16:37–41. doi: 10.7812/TPP/12-031 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18. Go AS, Magid DJ, Wells B, Sung SH, Cassidy‐Bushrow AE, Greenlee RT, Langer RD, Lieu TA, Margolis KL, Masoudi FA, et al. The cardiovascular research network: a new paradigm for cardiovascular quality and outcomes research. Circ Cardiovasc Qual Outcomes. 2008;1:138–147. doi: 10.1161/CIRCOUTCOMES.108.801654 [DOI] [PubMed] [Google Scholar]
  • 19. Magid DJ, Gurwitz JH, Rumsfeld JS, Go AS. Creating a research data network for cardiovascular disease: the CVRN. Expert Rev Cardiovasc Ther. 2008;6:1043–1045. doi: 10.1586/14779072.6.8.1043 [DOI] [PubMed] [Google Scholar]
  • 20. Solomon MD, Tabada G, Allen A, Sung SH, Go AS. Large‐scale identification of aortic stenosis and its severity using natural language processing on electronic health records. Cardiovasc Digit Health J. 2021;2:156–163. doi: 10.1016/j.cvdhj.2021.03.003 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21. Hicks KA, Mahaffey KW, Mehran R, Nissen SE, Wiviott SD, Dunn B, Solomon SD, Marler JR, Teerlink JR, Farb A, et al. 2017 cardiovascular and stroke endpoint definitions for clinical trials. J Am Coll Cardiol. 2018;71:1021–1034. doi: 10.1016/j.jacc.2017.12.048 [DOI] [PubMed] [Google Scholar]
  • 22. Gheorghiade M, Konstam MA, Burnett JC Jr, Grinfeld L, Maggioni AP, Swedberg K, Udelson JE, Zannad F, Cook T, Ouyang J, et al. Short‐term clinical effects of tolvaptan, an oral vasopressin antagonist, in patients hospitalized for heart failure: the EVEREST clinical status trials. JAMA. 2007;297:1332–1343. doi: 10.1001/jama.297.12.1332 [DOI] [PubMed] [Google Scholar]
  • 23. Konstam MA, Gheorghiade M, Burnett JC Jr, Grinfeld L, Maggioni AP, Swedberg K, Udelson JE, Zannad F, Cook T, Ouyang J, et al. Effects of oral tolvaptan in patients hospitalized for worsening heart failure: the EVEREST Outcome Trial. JAMA. 2007;297:1319–1331. doi: 10.1001/jama.297.12.1319 [DOI] [PubMed] [Google Scholar]
  • 24. O'Connor CM, Starling RC, Hernandez AF, Armstrong PW, Dickstein K, Hasselblad V, Heizer GM, Komajda M, Massie BM, McMurray JJ, et al. Effect of nesiritide in patients with acute decompensated heart failure. N Engl J Med. 2011;365:32–43. doi: 10.1056/NEJMoa1100171 [DOI] [PubMed] [Google Scholar]
  • 25. Teerlink JR, Cotter G, Davison BA, Felker GM, Filippatos G, Greenberg BH, Ponikowski P, Unemori E, Voors AA, Adams KF Jr, et al. Serelaxin, recombinant human relaxin‐2, for treatment of acute heart failure (RELAX‐AHF): a randomised, placebo‐controlled trial. Lancet. 2013;381:29–39. doi: 10.1016/S0140-6736(12)61855-8 [DOI] [PubMed] [Google Scholar]
  • 26. Massie BM, O'Connor CM, Metra M, Ponikowski P, Teerlink JR, Cotter G, Weatherley BD, Cleland JG, Givertz MM, Voors A, et al. Rolofylline, an adenosine A1‐receptor antagonist, in acute heart failure. N Engl J Med. 2010;363:1419–1428. doi: 10.1056/NEJMoa0912613 [DOI] [PubMed] [Google Scholar]
  • 27. Cormack J, Nath C, Milward D, Raja K, Jonnalagadda SR. Agile text mining for the 2014 i2b2/UTHealth cardiac risk factors challenge. J Biomed Inform. 2015;58:S120–S127. doi: 10.1016/j.jbi.2015.06.030 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28. Milward D, Bjareland M, Hayes W, Maxwell M, Oberg L, Tilford N, Thomas J, Hale R, Knight S, Barnes J. Ontology‐based interactive information extraction from scientific abstracts. Comp Funct Genomics. 2005;6:67–71. doi: 10.1002/cfg.456 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29. Curb JD, Ford CE, Pressel S, Palmer M, Babcock C, Hawkins CM. Ascertainment of vital status through the National Death Index and the Social Security Administration. Am J Epidemiol. 1985;121:754–766. doi: 10.1093/aje/121.5.754 [DOI] [PubMed] [Google Scholar]
  • 30. Brier GW. Verification of forecasts expressed in terms of probability. Mon Weather Rev. 1950;78:1–3. doi: [DOI] [Google Scholar]
  • 31. Lundberg SM, Lee S‐I. A unified approach to interpreting model predictions. Adv Neural Inf Proces Syst. 2017;30. [Google Scholar]
  • 32. Greene SJ, Lautsch D, Gaggin HK, Djatche LM, Zhou M, Song Y, Signorovitch J, Stevenson AS, Blaustein RO, Butler J. Contemporary outpatient management of patients with worsening heart failure with reduced ejection fraction: rationale and design of the CHART‐HF study. Am Heart J. 2022;251:127–136. doi: 10.1016/j.ahj.2022.05.016 [DOI] [PubMed] [Google Scholar]
  • 33. Docherty KF, Jhund PS, Anand I, Bengtsson O, Bohm M, de Boer RA, DeMets DL, Desai AS, Drozdz J, Howlett J, et al. Effect of dapagliflozin on outpatient worsening of patients with heart failure and reduced ejection fraction: a prespecified analysis of DAPA‐HF. Circulation. 2020;142:1623–1632. doi: 10.1161/CIRCULATIONAHA.120.047480 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34. Khan MS, Butler J, Greene SJ. Recognizing the significance of outpatient worsening heart failure. J Am Heart Assoc. 2020;9:e017485. doi: 10.1161/JAHA.120.017485 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35. Greene SJ, Felker GM, Butler J. Outpatient versus inpatient worsening heart failure: distinguishing biology and risk from location of care. Eur J Heart Fail. 2019;21:121–124. doi: 10.1002/ejhf.1341 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36. Greene SJ, Mentz RJ, Felker GM. Outpatient worsening heart failure as a target for therapy: a review. JAMA Cardiol. 2018;3:252–259. doi: 10.1001/jamacardio.2017.5250 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37. Okumura N, Jhund PS, Gong J, Lefkowitz MP, Rizkala AR, Rouleau JL, Shi VC, Swedberg K, Zile MR, Solomon SD, et al. Importance of clinical worsening of heart failure treated in the outpatient setting: evidence from the Prospective Comparison of ARNI with ACEI to Determine Impact on Global Mortality and Morbidity in Heart Failure Trial (PARADIGM‐HF). Circulation. 2016;133:2254–2262. doi: 10.1161/CIRCULATIONAHA.115.020729 [DOI] [PubMed] [Google Scholar]
  • 38. Heidenreich PA, Bozkurt B, Aguilar D, Allen LA, Byun JJ, Colvin MM, Deswal A, Drazner MH, Dunlay SM, Evers LR, et al. 2022 AHA/ACC/HFSA guideline for the management of heart failure: executive summary: a report of the American College of Cardiology/American Heart Association joint committee on Clinical Practice Guidelines. Circulation. 2022;145:e876–e894. doi: 10.1161/CIR.0000000000001062 [DOI] [PubMed] [Google Scholar]
  • 39. McDonagh TA, Metra M, Adamo M, Gardner RS, Baumbach A, Bohm M, Burri H, Butler J, Celutkiene J, Chioncel O, et al. 2021 ESC Guidelines for the diagnosis and treatment of acute and chronic heart failure. Eur Heart J. 2021;42:3599–3726. doi: 10.1093/eurheartj/ehab368 [DOI] [PubMed] [Google Scholar]
  • 40. Masson R, Ambrosy AP, Kheder K, Fudim M, Clare RM, Coles A, Hernandez AF, Starling RC, Ezekowitz JA, O'Connor CM, et al. A novel In‐hospital congestion score to risk stratify patients admitted for worsening heart failure (from ASCEND‐HF). J Cardiovasc Transl Res. 2020;13:540–548. doi: 10.1007/s12265-020-09954-x [DOI] [PubMed] [Google Scholar]
  • 41. Ambrosy AP, Bhatt AS, Gallup D, Anstrom KJ, Butler J, DeVore AD, Felker GM, Fudim M, Greene SJ, Hernandez AF, et al. Trajectory of congestion metrics by ejection fraction in patients with acute heart failure (from the Heart Failure Network). Am J Cardiol. 2017;120:98–105. doi: 10.1016/j.amjcard.2017.03.249 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42. Ambrosy AP, Cerbin LP, Armstrong PW, Butler J, Coles A, DeVore AD, Dunlap ME, Ezekowitz JA, Felker GM, Fudim M, et al. Body weight change during and after hospitalization for acute heart failure: patient characteristics, markers of congestion, and outcomes: findings from the ASCEND‐HF trial. JACC Heart Fail. 2017;5:1–13. doi: 10.1016/j.jchf.2016.09.012 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 43. Ambrosy AP, Pang PS, Khan S, Konstam MA, Fonarow GC, Traver B, Maggioni AP, Cook T, Swedberg K, Burnett JC Jr, et al. Clinical course and predictive value of congestion during hospitalization in patients admitted for worsening signs and symptoms of heart failure with reduced ejection fraction: findings from the EVEREST trial. Eur Heart J. 2013;34:835–843. doi: 10.1093/eurheartj/ehs444 [DOI] [PubMed] [Google Scholar]
  • 44. Ambrosy AP, Fonarow GC, Butler J, Chioncel O, Greene SJ, Vaduganathan M, Nodari S, Lam CSP, Sato N, Shah AN, et al. The global health and economic burden of hospitalizations for heart failure: lessons learned from hospitalized heart failure registries. J Am Coll Cardiol. 2014;63:1123–1133. doi: 10.1016/j.jacc.2013.11.053 [DOI] [PubMed] [Google Scholar]
  • 45. Eapen ZJ, Reed SD, Li Y, Kociol RD, Armstrong PW, Starling RC, McMurray JJ, Massie BM, Swedberg K, Ezekowitz JA, et al. Do countries or hospitals with longer hospital stays for acute heart failure have lower readmission rates?: Findings from ASCEND‐HF. Circ Heart Fail. 2013;6:727–732. doi: 10.1161/CIRCHEARTFAILURE.112.000265 [DOI] [PubMed] [Google Scholar]
  • 46. Mentz RJ, Cotter G, Cleland JG, Stevens SR, Chiswell K, Davison BA, Teerlink JR, Metra M, Voors AA, Grinfeld L, et al. International differences in clinical characteristics, management, and outcomes in acute heart failure patients: better short‐term outcomes in patients enrolled in Eastern Europe and Russia in the PROTECT trial. Eur J Heart Fail. 2014;16:614–624. doi: 10.1002/ejhf.92 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 47. Greene SJ, Fonarow GC, Solomon SD, Subacius H, Maggioni AP, Bohm M, Lewis EF, Zannad F, Gheorghiade M, Investigators A, et al. Global variation in clinical profile, management, and post‐discharge outcomes among patients hospitalized for worsening chronic heart failure: findings from the ASTRONAUT trial. Eur J Heart Fail. 2015;17:591–600. doi: 10.1002/ejhf.280 [DOI] [PubMed] [Google Scholar]
  • 48. Filippatos G, Angermann CE, Cleland JGF, Lam CSP, Dahlstrom U, Dickstein K, Ertl G, Hassanein M, Hart KW, Lindsell CJ, et al. Global differences in characteristics, precipitants, and initial management of patients presenting with acute heart failure. JAMA Cardiol. 2020;5:401–410. doi: 10.1001/jamacardio.2019.5108 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 49. Kristensen SL, Martinez F, Jhund PS, Arango JL, Belohlavek J, Boytsov S, Cabrera W, Gomez E, Hagege AA, Huang J, et al. Geographic variations in the PARADIGM‐HF heart failure trial. Eur Heart J. 2016;37:3167–3174. doi: 10.1093/eurheartj/ehw226 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 50. Lee DS, Gona P, Albano I, Larson MG, Benjamin EJ, Levy D, Kannel WB, Vasan RS. A systematic assessment of causes of death after heart failure onset in the community: impact of age at death, time period, and left ventricular systolic dysfunction. Circ Heart Fail. 2011;4:36–43. doi: 10.1161/CIRCHEARTFAILURE.110.957480 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 51. Henkel DM, Redfield MM, Weston SA, Gerber Y, Roger VL. Death in heart failure: a community perspective. Circ Heart Fail. 2008;1:91–97. doi: 10.1161/CIRCHEARTFAILURE.107.743146 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 52. Heidenreich PA, Spertus JA, Jones PG, Weintraub WS, Rumsfeld JS, Rathore SS, Peterson ED, Masoudi FA, Krumholz HM, Havranek EP, et al. Health status identifies heart failure outpatients at risk for hospitalization or death. J Am Coll Cardiol. 2006;47:752–756. doi: 10.1016/j.jacc.2005.11.021 [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Tables S1–S6

Click here for additional data file. (170.9KB, pdf)

Articles from Journal of the American Heart Association: Cardiovascular and Cerebrovascular Disease are provided here courtesy of Wiley

RESOURCES