Predicting incident heart failure from population-based nationwide electronic health records: protocol for a model development and validation study

Abstract

Introduction

Heart failure (HF) is increasingly common and associated with excess morbidity, mortality, and healthcare costs. Treatment of HF can alter the disease trajectory and reduce clinical events in HF. However, many cases of HF remain undetected until presentation with more advanced symptoms, often requiring hospitalisation. Predicting incident HF is challenging and statistical models are limited by performance and scalability in routine clinical practice. An HF prediction model implementable in nationwide electronic health records (EHRs) could enable targeted diagnostics to enable earlier identification of HF.

Methods and analysis

We will investigate a range of development techniques (including logistic regression and supervised machine learning methods) on routinely collected primary care EHRs to predict risk of new-onset HF over 1, 5 and 10 years prediction horizons. The Clinical Practice Research Datalink (CPRD)-GOLD dataset will be used for derivation (training and testing) and the CPRD-AURUM dataset for external validation. Both comprise large cohorts of patients, representative of the population of England in terms of age, sex and ethnicity. Primary care records are linked at patient level to secondary care and mortality data. The performance of the prediction model will be assessed by discrimination, calibration and clinical utility. We will only use variables routinely accessible in primary care.

Ethics and dissemination

Permissions for CPRD-GOLD and CPRD-AURUM datasets were obtained from CPRD (ref no: 21_000324). The CPRD ethical approval committee approved the study. The results will be submitted as a research paper for publication to a peer-reviewed journal and presented at peer-reviewed conferences.

Trial registration details

The study was registered on Clinical Trials.gov (NCT 05756127). A systematic review for the project was registered on PROSPERO (registration number: CRD42022380892).

STRENGTHS AND LIMITATIONS OF THIS STUDY.

• Large and nationwide dataset representative of the UK primary care population.

• Investigation of regression and machine learning techniques for the derivation of a prediction model for incident heart failure in the short and long term.

• Candidate variable data types are deliberately limited to ensure widespread applicability of the model given the reality of ‘missing’ data in routinely collected electronic health records.

• This study is designed to fill an implementation gap to enable electronic health records to provide decision support to primary care physicians.

• The derivation and validation work will be undertaken in datasets collected in the UK; therefore, further validation work may be pursued for international contexts.

Introduction

An estimated 64.3 million people are living with heart failure (HF) worldwide,¹ and the prevalence of HF is projected to increase.² HF is the most common cause of unplanned hospital admissions in older persons, and is associated with reduced quality of life and premature mortality.^3–6 Advances in the treatment of HF have offered improvements in prognosis,^7–9 however, many cases of HF present and are diagnosed and treated late in course of the disease.^{2 10}

International guidelines define four stages of HF: Stage A HF (at-risk for HF), Stage B HF (pre-HF; structural heart disease without symptoms), Stage C HF (symptomatic HF), and Stage D HF (advanced HF).^{7 11} Mortality increases with progression through the stages. Accordingly, guidelines recommend initiatives to identify individuals with Stage A and Stage B HF as evidence supports that the onset of symptomatic HF can be delayed or prevented by targeting modifiable risk factors.¹²

In the UK, 98% of the populace are registered in primary care and have electronic health records (EHRs).¹³ A decision tool that exploits routinely collected EHR data across a population to calculate HF risk could offer a scalable, efficient and cost-effective approach to identifying individuals with Stage A/B HF.¹⁴ Previous models applicable to community-based EHRs to predict HF risk have been limited. Models have seldom been externally validated,^{15 16} which prohibits an understanding of their generalisability. Many have been developed in curated prospective cohort studies, and their performance may not translate to EHR data.^{16 17} Others include laboratory results (eg, natriuretic peptide measurement),¹⁸ specialist investigations (eg, cardiac magnetic resonance (CMR))¹⁹ or observations (eg, blood pressure and body mass index)^{17 20} that are missing in the majority of primary care EHRs and which may limit their scalability and applicability across the population.²¹ Predictive models developed using deep learning have yet to report calibration performance and may be limited in clinical application by explainability.²² Furthermore, models have either provided risk prediction over short (6 months) or long prediction horizons (10 years),^{16 22} and therefore may not be used to both inform targeting of diagnostics and primary prevention initiatives.

The Clinical Practice Research Datalink (CPRD) is an ongoing primary care database, established in 1987, that comprises anonymised medical records and prescribing data from a network of General Practices (GPs) across the UK.¹³ CPRD undertakes over 900 checks covering the integrity, structure and format of the daily GP data collection and is an optimal tool for undertaking real-world, population-based evaluations of healthcare as well as the development of clinical prediction models.^{13 23}

Developing a prediction model for HF from routinely collected primary care EHR data could offer several advantages. A model created from widely available data in routinely collected EHRs could be translated into clinical practice by being embedded into existing clinical EHRs. Furthermore, a model embedded in EHRs could give risk prediction for incident HF over the next 1–10 years that is updated each time an individual’s clinical situation changes (eg, age, diagnoses recorded), which more accurately reflects the dynamic nature of disease pathogenesis and clinical decision making.

Research Aim

The aim of the study is to develop and validate a model for predicting incident HF from national primary care EHRs(figure 1). Specifically, we wish to develop a model that is widely applicable and scalable in routinely collected community-based EHRs, test its performance across a range of prediction horizons, and externally validate it in a geographically distinct primary care EHR dataset.

Figure 1.

Schematic reporesentation of the FIND-HF study.

Methods and analysis

Data sources and permissions

The derivation dataset for training and testing the model will be the CPRD-GOLD dataset. This is an ongoing primary care database, established in 1987, that comprises anonymised medical records and prescribing data contributed by GPs using Vision software.¹³ It contains data for approximately 17.5 million patients, with 30% of contributing practices in England.¹³ The included patients are broadly representative of the UK general population regarding age, sex and ethnicity.¹³ In order to contribute to the database, GPs and other health centres must meet prespecified standards for research-quality data (‘up-to-standard’).^{13 24}

To ascertain whether the prediction model is generalisable, we will externally validate its performance in the geographically distinct CPRD-AURUM dataset. This was launched in 2017 and encompasses only practices using EMIS Web software. It contains data for approximately 26.9 million patients and draws on data collected from practices in England only.²⁵ Any practices which previously contributed to CPRD-GOLD have been removed from the CPRD-GOLD cohort to ensure that these datasets reflect different populations. CPRD undertakes various levels of validation and quality assurance on the daily GP data collection comprising over 900 checks covering the integrity, structure and format of the data.²⁵ The included patients are broadly representative of the UK general population regarding age, sex, deprivation and geographical spread.²⁵

There is the possibility that patients may transfer from a practice in GOLD to a practice in AURUM or vice versa, but the proportion of transfers is small. In the study, we will ensure that the study period starts from registration with a practice and is censored from the date of transfer out. Therefore, there is no overlapping period for the same patient in the training/testing set and the validation set.

Recorded information in both datasets includes patients’ demography, clinical symptoms, signs, investigations, diagnoses, prescriptions, referrals, behavioural factors and test results entered by clinicians and other practice staff. All clinical information is coded using Read Codes in CPRD-GOLD and SNOMED clinical terms (CT) in CPRD-AURUM.^{26 27} In the proposed study, extracted patients will have patient-level data linked to Hospital Episode Statistics (HES) Admitted Patient Care (APC) and Diagnostic Imaging Dataset (DID), Office for National Statistics (ONS) Death Registration, patient-level deprivation and practice-level deprivation to provide a more comprehensive dataset. The CPRD dataset has been used to develop or validate a range of risk prediction models, including in cardiovascular disease.^{23 28}

Patient and public involvement

Patients and public were not involved in the design of this research. However, we are convening a Scientific Advisory Board to include representatives from patients and public involvement groups, clinical experts, national health system leaders and EHR software providers to provide context advice on the research, dissemination of results and translation of the findings into clinical practice.

Inclusion and exclusion criteria

The study population will comprise all available patients in CPRD-GOLD and CPRD-AURUM eligible for data linkage and with at least 1-year follow-up in the period between 2 January 1998 and 28 February 2022. Patients will be excluded if they were under 16 years of age at the date of the first registration in CPRD, diagnosed with HF before 2 January 1998, registered for less than 1 year in CPRD or ineligible for data linkage.

Outcome ascertainment

The models will be developed to predict new onset HF. HF will be defined as the first presence of one or more of the clinical codes related to HF developed by consensus with clinical members of the research team. Code lists for HF will include Read codes and SNOMED CT in CPRD datasets, and the 10th revision of the International Statistical Classification of Diseases and Related Health Problems (ICD-10) codes in HES APC events and underlying cause of death variable in the ONS Death Registration data file. The first record of HF within the study period will be taken as the date of diagnosis (the index date). To that effect, in our analytical cohorts, there are about 100 000 HF cases in CPRD-GOLD and 800 000 HF cases in CPRD-AURUM. Misclassified data can lead to systematic prediction errors and accuracy of data may vary over time,²⁹ but CPRD has converted older ICD codes to the newer version, increasing confidence in their validity. Using incidence density sampling,³⁰ we will match HF cases by year of birth (±5 years) and sex with up to five controls in the same practice on the index date without a diagnosis of HF on that date.

Predictor variables

A systematic review is being conducted to identify candidate predictors for inclusion (PROSPERO: CRD42022380892). The potential predictors will include age, sex, ethnicity and all disease conditions during follow-up. Candidate disease conditions will comprise hospitalised diseases, such as other cardiovascular diseases, obesity, diabetes mellitus, thyroid disorders, iron deficiency and anaemia, kidney dysfunction, electrolyte disorders, chronic lung disease, sleep-disordered breathing, hyperlipidaemia, gout, erectile dysfunction, depression, cancer and infection.⁷ Code lists for predictors will be used from publications if available, otherwise, the CPRD code browser will be used and codes checked by at least two clinicians. The code lists for predictors in GOLD and AURUM will be adapted from CALIBER and HDR UK repositories or publications. If none are available from these sources then new code lists will be developed using the OpenCodelists and checked by at least two clinicians.

For diagnoses, if medical codes are absent in a patient record, we will assume that the patient does not have that diagnosis, or that the diagnosis was not considered sufficiently important to have been recorded by the GP in case of symptoms.³¹ Ethnicity information is routinely collected in the UK NHS and so has increasingly high completeness,³² and we will include an ‘ethnicity unrecorded’ category where it is unavailable because missingness is considered to be informative.³³ Accordingly, we do not expect any missing data for any of the predictor variables in the analytical cohort.

Sample size

To develop a prognostic prediction model, the required sample size may be determined by three criteria suggested by Riley et al.³⁴ For example, suppose a maximum of 200 parameters will be included in the prediction model and the Cox-Snell generalised R² is assumed to be 0.01. A total of 377 996 patients will be required to meet Riley’s criterion (1) with global shrinkage factor of 0.95; this sample size also ensures a small absolute difference (Δ<0.05) in the apparent and adjusted Nagelkerke R² (Riley’s criterion (2)) and ensures precise estimate of overall risk with a margin of error <0.001 (Riley’s criterion (3)). According to the Quality and Outcomes Framework, the prevalence of HF in England is 0.91%. Given an HF prevalence of 0.91%, only 3439 patients will be expected to develop HF from 377 996 patients. Therefore, the number of patients in the CPRD dataset with HF will provide sufficient statistical power to develop and validate a prediction model with the predefined precision and accuracy.

Data analysis plan

Data preprocessing

The CPRD-GOLD and CPRD-AURUM data will be cleaned and preprocessed for model development and validation, respectively. For categorical variables, we will address data quality issues such as inconsistent formatting and encoding errors, ensure categories are properly defined and resolve any inconsistencies in their representation to maintain data integrity. For patient features with binary values (sex and disease conditions), 0 and 1 will be mapped to the binary values. Continuous variable (age) will be kept as continuous and we will employ statistical techniques to identify potential outliers (including the use of z-scores and inspection of the distribution of the variables). Preprocessed patient-level data in CPRD-GOLD will be randomly split into an 80:20 ratio to create development and internal validation samples using the Mersenne twister pseudorandom number generator.

Descriptive analysis

We will perform descriptive analyses of all variables and test the statistical difference between cases and controls using the t-test for normally distributed continuous variables, Wilcoxon rank sum test for non-normally distributed a continuous variable (age) and Pearson’s chi-squared test for categorical variables, using a p≤0.05 to represent significance.

Prediction model development

Our focus is on using the logistic regression model because it offers a more manageable approach for implementation, interpretation and training compared with machine learning (ML) algorithms. However, we will compare the performance of the logistic regression model to a broad range of supervised ML techniques, including random forest, neural network, support vector machine, discriminants analysis and naïve Bayes classifier. We will check the assumptions of each ML method to assess its quality and whether it is appropriate for the data. To examine the comparative performance of the ML algorithms, we will apply Cochran’s Q test, which allows for the evaluation of multiple MLs. The primary prediction window will set at 1 year.³⁵ We will also explore prediction models with the length of the prediction window set at 5 and 10 years.

Internal validation

We will evaluate the model performance using a validation cohort with internal bootstrap validation with 200 samples. The AUROC will be used to evaluate predictive ability (concordance index) with 95% CIs calculated using the DeLong method.³⁶ Youden’s index will be established for the outcome measure as a method of empirically identifying the optimal dichotomous cut-off to assess sensitivity, specificity, positive predictive value and negative predictive value. We will calculate the Brier score, a measure of both discrimination and calibration, by taking the mean squared difference between predicted probabilities and the observed outcome. Calibration will be assessed graphically by plotting predicted HF risk against observed HF incidence at 1, 5 and 10 years. Overall ML performance, including distance between the predicted outcome and actual outcome, will be measured. Decision Curve Analysis will be used to assess whether the predictive model would do more benefit than harm.

Clinical utility will be examined by using net benefit analysis, where the harms and benefits of using a model to guide treatment decisions will be offset to assess the overall consequences of using the FIND-HF model for clinical decision making.³⁶ The model will be compared at 1 year, 5 years and 10 years with model blind methods of performing echocardiography for all patients, or not performing echo for all patients, regardless of HF risk. We will assess the net benefit across the full range of possible threshold probabilities with an HF risk. A priori we will set an HF risk at 1, 5 and 10 years as being the threshold of clinical interest, to align with the incidence of HF at these time points in routine practice.

The same methods will be employed in subgroups by age (<65 years, ≥65 years; <75 years, ≥75 years), sex (women, men), ethnicity (White, Black, Asian, others and unspecified) and HF phenotype (HF with preserved ejection fraction, HF with reduced ejection fraction) to assess the model’s predictive performance in these clinically relevant groups.

External validation of model

The CPRD-AURUM dataset will then be used to externally validate the model performance to assess generalisability. A lack of external validation has hampered the implementation of previous prediction models for HF in routine clinical practice.³⁷ The prediction model will be applied to each individual in the external validation cohort to give the predicted probabilities of experiencing HF at 1, 5 and 10 years. Prediction performance will be quantified by calculating the AUROC, Brier score, the observed to expected ratio, and by using calibration plots, and the same aforementioned clinical utility and subgroup analysis will be conducted. We will compare the performance against previously published models for incident HF that have been externally validated and are scalable in EHRs.³⁸

Software

All analysis will be conducted through Stata and R.

Ethics and dissemination

The study has been approved by CPRD (ref no: 21_000324). Those handling data have completed University of Leeds information security training. All analyses will be conducted in concordance with the CPRD study dataset agreement between the Secretary of State for Health and Social Care and the University of Leeds.

The study is informed by the Prognosis Research Strategy (PROGRESS) framework and recommendations.³⁹ The subsequent research paper will be submitted for publication in a peer-reviewed journal and will be written following Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) reporting guidelines and the CODE-EHR best-practice framework.^{40 41}

If the model demonstrates evidence of clinical utility, it could be made readily available through EHR system providers. As such, each time the model is called within an EHR system, the risk score should be updated with new information so that prediction of an individual’s HF risk is updated contemporaneously. The model could be a built-in tool for use in GPs for the targeted identification of individuals at high risk of developing new-onset HF. Future rigorous prospective study will be needed to assess the clinical impact and cost-effectiveness of this risk model.¹⁴ At the point when utilisation in clinical practice is possible, the applicable regulation on medicine devices will be adhered to.⁴¹ When in clinical use, the model itself could also be reviewed and updated by a prespecified expert consensus group on an annual basis after incorporating evidence from postservice utilisation and the curation of more data. The model will have to be updated as population characteristics change, data quality of EHRs improves and new or additional risk factors emerge.

Ethics statements

Patient consent for publication

Not applicable.