Identification of undiagnosed atrial fibrillation patients using a machine learning risk prediction algorithm and diagnostic testing (PULsE-AI): Study protocol for a randomised controlled trial

Abstract

Atrial fibrillation (AF) is associated with an increased risk of stroke, enhanced stroke severity, and other comorbidities. However, AF is often asymptomatic, and frequently remains undiagnosed until complications occur. Current screening approaches for AF lack either cost-effectiveness or diagnostic sensitivity; thus, there is interest in tools that could be used for population screening. An AF risk prediction algorithm, developed using machine learning from a UK dataset of 2,994,837 patients, was found to be more effective than existing models at identifying patients at risk of AF. Therefore, the aim of the trial is to assess the effectiveness of this risk prediction algorithm combined with diagnostic testing for the identification of AF in a real-world primary care setting. Eligible participants (aged ≥30 years and without an existing AF diagnosis) registered at participating UK general practices will be randomised into intervention and control arms. Intervention arm participants identified at highest risk of developing AF (algorithm risk score ≥ 7.4%) will be invited for a 12‑lead electrocardiogram (ECG) followed by two-weeks of home-based ECG monitoring with a KardiaMobile device. Control arm participants will be used for comparison and will be managed routinely. The primary outcome is the number of AF diagnoses in the intervention arm compared with the control arm during the research window. If the trial is successful, there is potential for the risk prediction algorithm to be implemented throughout primary care for narrowing the population considered at highest risk for AF who could benefit from more intensive screening for AF.

Trial Registration: NCT04045639

1. Introduction

Atrial Fibrillation (AF) – the most common sustained heart arrhythmia [1] – is associated with a five-fold increase in stroke risk [2] and enhanced stroke severity compared to patients without AF, resulting in greater disability and higher mortality [3,4]. As AF can be intermittent and asymptomatic it is challenging to diagnose. Data indicate that only 70–85% of patients living with AF have been formally diagnosed [5,6]. Early identification of undiagnosed AF and appropriate management including anticoagulation therapy is essential to reduce stroke risk in patients with AF.

Multiple screening approaches are available for the identification of AF, all with varying cost-effectiveness and diagnostic ability [7]. Screening tends to be opportunistic (e.g. screening of patients attending their general practitioner (GP) for another reason), targeted (e.g. screening of higher-risk patients), or systematic (e.g. screening all patients aged >65 years), and can include simple pulse checking or more resource-intensive electrocardiogram assessment (ECG). European guidelines recommend opportunistic screening via pulse taking or ECG rhythm strip in patients ≥65 years of age, and only recommend systematic ECG screening in individuals ≥75 years or those at high risk of stroke [7]. Opportunistic screening is recommended because it is more likely to be cost-effective than systematic population screening approaches [[7], [8], [9]]. However, opportunistic screening relies on patients visiting their GP for a different reason and pulse checking often lacks diagnostic precision [7,10], resulting in many patients with intermittent AF having a missed diagnosis, and others undergoing unnecessary further testing.

These limitations, alongside further uncertainties about the benefits of screening and the effectiveness of treatment for AF in patients identified through screening are likely reasons for the absence of a national screening programme for AF in the UK [11]. Therefore, given the estimated size of the undiagnosed AF population (approximately 300,000 individuals in the UK [5,13]), and the absence of cost-effective systematic screening approaches for AF, there is interest in methods that are cost-effective, diagnostically precise, and able to be applied across a population to narrow the patient population at highest risk of AF that should undergo more intensive screening for AF.

Existing risk prediction models for AF include the CHARGE-AF [14], ARIC [15], Framingham AF [16], and SAAFE [17] models. However, some of these depend on ECG-derived data [15,16], include many variables, and are not implemented in clinical practice because none of them are automated.

Previously we reported on the development of an AF risk prediction algorithm that utilised machine learning techniques [18]. The algorithm was trained and tested based on retrospective data of patients (≥30 years) listed on the UK Clinical Practice Research Datalink (CPRD) GOLD, and validated against existing AF prediction models (CHARGE-AF [14], ARIC [15], Framingham AF [16]). Compared with the best-performing existing AF model (CHARGE-AF [14]), the machine learning algorithm was more effective at identifying patients with AF, reducing the number of patients needed to screen to identify one case of AF (with 75% sensitivity) from 13 to 9. However, whether this risk prediction algorithm when combined with diagnostic testing is accurate at identifying undiagnosed AF in a real-world primary care setting is currently unknown. Therefore, the Prediction of Undiagnosed atriaL fibrillation using a machinE learning AlgorIthm (PULsE-AI) randomised controlled trial is designed to assess the real-world ability of the machine learning AF risk prediction algorithm coupled with diagnostic testing (ECG ± KardiaMobile) for the identification of patients with AF compared to routine care.

2. Study objectives

The primary objective of the PULsE-AI trial is to assess the effectiveness of an AF risk prediction algorithm and diagnostic testing (ECG ± KardiaMobile) for the identification of patients with AF in primary care, compared to routine care. The primary outcome/endpoint is the prevalence of confirmed AF (% with diagnosed AF) during the research window in intervention and control arms. The secondary objective is to assess the economic impact (healthcare resource utilisation, life years, and quality-adjusted life years) of using the AF risk prediction algorithm and diagnostic testing (ECG ± KardiaMobile) for the identification of patients with AF.

3. Methods

3.1. Study design

The PULsE-AI trial is a prospective, randomised controlled trial conducted in the UK primary care setting. Eligible adult patients registered at participating general practices within the NIHR Clinical Research Network: West Midlands will be identified from medical records and then individually randomised to receive routine care plus the intervention/screening strategy (intervention arm) or routine care only (control arm). Diagnostic data will be extracted from medical records of both intervention and control arm participants at the beginning and end of the research window to determine the proportion of patients diagnosed with AF throughout the trial in the intervention and control arms.

The research window is defined as the time between the first collection of patient medical records at the beginning of the trial (06 June 2019) and the last collection following the completion of the intervention at the end of the trial. This research window will not be fixed but will be sufficient to allow participating practices enough time to conduct the intervention. The research window will be consistent across sites and trial arms to ensure the same time period for AF diagnosis remains the same for all participants.

This protocol has been designed in accordance with the Standardized Protocol Items: Recommendations for Interventional Trials (SPIRIT) guidelines and checklist (Supplementary Marerial). A schedule for enrolment, intervention and assessment (SPIRIT figure) is outlined in Table 1 . A CONsolidated Standards of Reporting Trials (CONSORT) diagram is shown in Fig. 1 .

Table 1.

A schedule for enrolment, intervention and assessment according to the Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) indications.

Timepoint	Study period
	Enrolment	Allocation	Post-allocation
			Research clinic	Two weeks post-research clinic	Three-year follow-up
	−T1	T0	T1	T2	T3
Enrolment
Eligibility determination	X
Assessment of AF risk	X
Treatment allocation		X
Informed consent			X
Interventions
12-lead ECG (or 6-lead for housebound participants)			X	Xa
KardiaMobile			X	X
Assessments
Baseline patient characteristics and medical history			X
Diagnosis of AF			X	X	Xb

AF: atrial fibrillation; ECG: electrocardiogram.

−T1: activities prior to randomisation/treatment allocation, including determination of patient eligibility from medical records and generation of an AF risk score; T0: randomisation/treatment allocation; T1: research clinic appointment; T2 completion of KardiaMobile monitoring period; T3: follow-up to assess the number of AF diagnoses in intervention arm participants beyond the research window.

^aIf unsuitable for KardiaMobile monitoring (i.e. without access to a compatible smartphone/tablet).

^bData on related events (e.g. stroke, mortality etc.) may be collected alongside AF diagnoses.

Fig. 1.

PULsE-AI Trial flow chart according to CONsolidated Standards of Reporting Trials.

AF: atrial fibrillation, ECG: electrocardiogram, EMIS: Egton Medical Information Systems, GP: general practitioner.

3.2. Study population

Adult patients (≥ 30 years) without a diagnosis of AF prior to 06 June 2019 will be identified from the medical records of participating general practices. Patients aged ≥30 years were included because this age group were included in the development of the machine learning risk-prediction algorithm. Patients will be eligible for participation in the trial if they have a complete set of key clinical measurements (height, weight, body mass index (BMI), systolic blood pressure (SBP), and diastolic blood pressure (DBP); i.e. a valid index date) recorded during a rolling 12-month window ending at any time during the 11 years prior to the start of the research window). All eligible patients will then be randomised into intervention and control arms in a 1:1 ratio via simple randomisation. Intervention and control group patients with a risk score ≥ 7.4% will form the primary analysis population, and all randomised patients will be included in a sensitivity analysis. Following randomisation and AF risk score generation, general practitioners (GPs) will review the lists of patients randomised to the intervention arm at highest risk of AF before letters of invitation are sent, to ensure all patients are suitable to participate in the trial. GPs may choose not to invite patients based on their clinical judgment (e.g. patients receiving end of life care, patients with active cancer, patients whose mental health may be negatively affected by being invited to participate in the trial) and the rationale for exclusion will be captured. The response rate among invited higher risk intervention participants is expected to be ~30%.

3.3. AF risk prediction algorithm

The AF risk prediction algorithm was developed using machine learning techniques and data from a retrospective cohort of 2,994,837 adult patients (aged ≥30 years) without a history of AF (prior to January 2006) and listed on the CPRD GOLD between January 2006 and December 2016 [18]. During the study period, 3.2% of the cohort were diagnosed with AF. Both baseline (patient demographics (age, sex, race, smoking status), history of antihypertensive use, type 1 or type 2 diabetes, and cardiovascular comorbidities) and time-varying (recent cardiovascular event(s), recent BMI and change in BMI, recent pulse pressure, change in SBP and DBP, and recent frequency of SBP, DBP, and BMI recordings) patient data were incorporated into the machine learning algorithm to generate a risk score for AF, with the three most important predictors of AF found to be recent heart failure, coronary heart disease, and myocardial infarction [18] (Supplementary Fig. S1). The algorithm was calibrated [19] during development, using cross-validation to prevent overfitting and random sampling to define training and holdout sets, ensuring the final set was broadly representative of what the model was trained on. The risk prediction algorithm will be run on data from all eligible patients with a valid index date following randomisation, and a risk score will be generated for each individual. A risk score of ≥7.4% was chosen as the threshold needed for the AF risk prediction algorithm as this corresponded to a sensitivity of 50% (Supplementary Table S1), as assumed in the sample size calculations.

3.4. Intervention group

Intervention group participants with a risk score ≥ 7.4% and suitable for participation in the study will be invited to attend a research clinic for diagnostic testing. Following informed consent, participant characteristics and clinical history will be recorded by the research nurse, and a 12‑lead ECG performed. Intervention arm participants not diagnosed with AF following the 12‑lead ECG will be provided with a KardiaMobile device (AliveCor Inc., California, USA), if they have access to a compatible smartphone or tablet. Those without access to a compatible smartphone or tablet will be offered a loan smartphone to use alongside the KardiaMobile for the study period and, if declined, will be invited for up to two further 12‑lead ECGs. Participants using the KardiaMobile will be asked to record their ECG twice daily (morning and evening), in addition to any time they feel unwell, for two weeks. Participants will be diagnosed with AF by a cardiologist if they have ≥30 s of arrhythmia, typical of AF, as per standard guidelines [12,20]. In addition, participants who are unable to attend the research clinic (e.g. because they are housebound or do not have available transport) will be offered a visit at home with a portable ECG, followed by the two weeks of KardiaMobile monitoring described above. All intervention arm participants with a cardiologist-confirmed diagnosis of AF (either from the 12‑lead ECG or KardiaMobile data) will have met the primary endpoint for the study and will be evaluated for anticoagulation therapy as per local procedures. Intervention arm participants may also be diagnosed with AF through routine clinical practice. In addition, participants diagnosed with other cardiac abnormalities will also be referred and treated according to local policies.

3.5. Control group

Control arm participants will have no direct contact with investigators during the study. Control arm participants will only be diagnosed with AF through routine clinical practice, and any diagnoses will be made according to local policies.

3.6. Statistical methods

3.6.1. Sample size

Owing to the pragmatic design of the trial, the recruitment of six study sites will result in an estimated 24,000 patients who meet the inclusion criteria. Based on data from the AF risk prediction algorithm development [18], it is assumed that approximately 1000 patients per arm will be at higher risk of AF (risk score ≥ 7.4%). Of these 1000 intervention arm participants, it is estimated that: ~30% will accept the invitation to attend the research clinic (see Supplementary Fig. S2 for a summary of estimated patient and participant numbers in each cohort of the trial); 1 in 16 of those who attend the clinic will be diagnosed with AF; and – over a six month period – 0.7% of those who do not attend the research clinic will be diagnosed with AF [18]. Thus, assuming an AF diagnosis rate of 2.4% in higher risk patients in the intervention arm and 0.7% in the control arm, and a 5% Type-I error rate, this yields a statistical power for the study of 88.5%. Power calculations were conducted using the ‘pow’ package in R version 3.4.2.

3.6.2. Statistical analysis

Demographics and clinical characteristics for all randomised participants, and for the primary analysis group of higher risk participants (risk score of ≥7.4%), will be summarised using descriptive statistics. If the baseline characteristics of the intervention and control arms are well balanced, the proportion of patients diagnosed with AF during the research window will be compared between the intervention and control arms via an unadjusted chi-squared comparison. Alternately, if there are baseline differences between arms, these will be adjusted for within a penalised logistic regression. A sensitivity analysis will also be performed on all participants randomised into intervention and control groups irrespective of predicted AF risk score. All statistical tests will be conducted at the 5% significance level.

The primary outcome will be utilised to inform an economic analysis designed to evaluate the health economic impact of a screening strategy using the AF risk prediction algorithm (combined with a diagnostic test(s)) for the identification of patients with undiagnosed AF (secondary objective). The cost-effectiveness model will compare the use of the AF risk prediction algorithm intervention as an add-on to current practice against current practice alone. The model will use AF incidence rates determined from this trial alongside data from published literature (for variables such as AF-related event rates and cost estimates) to evaluate outcomes over a patient's lifetime. The economic value of the AF risk prediction algorithm will be captured through the estimation of total costs, life years gained as a result of earlier diagnosis of AF, and quality-adjusted life years gained as a result of avoiding AF-related events due to earlier diagnosis.

3.6.3. Data management

In line with ethical approval requirements, pseudonymised patient-level data from participating study sites will be extracted from the Egton Medical Information System (EMIS) medical records of all eligible patients via a search and report function. Eligible patients without a valid index date will be excluded prior to the randomisation of patients at each study site. Following randomisation, data will be used to generate AF risk scores for all participants in the intervention and control arms, and those at highest risk of AF (risk score ≥ 7.4%) identified. Study sites will then de-anonymise records of the highest risk patients in the intervention arm for invitation. Once participants have signed informed consent, data collected during and after the research clinic appointment will be entered into the EMIS electronic medical records for each participant. All ECG data will be pseudonymised for cardiologist review. All KardiaMobile data will be uploaded to secure European Servers based on unique study identifiers. Pseudonymised data will be shared with the trial sponsor (and its delegates) using secure file transfer protocol (sFTP), which will ensure an end to end encryption of data extracted. Three years of additional follow-up data may be collected from the medical records of participants identified as higher risk by the AF risk prediction algorithm to evaluate the impact of the intervention on clinical outcomes, but will not involve direct patient contact. Consent for all pseudonymised data sharing outside of that collected at the research clinic has been provided by the general practice participating in the study.

3.7. Ethical approval and trial registration

Ethical approval for this study was granted on 28 February 2019 (Wales Research Ethics Committee 5 - Bangor). Health Research Authority (HRA) and Health and Care Research Wales (HCRW) approval for the study were granted on 09 April 2019. The trial was registered with clinicaltrials.gov (NCT04045639) on 05 August 2019. This manuscript is aligned with version 3 of the protocol; substantial protocol amendments were approved on 12 December 2019 and 08 February 2020.

4. Discussion

The PULsE-AI trial is designed to assess the performance of a machine learning AF risk prediction algorithm alongside diagnostic testing (12‑lead ECG ± KardiaMobile) at identifying and detecting patients with undiagnosed AF, and thus evaluate a complete screening strategy that could be applied in a current UK primary care setting. Whilst the AF risk prediction algorithm assessed in this study has been shown to reduce the potential number of patients needed to screen to detect undiagnosed AF by 31% [18] following theoretical validation, whether it is similarly effective at identifying patients at higher risk of AF within a real-world primary care setting is currently unknown.

Current AF screening approaches are limited by either cost-effectiveness or diagnostic precision [8,12]. Emerging technologies such as the KardiaMobile monitor and smartwatches are now widely available and increase the accessibility of home-based ECG recording; however, they can be expensive and may be technologically prohibitive to older patient populations who are at higher risk of AF. Indeed, the recent Apple Heart Study [21] invited people with access to a smartwatch to participate in a study. The study sought to measure the proportion of participants with an irregular pulse notification – and confirm AF or atrial flutter with an ECG. However, this study was skewed in favour of young to middle age adults who were most likely to have the technology, reducing the applicability of findings to older populations at highest risk of AF.

Machine learning techniques appear to be effective in the identification of undiagnosed AF [[22], [23], [24]], however all known studies involved the application of machine learning algorithms to ECG data. The novelty of this study is that the AF risk prediction algorithm does not require ECG data as an input and utilises routinely collected data contained within existing medical records. If successful, this approach which does not rely on existing ECG data or on patient access to emerging technologies could be of significant value to patients, clinicians, and budget-constrained healthcare systems alike.

The success of the study depends on several factors. First, the assumptions made during the sample size and power analysis calculations are correct. Calculations have been made based on observations from the AF risk prediction algorithm study [18], and yield a statistical power of 88%. Any changes in these assumptions could potentially affect the number of AF diagnoses made during the research window and therefore the primary outcome. Second, all calculations were made based on the population distribution as per the CPRD GOLD dataset used to develop the risk prediction algorithm. Therefore, if local populations vary significantly from the CPRD GOLD dataset (e.g. they are younger), more practices may need to be recruited to reach a sample size of approximately 1000 higher risk participants per arm. Third, AF will be diagnosed during the study via 12‑lead ECG or the KardiaMobile device. Whilst participants without a personal smartphone or tablet (or access to a device via a family member) will be offered a loan smartphone for the KardiaMobile monitoring period, some may lack confidence with technology and either decline the offer or are unable to use it correctly. Although these participants will be invited for subsequent 12‑lead ECGs, the absence of home-based ECG monitoring may reduce the ability to confirm the actual presence or absence of AF.

4.1. Limitations

This trial will be subject to several limitations. As with all multicentre studies, the generalisability of trial findings to patients/settings beyond the participating practices and surrounding areas is uncertain. Therefore, generalisations will be made with caution and both known and unknown differences in the study population that may impact the applicability of the trial will be discussed.

Due to the nature of the trial, patients may be ‘flagged’ as high risk of AF by the algorithm but are not subsequently diagnosed with AF. However, given the often-intermittent nature of AF, and/or the absence of a compatible smartphone in some participants, it will not be possible to confirm a genuine absence of AF in this subset of patients. Furthermore, it is possible that patients identified by the algorithm as higher-risk may have been picked up later by routine methods (lead-time bias). However, as both the intervention arm and the control arm have the same chance to identify AF patients opportunistically, we should be able to quantify this bias.

Due to the individually randomised design, both intervention and control participants will be present at study sites. It is, therefore, possible that physicians may increase their efforts to diagnose patients beyond what would be considered routine care as a result of being involved in an AF study (performance bias). To ensure the control arm closely reflected routine clinical care, we only recruited practices that were not active in cardiovascular-related research at the time of trial site recruitment. Practices were not restricted from participating in any local, regional or national AF detection initiatives during the course of the trial.

Every effort will be made to ensure that patients in the control arm and patients in the intervention arm will receive routine care as is usually delivered in clinical practice through appropriate training of practice researchers. Additionally, participants will be informed not to allow family or friends to use their KardiaMobile device as it could artificially inflate the number of diagnoses in either the control or intervention arms.

Finally, the COVID-19 pandemic may impact on trial recruitment because of the suspension of all non-essential services in primary care in England. Whether this will have an impact on the trial beyond simply extending the trial period is currently unknown.

5. Conclusion

Timely diagnosis of AF and optimal anticoagulation therapy (where required) can reduce the risk of AF-related events such as stroke. If the PULsE-AI trial is successful, the algorithm could become a valuable diagnostic adjunct, implementable in primary care for narrowing the population considered at highest risk of AF who should undergo more intensive testing (e.g. with a 12‑lead ECG), and therefore enable a more cost-effective strategy of screening for undiagnosed AF.

6. Trial status

Currently, six general practices are enrolled in the study. Patients at these practices have been randomised, and those at highest risk of AF identified. The first participant attended the research clinic in August 2019. Recruitment was placed on hold on 16 March 2020 due to the COVID-19 pandemic. The trial restarted on 21 July 2020, with some modifications (e.g. the provision of home visits) to minimise the risks to participants and the research team whilst operating within local and national guidelines.

Ethics approval and consent to participate

This study will be performed in accordance with the Declaration of Helsinki. Ethical approval has been obtained from the NHS Health Research Authority and Health and Care Research Wales (HCRW) (IRAS project ID: 252934). The trial has been registered with clinicaltrials.gov (NCT04045639).

Participating General Practices have already provided consent on behalf of their patients for pseudonymised data to be used for research. Informed consent will be obtained for all participants randomised to the intervention group who accept the invitation for diagnostic testing before the collection of any data from the research clinic appointment.

Funding

This trial is supported by Bristol Myers Squibb UK.

CDM is funded by a National Institute for Health Research (NIHR) Research Professorship (NIHR-RP- 2014-04- 026). CDM is also funded by the NIHR Applied Research Collaboration (West Midlands) and the NIHR School for Primary Care Research. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR, or the Department of Health and Social Care.

Author contributions

All authors contributed to the design of the study protocol. All authors read and approved the final manuscript. NRH, CA, LBH, AJC, DC, WD, UF, JG, LG, MH, CM, ACM, JR, DMS, and ATC are all members of the steering committee. NRH, LBH, UF, JG, LG, MH, SLawton, SLister, CM, BS, and DMS are all members of the logistics committee. AJC and ACM are responsible for endpoint adjudication. NRH, UF, and ATC are responsible for overall trial management.

Declaration of Competing Interest

AJC has received institutional grants and personal fees from Bayer, Boehringer Ingelheim, Pfizer/BMS and Daiichi Sankyo, and personal fees from Portola.

ACM has received consulting fees and a grant from Alliance Bristol-Myers-Squibb/Pfizer, and consulting fees from Bayer-Healthcare and Boehringer-Ingelheim.

DC is AI Research Director of Sensyne Health plc.

ATC receives consulting fees from AbbVie, ACI Clinical, Aspen, Bayer, Boehringer-Ingelheim, Bristol-Myers Squibb, Boston Scientific, CSL Behring, Daiichi-Sankyo, GlaxoSmithKline, GLG, Guidepoint Global, Johnson and Johnson, Leo Pharma, Medscape, McKinsey, Navigant, ONO, Pfizer, Portola, Sanofi, Takeda, Temasek Capital, TRN; advisory board membership with Bayer, Bristol-Myers Squibb, Daiichi-Sankyo, Johnson and Johnson, ONO, Pfizer, Portola, Sanofi; payments for lectures including speakers bureau services, payments for preparation of reports and payment for development of educational presentations from, Aspen, Bayer, Boehringer-Ingelheim, Bristol-Myers Squibb, Daiichi, GlaxoSmithKline, Johnson and Johnson, Medscape, Pfizer and Portola. He has advised the UK Government Health Select Committee, the all-party working group on thrombosis, the Department of Health, and the NHS, on the prevention of VTE. He is also an advisor to Lifeblood: the thrombosis charity and is the founder of the European educational charity the Coalition to Prevent Venous Thromboembolism.

NRH, UF, SLister, KGP and BS are employees of Bristol Myers Squibb UK.

LBH, JG, LG, MH, PM and DMS are employees of Health Economics and Outcomes Research Ltd. Health Economics and Outcomes Research Ltd. received fees from Bristol Myers Squibb UK in relation to this study.

CM and SLawton are employees of Keele Clinical Trials Unit. Keele Clinical Trials Unit received fees from Bristol Myers Squibb UK to support this study.

Appendix A. Supplementary data

Supplementary material