Role for machine learning in sex-specific prediction of successful electrical cardioversion in atrial fibrillation?

Abstract

Objective

Electrical cardioversion is frequently performed to restore sinus rhythm in patients with persistent atrial fibrillation (AF). However, AF recurs in many patients and identifying the patients who benefit from electrical cardioversion is difficult. The objective was to develop sex-specific prediction models for successful electrical cardioversion and assess the potential of machine learning methods in comparison with traditional logistic regression.

Methods

In a retrospective cohort study, we examined several candidate predictors, including comorbidities, biochemistry, echocardiographic data, and medication. The outcome was successful cardioversion, defined as normal sinus rhythm immediately after the electrical cardioversion and no documented recurrence of AF within 3 months after. We used random forest and logistic regression models for sex-specific prediction.

Results

The cohort comprised 332 female and 790 male patients with persistent AF who underwent electrical cardioversion. Cardioversion was successful in 44.9% of the women and 49.9% of the men. The prediction errors of the models were high for both women (41.0% for machine learning and 48.8% for logistic regression) and men (46.0% for machine learning and 44.8% for logistic regression). Discrimination was modest for both machine learning (0.59 for women and 0.56 for men) and logistic regression models (0.60 for women and 0.59 for men), although the models were well calibrated.

Conclusions

Sex-specific machine learning and logistic regression models showed modest predictive performance for successful electrical cardioversion. Identifying patients who will benefit from cardioversion remains challenging in clinical practice. The high recurrence rate calls for thoroughly informed shared decision-making for electrical cardioversion.

Key questions.

What is already known about this subject?

• No prediction tools that identifies patients with atrial fibrillation (AF) who may benefit from electrical cardioversion have been implemented in clinical practice. Accumulating evidence indicates that there are considerable sex differences in the epidemiology of AF. However, previous analyses have not accounted for the potential sex differences in prediction models.

What does this study add?

• Our sex-specific prediction models based on machine learning and logistic regression demonstrated different clinically important predictors between sexes, but the models did not demonstrate improved predictive ability compared with other existing models.

How might this impact on clinical practice?

• Identifying the patients who will benefit from cardioversion is a challenge in clinical practice. The high recurrence rate calls for thoroughly informed shared decision-making for electrical cardioversion.

Introduction

Atrial fibrillation (AF) is the most common sustained cardiac arrhythmia encountered in clinical practice. Electrical cardioversion is frequently performed to restore sinus rhythm and relieve symptoms in patients with persistent AF. Despite high rates of initial success of electrical cardioversion, more than half of the patients may have recurrence of AF within 1 year.^{1 2}

Electrical cardioversion is resource demanding and carries risks of thromboembolism and anaesthesia-related complications, and therefore, development of prediction models for successful cardioversion would be useful to inform shared decisions. Several characteristics have been identified as predictors of AF recurrence, such as high age, female sex, long duration of AF, heart failure, large left atrial size, hypertension, elevated body mass index, ischaemic heart disease, and chronic kidney disease.^2–4 However, the evidence of identification of patients who will benefit from cardioversion is weak and prediction of success is difficult in clinical practice.

Accumulating evidence indicates that there are considerable sex differences in the epidemiology of AF, presentation of AF, prognosis and in the response to antiarrhythmic therapy and radiofrequency ablation (RFA).^5–8 As the existing literature has examined ‘general’ predictors of successful cardioversion in cohorts including both sexes, the analyses have not accounted for the potential sex differences in prediction models. Development of separate models in women and men may be warranted to improve predictive accuracy. However, traditional regression approaches do not accommodate complex interaction between patient characteristics, but machine learning methods may be a promising new approach that gains acceptance in cardiovascular medicine.^9–11

Among patients with persistent AF, we aimed to develop sex-specific prediction models for successful electrical cardioversion and assess the potential of machine learning methods in comparison with traditional logistic regression.

Methods

Study population

We conducted a retrospective cohort study of consecutive patients diagnosed with persistent AF whose ongoing AF lasted for more than 48 hours or was of unknown duration. The patients were included with their first electrical cardioversion at Regional Hospital Central Jutland (Viborg Regional Hospital and Silkeborg Regional Hospital) in Denmark from August 2011 through March 2016. No exclusion criteria were applied. The criteria of 48 hours/unknown duration were applied because this cohort was originally established to examine use of oral anticoagulation and waiting time to electrical cardioversion.¹ From electronic medical records, we systematically collected clinical and laboratory data and information on medical history. The study population comprised both patients with a history of cardioversion and patients who underwent their first cardioversion ever. The patients were followed in a structured multidisciplinary AF clinic and were treated with anticoagulation in accordance with the European clinical guidelines.¹² Synchronised direct current electrical cardioversions were performed with paddles in anterior–posterior or anterior–lateral position. Energy levels ranged from 100 to 360 J using a biphasic defibrillator.

Candidate predictors

We used existing literature to select candidate predictors of successful cardioversion (table 1).^2–4 We assessed all information on potential predictors before the cardioversion procedure. Heart rate was measured on the date of cardioversion. The definition of excessive use of alcohol was more than eight drinks per week. Hypertension was defined as a history of hypertension or ongoing antihypertensive medication, diabetes mellitus as haemoglobin A1c level ≥48 mmol/mol or ongoing antidiabetic medication, and chronic obstructive pulmonary disease (COPD) as a pulmonary function test with irreversible airflow limitation, expressed as a FEV1/FVC ratio <0.7 after bronchodilatation, or ongoing inhalation treatment consistent with COPD. Echocardiographic data included left ventricular ejection fraction and left atrial diameter measured from the parasternal long axis. Echocardiograms were considered valid if the examination was performed within a period of 3 months before the cardioversion.

Table 1.

Baseline characteristics

	Women n=332	Men n=790
Age, years	71.0 (65.0–77.8)	67.0 (60.0–73.0)
Body mass index, kg/m2	27.0 (24.1–31.8)	27.8 (25.2–31.5)
Heart rate, min−1	93.0 (79.8–114.0)	91.0 (74.0–115.0)
Alcohol overuse, n (%)	23 (6.9)	110 (13.9)
Prior cardioversion and/or RFA, n (%)	99 (29.8)	292 (36.9)
Comorbidity, n (%)
Hypertension	195 (58.7)	481 (60.9)
Diabetes	28 (8.4)	114 (14.4)
COPD	58 (17.9)	78 (10.1)
Clinical scores, n (%)
CHA2DS2-VASc
0	23 (6.9)	83 (10.5)
1	10 (3.0)	200 (25.3)
≥2	299 (90.1)	507 (64.2)
HAS-BLED
0	63 (19.0)	299 (29.0)
1	192 (57.8)	360 (45.6)
≥2	77 (23.2)	201 (25.4)
Blood tests
Thyroid-stimulating hormone, IU/L	1.5 (0.9–2.4)	1.6 (1.1–2.4)
Haemoglobin, g/L	136.9 (127.2–148.1)	148.1 (136.9–156.2)
eGFR, mL/min/1.73 m2	66.0 (54.0–81.0)	72.0 (61.0–84.0)
Echocardiographic data
Left ventricular ejection fraction, n (%)
≥40 %	280 (84.6)	592 (75.0)
<40%	51 (15.4)	197 (25.0)
Left atrial diameter, cm	4.2 (3.8–4.5)	4.3 (4.0–4.7)
Medication, n (%)
Antiarrhythmic drugs
Class Ic	11 (3.4)	27 (3.5)
Class III	61 (18.8)	142 (18.3)
Beta-blocker	262 (80.6)	573 (73.9)
Non-dihydropyridine channel blocker	16 (4.9)	50 (6.5)
Digoxin	90 (27.8)	124 (16.0)
ACEI or ARB	148 (45.5)	392 (50.6)

Data are median (25%–75% percentiles) or counts (percentages).

Missing data: body mass index 0.5%; heart rate 1.0%; chronic obstructive pulmonary disease (COPD) 2.0%; thyroid-stimulating hormone 5.9%; haemoglobin 1.0%; estimated glomerular filtration rate (eGFR) 0.9%; left ventricular ejection fraction 0.2%; left atrial diameter 2.7%; antiarrhythmic drugs 2.0%; beta-blocker 2.0%; non-dihydropyridine channel blockers, 2.0%; digoxin 2.0%; ACE inhibitor (ACEI) 2.0%; angiotensin II receptor blocker (ARB) 2.7%.

CHA2DS2-VASc: Congestive heart failure, hypertension, age≥75 years, diabetes mellitus, prior stroke, tia or thromboembolism, vascular disease, age 65-74 years, and female sex. HAS-BLED: Hypertension, abnormal renal or liver function, prior history of stroke, prior major bleeding, labile INR, age>65 years, and drugs or alcohol.

RFA, radiofrequency ablation.

Outcome

The outcome was successful electrical cardioversion, defined as sinus rhythm immediately after the cardioversion and no documented recurrence of AF within 3 months. Following cardioversion, all patients were examined with telemetry for at least 2 hours. Afterwards, the patients could report recurrence of symptoms to the hospital that led to examination for AF. Over the 3-month follow-up, AF could be detected using ECG or Holter monitoring by clinical indication. At 3 months, all patients went to a control examination, which included examination with ECG.

Statistical methods

We performed all analyses in women and men separately.

We considered the following continuous predictors: age, body mass index, heart rate, thyroid-stimulating hormone (TSH) level, haemoglobin, estimated glomerular filtration rate (eGFR), and left atrial diameter. We also considered the following categorical predictors: alcohol use, history of prior cardioversion and/or RFA, hypertension, diabetes, COPD, low left ventricular ejection fraction (<40%), use of antiarrhythmic drugs, use of beta-blockers, use of non-dihydropyridine calcium channel blockers, use of digoxin, use of ACE inhibitor (ACEI), or angiotensin II receptor blocker (ARB).

First, we used a random forest algorithm.¹³ A random forest is composed of multiple individual decision trees that operate as an ensemble (online supplementary material). To derive the random forest, we split the data randomly into a 50% partition used for training and 50% used for validation. We tuned the number of iterations (ie, number of subtrees) and number of variables to examine randomly at each split. We used 500 iterations to obtain stable out-of-bag and validations errors. We then used the lowest validation error to determine the number of variables in each model. The random forest does not rely on selecting variables, but predictors have different relative importance in the prediction. Importance of a variable was assessed by minimal depth from the tree trunk. We displayed plots of importance score for each predictor.

Second, we applied logistic regression with backward elimination to estimate ORs with 95% CIs. To account for a relevant increase in a continuous covariate, we reported OR per population SD difference. We selected variables based on the Akaike information criterion, which is equivalent to selecting based on a p value<0.1570.¹⁴ We forced both age and atrial diameter into the models, as they are clinically important predictors.⁴

For both the random forest and logistic regression, we estimated the C statistic to assess model discrimination. We assessed calibration as the agreement between predicted and observed probabilities of successful cardioversion in deciles of predicted probabilities using the Hosmer-Lemeshow test.

We performed analyses on complete cases. Among women, 300 of 332 (90.4%) patients were complete cases, and among men, 717 of 790 (90.8%) patients were complete cases. All analyses were performed in Stata V.15.1.

Patient and public involvement

This study was performed without patient involvement. Patients were not invited to comment on the study design and were not consulted to develop patient relevant outcomes or interpret the results. Patients were not invited to contribute to the writing or editing of the manuscript for readability or accuracy.

Results

Baseline characteristics and cardioversion success

During the study period, 332 women and 790 men with persistent AF underwent electrical cardioversion. The median age was 71 years among the women and 67 years among the men. Fewer women (29.8%) than men (36.9%) had a history of prior cardioversion and/or RFA (p=0.02). Baseline characteristics are summarised in table 1.

Immediate restoration of sinus rhythm failed in 41 (12.4%) women and in 56 (7.1%) men (p=0.004). Electrical cardioversion was successful in 149 women (44.9%) and 394 men (49.9%) (p=0.13). Among the 233 female and 498 male patients without prior cardioversion or RFA, electrical cardioversion was successful in 106 women (45.5%) and 245 men (49.2%; p=0.35). No patients died or were lost to follow up.

Prediction of successful cardioversion using machine learning

In the final models, the out-of-back and prediction errors were 53.6% and 41.0% for the women and 50.4% and 44.8% for the men, respectively. Figure 1 shows the relative importance of each variable for the prediction of successful cardioversion in men and in women. Among women, the five most important predictors were age, haemoglobin, eGFR, hypertension, and antiarrhythmic class III drugs. C statistic of the random forest model for women was 0.59 (95% CI 0.51 to 0.68) and the model was well calibrated (Hosmer-Lemeshow p=0.94; figure 2A). The five most important predictors among men were haemoglobin, TSH, eGFR, age, and left atrial diameter. C statistic of the random forest model for men was 0.56 (95% CI 0.51 to 0.62) and the model was well calibrated (Hosmer-Lemeshow p=0.36; figure 2A).

Figure 1.

Importance score of predictor variables by sex. ACEI, ACE inhibitor; ARB, angiotensin II receptor blocker; BMI, body mass index; COPD, chronic obstructive pulmonary disease; eGFR, estimated glomerular filtration rate; Hgb, haemoglobin; LVEF, left ventricular ejection fraction; RFA, radiofrequency ablation; TSH, thyroid-stimulating hormone.

Figure 2.

Calibration plots. Agreement between predicted and observed probabilities of successful cardioversion in deciles of predicted odds.

Prediction of successful cardioversion using logistic regression

Among women, TSH, diabetes mellitus, and use of beta-blockers were retained as predictors in the multivariable model. We forced age and atrial diameter into the model. Table 2 shows the ORs for the selected model. The model had moderate discrimination with a C statistic of 0.60 (95% CI 0.54 to 0.67) and was well calibrated (Hosmer-Lemeshow p=0.42; figure 2B). Among men, backward elimination led to retainment of ventricular ejection fraction below 40% and use of ACEI or ARB as predictors. Additionally, we forced age and atrial diameter into the model. The final multivariable prediction model is given in table 2. Like the model among women, the discrimination was moderate, with a C statistic of 0.59 (95% CI 0.55 to 0.63) and the model was well-calibrated well (Hosmer-Lemeshow p=0.41; figure 2B).

Table 2.

Multivariable logistic regression models for successful cardioversion after electrical cardioversion

	OR (95% CI)*
Women
Age	1.06 (0.83 to 1.35)
Atrial diameter	0.85 (0.67 to 1.08)
Thyroid-stimulating hormone, IU/L	1.23 (0.96 to 1.58)
Beta-blockers	0.64 (0.36 to 1.15)
Diabetes mellitus	0.49 (0.20 to 1.22)
Men
Age	1.05 (0.90 to 1.22)
Left atrial diameter	0.74 (0.63 to 0.87)
LVEF <40% (vs ≥40 %)	1.46 (1.02 to 2.08)
ACEI or ARB	0.72 (0.53 to 0.97)

Among women, 300 of 332 (90.4%) patients were complete cases, and among men, 717 of 790 (90.8%) patients were complete cases.

*ORs associated with 1 SD increase were reported. SD for women: age 9.91; atrial diameter 0.56; thyroid-stimulating hormone 1.93. SD for men: age 10.12; atrial diameter 0.57.

ACEI, ACE inhibitor; ARB, angiotensin II receptor blocker; LVEF, left ventricular ejection fraction.

Comparison of machine learning and logistic regression predictions

The five most important variables according to the random forest model were different from the variables selected by the logistic regression model (figure 1 and table 2). Figure 3 shows the distribution of the predicted probabilities of successful electrical cardioversion for the random forest algorithm and logistic regression in men and women. Among the women, the prediction error of the multivariable logistic prediction model was 48.2% compared with 41.0% in the random forest algorithm, which corresponded to a difference of 7.2 percentage points in error rate. Among the men, the prediction error of the multivariable logistic prediction model was 46.6% compared with 44.8% in the random forest algorithm. This corresponded to a difference of 1.8 percentage points in error rate.

Figure 3.

Comparison between machine learning and logistic regression of predicted probabilities of successful electrical cardioversion, by sex.

Discussion

In this large real-world cohort study of consecutive patients with persistent AF, we demonstrated that electrical cardioversion was successful for 45% of the female patients and 50% of the male patients during 3 months of follow-up. A higher recurrence rate among women is in accordance with a recent study that demonstrated a higher burden of atrial fibrosis among women with AF, which is associated with structural remodelling and more advanced disease.¹⁵ The fact that most patients in our study were men may indicate a more conservative clinical approach in the management of women with AF. This tendency is consistent with other studies, in which women were scheduled for cardioversion less frequently than men.^{4 16}

We applied two approaches for the development of sex-specific prediction models for successful electrical cardioversion based on several candidate predictors available in routine clinical practice. The random forests and logistic regressions showed only moderate discriminative performance with C statistics of 0.59 and 0.60 among women, and 0.56 and 0.59 among men, respectively. Based on the clinical factors we included, the ability to predict which patients will benefit from electrical cardioversion is limited for both approaches. Furthermore, our results suggest that there are no substantial differences in the sex-specific predictive performance between the random forest and logistic regression.

Several studies have examined existing prediction models for successful/unsuccessful cardioversion or developed new prediction models.^17–21 However, we are not aware of any prediction models developed separately in women and men. Accordingly, we cannot compare our findings with the literature directly. Previous studies have supported that atrial diameter is an important predictor.^{2 4} Contrarily, our random forest analyses did not rank atrial diameter as one of the most important predictors (14th most important predictor among women and 5th most important among men). As we used left atrial diameter as a proxy for atrial size and the left atrium is not uniformly spherical, the atrial diameter may not fully reflect the volume of the left atrium. Volume measurements of the left atrium might be a stronger predictor of ineffective cardioversion than atrial diameter; however, to our knowledge, the predictive ability for successful cardioversion has not been compared between those two measures so far.

Machine learning methods have gained currency in the last years, and a staggered variety of approaches with different properties have been developed. In comparison with typical regression techniques, probably the most prominent property of the decision tree/random forest approach in the present study is the capability of handling high-order interactions between the investigated predictors (as illustrated in the example in the online supplementary material).²² Further details about statistical differences have been described elsewhere.²² Nevertheless, the presented data held little evidence that the random forest approach provided any meaningful improvement of the prediction of successful cardioversion. This finding agrees well with a recent systematic review, which examined the performance of machine learning compared with logistic regression for the development of prediction models and found no evidence of a superiority of machine learning.²³

Implications

Our results suggest that sex-specific identification of patients who will undergo successful cardioversion is challenging in routine clinical practice. Our sex-specific models found different clinically important predictors between sexes and approaches, but the models did not demonstrate improved predictive ability compared with other existing models. As our models only demonstrated moderate discrimination, further validation in external cohorts is needed.

Additionally, new approaches for developing prediction models should be considered. In a recent paper by Oto et al, the authors used a data mining algorithm to identify predictors of recurrence of persistent AF.²⁴ To our knowledge, no studies have so far applied data mining algorithms to develop sex-specific prediction models for successful electrical cardioversion. Another future approach may be the use of personalised computational modelling of arrhythmogenesis, which was recently applied among patients with persistent AF to identify ablation targets.²⁵ Theoretically, use of personalised computational models of the atria may be used to identify atrial substrate associated with a potential sustained effect of electrical cardioversion.

In contemporary medical practice, the shared decision of an electrical cardioversion needs to include a discussion on the high risk of recurrence of AF and willingness for exposure to antiarrhythmic drugs and ablation. Randomised trials have shown that rhythm control is not superior to rate control in the management of AF.^{26 27} Interestingly, these trials showed that mortality rates were significantly higher in women randomised to rhythm control as compared with men. Additionally, in relation to outcomes other than mortality, women assigned to rhythm control encountered worse outcomes compared with women in the rate control group.^{6 28 29} Therefore, since rhythm control is less beneficial in women and associated with more adverse events, patient sex may be important when deciding the optimal treatment strategy in AF.

Limitations

Our study had important limitations. Selection of patients for electrical cardioversion was based on shared decisions, which may reduce the generalisability of our results. For instance, our cohort included no multimorbid elderly patients and it is possible that patients with very enlarged atria and/or manifest heart failure were not scheduled for cardioversion. However, this way of recruiting patients reflects a real-world clinical setting. Information on other potential important predictors such as duration of AF or electrophysiological data was not available. We were not able to follow the patients with Holter monitoring or a loop/event recorder during the 3 months of follow-up and we had no information on the course of the patients’ symptoms.

Conclusions

In patients with persistent AF, sex-specific prediction of successful cardioversion is challenging. Machine learning and logistic regression models demonstrated modest predictive performance for successful electrical cardioversion. The high recurrence rate calls for thoroughly informed shared decision-making for electrical cardioversion, which should include a discussion about antiarrhythmic drug therapy and ablation in case of a failed cardioversion.