Development and validation of a deep-learning algorithm for rule-in and rule-out coronary artery disease based on electrocardiogram without evidence of myocardial ischemia

Graphical abstract

Development and validation of a deep-learning algorithm for rule-in and rule-out coronary artery disease based on raw electrocardiogram signals without evidence of myocardial ischemia. ECG, electrocardiogram; AUC, area under the receiver operating characteristic curve.

Highlights

• •An artificial intelligence algorithm based on ECG without myocardial ischemia evidence is predictive of CAD.
• •An AI-based decision-making protocol could achieve the guideline-recommended performance in guiding rule-out or rule-in further examination.
• •Limb leads/I-lead ECG with age and gender variables and without additional examinations, demonstrated robust CAD predictive performance.
• •Our externally validated algorithm based on ECG without myocardial ischemia evidence performed good in CAD detection.
• •An algorithm-based decision-making protocol may improve initial CAD triage accuracy and reduce non-essential tests for any settings with an ECG.

Abstract

Background

Current coronary artery disease (CAD) guidelines recommend to rule-out or rule-in patients for further examination by assessing a pretest probability (PTP) ≤ 5 % or ≥ 15 %. We developed and validated a deep-learning algorithm for rule-in or rule-out based on electrocardiogram (ECG) without myocardial ischemia evidence.

Methods

Between October 2019 and June 2022, data from two centers (Fuwai Hospital [Beijing] and Yunnan Fuwai Hospital) of CAD-suspected patients undergoing either coronary angiography or coronary computed tomography were used. Data from the Fuwai Hospital (Beijing) were used to train (randomly 90 %) and internally validate (randomly 10 %) a deep-learning algorithm to detect CAD (≥ 70 % stenosis) based on 12-lead ECGs. An algorithm-based decision-making protocol was established for rule-out or rule-in based on a predefined threshold allowing for a 95 % negative predictive value (NPV). Data from the Yunnan Fuwai Hospital were used to externally validate the performance of the decision-making protocol. The CAD prevalence was calculated in patients who were recommended to rule-in or rule-out.

Results

In internal validation set, area under the receiver operating characteristic curve (AUC) was 0.81 and the CAD prevalence of patients who were recommended rule-out and rule-in were 5 % (40/790) and 23 % (527/2253), respectively. In external validation set, the CAD prevalence of patients who were recommended rule-out and rule-in were 0 % (0/661) and 15 % (255/1699), respectively.

Conclusions

Our algorithm based on ECG without myocardial ischemia evidence performed good in CAD detection. An algorithm-based decision-making protocol could achieve the guideline-recommended performance in guiding rule-out or rule-in for further examination.

1. Introduction

Current American and European guidelines recommend pretest probability (PTP) stratification to guide further diagnostic testing [1,2]. Patients with a PTP ≤ 5 % can be assumed to have such a low probability of CAD that diagnostic testing can be deferred unless for other compelling reasons [2]. Whereas, patients in the initial diagnostic management with a PTP ≥ 15 % are generally recommended to undergo further assessment [3]. However, recent studies have demonstrated that, when tested with the updated Diamond-Forrester prediction tool recommended for PTP stratification, the true observed prevalence of obstructive CAD in patients with a PTP ≥ 15 % has often been < 5 % [4,5]. This PTP overestimation could lead to potential over-utilization of downstream non-invasive and invasive assessments, posing additional risks for patients as well as unnecessary resource expenditure for health care systems. Thus, there is an imperative need to improve the accuracy of PTP prediction to ensure cost effectiveness of the diagnostic workflow.

As a non-invasive, cheap and convenient examination tool, electrocardiogram (ECG) is widely used in clinical practice for the initial evaluation of suspected CAD. However, patients with clinically significant CAD often have a normal resting ECG [6]. Actually, apart from apparent ECG changes suggesting myocardial ischemia, various non-specific ECG abnormalities potentially linked to CAD are also often observed in ECGs that are otherwise considered “normal” for lacking clear evidence of myocardial ischemia [7]. Examples include inverted T waves, Wellens’ type T-wave, non-specific ST deviation with T-wave change, inverted U waves, etc [[8], [9], [10], [11], [12]]. However, the use of such non-specific ECG abnormalities for CAD assessment has been restricted by (i) limited studies validating their exact value and reliability in CAD diagnosis, (ii) the lack of consensus definitions or quantifiable severity grading for implementation, and (iii) poor reproducibility in human identification. With the evolution of artificial intelligence technology, deep learning algorithms have emerged as a promising tool for disease prediction by analyzing and interpreting ECG [[13], [14], [15], [16]].

Therefore, this study aimed to develop and validate a deep-learning algorithm to rule-in and rule-out suspected CAD patients for further assessment based on their ECGs without clear evidence of myocardial ischemia, in accordance with the guideline-recommended performance requirements.

2. Methods

2.1. Study design

We conducted a multicenter cross-sectional study. Data were retrospectively obtained from campuses of the Fuwai Hospital system, the Beijing Fuwai Hospital is in Northern China and the Yunnan Fuwai Hospital is in Southwest China. This study was approved by the institutional review boards of Fuwai Hospital.

2.2. Study population and data sources

We included all patients suspicion for CAD undergoing elective coronary angiography or coronary computed tomography angiography (cCTA) with at least one digital, standard 10-second, 12-lead ECG acquired at one of two centers before the tests. Exclusion criteria included the following: (i) prior percutaneous coronary intervention (PCI); (ii) prior coronary artery bypass graft (CABG); (iii) other heart diseases (e.g. congenital heart disease, valvular heart disease, or aortic disease); (iv) ECG was diagnosed as abnormal Q wave or ST-segment changed significantly (ST-segment elevation/depression > 0.1 mV in at least 2 contiguous ECG leads); (v) missing or unqualified data. ECG signals were recorded at a sampling rate of 500 Hz using multiple different machines. All ECGs were performed before coronary angiography or cCTA.

2.3. Study setting

Eligible ECGs from the Fuwai Hospital (Beijing) were randomly divided into training (90 %) and internal validation (10 %) sets for algorithm development, consistent with prior research setting experience [17]. The entire eligible data from the other center were divided into the external validation set. The internal validation set is employed to define the decision threshold aligned with the protocol's intended purpose (rule out PTP ≤ 5 %) recommended by guideline. Meanwhile, the external validation set is utilized to assess the protocol's practical application, specifically in guiding the decision to either rule in or rule out further examinations for CAD.

2.4. Labeling

All enrolled patients were dichotomized according to the presence of CAD, which was defined as luminal cross-sectional area stenosis of ≥ 70 % in at least 1 major epicardial vessel or ≥ 50 % in the left main stem based on coronary angiography or cCTA [18]. Two radiologists who were blinded to the study design independently reviewed each patient’s angiogram or cCTA to assess the degree of coronary artery stenosis, with any disputes settled via review by a third radiologist for consensus decision.

2.5. ECG pre-processing

ECGs were collected according to a standardized protocol (Supplementary material, Method 1). The quality of ECGs was assessed according to the established criteria in Supplementary material, Method 2. Unqualified ECGs were excluded from the final analysis. Qualified ECGs underwent further pre-processing to remove noise and ensure uniformity of signal quality (Supplementary material, Method 2), including eliminating power-line interference, drift noise and Z-score normalization.

2.6. Development of the models

Deep neural networks were used to design the ECG algorithm for CAD detection (Supplementary material Fig. S1). The model mainly consisted of a feature extraction module, a feature fusion module and a prediction module. Firstly, feature extraction module captured local and global features from the preprocessed ECGs. Subsequently, the local and global features were fused by feature fusion module. Finally, the prediction module took in the fused features and output prediction probability of CAD. During training phase, parameters of the model were updated to minimize prediction error between output and ground truth labeled through medical examination and checked by physicians. This process was repeated until the model obtained best performance on validation set. The details regarding the development of the model are provided in Supplementary material, Method 3.

In the determination of the threshold, in accordance with current guidelines recommending rule-out for patients with PTP ≤ 5 % [1,2], we assumed that the prevalence rate below this threshold was less than 5 %. The protocol was derived from the internal validation set and subsequently validated for obstructive CAD in the external validation set.

We also established three other CAD detection models. First, we fit a logistic regression model based on age, sex, ST deviation and T-wave abnormality for performance comparison. Second, we devised two simplified hybrid models incorporating both ECG and clinical variables to investigate the impact of using fewer leads in predicting CAD. The 6-lead algorithm includes data from six leads (I, II, III, aVR, aVL, and aVF) along with age and sex, while the I-lead algorithm utilizes data from the I-lead ECG, age, and sex.

2.7. Evaluating the models

We used the CAD prevalence of the rule-out group and rule-in group divided by the predefined threshold to evaluate whether the model can achieve the guideline-recommended performance. We also calculated the sensitivity, specificity, negative predictive value (NPV), positive predictive value (PPV) and area under the receiver operating characteristics curve (AUC). In the external validation cohort, we estimated Diamond-Forrester (D-F) model PTP using patient admission diagnoses to infer anginal symptom combined with age and sex. Comparative performance analyses between our algorithm and the D-F model were specifically conducted for the subgroup with intermediate PTP values (5–15 %).

2.8. Statistical analysis

Data were presented as mean ± standard deviation for continuous variables and percentages for discrete variables. Categorical variables were compared using chi-square or Fisher’s exact tests, and continuous variables were compared using t or Mann–Whitney U tests. We used Delong tests to compare the AUC of different models and to calculate the 95 % confidence intervals (CIs) of the AUC values. For sensitivity, specificity, NPV and PPV, the 95 % CIs were calculated using the Wilson score method. Pre-specified subgroup analyses were conducted according to age, sex, ST deviation and T-wave abnormality.

All comparisons were two-sided, with statistical significance defined as P < 0.05. Analyses were calculated using SAS version 9.4 (SAS Institute Inc.).

3. Results

3.1. Study data

Between November 2019 and April 2022, 26,856 patients with 30,425 ECGs met the criteria for inclusion were enrolled from the Fuwai Hospital (Beijing). The baseline characteristics were similar in the training set and internal validation set (Table 1). Between October 2019 and June 2022, 2360 eligible patients were enrolled from the other center for inclusion in the external validation set (Fig. 1). Compared with patients in the training set, those in the external validation set were younger and less likely to be male, have ST deviation and T-wave abnormality in ECGs.

Table 1.

Baseline characteristics.

Characteristics	Training N = 27382	Internal validation N = 3043	P-valuea	External validation N = 2360	P-valueb
Age	56.5 ± 12.2	56.4 ± 12.1	0.41	54.9 ± 11.7	＜0.001
Male	15,245 (55.7)	1711 (56.2)	0.56	1243 (52.7)	0.01
ST deviation	5854 (21.4)	647 (21.3)	0.88	194 (8.2)	＜0.001
T-wave abnormality	6879 (25.1)	744 (24.4)	0.42	493 (20.9)	＜0.001
Coronary stenosis (70 %)	5050 (18.4)	567 (18.6)	0.80	255 (10.8)	＜0.001
Diagnosed by Coronary angiography	2664 (52.8)	308 (54.3)		201 (78.8)
Diagnosed by Computed tomography	2386 (47.2)	259 (45.7)		54 (21.2)
Coronary test			0.44		＜0.001
Coronary angiography	3899 (14.2)	449 (14.8)		508 (21.5)
Computed tomography	23,483 (85.8)	2594 (85.2)		1852 (78.5)

Data presented as mean ± standard deviation or n ( %). No data were missing in Table 1.

^aP-value was obtained by comparison of the training and internal validation sets.

^bP-value was obtained by comparison of the training and external validation sets.

Fig. 1.

Study Flowchart. CABG, coronary artery bypass grafting; PCI, percutaneous coronary intervention.

3.2. Internal validation and determining the decision threshold for rule-in and rule-out

The AUC of the algorithm and other models are presented in Fig. 2. The algorithm achieved an AUC of 0.81 (95 % CI, 0.78–0.83) in the internal validation set, which was higher than the AUC of the logistic regression models based on age sex, ST deviation and T-wave abnormality (0.81 vs 0.72, P < 0.001).

Fig. 2.

Algorithm performance for detecting coronary artery disease and determination of decision threshold in the internal validation set. AUC, area under the receiver operating characteristic curve; NPV, negative predictive value; PPV, positive predictive value.

The decision threshold for rule-in and rule-out was 0.29 and had a 95 % NPV (95 % CI: 93–97), 23 % PPV (95 % CI: 22–25), 30 % specificity (95 % CI: 29–32), 93 % sensitivity (95% CI: 91–95) in internal validation set.

3.3. External validation of the decision threshold for further CAD examination

The numbers and percentages of rule-in and rule-out patients in the Internal Validation and External Validation Sets are shown in Supplementary material, Table S1. The discriminative performance of the predefined threshold in the external validation set is shown in Fig. 3. The CAD prevalence of patients who were recommended rule-out and rule-in were 0 % (0/661) and 15 % (255/1699), respectively. The protocol showed a sensitivity and PPV to rule-in CAD of 100 % and 15 %, respectively, whereas specificity and NPV to rule-out CAD was 31 % and 100 %. The discriminative performance of the decision-making protocol in subgroups are also showed in Fig. 3. The protocol had better discriminative performance in male patients and patients aged 60 or older, with ST deviation or T-wave abnormality in ECGs (Supplementary material, Table S2). For patients with PTP values between 5 % and 15 %, our model outperformed the conventional D-F model in the external validation population (Supplementary material, Fig. S2).

Fig. 3.

Ranges of prevalence of coronary artery disease in which our protocol can rule-in (red) or rule-out (green) further coronary artery disease examination. ST, ST deviation; T, T-wave abnormality, ST & T = 1 indicates patients have both ST deviation and T-wave abnormality; ST + T = 1 indicates patients have either ST deviation or T-wave abnormality; ST & T = 0 indicates patients have neither ST deviation nor T-wave abnormality.

3.4. Simplified ECG model performance

The AUC, sensitivity, specificity, PPV, NPV, accuracy and balanced accuracy of the simplified algorithm are presented in Table 2. The 6-lead and I-lead algorithms achieved an AUC of 0.75 (95 % CI, 0.72–0.78) and 0.71 (95 % CI, 0.68–0.74) in the external validation set.

Table 2.

Performance of simplified models for detecting coronary artery disease.

Models	Sen (95 % CI)	Spe (95 % CI)	PPV (95 % CI)	NPV (95 % CI)	Acc (95 % CI)	Balanced Acc (95 % CI)
External validation set (Max Sen + Spe)
6 leads algorithm + age + sex	0.72 (0.67–0.78)	0.66 (0.64–0.68)	0.21 (0.18–0.24)	0.95 (0.94–0.96)	0.67 (0.65–0.69)	0.69 (0.66–0.72)
I lead algorithm + age + sex	0.84 (0.79–0.88)	0.50 (0.48–0.52)	0.17 (0.15–0.19)	0.96 (0.95–0.97)	0.53 (0.51–0.55)	0.66 (0.64–0.69)
External validation set (Sen = 0.8)
6 leads algorithm + age + sex	0.80 (0.75–0.85)	0.56 (0.54–0.58)	0.18 (0.16–0.20)	0.96 (0.95–0.97)	0.58 (0.54–0.58)	0.68 (0.65–0.70)
I lead algorithm + age + sex	0.80 (0.75–0.85)	0.53 (0.51–0.55)	0.17 (0.15–0.19)	0.96 (0.95–0.97)	0.56 (0.54–0.58)	0.64 (0.66–0.69)

Sen, sensitivity; Spe, specificity; PPV, positive predictive value; NPV, negative predictive value; Acc, Accuracy; CI, confidence interval.

4. Discussion

In this multicenter cross-sectional study, we established a deep-learning algorithm that can rule-in and rule-out patients with suspected CAD for further assessment, based on their non-specific ECGs without clear myocardial ischemia evidence. Based on the decision threshold determined in the internal validation, the algorithm achieved excellent performance for both ruling out (prevalence is 0 %) and ruling in (prevalence is 15 %) suspected CAD in the external validation. The 6-lead algorithm achieved an AUC of 0.75 (95 % CI, 0.72–0.78) and the I-lead algorithm achieved an AUC of 0.71 (95 % CI, 0.68–0.74) in the external validation set.

Our algorithm based on 12 leads ECG could meet the guideline-recommended requirements for CAD PTP assessments. Previous studies have found that traditional tools based on age, gender, and nature of symptoms overestimate PTP of obstructive CAD, leading to over-examination [4,5,19,20]. To enhance tool performance, additional information is crucial. The 12-lead ECG is a low-cost, widely used, and non-invasive medical tool used on patients for both cardiac and noncardiac reasons, it remains an indispensable component of the initial evaluation. Although previous studies have shown 25 % to 50 % of patients with a history of chest pain due to documented CAD have normal resting ECG [21], certain ECG characteristics have been demonstrated to have predictive value for CAD. A few studies have explored the potential of deep learning in utilizing ECG information for CAD screening, but they often focused solely on internal validation populations, lacking external validation to affirm practical application value and generalizability [15,16]. Our study further elucidates the feasibility of using ECG, even in the absence of clear myocardial ischemia, for CAD detection. The algorithm had a well performance in the internal validation set (AUC = 0.81, 95 % CI 0.78–0.83). Moreover, we determined a decision threshold allowing for a 95 % NPV, aligning with guideline recommendations to establish a protocol. Applying the predefined threshold, the CAD prevalence of patients who were recommended rule-out and rule-in were 0 % (0/661) and 15 % (255/1699) in the external validation cohort, respectively. This implies that if our protocol classifies patients as negative, further tests may not be necessary. Conversely, if our protocol classifies patients as positive, even in the absence of clear ECG changes, their risk of disease warrants further examination. This approach not only reduces unnecessary patient examinations, minimizing invasive injury and economic burden, but also effectively identifies high-risk patients who require further investigation.

In the visualization analysis, we employed Layer-wise relevance propagation (LRP) to identify the ECG segments most relevant to our model's predictions (Supplementary material, Fig. S3). Interestingly, the model's key features of interest were concentrated in the QRS complex region, which aligns with findings from previous studies [22]. For patients without specific ST-segment changes, evidence of prior myocardial ischemia may manifest in the QRS waveform.

We also further explored the feasibility of using fewer leads ECG without clear myocardial ischemia to detect CAD. Both 6-lead and I-lead algorithms exhibited moderate performance (AUC is 0.75 and 0.71 respectively) in the external validation set. Previous research has suggested that algorithms might face challenges in extracting valuable features from single-lead ECG alone [16]. However, we discovered that limb leads/I-lead ECG, combined with age and gender variables and without additional examinations, demonstrated robust predictive performance. Algorithms incorporating fewer leads of ECG, along with age and gender, show promise as potential tools for early CAD screening in high-risk community populations, particularly leveraging wearable devices. Our algorithm could be further developed into a user-friendly mobile application for self-reporting, facilitating the assessment of CAD risk in advance of a medical visit. Future work involves refining and validating the algorithm based on community populations, ensuring its effectiveness in diverse settings.

The clinical implications of our algorithm are noteworthy, particularly in guiding outpatients with suspected CAD on whether further examinations are warranted. Several models have been developed to evaluate the PTP of CAD [[23], [24], [25], [26], [27], [28]]. The Diamond Forrester model is a landmark work published in 1979 aimed at determining the PTP of CAD based on age, gender, and type of chest pain [23]. While this model was simple and suitable for routine clinical use [25], its development population is outdated, leading to an overestimation of CAD risks in contemporary populations [4]. Compared to the conventional D-F model, our algorithm demonstrated superior predictive performance specifically in patients with PTP between 5 % and 15 % (Supplementary material, Fig. S2). We further examined our model's reclassification performance across all PTP strata (< 5 %, 5–15 %, and > 15 %) by D-F model (Supplementary material, Table S3). The algorithm demonstrated perfect sensitivity (0 % false negatives) with no missed CAD cases, though it required more diagnostic tests than conventional D-F model (6.7 vs. 5.6 angiography or cCTA per confirmed CAD case). We consider this modest increase in examination volume clinically justified given the potentially severe consequences of undiagnosed CAD, particularly in intermediate-risk patients where diagnostic uncertainty is greatest. Our algorithm demonstrates excellent discriminative performance, especially in male patients, those aged ≥ 60 years, and those with ST deviation or T-wave abnormality. The inclusion of ECGs recorded by multiple machines across different centers enhances the potential extrapolation of our algorithm. Although our model cannot replace the Diamond Forrester model without direct comparison data, it could complement it in clinical practice, contributing to improved risk assessment and decision-making for patients with suspected CAD.

5. Study limitation

Several limitations merit consideration in the present study. First, the prevalence of patients in this study was slightly higher than previous multi-center studies in Western countries. Further multi-center verification is needed. Second, CAD was defined based on coronary angiography or cCTA—the selection between these two modalities may bias the outcomes. But we believed that the bias was small, as 64-row cCTA was found to have reasonably consistent diagnostic performance when compared with coronary angiography [29]. Third, this is a retrospective study, and most of the patients are outpatients, resulting in the absence of many clinical variables in traditional prediction models, such as the D-F model. However, we derived PTP by inferring symptom from admission diagnoses to mitigate this constraint in our external validation cohort, enabling meaningful comparative assessment. While this limitation warrant consideration, it does not undermine the validity of our conclusions. We obtained definite clinical significance by comparing with the guideline-recommended requirements for CAD PTP assessments, which is sufficient to argue the hypothesis of this study.

6. Conclusions

We developed and validated a deep learning model to accurately rule-in and rule-out patients with suspected CAD based on nonspecific ECGs, meeting guideline-recommended requirements. Our finding suggests that this model is expected to be replicated in outpatient clinics or any settings equipped with an ECG to improve the triage accuracy of the initial CAD evaluation and reduce the utilization of non-essential treatment resources.

CRediT authorship contribution statement

Runchen Sun: Writing – review & editing, Writing – original draft, Validation, Project administration, Methodology, Formal analysis, Data curation, Conceptualization. Xiangqian Zhu: Writing – review & editing, Writing – original draft, Validation, Software, Methodology. Shen Lin: Writing – review & editing, Writing – original draft, Validation, Project administration, Methodology, Formal analysis, Conceptualization. Mengnan Shi: Writing – review & editing, Writing – original draft, Validation, Software, Methodology. Xuexin Yu: Writing – review & editing, Software, Methodology. Chang Liu: Writing – review & editing, Software, Methodology. Yaoguan Yue: Writing – review & editing, Software, Methodology. Juntong Zeng: Writing – review & editing, Methodology, Conceptualization. Yan Zhao: Writing – review & editing, Resources, Project administration. Xiaoqi Wang: Writing – review & editing, Resources, Project administration. Xiaocong Lian: Writing – review & editing, Software, Methodology. Xin Jin: Writing – review & editing, Software, Methodology. Zhe Zheng: Writing – review & editing, Resources, Project administration, Methodology, Conceptualization. Xiangyang Ji: Writing – review & editing, Software, Project administration, Methodology, Conceptualization.

Ethics approval

This study was approved by Ethics Committee of Fuwai Hospital, CAMS&PUMC(reference number: 2023–2252).

Funding

This work was supported by the National High Level Hospital Clinical Research Funding of Fuwai Hospital, Chinese Academy of Medical Sciences (No. 2022-GSP-GG-28, 2022-GSP-QN-10, 2023-GSP-RC-10). The funding sources were not involved in study design, data collection, analysis, interpretation of data, writing the report, and the decision to submit the article for publication.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary data

The following are the Supplementary data to this article: