Abstract
Aims
The clinical feasibility of artificial intelligence (AI)-based electrocardiography (ECG) analysis for predicting obstructive coronary artery disease (CAD) has not been sufficiently validated in patients with stable angina, especially in large sample sizes.
Methods and results
A deep learning framework for the quantitative ECG (QCG) analysis was trained and internally tested to derive the risk scores (0–100) for obstructive CAD (QCGObstCAD) and extensive CAD (QCGExtCAD) using 50 756 ECG images from 21 866 patients who underwent coronary artery evaluation for chest pain (invasive coronary or computed tomography angiography). External validation was performed in 4517 patients with stable angina who underwent coronary imaging to identify obstructive CAD. The QCGObstCAD and QCGExtCAD scores were significantly increased in the presence of obstructive and extensive CAD (all P < 0.001) and with increasing degrees of stenosis and disease burden, respectively (all Ptrend < 0.001). In the internal and external tests, QCGObstCAD exhibited a good predictive ability for obstructive CAD [area under the curve (AUC), 0.781 and 0.731, respectively] and severe obstructive CAD (AUC, 0.780 and 0.786, respectively), and QCGExtCAD exhibited a good predictive ability for extensive CAD (AUC, 0.689 and 0.784). In the external test, the QCGObstCAD and QCGExtCAD scores demonstrated independent and incremental predictive values for obstructive and extensive CAD, respectively, over that with conventional clinical risk factors. The QCG scores demonstrated significant associations with lesion characteristics, such as the fractional flow reserve, coronary calcification score, and total plaque volume.
Conclusion
The AI-based QCG analysis for predicting obstructive CAD in patients with stable angina, including those with severe stenosis and multivessel disease, is feasible.
Keywords: Artificial intelligence, Coronary artery disease, Electrocardiography, Stable angina
Graphical Abstract
Graphical Abstract.
Introduction
Despite advancements in various diagnostic techniques, electrocardiography (ECG) remains as the primary tool for the initial assessment of patients presenting with chest pain. Electrocardiography is particularly valuable for providing clinical insights into patients with acute chest pain. It plays a crucial role in the differential diagnosis of acute myocardial infarction and aids in determining the need for emergency invasive management of coronary artery disease (CAD).1 However, the use of ECG in diagnosing obstructive CAD in patients with stable angina is challenging. Electrocardiography findings in such cases are often non-specific and may even fall within normal ranges when interpreted using conventional rule-based methods.1,2 As a result, when obstructive CAD is clinically suspected, additional tests, such as functional stress tests or noninvasive coronary imaging, are frequently required. However, these methods use additional resources, incur higher costs, and can lead to delays in the diagnosis of obstructive CAD.1
Artificial intelligence (AI) has proven utility in the medical field, with real-world clinical applications becoming increasingly prevalent. The AI technology has been instrumental in interpreting ECG waveforms, particularly in deciphering ambiguous signals and their associations with clinical diagnoses and prognostic assessments.3–5 As yet, most AI ECG algorithms for CAD have primarily focused on identifying acute coronary syndromes, such as myocardial infarction, in patients presenting with acute chest pain, often in emergency department settings.6–10 However, our prior study deviated from this common focus by examining the potential of AI-based ECG analysis for predicting obstructive CAD in patients with stable angina.11 The findings suggested that an AI-based ECG analysis can substantially improve the prediction of obstructive CAD, achieving superior accuracy over that with conventional clinical risk factors. Nonetheless, this initial algorithm was developed in a relatively small patient cohort and was not subjected to externally validation.
In response to these limitations, our study aimed to develop a more comprehensive AI-based ECG analyzer, capable of predicting both the presence and severity of obstructive CAD, as well as the extent of CAD. To achieve this, we utilized a large cohort to develop an AI-based ECG analyzer and rigorously evaluated its performance. To specifically assess its performance in outpatients with stable angina, we performed external validation using an independent data set of lower-risk patients in addition to the developmental data set.
Supplementary Material
Contributor Information
Jiesuck Park, Department of Cardiology, Seoul National University Bundang Hospital, Gumi-ro 173beon-gil, Bundang-gu, Seongnam-si, Gyeonggi-do 13620, Republic of Korea; Department of Internal Medicine, Seoul National University College of Medicine, Daehak-ro, Jongno-gu, Seoul 03080, Republic of Korea.
Joonghee Kim, Department of Emergency Medicine, Seoul National University Bundang Hospital, Seongnam, Gyeonggi-do, 13620 Republic of Korea; ARPI Inc., Dolma-ro, Bundang-gu, Seongnam-si, Gyeonggi-do 13605, Republic of Korea.
Si-Hyuck Kang, Department of Cardiology, Seoul National University Bundang Hospital, Gumi-ro 173beon-gil, Bundang-gu, Seongnam-si, Gyeonggi-do 13620, Republic of Korea; Department of Internal Medicine, Seoul National University College of Medicine, Daehak-ro, Jongno-gu, Seoul 03080, Republic of Korea.
Jina Lee, Division of Cardiology, Severance Cardiovascular Hospital, Yonsei University College of Medicine, Yonsei University Health System, Seoul, Republic of Korea.
Youngtaek Hong, CONNECT-AI Research Center, Yonsei University College of Medicine, Seoul, Republic of Korea.
Hyuk-Jae Chang, Division of Cardiology, Severance Cardiovascular Hospital, Yonsei University College of Medicine, Yonsei University Health System, Seoul, Republic of Korea; CONNECT-AI Research Center, Yonsei University College of Medicine, Seoul, Republic of Korea; Ontact Health Inc., Ewhayeodae-gil, Seodaemun-gu, Seoul 03764, Republic of Korea.
Youngjin Cho, Department of Cardiology, Seoul National University Bundang Hospital, Gumi-ro 173beon-gil, Bundang-gu, Seongnam-si, Gyeonggi-do 13620, Republic of Korea; Department of Internal Medicine, Seoul National University College of Medicine, Daehak-ro, Jongno-gu, Seoul 03080, Republic of Korea; ARPI Inc., Dolma-ro, Bundang-gu, Seongnam-si, Gyeonggi-do 13605, Republic of Korea.
Yeonyee E Yoon, Department of Cardiology, Seoul National University Bundang Hospital, Gumi-ro 173beon-gil, Bundang-gu, Seongnam-si, Gyeonggi-do 13620, Republic of Korea; Department of Internal Medicine, Seoul National University College of Medicine, Daehak-ro, Jongno-gu, Seoul 03080, Republic of Korea; Ontact Health Inc., Ewhayeodae-gil, Seodaemun-gu, Seoul 03764, Republic of Korea.
Methods
Data availability
The data used in the present study cannot be made publicly available due to the ethical restrictions set by the Institutional Review Board of Seoul National University Bundang Hospital (https://bri.snubh.org/irb/), as patient and participant privacy would be compromised. A minimally anonymized data set can be requested by contacting the corresponding authors (yeonyeeyoon@snubh.org or cho_y@snubh.org). The AI-based algorithms described in this paper are intended for inclusion in an upcoming version of the ECG Buddy application (ARPI Inc.), which restricts their public release. However, the algorithms are available for research purposes. Interested researchers should contact J.K. (joonghee79@gmail.com) to request for access.
Study population
For development and internal validation of our algorithm, we screened patients aged 18 years and over who underwent invasive coronary angiography (CAG) or coronary computed tomography angiography (CCTA) to evaluate chest pain from 2011 to 2019 at Seoul National University Bundang Hospital (Seongnam, Gyeonggi-do, South Korea). The ECG data were retrospectively collected from records obtained within 6 months prior to CAG and CCTA. In instances of multiple ECG records per patient, all available data were included for model development. The ECGs were recorded at rest in the supine position using conventional ECG machines (PageWriter TC 30, and TC 70; Philips) with a 500 Hz sampling frequency. Consequently, 21 866 patients with a total of 50 756 ECG records were included. For model development, a random selection of 90% of the patients was allocated for training, with the remaining 10% designated for internal testing. Further, we allocated 25% of the training data specifically for hyperparameter optimization. The data split was conducted on a per-patient basis, ensuring no patient’s ECG records were divided across different sets.
For an external validation data set, we compiled data from five separate retrospective and prospective cohorts of outpatients undergoing CCTA or CAG due to suspected angina.12–16 Detailed information on the inclusion criteria for each study data set is provided in the Supplementary Methods S1. Within this pool, we assessed the availability of ECG records for patients from Severance Hospital (Seoul, South Korea) that were taken within 6 months prior to their procedures. To ensure the analysis focused on patients with stable angina, we excluded patients who were admitted to the emergency department with acute chest pain. These records were obtained at rest using conventional ECG machines (MAC 5500, MAC VU360; GE Healthcare) at a 500 Hz sampling frequency. When multiple ECGs were available, only the record closest to the CCTA or CAG date was selected. This process resulted in an external validation set of 4517 patients, each with a corresponding ECG record.
The study protocol was approved by the Institutional Review Board of Seoul National University Bundang Hospital (No. B-2304-823-002) and Severance Hospital (No. 4-202-1314) and conducted in accordance with the principles of the Declaration of Helsinki. The requirement for informed consent was waived because of the retrospective study design.
Network architecture
In previous reports, we presented a deep learning-based AI system, known as quantitative ECG (QCG™), which has shown effectiveness in diagnosing a spectrum of cardiac conditions, including acute coronary syndrome, ST-elevation myocardial infarction (STEMI), non-specific myocardial injury, and left ventricular dysfunction, etc.6,7 Building on this foundation, the current study extends the utility of QCG by incorporating features that enable the prediction of obstructive and extensive CAD. The network architecture of the QCG analyzer model consisted of (i) a common encoder for encoding input ECG image data and (ii) fully connected layers for predicting both obstructive CAD (QCGObstCAD) and extensive CAD (QCGExtCAD) (see Supplementary material online, Figure S1). For the purposes of training and validation in the present study, the ECG data were converted into portable network graphics image files. The encoder is a modified ResNet that integrates squeeze–excitation blocks into each residual block to enhance feature recalibration and incorporates a non-local network before the final residual block to capture long-range dependencies. The encoder was first pre-trained using various self-supervised tasks and then attached to the fully connected layers for fine-tuning. The Adam-W optimizer was used for model training, and Bayesian optimization was employed for hyperparameter search, aiming to enhance the predictive accuracy and efficiency of the model. The sigmoid outputs of the analyzer, namely, the QCG scores, were scaled between 0 and 100 for the presence of obstructive CAD (QCGObstCAD) and extensive CAD (QCGExtCAD).6,7,11 Hyperparameter optimization was further introduced for the final model, including batch size and learning rate, using the data splitting technique within the training data set.
Evaluation of quantitative electrocardiography scores for predicting obstructive and extensive coronary artery disease
The model’s efficacy in predicting obstructive and extensive CAD was assessed using an independent holdout internal test set and an external data set. In line with standard medical practice, when a patient underwent both CCTA and CAG, the CAG results were prioritized. Different criteria were applied in diagnosing obstructive CAD with CAG and CCTA due to CCTA's tendency to overestimate the severity of stenosis.17,18 This approach is based on prior studies indicating that CCTA generally shows a higher sensitivity, but a lower specificity for the diagnosis of obstructive CAD compared with CAG.19 Obstructive CAD was identified based on CAG and CCTA reports and defined as coronary lesions with diameter stenosis ≥50% on CAG or ≥70% on CCTA in major epicardial coronary arteries or their major branches. Severe obstructive CAD was defined as coronary lesions with diameter stenosis ≥70% on CAG or ≥90% on CCTA. Lesions with diameter stenosis of 50–69% on CAG and 70–89% on CCTA were classified as having intermediate stenosis, delineating them from severe obstructive CAD.
While the same criteria were used for defining obstructive CAD across both internal and external test sets, the definition of extensive CAD differed among data sets. In the internal test data set, as in the training data set, we calculated the sum of the degree of significant stenosis in all major epicardial arteries and their major branches. If there were lesions with severe obstructive stenosis in the proximal location of the left anterior descending or left main coronary arteries, a weighting factor of 1.2 was applied. Subsequently, this metric was normalized, with the value ≥ 0.8 defined as extensive CAD, indicating a high atherosclerotic burden. For further stratification, we defined intermediate burden as a normalized sum of the maximum diameter stenosis >0.2, but <0.8, and low burden as ≤0.2. In the external data set, which lacked detailed segmental stenosis information, we focused on the number of coronary arteries exhibiting obstructive CAD (one-vessel, two-vessel, or three-vessel disease). In the external data set, extensive CAD was defined as a multivessel disease, with the involvement of ≥2 vessels, and compared against cases with a one-vessel disease and no obstructive CAD.
The external data set contained additional data elements that varied according to the objectives of the original studies. These included the coronary artery calcification score (CACS)13 and total plaque volume on CCTA12,14,15 and the fractional flow reserve (FFR) on invasive CAG.15 Consequently, in the external validation process, we also evaluated associations between QCG scores and these additional parameters in subsets with available data. This additional analysis aimed to further elucidate the broader applicability and relevance of QCG scores in diverse clinical contexts and various cardiac health indicators.
Statistical analyses
All statistical analyses were conducted using R software (version 4.2.1; R Core Team). Continuous variables are summarized as the median (interquartile range) and categorical variables as numbers (percentages). Two-sided P-values < 0.05 were considered as statistically significant. Given the presence of multiple ECG images per patient in the internal data set, the analyses were conducted at the ECG image level for the internal validation test. Conversely, the external data set, with one ECG per patient, necessitated a patient-level analysis. We assessed the distribution of QCG scores in relation to the presence or absence of obstructive and extensive CAD, including stenosis severity and CAD burden. The diagnostic performance of the QCG score was evaluated by calculating the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) for every 10 units of increase in the QCG score. Additionally, the discriminatory capability of the QCG scores for predicting obstructive and extensive CAD was assessed by calculating the area under the receiver operating characteristic curve (AUC). We employed a bootstrapping method with 2000 iterations for estimating confidence intervals. Multivariate logistic regression analysis, adjusted for clinical variables such as age, sex, body mass index (BMI), hypertension, diabetes mellitus, dyslipidemia, and smoking status, was used to ascertain the independent predictive value of QCG scores for each outcome. The incremental predictive value of the QCG scores was determined by the improvement in the model performance relative to a clinical model, which included the same variables as those used for multivariate adjustment. A significant improvement was determined using DeLong test for AUC differences and the likelihood ratio test. We also plotted the distribution of the lesion characteristics (CACS, total plaque volume, and FFR) across QCG score tertiles (<20, ≤20 to <40, and ≥40) and assessed significant differences using the analysis of variance test.
Results
Baseline characteristics
Table 1 details the baseline characteristics of the internal and external data sets. The internal data set showed uniform demographics across the training and test groups, with a median age of 63 and about 60% male patients. Invasive CAG was the main method for CAD evaluation (77.1% in training, 77.2% in testing). The prevalence of obstructive CAD and severe obstructive CAD was nearly identical in the training (52.1% and 46.7%, respectively) and test (51.7% and 45.9%, respectively) subsets, with extensive CAD observed in 12.7% and 12.3%, respectively. The external data set, while clinically similar to the internal, mainly used CCTA (93.8%) and had a lower CAD prevalence (obstructive CAD at 6.6%, severe obstructive CAD at 3.5%, and extensive disease at 2.2%).
Table 1.
Baseline characteristics
| Internal data set | External data set | ||
|---|---|---|---|
| Training set | Test set | ||
| (n = 19 680) | (n = 2186) | (n = 4517) | |
| Clinical features | |||
| Age, years | 63 (53–73) | 63 (53–73) | 59 (52–66) |
| Male sex | 12 451 (63.3) | 1345 (61.5) | 2359 (52.2) |
| BMI, kg/m2 | 24 (22–27) | 25 (22–27) | 24 (23–26) |
| Hypertension | 9122 (46.4) | 1069 (48.9) | 2332 (51.6) |
| Diabetes mellitus | 4959 (25.2) | 560 (25.6) | 862 (19.1) |
| Dyslipidaemia | 6562 (33.3) | 712 (32.6) | 1912 (42.3) |
| Smoking status | 2901 (14.7) | 337 (15.4) | 636 (14.1) |
| Coronary artery disease | |||
| Invasive CAG | 15 178 (77.1) | 1688 (77.2) | 282 (6.2) |
| CCTA | 4502 (22.9) | 498 (22.8) | 4235 (93.8) |
| Obstructive CAD | 10 254 (52.1) | 1133 (51.8) | 296 (6.6) |
| Severe obstructive CAD | 9191 (46.7) | 1004 (45.9) | 159 (3.5) |
| Extensive CAD | 2491 (12.7) | 268 (12.3) | 99 (2.2) |
Values are presented as the median (interquartile range) or number (percentage), unless otherwise indicated. BMI, body mass index; CAD, coronary artery disease; CAG, coronary angiography; CCTA, coronary computed tomography angiography.
Distribution of quantitative electrocardiography scores
In the internal test data set, the distribution of QCG scores according to the presence and severity of obstructive and extensive CAD showed trends similar to those observed in the training data set (Figure 1). The QCGObstCAD scores were significantly higher in patients with obstructive CAD than in those without obstructive CAD [40 (21–65) vs. 12 (4–28), respectively; P < 0.001]. When further stratified according to stenosis severity, the QCGObstCAD scores gradually increased with the increasing stenosis severity [no stenosis, 12 (4–28); intermediate stenosis, 27 (14–44); severe stenosis, 43 (22–68); Ptrend < 0.001]. Similarly, the QCGExtCAD scores were significantly higher in patients with extensive CAD than in those without extensive CAD [19 (9–35) vs. 34 (21–54), respectively; P < 0.001]. The QCGExtCAD scores also demonstrated an incremental trend with the increasing CAD burden [low burden, 16 (8–30); intermediate burden, 27 (13–43); high burden, 34 (21–54); Ptrend < 0.001]. In the external data set, this trend was not diminished and was similarly observed, demonstrating that the QCGObstCAD score distinctly varied in accordance with the presence of obstructive CAD [43 (30–55) vs. 28 (17–38) for obstructive CAD vs. no obstructive CAD, respectively; P < 0.001], as well as with the severity of stenosis [no stenosis: 28 (17–38); intermediate stenosis, 38 (25–49); severe stenosis: 47 (36–61); Ptrend < 0.001]. Likewise, the QCGExtCAD scores significantly increased with the presence [38 (26–50) vs. 21 (12–30) for extensive CAD vs. no extensive CAD, respectively; P < 0.001] and extent of extensive CAD [no obstructive CAD, 21 (12–29); one-vessel disease, 31 (21–39); ≥2-vessel disease, 38 (26–50); Ptrend < 0.001] (Figure 1).
Figure 1.
Distribution of the QCG scores. The distribution of the QCG scores is plotted according to the presence of obstructive CAD and the severity and extent of the disease. In both the internal and external data sets, the QCG scores are significantly increased with the presence of obstructive CAD and higher degrees of stenosis and disease extent. CAD, coronary artery calcification; QCG, quantitative electrocardiography.
Predictive value of quantitative electrocardiography scores
We evaluated how the diagnostic performance of both QCG scores changes with every 10-unit increment and have presented these findings in Table 2. As expected, for both QCG scores, lower cut-off values corresponded with increased sensitivity and decreased specificity. In the external test set, which had a relatively lower prevalence of obstructive and extensive CAD, the PPV was notably low, while the NPV was significantly high.
Table 2.
Sensitivity, specificity, and positive and negative predictive values on different quantitative electrocardiography cut-offs for detection of obstructive and extensive coronary artery disease
| Internal data set (training) | Internal data set (test) | External data set | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Obstructive CAD | Obstructive CAD | Obstructive CAD | |||||||||||||||
| QCGObstCAD | Sen | Spe | Acc | PPV | NPV | QCGObstCAD | Sen | Spe | Acc | PPV | NPV | QCGObstCAD | Sen | Spe | Acc | PPV | NPV |
| 10 | 93.8 | 46.1 | 68.4 | 60.4 | 89.5 | 10 | 89.5 | 43.7 | 65.3 | 58.6 | 82.4 | 10 | 96.3 | 12.7 | 18.2 | 7.2 | 98.0 |
| 20 | 81.8 | 66.1 | 73.4 | 67.8 | 80.6 | 20 | 75.8 | 64.4 | 69.7 | 65.5 | 74.9 | 20 | 86.8 | 32.3 | 35.9 | 8.3 | 97.2 |
| 30 | 67.7 | 80.2 | 74.3 | 74.9 | 73.9 | 30 | 62.7 | 77.4 | 70.5 | 71.2 | 69.9 | 30 | 74.0 | 55.4 | 56.7 | 10.4 | 96.8 |
| 40 | 53.8 | 88.5 | 72.3 | 80.4 | 68.7 | 40 | 50.0 | 86.0 | 69.0 | 76.0 | 65.8 | 40 | 56.8 | 78.2 | 76.8 | 15.4 | 96.3 |
| 50 | 41.6 | 93.5 | 69.3 | 84.8 | 64.7 | 50 | 38.7 | 91.5 | 66.6 | 80.2 | 62.6 | 50 | 33.4 | 92.3 | 88.5 | 23.5 | 95.2 |
| 60 | 31.4 | 96.4 | 66.1 | 88.4 | 61.6 | 60 | 30.2 | 94.9 | 64.4 | 84.0 | 60.4 | 60 | 19.9 | 97.8 | 92.7 | 39.1 | 94.6 |
| 70 | 21.7 | 98.2 | 62.5 | 91.3 | 58.9 | 70 | 21.5 | 97.7 | 61.7 | 89.1 | 58.2 | 70 | 9.8 | 99.5 | 93.7 | 60.4 | 94.0 |
| 80 | 13.1 | 99.3 | 59.1 | 94.2 | 56.6 | 80 | 14.2 | 99.0 | 59.0 | 92.6 | 56.4 | 80 | 2.7 | 99.9 | 93.5 | 66.7 | 93.6 |
| Extensive CAD | Extensive CAD | Extensive CAD | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| QCGExtCAD | Sen | Spe | Acc | PPV | NPV | QCGExtCAD | Sen | Spe | Acc | PPV | NPV | QCGExtCAD | Sen | Spe | Acc | PPV | NPV |
| 10 | 92.1 | 24.3 | 33.6 | 16.2 | 95.1 | 10 | 90.1 | 28.4 | 37.1 | 17.0 | 94.6 | 10 | 97.0 | 19.0 | 20.7 | 2.6 | 99.6 |
| 20 | 77.5 | 50.3 | 54.0 | 19.8 | 93.4 | 20 | 75.8 | 51.6 | 55.0 | 20.4 | 92.9 | 20 | 82.8 | 46.6 | 47.4 | 3.4 | 99.2 |
| 30 | 61.3 | 69.0 | 68.0 | 23.9 | 91.8 | 30 | 57.1 | 69.0 | 67.3 | 23.1 | 90.8 | 30 | 69.7 | 75.1 | 74.9 | 5.9 | 99.1 |
| 40 | 46.6 | 81.1 | 76.3 | 28.1 | 90.5 | 40 | 41.9 | 80.0 | 74.6 | 25.5 | 89.4 | 40 | 41.4 | 92.5 | 71.4 | 11.0 | 98.6 |
| 50 | 31.6 | 89.6 | 81.7 | 32.6 | 89.2 | 50 | 30.6 | 88.8 | 80.7 | 30.9 | 88.7 | 50 | 24.2 | 98.1 | 96.5 | 22.6 | 98.3 |
| 60 | 18.3 | 95.7 | 85.1 | 40.1 | 88.1 | 60 | 18.3 | 95.0 | 84.2 | 37.3 | 87.7 | 60 | 10.1 | 99.6 | 97.7 | 37.0 | 98.0 |
| 70 | 7.0 | 99.0 | 86.4 | 52.7 | 87.0 | 70 | 7.3 | 98.8 | 86.0 | 50.0 | 86.7 | 70 | 4.0 | 100.0 | 97.9 | 100.0 | 97.9 |
| 80 | 0.7 | 99.9 | 86.4 | 62.7 | 86.4 | 80 | 1.6 | 100.0 | 86.2 | 88.9 | 86.2 | 80 | 1.0 | 100.0 | 97.8 | 100.0 | 97.8 |
Acc, accuracy; CAD, coronary artery calcification; NPV, negative predictive value; PPV, positive predictive value; QCG, quantitative electrocardiography; QCGObstCAD, QCG score for obstructive CAD; QCGExtCAD, QCG score for extensive CAD; Sen, sensitivity; Spe, specificity.
In our analysis, QCGObstCAD exhibited excellent predictive ability for obstructive CAD in the internal training set [AUC 0.830; 95% confidence interval (CI), 0.826–0.834; P < 0.001] and acceptable in the test set (AUC 0.781; 95% CI, 0.769–0.794; P < 0.001), as illustrated in Figure 2. A similar pattern was observed for severe obstructive CAD, with the training set performance showing an AUC of 0.822 (95% CI 0.818–0.826; P < 0.001) and the test set an AUC of 0.780 (95% CI 0.768–0.793; P < 0.001). The QCGExtCAD's predictive capability for extensive CAD was deemed acceptable, with AUC values of 0.707 (95% CI, 0.699–0.716; P < 0.001) in the internal training data set and 0.689 (95% CI, 0.664–0.714; P < 0.001) in the test data set. In the external data set, the predictive performance of QCGObstCAD for obstructive CAD (AUC, 0.731; 95% CI, 0.699–0.763; P < 0.001) and severe obstructive CAD (AUC, 0.786; 95% CI, 0.747–0.826; P < 0.001), as well as the performance of QCGExtCAD for extensive CAD (AUC 0.784, 95% CI 0.733–0.835; P < 0.001), all remained at an acceptable level.
Figure 2.
Predictive value of the QCG scores. QCGObstCAD shows a good predictive ability for obstructive CAD and severe stenosis in the internal data set. Similarly, QCGExtCAD shows a good predictive ability for high-burden CAD. The QCG scores also show a good predictive ability in the external data set. CAD, coronary artery calcification; QCG, quantitative electrocardiography; QCGObstCAD, QCG score for obstructive CAD; QCGExtCAD, QCG score for extensive CAD.
In the external data set, we evaluated the independent and incremental predictive values for both obstructive and extensive CAD. With the adjustment for conventional clinical risk factors (age, sex, BMI, hypertension, diabetes mellitus, dyslipidemia, and smoking), QCGObstCAD was an independent predictor for obstructive CAD (adjusted odds ratio per 10-point increase, 1.60; 95% CI, 1.47–1.74; P < 0.001) and severe obstructive CAD (adjusted odds ratio per 10-point increase, 1.87; 95% CI, 1.67–2.08; P < 0.001) (Figure 3). QCGExtCAD also demonstrated independent associations with extensive CAD (adjusted odds ratio per 10-point increase, 2.06; 95% CI, 1.76–2.42; P < 0.001). QCGObstCAD demonstrated an incremental predictive value over the clinical model for obstructive CAD (AUC, 0.716 vs. 0.763, P < 0.001; χ2 175.4 vs. 300.3, P < 0.001) and severe obstructive CAD (AUC, 0.751 vs. 0.816, P < 0.001; χ2 136.8 vs. 264.5, P < 0.001) (Table 3). Conversely, the integration of the clinical model over QCGObstCAD exhibited an incremental predictive value for obstructive CAD (AUC, 0.736 vs. 0.763, P < 0.001; χ2 241.0 vs. 300.3, P < 0.001) and severe obstructive CAD (AUC, 0.786 vs. 0.816, P = 0.001; χ2 219.0 vs. 264.5, P < 0.001). QCGExtCAD also exhibited an incremental predictive value over the clinical model in predicting extensive CAD (AUC, 0.794 vs. 0.840, P < 0.001; χ2 123.8 vs. 201.6, P < 0.001). Likewise, integrating clinical information over QCGExtCAD also increased the predictive value for extensive CAD (AUC, 0.784 vs. 0.840, P < 0.001; χ2 147.0 vs. 201.6, P < 0.001).
Figure 3.
Independent predictive value of the QCG scores. In the external data set, QCGObstCAD is an independent predictor of obstructive CAD and severe stenosis. Similarly, QCGExtCAD has an independent association with the multivessel disease. CAD, coronary artery calcification; QCG, quantitative electrocardiography; QCGObstCAD, QCG score for obstructive CAD; QCGExtCAD, QCG score for extensive CAD.
Table 3.
Incremental predictive value of the quantitative electrocardiography scores (external data set)
| AUC (95% CI) | P difference | χ2 | P difference | |||
|---|---|---|---|---|---|---|
| Obstructive CAD | ||||||
| Clinical modela | 0.716 (0.685–0.745) | Reference | — | 175.4 | Reference | — |
| QCGObstCADb | 0.736 (0.699–0.763) | 0.428 | Reference | 241.0 | <0.001 | Reference |
| Clinical + QCGObstCAD | 0.763 (0.733–0.792) | <0.001 | <0.001 | 300.3 | <0.001 | <0.001 |
| Severe obstructive CAD | ||||||
| Clinical modela | 0.751 (0.712–0.789) | Reference | — | 136.8 | Reference | — |
| QCGObstCADb | 0.786 (0.747–0.826) | 0.128 | Reference | 219.0 | <0.001 | Reference |
| Clinical + QCGObstCAD | 0.816 (0.780–0.853) | <0.001 | 0.001 | 264.5 | <0.001 | <0.001 |
| Extensive CAD | ||||||
| Clinical modela | 0.794 (0.749–0.839) | Reference | — | 123.8 | Reference | — |
| QCGExtCADb | 0.784 (0.733–0.835) | 0.725 | Reference | 147.0 | 0.001 | Reference |
| Clinical + QCGExtCAD | 0.840 (0.797–0.882) | <0.001 | <0.001 | 201.6 | <0.001 | <0.001 |
aInclusion of age, sex, body mass index, hypertension, diabetes mellitus, dyslipidaemia, and smoking status.
bPer 10-score increase.
AUC, area under the curve; CAD, coronary artery disease; CI, confidence interval; QCGExtCAD, QCG score for extensive CAD; QCGObstCAD, QCG score for obstructive CAD.
Associations between quantitative electrocardiography scores and other coronary artery disease characteristics
In the external data set, we examined subgroups with additional CAD characteristics: CACS (n = 3500) and total plaque volume (n = 782) assessed by CCTA and FFR (n = 59) measured through invasive CAG. We observed a significant decrease in FFR indicative of haemodynamically significant obstructive CAD with higher QCGObstCAD tertiles (P < 0.001) (Figure 4). CACS and the total plaque volume reflecting the CAD burden showed a gradual increase alongside higher QCGExtCAD tertiles (all P < 0.001).
Figure 4.
Association of the QCG scores with CAD features. There is a significant decrease in the FFR values within the highest tertiles of QCGObstCAD. Concurrently, markers of CAD burden, such as CACS and total plaque volume, exhibit a progressive increase with the ascending QCGExtCAD tertiles. CAC, coronary artery calcification; CAD, coronary artery calcification; FFR, fractional flow reserve; QCG, quantitative electrocardiography; QCGObstCAD, QCG score for obstructive CAD; QCGExtCAD, QCG score for extensive CAD.
Discussion
In the present study, we developed and validated novel ECG-based predictive indicators, QCGObstCAD and QCGExtCAD, for predicting obstructive and extensive CAD within a large cohort. Shifting from the typical focus on acute coronary syndrome, our work targets the detection of obstructive CAD and the assessment of extensive CAD risk in stable angina patients, broadening the utility of ECG diagnostics. Despite the lower prevalence and severity of CAD in an external data set, the QCG scores maintained an acceptable performance. Moreover, the QCG scores provided independent and incremental predictive values alongside traditional clinical variables for both obstructive and extensive CAD, enhancing the diagnostic utility of these clinical assessments.
In patients with stable angina, the diagnostic value of ECG is considerably limited owing to non-specific or normal ECG findings during rest.2,20,21 This limitation is primarily due to the conventional rule-based ECG interpretation's difficulty in detecting subtle changes indicative of obstructive CAD in stable angina. However, recent advancements in AI technologies, particularly deep learning-based ECG interpretation, have shown promise in overcoming these limitations. Unlike traditional computer-aided models that rely on predefined rules, AI-based approaches are capable of identifying nuanced patterns and comprehensive information within ECG features that may elude both human experts and conventional systems.4,5 Moreover, AI-based approaches extend beyond traditional diagnostic uses, detecting conditions like left ventricular dysfunction and predicting future cardiovascular events, such as atrial fibrillation.22 Given these advancements, we hypothesized that the AI-based ECG analysis could uncover the subtle or non-specific changes in stable angina, thereby improving obstructive CAD prediction. Our initial study on 723 stable angina patients introduced an AI model that utilizes deep learning-derived quantitative ECG features for obstructive CAD prediction,11 showing superior performance to clinical risk factors. However, this initial study, similar to others,23,24 was constrained by its small sample size and lack of external validation. Building on these promising results, the current study aimed to develop a more refined QCG analyzer with a larger cohort. We formulated the QCG scores to predict the presence and extent of obstructive CAD and validated their performance in both internal and external test data sets.
Tang et al.25 recently presented an AI ECG model focused on detecting obstructive CAD (diameter stenosis ≥ 50% on CCTA) in a sizable cohort of 10 538 patients with suspected CAD. This model showed a moderate efficacy (AUC 0.75), akin to our findings, yet its scope was restricted to diagnosing mild stenosis. Our study, however, extends the scope of QCG scores to various severities of CAD, including severe and extensive CAD, thereby enhancing the clinical utility of ECG-based predictive indicators. Furthermore, validating an AI ECG model in an independent study population is crucial for clinical application, as patient characteristics can influence model performance.4 Notably, the external test data set comprised patients with stable angina, deliberately excluding those admitted to the emergency room. This selection criterion resulted in a significantly lower prevalence of obstructive CAD in the external data set. Nevertheless, QCGObstCAD consistently demonstrated a strong predictive ability for obstructive CAD and maintained their discriminative power, providing an incremental predictive value beyond that for clinical risk factors.
Another key aspect of our study was its ambition to not only predict obstructive CAD, but to also evaluate its extent. QCGObstCAD showed a gradual increase with the increasing stenosis severity, and QCGExtCAD similarly increased in relation to the CAD extent. These results underscore the capacity of the QCG analyzer to automatically interpret ECG signals and generate QCG scores, providing quantifiable risk estimation for obstructive and extensive CAD. In subsets with additional CAD characteristics, we further observed an inverse correlation between QCGObstCAD and FFR and a positive correlation between QCGExtCAD and total plaque volume and CACS, although the QCG analyzer was not explicitly trained for these aspects. These findings support the QCG analyzer's potential as an early indicator for identifying high-risk patients, directing subsequent diagnostic and therapeutic interventions. In addition, the flexibility of the QCG analyzer in processing image-based ECG data enhances its usability. Electrocardiographies can be interpreted in various formats, ranging from photographs or printed outputs from conventional ECG recorders to digital ECG systems.6,7,11 Its adaptability ensures that the QCG analyzer is suitable for a wide range of clinical environments, from advanced medical centers equipped with digital ECG systems to resource-limited settings that rely on paper-based ECGs.
Limitations
The present study has some limitations. Rather than providing an optimal cut-off for predicting obstructive and extensive CAD, we detailed how the diagnostic performance varies with different scores, considering the potential impact of the pre-test probability of the study population on performance. However, it is important to note that all assessments were conducted in symptomatic patients. Consequently, this score should not be used for screening purposes in asymptomatic patients. Additionally, data were exclusively obtained from South Korea; accordingly, a multinational study is warranted to ascertain the performance of the QCG scores across diverse ethnicities and clinical settings. Another limitation of this study is the absence of an in-depth explainability analysis to elucidate the specific ECG features that influenced the AI model's predictions. While our focus was on the predictive performance, future work could benefit from incorporating model explainability to enhance the interpretability and clinical relevance of the findings. Although the generalizability of our QCG analyzer was supported by the independent external data set, this validation approach was retrospective. The clinical applicability of the QCG analyzer is expected to be enhanced with prospective studies assessing its influence on clinical decisions in angina patients. A study currently underway aims to evaluate the impact on decision-making for invasive angiography, with results that can be anticipated soon. Lastly, the exploration of the QCG scores’ predictive value for clinical outcomes remains beyond this study’s scope, inviting further research.
Conclusions
The present study demonstrated the feasibility of using a quantitative AI-based ECG analyzer, called the QCG analyzer, for predicting both the presence and extent of obstructive CAD in stable angina patients. The QCG analyzer was shown to provide an independent and incremental predictive value for obstructive CAD, encompassing severe and extensive CAD. Further prospective study is expected to state the potential of the newly proposed QCG scores serving as early indicators for the identification of high-risk patients who may benefit from subsequent diagnostic and therapeutic interventions for obstructive CAD.
Supplementary material
Supplementary material is available at European Heart Journal—Digital Health.
Funding
This research was supported by a grant from the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI) funded by the Ministry of Health and Welfare, Republic of Korea (grant number: RS-2023-00265933).
Data availability
In alignment with our commitment to ethical research practices and the conditions set by the Institutional Review Board (IRB) of Seoul National University Bundang Hospital, open access to research data without explicit participant consent is restricted. Consequently, the data from our study on diastolic dysfunction, collected retrospectively, cannot be publicly disclosed without constraints. Researchers seeking to engage in further study with the data set are advised to reach out to the corresponding author. Collaborative efforts can proceed within the IRB’s permitted scope or through the submission and approval of a new IRB application tailored to new research aims.
References
- 1. Gulati M, Levy PD, Mukherjee D, Amsterdam E, Bhatt DL, Birtcher KK, et al. 2021 AHA/ACC/ASE/CHEST/SAEM/SCCT/SCMR guideline for the evaluation and diagnosis of chest pain: a report of the American College of Cardiology/American Heart Association Joint Committee on Clinical Practice Guidelines. Circulation 2021;144:e368–e454. [DOI] [PubMed] [Google Scholar]
- 2. Mahmoodzadeh S, Moazenzadeh M, Rashidinejad H, Sheikhvatan M. Diagnostic performance of electrocardiography in the assessment of significant coronary artery disease and its anatomical size in comparison with coronary angiography. J Res Med Sci 2011;16:750–765. [PMC free article] [PubMed] [Google Scholar]
- 3. Alizadehsani R, Khosravi A, Roshanzamir M, Abdar M, Sarrafzadegan N, Shafie D, et al. Coronary artery disease detection using artificial intelligence techniques: a survey of trends, geographical differences and diagnostic features 1991–2020. Comput Biol Med 2021;128:104095. [DOI] [PubMed] [Google Scholar]
- 4. Avula V, Wu KC, Carrick RT. Clinical applications, methodology, and scientific reporting of electrocardiogram deep-learning models. JACC Adv 2023;2:100686. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Siontis KC, Noseworthy PA, Attia ZI, Friedman PA. Artificial intelligence-enhanced electrocardiography in cardiovascular disease management. Nat Rev Cardiol 2021;18:465–478. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Choi YJ, Park MJ, Ko Y, Soh MS, Kim HM, Kim CH, et al. Artificial intelligence versus physicians on interpretation of printed ECG images: diagnostic performance of ST-elevation myocardial infarction on electrocardiography. Int J Cardiol 2022;363:6–10. [DOI] [PubMed] [Google Scholar]
- 7. Kim D, Hwang JE, Cho Y, Cho HW, Lee W, Lee JH, et al. A retrospective clinical evaluation of an artificial intelligence screening method for early detection of STEMI in the emergency department. J Korean Med Sci 2022;37:e81. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Wu CC, Hsu WD, Islam MM, Poly TN, Yang HC, Nguyen PA, et al. An artificial intelligence approach to early predict non-ST-elevation myocardial infarction patients with chest pain. Comput Methods Programs Biomed 2019;173:109–117. [DOI] [PubMed] [Google Scholar]
- 9. Wu L, Huang G, Yu X, Ye M, Liu L, Ling Y, et al. Deep learning networks accurately detect ST-segment elevation myocardial infarction and culprit vessel. Front Cardiovasc Med 2022;9:797207. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Al-Zaiti S, Besomi L, Bouzid Z, Faramand Z, Frisch S, Martin-Gill C, et al. Machine learning-based prediction of acute coronary syndrome using only the pre-hospital 12-lead electrocardiogram. Nat Commun 2020;11:3966. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Park J, Yoon Y, Cho Y, Kim J. Feasibility of artificial intelligence-based electrocardiography analysis for the prediction of obstructive coronary artery disease in patients with stable angina: validation study. JMIR Cardio 2023;7:e44791. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Lee SE, Sung JM, Andreini D, Al-Mallah MH, Budoff MJ, Cademartiri F, et al. Differences in progression to obstructive lesions per high-risk plaque features and plaque volumes with CCTA. JACC Cardiovasc Imaging 2020;13:1409–1417. [DOI] [PubMed] [Google Scholar]
- 13. Cho I, Shim J, Chang HJ, Sung JM, Hong Y, Shim H, et al. Prognostic value of multidetector coronary computed tomography angiography in relation to exercise electrocardiogram in patients with suspected coronary artery disease. J Am Coll Cardiol 2012;60:2205–2215. [DOI] [PubMed] [Google Scholar]
- 14. Chang HJ, Lin FY, Gebow D, An HY, Andreini D, Bathina R, et al. Selective referral using CCTA versus direct referral for individuals referred to invasive coronary angiography for suspected CAD: a randomized, controlled, open-label trial. JACC Cardiovasc Imaging 2019;12:1303–1312. [DOI] [PubMed] [Google Scholar]
- 15. Stuijfzand WJ, van Rosendael AR, Lin FY, Chang HJ, van den Hoogen IJ, Gianni U, et al. Stress myocardial perfusion imaging vs coronary computed tomographic angiography for diagnosis of invasive vessel-specific coronary physiology: predictive modeling results from the Computed Tomographic Evaluation of Atherosclerotic Determinants of Myocardial Ischemia (CREDENCE) Trial. JAMA Cardiol 2020;5:1338–1348. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.National Information Society Agency. Open AI Dataset Project (AI-Hub). Available from: https://aihub.or.kr (1 December 2023).
- 17. Arbab-Zadeh A, Di Carli MF, Cerci R, George RT, Chen MY, Dewey M, et al. Accuracy of computed tomographic angiography and single-photon emission computed tomography-acquired myocardial perfusion imaging for the diagnosis of coronary artery disease. Circ Cardiovasc Imaging 2015;8:e003533. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18. Feuchtner G, Loureiro R, Bezerra H, Rocha-Filho JA, Sarwar A, Pflederer T, et al. Quantification of coronary stenosis by dual source computed tomography in patients: a comparative study with intravascular ultrasound and invasive angiography. Eur J Radiol 2012;81:83–88. [DOI] [PubMed] [Google Scholar]
- 19. Narula J, Chandrashekhar Y, Ahmadi A, Abbara S, Berman DS, Blankstein R, et al. SCCT 2021 expert consensus document on coronary computed tomographic angiography: a report of the Society of Cardiovascular Computed Tomography. J Cardiovasc Comput Tomogr 2021;15:192–217. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20. Chou R. High value care task force of the American College of Physicians. Cardiac screening with electrocardiography, stress echocardiography, or myocardial perfusion imaging: advice for high-value care from the American College of Physicians. Ann Intern Med 2015;162:438–447. [DOI] [PubMed] [Google Scholar]
- 21. US Preventive Services Task Force; Curry SJ, Krist AH, Owens DK, Barry MJ, Caughey AB, Davidson KW, et al. Screening for cardiovascular disease risk with electrocardiography: US Preventive Services Task Force recommendation statement. JAMA 2018;319:2308–2314. [DOI] [PubMed] [Google Scholar]
- 22. Attia ZI, Harmon DM, Behr ER, Friedman PA. Application of artificial intelligence to the electrocardiogram. Eur Heart J 2021;42:4717–4730. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23. Huang PS, Tseng YH, Tsai CF, Chen JJ, Yang SC, Chiu FC, et al. An artificial intelligence-enabled ECG algorithm for the prediction and localization of angiography-proven coronary artery disease. Biomedicines 2022;10:394. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24. Leasure M, Jain U, Butchy A, Otten J, Covalesky VA, McCormick D, et al. Deep learning algorithm predicts angiographic coronary artery disease in stable patients using only a standard 12-lead electrocardiogram. Can J Cardiol 2021;37:1715–1724. [DOI] [PubMed] [Google Scholar]
- 25. Tang P, Wang Q, Ouyang H, Yang S, Hua P. The feasibility of early detecting coronary artery disease using deep learning-based algorithm based on electrocardiography. Aging 2023;15:3524–3537. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
The data used in the present study cannot be made publicly available due to the ethical restrictions set by the Institutional Review Board of Seoul National University Bundang Hospital (https://bri.snubh.org/irb/), as patient and participant privacy would be compromised. A minimally anonymized data set can be requested by contacting the corresponding authors (yeonyeeyoon@snubh.org or cho_y@snubh.org). The AI-based algorithms described in this paper are intended for inclusion in an upcoming version of the ECG Buddy application (ARPI Inc.), which restricts their public release. However, the algorithms are available for research purposes. Interested researchers should contact J.K. (joonghee79@gmail.com) to request for access.
In alignment with our commitment to ethical research practices and the conditions set by the Institutional Review Board (IRB) of Seoul National University Bundang Hospital, open access to research data without explicit participant consent is restricted. Consequently, the data from our study on diastolic dysfunction, collected retrospectively, cannot be publicly disclosed without constraints. Researchers seeking to engage in further study with the data set are advised to reach out to the corresponding author. Collaborative efforts can proceed within the IRB’s permitted scope or through the submission and approval of a new IRB application tailored to new research aims.