Meta-Analysis of the Performance of AI-Driven ECG Interpretation in the Diagnosis of Valvular Heart Diseases

Abstract

Valvular heart diseases (VHDs) significantly impact morbidity and mortality rates world-wide. Early diagnosis improves patient outcomes. Artificial intelligence (AI) applied to electrocardiogram (ECG) interpretation presents a promising approach for early VHD detection. We conducted a meta-analysis on the efficacy of AI models in this context. We reviewed databases including PubMed, MEDLINE, Embase, Scopus, and Cochrane until August 20, 2023, focusing on AI for ECG-based VHD detection. The outcomes included pooled accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value. The pooled proportions were derived using a random-effects model with 95% confidence intervals (CIs). Study heterogeneity was evaluated with the I-squared statistic. Our analysis included 10 studies, involving ECG data from 713,537 patients. The AI algorithms mainly screened for aortic stenosis (n = 6), mitral regurgitation (n = 4), aortic regurgitation (n = 3), mitral stenosis (n = 1), mitral valve prolapse (n = 2), and tricuspid regurgitation (n = 1). A total of 9 studies used convolution neural network models, whereas 1 study combined the strengths of support vector machine logistic regression and multilayer perceptron for ECG interpretation. The collective AI models demonstrated a pooled accuracy of 81% (95% CI 73 to 89, I² = 92%), sensitivity was 83% (95% CI 77 to 88, I² = 86%), specificity was 72% (95% CI 68 to 75, I² = 52%), PPV was 13% (95% CI 7 to 19, I² = 90%), and negative predictive value was 99% (95% CI 97 to 99, I² = 50%). The subgroup analyses for aortic stenosis and mitral regurgitation detection yielded analogous outcomes. In conclusion, AI-driven ECG offers high accuracy in VHD screening. However, its low PPV indicates the need for a combined approach with clinical judgment, especially in primary care settings.

Valvular heart diseases (VHDs) affect 2.5% of the population and are a significant cause of morbidity and mortality.¹ Late detection or misdiagnosis of VHDs can lead to severe complications, including heart failure, arrhythmias, and increased risk of death.² The most prevalent VHDs are aortic stenosis (AS) and mitral regurgitation (MR), followed by aortic regurgitation, mitral stenosis, mitral valve prolapse (MVP), and tricuspid regurgitation.^3–5 As VHDs continue to progress and potentially worsen with age and co-morbid conditions, early detection becomes important for potential interventions.³ Currently, VHDs are suspected based on ambiguous clinical symptoms and inconsistent physical examination results, especially murmurs.⁶ Diagnostic echocardiography is the gold standard for assessing VHDs; however, it is resource-intensive, making it impractical for large-scale screening.⁶ Electrocardiograms (ECGs), in contrast, are ubiquitous and are typically the first tests performed for cardiovascular diseases, including in the primary care setting. However, their utility is currently limited to detecting only specific conditions based on the criteria established by human experts.⁷ However, the recent surge in artificial intelligence (AI) applications, especially machine learning and deep learning, offers a novel avenue for enhancing the use of ECG as a diagnostic tool for VHDs.⁷ This article provides a comprehensive review and meta-analysis of VHD detection using AI-driven ECG interpretation.

Methods

This systematic review was performed according to Cochrane Collaboration guidance. We followed the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) statement.⁸

We searched PubMed, MEDLINE, Cochrane Library, Embase, Elton B. Stephens Company, Web of Science, and Cumulated Index to Nursing and Allied Health Literature databases from the inception through August 20, 2023. Additional manual searches from the reference list of the included studies were performed. Search terms such as “artificial intelligence,” “deep learning,” “machine learning,” “valvular heart disease,” and “electrocardiogram” were used in various combinations. The identification and assessment of studies for inclusion was systematically performed with the inclusion and exclusion criteria described in the next section.

A total of 2 reviewers (S.S. and R.C.) independently selected and abstracted data from eligible studies. Discrepancies were resolved by discussion and consensus. The results were reviewed by senior investigators (S.V. and M. E.H.). The eligibility criteria were (1) AI use for screening or detection of VHDs; (2) use of ECG for screening or detection of VHDs; and (3) reporting of at least 1 outcome of interest, that is, positive predictive value (PPV), negative predictive value (NPV), sensitivity, specificity, or accuracy. All nonhuman studies and studies with incomplete information on the outcomes of interest or studies evaluating other cardiovascular conditions were excluded.

The outcomes assessed were accuracy (total correct predictions by AI ÷ total number of ECGs analyzed), sensitiveity (number of true VHDs detected by AI ÷ total number of confirmed VHDs), specificity (number of ÷ ECGs correctly interpreted by AI as negative for VHDs total number of normal ECGs), PPV (number of true VHDs ÷ detected by AI ÷ all VHDs detected by AI), and NPV (number of ECGs correctly interpreted by AI as negative for VHDs all ECGs reported as normal by AI). A subgroup analysis÷was conducted for individual types of VHDs, wherever feasible.

A qualitative bias evaluation was performed using the following key parameters for each study: (1) clear definition of the study population, (2) clear definition of the outcomes and outcome assessment, (3) independent assessment of the outcome parameters, and (4) identification of important confounders and prognostic factors. Publication bias was estimated visually by funnel plots.^9,10 If any bias was observed, further bias quantification was measured using the Begg–Mazumdar test,¹¹ Egger test,⁹ and Duval–Tweedie trim-and-fill method.¹² Sensitivity analyses were performed to assess the contribution of each study to the pooled estimate by excluding studies 1 at a time.

We used the random-effects model to pool effect sizes with 95% confidence intervals (CIs) using the MedCalc Statistical Software version 20.023 (MedCalc Software Ltd, 2021; Ostend, Belgium). The heterogeneity of effects was evaluated using the Higgins I² statistic with mild, moderate, and significant heterogeneity defined as 25%, 50%, and 75%, respectively.¹³

Results

Our analysis included 10 studies containing ECG data on 713,537 patients^{1,3–7,14–17} (Figure 1). A total of 6 studies used AI to screen for AS,^1,5,14–17 4 studies for MR,^1,5,6,15 3 studies for aortic regurgitation,^3,5,15 1 study for mitral stenosis,⁵ 2 studies for MVP,^4,7 and 1 for tricuspid regurgitation.⁵ A total of 9 studies used convolution neural network (CNN), and the remaining 1 study used a combination of support vector machine (SVM), logistic regression (LR) and multilayer perceptron (MLP).⁴ The characteristics of individual studies are listed in Table 1. Supplementary Table 1 lists the time interval between ECGs and echocardiography, verification method to confirm VHDs, and ECG system used in the studies.

Figure 1.

PRISMA 2020 flow diagram depicting the search strategy.

Table 1.

Study characteristics

Study	Year	Aim	Model	Training strategy	Testing strategy
Cohen-Shelly	2021	ECG screening for moderatesevere AS	CNN	129,788 random subjects	102,926 random subjects
Elias	2022	ECG identification of moderate or severe AS, AR, and MR	CNN (ValveNet)	43,165 patients	21,048 patients
Hata	2020	ECG classification of AS	CNN	128 ECG data	44 ECG data
Kwon	2020	ECG detection of significant AS	MLP and CNN	39,371 ECGs from 25,733 patients	10,865 ECGs from 10,865 patients (external validation)
Kwon JM	2020	ECG detection of MR	CNN	56,670 ECGs from 24,202 patients	10,865 ECGs of 10,865 patients (external validation)
Lin	2021	ECG prediction of MVP	SVM, LR, MLP	1654 subjects	552 subjects
Sawano	2022	ECG diagnosis of significant AR	2D-CNN, FC-DNN	19,136 ECGs from 10,460 patients	6,036 ECGs from 3,269 patients
Tison	2019	ECG detection of MVP	CNN-HMM–heuristic filter	170 manually segmented ECGs	36,186 sinus rhythm ECGs
Ulloa-Cerna	2022	ECG prediction of moderate or severe valvular disease (AS, AR, MR, MS, TR)	CNN with classification pipeline (min–max scaling, mean imputation, XGBoost classifier, and calibration)	2,232,130 ECGs from 484,765 adults	276058 patients
Vaid	2023	ECG identification of AS and MR	MLP and CNN	617,338 ECG-Echo pairs from 123,096 patients (MR) 617,338 ECG-Echo pairs for 128,628 patients (AS)	617,338 ECG-Echo pairs from 123,096 patients (MR) 617,338 ECG-Echo pairs for 128,628 patients (AS)

2D-CNN = two-dimensional convolutional neural network; AR = aortic regurgitation; AS = aortic stenosis; CNN = convolutional neural network; ECG = electrocardiogram; FC-DNN = fully connected deep neural network; HMM = hidden Markov model; LR = logistic regression; MLP = multilayer perceptron; MR = mitral regurgitation; MS = mitral stenosis; MVP = mitral valve prolapse; SVM = support vector machine; TR = tricuspid regurgitation.

No significant publication bias was observed in the primary and safety outcomes analyses using funnel plots. However, it should be noted that AI research, particularly, in its early phases of application in cardiovascular medicine, tends to highlight positive results. The sensitivity analysis revealed that a single study did not drive the results. We deemed all the studies at a high risk of bias because of the pooling of different VHDs with different underlying etiologies and variability of co-morbidities in the study groups.

The pooled accuracy of AI for the diagnosis of VHDs using ECG was 81% (95% CI 73 to 89, I² = 92%), sensitivity was 83% (95% CI 77 to 88, I² = 86%), specificity was 72% (95% CI 68 to 75, I² = 52%), PPV was 13% (95% CI 7 to 19, I² = 90%), and NPV was 99% (95% CI 97 to 99, I² = 50%) (Figure 2).

Figure 2.

Pooled accuracy, sensitivity, positive predictive value, and negative predictive value of AI-driven ECG diagnosis of valvular heart diseases from meta-analysis of the included studies.

The subgroup analysis for detection of AS showed a pooled accuracy of 86% (95% CI 73 to 96, I² = 93%), sensitivity of 88% (95% CI 81 to 93, I² = 68%), specificity of 71% (95% CI 65 to 76, I² = 32%), PPV of 12% (95% CI 9 to 16, I² = 15%), and NPV of 99% (95% CI 98 to 100, I² = 0%) (Table 2). Similarly, the subgroup analysis for screening of MR revealed a pooled accuracy of 86% (95% CI 62 to 99, I² = 95%), sensitivity of 88% (95% CI 83 to 92, I² = 28%), specificity of 69% (95% CI 61 to 76, I² = 55%), PPV of 23% (95% CI 5 to 48, I² = 95%), and NPV of 97% (95% CI 90 to 100, I² = 85%) (Table 2).

Table 2.

Pooled outcomes for types of valvular heart diseases

Pooled rates	Accuracy	Sensitivity	Specificity	PPV	NPV
AS	86.36 (73.05 to 95.62) I2 = 93.22% 5 studies	87.56 (81.27 to 92.70) I2 = 68.60% 4 studies	70.88 (65.36 to 76.10) I2 = 32.26% 4 studies	12.21 (8.96 to 15.88) I2 = 15.39% 4 studies	98.84 (97.57 to 99.65) I2 = 0.00% 4 studies
MR	86.00 (62.03 to 98.94) I2 = 95.97% 3 studies	87.50 (82.78 to 91.56) I2 = 28.48% 3 studies	68.73 (60.65 to 76.27) I2 = 55.71% 3 studies	22.96 (5.44 to 47.85) I2 = 95.46% 3 studies	96.89 (89.79 to 99.92) I2 = 85.53% 3 studies

Values in pooled proportions (95% confidence intervals).

AS = aortic stenosis; MR = mitral regurgitation; NPV = negative predictive value; PPV = positive predictive value.

Discussion

Our study fills a critical gap by providing the most comprehensive meta-analysis to date evaluating the value of ECG-based AI for screening for VHDs. The pooled accuracy, sensitivity, specificity, and NPV of AI were exceptionally high, confirming the significant potential of this technology. However, the PPV was notably low. Several factors could contribute to this low PPV. A potential reason could be the inherent variability in ECG recordings. In addition, factors such as patient positioning, electrode placement, and minor arrhythmias can affect ECG readings, potentially influencing the AI’s interpretation.¹⁸

The observed low PPV, juxtaposed with a high NPV, has significant clinical implications. Although AI-driven ECG interpretation exhibits a strong potential as a first-line screening tool to exclude VHDs, its current capacity to positively diagnose the disease is limited.¹ Compared with traditional diagnostic methods, AI-driven ECG interpretation provides faster results; however, reliability remains a concern. In primary care or first-contact medical scenarios where ruling out VHDs is paramount, AI-driven ECG interpretation could be instrumental. Nevertheless, for diagnostic purposes or in specialty clinics, it may be prudent not to rely solely on AI for definitive diagnosis. Collaborative diagnosis, where AI interpretation is vetted with clinical judgment, may mitigate this limitation.

Another combined screening method for detecting VHDs could enhance the PPV, as demonstrated by Ulloa-Cerna et al.⁵ Furthermore, the significant correlation observed by Vaid et al¹ between AI-detected AS and patients who underwent transcatheter procedures underscores the benefits of incorporating AI into clinical practice. This is supported by the high pooled accuracy observed in our study.

The subgroup analyses for detecting AS and MR yielded similar results. The significance of these findings is heightened given the increasing importance of early VHD diagnosis, especially because previous studies have highlighted improved outcomes with timely intervention in asymptomatic patients.¹⁹ Especially in resource-limited settings where advanced diagnostic tools such as echocardiography may not be readily available, the potential for AI-driven ECG interpretation is paramount. Such settings could benefit immensely from the cost-effectiveness and rapidity of AI-driven ECGs.

Given the increasing popularity and accessibility of wearable health devices, such as smartwatches and fitness bands, the potential for AI-driven ECG interpretation for patient self-monitoring becomes even more significant. It can empower patients to monitor their heart health and seek timely medical interventions. However, the accuracy of readings from these devices, potential data overload for health care professionals, and patient anxiety from false alarms are challenges that need addressing.

Notably, 1 of the pioneering studies by Tison et al⁷ used a fusion of CNN and hidden Markov models for MVP detection, revealing an area under the receiver operating characteristic of 0.77 (95% CI 0.76 to 0.78). This accuracy was slightly lower for MVP than for other cardiac pathologies, likely attributed to the subtle impact of MVP on ECG.⁷ This was further corroborated by a subsequent study by Lin and Zeng,⁴ which used different machine learning models such as SVM, LR, and MLP.

The heterogeneity in our analysis stemmed largely from the diverse AI models used across studies. For instance, although CNN models were predominant, the synergistic application of SVM, LR, and MLP in the included studies warrants attention. SVM is particularly adept at high-dimensional spaces, making it apt for intricate ECG data. LR offers probabilistic outcomes, presenting a nuanced risk assessment, whereas MLP identifies nonlinear relations in ECG signals. Their combined utility has the potential to provide a more comprehensive interpretation of ECG data, capturing informational facets that CNNs may overlook. Yet, as we venture further into the realm of AI-driven health care, it is crucial to ensure that patient consent is obtained transparently, especially when it comes to data sharing and AI analysis. Ethical guidelines need to be established to ensure that the patients’ rights and privacy are not compromised.

In envisioning future AI-driven ECG interpretation, we anticipate a shift toward studies that combine advanced AI methods with comprehensive clinical validation. This would ideally include real-time validation of AI across diverse patient groups, confirming the method’s broad applicability. Consistently improving the AI in response to real-world data can enhance their precision and versatility. Moreover, collaborations between AI experts and clinicians are necessary to ensure that AI remains relevant to clinical practice.

Our study has several limitations. First, the lack of external validation in some studies curtails the generalizability of our results. These limitations imply that although AI-driven ECG offers a promising avenue for VHD screening, its application in diverse clinical settings requires cautious optimism. Second, the analysis was limited by the inherent bias of the retrospective design of the included studies. In addition, the use of identical data sets for training and testing in some studies could have inadvertently inflated the reported accuracies. Moreover, the limited studies on VHDs other than AS and MR rendered us incapable of conducting a comprehensive subgroup analysis for all VHD types.

In conclusion, although AI-driven ECG offers high accuracy in VHD screening, its low PPV limits definitive diagnosis. Collaboration between AI and clinical judgment can optimize its clinical utility, especially in primary care scenarios. Moving forward, further research should delve into optimizing AI algorithms, considering diverse age groups, ethnicities, and geographical locations. Exploring their utility in diverse patient populations and collaborating with interdisciplinary teams, including cardiologists, data scientists, and ethicists, will be crucial. Assessing their potential in tandem with other diagnostic tools is also recommended.