Abstract
Introduction:
The interpretation of electrocardiograms (ECGs) involves dynamic interplay between computerized ECG interpretation (CEI) software and human overread. However, the impact of computer ECG interpretation on the performance of healthcare professionals remains largely unexplored. The aim of this study was to evaluate the interpretation proficiency of various medical professional groups, with and without access to the CEI report.
Methods:
Healthcare professionals from diverse disciplines, training levels, and countries sequentially interpreted 60 standard 12-lead ECGs, demonstrating both urgent and non-urgent findings. The interpretation process consisted of two phases. In the first phase, participants interpreted 30 ECGs with clinical statements. In the second phase, the same 30 ECGs and clinical statements were randomized and accompanied by a CEI report. Diagnostic performance was evaluated based on interpretation accuracy, time per ECG (in seconds [s]), and self-reported confidence (rated 0 [not confident], 1 [somewhat confident], or 2 [confident]).
Results:
A total of 892 participants from various medical professional groups participated in the study. This cohort included 44 (4.9%) primary care physicians, 123 (13.8%) cardiology fellows-in-training, 259 (29.0%) resident physicians, 137 (15.4%) medical students, 56 (6.3%) advanced practice providers, 82 (9.2%) nurses, and 191 (21.4%) allied health professionals. The inclusion of the CEI was associated with a significant improvement in interpretation accuracy by 15.1% (95% confidence interval, 14.3 to 16.0; P<0.001), decrease in interpretation time by 52 s (−56 to -48; P<0.001), and increase in confidence by 0.06 (0.03 to 0.09; P=0.003). Improvement in interpretation accuracy was seen across all professional subgroups, including primary care physicians by 12.9% (9.4 to 16.3; P=0.003), cardiology fellows-in-training by 10.9% (9.1 to 12.7; P<0.001), resident physicians by 14.4% (13.0 to 15.8; P<0.001), medical students by 19.9% (16.8 to 23.0; P<0.001), advanced practice providers by 17.1% (13.3 to 21.0; P<0.001), nurses by 16.2% (13.4 to 18.9; P<0.001), allied health professionals by 15.0% (13.4 to 16.6; P<0.001), physicians by 13.2% (12.2 to 14.3; P<0.001), and non-physicians by 15.6% (14.3 to 17.0; P<0.001).
Conclusion:
CEI integration improves ECG interpretation accuracy, efficiency, and confidence among healthcare professionals.
Keywords: computer interpretation, ECG analysis, ECG competency, ECG interpretation, medical professionals
INTRODUCTION
In contemporary medical practice, electrocardiogram (ECG) interpretation involves a synergistic blend of human expertise and computerized ECG interpretation (CEI) software. Since the mid-1900s, CEI software has been extensively utilized to process the vast majority of recorded ECGs (1–3), with the aim of improving clinician interpretation, reducing errors, streamlining clinical workflow, and minimizing healthcare expenses (3–6).
Despite the widespread adoption of CEI software in conjunction with human overread, the impact of CEIs on the overall ECG interpretation performance across various medical professionals remains largely unexplored. While previous research has demonstrated the potential benefits of CEI programs in enhancing ECG interpretation accuracy and efficiency, it has also highlighted limitations in diagnostic performance and negative influence on decision making (7–19). Moreover, prior work evaluating the impact of CEI have typically been limited by small sample sizes and a narrow focus on expert ECG interpreters, failing to encompass the broader range of medical professional groups who interpret ECG (19–28).
This study aims to evaluate how CEI influences the interpretation performance of diverse group of healthcare professionals from varying disciplines and training levels. In doing so, we hope to gain insights that can enhance diagnostic performance of CEI software, address gaps in ECG interpretation proficiency, and prepare for the evolving role of diagnostic and predictive analytic models in medical practice.
METHODS
Study Participants
Participants included a diverse group of healthcare professionals participating in the EDUcation Curriculum Assessment for Teaching Electrocardiography (EDUCATE) Trial (29). Interested participants registered, enrolled, and provided electronic informed consent. We have previously described participant registration and enrollment (29).
Eligible participants met the following criteria: completion of professional training or current enrollment in a training program, and age 18 years or older. The participant pool comprised various healthcare groups, including medical students, resident physicians, cardiology fellows-in-training, primary care physicians (including family medicine, internal medicine, and emergency medicine physicians), nurses, advanced practice providers (including physician assistants and nurse practitioners), and allied health professionals (including ECG technicians, emergency medicine technicians, and paramedics). Physician providers were defined as primary care physicians, cardiology fellows-in-training, and resident physicians, while non-physician providers were defined as advanced practice providers, nurses, and allied health professionals. Practicing cardiologists and emergency medicine physicians were excluded from participation in the study.
Participant information was de-identified and securely stored. The confidentiality of individual participants and their institutional and program affiliations was maintained. Participation was voluntary and no financial incentives or continuing medical education credits were provided. Participants received one year of free access to an online educational platform as the sole study participation incentive.
Study Design
The study focused on the sequential interpretation of standard 12-lead ECGs. These ECGs included both urgent and non-urgent findings commonly taught in medical training (Table 1). The interpretation process consisted of two phases. In the first phase, participants analyzed 30 ECGs accompanied solely by a clinical statement, providing essential contextual information (e.g., “75-year-old male with chest pain”). In the second phase, the same 30 ECGs and clinical statements were re-presented in a random order, with addition of the CEI report.
Table 1.
ECG findings evaluated and their prevalence in the test set.
| Urgent Findings (15) | Non-Urgent Findings (54) |
|---|---|
| ST-elevation myocardial infarction (5) | Normal sinus rhythm (15) |
| ST-T changes suggesting ischemia (2) | Sinus bradycardia (4) |
| Ventricular tachycardia (1) | Sinus tachycardia (1) |
| Ventricular fibrillation (1) | Sinus arrhythmia (1) |
| Supraventricular tachycardia (1) | Atrial fibrillation (3) |
| Second-degree AV block, Mobitz type II (1) | Atrial flutter (1) |
| Third-degree AV block (1) | Multifocal atrial tachycardia (1) |
| Prolonged QT interval (1) | AV junctional rhythm (2) |
| Ventricular preexcitation (1) | Premature atrial complex (1) |
| Pericarditis (1) | Premature ventricular complex (1) |
| First-degree AV block (4) | |
| Second-degree AV block, Mobitz type I (1) | |
| Right atrial abnormality/enlargement (2) | |
| Left atrial abnormality/enlargement (2) | |
| Right bundle branch block (1) | |
| Left bundle branch block (2) | |
| Left anterior fascicular block (1) | |
| Left ventricular hypertrophy (3) | |
| Early repolarization (1) | |
| Age indeterminate or old myocardial infarction (5) | |
| Incorrect electrode placement (1) | |
| Atrial pacing (1) |
Study Approval
This study was approved by the educational review committee and institutional review boards (IRBs) at the Mayo Clinic, as well as IRBs at Washington University School of Medicine in St. Louis, SUNY Upstate Medical University, and Keck School of Medicine at the University of Southern California. WCG IRB, an external IRB review company, also reviewed and approved this work.
Study Support
GE Healthcare (Milwaukee, WI) provided all ECGs used in this study. The company was not involved in study implementation, data analysis, or final reporting of study results.
ECG Selection and Preparation
GE Healthcare (Milwaukee, WI) made available a comprehensive collection of standardized 12-lead ECGs. Each ECG consisted of three 10-second rhythm strips (leads II, V1, and V5) and followed a standard paper speed of 25 mm/s and voltage calibration of 10 mm/mV.
The goal of the selection process was to include 12-lead ECGs that covered commonly taught urgent and non-urgent interpretation findings, with the possibility of multiple findings present on a single ECG. Signal quality was prioritized in the selection, and ECG selection was independent of the accuracy of the CEI statements.
All ECGs were accompanied by brief clinical statements generated by the investigators (A.H.K. and A.M.M.). These statements included basic information such as the patient’s age, sex, and a concise description of the clinical context, including the patient setting, chief complaint, or clinical diagnosis. The first set of 30 ECGs included clinical statements but did not include a CEI report, while the second set of the same 30 ECGs included both the clinical statements and a CEI report. The CEI report provided measurements such as ventricular rate, PR interval, QRS duration, QT/QTc intervals, and P-R-T axes, along with direct and unaltered interpretive statements provided by the manufacturer. The order of the second set of 30 ECGs was randomized to ensure it differed from the order of the first set of ECGs. The clinical statements used for the first set were the same as those used for the second set.
A panel consisting of three board-certified cardiologists (P.A.N., N.S.A., and A.M.M.) and one cardiology fellow-in-training (A.H.K.) established the reference standard for ECG interpretation through consensus agreement.
Study Protocol
Once enrolled, we instructed registrants via email to complete a baseline survey and independently interpret the prepared ECGs.
Baseline Survey.
Participants completed a baseline survey to collect general characteristics information regarding demographics, profession, work setting, clinical experience, average weekly ECG interpretations, prior dedicated ECG training, expert ECG interpretation supervision, independent ECG interpretation comfort, and ECG interpretation responsibility in the workplace.
ECG interpretation.
Participants were allowed 10 minutes to complete three interpretation tasks for each ECG. They were expected to (i) estimate the ventricular rate within a range (i.e., <40 beats/minute, 40-80 beats/minute, 81-120 beats/minute, 121-160 beats/minute, or >160 beats/minute), (ii) determine the mean QRS axis (i.e., normal axis [−30° to +90°], right axis deviation [+90° to +180°], left axis deviation [−30° to −90°], right superior axis [−90° to −180°], or indeterminate axis), and (iii) provide a full interpretation using from a list of predefined multiple choice options. No free-text responses were allowed. Figure 1 demonstrates the display format and scoresheet presentation for each ECG.
Figure 1. ECG question presentation format.
This question incorporates the computerized ECG interpretation report with its interpretive statements and measurements.
Participants were presented with one ECG at a time and were required to complete all interpretation tasks before proceeding to the next ECG. After submitting their answers, or when the allotted time (10 minutes) expired, participants rated their confidence level as confident, somewhat confident, or not confident. Participants were unable to revisit previous questions and were unaware of their interpretation performance as they progressed through the test. They were not required to interpret both ECG sets in a single session; they had the option to pause and resume at their convenience to complete the remaining ECGs.
Performance Metrics
Participants’ interpretation accuracy was evaluated based on the percentage of correctly identified ECG findings using a binary measure: one point for a correct response and zero points for an incorrect response. The maximum point value for the test was 69 points (i.e., 1 point for 69 interpretative findings found on 30 ECGs). No weighted-point values for specific findings or penalties were incorporated into the evaluation. Interpretation time per ECG (seconds [s]) and self-reported confidence (rated on an ordinal scale of 0 [not confident], 1 [somewhat confident], or 2 [confident]) were also recorded.
Statistical Analysis
A series of analyses were conducted to offer a multifaceted understanding of the results. The impact of CEI statements on interpretation accuracy, interpretation time per ECG, self-reported confidence, determination of ventricular rate and mean QRS axis, and identification of specific ECG findings was examined. The effect of inaccurate CEI statements on participants′ interpretation accuracy was explicitly scrutinized. The identification of ST-elevation myocardial infarction (STEMI) was solely based on whether its presence was acknowledged, excluding the precision of STEMI localization. Additionally, the influence of CEI statements on the identification of emergencies was specifically assessed. Emergencies constituted a composite category that included STEMI (presence only), ventricular tachycardia, ventricular fibrillation, and third-degree AV block.
Participant study data was exported from online learning management software and imported into statistical analysis software. Statistical analyses were performed using MedCalc for Windows, version 19.4 (MedCalc Software, Ostend, Belgium). Descriptive statistics were used to summarize survey data, with nominal and continuous variables reported as a count (percent of total) and mean, respectively. T-test analyses were used to compare performance metrics. Statistical significance was determined based on a two-tailed alpha <0.05.
RESULTS
Study Participants
Table 2 presents the baseline characteristics of study participants. The study included 892 medical professionals (40.8% female). Most participants were aged 26-34 years (522, 58.5%), and came from the United States (519, 58.2%). The participant cohort included diverse medical professionals: 44 (4.9%) primary care physicians, 123 (13.8%) cardiology fellows-in-training, 259 (29.0%) resident physicians, 137 (15.4%) medical students, 56 (6.3%) advanced practice providers, 82 (9.2%) nurses, and 191 (21.4%) allied health professionals. Physicians and non-physicians comprised 47.8% (426) and 36.9% (329) of all participants, respectively.
Table 2.
Participant characteristics.
| Characteristic (N=892) – No. (%) | |||
|---|---|---|---|
| Age distribution | Average ECG interpretations | ||
| 18-25 years | 127 (14.2) | 0 per week | 149 (16.7) |
| 26-35 years | 522 (58.5) | 1-10 per week | 491 (55.0) |
| 36-45 years | 143 (16.0) | 11-25 per week | 174 (19.5) |
| 46-55 years | 66 (7.4) | >25 per week | 78 (8.7) |
| >55 years | 34 (3.8) | ECG interpretation responsibility | |
| Sex | Directly impacts patient care | 600 (67.3) | |
| Female | 364 (40.8) | Indirectly impacts patient care | 113 (12.7) |
| Male | 528 (59.2) | No impact on patient care | 42 (4.7) |
| Location | Not applicable | 137 (15.4) | |
| United States | 519 (58.2) | ECG interpretation comfort | |
| Outside United States | 373 (41.8) | Uncomfortable | 384 (43.0) |
| Professional group | Somewhat comfortable | 388 (43.5) | |
| Primary care physicians | 44 (4.9) | Comfortable | 120 (13.5) |
| Cardiology fellows-in-training | 123 (13.8) | Dedicated ECG training | |
| Resident physicians | 259 (29.0) | 0 hours | 399 (44.7) |
| Medical students | 137 (15.4) | <5 hours | 242 (27.1) |
| Advanced practice providers | 56 (6.3) | ≥5 hours | 251 (28.1) |
| Nurses | 82 (9.3) | Expert ECG interpreter supervision | |
| Allied health professionals | 191 (21.4) | None | 367 (41.1) |
| Physicians1 | 426 (47.8) | Rarely | 241 (27.0) |
| Non-physicians1 | 329 (36.9) | Somewhat often | 183 (20.5) |
| Direct clinical experience | Often | 70 (7.8) | |
| 0-3 years | 435 (48.8) | Very often | 31 (3.5) |
| 4-7 years | 212 (23.8) | Value of computer ECG interpretation | |
| 8-10 years | 74 (8.3) | Unhelpful | 220 (24.7) |
| 11-20 years | 107 (12.0) | Somewhat helpful | 517 (58.0) |
| >20 years | 64 (7.2) | Helpful | 155 (17.4) |
ECG Interpretation Performance
Results in Tables 3 and 4, Figure 2, and Supplemental Table 1 show participants’ interpretation performance metrics with and without the CEI.
Table 3.
Interpretation performance for various evaluated metrics.
| Metric | Without CEI | With CEI | Mean Difference | P Value |
|---|---|---|---|---|
| Overall interpretation accuracy | 46.8 ± 18.7 | 62.0 ± 19.4 | ↑ 15.1 ± 13.1 | <0.001 |
| Confidence | 0.88 ± 0.53 | 0.94 ± 0.30 | ↑ 0.06 ± 0.48 | 0.003 |
| Interpretation time | 145 ± 67 | 93 ± 27 | ↓ −52 ± 63 | <0.001 |
| Ventricular rate | 79.4 ± 18.4 | 89.7 ± 17.1 | ↑ 10.3 ± 15.5 | <0.001 |
| Mean QRS axis | 72.4 ± 22.2 | 74.4 ± 23.5 | ↑ 1.9 ± 13.9 | 0.065 |
| Primary rhythm | 56.0 ± 19.8 | 70.2 ± 19.5 | ↑ 14.2 ± 15.1 | <0.001 |
| Sinus rhythm | 69.1 ± 25.5 | 75.1 ± 22.6 | ↑ 6.0 ± 19.1 | <0.001 |
| Atrial fibrillation | 49.8 ± 36.9 | 77.5 ± 31.0 | ↑ 27.7 ± 36.4 | <0.001 |
| Premature atrial/ventricular complexes | 64.0 ± 35.4 | 73.2 ± 32.5 | ↑ 9.2 ± 28.2 | <0.001 |
| AV block | 43.6 ± 25.1 | 70.0 ± 24.7 | ↑ 26.4 ± 25.5 | <0.001 |
| ST-elevation myocardial infarction1 | 65.9 ± 29.7 | 76.5 ± 27.1 | ↑ 10.6 ± 24.2 | <0.001 |
| Emergencies2 | 70.2 ± 26.8 | 78.8 ± 23.7 | ↑ 8.6 ± 19.0 | <0.001 |
| Bundle branch block | 57.4 ± 37.2 | 78.5 ± 31.3 | ↑ 21.2 ± 34.6 | <0.001 |
| Left ventricular hypertrophy | 18.2 ± 25.9 | 33.7 ± 28.7 | ↑ 15.5 ± 29.8 | <0.001 |
| Pericarditis | 51.5 ± 50.0 | 38.1 ± 48.6 | ↓ −13.3 ± 45.3 | <0.001 |
ST-elevation myocardial infarction recognition, not localization, was evaluated.
Emergencies included ST-elevation myocardial infarction1, ventricular tachycardia, ventricular fibrillation, and third-degree AV block.
Abbreviations: CEI, computer ECG interpretation.
Table 4. Interpretation performance by medical professional group.
Mean difference in interpretation accuracy, ventricular rate, mean QRS axis, confidence, and time between the presence and absence of the computerized ECG interpretation are shown.
| Group | N | Interpretation Accuracy (%) | Ventricular Rate (%) | Mean QRS Axis (%) | Confidence (range: 0-2) | Interpretation Time (s) | |||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Δ | P-Value | Δ | P-Value | Δ | P-Value | Δ | P-Value | Δ | P-Value | ||
| Primary care physicians | 44 | ↑ 12.9 ± 11.4 | 0.003 | ↑ 7.1 ± 11.9 | 0.016 | ↓ −0.1 ± 10.5 | 0.985 | ↓ −0.03 ±0.51 | 0.708 | ↓ −52 ± 67 | <0.001 |
|
| |||||||||||
| Cardiology fellows-in-training | 123 | ↑ 10.9 ± 9.9 | <0.001 | ↑ 7.1 ± 13.4 | <0.001 | ↑ 2.2 ± 8.8 | 0.274 | ↓ −0.19 ± 0.48 | 0.001 | ↓ −60 ± 65 | <0.001 |
|
| |||||||||||
| Resident physicians | 259 | ↑ 14.4 ± 11.6 | <0.001 | ↑ 7.5 ± 12.4 | <0.001 | ↓ −0.3 ± 12.2 | 0.829 | ↑ 0.10 ± 0.46 | 0.005 | ↓ −51 ± 61 | <0.001 |
|
| |||||||||||
| Medical students | 137 | ↑ 19.9 ± 18.2 | <0.001 | ↑ 11.4 ± 17.5 | <0.001 | ↑ 0.1 ± 16.5 | 0.971 | ↑ 0.20 ± 0.48 | <0.001 | ↓ −46 ± 67 | <0.001 |
|
| |||||||||||
| Advanced practice providers | 56 | ↑ 17.1 ± 14.4 | <0.001 | ↑ 13.8 ± 15.0 | <0.001 | ↑ 2.3 ± 12.8 | 0.623 | ↑ 0.18 ± 0.45 | 0.006 | ↓ −42 ± 55 | <0.001 |
|
| |||||||||||
| Nurses | 82 | ↑ 16.2 ± 12.4 | <0.001 | ↑ 11.9 ± 17.6 | 0.002 | ↑ 5.4 ± 16.2 | 0.273 | ↑ 0.28 ± 0.46 | <0.001 | ↓ −40 ± 58 | <0.001 |
|
| |||||||||||
| Allied health professionals | 191 | ↑ 15.0 ± 11.5 | <0.001 | ↑ 14.6 ± 17.7 | <0.001 | ↑ 4.9 ± 15.9 | 0.061 | ↓ −0.04 ± 0.43 | 0.320 | ↓ −59 ± 62 | <0.001 |
|
| |||||||||||
| Physicians1 | 426 | ↑ 13.2 ± 11.2 | <0.001 | ↑ 7.3 ± 12.6 | <0.001 | ↑ 0.4 ± 11.2 | 0.717 | ↑ 0.00 ± 0.49 | 0.738 | ↓ −54 ± 63 | <0.001 |
|
| |||||||||||
| Non-physicians1 | 329 | ↑ 15.6 ± 12.2 | <0.001 | ↑ 13.8 ± 17.3 | <0.001 | ↑ 4.6 ± 15.5 | 0.034 | ↑ 0.08 ± 0.46 | 0.013 | ↓ −52 ± 61 | <0.001 |
|
| |||||||||||
| All participants | 892 | ↑ 15.1 ± 13.1 | <0.001 | ↑ 10.3 ± 15.5 | <0.001 | ↑ 1.9 ± 13.9 | 0.065 | ↑ 0.06 ± 0.48 | 0.003 | ↓ −52 ± 63 | <0.001 |
Physicians include primary care physicians, cardiology fellows-in-training, and physician residents, whereas non-physicians include advanced practice providers, nurses, and allied health professionals.
Figure 2. Mean difference in performance with and without the computer ECG interpretation.
1 Physicians include primary care physicians, cardiology fellows-in-training, and physician residents, whereas non-physicians include advanced practice providers, nurses, and allied health professionals.
2 ST-elevation myocardial infarction was only assessed for recognition of its presence and not for localization accuracy.
3 Emergencies included STEMI3, ventricular tachycardia, ventricular fibrillation, and third-degree AV block.
Abbreviations: CI, confidence interval; CEI, computer ECG interpretation; STEMI, ST-elevation myocardial infarction.
The inclusion of CEI led to an overall increase in ECG interpretation accuracy by 15.1% (14.3 to 16.0, P<0.001) (Table 3). Overall ECG interpretation accuracy increased across all medical professional subgroups: primary care physicians 12.9% (9.4 to 16.3; P=0.003), cardiology fellows-in-training 10.9% (9.1 to 12.7; P<0.001), resident physicians 14.4% (13.0 to 15.8; P<0.001), medical students 19.9% (16.8 to 23.0; P<0.001), advanced practice providers 17.1% (13.3 to 21.0; P<0.001), nurses 16.2% (13.4 to 18.9; P<0.001), allied health professionals 15.0% (13.4 to 16.6; P<0.001), physicians 13.2% (12.2 to 14.3; P<0.001), and non-physicians 15.6% (14.3 to 17.0; P<0.001) (Table 4 and Supplemental Table 1).
CEI inclusion was associated with a significant 10.3% (9.3 to 11.4; P<0.001) increase in accuracy for determining ventricular rate and a modest 1.9% (1.0 to 2.8; P=0.065) increase for determining mean QRS axis (Table 3 and Supplemental Table 1).
The inclusion of the CEI report led to an overall increase in primary rhythm interpretation accuracy by 14.2% (13.2 to 15.1; P<0.001) for all participants, with consistent improvements observed in primary rhythm interpretation across all subgroups (Table 3 and Figure 2). The detection of STEMI and emergencies also increased for all subgroups with the CEI, with increases of 10.6% (9.0 to 12.2; P<0.001) and 8.6% (7.4 to 9.9; P<0.001) for all participants, respectively (Tables 3 and Figure 2). With the CEI, significant improvements (P<0.001) in accuracy were also seen for the detection of sinus rhythm, atrial fibrillation, premature atrial and ventricular complexes, AV block, STEMI, emergencies, bundle branch block, left ventricular hypertrophy, and pericarditis.
Average interpretation time per ECG decreased from 145 ± 67 s to 93 ± 27 s with the CEI, amounting to an average reduction of 52 s (−56 to −48; P<0.001) (Table 3). Participants’ self-reported confidence modestly increased from 0.88 ± 0.53 to 0.94 ± 0.30 with the CEI (P=0.003) (Table 3). Most groups, except for cardiology fellows-in-training, demonstrated greater confidence with the CEI (Table 4 and Supplemental Table 1).
Among the 69 interpretive findings included in the test, the CEI report was deemed incorrect for 7 (10.1%) of them. The presence of incorrect CEI reports did not negatively impact the interpretation accuracy of all participants (−0.5 ± 14.6, P=0.677). Similarly, the influence of incorrect CEI reports on interpretation accuracy did not reach statistical significance among the professional groups (P>0.05) (Table 5).
Table 5.
Incorrect computerized ECG interpretation influence on interpretation accuracy.
| Group | N | Incorrect CEI Impact (%) | |
|---|---|---|---|
| Δ | P-Value | ||
| Primary care physicians | 44 | ↓ −1.9 ± 14.3 | 0.735 |
|
| |||
| Cardiology fellows-in-training | 123 | ↓ −4.9 ± 14.5 | 0.064 |
|
| |||
| Resident physicians | 259 | ↓ −0.1 ± 16.3 | 0.962 |
|
| |||
| Medical students | 137 | ↑ 0.0 ± 13.9 | 1.000 |
|
| |||
| Advanced practice providers | 56 | ↓ −1.3 ± 12.3 | 0.732 |
|
| |||
| Nurses | 82 | ↑ 0.7 ± 13.3 | 0.851 |
|
| |||
| Allied health professionals | 191 | ↑ 1.3 ± 13.7 | 0.578 |
|
| |||
| Physicians1 | 426 | ↓ −1.6 ± 15.7 | 0.309 |
|
| |||
| Non-physicians1 | 329 | ↑ 0.7 ± 13.4 | 0.698 |
|
| |||
| All participants | 892 | ↓ −0.5 ± 14.6 | 0.677 |
Physicians include primary care physicians, cardiology fellows-in-training, and physician residents, whereas non-physicians include advanced practice providers, nurses, and allied health professionals.
DISCUSSION
Our study evaluated the impact of the CEI on ECG interpretation performance among a diverse group of healthcare professionals. The results demonstrate that including the CEI led to a substantial improvement in ECG interpretation accuracy, efficiency, and confidence, indicating a positive influence of the CEI on interpretation proficiency. Furthermore, our study demonstrated significant enhancements in primary rhythm analysis and the detection of critical and emergent findings, highlighting the potential benefits of CEI software in improving patient care.
Comparison to Prior Work
Our study adds to existing literature by assessing the effect of CEI on a broad spectrum of medical professionals. While previous research has mainly focused on comparing the performance of various CEI programs against each other and against expert interpreters (7–28), there is a scarcity of studies that directly probe the influence of CEI on the interpretative accuracy of non-expert interpreters. As such, this study fills a critical gap in research, shedding light on an area that has received limited attention.
Our findings align with prior studies, demonstrating that CEI software improves interpretation accuracy and reduces interpretation time (6–8, 25–27). In addition, similar to our findings, prior research has reported no significant impact on resident physicians’ interpretations in the presence or absence of erroneous CEIs, similar to our observations (28). Despite previous research highlighting the negative impact of incorrect CEI on atrial fibrillation diagnosis, it is worth noting that our study, which had a limited selection of ECGs, did not include any incorrect CEI reports for atrial fibrillation (14–15).
Study Implications
The findings of our study have significant relevance for clinical practice and the future of ECG interpretation. Improvements in rhythm analysis and detection of critical ECG findings, highlight the potential advantages of CEI in enhancing human diagnostic skills and clinical decisionmaking abilities. These implications are particularly important considering that non-expert medical professionals, who often perform initial ECG interpretation in frontline patient care, have demonstrated several notable improvements when using CEI. Furthermore, the observation that CEI reports significantly enhance both the efficiency and accuracy of ECG interpretations emphasizes the instrumental role CEI software can play in clinical workflows, facilitating rapid and precise ECG interpretations in genuine clinical practice.
Our findings revealed that cardiology fellows-in-training derived the least benefit from the CEI report compared to other groups. This outcome is likely due to their higher level of ECG interpretation expertise, medical training, and clinical responsibility for cardiovascular care. This finding also suggests that as the interpreter’s skill level improves, the added value of the CEI becomes more limited, potentially reaching a threshold where the impact of CEI software on performance diminishes. However, our findings clearly highlight the overall value of CEI for healthcare professionals who typically do not possess strong ECG interpretation skills.
Therefore, it is important to consider potential variations in the usefulness of CEI reports based on the interpreter’s initial skill level during the development of CEI.
Similar to prior work, our study also reveals significant gaps in ECG interpretation performance among various medical professional groups, even in the context of CEI usage (30). While CEI plays a role in improving ECG interpretation, it does not fully supplant the impact of ECG interpretation deficiencies. Hence, there is a need to enhance ECG interpretation performance through other avenues, including broad-based educational initiatives (31). For maximizing the clinical utility of ECGs, robust training programs may be necessary to adequately bridge proficiency gaps.
Lastly, as we venture further into a new transformative era of diagnostic medicine, characterized by the prevalence of automation and artificial intelligence (AI) technologies, there is a growing need for a comprehensive understanding of the dynamic between healthcare professionals and CEI. This understanding becomes particularly significant in light of the rapid development of AI-enhanced ECG (AI-ECG) models (32–33). At present, the existing CEI software, largely reliant on rule-based algorithms – a rudimentary form of AI – does adequately encompass the evolving landscape of recent AI-ECG advancements. However, existing CEI algorithms, much like their emerging AI counterparts, analyze and interpret ECG data, generating interpretive statements for its users. As we approach the eventual integration of advanced AI-ECG models into CEI, it will become increasingly important to evaluate the effectiveness of these novel algorithms relative to their rule-based predecessors, and to understand their potential influence on clinical decisionmaking processes.
Study Strengths and Limitations
Our study has several key strengths that enhance the reliability and significance of our findings. Firstly, the direct comparison of ECG interpretation performance with and without the CEI report allowed us to evaluate the specific impact of the CEI report on accuracy, efficiency, and confidence. Secondly, we examined a large and diverse sample population of medical professionals who regularly perform ECG interpretation in clinical practice. Thirdly, we utilized a wide selection of different ECGs, incorporating commonly taught and encountered urgent and non-urgent findings. Lastly, the use of a virtual testing environment provided convenience, accessibility, and flexibility, enabling a broader participant pool by eliminating the need for physical travel and testing schedule constraints.
However, certain limitations should be considered. The controlled testing environment, which does not replicate the complexities of real-world clinical practice, and the exclusion of practicing cardiology and emergency medicine physicians may limit the generalizability of our findings. Additionally, our study design, involving sequential ECG assessments rather than a randomized controlled trial, may introduce potential confounding factors that could inflate the observed positive impact of the CEI. Lastly, our use of a small selection of commonly taught urgent and non-urgent findings does not fully capture the breadth or complexity of ECGs that may be encountered in clinical practice.
Future Directions
Future research should focus on conducting comprehensive evaluations of the effectiveness of current CEI programs, considering their limitations, biases, and impact on interpretation. It should specifically aim to identify the specific elements of CEI reports that have the greatest positive and negative influences on interpreter performance. Additionally, it will be crucial to investigate the effects of incorporating external clinical variables and predictive analytics, including novel AI-ECG models, into CEI reports on interpreter performance. Lastly, exploring the intricate interplay between training, experience, and other cognitive factors may assist medical educators in designing educational initiatives that effectively enhance ECG interpretation proficiency.
CONCLUSION
CEI integration improves ECG interpretation accuracy, efficiency, and confidence among healthcare professionals.
Supplementary Material
Supplemental Table 1. Participant performance.
1 Physicians include primary care physicians, cardiology fellows-in-training, and physician residents, whereas non-physicians include advanced practice providers, nurses, and allied health professionals.
Abbreviations: CEI, computer ECG interpretation.
Support:
The study was supported by GE Healthcare (Milwaukee, WI) and NIH T32 HL007111. The EKG Guy learning and research management software platform was utilized to conduct, collect, and export data for analysis.
Disclosures:
A.H.K. is the founder and CEO of The EKG Guy, and has received research funding from GE Healthcare (Milwaukee, WI).
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Declaration of interests
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
REFERENCES
- 1.Taback L, Marden E, Mason HL, et al. Digital recording of electrocardiographic data for analysis by a digital computer. IRE Trans Med Electro 6:167–171, 1959 [Google Scholar]
- 2.Pipberger HV, Freis ED, Taback L, et al. Preparation of electrocardiographic data for analysis by digital electronic computer. Circulation 21:413–418, 1960 [DOI] [PubMed] [Google Scholar]
- 3.Rautaharju PM. Eyewitness to history: landmarks in the development of computerized electrocardiography. J Electrocardiol 49:1–6, 2016 [DOI] [PubMed] [Google Scholar]
- 4.Estes NA 3rd. Computerized interpretation of ECGs: supplement not a substitute. Circ Arrhythm Electrophysiol 6(1):2–4, 2013 [DOI] [PubMed] [Google Scholar]
- 5.Kashou AH, Mulpuru SK, Deshmukh AJ, et al. An artificial intelligence-enabled ECG algorithm for comprehensive ECG interpretation: Can it pass the ‘Turing test’? Cardiovasc Digit Health J 2(3):164–170, 2021 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Schlapfer J, Wellens HJ. Computer-Interpreted Electrocardiograms: Benefits and Limitations. J Am Coll Cardiol 70(9):1183–1192, 2017 [DOI] [PubMed] [Google Scholar]
- 7.Hongo RH, Goldschlager N. Status of computerized electrocardiography. Cardiol Clin 24:491–504, 2006 [DOI] [PubMed] [Google Scholar]
- 8.Gregg RE, Deluca DC, Chien CH, et al. Automated serial ECG comparison improves computerized interpretation of 12-lead ECG. J Electrocardiol 45:561–565, 2012 [DOI] [PubMed] [Google Scholar]
- 9.Willems JL, Abreu-Lima C, Arnaud P, et al. The diagnostic performance of computer programs for the interpretation of electrocardiograms. N Engl J Med 325(25):1767–1673, 1991 [DOI] [PubMed] [Google Scholar]
- 10.Tsai TL, Fridsma DB, Gatti G. Computer decision support as a source of interpretation error: the case of electrocardiograms. J Am Med Inform Assoc 10:478–483, 2003 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Guglin ME, Thatai D. Common errors in computer electrocardiogram interpretation. Int J Cardiol 106:232–237, 2006 [DOI] [PubMed] [Google Scholar]
- 12.Shah AP, Rubin SA. Errors in computerized electrocardiogram interpretation of cardiac rhythm. J Electrocardiol 40:385–390, 2007 [DOI] [PubMed] [Google Scholar]
- 13.Taggar JS, Coleman T, Lewis S, Heneghan C, Jones M. Accuracy of methods for diagnosing atrial fibrillation using 12-lead ECG: a systematic review and meta-analysis. Int J Cardiol 184:175–183, 2015 [DOI] [PubMed] [Google Scholar]
- 14.Bogun F, Anh D, Kalahasty G, et al. Misdiagnosis of atrial fibrillation and its clinical consequences. Am J Med 117:636–642, 2004 [DOI] [PubMed] [Google Scholar]
- 15.Hwan Bae M, Hoon Lee J, Heon Yang D, et al. Erroneous computer electrocardiogram interpretation of atrial fibrillation and its clinical consequences. Clin Cardiol 35:348–353, 2012 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Poon K, Okin PM, Kligfield P. Diagnostic performance of a computer-based ECG rhythm algorithm. J Electrocardiol 38:235–238, 2005 [DOI] [PubMed] [Google Scholar]
- 17.Garvey JL, Zegre-Hemsey J, Gregg R, et al. Electrocardiographic diagnosis of ST segment elevation myocardial infarction: an evaluation of three automated interpretation algorithms. J Electrocardiol 49:728–732, 2016 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Garg A, Lehmann MH. Prolonged QT interval diagnosis suppression by a widely used computerized ECG analysis system. Circ Arrhythm Electrophysiol 6:76–83, 2013 [DOI] [PubMed] [Google Scholar]
- 19.Brailer DJ, Kroch E, Pauly MV. The impact of computer-assisted test interpretation on physician decision making: the case of electrocardiograms. Med Decis Making 17(1):80–86, 1997 [DOI] [PubMed] [Google Scholar]
- 20.Landelius J, Nordgren L. Evaluation of a computer programme for interpretation of 12-lead electrocardiograms. Ups J Med Sci 84:37–46, 1979 [DOI] [PubMed] [Google Scholar]
- 21.Shirataka M, Miyahara H, Ikeda N, et al. Evaluation of five computer programs in the diagnosis of second-degree AV block. J Electrocardiol 25(3):185–195, 1992 [DOI] [PubMed] [Google Scholar]
- 22.Caceres CA, Hochberg HM. Performance of the computer and physician in the analysis of the electrocardiogram. Am Heart J 79:439–443, 1970 [DOI] [PubMed] [Google Scholar]
- 23.Swartz MH, Teichholz LE. The Marquette 12SL ECG analysis program: evaluation of physician changes. In: Computers in cardiology. IEEE Computer Society 437–440, 1982 [Google Scholar]
- 24.Carel RS. Drori Y. Sources of common computer cardiologist discord in ECG interpretations in an ambulatory population. Med Bioi Eng Comput 20:264–268, 1982 [DOI] [PubMed] [Google Scholar]
- 25.Caceres CA. Electrocardiographic coding: an overview. Eur J Cardiol 8:145–153, 1978 [PubMed] [Google Scholar]
- 26.Milliken JA. Pipberger H, Pipberger HV. et al. The impact of an ECG computer analysis program on the cardiologist’s interpretation. A cooperative study. J Electrocardiol 16:141–149, 1983 [DOI] [PubMed] [Google Scholar]
- 27.Grauer K, Kravitz L, Ariet M, et al. Potential benefits of a computer ECG interpretation system for primary care physicians in a community hospital. J Am Board Fam Pract 2(1):17–24, 1989 [PubMed] [Google Scholar]
- 28.Southern WN, Arnsten JH. The effect of erroneous computer interpretation of ECGs on resident decision making. Med Decis Making 29:372–376, 2009 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Kashou AH, Noseworthy PA, Beckman TJ, et al. Education curriculum assessment for teaching electrocardiography: Rationale and design for the prospective, international, randomized controlled EDUCATE trial. J Electrocardiol In Press, 2023 [DOI] [PubMed] [Google Scholar]
- 30.Kashou AH, Noseworthy PA, Beckman TJ, et al. ECG Interpretation Proficiency of Healthcare Professionals. Curr Probl Cardiol 48(10):101924, 2023 [DOI] [PubMed] [Google Scholar]
- 31.Kashou AH, Noseworthy PA, Beckman TJ, et al. Exploring Factors Influencing ECG Interpretation Proficiency of Medical Professionals. Curr Probl Cardiol 13;48(10):101865, 2023 [DOI] [PubMed] [Google Scholar]
- 32.Kashou AH, May AM, Noseworthy PA. Artificial Intelligence-Enabled ECG: a Modern Lens on an Old Technology. Curr Cardiol Rep 22(8):57, 2020 [DOI] [PubMed] [Google Scholar]
- 33.Siontis KC, Noseworthy PA, Attia ZI, et al. Artificial intelligence-enhanced electrocardiography in cardiovascular disease management. Nat Rev Cardiol 18, 465–478, 2021 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Supplemental Table 1. Participant performance.
1 Physicians include primary care physicians, cardiology fellows-in-training, and physician residents, whereas non-physicians include advanced practice providers, nurses, and allied health professionals.
Abbreviations: CEI, computer ECG interpretation.