Electrocardiographic diagnosis of ST segment elevation myocardial infarction: An evaluation of three automated interpretation algorithms

Abstract

Objective

To assess the validity of three different computerized electrocardiogram (ECG) interpretation algorithms in correctly identifying STEMI patients in the prehospital environment who require emergent cardiac intervention.

Methods

This retrospective study validated three diagnostic algorithms (AG) against the presence of a culprit coronary artery upon cardiac catheterization. Two patient groups were enrolled in this study: those with verified prehospital ST-elevation myocardial infarction (STEMI) activation (cases) and those with a prehospital impression of chest pain due to ACS (controls).

Results

There were 500 records analyzed resulting in a case group with 151 patients and a control group with 349 patients. Sensitivities differed between AGs (AG1 = 0.69 vs AG2 = 0.68 vs AG3 = 0.62), with statistical differences in sensitivity found when comparing AG1 to AG3 and AG1 to AG2. Specificities also differed between AGs (AG1 = 0.89 vs AG2 = 0.91 vs AG3 = 0.95), with AG1 and AG2 significantly less specific than AG3.

Conclusions

STEMI diagnostic algorithms vary in regards to their validity in identifying patients with culprit artery lesions. This suggests that systems could apply more sensitive or specific algorithms depending on the needs in their community.

Introduction

Electrocardiography (ECG) remains the key element in establishing the diagnosis of acute ST segment elevation myocardial infarction (STEMI). 12-lead electrocardiogram (12-lead ECG) diagnosis of myocardial injury with ST segment elevation identifies a group of patients that require coronary intervention in a time-critical manner. Current guidelines promote the use of ECGs very early in the evaluation process of patients with symptoms that may represent acute cardiac ischemia and injury. Symptoms triggering ECG acquisition include chest pain, shortness of breath, diaphoresis, and other anginal equivalents [1]. ECG is incorporated into emergency department triage algorithms and is also used by paramedics in the evaluation of patients with chest pain. ST-elevation myocardial infarction (STEMI) patients are thus identified in the prehospital environment and transported by emergency medical services (EMS) directly to hospitals that are capable of establishing myocardial reperfusion by performing catheter-based primary coronary intervention (PCI) or surgical revascularization [2].

Computerized diagnostic algorithms have been developed to assist with the interpretation of ECGs. These algorithms are particularly useful when used in the prehospital environment where paramedics may have less experience in interpreting ECGs than a typical emergency physician or cardiologist. Computerized ECG measurements are instantaneous, eliminate human bias, and can precisely measure waveforms to a resolution of 10 µV, significantly more than the human eye [3,4].Moreover, when the diagnostic interpretation algorithm suggests STEMI, paramedics execute specific actions to optimize care for their patients. These actions may include administration of specific medicines, activation of cardiac catheterization laboratory from the field, and subsequent routing of patients directly to PCI-capable STEMI receiving centers. As with any diagnostic test, a balance exists between the sensitivity and specificity of the 12-lead ECG in establishing the diagnosis of STEMI. To date, prehospital computer algorithm interpretation is not very sensitive for STEMI identification though previous studies are limited. Currently, the standard ECG has limited sensitivity (30–70%) and specificity (70–100%) [5–9]. One goal for screening ECGs in the prehospital setting is to capture all potential STEMI patients (high sensitivity) so they may be appropriately treated in a timely manner. However, over-utilization of resources may result if the interpretation algorithms have lower specificity and ‘over call’ STEMI. Inappropriate cardiac catheterization activations may result in patients receiving invasive procedures who do not need them, distrust of EMS providers for clinical decision-making, and unnecessary costs [10–12].Improved diagnostic methods and algorithms for prehospital electrocardiography are needed.

The objective of this study was to assess the validity, in the prehospital environment, of three different computerized ECG interpretation algorithms with respect to the algorithms’ ability to correctly identify STEMI patients, a highly vulnerable population at highest risk for cardiac death and complications if they do not receive emergent cardiac intervention.

Methods

This was a retrospective validation study using patients who had been previously enrolled in a local STEMI registry or who had been transported to one of three emergency departments with a prehospital impression of cardiac chest pain related to myocardial ischemia/infarction. Patients were included if they were transported by the local EMS system between 1/1/12 and 12/31/13. This study received institutional review board approval from Carolinas Medical Center (Charlottte, NC).

Study population

The regional STEMI system of care which produced the patients for this study includes three hospitals with capability of providing 24-h cardiac catheterization facilities. Patients were enrolled in this study if they were transported by EMS to the regional academic hospital which was one of the three available 24-h cardiac catheterization facilities. The county under study was served by a single all-advanced life support (ALS) municipal EMS Agency. These entities have worked cooperatively in establishing a system of care for STEMI patients that has been in place since 2005. The institutions have successfully participated in the North Carolina state-wide system of care for STEMI patients, the Regional Approach to Cardiovascular Emergencies (RACE) initiative.

There were two patient groups enrolled in this study, those with verified prehospital STEMI activation and those with a prehospital impression of chest pain potentially due to ACS. STEMI patients were abstracted from the regional STEMI database of which the activation criteria were as follows: patient with a chief complaint consistent with the signs and symptoms of ACS, a 12-lead ECG with a computerized diagnostic statement of ACUTE MI, and a paramedic over-read which identifies > 1 mm of ST segment elevation in contiguous leads. STEMI activation patients may have any of the following outcomes: PCI, catheterization with no intervention, surgery, medical management, or canceled STEMI activation. The second group of patients was defined as those having a prehospital impression of chest pain potentially due to ACS, at least one 12-lead ECG performed, and no prehospital STEMI system activation. Patients were not excluded based on ECG diagnostic statements or hospital course of care.

The main outcome variable was documentation of a culprit coronary artery upon catheterization. Culprit coronary arteries were identified by review of the cardiac catheterization report made by the cardiologist performing the procedure. If not specifically mentioned within the report, culprit coronary arteries were identified as the stenosed coronary artery responsible for local ischemia seen on catheterization. These data were obtained from the regional STEMI registry and were classified dichotomously as culprit artery, either yes or no. Secondary outcomes included the occurrence of catheterization and whether or not the patient met STEMI activation criteria.

The main independent variables of interest were diagnostic algorithms. There were three proprietary diagnostic algorithms assessed in this study labeled as algorithm 1 (AG1), algorithm 2 (AG2), and algorithm 3 (AG3). Upon identification of a cohort of eligible patients, study investigators extracted the first 12-lead ECG acquired on each patient and de-identified that electronic file. If the first ECG was determined by the study investigators to be of poor quality; incomplete or excessive artifact, subsequent ECGs were reviewed and the best quality tracing was selected. A data file with only the de-identified electronic ECGs was sent to the industry partner to be run through all three algorithms. Study investigators were blinded to the properties of all algorithms. Each algorithm returned an assessment of whether the 12 lead met prespecified criteria for an ACUTE MI. After processing, diagnostic statements were matched back to their patient of origin to assess outcome and control for other independent variables. Patient baseline characteristics including age, gender, and race were also collected.

Data analysis

We selected 149 consecutive STEMI activations (cases) and a random sample of 351 patients with a prehospital impression of chest pain potentially due to ACS and no STEMI system activation (controls). Sample size selection was determined based on the number of cases available for review in the STEMI registry balanced with the desire to have narrow confidence intervals around resulting sensitivity and specificity estimates. Initial descriptive analyses were performed using means and proportions where appropriate. Primary analysis assessed the validity of each algorithm by examining the sensitivity, specificity, positive and negative likelihood ratios to detect a patient with a culprit coronary artery, the need for catheterization or meeting the STEMI activation criteria. Algorithms were then compared to each other using McNemar’s chi square to assess for algorithm differences in sensitivity and specificity. To adjust for multiple testing, alpha was set at 0.008. Logistic regression was performed with each algorithm as an independent variable, as well as age and gender, to assess area under the receiver operating characteristic (A-ROC) curve. Likelihood ratio tests were compared between the three main models of types of algorithms when controlling for demographics of the study population. Secondary analysis on the two remaining outcome variables repeated the modeling for the main outcome variable.

Results

There were 500 records analyzed and Table 1 displays basic demographic and clinical characteristics of this study population. The target recruitment levels for cases and controls were not met due to misclassification during initial coding, resulting in a case group with 151 patients and a control group with 349 patients. There were 273 (54.6%) male patients and the average age of the study population was 59.7 years (95% CI 583.−61.1). Males were overrepresented in the cases as compared to the controls. 163 (32.8%) patients received cardiac catheterization during the course of their hospital stay and of these, 145 (29.1%) patients had an identified culprit artery. As expected, the frequency of catheterization and culprit artery identification was higher in the case group as compared to controls.

Table 1.

Basic demographics and clinical characteristics.

	Total population n (col%)	Prehospital STEMI activation n (row%)	Not a prehospital STEMI activation n (row%)
Culprit artery
Yes	145 (29.0)	118 (81.4)	27 (18.6)
No	355 (71.0)	33 (9.0)	322 (91.0)
LAD	43 (29.7)	34 (79.1)	9 (20.9)
RCA	37 (25.5)	32 (86.5)	5 (13.5)
Lt-circumflex	9 (6.2)	8 (88.9)	1 (11.1)
Lt-main	0	-	-
Multi-vessel	56 (38.6)	44 (78.6)	12 (21.4)
Catheterization
Yes	163 (32.6)	124 (76.1)	39 (23.9)
No	337 (67.4)	27 (8.0)	310 (92.0)
STEMI activation
Yes	151 (30.2)	—	—
No	349 (69.8)	—	—
Gender
Male	273 (54.6)	104 (38.1)	169 (61.9)
Female	227 (45.4)	47 (20.7)	180 (79.3)
Age (95% CI)	59.7 (58.3–61.1)	60.2 (57.8–62.5)	59.5 (57.7–61.3)

STEMI = ST-elevation myocardial infarction.

LAD = left anterior descending.

RCA = right coronary artery.

Multi-vessel = more than one coronary artery with occlusion and infarct recurring PCI.

Table 2 displays the results of the validation analysis. Through all analyses, AG3 consistently had the lowest estimated sensitivity and the highest specificity. When assessing the ability to identify a patient with a culprit artery, the range of sensitivities among the three algorithms was 0.69–0.62 while the range of specificities was 0.89–0.95. Statistical differences in sensitivity were found when comparing AG1 to AG3 (0.69 vs 0.62, p = 0.0039) and AG2 to AG3 (0.68 vs 0.62, p = 0.0027). Similar patterns were found when assessing differences in specificity, with AG1 and AG2 significantly less specific than AG3 (0.89 vs 0.95, p = 0.0002 and 0.91 vs 0.95, p = 0.0002).

Table 2.

Validation of three different algorithms.

	AG1	AG2	AG3
Culprit artery
Sensitivity	0.69 (0.61–0.76)	0.68 (0.60–0.76)	0.62 (0.54–0.70)
Specificity	0.89 (0.85–0.92)	0.91 (0.87–0.94)	0.95 (0.92–0.97)
LR+	6.42 (4.67–8.84)	7.55 (5.33–10.70)	12.21 (7.65–19.49)
LR−	0.35 (0.27–0.44)	0.35 (0.27–0.44)	0.40 (0.32–0.49)
A-ROC	0.79 (0.75–0.83)	0.79 (0.75–0.84)	0.78 (0.74–0.83)
Catheterization
Sensitivity	0.66 (0.58–0.73)	0.65 (0.57–0.72)	0.57 (0.49–0.64)
Specificity	0.91 (0.87–0.93)	0.93 (0.89–0.95)	0.95 (0.93–0.97)
LR+	7.11 (5.00–10.13)	8.74 (5.90–12.95)	12.78 (7.66–21.33)
LR−	0.38 (0.35–0.47)	0.38 (0.35–0.47)	0.45 (0.38–0.54)
A-ROC	0.78 (0.74–0.82)	0.79 (0.75–0.83)	0.76 (0.72–0.80)
STEMI activation
Sensitivity	0.79 (0.72–0.85)	0.79 (0.71–0.85)	0.67 (0.59–0.74)
Specificity	0.95 (0.91–0.97)	0.97 (0.94–0.98)	0.98 (0.96–0.99)
LR+	14.60 (9.36–22.77)	22.92 (13.06–40.21)	33.35 (15.88–70.03)
LR−	0.22 (0.16–0.30)	0.22 (0.16–0.30)	0.34 (0.27–0.42)
A-ROC	0.87 (0.84–0.90)	0.88 (0.84–0.91)	0.82 (0.79–0.86)

Values in parenthesis are 95% CI.

LR = likelihood ratio.

A-ROC = area under the receiver operating characteristic curve.

When assessing the probability of the presence of a culprit artery (LR+), the range of likelihoods was 6.42–12.21 with confidence intervals overlapping for all three algorithms. This range of likelihoods indicates a moderate to large increase in the likelihood of disease, if a 12-lead ECG was STEMI positive. The ranges of likelihoods for the probability of no culprit artery being present (LR−) were 0.35–0.40, again with overlapping confidence intervals. This range of likelihoods indicates a small decrease in the likelihood of disease, if a 12-lead ECG was STEMI negative.

The overall ability to discriminate between individuals with a culprit artery and those without was fair, with an area under the ROC curve between 0.78 and 0.79 for the three algorithms. The ability for the algorithms to discriminate improved to good when assessing the outcome variable STEMI activation. The ranges of A-ROC values were 0.82–0.87.

Discussion

Accurate ECG interpretation is critical to rapid clinical decision-making that directly impacts patient outcomes. Computerized algorithms have been developed to support clinical decision-making in emergency cardiac care settings, where time is of the essence. This study was novel as it is the first known to compare three computerized algorithms for diagnostic accuracy and clinical outcomes. The primary outcome was culprit artery and the secondary outcomes were STEMI criteria activation and occurrence of cardiac catheterization.

Kudenchuck et al. (1998) conducted a study that compared computerized algorithm results to an expert electrocardiographer in determining STEMI [13]. Prehospital and hospital records were abstracted and the documented final diagnosis was the primary outcome of interest to determine patients with acute infarction. Of 391 patients with evidence for acute myocardial infarction, only 202 (52%) were identified by computerized algorithm compared with 259 (66%) by expert electrocardiographer (p < .001).

A more recent study was done by Bhalla et al. (2013). In this, investigators conducted a retrospective cross-sectional study to determine the sensitivity and specificity of prehospital ECGs for STEMI identification [14]. The primary outcomes of interest were ED physician’s ECG interpretation of STEMI and cardiac catheterization laboratory activation. The rationale for these outcomes was to determine whether computerized interpretation could be used to make the same decision as the ED physician in terms of STEMI criteria activation.

Given the context of these prior studies, our work from a clinical perspective changes in diagnostic algorithm specificity may be more clinically important than the smaller changes in sensitivity that we observed. If specificity can be raised to 95%, systems that consider fibrinolytic therapy as their reperfusion therapy of choice may have greater confidence in the algorithms’ ability to identify patients for this therapy. The difference in observed sensitivity among the three algorithms, ranged 0.62–0.69 has implications for systems directing patients to bypass closer hospital and deliver them to PCI centers. While the differences in range were similar for sensitivity and specificity in identifying culprit artery, meeting a threshold of 95% specific adds to the confidence in using the diagnostic statement to help direct clinical care.

Future research should focus on strategies that improve computerized algorithm accuracy for STEMI diagnoses. These may include incorporation of gender and age indices in algorithms, as indicated in the most recent guidelines for defining acute myocardial infarction [9]. Electrocardiographic changes of ischemia, along with gender and age, improve the sensitivity and specificity of ECG for diagnosis of acute myocardial infarction [15].

Another strategy for improving computerized STEMI detection would be incorporating prior ECG findings into computerized algorithms since serial ECGs can significantly increase the sensitivity for a STEMI diagnosis [6,16,17]. Prior studies show that serial or continuous ECGs improve the diagnostic accuracy for STEMI since a single static “snap-shot” 12-lead ECG may miss dynamic ischemic changes. Current guidelines recommend comparison of an acute ECG with a previously acquired ECG to ensure ECG ischemic changes are new, but this is not a common practice in the prehospital environment [1,9]. We recently examined the benefit of augmenting prehospital ECG findings with that of the initial hospital ECG and found a significant increase in sensitivity (79.9%) and decrease in specificity (61.2%) when ambulance ECGs were considered in conjunction with the initial ECG acquired in the hospital (p < 0.05) [16]. Prior ECG could also help distinguish new changes of ischemia from chronic conditions such as left bundle branch block or left ventricular hypertrophy which mimic STEMI. This represents a logistical challenge, but with the advent of electronic health records and Cloud technology may be more feasible than before.

Limitations

This was a retrospective review of a previously existing quality improvement data set in a single county and there are several limitations to this study. Consecutive patients were selected into the study based on date to ensure an adequate sample. Lack of randomization may have led to a male predominance in the case cohort and potential selection bias; however, there is no evidence of such bias in the information presented. Further, the gold standard used in this validation was the presence of a culprit artery upon cardiac catheterization. While quantitatively an appropriate standard from a qualitative perspective, the presence or absence of a culprit artery may not be the best standard to base a prehospital STEMI system on. Also, we did not follow up with patients who did not go to the catheterization lab to determine if there was a missed STEMI. This is clearly a limitation of the retrospective nature of this study. We attempted to address this limitation by analyzing both STEMI activation and catheterization only. Finally, the results from this analysis may not be generalized to other systems. The system that produced the cases and controls used in this analysis is part of a regional STEMI network. Identification of STEMI patients may vary in other systems that have differing levels of prehospital clinical care.

Conclusions

This study assessed the validity of three different computerized ECG interpretation algorithms. Each algorithm had differing test characteristics when assessed against the presence of a culprit artery. This demonstrates that computerized ECG algorithms vary and may be designed to optimize sensitivity or specificity. Individual applications may vary, and local needs should be considered because they could favor use of a higher sensitivity algorithm vs a higher specificity algorithm.