Added Value of a Resting ECG Neural Network That Predicts Cardiovascular Mortality

Abstract

Background: The resting 12‐lead electrocardiogram (ECG) remains the most commonly used test in evaluating patients with suspected cardiovascular disease. Prognostic values of individual findings on the ECG have been reported but may be of limited use.

Methods: The characteristics of 45,855 ECGs ordered by physician's discretion were first recorded and analyzed using a computerized system. Ninety percent of these ECGs were used to train an artifical neural network (ANN) to predict cardiovascular mortality (CVM) based on 132 ECG and four demographic characteristics. The ANN generated a Resting ECG Neural Network (RENN) score that was then tested in the remaining ECGs. The RENN score was finally assessed in a cohort of 2189 patients who underwent exercise treadmill testing and were followed for CVM.

Results: The RENN score was able to better predict CVM compared to individual ECG markers or a traditional Cox regression model in the testing cohort. Over a mean of 8.6 years, there were 156 cardiovascular deaths in the treadmill cohort. Among the patients who were classified as intermediate risk by Duke Treadmill Scoring (DTS), the third tertile of the RENN score demonstrated an adjusted Cox hazard ratio of 5.4 (95% CI 2.0–15.2) compared to the first RENN tertile. The 10‐year CVM was 2.8%, 8.6% and 22% in the first, second and third RENN tertiles, respectively.

Conclusions: An ANN that uses the resting ECG and demographic variables to predict CVM was created. The RENN score can further risk stratify patients deemed at moderate risk on exercise treadmill testing.

Cardiovascular disease remains the most common cause of death in the United States. ¹ , ² , ³ The ability to predict cardiovascular mortality (CVM) helps risk stratify and guide the management of patients suspected of having cardiovascular disease. ⁴ Several noninvasive predictors of CVM, such as stress imaging, coronary calcium measurements ⁵ and C‐reactive protein levels ⁶ are available but can be costly and may not be immediately accessible.

The resting electrocardiogram (ECG) has long been used in the management of patients with cardiovascular disease. Although individual characteristics of the ECG have been shown to be predictive of CVM, ⁷ , ⁸ , ⁹ a method to encompass all of the available information on an ECG to reliably predict mortality has not yet been devised.

Artificial neural networks (ANN) employ algorithms that are modeled after the biologic system of neuronal connections. ¹⁰ , ¹¹ The basic element in an ANN is the artificial node which contains various weighted inputs. Multiple layers of these nodes are connected mathematically to create an ANN which is then trained to predict an outcome based on sample cases.

Predictive ANN have been successfully created in several medical fields. ¹² , ¹³ , ¹⁴ , ¹⁵ , ¹⁶ One of the first ANN that used ECG data combined with clinical information predicted acute myocardial infarctions (AMI) in the emergency room as accurately as physicians. ¹⁷ ANN have also successfully been used to accurately classify and distinguish ventricular hypertrophy and AMI. ¹⁸

Although several ANN have since been trained to aid in the diagnosis of AMI, ¹⁹ , ²⁰ , ²¹ , ²² , ²³ none that use solely ECG characteristics to predict CVM have yet been reported. Currently available computing power has recently made it possible to create large ANNs that can train using the thousands of examples needed to make reliable outcome predictions. We hypothesized that an ANN using multiple variables from the resting ECG and four demographic factors could be trained with a large ECG dataset to reliably predict CVM and that this ANN would demonstrate added prognostic value above traditional exercise stress testing.

METHODS

Study Population

The creation and initial testing of the ANN involved a retrospective analysis of 45,855 ECGs obtained between 1987 and 2000 that were ordered for usual clinical indications at the Palo Alto (Calif) Veterans Affairs Medical Center. Added value of the neural network was determined in a cohort of 2, 189 patients who were referred for clinical treadmill testing during the same time period. Information on coronary risk factors, symptoms, medications and prior cardiac events were collected before exercise testing. The subjects gave written informed consent and the study was approved by Stanford University Institutional Review Board.

ECG Analysis

An ECG system (GE Marquette) validated by the United States Food and Drug Administration was used to collect, store and analyze the resting ECGs. The recorded data included the timing and voltages at each point of the PQRST complex on the basic eight leads with derivation of the remaining four leads. The system was able to flag rhythm abnormalities, measure standard intervals and perform waveform analysis to provide the basic electrocardiographic interpretations (GE 12 SL analysis program, http://www.gehealthcare.com). Standardized computerized ECG criteria as described by the GE 12‐lead electrographic analysis program were used for the diagnosis of Q waves, ST changes, and bundle branch blocks. From these measurements, the Romhilt‐Estes criteria for left ventricular hypertrophy (LVH), ²⁴ cardiac infarction injury scores (CIIS) ²⁵ and Selvester ²⁶ scores were calculated.

Exercise Testing

Subjects underwent symptom‐limited treadmill testing using an individualized ramp treadmill protocol and exercised to maximum exertion. ²⁷ Twelve‐lead ECG data were recorded at 500 samples per second during exercise test. Visual ST‐segment depression and ST slopes were measured and read by the senior authors (V.F., J.M.). The Duke Treadmill Score (DTS) ²⁸ was calculated by converting METs estimated from the individualized ramp protocol to minutes on the Bruce protocol.

Outcome Variables

The primary outcome evaluated for this study was CVM. CVM data was gathered from the Social Security Death Index and California Death Registry, and the cause of death was determined from the registry classification and confirmed by Veterans Affairs medical records. Death status was determined as of March 2006. CVM was defined as death from a clearly identifiable CV cause or death of subjects with a history of CV disease and no identifiable non‐CV cause for death. Classification was made with the observers blinded to the test results and resolved by consensus of two observers; conflicts were resolved by the senior author (V.F.).

Neural Network Creation and Training

Network creation and training was performed using commercially available software (Brainmaker Pro v3.75, California Scientific, http://www.calsci.com). An ANN was created with four demographic input variables (age, sex, race and Body Mass Index (BMI)) as well as 132 computer‐generated ECG input variables consisting of quantitative characteristics (rate, intervals, ECG wave durations and amplitudes, P, QRS and T axis), qualitative characteristics (rhythm, presence of abnormal Q‐waves, pacing) and scores (Romhilt‐Estes, CIIS, Selvester) from the 12 standard leads (Table 1). The ANN was formed with one layer of 70 hidden nodes and a single output node representing CVM.

Table 1.

Summary of the 136 Input Variables Used for Creation of the Artificial Neural Network

Variable Name	Number	Variable Type
Age, BMI	2	Continuous
Other demographics (gender, race)	2	Categorical
Heart rate (ventricular, atrial)	2	Continuous
Rhythm determinants (sinus, AF)	2	Binary
Presence of PVC	1	Binary
Evidence of pacing	1	Binary
Left or right QRS axis deviation	2	Binary
Abnormal P‐wave axis	1	Binary
Major intervals (PR, QT, QTc)	3	Continuous
Left or right bundle branch blocks, IVCD	3	Binary
P‐wave duration 12 leads	12	Continuous
Q‐wave duration 12 leads	12	Continuous
QRS duration 12 leads	12	Continuous
R‐wave duration 12 leads	12	Continuous
Hypertrophy (LV, RV, LA)	3	Binary
LVH criteria (Minnesota, Romhilt‐Estes, Sokolow‐Lyon)	3	Categorical
ST depression	1	Binary
Q‐wave amplitude 12 leads	12	Continuous
QRS amplitude 12 leads	12	Continuous
T‐wave amplitude 12 leads	12	Continuous
Acute myocardial infarction	1	Binary
ST elevation (anterior, inferior, lateral, II, V2, V5, any)	7	Binary
Presence of Q waves (anterior, septal, lateral, inferior, inferoposterior, any)	5	Binary
Infarct scores (CIIS, Selvester)	2	Continuous
J‐point elevation II, V2, V5, aVR	4	Continuous
ST slope	1	Continuous
Ventricular activation time V5, V6	2	Continuous
QT and QTc dispersion (all leads, precordial)	4	Continuous

BMI = body mass index; AF = atrial fibrillation; PVC = premature ventricular contraction; IVCD = intraventricular conduction delay; LV = left ventricular; RV = right ventricular; LA = left atrial; LVH = left ventricular hypertrophy; CIIS = cardiac injury infarct score.

Of the 45,855 ECGs available, 4586 (10%) randomly selected ECGs were set aside for testing and were not used in training (Fig. 1). From the remaining 41,269 ECGs, all 4561 ECGs associated with a cardiovascular death and an equal number of randomly selected ECGs not associated with cardiovascular death were used for training. It was decided not to exclude patients with LBBB or paced rhythms because these characteristics themselves may contain prognostic value that can be accounted for by neural networks. The network was presented with the training ECGs and, if CVM prediction was incorrect, a back‐propagation algorithm was used to readjust the randomly assigned input weights. A constant training tolerance of 10% and a constant learning rate of 1.0 were used. Each training ECG was presented to the network over 15,000 times and training was stopped once the area under the ROC curve predicting CVM was maximized.

Figure 1.

Method for selecting the training and testing sets of ECGs for neural network creation and validation.

Statistical Analysis in the Neural Network Testing Group

After the completion of training, the four demographic variables and the 132 variables derived from the ECGs of the testing set were presented to the network. The ANN produced an output between 0 and 1 for each ECG which we will term the Resting ECG Neural Network (RENN) score.

Analysis of Variance (ANOVA) and chi‐squared tests were used to compare the differences in continuous and categorical values, respectively, between the different RENN tertiles. The prognostic significance of the RENN score was subsequently evaluated using univariate and multivariable Cox proportional hazards analysis. Discriminative accuracy was assessed using the right‐censored concordance index (C index).

Next, a Cox hazard model containing the four demographic factors and 11 highly predictive ECG variables (ventricular rate, QTc, QRS duration, left axis deviation, atrial fibrillation, presence of Q‐waves, T‐wave amplitude in aVR, ST segment deviation, ST segment slope, CIIS and Romhilt LVH score) chosen by the investigators was created using the training set. Although attempt at stepwise regression was made, the sheer size of the dataset and the number of variables being studied made this task computationally prohibitive. After multiple efforts at forward and backward stepwise regression with various statistical packages, the models failed to converge. A Cox score based on the model with the 11 originally chosen variables was subsequently calculated for each ECG in the testing set. To compare the predictive power of the RENN score to the QRS duration, Romhilt LVH criteria, CIIS and Cox scores, each measurement was divided into tertiles. The second and third tertiles of each group were compared to the first respective tertile by calculating unadjusted hazard ratios. In addition, the Cox model and the neural network model were compared with penalized chi‐square measurements.

To demonstrate the added value of the RENN score to the Cox model, Cox scores for each of the ECGs in the testing set were calculated. The parameters for the model are based on the Cox hazard regression model created using the training set. The Cox model scores for the testing ECG set were computed by summing the intercept of the Cox model plus each variable multiplied by its corresponding parameter. A matrix of Cox score tertiles by RENN score tertiles, resulting in a total of nine subcategories, was constructed. A hazard ratio for CVM was calculated for each of the nine resulting subcategories. No adjustment was made for the demographic variables at this point because these variables were included as part of both the Cox and RENN models.

Statistical Analysis in the Exercise Treadmill Testing Cohort

The relationship between tertiles of the RENN Score and CVM in the patients who underwent exercise treadmill testing was evaluated according to the Kaplan‐Meier method. Multivariate Cox proportional analyses adjusted for age, sex, hyperlpidemia, diabetes, smoking status and hypertension were used to evaluate associations between RENN tertiles and CVM. Similar analyses were performed in the subgroup of patients who were deemed at intermediate risk by DTS.

SAS version 9.1 as well as the “Design” and “Hmisc” libraries in S‐Plus version 7.0 (Insightful Corp, Seattle, WA) were used for statistical analysis. The proportional hazards assumption was evaluated by the scaled Schoenfeld residual. No violations were observed. The authors had full access to the data and take responsibility for its integrity. All authors have read and agree to the manuscript as written.

RESULTS

Analysis of RENN Score in the Test Group

Over a mean of 7.5 years, there were a total of 449 cardiovascular deaths in the test group. The RENN scores that were calculated in the testing group followed a trimodal distribution (Fig. 2), with the largest peak at a score of 0.17 and a mean score of 0.44. The baseline demographics, ECG characteristics and cardiovascular deaths of the test population divided by RENN tertiles are presented in Table 2. Only 7.2% of the ECGs in the first RENN tertile had any of the major ECG abnormalities noted in the table, while 10% and 49.5% of the ECGs in the second and third RENN tertiles contained any of these abnormalities (p < 0.0001).

Figure 2.

Distribution of RENN scores in the test population. RENN = resting ECG neural network.

Table 2.

Baseline Demographics, ECG Characteristics and Cardiovascular Deaths of the Test Population by RENN Tertiles

	RENN Tertile 1 n = 1409	RENN Tertile 2 n = 1409	RENN Tertile 3 n = 1407	P‐Value
Demographics
Age	45 ± 12	57 ± 13	66 ± 11	<0.0001
Female	13 (0.9)	15 (0.9)	3.4 (0.5)	<0.0001
BMI	31.4 ± 5	32.0 ± 5	31.4 ± 4	0.12
Hispanic	8.2 (0.7)	4 (0.5)	4.7 (0.6)	<0.0001
Black	20 (1.0)	7.7 (0.7)	10 (0.8)	<0.0001
ECG Abnormalities
Heart rate > 100	2.8 (0.4)	5.5 (0.6)	9.1 (0.8)	<0.0001
Left axis deviation	2.6 (0.4)	6.7 (0.7)	19.5 (1.1)	<0.0001
Right axis deviation	2.3 (.4)	2.0 (0.4)	2.5 (0.4)	0.66
PR > 200 ms	3.3 (0.5)	3.8 (0.5)	12 (0.9)	<0.0001
QRS > 120 ms	0.7 (0.2)	2.6 (0.4)	15 (1.0)	<0.0001
QTc > 450 ms	3.3 (0.5)	9.2 (0.8)	29 (1.2)	<0.0001
LBBB	0	0.1 (0.07)	3.3 (0.5)	<0.0001
RBBB	0.5 (0.2)	2.1 (0.4)	7.9 (0.7)	<0.0001
Paced rhythm	0	0	1.2 (0.3)	<0.0001
Atrial fibrillation	0	0.1 (0.1)	0.3 (0.2)	0.08
PVC	0.7 (0.2)	2.3 (0.4)	7.5 (0.7)	<0.0001
LVH	1.9 (0.4)	1 (0.2)	11.0 (0.8)	<0.0001
RVH	0.3 (0.1)	0.4 (0.2)	0.3 (0.1)	0.93
LAE	1.1 (0.3)	2.7 (0.4)	7.5 (0.7)	<0.0001
Upright T‐wave aVR	0.07 (0.07)	1.1 (0.3)	14 (1.0)	<0.0001
Q waves	3.8 (0.5)	6.6 (0.7)	27 (1.1)	<0.0001
Any ECG abnormality	7.2 (0.7)	10.0 (0.8)	49.5 (1.3)	<0.0001
Cardiovascular deaths	2.3 (0.4)	5.8 (0.6)	19 (1.0)	<0.0001

*Values are mean ± standard deviation or percent of subjects (standard error).

RENN = Resting ECG Neural Network score; T = tertile; BMI = body mass index; LBBB = left bundle branch block; RBBB = right bundle branch block; PVC = premature ventricular contraction; LVH = left ventricular hypertrophy; RVH = right ventricular hypertrophy; LAE = left atrial enlargement; Q waves refers to pathologic Q waves in any lead; Any ECG Abnormality refers to the abnormalities in this table. P‐value represents significance from the ANOVA for continuous variables and the chi‐square tests for categorical variables across the RENN tertiles. P < 0.05 was considered statistically different.

The C index of the RENN score in the test sample was 0.79. Each RENN tertile was evaluated for association with CVM by measuring Cox hazard ratios where the first tertile was used as a reference (Fig. 3). The second and third RENN tertiles had Cox proportional hazard ratios of 3.0 (95% CI 2.0–4.5) and 11.6 (8.0–16.8), respectively, when compared to the first tertile. The hazard ratios for the second and third RENN tertiles were higher than the respective Cox model tertiles, a difference which was most pronounced in the third tertile. These hazard ratios were in turn higher than the hazard ratios for the respective QRS, LVH and CIIS tertiles. In addition, although the Cox model performed well in this validation sample with a penalized chi‐square of 137.8, the neural network model performed better with a penalized chi‐square of 158.8.

Figure 3.

Cox proportional hazard analysis of QRS, LVH, CIIS, Cox Score and RENN Score tertiles in the test group. Hazard ratios are expressed relative to the first tertile of each respective group. Line bars represent 95% confidence intervals. QRS = maximal QRS duration; LVH = Romhilt left ventricular hypertrophy score; CIIS = cardiac infarction injury score; Cox = Cox model score; RENN = resting ECG neural network score.

To evaluate the additive value of the RENN score above the Cox model, with particular attention to the middle Cox scores, scores for each ECG in the testing set were calculated using the aforementioned Cox model. The scores were then divided into tertiles which were further subdivided by RENN tertiles to form a matrix of nine cells (Table 3). The Cox hazard ratios for each cell, relative to the cell at the intersection of the lowest Cox and RENN tertiles, were calculated. Of note, ECGs originally placed in the second Cox tertile, what could be considered an intermediate risk category, were then reclassified as having hazard ratios of 2.9 (95% CI 1.4–6.1), 4.8 (95% CI 2.5–9.4) and 16.4 (95% CI 8.9–30) by the RENN score.

Table 3.

Cox Proportional Hazard Ratios of Cox Score Tertiles Stratified by RENN Score Tertiles in the Test Population

Tertile	RENN 1	RENN 2	RENN 3
COX 1	1.0	3.8	13.6
	n = 865	n = 375	n = 167
COX 2	2.9	4.8	16.4
	n = 463	n = 541	n = 404
COX 3	2.1	6.2	22
	n = 79	n = 493	n = 836

Bold numbers indicate the Cox proportional hazard ratio compared to the ECGs with RENN and Cox scores in the first tertile. Numbers preceded by n = indicate the total number of ECGs in each cell.

RENN = resting ECG neural network score; COX = score obtained from the Cox model.

Analysis of the RENN Score in the ETT Group

The 2189 subjects in the exercise stress testing group were followed for an average of 8.6 years and had total of 156 cardiovascular deaths. The mean age of the predominantly male cohort was 56 years, 36% had established coronary heart disease, 11% had diabetes, 49% were hypertensive and 15% were on nitrate therapy (Table 4).

Table 4.

Baseline Demographic and Clinical Characteristics of Patients who Underwent Exercise Treadmill Testing

	Exercise Testing Population n = 2189
Demographics
Age	56.4 ± 11.7
Women	73 (3.3)
Black	252 (12.0)
Hispanic	146 (6.9)
Body mass index	28.3 ± 5.3
Current smoker	636 (25.4)
Ever smoker	1471 (67.2)
Past Medical History
Coronary artery disease	791 (36.1)
Myocardial infarction	519 (23.7)
Percutaneous coronary intervention	141 (6.4)
Coronary artery disease bypass graft	183 (8.4)
Diabetes	243 (11)
Hypertension	1086 (50)
Stroke	98 (4.5)
Atrial fibrillation	87 (4.0)
COPD	160 (7.3)
Claudication	124 (5.7)
Medication Use
Beta blocker	397 (18.1)
Nitrates	327 (14.9)
ACE‐I	248 (11.3)
Exercise Test Variables
Baseline heart rate	77.0 ± 15
Initial systolic blood pressure, mmHg	134.6 ± 21
Initial diastolic blood pressure, mmHg	82.8 ± 12
Exercise angina	270 (12.3)
ST‐segment deviation ≥ 1 mm	455 (21)
Metabolic equivalents (METS)	7.9 ± 3.5
Borg's perceived exertion score	17.3 ± 2.3
Duke Treadmill Score	5.2 ± 6.5
Mean RENN Score	0.45 ± 0.2

Values represent mean ± standard deviation or total number (percent).

There was a significant association between RENN Score tertiles and CVM in the overall ETT group. To demonstrate the added value of the RENN Score above the DTS, multivariate Cox hazard regression analysis was performed in the subgroup of patients with moderate risk DTS ( Fig. 4 ). The hazard ratio for the second and third tertiles compared to the first tertile in this subgroup were 2.7 (95% CI 0.9–8.1) and 5.4 (95% CI 2.0–15.2). Kaplan Meier curves were then plotted for survival from CVM for each of the RENN score tertiles in both the overall ETT group and the moderate‐risk group (Fig. 5). An early separation of the plots was demonstrated in each group and the degree of separation grew significantly over time (P < 0.0001). Of note, in the moderate‐risk DTS subgroup, the 10‐year CVM rates were 2.8%, 8.6% and 22% in RENN tertiles 1, 2 and 3, respectively.

Figure 4.

Cox hazard ratios of RENN score tertiles in all ETT subjects and in subjects with moderate risk DTS. Hazard ratios are expressed relative to the first tertile of each respective group and are adjusted for age, sex, diabetes, hyperlipidemia, smoking status and hypertension. Error bars represent lower 95% confidence limit. ETT = Exercise Tolerance Testing; DTS = Duke Treadmill Score.

Figure 5.

Kaplan‐Meier curves depicting the association between RENN score tertiles and cardiovascular mortality in (A) all subjects who underwent ETT and (B) only subjects with a moderate risk DTS. Log‐rank test P < 0.001. RENN = resting ECG neural network; DTS = Duke Treadmill Test.

DISCUSSION

Although the resting ECG is the most widely used cardiovascular diagnostic test, its ability to accurately predict long‐term outcomes has been questioned. ²⁹ , ³⁰ , ³¹ , ³² A few features, such as the presence of Q‐waves or a wide QRS, are clues to poor prognosis. However, more subtle clues such as low amplitudes and, more importantly, the relationship between these markers are difficult to interpret. With improving computing power, the application of ANN is gaining popularity. These networks have been developed for various clinical scenarios and in some cases have outperformed physicians ¹⁷ and traditional statistical techniques such as multiple logistic regression models. ¹⁶

A novel risk‐stratifier, the RENN score, has been created with the aid of an ANN. The RENN score's ability to stratify risk of CVM for several years after it has been recorded argues that the resting ECG indeed contains sufficient information to make reliable long‐term outcome predictions. One important question that often follows in the use of ANN is whether or not it can outperform traditional statistical approaches. Recent reports have suggested that limiting the analysis to simply comparing the discriminative accuracy with ROC curves or Concordance indices may be underestimating additive value of one test over the other. ³³ One study looking at the additive value of CRP to a clinically‐based Cox hazard model predicting cardiovascular events in women demonstrated that in certain subgroups, CRP was able to further risk‐stratify. ³⁴ Similarly, our analysis revealed that the RENN score was able to not only outperform, but also provide further risk‐stratifying power to the Cox score in a large number of ECGs (Table 2).

More importantly, we sought to determine the additive value of this novel risk score above the DTS, a well established noninvasive risk stratifier. Of greatest clinical relevance is the question of managing patients with moderate risk DTS scores. Our analyses revealed that the RENN score is able to identify patients in this intermediate subgroup who are at very high and very low risks of 10‐year CVM and could potentially help further guide management of this intermediate risk category.

There are several advantages to the approach that has been presented. One strength is that the interpretation of the ECG requires no human intervention. By including all available variables in the creation of the neural network, the bias of selecting only the variables believed to be important is minimized. Current processing power permits a more inclusive approach to the generation of large neural networks which have the particular advantage of accounting for interactions between numerous variables. The determination of the ECG characteristics and calculation of the RENN score were completely automated and objective. A scoring system such as this could be integrated into currently available ECG recording devices and provided to the physician instantaneously. The marginal cost of obtaining this score may also make this a candidate as a potential tool for screening for cardiovascular disease.

There are a few limitations of this approach worth noting. As with any neural network, it is difficult to determine which individual factors or combination of input variables are most important in determining the outcome of interest. The authors have chosen the ANN model for analysis, however, there are alternative machine‐learning algorithms that could be used to predict CV mortality. Robust Bayesian classification using exercise ECG and stress echocardiography characteristics in patients after an AMI performed comparably to a neural network in long‐term prediction of CV mortality. ³⁵ Future work will be directed to identify other machine‐learning algorithms that may outperform the RENN using a large resting ECG dataset.

A further limitation of this study is that the ANN was created and tested in a predominantly male, Caucasian veteran population. Analysis using a different population will be necessary to further validate this approach.

In summary, we have developed a novel cardiovascular risk stratifier, the RENN score, with the aid of ANN. This score can be obtained from objective demographic information and variables derived from the automated, computerized ECG with minimal additional effort. This novel score significantly outperforms traditional regression models and provides additive prognostic value above the DTS. Identifying patients at high risk for CVM may lead to more aggressive risk factor modification and potentially more invasive assessment of cardiovascular disease. The use of the RENN score could be a cost‐effective way of identifying high‐risk patients.