Assessment of Reproducibility – Automated and Digital Caliper ECG Measurement in the Framingham Heart Study

Abstract

Background

Digitized electrocardiography permits the rapid, automated quantification of electrocardiograms (ECGs) for analysis. Community- and population-based studies have increasingly integrated such data. Assessing the reproducibility of automated ECG measures with manual measures is a critical step in preparation for using automated measures for research purposes. We recently established an ECG repository of digitally recorded ECGs for the Framingham Heart Study and we sought to assess the reproducibility of automated and manual measures.

Methods

We selected 185 digitally recorded ECGs from routine visits of Framingham Heart Study participants spanning from 1986 to 2012. We selected the following ECG measures for their relevance to clinical and epidemiologic research: P wave duration, P wave amplitude, and PR interval in lead II; QRS duration and R wave amplitude in lead V6; and QT interval in lead V5. We obtained automated values for each waveform, and used a digital caliper for manual measurements. Digital caliper measurements were repeated in a subset (n=81) of the samples for intrarater assessment.

Results

We calculated the intraclass correlation coefficient (ICC) values for the interrater and intrarater assessments. P wave duration had the lowest interrater ICC (r=0.46) and lowest intrarater ICC (r=0.57). R wave amplitude had the highest interrater and intrarater ICC (r=0.98) indicating excellent reproducibility. The remaining measures had interrater and intrarater ICCs of r≥0.81.

Conclusions

The interrater reproducibility findings for P wave amplitude, PR interval, QT interval, QRS duration, and R wave amplitude were excellent. In contrast, the reproducibility of P wave duration was more modest. These findings indicate high reproducibility of most automated and manual ECG measurements.

Introduction

Community- and population-based studies have increasingly integrated automated, computer-based analysis ECG quantification. Such analysis permits efficiently developing an ECG database comprised of a large body of data with readily accessible and reproducible measures. Establishing reproducibility between manual and automated measures is essential prior to integrating automated measures. We consequently sought to determine the intrarater and interrater reproducibility of manual and automated ECG measurements of specific waveforms in the Framingham Heart Study ECG repository.

Methods

Participants

The Framingham Heart Study is a community-based study that was initiated in 1948 to identify incident cardiovascular disease and its risk factors (1). There has been prospective expansion of the Framingham Heart Study with subsequent enrollment of the Offspring Cohort in 1971, the Third Generation Cohort in 2002, and the multiracial Omni cohorts in 1994 and 2002 (1,2). Participants have ECGs as part of every Framingham Heart Study examination. In 1985, the Framingham Heart Study adopted a digital ECG recording system. Digitally recorded ECGs from 1986 to the present have been converted for contemporary analysis with the MUSE 8 ECG Management System (General Electric, Fairfield, CT), forming a repository of digitally recorded ECGs extending from 1986 to present (1).

In the current analysis, we sampled Framingham Heart Study ECGs spanning from 1986 to 2012. We randomly selected 50 ECGs from each of the following periods: 1986 to 1990, 1991 to 2000, and 2001 to 2010. We then randomly selected an additional 35 ECGs from 2011 to 2012. This approach limited overrepresentation of any single time period to account for temporal changes in ECG acquisition and recording techniques. ECGs were excluded if they had a paced rhythm, atrial fibrillation, or upon review had a technically inadequate tracing. The sample was not intended to be representative of the Framingham Heart Study or the ECG repository.

The digitally recorded ECGs were recorded at either 250 or 500 samples per second with a filter of 150Hz. They were printed on standard ECG paper at 25mm/s and 0.1 mV/mm, followed by transformation for contemporary analysis by the MUSE 8 ECG Management System (General Electric, Fairfield, CT) (1). P wave duration, P wave amplitude, PR interval, QT interval, QRS duration, and R wave amplitude were selected for study in specific leads because of their clinical significance. P wave duration, P wave amplitude and PR interval were measured in lead II because these waveforms in lead II can be used in evaluating for left atrial enlargement, right atrial enlargement, and sinus rhythm, respectively (3). QT interval was measured in V5 because this is one of the recommended leads for determining QT prolongation (4). QRS duration was measured in V6 because the QRS complex in this lead can be used to recognize certain bundle branch morphologies (5). R wave amplitude was measured in V6 because certain methods of evaluating for left ventricular hypertrophy involve the R wave amplitude in lead V6 (5). R wave amplitude in lead V6 has been measured manually as part of the standardized Framingham Heart Study examination. Prior studies from the Framingham Heart Study evaluating left ventricular hypertrophy have used R wave amplitude in V6 as a method of determining left ventricular hypertrophy (6,7). The Boston University School of Medicine Institutional Review Board approved each Framingham Heart Study examination and all participants provided written, informed consent.

Measurement Protocol

A single individual (GMB) used digital calipers to make manual measurements by manipulating a computer mouse. Images were maximally enlarged as allowed by the Muse 8 Management System (General Electric, Fairfield, CT) (8). The digital caliper measurements were performed on the first complete waveform. Only waveforms in sinus rhythm were included. Incompletely recorded beats and premature ventricular beats were excluded and the next complete sinus waveform in sequential order was measured. P wave duration was measured in lead II. Measurement was conducted from the onset of the P wave, defined as the initial deflection from the isoelectric baseline of the TP segment, to the offset of the P wave, defined as the return of the P wave to the isoelectric baseline of the PR interval. P wave amplitude was measured in the same lead II waveform from the onset of the P wave to its highest amplitude. PR interval measurements were performed in the same lead II waveform. The PR interval was measured from the onset of the P wave to the onset of the QRS complex, defined as the initial deflection from the baseline of the PR interval. The QT interval was measured in lead V5, and determined as the onset of the QRS complex to the end of the T wave, defined as the return of the T wave to the isoelectric baseline of the TP segment. QRS duration was measured in lead V6, and determined as the onset of the QRS complex to the return of the complex to the isoelectric baseline of the ST segment. R wave amplitude was measured in lead V6, and quantified from the onset of the QRS complex to the highest vertical point of the R wave.

Blinded, repeated measures were obtained on different days for assessment of intrarater reproducibility. Intrarater measurements were performed on the same waveforms as those measured initially. The intrarater assessment included 20 ECGs from 1986 to 1990, 1991 to 2000, and 2001 to 2010, and 21 from 2011 to 2012 (total n=81) from the original interrater assessment. Intrarater ECGs were selected randomly from the initial pool of tracings. The automated measures were recorded by the MUSE 8 ECG Management System (General Electric, Fairfield, CT). The MUSE 8 ECG Management System program provides median, lead specific measures from digitally recorded ECGs.

Definitions

The definition of hypertension for this study was systolic blood pressure ≥140 mmHg, diastolic blood pressure ≥90 mmHg, or current use of antihypertensive medications for treatment of high blood pressure. Diabetes was defined as the use of oral hypoglycemic agents, insulin, or fasting blood glucose ≥126 mg/dL. Cardiovascular disease was defined as the presence of coronary heart disease, stroke, peripheral vascular disease, and/or congestive heart failure (9).

Statistical analysis

Means, standard deviations and descriptive statistics of continuous variables and the distributions of categorical variables were performed for the sample cohort of this study. ICCs were used to quantify both interrater and intrarater assessments of the six ECG measures. The interrater assessment calculated ICCs comparing the automated measurements with digital caliper measurements of the 185 selected ECGs. The intrarater assessment calculated ICCs comparing the digital caliper measurements performed on 81 ECGs from the interrater assessment to repeated digital caliper measurements of the same 81 ECGs that were recorded in blinded fashion and on different days (n=81). An ICC of 1.0 indicates perfect correlation, 0.75 excellent correlation, and 0.5 moderate correlation (10). All statistical analyses were performed with SAS 9.2 (SAS Institute, Cary, NC). To assess the reproducibility further, we evaluated the mean differences of the interrater and intrarater measures.

Results

Study subjects

Complete demographic data were available from 184 of the 185 participants at the time this study was conducted. Of these 184 participants, the mean age was 60 years, 59% of the participants were women, the mean BMI was 27.1, 16% had cardiovascular disease, 11% had diabetes, and 50% had hypertension (Table 1).

Table 1.

Descriptive characteristics of Framingham Heart Study participants included in the electrocardiogram reproducibility analysis. (N=185)

Clinical Characteristic
Age, years	60 ± 17
Women	108 (58.7%)
Body Mass Index	27.1 ± 4.9
Hypertension	92 (50.0%)
Diabetes	19 (10.6%)
Cardiovascular disease	30 (16.2%)

Continuous variables described as mean±SD and categorical as number (%).

Assessment of correlations

Correlations of the automated and digital caliper measurements are listed in Table 2. P wave duration had the lowest correlation (r=0.47, P<0.001). Correlations of P wave amplitude, PR interval, QRS duration, QT interval, and R wave amplitude all had a correlation of r≥0.81 (P<0.001). The correlation of PR interval and QT interval were both r=0.81 (P<0.001). R wave amplitude had the strongest correlation (r=0.98, P<0.001). Figures 1A–1F graphically demonstrate the correlation of the automated measure and digital caliper values for P wave duration, P wave amplitude, PR interval, QRS duration, QT interval, and R wave amplitude using scatter plots. The higher correlation value of R wave progression (r=0.98) is represented by the excellent correlation of the data points with the trendline shown in Figure 1F. The lower correlation of P wave amplitude (r=0.47) is represented by the more variable correlation of the data points with the trendline shown in Figure 1A.

Table 2.

Correlation between automated and digital caliper measures (in select leads).

	P wave duration (II)	P wave amplitude (II)	PR interval (II)	QRS duration (V6)	QT interval (V5)	R wave amplitude (V6)
Correlation	0.47 ‡	0.85 ‡	0.81 ‡	0.86 ‡	0.81 ‡	0.98 ‡

Correlations are between automated measures and measures using digital calipers as described by text.

^*P<0.05,

^†P<0.01,

^‡P<0.001.

Figure 1.

Scatter plots showing correlations comparing automated measure and digital caliper values for (A) P Wave Duration; (B) P Wave Amplitude; (C) PR Interval; (D) QRS Duration; (E) QT Interval; (F) R wave Amplitude.

Interrater and intrarater correlation coefficients

The interrater ICC values comparing automated measurements to digital caliper measurements are listed in Table 3. The lowest interrater ICC was for P wave duration (r=0.46). The remaining ICCs in the interrater assessment were r≥0.81, with PR interval and QT interval both having ICC values of r=0.81. R wave amplitude had an excellent interrater ICC (r=0.98).

Table 3.

Intraclass correlation coefficients (95% confidence interval) of ECG measures in selected leads for inter- and intrarater assessment.

	Interrater assessment*	Intrarater assessment†
P wave duration (II)	0.46 (0.35, 0.57)	0.57 (0.42, 0.71)
P wave amplitude (II)	0.85 (0.81, 0.89)	0.97 (0.96, 0.98)
PR interval (II)	0.81 (0.75, 0.85)	0.97 (0.95, 0.98)
QT interval (V5)	0.81 (0.75, 0.85)	0.94 (0.92, 0.96)
QRS duration (V6)	0.83 (0.79, 0.87)	0.96 (0.95, 0.98)
R wave amplitude (V6)	0.98 (0.97, 0.99)	0.98 (0.96, 0.98)

^*Interrater assessment evaluated intraclass correlation coefficients using values obtained from automated measures and digital calipers.

^†Intrarater assessment evaluated intraclass correlation coefficients using values obtained from digital calipers only.

The intrarater ICC values comparing repeated digital caliper measurements are listed in Table 3. The intrarater assessment compared the digital caliper measurements made in blinded fashion and on different days from 81 of the original 185 ECGs. The lowest ICC was P wave duration (r=0.57), consistent with only moderate reproducibility. PR interval, P wave amplitude, QRS duration, QT interval, and R wave amplitude had ICC values of r≥0.94, with R wave amplitude having a value of r=0.98. These ICC values demonstrate excellent intrarater reproducibility and were mostly superior to the interrater assessments.

The mean difference for the intrarater measures for R wave amplitude in V6 was 11.6 μV, and was the largest of all the calculated values. In contrast, the mean difference for the interrater measures for R wave amplitude in V6 was −0.8 μV. The interrater and intrarater mean differences for all waveforms are summarized in Table 4. The second largest mean difference was the interrater P wave duration in lead II, 5.2 ms. The mean difference with the smallest absolute value was the intrarater P wave amplitude in lead II, 0.1 μV.

Table 4.

Mean difference and standard deviation of interrater and intrarater measures

Variable	Mean difference	Standard Deviation
P wave duration interrater measures in lead II, (ms)	5.2	17.3
P wave duration intrarater measures in lead II, (ms)	−3.6	15.9
P wave amplitude interrater measures in lead II, (μV)	−1.8	20.8
P wave amplitude intrarater measures in lead II, (μV)	0.1	9.7
PR interval interrater measures in lead II, (ms)	−2.0	19.8
PR interval intrarater measures in lead II, (ms)	1.7	7.8
QT interval interrater measures in lead V5, (ms)	5.0	19.7
QT Interval intrarater measures in lead V5, (ms)	3.7	11.5
QRS duration interrater measures in lead V6, (ms)	3.2	10.6
QRS duration intrarater measures in lead V6, (ms)	−0.6	5.5
R wave amplitude interrater measures in lead V6, (μV)	−0.8	89.4
R wave amplitude intrarater measures in lead V6, (μV)	11.6	101.1

μV=microvolts, ms=milliseconds

Discussion

In the present analysis, we assessed intrarater and interrater reproducibility of ECG measures in selected tracings and waveforms from a large, community-based epidemiologic study. The ECG is a long-standing component of the Framingham Heart Study, and the present investigation provides an opportunity to assess the reproducibility of automated and digital caliper measures. This present analysis seeks to allow for the inclusion of specific automated measures as part of Framingham Heart Study investigations.

In our analysis, we determined that the highest levels of interrater and intrarater reproducibility were found in quantifying R wave amplitude in lead V6. The correlation of automated and digital caliper measures for R wave amplitude was very robust (r=0.98, P<0.001). This strong correlation is clearly demonstrated by the corresponding scatter plot (Figure 1F). This scatter plot compares the automated measure and digital caliper values of R wave amplitude. The values in the scatter plot for R wave amplitude correlate excellently with the trendline. Both the R wave amplitude interrater and intrarater ICC values were r=0.98, similarly indicating excellent reproducibility. We attribute the strong ICC determination of reproducibility to the easily identified initial onset and peak of the R wave. Clinical applications incorporating R wave amplitude measurements include determining left ventricular hypertrophy, as well as monitoring longitudinal, prospective changes in R wave amplitude in association with cardiovascular risk factor accumulation.

The correlation values of P wave amplitude, PR interval, QRS duration and QT interval were between 0.81 and 0.86. The interrater ICC values for P wave amplitude, PR interval, QRS duration and QT interval were between 0.81 and 0.85. The intrarater ICC values for P wave amplitude, PR interval, QRS duration and QT interval also demonstrated excellent reproducibility with values between 0.94 and 0.97. These assessments provide us with similar confidence in integrating these values with Framingham Heart Study data.

In contrast, P wave duration consistently had the lowest values of correlation. The P wave duration interrater ICC value was r=0.46, and the intrarater ICC value was r=0.57. The P wave duration interrater ICC value was the lowest of all the ICC values in the study, and the only ICC value that showed less than moderate reproducibility. Likewise, the P wave duration intrarater ICC was the lowest value in the intrarater analysis and the only intrarater ICC value that correlated with less than excellent reproducibility, or a value of less than r=0.75. We note that P wave duration had lower values of correlation while other measurements involving the P wave, i.e. P wave amplitude and the PR interval, were superior.

The P wave is a low-amplitude waveform and our data suggest that there may be potential challenges interpreting automated and digital measurements. The interrater ICC was only slightly inferior to the intrarater ICC, suggesting the challenges in P wave duration measurement are similar for both automated and digital measurements.

Overall, the lowest correlation values and interrater ICCs were found in P wave duration, PR interval, and QT interval. We examined the ECGs with the largest discrepancy among the automated measure and digital caliper values for these three intervals. We observed that the waveforms had significant baseline artifact. P waves had low amplitudes in ECGs that demonstrated discrepant P wave duration and PR interval measurements. Likewise, tracings with discrepant QT interval measurements had low amplitude T waves. Coarse tracings and low amplitudes often made determining P wave and T wave onset and offset less discreet, resulting in greater variability in reproducibility of measures. It is likely that the diminished amplitudes of these intervals contributed to the lower correlation values and interrater ICCs for P wave duration, PR interval and QT interval.

Scrutiny of our data further identified a contradictory interpretation by the automated measurement. We found an automated measure with a median PR interval of 0 ms, despite evident P waves and automated measures of P wave duration of 154 ms and P wave amplitude of 43 μV in the same waveform. We suspect that this discordance contributed to the lower PR interval interrater ICC (r=0.81) relative to the superior intrarater ICC (r=0.97). Values reported as 0 in the automated measures indicate that a specific waveform was not identified by the MUSE 8. It is important that when data is collected from automated measures that values of 0 are not included in calculating statistics to avoid falsely altering the calculations. We suggest quality control assessments to identify such discrepancies when using automated measures.

In our analysis, the mean differences of the intrarater measures were closer to zero than the interrater measures, with the exception of R wave amplitude. It is likely that the large amplitude of the R wave contributed to the sizeable differences in standard deviation in the mean differences in the interrater and intrarater assessments.

There are several strengths to our study. The MUSE 8 ECG Management System displays high resolution digitized ECGs that enhance the accuracy and precision of manipulating a digital caliper. The ECGs included in our investigation were digitally recorded using a standardized technique. Furthermore, the tracings were collected from 1986 to 2012, limiting bias from temporal trends in ECG acquisition. Also, we included measurements of six widely used ECG landmarks with varying levels of clinical and investigatory significance.

Our study also has several limitations. First, a single reader conducted the digital caliper measurements. Inclusion of more than one reader would have facilitated an additional determination of interrater measurements between two individuals. However, our foremost goal was to report interrater reproducibility between the digital caliper and automated measurements. Having only a single reader introduces the potential for systematic error, but also assures a consistent technique for quantifying intrarater assessments. Second, the sample size of selected ECGs was relatively limited. The interrater ICC assessments included 185 ECGs and the intrarater ICC assessments included 81 ECGs. Third, the number of measured waveforms in specific leads was limited. We selected specific waveforms based on their relevance to prior studies from the Framingham Heart Study investigations (6,7). Measurements of more waveforms and measurements of the same waveforms in more than one lead were not collected. Fourth, the limited sample size only included ECGs in sinus rhythm and excluded ECGs with paced rhythms or in atrial fibrillation. Therefore, this study is not representative of ECG reproducibility assessments in cohorts including arrhythmias and paced rhythms. Fifth, Framingham Heart Study participants are primarily of European descent. Hence, generalizability of our results to more racially and ethnically diverse cohorts is unknown. Sixth, enlargement of waveforms may have enhanced reading but simultaneously amplified background noise that could make waveform onset and offset less discrete. We lacked a standardized protocol for distinguishing waveforms and background noise beyond what is described in our methods section. As a result our measures may have included artifact and both systematic and random errors. Next, we did not compare the reproducibility results of this study to those of other ECG repositories or to the standards required for the FDA approval of computerized ECG interpretation. The scope of our study was to internally validate automated measures at the Framingham Heart Study for future studies. Finally, the measurements from the ECGs provided did not compare serial ECGs of the same individual. Therefore, the reproducibility assessment here does not apply to repeated ECGs of the same individual or cohort.

Conclusion

Overall, the findings from this study show excellent interrater correlation of automated and digital caliper measurements of PR interval, P wave amplitude, QRS duration, QT interval and R wave amplitude. P wave duration had more limited reproducibility. Our results provide us with strong confidence in introducing automated measures to Framingham Heart Study data. Integrating rapidly acquired waveforms through digital ECG platforms will enhance Framingham Heart Study data acquisition, save valuable investigator time, and permit novel analyses that may guide identification of cardiovascular disease and its risk factors.