Clinical validation and evaluation of a novel six-lead handheld electrocardiogram recorder compared to the 12-lead electrocardiogram in unselected cardiology patients (EVALECG Cardio)

Abstract

Aims

Handheld electrocardiogram (ECG) monitors are increasingly used by both healthcare workers and patients to diagnose cardiac arrhythmias. There is a lack of studies validating the use of handheld devices against the standard 12-lead ECG. The Kardia 6L is a novel handheld ECG monitor which can produce a 6-lead ECG. In this study, we compare the 6L ECG against the 12-lead ECG.

Methods and results

A prospective study consisting of unselected cardiac inpatients and outpatients at Leeds Teaching Hospital NHS Trust. All participants had a 12- and 6-lead ECGs. All ECG parameters were analysed by using a standard method template for consistency between independent observers. Electrocardiograms from the recorders were compared by the following statistical methods: linear regression, Bland–Altman, receiver operator curve, and kappa analysis. There were 1015 patients recruited. The mean differences between recorders were small for PR, QRS, cardiac axis, with receiver operator analysis area under the curve (AUC) of >80%. Mean differences for QT and QTc (between recorders) were also small, with AUCs for QT leads of >75% and AUCs for QTc leads of >60%. Key findings from Bland–Altman analysis demonstrate overall an acceptable agreement with few outliers instances (<6%, Bland–Altman analysis).

Conclusion

Several parameters recorded by the Kardia 6L (QT interval in all six leads, rhythm detection, PR interval, QRS duration, and cardiac axis) perform closely to the gold standard 12-lead ECG. However, that consistency weakens for left ventricular hypertrophy, QRS amplitudes (Lead I and AVL), and ischaemic changes.

Graphical Abstract

Introduction

Handheld electrocardiogram (ECG) machines are low-cost, compact and are now used increasingly by physicians and patients for screening and diagnostic purposes.¹ These recorders have been shown to be very useful for the detection of atrial fibrillation,² diagnosis of patients with palpitations,³ and more recently the evaluation of the QT interval.⁴ The first devices were single lead monitors, but now the Kardia 6L (AliveCor Inc., Mountain View, CA, USA) has been approved for rhythm diagnosis and recently received FDA approval for QT analysis.⁵ A recent study showed how the Kardia 6L with machine learning can be useful in the diagnosis of long QT syndrome.⁴

The 12-lead ECG remains the gold standard, however, can be challenging to perform for a number of reasons including training, a suitable private clinical environment to perform the recording, time and also cables that need thorough cleaning (particularly relevant with COVID-19). Therefore, the use of simpler, smaller, and portable handheld recorders is very attractive for clinical use. These devices have been used for diagnostic purposes beyond rhythm and QT analysis.⁶

To our knowledge, a large validation study comparing the novel Kardia 6L with the standard 12-lead ECG has never been performed in unselected cardiology patients. Validating this recorder is important since the 12-lead ECG recording system is different to the Kardia 6L (e.g. no chest leads on Kardia 6L), and hence could lead to diagnostic errors. We hypothesized that there would be no significant differences in diagnostic accuracy between the 12-lead ECG recorder and Kardia 6L, and hence designed a prospective study to compare the two different modalities.

Methods

A prospective study was designed to compare the diagnostic ability of the Kardia 6L against the 12-lead ECG. Patients who attended the outpatient cardiology clinics or were inpatients at the Leeds General Infirmary, Leeds, UK and required a 12-lead ECG for clinical reasons were approached by the research team. We did not perform asymptomatic screening ECGs (e.g. for the detection of atrial fibrillation). The only exclusion criteria were refusal or inability to provide informed consent. Following informed consent, baseline patient demographics were collected, and the ECGs were obtained by either a research doctor or nurse who were trained in obtaining 12-lead ECGs and Kardia 6L recordings.

Electrocardiogram acquisition

A standard 12-lead ECG was performed first, followed by the Kardia 6L recordings which were performed immediately after the 12-lead ECG acquisition (generally within a minute). For the latter care was taken to ensure that the device was held in the correct orientation so that transposition was avoided. There were two handheld electrodes and a third electrode was placed on the left thigh (or the left ankle if this was not possible). Isopropyl alcohol swabs were used to ensure appropriate contact. For the Kardia 6L recording, the device was programmed for a maximum recording time of 30 s, but this could be terminated early if a good quality recording had been obtained (by visual assessment) or repeated if the recordings were poor. The 12-lead ECG was obtained with a MAC 550 (GE Healthcare, WI, USA). The filter settings for the 12 leads were 0.05–100 Hz and 0.05–40 Hz for the Kardia 6L. The sampling rate stated by the manufacturer for the Kardia 6L was 300/s with a 16-bit dynamic range and system resolution of <1 μV. Both recordings were obtained with a sweep speed of 25 mm/s, and an amplitude of 1 mm/mV.

The 12-lead ECG was directly printed onto ECG graph paper, and the Kardia 6L traces were stored as a PDF on a mobile phone and then printed onto plain paper (graph paper was not used as the printout included the gradations).

Electrocardiogram analysis

Three experienced observers (one cardiologist and two cardiac physiologists) performed the ECG analysis independently of each other. Each ECG was analysed twice, and to ensure quality control interobserver and intraobserver analysis coefficient of variation analysis was calculated on a sample of ECGs.

Several measures were taken to prevent classification bias within this study. Electrocardiograms were analysed in a systematic manner and were separated into two groups: a batch of 12-lead ECGs and a batch of Kardia 6L ECGs. The ECGs were anonymized and numbered; these were analysed in packs of 10. In each pack, the 12-lead ECGs were analysed first followed by the Kardia 6L ECGs. Analysis within each pack took place in a random order so that the obtained measurements were not emulated.

The parameters analysed included PR and QT interval, QRS duration, Axis and QRS amplitude, rate, rhythm, and ischaemia. Electrocardiogram quality was graded as good (5–6 leads available for analysis), acceptable (3–4 leads available for analysis), or poor (1–2 leads available for analysis) for the limb lead recordings on the 12-lead and Kardia 6L ECG. For the PR interval and QRS duration, the complex with the clearest recording was chosen. The cardiac axis was calculated using leads I and III and computed using an online calculator (https://en.my-ekg.com/calculation-ekg/heart-axis-calculator.php). The QRS duration was measured for each lead by measuring the peak to trough of the QRS complex. For QT analysis, each lead was analysed using the tangent method.⁷ QTc was calculated using Bazett’s formula. All recordings were performed manually and stored on the study database. For QT analysis, the average measurement of three complexes was taken.

Statistical analysis

Once the ECGs were double reported, for each quantitative analysis the two observers’ recordings were plotted on a scatter plot. Any outliers on the scatterplots (>40 ms) were rechecked and reanalysed if appropriate by a third observer. The values were then averaged and compared against each other (12-lead vs. Kardia 6L).

The descriptive statistics are shown in Table 1 for 12 ECG and Kardia 6L. Qualitative parameters are presented as the percentage of the sample and the continuous parameters using the mean and standard deviation (SD) (Table 1).

Table 1.

Study patients baseline characteristics

	n = 1015
Age (years)	62 ± 17
Male	634 (62.4%)
Ethnicity (Caucasian: South Asian: Black: Other)	767 (75.6%): 62 (6.1%): 14(1.4%): 172(16.9%)
Diabetes	193 (19%)
Hypertension	385 (37.9%)
Previous coronary disease	278 (27.4%)
History of arrhythmia (atrial or ventricular)	258 (25.4%)
Known atrial fibrillation	207 (20.4%)
BMI	28.6 ± 9
Outpatients: inpatients	613 (60.4%): 402(39.6)
Indication for ECG
Valve disease	116 (11.4%)
Arrhythmia	116 (11.4%)
Heart failure	262 (25.8%)
Coronary disease	478 (47.1%)
Inherited arrhythmia assessment	110 (10.8%)

Various statistical methods were used to perform the comparison of the 12-lead ECG vs. Kardia 6L ECG. The principal analysis was agreement between the 12-lead ECG and 6L Kardia continuous measurements performed through a Bland–Altman method.⁸ Left ventricular hypertrophy was assessed using both the Sokolov–Lyon and modified Cornell criteria. ST-segment elevation or depression was recorded as being present if there was 1 mm elevation or depression of the ST-segment in any lead, and T-wave inversion was noted if the T-wave was inverted beyond 1 mm except in lead V1. Secondly, the Kappa statistic was used to assess the inter-modality reliability for categorical variables (the closer to 1 the better the association). To compare continuous data, the initial comparison was done through a coefficient of determination of r² derived from a linear regression which indicated the amount of variability of 12-lead ECG measurement explained by the Kardia 6L ECG. Finally, the performance of the Kardia 6L vs. 12 ECG (considered as the gold standard) was assessed through sensitivity (Se), specificity (Sp), binary receiver operating characteristic curve, and area under the curve (AUC) for QT/QTc <360 and QT/QTc >460 classified as abnormal and QT/QTc between 360 and 460 inclusive classified as normal QT/QTc.⁹

Statistical analysis was performed with Minitab statistical software (version 19, State College, PA, USA).

Sample size

The sample size calculation was based on the Bland–Altman plot.¹⁰ The SD from the 12-lead ECG (5.2 ms)¹¹ as well as the difference between handheld and 12-lead ECGs measurements of QTc were used in the calculation. A limit of agreement of 2.2 times SD was used as the approximation of the difference (2.2 × 5.2 = 11.44 ms).

We calculated that a sample size of 538 would yield a power of 80% and 665 for a power of 90% based on the above calculations. Based on these calculations, the study planned to recruit 1000 participants.

Results

A total of 1015 patients were recruited for this study. The baseline demographics are presented in Table 1. The left thigh was used in >99% of patients.

The mean intraobserver and interobserver coefficient of variation for the three observers was less than 1.5% (Supplementary material online, File S1).

The Kardia 6L ECG performed well for rhythm recognition and T-wave changes, but less well for ischaemia (Table 2, Supplementary material online, File S2).

Table 2.

Kappa statistic for qualitative data

ECG rhythm and abnormalities	Kappa statistic
Sinus rhythm	0.779
Atrial fibrillation	0.986
Atrial flutter	0.917
IVCD	0.970
Junctional rhythm	1.000
Paced	0.894
Left ventricular hypertrophy	0.678
2nd degree or 3rd degree AV block	1.000
Ectopy	0.666
T-wave changes	0.895
ST elevation	0.421
ST depression	0.340

IVCD, intra ventricular conduction delay.

It was possible to measure the PR, QRS duration, axis, and QRS amplitude in most of the patients using the Kardia 6L (>70% for PR, but >90% for the other parameters). The mean differences for these parameters were small (Tables 3 and 4, Supplementary material online, File S2).

Table 3.

Percentage of leads analysis possible and fitted logistic linear regression analysis comparing 12-lead ECG and 6L measurements

	% ECG leads that could be analysed		R²
Leads	12L	6L
PR interval	75.9%	71.5%	83.4%
QRS duration	92.2%	87.6%	85.0%
Axis	91.6%	89.9%	78.0%
QRS amplitude Lead I	97.9%	97.1%	59.5%
QRS amplitude Lead II	98.1%	97.3%	73.9%
QRS amplitude Lead III	98.0%	97.1%	73.6%
QRS amplitude Lead AVR	97.9%	97.2%	70.4%
QRS amplitude Lead AVL	98.1%	97.4%	67.4%
QRS amplitude Lead AVF	98.1%	97.1%	75.8%
QT Lead I	67.9%	71.6%	75.8%
QTc Lead I	67.9%	71.5%	46.2%
QT Lead II	76.6%	72.8%	77.8%
QTc Lead II	76.6%	72.8%	48.1%
QT Lead III	55.4%	51.4%	72.9%
QTc Lead III	55.4%	51.3%	48.5%
QT Lead AVR	68.6%	71.9%	73.5%
QTc Lead AVR	68.6%	71.8%	44.1%
QT Lead AVL	58.2%	66.1%	75.5%
QTc Lead AVL	58.2%	66.0%	46.1%
QT Lead AVF	61.5%	57.0%	72.9%
QTc Lead AVF	61.5%	56.9%	45.8%
Longest QT	80.4%	75.9%	78.8%
QTc longest lead	80.4%	75.9%	51.7%

Table 4.

Bland–Altman and receiver operator analysis for quantitative data

Leads	Mean	SD	CI upper limit	CI lower limit	Outliers (%)	AUC
	Bias
PR interval (ms)	0.76	12.00	24.28	−22.76	13 (1.8)	0.91
QRS duration (ms)	0.29	8.47	16.89	−16.32	21 (2.4)	0.98
Axis	4.24°	22.11°	47.57°	−39.08°	34 (4.3)	0.85
QRS amplitude Lead I (mm)	−1.47	2.91	4.23	−7.17	41 (4.2)	NA
QRS amplitude Lead II (mm)	0.96	2.07	5.02	−3.10	31 (3.1)	NA
QRS amplitude Lead III (mm)	−0.16	2.73	5.18	−5.50	40 (4.1)	NA
QRS amplitude Lead AVR (mm)	−0.33	1.80	3.19	−3.85	34 (3.4)	NA
QRS amplitude Lead AVL (mm)	−1.13	2.56	3.90	−6.15	39 (4.0)	NA
QRS amplitude Lead AVF (mm)	0.94	2.11	5.06	−3.19	37 (3.7)	NA
QT Lead I (ms)	6.29	21.68	48.69	−36.21	29 (4.0)	0.79
QTc Lead I (ms)	−0.27	28.17	54.93	−55.48	29 (4.0)	0.70
QT Lead II (ms)	7.03	19.81	45.87	−31.80	22 (3.0)	0.82
QTc Lead II (ms)	0.62	26.82	53.19	−51.95	37 (5.1)	0.69
QT Lead III (ms)	6.47	23.99	53.48	−40.54	19 (3.7)	0.79
QTc Lead III (ms)	1.15	28.67	57.35	−55.05	25 (4.9)	0.66
QT Lead AVR (ms)	7.06	21.18	48.57	−34.46	20 (2.8)	0.81
QTc Lead AVR (ms)	−0.03	27.11	53.10	−53.15	37 (5.2)	0.70
QT Lead AVL (ms)	5.45	22.50	49.56	−38.65	21 (3.2)	0.80
QTc Lead AVL (ms)	−2.02	28.38	53.61	−57.65	28 (4.2)	0.67
QT Lead AVF (ms)	8.49	23.00	53.57	−36.59	17 (3.0)	0.77
QTc Lead AVF (ms)	2.35	29.14	59.46	−54.76	31 (5.4)	0.64
Longest QT (ms)	11.76	20.69	52.31	−28.79	24 (3.2)	0.80
QTc longest (ms)	5.71	27.11	58.85	−47.42	43 (5.7)	0.74

AUC, area under the curve (the normal range was set at a QT/QTc of between 360 and 460 ms; CI, 95% confidence interval for upper and lower limit on Bland–Altman analysis; outlier percentage compared to the number of ECGs analysed rather than total sample); SD, standard deviation; NA, not applicable.

The median QT interval in the sample was 380 ms (interquartile range 360–420 ms, and minimum and maximum QT of 260 ms and 640 ms, respectively). The correlation coefficient for QT measurements between the Kardia 6L ECG and 12-lead ECG was good. QT interval measurements for the Kardia 6L could be performed in 55–77% of the leads with good correlation for the QT interval (r² > 70%), but less so for QTc. The mean differences between Kardia 6L and the 12-lead ECG for QT and QTc were small, the AUC was >75% for QT but less for QTc although overall >60% (for the AUC, we defined a normal QT interval between 360 and 460 ms and any values outside this range were considered abnormal). For a large number of samples, the number of outliers (<6%) on the Bland–Altman analysis was small (Tables 3 and 4, Supplementary material online, File S2).

Discussion

To our knowledge, this is the first large scale comparison of the Kardia 6L ECG monitor against the standard 12-lead ECG in an unselected cohort of cardiac patients. The key findings were: (i) the strong correlations observed between the Kardia 6L and 12-lead ECG for most measurements including QT and QTc [the r² (coefficient of determination) reported in the majority of measurements is above 70%], (ii) that several parameters yielded an AUC above 0.8 which shows that the Kardia 6L ECG performs very closely to the 12 ECG, and far from the random diagnosis for the thresholds established, (iii) few outliers and little bias in the Bland–Altman analysis (<6% in all parameters) when compared with the 12-lead ECG, (iv) excellent correlation with rhythm analysis (Kappa 0.78 for sinus rhythm for example), but (v) the Kardia 6L performed poorly when compared with the 12-lead ECG for ischaemia analysis, assessment of left ventricular hypertrophy and QRS amplitudes.

There are many advantages of handheld ECG monitors over a standard 12 leads. The portability, size, cost, and storage and transfer of data makes these monitors potentially useful in practices where ECG recorders are not available or difficult to perform [i.e. need for a private room with couch, operator with enough experience, consumables (e.g. ECG electrodes), cleaning equipment (for cables can be time-consuming)], and the recording can be performed almost immediately whilst the patient is seated. Many companies have incorporated algorithms for rhythm analysis, and these can be particularly attractive for less experienced healthcare workers and patients. The portability means that patients can also use the devices and capture rhythm diagnostics. Moreover, these devices can be used in areas of the world where ECG equipment is not available. Furthermore, mobile phone technology which is widely available worldwide can be used to transfer data quickly for a second opinion from these devices.¹² There has been an explosion in the utility as well as the number of handheld monitors. However, it is important that these devices are adequately tested against the gold standard before being used as they may have shortcomings. It is important to note that the ECG is not acquired in the same way as the 12 leads. Ischaemia diagnosis and QT measurements clinically rely on the availability of precordial leads. Peripheral leads provide an estimate of the overall ‘far-field’ cardiac de-/repolarization vector. In contrast, the precordial leads, referenced to the central Wilson terminal, provide a rather local ‘near-field’ assessment of de-/repolarization changes. Precordial leads are thus more sensitive and specific for ischaemia diagnosis. QT times vary between leads of the 12-lead ECG. The mean QT time of all 12 leads appears to be the best estimator for a cardiac QT time.¹³ Both peripheral (aVF) and precordial (V3) leads have shown a rather good correlation with the mean QT time. Estimating the QT time from peripheral only is likely to obtain incorrect estimates of the real underlying QT time.

In our study, rhythm analysis and interpretation with the Kardia 6L was excellent, and this is consistent with previous reports.³ However, assessment for ischaemia was poor, suggesting the importance of chest leads in this analysis, and that the 6L cannot replace the 12-lead ECG for this indication.

QT interval analysis has been a major focus with handheld ECG devices. Recently, a large study compared machine learning calculated QT in a prototype of the Kardia 6L against the 12-lead ECG.⁴ This was tested in patients with known long QT syndrome in whom the gold standard QT interval was obtained from cardiologist over-read 12-lead ECGs where it has been noted that there can be a disagreement of 70 ms between observers.¹⁴ When it comes to the use of this technology it is important that ‘real life’ scenarios are used to compare these modalities, i.e. generalists interpreting on standard ECG paper. A machine generated value may be incorrect if there is artefact, and many clinicians will manually remeasure. Prior studies have not performed a lead against lead comparison, which we report for the first time. Our data suggest that in approximately 20% of cases it is not possible to make an accurate assessment either because of a poor recording, unclear baseline or unclear T-wave (e.g. flat T-wave). Recent short reports have nevertheless shown that measurement of QTc and PR interval and QRS duration appear to show good agreement between the handheld and 12-lead ECG. In a two-lead handheld smartphone ECG monitor, there was reasonable agreement in the QTc in hospitalized patients started on dofetilide.¹⁵ In an athletic population, there was relatively good agreement between the Kardia 6L and 12-lead ECGs with a caveat that the Kardia 6L recordings were shorter.¹⁶ It is important to note that individual lead analysis was not performed in this; our data show that there can be variance dependent on where measurements are taken. More recently, in a population of patients being monitored for long QT syndrome, the Kardia 6L performed well compared to the 12-lead ECG despite the difference in the patients’ position (seated vs. supine).¹⁷ Our data together with previous studies does suggest that QT analysis is feasible with these smartphone-based monitors.

Limitations

There are several limitations to this study. The recordings made by each device were sequential rather than simultaneous, so the same complexes were not being compared. Furthermore, the assessments were performed on standard ECG graph paper at 25 mm/s (it was not possible to program the Kardia 6L monitor to a slower sweep speed) and this could have introduced some error. Nevertheless, the analysis suggests that at least for screening for QT abnormalities the handheld ECG performs well. This probably explains why the QTc measurements were worse because the heart rate added additional variability. The location of the third electrode on the Kardia 6L could have an impact on the ECG vectors. We performed the majority of the recordings with the third electrode placed on the thigh but used the left ankle in a small proportion of patients which could have introduced an error. We accept that the fact that the two ECG recordings look different could lead to observer bias. Unfortunately, it is not possible to remove this bias completely even if the 12-lead ECG was cut in half to show the limb leads only, as the paper and layout of the two outputs are different. Furthermore, we wanted to analyse the chest lead vectors as well and hence separating the 12 leads could have led to more errors.

Conclusion

In conclusion, in this study, we validated the 6L ECG against the 12 leads and found that several parameters were measured consistently including QT intervals. Handheld ECG monitoring is increasingly being used both by health care workers and patients. We have shown that the Kardia 6L performs well for many parameters including rhythm and QT analysis. The rapid development of handheld technology and accessibility is exciting, but it is essential that novel technology is assessed in large studies against the gold standard (i.e. 12-lead ECG) as the potential of harm to the patient from misdiagnosis is concerning especially if these devices do not perform well for certain analysis like ischaemia in the case of the Kardia 6L monitor.

Supplementary material

Supplementary material is available at European Heart Journal - Digital Health online.

Funding

N.A. is a Biosense Webster Research Fellow. M.H.T. has received an unrestricted educational grant from Biosense Webster.

Conflict of interest: none declared.

Data availability

All data is included in the submission/manuscript file.