Clinical Evaluation of the ECG Noise Extraction Tool as a Component of ECG Analysis Algorithms Evaluation

Abstract

Objective:

The aim of this work is to demonstrate the performance of the ECG noise extraction tool (ECGNExT) which provides estimates of ECG noise that are not significantly different from the inherent noise in an ECG generated by motion artifacts and other sources. In addition, this paper elaborates on use of ECGNExT in an algorithm evaluation context comparing two QRS detection algorithms.

Methods:

140 simultaneous pairs of clean ECGs and ECGs corrupted with motion-induced noise from 29 participants under five different and separate motion conditions were collected and analyzed. Estimates of the noise component of the ECGs recorded with noise were obtained using ECGNExT and were then added to the clean ECGs yielding estimated ECGs with noise. Root mean squared error (RMSE) between the recorded and estimated ECGs with noise was calculated for temporal comparison, and band powers of the signals were calculated for spectral comparison.

Results:

A t-test revealed that the mean RMSE < 150-microvolts with p-value < 0.001 and $\mathit{\alpha }=0.01$, and equivalence tests showed that the band powers of the two ECGs were statistically equivalent with $\mathit{\alpha }=0.01$.

Conclusion:

ECGNExT can reliably estimate the underlying ECG noise while preserving temporal and spectral features.

Significance:

We previously proposed ECGNExT as a component of ECG analysis algorithm testing during noise conditions and reported its performance based on simulated ECG data. This work provides additional support of the performance and functionality of the ECGNExT algorithm from a study with pairs of simultaneously recorded ECGs with and without noise from human subjects.

I. Introduction

Wearable electrocardiograph (ECG) devices for monitoring cardiac activity and detecting arrhythmias are increasingly used in clinical and home healthcare applications [1–4]. These devices have drawn attention [5, 6] due to their compact form factor that facilitates long-term ECG detection and recording [7]. However, a challenge in these devices is the noise that contaminates the ECGs at the point of detection, particularly motion-induced noise for wearable devices [8] that could lead to missed or false detection of significant cardiac events [9]. Depending on the location of a wearable device on the body and the coupling of its sensors with the skin, the sources of noise can be a combination of various factors including motion [8], [10], myoelectric activity [11], electrode-to-skin coupling/movement [12], and other sources.

The effects of noise in ECGs can lead to various performance degradations for analysis algorithms such as false or incorrect detection of events or arrhythmias [9, 13], inaccurate heart rate and heart-rate variability estimations [14], and other adverse outcomes that can impact the diagnosis of relevant cardiac conditions. To assess the performance of ECG processing algorithms, international standards (e.g., ANSI/AAMI EC57, ANSI/AAMI/IEC 60601-2-47) [15, 16] include certain tests to be conducted using previously recorded databases, such as the MIT-BIH Arrhythmia database [17–20].

Applying additive noise to annotated ECGs from previously recorded databases is one approach that can be used to assess impacts of noise on an arrhythmia analysis algorithm’s performance. A common set of signals for this is the MIT-BIH Noise Stress Test (NST) Database [21] available via PhysioNet [20]. These noise datasets include half-hour recordings of muscle artifacts, electrode motions, and baseline wander and can be added to clean ECGs to simulate a noisy ECG. However, due to device design differences for new wearable ECG devices such as anatomical sensor location (e.g., wrist [22] vs. chest [2]), skin-sensor interface differences (e.g., contact [2, 22] vs. non-contact capacitive [23] electrodes), sensor modalities (e.g., wet vs. dry [22] electrodes), or other characteristics (e.g., hardware and software impacting ECG acquisition, filtering, and processing) it may not always be clear if the simulated noisy ECGs using previously recorded datasets may be representative of the ECGs recorded by a device during realistic use conditions. Using the noise samples from databases with ECGs recorded with different sensor technologies may not account for variations due to the different factors.

Obtaining realistic estimates of underlying signal noise from ECGs recorded with the device that will be used can provide more representative noise signals to be added to clean ECGs for testing the algorithms associated with the device of interest. This approach provides the noise samples to be relevant to the application a device is designed for, but also accounts for variations stemming from differences in device uses (e.g., electrode technology, electrode location, patient activity, and other factors that may impact the noise/artifacts introduced on the ECG). This is possible because the noise estimates are obtained using the same sensing approach used by the device in question.

To that end, we previously proposed an ECG noise extraction tool (ECGNExT) to obtain samples of realistic noise from an ECG [24]. This tool provides a means to extract the noise present in an ECG recorded from any device, that can then be added to annotated reference ECGs to provide different signal-to-noise ratios (SNRs) for testing. In our previous work, the validation of the algorithm was limited to simulated ECGs where the noise was added from the NST database to those simulated ECGs.

In this paper, we present a clinical performance analysis of the algorithm proposed in [24]. We recorded simultaneous ECGs at two locations (wrist and collar bone) under certain motion conditions at the wrist site that allows for a comparison of the noise estimates from ECGNExT. In addition to showing that the noise estimated by the algorithm is representative of the underlying ECG noise that could impact the performance of an ECG processing algorithm, we also provide an example of utilization of the ECGNExT algorithm in the context of performance evaluation of two simple heartbeat detection algorithms. The performances of these heartbeat detection algorithms are evaluated against varying SNRs of the ECGs affected by varying amplitudes of noises estimated by the ECGNExT algorithm.

II. Methods

A. Algorithm Description

The ECG noise extraction tool (ECGNExT) estimates samples of noise from an ECG recorded during noise conditions. A detailed description is available in [24]. As illustrated in Fig. 1, the inputs of the algorithm used in this paper are a ‘noisy’ ECG $\left(EC{G}_{NOISY}\right)$, the bandpass-filtered version of $EC{G}_{NOISY}\left(EC{G}_{NOIS{Y}_{BP}}\right)$, and the position (indices) of the R peaks, marking each heartbeat. ECGNExT is designed to accept accurate QRS (R-peak) locations as an input. Here, we used the publicly available Pan-Tompkin’s algorithm [26] for detecting R-peaks. ECGNExT users can use more advanced R-peak detectors, or, for example, a separate (cleaner) ECG channel for R-peak detection [24]. $EC{G}_{NOIS{Y}_{BP}}$ is a band-pass [0.5 to 40 Hz] filtered version of $EC{G}_{NOISY}$ used to create a median-beat template. A residual signal, $EC{G}_{NOISY}$ residuals, expected to contain high-frequency and low-frequency is computed by subtracting $EC{G}_{NOIS{Y}_{BP}}$ from $EC{G}_{NOISY}$. The QRS estimator subsystem in the algorithm estimates a template of the PQRST complex of each beat by aligning the QRS complexes available in $EC{G}_{NOIS{Y}_{BP}}$ using the R peak indices and extracting the region around the R peak that includes the PQRST segments for every beat. For each R peak, this region starts at a distance of 1/3 of the smallest R-to-R interval (RRI) to the left of the R peak and ends at a distance of 2/3 of the smallest RRI to the right of the R peak. Finally, a median-beat template is created by obtaining a sample-by-sample median of the segments for each beat in a 60 s ECG record. Then the Synthetic ECG Generator subsystem generates a synthetic ECG by repeating the template at each of the input R peak indices, Fig. 1. Subtracting the Synthetic ECG from $EC{G}_{NOIS{Y}_{BP}}$ yields an initial estimate of the noise in the ECG, from which a period of +/− 40 ms around the R-peak is removed by the QRS Removal subsystem to mitigate any residual differences between the synthetic and recorded ECG for each beat. The resulting signal is added to $EC{G}_{NOISY}$ residuals from which QRS complexes are similarly removed by the same QRS removal subsystem. The result is $NOIS{E}_{EST}$, the final output of the algorithm, which contains a synthetically extracted noise signal similar to that impacting the original ECG. Further details of the process are available in our previous work [24] and MATLAB (The MathWorks, Inc.) code implementing the algorithm is available at: https://github.com/dbp-osel/ECGNExT [25].

Fig. 1:

Block diagram of the ECGNExT Algorithm.

B. Study Design

The study was approved under FDA IRB Study #2020-CDRH-007. Informed consent was received from 30 participants (22 male, 8 female, age: 41+/−9) who were then enrolled in the study. Consistent with the protocol, all participants were willing and able to perform a series of simple instructed motions, had no known cardiac conditions or active implantable devices, and were not pregnant. Gel electrodes were placed on the right collar bone, right wrist, and two on the left iliac bone for reference. Single-lead ECGs were simultaneously recorded from the right collar bone and right inner wrist sites using the left iliac bone electrodes as references. All recordings were performed using a two-channel bio amplifier (FE232) and a PowerLab 16/35 data acquisition unit from ADInstruments. The sampling rate was set to 1000 samples per second and the “Mains Filtering” feature of the unit (a filtering feature offered by the data acquisition software (LabChart8) to attenuate powerline noise) was set to active.

Participants were asked to sit still and calm in a chair and perform the following five independent and separate activities with their right hand for a duration of at least one-minute each.

1. Finger Tapping: Tapping their fingers on the chair arm where they were sitting.

2. Arm Waving: Waving their arm from elbow down.

3. Wrist Rotating: Rotations around the wrist area without moving the rest of the arm.

4. Electrode Tapping: Tapping on the outer electrode surface on the right wrist using their left hand.

5. Fist Making: Different activities involving making a fist and other finger motions.

Each activity was performed to isolate movement to the wrist location only. Pairs of ECGs from collar bone $\left(EC{G}_{CLEAN}\right)$ and wrist $\left(EC{G}_{NOISY}\right)$ locations were simultaneously recorded from each participant. $EC{G}_{CLEAN}$ is expected to be free from artifacts induced by the motion, while $EC{G}_{NOISY}$ is expected to contain artifacts induced by the motion. From these recordings, motion-induced artifacts can then be isolated from $EC{G}_{NOISY}$ by removing $EC{G}_{\mathit{CLEAN}}$. Each activity was preceded by one-minute periods without any motion for collecting baseline ECGs. Of the 30 participants one dataset did not cover the entire set of activities as listed above and was not included in the analysis. A total of 145 (5 activities * 29 participants) pairs of $EC{G}_{CLEAN}$ and $EC{G}_{NOISY}$ signals with lengths of 60 s were collected as the activities were being performed. While the data collection for each activity lasted for 60 s, 24 $EC{G}_{\mathit{CLEAN}}$ segments from 10 participants had significant noise and motion artifacts causing errors in R peak detection which are necessary for the study analysis. Of these 24 signals, 5 (4 from one participant and 1 from another) were determined to be unusable for the entire 60 s signal length and were not included in further analyses. This reduced the total ECGs studied to 140. At least 30 s of usable ECGs were identified from the remaining 19 corrupt $EC{G}_{CLEAN}$ signals and were included in the analyses. A breakdown of total number of ECGs analyzed for each activity is provided in Table I.

Table I:

Summary of ECG segments in each activity and number of outliers caused by $EC{G}_{\mathit{CLEAN}}$ and $EC{G}_{{\mathit{CLEAN}}_{BP}}$.

Activity Type	Act. Code	# Of ECGs	# Of ECGs < 60 s	# Of cases with RMSE > 150 μV
				$EC{G}_{\mathit{CLEAN}}$	$EC{G}_{{\mathit{CLEAN}}_{BP}}$
Finger Tap	ACT1	28	4	6	2
Arm Wave	ACT2	28	6	5	2
Wrist Rotate	ACT3	27	3	3	1
Electrode Tap	ACT4	28	3	3	2
Make Fist	ACT5	29	3	1	1
Total		140	19	18	8
No Motion	NoMot	145	12	10	3

Fifteen datasets (including one from the 14 shortened ECGs) had $EC{G}_{CLEAN}$ signals with significant noise and motion artifacts throughout their recording periods while their R peaks could still be detected. A sample noisy $EC{G}_{\mathit{CLEAN}}$ (left panel of Fig. 2) is compared with a noise-free $EC{G}_{CLEAN}$ (right panel of Fig. 2). Existence of levels of noise in $EC{G}_{\mathit{CLEAN}}$ such as what is depicted in the left panel of Fig. 2 prevents us from accurately investigating the error associated with $NOIS{E}_{EST}$. Therefore, we applied a band-pass filter with a pass band between 0.5 and 40 Hz to all $EC{G}_{CLEAN}$ signals. We denote the resulting band pass-filtered $EC{G}_{CLEAN}$ as $EC{G}_{CLEA{N}_{BP}}$. The results based on both scenarios of using $EC{G}_{CLEAN}$ and $EC{G}_{CLEA{N}_{BP}}$ in (1) are summarized in the tables of the Results section and in the Appendix, whereas the figures report results based on $EC{G}_{CLEA{N}_{BP}}$ scenario only unless otherwise indicated.

Fig. 2:

$EC{G}_{CLEAN}$ from two participants while performing finger tapping. The left resulted in poor R peak detection whereas the right is acceptable.

C. Extracted Noise Analysis

First R-peaks were detected from $EC{G}_{{\mathit{CLEAN}}_{BP}}$ using Pan-Tompkins QRS detection algorithm [26]. $EC{G}_{NOISY}$ signals along with the R-peak locations were passed through the ECGNExT algorithm to estimate the recorded noise and other induced motion artifacts. We then obtained $EC{G}_{NOIS{Y}_{EST}}$ as in (1) to compare it with $EC{G}_{NOISY}$.

$$EC{G}_{NOIS{Y}_{EST}}=EC{G}_{CLEAN}+{\text{NOISE}}_{EST}$$ (1)

For reference, see Fig. 3 for a 10-second sample of these signals collected during a finger tapping activity from one participant. From the top in the figure, panel 1 shows $EC{G}_{CLEAN}$, panel 2 shows $EC{G}_{NOISY}$, panel 3 shows $NOIS{E}_{EST}$, and panel 4 shows $EC{G}_{NOIS{Y}_{EST}}$.

Fig. 3:

Sample signals for one subject during finger tapping activity. (Top to bottom) panel 1 is a clean $ECG(EC{G}_{\mathit{CLEAN}}$), panel 2 is the recorded noisy $ECG\left(EC{G}_{NOISY}\right)$ from the wrist panel 3 is the estimated noise $(\left.{NOISE}_{EST}\right)$ from $EC{G}_{NOISY}$, and panel 4 is the estimated noisy $ECG\left(EC{G}_{NOI{Y}_{EST}}\right)$ obtained from adding $NOIS{E}_{EST}$ to the $EC{G}_{CLEAN}$.

To demonstrate that the estimated noise output from ECGNExT algorithm is representative of the motion artifact in the recorded ECG, we tested the hypothesis that the root mean squared error (RMSE) between $EC{G}_{NOISY}$ and $EC{G}_{NOIS{Y}_{EST}}$ as defined in (2) was less than 150 μV and 50 μV using a one-side T-test. These thresholds were selected from the ECG monitoring standard IEC 60601-1-27 for rejecting QRS complexes and internal noise allowed in ambulatory ECG per the standard IEC 60601-2-47, respectively [15].

$$RMSE=\sqrt{\frac{\sum _{i=1}^N{\left(EC{G}_{NOIS{Y}_i}-EC{G}_{NOIS{Y}_{ES{T}_i}}\right)}^2}{N}}$$ (2)

Here, $\mathrm{N}$ is the number of samples in an ECG segment recorded during an activity from each participant. Because the primary difference between $EC{G}_{CLEAN}$ and $EC{G}_{NOISY}$ is the presence of motion-induced artifacts, by subtracting $EC{G}_{CLEAN}$ from $EC{G}_{NOISY}$ we can generate a direct estimate of the recorded noise for evaluation of $NOIS{E}_{EST}$. We measured and visually compared the recorded noise $\left(NOIS{E}_{REC}\right)$ with $NOIS{E}_{EST}$, where $NOIS{E}_{REC}$ is the difference between $EC{G}_{NOISY}$ and $EC{G}_{CLEAN}$:

$$NOIS{E}_{REC}=EC{G}_{NOISY}-EC{G}_{CLEAN}$$ (3)

The difference between ${NOISE}_{EST}$ and ${NOISE}_{REC},NOIS{E}_{\mathit{diff}}$, as defined in (4) is also reported.

$$NOIS{E}_{\mathit{diff}}=NOIS{E}_{REC}-NOIS{E}_{EST}$$ (4)

The QRS complexes of $EC{G}_{\mathit{CLEAN}},EC{G}_{NOISY}$, and $NOIS{E}_{REC}$ were removed when used in (1) to (4) because $NOIS{E}_{EST}$ has its QRS removed as part of the ECGNExT algorithm.

To evaluate differences in spectral content between the estimated and recorded signals, spectral powers $P_1$ and $P_2$ were obtained for $EC{G}_{NOISY}$ and $EC{G}_{NOIS{Y}_{EST}}$, respectively. For this purpose, the signals were detrended after their means were removed, and then their power spectral densities were calculated based on Welch’s windowing method [27] using the MATLAB function pwelch.m with a window length of 30 s and 50% overlap. Finally, spectral powers P₁ and P₂ were obtained in three bands of 0.5 – 5 Hz, 5 – 25 Hz, and 25 – 40 Hz [28, 29] by calculating the area under the obtained power spectral density curves with widths defined by the mentioned frequency bands. $EC{G}_{NOIS{Y}_{EST}}$ here was based on $EC{G}_{{\mathit{CLEAN}}_{BP}}$ and QRS complexes were not removed from $EC{G}_{NOISY}$ and $EC{G}_{NOIS{Y}_{EST}}$ when obtaining P₁ and P₂. For the sample power curves in Fig. 8, the involved signals were first normalized based on their standard deviations before their powers were estimated, whereas the rest of frequency analysis was based on unnormalized signals. Please note that $EC{G}_{CLEAN}$ was used instead of $EC{G}_{CLEA{N}_{BP}}$ as a reference signal when plotting Fig. 8 to cover for all the frequency components involved. In contrast, $EC{G}_{CLEA{N}_{BP}}$ was used as a reference when showing aggregate powers of the signals involved in the mentioned three bands in the boxplots of Fig. 9 to be consistent with the rest of figures for time-domain analyses. The differences between $P_1$ and $P_2$, were then analyzed through an equivalence test with an equivalence interval (EI) as defined in (5) for means of $P_1$ and $P_2$ at $\alpha =0.01$ in the mentioned frequency bands:

$$EI=\left[\begin{array}{ll}{\delta }_L& {\delta }_U\end{array}\right]\left(\mathit{in}\mu V^2\right)$$ (5)

${\delta }_L$ and ${\delta }_U$ are the lower and upper bounds of the EI in (5) and used to define the alternative hypothesis as:

$${\delta }_L<{\mu }_{P_1}-{\mu }_{P_2}<{\delta }_U$$ (6)

where ${\mu }_{P_1}$ and ${\mu }_{P_2}$ are the means of $P_1$ and $P_2$, respectively. We used the two one-sided test (TOST) method implemented by a code provided in MathWorks^® Exchange repository [30]. The test result is a confidence interval (CI) and if it lies within the proposed EI, then the null hypothesis is rejected. In this study, we use the common standard deviation of $P_1$ and $P_2,\sigma$, to define ${\delta }_L$ and ${\delta }_U$ as $-0.36\sigma$ and $0.36\sigma$, respectively, as proposed in [31] with:

$$\sigma =\sqrt{\frac{\left(N_1-1\right){{\sigma }_{P_1}}^2+\left(N_2-1\right){{\sigma }_{P_2}}^2}{N_1+N_2-2}}$$ (7)

where ${\sigma }_{P_1}$ and ${\sigma }_{P_2}$ are the standard deviations of $P_1$ and $P_2$, respectively, with their corresponding number of samples denoted by $N_1$ and $N_2$.

Fig. 8:

Sample power curves comparing spectral characteristics of the $EC{G}_{NOISY}$ (dark grey dashed line) and $EC{G}_{NOIS{Y}_{EST}}$ (grey solid line).

Fig. 9:

Summary of absolute powers of $EC{G}_{NOISY}$ (black) and $EC{G}_{{\mathit{NOISY}}_{EST}}$ (grey) powers in three bands ordered from top to bottom as in Table III, grouped based on each activity.

When obtaining $P_1$ and $P_2$, we considered both cases of $EC{G}_{NOISY}$ and $EC{G}_{NOIS{Y}_{EST}}$ with and without QRS complexes. For the case of removing QRS, the QRS components of $EC{G}_{CLEAN}$ were first removed and then $NOIS{E}_{EST}$ was added to it to obtain $EC{G}_{NOIS{Y}_{EST}}$ per (1). In the case of $EC{G}_{NOIS{Y}_{EST}}$ with QRS, the $EC{G}_{\mathit{CLEAN}}$ was cut short to comply with the length of ${NOISE}_{EST}$ to allow addition per (1). These two cases were examined with both scenarios of using $EC{G}_{CLEAN}$ and $EC{G}_{CLEA{N}_{BP}}$ in (1). The outcomes of this equivalence test are summarized in the Results section for the only scenario where $EC{G}_{{\text{CLEAN}}_{BP}}$ is used in (1) without removing the QRS components of the resulting $EC{G}_{NOIS{Y}_{EST}}$ and the $EC{G}_{NOISY}$. The outcomes for other scenarios are summarized in the Appendix.

D. Sample Size Analysis

The number of participants for this study is based on simulated results from the previous work [24] where an RMSE of less than 50 μV was suggested based on the IEC 60601–2-47 standard [15]. We used “sampsizepwr.m” function from MATLAB to obtain the minimum number of datasets to ensure a power of 0.9 at $\alpha =0.01$ for the proposed alternative hypothesis. At least 76 ECG pairs were calculated based on these conditions. It was initially unclear how many of the activity periods would provide clean and noisy ECGs acceptable for our analysis but estimated that 3 segments could be obtained from each participant. Therefore, a minimum of 26 subjects was determined to be needed with additional subjects to be included in case of subject drop out.

E. ECGNExT in a Sample ECG Analysis Algorithm Evaluation Context

This section presents an example utilization of ECGNExT, mimicking a hypothetical design problem, where a developer needs to compare two available ECG analysis algorithms to determine which one performs better in the presence of noise. To that end, we consider two example heartbeat detection algorithms, Pan-Tompkins [26] and Perry [32] – a QRS detection algorithm available in the WFDB software package [33] of PhysioNet [20]. We evaluate their QRS detection performance in presence of the realistic noise estimates captured during this study. We used the 145 ${NOISE}_{EST}$ segments that were obtained using ECGNExT algorithm as described earlier. Adding each ${NOISE}_{EST}$ segment at a time with a single selected $EC{G}_{CLEAN}$ segment from the pool of ECGs collected during no-motion episodes yielded 145 SNR-controlled $EC{G}_{NOIS{Y}_{EST}}$ signals. These $EC{G}_{NOIS{Y}_{EST}}$ signals were then fed to both QRS detection algorithms at SNR levels varying from −12 dB to 12 dB in steps of 6 dB to evaluate their performances at each SNR level and with each $EC{G}_{NOIS{Y}_{EST}}$ signal. Fig. 4 illustrates the processes of obtaining the ${NOISE}_{EST}$ from $EC{G}_{NOISY}$ using ECGNExT algorithm, making the $EC{G}_{NOIS{Y}_{EST}}$, and performance evaluation of the QRS detection algorithms.

Fig. 4:

Workflow of algorithm evaluation using the noise estimates from ECGNExT algorithm.

Detection rate (in percentage), false-detection rate (in beats per minute (bpm)), and heart rate (HR) in bpm were used to compare the performances of the Pan-Tompkins and Perry algorithms at different SNRs. Detection rate was defined as number of correctly detected R peaks divided by the number of ground truth (GT) R peaks. Only one peak detected within 150 ms of each GT R peak was considered a correctly detected R peak per ECG standard EC57 clause 4.32 [16] and the rest were reported as false detections. Number of false detections within the first and last GT R peaks divided by the time between these two GT R peaks defines the false-detection rate in this paper. HR reported in this paper is the average of reciprocals of RRIs.

III. Results

A total of 140 ECGs from 29 participants were included in the analyses. Table I provides a breakdown of the number of recordings used from each activity after excluding some ECGs as described in the Methods. In addition, the table provides the number of ECGs that resulted in RMSEs greater than 150 microvolts when using $EC{G}_{CLEAN}$ versus $EC{G}_{{\mathit{CLEAN}}_{BP}}$ in (1).

A. Time Domain Results

Fig. 5 provides examples of $EC{G}_{NOISY}$ signals for no motion (topmost panel) and the rest of activity types in the rest of the panels as labeled in Table I. The signal related to fist making (ACT5) appears to be the noisiest. Random spikes in the signal affected by electrode tapping (ACT4) due to tapping events are also evident in the figure.

Fig. 5:

Sample $EC{G}_{NOISY}$ signals for each motion type as encoded in Table I.

The null hypothesis of having an RMSE of 150 μV was rejected with a p-value < 0.001 as we conducted the one-sided t-test with an $\alpha =0.01$, supporting that $EC{G}_{ERROR}$ remains below 150 μV on average. Table II provides a summary of the RMSE statistics based on the two scenarios of using $EC{G}_{CLEAN}$ and $EC{G}_{CLEA{N}_{BP}}$. Rows 1 and 3 are based on the case when $EC{G}_{\mathit{CLEAN}}$ was used in (1) for no motion and activity periods, respectively. Similarly, Rows 2 and 4 are based on the case when $EC{G}_{{\mathit{CLEAN}}_{BP}}$ was used instead in (1) for no motion and activity periods, respectively. The one-sided $\mathrm{t}$-test results are based on data summarized in the fourth row of the table.

Table II:

RMSE statistics for various scenarios of using the $EC{G}_{\mathit{CLEAN}}$.

No.	Reference ECG	RMSE Statistics (μV)		Activity
		Mean	STD
1	$EC{G}_{\mathit{CLEAN}}$	62.7	78.2	N
2	$EC{G}_{{\mathit{CLEAN}}_{BP}}$	34.6	66.7
3	$EC{G}_{\mathit{CLEAN}}$	78.4	84.6	Y
4	$EC{G}_{{\mathit{CLEAN}}_{BP}}$	56.2	96.5

Number of $EC{G}_{CLEA{N}_{BP}}$ signals causing the outliers is provided in Table I. We also performed the t-test with $RMSE=50\mu V$. However, this condition was too slim to reject the null hypothesis.

Fig. 3 shows sample signals involved in the process to observe the similarities between $EC{G}_{NOISY}$ (the second panel from the top) and $EC{G}_{NOIS{Y}_{EST}}$ (the fourth panel from the top). As can be seen, the effects of $NOIS{E}_{EST}$ (the third panel from the top) on the $EC{G}_{CLEAN}$ (the first panel from the top) is comparable to that of motion noise effects observed in the $EC{G}_{NOISY}$. This is despite the loss of time alignment between $EC{G}_{NOIS{Y}_{EST}}$ and $EC{G}_{NOISY}$ due to QRS removal which shows that the algorithm preserves the overall morphology of the underlying ECG noise.

Fig. 6 shows an overlay of a sample $EC{G}_{\text{NOISY}}$ after QRS removal only in contrast with $NOIS{E}_{EST}$. As can be seen, the effects of T-waves are attenuated in ${NOISE}_{EST}$ as opposed to $EC{G}_{NOISY}$ with only QRS removed. This illustrates an advantage offered by the algorithm which avoids altering the T-Waves in an ECG after adding the estimated noise for algorithm evaluation.

Fig. 6:

Overlay of $EC{G}_{NOISY}$ and $EC{G}_{NOIS{Y}_{EST}}$ for one subject during a finger tapping episode.

Fig. 7 shows boxplots of the RMS values of $NOIS{E}_{EST}$ and ${NOISE}_{REC}$ on the top panel, and the RMS values of ${NOISE}_{\mathit{diff}}$ on the bottom. The boxplots are based on data summarized on row 4 of Table II but grouped based on activity types. A good conformity can be seen between $NOIS{E}_{EST}$ and $NOIS{E}_{REC}$ from the top panel of the figure as $NOIS{E}_{EST}$ closely follows the $NOIS{E}_{REC}$ pattern in terms of mean values and the variations around the mean for each activity. The difference between $NOIS{E}_{EST}$ and $NOIS{E}_{REC}$, captured by $NOIS{E}_{\mathit{Diff}}$, remains around 50 μV on average for each activity (except arm waving) based on the bottom panel of the figure. The largest difference is displayed for Arm Wave activity in the bottom panel of the figure. However, that difference is because of the activity affecting the $EC{G}_{\mathit{CLEAN}}$ electrode which is influenced more by the Arm Wave activity than other activities and introduces additional noise in the comparison that is not part of the ${NOISE}_{\text{EST}}$ obtained from ECGNExT algorithm. Rather, those noises are part of the $EC{G}_{\mathit{CLEAN}}$ since in arm waving activity the noise could not be completely isolated to the wrist area.

Fig. 7:

RMS values for $NOIS{E}_{EST}$ in grey and $NOIS{E}_{REC}$ in black (top panel); RMS values for $NOIS{E}_{diff}$ (bottom panel).

B. Frequency Domain Results

Table III summarizes the power statistics of $EC{G}_{NOISY}$ and $EC{G}_{NOIS{Y}_{EST}}$ at three different frequency bands and indicates that the CI’s are within their corresponding EI’s. Comparing the obtained CI’s of the mean difference for each spectral band power as listed in Table III and their corresponding EI’s defined in (5), the null hypothesis can be rejected with an $\alpha =0.01$. This means that spectral characteristics of the $EC{G}_{NOISY}$ and $EC{G}_{NOIS{Y}_{EST}}$ are not significantly different on average for the listed frequency bands. This can be observed in the sample spectral plot in Fig. 8 as curves related to $EC{G}_{NOISY}$ (dashed dark grey) and $EC{G}_{NOIS{Y}_{EST}}$ (solid grey) are closely following each other. In addition, the aggregate spectral characteristics depicted in the boxplots of Fig. 9 suggest that both $EC{G}_{NOISY}$ and $EC{G}_{NOIS{T}_{EST}}$ exhibit similar patterns across all the study participants and for each activity for most of the case. $EC{G}_{CLEA{N}_{BP}}$ was used in (1) to obtain $EC{G}_{{\mathit{NOISY}}_{EST}}$ and the QRS components were not removed from $EC{G}_{NOISY}$ and $EC{G}_{NOIS{Y}_{EST}}$ when obtaining the power for the contents of Table III. Results of the other combinations described in II.C are provided in the Appendix.

Table III:

Summary of the spectral characteristics of $EC{G}_{\mathit{NOISY}}$ and $EC{G}_{{\mathit{NOISY}}_{EST}}$ signals.

No.	ECG	QRS	Frequency Bands	Power Statistics				CI × 105 (μV2)	EI × 105 (μV2)
				Mean × 105 (μV2)		STD × 105 (μV2)
				$EC{G}_{\mathit{NOISY}}$	$EC{G}_{{\mathit{NOISY}}_{EST}}$	$EC{G}_{\mathit{NOISY}}$	$EC{G}_{{\mathit{NOISY}}_{EST}}$
1	$EC{G}_{{\mathit{CLEAN}}_{BP}}$	Y	0.5 – 5 Hz	0.8814	0.9132	3.9565	3.8709	[−1.120 1.056]	[−1.4090 1.4090]
2			5 – 25 Hz	0.4133	0.4729	1.1114	1.0470	[−0.360 0.241]	[−0.3887 0.3887]
3			25 – 40 Hz	0.0256	0.0265	0.0571	0.0507	[−0.016 0.014]	[−0.0194 0.0194]

C. Algorithm Demo Test Results

Fig. 10 summarizes the performance of the two R-peak detection algorithms (Pan-Tompkins and Perry) when processing the $EC{G}_{NOIS{Y}_{EST}}$ with varying SNRs. Performance improvements against SNR can be observed in Fig. 10.

Fig. 10:

Performance metrics reported for Pan-Tompkins (black) and Perry (grey) QRS detection algorithms.

IV. Discussions

An RMSE, our estimate of difference between $EC{G}_{NOISY}$ and $EC{G}_{NOIS{Y}_{EST}}$, of less than 150 μV shows that the noise estimated by the ECGNExT algorithm (available online at: https://github.com/dbp-osel/ECGNExT [25]) are representative of the inherent noise in the ECGs collected under motion conditions. This means that a clean test ECG signal combined with noises estimated by ECGNExT algorithm can provide representative signals to an otherwise noisy ECG that an ECG analysis algorithm can be exposed to. It follows that relevant performance tests can be conducted on an ECG analysis algorithm using noise estimates from ECGNExT algorithm in conjunction with a clean ECG.

While an RMSE of 150 μV is a generous threshold, it needs to be emphasized that it allows some room for errors due to the inherent differences in ECGs collected from two different anatomical sites $-EC{G}_{NOISY}$ from the wrist vs. $EC{G}_{\mathit{CLEAN}}$ from the collar bone involved in $EC{G}_{NOIS{Y}_{EST}}$, and for any noise present in $EC{G}_{CLEAN}$. This is evident from the second row of Table II where an RMSE of up to about 100 μV for the no motion case alone is observed. This shows that part of the difference between $EC{G}_{NOISY}$ and $EC{G}_{NOIS{Y}_{EST}}$ in our analyses is due to differences in location of electrode placement. Comparing the second and fourth rows, there is an increase of 20 μV in mean RMSE as we introduce motion in the ECG collected from the wrist. This also implies that the differences in ECG collection sites contribute more in the RMSE than motion artifacts. Thus, the algorithm tends to produce repeatable output under motion and no motion conditions.

The larger difference between $NOIS{E}_{REC}$ and $NOIS{E}_{EST}$ observed in the bottom panel of Fig. 7 related to the Arm Wave activity is due to the increased noise in $EC{G}_{\mathit{CLEAN}}$. This is because the Arm Wave activity is less isolated to the wrist area and affects $EC{G}_{CLEAN}$ collected from the collar bone. Thus, using these $EC{G}_{CLEAN}$ segments to obtain $EC{G}_{NOIS{Y}_{EST}}$ results in increased amounts of noise from the $EC{G}_{\mathit{CLEAN}}$ which is not accounted for by the ECGNExT algorithm. On the other hand, the RMS Error between ${NOISE}_{REC}$ and ${NOISE}_{EST}$ (bottom panel of Fig. 7) for electrode tapping is relatively small despite the noise magnitude asserted by this activity (5^th panel from the top in Fig. 5, top panel of Fig. 7, and boxplots of Fig. 9). This is because this activity has a better isolation from the shoulder where $EC{G}_{CLEAN}$ is recorded and thus $NOIS{E}_{EST}$ from ECGNExT follows $NOIS{E}_{REC}$ very closely.

Frequency analysis of $EC{G}_{NOISY}$ and $EC{G}_{NOIS{Y}_{EST}}$ reveal similar outcomes. That is, the ECGNExT algorithm preserves the spectral components of the noise as shown by the equivalence testing. In addition, the power curves of Fig. 8 show that both samples from $EC{G}_{NOISY}$ and $EC{G}_{NOIS{Y}_{EST}}$ exhibit very similar patterns. Similarly, the boxplots of the aggregate powers in Fig. 9 capture the overall conformity of the spectral powers of $EC{G}_{NOIS{Y}_{EST}}$ to that of $EC{G}_{NOISY}$. The slight, but consistent increase in powers of $EC{G}_{NOIS{Y}_{EST}}$ can again be attributed to inherent differences in recording locations where noises in $EC{G}_{CLEAN}$ also contribute based on (1). This can further be supported by noting the same increases in the no motion case.

The response of ECG analysis algorithms to motion noise presented in III.C highlights the effects of ECGs with different SNRs on the performance of the algorithm in question. This also outlines a possible application of the ECGNExT algorithm in an ECG analysis algorithm evaluation context. Based on the plots of Fig. 10, the performances of the two sample algorithms improve as the ECG SNR increases. While the performance improvement is not necessarily linear as the SNR is increased, the overall pattern shows an improving trend in the device performance as the SNR increases. Here, the ECGNExT algorithm facilitates a comparative study where a realistic noise relevant to the device application is made available using the algorithm that helps derive application-relevant insights. For example, we can observe that the Pan-Tompkins algorithm is more robust to higher levels of noise as opposed to Perry algorithm. Although the former presents higher false-detection rates, it reaches to the levels of false-detection rates from the latter at the SNR of 0dB. This, combined with the better detection rate offered by the Pan-Tompkins algorithm demonstrates overall better performance over Perry algorithm under the same ECG noise conditions. On the other hand, Perry algorithm tends to demonstrate better HR estimation performance owing to its lower false-detection rates even under higher levels of noise, for example.

One of the main advantages of the ECGNExT algorithm is that it estimates representative noise that can be collected under similar conditions that an ECG device will be used. This allows for comprehensive tests without burdensome ECG collection from target patients populations an algorithm under test is intended for. This is feasible by using the ECGNExT algorithm to obtain realistic noise estimates from healthy population which then can be added to annotated reference ECGs containing pathologies that a device is expected to encounter in clinical use. In addition, this algorithm can extensively remove morphological characteristics of the ECG (e.g., T-wave effects) that the noises can carry from the source ECG. This feature helps avoid the adverse effects that those ECG components may have on an algorithm being evaluated using noise estimates from the ECGNExT algorithm. Such adverse effects are prevented because the internal mechanism of the ECGNExT algorithm attenuates most of the morphological features related to physiological characteristics of the underlying noisy ECG.

One of the main limitations of this study is the challenge of noise in the reference ECG, $EC{G}_{CLEAN}$, that could not be fully avoided in the study. Our method extracts any noise from $EC{G}_{NOISY}$, and the error quantification used in this paper assumes $EC{G}_{CLEAN}$ to be free from any noise. However, any noise included in $EC{G}_{CLEAN}$ would be measured as error, even in the hypothesis that the ECGNExT algorithm would work perfectly. This creates uncertainty in the evaluation of the results, as it is not possible to discriminate errors due to incorrect ECGNExT output from those apparent errors due to noise in $EC{G}_{CLEAN}$. As a mitigation effort we excluded recordings that showed significant noise in $EC{G}_{CLEAN}$, and used the band-pass filtered $EC{G}_{CLEAN},EC{G}_{CLEA{N}_{BP}}$, instead of $EC{G}_{CLEAN}$ in (1) because $EC{G}_{CLEAN}$ in this equation needs to be ideally clean so that $EC{G}_{NOIS{Y}_{EST}}$ would not carry artifacts from $EC{G}_{CLEAN}$. That is, $EC{G}_{CLEA{N}_{BP}}$ helps get ECGs closer to ideally clean ECGs. While $EC{G}_{CLEA{N}_{BP}}$ helped significantly reduce the number of RMSE outliers, a few outliers remain. This is due to noises that remain in $EC{G}_{CLEA{N}_{BP}}$ that do not appear in $EC{G}_{NOISY}$, and in some cases, due to a filtering artifact that introduces an end-effect anomaly to $EC{G}_{CLEAN}$ that is not present in either $EC{G}_{CLEAN}$ or $EC{G}_{NOISY}$.

Another limitation is that the motion artifacts introduced at the wrist ECG may not be inclusive of the broad range of all ECG noise scenarios. Since the focus of the study is to evaluate noise that is at a similar amplitude range to QRS complex to challenge a given algorithm within their detection limits, certain body movements such as walking, running, sitting-standing, or other whole-body movements were not considered in our study. This is because they would exhibit overwhelming levels of noise leading to QRS characteristics not being identified. Rather, the artifacts considered here are representative signal patterns influencing the ECG. This was intentionally limited to these representative artifacts to avoid losing QRS complexes entirely due to factors such as saturation and/or other noise amplitude effects that are beyond the limits of what the ECG analysis algorithms may detect. In addition, a reference clean ECG as an evaluation baseline needed for the purpose of this paper could not be maintained under these whole-body motion conditions. The excluded forms of motions may be relevant for designing other studies to apply ECGNExT for studying a device under different scenarios where they may be used in conjunction with their relevant motion conditions.

Finally, while the study was conducted on up to 60 s-long signals, the method could be implemented in a windowing approach to apply the algorithm over longer signals.

V. Conclusion

A performance evaluation of the ECGNExT algorithm was presented through comparative analyses with simultaneously recorded ECGs with and without noise. The analyses compared both temporal and spectral characteristics of a noisy ECG to that of a clean ECG contaminated with noise estimates from the same noisy ECG where the noise estimates were obtained using the ECGNExT algorithm. The results indicate that the ECGNExT algorithm can estimate the noises with sufficient similarity to the actual underlying noise in an ECG enabling an effective and modernized approach for use when evaluating ECG analysis algorithms robustness to noise. A realistic noise estimate obtained using the ECGNExT algorithm supports efficient test methods with clinical relevance as visualized in the sample comparative study of QRS detection performance evaluation.

Appendix

The table below provides the remaining results of the spectral analysis of the $EC{G}_{NOISY}$ and $EC{G}_{NOIS{Y}_{EST}}$ that were not included in Table III. These details are for the remaining scenarios that were examined and described in II.C.

No.	ECG	QRS	Frequency Bands	Power Statistics				CI × 105 (μV2)	EI × 105 (μV2)
				Mean × 105 (μV2)		STD × 105 (μV2)
				$EC{G}_{\mathit{NOISY}}$	$EC{G}_{{\mathit{NOISY}}_{EST}}$	$EC{G}_{\mathit{NOISY}}$	$EC{G}_{{\mathit{NOISY}}_{EST}}$
1	$EC{G}_{\mathit{CLEAN}}$	N	0.5 – 5 Hz	0.8134	0.9683	3.9739	3.8414	[−1.242 0.932]	[−1.4070 1.4070]
2			5 – 25 Hz	0.2099	0.2178	1.0436	1.0377	[−0.297 0.281]	[−0.3746 0.3746]
3			25 – 40 Hz	0.0127	0.0136	0.0507	0.0508	[−0.015 0.013]	[−0.0183 0.0183]
4		Y	0.5 – 5 Hz	0.8814	0.9546	3.9565	3.8746	[−1.162 1.016]	[−1.4097 1.4097]
5			5 – 25 Hz	0.4133	0.4731	1.1114	1.0470	[−0.360 0.240]	[−0.3887 0.3887]
6			25 – 40 Hz	0.0256	0.0286	0.0571	0.0509	[−0.018 0.012]	[−0.0195 0.0195]
7	$EC{G}_{{\mathit{CLEAN}}_{BP}}$	N	0.5 – 5 Hz	0.8134	0.8713	3.9739	3.8260	[−1.143 1.027]	[−1.4042 1.4042]
8			5 – 25 Hz	0.2099	0.2178	1.0436	1.0377	[−0.297 0.281]	[−0.3746 0.3746]
9			25 – 40 Hz	0.0127	0.0134	0.0507	0.0507	[−0.015 0.013]	[−0.0182 0.0182]