Noninvasive Fetal ECG: the PhysioNet/Computing in Cardiology Challenge 2013

Abstract

The PhysioNet/CinC 2013 Challenge aimed to stimulate rapid development and improvement of software for estimating fetal heart rate (FHR), fetal interbeat intervals (FRR), and fetal QT intervals (FQT), from multichannel recordings made using electrodes placed on the mother’s abdomen. For the challenge, five data collections from a variety of sources were used to compile a large standardized database, which was divided into training, open test, and hidden test subsets. Gold-standard fetal QRS and QT interval annotations were developed using a novel crowd-sourcing framework.

The challenge organizers used the hidden test subset to evaluate 91 open-source software entries submitted by 53 international teams of participants in three challenge events, estimating FHR, FRR, and FQT using the hidden test subset, which was not available for study by participants. Two additional events required only user-submitted QRS annotations to evaluate FHR and FRR estimation accuracy using the open test subset available to participants.

The challenge yielded a total of 91 open-source software entries. The best of these achieved average estimation errors of 187bpm² for FHR, 20.9 ms for FRR, and 152.7 ms for FQT. The open data sets, scoring software, and open-source entries are available at PhysioNet for researchers interested on working on these problems.

1. Introduction

Since the late 19th century, decelerations of fetal heart rate (FHR) have been known to be associated with fetal distress. The 14th annual PhysioNet/Computing in Cardiology Challenge was aimed at accelerating the development of accurate algorithms for locating fetal QRS (FQRS) complexes and estimating fetal QT intervals in noninvasive fetal electrocardiograms (FECGs), acquired using electrodes placed on the mother’s abdomen. Unlike direct FECGs obtained using a fetal scalp electrode, noninvasive FECGs can be observed throughout the second half of pregnancy with negligible risk, but it is often difficult to detect fetal QRS complexes in non-invasive FECGs, since maternal QRSs are usually of greater amplitude.

Beyond FHR, features such as FHR variability and fetal QT interval may be useful independent indicators of fetal status. There are no accepted techniques for assessing such features from non-invasive FECGs, however. Multiple challenge events were designed to test basic FHR estimation accuracy, as well as accuracy in measurement of inter-beat (RR) and QT intervals needed as a basis for derivation of other FECG features. The PhysioNet/CinC 2013 Challenge was therefore conducted as five different events:

Events 1 and 2: Open-Source FHR and FRR Estimation

Each open-source entry produced a set of estimated R-peak locations that was used to construct an FHR (Event 1) and an FRR (Event 2) time series for each recording in the hidden test set.

Event 3: Open-Source FQT Estimation

Each open-source entry produced an estimate of the median QT interval for each recording in the hidden test set.

Events 4 and 5: FHR and FRR Estimation

Participants submitted a set of estimated R-peak locations that was used to construct an FHR (Event 4) and an FRR (Event 5) time series for each recording in the open test set.

2. Challenge data

Data for the challenge consist of a collection of one-minute, four-channel non-invasive FECGs sampled at 1 kHz. The data were obtained from multiple sources using a variety of instrumentation with differing frequency response, resolution, and configuration. The 447 records used in the challenge were drawn from five data collections (Table 1).

Table 1.

FECG database reference.

Database Name	N records
Abdominal and Direct FECG [1]	25
Artificially Generated FECGs [2]	20
Non-Invasive FECG [3]	14
Ukraine Non-Invasive FECG	340
Private Scalp FECG Database	48

The 447 records were partitioned into three subsets. Training set A contains 75 records; its reference annotations were provided to the participants. Open test set B contains 100 records; its signals, but not its reference annotations, were available for study by the participants. Hidden test set C contains the remaining 272 records; it was withheld from the competitors for the purpose of testing the open-source entries (Events 1, 2, and 3). Sets A and B, and the reference annotations for set A, remain freely available [4]. Figure 1 shows a short excerpt of a record from set A.

Figure 1.

A four-second excerpt of a one-minute record from the challenge training set; the red circles mark the locations of the fetal QRS complexes, coinciding in three cases with the larger-amplitude maternal QRS complexes.

2.1. Reference annotations

The reference annotations used for the challenge were revised during the competition. During the first phase of the challenge, preliminary scores were based on reference QRS annotations derived by the authors of the respective data sets. During the later stages of the challenge, the original annotations were replaced by a set of crowd-sourced reference annotations, derived by applying a novel probabilistic voting algorithm [5, 6] to multiple sets of manual annotations and those made by the open-source entries in Phase 1. For Event 3, reference QT durations were also derived by applying the probabilistic voting algorithm to manual annotations done by seven observers.

3. Scoring criteria

Entries were scored using tach, mxm, and ann2rr (components of the WFDB software package [7]) and custom software developed for the challenge. The software processed the reference annotations and the test annotations (i.e., those generated by the participant’s entry) to obtain a score for each test record, then calculated an average score for all records in the test set.

3.1. FHR time series estimation (Events 1 and 4)

Reference and test FHR time series were derived from the reference and test FQRS annotations by tach, which used the IPFM method [8] to obtain 12 uniformly sampled and smoothed instantaneous FHR estimates for each record. The score for each record was the mean squared error (in bpm²) between the fetal heart rate signals estimated from the reference and test annotations. (For records with no test annotations, the score was 8000.) Any test annotations that preceded the first reference QRS, and any that followed the last reference QRS, were ignored. The final aggregate score for the event was the average score for all records.

3.2. FRR time series estimation (Events 2 and 5)

Reference and test FRR time series were obtained from the reference and test FQRS annotations using ann2rr. Scoring for Events 2 and 5 was performed by comparing these time series using mxm, which derived the root mean square difference (in milliseconds) between corresponding RR intervals. (For records with no test FRR intervals, the score was 200.) Any test RR intervals that preceded the first reference RR, and any that followed the last reference RR, were ignored. The final aggregate score for the event was the average score for all records.

3.3. FQT interval estimation (Event 3)

Scoring for Event 3 was the root mean square difference between the reference and test FQT intervals for all records, in milliseconds.

4. Results

Participants submitted a total of 208 sets of FQRS annotations for Events 4 and 5, and 91 open-source entries for Events 1, 2, and 3 during the challenge period. Results for events 1, 2, 4 and 5 are shown in Fig. 2. Table 2 shows the top scores and the teams who achieved them for all five events.

Figure 2.

Scores for events 1,2, 4 and 5.

Table 2.

Top scores for Challenge events. Results in parentheses are from unofficial participants.

Event	Winner	Score
Event 1	Varanini [9] (Behar [10])	187.091 (179.439)
Event 2	Varanini [9] (Behar [10])	20.975 (20.793)
Event 3	Podziemski [11]	152.71
Event 4	Andreotti [12]	18.083
Event 5	Andreotti [12]	4.337

5. Discussion

The PhysioNet/CinC 2013 challenge attracted a total of 53 teams attempting non-invasive extraction of fetal ECG information from maternal abdominal leads. Most teams employed on a two-step approach, where the first step typically consisted of the removal of the maternal QRS, followed by a second step with the aim of the extracting the fetal QRS. The removal of the maternal component was achieved using techniques that included subspace decomposition or reconstruction [13–20], adaptive filtering/averaging [12, 21–25], de-noising via wavelets [26–30], and a fusion of several approaches [10]. The final step of fetal QRS detection used a variety of approaches including matched filtering [14,15,31], Christov’s beat detection [32], entropy [14, 33], RS slope [11], expectation weighting [34], echo state recurrent neural network [35], or fusion of multiple methods [36].

Interestingly, most of the top-performing entries in the challenge made use of strategies that differentiated them from their competitors. For instance, Behar et al [10] instead of relying on a single technique for maternal ECG extraction, decided to implement a fusion of several different extraction methods. Another successful approach was the use adaptive QRS templates for the either the mother, fetus, or both; thus allowing for realistic non-stationary conditions [12].