Overcoming Barriers to Quantification and Comparison of Electrocardiographic Imaging Methods: A Community-Based Approach

Abstract

There has been a recent upsurge in the development of electrocardiographic imaging (ECGI) methods, along with a significant increase in clinical application. To better assess the state-of-the-art, enable reliable progress, and facilitate clinical adoption, it is important to be able to compare results in a comprehensive manner, scientifically and clinically. However, studies vary in modeling choices, computational methods, validation mechanisms and metrics, and clinical applications, making unified evaluation and comparison of ECGI a critical challenge.

This paper describes initial results of a project to address this challenge via a community-based approach organized by the Consortium for Electrocardiographic Imaging (CEI). We detail different aspects of this collective effort including a data sharing repository, a platform for comparison of different algorithms and modeling approaches on the same datasets, several active workgroups and progress made along these directions. We also summarize the results from groups participating in this collaboration and contributing solutions by applying their methods to the same dataset for comparison.

1. Introduction

Electrocardiographic imaging (ECGI) involves the development of methods to non-invasively image the electrical activity of the heart from electrocardiographic (ECG) data. ECGI has gained increasing attention from both research and industry communities due to its potential for clinical application in the diagnosis and treatment planning of cardiac arrhythmias, such as atrial fibrillation and premature ventricular contractions (PVC). However, its application to clinical practice has been limited in part by the lack of consensus on the elements that form the ECGI pipeline. Each research study reports on a different choice of geometry segmentation approaches, source models and computational methods. Moreover, every method is typically tested on a newly acquired set of data, using different validation metrics and with a different clinical application in mind. This lack of standardization and unbiased comparison is one of the major challenges to the progress of ECGI. To overcome this limitation, a group of researchers in the ECGI community formed the Consortium for Electrocardiographic Imaging (CEI). The first CEI meeting was held around the Computing in Cardiology Conference (CinC) in 2014. Since then, this community-based effort has continued and the CEI has grown in both the number of members and the resources available.

The objective of CEI is to facilitate collaboration and sharing among groups in the ECGI community to promote standardization. We do so through four main efforts: facilitating communication and dissemination, the creation of a common data repository, the organization of hackathon events and the ongoing workgroups on three ECGI topics. A website is also maintained, www.ecg-imaging.org, with information about current events, lists of publications and access to all our community efforts.

CEI hosts the EDGAR data repository, an ECGI data collection that currently contains 15 datasets including data from simulations, animal experiments and human mapping recordings [1]. The dataset includes a geometric model after various stages of processing as well as associated electrode data. The EDGAR database has been recently improved with a new user interface that can be accessed online at edgar.sci.utah.edu. We encourage all researchers in the ECGI community to use these datasets in their work and to contribute new datasets to EDGAR.

Similarly, in order to promote the discussion, comparison and sharing of software solutions, CEI continues organizing hackathon events. The last one happened during Computing in Cardiology 2017 in Rennes, France, with the general topic of pre- and post-processing techniques for ECGI data. It included subtopics ranging from baseline wander removal, beat averaging, estimation of activation times and QRST complex removal for imaging of atrial fibrillation.

In order to collectively study specific ECGI components, CEI has formed three working groups: the Forward Modeling Workgroup, the PVC Reconstruction Work-group and Atrial Arrhythmias Workgroup. The Forward Modeling and the PVC workgroups have established collaborative efforts using COVALIC, a web-platform to collect submissions from participants and compare them using common metrics.

In this paper, we first describe the workgroups that are being developed within CEI, then we describe the online platform that enables them and the latest progress that has been posted on it. Finally, we give a few examples of how this community approach is beginning to provide insights into the ECGI state-of-the-art.

2. The CEI Workgroups

Forward Modeling Workgroup

Evaluation of variability within the forward modeling pipeline in ECGI is necessary since each group has their own set of tools, techniques and best practices to develop forward models. These varied approaches can produce considerable variability in the ECGI results, yet there has not been an effort to characterize its effects. The aim of the Forward Modeling workgroup is to characterize the variation and uncertainty in each stage of the forward modeling pipeline using common geometries processed by different research groups. We have compared the agreement among segmentations of a patient based on submissions from multiple groups with the COVALIC system and are expanding the analysis to multiple subjects.

Pre-ventricular Contraction Workgroup

The workgroup on PVC focuses on the inverse solutions within the ECGI pipeline. Its particular application, localization of PVCs, has been selected since it is the application of ECGI with most available data and most progresses in recent years. The PVC workgroup compares solutions obtained with different methods and groups on a single dataset. We have recently opened up its COVALIC interface and the first participants have started posting their results.

Atrial Arrhythmias Workgroup

The atrial arrhythmias workgroup has available data of both human and simulation cases with atrial fibrillation that is now available at the EDGAR repository. The aim of the workgroup is to use this dataset to test the performance metrics and resolution of inverse methods used in the ECGI community for atrial arrhythmia.

Since their launch during the 2016 CinC Conference in Vancouver, Canada, the primary objective of the atrial arrhythmias workgroup has been to write a consensus paper that summarizes the current state of the art in functional imaging of atrial arrhythmias and to establish a consensus regarding future directions and perspectives.

3. The COVALIC Platform

To enable the workgroup efforts, we have partnered with Kitware Inc., and adapted their COVALIC platform to the needs of the workgroups. This platform, available at https://challenge.kitware.com, allows individuals to download a test dataset, run their methods and upload the corresponding results for evaluation with predetermined metrics. COVALIC was originally designed for competitive challenges and can automatically score and compare submissions and post them online. In the CEI structure each workgroup, or “challenge”, is separated into different phases that compare different steps/aspects of the work-group objectives, e.g. different datasets in the Forward Modeling workgroup and different source models for the PVC workgroup. Each submission is then scored based on the metrics relevant to each problem. Since our primary objective is collaborative rather than competitive, we have been reinterpreting some of the COVALIC terminologies and working with Kitware to make some modifications to the platform.

Forward Modeling Workgroup

In the forward modeling workgroup, we are currently working on quantifying the variation of tissue segmentation, the first step of the model building process used in ECGI. We have set up a COVALIC challenge to receive and compare segmentations from multiple research groups of data from three subjects. Currently, the three datasets are available and several submissions have already been received for one of them [2]. Fig. 1 presents comparison of segmentations from four different research groups of the ventricles and torso of one subject.

Figure 1.

Forward Modeling challenge results. Comparison of 5 segmentations of the myocardium and the torso surface. The solid colors indicate the aggregate segmentation and the outlines each of the segmentations.

PVC Workgroup

The most significant advance of the PVC reconstruction workgroup is that the COVALIC platform is up and ready to receive submissions from all participants. The first comparison will be on a simulation dataset provided by the Karlsruhe Institute of Technology and available in EDGAR [3]. It was generated with a cellular automaton and contains multiple heartbeats paced at different locations. This dataset also includes ground truth and forward models for multiple source models: extracellular potentials on a pericardial surface, extracellular potentials on an endocardial+epicardial surface, transmembrane potentials on an endocardial+epicardial surface and also transmembrane potentials in a volumetric mesh.

The ground truth consists of potentials, activation times and coordinates of the pacing site. Participants have the flexibility to compare their results in all the forms of ground truth or limit their contribution to a subset. To obtain a numerical comparison, we have currently implemented the basic metrics of relative error, correlation between true and reconstructed potential maps, correlation between activation times and localization error of the pacing site. Once we have enough submissions in the synthetic scenario, we will open the “challenge” to real mapping experiments associated to this same geometry.

4. Current Results

4.1. Forward Modeling Workgroup

We received four segmentations of the ventricles and torso (from University of Utah, Inria Research Center at Bordeaux, Inria Research Center at Sophia Asclepios, Nijmegen University, and Oregon Health and Science University) for one of the subjects. Additionally, two groups submitted segmentations for the lungs. Each tissue segmentation was evaluated by generating an aggregate segmentation using the STAPLE algorithm [4], and then comparing each segmentation to the aggregate using the DICE coefficient.

The DICE coefficient of each of the submitted segmentations were 0.74, 0.74, 0.79, 0.69, and 0.85 for the ventricles, 1.0, 0.99, 0.98, 0.99, and 1.0 for the torso, 0.94 and 0.97 for the right lung, and 0.93 and 0.97 for the left lung. The lower DICE coefficients in the ventricles indicate that there is much less agreement in the segmentation of the ventricles than the torso. The specificity of each of the ventricular segmentations was virtually identical, though this measurement may be skewed by the relatively low number of myocardial pixels in the total scan. The sensitivity of each of segmentations roughly corresponded to the correlation compared to the aggregate segmentation. Segmentations of the ventricles with a higher sensitivity segmented a larger region for the ventricles.

Qualitative comparison of segmentation can be seen in Figure 1 and indicates that the biggest differences in the segmentation is the choice of where to separate the atria and ventricles. This creates some discrepancies around the valves and in the outflow track regions. Another area of variability is the right lateral and posterior free wall and the right ventricle apex. Furthermore, there were some differences in the two submitted segmentations of lungs, especially in the region anterior to the heart. These areas of high variability may produce more differences in the forward model pipeline and the subsequent ECGI calculations.

Further work is needed to evaluate the effect of these segmentation variations on the subsequent forward model. To test the effect of the variation, we will have various research groups continue the modeling pipeline with the aggregate segmentation and with the submitted segmentations. To do so, we will first continue with the next stage of the modeling pipeline: generating a computation mesh.

4.2. Premature Ventricular Contraction Reconstruction Workgroup

At the time of writing this report, solutions using four different inverse methods applied to all source models have been posted. These submissions already exemplify how this repository can be used for comparison of ECGI solutions. Of course, the conclusions that can be drawn are limited and a better analysis will require the contribution from more groups. We encourage everyone working on new ECGI algorithms for PVC localization to submit their results and work together with the objective of writing a joint consensus paper.

We show results for the two extracellular source models posted in COVALIC in Figure 2. The four plots in this figure correspond to the metrics currently evaluated in the PVC workgroup: relative error of potentials, temporal correlation of potentials, correlation of activation times and error in localization of the PVC site. Within each plot, we show the distribution of the results using the extracellular source models on the pericardial geometry (in blue) and the endocardial+epicardial geometry (orange). Each black point is an individual solution, the red dots correspond to the group means and the shaded areas the distribution estimate of the solutions in each source model.

Figure 2.

PVC challenge results. Comparison of four inverse methods applied to the pericardial (blue) and endocardial+epicardial (orange) geometries with an extracellular source model.

The first observation that can be drawn from these plots is that the quality of the results drops when the methods are applied to the endocardial+epicardial geometries. The increase in variance is most dramatic in the relative error, but a similar effect can be seen in the other metrics.

Across both geometries, 0th order Tikhonov is the inverse method that provides better results overall. The increase in error variance observed in the endocardial+epicardial geometry is slightly more pronounced for 0th order Tikhonov although it remains the best performing algorithm. This result suggests that the added regularization in the 2nd order Tikhonov and splines helps stabilize the solutions when the inverse problem is more illposed but it is too strong to allow for accurate results.

The final observation is that both relative error and the temporal correlation of potentials do not match the results in pacing site localization, which is the ultimate clinical goal. This observation suggests that the error metrics applied the reconstructed potentials are not valid metrics to evaluate the quality of the results.

5. Conclusions

CEI continues developing tools to facilitate the collaboration and sharing of data and software solutions among the ECGI research community. The latest developments include the online COVALIC system with the Forward Models and PVC Localization workgroups. Some groups have already started submitting their solutions. However these efforts require the widespread collaboration of members in the ECGI community in order to have an impact. We encourage all researchers on ECGI to participate in the workgroups, hackathons and share data and software through CEI.