Hybrid Neural State-Space Modeling for Supervised and Unsupervised Electrocardiographic Imaging

Abstract

State-space modeling (SSM) provides a general framework for many image reconstruction tasks. Error in a priori physiological knowledge of the imaging physics, can bring incorrectness to solutions. Modern deep-learning approaches show great promise but lack interpretability and rely on large amounts of labeled data. In this paper, we present a novel hybrid SSM framework for electrocardiographic imaging (ECGI) to leverage the advantage of state-space formulations in data-driven learning. We first leverage the physics-based forward operator to supervise the learning. We then introduce neural modeling of the transition function and the associated Bayesian filtering strategy. We applied the hybrid SSM framework to reconstruct electrical activity on the heart surface from body-surface potentials. In unsupervised settings of both in-silico and in-vivo data without cardiac electrical activity as the ground truth to supervise the learning, we demonstrated improved ECGI performances of the hybrid SSM framework trained from a small number of ECG observations in comparison to the fixed SSM. We further demonstrated that, when in-silico simulation data becomes available, mixed supervised and unsupervised training of the hybrid SSM achieved a further 40.6% and 45.6% improvements, respectively, in comparison to traditional ECGI baselines and supervised data-driven ECGI baselines for localizing the origin of ventricular activations in real data.

I. Introduction

Electrocardiographic imaging (ECGI), in analogy to tomographic imaging, aims to computationally reconstruct electrical potential signals of the heart from non-invasive external observations (i.e., body-surface ECGs) [1]. The forward imaging physics between the cardiac sources and their ECG measurements is relatively well-understood. Given heart and thorax geometrical meshes, the forward operator can be numerically constructed by solving the governing equations for quasi-static electromagnetic fields [2]. The inference/optimization of the inverse solutions given ECG measurements, however, is challenging due to the ill-posedness of the problem. A significant amount of effort in ECGI literature has focused on regularization techniques to leverage prior knowledge of cardiac electrical activity. Examples include spatial and/or temporal smoothness [3], [4] or sparsity [5], [6] of the solution, as well as a priori physiological knowledge about the spatiotemporal dynamics of the solution [1], [7], [8]. Many of these existing works can be represented as special cases within the general framework of state-space modeling (SSM) [9], where the prior knowledge can be expressed in a dynamic transition function that makes a prediction of the system state to constrain the inverse solution. Solving a maximum a posteriori (MAP) optimization, this prediction can then be corrected by the actual observations (i.e., ECG data) to obtain an optimal reconstruction of the system state. Such a framework provides flexibility in explicitly incorporating various spatiotemporal knowledge about the system, such as complex partial differential equations (PDEs) describing the electrical dynamics of the cardiac system [1].

However, the traditional SSM framework faces several challenges. First, the reliance on strong structural and parameter assumptions in the state-space functions – although representing general knowledge about the systems – introduces potential errors when applied to individual subjects. Second, state-space Bayesian filtering is typically applied independently to each set of ECG observations to estimate the corresponding cardiac electrical activity, while the prior knowledge as described in the state-space functions is kept fixed. Even if information across multiple sets of ECG data from the same subject may indicate errors in the prior knowledge, they are not assimilated to refine the state-space functions for improving future estimations.

In recent years, traditional ECGI approaches have been increasingly replaced by data-driven machine-learning approaches that extract information from data without needing sophisticated prior knowledge or physics-based modeling [10]–[12]. Its blackbox model however lacks interpretability, and training requires large amounts of labeled data – in the form of electrical activity measured on the heart surface – that are typically not available in the real world. While it is clear that SSM frameworks naturally complement these limitations, direct integration of traditional SSM framework (with fixed state-space functions) with data-driven learning is challenging for ECGI and the attempt has been limited.

In this paper, we propose a novel hybrid SSM framework for ECGI to leverage the advantage of state-space formulations in data-driven learning, with partially-learnable state-space functions and unsupervised Bayesian filtering strategies. We follow structured state-space modeling to leverage the physics-based forward emission function, such that the learning is supervised by the known physics free from the need of labeled data of cardiac electrical activity. In the meantime, instead of completely fixed state-space functions, we introduce neural modeling of the dynamic transition function (to predict the signal propagation in the heart) and the associated Bayesian filtering strategy. In an initial publication [13], we demonstrated the proof-of-concept of learning this hybrid SSM in an unsupervised fashion across subjects on in-silico data, followed by its fine-tuning on in-vivo data. In this work, we formalize two new scenarios to improve the learning of this hybrid SSM: 1) Unsupervised learning where, when previous ECG observations from the same subject accumulate without ground truth recordings in the heart, the hybrid SSM utilizes these past ECG data to improve its future ECGI solutions on the same individual; and 2) Combined supervised and unsupervised learning where, when both in-silico simulation and clinical ECG observations are available, simulation data corresponding to different rhythms can be generated to augment the learning. In both scenarios, we develop the hybrid SSM to take ECG observations and reconstruct the epicardial and endocardial extracellular potentials.

To elaborate, in the scenario of unsupervised Bayesian filtering across different ECG observations on the same subject, the learning does not require any a priori data of cardiac electrical activity. Instead of relying on potentially incorrect prior assumptions in fixed SSM functions, we allow part of the state-space functions to be learnable to accumulate knowledge from past data of an individual subject. Compared to independently applying Bayesian filtering with a fixed SSM, we show that the learnable SSM can accumulate knowledge from unsupervised Bayesian filtering across a (small) number of ECG observations, and thus deliver improved ECGI results on new ECG observations. We demonstrated this via both in-silico and in-vivo experiments.

Furthermore, the neural formulation of the SSM allows it to be supervised by paired data of ECG observation and its corresponding cardiac electrical activity when such data are available. This allows the integration of supervised data-driven learning utilizing available data of cardiac electrical activity (e.g. via in-silico simulation), with unsupervised Bayesian filtering utilizing only ECG data (e.g. in typical clinical settings). In this scenario, the same learnable state space functions are supervised by ground truth of electrical cardiac data when available (i.e., data-driven loss), along with supervision by ECG data via the physics-based forward operator (i.e., physics-based loss). On in-vivo test data for human subjects, we demonstrated that this mixed-loss SSM allowed us to combine simulated data – either from the same subject or from other subjects – with real ECG data of the subject to improve ECGI performance over using either model alone.

II. Related Works

Cardiac potential ${\mathbf{X}}_t$ generates time-varying voltages on the body surface ${\mathbf{Y}}_t$ following the quasi-static electromagnetic theory [14]. Given a pair of heart and torso geometry, the governing physics could be numerically approximated as [1]:

$${\mathbf{Y}}_t=\mathbf{H}{\mathbf{X}}_t\forall t\in \left\{1,\dots ,T\right\},$$ (1)

where ${\mathbf{X}}_t$ represents electrical potentials on the heart mesh and ${\mathbf{Y}}_t$ the electrical potentials on the torso mesh, at time instant $t.\mathbf{H}$ defines the forward imaging physics obtained over subject-specific heart-torso geometry and the associated tissue conductivity. ECGI addresses the reconstruction of ${\mathbf{X}}_{1:T}$ from given measurements of ${\mathbf{Y}}_{1:T}$

A. Traditional Physics-based ECGI Approaches

Based on (1), various numerical optimization and statistical inference methods can be used to seek the inverse solution ${\stackrel{\mathbf{ˆ}}{\mathbf{X}}}_t$ that minimizes the fitting of the observational data ${\mathbf{Y}}_t$, while satisfying certain constraints $ℛ\left({\mathbf{X}}_t\right)$ about ${\mathbf{X}}_t$. Representative constraints include the the smoothness of the solution in space and time at different orders of derivatives [15], such as Tikhonov regularization [3], [16], truncated singular value decomposition (SVD) [17], and spatiotemporal regularization [4]. Other constraints consider the sparsity of the cardiac signal in the gradient domain by L1 norm [5] and total variance [6]. Alternatively, model-based regularization encodes a priori physiological knowledge about the spatiotemporal dynamics of the solution, including logistic functions [7] and step jump functions [18] to describe the activation of action potential and 3D electrophysiological simulation model [1], [19], [20] or network-based representation [8] of spatiotemporal propagation of action potential for transmural sources throughout the myocardium.

State-Space Modeling & Bayesian Filtering:

The various existing works discussed above can be conceptually summarized in a general state-space framework, where different types of constraints $ℛ\left({\mathbf{X}}_t\right)$ can be translated to designs of different dynamic transition functions. For instance, temporal smoothness of ${\mathbf{X}}_t$ can be described by ${\mathbf{X}}_t={\mathbf{X}}_{t-1}+{\omega }_t$ which assumes that ${\mathbf{X}}_t$ changes minimally over consecutive time frames and ${\omega }_t$ is a noise term. If no temporal constraint is imposed, the dynamic transition function can be simply interpreted as ${\mathbf{X}}_t={\omega }_t$. For instance, the spatial smoothness constraint for ${\mathbf{X}}_t$ [3], [4] can be interpreted as a Gaussian assumption for ${\omega }_t$, and the spatial sparsity constraint [5], [6] can be summarized as a Laplacian assumption for ${\omega }_t$. Alternatively, the dynamic transition function in model-based regularization can be translated as PDEs [1], [7] or neural networks trained from such PDEs [8]. In general, while simpler functions (e.g., smoothness) have limited capacity to express rich knowledge, complex functions (e.g., PDEs of action potential propagation) are associated with increased structural and parameter assumptions and thus a higher risk of introducing errors when applied to an individual subject.

In traditional Bayesian filtering, after the choice of the dynamic transition function is made, it is kept fixed to provide predictions that constrain the estimation given ECG data. Incorrect assumptions may thus lead to incorrect estimations; furthermore, Bayesian filtering is typically performed independently across different ECG data, with a missed opportunity to pull knowledge across data to refine the dynamic transition function to improve its application to future ECG data.

B. Data-Driven ECGI approaches

As in many other areas, deep learning ECGI methods have started to show great promise [11], [12], [21]–[23]. These approaches typically learn a direct inverse mapping using data pairs of signals on the body and the heart. For instance, fully connected layers have been used to learn the temporal correlation between body and heart potentials from the sequence of ECG data and reconstruct heart potentials in future timesteps [24]–[26]. Long short-term memory (LSTM) networks [27] have been used to learn sequence-to-sequence relationship between ECGs and cardiac transmembrane potentials (TMPs) [11]. Graph convolutional neural networks (GCNNs) have also been adopted to learn inverse mapping as a function of given geometry to improve generalization across subjects [12].

These data-driven approaches do not incorporate imaging physics within the forward model, and often require a large amount of pairs of ${\mathbf{X}}_t$ and ${\mathbf{Y}}_t$ to train. Because real cardiac electrical activity ${\mathbf{X}}_t$ is hardly available in practice, training on simulated data has been commonly adopted in existing ECGI models [11], [12], [25]. The generalizability of the resulting models to real data, however remains to be tested.

III. Preliminary: Bayesian Filtering

The traditional SSM for ECGI consists of a prior dynamic transition function and an emission function:

$$Transitionfunction:{\mathbf{X}}_t=f\left({\mathbf{X}}_{t-1}\right)+w_t,$$ (2)

$$\mathit{Emission}\mathit{function:}{\mathbf{Y}}_t=\mathbf{H}{\mathbf{X}}_t+{\nu }_t$$ (3)

where ${\mathbf{X}}_t$ is the heart signal and ${\mathbf{Y}}_t$ is the body surface potential at time $t$, and $w_t$ and ${\nu }_t$ are noise variables introduced to account for modeling errors in these two functions. The transition function $f(\cdot )$ is typically designed to describe prior knowledge about ${\mathbf{X}}_t$ as discussed earlier. Based on this SSM, Bayesian filtering involves iterative estimations following a two-step procedure below:

$$\mathit{Prior}\mathit{prediction:}{\mathbf{X}}_t^{-}=f\left({\mathbf{X}}_{t-1}\right),$$ (4)

$$\mathit{MAP}\mathit{estimation:}{\stackrel{\mathbf{ˆ}}{\mathbf{X}}}_t={\mathbf{X}}_t^{-}+{\mathbf{K}}_t\left({\mathbf{Y}}_t-\mathbf{H}{\mathbf{X}}_t^{-}\right),$$ (5)

where the prediction in (4) leverages the prior assumption in the SSM transition function in (2), and the MAP estimation in (5) utilizes the Kalman gain ${\mathbf{K}}_t={\mathbf{P}}_t^{-}{\mathbf{H}}^T{\left(\mathbf{H}{\mathbf{P}}_t^{-}{\mathbf{H}}^T+{\mathbf{R}}_{\nu }\right)}^{-1}({\mathbf{P}}_t^{-}$ is the covariance of ${\mathbf{X}}_t$) to optimally weigh the prior prediction with its discrepancy to observed ${\mathbf{Y}}_t$.

Typically the state-space functions are kept fixed when the above Bayesian filtering is applied to different observations of ${\mathbf{Y}}_{1:T}$. This means that the knowledge of the discrepancy between the prior assumption and the observations, while present in each of such observations, is not being utilized to refine the prior assumptions nor to improve future estimations.

IV. Hybrid SSM for Neural Filtering

In this work, we present a novel hybrid SSM framework and realize it with an encoding-decoding network to make the state-space function and the associated Bayesian filtering strategy learnable. Specifically, the decoder will model the SSM in (2) and (3) as a generator network, and the encoder will learn the Bayesian filtering in (4) and (5) as an inference network.

A. Decoder: Hybrid SSM

The decoder (as illustrated in blue boxes in Fig. 1) models state-space functions (2) and (3) as:

$$Transition:\left\{\begin{array}{rr}\frac{d{\mathbf{s}}_t}{dt}={ℱ}_{\varphi }\left({\mathbf{s}}_t\right),& \text{Temporal}\text{Transition,}\\ {\mathbf{X}}_t={𝒟}_{\eta }\left({\mathbf{s}}_t\right),& \text{Spatial}\text{Decoding,}\end{array}\right.$$ (6) (7)

$$Emission:{\mathbf{Y}}_t=\mathbf{H}{\mathbf{X}}_t,$$ (8)

where ${\mathbf{s}}_t\in {\mathbb{R}}^{V\times M}$ is the state of the latent dynamics at time $t,$ $V$ is the number of nodes (~60, depending on the anatomy of the subject) on the sparse heart mesh and $M$ is the feature dimension (64 for all subjects) in the latent space. Here we introduce spatial-temporally disentangled neural modeling of the transition function in Equation (2). Specifically, we introduce neural functions ${ℱ}_{\varphi }$ to model the temporal dynamics of cardiac electrical activity in a lower-dimensional latent space, and ${𝒟}_{\eta }$ to model the generation of cardiac electrical activity from this latent space. Both $\varphi$ and $\eta$ are learnable parameters. We preserve the physics-based emission function leveraging the forward operator $\mathbf{H}$ to supervise the learnable SSM with ECG data without labeled data of cardiac electrical activity. We term this a hybrid SSM.

Fig. 1.

Overview of the presented hybrid neural SSM. The size of the feature map follows the format of #Vertices x #Feature x #Time. The green boxes (${ℰ}_{\mathit{\rho }}$ and ${𝒢}_{\mathit{\gamma }}$) belong to the encoder in Section IV-B and the blue boxes (${ℱ}_{\mathit{\varphi }},$ ${𝒟}_{\mathit{\eta }}$, and $\mathbf{H}$ operator) belong to the decoder in Section IV-A. The purple dashed box is the unsupervised learning of Bayesian filtering in Section V-A and the orange dashed box is the combined supervised and unsupervised learning in Section V-B.

Latent Dynamic Transition with Neural ODEs:

We chose to model the latent dynamics of cardiac electrical activity with a neural ODE due to its ability to handle irregular data availability. Given that the temporal resolution of ECG varies, the ability to propagate forward indefinitely without requiring data allows the transition models’ temporal resolution to be decoupled from that of the available ECG. In data settings with high density, both spatially and temporally, the NODE also has the advantage of an $O\left(1\right)$ memory cost in its forward solution with the usage of the adjoint method [28].

More specifically, as illustrated in Fig. 1, the dynamics of the latent variable ${\mathbf{s}}_t$ of ${\mathbf{X}}_t$ can be described by numerically integrating the neural ODE function ${ℱ}_{\varphi }$ in Equation (6) as:

$${\mathbf{s}}_t={\mathbf{s}}_{t-1}+{\int }_{t-1}^t{ℱ}_{\varphi }\left({\mathbf{s}}_{\tau }\right)d\tau .$$ (9)

We implemented the ODE function as a stack of multi-layer perceptrons and activation function. We then apply a fourth-order Runge-Kutta solver [29] to solve the ODE on $[t-1,t]$ to obtain the prediction of the current state ${\mathbf{s}}_t$.

Spatial Decoding via GCNNs:

As heart signals live on a 3D non-Euclidean geometry, we describe the spatial decoding model from the learned latent space to the reconstruction ${\mathbf{X}}_t$ with a stack of GCNNs ${𝒟}_{\eta }$ with learnable parameters $\eta$.

As detailed in our previous work [12], we represent the triangular mesh of the heart as an undirected graph. Note that the number of vertices of the heart graph varies among different subjects. The spatial decoding is performed over hierarchical graph representations of the heart geometry obtained by a specialized mesh coarsening method [30] to preserve the topology of the geometry.

Following the previous work [13], we use the graph convolution with a continuous spline kernel across graphs [31], then introduce residual blocks to make the network deeper and more expressive.

Emission to ECG Data:

The emission from ${\mathbf{X}}_t$ to ${\mathbf{Y}}_t$ employs the physics-based forward operator $\mathbf{H}$. This allows us to supervise the hybrid SSM with this physics operator and observed ECG data, rather than data of cardiac electrical activity that is rarely available in practice. Indeed the physics operator may have errors, which may degrade the performance of the hybrid SSM compared to a neural network that is fully supervised by cardiac electrical activity data. However note that such data is hardly available in practice, thus the potential performance drop of the hybrid SSM will be in exchange for a significant advantage in terms of real-world applicability. We will compare the presented hybrid SSM with fully-supervised baselines in the experiments.

B. Encoder: Neural Bayesian Filtering

We then use the encoder network (as illustrated in green boxes in Fig. 1) to learn Bayesian filtering:

$$Predictionfunction:{\mathbf{s}}_t^{-}={\widehat{s}}_{t-1}+{\int }_{t-1}^t{ℱ}_{\varphi }\left({\mathbf{s}}_{\tau }\right)d\tau ,$$ (10)

$$Estimationfunction:{\widehat{\mathbf{s}}}_t={𝒢}_{\gamma }\left({\mathbf{z}}_t^h,{\mathbf{s}}_t^{-}\right),$$ (11)

where ${\mathbf{z}}_t={ℰ}_{\rho }\left({\mathbf{Y}}_t\right),$ ${\mathbf{z}}_t\in {\mathbb{R}}^{V\times M}$ is the latent embedding of the observation ${\mathbf{Y}}_t,$ $V$ is the number of nodes (~60, depending on the anatomy of the subject) on the sparse heart mesh and $M$ is the feature dimension (64 for all subjects) in the latent space, and the prediction of the latent state ${\mathbf{s}}_t^{-}$ is modeled as the solution to the neural ODE in Equation (6) given the previous state estimation ${\stackrel{\mathbf{ˆ}}{\mathbf{s}}}_{t-1}$. The encoder consists two parts: the spatial embedding function ${ℰ}_{\rho }$ encodes ${\mathbf{Y}}_t$ to a latent variable ${\mathbf{z}}_t^h$, and the correction function ${𝒢}_{\gamma }$ combines ${\mathbf{s}}_t^{-}$ with the embedded ${\mathbf{z}}_t^h$ to get an optimal state estimation ${\stackrel{\mathbf{ˆ}}{\mathbf{s}}}_t$.

GCNNs for Embedding Observed Data:

As torso signals ${\mathbf{Y}}_t$ also live on 3D geometry, we follow a similar design in Section IV-A to describe the spatial embedding function ${ℰ}_{\rho }$ with a stack of GCNNs with learnable parameters $\rho$. The geometry of the torso is also represented as an undirected graph, where edges and vertices are defined similarly as described in Section IV-A. For torso graphs on different subjects, the number of vertices is the same but the edge attributes are different. To learn the relationship between heart and torso, we follow [12] and assume the linearity to hold between ${\mathbf{X}}_t$ and ${\mathbf{Y}}_t$ in the latent space during inverse imaging. We construct a bipartite graph between the graph embedding of the heart and torso geometry: the edge attribute $\mathbf{u}(i,j)$ between torso vertex $i$ and heart vertex $j$ describes their relative geometrical relationship. Using spline convolution, we model the latent representation ${\mathbf{z}}^h\left(i\right)$ on vertex $i$ of the latent heart mesh as a linear combination of latent representation ${\mathbf{z}}^b\left(j\right)$ across all vertices $j$ of the latent torso mesh as:

$${\mathbf{z}}^h\left(i\right)=\sum _j{\mathbf{z}}^b\left(j\right)\cdot \widehat{h}\left(\mathbf{u}\left(i,j\right)\right),$$ (12)

where $\stackrel{\mathbf{ˆ}}{\mathbf{h}}$’s are learned as the spline convolution kernel.

Latent Dynamics Correction with GCN-GRU:

The final estimation of the latent variable ${\stackrel{\mathbf{ˆ}}{\mathbf{s}}}_t$ is given as a weighted combination of the latent representation ${\mathbf{z}}_t^h$ of the measurement ${\mathbf{Y}}_t$ on the heart, and the prediction of the latent dynamics ${\mathbf{s}}_t^{-}$. This is achieved by a Gated Recurrent Unit (GRU) cell [32] whose underlying architecture contains GCN layers as both the hidden state and input to the cell are graphs. We call it a GCN-GRU cell and denote it as ${𝒢}_{\gamma }$. Specifically, each linear operator in the regular GRU cell is replaced by GCN:

$$\begin{array}{l}{\mathbf{r}}_t=\sigma \left(G_{z1}\left({\mathbf{z}}_t^h\right)+G_{s1}\left({\mathbf{s}}_t^{-}\right)\right),\\ {\mathbf{g}}_t=\sigma \left(G_{z2}\left({\mathbf{z}}_t^h\right)+G_{s2}\left({\mathbf{s}}_t^{-}\right)\right),\\ {\mathbf{n}}_t=\tanh \left(G_{z3}\left({\mathbf{z}}_t^h\right)+\mathbf{r}\odot G_{s3}\left({\mathbf{s}}_t^{-}\right)\right),\\ {\stackrel{\mathbf{ˆ}}{\mathbf{s}}}_t=\left(1-{\mathbf{g}}_t\right)\odot {\mathbf{n}}_t+{\mathbf{g}}_t\odot {\mathbf{s}}_t^{-},\end{array}$$ (13)

where $\sigma (\cdot )$ is the sigmoid function, ${\left\{G_{zi},G_{si}\right\}}_{i=1}^3$ are GCN operators, and $\gamma ={\left\{z_i,s_i\right\}}_{i=1}^3$ are learnable parameters.

V. Learning the Hybrid SSM

The presented encoder-decoder as described in Section IV is denoted as ${\mathit{F}}_{\theta }$, where $\theta$ are learnable parameters. We consider the following two scenarios in learning ${\mathit{F}}_{\theta }$. First, we learn ${\mathit{F}}_{\theta }$ without the supervision of ground truth cardiac signals $\mathbf{X}$, where the loss function (which we call physics-loss) between the input body surface potentials $\mathbf{Y}$ and the output of the emission model $\stackrel{\mathbf{ˆ}}{\mathbf{Y}}$ provides an alternative supervision to the model. Second, we introduce mixed-loss by combining unsupervised learning on in-vivo real data when ground truth data $\mathbf{X}$ is not available, with data-driven learning on synthetic data when ground truth data $\mathbf{X}$ are available.

A. Unsupervised Learning of Hybrid SSM

Traditional methods seek to find a solution ${\mathbf{X}}_t$ that minimizes the fitting of ${\mathbf{Y}}_t$ while satisfying constraints $ℛ\left({\mathbf{X}}_t\right)$ :

$$\arg \underset{{\mathbf{X}}_t}{\text{min}}\sum _{t=1}^T{\Vert {\mathbf{Y}}_t-\mathbf{H}{\mathbf{X}}_t\Vert }_2^2+\lambda ℛ\left({\mathbf{X}}_t\right),$$ (14)

where the regularization parameter $\lambda$ is often empirically tuned. The proposed physics-loss (as illustrated in purple dashed boxes in Fig. 1) utilizes the physics-informed relationship in (14) to supervise the neural function ${\stackrel{\mathbf{ˆ}}{\mathbf{X}}}_t={\mathit{F}}_{\theta }\left({\mathbf{Y}}_t\right)$ of the hybrid SSM, where $\mathit{F}=\{ℰ,ℱ,𝒢,𝒟\}$ and $\theta =\{\rho ,\varphi ,\gamma ,\eta \}$. The optimization problem in (14) is then converted to learning parameters of ${\mathit{F}}_{\theta }\left({\mathbf{Y}}_t\right)$ for:

$$\arg \underset{\theta }{\text{min}}\sum _{i=1}^N\sum _{t=1}^T{\Vert {\mathbf{Y}}_t^i-\mathbf{H}{F}_{\theta }\left({\mathbf{Y}}_t^i\right)\Vert }_2^2+\lambda ℛ\left(F_{\theta }\left({\mathbf{Y}}_t^i\right)\right),$$ (15)

where $N$ is the number of data points. While different choices of $ℛ\left({\mathbf{X}}_t\right)$ can be considered, as a proof of concept we adopt the popular Laplacian smoothing over the heart geometry, and $\lambda$ is empirically tuned as in (14).

B. Combined Supervised and Unsupervised Learning

The neural formulation of the SSM allows it to be learned with supervision when partial data of cardiac electrical activity ${\mathbf{X}}_t$ are available. In this scenario, we propose mixed-loss (as illustrated in orange dashed boxes in Fig. 1) that combines the strength of both the physics loss in (15) and data-driven learning. Specifically, the learnable state-space function is supervised by the in-silico data with ground truth ${\mathbf{X}}_t$ available, along with the same physic loss in (15) for in-vivo ECG data without ${\mathbf{X}}_t$. Additionally, we ask the physics loss to be satisfied on in-silico data as well. In summary, the mixed-loss can be represented as follows:

$$\begin{array}{l}\text{arg}\underset{\theta }{\text{min}}\sum _{i=1}^N\sum _{t=1}^T{\Vert {\mathbf{Y}}_t^i-\mathbf{H}{\mathit{F}}_{\theta }\left({\mathbf{Y}}_t^i\right)\Vert }_2^2+\lambda ℛ\left({\mathit{F}}_{\theta }\left({\mathbf{Y}}_t^i\right)\right)\\ +{\mu }^i{\Vert {\mathbf{X}}_t^i-{\mathit{F}}_{\theta }\left({\mathbf{Y}}_t^i\right)\Vert }_2^2,\end{array}$$ (16)

where ${\mu }^i$ is 0 when there is no available ${\mathbf{X}}^i$, and non-zero and empirically determined when ${\mathbf{X}}^i$ is available for sample ${\mathbf{Y}}^i$.

VI. Experiments and Results on Unsupervised Learning of Hybrid SSM

In this set of experiments, we demonstrate that when the hybrid SSM is trained to do unsupervised Bayesian filtering on multiple data from the same subject, its performance on new data increases and improves over independent ECGI solutions on the same data. In this setting, the hybrid SSM does not need the supervision of ${\mathbf{X}}_t$. This represents a common use scenario that, as multiple ECG data from the same individual are collected over time, the hybrid SSM keeps refining the learned function of the underlying cardiac system for future use on the same subject. We tested this on both simulation and in-vivo data.

A. Models, Baselines, and Evaluation Metrics

Our presented hybrid SSM contains the following components: a spatial embedding network with three GCNN blocks, two standard convolutional layers, and a spline convolutional inverse mapping; a neural-ODE-based latent transition model with four linear layers; a correction model with a GCN-GRU cell; and a spatial decoder with two standard convolutional layers and four GCNN blocks, with 1,051,594 parameters in total. We used ELU activation [33] for most of the layers and the hyperbolic tangent function for the last layer of the neural function.

In this set of experiments, the model is supervised only by the ECG data through the forward operator via the loss function in Equation (15): in other words, the model is unsupervised (by the label ${\mathbf{X}}_t$). We call this unsupervised model U-SSM. The network parameter is optimized by Adam optimizer [34] with a learning rate of 5×10⁻⁴ and a learning rate scheduler decreasing the learning rate every 200 epochs with a decay rate of 0.5. We compared the performance of unsupervised SSM with classic ECGI approaches utilizing three different types of dynamic transition functions as temporal constraints: 1) no temporal constraints which correspond to second-order Tikhonov regularization [3], 2) temporal smoothness constraint, and 3) transmural constraint [35]. We also trained a data-driven version of the same hybrid SSM (we call this model S-SSM) where the hybrid SSM is supervised by available cardiac data ${\mathbf{X}}_t$ corresponding to the ECG data.

B. Simulation-Data Experiments

1). Data and Training:

We considered three human heart-torso meshes, with 448, 475, 480 heart nodes and 120, 120, 120 torso nodes respectively. For each heart, we considered two different spatial configurations of scar tissues, resulting in total six subject-specific hearts for training and testing the hybrid SSMs. On each heart, we simulated realistic spatiotemporal propagation sequences of action-potentials by the Aliev-Panfilov model [36] considering different origins of activation. To obtain the extracellular potential on the heart surface from volumetric action potential, we obtained the forward operator by solving Possoin’s equation using the coupled meshfree method and boundary element methods as described in [1], [37], [38]. The forward operator from the heart-surface extracellular potential ${\mathbf{X}}_t$ to body-surface potential ${\mathbf{Y}}_t$ was then obtained using the open-source SCIRun toolkit solving the Laplace equation using the boundary element method [39]. On the simulated body-surface potential, 30 dB Gaussian noises were added for reconstructing ${\mathbf{X}}_t$. Because the action potential simulated by the Aliev-Panfilov model was unitless in both amplitude and time, the generated signals on the heart and torso were also unitless. Specifically, in our experiments, we considered the depolarization process that was downsampled in time and represented by 65 discrete time steps. To generate disjoint training and testing sets for each subject, we partition the data based on the locations of activation origins. To examine the performance of the U-SSM as the number of training data increases, we randomly sampled the simulated data with an increasing number of origins (25, 50, 75, 100).

We also considered simulation data generated from a different pipeline on the three healthy subjects. For these simulations, each bi-ventricular mesh of targeted resolution 1200μm was equipped with rule-based fibers [40] and a physiologically-detailed His-Purkinje system representing the cardiac conduction system [41]. Universal ventricular coordinates were computed for spatial navigation [42], [43]. On each heart, simulations of the trans-membrane voltages were run using a reaction-Eikonal model in monodomain formulation without diffusion using CARPentry [44], [45] considering 500 different origins of activation evenly sampled across the entirety of the endocardium (see Figure 2). Automated sampling of activation sites was facilitated using the universal ventricular coordinates in open-source meshtool [46]. Cellular dynamics within the heart were modeled using the Mitchell-Schaeffer ionic model with a set membrane voltage of −86.2 mV and a plateau of 40 mV [47]. Heart conductivities were assigned according to [48] and the torso conductivity was assigned a value of 0.22 S m⁻¹ [49]. Conduction velocities within the myocardium were assigned 0.6 ms⁻¹ with an off-axis ratio of 4:2:1. A general conduction velocity of 2.0 ms⁻¹ was prescribed within the His-Purkinje System, with slight deviations assigned in the fascicular branches. An anterograde and retrograde delay of 8 ms and 3 ms, respectively, was assigned. The simulated transmembrane voltages were rescaled to [0, 1], and we followed the similar steps mentioned above to obtain extracellular potentials and then 120-lead ECGs. Therefore, the generated signals on the heart and torso were all unitless. The sampling frequency was 1000 Hz. The temporal dimension was 600 discrete time steps and then downsampled to 200 discrete time steps. The three subjects have the spatial dimensions of 350, 350, 350 on the heart and 120, 120, 120 on the torso and we consider the depolarization process represented by 90 discrete time steps. For each subject, we randomly sampled the simulated data with an increasing number of origins (100, 200, 300, 400) to examine the performance of the U-SSM with an increasing number of training data.

Fig. 2.

Overview of the sampled pacing locations (black) for each heart geometry.

2). Evaluation Metrics:

The accuracy of ECGI solutions in reconstructing ${\mathbf{X}}_t$ was measured by the mean square error (MSE), spatial correlation coefficient (SCC), and temporal correlation coefficient (TCC) between the reconstructed and actual potential sequence on the heart surface. While MSE measures the quantitative errors of the reconstructed signals on the heart, the SCC and TCC measure the correlation between the reconstructed and reference signals. We considered the correlation for both spatial signals at each time instance (SCC) and temporal signals at each location of the heart (TCC).

3). Results:

Fig. 3 summarizes the quantitative metrics of the four models with respect to different numbers of training data. The reconstruction accuracy of the U-SSM surpassed the three ECGI baselines as the data availability increased. Note that the TCC of the U-SSM was better than the Tikhonov and temporal smooth ECGI baselines even when the training dataset is small among all subjects (25% training data). The SCC by the U-SSM outperformed all three ECGI baselines when there were over 50 samples in training. The S-SSM set up a strong upper bound for the performance of the U-SSM, even at a small training size of 25. Note that the U-SSM’s performance is approaching that of the supervised model as the number of unlabeled data increases. Fig. 4 provides visual examples demonstrating improved signal propagation patterns and scar locations obtained by the U-SSM. Notice that the U-SSM shows better reconstruction details of the signal propagation on the endocardial surface compared to all ECGI baseline models.

Fig. 3.

Performance of the proposed model with respect to different data availability in training (right side of the vertical dashed line) and the four baseline models (left side of the vertical dashed line).

Fig. 4.

Visual examples of reconstructed electrical activity among four comparison models. Both epicardial and endocardial surfaces are presented. The arrow indicates the pacing site location. The color bar shows the scaled range of the signal since it is unitless in synthetic data. The U-SSM is trained on 50% of the full set. The MSE value is shown for each model at each timestep. The U-SSM showed better performance in predicting signal propagation and scar localization (circled) compared to the three ECGI baselines. The S-SSM demonstrated an upper bound in both signal propagation prediction and scar localization.

4). Alternative Architecture of Latent Transition Functions:

As mentioned in Section IV-A, we chose the neural ODE as the latent dynamic transition function because of its generality as a continuous dynamic model, which has the advantage of decoupling the transition model’s temporal resolution from that of the available ECG measurements, to accommodate irregularly sampled time-series data, and to handle missing data in time – the latter two scenarios however are not commonly expected in the setting of ECGI. Nevertheless, it is important to note that the presented hybrid SSM is agnostic to specific choices of function architectures or types for the latent dynamic transitions. To demonstrate this, we investigated an alternative for temporal modeling: ${\mathbf{s}}_t^{-}={\stackrel{\mathbf{ˆ}}{\mathbf{s}}}_{t-1}+\mathrm{\Delta }{\mathbf{s}}_t$, where $\mathrm{\Delta }{\mathbf{s}}_t$ is given by GRU (denoted as GRU-res):

$$\begin{array}{l}{\mathbf{s}}_{t-1}^{\left(1\right)}=\mathrm{E}\mathrm{L}\mathrm{U}\left({\mathit{\alpha }}_1{\stackrel{\mathbf{ˆ}}{\mathbf{s}}}_{t-1}+{\mathit{\beta }}_1\right),{\mathbf{g}}_{t-1}=\sigma \left({\mathbf{W}}_1{\mathbf{s}}_{t-1}^{\left(1\right)}+{\mathbf{b}}_1\right)\\ {\mathbf{s}}_{t-1}^{\left(2\right)}=\mathrm{E}\mathrm{L}\mathrm{U}\left({\mathit{\alpha }}_2{\stackrel{\mathbf{ˆ}}{\mathbf{s}}}_{t-1}+{\mathit{\beta }}_2\right),{\mathbf{h}}_{t-1}=\mathrm{E}\mathrm{L}\mathrm{U}\left({\mathbf{W}}_2{\mathbf{s}}_{t-1}^{\left(2\right)}+{\mathbf{b}}_2\right)\\ {\tilde{\mathbf{s}}}_{t-1}={\mathit{\alpha }}_3{\stackrel{\mathbf{ˆ}}{\mathbf{s}}}_{t-1}+{\mathit{\beta }}_3,\\ & \mathrm{\Delta }{\mathbf{s}}_t=\left(1-{\mathbf{g}}_{t-1}\right)\odot \left({\mathbf{W}}_3{\tilde{\mathbf{s}}}_{t-1}+{\mathbf{b}}_3\right)+{\mathbf{g}}_{t-1}\odot {\mathbf{h}}_{t-1},\end{array}$$

where ${\mathit{\theta }}_z={\left\{{\mathbf{W}}_i,{\mathbf{b}}_i,{\mathit{\alpha }}_i,{\mathit{\beta }}_i\right\}}_{i=1}^3$ are learnable parameters of the dynamic function [50]. We compared GRU-res with the proposed method (denoted as Neural ODE) on a subset of synthetic data in Section VI-B with changing of origin. Fig. 5 summarizes the quantitative results of the two temporal models with respect to different numbers of training data. The reconstruction accuracy of GRU-res is compatible with Neural ODE and has a similar trend with increasing training data. This proves that hybrid SSM was not affected by the choice of latent dynamic models.

Fig. 5.

Summary of average performance on in-silico data among three comparison models with respect to different temporal models.

C. Real-Data Experiments

1). Data and Training:

We then repeated the evaluation and comparison experiments on in-vivo 120-lead ECG data from three subjects with premature ventricular contraction (PVC) on structurally healthy hearts [51] (termed as Healthy 1, Healthy 2, and Healthy 3), which have been made available at EDGAR [52], and three post-infarction subjects [52], [53] (termed as Post 1, Post 2, and Post 3), all undergoing ventricular pacing. The healthy subjects have the same geometry as the geometry of the three healthy subjects used in Section VI-B. The body surface potentials of the three healthy subjects are sampled at 2000 Hz and further downsampled using polyphase filtering to 201 steps, with the depolarization represented in 90 steps used in experiments. The heart-torso geometry of the three subjects with post-infarction is the same as the geometry of the three subjects with scar tissues used in Section VI-B. The body surface potentials of the three subjects with post-infarction are sampled at 1000 Hz and further downsampled using polyphase filtering to 101 steps, with the depolarization represented in 65 steps used in experiments. For each subject, patient-specific heart torso geometry was extracted from computed-tomography images, on which extracellular potential signals were reconstructed from ECG data. Each healthy subject contains ECG data acquired from 21, 14, and 17 different origins of pacing, respectively. Each postinfarct subject contains ECG data acquired from 22, 20, and 23 different origins of pacing respectively. For each subject, we randomly left out four sites for testing. Because the S-SSM cannot be directly trained on in-vivo data due to lack of ${\mathbf{X}}_t$, we considered directly applying the S-SSM trained on the complete in-silico dataset in Section VI-B to in-vivo data. Similarly, we also evaluated existing data-driven methods, such as ST-GCNN [12], Euclidean method [11], and MARS [23], by training them on the same in-silico dataset and testing on in-vivo data. We also considered ECGI with the second-order Tikhonov regularization and transmural regularization in comparison. To examine the performance of the U-SSM given different training data availability, we randomly selected 50% and 25% of the training data on each subject. All results below are reported on the same four test cases each on the six subjects totaling 24 cases.

2). Evaluation Metrics:

Due to the lack of in-vivo measurement of heart-surface potential, quantitative accuracy was measured by the Euclidean distance between the reconstructed origins of the activation and the known sites of pacing. To localize the site of the earliest activation by the steepest descent of the reconstructed extracellular potential during depolarization in itself presents non-trivial challenges [54]. We thus resorted to manual identification of the region of the earliest activation by visual inspection of the reconstruction electrograms sequence, and the centroid of the identified region was calculated as the site of the earliest activation.

3). Results:

Table I summarizes the localization errors for the early activation site with respect to different amounts of training data. Directly applying supervised data-driven models trained on simulation data to in-vivo data (supervised baselines) had a worse overall localization error (S-SSM: 44.00 ± 12.22, ST-GCNN: 49.12 ± 20.81, Euclidean: 48.90 ± 15.26, MARS: 38.48 ± 25.04) than the Tikhonov (40.26 ± 22.69) and transmural-based regularization (33.97 ± 20.79) ECGI baselines, highlighting the challenges of generalization when training an ECGI reconstruction network supervised by simulation data. When using 25% of the full training set, the overall localization accuracy of the U-SSM (37.51 ± 15.69) was comparable with the Tikhonov method (40.26 ± 22.69) and less accurate than transmural-based regularization (33.97± 20.79). The accuracy of U-SSM surpassed the Tikhonov ECGI baseline when the training data increased (25%: 37.51 ± 15.69, 50%: 34.71 ± 17.88, 100%: 33.01 ± 18.18). When using the full training data, the accuracy of U-SSM became comparable with the transmural-based ECGI baseline. Fig. 6 provides visual examples.

TABLE I.

Summary of localization errors for the origin of ventricular activation in each subject. (unit: mm. paired p-test is performed between ecgi tikhonov and other comparison models. nss: non-subject-specific)

Type	Model	Post 1	Post 2	Post 3	Healthy 1	Healthy 2	Healthy 3	Overall	p-value
Non-ML ECGI	Tikhonov	33.87(17.81)	51.60(12.07)	39.09(39.05)	22.49(20.78)	44.69(26.60)	49.82(15.01)	40.26(22.69)	/
	Transmural	30.63(17.50)	38.00(16.40)	14.19(11.46)	25.56(24.86)	30.75(13.68)	64.65(5.45)	33.97(20.79)	0.321
Supervised Baselines	S-SSM	41.33(20.35)	45.39(16.98)	50.70(6.41)	35.82(7.18)	38.15(19.80)	39.16(23.51)	44.00(12.22)	0.481
	ST-GCNN	34.05(18.65)	62.41(22.31)	65.90(23.50)	46.11(18.76)	30.59(11.87)	55.67(13.47)	49.12(20.81)	0.165
	Euclidean	46.83(28.39)	56.75(16.04)	53.10(20.44)	39.82(15.28)	49.43(8.21)	47.47(9.74)	48.90(15.26)	0.129
	MARS	39.69(21.79)	43.40(47.43)	37.37(23.49)	29.97(24.13)	32.30(15.02)	48.11(26.05)	38.48(25.04)	0.797
U-SSM	25%	39.68(19.82)	48.91(19.26)	32.11(16.23)	28.32(17.01)	33.47(11.98)	42.59(18.17)	37.51(15.69)	0.628
	50%	37.42(20.02)	44.82(27.66)	26.09(8.63)	25.96(21.27)	32.58(11.74)	41.42(21.11)	34.71(17.88)	0.352
	100%	35.12(19.55)	39.70(21.47)	26.82(27.66)	24.85(19.38)	33.16(13.90)	38.41(17.73)	33.01(18.18)	0.228
M-SSM	25%	31.41(16.61)	32.50(23.76)	36.75(12.79)	18.14(5.02)	31.88(13.29)	28.19(12.76)	29.81(13.83)	0.060
	50%	24.45(17.76)	30.02(16.04)	35.22(14.08)	15.97(6.84)	29.55(15.26)	25.44(13.42)	26.77(13.81)	0.017
	100%	23.23(13.96)	27.20(16.32)	25.34(7.65)	13.16(1.57)	29.29(13.39)	25.28(9.85)	23.92(11.03)	0.003
	nss	26.66(14.93)	24.79(7.98)	24.55(11.13)	24.52(20.02)	33.35(13.79)	27.34(12.58)	26.87(12.09)	0.014
F-SSM	25%	36.48(22.22)	39.06(18.94)	42.54(15.13)	25.09(4.35)	30.58(12.14)	35.59(17.31)	34.89(14.51)	0.334
	50%	43.10(27.05)	32.28(18.23)	40.64(19.63)	23.76(4.44)	29.94(12.63)	35.68(12.29)	34.23(15.60)	0.289
	100%	31.51(14.89)	39.29(20.28)	37.14(18.74)	22.00(3.04)	29.48(11.62)	28.25(10.64)	31.28(13.09)	0.100
	nss	32.31(17.08)	44.40(3.76)	21.61(5.10)	26.66(19.84)	34.51(11.72)	30.99(11.48)	31.75(12.56)	0.115

Fig. 6.

Visual examples of reconstructed electrical activations on in-vivo data (Unit: mm; ss: subject-specific; nss: non-subject-specific). The arrows are predicted pacing sites and the dots highlight actual pacing sites. The localization error is shown for each model. The U-SSM and both subject-specific and non-subject-specific M-SSM showed generally better accuracy in early activation localization among all comparison models.

This set of experiments demonstrates that the presented U-SSM can learn to perform unsupervised Bayesian filtering on individual hearts, with a reconstruction accuracy higher than the supervised ECGI networks trained on simulation data and traditional ECGI methods with a similar regularization choice, and on par with traditional ECGI methods using more advanced regularization options.

VII. Experiments and Results on Combined Learning of Hybrid SSM

In this set of experiments, we demonstrate the use scenario for the hybrid SSM to leverage available simulation data – either from different subjects or from the same subject – and combine it with in-vivo ECG data available on a subject, to improve ECGI performance than using either data alone.

A. Models, Baselines, and Evaluation Metrics

We considered two specific experimental settings. For each subject on whom to test the hybrid SSM, we combined unsupervised training using in-vivo ECG data with supervised training on in-silico data 1) generated on this specific subject, versus 2) generated from two different subjects. This helps us test if we need to generate subject-specific simulation data for a subject in order to use this mixed-loss SSM (we call this model M-SSM).

For the three healthy subjects and the three post-infarction subjects, the simulation and in-vivo data are as described in Sections VI-B and VI-C. For each healthy subject, we chose the same 400 origins in training as in Section VI-B, and the same selection of four pacing sites for testing and the rest for training on in-vivo data as in Section VI-C. For each post-infarction subject, we chose one configuration of scar tissue with the same 100 origins in training as in Section VI-B, and the same selection of four pacing sites for testing and the rest for training on in-vivo data as in Section VI-C. All results below are reported on the same in-vivo test cases each on the six subjects totaling 24 cases.

Similar to Section VI-C, we compared the M-SSM with: 1) ECGI with second-order Tikhonov regularization and transmural-based regularization, 2) existing supervised data-driven models (S-SSM, ST-GCNN, Euclidean, and MARS) by training on in-silico data and testing on in-vivo data, and 3) unsupervised fine-tuning of the same supervised SSM on in-vivo data using the physics-based loss in (15) (we call this F-SSM). Similar to Section VI-C, we also examine the performance of the M-SSM concerning the change in data availability of in-vivo data by randomly selecting 50% and 25% of in-vivo data in training on each subject. We continued to use the Euclidean distance of pacing site localization similar to that in Section VI-C.

B. Combining with Subject-Specific Simulation Data

Table I shows localization errors of early activation sites among the comparison models. The M-SSM demonstrated the best overall localization accuracy (23.92 ± 11.03) among all comparison models. Note that directly applying supervised data-driven models trained on simulation data to in-vivo data showed worse overall performance (S-SSM: 44.00 ± 12.22, ST-GCNN: 49.12 ± 20.81, Euclidean: 48.90 ± 15.26, MARS: 38.48 ± 25.04) than both M-SSM and F-SSM (31.28 ± 13.09). When using 25% of the full training set, the overall performance of the M-SSM (29.81 ± 13.83) was better than the Tikhonov baseline (40.26 ± 22.69) and the transmural-based regularization baseline (33.97 ± 20.79). In Post 1, 2, 3, and Healthy 1, 3, the performance of the M-SSM using 25% of the full training set (Post 1: 31.41 ± 16.61, Post 2: 32.50 ± 23.76, Post 3: 36.75 ± 12.79, Healthy 1: 18.14 ± 5.02, Healthy 3: 28.19 ± 12.76) was even better than the F-SSM using the full training set (Post 1: 31.51 ± 14.89, Post 2: 39.29 ± 20.28, Post 3: 37.14 ± 18.74, Healthy 1: 22.00 ± 3.04, Healthy 3: 28.25 ± 10.64). The overall localization accuracy of both M-SSM (25%: 29.81 ± 13.83, 50%: 26.77 ± 13.81, 100%: 23.92 ± 11.03) and F-SSM (25%: 34.89 ± 14.51, 50%: 34.23 ± 15.60, 100%: 31.28 ± 13.09) showed an increasing trend with the increase in data availability. Fig. 6 provides visual examples of reconstructions of early activation.

C. Combining with Non-Subject-Specific Simulation data

As summarized in Table I, the non-subject-specific M-SSM showed a better localization accuracy (26.87 ± 12.09) in early activation sites in comparison with the Tikhonov baseline (40.26 ± 22.69) and transmural-based regularization baseline (33.97 ± 20.79), even though there is a small reduction of performance compared to the subject-specific M-SSM (23.92 ± 11.03). The overall performance of the non-subject-specific M-SSM in Post 1, 2, and all Healthy subjects (Post 1: 26.66 ± 14.93, Post 2: 24.79 ± 7.98, Healthy 1: 24.52 ± 20.02, Healthy 2: 33.35 ± 13.79, Healthy 3: 26.87 ± 12.09) was better than F-SSM (Post 1: 32.31 ± 17.08, Post 2: 44.40 ± 3.76, Healthy 1: 26.66 ± 19.84, Healthy 2: 34.51 ± 11.72, Healthy 3: 31.75 ± 12.56), while the performance of F-SSM was better in Post 3 (21.61 ± 5.10) than M-SSM (24.55 ± 11.13). Visual examples in Fig. 6 show that the non-subject-specific M-SSM also improved the early activation localization.

This set of experiments demonstrated that the hybrid SSM allows us to combine supervised training using simulation data and unsupervised training using in-vivo data, to obtain improved performance over using either data alone. It can utilize non-subject-specific simulation data which reduces the burden of having to conduct subject-specific in-silico simulations.

VIII. Conclusions and Discussion

We have presented a novel hybrid SSM framework for ECGI to leverage the advantage of state-space formulations in data-driven learning, with partially learnable state-space functions and unsupervised Bayesian filtering strategies. We demonstrated its improved ECGI performance under unsupervised Bayesian filtering on both in-silico and in-vivo experiments and under mixed-loss SSM setting on in-vivo data combined with simulation data either from the same subject or from other subjects. Our method is general for problems with spatiotemporal data with disentangled dynamic transition and spatial emission. Future studies will extend its application to other problems that fall into this category.

Several aspects of the presented hybrid SSM framework can be further investigated. One of the challenging factors exists in the subject-specific nature of Bayesian filtering, in which a critical component is that it accumulates the information of each subject. One may naively consider learning the knowledge shared across subjects by training the model on different subjects together. However, we found the improvement in reconstruction was marginal. How to model diverse dynamics across subjects with few-shot data remains an open question.

There are various choices of source models to represent the cardiac electrical activity in existing ECGI approaches, including heart surface potentials [24]–[26], or transmembrane voltages defined on the volumetric mesh of the heart [1], [8], [11], [55], [56]. Our study is based on the former because this is the most common formulation and the one used in commercial systems. Further extension of the hybrid SSM to the volumetric representation of the cardiac electrical activity will bring challenges in building appropriate hierarchical graph representations for spatial decoding. Furthermore, the substantial increase of the size of the graph can also lead to a higher computational cost of training the hybrid SSM.

This work uses Laplacian smoothing as the regularization in the objective function because this is one of the most popular choices and the Laplacian operator can be directly derived given the heart geometry. When using the Laplacian operator, the reconstruction accuracy of hybrid SSM was lower compared to traditional methods using more advanced regularization with small training data and increased to be comparable with the transmural-based method when more data were available. However, many alternative constraints on cardiac electrical activities as introduced in Section II-A, such as spatial sparsity [5], [6] and PDEs [1], [7], have not been investigated. Future works will seek to better incorporate physiological knowledge into the hybrid SSM by tackling challenges in the modeling of spatial and temporal gradients to make physics-informed neural networks.

The two major scenarios of hybrid SSM are 1) unsupervised learning of hybrid SSM when previous ECG observations from the same subject accumulate, and 2) combined supervised and unsupervised learning when both in-silico simulation and clinical ECG observations are available. In both scenarios, the hybrid SSM is unsupervised on the real data, meaning that all it requires is past ECG data from the same subjects. Because no label (actual knowledge about the underlying conditions) is required, in theory, ECG observations under any arrhythmic conditions can be used such as sinus rhythm, pacing, or various arrhythmic conditions. In our experiments, we considered pacing rhythms primarily because of the availability of data. In future works, we will investigate the general applicability of the hybrid SSM to various ECG conditions owing to its unsupervised nature.

We considered the localization of the pacing site by manual identification of the earliest activation region as the quantitative evaluation of the proposed method on in-vivo data. The ECGI validation remains an open question as the challenges come from the choice of the implementation methods, the unavailability of in-vivo data, and variations in clinical application interest [57]. As one of the typical local comparison metrics, the estimation of activation time and site from reconstructed electrograms itself is a difficult problem that is investigated among many groups [58], [59]. Future works will investigate more generalized and automated localization methods for pacing and alternative metrics based on the corresponding arrhythmic conditions.

A complete heart signal has two main stages: depolarization and repolarization, in which using ECGI to detect the time of electrical activation and recovery of local cardiac tissue from transmembrane voltages or extracellular potentials remains an open question. Our study only considered the depolarization stage in all experiments, given the limitation of the neural network in describing complex dynamics. Future works will explore more advanced neural networks to describe the exact physical processes.

Ideally, the performance of hybrid SSM should keep improving with the increasing of available training data. However, we noticed a common phenomenon in our experiments that improvement of performance slowed down when the number of training data was above a certain amount. Future work on the hybrid SSM will investigate whether the hybrid SSM is sufficiently expressive for the data governed by physics equations.