CHeart: A Conditional Spatio-Temporal Generative Model for Cardiac Anatomy

Abstract

Two key questions in cardiac image analysis are to assess the anatomy and motion of the heart from images; and to understand how they are associated with non-imaging clinical factors such as gender, age and diseases. While the first question can often be addressed by image segmentation and motion tracking algorithms, our capability to model and answer the second question is still limited. In this work, we propose a novel conditional generative model to describe the 4D spatio-temporal anatomy of the heart and its interaction with non-imaging clinical factors. The clinical factors are integrated as the conditions of the generative modelling, which allows us to investigate how these factors influence the cardiac anatomy. We evaluate the model performance in mainly two tasks, anatomical sequence completion and sequence generation. The model achieves high performance in anatomical sequence completion, comparable to or outperforming other state-of-the-art generative models. In terms of sequence generation, given clinical conditions, the model can generate realistic synthetic 4D sequential anatomies that share similar distributions with the real data. The code and the trained generative model are available at https://github.com/MengyunQ/CHeart.

I. Introduction

CARDIAC imaging plays an essential role in cardiovascular image diagnosis and management [10]. Imaging modalities such as cine cardiac magnetic resonance (CMR) or ultrasound scans reveals the anatomical structure of the heart as well as its contracting and relaxing patterns [26]. A classical but long-standing research problem is to explore the associations between the three-dimensional (3D) cardiac anatomy and other non-imaging clinical factors, such as age, gender, diseases [5]. Besides 3D anatomical information, the temporal dynamic motion of the heart also contains information that is useful for clinical diagnosis and therapy selection [20], [32], [47]. It is of particular interest to develop computational tools that can bridge between spatial-temporal imaging features and non-imaging clinical factors. In this work, we aim to improve our understanding of the spatial-temporal cardiac anatomy and clinical factors from a generative modelling perspective. We propose a conditional generative model to model the interaction between imaging features and clinical factors. Given clinical factors as conditions, the proposed model can generate corresponding 4D spatial-temporal cardiac anatomies. We demonstrate that the generated 4D anatomies are realistic and consistent with real data distribution.

Lately, the field of conditional generative modelling has made tremendous progress, greatly driven by deep learning methods such as conditional generative adversarial networks (GAN) [34], conditional variational autoencoders (VAEs) [29], [48], flow-based models [41] and diffusion models [36]. These approaches enable efficient approximation of the underlying conditional distributions and generation of high-quality samples. Improvements in conditional generative models have been characterised by numerous developments in different generation tasks: image-to-image translation [13], [24], [27], style and lyrics-to-music generation [16] and text-to-image synthesis [12].

Apart from generating static images [36], generative models have also been applied to sequential data, such as videos [46], [53] and music [16]. In these applications, it is important to learn a model that is able to capture the inner connection of temporal sequences. To this end, long short-term memory (LSTM) [28], [52] and transformers [56] have been explored to learn the sequential progression of the latent representations of the samples. Some work also introduces spatiotemporal convolution and attention layers to learn temporal world dynamics from a collection of videos [46]. Sequential data contain both structural variations and motion information. Disentangled representation learning approaches such as DiSCVAE [59] have been proposed to separate representations of the motion features from the structural features.

In the field of medical imaging, several papers have explored incorporating non-imaging clinical factors into the image generation process. Dalca et al. [15] proposed a learning framework for building deformable brain image templates conditioned on age. Xia et al. [54] developed a model to generate synthetic brain images conditioned on age and the status of Alzheimer’s disease. For cardiac images, Biffi et al. [7] presented LVAE for interpretable classification of anatomical shapes into different clinical conditions. Krebs et al. [30] proposed to learn a probabilistic motion model for spatio-temporal cardiac image registration. Reynaud et al. [40] proposed a causal generative model to generate synthetic 3D ultrasound videos conditioned on a given input image and an expected ejection fraction. Campello et al. [9] proposed a conditional generative model in cardiac imaging to extract longitudinal patterns related to aging. Duchateau et al. [17] built a scheme for synthesizing pathological cardiac sequences from real healthy sequences. Amirrajab et al. [1] developed a framework for simulating cardiac MRI with variable anatomical and imaging characteristics. For cardiac temporal modeling scheme, some work [57], [60], [61] showed dynamic cardiac data could be described by low-dimensional latent representations, i.e. a conditional autoencoder to capture latent representations of data [61] or temporal smoothness applied as a regularisation term in the reconstruction loss function [60], [61]. These works provide useful insights for conditional medical image generation. However, the generation of a sequence of spatial-temporal cardiac anatomies from multiple clinical factors has been less explored.

In this work, we propose a conditional generative model that can generate realistic cardiac anatomical sequences conditioned on non-imaging factors including age, gender, weight, height and blood pressure. We name the Conditional Heart generation model as CHeart. The model employs a variational autoencoder to learn the latent representations for cardiac anatomies and a condition encoder to embed the clinical conditions into a condition latent vector. Then, a Temporal Module is designed to generate the condition-related sequential latent space based on the anatomy latent representations and the condition latent vector. The proposed model demonstrates a high diversity and fidelity in the generation, evaluated using structural overlaps and surface distance metrics, as well as clinical measure (ventricular volume and mass) distributions. The main contributions in this work are summarised as follows:

• We propose a spatial-temporal generative model for 3D cardiac anatomy that accounts for both the spatial variations and the temporal variations i.e. motion during the cardiac cycle.

• We leverage both imaging data and non-imaging clinical data to train the model, which allows the model to generate cardiac anatomical sequences conditioned on multiple clinical factors.

• We introduce a temporal module into the latent space of cardiac anatomy and conditions to model the complex sequential patterns of a beating heart.

• We demonstrate that the model can generate highly realistic and diverse cardiac anatomical sequences that follow the real data distributions.

II. Methods

The proposed generative model takes non-imaging clinical factors as input and generates a cardiac anatomical sequence. Fig. 1 illustrates the overall framework. The following sections provide more technical details. First, we introduce the conditional generative model. Then, we describe the temporal module for learning the sequential latent representations due to cardiac motion. Lastly, we demonstrate two applications of the generative model at the inference stage: anatomical sequence completion and anatomical sequence generation.

Fig. 1.

Overview of the CHeart model including training and inference stages. During training, an encoder is applied to learn the latent representations z_c, z₀ for the clinical conditions c and anatomy at the first time frame x₀. A temporal module models the trajectory of $z_{0:T-1}^c$ in the latent space across the temporal dimension from the initial latent vectors z_c and z₀. The decoder then generates the 4D cardiac anatomy sequence x_0:T–1 from the latent vectors on the temporal trajectory. The training process enables two inference mechanisms at test time: sequence completion and sequence generation. In sequence completion, the model is given x₀ and c, and generates the remaining sequence of anatomies in the cardiac cycle. In sequence generation, a random latent code z₀ sampled from the prior distribution and c are given to the model and the temporal module to generate the latent vector sequence $z_{0:T-1}^c$, which are used to generate synthetic cardiac anatomical sequence $x_{0:T-1}^{\prime }$.

A. Conditional generative model

Assume that we observe a dynamic sequence of anatomies of a subject, x_t(t = 0, 1, …, T – 1), where x_t denotes the anatomical segmentation at the t-th frame and T denotes the total number of time frames in a sequence. We also observe some clinical conditions c for this subject, where c could include factors such as age, gender, weight, height, blood pressure etc. Our aim is to learn the probability distribution of the anatomy x conditioned on c with a chosen model, p_θ(x|c), where θ denotes the model parameters. We seek a model p_θ(x|c) which is sufficiently flexible to be able to describe the data x. Deep neural networks have often been used for this modelling due to its complex modelling capacity [21], [29], [48]. Without losing generality, we first attempt to learn the distribution of anatomy at the first time frame, p_θ(x₀|c), which is often the end-diastolic (ED) frame in cardiac imaging.

We adopt the conditional β-VAE model [21], [29], [48] to learn the data distribution. The condition c is embedded as a condition latent vector z_c by the MLP, which integrates multiple clinical factors and enables exploration across the conditional latent space. The model consists of a decoder p_θ(x₀|z₀, z_c) and an encoder q_ϕ(z₀|x₀, z_c). The decoder p_θ(x₀|z₀, z_c) with parameters θ maps the latent variables z₀, z_c to the anatomy x₀. We assume a prior distribution p(z₀) over the latent variable z₀. The prior and the decoder together define a joint distribution, denoted as p_θ(x₀, z₀|z_c), which is parameterized by θ.

To turn the intractable posterior inference and learning problem into a tractable problem, we introduce a parametric encoder model q_ϕ(z₀|x₀, z_c) with ϕ as the variational parameters, which approximates the true but intractable posterior distribution p_θ(z₀|x₀, z_c) of the generative model, given an input x₀ and condition space z_c:

$$q_{\varphi }\left(z_0\mid x_0,z_c\right)\approx p_{\theta }\left(z_0\mid x_0,z_c\right)$$ (1)

where q_ϕ(z₀|x₀, z_c) often adopts a simpler form, e.g. the Gaussian distribution. By introducing the approximate posterior q_ϕ(z₀|x₀, z_c), the log-likelihood of p_θ(x₀|z_c) can be formulated as:

$$\begin{array}{l}\log p_{\theta }\left(x_0\mid z_c\right)={\mathbb{E}}_{z_0\sim q_{\varphi }\left(z_0\mid x_0,z_c\right)}\log \left[p_{\theta }\left(x_0\mid z_c\right)\right]\hfill \\ \begin{array}{ll}\hfill & \begin{array}{c}={\mathbb{E}}_{z_0\sim q_{\varphi }\left(z_0\mid x_0,z_{\text{c}}\right)}\log \left[\frac{p_{\theta }\left(x_0,z_0\mid z_c\right)}{q_{\varphi }\left(z_0\mid x_0,z_c\right)}\right]\\ \end{array}\hfill \\ \hfill & +{\mathbb{E}}_{z_0\sim q_{\varphi }\left(z_0\mid x_0,z_c\right)}\log \left[\frac{q_{\varphi }\left(z_0\mid x_0,z_c\right)}{p_{\theta }\left(x_0\mid z_0,z_c\right)}\right]\hfill \end{array}\hfill \end{array}$$ (2)

where the second term denotes the Kullback-Leibler (KL) divergence D_KL(q_ϕ || p_θ), between q_ϕ(z₀|x₀, z_c) and p_θ(z₀|x₀, z_c). It is non-negative and zero only if the approximate posterior q_ϕ(z₀|x₀, z_c) equals the true posterior distribution p_θ(z₀|x₀, z_c). Due to the non-negativity of the KL divergence, the first term in Eq. 2 is the lower bound of the evidence log[p_θ(x₀|z_c)], known as the evidence lower bound (ELBO). Instead of optimising the evidence log[p_θ(x₀|z_c)] which is often intractable, we optimise the ELBO:

$$\underset{\theta ,\varphi }{\max }ELBO=\log \left[p_{\theta }\left(x_0\mid z_c\right)\right]-D_{KL}$$ (3)

To better control the encoding representation capacity and encourage more efficient latent encoding, we adopt β-VAE by modifying VAE with an adjustable hyperparameter β [21]. As a result, the loss function of the generative model is formulated as:

$$\begin{array}{c}{\mathcal{L}}_{\theta ,\varphi }=-{\mathbb{E}}_{z_0\sim q_{\varphi }\left(z_0\mid x_0\right)}\log \left[p_{\theta }\left(x_0\mid z_0,c\right)\right]\\ +\beta \cdot D_{KL}\left[q_{\varphi }\left(z_0\mid x_0,c\right)\parallel p_{\theta }\left(z_0\right)\right]\end{array}$$ (4)

where the sign is negated so as we can minimise the loss function.

In practice, we use the reconstruction loss for the first term., i.e. how accurate the generative model p_θ(x₀) can be for reconstructing the anatomy x₀ from the latent vector z₀ using the decoder. The reparameterization trick is applied to replace the subscript of the expectation and express the random variable z₀ ~ q_ϕ(z₀|x₀, z_c) as some differentiable and invertible transformation of another random variable ϵ, so the expectation does not rely on q itself.

B. Motion modelling in the latent space

In the previous section, we modelled q_ϕ(z₀|x₀, z_c) and p_θ(x₀|z₀, z_c) for the first frame x₀ in a sequence. Here, to model the whole anatomical sequence x₀, x₁, …, x_T–1 on the clinical conditions c, we design a Temporal Module constructed using a one-to-many LSTM structure [49] with parameters ω, which generates the condition-related sequential latent codes based on z₀ and z_c. The detailed structure of the temporal module is illustrated in Fig. 2.

Fig. 2.

The temporal module for generating the sequential latent codes z_0:T–1, constructed with a one-to-many long short-term memory (LSTM) structure.

LSTM [22] is a variant of recurrent neural networks that consists of gating mechanisms and cell memory blocks. The first LSTM cell of the module takes the concatenation of the anatomy latent representation z₀ and the condition latent representation z_c as input, which is denoted as $z_0^c$. With the hidden state h₀ and cell state cell₀ being initialised to zero, it infers the latent $z_1^c$ at the next time frame. Each following LSTM cell, with shared weights, takes $z_{t-1}^c$ as input, updates the hidden state h_t and cell state cell_t, and infers the latent $z_t^c$. All the LSTM cells have shared weights. Each latent code $z_t^c$ contains information of both the anatomy at time t and the clinical conditions c. The cardiac anatomy of a dynamic sequence forms a temporal sequence $z_t^c$ in the latent space, where t = 0, 1, …, T. After the temporal module computes the latent codes $z_{0:T-1}^c$ across all the time frames, the decoder generates the anatomical sequence $x_t^{\prime }$ from $z_t^c$, illustrated in Fig. 1.

The overall loss function for modelling the anatomical sequence generation is formulated based on Eq.4:

$$\begin{array}{c}{\mathcal{L}}_{\theta ,\varphi ,\omega }=-{\displaystyle \sum _{t=0}^{T-1}}{\mathbb{E}}_{z_0\sim q_{\varphi }\left(z_0\mid x_0\right)}\left(\log \left(p_{\theta }\left(x_t\mid z_t,z_c\right)\right)\right)\\ +\beta D_{KL}\left(q_{\varphi }\left(z_0\mid x_0,z_c\right)\parallel p_{\theta }\left(z_0\right)\right)\end{array}$$ (5)

The training loss function is composed of two parts: 1) the reconstruction accuracy at all time frames, where we use cross-entropy for evaluating the performance in reconstructing the segmentation maps; 2) the KL divergence term, penalising the discrepancy between the learned prior and posterior distributions. The whole training process is performed end-to-end, with the encoder, temporal module and decoder being trained together. The VAE enables the model to learn a low-dimensional latent space that captures the underlying anatomical variations. By incorporating the temporal module, the model can effectively model the temporal dynamics in the cardiac images, enabling the generation of anatomically consistent and coherent sequences over time.

C. Inference

To demonstrate the performance of the proposed generative model at the inference stage, we carry out two benchmark tasks, namely anatomical sequence completion and anatomical sequence generation, as shown in the right panel of Fig. 1.

In anatomical sequence completion, the model is given the anatomy at the first time frame x₀ and clinical conditions c. It is asked to generate the remaining sequence of anatomies across the cardiac cycle. The model maps x₀ and c to their latent representations z₀ and z_c, predicts the sequential latent codes $z_{0:T-1}^c$ through the temporal module and finally reconstructs the full sequence of cardiac anatomy $x_{0:T-1}^{\prime }$ using the shared-weight decoders.

In anatomical sequence generation, the model is only conditioned on the clinical factors c and it does not require any anatomy as input. Since the model has learnt the distribution of anatomical latent variable p_z0, we can draw samples z₀ in the latent space from a Gaussian distribution 𝒩(0, 1) and concatenate it with the clinical latent code z_c. We then provide the concatenated latent code $z_0^c$ to the temporal module to predict $z_{0:T-1}^c$ and generate the full anatomical sequence $x_{0:T-1}^{\prime }$ using the decoder.

D. Evaluation

To evaluate the conditional generative model, we use quantitative measures to assess the generated anatomy, as well as use clinical measures to assess the distribution similarity.

First, we employ the Dice coefficient, the Hausdorff distance (HD) and the average symmetric surface distance (ASSD) which compare the similarity of the generated cardiac anatomy to the ground truth anatomy associated with the same clinical conditions.

Second, we derive five imaging phenotypes including the left ventricular myocardial mass (LVM), LV end-diastolic volume (LVEDV), LV end-systolic volume (LVESV), right ventricular end-diastolic volume (RVEDV) and RV end-systolic volume (RVESV). We evaluate differences between generated data and real data with the same clinical conditions, denoted as d_phenotype. Furthermore, these phenotypes are closely associated with age and gender [5]. We calculate the distributions of the imaging phenotypes against age and gender, and compare the generated data to the real data. The comparison is illustrated qualitatively using density plots and quantitatively using the Kullback–Leibler (KL) divergence and Wasserstein distance (WD). The KL divergence [14] is an information-theoretic measurement of the similarity between two probability mass functions. Similarly, WD [2] measures the distance between two probability distributions and can be computed as:

$$\text{WD}=\underset{\gamma \sim \prod (P,Q)}{\inf }{\mathbb{E}}_{(u,v)\sim \gamma }[\parallel u-v\parallel ]$$ (6)

where ∏(P, Q) is the set of all joint distributions over u and v. WD can be seen as the minimum work needed to transform one distribution to another, where work is defined as the amount of mass that must be moved from u to v to transform P to Q and the distance to be moved.

III. Experiments

A. Data sets

A short-axis 3D cardiac MR dataset of 1,383 subjects was used, acquired from Hammersmith Hospital, Imperial College London. Each cardiac cine image sequence comprises 20 time frames (T = 20) covering one complete cardiac cycle, with a spatial resolution of 1.25 mm × 1.25 mm × 2 mm. The temporal resolution ranges from 0.041 to 0.048 seconds per frame, accommodating variations in the heart rate. The cardiac anatomy is described by the image segmentation map with four labels: background, the left ventricle (LV), myocardium (Myo) and the right ventricle (RV). Ground truth segmentation at end-diastolic (ED) and end-systolic (ES) frames was generated by using a multi-atlas segmentation method [3], then quality controlled and manually corrected by an experienced cardiologist using itkSNAP [58]. A state-of-the-art nnU-net model [23] was trained using the ED and ES segmentation and then deployed to all time frames generating the 3D-t segmentation, followed by manual quality control. To eliminate the influence of image orientations in the generation, all 3D-t segmentation were rigidly aligned to a template space using MIRTK [42], [44] and cropped to a standard size of 128 × 128 × 64. In this way, the generative model will focus on learning subject-specific variations of the anatomy instead of image orientations.

In terms of demographic information, all subjects were healthy volunteers, with 775 females and 608 males, aged between 18-73 years old, weighed between 33-131 kg, with height between 142-195 cm and systolic blood pressure (SBP) between 79-183 mm Hg. When incorporating the clinical information into the model, age was represented as a categorical factor with seven age groups with an interval of 10 years, from 10 to 80 years old. The dataset was randomly split into three subsets for training (n = 968), validation (n = 138) and test (n = 277).

B. Experimental setup

1). Implementation

The model was implemented in PyTorch [37]. The encoder q_ϕ, consisted of four 3D convolution layers, one flatten layer and one bottleneck layer, outputting the latent code z₀. The condition mapping network was constructed using an MLP, outputting latent code z_c for input conditions c. A latent dimension of 32 was used for both z₀ and z_c, and 64 for the concatenated latent vector $z_0^c$. The decoder consisted of one flatten layer and four 3D transposed convolution layers. All convolution and transposed convolution layers in the encoder and the decoder used a kernel size of 4. The temporal module was built with one-layer LSTMCells. The regularisation weight β in β-VAE was set to 0.001. The model was trained using the Adam optimiser with a learning rate of 5·10^–4 and a batch size of 8. It was trained for 500 epochs and an early stopping criterion was used based on the validation set performance. The training took 17 hours on an NVIDIA RTX A6000 GPU.

2). Baseline methods

Currently, there is no other existing work for performing conditional generation of 3D-t cardiac anatomies. For comparison, we implemented the following baseline generation methods developed in other application domains, extending them from 2D image generation to 3D-t data generation:

CGAN

A conditional version of the generative adversarial network (GAN) originally developed for MNIST images [34]. Note that the model can only perform cardiac sequence generation, not sequence completion.

CVAE

The conditional generative model CVAE [48]. It was modified to adapt to this application. CVAE applied condition incorporation by concatenating conditions and anatomies in both the encoder and decoder.

CVAE-GAN

A conditional variational generative adversarial network proposed in [6]. It is a general learning framework that combines a VAE with a GAN for synthesizing natural images in fine-grained categories.

PCA

The principal component analysis (PCA) [25]. It is a classical method for dimensionality reduction, which aims to preserve as much of the variation in data as possible using the principal components. Note that the PCA is only used for performing sequence completion, but not for sequence generation.

C. Sequence completion

A well-known challenge to generative modelling is the difficulty in evaluation, as we normally do not have access to the ground truth data distribution, e.g. the distribution of all possible cardiac anatomies in our case. Therefore, we adopt anatomical sequence completion as a surrogate task for evaluating the model performance. The sequence completion experiments were conducted to assess the ability of capturing the sequential information given the first frame of a cardiac anatomy sequence. One example of sequence completion is shown in Fig. 3. It can be seen in the figure that the generated anatomies across time frames maintain the same heart structures as the ED frame and capture the temporal motion pattern through time, contracting first and then expanding.

Fig. 3.

An example of sequence completion, arranged in two rows with the left-to-right and top-to-bottom order. With the end-diastolic (ED) frame in time t = 0 and conditions c as input, the model generates the remaining anatomical sequence at time frame t = 1–19, shown within the gray box. The top row depicts anatomy images at time frame t = 0 – 9, and the bottom row depicts at time frame t = 10 – 19.

The sequence completion accuracy is evaluated between the generated anatomy and ground truth across the whole sequence in terms of the Dice metric, HD and ASSD for three structures: LV, Myo and RV. Table I reports the sequence completion accuracy of the proposed model and compares it to other generative models including CVAE-GAN [6], CVAE [48] and PCA [25]. It shows that the proposed model achieves a good sequence completion accuracy with an average Dice metric of 0.874, HD of 5.842 mm and ASSD of 1.462 mm, which is comparable to or outperforms the other three generative models in most metrics. In addition, we conducted evaluations at the basal, mid-cavity, and apical slices. The proposed model achieved an average Dice metric of 0.929, 0.927, and 0.878 for LV at the three locations, surpassing the corresponding metrics of the other three generative models.

Table I. The Sequence Completion Performance of Different Models in terms of Dice, Hausdorff distance (HD), average symmetric surface distance (ASSD). Mean and standard deviation are reported. Asterisks indicate statistical significance (* : P ≤ 0.05) when using a paired Student’s t-test comparing the performance of the proposed method to other methods.

	Dice (unit: 1)
	LV	Myo	RV	Average
CVAE-GAN [6]	0.845*±0.028	0.697*±0.054	0.832*±0.028	0.791*±0.032
CVAE [48]	0.900±0.023	0.800*±0.040	0.894±0.023	0.864*±0.026
PCA [25]	0.906±0.022	0.810±0.038	0.901±0.023	0.872±0.025
Proposed	0.908 ±0.023	0.814 ±0.037	0.902 ±0.021	0.874 ±0.024
	HD (unit: mm)
	LV	Myo	RV	Average
CVAE-GAN [6]	10.361*±1.475	9.571*±1.379	14.070*±3.736	11.334*±1.849
CVAE [48]	5.920*±1.335	5.891*±1.055	6.525±1.076	6.112*±1.049
PCA [25]	5.517 ±1.029	5.710±1.125	6.165 ±1.072	5.797 ±0.978
Proposed	5.535±1.180	5.576 ±0.955	6.445±1.067	5.842±1.017
	ASSD (unit: mm)
	LV	Myo	RV	Average
CVAE-GAN [6]	2.120*±0.390	1.670*±0.236	2.244*±0.399	1.983*±0.306
CVAE [48]	1.657*±0.348	1.376*±0.212	1.622*±0.305	1.461±0.280
PCA [25]	1.565±0.324	1.319*±0.221	1.519 ±0.301	1.490±0.305
Proposed	1.535 ±0.330	1.298 ±0.208	1.620±0.323	1.462 ±0.266

We also performed paired student’s t-tests between the results generated by our method and those of competing methods. The performance metrics of the proposed model marked with asterisk in Table I were significantly better than other methods at a p value smaller than 0.05. On a different cardiac MR dataset, [4] reports an average Dice metric of 0.94, 0.88, 0.90 for LV, myocardium and RV, respectively, for interobserver variability in manual cardiac image segmentation (Table 3 of [4]). The Dice metric of the proposed generative model is close to this value, which indicates its high performance and capability for anatomical sequence completion.

Table III. Comparison of sequence generation performance among CGAN, CVAE, CVAE-GAN and the proposed model. The clinical measures derived from each real sample are compared to those derived from 20 synthetic samples of exactly the same conditions. The mean and the minimal differences of the clinical measures are reported here.

Model	dLVEDV (mL)		dLVESV (mL)		dRVEDV (mL)		dRVESV (mL)		dLVM (g)
	mean	best/min	mean	best/min	mean	best/min	mean	best/min	mean	best/min
CGAN [34]	35.58±20.33	15.66±16.67	20.06±9.71	19.74±9.72	51.47±25.25	14.71±17.12	17.57±12.19	17.04±12.18	38.26±19.15	10.40±11.23
CVAE [48]	35.74±16.99	4.91 ±9.84	13.92±6.06	1.87±3.46	44.97±21.58	6.46 ±12.92	19.49±9.21	2.86±5.74	23.07±9.96	2.70 ±4.33
CVAE-GAN [6]	51.32±20.40	6.33±11.96	19.80±6.53	1.69 ±2.57	48.94±28.66	8.28±17.52	25.26±10.99	2.57 ±4.11	51.03±11.40	8.29±7.91
Proposed	25.93 ±17.47	6.87 ±12.09	11.74 ±8.41	3.54±6.25	34.63 ±21.31	6.88±12.87	15.54 ±11.33	5.12±9.19	17.34 ±9.89	2.95±5.62

D. Sequence generation

Apart from the sequence completion task, we also perform anatomical sequence generation and evaluate how close the generated anatomical sequences are to the real data. In this experiment, we generate new synthetic anatomies of the heart by providing the clinical conditions as the only input to the model. Given the stochastic nature of the VAE generation, for each set of input conditions, multiple anatomical sequences can be generated. We draw 20 random samples from the Gaussian distribution of the latent vector, and correspondingly generate 20 synthetic anatomical sequences for this input condition set.

We first compare the synthetic anatomies to the real anatomy with the same clinical conditions and evaluate the mean similarity and the best similarity across 20 samples, in terms of the Dice metric, HD, ASSD and differences of clinical measures. This is similar to the random average or random best evaluation in other recent generation works in computer vision [38]. Table II shows that the proposed model achieves a reasonably good sequence generation accuracy with a mean Dice metric of 0.713, HD of 10.940 mm and ASSD of 3.023 mm. We also reported the best value of each measurement, with a significantly improved maximum Dice of 0.793, minimum HD of 8.166 mm, and ASSD of 2.049 mm. This perhaps means the proposed method can capture a wide variation of anatomies and thus draw a sample that is close to the real sample. When we compare the differences of clinical phenotypes, Table III shows that our model achieved the lower measurement difference with a mean difference of 25.93 mL, 11.74 mL, 34.63 mL, 15.54 mL and 17.34 g and minimum difference of 6.87 mL, 3.54 mL, 6.88 mL, 5.12 mL and 2.95 g for LVEDV, LVESV, RVEDV, RVESV and LVM, respectively. The results of mean and best values indicate that our model achieves similar (Dice) or better sequence generation accuracy (HD, ASSD, difference in clinical measures) compared to other methods. The best values of the metrics indicate the high fidelity of the proposed generative model, which refers to the degree to which the generated samples resemble the real ones [35], [43]. It is important to acknowledge that in anatomical sequence generation, the model is not expected to replicate existing anatomies. But instead, the model generates a plausible anatomy that fulfils certain conditions, which is compared to a real anatomy with the same conditions.

Table II. Comparison of sequence generation performance between CGAN, CVAE, CVAE-GAN and the proposed model, in terms of mean and best Dice metric and contour distance metrics for the average performance over LV, RV and Myo. The best value across 20 samples for Dice metric (maximum), HD (minimum) and ASSD (minimum) are reported. Asterisks indicate statistical significance (*: p ≤ 0.05) when using a paired Student’s t-test comparing the performance of the proposed method to other methods.

Model	Dice (unit: 1)		HD (unit: mm)		ASSD (unit: mm)
	mean	best/max	mean	best/min	mean	best/min
CGAN [34]	0.713 ±0.061	0.717*±0.061	15.533*±2.258	13.956*±2.326	3.004 ±0.714	2.862*±0.712
CVAE [48]	0.694±0.056	0.789±0.049	11.461*±1.809	8.321±1.536	3.380*±0.710	2.317*±0.540
CVAE-GAN [6]	0.645*±0.052	0.774±0.039	16.844*±2.008	12.105*±1.815	3.693*±0.709	2.185±0.394
Proposed	0.713 ±0.058	0.793 ±0.052	10.940 ±2.343	8.166 ±1.621	3.023±0.757	2.049 ±0.521

Further, we visualised two examples of anatomical sequence generation in Fig. 4. For each example, we show five random synthetic samples which share the same clinical conditions as the real sample. It illustrates that the LV and RV structures look realistic and their shapes share a high similarity to the real anatomy. The contracting pattern of the ventricles and myocardium from ED to ES frame also looks realistic and similar to the real sample. This demonstrates our model can capture the overall anatomy and temporal dynamics of the heart during generation. The five samples with the same conditions also present certain degrees of variations, which demonstrates the diversity of synthetic data. This is due to the Gaussian sampling part of the generation process and reflects the individual differences between two hearts even if they are of the same gender and age, which can be caused by genetic, environmental, lifestyle and many other factors that are not easily accounted for by the model.

Fig. 4.

Visualisation of synthetic anatomies (last five columns) generated by the model, compared to the real anatomy (first column) with the same clinical conditions (text annotation). The whole anatomical sequence is generated but only ED and ES frames are shown here. The first and second rows of each example show the ED and ES frames of the cardiac anatomical sequence.

To further evaluate whether fidelity and diversity of the generated samples with respect to the real samples, we assess the distance between their distributions, conditioned on age, a common factor of interest in clinical research. In addition to quantitative assessments, we conducted qualitative comparisons by evaluating the distributions of five clinical measures for both real and synthetic anatomies against age, including LVM, LVEDV, LVEV, RVEDV, and RVEF, illustrated in Fig. 5. Compared to other methods, the synthetic data distributions from our model closely resemble the real distributions and cover the full variability of the real samples. Table IV reports the KL divergence and Wasserstein distance between synthetic and real data distributions. The proposed model achieves the best KL or WD metrics in most clinical measurements, with KL divergence values of 0.034, 0.043, 0.034, 0.039, 0.031, and WD values of 15.053, 5.773, 12.214, 9.182, 9.215 for LVEDV, LVESV, RVEDV, RVESV, and LVM, respectively. These results demonstrate that the synthetic data generated by our model maintains a distribution against age that is similar to the real data.

Fig. 5.

Distributions of clinical measures for real data and synthetic data. Each graph displays a kernel density plot of an imaging phenotype (LVM, LVEDV, LVESV, RVEDV, RVESV) against age. For each plot, the x-axis denotes age and the y-axis denotes the value of the imaging phenotype. Darker areas in the plot indicate the regions where the data is more concentrated. Lighter areas show the regions where the data is sparser.

Table IV. KL divergence and Wasserstein distance between synthetic data distribution and real data distribution.

Distribution similarity	Kullback–Leibler (KL) divergence					Wasserstein Distance (WD)
	LVEDV	LVESV	RVEDV	RVESV	LVM	LVEDV	LVESV	RVEDV	RVESV	LVM
CGAN [34]	0.023 ±0.001	0.019 ±0.001	0.036±0.004	0.022 ±0.001	0.050±0.006	33.687±1.173	19.982±0.025	41.643±4.161	17.434±0.036	35.395±2.933
CVAE [48]	0.039±0.005	0.041±0.004	0.042±0.004	0.034±0.003	0.030±0.003	11.929 ±2.116	7.017±0.964	14.680±2.869	9.665±1.051	10.365±1.703
CVAE-GAN [6]	0.153±0.025	0.023±0.003	0.046±0.006	0.064±0.008	0.098±0.019	27.001±2.809	8.425±1.771	24.614±3.202	9.748±2.675	43.251±4.566
Proposed	0.034±0.002	0.043±0.002	0.034 ±0.002	0.039±0.002	0.031 ±0.002	15.053±3.597	5.773 ±1.358	12.214 ±2.408	9.182 ±2.145	9.215 ±1.713

E. Temporal dynamics

The proposed model encoded the anatomical and clinical information $z_0^c$ of the first frame (ED) and generated the latent vectors $z_t^c$ for the following frames by the temporal module. We use the dimensionality reduction technique, t-distributed stochastic neighbor embedding (t-SNE) [31], to visualise the latent space $z_t^c$ of the generated anatomical sequences, as shown in Fig. 6. The sequential latent codes $z_{0:T-1}^c$ start at ED (t = 0) and move along a cyclic path in the latent space. It shows that the generative model can capture the temporal dynamics of the anatomy during the heartbeat and form a cyclic pattern as a real heart [45]. More overlapped areas between frames 9 to 18 show that the variation of anatomies is smaller in the relaxation stage, which demonstrates the nonlinear trajectories of cardiac motion. We plotted one example of the anatomical sequence at time frame 0, 3, 4, 6, 9, 12, 15, 18 in the figure. Through the time frames, the anatomies present first decreased and then increased LV volumes. The thickness of Myo has the opposite trend, which is consistent with the contraction and relaxation pattern of the heart [19].

Fig. 6.

T-distributed stochastic neighbor embedding (t-SNE) visualization of latent space for generated anatomical sequences from frame 0 to frame 18. Each dot represents a single time frame of a sample, with colors indicating the frame index. A sequence of anatomies, decoded from a corresponding sequence of latent codes, belonging to one subject, is visualised in the figure.

F. Condition manipulation

With the conditional generative model, we are able to simulate the change of anatomy when certain conditions (e.g. age) change. Fig. 7(a) shows a series of generated anatomies during ageing, when the condition age increases but all the other conditions as well as the latent vectors drawn from the Gaussian distribution are fixed. The difference map comparing the aged anatomy to the anatomy at 10-20 years old shows subtle changes to the LV and RV structures. We further generate 200 random samples of the synthetic ageing anatomies and derive the clinical measures. Fig. 7(b) illustrates the longitudinal evolution of these measures, stratified by gender. We observe a longitudinally increasing trend in LVM during ageing and a decreasing trend in LVEDV, consistent with findings in clinical literature [18] (Figure 3 of [18]). It demonstrates the potential of using this model for simulating anatomical data distributions. However, we need to be cautious in interpreting this result, as our training data is cross-sectional instead of longitudinal and also the mechanism of cardiac ageing is complex, confounded by more factors (genetics, lifestyle etc) than the five conditions we used in this work.

Fig. 7.

(a) An example of the synthetic cardiac anatomy during ageing. The first and third rows show the cardiac anatomies at end-diastolic (ED) and end-systolic (ES) frames.

The second and fourth rows show the difference maps between the aged anatomy 20-80 years old and the anatomy at 10-20 years old. (b) The simulated evolution of clinical measures (LVM, LVEDV, LVESV, RVEDV, RVESV) by generating 200 samples of gender-specific ageing cardiac anatomy and plotting their mean measures with 95% confidence interval.

IV. Discussion

The proposed model is built upon a β-VAE for learning the latent space of the cardiac anatomy. It integrates a conditional branch to model the influence of multiple clinical factors on the generation process and uses a temporal module to model the temporal relationship of anatomical latent vectors during cardiac motion. The experiments demonstrate good performance in both anatomical sequence completion and sequence generation tasks, qualitatively and quantitatively. The model enables condition manipulation for demonstrating the impact of clinical factors on anatomical shape variation. When using the common clinical measures (ventricular volumes and mass) for evaluation, the distribution of generated anatomies is close to the real data distribution visually (Fig. 5) and quantitatively (Table IV), which indicate both the fidelity and diversity of the generation. While the model performs well in generating anatomically coherent structures, further improvement can be made in terms of achieving a closer similarity between the distribution of generated anatomies and real data distribution. There is also potential for further exploration of the relationship between cardiac motion and clinical conditions

We foresee there are several potential downstream tasks for the generative cardiac anatomy model, including discovering patterns in large datasets, facilitating out-of-distribution detection and generating synthetic data etc. First, by training a generative model on a large dataset of cardiac anatomies, the trained model can capture complex patterns and variations of the anatomy associated with different clinical factors. This knowledge can be valuable for understanding population-level characteristics, identifying risk factors and informing public health strategies. Second, by learning the distribution of normal cardiac anatomy and dynamics, the proposed model can identify patterns of a given anatomy that deviate from the norm, indicating potential anomalies that require further investigation. More importantly, the proposed method is a conditional generative model, which means it can learn the norm specifically for certain conditions (e.g. a gender and age group) and evaluate the deviation from the norm in a personalised manner. Third, the trained generative model can provide a large amount of synthetic data for other tasks. Synthetic data can be used for performing data augmentation for training machine learning models [8], creating synthetic fair data to improve the fairness of prediction models [11], [50], or used as digital anatomies for performing in-silico trials [55]. Diverse and realistic synthetic data will alleviate the data scarcity issue in the medical field, where real data are often limited or not easy to share. This includes the creation of synthetic data for privacy-preserving research [39], [51].

There are a few limitations of this work. The first limitation is the high computational cost during training to learn the spatio-temporal patterns from 4D data, even after cropping the images to 128 × 128 × 64 and using sequences of only 20 time frames. An interesting future direction is to reduce the computational complexity of high-dimensional and high-resolution medical imaging data. Second, here we use a segmentation map as a representation of the anatomy so that the generative model can focus on learning the variations of anatomy, instead of intensity image styles. Future explorations could be extended to the generation of intensity images for the heart [1] or using mesh as a representation for the anatomy [33], which may be computationally more efficient. Third, we use a cross-sectional imaging dataset of mainly healthy volunteers for training the generative model, due to the challenge of curating large-scale longitudinal datasets with high spatial resolution. It would be interesting to extend this to longitudinal and clinical imaging cohorts with cardiac diseases in the future.

V. Conclusion

In this work, we propose a novel conditional generative model that is able to synthesise spatial-temporal cardiac anatomies given clinical factors as input. It demonstrates the feasibility of generating highly realistic synthetic 3D-t anatomies for the heart that captures both the anatomical variations and motion of the heart. The work paves the way for further generative modelling research in cardiac imaging, such as incorporating disease types or representing anatomy as meshes. It also has the potential to be applied to downstream tasks, such as performing data augmentation based on various anatomies, building condition-specific atlases and performing biomechanical modelling of the heart etc.