GANcMRI: Cardiac magnetic resonance video generation and physiologic guidance using latent space prompting

Abstract

Generative artificial intelligence can be applied to medical imaging on tasks such as privacy-preserving image generation and superresolution and denoising of existing images. Few prior approaches have used cardiac magnetic resonance imaging (cMRI) as a modality given the complexity of videos (the addition of the temporal dimension) as well as the limited scale of publicly available datasets. We introduce GANcMRI, a generative adversarial network that can synthesize cMRI videos with physiological guidance based on latent space prompting. GANcMRI uses a StyleGAN framework to learn the latent space from individual video frames and leverages the timedependent trajectory between end-systolic and end-diastolic frames in the latent space to predict progression and generate motion over time. We proposed various methods for modeling latent time-dependent trajectories and found that our Frame-to-frame approach generates the best motion and video quality. GANcMRI generated high-quality cMRI image frames that are indistinguishable by cardiologists, however, artifacts in video generation allow cardiologists to still recognize the difference between real and generated videos. The generated cMRI videos can be prompted to apply physiologybased adjustments which produces clinically relevant phenotypes recognizable by cardiologists. GANcMRI has many potential applications such as data augmentation, education, anomaly detection, and preoperative planning.

1. Introduction

Cardiac magnetic resonance imaging (cMRI) is a high-quality imaging modality for visualizing cardiac form and function, offering an unparalleled ability to capture specific details related to cardiovascular disease while sparing patients from the risks of radiation exposure. However, the application of cMRI in routine clinical practice is constrained by its high cost and lengthy acquisition time, making it impractical for sick patients that cannot hold still (‘breath holds’) or serve as the primary modality for serial measurement of disease progression. Imaging research is also hindered by the limited number of publicly available large-volume datasets.

Numerous studies have explored the tradeoff between acquisition time and resolution quality in MRI research, and methods like k-space undersampling approaches including partial Fourier (Noll et al., 1991; Peters et al., 2000; Ahn et al., 1986), sliding window (van Vaals et al., 1993; Foo et al., 1995; Korosec et al., 1996), parallel imaging (Pruessmann et al., 1999; Kozerke et al., 2004), and compressed sensing (Lustig et al., 2007; Usman et al., 2011; Jung et al., 2009) are in use to expedite scans while preserving image quality. These techniques seek to leverage less data but still yield high-quality images using advanced reconstruction algorithms.

These traditional techniques do not utilize previously collected data samples and generative machine learning to facilitate high-resolution reconstruction. However, because of the recent significant advancements in generative deep learning and synthetic data generation, this data-driven approach could pave the way for enhanced accuracy, resolution, and efficiency. Such approaches have been applied in brain MRI and knee MRI (Pinaya et al., 2022; Astuto et al., 2021), and could be similarly leveraged in cardiac MRI analysis and interpretation.

In this paper, we present GANcMRI, a cMRI specific StyleGAN model capable of generating cMRI videos, improving their temporal resolution, and visualizing disease progression with related physiologic prompting. We formulate video generation, similarly to Tian et al. (2020), as the problem of finding a suitable trajectory through the latent space across time of a pretrained image generator. Using our ED-to-ES and frame-to-frame methods, we find a trajectory in the latent space that corresponds to the progression over time. Starting from a random point in the latent space, we use this trajectory to generate a sequence of latent space points that correspond to a sequence of frames comprising a video. In addition to generating synthetic videos, we use Frame-to-frame method to increase the temporal resolution of real videos, by first projecting each of their frames to the latent space and interpolating the intermediate frames. Finally, we also propose a method for synthetic physiological prompting of cMRIs using latent space calculus to manipulate images. To demonstrate the performance of the approach, we verify that the synthetic videos have high quality based on their FID and FVD scores as well as provide blinded images and videos to domain experts (cardiologists) to evaluate whether they can identify generated vs. real images and if there are notable artifacts or changes from GANcMRI.

2. Related Works

Generative AI in MR

Recent advances in deep learning, specifically Generative adversarial networks (GANs) (Goodfellow et al., 2014), and Diffusion models (Dhariwal and Nichol, 2021; Rombach et al., 2022), allowed for high quality 2D medical image generation and the results have proven useful in various medical applications. Generative AI has been used to increase the size of image datasets (Diller et al., 2020; Pinaya et al., 2022), translate from one imaging modality to the other (Osman and Tamam, 2022; Li et al., 2022), privacy preservation, and superresolution (Wahid et al., 2022; Chen et al., 2018a,b)

Video data generation

While the achievements in image synthesis have been noteworthy, video synthesis is more complex because of the need to accurately model dynamics and the progression of time. To avoid expensive conv3d layers, the current state-of-art methods for video synthesis are predominantly leveraging architectures initially designed for image synthesis, such as StyleGAN (Karras et al., 2019) or Latent Diffusion (Ho et al., 2020). These methods are subsequently fine-tuned for video applications (Blattmann et al., 2023; Fox et al.), or undergo slight architectural modification to include the temporal dimension (Brooks et al., 2022; Skorokhodov et al., 2022). To the best of our knowledge, as of the current date, no papers have explored the application of generative AI in producing synthetic medical images with the temporal dimension.

StyleGAN2

StyleGAN introduced by Karras et al. (2019) is an architecture that uses concepts from classical GANs (Goodfellow et al., 2014), variational autoencoders (Kingma and Welling, 2022) and neural style transfer (Gatys et al., 2015) to produce high-quality images while having a well-structured latent space. StyleGAN2 is an improved version that allows for projecting real images to the latent space and semantic editing (Karras et al., 2020). StyleGAN achieved state of the art performance on different medical datasets (Woodland et al., 2022), and has also been used for the generation of volumetric images (Hong et al., 2021).

3. Methods

3.1. Data

We constructed our dataset using imaging data from the UK Biobank (UKBB) cardiac MRI cohort (Littlejohns et al., 2020). Our dataset consists of 45, 531 unique 4-chamber long axis (LAX) cine cMRIs from different UKBB participants. The cMRI videos, originally varying in height (144 − 210, avg ~ 207) and width (114 − 210, avg ~ 169), and with 50 frames, were uniformly resized to [256 × 256 × 50] for robust data processing. We select the first frame of the image to be the end-diastole (ED) frame (as described in UKB protocol by Littlejohns et al. (2020)), and the frame in which the LV area is the smallest (measured by ukbb-cmr (Bai et al., 2018)) as end-systole (ES) frame. 80−10−10 train-val-test split was used giving us 36, 425 videos in training dataset, 4, 553 videos in validation dataset and 4, 553 videos in test dataset. Each frame of the cMRI video was saved as a separate image giving us in total 2, 276, 550 images (1, 821, 250 in train, 227, 650 in val and, 227, 650 in test). While the image generator was trained on the full dataset, the number of video files used to develop video generation methods is given in Table 2.

3.2. Image generation

We train the StyleGAN2 architecture from scratch using TensorFlow’s official implementation (Karras et al.), on grayscale cMRI frames of resolution 256×256. To determine the best training checkpoint and to evaluate the performance of the image generation we use Frechet Inception Distance (FID) introduced by Heusel et al. (2018). Because FID uses the InceptionV3 network pretrained on RGB images, we convert our grayscale images to RGB before computing FID. However, since there is a potential issue with using a RGB network on grayscale images, we introduce a task-specific metric cFID. The only difference between cFID and FID is the backbone. Namely, cFID employs the encoder output (conv4 activations) from ukbb_cmr’s (Bai et al., 2018) pretrained fully convolutional network, trained for four-chamber segmentation in long-axis view. The conv4 layer activations are reshaped from (batch_size,height,width,num_channels) = (batch_size,16, 16, 256) to (batch_size,256, 256), then the mean is taken across channels resulting in the shape (batch_size, 256), i.e. for each frame we get 256 features.

Additionally, synthetic image quality and similarity to real distribution is assessed by a clinical cardiologist and cardiac imager with > 5 years of experience in cMRI. He was presented with a 100 pairs of real and fake frames, and he was asked to select a real one between the two (evaluation UI is shown in Figure 6).

3.3. Video generation

We treat videos as time-continuous signals x(t) in the image space $X\subseteq {ℝ}^{h\times w}$, such that $\forall t\in {ℝ}^{+}x(t)\in X$. Let $W\subseteq {ℝ}^{14\times 512}$ be a latent space learned by StyleGAN2 for image generation, and let F be a function that maps w ∈ W to X. Assuming that F is a time-signal preserving function then ∀ t F(w(t)) = x(t) i.e. every real video can be represented as a sequence of latent points. This relationship allows us to define videos in the latent space learned by StyleGAN as shown in Figure 1. Therefore, we focus on modeling w(t) signal in the latent space W, which is easier than modeling x(t), because any point in W corresponds to a valid cMRI frame. Note the that first frame corresponds to t = 1.

Figure 1:

A continuous signal in latent space corresponds to the continuous signal in the image space that represents a video.

3.3.1. ED-to-ES model

Cardiac function is a cyclical filling and pumping process, typically defined by maximum relaxation at end-diastole (ED) and maximum contraction at endsystole (ES). We introduce ED-to-ES method, modeling the time-signal in the latent space as:

$$w(t)=w(1)+t{k}_{ED\to ES}$$

Here, k_ED→ES is the trajectory from ED to ES, and its negative represents the ES to ED trajectory, enabling full cardiac cycle simulation.

To find k explicitly we first need a way to move from the image space to the latent space, in other words, we need to project real images onto the latent space. We use the projector from the original implementation of StyleGAN2 (P : X → W s.t. P(x) = w). Then k_ED→ES is calculated as follows. For each video, we project ED frame and ES frame to the latent space, and then take a mean (across videos) of the difference between ES latent point and ED latent point, i.e:

$$k_{ED\to ES}=\frac{{\sum }_{j=0}^{N}P\left(E{S}_{-}frms[j]\right)-P\left(E{D}_{-}frms[j]\right)}{N}$$

Where N is the number of videos, ES_frms is a list of ES frames, ED_frms is a list of ED frames, and ED_frms[j] and ES_frms[j] are from the same video.

3.3.2. Frame-to-frame model

Second, we propose Frame-to-frame model.

$$w(t)=w(t-1)+k_{i\to i+1}$$

In this case, we will have a trajectory k for each movement in time from w(i) to w(i + 1), in other words, we will model movement in time for every frame (50 frames), instead of just ED to ES (Figure 2). Similarly to the first method, we calculate k_i→i+1 by taking a mean across videos of the difference between i + 1-th frame corresponding latent point and i-th frame corresponding latent point.

$$k_{i\to i+1}=\frac{\sum _{j=0}^{N}P(\mathit{\text{videos}}[j][i+1])-P(\mathit{\text{videos}}[j][i])}{N}$$

Where videos is a list of videos, and videos[j][i] is the i-th frame from the j-th video. However, when using Frame-to-frame method we have to use multiple trajectories (k_i→i+1) and the transition between them might not be smooth, which would result in video jitter. To fix the sharp transition, we fit a PCA model PCA(·), computing 32 principal components of the dataset K = {k_1→2 ...k_i→i+1 ...k_49→50}, and use this pretrained model to update our frame-to-frame trajectories:

Figure 2:

ED-to-ES method finds a latent direction that corresponds to the movement of the heart from end-diastole to end-systole in the image space. Frame-to-frame method finds 49 latent directions corresponding to the movement of the heart from frame to frame in the image space

$$k_{i\to i+1}^{\prime }=k_{i\to i+1}-PC{A}^{-1}\left(PCA\left(k_{i\to i+1}\right)\right)$$

3.3.3. Applying ED-to-ES and Frame-to-frame for video generation

To generate a synthetic video resembling cMRIs in the UK Biobank, we first need to get the starting latent code w(1) that corresponds to the ED frame in the image space. We do so by picking a random point in the latent space w and then moving along the direction from ED to ES.

$$w(1)=w-0.7k_{ED\to ES}\text{ s.t. }{w}_0\sim 𝒩(0,1)$$

Next, we use either ED-to-ES or Frame-to-frame to predict future timesteps resulting in a 50 frame video. When using ED-to-ES method we use k_ED→ES trajectory to generate the first 25 frames, and −k_ED→ES to generate the last 25 frames, while for the Frame-to-frame method, we use a different trajectory for every frame transition.

3.3.4. Temporal super-resolution

Using the latent space projection of real cMRI frames we can increase the temporal resolution of cMRI. Given that {w₁,...w_M} w_i = P(i − th frame) are latent codes for all frames of a single cMRI video, (M is the number of frames) we compute the transition trajectories for this single video

$$t_{i\to i+1}=w_{i+1}-w_i$$

Then, we fit a PCA model (and call it tPCA) on the dataset of all transitions calculating the first (M − 6) principal components of the dataset T = {t_1→2 ...t_49→50}, and update the transition trajectories:

$$t_{i\to i+1}^{\prime }=t_{i\to i+1}-tPC{A}^{-1}\left(tPCA\left(t_{i\to i+1}\right)\right)$$

Finally, starting from w₁ we can compute latent codes for any number of intermediate frames. For $h\in ℝ$

$$w_h=w_1+\left(\sum _{i=1}^{\mathit{\text{floor(h)}}}{t}_{i\to i+1}\right)++((h-\mathit{\text{floor}}(h))t_{\mathit{\text{floor}}(h)\to ceil(h)}$$

For example, to compute a frame that comes in between of 4 and 5 we take h = 4.5.

$$w_{4.5}=w_1+t_{1\to 2}+t_{2\to 3}+t_{3\to 4}+0.5t_{4\to 5}$$

We conduct two experiments with temporal super-resolution. First, we reduce a 50-frame video to 25 frames by removing every second frame, then we use GANcMRI to interpolate intermediate frames and restore it to 50-frames. We also apply the same process to the original 50-frame video to obtain a 100-frame video, that we use for visual quality assessment.

3.3.5. Evaluation

To evaluate the video generation performance of the ED-to-ES and the Frame-to-frame methods, we use Frechet Video Distance (FVD) introduced by Unterthiner et al. (2019) from the implementation by Skorokhodov et al. (2022). When calculating FVD we sample 16 equally spaced frames from both the synthetic and real videos.

We assess temporal super-resolution performance using the mean structural similarity index (meanSSIM). meanSSIM is an average of SSIM computed between the real (removed) and the interpolated frames.

Additionally, a cardiologist conducts visual quality testing on 100 video sets using a custom UI (Figure 7). Each set consists of four cardiac MRI videos: a 50-frame real video, a 100-frame AI-enhanced video (temporal super-resolution), a 50-frame fully AI-generated video using ED-to-ES method, and a 50-frame fully AI-generated video using Frame-to-frame method.

3.4. Physiologic guidance

We aim to find the latent space trajectory k_{pheno_low→pheno_high} (for arbitrary phenotype), such that if we move the latent point along this trajectory the phenotype value in the corresponding image will increase. To do so, we first find the mean of the phenotype values and then classify all the cMRIs with the phenotype value lower than the mean as low, and all the cMRIs with the phenotype value higher than the mean high. Next, we embed ED frames of low cMRIs and high cMRIs and get lists low latents and high latents. Finally, we calculate k as follows:

$$\begin{gathered} k_{\mathit{\text{pheno\_low}}\to \mathit{\text{pheno\_high}}}= \\ =\frac{{\sum }_{l=0}^L\mathit{\text{high\_latents}}[l]}{L}-\frac{{\sum }_{s=0}^S\mathit{\text{low\_latents}}[s]}{S} \end{gathered}$$

Given that w is a latent code of any real or the generated image, we can increase the value of the phenotype in w by adjusting it as follows: w_adjusted = w + c k_{pheno_low→pheno_high.} To verify the accuracy, we use automated measurement methods on F(w_adjusted) and observe that there is a linear relationship between c and a measurement obtained using these methods. We focus on ED frames and the physiologic adjustments of left ventricular sphericity index (calculated in the same manner as Vukadinovic et al. (2023)) and left ventricular area, but the described method can be applied for any phenotype.

4. Results

4.1. Synthetic single frame cMRI images are indistinguishable from the real ones

Image generation performance was evaluated with FID, cFID, and the visual quality check was performed by a cardiologist. FID was computed between 50, 000 random samples from real image distribution and 50, 000 random samples from synthetic image distribution [FID = 92.58 cFID = 20.03].

In the evaluation of 100 pairs of fake and real images, the clinical cardiologist incorrectly identified the fake image as real in 60% of the cases and correctly identified the real image in the remaining 40% of cases. Figure 3 shows a sample of 4 real and 4 fake images.

Figure 3:

Top row: AI generated images. Bottom row: Real images.

4.2. Frame-to-frame video generation outperforms other approaches

All three methods for video generation (ED-to-ES, Frame-to-frame and, super-resolution) yielded high-quality videos with smooth transitions. We made fully synthetic ED-to-ES generated videos and fully synthetic Frame-to-frame generated videos publicly available in the supplementary material.

Frechet Video Distance (FVD) was computed between each two pairs of distributions (Table 1). FV D gives two indicators that Frame-to-frame model is superior to ED-to-ES model. First,

$$FVD\text{ (Real, Frame-to-frame) }<FVD\text{ (Real, ED-to-ES) }$$

means that frame-to-frame distribution is closer to the real distribution. And

$$FVD\text{ (F-to-f, F-to-f) }<FVD\text{ (ED-to-ES, ED-to-ES })$$

shows us that the distribution of videos produced by Frame-to-frame method is more stable than ED-to-ES method.

Table 1:

FVD for each two pairs of distributions

Real	Real 4.01	ED-to-ES 301.83	F-to-f 283.53
ED-to-ES		35.28	95.16
Frame-to-frame			8.81

The super-resolution quality was assessed separately, as described in methods 3.3.5, with meanSSIM. MeanSSIM computed on 100 video samples is 0.7.

All methods were evaluated in a visual quality test. Frame-to-frame videos were a clear winner, being assigned the highest ranking among synthetic methods in 50% of cases (Figure 4). Interestingly, the superresolution videos with 100 frames performed worse than their 50-frame competitors. While synthetic videos may appear convincingly real upon initial observation, a cardiologist was able to successfully distinguish all real videos from their synthetic counterparts with absolute accuracy.

Figure 4:

A cardiologist ranked the quality of real and synthetic cMRIs in a side-by-side comparison.

Finally, to showcase our method’s efficiency and scalability, we benchmarked it using an NVIDIA RTX 2080 Ti GPU and i9–9820X CPU. Our frame-to-frame approach took 0.73s ± 0.02s to produce a 50-frame 256×256 grayscale cMRI video in 100 trials and ED-to-ES took 0.71s ± 0.01s.

4.3. GANcMRI accurately reflects physiologic adjustment

As explained in methods 3.4, we perform morphological adjustment by moving along the trajectory k_{pheno low→pheno high}, and we do so by adding a scaled value of the trajectory to the latent code corresponding to cMRI image. In our experiments, we picked a scalar c to be in the interval from [−2, 3], because scalars outside of this range result in abnormally small/big phenotype values. Upon visual inspection, it is easy to see (Figure 5) that moving along the trajectory k_{sphericity_low→sphericity_high} in the latent space GANcMRI generates ED frames ranging from low sphericity index to high sphericity index.

Figure 5:

By increasing the scalar C we move along the latent space direction that corresponds to increasing the sphericity index in image space

To confirm the relationship numerically, we perform a pearson R test between the values of c and the actual phenotype value of the synthetic images corresponding to the modified latent vector. Indeed, the correlation is strong, resulting in pearson R = 0.98 and p value = 7.8·10⁻³⁵ for left ventricular sphericity index, and pearson R = 0.89 and p value = 6.9·10⁻¹⁸ for left ventricular area (Figures 8, 9).

These results enable conditional synthetic cMRI video creation by pre-selecting heart properties. The code and the weights to run conditional cMRI video generation with GANcMRI are available at https://github.com/vukadinovic936/GANcMRI

5. Discussion

We utilized a generative machine learning approach, the substantial UK Biobank cMRI data, and latent space calculus to generate cMRI images and videos. Nearing the level of being indistinguishable by clinicians, optimally generated cMRI videos can open new avenues in cardiac imaging research and clinical practice. We were able to generate individual synthetic cMRI frames of a quality indistinguishable from real ones. By interpreting videos as continuous signals within the image latent space, we model time trajectories on a frame-to-frame basis to produce cMRI videos of higher temporal resolution and superior quality compared to other video generation methods, although they remain distinguishable from the real ones. Finally, we demonstrate physiologic guidance can be applied to synthetic cMRIs to generate clinically relevant changes in the synthetic videos.

GANcMRI is a versatile tool with multiple downstream applications: it enables conditional video generation by tweaking physiological attributes of initial frames and employing Frame-to-frame model for time progression; when combined with language models, like GPT, it facilitates physiologic guidance with natural language prompts; it can produce synthetic datasets, thus mitigating privacy issues; and, it possesses the capacity for spatial super-resolution, transforming low-res MRI scans into higher resolution 256 × 256 resolution images.

A few limitations still persist. While our synthetic image generation methods are sufficient to be indistinguishable from natural images, limitations in generated videos still allow them to be identified. Generated videos present smoother transitions than the actual cardiac motion and lack the Brownian motion of the blood pool, making them distinguishable by cardiologists. Given our limited information approach, Brownian motion was unable to be simulated, however future iterations might be able to more closely replicate actual cardiac motion with cardiacphase specific motion. In our current model, the time progress trajectory is approximated with linear trajectories which may not capture complex temporal dynamics sufficiently. In our experiments, we did not identify any non-physiologic artifacts or unnatural images, however a more thorough evaluation was bottlenecked by cardiologist time. A more detailed evaluation and ablation studies are warranted in the future if ever put into clinical practice.

Further research should be undertaken to optimize the quality of cMRI imaging. Data-driven generative approaches might decrease the amount of necessary data, allowing for faster and cheaper scans. Super-resolution should be approached with caution given the potential for hallucination, however, our physiological prompting within the embedding space allows the generation of reasonably looking attributes consistent with cardiovascular disease. Further work remains to leverage polynomial regression to better approximate trajectories and the architecture to improve the model’s understanding of temporal dimension.

Appendix A. Implementation and Evaluation details

Table 2:

Number of files used for each method

Method	Number of files
ED-to-ES	547
Frame-to-Frame	151
Super-res	1
Spher phys	139 (68 low, 71 high)
LV phys	149 (83 low, 66 high)

Figure 6:

User interface for evaluating the quality of synthetic cMRI frames. 1 real and 1 fake frame are shown and the evaluator is asked to choose the frame that he/she thinks is real.

Figure 7:

User interface for evaluating the quality of synthetic cMRI videos. The evaluator is shown a real video, ED-to-ES generated video, Frame-to-frame generated video and super-resolution video. They click the radio button next to the video they think is real, and assign rank from 1 to 4 in order of quality to all videos.

Figure 8:

As the scalar C increases, LV area increases

Figure 9:

Moving along the direction of k_{small_lv_area→big_lv_area} is correlated with the increase in lv area.