Brain Lesion Synthesis via Progressive Adversarial Variational Auto-Encoder

Abstract

Laser interstitial thermal therapy (LITT) is a novel minimally invasive treatment that is used to ablate intracranial structures to treat mesial temporal lobe epilepsy (MTLE). Region of interest (ROI) segmentation before and after LITT would enable automated lesion quantification to objectively assess treatment efficacy. Deep learning techniques, such as convolutional neural networks (CNNs) are state-of-the-art solutions for ROI segmentation, but require large amounts of annotated data during the training. However, collecting large datasets from emerging treatments such as LITT is impractical. In this paper, we propose a progressive brain lesion synthesis framework (PAVAE) to expand both the quantity and diversity of the training dataset. Concretely, our framework consists of two sequential networks: a mask synthesis network and a mask-guided lesion synthesis network. To better employ extrinsic information to provide additional supervision during network training, we design a condition embedding block (CEB) and a mask embedding block (MEB) to encode inherent conditions of masks to the feature space. Finally, a segmentation network is trained using raw and synthetic lesion images to evaluate the effectiveness of the proposed framework. Experimental results show that our method can achieve realistic synthetic results and boost the performance of down-stream segmentation tasks above traditional data augmentation techniques.

1. Introduction

Mesial temporal lobe epilepsy (MTLE) is one of the most common brain diseases and affects millions of people worldwide [11]. First-line treatment for MTLE is anti-seizure medicine but up to 30% of patients do not achieve seizure control, in these patients resective neurosurgery may be curative [15]. As a minimally invasive treatment, laser interstitial thermal therapy (LITT) can accurately locate and ablate target lesion structures within the brain [17]. LITT has been shown as an effective treatment for MTLE, and ablation of specific structures can be predictive of seizure freedom [16]. Region of interest (ROI) segmentation needs to be performed to enable quantitative analyses of LITT [18] (e.g., lesion volume quantification and ablation volume estimation). However, manual delineation is inevitably time-consuming and requires domain knowledge expertise. Automated lesion segmentation could improve the speed and reliability of lesion segmentation for this task.

In the literature, some segmentation methods for the post-ablation area have already been exploited. Ermiş et al. [2] developed Dense-UNet to segment resection cavities in glioblastoma multiforme patients. Pérez-García et al. [14] proposed an algorithm to simulate resections from preoperative MRI and utilized synthetic images to assist the brain resection cavity segmentation. Although the segmentation performance seems to be satisfied, it can be constrained by a small-scale dataset. Also, generated images of the rule-based resection simulation method can be less diverse, and imperfect synthetic results may compromise the performance of the segmentation model.

To mitigate the huge demand of images for training CNNs, methods utilizing generative adversarial network (GAN) [4] have been presented. Han et al. [6] generated 2D brain images with tumours from random noise to create more training samples. Kwon et al. [10] implemented a 3D α-GAN for brain MRI generation, including tumour and stroke lesion simulation. While these methods demonstrate the potential of GANs, there are some limitations. First, not all synthetic brain images have corresponding lesion masks, which means these methods may not be suitable to use for some down-stream tasks such as lesion segmentation. Additionally, these methods need extensive training samples to generate realistic results, which implies that the generalizability and robustness of these networks can not be ensured when the number of training samples is limited. Recently, Zhang et al. [19] designed a lesion-ware data augmentation strategy to improve brain lesion segmentation performance. However, its effectiveness still can be affected due to limited training samples.

To address the aforementioned issues, we develop a novel progressive brain lesion synthesis framework based on an adversarial variational auto-encoder, and refer it as PAVAE. Instead of simulating lesions directly, we decompose this task into two smaller sub-tasks (i.e., mask generation and lesion synthesis) to alleviate the task difficulty. For mask generation, we utilize a 3D variational auto-encoder as the generator to avoid mode collapses. We adopt a WGAN [1] discriminator with the gradient penalty [5] to encourage the generator to give more distinct results. We also design a condition embedding block (CEB) to encode semantic information (i.e., lesion size) to guide mask simulation. For lesion generation, we utilize a similar structure except replacing the CEB with a mask embedding block (MEB), to encode the shape information provided by masks to guide lesion synthesis. In the inference stage, we first sample from a Gaussian distribution to form the shape latent vector for mask simulation. Next, we combine the generated mask with an intensity latent vector sampled from a Gaussian distribution, and feed them into the lesion synthesis network to create the lesion image. Finally, we create new post-LITT ablation MR images from the generated lesion images. We train a lesion segmentation model using the framework nnUNet [7] to show the effectiveness of our method in synthesizing training images.

2. Methodology

Overall, the brain lesion synthesis task is decomposed into two smaller sub-tasks as described in Section 2.1. First, we design an adversarial variational auto-encoder to generate binary masks. To assist mask generation, we present a CEB to help encode mask conditions into the feature space, so that mask simulation can be guided by high-level semantic information. We adopt a similar architecture to generate lesions guided by binary lesion masks. Lesion masks are embedded into the feature space using a MEB. These additional blocks are described in Section 2.2. Finally, all models are trained using a four-term loss function as described in Section 2.3 to ensure reconstructions are accurate, latent spaces are approximately Gaussian, and the real and simulated distributions are similar.

2.1. Model Architecture

Fig. 1 illustrates our progressive adversarial variational auto-encoder for brain lesion synthesis. We design a progressive 3D variational auto-encoder to approximate both shape and intensity information of post-LITT ablation lesions as Gaussian distributions. Besides, a following discriminator can ensure that generated images are more realistic. The kernel size of all convolutional layers is set to 3 × 3 × 3, and Instance Normalization (InstanceNorm) and Leaky Rectified Linear Unit (LeakyReLU) are used after each convolutional layer. For the last convolutional layer, we use a Sigmoid function to scale output values between 0 and 1.

Fig. 1. The pipeline of our proposed framework for progressive brain lesion synthesis.

Our method contains two separate networks with similar structures for (a) mask simulation and (b) lesion synthesis, respectively. For the inference (c), we sample shape latent vectors and intensity latent vectors from Gaussian distributions to generate new lesions.

New lesion synthesis is performed as shown in Fig. 1 (c). We first randomly sample from a Gaussian distribution to build shape latent vectors which are input into D_S to generate new masks. Next, new masks and intensity latent vectors sampled from a Gaussian distribution are used as input for D_I to generate new lesions. Here new masks are responsible for controlling new shapes and intensity latent vectors are responsible for intensity patterns.

2.2. Condition and Mask Embedding Blocks

To add additional guidance for models in order to generate better results, we propose two separate modules shown in Fig. 2, a CEB and a MEB. For MEB, we follow the approach presented by SPADE [12]. First, masks are resized to the feature map resolution using nearest-neighbor downsampling. Next, learned scale and bias parameters are produced by three 3D convolutional layers. Finally, the normalized feature maps are modulated by the learned scale and bias parameters. For CEB, the structure is similar to MEB, but all 3D convolutional layers are replaced with linear layers since all input conditions are vectors.

Fig. 2. The structure of (a) condition embedding block (CEB) and (b) mask embedding block (MEB).

2.3. Loss Functions

To optimize the encoder, decoder and discriminator so that reasonable masks and realistic lesions are generated, four loss functions are utilized in our work: reconstruction Loss ℒ_Rec, KL divergence ℒ_KL, and GAN specific losses ℒ_G and ℒ_D. First, the reconstruction loss ℒ_Rec is used to ensure outputs have high fidelity to the ground truth images. The Kullback-Leibler (KL) loss ℒ_KL is imposed on the model to minimize the KL divergence between the intractable posterior distribution and the prior distribution (i.e., Gaussian distribution in latent space). Furthermore, we add Wasserstein loss functions (ℒ_G and ℒ_D) to the GAN in order to prevent results generated by the decoder from being fuzzy.

The reconstruction loss implemented in our framework is mean squared error (MSE) defined as:

$${\mathcal{L}}_{Rec}={\displaystyle {\sum }_i{\Vert x_g^{(i)}-x_r^{(i)}\Vert }^2},$$ (1)

where $x_r^{(i)}$ refers to the i^th real image within a mini-batch, and $x_g^{(i)}$ denotes the i^th generated image obtained from the decoder. MSE guarantees that real images and synthetic images look similar in general. However, synthetic images may lose some detailed information, which will make them appear indistinct.

The KL loss is defined as the KL divergence 𝒟_KL between the posterior distribution q (z|·) and the prior distribution p (z), which is formulated as:

$${ℒ}_{KL}={{\displaystyle \sum }}_i{\mathcal{D}}_{KL}\left[q\left(z|x_r^{(i)}\right)\parallel p(z)\right],$$ (2)

where $q\left(z|x_r^{(i)}\right)$ is the posterior latent distribution under the condition of $x_r^{(i)}$, and p (z) is a normal distribution for the latent vector z. By minimizing the KL divergence between the two distributions, the conditional distribution of the latent vector z approximates a Gaussian distribution.

To avoid generating images with blurriness and instability during training, we deploy the loss functions from WGAN [1] instead of the original GAN. The Wasserstein loss can be defined as:

$${ℒ}_D={\mathbb{E}}_{x_g~{\mathbb{P}}_g}[D(x_g)]-{\mathbb{E}}_{x_r~{\mathbb{P}}_r}[D(x_r)]+\lambda {\mathbb{E}}_{\widehat{x}~{\mathbb{P}}_{\widehat{x}}}[{(\parallel {\nabla }_{\widehat{x}}D(\widehat{x}){\parallel }_2-1)}^2],$$ (3)

$${ℒ}_G=-{\mathbb{E}}_{x_g~{\mathbb{P}}_g}\left[D\left(x_g\right)\right].$$ (4)

Compared with original GAN using a discriminator to differentiate whether images are real or fake, WGAN uses the Wasserstein distance to directly estimate the difference between two distributions ℙ_r and ℙ_g. This Wasserstein distance can be formulated as ${\mathcal{D}}_W={\mathbb{E}}_{x_r~{\mathbb{P}}_r}\left[D\left(x_r\right)\right]-{\mathbb{E}}_{x_g~{\mathbb{P}}_g}\left[D\left(x_g\right)\right]$, where 𝔼 denotes the maximum likelihood estimation and D(·) denotes the discriminator. In addition, we further include a gradient penalty regularization [5] to constrain ℒ_D to satisfy the 1-Lipschitz condition, so that ℒ_D will remain stable during network training. The gradient penalty regularization is formalized as $\lambda {\mathbb{E}}_{\widehat{x}~{\mathbb{P}}_{\widehat{x}}}\left[{\left(\parallel {\nabla }_{\widehat{x}}D(\widehat{x}){\parallel }_2-1\right)}^2\right]$, where $\widehat{x}$ denotes random interpolation between real samples and generated samples, and λ is a weighting factor. In our experiments, we fix the value of λ to 10.

3. Experiments

3.1. Dataset

In this study, 47 T1-weighted MRI scans of 47 patients are collected from a high-volume epilepsy surgery center which has already established expertise in using LITT for MTLE. Consecutive patients are included if they received LITT for MTLE and had concordant semiology, scalp electroencephalography (EEG) and structural MRI features of mesial temporal sclerosis, or had seizure onset confirmed within the hippocampus following stereo-EEG (SEEG) investigation. Ethical approval for the study was provided by institutional review board approval for the retrospective use of anonymized imaging. All T1-weighted images are first aligned to the MNI152 brain template [3]. A random split of the dataset is performed, keeping 20% (10 cases) of the whole dataset as the test set with the remaining 80% (37 cases) being used as the training set.

3.2. Implementation Details

Our framework is implemented within PyTorch 1.10.0 [13]. For network training, encoder and decoder layers are treated as the generator and optimized together. To optimize the generator and discriminator networks, we use two Adam optimizers [8]. The initial learning rate is set to 5e-5, and the batch size is set to 13 due to GPU memory limitations. For each model, we train for 1000 epochs individually using only data in the training set. For input images, we extract a 64 × 64 × 64 cube from raw MRI scans corresponding to a ROI containing the mask to ensure the entire lesion area is included within the image.

3.3. Evaluation Metrics

All metrics are evaluated and reported on the test set. First, we evaluate the lesion synthesis performance using three metrics, i.e., peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and normalized mean square error (NMSE). Moreover, to prove that generated brain lesions can further boost the performance of down-stream tasks, we use four metrics to measure brain lesion segmentation performance: Dice coefficient, Jaccard index, the average surface distance (ASD), and the 95% Hausdorff Distance (95HD).

3.4. Experimental Results

Comparison of Lesion Synthesis Performance

We first qualitatively compare the synthetic results of our framework with other existing methods. Specifically, we employ 3D VAE and 3D VAE w/ WGAN-GP as baseline models. For 3D VAE, only ℒ_Rec and ℒ_KL is utilized for model training. For 3D VAE w/ WGAN-GP, the structure is similar to the lesion synthesis network and all loss functions are utilized for training, but MEB has not been included. Note that all models are implemented with 3D convolution and 3D InstanceNorm layers. Triplanar views of synthetic lesions are shown in Fig. 3. Here, 3D VAE w/ WGAN-GP refers to 3D VAE followed by a WGAN-GP discriminator, PAVAE (Syn Mask) indicates lesion synthesis utilizes generated masks derived from the mask synthesis network. PAVAE (Real Mask) indicates that real masks are utilized to guide lesion synthesis. As can be found in Fig. 3, 3D VAE always generates fuzzy lesion images. Also, small lesions seem to be diffused, indicating the model has trouble in simulating small lesions. When a WGAN-GP discriminator is added to 3D VAE, results are clearer for big lesions. However, WGAN-GP still can not simulate small lesions. For our model, using synthetic masks to guide lesion generation i.e., PAVAE (Syn Mask), we observe even small lesions are successfully generated. Utilizing real masks for guidance results in synthetic lesions are closest to the ground truth among all compared methods (the rightmost column in Fig. 3). This highlights that the lesion synthesis network can generate realistic image intensity when provided a realistic lesion mask.

Fig. 3. Qualitative comparison for different generative models.

Quantitative results shown in Table 1 indicate that neither 3D VAE nor 3D VAE w/ WGAN-GP can achieve high SSIM and low NMSE simultaneously. As for our model, we obtain relatively good synthetic results merely using generated masks. If we replace generated masks with real masks, the final results achieve the highest PSNR and SSIM, and lowest NMSE among all methods.

Table 1. Quantitative comparison of synthetic results for different generative models.

Method	Metrics
	PSNR[dB]	SSIM[%]	NMSE
3D VAE [9]	21.40	10.18	76.34
3D VAE w/ WGAN-GP [5]	22.05	92.15	152.87
PAVAE (Syn Mask)	23.67	94.90	76.98
PAVAE (Real Mask)	32.74	99.29	15.68

Comparison of Lesion Segmentation Performance

For the purpose of proving the effectiveness of synthetic results generated by our framework, we conduct the brain segmentation model based on nnUNet. For the training set and test set split, we follow the same manner with the lesion synthesis task. We generate 100 synthetic lesion images using CarveMix [19] and PAVAE individually, and combine them with raw images to create new training datasets. We use Dice loss and Cross-entropy loss to train nnUNet for 200 epochs. Both quantitative and qualitative results are shown in Table 2 and Fig. 4. Here, NoDA means no data augmentation strategy is employed and only the real training dataset (37 samples) is used to train the model. TDA means traditional data augmentation strategies implemented in nnUNet, including random flip and rotations, and etc.. As shown in Table 2, our method has the best performance for three metrics (Dice, Jaccard, 95HD) and for the final metric ASD, only CarveMix has a slightly lower value. Furthermore, in Fig. 4, we can observe that all the results of the competing methods over-segment the LITT ablation volume. Our method yields accurate segmentation results, which most closely resembles the expert annotation.

Table 2. Comparison of segmentation results using different data augmentation techniques during training.

Method	Metrics
	Dice[%]	Jaccard[%]	ASD[voxel]	95HD[voxel]
NoDA [7]	66.69	51.19	1.17	3.52
TDA [7]	72.25	57.67	1.08	3.15
CarveMix [19]	73.29	58.77	0.97	3.24
PAVAE (Syn Mask)	74.18	59.95	1.00	2.77

Fig. 4. Segmentation results from different training datasets for nnUNet models.

4. Discussion and Conclusion

Building a 3D generative model may face several problems. The biggest ones can be mode collapses due to limited training samples and increasing computational complexity compared with 2D generative models. To tackle these issues, we have presented a progressive adversarial variational auto-encoder for brain lesion synthesis, which can generate reasonable masks and realistic brain lesions in a step-wise fashion. We further develop two types of blocks (i.e., CEB and MEB) to utilize both semantic and shape information to facilitate this lesion synthesis.

Experimental results show that our framework can create high-fidelity brain lesions and boost the down-stream segmentation model training compared with existing methods. However, as can be found in the quantitative results (Fig. 3), ground truth masks are still able to synthesize more realistic lesion images indicating a potential room for improvement when creating masks. Besides, all data used in this study was from a single center, further validation is required to evaluate its effectiveness on multi-center data and especially data acquired on different MRI scanners.