Unsupervised anomaly detection with deep generative models: an experimental analysis of model variability and mitigation strategies

Abstract.

Purpose

Unsupervised anomaly detection allows identifying anomalies from unlabeled data, making it useful for neuroimaging analysis and computer-aided diagnosis. Given an individual’s scan, we use a generative model to construct a subject-specific image of healthy appearance and compare both images with identify anomalies. Such approach has drawbacks as the reconstructions are imperfect, and model variability is not taken into account.

Approach

We study model variability arising from using different random seeds during training and explore strategies to mitigate the effect of unwanted reconstruction errors and variability. The strategies include model ensembling, anomaly map normalization, and anomaly map designs based on single or multiple pseudo-healthy reconstructions. We compare these approaches in the context of dementia-related anomalies on 3D FDG PET from ADNI using variational autoencoder models.

Results

Our experiments suggest that variance between models can be reduced by aggregating their reconstructions in a $Z$-score based anomaly map. This strategy is highly effective, but computationally expensive, as it requires training several instances of the model. An alternative strategy is to normalize the anomaly map using statistics computed from a healthy validation set. We show that this normalization strategy substantially reduces performance variability across models and even increases anomaly detection performance in certain cases.

Conclusion

We highlight a largely overlooked source of variability in deep generative models, their random seed, which can lead to substantially different anomaly detection performance and biased evaluation. Our experimental analysis emphasizes that it is crucial to design robust anomaly maps and to mitigate the impact of randomness-induced variability, by accounting for model reconstruction errors for instance using $Z$-score ensembling, or healthy-control normalization, to support more stable and trustworthy deployment in real-world clinical practice.

1. Introduction

Machine learning methods have shown promise in the quantitative analysis of neuroimaging data.¹ They enable the detection of subtle changes in brain patterns amid natural variations, a critical aspect in the early diagnosis of neurodegenerative disorders. Within the unsupervised anomaly detection (UAD) framework, machine learning algorithms are used to identify rare events from unlabeled data, facilitating the detection of various anomalies,^2–4 including those associated with different dementia subtypes such as Alzheimer’s disease.^5–7

Reconstruction-based anomaly detection methods rely on generating a subject-specific image of healthy appearance and then comparing the subject’s real image with this synthetic image, resulting in a personalized map of anomalies. First, generative models such as variational autoencoders (VAEs), generative adversarial networks (GANs), or diffusion models are trained exclusively on healthy images to learn the healthy distribution. Second, at inference time, the model is fed a patient’s image (with potential anomalies) and reconstructs the image based on its learned healthy distribution.² Ideally, the model should reconstruct healthy tissue only, and we hypothesize that the reconstructed image is a subject-specific pseudo-healthy representation of the input image.^2,8,9 Finally, anomaly maps can be defined as the reconstruction error, or residual, that is, the voxel-wise difference between the patient’s input image and the model’s output image, such that high values indicate anomalies.^2,8,9 However, this residual-based approach has several shortcomings: residuals depend on the underlying raw intensities, there is no natural thresholding mechanism, and it is hard to distinguish between reconstruction errors due to genuine pathological anomalies and reconstruction errors due to the model’s imperfect reconstructions, leading to false positive detections even in healthy tissues.^10–12

Several approaches have been proposed to address these issues: postprocessing techniques,^8,13,14 leveraging multiple reconstructions,^15–17 alternative metrics,¹⁸ and uncertainty estimation techniques.^11,12,19 There is no standard postprocessing procedure, neither of the pseudo-healthy reconstruction, nor of the anomaly map. Common steps include applying a body contour mask, morphological operations, median kernel filtering, normalization, thresholding of the anomaly map, connected component analysis, and adaptive histograms.^8,13,14 To account for the model’s reconstruction errors, several studies have used multiple reconstructions rather than a single reconstruction, which can easily be done in VAEs and diffusion models. Some approaches compare the average reconstruction with the input image.^8,15 Patel et al.¹⁶ fit a kernel density estimation independently for each voxel position to obtain a probability density function for intensities. Behrendt et al.¹⁷ considered the covariance of pixels between reconstructions and voxels of the image. Huijben et al.¹⁸ explored alternative metrics to the absolute reconstruction error, including metrics based on components of the structural similarity (SSIM),²⁰ perceptual metrics, and metric ensembles. Other works are based on uncertainty estimation, in which the anomaly maps are normalized by uncertainty or the estimated variance of individual pixels.^11,12,19

In this work, we unveil various sources of variability in a reconstruction-based anomaly detection pipeline and study their impact on reconstruction errors and anomaly detection performance. We focus on variability arising from the random seed used during training (which affects weight initialization, data augmentation, and data ordering) and explore strategies including model ensembling and anomaly map normalization to improve the reliability of anomaly detection pipelines and their evaluation. We compare these approaches in the context of dementia-related anomalies on 3D FDG PET using a variational autoencoder. Our experimental comparative analysis emphasizes that it is crucial to design robust anomaly maps and to mitigate the impact of randomness-induced variability to support more stable and trustworthy deployment in real-world clinical practice.

2. Methods

2.1. Generative Model

To generate subject-specific pseudo-healthy images, we rely on VAEs,²¹ specifically a $\beta$-VAE.²² The $\beta$-VAE model is built around two components: an encoder and a decoder. The encoder $q_{\varphi }(z|x)$ maps each observation $x\in \mathcal{X}$ into a compact latent representation $z\in {\mathbb{R}}^d$. The decoder $p_{\theta }(x|z)$ then attempts to reconstruct the observation from this latent code. A common formulation assumes that the encoder outputs a Gaussian distribution, $q_{\varphi }(z|x)=\mathcal{N}(\mu (x),\sigma (x))$, whereas the decoder models the likelihood as $p_{\theta }(x|z)=\mathcal{N}(\mathcal{D}(z),1)$, where $\mu$, $\sigma$, and $\mathcal{D}$ are implemented as neural networks. Training consists of minimizing a loss composed of a reconstruction term and a regularization term:

$$\mathcal{L}=\sum _{x\in \mathcal{X}}{\mathbb{E}}_{z\sim q_{\varphi }(z|x)}[\log p_{\theta }(x|z)]-\beta \mathrm{KL}(q_{\varphi }(z|x)||p(z)).$$ (1)

The first part encourages the decoder to accurately reconstruct $x$ from $z$, whereas the second part penalizes deviations of the encoder’s output distribution from the prior $p(z)=\mathcal{N}(0,I)$ using the Kullback–Leibler (KL) divergence. The coefficient $\beta$ adjusts how strongly this KL term influences the learned latent space. The training and inference process are depicted in Fig. 1. Note that in most cases, and unless stated otherwise, during training, we sample the latent code, $z\sim \mathcal{N}(\mu (x),\sigma (x))$ using the reparameterization trick, whereas at inference we set $z=\mu (x)$ as shown in Fig. 1.

Fig. 1.

Typical procedure for VAE-based pseudo-healthy reconstruction and residual-based anomaly map. For training (top), the model is shown exclusively healthy data, and the latent code is sampled using the reparameterization trick. For inference (bottom), the model is fed an image with potential anomalies ($\mathbf{x}$) and reconstructs the image based on its learned distribution, using $z=\mu (x)$, and obtaining a pseudo-healthy reconstruction ($\hat{\mathbf{x}}$). An anomaly map can be obtained by comparing the input image and its pseudo-healthy reconstruction, typically using the residual, $\mathbf{r}(\mathbf{x})=\mathbf{x}-\hat{\mathbf{x}}$.

The model is trained exclusively on healthy data to learn the healthy distribution. At inference time, the model is fed an image with potential anomalies, denoted by $\mathbf{x}$, and reconstructs the image based on its learned healthy distribution. We call this image the pseudo-healthy reconstruction, and denote it by $\hat{\mathbf{x}}$. By comparing the input image $\mathbf{x}$ and its pseudo-healthy reconstruction $\hat{\mathbf{x}}$, we can identify anomalies, as shown in Fig. 1.

We choose to use VAE models because they are computationally lightweight compared with diffusion models, making it feasible to run extensive experiments on 3D images. Moreover, their inherently imperfect reconstructions make them well-suited for studying the impact of various sources of variability on the reconstruction error. We also implement two baseline models for comparison, the f-AnoGAN²³ and a latent diffusion model (LDM),²⁴ which we present in further detail in Sec. 3.1.

2.1.1. Obtaining multiple pseudo-healthy reconstructions

In this paper, we study various sources of variability in a pseudo-healthy reconstruction pipeline. We are therefore interested in obtaining multiple reconstructions ${\hat{\mathbf{x}}}_i$ per input image $\mathbf{x}$ rather than a single reconstruction $\hat{\mathbf{x}}$. We do so in two different ways: (i) by training multiple models with varying random seeds, i.e., a deep ensemble, and (ii) by sampling multiple times in the model’s latent space, i.e., Monte Carlo sampling.

• i.Deep ensemble: training models with varying random seeds

  We train $N_s=11$ models with random seeds ranging from 0 to 10. All models are trained with an identical training procedure on the same dataset. The only sources of variation are in the random initialization of model weights, the stochastic nature of data augmentation (with fixed parameters but random sampling), and the order in which data is shown during training. This setup allows studying the variability of a $\beta$-VAE with respect to its random seed, resulting in 11 distinct models and therefore 11 different reconstructions for each input image. We denote by ${\mathit{\mu }}_s(\hat{\mathbf{x}})$ and ${\mathit{\sigma }}_s(\hat{\mathbf{x}})$ the mean and standard deviation of the $N_s$ reconstructions $\{{\hat{\mathbf{x}}}_i{\}}_{1\le i\le N_s}$ obtained for an input image $\mathbf{x}$. This process is represented in Fig. 2.

• ii.Monte Carlo sampling in the latent space

  It is also possible to obtain multiple reconstructions from a single trained model, rather than training multiple models to obtain multiple reconstructions. Recall that, in a VAE, an image $\mathbf{x}$ is encoded to obtain the mean $\mu (\mathbf{x})$ and standard deviation $\sigma (\mathbf{x})$ of the latent distribution. At inference time, rather than setting $\mathbf{z}=\mu (\mathbf{x})$, we can sample $N_l=10$ latent codes ${\mathbf{z}}_i\sim \mathcal{N}(\mu (\mathbf{x}),\sigma (\mathbf{x}))$ and decode each to obtain the corresponding reconstructions ${\hat{\mathbf{x}}}_i$. With this approach, we can obtain multiple reconstructions “for free” by introducing stochasticity only at the latent sampling stage, a natural property of VAEs. We denote by ${\mathit{\mu }}_l(\mathbf{x})$ and ${\mathit{\sigma }}_l(\mathbf{x})$ the mean and standard deviation of the $N_l$ reconstructions $\{{\hat{\mathbf{x}}}_i{\}}_{1\le i\le N_l}$ obtained for an input $\mathbf{x}$. This process is represented in Fig. 3.

Fig. 2.

Deep ensembling with $N_s$ models trained with varying random seeds to obtain multiple pseudo-healthy reconstructions. We denote by ${\mathrm{VAE}}_i$ the model trained with random seed $i$ and ${\hat{\mathbf{x}}}_i$ the pseudo-healthy reconstruction for an input image $\mathbf{x}$. The mean ${\mathit{\mu }}_s(\hat{\mathbf{x}})$ and standard deviation ${\mathit{\sigma }}_s(\hat{\mathbf{x}})$ of all reconstructions $\{{\hat{\mathbf{x}}}_i{\}}_{1\le i\le N_s}$ are computed to obtain two different types of anomaly maps: ${\mathbf{r}}_{\mathrm{seeds}}(\mathbf{x})=\mathbf{x}-{\mathit{\mu }}_s(\hat{\mathbf{x}})$ and ${\mathbf{z}}_{\mathrm{seeds}}(\mathbf{x})=\frac{\mathbf{x}-{\mathit{\mu }}_s(\hat{\mathbf{x}})}{{\mathit{\sigma }}_s(\hat{\mathbf{x}})}$.

Fig. 3.

Monte Carlo sampling in the latent space at inference time to obtain $N_l$ pseudo-healthy reconstructions using a single trained VAE model. At inference time, for an input image $\mathbf{x}$, we encode it to obtain $\mu (\mathbf{x})$ and $\sigma (\mathbf{x})$, and sample a latent code ${\mathbf{z}}_i\sim \mathcal{N}(\mu (\mathbf{x}),\sigma (\mathbf{x}))$ using the reparameterization trick, which we decode to obtain ${\hat{\mathbf{x}}}_i$, the pseudo-healthy reconstruction. By repeating the process $N_l$ times, we obtain $N_l$ reconstructions $\{{\hat{\mathbf{x}}}_i{\}}_{1\le i\le N_l}$, from which we compute the mean ${\mathit{\mu }}_l(\hat{\mathbf{x}})$ and standard deviation ${\mathit{\sigma }}_l(\hat{\mathbf{x}})$, to obtain two different types of anomaly maps: ${\mathbf{r}}_{\mathrm{latent}}(\mathbf{x})=\mathbf{x}-{\mathit{\mu }}_l(\hat{\mathbf{x}})$ and ${\mathbf{z}}_{\mathrm{latent}}(\mathbf{x})=\frac{\mathbf{x}-{\mathit{\mu }}_l(\hat{\mathbf{x}})}{{\mathit{\sigma }}_l(\hat{\mathbf{x}})}$.

2.2. Dataset

Our dataset consists of 3D FDG PET scans from the ADNI database.^25–27 For our main dataset, used for training, validation, and testing for our main experiments, we select images that are co-registered, averaged, and uniformized to the same resolution (8 mm FWHM), to minimize heterogeneity due to the use of different scanners. For a sensitivity analysis which we present in Sec. 3.4.1, we use additional validation and testing datasets. For these, we include images that are coregistered, and averaged, but not uniformized to the same resolution (with effective resolution ranging from 4.5 to 7.5 mm FWMH, see Ref. 28 for more detail). We process all data using the pet-linear pipeline from Clinica,²⁹ which affinely registers to the MNI space and normalizes in intensity using the average PET uptake in the cerebellum and pons regions and crops each image. We select images from stable healthy participants that are constantly labeled as cognitively normal (CN) in ADNI for a 3-year interval. We use 545 images from 302 healthy subjects for training, 25 images from 25 healthy subjects for validation. For testing, we use 50 images from 50 healthy subjects and also include 50 images from 50 Alzheimer’s disease (AD) patients. These data splits were performed at the subject’s level and stratified by sex and age to avoid data leakage and reduce biases. The validation and testing datasets will be further described in Sec. 2.4.3.

2.3. Evaluation Procedure

In this section, we detail our evaluation procedure. The different steps presented here are summarized in the diagram in Fig. 4.

Fig. 4.

Diagram of our evaluation procedure and corresponding metrics. Plain arrows correspond to our typical pseudo-healthy reconstruction pipeline, dashed arrows to our anomaly simulation procedure, and colored lines to the metric computation. It is represented in the simple case of residual-based anomaly maps and single reconstruction, but the framework is general and can be adapted to all situations described in the paper.

2.3.1. Healthy-to-healthy reconstruction

We first evaluate our model’s ability to reconstruct healthy images. We use the SSIM²⁰ and mean squared error (MSE) metrics. We denote by hhSSIM (resp. hhMSE), the SSIM (resp. MSE) between a healthy input $\mathbf{x}$ and its reconstruction $\hat{\mathbf{x}}$. We compute these within the brain only, using a brain mask based on the MNI template.

2.3.2. Abnormal-to-healthy reconstruction

We then wish to evaluate our model’s ability to reconstruct abnormal images as healthy and anomaly detection performance. Because ground truth pairs of healthy and abnormal images are unavailable, we rely on the simulation framework introduced by Hassanaly et al.⁷ to simulate realistic hypometabolism characteristic of various dementia subtypes with different degrees of severity. In practice, we select a realistic pathology mask $M$ and reduce the image intensity within this mask by a given severity percentage $p$. Formally, let us denote by ${\mathbf{x}}^{\prime }$ the simulated abnormal image obtained from a healthy image $\mathbf{x}$, we set ${\mathbf{x}}^{\prime }=\mathbf{x}\odot \overline{M}+(1-p)·\mathbf{x}\odot M,$ where $\odot$ is the voxel-by-voxel multiplication, and $\overline{M}$ is the complement of the pathology mask $M$, i.e., the region of the brain outside the pathology mask. Note that to obtain a realistic simulated image, we smoothed the mask with a Gaussian convolution filter with $\sigma =5$.

This framework is used to quantitatively evaluate our models. It allows evaluating the abnormal-to-healthy reconstruction quality by comparing the pseudo-healthy image generated from simulated abnormal input $\widehat{{\mathbf{x}}^{\prime }}$ and the ground-truth healthy image $\mathbf{x}$. Similarly as above, we use the SSIM and the MSE metrics, which we now denote by ahSSIM and ahMSE.

2.3.3. Voxel-level anomaly map

Once we have, for each input $\mathbf{x}$, its single pseudo-healthy reconstruction $\hat{\mathbf{x}}$, or its multiple pseudo-healthy reconstructions $\{{\hat{\mathbf{x}}}_i{\}}_i$, we design our anomaly maps in different manners.

Our baseline is the residual-based anomaly map $\mathbf{r}=\mathbf{x}-\hat{\mathbf{x}}$. Given the limitations of residual-based anomaly maps,¹⁰ we include in our experiments SSIM-based anomaly maps, where we compute the SSIM, with window size 11, between the input and the pseudo-healthy reconstruction(s).^17,18

Moreover, to account for the model’s reconstruction errors, we design anomaly maps that are obtained from multiple reconstructions rather than a single reconstruction. These are designed in different ways: the difference (or SSIM) between the input image and the mean of the reconstructions and a $Z$-score with the multiple reconstructions. Recall that we denote by ${\mathit{\mu }}_s(\hat{\mathbf{x}})$ and ${\mathit{\sigma }}_s(\hat{\mathbf{x}})$ the mean and standard deviation of the reconstructions $\{{\hat{\mathbf{x}}}_i{\}}_{1\le i\le N_s}$ obtained with the $N_s$ models with different random seeds. Similarly, we denote by ${\mathit{\mu }}_l(\hat{\mathbf{x}})$ and ${\mathit{\sigma }}_l(\hat{\mathbf{x}})$ the mean and standard deviation of the reconstructions $\{{\hat{\mathbf{x}}}_i{\}}_{1\le i\le N_l}$ obtained by sampling $N_l$ times in the latent space.

In total, there are eight different anomaly map designs, globally denoted by $\mathbf{a}(\mathbf{x})$ and represented in Fig. 5:

Fig. 5.

Summary table of all possible anomaly maps, obtained from a single trained VAE model, from the deep ensemble with varying random seeds, or from Monte Carlo sampling in the latent space, and compared and aggregated in different manners. The top row corresponds to raw anomaly maps, and the bottom row to anomaly maps normalized using the healthy control validation set. We represent a central axial slice of the anomaly maps obtained for an input image with simulated AD at severity 30%.

• i.$\mathbf{r}(\mathbf{x})=\mathbf{x}-\hat{\mathbf{x}}$,
• ii.${\mathbf{r}}_{\mathrm{SSIM}}(\mathbf{x})=1-\mathrm{SSIM}(\mathbf{x},\hat{\mathbf{x}})$,
• iii.${\mathbf{r}}_{\mathrm{latent}}(\mathbf{x})=\mathbf{x}-{\mathit{\mu }}_l(\hat{\mathbf{x}})$,
• iv. ${\mathbf{z}}_{\mathrm{latent}}(\mathbf{x})=\frac{\mathbf{x}-{\mathit{\mu }}_l(\hat{\mathbf{x}})}{{\mathit{\sigma }}_l(\hat{\mathbf{x}})}$
• v. ${\mathbf{r}}_{{\mathrm{latent}}_{\mathrm{SSIM}}}(\mathbf{x})=1-\mathrm{SSIM}(\mathbf{x},{\mathit{\mu }}_l(\hat{\mathbf{x}}))$
• vi. ${\mathbf{r}}_{\mathrm{seeds}}(\mathbf{x})=\mathbf{x}-{\mathit{\mu }}_s(\hat{\mathbf{x}})$
• vii. ${\mathbf{z}}_{\mathrm{seeds}}(\mathbf{x})=\frac{\mathbf{x}-{\mathit{\mu }}_s(\hat{\mathbf{x}})}{{\mathit{\sigma }}_s(\hat{\mathbf{x}})}$
• viii. ${\mathbf{r}}_{{\mathrm{seeds}}_{\mathrm{SSIM}}}(\mathbf{x})=1-\mathrm{SSIM}(\mathbf{x},{\mathit{\mu }}_s(\hat{\mathbf{x}}))$

In addition, we eventually normalize each anomaly map $\mathbf{a}(\mathbf{x})$ using a healthy validation set. We set

$$\tilde{\mathbf{a}}(\mathbf{x})=\frac{\mathbf{a}(\mathbf{x})-{\mathit{\mu }}_{\mathrm{HC}}(\mathbf{a})}{{\mathit{\sigma }}_{\mathrm{HC}}(\mathbf{a})},$$

where ${\mathit{\mu }}_{\mathrm{HC}}(\mathbf{a})$ and ${\mathit{\sigma }}_{\mathrm{HC}}(\mathbf{a})$ are the mean and standard deviation of the anomaly map $\mathbf{a}$ across all images in a healthy control validation set (HC). We represent this normalization step in the diagram in Fig. 6 in the simple case of the residual anomaly map, and all anomaly maps derived from this process at the bottom row of Fig. 5. Note that the choice of healthy validation set is explored in a sensitivity analysis in Sec. 3.4.1.

Fig. 6.

Diagram of the normalization of anomaly maps using the anomaly maps obtained from a healthy control validation set. The diagram represents the simple case in which the anomaly map is a simple residual between input and output, but the process can be extended for all anomaly map types. We set $\tilde{\mathbf{r}}(\mathbf{x})=\frac{\mathbf{r}(\mathbf{x})-{\mathit{\mu }}_{\mathrm{HC}}(\mathbf{r})}{{\mathit{\sigma }}_{\mathrm{HC}}(\mathbf{r})}$, where ${\mathit{\mu }}_{\mathrm{HC}}(\mathbf{r})$ and ${\mathit{\sigma }}_{\mathrm{HC}}(\mathbf{r})$ are the mean and standard deviation of $\mathbf{r}$ across all images in a healthy validation set (HC).

We evaluate the voxel-level anomaly detection performance by comparing the unthresholded anomaly map with the simulation mask. For this, we compute the voxel-wise average precision, denoted by ${\mathrm{AP}}_{\mathrm{vox}}$ , and an estimate of the best possible Dice index, denoted by $\lceil \mathrm{Dice}\rceil$.

2.3.4. Sample-level anomaly score

We design sample-level anomaly scores as the mean of voxels of the absolute value of our anomaly map $|\mathbf{a}|$. We evaluate these using sample-wise area under the receiver operating characteristic curve (AUC) and average precision (${\mathrm{AP}}_{\mathrm{s}}$) (healthy versus abnormal, for each evaluation group separately). We binarize the scores using the 95^th percentile of the scores on the healthy validation set and compute the precision and recall for each class separately.

2.4. Implementation Detail

2.4.1. VAE architecture and training

In terms of architecture, we choose the 3D $\beta$-VAE proposed in Hassanaly et al.³⁰ because it showed good performance on a benchmark³⁰ comparing VAE variants focused on detecting dementia-related lesions in FDG PET images.

Its encoder is composed of five convolutional layers and a dense layer, and its decoder is symmetrical, by adding an upsampling layer before each convolution. After each convolutional layer is a batch normalization and a swish activation function, except for the last layer, in which there is no batch normalization, and the activation function is a sigmoid.

We perform data augmentation during training: random noise (with standard deviation 2) and random blur (with standard deviation 0.02) from TorchIO.³¹ We found that such change slightly increases the model performance in terms of healthy-to-healthy reconstruction, with the hhSSIM increasing from $0.860\pm 0.025$ to $0.874\pm 0.022$, and anomaly detection on simulated images, with the ${\mathrm{AP}}_{\mathrm{vox}}$ on AD-30 increasing from $0.407\pm 0.084$ to $0.469\pm 0.080$.

We keep all other training parameters the same: the latent space has size 256, we set $\beta =10$, we train the model for 200 epochs, with learning rate ${10}^{-4}$ and batch size 8 using ClinicaDL.³² Note that our images undergo min–max normalization from TorchIO³¹ before being passed through the neural network layers, so that the each voxel’s intensity is between 0 and 1.

2.4.2. Baseline implementation

To compare with the VAE, we implement two baseline approaches: the f-AnoGAN proposed by Schlegl et al.²³ and an LDM proposed by Graham et al.²⁴

For the training of the f-AnoGAN, we first train a Wasserstein GAN with gradient penalty to produce images from a latent variable $z\sim \mathcal{N}(0,I)$, and then, we train an encoder to map healthy images to latent embeddings. We train the GAN for 400 epochs, and the encoder for 120 epochs.

For the LDM, we attempt to remain as close as possible to the formulation used in Graham et al.²⁴ The training has two phases: first a VQ-VAE³³ is trained to compress the 3D image, and then, the diffusion model is trained on these latent embeddings to learn the healthy distribution. At inference, the input image is encoded, its embedding is noised to a chosen noise level $t\in [0,1]$, passed through the diffusion model, and then decoded to obtain a pseudo-healthy reconstruction. Our VQ-VAE has three convolutional blocks, to obtain a spatial latent representation of size (21,26,22). We use a codebook size of 512 following Hassanaly et al.³⁰ and 256 for the embedding dimension. We select a specific noise level $t=0.7$ so that we can evaluate the pseudo-healthy reconstruction quality, whereas Graham et al.²⁴ aggregate multiple anomaly maps with different noise levels and evaluate solely the detection capabilities.

2.4.3. Summary of validation and testing datasets

Using the simulation framework described in Sec. 2.3.2,⁷ and the healthy test set, we obtain five evaluation groups which differ in lesion size, location, and severity: three groups of Alzheimer’s disease (AD) with increasing degrees of severity (AD-15, AD-30, AD-50) and two groups of medium severity for two other dementia subtypes, namely, the behavioral variant of fronto-temporal dementia (bvFTD-30), and the semantic-variant of primary progressive aphasia (svPPA-30), all shown in Fig. 7.

Fig. 7.

Summary table of the anomalies simulated (AD, bvFTD, and svPPA) with varying degrees of severity for the same subject, with the corresponding healthy image (right). Top: binary pathology mask used for the hypometabolism simulation. Bottom: simulated abnormal image, with the pathological region circled in red. The images are sagittal slices (at index 60) of the same subject, overlayed with the MNI template.

We also perform a sensitivity analysis in Sec. 3.4.1 to study the impact of the healthy-control normalization group on the anomaly map normalization , i.e., going from $\mathbf{a}(\mathbf{x})$ to $\tilde{\mathbf{a}}(\mathbf{x})=\frac{\mathbf{a}(\mathbf{x})-{\mathit{\mu }}_{\mathrm{HC}}(\mathbf{a})}{{\mathit{\sigma }}_{\mathrm{HC}}(\mathbf{a})}$. For these experiments, we use two additional validation groups and five additional testing groups. The first additional validation group consists of images with simulated low-severity AD, denoted by val-AD-15. The second additional validation group consists of the healthy validation images with mixed resolutions, rather than uniformized resolutions, denoted by val-CN-mres. Finally, the five additional testing groups consist in the same test images as described above, but in their mixed resolution version.

3. Experiments and Results

3.1. Baseline Approaches

We first compare our VAE’s performance with two baseline models, the f-AnoGAN proposed by Schlegl et al.²³ and an LDM proposed by Graham et al.²⁴ Both models are trained with random seed 0, and we therefore compare their performance with the $\beta$-VAE trained with random seed 0 as well.

We report the abnormal-to-healthy reconstruction quality and the anomaly detection performance on AD-30 images in Fig. 8. We observe that all models have similar performance, showing that the $\beta$-VAE, despite its blurry reconstructions, generate good pseudo-healthy reconstructions enabling good anomaly detection performance. We note that the ahMSE is lower for the $\beta$-VAE than for the other models, whereas the ahSSIM is higher for the baseline models.

Fig. 8.

Reconstruction and anomaly detection performance on AD-30 images for baseline models: $\beta$-VAE with seed 0 ($\beta {\text{-}\mathrm{VAE}}_0$), f-AnoGAN, and latent diffusion model (LDM). The abnormal-to-healthy reconstruction performance (in blue) is assessed using the ahMSE and the ahSSIM. The anomaly detection performance is assessed using the ${\mathrm{AP}}_{\mathrm{vox}}$ and the $\lceil \mathrm{Dice}\rceil$ for raw residual map $\mathbf{r}$ (in orange) and for normalized residual map $\tilde{\mathbf{r}}$ (in green).

In the following sections, we will focus on the $\beta$-VAE.

3.2. Variance of Models Trained with Different Random Seeds

We observe a large variability in the performance of VAE models trained with different random seeds, with considerably greater variability in anomaly detection metrics, ${\mathrm{AP}}_{\mathrm{vox}}$ and $\lceil \mathrm{Dice}\rceil$, than in abnormal-to-healthy reconstruction metrics, ahSSIM and ahMSE. The results for the evaluation group AD-30, which is representative of the other groups, for the residual-based anomaly map $\mathbf{r}$ are shown in Fig. 9. Similar patterns are observed for all groups and anomaly map designs and are displayed in Figs. 15 and 16 in the Appendix B.

Fig. 9.

Reconstruction and anomaly detection performance on AD-30 images for residual map $\mathbf{r}$.

Fig. 15.

Reconstruction metrics per model trained with different random seed for all groups.

Fig. 16.

Average precision per model trained with different random seeds for all groups and all anomaly maps.

Note that this variability in performance is not due to the VAEs’ convergence, as all models had converged properly as shown by the learning curves displayed in Fig. 14 in the Appendix A.

Fig. 14.

Learning curves for all VAE models trained.

In the context of deep learning workflows for unsupervised anomaly detection, the impact of random seeds is often overlooked, with relatively few papers using deep ensembles, implying that the same model architecture can yield substantially different results depending on its initialization, potentially leading to substantial over- or under-performance, and biased evaluation.

3.3. Strategy 1: Z-Score to Aggregate Outputs from Models Trained with Different Random Seeds

In Fig. 10, we plot the ${\mathrm{AP}}_{\mathrm{vox}}$ scores obtained with residual anomaly maps of individual models $\mathbf{r}$ (in blue) and with the proposed ${\mathbf{z}}_{\mathrm{seeds}}$ (in orange). We observe that ${\mathbf{z}}_{\mathrm{seeds}}$ outperforms all individual models for AD-15, AD-30, and svPPA-30 in terms of mean and median. For bvFTD-30, it outperforms all models in terms of median, and all but two in terms of mean. For AD-50, it outperforms all but two in terms of mean, and all but four models in terms of median.

Fig. 10.

${\mathrm{AP}}_{\mathrm{vox}}$ for all groups of residual-based anomaly maps $\mathbf{r}$ of models trained with random seeds 0 to 10 (in the $x$-axis, in blue) compared with ${\mathbf{z}}_{\mathrm{seeds}}$ anomaly maps (in orange) built from the predictions of all models.

Similar results were observed with other anomaly map designs than residuals (see Fig. 17 in the Appendix B).

Fig. 17.

AP for all groups of all anomaly maps of models trained with random seeds 0 to 10 (in the $x$-axis, in blue) compared with ${\mathbf{z}}_{\mathrm{seeds}}$ anomaly maps (in orange) built from the predictions of all models.

3.4. Strategy 2: Normalization of Anomaly Maps Using the Healthy Validation Set

Because training a deep ensemble is computationally expensive, we explore alternative strategies to reduce variance and mitigate the impact of random seed initialization.

We turn to normalizing the anomaly map using a healthy validation set, which not only allows for reducing the effect of the random seed on the performance variance, but should also account for a portion of the reconstruction error due to model imperfections and should capture the variability present in healthy data.

In Fig. 11, we report the ${\mathrm{AP}}_{\mathrm{vox}}$ for $\mathbf{r}$ and $\tilde{\mathbf{r}}$ across different random seeds for all evaluation groups. We observe a clear reduction in performance variability: the worst-performing models show substantial improvement, whereas the best-performing models maintain similar performance. We even observe a general increase in ${\mathrm{AP}}_{\mathrm{vox}}$ (from 0.1 to 0.3) for the svPPA-30 group.

Fig. 11.

${\mathrm{AP}}_{\mathrm{vox}}$ without (left) and with (right) healthy-control validation-based normalization.

We observe from Fig. 8, that normalizing the anomaly maps using a healthy control validation set also improves the anomaly detection performance for the two baseline models on AD-30, the f-AnoGAN²³ and the LDM.²⁴ This suggests that our proposed strategy may generalize beyond VAEs.

Results for other anomaly maps are presented in Fig. 18 in the Appendix B.

Fig. 18.

AP without (left) and with (right) healthy-control validation-based normalization for all anomaly maps.

3.4.1. Sensitivity analysis: impact of the validation set quality

We then conduct a sensitivity analysis to assess the robustness of our normalization procedure. Our investigation focuses on two specific scenarios: (i) the case where the validation set is contaminated and contains anomalies and (ii) the case where a distribution shift occurs between the training and test sets.

To examine the first scenario, we use the val-AD-15 validation set, in which we introduced low-severity AD anomalies into all images. As shown in the top two rows of Table 1(a), the performance degrades when normalization is performed using val-AD-15 instead of val-CN. The drop is especially large for the svPPA-30 group, which is consistent with the fact that detecting small-scale anomalies is even more challenging. This experiment confirms that the normalization step requires a validation set that is as clean and anomaly-free as possible.

Table 1.

Mean (± std) ${\mathrm{AP}}_{\mathrm{vox}}$ across the 11 models trained with different random seeds for residual maps normalized using different healthy control validation groups: the usual healthy-control validation group (val-CN), images with simulated low-severity AD (val-AD-15), and healthy-control validation images with mixed (rather than uniform) resolutions (val-CN-mres). The highest values (in terms of mean) are highlighted in bold.

(a) Evaluation on our usual testing sets with images with uniform resolutions
	AD-15	AD-30	AD-50	svPPA-30	bvFTD-30
val-CN	$\mathbf{0.268}\pm \mathbf{0.034}$	$\mathbf{0.526}\pm \mathbf{0.061}$	$\mathbf{0.826}\pm \mathbf{0.029}$	$\mathbf{0.319}\pm \mathbf{0.105}$	$\mathbf{0.439}\pm \mathbf{0.092}$
val-AD-15	$0.209\pm 0.013$	$0.340\pm 0.052$	$0.786\pm 0.038$	$0.098\pm 0.051$	$0.381\pm 0.080$
val-CN-mres	$0.206\pm 0.022$	$0.377\pm 0.062$	$0.754\pm 0.044$	$0.216\pm 0.087$	$0.279\pm 0.078$
(b) Evaluation on the testing sets with images with mixed resolutions
	AD-15-mres	AD-30-mres	AD-50-mres	svPPA-30-mres	bvFTD-30-mres
val-CN	$\mathbf{0.256}\pm \mathbf{0.041}$	$\mathbf{0.449}\pm \mathbf{0.089}$	$\mathbf{0.748}\pm \mathbf{0.072}$	$\mathbf{0.153}\pm \mathbf{0.086}$	$\mathbf{0.334}\pm \mathbf{0.108}$
val-AD-15	$0.208\pm 0.015$	$0.304\pm 0.053$	$0.68\pm 0.082$	$0.055\pm 0.03$	$0.316\pm 0.098$
val-CN-mres	$0.226\pm 0.037$	$0.381\pm 0.085$	$0.718\pm 0.075$	$0.131\pm 0.075$	$0.263\pm 0.087$

For the second scenario, we introduce a distribution shift using data with nonuniform resolution, which we refer to as mixed resolution (mres). Our goal is to determine whether normalization should rely on data closer to the training distribution or closer to the inference distribution. In all experiments, the models are trained on images with uniform resolution, and for normalization, we use either uniform-resolution data (val-CN) or mixed-resolution data (val-CN-mres). In Table 1(a), the test set also has uniform resolution, whereas in Table 1(b), the test set uses mixed resolutions. Across both settings, we observe that normalizing with images with uniform resolution (val-CN) consistently outperforms normalization with mixed resolutions (val-CN-mres). This suggests that normalization should be performed using data that match the training distribution rather than the inference distribution.

3.5. Strategy 3: Anomaly Map Designs

As an alternative approach to reduce variability and mitigate reconstruction errors, we explore other anomaly map designs and report these in Fig. 12 for the normalized anomaly maps, and in Fig. 19 for the raw anomaly maps.

Fig. 12.

Mean ${\mathrm{AP}}_{\mathrm{vox}}$ across models for each normalized anomaly map and each evaluation group.

Fig. 19.

Mean AP across models for each anomaly map type (without healthy-control validation-based normalization) and each evaluation group.

First, we test replacing the standard voxel-wise intensity difference with the SSIM between the input and the pseudo-healthy reconstruction, with ${\mathbf{r}}_{\mathrm{SSIM}}$, ${\mathbf{r}}_{{\mathrm{latent}}_{\mathrm{SSIM}}}$, and ${\mathbf{r}}_{{\mathrm{seeds}}_{\mathrm{SSIM}}}$. In our experiments, this approach does not lead to performance improvements, on the contrary, SSIM-based maps consistently yield worse anomaly detection performance compared with standard residual maps. We also evaluate anomaly maps based on multiple reconstructions: ${\mathbf{r}}_{\mathrm{latent}}$, ${\mathbf{z}}_{\mathrm{latent}}$, ${\mathbf{r}}_{\mathrm{seeds}}$, and ${\mathbf{z}}_{\mathrm{seeds}}$. We find that the usual residual $\mathbf{r}$ and ${\mathbf{r}}_{\mathrm{latent}}$ produced comparable results, whereas ${\mathbf{z}}_{\mathrm{latent}}$ perform poorly, likely due to unstable and low standard deviation values. As previously shown, ${\mathbf{z}}_{\mathrm{seeds}}$ and ${\mathbf{r}}_{\mathrm{seeds}}$, even though computationally expensive, remain very strong-performing.

Furthermore, we quantify the reduction in performance variability and confirm that all types of anomaly maps benefit from this validation-based normalization. In terms of mean ${\mathrm{AP}}_{\mathrm{vox}}$ between subjects, this normalization reduces variability by at least 20%, with an average reduction of more than 50% across all evaluation groups and anomaly map designs, and up to a 10-fold decrease in the best cases. We note that there are some outlier cases for the bvFTD-30 and svPPA-30 evaluation groups in which the normalization increases variability.

3.6. Validation on Real Alzheimer’s Data

Finally, we validate our approach on both simulated dementia and real Alzheimer’s data from the ADNI database, both qualitatively by visually inspecting the anomaly maps and quantitatively using anomaly scores and comparing to other baseline approaches.

In Fig. 13, we display examples of anomaly maps for two AD patients, in Table 2, we show classification performance on anomaly scores from real AD patients for $\mathbf{r}$, $\tilde{\mathbf{r}}$, and ${\mathbf{z}}_{\mathrm{seeds}}$, and in Table 3, we show classification performance for the normalized residual $\tilde{\mathbf{r}}$ across all real and simulated testing groups, and in Table 4, we display the same results for the raw residual $\mathbf{r}$.

Fig. 13.

Input image $\mathbf{x}$, pseudo-healthy reconstruction $\hat{\mathbf{x}}$, normalized residual $\tilde{\mathbf{r}}$, and thresholded anomaly map for two real AD patients.

Table 2.

Classification performance for the AD testing group versus healthy, for the different models ($\beta$-VAE with seed 0, f-AnoGAN and LDM), with raw residuals, normalized residuals, and $Z$-score across random seeds (${\mathbf{z}}_{\mathrm{seeds}}$) for the $\beta$-VAE.

		Unthresholded		Thresholded
		AUC	${\mathrm{AP}}_{\mathrm{s}}$	Precision	Recall
$\beta \text{-}{\mathrm{VAE}}_0$	$\tilde{\mathbf{r}}$	0.821	0.859	0.814	0.700
	$\mathbf{r}$	0.820	0.861	0.944	0.340
	${\mathbf{z}}_{\mathrm{seeds}}$	0.744	0.770	0.875	0.280
f-AnoGAN	$\tilde{\mathbf{r}}$	0.762	0.771	0.864	0.380
	$\mathbf{r}$	0.756	0.770	0.833	0.300
LDM	$\tilde{\mathbf{r}}$	0.793	0.812	0.850	0.340
	$\mathbf{r}$	0.801	0.827	0.805	0.660

Table 3.

Classification performance (abnormal versus healthy, for each evaluation group separately), for the different models ($\beta$-VAE with seed 0, f-AnoGAN, and LDM), with normalized residuals $\tilde{\mathbf{r}}$.

		Unthresholded		Thresholded
		AUC	${\mathrm{AP}}_{\mathrm{s}}$	Precision	Recall
test-AD	$\beta \text{-}{\mathrm{VAE}}_0$	0.821	0.859	0.814	0.700
	f-AnoGAN	0.762	0.771	0.864	0.380
	${\mathrm{LDM}}_{0.7}$	0.801	0.827	0.805	0.660
AD-15	$\beta \text{-}{\mathrm{VAE}}_0$	0.806	0.791	0.750	0.480
	f-AnoGAN	0.666	0.617	0.727	0.160
	${\mathrm{LDM}}_{0.7}$	0.764	0.732	0.742	0.460
AD-30	$\beta \text{-}{\mathrm{VAE}}_0$	1.000	1.000	0.862	1.000
	f-AnoGAN	0.948	0.909	0.929	0.780
	${\mathrm{LDM}}_{0.7}$	0.992	0.992	0.862	1.000
AD-50	$\beta \text{-}{\mathrm{VAE}}_0$	1.000	1.000	0.862	1.000
	f-AnoGAN	1.000	1.000	0.943	1.000
	${\mathrm{LDM}}_{0.7}$	1.000	1.000	0.862	1.000
bvFTD-30	$\beta \text{-}{\mathrm{VAE}}_0$	0.955	0.952	0.849	0.900
	f-AnoGAN	0.812	0.755	0.824	0.280
	${\mathrm{LDM}}_{0.7}$	0.911	0.879	0.833	0.800
svPPA-30	$\beta \text{-}{\mathrm{VAE}}_0$	0.685	0.663	0.667	0.320
	f-AnoGAN	0.576	0.565	0.571	0.080
	${\mathrm{LDM}}_{0.7}$	0.637	0.615	0.619	0.260

Note: The highest values per evaluation group are highlighted in bold.

Table 4.

Classification performance (abnormal versus healthy, for each evaluation group separately), for the different models ($\beta$-VAE with seed 0, f-AnoGAN, and LDM), with raw residuals $\mathbf{r}$.

		Unthresholded		Thresholded
		AUC	${\mathrm{AP}}_{\mathrm{s}}$	Precision	Recall
test-AD	$\beta \text{-}{\mathrm{VAE}}_0$	0.820	0.861	0.944	0.340
	f-AnoGAN	0.756	0.770	0.833	0.300
	${\mathrm{LDM}}_{0.7}$	0.793	0.812	0.850	0.340
AD-15	$\beta \text{-}{\mathrm{VAE}}_0$	0.810	0.802	0.917	0.220
	f-AnoGAN	0.693	0.649	0.625	0.100
	${\mathrm{LDM}}_{0.7}$	0.695	0.669	0.571	0.080
AD-30	$\beta \text{-}{\mathrm{VAE}}_0$	1.000	1.000	0.980	1.000
	f-AnoGAN	0.947	0.902	0.923	0.720
	${\mathrm{LDM}}_{0.7}$	0.989	0.989	0.938	0.900
AD-50	$\beta \text{-}{\mathrm{VAE}}_0$	1.000	1.000	0.980	1.000
	f-AnoGAN	1.000	1.000	0.943	1.000
	${\mathrm{LDM}}_{0.7}$	1.000	1.000	0.943	1.000
bvFTD-30	$\beta \text{-}{\mathrm{VAE}}_0$	0.933	0.934	0.962	0.500
	f-AnoGAN	0.849	0.782	0.786	0.220
	${\mathrm{LDM}}_{0.7}$	0.906	0.891	0.885	0.460
svPPA-30	$\beta \text{-}{\mathrm{VAE}}_0$	0.621	0.610	0.667	0.040
	f-AnoGAN	0.564	0.551	0.500	0.060
	${\mathrm{LDM}}_{0.7}$	0.612	0.603	0.571	0.080

Note: The highest values per evaluation group are highlighted in bold.

From Table 2, we find that anomaly scores based on normalized residuals improve classification performance compared with anomaly scores based on raw residuals. We note that this holds across all models, including the f-AnoGAN and the LDM. Moreover, we find that the classification of anomaly scores based on $\tilde{\mathbf{r}}$ is higher than for anomaly scores based on $\mathbf{r}$, which is in turn higher than anomaly scores based on ${\mathbf{z}}_{\mathrm{seeds}}$.

Overall, as shown in Table 3, our anomaly scores demonstrate good classification abilities across all evaluation groups, both with real and synthetic anomalies, with slightly worse scores for lesions of low severity (AD-15) and small size (svPPA-30).

4. Discussion

In this work, we highlight a largely overlooked source of variability in deep learning models for medical image synthesis: their random seed, which influences the random initialization of model weights, data augmentation (fixed parameters but random sampling) and the order in which data are presented to the model. We study the impact of the random seed of variational autoencoders on model performance, in terms of pseudo-healthy reconstruction quality, and in terms of anomaly detection performance. Because this source of variance is clearly undesirable, especially in the context of model evaluation for tasks such as anomaly detection, we present several approaches to mitigate its impact. We propose three different solutions, which we evaluate on 3D FDG PET in the context of dementia-related anomaly detection.

The first strategy proposed is to train a deep ensemble of models with different random seeds and to aggregate their reconstructions using a $Z$-score. This strategy is highly effective, but computationally expensive, as it requires training several replicates of the model. Note that the current study combines the outputs of 11 replicates, and it would be interesting to better understand the impact of the number and quality of the replicates on the performance gains.

As a more efficient alternative, we propose normalizing the anomaly maps using statistics computed from healthy controls, by means of a $Z$-score. We show that this normalization strategy substantially reduces performance variability across models and even increases anomaly detection performance in certain cases.

In our sensitivity analysis, we studied the impact of impure data, and distribution shifts in our normalizing set, and found that the normalizing data should be as clean and anomaly-free as possible, and match the training distribution rather than the inference distribution.

A critical component of the normalization strategy is the preprocessing of our images, and specifically the affine registration of all images to a common template, which enables voxel-wise comparison across subjects. Compared with standard clinical $Z$-score approaches, which rely entirely on population-level normalization and are thus highly sensitive to registration quality, our approach relies on subject-specific pseudo-healthy reconstruction, resulting in greater robustness to template misalignment. Moreover, although affine registration already yields substantial performance gains in our experiments, nonlinear registration could in principle improve spatial correspondence between subjects, leading to more accurate estimates of voxel-wise variability. However, in the context of pathologies involving severe cortical atrophy, nonlinear registration may inadvertently deform anomalous regions to conform to the template, potentially suppressing pathological signal. Investigating this trade-off constitutes a interesting direction for future work.

Moreover, note that if a lesion happens to coincide with a region that is poorly reconstructed in healthy subjects, resulting in high reconstruction variance on the healthy control normalization set, the detection will be more challenging in that specific area. We argue, however, that this behavior is reasonable: if a region is consistently poorly reconstructed in healthy subjects, it is likely that the model’s reconstruction of that same region in pathological subjects will also be unreliable, making any anomaly score derived from it inherently untrustworthy. The normalization thus appropriately discounts these uncertain regions rather than generating potentially spurious anomaly detections.

As a third strategy, we explore several anomaly map designs and find that, in our setting, SSIM-based maps do not outperform residual-based maps and that maps obtained by leveraging multiple reconstructions perform similarly to the usual residual maps.

We believe that the under-performance of SSIM-based maps can be attributed to its structure component, one of the three components of the SSIM alongside luminance and contrast. As visible in Fig. 5 where anomaly maps are qualitatively compared, SSIM-based maps exhibit large values near the cortical contours, precisely where the VAE struggles to preserve fine structural detail due to the blurry nature of its reconstructions. These artifacts at the cortical boundaries are not of pathological origin, yet they dominate the SSIM-based anomaly map and degrade detection performance. This interpretation is consistent with the findings of Huijben et al.,¹⁸ who analyzed the three SSIM components separately in the context of VAE-based anomaly detection on FLAIR images with synthetic lesions and showed that the poor performance of the structure component overshadows the contributions of the luminance and contrast components when the full SSIM score is used. Together, these results suggest that although SSIM is a principled perceptual metric, its structure component is ill-suited to the blurry reconstruction regime typical of VAEs, making simple residual-based maps a more reliable choice in this setting.

A further limitation of this work concerns the metrics used to evaluate anomaly detection performance.³⁴ In particular, our evaluation does not distinguish between hypo- and hyper-metabolism, a distinction that is clinically meaningful in the context of dementia and FDG PET imaging, in which specific patterns of reduced or increased uptake carry distinct diagnostic significance. However, we note that this limitation applies specifically to the quantitative evaluation framework and does not preclude such distinctions from being made in practice. Indeed, one of the key advantages of the proposed approach is the interpretability of the resulting anomaly maps: by visually inspecting the signed residuals, a clinician can differentiate between regions of hypo- and hyper-metabolism, providing richer diagnostic information than what is captured by the aggregate performance metrics reported here.

Finally, the comparison with two baseline models, the f-AnoGAN²³ and the LDM,²⁴ shows that normalizing the anomaly maps seems beneficial to other models than VAEs. Future works could assess whether these findings generalize to other types of models.

Beyond PET-to-PET reconstruction approaches such as the one presented in this work, another well-recognized paradigm for pseudo-healthy PET synthesis relies on anatomically conditioned models, typically UNet-based architectures, that take a structural MRI as input and synthesize a pseudo-healthy PET image.^35–37 These approaches offer a conceptual advantage for anomaly detection: because the input and output modalities differ, there is a natural domain gap that reduces the risk of anomalous PET signal being carried through into the reconstruction, a known challenge in reconstruction-based UAD. Furthermore, skip connections between the MRI encoder and PET decoder can enforce stronger anatomical fidelity, yielding sharper and more spatially precise pseudo-healthy images. However, these approaches require paired MRI-PET data at inference time, which is not always available, whereas PET-to-PET models such as ours can operate from PET alone. More fundamentally, directly observing the PET signal allows the model to capture subject-specific functional and metabolic patterns that anatomy alone may not predict, which is particularly relevant for detecting subtle or early-stage hypometabolism that precedes detectable structural changes. Both paradigms therefore present complementary trade-offs, and future work combining both modalities within a unified framework could be a promising direction.

5. Conclusion

In this work, we study variability in deep generative models used for pseudo-healthy reconstruction by comparing 11 VAEs trained with different random seeds. Our experiments were conducted on 3D brain FDG PET images with real and simulated dementia-related anomalies with different lesion size, location, and severity. We find that while the random seed has little impact on the reconstruction quality variability, it induces large variability in anomaly detection performance. We explore different strategies to mitigate the impact of the random seed on model performance variability. We demonstrate that normalizing the anomaly maps by means of a $Z$-score using the anomaly maps computed on a healthy control validation set substantially reduces performance variability across models and even increases anomaly detection performance in certain cases.

6. Appendix A: Learning Curves

In Fig. 14, we display the learning curves of the 11 VAEs trained with random seeds 0 to 10, showing that all models had converged properly.

7. Appendix B: Additional Results: All Evaluation Groups and Anomaly Maps

7.1. Variance of Models Trained with Different Random Seeds

In this section, we show additional results regarding the variance of models trained with different random seeds. In Fig. 15, we show the variance in terms of abnormal-to-healthy reconstruction metrics for all evaluation groups. In Fig. 16, we show the variance in terms of anomaly detection performance, using the average precision, for all evaluation groups and all anomaly map types.

7.2. Z-Score to Aggregate Outputs from Models Trained with Different Random Seeds

In Fig. 17, we show additional results regarding our Z-score to aggregate outputs from models trained with different random seeds. We display the average precision for all evaluation groups and different anomaly map designs.

7.3. Normalization of Anomaly Maps Using the Healthy Validation Set

In Fig. 18, we show the results of normalizing the anomaly maps using the healthy validation set for all evaluation groups and all anomaly map types.

7.4. Anomaly Map Design

In Fig. 19, we show the mean AP across models for each raw anomaly map type and each evaluation group.

7.5. Validation on Real Data

In Table  4, we display the classification performance for the different models with raw residuals for each evaluation group.

Biographies

Maëlys Solal is a PhD candidate in the ARAMIS Lab, a joint laboratory between Sorbonne University, CNRS, Inria, Inserm, and AP-HP within the Paris Brain Institute. She received her master’s degree in computer science from the University of Oxford in 2022 and her master’s degree in cognitive science from the ENS - PSL in 2023. Her research focuses on deep learning and generative models for the analysis of medical imaging and in particular their applications to brain disorders.

Pascaline André is a PhD candidate in the ARAMIS Lab, a joint laboratory between Sorbonne University, CNRS, Inria, Inserm, and AP-HP within the Paris Brain Institute. She received her master’s degree in statistics from the University of Paris Saclay in 2023. Her research focuses on the statistical evaluation of machine learning models for medical image analysis.

Ninon Burgos is a CNRS researcher in the ARAMIS Lab, a joint laboratory between Sorbonne University, CNRS, Inria, Inserm, and AP-HP the Paris Brain Institute, France. She completed her PhD at University College London, UK, in 2016, and her Habilitation degree from Sorbonne University in 2022. Her research focuses on the development of computational imaging tools to improve the understanding and diagnosis of dementia.

Funding Statement

The research leading to these results has received funding from the French government under management of Agence Nationale de la Recherche as part of the “Investissements d’avenir” program (reference ANR-19-P3IA-0001 - PRAIRIE - and reference ANR-10-IAIHU-06 - IHU ICM) and the “France 2030” program (reference ANR-23-IACL-0008, PRAIRIE-PSAI) and from the European Union’s Horizon Europe Framework Program (grant number 101136607, project CLARA). This research was also funded by the Agence Nationale de la Recherche under the project “ANR-23-CE45-0005-01.” This work was granted access to IDRIS HPC resources under the allocation AD011011648 made by GENCI. Data collection and sharing for the ADNI is funded by the National Institute on Aging (National Institutes of Health Grant U19AG024904). The grantee organization is the Northern California Institute for Research and Education. In the past, ADNI has also received funding from the National Institute of Biomedical Imaging and Bioengineering, the Canadian Institutes of Health Research, and private sector contributions through the Foundation for the National Institutes of Health (FNIH).

Disclosures

The authors declare that there are no financial interests, commercial affiliations, or other potential conflicts of interest that could have influenced the objectivity of this research or the writing of this paper.

Code and Data Availability

The data preprocessing pipelines are available in Clinica (https://github.com/aramis-lab/clinica). The VAE models are implemented in Pythae (https://github.com/clementchadebec/benchmark_VAE). The code for model training and inference is available on ClinicaDL (https://github.com/aramis-lab/clinicaDL).

Data used in preparation of this article were obtained from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database, which can be obtained by requesting access on the LONI website (https://adni.loni.usc.edu/data-samples/access-data).