Survey on sampling conditioned brain images and imaging measures with generative models

Abstract

Generative models have become innovative tools across various domains, including neuroscience, where they enable the synthesis of realistic brain imaging data that captures complex anatomical and functional patterns. These models, such as Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), and diffusion models, leverage deep learning to generate high-quality brain images while maintaining biological and clinical relevance. These models address critical challenges in brain imaging, e.g., the high cost and time required for data acquisition and the frequent imbalance in datasets, particularly for rare diseases or specific population groups. By conditioning the generative process on variables such as age, sex, clinical phenotypes, or genetic factors, these models enhance dataset diversity and provide opportunities to study underrepresented scenarios, model disease progression, and perform controlled experiments that are otherwise infeasible. Additionally, synthetic data generated by these models offer a potential solution to data privacy concerns, as they provide realistic non-identifiable data. As generative models continue to develop, they hold significant potential to substantially advance neuroscience by augmenting datasets, improving diagnostic accuracy, and accelerating the development of personalized treatments. This paper provides a comprehensive overview of the advancements in generative modeling techniques and their applications in brain imaging, with a particular emphasis on conditional generative methods. By categorizing existing approaches, addressing key challenges, and highlighting future directions, this paper aims to advance the integration of conditional generative models into neuroscience research and clinical workflows.

Introduction

Generative models have emerged as a powerful tool across various domains, offering practical synthesized complex and high-dimensional data. In neuroscience, they present a paradigm shift, enabling the generation of realistic brain imaging data that capture intricate anatomical and functional patterns. Compared to traditional statistical approaches, recent generative models leverage deep learning techniques to achieve high fidelity and flexibility. Representative models such as Variational Autoencoders (VAEs) [1], Generative Adversarial Networks (GANs) [2], and diffusion models [3] have demonstrated remarkable success in creating highly realistic brain images while preserving essential biological and clinical patterns [4–6].

Brain imaging, which often requires sophisticated collection techniques such as magnetic resonance imaging (MRI) and positron emission tomography (PET), is inherently expensive and time-consuming to acquire [7, 8]. Furthermore, real-world datasets frequently suffer from label imbalance, particularly for rare diseases or underrepresented demographic groups [9]. Generative models are able to address these challenges by producing synthetic brain images that enhance both size and diversity of datasets. For example, generative models trained on limited data, such as scans from patients with rare neurological disorders, can generate diverse synthetic images that preserve biologically relevant features [10]. By conditioning the generation process on variables such as age, sex, clinical phenotypes, or genetic factors, these models provide new opportunities for studying underrepresented scenarios and performing experiments that would otherwise be infeasible. These advances allow researchers to augment datasets and even model disease progression in a controlled manner [11]. Additionally, synthetic data offer a potential solution for data privacy concerns, as they allow the use of realistic non-identifiable data [12].

Beyond data augmentation, generative models offer advanced capabilities for enhancing image quality and resolution, facilitating the extraction of detailed structural and functional information from low-quality scans [13, 14]. Additionally, these models can harmonize imaging data collected from different platforms, reducing inconsistencies and enabling more reliable comparisons across studies [15, 16]. By addressing these technical and practical challenges, generative models have become indispensable in brain imaging research, allowing deeper insights into neurological conditions and their progression.

As these technologies continue to advance, they hold immense potential to revolutionize neuroscience by improving our understanding of brain disorders, enhancing diagnostic accuracy, and accelerating the development of personalized treatment strategies. This paper provides a comprehensive overview of the methodologies and applications of generative models in brain images and measures, with a particular focus on conditional generation methods. By categorizing existing approaches and addressing emerging challenges, this review aims to guide future research and support the integration of generative models into neuroscience workflows.

The remainder of this paper is structured as follows. In Sect. 2, we introduce the fundamental concepts of generative models and their conditional variants. Section 3 presents a detailed review of their applications in brain imaging and imaging measures, categorized by methodological approaches. Finally, Sect. 4 summarizes the review, discusses current challenges, and highlights future directions for the use of generative models in neuroscience.

Fundamentals of conditional generation

Generative models

Generative models are machine learning frameworks designed to learn the underlying distribution of data , i.e., , and generate new samples that mimic real-world data, , where represents learnable parameters of a generative model. They are useful for synthesizing high-dimensional and complex data and have become invaluable across various domains such as computer vision, natural language processing and neuroimaging. In this section, we introduce three popular generative models: Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), and diffusion models.

Variational Autoencoders. As shown in Fig. 1a , autoencoders are neural networks that encode input data as a latent representation and reconstruct the original input from the . VAEs are variants of the autoencoder that learn a probabilistic distribution of latent representations instead of fixed representations [1]. VAEs consist of two key components, encoder and decoder . The approximates a posterior distribution and approximates conditional likelihood distribution where parameters and are learned by neural networks. VAEs are trained by maximizing the evidence lower bound (ELBO) on the marginal data log-likelihood , expressed as:

1

where the first term indicates reconstruction loss and second term is the Kullback–Leibler (KL) divergence between the posterior distribution and the prior distribution . When properly trained, the decoder serves as a generative model by decoding random samples from the latent space . VAEs can generate diverse data distributions, but often produce low-quality samples due to regularization of the latent space [17].

Fig. 1.

Overview of different generative models. a Variational Autoencoders (VAEs) [1] map data samples x to latent distribution of z using encoder, and generate outputs by sampling from this space via a decoder. b Generative Adversarial Networks (GANs) [2] train a generator G and a discriminator D adversarially, where the generator creates synthetic data, and the discriminator distinguishes between real and synthetic data. c Diffusion models involve two processes: a forward process (solid arrow) that progressively add noise to the data, and a reverse process (dashed arrow) that removes the noise to generate clean outputs

Generative Adversarial Networks. GANs are generative models designed to sample from unknown data distributions [2]. As illustrated in Fig. 1b , GANs consist of two neural networks, a generator G, which maps a noise vector to synthetic sample , and a discriminator D, which estimates the probability that a given sample originates from real data rather than from G.

The training process is formulated as a two-player minimax game with the value function V(G, D):

2

to balance the G and D as a Nash equilibrium. In this adversarial framework, G learns to generate realistic data that can deceive D, while D improves its ability to distinguish real from synthetic samples. This dynamic is often analogized to counterfeiters versus police, where each network improves in response to the other. This adversarial training process allows GANs to generate highly realistic synthetic data. However, GANs have poor mode coverage, which means they may fail to represent the full diversity of the data distribution [18].

Diffusion models. Diffusion models are the frontier of generative models which are capable of approximating highly complex data distributions. They operate through two key processes, i.e., forward process and reverse process (see Fig. 1c). The forward process gradually adds Gaussian noise to an input, transforming it into complete Gaussian noise over a series of steps. The reverse process recovers the original representation by denoising the Gaussian noise iteratively. This can be analogized to obscuring an image with fog and then learning to remove the fog step by step to reconstruct the original scene.

Diffusion models can be understood from two perspectives: the variational perspective [3, 19] and the score-based perspective [20, 21]. In this paper, we focus on the variational perspective with the Denoising Diffusion Probabilistic Model (DDPM) [3] being a prominent example. In DDPM, forward process at step t and approximate posterior are defined as follows:

3

where T is total number of diffusion steps, are the variance schedule across diffusion steps, and is the identity matrix. The represents the latent variable in time step t and represents the normal distribution of mean and covariance . The reverse process starting from is formulated as:

4

where and are reverse model parameters. DDPM is trained by optimizing variational bound on negative log-likelihood:

5

Through this training process, the diffusion models learn to reverse the noising steps, allowing the generation of realistic and high-quality samples. However, both training and sampling from diffusion models remain computationally expensive.

Conditional generation: concepts and mechanisms

In standard (unconditional) generative models, the synthesis process relies solely on the learned latent representation of the dataset, without external control over the output (Fig. 2a). In contrast, conditional generative models incorporate auxiliary inputs, referred as conditioning variables, to guide the generation process. These variables include categorical labels (e.g., disease status), continuous values (e.g., age or symptom severity), or even other modalities (e.g., images). By integrating such information, conditional models enable the generation of data that aligns with the desired condition, opening possibilities for applications in neuroscience. For example, conditional models can be used to generate synthetic Magnetic Resonance (MR) images representing specific age groups or disease stages, allowing researchers to simulate rare or challenging scenarios, such as advanced stages of neurodegeneration.

Fig. 2.

A schematic view of various conditioning mechanisms. Given a latent vector z, the generator G samples a synthetic sample , given conditions c and . a Unconditional generation: the model generates outputs without any conditional input. b Conditional generation via concatenation: the conditional variables are concatenated with the generator’s input. c Conditional generation via intermediate layers: the conditional variables are introduced into the intermediate layers of the model. d Hybrid conditional generation: the conditional variables are utilized both at the input and within the intermediate layers of the model

In Fig 2, we summarize the mechanisms of conditional generation which involve architectural and training modifications to effectively integrate conditioning variables into generative models. One of the simplest and popular methods is concatenating the condition with the latent input (e.g., noise vector) fed to the generator (Fig 2b). This approach enables the model to learn a mapping between the input conditions and the generated data. For example, in conditional GANs (cGANs), both the generator and discriminator are conditioned on the same auxiliary information, ensuring that the generated images adhere to the desired criteria [22]. Another approach introduces the conditioning variables into the intermediate layers of the model (in Fig. 2c). Feature-wise transformations, such as adaptive instance normalization (AdaIN) [23], dynamically adjust the generator’s parameters based on the conditioning inputs, allowing fine-grained control over the synthesis process. Unlike GAN-based models, advanced generative models such as diffusion models or transformers leverage attention mechanisms to incorporate conditioning information [24]. In some cases, the conditioning variable can be utilized in multiple ways, depending on its characteristics and the specific requirements of the task (in Fig. 2d). Additionally, in specific tasks such as image translation (e.g., cross-modal synthesis), images themselves can serve as the conditioning variable without the need for a random noise vector . These mechanisms ensure that the generation process focuses on specific aspects of the conditioning inputs, improving the fidelity and alignment of the generated data with the desired condition attributes.

Conditional generation in brain imaging

Conditional generative models have been extensively applied in brain imaging, each offering unique strengths and capabilities. This section categorizes these applications based on the underlying model architecture: i) Conditional Variational Autoencoders (cVAEs), ii) Conditional Generative Adversarial Networks (cGANs), iii) Conditional Diffusion Models, and iv) Other Conditional Generative Approaches including encoder-decoder models and transformer-based methods.

The conditional generative models reviewed in this paper have been applied to a wide variety of brain imaging datasets. Large and diverse datasets are essential for training effective generative models, however, in the medical domain, data collection is often constrained by privacy and the high cost of expert annotations. Several public neuroimaging datasets have been curated to address these challenges; the Alzheimer’s Disease Neuroimaging Initiative (ADNI), the Brain Tumor Segmentation (BraTS) challenge datasets, and the Open Access Series of Imaging Studies (OASIS) are particularly prominent, each serving different research purposes.

ADNI [26] is a longitudinal study designed to identify biomarkers of Alzheimer’s Disease (AD) progression. Spanning multiple phases, it includes over 2,500 participants with a distribution across cognitively normal individuals, mild cognitive impairment (MCI), and AD patients. The dataset provides structural and functional MRI, PET imaging, genetic data, cognitive assessments, cerebrospinal fluid (CSF) biomarkers, and extensive clinical and demographic information. ADNI has become a cornerstone resource for AD progression modeling and biomarker validation. BraTS [33] is a series of datasets developed as part of the Brain Tumor Segmentation challenges, aiming to benchmark algorithms for brain tumor segmentation. It primarily consists of multi-parametric MRI scans, including native T1, contrast-enhanced T1 (T1-CE), T2, and FLAIR sequences. Annotated tumor regions are provided by expert radiologists. The BraTS datasets, updated annually since 2012, cover a range of glioma types and include pre- and post-operative imaging, facilitating research into tumor growth modeling and survival prediction. OASIS [29] is a collection of datasets focused on aging and cognitive decline. OASIS-3, the most extensive release, spans 15–30 years of longitudinal imaging and clinical data collection from over 1,000 participants. It includes T1-weighted, T2-weighted, FLAIR, ASL, SWI, resting-state BOLD, and diffusion imaging (DTI), along with PET imaging using various tracers. Comprehensive clinical and cognitive assessments accompany the imaging data, making OASIS valuable for studying both normal aging and the early stages of dementia.

Based on these datasets and model architectures, the following subsections are organized by the type of conditional generative models. A summary of the major datasets utilized across the reviewed studies is provided in Table 1. The conditional generative models are summarized in Tables 2–5, providing an overview of their applications.

Table 1.

Summary of datasets used in multiple papers reviewed in this survey

Dataset name	Data type	Size	Description
UKB [25]	MRI, fMRI, DXA, genetic data, biomarker, clinical assessments, lifestyle evaluations	500,000 subjects	Biomedical database for health and disease research
ADNI [26]	MRI, fMRI, PET, biomarker, genetic data, clinical assessments, demographics	1,500 subjects (ADNI 4)	Dataset for earlier AD diagnosis and biomarker validation
AIBL [27, 28]	MRI, PET, biomarker, lifestyle evaluation	3,045 subjects	Dataset for delay onset, prevent or treat AD
OASIS [29, 30]	MRI, PET, clinical and cognitive assessments, demographics	1,378 subjects (OASIS-3)	Dataset for AD prediction and brain aging analysis
HCP [31]	MRI, fMRI, DTI, MEG, behavioral and cognitive assessment	1,206 subjects	Dataset for human connectome characterization
IXI [32]	MRI, PD, MRA, DWI, demographics	600 subjects	Brain MRI dataset of normal, healthy subjects
BraTS [33, 34]	MRI	2,200 subjects (BraTS 2024)	Dataset for brain tumor segmentation

This table includes all datasets employed in more than one of the reviewed studies. For datasets with multiple versions, statistics from a representative version are summarized. When multiple versions exist, the earliest and most recent publications are cited

Table 2.

Summary of conditional variational autoencoders (cVAEs) in neuroimaging reviewed in this survey

Model	Dataset	Condition	Task
Lawry et al. [37]	UKB, ADNI	Age, intracranial volume	Removing confounding variables
An et al. [15]	ADNI, AIBL, MACC	Site	Data harmonization
Sidulova et al. [38]	ABIDE	Age, sex, label	Functional connectivity analysis
Xuetong et al. [39]	OASIS	Age, sex, intracranial volume	Normative modeling
Lu et al. [40]	HCP	Label	Individual identification

Most cVAE applications aim to mitigate the influence of confounding attributes by incorporating them as conditioning variables. Dataset references are listed below, with the first paper cited for multiple versions: UKB [25], ADNI [26], AIBL [27], MACC [41], ABIDE [42], OASIS [29], HCP [31]

Table 5.

Summary of other conditional generative models in neuroimaging reviewed in this survey

Model	Dataset	Condition	Task
Song et al. [14]	*, ADNI, IXI	Image	One-to-one MRI synthesis and superresolution
Han et al. [94]	BraTS	Image	Multi-to-one MRI synthesis
Liu et al. [95]	*	Image, Mask	Brain lesion segmentation with synthesized DWI
Baek et al. [96]	ADNI	Style	Missing imaging measures imputation
Beak et al. [97]	ADNI	Modality	Missing imaging measures imputation
Gui et al. [98]	BraSyn	Image	Non-contrast MRI to contrast-enhanced MRI synthesis
Ngo et al. [99]	NeuroQuery, IBC	Text	Text to brain activation map synthesis

Dataset references are listed below, with the first paper cited for multiple versions. An asterisk (*) indicates studies that produce new data for model training: ADNI [26], IXI [32], BraTS [33], BraSyn [100], NeuroQuery [101], IBC [102]

Conditional variational autoencoders (cVAEs)

Conditional Variational Autoencoders (cVAEs) [35, 36] enhance the traditional VAE framework by incorporating conditioning variables into the latent space. This enhancement enables the generation process to be guided by specific attributes, providing control over the output. Although cVAEs are less prevalent in neuroimaging in recent years, they have shown effectiveness in several applications. Specifically, cVAEs are designed to disentangle the effect of specific attributes by incorporating them as conditional inputs. A summary of the reviewed cVAE papers is provided in Table 2.

One notable application of cVAEs is the generation of synthetic brain images while controlling for confounding factors, such as age and intracranial volume [37]. By reducing bias through conditioning, cVAEs have been employed to analyze functional connectivity in Autism Spectrum Disorder, achieving a 3–10 % improvement in Pearson correlation between reconstructed and input samples [38]. They have also been used to identify brain dysfunction in Alzheimer’s Disease via normative modeling, where conditioning combined with adversarial learning led to a 6 % increase in AUROC [39].

Furthermore, cVAEs have been utilized as harmonization, enabling standardized imaging across multiple sites and supporting consistent cross-study analyses [15]. They have also been applied to capture inter-subject variability, facilitating individual identification [40].

Conditional generative adversarial networks (cGANs)

Conditional Generative Adversarial Networks (cGANs) [22] are the most widely used generative models in neuroimaging, as they produce high-resolution, realistic images conditioned on auxiliary variables. In most applications, cGANs guide the image generation process by incorporating specific attributes or modalities as conditioning information. Table 3 provides an overview of the reviewed cGAN studies, highlighting key details.

Table 3.

Summary of conditional generative adversarial networks (cGANs) in neuroimaging reviewed in this survey

Model	Dataset	Condition	Task
Zhuang et al. [43]	Neurovault	Label	fMRI generation
Jung et al. [44]	ADNI	Label	fMRI generation
Uzunova et al. [10]	BraTS, IXI, LONI LPBA40	Image	Brain tumor MRI generation
Du et al. [45]	BraTS	Image	Pseudo-healthy MRI generation
Tang et al., [46]	BCP	Age	4D infant brain atlas generation
Jaouen et al. [47]	BraTS	Image	T1 to T2 synthesis
Fu et al. [13]	*	Image	Low dose PET to full dose PET synthesis
Ang et al. [48]	*	Image	MRI to CT synthesis
Chen et al. [49]	*	Image	MRI to CT synthesis
Ren et al. [50]	HCP	Image, q-space information	MRI to DWI synthesis
Pinetz et al. [52]	*, BraTS	Image, Metadata	Contras-enhanced MRI generation
Hu et al. [53]	BraTS, IXI	image, k-space features	One-to-one and multi-to-one MRI synthesis
Tang et al. [54]	BigNeuron	Skeleton of neuron	3D skeleton to neuron image synthesis
Liu et al. [51]	BraTS	Image, Modality	Multi-to-multi MRI synthesis
Zhang et al. [8]	ADNI	Image	PET synthesis using multi-modal MRI with missing modalities
Yu et al. [55]	*	Image	Mouse brain segmentation with missing modality imputation
Pan et al. [56]	ADNI	Image	Diagnosis using MRI and PET with missing modality imputation
Nguyen et al. [57]	NFBS, [58]	Image	Inpainting-based brain tumor segmentation
Xin et al. [7]	BraTS	Image, Modality	Multimodal MRI synthesis from single MRI
Ferreira et al. [59]	BraTS	Image, Label	Brain tumor segmentation with data augmentation
Hamghalam et al. [60]	BraTS	Image	Brain lesion segmentation with enhanced images
Wegmayr et al. [11]	ADNI, AIBL	Image	Prediction of MCI-AD conversion using synthetic aged images
Liu et al. [16]	UKB, PPMI, ADNI, ABCD, ICBM	Image, Style	Data harmonization

cGAN applications leverage conditioning variables to enable high-fidelity image synthesis and cross-modal translation. Dataset references are listed below, with the first paper cited for multiple versions. An asterisk (*) indicates studies that produce new data for model training: Neurovault [61], ADNI [26], BraTS [33], IXI [32], LONI LPBA40 [62], BCP [63], HCP [31], BigNeuron [64], NFBS [65], AIBL [27], UKB [25], PPMI [66], ABCD [67], ICBM [68]

cGANs have been used to synthesize fMRI data based on given labels [43, 44], simulate MR images with tumors [10], and create both pseudo-healthy and pseudo-pathologic MR images [45]. They can also produce longitudinal brain images, such as 4D brain atlases conditioned on age with tissue segmentation map [46].

A key strength of cGANs is their capability for cross-modal synthesis, facilitating data translation between imaging modalities. Examples include transforming T1-weighted images to T2-weighted images [47], converting low-dose PET scans to full-dose PET scans [13], synthesizing CT scans from MR images [48, 49], generating DWI from MR images [50], and generating multi-modal images from a single input [51]. Beyond imaging, cGANs have been applied to integrate non-imaging modalities, such as generating low-dose MR images using metadata [52] or synthesizing MR images conditioned on k-space features [53]. They have even been used to generate neuron images from the skeleton of neurons [54]. cGANs are also effective for imputing missing modalities in medical imaging, including reconstructing missing modalities in MR images [8], imputing missing mouse MR images [55], and completing unpaired MR images or PET scans to MRI-PET pairs [56].

In segmentation tasks, cGANs have been widely applied for data augmentation, improving the performance of the model for brain tumor segmentation [7, 57, 59] and brain lesion segmentation [60]. Additionally, cGANs support longitudinal data synthesis, such as generating follow-up MR images to predict Mild Cognitive Impairment (MCI) to Alzheimer’s Disease (AD) conversion [11]. Furthermore, cGANs play a crucial role in harmonizing multi-site imaging datasets, ensuring consistent cross-study analyses and improve data comparability [16].

Quantitative evaluations further demonstrate the effectiveness of cGANs. In [10], the inclusion of synthetic tumor MR images led to a substantial improvement in segmentation performance, raising the Dice score from 0.01 to 0.70 when only healthy data were originally available. In [45], synthetic pseudo-healthy and pseudo-pathological data improved the Dice score for legion segmentation by 14 %. Similarly, cGAN-based harmonization in [16] resulted in a 2–9 percentage points improvement in Dice score.

Conditional diffusion models

Conditional diffusion models [69] utilize conditioning variables during the reverse process, iteratively transforming noise into data with exceptional fidelity and fine-grained control. These models have demonstrated remarkable versatility and precision across a wide range of brain imaging research. By integrating diverse sources of conditioning information, conditional diffusion models enable targeted synthesis and cross-modal translation with enhanced realism and control. Their applications are detailed in Table 4.

Table 4.

Summary of conditional diffusion models in neuroimaging reviewed in this survey

Model	Dataset	Condition	Task
Jeong et al. [70]	*	Image	Adjacent slice conditioned CT generation
Peng et al. [71]	ADNI, [74]	Image	MRI generation
Han et al. [72]	SegTHOR, OASIS	Image, Mask	CT and MRI generation
Litrico et al. [73]	OASIS	Image, Time interval, Cognitive status, Age	MRI generation
Tapp et al. [75]	SynthRad	Image	MRI to CT synthesis
Dayarathna et al. [76]	*	Image	Ultra low field MRI to high field MRI synthesis
Jiang et al. [77]	BraTS, IXI	Image	Multi-to-one MRI synthesis
Hu et al. [78]	*	mGRE signal	MRI synthesis
Bae et al. [79]	*, ISLES, UniToBrain	Image, Temporal distance, Category of known scans	Temporal inpainting in 4D CT perfusion
Cho et al. [80]	ADNI, OASIS	Image, Age, Label	Longitudinal imaging measures generation
Xiao et al. [81]	TADPOLE	Image, Demographics, Baseline diagnosis, Time difference	Cortical thickness trajectory prediction
Kebaili et al. [82]	BraTS	Tumor characteristics	MRI and mask generation for tumor segmentation
Guo et al. [83]	WMH	Image	Brain segmentation
Zong et al. [84]	ADNI	Image	Brain network generation
Jeon et al. [85]	ADNI	Label, Genetic information	MRI generation

Conditional diffusion models are utilized in scenarios requiring high-fidelity generation guided by specific attributes or contextual information. Dataset references are listed below, with the first paper cited for multiple versions. An asterisk (*) indicates studies that produce new data for model training: ADNI [26], SegTHOR [86], OASIS [29], SynthRad [87], BraTS [33], IXI [32], ISLES [88], UniToBrain [89], TADPOLE [90], WMH [91]

A significant application of conditional diffusion models is the synthesis of 3D imaging data, such as MR images and CT scans, guided by contextual information from 2D slices [70, 71]. These models can also generate images aligned with the specific masks, enabling the creation of medical images with segmentation map [72]. Additionally, patient-specific information, such as cognitive status and age, can serve as conditioning variables to synthesize degenerated brain MR images, supporting studies on neurodegnerative progression [73].

Figure 3 presents a representative example of a clinically-informed conditional diffusion model, illustrating the integration of genetic information and labels into intermediate layers via the attention mechanism (Fig. 2c) for MR image generation [85]. The comparison of outputs from multiple generative models highlights how different conditioning approaches influence the visual and structural characteristics of generated MR images, ultimately enhancing disease-specific features and visual fidelity. Notably, this model achieves state-of-the-art performance on some metrics while performing on par with other approaches in others, effectively capturing AD-specific variations and improving accuracy in downstream classification tasks [22, 24, 92, 93]. This visually illustrates the strength of conditional diffusion models in leveraging rich contextual variables to generate clinically meaningful images.

Fig. 3.

Real images and the conditioned(labels and genotype) samples of brain MRI generated from various generative models (cGAN [22], AttnGAN [92], StackGAN [93], LDM [24], Jeon et al. [85]) for three diagnostic groups (Cognitive Normal (CN), Mild Cognitive Impairment (MCI), Alzheimer’s Disease (AD)). (from [85]; with permission)

Beyond single-modal image synthesis, conditional diffusion models have demonstrated superior performance in cross-modal synthesis, facilitating translations between imaging modalities. For instance, they have been successfully applied to MRI-to-CT [75] and low-field to high-field MRI synthesis [76]. These models can also create multi-modal MR images [77, 78]. Conditional diffusion models address modality-specific gaps in medical imaging datasets and enhance data accessibility for diverse research needs.

In temporal applications, conditional diffusion models have been used to impute missing time points in 4D CT perfusion imaging [79], and predict trajectories of imaging measures, such as changes in cortical thickness over time [80, 81]. They also demonstrate strong performance in segmentation tasks, including tumor segmentation [82] and brain segmentation [83]. Constructing brain network is an another emerging application of conditional diffusion models. By integrating structural and functional brain imaging data, these models facilitate the modeling of brain connectivity, advancing our understanding of neural interactions [84].

Quantitative evaluations further support the effectiveness of conditional diffusion models. In [71], Fréchet Inception Distance (FID) of synthetic MRI was six times lower than that of images generated by a non-conditional diffusion model, indicating significant improved image quality. In [85], incorporating genetic data into the conditioning process led to nearly a 10 percentage points higher in downstream classification tasks. Additionally, in [79], conditioning on the relative order of the target CT scan with respect to two known scans substantially improved the quality of perfusion parameter maps, reducing the RMSE by half, particularly in cases involving long acquisition intervals.

Other approaches

Other conditional generative approaches, summarized in Table 5, including encoder-decoder frameworks and transformer-based models, both of which have been successfully applied to brain imaging tasks, offering unique strengths and versatility.

Encoder-decoder architectures have been widely used, such as cross-modal translation [14, 94], segmentation [95], and imputing missing brain imaging measures [96, 97]. These models are particularly advantageous for managing latent space, enabling better alignment with desired downstream tasks. This flexibility allows encoder-decoder frameworks to address various imaging challenges, offering fine-grained control and adaptability. For example, in Fig. 4, the authors presented a framework that generates unobserved brain imaging measures for specific subjects using their existing measures, which transfers modality-specific style (Fig. 2d) while preserving AD-specific content. The visual comparison in Fig. 4 demonstrate the imputation result demonstrated high generative quality with small average Cohens d between the generated measures and real ones, significantly enhancing downstream Alzheimer classification task [96].

Fig. 4.

Visual comparison of observed measurement (Top) and imputed value (Middle) on the inner view of the left hemisphere from a Cognitive Normal (CN) subject. Imputed values were generated from the subject’s available cortical thickness data using a conditional encoder-decoder framework. All measurements were standardized. The bottom row shows the distance between real and imputed data, indicating high structural fidelity of the generated output. (from [96]; with permission)

By leveraging attention mechanisms, transformer-based models introduce innovative capabilities to conditional generation tasks as well. For example, they have been utilized for dose-variant autoregression to synthesize T1Gd images [98] and for generating brain activation maps from free-text queries [99]. These models also outperform in tasks such as segmentation and cross-modal synthesis, providing adaptability and scalability to novel applications. Although transformer-based approaches are currently less prevalent than cGANs or diffusion models in neuroimaging, their complementary capabilities, such as handling complex relationships among variables and integrating diverse data types, make them a valuable addition to conditional generation of neuroimaging data.

Quantitative results highlight the effectiveness of these models. In [96], imputation using domain-agnostic embedding improved downstream Alzheimer’s Disease classification accuracy by 3 percentage points. In [98], generating contrast-enhanced brain MR images through sequential steps led to a 10 % increase in PSNR and in 5 % improvement in Dice score for tumor segmentation tasks.

Discussion and conclusion

Conditional generative models have emerged as powerful tools in neuroimaging field. These models are applied across a variety of tasks, including data augmentation, imputation, image translation, segmentation, and data harmonization. Among these techniques, cGANs have been particularly popular due to their superior performance and versatility. In recent years, there has been a notable surge in the adoption of conditional diffusion models, reflecting their growing prominence and potential for generating high-fidelity neuroimage and image-derived measures.

For practical utilization, conditional generative models still face several limitations in brain imaging. First, the fidelity and clinical plausibility of synthetic images remain challenging, particularly for complex structures such as small lesions or subtle anatomical variations [60]. This limitation reduces their utility in applications that require precise details, such as detecting early-stage neurodegenerative changes. Second, effectiveness of conditioning can be inconsistent across datasets and tasks [50]. For instance, models conditioned on demographic or clinical variables may struggle to generate realistic images when these variables are sparsely represented in the training data. Third, computational demands are a significant barrier, particularly for diffusion models, which require extensive computational resources for training and inference. Fourth, there is a lack of robust evaluation metrics tailored to brain imaging tasks. Although metrics such as Fréchet Inception Distance (FID), Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM) are commonly used, they may not adequately capture the biological plausibility or clinical utility of synthetic images [103]. Finally, ethical and regulatory concerns surrounding the use of synthetic data remain significant. Potential risks include the propagation of biases due to imbalanced training data, inadvertent generation of identifiable features posing privacy threats, and limited clinical acceptance stemming from insufficient validation of synthetic data. Addressing these challenges is essential for the responsible and effective integration of generative models in medical practice.

Despite the challenges listed above, conditional generative models present several exciting opportunities to facilitate brain imaging research. Integration with multi-modal data is a promising direction, where models can jointly process diverse data types, such as imaging, genomics, and clinical metadata, to generate comprehensive and contextually enriched datasets [81, 85]. Additionally, the application of transformer-based architectures in conditional generation offers untapped potential to improve long-range dependencies and complex feature conditioning. Another key opportunity lies in the development of adaptive and explainable generative models, which may provide insight into the relationships between conditioning variables and generated outputs, enhancing their interpretability and trustworthiness in clinical applications. Furthermore, personalized medicine could benefit significantly from conditional generation by enabling the simulation of individual-specific disease trajectories or treatment responses [80]. These synthetic samples will help deal with privacy concerns by ensuring models generate non-identifiable synthetic data, which will be critical for ethical compliance and wider adoption in sensitive domains like healthcare [44, 70].

Recent progress in computational efficiency through techniques such as model distillation and hardware acceleration will foster generative models more accessible to researchers with limited computational resources [13, 75, 82]. In parallel, determining the appropriate use of standardized metrics [104] and developing novel evaluation criteria tailored to clinical relevance and biological correctness in medical imaging [105] are essential for fostering a robust and meaningful research environment in neuroimaging. By embracing these opportunities, conditional generative models will further help develop transformative research in neuroimage analysis and advance scientific discoveries in understanding brain functions.

Funding

Open Access funding enabled and organized by Pohang University of Science and Technology (POSTECH). Open Access funding enabled and organized by Pohang University of Science and Technology (POSTECH). This work was supported by the National Research Foundation of Korea (NRF) funded by Korea government (MSIT) (Grant Numbers NRF-2022R1A2C2092336) and Institute of Information & communications Technology Planning & Evaluation (IITP) funded by Korea government (MSIT) (Grant Numbers RS-2022-II2202290 and RS-2019-II191906 (Artificial Intelligence Graduate School Program at POSTECH)).

Declarations

Conflict of interest

The authors have declared that no Conflict of interest exists.

Ethical approval

No human or animal subjects are involved in this study.

Consent to participate

No human or animal subjects are involved in this study.

Consent to publish

No human or animal subjects are involved in this study.