Updated Primer on Generative Artificial Intelligence and Large Language Models in Medical Imaging for Medical Professionals

Abstract

The emergence of Chat Generative Pre-trained Transformer (ChatGPT), a chatbot developed by OpenAI, has garnered interest in the application of generative artificial intelligence (AI) models in the medical field. This review summarizes different generative AI models and their potential applications in the field of medicine and explores the evolving landscape of Generative Adversarial Networks and diffusion models since the introduction of generative AI models. These models have made valuable contributions to the field of radiology. Furthermore, this review also explores the significance of synthetic data in addressing privacy concerns and augmenting data diversity and quality within the medical domain, in addition to emphasizing the role of inversion in the investigation of generative models and outlining an approach to replicate this process. We provide an overview of Large Language Models, such as GPTs and bidirectional encoder representations (BERTs), that focus on prominent representatives and discuss recent initiatives involving language-vision models in radiology, including innovative large language and vision assistant for biomedicine (LLaVa-Med), to illustrate their practical application. This comprehensive review offers insights into the wide-ranging applications of generative AI models in clinical research and emphasizes their transformative potential.

INTRODUCTION

The advent of Chat Generative Pre-trained Transformer (ChatGPT), a powerful tool developed by OpenAI, has garnered interest in various generative artificial intelligence (AI) models in the field of medicine. Generative AI, which includes Generative Adversarial Networks (GANs) [1,2], diffusion models [3,4,5], and Large Language Models (LLMs) [6,7,8,9,10], has shown tremendous potential for a wide range of medical applications.

Medical imaging has emerged as a prominent area that can be explored using generative models. GANs and diffusion models have been used in studies focusing on image reconstruction and quality enhancement [11]. The paramount importance of maintaining privacy in medical research has also led to the use of synthetic data. Owing to their ability to produce synthetic medical data that mirror real-world characteristics, generative models offer innovative solutions for data privacy, thereby enabling researchers to conduct studies without compromising patient confidentiality.

The vision-language model (VLM) and LLMs, such as ChatGPT, have accelerated the developmental pace and facilitated the development of numerous applications in the medical field that were previously unimaginable. The increase in the use of AI-driven models in the medical field has necessitated a comprehensive understanding of generative AI models among researchers and practitioners, including general radiologists seeking to leverage these advancements.

Therefore, this review aims to provide a concise overview of the fundamental principles and applications of generative AI models, with a particular focus on medical imaging. This review explores the introduction of basic models, such as variational autoencoders (VAEs), GANs, diffusion models, and their variants, and the underlying mechanisms that drive their generative capabilities. Furthermore, this review also showcases how these models combine their capabilities with the power of LLMs to understand images.

Overview of Common Generative AI Models: VAE, GAN, Diffusion Models, LLMs, and VLMs

In contrast to deterministic models, such as regression models and classifiers, wherein the outcome is completely determined by the parameter and initial input values, generative models incorporate randomness or unpredictability while predicting the answers. Consequently, generative models are referred to as probabilistic or stochastic models. Probabilistic models incorporate probability (chance) to address uncertainty and consider the likelihood of obtaining different outcomes during the prediction process. Stochastic models inherently involve randomness or unpredictability in their prediction processes. These generative models can be categorized according to the type of data process (i.e., vision, language, or both) or the manner in which they train and generate samples (i.e., explicit and implicit models). Explicit models are probabilistic generative models that define the probability distribution of the modeled data. In contrast, implicit models generate data directly via stochastic or random processes without explicitly defining the probability distribution. Figure 1 illustrates the categories of commonly used generative AI models.

Fig. 1. Categories and examples of generative AIs. AI = artificial intelligence, VLM = vision-language model, VAE = variational auto-encoder, GPT = generative pre-trained transformer, CLIP = contrastive language-image pretraining, PaLM = pathways language model, LLaVA = large language and vision assistant, LLaMA = large language model Meta AI, GAN = generative adversarial network.

Vision generative models, such as VAEs [12], GANs [1,2,13], and diffusion models [3,4,5], are defined as models that only process images. Figure 2 depicts the typical architecture of vision-generative models. Remarkable progress in natural language processing (NLP) has facilitated the processing and generation of a massive amount of data by language-generative models, such as ChatGPT, BARD, pathways language model (PaLM) [14], and large language model Meta AI (LLaMA) [15]. Recent advances in multimodal processing in deep learning have also facilitated the processing of natural language and computer vision data by large models. Consequently, AI models can generate language from images, images from language, or both. Examples of VLMs include GPT-4 [16], DALL-E [17,18], and large language and vision assistant (LLaVA) [19].

Fig. 2. Typical architectures of generative models for medical images. A: Variational auto-encoder. B: Generative adversarial network. C: Diffusion model. x = input image, z = latent space, N = normal distribution, ε = noise, x’ = reconstructed image, T = timestep.

Variational Auto-Encoder (VAE)

VAEs are generative models that employ an encoder-decoder architecture with a prior distribution (existing distribution). The encoder maps each input image onto a latent space. The encoded latent feature is subsequently used by the decoder to generate an image. VAEs are trained by approximating the distribution of the encoded latent features to a known distribution (e.g., a normal distribution) and reconstructing the image to resemble the given input. An auto-encoder [20] possesses a similar architecture; however, it focuses on the efficient learning of latent representations rather than image generation.

Variants of VAE Models

VAEs, which are a class of generative models based on the principles of variational inference, have several variants. Conditional VAEs (CVAEs) [21] are an extension of VAEs wherein the encoder and decoder are conditioned on additional information, such as labels or other data. VAEs are particularly useful for tasks such as conditional image generation (e.g., generating images of a particular class). VAEs with arbitrary conditioning (VAEACs) [22] enable flexible conditioning (i.e., conditioning on arbitrary subsets of observed variables). Consequently, VAEACs have been used for various tasks such as inpainting, denoising, or feature prediction. Disentagled VAE (beta-total correlation VAE, β-TCVAE) [23] introduces an additional term in the loss function that facilitates the learning of disentangled representations by minimizing the total correlation between different latent variables. Hierarchical VAEs [24] are characterized by a hierarchy of latent variables, with each level capturing a different level of abstraction in the data. Hierarchical VAEs can model complex data distributions more effectively and capture abstract representations at higher levels of hierarchy. Sequential VAEs, such as deep recurrent attentive writers (DRAW) [25] and variational recurrent neural networks (VRNN) [26], can handle sequential data, such as text or time series.

Generative Adversarial Network (GAN)

GANs comprise two competing networks: a generator, which forges a realistic fake image from a given latent feature, and a discriminator, which distinguishes fake images from real images. The generator attempts to deceive the discriminator via adversarial training, thereby improving image generation. Mode collapse, a phenomenon characterized by the production of a limited variety of samples by the generator, and training instability are the primary limitations of standard GANs.

Variants of GAN Models

Different variants of GANs, such as deep convolutional GAN (DCGAN), conditional GAN (cGAN), progressive growing GAN (PGGAN), style based GANs (StyleGAN), cycle-consistent GAN (CycleGAN), and StarGAN, are characterized by unique characteristics and applications.

DCGANs integrate convolutional neural networks (CNNs) into the architecture of GANs [27]. Transposed convolutional layers are used by the generator to produce images, and common convolutional layers are used by the discriminator to distinguish between the real and generated images. This marks a significant step forward, as most GANs developed prior to DCGANs were based on fully connected layers. Compared with standard GANs, DCGANs yield a significant improvement in the stability of training GANs and facilitate the generation of higher-quality images.

In contrast to standard GANs that generate data from random noise, cGANs [28] generate data conditioned on additional information, such as class labels, data from other modalities, or text descriptions. This conditionality enables the generation of targeted types of images, such as images of a specific class.

PGGANs [29] progressively train generator and discriminator networks, starting from low-resolution images and gradually increasing to higher resolutions. This approach enables the networks to learn large-scale structures initially and then learn finer details progressively as the training progresses. PGGANs achieve more stable training compared with that of a high-resolution GAN from the beginning by focusing on lower resolutions initially and then gradually increasing the complexity.

StyleGANs [2,30,31] introduce a novel architecture, such as a style-based generator, that facilitates unprecedented control over the style and content of the generated images. This key innovation in StyleGANs facilitates separate control over high-level attributes (such as the contour and shape of the brain) and stochastic variations (such as brain sulci) in the generated images. StyleGANs [2] can also introduce a mapping network that transforms a latent code (random input) into an intermediate latent space. This intermediate space can capture the ‘style’ of the generated image. In addition to style, noise inputs can be added at various layers to introduce stochastic variations that are not controlled by the latent code. Improved versions, such as StyleGAN2 [30] and StyleGAN3 [31], have been developed in addition to the original StyleGAN.

CycleGAN [32], which was specifically designed for image-to-image translation where paired examples are not available, uses two-generator networks. Generator G_X learns to map from domain X to domain Y, whereas generator G_Y learns reverse mapping from domain Y to domain X. CycleGAN also comprises two discriminators, D_X and D_Y (Fig. 3). Cycle consistency loss is a crucial component of CycleGAN that ensures the translation of an image from domain X to domain Y and then back to domain X such that the generated image resembles the original image from domain X (and vice versa for domain Y). This component acts as a proxy for the paired training data. Moreover, its ability to work without paired examples facilitates its incorporation into a wide range of real-world scenarios where paired training data are scarce or difficult to obtain. For instance, CycleGANs can convert the reconstruction kernel of computed tomography (CT) images or denoise low-dose CT [33,34,35,36,37,38].

Fig. 3. CycleGAN architecture to generate high-dose CT images from low-dose CT using an unpaired dataset. CycleGAN = cycle-consistent generative adversarial network, CT = computed tomography, D = discriminator, G_A = high-dose CT generator from low-dose CT, G_B = low-dose CT generator from high-dose CT, x = low-dose CT image, y = high-dose CT image, X = domain of low-dose CT, Y = domain of high-dose CT, ‘ = generated image from real image, “ = generated image from generated image.

StarGAN [39], a versatile GAN that can perform multiple image-to-image translations simultaneously, can handle multiple domains when translating between multiple medical imaging modalities, such as multi-contrast magnetic resonance (MR) images and multi-view echocardiography [40,41]. StarGAN is particularly useful in scenarios encompassing diverse datasets and imaging equipment.

Diffusion Model

The diffusion model, a newer generative model, uses noise distribution and operates in forward and reverse processes based on a Markov chain, a discrete time-stochastic process with Markov property. This property dictates that the future state of the process is determined solely by its current state and is independent of its history. The transition from one state to another is determined by the probability associated with the current state. Owing to this property, a Markov chain can efficiently describe the transition between two states over time (or steps) by multiplying the initial probability by the probabilities of subsequent steps. Thus, it is a concise and easy method that can be used to determine dynamic changes in the data. Random noise (i.e., Gaussian noise) is added to a given image in multiple scheduled steps in the forward process of a diffusion process. In contrast, the model denoises the given image and predicts the image from the previous step in the reverse process. The model is trained by predicting “the noise” from the given image, and images are generated by denoising a random noise step by step (a Markov chain). Figure 4 presents an example of the generation of three-dimensional medical images from two-dimensional slices using the mask inpainting technique of a diffusion model sampler.

Fig. 4. 2D slice-wise generation of 3D CT volume using the mask inpainting technique of the denoising diffusion probabilistic model sampler. A: Training 3D diffusion model. B: 3D slice generation using diffusion sampling. D = dimensional, CT = computed tomography, s_t = t^th slice of CT image, Mask for s_t = empty mask for generative inpainting to generate s_t slice.

Variants of Diffusion Models

Variants of diffusion models include score-based generative models, denoising diffusion probabilistic models (DDPMs) [3], and denoising diffusion implicit models (DDIMs). The “score,” which refers to the gradient of the log probability density of the data in score-based models [5], serves as a guiding factor for the forward (where noise is added to the data) and reverse (where the data are recovered from noise) processes utilizing stochastic differential equations.

DDPMs are based on the principle of a diffusion process, which is a Markov chain that gradually adds noise to data over a series of steps. This process transforms data into a simple known distribution (typically Gaussian noise). Forward diffusion is a fixed process that is not learned, whereas the reverse process is learned by a neural network trained to predict the noise added at each step of the forward process. The model can reconstruct the original data from noise via iterative denoising.

DDIMs modify the traditional diffusion process to facilitate efficient and deterministic generation of samples [4]. In contrast to DDPMs, which explicitly model noise at each step of the diffusion process, DDIMs use an implicit modeling approach. Consequently, DDIMs can define a deterministic trajectory for the reverse process (from noise to data) without explicitly modeling the noise distribution at each step. The deterministic non-Markovian reverse process of DDIMs is one of its unique features. The reverse process involves the addition of a small amount of random noise at each step in DDPMs, making the process inherently stochastic. In contrast, DDIMs follow a deterministic path. Consequently, the output remains the same, given a starting noise. DDIMs can generate samples more rapidly, while ensuring minimal changes in the quality of the generated image. This is particularly beneficial for reducing computational overhead.

Large Language Model (LLM)

LLMs, such as ChatGPT and GPT-4 [16], are based on transformer architecture [42]. The transformer, the core component of LLMs, enables these models to understand and generate human-like text. Words are processed simultaneously in the transformer, rather than one after the other, which makes it a great tool for understanding the context of a language. Transformers use a mechanism known as “attention” to weigh the relevance of different words when generating responses or predictions. LLMs train on vast amounts of text data and scale the model architecture to a larger size.

GPT is an LLM that uses a transformer decoder, a specific part of the transformer architecture. As it comprises a decoder-only architecture, it resembles an expert chef who does not have to learn a specific recipe (encoder) since they have already learned hundreds of recipes. Consequently, GPT-based LLMs [6,7,8,16], such as ChatGPT and GPT-4, can effectively generate human-like text by combining the power of the transformer decoder with the broad knowledge learned from extensive training data. GPT can generate texts based on a description or a single or few examples. Figure 5 presents zero-shot, one-shot, and few-shot generation examples of GPT.

Fig. 5. Description of zero-shot, one-shot, and few-shot generation and prompting. A: Description of zero-shot, one-shot, and few-shot generation. B: Prompting examples.

Bidirectional Encoder Representations from Transformers (BERTs) [10] are pre-trained models that use the bidirectional encoder of the transformer architecture, which facilitates the consideration of the left and right contexts during training. BERT is trained on a process known as the masked language model, which masks random words in a sentence. The model predicts the masked words based on the context provided by the surrounding sentences and words. This process has enabled BERT to capture deep contextual representations, resulting in superior performance in various NLP tasks. BERT is a powerful tool for NLP tasks, such as language understanding, sentiment analysis, and question-answering systems, owing to its ability to understand context.

GPT and BERT are built on the transformer architecture and utilize attention mechanisms, which significantly improve NLP tasks. These models have facilitated the completion of more accurate and context-aware language-modeling tasks, making them valuable assets in the domain of NLP. The capabilities of these models have given rise to new avenues for various applications, from the generation of coherent text to understanding and processing languages in a meaningful and contextually relevant manner. LLMs can help create a patient-friendly language for reports and discussions, improve communication with patients, and enhance the understanding of medical conditions and imaging results among patients [43,44,45].

Large Language Model to Vision-Language Model

VLM [46] has been introduced as an AI model that combines the power of natural language understanding with images or visual understanding in recent years. VLMs function by learning from a large amount of text and image data in a manner similar to that of a student learning by reading books and text accompanied by pictures. Owing to their ability to quickly analyze medical reports and images simultaneously, these models can be used as aids by radiologists.

The tasks performed in the field of VLM, which is a multidisciplinary field that combines computer vision and NLP, can be classified into two main categories: generation and perception.

Generation tasks can be grouped into four categories: visual question answering (VQA), visual reasoning, visual captioning, and visual generation. Figure 6A presents the generation tasks in VLM. AI models are presented with a visual input (an image or video) and a question related to the input in VQA [47]. The AI model provides correct responses based on its understanding of the questions and visual input. Visual reasoning requires the deduction of cognitive insights or commonsense knowledge from images by the AI models [48]. AI models create relevant and descriptive captions for the visual inputs provided in visual captioning [49]. Visual generation involves the generation of visual output from a given textual input [50].

Fig. 6. Generation and perception tasks of language-vision models. A: Generation language-vision models. B: Perception language-vision models. CXR = chest radiograph.

Perception tasks can be grouped into three categories: image recognition, visual grounding, and visual retrieval. Figure 6B presents the perception tasks in VLM. contrastive language-image pretraining (CLIP) [51] and a large-scale image and noisy-text embedding (ALIGN) [52] have transformed traditional image recognition tasks into language-vision tasks via image recognition, thereby enabling the recognition of unseen concepts by AI. Visual grounding involves the prediction of the bounding box that corresponds to a text query within an image [53,54]. Image-text retrieval involves the retrieval of the most relevant text or image from a large corpus based on the query provided [55].

General segmentation with prompting uses textual prompts to guide the AI in the identification and outlining of specific parts of an image. The Segment Anything Model (SAM) developed by Meta [56] has exhibited notable performance in this area. This approach can also be applied to the domain of medical imaging [57,58,59,60,61], making it a promising avenue for future research.

Outlook of Foundation Models in Radiology

A foundation model is a large model that is pre-trained on a vast amount of data. Foundation models can be used for diverse downstream tasks via fine-tuning or zero-shot methods [62]. Representative language foundation models include BERT [10] and the GPT series [6,7,8]. Several multi-modal foundation models of VLM, such as DALL-E [17,18], Flamingo [63], and Florence [64], have also been introduced.

A research team from Microsoft recently reported a significant breakthrough in the biomedical domain with the introduction of LLaVA for BioMedicine (LLaVA-Med), a VLM [65]. A research team from Google introduced its counterpart, Med-PaLM, around the same time [66]. These foundation medical VLMs leverage the ability to understand images and interpret text to perform various tasks in the domain of biomedical imaging. The primary objective of these models is to assist in answering open-ended research questions related to biomedical images. To this end, the models learned from a vast biomedical figure-caption dataset extracted from PubMed Central, a large biomedical literature database. The training of LLaVA-Med was performed using a curriculum learning method, wherein learning tasks are presented in a sequence of increasing difficulty, in a manner similar to how humans learn, starting with simple concepts and progressing to more complex ones. The model initially focused on aligning biomedical vocabulary using figure-caption pairs. Subsequently, the model learned to understand open-ended conversational semantics using the data generated by GPT-4, another powerful AI language model. This training process enabled the model to acquire biomedical knowledge gradually, in a manner similar to how a layperson learns. This led to the development of an AI system capable of answering inquiries regarding biomedical images and open-ended research questions. These foundation models of VLMs perform well on standard biomedical VQA datasets [65,66]. The advantage of this system lies in its multimodal approach, as it can comprehend textual and visual information.

Attempts have been made to construct such foundation models in the field of radiology. One group of researchers trained a vision foundation model on 100 million medical images, including radiographs, CT images, MR images, and ultrasound images [67]. Another group of researchers trained a self-supervised network on 4.8 million chest radiographs [68]. However, the networks were not scalable despite these studies training their networks on a vast amount of data and demonstrating their diverse utility. A foundation VLM was implemented in the field of radiology in a recent study [69].

Potential and Application of Generative Models in Clinical Imaging Research

Generative models have demonstrated versatility in numerous tasks, such as denoising, image reconstruction, and vital image analysis. Moreover, they have also been utilized in a technique called “inversion” to transform real data, such as images, into a latent space representation. These models have emerged as a promising approach that can address multiple challenges in medical research by generating medical data.

Denoising, Image Reconstruction, Inter-Modality Synthesis, and Imaging Analysis Tasks

Promising applications of GAN in the field of radiology include image quality improvement via post-processing and faster image acquisition. Radiation exposure and the increased risk of developing cancer associated with CT image acquisition are obstacles faced in the field of radiology. Deep learning-based image reconstruction, particularly using GANs, may be a potential solution to overcome this obstacle. Successful applications of GANs include CT denoising [70,71], artifact reduction [72], and improving the accuracy of radiation therapy planning [73,74]. MRI modalities often require long acquisition times. Deep learning techniques, including GANs, have been used to enable high-quality image reconstruction from undersampled k-space data [75,76,77], effectively speeding up MRI acquisition and improving image quality. In addition, GANs have also been utilized to convert low-magnetic-field MR images into high-magnetic-field MR images [78].

Intermodality synthesis, which has been used extensively to optimize imaging processes, involves converting images between different modalities or sequences [79]. GANs play a crucial role in this process and have been used to synthesize CT images from MR images [80,81], thereby reducing radiation exposure and diminishing the requirement for additional image acquisitions. GANs can also compensate for missing MRI sequence data [82] and support training across multiple modalities [83], thereby facilitating image analysis across various modalities and sequences.

GANs are versatile and effective tools for advancing medical imaging and analysis. Moreover, they have effectively improved the deep learning performance for various radiology tasks, including lesion detection, organ segmentation, and the prediction of patient outcomes, via data augmentation [84,85,86,87,88,89]. GANs have also been used in image registration to yield more accurate results. They have been used successfully in MR-to-transrectal ultrasound image registration and image registration of MR and CT images for thoracic and abdominal organs [90,91]. Furthermore, GANs have also been used to identify abnormal lesions in medical images by learning the data distribution of normal images via unsupervised or semi-supervised learning [92,93,94,95,96,97]. The progression of Alzheimer’s disease has been modeled using GANs via the utilization of MRI data to predict disease changes over longitudinal examinations [98].

Diffusion models have been used effectively in various CT imaging applications, such as CT denoising [99,100,101,102], CT kernel conversion [103], and accelerating MRI acquisition time [104,105,106]. Diffusion models have also demonstrated potential for intermodality synthesis, notably generating synthetic CT images from MR images [107,108,109]. Moreover, these models have been applied to other domains of radiology, such as automatic lesion and organ segmentation [110,111,112,113], image registration [114], and anomaly detection [115,116,117,118,119].

Inversion of Generative Models

Certain real data (e.g., images) can be inverted into a latent space using a technique known as inverse mapping of generative models or inversion of generative models [120]. The inverted latent features are manipulated and edited subsequently [121,122,123]. Figure 7 presents an example of the manipulation of an inverted latent feature and the editing of the images using the generative model inversion technique. GANs have been used for inversion in the medical domain. Ren et al. [124] used GAN inversion to generate tumor-like stimuli with specific shapes, sizes, and realistic textures in mammograms in a controlled manner. Fetty et al. [125] reported that GAN inversion can be used to manipulate the imaging modality (MRI to CT and vice versa) and sex of the patient. Lee et al. [126] demonstrated that GAN inversion can be used to control the presence and severity of a disease by manipulating the latent space. These findings indicate that GAN inversion can address important issues of deep learning in medical imaging and help develop other deep-learning networks by providing images of diverse disease severities.

Fig. 7. Manipulating the inverted latent feature and editing images with the GAN inversion. GAN = generative adversarial network, x = dataset consists of real images, x’ = dataset consists of generated images, W = latent space of dataset x mapped by the inversion encoder, z = latent feature of the real image, n_t = editing parameter of latent feature z, z + n_t = edited latent feature z.

Solving Clinical Research Challenges using Generative Model-Based Synthetic Data

Stringent regulations, such as the Health Insurance Portability and Accountability Act (HIPAA) in the United States and the General Data Protection Regulation (GDPR) in Europe, have restricted the sharing of sensitive patient data, thereby impeding the creation of comprehensive datasets for research [127,128]. Synthetic data sharing has safeguarded patient privacy and facilitated research collaboration [129]. Imbalances in medical datasets, particularly in datasets of underrepresented rare diseases, can skew machine-learning predictions. Synthetic data has been used to rectify this imbalance using methods such as the synthetic minority oversampling technique (SMOTE) by addressing class imbalances [130]. Health inequities are amplified by real-world data bias caused by factors such as age, sex, and socioeconomic status. DEbiasing CAusal Fairness (DECAF) mitigates this bias by leveraging causal structures to synthesize fairer data [131]. Synthetic data, which have been used in clinical simulation tests for AI models in diverse settings, aid in identifying and rectifying potential errors before deployment, thereby minimizing “alert fatigue” and bolstering trust in AI models [132]. Synthetic control arms utilizing external patient-level data are efficient and ethical alternatives that can be used in clinical trials to save resources [133,134].

Pitfalls and Limitations of Generative AI

Generative AI models have demonstrated impressive capabilities in the field of medicine. However, despite their notable strengths, these models are associated with several limitations and potential drawbacks that warrant careful consideration during their application.

First, mode collapse, a phenomenon wherein the generator fails to provide a wide array of outputs, is a primary limitation of GANs that often results in repetitive samples or a limited set of variations [135,136,137]. This limitation influences the diversity and richness of the generated data.

Second, the production of high-quality and accurate samples that mirror the complexity of medical training data remains a challenge. The output may lack coherence or exhibit artifacts in some instances, which can undermine its reliability and usability in medical applications [31,138,139].

Third, these models tend to assimilate the biases present in the training data, which can lead to overfitting issues and a limited ability to generalize to new or unseen medical datasets. The assessment of the uncertainty or reliability of the generated medical samples is a significant challenge that hinders the quantification of predictive confidence and impacts their reliability in critical decision-making processes in healthcare.

Fourth, ethical concerns regarding the use of these models to generate false or misleading medical information, which can compromise authenticity and trustworthiness, have arisen.

Fifth, the computational demands for training these models are substantial and require significant resources and time, thereby limiting their accessibility and scalability, particularly in resource-constrained healthcare environments.

Sixth, the interpretability and control of these models remain challenging, particularly in complex architectures such as GANs. The opaque nature of these models impedes their fine-tuning, understanding, and decision-making processes, which limits their practical applications in medical research and clinical settings [11]. Hallucinations, characterized by the generation of unrealistic or spurious outputs, challenge the credibility and reliability of generated content [140,141]. The outputs, while resembling authentic data, may contain elements or details that are absent in the training data, posing risks to medical decision-making and research.

Lastly, robust generalization of these generative models to unseen medical data distributions remains a challenge that must be surmounted to enable their reliable and ethical use in medical research, diagnosis, and treatment planning.

Thus, the adoption of generative AI in clinical practice should be monitored by experts who can determine the reliability of the generated results. Furthermore, physicians and researchers in the medical field should be aware of the pitfalls and limitations of generative AI and exercise caution while implementing them. Guidelines for the use of generative AIs may facilitate a more comprehensive understanding of their advantages and pitfalls [142,143,144].

CONCLUSION

The landscape of generative AI models encompasses various categories, including vision models (VAEs, GANs, and diffusion models) and language models (LLMs and VLMs). Each category comprises variants and applications tailored for specific tasks, particularly medical imaging and analysis. LLMs, such as ChatGPT and GPT-4, built on the transformer architecture, excel in human-like text generation. In contrast, BERT, which is based on transformers, focuses on bidirectional language understanding. Recent advancements in VLMs have facilitated the execution of tasks, such as VQA, image recognition, and retrieval.

Generative models can significantly enhance image quality in clinical research, expedite MRI acquisition, generate synthetic data for rare diseases, and alleviate concerns regarding patient/data privacy in clinical AI research. Furthermore, the inversion of generative models facilitates the manipulation and editing of features within the latent space, thereby offering insights into latent representations, enabling control over data generation, and supporting various image manipulation tasks in the field of medicine.

Nevertheless, despite these benefits, generative AI is associated with multiple challenges, such as mode collapse, bias amplification, interpretability issues, computational demands, ethical concerns, and the potential to generate unrealistic data (hallucinations). Expert validation and awareness of these limitations play a crucial role in ensuring its responsible use in the medical context. Striking a balance between the potential benefits and disadvantages of generative AI is essential for the ethical and effective integration of these models into healthcare and research practices.