A Current Review of Generative AI in Medicine: Core Concepts, Applications, and Current Limitations

Abstract

Purpose of Review

This review aims to offer a foundational overview of Generative Artificial Intelligence (AI) for healthcare professionals without an engineering background. It seeks to aid their understanding of Generative AI’s current capabilities, applications, and limitations within the medical field.

Recent Findings

Generative AI models, distinct from discriminative models, are designed to create novel synthetic data. Key model families discussed include diffusion models for generating images and videos, Large Language Models (LLMs) for text, and Large Multimodal Models (LMMs) capable of processing multiple data types. Recent applications in healthcare are diverse, encompassing general uses like generating synthetic medical images, automating clinical documentation, and creating synthetic audio/video for training. More specialized applications include leveraging Generative AI models as backbones for diagnostic aids, enhancing information retrieval through Retrieval-Augmented Generation (RAG) pipelines, and coordinating multiple AI agents in complex workflows.

Summary

Generative AI holds significant transformative potential in medicine, enhancing capabilities across imaging, documentation, education, and decision support. However, its integration faces substantial challenges, including models’ knowledge limitations, the risk of generating incorrect or uncertain “hallucinated” outputs, inherent biases from training data, difficulty in interpreting model reasoning (“black box” nature), and navigating complex regulatory and ethical issues. This review offers a balanced perspective, acknowledging both the promise and the hurdles. While Generative AI is unlikely to fully replace physicians, understanding and leveraging these technologies will be crucial for medical professionals navigating the evolving healthcare landscape.

Introduction

When physicians hear the term “generative artificial intelligence (AI),” some might picture the interfaces of popular proprietary chatbots like ChatGPT [1], Claude [2], or Gemini [3] and view Generative AI as just another tech toy whose popularity will eventually fade. Others, more intrigued by the convergence of medicine and technology, may think of recent advances in AI and wonder how these innovations might reshape the field. Among this latter group, optimists might eagerly imagine AI as the solution to all challenges of their current practice or as a tool to push medicine beyond its current limitations. Conversely, some may echo the cautious stance of Geoffrey Hinton, one of the pioneers of modern AI and a recent Nobel Prize laureate in physics, feeling uneasy about the potential for AI to replace traditional roles in medicine [4]. However, there is also a group of physicians who aim to maintain a balanced perspective on AI—a view not swayed by the relentless advocacy for or against it in media and social channels but grounded in scientific evidence, thoughtful reasoning, and a continuous assessment of AI’s current and future role in medicine.

Unfortunately, building a balanced view of Generative AI is challenging. The field is advancing at an unprecedented pace, driven by substantial financial and intellectual investment from a range of stakeholders inside and outside of healthcare [5]. Keeping up with these rapid changes is no easy task for physicians. Medical communities are not accustomed to rapid shifts and overnight trends. This field is grounded in the methodical pace of multi-year clinical trials, lengthy peer-review processes, and gradually evolving practice guidelines, which may not update for months, if not years. This deliberate approach exists for an important reason: medicine—and healthcare overall—deals with human lives, and the last thing desired is to hastily adopt a poorly vetted diagnostic or therapeutic innovation that could endanger patients. This is in contrast to AI where the mantra is to ‘fail early and often’. Many of today’s AI advancements were barely on the radar of even the most expert engineers just one year ago. Not only is AI’s impact escalating daily, but the speed of its evolution surpasses that of any recent technological trend. Given this context, the conventional methods of responding to innovation in medicine will not suffice for navigating the swift and transformative rise of AI.

Despite the rapid advances in this field, much prior work was aimed at leveraging the intersection of Generative AI and medicine [6–9]. While valuable, many of these efforts overlook a critical need: to explain the core concepts, opportunities, and applications of Generative AI in a way that is practical for physicians without an engineering background. This review outlines the generative AI landscape for physicians interested in the technology, but who are not programmers or technical experts, by providing a foundational understanding of common generative AI terminology. We begin by defining generative AI, explaining how it differs from discriminative AI, and reviewing the building blocks of generative AI algorithms, along with the general concepts in their development and deployment. We then discuss the modality-based applications of generative AI in medicine, followed by several specific use cases. Finally, we thoroughly examine the current limitations of generative AI in medicine and review some of the proposed solutions. Table 1 contains a glossary of all technical terms introduced in this review.

Table 1.

A glossary of key terms introduced in this review

Term	Definition
Artificial Intelligence (AI)	Algorithms that mimic human intelligence, performing tasks such as image recognition, language understanding, and decision-making.
Machine Learning (ML)	A subset of AI involving algorithms that learn patterns from data to make predictions or decisions without being explicitly programmed.
Deep Learning (DL)	A subset of ML algorithms using neural networks to process complex data; the most common form of ML used in Generative AI.
Neural Networks	Computational models that capture complex patterns and relationships in input data by processing it through multiple layers of mathematical operations.
Training	The process of optimizing the parameters of mathematical operations in neural networks by exposing them to a dataset of input data paired with the corresponding expected outputs.
Inference	The process of applying a trained AI model to an unseen cohort of input data during test time or actual deployment.
Zero-Shot Learning	A method where AI models perform inference without explicit task-specific training (i.e., out-of-the-box), leveraging their general knowledge learned during pre-training.
Fine tuning	Adapting a pre-trained model to a specific task or dataset by further training a subset of its parameters on task-specific data. The goal is to maintain (part of) the model’s prior knowledge while giving it new capabilities.
Graphics Processing Unit (GPU)	Specialized hardware designed to accelerate the training and inference of deep learning models; they can be purchased for in-house systems or accessed through subscription-based cloud services such as Amazon Web Service or Google Cloud.
Generative AI	AI models that generate new content, such as text, images, or audio, by learning patterns from training data.
Discriminative AI	AI Models that classify or predict outcomes by distinguishing between categories, used in tasks like classification, regression, etc.
Convolutional Neural Networks (CNNs)	A class of deep learning models designed to process and analyze grid-like data structures, such as images, by applying convolutional operations to extract spatial hierarchies and features.
Transformers	A class of deep learning models based on self-attention mechanisms, enabling them to process and model sequential data efficiently by capturing long-range dependencies.
Self-attention	A key mechanism in transformer models that compute relationships between all elements of an input sequence (e.g., different words in a sentence or different patches of an input image), enabling the models to better capture dependencies and contextual relevance in the input data.
Embedding	A compressed representation of data in a low-dimensional vector space that preserves semantic or contextual relationships, facilitating efficient input to downstream neural networks.
Encoder	A neural network module that transforms raw input data into dense embeddings by capturing essential features, serving as a critical step in representation learning for downstream tasks.
Decoder	A neural network module that reconstructs meaningful outputs, such as classifications, sequences, or images, from input embeddings by leveraging learned representations of the data.
U-Net	A neural network architecture with an encoder-decoder structure that receives an image as input and produces another image as output; often used for tasks like image segmentation or generation.
Generative Adversarial Networks (GANs)	Models that generate synthetic imaging data through an adversarial competition between a generator (e.g., a U-Net) and a discriminator (a classifier that guides the generator to improve its outputs); historically popular for image generation tasks, but now less commonly used due to advancements in diffusion models.
Diffusion Models	Models that generate synthetic imaging by learning to reverse noise into meaningful images; similar to GANs, internally leverages U-Net models.
Conditional Generation	Modifying Generative AI models (e.g., a diffusion model) to create outputs based on input signals rather than random generation of synthetic data; for example, generating an image based on a textual description or modifying an image according to a given style.
Super-Resolution Models	Conditional Generative AI models that enhance the quality of an input image (i.e., the condition) by increasing its resolution; e.g., by converting a 64 × 64-pixel input image into a 1024 × 1024-pixel output.
Token	A fundamental unit of data, such as words, subwords, or characters, used as inputs for natural language processing models. Different LLMs can have different tokenization schemas.
Large Language Models (LLMs)	Generative Pretrained Transformers (GPTs) like ChatGPT and Claude models that predict all tokens in an input sequence, predicting the next token that is most likely to complete the sequence in a meaningful way.
Large Multimodal Models (LMMs)	Extensions of LLMs that accept multiple input data formats (e.g., text, images, audio) for integrated and enhanced functionality, such as understanding an input movie or captioned figure.
Prompt Engineering	Techniques for crafting user inputs (prompts) that can better guide the generation of LLMs or LMMs for achieving desired outputs.
Reinforcement Learning with Human Feedback (RLHF)	A fine-tuning method for aligning model outputs with human preferences and ethical standards, often applied to LLMs and LMMs before their public release.
Open-weight AI Models	Models with publicly accessible pretrained weights, enabling customization and secure deployment within institutional networks.
Proprietary AI Models	Commercial models that are often provided through web-based Application programming interfaces, are subscription-based, and offer state-of-the-art performance despite raising privacy concerns in healthcare.
Hallucinations	Erroneous or fabricated outputs generated by AI models, such as LLMs citing non-existent references or diffusion models introducing artifacts when converting a T1 magnetic resonance imaging sequence to a T2 sequence.
Retrieval-Augmented Generation (RAG)	A technique to ground LLMs or LMMs by integrating relevant information retrieved from external knowledge sources into their generative process.
Multi-Agent Setups	Systems where multiple AI models (e.g., LLMs or LMMs) collaborate on complex tasks, have the capacity to call each other, and one model’s output is an input to another; could be designed with or without humans in the loop.
Interpretability	The ability to understand and explain AI model decisions, essential for building trust in medical applications.
Saliency Maps	Visualization tools highlighting the most influential parts of an input (e.g., an image) that affect an AI model’s predictions, improving explainability of the model.
Bias	Systematic errors in AI outputs caused by suboptimal training data, inappropriate model and training choices, or inadequate model evaluation leading to reduced fairness or reliability in inference time.

Core Concepts of Generative AI

Generative AI Vs. Discriminative AI

AI encompasses computer algorithms that mimic aspects of human behavior, enabling machines to perform tasks that would typically require human intelligence, such as recognizing images, processing language, and making decisions (Fig. 1A). Within AI, machine learning (ML) represents a subset of algorithms that can “learn” from patterns in data, continually refining their ability to make predictions or decisions without needing to be explicitly programmed for each new task. For instance, an ML model trained on patient records might learn to identify specific patterns that suggest certain diagnoses. An even more specialized subset of ML, known as deep learning (DL), uses many layers (hence ‘deep’) of interconnected nodes or “neurons” (which to some extent simulate the human brain’s structure). These DL models may solve complex tasks like interpreting MRI scans or analyzing vast datasets of patient information—a topic that will be further explored in the next Sect. [10]. In this review, the word ‘model’ is generally used to refer to the layers and nodes that can perform some ML task.

Fig. 1.

(A) Artificial intelligence (AI) vs. machine learning vs. deep learning vs. Generative AI. (B) Generative AI (right) vs. discriminative AI (left). Radiographs are adopted from Radiopedia.org; case courtesy of Sajoscha A. Sorrentino (rID: 14979) and Henry Knipe (rID: 31525)

In ML and DL, models are often categorized as either discriminative or generative (Fig. 1B). Discriminative models are designed to classify or predict outcomes based on input data, helping to distinguish between categories or predict results. For example, classifiers, which are commonly used in diagnostic imaging, might classify a radiology image as showing a “normal” or “abnormal” condition. Regressors predict continuous values, such as estimating a patient’s age based on their images. Other discriminative models include segmentation models, which classify pixels in an image as being of one type or another, like outlining a tumor in a CT scan. Similarly, object detection models combine classification and localization, identifying not only the category but also the exact position of objects within an image—useful for detecting and identifying abnormalities in chest X-rays. These models work with existing data to identify patterns, distinguishing features that are already present.

In contrast, Generative AI models do more than classify; they generate synthetic data that closely resembles real data but are novel in comparison to the training data [11]. This unique capability enables generative models to produce human-like text, realistic images, and even synthetic audio. For instance, Generative Pre-trained Transformers (GPTs), or as they are more commonly called, large language models (LLMs), can generate coherent text for tasks such as clinical documentation or medical question-answering, which can aid medical professionals by streamlining their practice routine. Similarly, diffusion models are a modern family of generative models that can generate complex images and videos, which can simulate rare medical conditions or variations for clinical or educational purposes. Unlike discriminative models, generative models are valuable for creating new content and expanding data resources, enhancing both clinical and research capacities in healthcare.

Building Blocks of Generative AI Models

What Is a Neural Network??

A model (neural network) can be thought of as a complex, layered equation designed to process data and learn patterns (Fig. 2). In traditional equations, one typically knows the constants and coefficients beforehand and solves for unknown variables. However, in a neural network, this process is reversed: the algorithm starts with known data points, such as images, text, or numerical values, which serve as the variables in the equation, while the model must learn the best coefficients and operations by examining many examples of what the developers want the model to learn. This learning process, called the training or learning phase, is when the neural network adjusts its internal parameters—its coefficients and structure—to represent the data accurately. Through repeated adjustments and feedback, the network learns by minimizing the difference between its predictions and the actual outcomes in the training set. This process requires a large number of training examples that are to be learned from. Collecting such data involves a lot of effort to identify and confirm that the training set is correct. In the case of segmentation, for example, it requires that all the structures or abnormalities of interest have been contoured.

Fig. 2.

An example of a three-layer neural network used to predict the prognosis of patients with COVID-19, represented mathematically. The neural network takes three inputs: gender (X1, where 0 = male and 1 = female), age (X2, expressed in years and divided by 100), and admission-time white blood cell (WBC) count (X3, expressed as WBC count/10,000). The network computes a final output (YF) between 0 and 1, where values greater than 0.5 indicate a poor prognosis. Panel (A) shows the expanded mathematical representation of the equations for each layer, while panel (B) illustrates the same equations in a layer-wise format, including the flow of inputs through the network. Nonlinear activation functions (e.g., Sigmoid) are applied at each layer, with weights and biases determining the network’s behavior

Once trained, a neural network effectively functions as a complex, yet standard equation: new data points serve as variables, are processed through intricate layers of parameters and operations, and yield meaningful outputs from the network. For example, a neural network trained to identify animals from images could take pixel values of an input image as variables, process them through its internal layers, and output a single value between 0 and 1. This output will be closer to 0 if the image is a cat or to 1 if the image is a dog.

Neural networks have unique characteristics that make them powerful for tasks such as image recognition, language processing, and complex decision-making. One key feature is their inherent nonlinearity, derived from their interconnected chains of neurons and layers. Each neuron functions as a separate equation, and when these neurons connect, they form a larger equation known as a layer. Similarly, each layer acts as a linear transformation that receives inputs from the previous layer, calculates the output, applies a nonlinearity, and sends its output to the next layer. These interconnections and nonlinearities enable neural networks to capture complex relationships in data that simpler, linear ML models cannot. However, this complexity requires substantial data to effectively optimize each layer’s parameters across a wide range of inputs. The training process is computationally intensive, often requiring specialized hardware, such as graphics processing units (GPUs), to handle the vast number of calculations efficiently. Over the past decade, the availability of large datasets, rapid advancements in GPUs, and breakthroughs in DL algorithms have made neural networks not only feasible but also dominant tools in AI, achieving remarkable performance across diverse applications [12].

Convolutional Networks Vs. Transformer Networks

There are many ways to connect neurons and layers to build a neural network, but over time, researchers have found schemas that work better than random connections in building these networks. When schemas have a similar structure, they are referred to as having similar “architectures.” Two of the most influential architectures in today’s world of DL are convolutional neural networks (CNNs) and transformers (Fig. 3). CNNs are rooted in “convolutions,” which focus on analyzing localized patterns, while transformers rely on “self-attention” blocks that help the model understand how each part of the data relates to every other part.

Fig. 3.

Comparison of Convolutional Neural Networks (CNNs) and Transformer-based Neural Networks for image classification. In CNNs, convolutional operations focus on processing adjacent pixels in local neighborhoods independently, with limited attention to remote regions of the image. In contrast, transformers “patchify” images and employ an attention mechanism, allowing features from each patch to be enhanced by attending to features from all other patches (this mechanism is similar to how transformers process text data, where each token attends to all other tokens in a sequence). While this figure illustrates the classification function, both models can also be used for other discriminative or generative tasks. The final classification is performed by a simple classification head, such as a linear neural network, which processes features from either the CNN or the transformer

CNNs have been especially popular in image processing due to their ability to capture spatial patterns and hierarchical structures. Each layer in a CNN looks for specific features in the data, beginning with simple edges or textures and moving to more complex shapes in higher layers. CNNs leverage a grid-like pattern to process images, allowing them to detect even subtle changes in the fine texture of an image, making them invaluable in fields like radiology and dermatology. The convolutional operations within CNNs create filters that focus on small, localized portions of the image at a time, making CNNs efficient and computationally lighter than other architectures such as transformers [13].

Transformers, on the other hand, have revolutionized tasks that require understanding context and sequence, such as language processing. Their core feature, the attention mechanism, enables them to assess the importance of each part of the input data relative to others, and to learn the importance of their order or distance within the sequence [14]. This mechanism excels at identifying relationships and dependencies over a long range of data, allowing transformers to comprehend the ‘meaning’ of sentences or paragraphs by analyzing all parts of the text together rather than focusing on one segment at a time. This approach has enabled transformers to outperform CNNs in tasks that require contextual understanding, and it has opened the door to applications across a wide range of domains, including text generation, medical documentation, and more.

Despite these foundational differences, both CNNs and transformers have found applications outside their traditional domains. While CNNs are exceptional at detecting localized features and textures within images, transformers can also process images effectively, but they will focus more on geometrical patterns (e.g., distance between objects) and long-range dependencies within data. Similarly, CNNs have shown promise in processing sequential data, though their ability to capture long-term dependencies is limited compared to transformers. The distinct strengths of each architecture provide flexibility and complementarity, enabling DL practitioners to select models based on the specific characteristics of their data and the goals of their tasks [15, 16].

How Do Neural Networks Represent Data??

In order for neural networks to perform complex tasks, they should ideally be able to distinguish meaningful information from irrelevant content (noise) within the input data. This process of filtering out noise and focusing on important features is crucial, as the network needs to identify only the signals that are relevant to the specific task it is trained for. For instance, if a model is trained to detect liver tumors on an abdominal CT scan, structures such as the intestines or bones might be present in the scan but are less relevant to this task and are thus considered as noise. The model must learn to focus on the liver and ignore these unrelated regions. This ability to distinguish signal from noise allows the network to create representations, or internal summaries, of the data that highlight only the most relevant features.

These representations are typically represented as an embedding, where essential information is converted into a series of numerical values, typically a vector (or matrix if the input is higher than 1 dimension). Though these numbers may seem meaningless to a human observer, they capture the essence of that data element needed to perform the task. For example, in a brain MRI scan, a model trained to segment a tumor will transform the complex, high-dimensional scan data into a simplified representation that emphasizes only the features needed to locate the tumor. This embedding serves as a concise but highly informative version of the input data, making it easier for the model to complete its task efficiently [17, 18].

Different types of neural networks use various methods to create and process these representations. Some models employ internal, straightforward mechanisms, while more advanced architectures use separate networks to handle different aspects of representation. For example, in a U-Net—a model commonly used for medical image segmentation and generation—two specialized networks work together: an encoder and a decoder [19]. The encoder transforms the input scan into a compressed embedding by isolating the relevant features, while the decoder uses this embedding to reconstruct a new image or a segmentation map, where each area of interest is highlighted. This encoding-decoding structure enables the model to create a detailed, clinically useful output, such as a tumor outline within the original scan.

In language processing, transformers rely on embeddings to represent words in a way that reflects their meanings and relationships. Embedding models within transformers analyze large amounts of text data to understand word associations based on context. Each word or phrase is represented as an embedding—a list of values that reflects various characteristics. For example, embeddings for “car” and “motorcycle” will be relatively similar because they both relate to transportation, while the embedding for “ice cream” will be very different, capturing its association with food. Mathematically, the embeddings of “car” and “motorcycle” are closer to each other in this multidimensional space than they are to “ice cream,” helping the model understand context and meaning [20]. Furthermore, embeddings capture not only semantic meaning but also deeper relational and contextual nuances. A notable example is the analogy where the distance and direction between “man” and “woman” mirrors that of “king” and “queen.” This relationship demonstrates how embeddings can encode higher level structures and comprehend complex relationships in language.

Common Generative AI Models

In recent years, the landscape of Generative AI has evolved rapidly, with advances in architectures shifting the focus toward increasingly capable and versatile models. Although older generative models like Generative Adversarial Networks (GANs) and Long Short-Term Memory (LSTM) networks played pivotal roles in early AI development [21, 22], today’s Generative AI field is dominated by three primary families of models: diffusion models, LLMs and LMMs (Fig. 4). Each of these families excels in unique aspects of data generation and interpretation, outperforming earlier model architectures like GANs and LSTMs in both performance and efficiency. The following sections will explore these model types, their core functionalities, and their primary applications in medical and other fields.

Fig. 4.

An overview of diffusion models (A), large language models (B), and large multimodal models (C)

Diffusion Models

Diffusion models are a class of generative models primarily focused on creating visual data, such as realistic images or complex video sequences. Although they have also been explored in text generation and other domains, their most common use is in generating high-quality, diverse visual outputs. The underlying architecture of diffusion models often relies on U-Net, a type of neural network that specializes in capturing and reconstructing spatial details through a combination of encoding and decoding layers [23].

The primary process of a diffusion model involves learning to generate images from random noise. During training, these models are fed millions of images, which are degraded by adding noise until the images are nearly indistinguishable from random noise. The model then learns to reverse this process, transforming noise back into meaningful images step-by-step. This “denoising” capability allows the diffusion model to effectively reconstruct and generate new visual data from pure randomness, synthesizing images that appear authentic and varied. For instance, a diffusion model trained on nature images can generate an entirely new, realistic landscape from random noise, even if the specific scene it generates was never seen during training [24].

Diffusion models can be either unconditioned, generating images purely based on learned data patterns, or conditioned, where additional input directs the model’s output. For example, a diffusion model trained on seasonal images could be conditioned to generate scenes that depict winter or summer landscapes based on user specifications [25]. These models can even be conditioned on textual prompts, as seen in models like Stable Diffusion, where a user might describe an image’s style or content, and the model generates a visually aligned output [26]. Diffusion models can also perform image-to-image transformations, receiving one type of image and transforming it into another, such as converting a T1-weighted MRI sequence into a T2-weighted MRI sequence in medical imaging. Though diffusion models are large and require significant computational resources, they are relatively less data-intensive than transformer-based models, thanks to their convolutional network foundation rather than a transformer-heavy architecture.

Large Language Models (LLMs)

LLMs such as ChatGPT, Gemini, and Claude are models that specialize in understanding and generating human-like textual outputs. They are typically trained on extensive datasets sourced from the internet (e.g., websites, online books, or articles) and are designed to predict the next word (token) in a sequence based on preceding context [27]. This predictive capability, often termed ‘next-token prediction,’ is the fundamental mechanism through which LLMs generate coherent and contextually appropriate text. For simplicity, one can think of each token as a word or meaningful sub-word, though the precise definition of a token is more nuanced [28].

These models operate on transformer architectures, which was previously discussed as excelling at capturing long-range dependencies and contextual nuances in language. Transformers enable LLMs to consider each part of the text in relation to every other preceding part, making them effective at understanding complex sentence structures and capturing the context of medical conversations. Due to their transformer-based design, LLMs require vast datasets and significant computational power for training, making their development resource-intensive. However, once trained, these models can perform a range of tasks beyond text completion. Techniques applied post-training allow LLMs to adapt to functions like question answering, summarizing patient notes, or generating detailed explanations—tasks that have direct applications in medical documentation and patient communication [14].

Large Multimodal Models (LMMs)

LMMs represent an extension of LLMs; unlike LLMs, which only handle language, LMMs are designed to interpret diverse data types, such as images, text, and audio. While their output is often text, their ability to process multiple input types allows LMMs to analyze and summarize complex information, such as a combination of visual and textual patient records. For instance, an LMM might analyze both a chest radiograph and accompanying medical notes to provide a more integrated and accurate output. The training of LMMs also mirrors that of LLMs, relying on large, diverse datasets and transformer architecture to capture dependencies and relationships across modalities. However, the process of collecting and curating data for developing LMMs is naturally more resource-intensive and complicated than LLMs [29].

Although LMMs, LLMs, and diffusion models are often discussed as separate categories, the boundaries between them are becoming increasingly fluid. For example, some LMMs incorporate diffusion model components to enhance visual generation capabilities, while text-to-image diffusion models use transformer architectures to encode textual prompts, conditioning their output on those prompts. Additionally, different terms sometimes refer to similar concepts within Generative AI. For instance, ‘vision-language model’ (VLM) is often used to describe a model that handles only text and image data, differentiating it from the broader category of LMMs.

From Development To Deployment of Generative AI Models

Understanding how Generative AI models are developed and deployed provides us with critical context for assessing their real-world applications in medicine (Table 2). While training these models involves substantial resources and technical expertise, the deployment phase presents additional considerations around accessibility, privacy, and customization. Lastly, the choice between proprietary and open-weight models plays a pivotal role in determining which approach aligns best with clinical needs, ethical requirements, and institutional resources.

Table 2.

Comparative overview of key considerations in the development and deployment of generative AI models for real-world applications in medicine. Abbreviations: GPU: graphical processing unit, CPU: central processing unit

Considerations	Development	Deployment
Phases	● Pre-training: Data collection and curation ● Training: Resource-intensive process requiring specialized hardware Post-training: Alignment with policies or adaptation for specific applications	● Out-of-the-box: Using pre-trained models as-is ● Fine-tuning: Training a subset of model parameters for specialised tasks
Data Requirements	● Requires large amounts of unlabeled data (model learns next-token prediction) ● Often collected from the internet and extensively curated ● Licensing and copyright issues may limit data access	● Out-of-the-Box: No training data needed ● Fine-Tuning: Requires a smaller, labeled dataset often curated in-house
Computational Requirements	● Large GPU clusters (hundreds to thousands of GPUs)	● In-house hosting: Small number of GPUs (sometimes one GPU or even a powerful CPU is sufficient) ● Cloud hosting: No in-house hardware needed
Time Requirements	● Weeks to months for training	● Out-of-the-box: Immediate use ● Fine-tuning: Several days to weeks
Key Technical Challenges	● Collection and curation of massive datasets for training ● Maintaining model stability during extended training phases	● Out-of-the-box: Limited model knowledge for specialized fields ● Fine-tuning: Risk of ‘catastrophic forgetting’
Feasibility	● Mostly limited to the tech industry or academia due to resource demands and expertise	● Out-of-the-Box: High accessibility via in-house deployment or cloud platforms; requires basic knowledge of AI and prompt engineering ● Fine-Tuning: Requires moderate resources and technical expertise
Environmental Impact	● Significant energy consumption due to large GPU clusters running for extended periods	● Out-of-the-Box: Negligible impact ● Fine-Tuning: Minimal to moderate energy consumption, depending on the scale

Development

Regardless of architecture, most modern Generative AI models undergo three main phases: pre-training, training, and post-training.

Pre-Training focuses on gathering and preparing large amounts of data to provide the model with a broad understanding of its target domains. For LLMs and LMMs, pre-training often involves collecting publicly available data from internet sources and filtering out irrelevant or low-quality content to ensure alignment with the model’s intended applications. For example, building a medical LLM may require identifying reputable sources and removing counter-productive non-medical data, thereby enhancing relevance and applicability in healthcare settings. Addressing copyright and ethical issues is also crucial, ensuring only licensed, permitted data is included.

After data collection, the raw data should be cleaned and standardized. For instance, duplicated content might appear across different websites, so additional copies must be removed from the dataset. Additionally, text data should be broken down into smaller units, or tokens, allowing efficient processing of language patterns. This human data cleaning requires substantial resources; hence, Generative AI companies often employ data engineers and sometimes leverage pre-trained LLMs to accelerate this phase [30].

Once the training datasets are ready, the training phase begins, a compute-intensive process that typically requires specialized hardware, such as large clusters of GPUs. Models are trained iteratively over weeks to months, with many engineers monitoring the process to identify and correct errors and address other unexpected issues. Training large generative models incurs significant time and electrical energy (which entails environmental costs). For instance, training a single large model can consume as much electricity as small cities, which has led to discussions on AI’s environmental impact [31]. A general model that has been trained on a large corpus of data is often called a ‘foundation’ model as it serves as the foundation for building applications.

In the post-training phase, the model is prepared for specific applications and adapted to meet ethical and social expectations. For example, while an LLM may initially learn to predict the next word in a sequence, additional steps are needed to enable it for chat-based use cases where it can respond to user questions. Such specific applications are often taught through instruction-tuning, a less extensive training phase focused on changing the model’s style (e.g., from sentence completion to question-answering) rather than teaching new knowledge [32]. Models also require alignment with social norms before public release (e.g., replying professionally and avoiding biased, unethical or dangerous responses). Further fine-tuning using techniques like reinforcement learning with human feedback (RLHF) may be used to guide the model towards desirable content and style [33].

Deployment

While the development of Generative AI models is largely restricted to tech industry or academia due to complexity and resources, anyone can access and use these models for their purposes. Therefore, understanding the different options available for deploying these models for personal or professional use is essential.

It is straight-forward to use a foundation model without modification (additional training). Many foundation models are accessible via cloud-based platforms, reducing the need for institutions to maintain extensive hardware resources. Users interact with these models by entering prompts and receiving outputs tailored to their needs. Since the advent of ChatGPT, many resources have focused on enhancing users’ abilities to get better results by teaching prompt engineering—a skill for crafting more effective prompts to achieve desired outputs [34]. However, a critical assumption when using models out-of-the-box is that the model already possesses the necessary knowledge to answer specific queries, an assumption that doesn’t always hold, especially in specialized fields like medicine.

Alternatively, fine-tuning offers a middle ground between full-scale training of a foundation model and out-of-the-box use. In this process, users train a subset of the foundation model’s parameters on a targeted dataset to improve performance for specific applications. For instance, fine-tuning a general language model on radiology reports can make it more effective in summarizing findings or generating structured impressions in a radiological context [35].

While fine-tuning aligns a model with specialized tasks, it requires additional computational resources and a curated dataset, albeit not as intensely as initial training. It also can mostly be performed on ‘open-weight’ foundation models (like open-source software, but rather than downloading the source code, one downloads the weights and architecture of the foundation model). Users must therefore account for the cost of data collection and processing for fine-tuning, as well as the computational demand of additional training. Research shows that fine-tuned models can sometimes experience “catastrophic forgetting,” where they lose accuracy on previously learned tasks or develop tendencies to mimic the style of fine-tuning data rather than adding new knowledge [36].

A.4.3. Proprietary vs. Open-weight Generative AI Models.

The choice between proprietary and open-weight Generative AI models adds another layer of complexity to deployment planning. This decision impacts data control, cost, and customization—all key considerations in healthcare applications (Table 3).

Table 3.

Comparative overview of open-weight and proprietary models

Aspect	Open-weight Models	Proprietary Models
Data Security	● Can be operated within a secure institutional network ● High compliance with internal privacy regulations	● Data shared externally ● Potential privacy risks ● Limiting applicability for tasks involving protected health information
Control	● Full control over data and model architecture (if the code is also available) ● Adaptable to institutional needs	● Limited control over training data and model architecture ● Limited model customization options during inference
Performance	● Slightly inferior performance than proprietary models, but improving rapidly due to the growth of AI community ● Might require technical expertise for inference, monitoring, and updates	● State-of-the-art performance (lead most benchmarks) ● Ease of use in inference ● Regular updates, security patches, fine-tuning options for specific tasks
Cost	● High initial costs (hardware, expertise) ● No direct costs or subscription fees after the initial setup ● Cost-effective for long-term use and larger institutions	● Lower upfront costs with minimal setup required ● Subscription-based cost after the initial setup ● Cost-effective for short-term or smaller institutions
Maintenance	● Institution responsible	● Provider responsible
Scalability	● Hardware-dependant	● Scales via provider infrastructure

Proprietary models typically do not allow access to the actual weights in the model and are offered via subscription-based access to cloud services, such as OpenAI’s ChatGPT or Google’s Gemini. These models are maintained by companies that provide regular updates, security patches, and optionally can provide fine-tuning options for specific tasks. Although proprietary models offer state-of-the-art performance and ease of use, data privacy is a notable concern. For healthcare institutions, sending sensitive patient information to an external service may conflict with regulations like Health Insurance Portability and Accountability Act (HIPAA), limiting applicability for tasks involving protected health information. Some AI companies may offer contracts to deploy containerized models within institutional firewalls, but getting approval of these contracts requires considerable resources, often unavailable to smaller institutions.

On the other hand, proprietary models relieve institutions of the technical demands of model development and operation, as the provider manages computational infrastructure. Subscription fees may be lower in the short term than investing in GPU servers, particularly if the need for these is sporadic, making proprietary models financially attractive for many use cases.

Alternatively, open-weight models, such as Stable Diffusion from Stability AI, LLaMa from Meta or LLaVa models from Microsoft, provide complete control over data and enable compliance with internal privacy standards since they can be operated within a secure institutional network [26, 37, 38]. For healthcare, open-weight models are appealing for tasks involving sensitive data as they facilitate compliance with stringent privacy regulations while providing advanced AI capabilities. They may also be more customizable towards an institution’s primary goals and needs, whereas previously mentioned, proprietary models are not as customized intentionally towards a single intended use case.

However, using open-weight models requires substantial hardware investments, especially for fine-tuning. Although some cloud platforms, such as Amazon Web Services, offer GPU resources for hire, costs can escalate quickly, making long-term, large-scale use expensive. Additionally, hosting these models internally demands more technical expertise, as ongoing maintenance, updates, and troubleshooting become the institution’s responsibility rather than that of an external provider (leveraging proprietary models also requires some degree of local domain expertise to collaborate with external proprietary model providers). Furthermore, the performance of open-weight models are often slightly inferior to proprietary alternatives, although the gap is closing rapidly [39].

Another drawback of open-weight models is their performance. At the time of writing this manuscript, proprietary models lead most available benchmarks, outperforming their open-weight counterparts. However, the landscape is changing rapidly, and it is speculated that this gap will narrow and eventually close. Furthermore, there is no evidence to suggest that either approach provides value in terms of improving operations or clinical outcomes, when considering the cost of proprietary collaboration or internal open-weight model development and deployment.

In summary, the choice between proprietary and open-weight Generative AI models involves balancing privacy, control, performance, and cost. For some medical applications, proprietary models provide accessible and powerful tools but may raise data security concerns. Open-weight models, while often requiring higher initial investments, offer enhanced control and compliance benefits crucial for privacy-sensitive applications in healthcare.

Applications of Generative AI in Medicine

The applications of Generative AI models in medicine are extensive and continue to expand, with each advancement opening new possibilities in medical imaging. Here, we categorize these applications into two arbitrary groups: general applications, which directly utilize the synthetic (i.e. words or pixels put together in new and novel ways) output generated by a single Generative AI model, and special applications, which may not produce synthetic data but still heavily rely on Generative AI models or benefit from more complex configurations involving multiple models. We will then highlight a few examples from the medical literature for each of these categories (Fig. 5).

Fig. 5.

An overview of general and special applications of generative artificial intelligence (AI) models in medicine

General Applications

Image Generation and Enhancement

Generative AI has shown transformative potential in medical imaging by addressing various challenges in data generation and quality enhancement. For instance, the creation of synthetic medical images using diffusion models can help overcome limitations associated with small or biased datasets by generating realistic images to augment training data [40]. Diffusion models offer various ways to expand real data pools; for example, CXR-IRGen is a diffusion model capable of generating synthetic but realistic chest radiographs based on human prompts [41]. The Multitask Brain Tumor Inpainting model is another conditional diffusion model that can add tumoral lesions to tumor-free brain MRI scans or, conversely, remove tumors from input scans [42]. Moreover, Generative AI models have the potential to highlight and address biases in medical datasets by transforming one radiograph into another, preserving all original features while altering attributes like the synthetic patient’s race [43].

Beyond expanding training datasets for other AI models, image generation models have other clinically relevant applications. For example, super-resolution models leverage diffusion-based approaches to produce higher-resolution versions of CT, MR, ultrasound, or histopathologic images, addressing issues such as low resolution due to equipment constraints [44–46]. This technology could also facilitate faster radiology studies by capturing lower-resolution images and subsequently converting them back to high resolution using AI. Generative AI models can also convert one type of radiologic image to another, such as CT images to MRIs [47]. Additionally, diffusion models have shown promise for rotating radiographs in 3D space, enabling the visualization of a single radiograph from multiple angles without requiring additional images [48]. Finally, with some modifications, image generation models can be applied to discriminative tasks, as demonstrated in studies where diffusion models performed well in segmentation and anomaly detection tasks [49, 50].

Text Generation and Clinical Documentation

Generative AI, particularly LLMs, is increasingly transforming how medical text is created and utilized across various applications. In clinical practice, systems powered by LLMs are being developed to handle clinical note generation from patient-provider conversations, a process that has demonstrated potential in improving both documentation accuracy and clinician efficiency [51, 52]. As another instance, specialized models like HEAL and Mistral-7B have shown effectiveness in generating detailed and contextually appropriate medical notes, such as discharge summaries, which contribute to more streamlined handoffs and continuity of care [52, 53]. Other innovations include automated generation of radiology reports where LLMs can structure reports by analyzing imaging findings, emphasizing critical abnormalities and providing relevant clinical interpretations [54, 55]. These applications suggest that generative AI can reduce clinicians’ cognitive load, minimize errors, and standardize documentation quality. However, evidence also indicates that incorporating LLMs into clinical workflows may increase clinicians’ workload, or that collaboration between LLMs and clinicians may result in worse performance than when each acts independently [56–59].

LLMs are also being adopted to support patient communications, where they are used to translate complex medical information into accessible language, enabling patients to better understand their health conditions and test results [60]. This direct communication enhances patient engagement and autonomy, particularly when discussing diagnosis and treatment options, and fosters a more collaborative relationship between patients and healthcare providers [61]. In medical education, LLMs offer a unique solution to streamline learning for students, enabling personalized study resources and assisting with the preparation for exams. For instance, LLMs can generate exam-style questions and answer explanations that align with real-world medical scenarios, helping students practice critical thinking and clinical decision-making skills in preparation for licensing exams [62, 63]. Additionally, these models can serve as virtual tutors, capable of responding to complex questions about medical content in a way that complements traditional educational resources. This support alleviates some pressures on clinical educators and provides students with flexible, on-demand learning options [62].

Audio and Video Generation

Although generating audio and video in medicine is not as explored as generating text and imaging data, these models hold significant value across various applications. Audio and video generation models in medicine are enabling new ways to enhance clinical training, patient communication, and medical education [64]. Similar to text and image generation models, these AI tools can generate realistic surgical videos, clinical sounds, or transcriptions of medical notes to expand the datasets used for downstream DL models, improving performance in multimodal tasks [65–69]. They can also create high-fidelity synthetic surgical or procedural videos, as well as simulations of heart, lung, or procedural sounds for medical training, exposing students to rare clinical scenarios [70, 71]. These models also enrich educational content by generating interactive patient avatars, allowing learners to practice handling difficult conversations, such as end-of-life discussions, across diverse cultural and emotional contexts [72, 73]. Additionally, they have the potential to support live translation in healthcare, breaking down language barriers in real time and facilitating more empathetic, culturally sensitive patient interactions [74].

Another application that makes video generation models particularly interesting is their potential to advance surgical and interventional practices. For instance, synthetic video models can enhance preoperative planning and assist surgeons in real time by providing tailored visual guides for instrument placement and specific procedural steps [67, 75]. They also open a new avenue for patient education by generating accessible, explanatory videos from text, which improves patient understanding of complex medical procedures [76]. This ability to generate precise, task-specific video content has promising implications for both personalized surgical planning and improved patient care [75, 77].

Special Applications

Diagnostic, Prognostic, and Treatment Planning Aids

While discriminative AI was priorly distinguished from Generative AI and while clinical diagnosis is mostly a discriminative task, Generative AI models may still be valuable as backbones in complex data-processing pipelines to facilitate accurate and efficient patient assessments. These applications leverage Generative AI’s capacity to integrate diverse data sources—such as imaging, text, and health records—into LMMs, allowing for enriched, context-aware decision support. For instance, recent work with multimodal models trained to interpret electrocardiograms (ECGs) illustrates how an instruction-tuned LMM significantly improves accuracy in identifying a range of cardiac abnormalities from ECG images, resulting in an average 15–30% accuracy improvement over alternative discriminative models when limited ECG data is available [78]. Similarly, LMMs can offer detailed interpretations of complex imaging data like CT scans to understand nuanced anatomical and pathological features, yielding superior performance in orthopedic diagnoses and supporting clinician decision-making in real time [79].

In neurology, combining radiology reports with brain MRI data has shown promise in enhancing the interpretability and accuracy of lesion detection models. The ReportGuidedNet model, for example, integrates textual information from radiology reports to refine attention on MRI features, enabling it to distinguish among numerous brain conditions with high accuracy. ReportGuidedNet notably outperforms comparable models by producing more accurate attention maps and performing consistently across diverse datasets, indicating that such multimodal models can generalize effectively and provide clinicians with interpretable visual aids for identifying and localizing brain abnormalities [80]. Moreover, models that predict patient trajectories, such as the ETHOS transformer, demonstrate the utility of generative architectures for forecasting health outcomes. ETHOS employs a zero-shot learning approach to assess potential future events in patient care, drawing on large volumes of patient health data to simulate treatment paths and address biases, enabling a more individualized and adaptive approach to care planning [81].

Retrieval-Augmented Generation (RAG)

LMMs are impressive in their ability to understand and generate human-like responses across a wide range of queries. However, these models face significant limitations in healthcare settings where accuracy, up-to-date knowledge, and a large “context window” (the amount of information they can process at once) are essential. When the input data exceeds the context window, the model must truncate or compress the content, often leading to oversimplified or inaccurate responses. Additionally, even LLMs with larger context windows may see degraded performance when too much data is included, leading to inefficient or irrelevant answers. Another critical issue with LLMs is their reliance on static training data, which means they are not equipped to incorporate newer information beyond their training cutoff. This can result in “hallucinations,” where the model generates plausible-sounding but incorrect or outdated information. In healthcare, where guidelines and treatment options frequently evolve, this knowledge cutoff poses a challenge to model reliability and clinical utility [82].

Retrieval-Augmented Generation (RAG) is an approach that addresses these issues by allowing LLMs to dynamically access relevant external information as needed, Fig. 6 [8]. Instead of depending solely on its training data, a RAG pipeline begins by encoding both a large corpus of documents and the user’s query into mathematical representations known as embeddings. These embeddings, which capture semantic meaning, allow the system to perform mathematical operations to identify and retrieve the documents most relevant to the user’s query. Once the relevant documents are identified, they are provided to the LLM as context, enabling the model to generate responses that are not only more accurate and contextually appropriate but also capable of incorporating recent or rarely used information. This retrieval step ensures that the LLM’s answers reflect up-to-date knowledge and align closely with the user’s specific question.

Fig. 6.

A conceptual overview of the Retrieval-Augmented Generation (RAG) process, illustrating its two main stages: database creation (left) and knowledge retrieval (right). In the Database Creation stage, raw documents (e.g., research papers or other textual content) are divided into smaller segments using a text-splitting mechanism to create manageable chunks. These segments are processed by a language model to generate dense vector embeddings that numerically represent the content. These embeddings are then stored in a vector database for efficient retrieval. In the Knowledge Retrieval stage, a user’s question is converted into a query embedding by the same language model. The vector database is queried to calculate the cosine similarity between the query embedding and stored document embeddings, ranking the documents based on relevance. The top-ranked documents are retrieved and used to augment the user’s question by adding contextual information. This enriched query is then submitted to the language model, which generates a detailed and accurate response by leveraging both the context and the user’s original question

RAG models provide significant advancements in overcoming the limitations of traditional large language models (LLMs), where accuracy and timeliness are critical [83]. For instance, Almanac, a RAG-augmented language model designed for clinical medicine, retrieves and synthesizes information from curated medical sources, enabling it to provide factual and guideline-compliant recommendations across diverse specialties, underscoring RAG’s potential to increase model trustworthiness in critical care scenarios [84]. Self-BioRAG is another RAG-based system that demonstrated enhanced performance in generating contextually relevant, accurate, and uptodate biomedical explanations when producing long-form answers to medical questions [85]. Similarly, RAG has shown practical utility in supporting clinical decision-making by integrating the latest guidelines into diagnostic reasoning, exemplified by a custom-built ChatGPT model that produces guideline-compliant recommendations for nephrology cases [86].

Multi-Agent Setups

A multi-agent setup in AI refers to a system where multiple “agents” work together to achieve a common goal, with each agent performing a specialized role within the workflow [87]. In this context, an “agent” is typically a distinct AI model or module that is trained to handle a specific type of task or function, such as diagnosing symptoms, retrieving relevant information, or even coordinating with other agents to refine the final output. Unlike traditional systems that rely on a single LLM or LMM to address complex tasks, multi-agent frameworks can distribute different aspects of a task among specialized agents, and sometimes a mixture of AI agents and humans, allowing for more dynamic and efficient processing (Fig. 7). This modular approach reduces the cognitive load on each individual model, thus enabling AI systems to handle more complex workflows.

Fig. 7.

An example of a multi-agent framework for radiology report generation. The framework illustrates a collaborative process where specialized agents perform distinct roles: Agent 1 (Planner) strategizes based on input data, Agent 2 (Draft Writer) creates a preliminary draft, and Agent 3 (Editor) finalizes the report. Agents utilize external tools, web searches, and internal communication to iteratively refine the output, resulting in an accurate and coherent radiology report

Multi-agent frameworks are a recent trend in healthcare that have already shown potential in supporting complex clinical applications. For example, a hypothetical multi-agent system designed for sepsis management could include agents dedicated to specific tasks like monitoring patient vital signs, assessing diagnostic indicators, and recommending treatments. In such a setup, one agent might continuously analyze lab results, while another monitors physiological data to detect early warning signs. This distributed approach ensures that critical steps in sepsis management are coordinated seamlessly, improving both response time and patient outcomes in high-stakes situations [88]. Similarly, ClinicalAgent, a multi-agent framework for clinical trials, exemplifies how multi-agent systems can streamline eligibility screening and data analysis, enabling clinical trial coordinators to quickly and accurately identify suitable participants based on a range of up-to-date medical information [89].

Multi-agent frameworks are also becoming valuable tools in medical education and research. In educational applications, MEDCO (Medical Education Copilots) leverages multiple agents that embody roles such as “expert physician,” “radiologist,” and “virtual patient” to simulate real-world medical scenarios. This framework provides students with an interactive learning environment that encourages interdisciplinary collaboration and hands-on problem-solving [90]. Research applications also benefit from multi-agent setups. For instance, AI Hospital uses a multi-agent framework to simulate patient-provider interactions, allowing researchers to assess diagnostic approaches under realistic conditions. By incorporating agents that play roles such as patient, physician, and diagnostic reviewer, AI Hospital enables iterative discussions that help to refine diagnostic conclusions, mitigating biases and improving the reliability of clinical decision-making [91].

Current Limitations of Generative AI

Despite the impressive achievements of Generative AI in medicine and other fields, it is crucial to acknowledge the key limitations and ongoing challenges that impact its responsible integration into healthcare (Table 4). In this section, Five of these challenges will be reviewed, followed by strategies that can, so far, be used to mitigate them.

Table 4.

Overview of key challenges in applying generative AI in healthcare and potential strategies to mitigate them. Abbreviations: RAG: retrieval augmented generation; FDA: food and drug administration

Challenge	Potential Solution / Mitigation Strategies
Limitation in Knowledge	● Use RAG to integrate real-time data queries ● Enable real-time internet search capabilities ● Fine-tune models using recent, curated datasets
Uncertainties and Hallucinations	● Employ RAG pipelines to ground responses in verified data ● Develop models with uncertainty quantification capabilities ● Provide confidence levels or reliability scores for outputs
Bias and Fairness	● Use inclusive training datasets representing diverse demographics ● Generate synthetic data to address underrepresentation in datasets ● Create and apply diverse benchmarks for evaluation
Model Interpretability	● Develop visualization tools, such as heatmaps and saliency maps, for image-based models ● Explore methods to elucidate internal concepts in text-generative models
Regulatory and Ethical Issues	● Adapt risk-based regulatory frameworks (e.g., the FDA’s Total Product Life Cycle approach) ● Continuously monitor models in real-world settings ● Enhance regulatory resources

Limitation in Knowledge

Generative AI models like LLMs and LMMs rely on training data that is static—limited to information available at the time of training. This “knowledge cutoff” means that these models lack awareness of recent medical advancements, evolving clinical guidelines, and newly identified conditions or treatments introduced after the model’s training. In a clinical context, this limitation poses critical risks, as models might provide guidance based on outdated or incomplete information. Additionally, LLMs and LMMs are built from large but far from comprehensive datasets. Many critical and specialized medical resources may be absent from the training data, which is particularly concerning for rare diseases or cutting-edge research areas. Without access to the latest knowledge, these models may produce responses that are speculative or incomplete when asked about recent developments or niche medical fields [92].

Despite these limitations, efforts are underway to bridge this knowledge gap [93]. One approach that was introduced before is RAG; a method where the model can query external databases for the latest information and incorporate it into its contextual knowledge for generating responses. Furthermore, some LLMs like ChatGPT and LLM-based tools such as Perplexity now have real-time internet search capabilities, enabling them to fetch up-to-date information from the web when generating answers. Lastly, medical institutions can fine-tune models using recent data sources to extend their knowledge. However, as mentioned earlier, fine-tuning has its drawbacks, including the potential for introducing biases, catastrophic forgetting, or learning the style of the new training data rather than the knowledge within it [94].

Uncertainties and Hallucinations

Another major challenge with Generative AI models is their propensity to produce “hallucinations”—confident yet incorrect or fabricated responses. These errors often stem from the models’ goal to generate plausible language patterns rather than verified information [92]. Since LLMs are highly adept at mimicking human language, they may present incorrect information in a way that appears highly credible. Moreover, Generative AI models rarely acknowledge uncertainty or gaps in their knowledge, preferring instead to produce a confident answer even when they lack sufficient information. This issue is especially concerning in medicine, where AI-generated outputs that sound authoritative can mislead healthcare professionals and impact clinical decisions.

Mitigating hallucinations and uncertainties involves multiple strategies. RAG pipelines, for instance, help reduce hallucinations by grounding model responses in verified data. However, even if a model has access to sufficient knowledge to generate a correct output, there is no guarantee it will do so. Therefore, RAG pipelines must also incorporate methods to estimate and communicate uncertainty [95]. Uncertainty quantification is an active area of DL research that explores techniques for assigning confidence levels to AI outputs, which could assist users in assessing the reliability of each response [96]. While a detailed discussion of mathematical techniques for quantifying uncertainty in Generative AI models is beyond the scope of this review, healthcare users of AI should be aware of the potential for indicating the certainty of model outputs and actively request their engineering partners to equip proposed models with such capabilities [97].

Bias and Fairness

Generative AI models inherent biases from several sources: the training data, model design choices, and the evaluation methods used prior to public release [98–100]. Regardless of the origin, a biased AI model can produce skewed outputs that inadvertently perpetuate or even amplify existing biases in healthcare. Since Generative AI models are typically (pre)trained on large datasets—often sourced from the internet or specific institutions—any demographic or cultural biases present in these data can influence the model’s behavior. For instance, a generative model trained predominantly on data from one region or demographic group may generate outputs that are less relevant or even misleading for patients from different backgrounds. Such bias not only reduces the model’s utility across diverse populations but also risks perpetuating health inequities [101].

Addressing bias requires both proactive and reactive strategies. Proactively, training datasets must be more inclusive, representing a broad spectrum of demographics, medical conditions, and treatment approaches. Notably, some researchers are leveraging Generative AI itself to address biases in training datasets—e.g., by generating synthetic data points from underrepresented populations to enhance diversity and balance within training data [43]. There also exists techniques to identify and measure biases within already trained models. A prominent trend in research is to create datasets specifically for evaluating Generative AI models across diverse tasks and benchmarks. However, as these benchmarks are often publicly accessible, newer models could inadvertently be exposed to them during training [102]. Thus, caution is warranted when interpreting model performances against online benchmarks. An unexpectedly high performance on a benchmark could sometimes indicate that the benchmark data leaked into the model’s training set [98].

Model Interpretability

Generative AI models, and neural networks in general, are often considered “black boxes” due to their complex structures and the difficulty in tracing how they arrive at specific outputs. In healthcare, this lack of interpretability poses a challenge, as understanding the rationale behind AI-generated insights is essential for clinical trust and accountability. Physicians and other healthcare professionals need assurance that AI tools base their recommendations on medically sound reasoning. Yet, current Generative AI models frequently lack mechanisms to explain their outputs in terms that are easily understood by humans.

To address this, researchers are actively exploring ways to make AI models more interpretable. For image-based discriminative models, there is a well-established body of research on visualization tools, such as heatmaps and saliency maps, that highlight areas of focus—indicating which parts of an image influence a model’s prediction (e.g., revealing whether a classifier is focused on a tumor within a scan when identifying cancer or is mistakenly focusing on irrelevant parts of the image) [103]. However, such techniques are less frequently applied to image or video generation models, where the output itself is visual. Furthermore, existing research has also pointed out limitations and biases in these vision-based interpretability techniques [104].

Understanding the inner workings of text generative models, such as LLMs and LMMs, is even more challenging. Some recent research has proposed promising methods to elucidate the internal concepts that an LLM uses to generate responses [105, 106], yet there is a long road ahead to comprehensively interpret the driving forces behind generations in LLMs, LMMs, or diffusion models.

Regulatory and Ethical Issues

The rapid integration of Generative AI in healthcare introduces complex regulatory and ethical challenges. Due to the inherent black-box nature of models like LLMs, assessing and ensuring their reliability in clinical settings becomes problematic, particularly as these models may make autonomous, data-driven inferences without transparent reasoning. This opacity limits clinicians’ ability to interpret AI-generated outputs and verify their accuracy, raising concerns about accountability when outcomes are suboptimal or adverse. Ethical issues also arise from the risk of AI models introducing or perpetuating biases due to unbalanced training datasets, which could lead to inequities in patient care across different demographic groups. Furthermore, the evolving nature of AI systems means they require ongoing monitoring to maintain efficacy and safety, a process that can be challenging to standardize across diverse healthcare environments.

To address these challenges, regulatory bodies like the Food and Drug Administration (FDA) have attempted to evolve their frameworks and methodologies to keep pace with AI advancements. As an example, the FDA has recently introduced a risk-based regulatory approach to assess AI-enabled medical devices, tailoring requirements based on the AI’s intended use and associated risk to patient safety. Key components of this adaptive framework include a “total product life cycle” approach that monitors an AI system from pre-market approval through to post-market surveillance, which involves continuous evaluation of performance in real-world settings [107]. Regardless, bodies like the FDA face substantial challenges in regulating Generative AI effectively. Rapid, autonomous learning, continual updates, and potential unintended effects of Generative AI tools require new approaches for long-term monitoring, accountability, and risk mitigation. In such a rapidly changing environment, the FDA’s own resources must also evolve to manage the increased volume of applications and novel AI models submitted for review [108].

Conclusions

This review attempted to delineate the landscape of generative AI in medicine by systematically addressing its foundational concepts, practical applications, and inherent limitations. In Section A, we defined generative AI and detailed its core architectures—including convolutional neural networks and transformers—emphasizing their ability to generate synthetic outputs by learning from large datasets, in contrast to discriminative models which primarily classify or predict from existing data.

In Section B, we examined the diverse applications of these models in healthcare, such as the generation of synthetic medical images, automated clinical documentation through large language models, and the use of retrieval-augmented generation to enhance diagnostic support and patient communication. These examples illustrate how generative AI is being integrated to improve clinical workflows, education, and decision support.

In Section C, we highlighted the current challenges facing generative AI in medicine. These include the limitations imposed by static training data (leading to knowledge gaps), the risks of hallucinated outputs and inherent biases, issues with model interpretability, and evolving regulatory and ethical concerns.

Collectively, our review supports a data-driven approach for integrating generative AI into clinical practice. Balancing the benefits of Generative AI with its limitations demands a multifaceted approach, encompassing regulatory oversight, technical improvements, and a shift toward transparency in AI-driven decisions. While these actions often fall within the purview of technical stakeholders and healthcare leadership, we conclude this report by emphasizing the responsibility of individual practitioners facing this new technology.

Author Contributions

Po.R, B.K., and S.F drafted the articles. Pa.R., M.S, and M.M. prepared the figures and tables and proorfread the main text. B.J.E. led the research and ensured the integrity of the final submission.

Funding

None.

Data Availability

No datasets were generated or analysed during the current study.

Declarations

Competing Interests

The authors declare no competing interests.

Conflict of interest

None.