Generative large language models in medicine: a scoping review of recent methodological advances

Abstract

Generative large language models (LLMs) are rapidly transforming medicine, demonstrating unprecedented capability across a broad spectrum of clinical and biomedical tasks. While prior literature has extensively investigated their applications, the methodological foundation underpinning these models remains comparatively underexamined. In this review, we provide a mechanically grounded analysis of recent methodological advances shaping the development and deployment of generative LLMs in healthcare. We categorize the technical landscape into three principal pillars: pretraining, fine-tuning, and prompt engineering, and examine their key architectures, subtypes, and adaptation strategies based on literature published between 2023 and 2025. We further discuss the emerging directions, including efficient model infrastructures and LLMs-powered multi-agent systems, alongside critical challenges related to bias, generalization, and evaluation. By tracing the evolutionary trajectories of these methodologies, this scoping review provides a mechanism-centered framework to inform responsible model development and deployment in medical settings, tailored to task complexity, data characteristics, and resource constraints.

Introduction

Amid the rapid advances in artificial intelligence (AI), generative large language models (LLMs) have emerged as a revolutionary, paradigm-shift technology^1–3. Built primarily on Transformer architectures, these large-scale neural networks are trained on massive corpora to learn contextualized representations that generalize across tasks and modalities⁴. Flagship systems, exemplified by OpenAI’s ChatGPT⁵, have demonstrated unprecedented capabilities in understanding, reasoning, and multimodal generation⁶. In medicine, these advances are beginning to unlock new possibilities of efficiency, effectiveness, and intelligence that were unimaginable merely a few years ago. From visual-language foundation models advancing computational pathology^7,8 to prompt-engineered chatbots supporting outpatient interactions⁹, from automated summarization of clinical documentation^10–12 to adaptive decision-support systems¹³, LLMs have been rapidly explored across the healthcare continuum. Beyond individual applications, LLMs are increasingly embedded in biomedical research and clinical workflows as informative, assistive collaborators^14,15. As this rapid evolution is mirrored by the exponential rise in scientific publications, a systematic examination of the methodological foundations has become increasingly critical. Much like a high-resolution Computed Tomography (CT) scan to reveal the underlying structure, elucidating these mechanisms thoroughly is essential for informed and responsive deployment of LLMs, aligned with task complexity, data characteristics, and available computational and human resources.

To this end, our review examines the recent methodological advances of generative LLMs in medicine. While a growing body of reviews has surveyed progress of LLMs in healthcare, most concentrated on their downstream use cases^13,16–19. Only a few have examined the methodological foundations. For example, Wang et al.²⁰ discussed fine-tuning and zero-shot prompting strategies in biomedical contexts, but provided limited coverage of pretraining methodologies. Kan et al.²¹ and Zhou et al.²² investigated a broad spectrum of training and adaptation mechanisms, but provided relatively shallow technical analysis of pretraining design. Distinct from prior works, this review delivers an in-depth and integrative analysis of the methodological landscape of generative LLMs. We organize the training and adaptation strategies into three core pillars: pretraining, fine-tuning, and prompting engineering, and systematically dissect their technical foundations, including backbone architectures, training strategies, and representative subtypes. By synthesizing the latest advances, we delineate the evolutionary trajectories of these methodologies and discuss emerging directions together with key challenges. This review offers a coherent and forward-looking framework to inform effective and responsible strategies for model selection, development, and deployment, supporting the advancement of more intelligent, equitable, and human-AI collaborative healthcare systems.

Results

Search results and article selection

Our search across three literature databases (PubMed, Web of Science [WOS], and arXiv) yielded 205,865 records, with 179,089 retained after deduplication. Title and abstract-based screening using an LLM-enabled automated pipeline²³ identified 4527 studies. Human reviewers subsequently assessed titles, abstracts, and full texts when required to determine eligibility. Ultimately, 90 studies that met the inclusion/exclusion criteria were included in the final in-depth analysis (Fig 1).

Fig. 1. Literature retrieval and selection workflow incorporating LLM-assisted screening and human eligibility review.

PRISMA-style flow diagram of study selection process. From 205,865 records retrieved (PubMed, Web of Science, arXiv; Jan 2023 -Sep 2025), 26,776 duplicates were removed (n = 179,089). LLM-assisted title/abstract screening excluded 174,562 records not meeting methodological, domain, or study-type criteria (n = 4527). Human eligibility assessment further excluded 4437 records, yielding 90 studies included in the scoping review.

Methodological advancement

The methodological foundations of generative LLMs in medical fields primarily encompass pretraining, fine-tuning, and prompting techniques. These approaches are designed to tackle the distinctive challenges of biomedical tasks. Pretraining allows models to learn foundational knowledge from large-scale corpora, while fine-tuning adapts pretrained models for task- or domain-specific objectives. Complementally, prompting techniques allow models to be flexibly repurposed for new tasks without explicit parameter updates, leveraging their inherent generalization and in-context learning capabilities. In this section, we review recent advances across these three methodological paradigms. Key concepts introduced throughout this section are defined in Supplementary Table S1.

Pre-training

Pretraining constitutes the foundational stage in the development of generative LLMs, in which models are trained on large-scale datasets to acquire generalizable representations that can be adapted to downstream tasks^24,25. This process typically involves a sequence of data curation, architecture design, model training, and validation^26,27. Among these components, architecture design and training objectives are tightly coupled and jointly determine model capacity, efficiency, and generalization.

Contemporary generative models are predominantly built on Transformer architectures²⁸, which leverage self-attention mechanisms to capture long-range dependencies and contextual relationships without sequential constraints, providing the flexibility and scalability required for large-scale pretraining across diverse data modalities. Though GPT-style generative LLMs are fundamentally decoder-only architectures, extending to medical domains and multimodal settings typically require modality-specific encoders to transform non-text inputs into representations compatible with the language model backbone.

At the core of pre-training is self-supervised learning (SSL)^29,30, which enables LLMs to acquire linguistic, semantic, and contextual representations from large-scale unlabeled corpora without requiring manual annotation. Core SSL objectives are token-level prediction, including autoregressive and masked language modeling. In addition, auxiliary SSL objectives, such as contrastive and matching-based alignment are incorporated, particularly in multimodal settings, to enhance cross-modal correspondence and semantic grounding.

Based on modalities presented in the training data, pretraining strategies can be broadly classified into two categories: pretraining of unimodal LLMs and multimodal LLMs (Fig. 2).

Fig. 2. Pretraining of unimodal and multimodal generative LLMs for medical downstream tasks.

Pre-training three types of modality-specific generative LLMs: A Unimodal text models: text data, including clinical notes, tables, practice guidelines, and biomedical literature, are tokenized and processed through a text encoder-decoder pipeline, supporting downstream tasks such as medical chatbots, information extraction, text summarization, and decision support. B Unimodal image models: 2D/3D medical images, spanning histopathology slides, CT, MRI, and other imaging modalities, are processed through a vision encoder-decoder, enabling image classification, segmentation, retrieval, and synthesis. C Multimodal models: heterogeneous data streams, including text, images, multi-omics, and other data modalities, are each processed through modality-specific encoders; the resulting representations are unified through a feature alignment and fusion module before a shared decoder, supporting cross-modal tasks such as multimodal image classification, medical VQA, radiology report generation, image synthesis and enhancement.

Pretraining of unimodal generative LLMs

Pretraining of a unimodal generative model (PUGM) involves training the model on data from a single modality, such as text or image, to learn modality-specific representations (Unimodality section in Table 1 and Fig. 2). For example, SurgNet³¹ was pretrained on large-scale, unlabeled surgical images based on a Transformer backbone. Ma et al.³² introduced a masked autoencoder (MAE) within the SWIN Transformer framework for pathology pretraining using cytopathology and immunohistochemical image datasets. TransformEHR³³ employed a cross-attention mechanism to dynamically weight encoder representations, while its decoder autoregressively generates diagnostic codes. PABLO³⁴ combined a BERT-based encoder with a masked language modeling decoder for non-accidental trauma prediction. These studies illustrate how unimodal pretraining enables generative models to capture modality-specific structure and semantics, establishing a foundation for downstream medical tasks.

Table 1.

Representative studies in pre-training generative LLMs

Model modality	Model	Data modality	Data source	Model architecture			Pretraining strategy	Pretraining scope
				Encoder	Feature alignment and fusion	Decoder
Uni-modality	SurgNet31	Vision	Surgical video	Transformer encoder	–	Transformer decoder	Auto-encoding masked prediction	Encoder + Decoder
	Ma’s modal32	Vision	Microscopy images	Residual Channel Attention Swin-Transformer (RCAST) encoder	–	Transformer decoder	Auto-encoding masked prediction	Encoder + Decoder
	Huang’s modal34	Language	Visit and inpatient admission records	Transformer encoder (Cross attention)	–	Transformer decoder	Auto-encoding masked prediction	Encoder + Decoder
	TransformEHR33	Language	EHR (structured data)	BERT	–	Transformer decoder	Auto-encoding masked prediction	Encoder + Decoder
Multi-modality	Med-MLLM36	Vision & Language	CT/X-Ray images + clinical notes	ViT + Transformer encoder	Soft image-text alignment	Transformer decoder	Alignment prediction	Encoder + Decoder
	CONCH7	Vision & Language	Pathological images + caption	ViT + Transformer encoder	Contrastive alignment	Transformer decoder	Contrastive learning + autoregressive language modeling	Full training
	UMD37	Vision & Language	Medical imagesa + caption	ViT + Transformer encoder	Contrast alignment	Transformer decoder	Contrastive learning	Full training
	Zhou’s model35	Vision & Language	Chest X-ray + radiology report	ViT + BERT	Concatenation	Transformer decoder	Auto-encoding masked prediction	Full training
	Med-VLP41	Vision & Language	Medical images + caption	CLIP-ViT-B + CXR-BERT	Contrastive alignment	Transformer decoder	Contrastive learning	Encoder
	BiomedGPT47	Vision & Language	Multimodality	ResNet + BERT	Transformer	Transformer decoder	Auto-encoding masked prediction	Full training
	DeepDR-LLM43	Vision & Language	Retinopathy + EHR (structured and unstructured data)	ViT	Concatenation	Llama decoder	Auto-regressive masked Learning	Decoder
	skinGPT-445	Vision & Language	Skin image + notes	ViT + Q-Former	Linear alignment	Llama decoder	Alignment prediction	Encoder + Decoder
	PathChat44	Vision & Language	Medical images + Case report	ViT	MLP	Llama decoder	Auto-regressive masked Learning	Decoder
	MISS38	Vision & Language	Radiology images + QA pairs	ViT + JTM encoder	Cross attention	Transformer decoder	Auto-encoding masked prediction	Encoder
	MedGemma39	Vision & Language	Medical images + text	SigLIP + text_tokenizer	Concatenation	Transformer decoder	Auto-regressive language modeling + image-text alignment	Encoder + Decoder

^aMedical images include various clinical imaging modalities, such as CT, MRI, and PET.

CT computed tomography, JTM joint text-multimodal encoder, MLM masked language modeling, MLP multilayer perceptron, MRI magnetic resonance imaging, QA question and answer, UMD unified medical multi-modal diagnostic framework, ViT vision transformer, VSS visual state space.

Pretraining of multimodal generative LLMs

Pretraining of a multimodal generative model (PMGM) involves jointly learning from diverse data modalities to enable cross-modal understanding and generation (Multi-modality section in Table 1 and Fig. 2). Compared with PUGM, multimodal pretraining poses additional challenges, including modality heterogeneity, data alignment, and feature fusion. Architecturally, PMGM typically comprises modality-specific encoders to extract unimodal representations, projection or fusion modules to integrate and align features, and a unified decoder to support coherent multimodal generation and reasoning. Based on the type of input, multimodal generative models can be broadly categorized into vision-centric, language-centric, and vision-language models, each reflecting different design trade-offs in modality grounding and generative capabilities.

In PMGM architectures, the encoder is responsible for transforming each input modality into a shared latent space using modal-specific neural networks. Vision-centric models typically employ convolutional neural networks (CNNs) or vision transformer (ViT) as encoders. In contrast, vision-language models ingest both images and text, in which language components are often handled by BERT or Transformer encoders, while visual inputs are processed by ViT or CNNs. A wide range of architectural combinations has been explored, including ViT + BERT (e.g., Zhou’s³⁵ model), ViT + Transformer encoder (e.g., Med-MLLM³⁶, CONCH⁷, and UMD³⁷), and ViT + JTM encoder (MISS³⁸). More recent designs also incorporate multimodal contrastive backbones, such as MSigLIP paired with a text tokenizer within Gemma^39,40, or CLIP-ViT-B combined with CXR-BERT in Med-VLP⁴¹, underscoring the central role of ViT-based architectures in visual representation learning. Additionally, some vision-language models operate solely on visual inputs while generating textual outputs, such as Huang’s model⁴² for chest radiograph interpretation, DeepDR-LLM⁴³ for primary diabetes care, and PathChat⁴⁴ for pathology-oriented clinical dialogue. These architectures illustrate diverse encoder designs employed in PMGM to support cross-modal representation learning.

Following feature extraction by encoders, PMGM architectures typically employ one latent feature projector to integrate representations across modalities. This component serves as a critical bridge between the encoder and decoder, supporting effective cross-modal alignment and fusion. Feature alignment aims to reduce cross-modal representation discrepancies and establish semantically consistent latent spaces. The strategies of feature alignment include soft image-text alignment (Med-MLLM³⁶), linear alignment (SkinGPT⁴⁵), and contrastive alignment (CONCH⁷, UMD³⁷, and Med-VLP⁴¹). The aligned representations are subsequently fused, i.e., integrated into a unified representation. One straightforward fusion approach is direct feature merging, in which visual features and textual tokens are jointly fed into the decoder (e.g., Zhou’s model³⁵). Feature concatenation remains a widely used fusion strategy, employed in architectures such as DeepDR-LLM⁴³. Cross-attention-based fusion has also been extensively explored, allowing models to dynamically attend to relevant information across modalities (e.g., Chen’s model⁴⁶ and MISS³⁸). More recently, an emerging trend involves the use of dedicated neural networks as feature projectors, enabling more expressive and learnable cross-modal integration in frameworks like BiomedGPT⁴⁷.

In PMGM architectures, the decoder is typically implemented as a Transformer-based generative module. Serving as the core component for sequence generation, the Transformer decoder autoregressively produces outputs conditioned on integrated multimodal representations provided by the encoders and feature projectors. This design enables flexible and scalable generation across modalities while maintaining compatibility with pretraining and adaptation strategies^{35–39,41,45,46}.

Fine-tuning

Fine-tuning refers to the adaptation of a pretrained LLM by updating all or a subset of parameters to adapt the model to domain-specific knowledge, task requirements, or alignment objectives. In medical applications, fine-tuning plays a critical role in bridging general language understanding and clinically meaningful performance. Post-training adaptation of LLMs usually involves three major fine-tuning strategies: instruction tuning, which enhances task execution and reasoning capabilities by training model on instruction-response pairs; preference tuning, which aligns model behavior with human preference and desired behaviors; and parameter-efficient fine-tuning (PEFT), which facilitates efficient and scalable adaptation by updating only a small subset of model parameters.

Instruction tuning

Instruction tuning adapts LLMs to better follow natural-language task specification using instruction-response pairs⁴⁸. It adjusts model weights to better align with task-specific goals. Based on the instruction source, instruction tuning can be categorized into supervised instruction tuning and self-instruct tuning (Table 2).

Table 2.

Representative studies in instruction tuning

Category	Author (publication year)	Data modality	Base model	Biomedical task
Supervised instruction tuning	Li et al. (2024)49	Text	GPT-3.5	Information extraction
	Li et al. (2025)57	Text	GPT-3.5, Llama-2	Adverse event extraction
	Singhal et al. (2025)192	Text	Med-PaLM 2	QA
	Cui et al. (2025)193	Text	Llama-3.1-8B-Instruct	Temporal reasoning
	Li et al. (2024)43	Text, image	Llama	Disease diagnosis
	Wu et al. (2025)194	Text, image	MedLLaMA	VQA, report generation, etc.
	Peng et al. (2025)195	Text, image	BiomedGPT	Image classification and captioning, VQA, etc.
	Yao et al. (2025)196	Text, image	CLIP	Disease diagnosis
	Li et al. (2025)182	Text, image	BiomedGPT, EchoCLIP, Gemini-1.5, etc.	Disease diagnosis
Self-instruct tuning	Tran et al. (2024)59	Text	GPT-4 (data generation), LLaMA 1 and 2	QA, IE, etc.
	Zhang et al. (2024)197	Text	GPT-4-Turbo (data generation), Llama-3	QA
	Li et al. (2025)63	Text	GPT-4 (data generation), GPT-3.5	QA
	Kim et al. (2025)198	Text	GPT-4 (data generation), Llama-3	Medical reasoning
	Wu et al. (2025)165	Text	GPT-4 (data generation), Llama 3	Multilingual QA, text summarization, etc.
	Cui et al. (2024)199	Text, image	GPT-4V (data generation), LLaVA-Med	VQA
	Li et al. (2023)200	Text, image	GPT-4 (data generation), LLaVA-Med	VQA

CLIP contrastive language-image pre-training, GPT generative pre-trained transformer, IE information extraction, Llama large language model Meta AI, LLaVA large language and vision assistant, QA question and answer, VQA visual question and answer.

Supervised instruction uses human-curated instruction-response pairs to fine-tune model parameters through gradient-based optimization^49–58 (Fig. 3A). For example, BioInstruct utilized supervised instruction tuning for biomedical natural language processing (NLP), achieving substantial performance gains when training data were closely aligned with downstream objectives⁵⁹. While this approach offers high precision and reliability, its scalability is limited by the availability of high-quality labeled data and expert annotation cost^60–62.

Fig. 3. Schematic illustration of two instruction tuning methods.

A Supervised instruction tuning, where instruction-response pairs are derived from human-annotated datasets and structured into Input, Output, and Context components for gradient-based model adaptation, potentially updating the attention, feed-forward, and normalization layers. B Self-instruct tuning, where instruction-response pairs are generated from seed instructions from LLMs to form a scalable, synthetic corpus, followed by the same model adaptation process.

Self-instruct tuning introduces a scalable alternative by automating instruction-response generation using LLMs themselves^59,63–67 (Fig. 3B). In this approach, a seed model generates synthetic instructions and corresponding responses, which are subsequently curated and used for fine-tuning^68,69. Self-instruct approaches have been shown to substantially improve instruction-following ability and zero-shot generalization across tasks such as classification, reasoning, and text generation⁶⁹. Early work demonstrated that the vanilla GPT-3 model struggled with instruction adherence, whereas self-instruct tuning effectively mitigated these issues⁶⁹. Subsequent studies further showed that incorporating refinement strategies, such as human expert review or distillation from teacher models, can markedly enhance performance compared with training on unrefined synthetic data alone⁶⁹. More recent extensions, including multimodal self-instruct, have expanded the approach to visual reasoning tasks⁷⁰. Despite its scalability advantages, self-instruct tuning poses challenges related to instruction quality and error propagation, as inaccuracies in synthetic data can be amplified during fine-tuning¹⁹.

Preference tuning

Preference tuning is typically applied after instruction tuning to further align LLMs with human preferences and desired behavioral norms. While instruction tuning improves task comprehension and response structure, preference tuning refines more nuanced aspects of model behaviors, such as consistency and response style. In medical applications, preference tuning contributes to reducing undesirable behaviors, including incoherent reasoning or overconfident outputs, though it does not independently eliminate hallucinations or bias⁷¹.

Preference tuning is commonly formulated as a reinforcement learning process, in which a pretrained policy is to maximize reward signals derived from human or proxy preferences⁷². Based on model optimization mechanisms, current preference tuning can be categorized into reinforcement learning from human feedback (RLHF), direct preference optimization (DPO), and supervised fine-tuning (SFT)-like approaches. RLHF normally follows a three-stage pipeline, including SFT, reward modeling, and reinforcement learning^73–75. DPO formulates preference learning as a classification-based objective, enabling closed-form policy optimization without explicit reward modeling^76–78. More recent preference optimization methods, such as odds ratio preference optimization (ORPO)⁷⁹, further simplify alignment by eliminating the need for a separate reference model while maintaining competitive performance. These preference tuning techniques have been extensively applied to healthcare-oriented LLMs. For instance, Yang et al.⁸⁰ proposed Zhongjing, a Chinese medical LLM aligned through RLHF using expert feedback and real-world multi-turn medical dialogues, improving instruction-following ability and response safety. Savage et al.⁸¹ systematically evaluated DPO-based methods across clinical reasoning and urgency triage tasks, demonstrating consistent performance gains. Izhar et al.⁸² proposed ORPO as a hybrid alignment strategy for continual radiology report generation, bypassing the need for RLHF alignment.

Parameter efficient fine-tuning

Adapting large pre-trained language models to specialized medical tasks is often constrained by limited labeled data and high computational costs. Parameter efficient fine-tuning (PEFT) addresses these challenges by adjusting only a small portion of model parameters while keeping the majority of weights frozen⁸³, thereby markedly reducing training overheads without substantially compromising performance. PEFT comprises four primary subtypes: adapter tuning, prefix tuning, prompt tuning, and low-rank adaptation (LoRA) (Fig. 3)⁸⁴, each of which offers distinct trade-offs between parameter efficiency, computational complexity, and model performance (Table 3)⁸⁵.

Table 3.

Comparative analysis of full fine-tuning and parameter-efficient fine-tuning (PEFT) methods

PEFT methods	Trainable parameters	Computational cost	Generalization
Full Fine-Tuning	High	High	Moderate–High
Adapter Tuning	Moderate	Moderate	High
Prefix Tuning	Very Low	Low	Moderate
Prompt Tuning	Very Low	Low	Low–Moderate
LoRA	Low	Low	High

(1) Adapter Tuning: Adapter tuning introduces lightweight, trainable modules (called adapters) into transformer blocks while freezing the backbone models⁸⁶ (Fig. 4A). This approach represents the early adopter of PEFT and has demonstrated strong performance across diverse generative and vision-language tasks. For example, Adapt-cMolGPT⁸⁷ employed down-and-up-projection adapters to improve the drug-like molecular compound generation, while DeepDR-LLM⁴³ integrated multilayer perceptron-based adapters to support personalized diabetes management. MAKEN⁸⁸ adopted low-rank adapters to calibrate medical image representations, facilitating efficient visual feature learning for medical report generation. Reg2RG⁸⁹ utilized a six-layer transformer decoder-based adapter to compress and project both local region-level features and global volumetric features from the 3D encoder into the LLM embedding space, thereby enhancing vision-language alignment for CT report generation. Adapter tuning has also been widely adopted for medical image segmentation. Medical SAM Adapter⁹⁰ injected domain knowledge into the Segment Anything Model (SAM) via lightweight bottleneck adapters and prompt-conditioned Hyper-Adapter, enabling efficient and interactive segmentation. Extensions such as MA-SAM⁹¹ and 3DSAM-adapter⁹² further adapted SAM to volumetric and temporal medical data by introducing 3D adapter modules that capture spatial context while preserving the frozen backbone.

Fig. 4. Schematic overview of four parameter-efficient fine-tuning (PEFT) methods.

A Adapter Tuning, B Prefix Tuning, C Prompt Tuning, D LoRA.

(2) Prefix Tuning: Prefix tuning adds tunable parameters to multi-head attention layers⁹³, enabling task adaptation while keeping the backbone model weights frozen (Fig. 4B). In medical applications, Van Sonsbeek et al.⁹⁴ used visual prefixes for medical visual QA, and Chen et al.⁹⁵ introduced MILE-Prefix, attaching prefixes to joint text-multimodal encoders and decoders, substantially reducing training costs in resource-constrained medical domains.

(3) Prompt Tuning: Prompt tuning adapts pretrained language models with continuous, learnable vectors, commonly referred to as soft prompts, that are prepended to the input and optimized via backpropagation^96,97 (Fig. 4C). Recent advancements have tailored prompt tuning to address challenges inherent to biomedical data, including semantic ambiguity in textual records and computational bottlenecks in multimodal analysis. For instance, Medical short text classification via Soft Prompt-tuning⁹⁸ (MSP) enriched the soft prompts with professional medical concepts to bridge gaps between noisy lay queries and specialized medical terminology. For large-scale architectures, GatorTronGPT⁹⁹ demonstrated that soft prompts can effectively adapt a frozen 20-billion parameter model to diverse clinical tasks without the cost of full fine-tuning. In addition, prompt tuning has been extended to visual and multimodal settings. For instance, Fine-Grained Prompt Tuning¹⁰⁰ (FPT) summarized fine-grained pathological features into compact prompt representations to enable high-resolution classification on consumer hardware, while MRG-LLM¹⁰¹ employed instance-conditioned prompt transformations to generate patient-specific radiology reports. These studies highlight prompt tuning as a scalable and flexible PEFT strategy for adapting foundation models to medical tasks under stringent resource constraints.

(4) Low-Rank Adaptation: Low-rank adaptation (LoRA) injects trainable low-rank matrices into each layer of the Transformer architecture to approximate the weight updates¹⁰² (Fig. 4D). Due to its ability to achieve comparable or in some cases even superior performance to the full fine-tuning while being easy to implement¹⁰³, LoRA has become one of the most widely adopted PEFT techniques in addressing medical tasks. For example, BrainGPT augmented Mistral-7B-v0.1 with neuroscience-specific knowledge through LoRA¹⁰⁴; PRIMERA, LongT5, and Llama-2 have been fine-tuned with LoRA for medical evidence summarization¹⁰⁵; GatorTron has been adapted with LoRA for clinical temporal relation extraction¹⁰⁶; MiniGPT-4 has been fine-tuned with LoRA to develop RRG-LLM for radiology report generation¹⁰⁷; and real-time detection transformer (RT-DETR) has been enhanced with LoRA for full-body musculoskeletal ultrasound analysis¹⁰⁸.

A growing family of LoRA variants has been proposed to improve adaptability and address task-specific challenges in medical applications. To mitigate memory constraints, quantized low-rank (QLoRA)¹⁰⁹ integrated low-rank adaptation with 4-bit NormalFloat and double quantization. QLoRA has been applied to fine-tune Llama2-70B-chat-hf for Japanese medical question answering⁴⁸ and to fine-tune FLAN-T5 for diverse clinical text summarization tasks¹¹⁰, achieving performance comparable to or exceeding that of medical experts. Beyond clinical text, PH-LLM¹¹¹ employed QLoRA on social media data for real-time public health infoveillance, illustrating its utility in population-level monitoring. To support multi-task medical applications, MOELoRA¹¹² combined mixture-of-experts (MOE) architectures and LoRA, allowing each expert to capture task-specific knowledge while retaining high parameter efficiency through a task-guided gating mechanism. AdaLoRA¹¹³ improved efficiency by allocating rank based on singular-value importance and pruning less informative components. Building on this idea, SA-MDKIF¹¹⁴ injected medical domain knowledge by assigning an AdaLoRA module to each medical skill. SeLoRA¹¹⁵ introduced self-expanding LoRA, dynamically adjusting rank allocation to better support high-fidelity medical image synthesis. These variants demonstrate how LoRA-based methods can be effectively extended to balance efficiency, adaptability, and performance across diverse medical tasks.

Prompting engineering

Prompting engineering steers LLMs toward desired outputs by modifying the input alone, without updating model parameters. As a lightweight adaptation approach, it plays an important role in enhancing the accuracy, versatility, and applicability of LLMs. Early prompting techniques relied on manually crafted templates or detailed contextual descriptions to elicit desired responses¹¹⁶, often requiring substantial domain expertise and iterative refinement. As the field has advanced, prompting methodologies have evolved toward more efficient and automated strategies. Contemporary prompting techniques can be broadly categorized into in-context learning (ICL), chain-of-thought (CoT), and retrieval-augmented generation (RAG), each offering different levels of automation, interpretability, and generalization (Fig. 5).

Fig. 5. Schematic overview of three prompting methods.

A In-context learning, which leverages a set of demonstration examples (typically few-shot QA pairs); B Chain-of-thought prompting, which breaks down complex inputs into sequential reasoning steps; and C Retrieval-Augmented Generation, which dynamically retrieves relevant information from an external (vector) database to enrich the integrated input before processing.

In-context learning

In-context learning (ICL) enables LLMs to perform new tasks by conditioning on task instructions or example demonstrations provided directly in the input prompt, without updating model parameters (Fig. 5A). ICL supports both zero-shot learning, which relies solely on task-specific instructions without any labeled examples, and few-shot learning, where a small set of labeled examples is provided to reduce ambiguity and improve performance. In biomedical applications, ICL has demonstrated strong effectiveness in tasks such as QA¹¹⁷ and information extraction^118,119. While few-shot examples are often selected randomly, recent studies showed that example selection critically affected performance. For example, k-nearest neighbor-based retrieval of semantically similar examples significantly improved histopathology classification accuracy¹²⁰. DDI-JUDGE selected similarity-guided few-shot examples for drug-drug interaction prediction, outperforming leading LLMs without few-shot prompting¹²¹.

Chain-of-thought

Chain-of-thought (CoT) prompting decomposes complex questions into smaller, manageable steps, guiding LLMs to address sub-questions sequentially (Fig. 5B). This structured approach simplifies the problem-solving process, particularly effective for tasks requiring multi-step reasoning, such as mathematical questions, logical inference, and medical QA. For example, Lucas et al. improved accuracy on USMLE Sample Exam questions using an ensemble CoT method that iteratively aggregated step-by-step answers into new prompts¹²². Ott et al. curated multiple scientific/medical QA datasets for CoT training, seeking to enhance LLM robustness and explainability¹²³. ZeroTuneBio¹²⁴ incorporated CoT reasoning into a three-stage name entity recognition (NER) framework, enabling zero-shot entity extraction without task-specific fine-tuning. These studies demonstrate that structured reasoning via CoT can simultaneously boost interpretability, robustness, and accuracy in complex biomedical inference tasks.

Retrieval-augmented generation

Retrieval-augmented generation (RAG) enhances LLMs by integrating external knowledge retrieval into the generation process, enabling more accurate, grounded, and contextually relevant responses (Fig. 5C)¹²⁵. By dynamically accessing up-to-date or domain-specific information at inference time, RAG mitigates limitations of static pretraining and is particularly valuable in rapidly evolving medical domains where factual accuracy and currency are critical.

The effectiveness of RAG depends on the design of the retrieval pipeline. Modern RAG systems predominantly rely on embedding-based dense retrieval^{120,121,126–133}. Retrieval performance is highly sensitive to the choice of embedding model, which determines representational quality, domain coverage, and computational cost. Commonly used embedding models include OpenAI’s text-embedding-ada-002 and text-embedding-3-small/large¹³⁴, sentence-transformers models¹³⁵, BGE‑large‑en-v1.5¹³⁶, among others. These models vary in vector dimensionality, multilingual capability, and domain specialization, which influence both retrieval accuracy and computational efficiency. In addition, in large-scale or continuously updated knowledge bases, practical considerations such as retrieval latency, scalability, and index freshness are equally important. To address these challenges, high-performance vector indexing systems, such as FAISS^137,138, ScaNN^139,140, and Milvus^141,142, have been widely adopted to enable high-throughput and low-latency vector search.

Leveraging these rapidly evolving technologies, several innovative RAG frameworks have demonstrated their versatility in medical applications. For example, RISE¹²⁶ addressed diabetes-related inquiries through a RAG pipeline comprising query rewriting, information retrieval, summarization with factual and safety checks, and final response generation to improve the accuracy and safety of LLM-generated responses. Self-BioRAG¹²⁷ integrated RAG with self-reflection to improve biomedical reasoning through explanation generation, selective retrieval of domain-specific documents, and reflective response refinement. Other applications integrated external biomedical resources more directly via relevant domain tools: GeneGPT¹²⁸ leveraged NCBI Web APIs of E-utils for biomedical databases access and the BLAST tool for DNA sequence alignment, while RefAI¹²⁹ incorporated real-time PubMed retrieval into GPT-4 to produce up-to-date biomedical summaries.

Beyond standalone architectures, RAG has been increasingly integrated with other complementary prompting strategies to further enhance reasoning and task performance. For example, Kresevic et al. combined RAG, few-shot learning, and CoT reasoning to improve accuracy in Hepatitis C-related QA¹³⁰. Similarly, reguloGPT utilized RAG and CoT prompting to construct knowledge graphs for molecular regulatory pathways¹³¹, while the RT-framework coupled retrieval modules and CoT-based few-shot learning to enhance biomedical named entity recognition¹³². Representing a further evolution, RAG has also been increasingly embedded in multi-agent systems (MASs). BioRAGent¹³³, for instance, decomposes biomedical QA into coordinated agents for query interpretation, retrieval, and verification, achieving superior performance over state-of-the-art LLMs while improving usability and user experience.

Discussion

As generative LLMs continue to evolve, their integration in medicine is increasingly shaped by advances in pretraining, fine-tuning, and prompting, together with growing demands for robustness, safety, and reliability^{16,17,143,144}. Recent progress has shifted from purely scale-driven expansion toward more efficient architectures, multi-agent orchestration, and evidence-grounded generation^145,146. In parallel, increasing model complexity and broader deployment into the high-stakes clinical settings introduce new challenges related to bias and accountability, underscoring that technical progress alone is insufficient if without rigorous, safety-centered evaluation and validation to support trustworthy real-world implementation. This section discusses key emerging directions alongside the inherent challenges, with the aim to inform more robust, safety-conscious development and deployment of generative LLMs in health systems.

Recent advances in pretraining architectures reflect the transition from parameter scaling toward prioritizing efficiency, adaptability, and long-context reasoning. Google’s Titans, for instance, introduces a memory-augmented attention mechanism that allows models to learn and retrieve long-range context during inference, extending effective window lengths into the multi-million-token range¹⁴⁷. In parallel, Meta’s Byte Latent Transformer (BLT) enhances efficiency through byte-level latent patches, replacing conventional tokenization, which matches token-based model performance at substantially lower inference cost¹⁴⁸. DeepSeek-V3 employs a sparse mixture-of-experts (MOE) architecture with Multi-head Latent Attention (MLA) to sustain high accuracy on long-context reasoning tasks while activating only a fraction of parameters per token¹⁴⁹. The architectural evolution of general-domain LLMs presents transformative opportunities for the medical domain. Specifically, long-context architectures exemplified by Titans are well-suited for longitudinal clinical reasoning, enabling coherent integration of multi-year, heterogeneous patient records that exceed the conventional context limits. By supporting stable retrieval and reasoning over extended temporal horizons, these architectures unlock new potentials for improving diagnostic continuity, medication reconciliation, and holistic patient understanding. Concurrently, parameter-efficient architectures, including sparse MOE models, provide a scalable pathway toward high-level reasoning with substantially reduced computational overhead. Such efficiency is especially advantageous in real-world healthcare settings, where scalability, latency, and resource limitations remain significant barriers. Collectively, the architectural advances suggest that the next phase of medical LLM development will extend beyond scaling and domain-specific pretraining toward systematic adaptation of the innovative architectures, making them to accommodate the complexity and longitudinal nature of medical data, while remaining response to practical needs in the real-world healthcare settings.

An emerging frontier in medical AI is multi-agent systems (MASs), which are coordinated ensembles of specialized AI agents that collaboratively reason, retrieve evidence, verify outputs, and synthesize decisions. Recent work across diverse clinical applications, including ClinicalAgent for trial design¹⁵⁰, CARE-AD for Alzheimer’s disease prediction¹⁵¹, and MedAgent-Pro for hierarchical diagnostic planning¹⁵², demonstrated that multi-agent models improved factual accuracy, interpretability, and safety compared with the single-agent models. Multi-agent frameworks represent an evolution from linear prompting toward collaborative and self-regulating reasoning. Specialized agents, such as planner, retriever, and evaluator, exchange context and intermediate outputs through structured dialogue or API-mediated communication^150–153. Reflexive mechanisms, such as self-retrieval, cross-examination, and iterative critique, allow systems to recognize uncertainty, query external knowledge sources, and refine conclusions, thereby reducing hallucination and error propagation. More recent architectures incorporate hierarchical oversight, including supervisory agents that audit reasoning quality, enforce factual consistency, and manage safety constraints. The vision for multi-agentic medical AI centers on building governed cognitive collectives—transparent, auditable, and modular ecosystems designed to collaborate with clinicians and scientists. Such systems could support distributed reasoning, continuous knowledge synthesis, and adaptive decision support while maintaining accountability. Realizing this vision will require advances in standardized inter-agent coordination protocols, benchmarking methodologies that quantify collective reasoning gains, and rigorous safety and reliability evaluation in prospective and workflow-integrated clinical settings. As these foundations mature, LLMs-powered MASs are poised to become a cornerstone of trustworthy, human-aligned, and continuously improved clinical AI, shifting the focus from isolated predictive models toward auditable, system-level intelligence.

The pursuit of universal health AI^154,155, closely aligned with generalist medical AI (GMAI)^156,157, aims to build systems capable of flexible, multimodal, and context-aware reasoning across diverse healthcare environments. Achieving this vision requires addressing persistent challenges in robustness, task transferability, and equity across diverse medical environments. Recent frontier foundation models, ranging from proprietary systems like OpenAI’s GPT-5¹⁵⁸ and Google’s Gemini 3 Pro¹⁵⁹, to open-weights models including DeepSeek’s R1¹⁶⁰, Meta’s Llama4¹⁶¹, and Moonshot’s Kimi K2 Thinking¹⁶², reflect a new generation of reasoning-centric architectures. By integrating long-context processing, retrieval-augmented inference, and multimodal understanding, these models signal a shift toward explicit reasoning and problem solving with high transparency and reliability^163,164. Yet, architectural advances alone are insufficient to realize universal health AI in practice. Real-world impact depends on rigorous validation across heterogeneous populations, institutions, and clinical workflows, together with systematic assessments of bias, safety, and unintended consequences^165,166. Progress will require architectures that balance general-purpose reasoning with deep domain expertise, supported by standardized benchmarks for fairness, transparency, and real-world efficacy^161,165,167. Ultimately, universal health AI will emerge as an interoperable ecosystem of governed, adaptive intelligence, continuously learning from diverse data and guided by ethical and human-centered oversight^16,144.

Despite rapid advances, several foundational challenges continue to limit the medical readiness of LLMs and multimodal AI systems.

In current healthcare applications, most RAG systems primarily rely on textual sources, such as clinical notes, practice guidelines, or biomedical literature, to condition model outputs^168,169. However, text-only retrieval is inherently limited for many clinical tasks. Documentation practices vary widely across clinicians, institutions, and EHR systems, potentially introducing semantic drift and inconsistency that can propagate through downstream reasoning^170–172. Text-based retrieval may not always be grounded in the observable, image-based manifestations of disease, limiting its utility for visually driven clinical reasoning¹⁷³. By contrast, real-world clinical decision making frequently depends on image-based reasoning. For example, comparing visually similar radiology studies or whole-slide pathology images to contextualize a new case before generating a report. Although content-based image retrieval (CBIR) has been studied for decades, it remains one of the persistently under-implemented technologies in medical AI^174,175. Despite significant technical advances, CBIR systems remain largely experimental and are rarely integrated into hospital information systems or linked with RAG-based pipelines¹⁷⁶. Consequently, the existing medical RAG applications remain text-only^{126,127,129,133,177}, with no truly multimodal, image-aware retrieval frameworks capable of grounding LLM responses in radiologic, pathologic, or other visual data.

Even state-of-the-art LLMs remain prone to hallucination when operating under clinical uncertainty, particularly in scenarios involving incomplete, ambiguous, or conflicting evidence^178–180. In such settings, models can generate plausible-sounding but unsupported statements, fabricate diagnoses, or conflate temporal events, creating substantial risks for downstream decision-making¹⁷⁸. These errors stem from the probabilistic nature of LLM text generation, which prioritizes linguistic coherence over evidentiary accuracy. While current alignment methods, such as RLHF and DPO, reduce overt factual errors, they fail to fully prevent clinically dangerous hallucinations because they are typically trained on general-domain rather than clinical preference data. Mitigating this limitation requires evidence-grounded reasoning pipelines that constrain model outputs to retrieved, verifiable sources, alongside explicit uncertainty quantification, real-time retrieval validation, and human-in-the-loop review in high-stakes contexts^179,181.

Despite rapid methodological progress, rigorous evaluation of generative LLMs remains challenging. Most existing benchmarks rely on retrospective datasets, static question-answering tasks, or narrow domain-specific evaluations that may inadequately reflect the complexity and diversity of real-world clinical practice. As highlighted in our prior multimodal LLM evaluation study¹⁸² and other relevant research^183,184, strong benchmark performance does not necessarily translate into reliable clinical utility across diverse patient populations and prospective clinical settings, yielding critical patient safety implications. Findings based on NOHARM (Numerous Options Harm Assessment for Risk in Medicine)¹⁸⁵, a benchmark centered on patient-level harm assessment, demonstrates that widely used LLMs can produce severely harmful medical recommendations at nontrivial rates, emphasizing clinical safety as a distinct performance dimension necessitating explicit measurement. To address these issues, MedHELM¹⁸⁶, an extensible evaluation framework, integrates clinician-validated taxonomy, real-world EHR data, and cost-performance analysis to characterize LLMs’ strengths and limitations across medical tasks. Future evaluation frameworks should therefore extend beyond retrospective benchmarking to incorporate prospective validation, calibration-aware assessment, explicit measurement of omission errors, and standardized harm weighted reporting capable of supporting trustworthy real-world clinical deployment.

Current LLMs primarily learn statistical associations from large-scale training corpora rather than explicit causal relationships, limiting their ability to distinguish correlation from causation in complex clinical settings. Consequently, models may generate clinically plausible recommendations that fail to accurately reflect underlying biological mechanisms, intervention effects, or longitudinal disease dynamics. This limitation is particularly critical in medicine, where diagnosis and treatment decisions often rely on mechanistic understanding, temporal progression, and causal inference rather than pattern recognition alone. The absence of explicit causal modeling may therefore lead to erroneous recommendations in scenarios involving confounding variables, rare diseases, or complex intervention-response relationships. Recent studies have increasingly emphasized the need to integrate causal inference, mechanistic biomedical knowledge graphs, and longitudinal patient modeling into foundation AI systems to improve clinically meaningful reasoning and decision support^187,188. Although these emerging approaches may help bridge the gap between associative prediction and mechanistic inference, achieving reliable causal reasoning remains an open challenge for future medical foundation models.

Establishing ethical and trustworthy deployment of LLMs in medicine remains a critical barrier to real-world adoption. Persistent concerns, ranging from data privacy and demographic bias to limited transparency and diffuse accountability, highlight the need for governance frameworks that embed ethical safeguards throughout the AI model lifecycle¹⁸⁹. Growing evidence shows that biases inherent in training data can propagate into unequal diagnostic or treatment recommendations, potentially reinforcing the existing disparities across populations and clinical settings. At the same time, LLMs’ tendency to generate fluent but incorrect or fabricated information underscores the importance of rigorous pre-deployment validation, continuous performance monitoring, and safeguards against overconfident outputs in high-stakes clinical scenarios. Addressing these limitations requires transparent documentation of data provenance, reproducible and auditable development pipelines, and clearly defined human-in-the-loop oversight for sensitive decision-support tasks. Standardized reporting tools, such as model cards and structured fairness assessments, are essential for enabling traceability, accountability, and informed clinical governance¹⁹⁰. Ultimately, translating ethical principles into operational practice is critical to ensure that generative and multi-agent clinical AI systems remain safe, equitable, and aligned with patient needs and societal values.

This review has a few limitations. First, despite a comprehensive search across multiple databases, the rapid development in generative LLMs made it possible that relevant models or emerging methodologies are not fully captured in this review. Second, as this study prioritizes the mechanistic decoding of the LLM development and adaptation strategies, it does not include quantitative meta-analyses or direct model comparisons. Future work could complement the current qualitative synthesis with granular meta-analysis to enhance quantitative and comparative insights. Third, although systematic efforts were made to ensure consistency in literature screening and study categorization across reviewers, the synthesis and thematic organization of evidence inevitably involved a degree of subjective judgment. Therefore, the findings of this scoping review should be interpreted as a structured, descriptive synthesis of the recent methodological advances and their underlying mechanisms rather than a quantitative assessment of model performance.

The methodological landscape of generative LLMs is undergoing rapid evolution. As outlined in this review, recent advances extend beyond model scaling to encompass more efficient backbone architectures, more effective adaptation techniques, and increasingly sophisticated multi-agent frameworks. These developments reflect a paradigm shift toward highly efficient, adaptable, and context-aware generative AI systems in support of complex reasoning and task execution across diverse medical settings. Sustained human-AI collaboration, embedding human expertise and patient needs into the development and deployment life cycle, is essential to translate methodological innovation into trustworthy, equitable, and clinically meaningful outcomes. With responsible deployment, rigorous validation, and careful governance, generative LLMs have the potential to become a foundational pillar of next-generation, human-centered healthcare at scale.

Methods

Literature retrieval

We conducted a comprehensive literature retrieval across PubMed, Web of Science (WOS), and arXiv, using search terms relevant to generative LLMs in medicine (Supplementary Section 1). To capture the most recent methodological advances following the emergence and widespread adoption of generative LLMs, we restricted the publication time to January 2023 through September 2025. Database-specific query strategies and initial results are detailed in Supplementary Section 2. The Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) Checklist is available at Supplementary Table S2.

Eligibility criteria and scope

Studies were included if they satisfied all of the following criteria: (1) the proposed LLMs adopted a Transformer decoder; (2) the methodological contribution pertained to pretraining, fine-tuning, or prompt engineering; (3) the study was situated within the medical domain, including both clinical medicine (such as cardiology and surgery) and basic research (such as physiology and genetics); and (4) the work was original research rather than reviews, surveys, editorials, or commentaries. Notably, to reflect the rapidly evolving landscape of generative LLMs in medicine, this review extends beyond language-only models to encompass multi-modal architectures that accommodate diverse data modalities, including medical imaging, audio, video, and multi-omics.

Literature screening and selection

To balance efficiency and rigor in study selection, we employed a two-stage approach combining LLM-assisted screening with human expert review. Specifically, a multi-agent system (MAS)-enabled literature review pipeline²³, built on top of our previous study¹⁹¹, was utilized for title/abstract-based screening. The MAS comprised two open-sourced (Llama 3-8B and Qwen 3-8B) serving as independent voter agent and one proprietary LLM (GPT-5) serving as an arbitrator. Studies receiving concordant inclusion decisions from both voter agents were retained. For studies with inconsistent decisions, the arbitrator agent reviewed title/abstract and made a final determination. All studies approved by the MAS were further reviewed by human experts for eligibility assessment, based on title/abstract and full texts when necessary. To refine the final corpus, studies were further evaluated on their academic influence and methodological rigor, with additional consideration given to citation counts and/or publication venue characteristics (such as journal impact factors and conference standing within the field). Throughout the screening process, regular discussions were conducted to harmonize study categorization and criteria alignment.

Result synthesis

For the studies included in the final analysis, we extracted key methodological characteristics, including model architectures, training data source and modalities, development and adaptation strategies, and reported performance. Based on these analyses, we organized the field into a hierarchical structure, delineated major methodological categories, identified model subtypes and variants, and summarized their distinctive features. Through this synthesis, we sought to develop a technical roadmap that characterizes the evolving landscape of generative LLM development and deployment in medicine.

Supplementary information

Supplementary Information ^{(212.6KB, pdf)}

Author contributions

Fang Li: conceptualization, investigation, visualization, writing - drafting and revision; Jianfu Li: conceptualization, investigation, writing - drafting and revision; Weiguo Cao: investigation, visualization, writing - drafting and revision; Haifang Li: investigation, visualization, writing - drafting and revision; Yiming Li: investigation, visualization, writing - drafting and revision; Zenan Sun: investigation, visualization, writing - drafting and revision; Dequan Chen: conceptualization, investigation, writing - critical review; Tiehang Duan: investigation, writing - drafting and revision; Pengze Li: investigation, visualization; Hamid Tizhoosh: conceptualization, writing - critical review; Cui Tao: funding acquisition, conceptualization, supervision, writing - critical review. All authors reviewed and approved the final manuscript.

Data availability

The data supporting the findings of this study are available within the manuscript and its supplementary files.

Code availability

The Python codes for LLM-assisted literature screening are available at the Github: https://github.com/Tao-AI-group/Automatic_Review.

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at https://doi.org/10.1038/s44401-026-00137-5.