Abstract
Objectives
Large language models (LLMs) have shown potential in biomedical applications, leading to efforts to fine-tune them on domain-specific data. However, the effectiveness of this approach remains unclear. This study aims to critically evaluate the performance of biomedically fine-tuned LLMs against their general-purpose counterparts across a range of clinical tasks.
Materials and Methods
We evaluated the performance of biomedically fine-tuned LLMs against their general-purpose counterparts on clinical case challenges from NEJM and JAMA, and on multiple clinical tasks, such as information extraction, document summarization and clinical coding. We used a diverse set of benchmarks specifically chosen to be outside the likely fine-tuning datasets of biomedical models, ensuring a fair assessment of generalization capabilities.
Results
Biomedical LLMs generally underperformed compared to general-purpose models, especially on tasks not focused on probing medical knowledge. While on the case challenges, larger biomedical and general-purpose models showed similar performance (eg, OpenBioLLM-70B: 66.4% vs Llama-3-70B-Instruct: 65% on JAMA), smaller biomedical models showed more pronounced underperformance (OpenBioLLM-8B: 30% vs Llama-3-8B-Instruct: 64.3% on NEJM). Similar trends appeared across CLUE benchmarks, with general-purpose models often achieving higher scores in text generation, question answering, and coding. Notably, biomedical LLMs also showed a higher tendency to hallucinate.
Discussion
Our findings challenge the assumption that biomedical fine-tuning inherently improves LLM performance, as general-purpose models consistently performed better on unseen medical tasks. Retrieval-augmented generation may offer a more effective strategy for clinical adaptation.
Conclusion
Fine-tuning LLMs on biomedical data may not yield the anticipated benefits. Alternative approaches, such as retrieval augmentation, should be further explored for effective and reliable clinical integration of LLMs.
Keywords: large language models (LLMs), biomedical fine-tuning, domain-specific adaptation, hallucination in AI models, benchmarking
Introduction
Large language models (LLMs) have shown remarkable potential for various applications, including in the biomedical domain.1–5 These models can serve as knowledge sources, aid in information retrieval from patient notes, assist with data structuring or coding, and support patient anamnesis.3–5 To enhance LLMs’ performance on domain-specific tasks, several initiatives have focused on fine-tuning these models using biomedical data.6–11
Recent approaches include BioMistral-7b,8 based on Mistral 7b,12 and OpenBioLLM,7 which uses the Llama3 (Large Language Model Meta AI) models as a foundation.11 These efforts aim to create specialized models for biomedical applications.
However, assessing the performance of fine-tuned models presents challenges. While using exam questions, such as those from the United States Medical Licensing Examination (USMLE), is common,7–9,11,13 this approach may not accurately reflect a model’s performance in real clinical practice. Additionally, the prolonged availability of these questions online risks data leakage and benchmark corruption.14
The latest general-purpose LLMs, like Llama 3 and Mistral, are trained on vast amounts of web data, likely including substantial medical information. This raises questions about the added value of fine-tuning, with limited availability of domain-specific data, research groups may struggle to introduce novel information not already present in the training data of large AI companies. Consequently, biomedical LLMs fine-tuned on similar web-based data might face issues such as redundant learning or even performance degradation due to catastrophic forgetting.15
To address these concerns, we conducted a comparative study of recent biomedical LLMs and their general-purpose baseline models. Our evaluation used data from recent benchmarks, specifically chosen to likely be outside the fine-tuning process of the biomedical models, ensuring a fair assessment.
We hypothesized that biomedical models would outperform their general-purpose counterparts given the domain-specific nature of the tasks. Our findings challenge this hypothesis and provide insights into the complex relationship between model architecture, training data, and task-specific performance in the medical domain.
Methods
The benchmarks used in this study were carefully selected to represent a wide range of clinical tasks while ensuring they were likely outside the fine-tuning datasets of biomedical models. We chose recently published case vignettes and newly developed benchmarks to minimize the risk of data contamination. The biomedical and general-purpose models were selected to represent the state-of-the-art in both categories, covering different model sizes and architectures to ensure a comprehensive comparison.
Benchmarks
The following benchmarks were used:
Clinical case challenges
We evaluated the performance of biomedical and general-purpose LLMs using clinical challenges from two medical journals: the New England Journal of Medicine (NEJM) and the Journal of the American Medical Association (JAMA). These case vignettes represent real-world clinical scenarios and cover a wide range of medical specialties and conditions. The NEJM dataset consists of 347 questions, while the JAMA dataset contains 140 questions. These case vignettes, presented with multiple-choice answers, cover a wide range of medical knowledge and provide a comprehensive assessment of the models' performance on unseen medical data. Models are evaluated by accuracy of correctly answered questions.
MeDiSumQA
This task involves answering questions based on discharge summaries from the MIMIC-IV database.14 It tests the model’s ability to extract relevant information from lengthy clinical documents and provide accurate, patient-friendly responses.16
MeDiSumCode
This task evaluates the model’s capability to assign appropriate ICD-10 codes to diagnoses and procedures mentioned in discharge summaries. The task was introduced together with MeDiSumQA as part of the CLUE benchmark and requires both accurate information extraction and a deep understanding of medical coding systems.16
MedNLI
Based on the MIMIC-III dataset,17 this natural language inference task assesses the model’s ability to determine the logical relationship between a premise (a sentence from a clinical note) and a hypothesis.18
MeQSum
This task involves summarizing consumer health queries, testing the model’s ability to understand lay language and reformulate it into concise, medically sound queries.19
ProblemSummary
Using clinical notes organized according to the SOAP (Subjective, Objective, Assessment, Plan) principle, this task requires models to predict a patient’s current health problems based on the Subjective and Assessment sections.20
LongHealth
This task uses 20 fictional patient records to test the model’s ability to handle long documents, answer questions about them, and identify when information is not available in the given context.21 Evaluation is split into three sub-tasks: (1) Answering questions about long document. (2) Handling increased input length with unrelated documents. (3) Identifying when information is not available.
Evaluation metrics
MeDiSumQA, MeDiSumCode, and LongHealth require model with longer context size, limiting the number of models that can be evaluated on these benchmarks.
For text generation tasks (MeDiSumQA, MeQSum, and ProblemSummary), ROUGE scores (ROUGE-1, ROUGE-2, ROUGE-L), and BERTScore are used to evaluate the quality and semantic similarity of generated outputs. ROUGE (Recall-Oriented Understudy for Gisting Evaluation) scores evaluate the overlap of n-grams, which are contiguous sequences of n items (usually words) from a given text, between the generated text and reference text. These scores range from 0 to 1, with scores closer to 1 indicating a higher overlap and better performance. BERTScore evaluates semantic similarity using BERT (Bidirectional encoder representations from transformers) embeddings. The score also ranges from 0 to 1, with higher scores indicating greater semantic similarity. The MeDiSumCode task is assessed using F1-scores for exact and approximate matches of ICD-10 codes, as well as the ratio of valid codes generated. MedNLI uses accuracy to evaluate the model’s ability to classify relationships between premises and hypotheses. For tasks involving entity recognition and medical concepts (ProblemSummary), F1-scores for UMLS entity extraction are also employed. The LongHealth task primarily uses accuracy across its subtasks to evaluate comprehension and question-answering abilities on long clinical documents.16
Models evaluated
Generalist models
Llama (Large Language Models by Meta AI) is a series of foundation models developed by Meta AI. We evaluated models based on the two latest generations of Llama. Llama 2, an improved version of the original Llama model released later in 2023, features enhanced training on a larger dataset and improvements to the model architecture.22 The Llama 2 models were released as base models as well as finetuned chat models. Llama 3, the most recent iteration released in 2024,23 incorporates further advancements in training techniques and utilizes the largest training dataset within the Llama family. Like Llama 2, Llama 3 was released as both a base model and a fine-tuned chat variant.
Mistral 7B, an open-source language model developed by Mistral AI, was released in 2023. It uses novel architectural features such as grouped-query attention and sliding window attention, allowing for efficient processing of long sequences.12 For all base models, the chat/instruction tuned versions, provided by the developers were used (Llama-2-7b-chat-hf, Llama-2-70b-chat-hf, Llama-3-8B-Instruct, Llama-3-70B-Instruct and Mistral-7B-Instruct-v0.2).
Biomedical models based on llama
Meditron-7B, med42, and ClinicalCamel-70B are models based on Llama 2 with 7B parameters or 70B parameters.10,24,25 OpenBioLLM-70B and OpenBioLLM-8B are the most recent biomedical LLMs, based on Llama 3. They currently achieve state of the art on USMLE question answering. BioMistral-7B and JSL-MedMNX-7B-SFT are finetuned versions of Mistral-7B using biomedical data.12
Evaluation procedure
The evaluation of the models was conducted using a combination of methods tailored to each benchmark. For the clinical case challenges from NEJM and JAMA, we employed a GPU server equipped with four NVIDIA A100 GPUs. All models were evaluated in their version that is available through the Hugging Face transformers library (version 4.40.1). For compatible models, vLLM (version 0.4.2) was used, which provides a parallelized engine around the Hugging Face models for faster inference. All models were tested with identical parameters: temperature set to 0, top P at 0.95, and both frequency and presence penalties at 0. The 70B models were run across all four GPUs using Ray (version 2.21.0), while the smaller models were executed on a single A100 GPU. The model Meditron-7b did not produce comprehensible outputs in the initial prompt setup. As the model was trained with a fixed system prompt, the order of the prompt parts was changed to provide the actual clinical vignette at the very end. This was deemed necessary to ensure a fair evaluation of the model capabilities.
All other benchmark evaluations were performed on an NVIDIA DGX node with 8 A100 GPUs. The Hugging Face Text Generation Inference Toolkit (v2.1.0) was used for model inference. Larger models were distributed across up to four GPUs, while smaller models were loaded onto a single GPU. Whenever the tokenizer provided a chat template, it was used to format the model input. For models where the template was described in the model card, the template was added manually. All models were tested with the Hugging Face default parameters (temperature 1, top P 0.95, and no penalties).
Results
Clinical case challenges
On the JAMA case challenges, OpenBioLLM-70B achieved the highest accuracy at 66%, followed by Llama-3-70B-Instruct at 65%. JSL-MedMNX-7B-SFT achieved 54% accuracy, slightly higher than its base model Mistral-7B-Instruct-v0.2 with 52%. Notably, Llama-3-8B-Instruct (57%) outperformed its biomedical counterpart OpenBioLLM-8B (18%). BioMistral-7B (28%) underperformed compared to Mistral-7B-Instruct-v0.2. In contrast med42-70B and JSL-MedMNX-7B-SFT performed better than their respective baseline models.
For the NEJM cases, OpenBioLLM-70B, and Llama-3-70B-Instruct both achieved 74% accuracy. Llama-3-8B-Instruct (64%) again outperformed OpenBioLLM-8B (30%). JSL-MedMNX-7B-SFT and Mistral-7B-Instruct-v0.2 performed similarly (47% and 46%, respectively). BioMistral-7B (37%) again underperformed compared to its base model. Meditron-7b showed poor performance on both datasets. Table 1 provides an overview of model accuracy in the clinical case vignettes. Figure 1 provide an overview of the performance on all evaluated tasks.
Table 1.
Individual results on the clinical case challenges.a
| JAMA Case Challenges | NEJM Case Challenges | |
|---|---|---|
| Llama 2 Models | ||
| Llama-2-7b-chat-hf | 44.3% (36%-53%) | 26.8% (22%-32%) |
| Llama-2-70b-chat-hf | 45.7% (38%-54%) | 43.5% (38%-49%) |
| Meditron-7B | 12.9% (7%-18%) | 4.9% (3%-7%) |
| ClinicalCamel-70b | 49.3% (41%-57%) | 38.9% (34%-44%) |
| Med42-70b | 52.9% (44%-61%) | 56.5% (51%-61%) |
| Llama 3 Models | ||
| Llama-3-8B-Instruct | 57.1% (49%-65%) | 64.3% (59%-69%) |
| Llama-3-70B-Instruct | 65% (57%-73%) | 74.6% (70%-79%) |
| OpenBioLLM-8B | 17.9% (12%-24%) | 30% (25%-35%) |
| OpenBioLLM-70B | 66.4% (59%-74%) | 74.1% (70%-78%) |
| Mistral -7B Models | ||
| Mistral-7B-Instruct-v0.2 | 52.1% (44%-60%) | 46.4% (41%-52%) |
| JSL-MedMNX-7B-SFT | 53.6% (45%-63%) | 46.7% (41%-52%) |
| Biomistral-7B | 27.9% (21%-35%) | 37.2% (32%-42%) |
This table presents the performance of several LLMs on clinical case challenges from JAMA and NEJM. The results are shown as percentages, with 95% confidence intervals in parentheses. Models are grouped by their base architecture (Llama 2, Llama 3, and Mistral-7B), and include both general-purpose and biomedical-specific variants. The highest scores for each model group are highlighted in bold.
Figure 1.
Comparative performance of selected general-purpose vs biomedical LLMs with longer context size. This figure presents heatmaps showing the performance of selected general-purpose and biomedical LLMs across various clinical tasks. The models are divided into two panels based on size: (A) (7B/8B models) and (B) (70B models). The y-axis lists the LLMs, with general-purpose models (eg, Llama-3-70B-Instruct, Llama-3-8B-Instruct, Mistral-7B-Instruct-v0.2) displayed first, followed by biomedical models (eg, OpenBioLLM-70B, OpenBioLLM-8B, JSL-MedMNX-7B-SFT, Biomistral-7B). The x-axis represents the clinical tasks, including JAMA Cases, NEJM Cases, MedNLI, Problem Summary, MeQSum, LongHealth, MeDiSumQA, and MeDiSumCode. The numbers in each cell correspond to performance scores for each model-task pair, with higher scores reflecting better performance (maximum score: 100). Separate color scales are used for each panel to ensure visibility of fine-tuning gains within each size group, avoiding scale-related distortions between smaller and larger models.
MedNLI task
OpenBioLLM-70B achieved the highest accuracy with 80.85%, closely followed by the generalist model Llama-3-70B-Instruct (79.37%) and JSL-MedMNX-7b-SFT (79.3%). Notably, OpenBioLLM-8B (44.93%) underperformed compared to Llama-3-8B-Instruct (74.08%). BioMistral-7B (62.75%) also showed lower accuracy than its base model Mistral-7B-Instruct-v0.2 (69.93%).
ProblemSummary
Llama-3-70B-Instruct outperformed all models across ROUGE and BERT F1 scores. OpenBioLLM-70B showed lower performance than its base model, particularly in ROUGE scores. Similarly, JSL-MedMNX-7b-SFT and BioMistral-7B achieved lower scores than Mistral-7B-Instruct-v0.2. Also, Meditron-7B and ClinicalCamel-70B each achieved lower scores than their generalist models counterparts.
MeQSum
Llama-2-7B-chat-hf achieved the highest BERT F1 score (75.98) on MeQSum, slightly outperforming larger and more recent models. All biomedical LLMs underperformed compared to their respective generalist models across all metrics.
LongHealth
Models based on Llama 2 did not support enough context to be evaluated on LongHealth, MeDiSumQA, and MeDiSumCode. Llama-3-70B-Instruct demonstrated superior performance across all LongHealth tasks. OpenBioLLM-70B showed lower scores than its base model, particularly in Task 3, which evaluates how prone to hallucinating non-existing information models are. JSL-MedMNX-7b-SFT underperformed compared to Mistral-7B-Instruct-v0.2, except for Task 3, where JSL-MedMNX-7b-SFT achieved a higher score than Mistral-7B-Instruct-v0.2. BioMistral-7B showed consistently worse performance compared to Mistral-7B-Instruct-v0.2.
MeDiSumQA
Llama-3-70B-Instruct achieved the highest scores across all MeDiSumQA metrics. OpenBioLLM-70B showed comparable but slightly lower performance. BioMistral-7B and JSL-MedMNX-7b-SFT both underperformed compared to Mistral-7B-Instruct-v0.2, with notably lower ROUGE and BERT F1 scores.
MeDiSumCode
Llama-3-70B-Instruct significantly outperformed all other models in MeDiSumCode, achieving the highest scores across all metrics. OpenBioLLM-70B showed lower performance than its base model, particularly in EM F1 and AP F1 scores. BioMistral-7B underperformed compared to Mistral-7B-Instruct-v0.2 (68.76), while JSL-MedMNX-7b-SFT (68.48) showed similar performance to its base model in Valid Code Accuracy.
Tables 2 and 3 show individual metrics of all tasks except clinical case vignettes. Figure 1 shows the comparative performance of the models across tasks. Figure 2 provides an overview of all models and their metrics on the individual tasks.
Table 2.
Results on MedNLI, ProblemSummary and MeQSum.a
| MedNLI | ProblemSummary |
MeQSum |
|||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Mean score | Acc | R-L | R-1 | R-2 | BERT F1 | UMLS F1 | R-L | R-1 | R-2 | BERT F1 | |
| Llama 1 Models | |||||||||||
| MedAlpaca 7B | 27.72 | 22.8 (20.5-25.0) | 14.2 (13.0-15.4) | 17.6 (16.2-19.1) | 5.7 (4.7-6.6) | 66.4 (65.8-67.1) | 18.3 (16.3-20.3) | 28.5 (27.1-29.8) | 31.5 (30.2-32.8) | 14.5 (13.4-15.7) | 69.2 (68.4-70.0) |
| PMC-Llama-7B | 17.31 | 21.2 (19.1-23.2) | 8.4 (7.3-9.4) | 10.6 (9.4-11.8) | 3.4 (2.7-4.0) | 54.4 (52.0-56.6) | 14.2 (12.3-16.2) | 6.0 (6.7-6.4) | 6.6 (6.3-7.0) | 2.1 (1.9-2.3) | 35.4 (34.8-35.9) |
| Llama 2 Models | |||||||||||
| Llama-2-7b-chat-hf | 36.91 | 41.2 (38.7-43.8) | 17.5 (16.0-18.9) | 22.3 (20.5-23.9) | 6.9 (5.6-8.2) | 66.3 (65.6-67.0) | 21.7 (19.5-23.9) | 36.4 (35.2-37.7) | 39.9 (38.7-41.2) | 18.1 (16.9-19.2) | 76.0 (75.4-76.5) |
| Llama-2-70b-chat-hf | 42.9 | 61.7 (59.3-64.0) | 14.4 (13.5-15.4) | 19.8 (18.5-21.0) | 6.1 (5.4-6.9) | 65.1 (64.5-65.6) | 21.4 (19.5-23.2) | 34.9 (33.7-36.2) | 38.8 (37.6-40.0) | 18.5 (17.3-19.6) | 74.3 (73.8-74.9) |
| Meditron-7B | 13.04 | 2.4 (1.6-3.2) | 11.3 (10.4-12.2) | 13.4 (12.1-14.7) | 4.8 (4.1-5.6) | 63.4 (62.6-64.1) | 15.2 (13.3-17.1) | 6.8 (6.4-7.2) | 7.9 (7.5-8.3) | 2.3 (2.0-2.5) | 43.4 (42.9-43.8) |
| ClinicalCamel-70b | 35.74 | 64.7 (62.2-67.2) | 8.7 (7.3-10.0) | 10.9 (9.3-12.5) | 3.7 (2.9-4.6) | 60.0 (59.0-60.9) | 12.4 (10.6-14.2) | 16.9 (15.7-18.2) | 18.8 (17.4-20.2) | 8.5 (7.4-9.6) | 49.3 (48.0-50.6) |
| Llama 3 Models | |||||||||||
| Llama-3-8B-Instruct | 48.38 | 74.1 (71.7-76.4) | 22.7 (21.3-24.3) | 28.5 (26.8-30.3) | 9.9 (8.6-11.2) | 71.5 (70.7-72.2) | 25.3 (23.1-27.5) | 32.2 (31.1-33.3) | 36.5 (35.3-37.6) | 16.4 (15.3-17.4) | 72.7 (72.2-73.3) |
| Llama-3-70B-Instruct | 52.38 | 79.4 (77.2-81.5) | 25.5 (23.8-27.2) | 33.2 (31.2-35.2) | 13.0 (11.4-14.7) | 73.0 (72.2-73.8) | 29.2 (26.8-31.6) | 36.6 (35.4-37.7) | 40.2 (38.9-41.4) | 19.3 (18.1-20.4) | 75.7 (75.2-76.3) |
| OpenBioLLM-8B | 33.20 | 44.9 (42.1-47.6) | 10.9 (9.7-12.0) | 13.7 (12.3-15.1) | 4.0 (3.3-4.8) | 64.2 (63.5-64.9) | 15.7 (14.0-17.5) | 26.2 (24.9-27.5) | 29.4 (28.0-30.7) | 14.0 (13.0-15.0) | 62.4 (60.8-63.9) |
| OpenBioLLM-70B | 47.54 | 80.8 (78.8-82.9) | 12.1 (11.0-13.2) | 16.6 (15.2-18.0) | 5.6 (4.8-6.3) | 66.5 (65.9-67.2) | 17.7 (16.0-19.4) | 30.7 (29.5-31.8) | 34.3 (33.2-35.5) | 15.5 (14.5-16.5) | 72.0 (71.4-72.6) |
| Med 42 | 47.26 | 75.4 (73.3-77.7) | 19.8 (18.3-21.3) | 25.1 (23.5-26.8) | 7.8 (6.7-8.8) | 67.8 (67.1-68.5) | 26.3 (24.1-28.4) | 29.3 (27.9-30.7) | 32.0 (30.6-33.5) | 15.9 (14.6-17.2) | 70.9 (70.2-71.7) |
| Mistral-7B Models | |||||||||||
| Mistral-7B-Instruct-v0.2 | 46.47 | 70.0 (67.4-72.3) | 19.6 (18.2-21.0) | 25.6 (23.8-27.3) | 8.9 (7.6-10.2) | 69.6 (68.9-70.4) | 22.1 (19.8-24.4) | 33.5 (32.3-34.8) | 37.4 (36.3-38.6) | 16.6 (15.5-17.6) | 73.5 (72.9-74.0) |
| JSL-MedMNX-7B-SFT | 49.40 | 79.3 (77.1-81.5) | 19.5 (18.2-20.9) | 25.6 (23.9-27.3) | 8.2 (7.1-9.4) | 67.6 (66.9-68.3) | 25.7 (23.6-28.0) | 32.5 (31.3-33.6) | 36.6 (35.4-37.7) | 16.2 (15.2-17.3) | 73.0 (72.5-73.5) |
| Biomistral-7B | 40.59 | 62.8 (60.1-65.2) | 16.9 (14.9-18.9) | 20.8 (18.4-23.2) | 8.4 (6.9-9.8) | 59.0 (55.3-62.4) | 20.2 (17.8-22.7) | 25.9 (24.9-26.9) | 28.5 (27.4-29.5) | 13.3 (12.4-14.2) | 67.9 (67.3-68.6) |
This table displays the performance of different LLMs on three clinical tasks: MedNLI (natural language inference), ProblemSummary (summarizing patient problems), and MeQSum (summarizing medical questions). Various metrics are reported, including accuracy (Acc), ROUGE scores (R-L, R-1, R-2), BERT F1, and UMLS F1. The mean score across all tasks is also provided. The best performance for each model group and metric is highlighted in bold.
Table 3.
Results on LongHealth. MeDiSumQA and MeDiSumCode.a
| LongHealth |
MeDiSumQA |
MeDiSumCode |
||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Mean score | Task 1 | Task 2 | Task 3 | R-L | R-1 | R-2 | BERT F1 | UMLS F1 | EM F1 | AP F1 | Valid Code Acc | |
| Llama 3 Models | ||||||||||||
| Llama-3-8B-Instruct | 40.42 | 66.7 (62.1-71.3) | 66.6 (64.6-68.7) | 56.3 (39.5-43.7) | 22.4 (21.2-23.8) | 28.2 (26.8-29.5) | 9.6 (8.6-10.7) | 68.6 (68.0-69.2) | 22.7 (20.8-24.6) | 3.9 (3.2-4.6) | 17.5 (16.0-19.0) | 61.9 (59.5-64.4) |
| Llama-3-70B-Instruct | 56.08 | 82.3 (78.5-85.8) | 77.9 (76.1-79.8) | 91.7 (90.5-92.9) | 26.2 (24.9-27.6) | 32.5 (31.1-34.0) | 12.0 (10.8-13.1) | 70.3 (69.6-70.9) | 25.8 (23.8-27.6) | 19.6 (18.2-21.0) | 39.2 (37.6-40.7) | 93.9 (92.9-94.9) |
| OpenBioLLM-8B | 25.37 | 37.0 (32.2-41.6) | 41.8 (39.6-44.0) | 1.6 (1.0-2.1) | 22.9 (21.5-24.3) | 28.0 (26.3-29.5) | 10.4 (9.2-11.7) | 68.7 (68.0-69.4) | 22.1 (20.2-24.0) | 0.8 (0.6-1.1) | 4.8 (4.0-5.7) | 51.1 (47.0-55.2) |
| OpenBioLLM-70B | 45.64 | 81.2 (77.3-84.7) | 75.6 (73.7-77.5) | 62.9 (60.7-65.0) | 21.85 (20.6-23.0) | 27.8 (26.5-29.1) | 9.5 (8.5-10.5) | 68.4 (67.8-69.0) | 22.4 (20.6-24.2) | 7.3 (6.6-8.0) | 20.2 (19.1-21.3) | 73.6 (71.6-75.6) |
| Mistral-7B Models | ||||||||||||
| Mistral-7B-Instruct-v0.2 | 36.4 | 65.3 (60.5-70.0) | 56.0 (51.1-60.9) | 28.0 (23.4-32.6) | 21.8 (20.6-22.9) | 27.5 (26.2-28.8) | 9.2 (8.3-10.1) | 68.4 (67.0-69.0) | 20.3 (18.6-21.9) | 3.1 (2.6-3.6) | 18.2 (16.9-19.5) | 68.7 (66.9-70.5) |
| JSL-MedMNX-7B-SFT | 33.86 | 51.7 (46.8-56.6) | 40.2 (38.1-42.3) | 52.2 (50.1-54.4) | 15.9 (15.0-16.9) | 20.9 (19.8-22.0) | 6.7 (6.0-7.4) | 65.9 (65.4-66.4) | 17.3 (16.0-18.6) | 2.8 (2.2-3.4) | 13.2 (11.9-14.6) | 68.6 (65.9-71.4) |
| Biomistral-7B | 23.94 | 39.0 (34.1-43.9) | 34.2 (32.2-36.2) | 7.8 (6.6-9.0) | 14.6 (13.4-15.8) | 17.8 (16.4-19.2) | 5.5 (4.6-6.3) | 59.0 (57.2-60.7) | 16.8 (14.9-18.5) | 1.7 (1.2-2.1) | 9.9 (8.6-11.1) | 54.6 (51.3-58.0) |
This table presents the performance of selected LLMs on three additional clinical tasks: LongHealth (handling long clinical documents), MeDiSumQA (question answering based on medical summaries), and MeDiSumCode (medical coding). For LongHealth, results are broken down into three subtasks. MeDiSumQA reports ROUGE scores, BERT F1, and UMLS F1. MeDiSumCode includes metrics specific to medical coding tasks. The highest scores for each model group and metric are highlighted in bold.
Figure 2.
Comparative performance of the domain specific and general-purpose base models on the two clinical case vignette series (A), LongHealth (B), MeDiSumQA (C), MedNLI (D), MeQSum (E), ProblemSummary (F), and MeDiSumCode (G). Bars represent the respective metric and error bars represent 95% confidence intervals obtained through bootstrapping.
Discussion
Our comprehensive evaluation of LLMs across clinical tasks has yielded unexpected insights regarding the performance of biomedically fine-tuned models. Contrary to our initial hypothesis, biomedical models generally underperformed compared to their general-purpose counterparts across various tasks. This suggests that fine-tuning LLMs on biomedical data may not provide the expected benefits and may even decrease rather than improve performance.
Several factors may contribute to this underperformance. The superior performance of general-purpose models might stem from their exposure to a more diverse range of topics and linguistic structures during pre-training. This broader knowledge base could enable more flexible reasoning and better generalization to novel tasks. Additionally, the fine-tuning process for biomedical models might inadvertently introduce biases or overly narrow the models’ focus, potentially limiting their ability to integrate broader contextual information crucial for complex clinical reasoning.
One explanation for the discrepancy between the benchmark results of the biomedical model releases and our own could be overfitting to specific medical datasets during fine-tuning, leading to reduced generalization capabilities. This could be due to data leakage between training and test sets,26 a risk that increases with the growing size of training datasets, making it increasingly challenging for researchers to verify data integrity, especially as some models evaluated (OpenBioLLM or JSL-MedMNX-7b-SFT) do not report their training data. Overfitting can also occur indirectly through repeated evaluation on common test datasets, such as USMLE or MMLU, which may inadvertently select for models that perform well on these specific benchmarks.
A key factor to consider is the potential loss of general knowledge during the fine-tuning process, a phenomenon known as catastrophic forgetting.14 Our findings highlight the delicate balance required when adapting general-purpose models to specialized domains without compromising their broad capabilities. Potential evidence supporting this hypothesis comes from the observation that biomedical LLMs that only underwent supervised fine-tuning (ClinicalCamel-70B, OpenBioLLM-8B/70B)7,25,27 showed a smaller performance decrease compared to their general-purpose counterparts than models that underwent continued pretraining (BioMistral-7B, Meditron-7B).8,24
Interestingly, larger models exhibited smaller performance gaps between biomedical and generalist versions. Given that both were trained with the same amount of data, the overall changes in model weights of larger models might have been smaller, potentially reducing the risk of catastrophic forgetting. These observations suggest that using only fine-tuning, rather than continued pretraining, may be preferable when adapting LLMs for specific domains. However, the data presented in our analysis is insufficient for definitive conclusions, and further research is needed to investigate these phenomena thoroughly.
The consistent strong performance of Llama-3-70B-Instruct across all benchmarks is particularly noteworthy. As the latest iteration of Llama models, Llama 3 was trained on an unprecedented 15 trillion tokens of data, a 7-fold increase compared to Llama 2 and substantially more than previously believed optimal.22,23,28 With this vast amount of data, it is highly likely that nearly all freely available biomedical texts on the internet are included in the training data of these general-purpose models, enabling them to inherently capture sufficient medical knowledge. Consequently, unless biomedical LLMs are fine-tuned on novel, previously unavailable data (eg, copyrighted scientific papers, proprietary hospital data), fine-tuning on publicly accessible biomedical data may not add new knowledge. Instead, it may risk the model forgetting valuable information through continued fine-tuning.
Performance differences between biomedical and general LLMs vary depending on the task. While the performance of OpenBioLLM 70B and Llama3 70B appears to be on par for the clinical vignettes, which have a uniform multiple-choice format, OpenBioLLM performed significantly worse than Llama3 70B on the MeDiSumCode benchmark, which requires in-depth knowledge of the ICD coding system with over 70 000 individual codes. This discrepancy suggests that the benefits of biomedical fine-tuning may be task-dependent, with potentially greater advantages in highly specialized medical tasks that require deep domain knowledge.
One critical finding of our study is the higher risk of hallucinations observed in biomedical LLMs. The results from LongHealth Task 3, which evaluates hallucination tendencies, raise important concerns about the reliability of LLMs in clinical applications. The superior performance of general-purpose models in this aspect is particularly intriguing and warrants further investigation. Recent work has highlighted the critical nature of this issue in healthcare AI, emphasizing the need for robust strategies to mitigate hallucination risks.29,30
These results could have significant implications for the development and application of LLMs in healthcare. They challenge the prevailing assumption that domain-specific fine-tuning is universally beneficial for specialized tasks. Instead, our findings suggest that the relationship between model performance and domain adaptation is more nuanced and complex than previously thought. This complexity may stem from the intricate interplay between a model’s general knowledge and its ability to apply that knowledge in specific contexts.
Interestingly, larger models exhibited smaller performance gaps between biomedical and generalist versions. Given that both were trained with the same amount of data, the overall changes in model weights of larger models might have been smaller, potentially reducing the risk of catastrophic forgetting. These observations suggest that using only fine-tuning, rather than continued pretraining, may be preferable when adapting LLMs for specific domains. However, further research is needed to investigate these observations more thoroughly.
Limitations
Our comparison has limitations. The two sets of case vignettes have been freely available on the web and might thus have partly been included in the training data of recent LLMs, leading to an overestimation of the LLM performance. However, since the domain-specific models are based on the general-purpose LLMs, this would not be an advantage. Rather, the inferior performance of the biomedical LLMs, could be an additional argument for the presence of catastrophic forgetting. Furthermore, while the benchmarks used cover a range of clinical tasks, they may not fully represent the complexity and diversity of real-world clinical scenarios. Specifically, they do not cover detailed medical knowledge such as nuanced diagnostic criteria, extensive patient history considerations, and comprehensive treatment recommendations. In addition, diagnosis in complicated cases typically involves an iterative process of interpreting and acquiring additional diagnostic information about the patient, a process that is not easily replicated in the multiple-choice question format.
Conclusion
In conclusion, our study challenges prevailing assumptions about the effectiveness of biomedical fine-tuning for LLMs, with potential implications for domain-specific adaptation in general. Rather than continued pre-training or fine-tuning, alternative approaches such as retrieval-augmented generation are worth exploring to enhance the biomedical capabilities of LLMs without compromising their general knowledge. Recent studies have shown promising results for these techniques.31,32
While biomedical LLMs perform well on widely used benchmarks such as the USMLE or MMLU, our evaluation reveals variable performance across tasks and models. This highlights the need for more rigorous, task-specific evaluation frameworks for healthcare LLMs. These evaluations should focus on clinical support tasks such as text summarization, information retrieval, and data structuring. We believe that using LLMs for these applications could provide more noticeable relief to healthcare workers than using LLMs as (potentially unreliable) knowledge bases.
Supplementary Material
Contributor Information
Felix J Dorfner, Charité—Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin 10117, Germany; Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital and Harvard Medical School, Charlestown, MA 02129, United States.
Amin Dada, Institute for AI in Medicine (IKIM), University Hospital Essen (AöR), Essen 45131, Germany.
Felix Busch, Department of Radiology, Klinikum Rechts Der Isar, Technical University Munich, Munich 81675, Germany.
Marcus R Makowski, Department of Radiology, Klinikum Rechts Der Isar, Technical University Munich, Munich 81675, Germany.
Tianyu Han, Department of Diagnostic and Interventional Radiology, University Hospital Aachen, Aachen 52074, Germany.
Daniel Truhn, Department of Diagnostic and Interventional Radiology, University Hospital Aachen, Aachen 52074, Germany.
Jens Kleesiek, Institute for AI in Medicine (IKIM), University Hospital Essen (AöR), Essen 45131, Germany; Cancer Research Center Cologne Essen (CCCE), West German Cancer Center Essen, University Hospital Essen (AöR), Essen 45147, Germany; German Cancer Consortium (DKTK, Partner Site Essen), Heidelberg, Germany; Department of Physics, TU Dortmund, Dortmund 44227, Germany.
Madhumita Sushil, Bakar Computational Health Sciences Institute, University of California, San Francisco, San Francisco, CA 94158, United States.
Lisa C Adams, Department of Radiology, Klinikum Rechts Der Isar, Technical University Munich, Munich 81675, Germany.
Keno K Bressem, Department of Radiology, Klinikum Rechts Der Isar, Technical University Munich, Munich 81675, Germany; German Heart Center Munich, Technical University Munich, Munich 80636, Germany.
Author contributions
Felix J. Dorfner (Conceptualization, Formal analysis, Investigation, Methodology, Visualization), Amin Dada (Methodology), Marcus R. Makowski (Resources), Daniel Truhn (Methodology), Lisa Adams (Project administration, Resources, Supervision, Visualization), and Keno K. Bressem (Conceptualization, Formal analysis, Investigation, Methodology, Project administration, Visualization)
Supplementary material
Supplementary material is available at Journal of the American Medical Informatics Association online.
Funding
This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. LCA is a is an Albrecht-Struppler-Clinician Scientist Fellow, funded by the Federal Ministry of Education and Research (BMBF) and the Free State of Bavaria under the Excellence Strategy of the Federal Government and the Länder, as well as by the Technical University of Munich - Institute for Advanced Study.
Conflicts of interest
None declared.
Data availability
The datasets analyzed during the current study are publicly available. Publicly available benchmark datasets, such as MIMIC-III and MIMIC-IV, can be accessed through the PhysioNet repository (https://physionet.org/).
References
- 1. Eriksen AV, Möller S, Ryg J. Use of GPT-4 to Diagnose Complex Clinical Cases. Massachusetts Medical Society; 2023.
- 2. Rajpurkar P, Chen E, Banerjee O, Topol EJ. AI in health and medicine. Nat Med. 2022;28:31-38. [DOI] [PubMed] [Google Scholar]
- 3. Thirunavukarasu AJ, Ting DSJ, Elangovan K, Gutierrez L, Tan TF, Ting DSW. Large language models in medicine. Nat Med. 2023;29:1930-1940. [DOI] [PubMed] [Google Scholar]
- 4. Goh E, Gallo R, Hom J, et al. Large language model influence on diagnostic reasoning: a randomized clinical trial. JAMA Netw Open. 2024;7:e2440969. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Blease CR, Locher C, Gaab J, Hägglund M, Mandl KD. Generative artificial intelligence in primary care: an online survey of UK general practitioners. BMJ Health Care Inf. 2024;31:e101102. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Lee J, Yoon W, Kim S, et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics. 2020;36:1234-1240. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Pal A, Openbiollms M. Advancing Open-Source Large Language Models for Healthcare and Life Sciences. Hugging Face Repository; 2024.
- 8. Labrak Y, Bazoge A, Morin E, Gourraud P-A, Rouvier M. Biomistral: a collection of open-source pretrained large language models for medical domains, arXiv, arXiv:2402.10373, 2024, preprint: not peer reviewed.
- 9. Wu C, Lin W, Zhang X, Zhang YA, Xie W, Wang Y. PMC-LLaMA: toward building open-source language models for medicine. J Am Med Inform Assoc. 2024;31:1833-1843. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Christophe C, Kanithi PK, Munjal P, et al. Med42—evaluating fine-tuning strategies for medical llms: full-parameter vs parameter-efficient approaches, arXiv, arXiv:2404.14779, 2024, preprint: not peer reviewed.
- 11. Han T, Adams LC, Bressem KK, Busch F, Nebelung S, Truhn D. Comparative analysis of multimodal large language model performance on clinical vignette questions. JAMA. 2024;331:1320-1321. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Jiang AQ, Sablayrolles A, Mensch A, et al. Mistral 7B, arXiv, arXiv:2310.06825, 2023, preprint: not peer reviewed.
- 13. Kung TH, Cheatham M, Medenilla A, et al. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLoS Digit Health. 2023;2:e0000198. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Golchin S, Surdeanu M. Time travel in llms: tracing data contamination in large language models, arXiv, arXiv:2308.08493, 2023, preprint: not peer reviewed.
- 15. Kirkpatrick J, Pascanu R, Rabinowitz N, et al. Overcoming catastrophic forgetting in neural networks. Proc Natl Acad Sci USA. 2017;114:3521-3526. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Dada A, Bauer M, Contreras AB, et al. CLUE: a clinical language understanding evaluation for LLMs, arXiv, arXiv:2404.04067, 2024, preprint: not peer reviewed.
- 17. Johnson AE, Pollard TJ, Shen L, et al. MIMIC-III, a freely accessible critical care database. Sci Data. 2016;3:160035-160039. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18. Romanov A, Shivade C. Lessons from natural language inference in the clinical domain, arXiv, arXiv:1808.06752, 2018, preprint: not peer reviewed.
- 19.On the summarization of consumer health questions. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics; 2019.
- 20.Summarizing patients’ problems from hospital progress notes using pre-trained sequence-to-sequence models. In: Proceedings of COLING. International Conference on Computational Linguistics. NIH Public Access; 2022. [PMC free article] [PubMed]
- 21. Adams L, Busch F, Han T, et al. LongHealth: a question answering benchmark with long clinical documents, arXiv, arXiv:2401.14490, 2024, preprint: not peer reviewed.
- 22. Touvron H, Martin L, Stone K, et al. Llama 2: open foundation and fine-tuned chat models, arXiv, arXiv:2307.09288, 2023, preprint: not peer reviewed.
- 23. Meta A. Introducing meta llama 3: the most capable openly available llm to date Meta AI. 2024. [Google Scholar]
- 24. Chen Z, Cano AH, Romanou A, et al. Meditron-70b: scaling medical pretraining for large language models, arXiv, arXiv:2311.16079, 2023, preprint: not peer reviewed.
- 25. Toma A, Lawler PR, Ba J, Krishnan RG, Rubin BBClinical camel: an open expert-level medical language model with dialogue-based knowledge encoding, arXiv, arXiv:2305.12031, 2023, preprint: not peer reviewed.
- 26. Balloccu S, Schmidtová P, Lango M, Leak DO. cheat, repeat: data contamination and evaluation malpractices in closed-source llms, arXiv, arXiv:2402.03927, 2024, preprint: not peer reviewed.
- 27. Han T, Adams LC, Papaioannou J-M, et al. MedAlpaca—an open-source collection of medical conversational AI models and training data, arXiv, arXiv:2304.08247, 2023, preprint: not peer reviewed.
- 28. Hoffmann J, Borgeaud S, Mensch A, et al. Training compute-optimal large language models, arXiv, arXiv:2203.15556, 2022, preprint: not peer reviewed.
- 29. Pal A, Umapathi LK, Sankarasubbu M. Med-halt: medical domain hallucination test for large language models, arXiv, arXiv:2307.15343, 2023, preprint: not peer reviewed.
- 30. Ahmad MA, Yaramis I, Roy TD. Creating trustworthy llms: dealing with hallucinations in healthcare AI, arXiv, arXiv:2311.01463, 2023, preprint: not peer reviewed.
- 31. Lewis P, Perez E, Piktus A, et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. Adv Neural Inf Process Syst. 2020;33:9459-9474. [Google Scholar]
- 32. Xiong G, Jin Q, Lu Z, Zhang A. Benchmarking retrieval-augmented generation for medicine, arXiv, arXiv:2402.13178, 2024, preprint: not peer reviewed.
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
The datasets analyzed during the current study are publicly available. Publicly available benchmark datasets, such as MIMIC-III and MIMIC-IV, can be accessed through the PhysioNet repository (https://physionet.org/).