The evaluation illusion of large language models in medicine

Abstract

While large language models (LLMs) hold promise for transforming clinical healthcare, current comparisons and benchmark evaluations of large language models in medicine often fail to capture real-world efficacy. Specifically, we highlight how key discrepancies arising from choices of data, tasks, and metrics can limit meaningful assessment of translational impact and cause misleading conclusions. Therefore, we advocate for rigorous, context-aware evaluations and experimental transparency across both research and deployment.

Large language models (LLMs) hold transformative potential for re-imagining complex workflows in medicine including clinical decision support, health record documentation and retrieval, and patient communication^1–3. However, the open-ended generation capabilities facilitated by foundation models can be nuanced to evaluate, particularly in medicine. Many current experimental designs fall short of adequately comparing and validating the real-world utility, safety, and contextual effectiveness of both models and end-to-end tools. The breakneck speed of LLM research, translation, and dissemination has often outpaced rigorous evaluation and review, which can lead to a disconnect between underlying technological capabilities and public perception. As the community of authors, reviewers, consumers, and purveyors of clinical NLP research expands exponentially, it is imperative that this community be able to precisely communicate and reason about claims on LLM performance in clinical settings. For example, the lack of rigorous empirical evaluations can led to misleading conclusions about whether bespoke and clinically fine-tuned LLMs outperform general-purpose LLMs^4,5. Achieving this goal requires being upfront about the research questions a specific experiment design is equipped to address, factoring in the common gaps between evaluation in research and what’s required for robust workflow integration. In this work, we describe key discrepancies arising in current evaluation of LLMs in medicine and the real-world desiderata for robust evaluation (Fig. 1). These observations are tailored towards autoregressive LLMs, both general-purpose and medically fine-tuned⁶.

Fig. 1. Overview of the four major discrepancies in current evaluation of clinical large language models.

Discrepancies in data, tasks, automated metrics, and translational impact between the current status quo and real-world deployment lead to insufficient or misleading evaluations of large language models in medical contexts.

Discrepancy in data

A strong indicator of the potential for the success of a clinical LLM tool, irrespective of the specific clinical task, is whether the evaluation data reflects the real-world data on which the tool will be deployed⁷. A recent survey across a broad range of clinical tasks found that only 5% of recent evaluations of LLMs in medicine have been conducted on actual electronic health record (EHR) data⁸. The rest were evaluated on benchmarks consisting of data like clinical vignettes, subject matter expert-generated questions, or multiple choice medical question-answering exams. However, real EHR data comes with the added complexity of clinical jargon, fragmentation in data collection, heterogeneity across healthcare institutions, and harder-to-process formats (e.g., scanned faxes). Moreover, evaluation datasets themselves introduce bias through patient population selection, institutional practices, and data collection protocols that may not represent the diversity of real-world deployment contexts⁹. These dataset biases can systematically favor certain model architectures or training approaches while masking performance disparities across demographic groups¹⁰. For most tasks in medicine, having all relevant clinical information synthesized into a short paragraph would be highly unrealistic and time-consuming to produce.

Discrepancy in tasks

Using EHR data alone is insufficient for construct validity, i.e., measuring the underlying target of evaluation¹¹. For example, the popular MedNLI benchmark aims to measure how well models can interpret EHR text; given a pair of clinical sentences (a premise and a hypothesis about a patient), the goal is to see whether the hypothesis can be inferred, refuted, or neither by the premise¹². However, using only the hypothesis and no premise (which should be equivalent to random), one can achieve approximately double the F1 score of a random baseline, indicating significant artifacts in the dataset¹³. The most popular category of benchmarks are multiple-choice exams, whose questions by design must include all relevant information¹⁴. However, this doesn’t mirror clinical practice, in which additional measurements must be continuously elicited by the clinician¹⁵; unfortunately, language models have been shown to have significantly poorer performance in partial information scenarios in which they need to interactively uncover more evidence for diagnosis^16,17. Further exacerbating this discrepancy is the fact that high performance on multiple-choice benchmarks can be driven by flawed rationales¹⁸.

Discrepancy in automated metrics

Researchers often continue to rely on automated metrics as a source of evaluation for open-ended generation tasks. While such metrics from the model development process may serve as a valuable first step–for example to filter out tools that may not meet the most preliminary utility threshold—reliance on crude criterion is misguided. There is strong evidence to believe that accuracy measured using automated metrics lifted from the natural language processing (NLP) literature, such as ROUGE, BLEU, BERTScore, and its variants^19–21 do not reflect accuracy when measured via human evaluations across many clinical text generation tasks. To quantify this notion, we conducted a scoped literature search that measure how well automated metrics correlate with human perceptions for clinical summarization tasks. Papers were initially filtered from arXiv, the most popular computer science archive, and Google Scholar, based on relevant terms in the title and abstract. For full literature search details, see Supplementary Note 1 and Tables 1, 2. We report the correlations for three commonly used metrics across the dimensions of completeness and correctness in Table 1; further dimensions and metrics can be found in the supplemental table. Unfortunately, many evaluations found that automated metrics only loosely reflected human preference, with correlations sometimes even being negative. Similarly, automated readability scores (e.g., based on syllables per word) have correlations of 0.1–0.3 with layperson perception²². This suggests that even preliminary evaluations warrant small-scale human-evaluations²³. While the clinical NLP community has explored improved measures and metrics (e.g., DocLens), we present a call to the broader NLP community to test out these new measures of accuracy across use cases and develop better metrics for tasks like readability²⁴.

Table 1.

Literature search on concordance of common automated metrics for clinical summarization

Metric Name	Dimension	Pearson Correlation	Spearman Correlation
BERTScore	Completeness	0.2832, 0.4433	0.15–0.234, 0.64535
	Correctness	0.2332, 0.5233, 0.02236	0.15–0.234, 0.53035, −0.7736
BLEU	Completeness	0.2232	0.2–0.2534, 0.68535
	Correctness	0.1332	0.1–0.1534, 0.44735
ROUGE	Completeness	0.3032, 0.4233, 0.47937	0.32624, −0.38924, 0.2–0.2534, 0.61035
	Correctness	0.1632, 0.5333, −0.0136, 0.41637	0.15–0.234, 0.47935, −0.6636

Correlation of completeness and correctness measures between human evaluators and three common automated metrics (BERTScore, BLEU, and ROUGE), as measure on clinical summarization tasks.

Several benchmarking efforts, such as HealthBench²⁵, MedArena²⁶, have recently focused on grounded rubric-based or preference-based human-in-the-loop evaluations as an alternative. However, challenges persist. For example, HealthBench consists largely of synthetically expert-generated examples; evaluation requires custom rubrics per example, which is difficult to scale more broadly, particularly considering changing guidelines and complex patient context.

Discrepancy in translational impact

Finally, there is a disconnect between the dimensions emphasized in current evaluations and the potential for utility of LLM deployments in healthcare. For example, for in-basket messaging applications, preliminary results have shown benefits not in saved time for drafting responses, the primary motivation for the pilot studies, but in the perceived cognitive burden of the task^27–29, for which tangible and standardized metrics have not been established. Current human evaluations fall short of rigorous experiment design to ensure that studies can surface relevant insights (tangibly measured with statistical significance) on all evaluation targets, such as utility, edge cases, across clinical domains and specialties of interest, patient representation, etc. There is a broad swath of classical NLP and informatics literature emphasizing the need for extrinsic evaluations, both in-situ and in-vivo evaluations^30,31. While an intrinsic evaluation might measure if a summary is accurate, an extrinsic evaluation could test whether real-world clinicians’ workflows are improved in time and accuracy with access to the summary. For question-answering and broadly close-ended applications of clinical LLMs, we emphasize that the bar for evidence must be higher and require a discussion of failure modes, and rare edge cases to help the broader community concretely operationalize potential safety features, enabling the healthcare AI research community to triage algorithmic improvements. While this may entail increased cognitive burden and deliberation for staff responsible for evaluations, it can increase the chance of success of a tool once deployed⁷.

Bringing clinical and human context into evaluation

Clinical LLMs are unlikely to be deployed as standalone tools in the foreseeable future highlighting the need for evaluations that account for the complexity of human-AI collaborations. Here rigorous experiment design must include standard practices such as on-boarding and training of clinicians to the use of the AI tool, and transparently reporting these steps as part of the study design due to their significant impact on the downstream (perceived and actual) utility of the tool, ability of researchers and the broader healthcare community to assess a piece of evidence, and in general raising the bar for the quality of reported evidence of clinical LLM tools.

Meaningful integration of LLMs in healthcare requires a multidisciplinary approach to rigorous evaluation, with vigilance from authors, reviewers, and readers. We must incentivize researchers to be forthcoming about the set of discrepancies present in their work, not just for other researchers, but so non-computational audiences don’t have to read between the lines to infer the nuances in a piece of evidence. While every work cannot comprehensively evaluate tools end-to-end, the community has a strong mandate to be upfront and precise about the scope and limitations of experimental evidence provided. This shift requires strategic investments in interdisciplinary expertise—combining implementation scientists, clinical NLP researchers, data scientists, and healthcare practitioners—to create sophisticated, context-aware evaluation methodologies. By establishing standards that systematically validate the utility, limitations, and potential impacts of AI technologies, we can responsibly harness the transformative potential of LLMs and AI more broadly while providing grounded evaluations for impactful systems.

Author contributions

M.A., I.Y.C., F.G. and S.J. all contributed to the writing of the work. F.G. prepared Table 1.

Data availability

No datasets were generated or analyzed during the current study.

Competing interests

M.A. is a co-founder of Layer Health.

Supplementary information

The online version contains supplementary material available at 10.1038/s41746-025-01963-x.