Improving electronic health record processing of large language models via retrieval-augmented generation: A case study on dietary supplements

Abstract

Large language models (LLMs) excel in natural language processing (NLP) but struggle with domain-specific complexities in electronic health records (EHRs). We demonstrate that retrieval-augmented generation (RAG) enhances LLMs for dietary supplement (DS) information extraction. By testing models like Llama-3 with diverse retrievers on tasks including entity recognition and usage classification, task-aligned retrieval outperforms reliance on model size or specialization. Smaller general models paired with optimized retrievers match or exceed specialized counterparts-structured retrieval aids complex tasks (e.g., triple extraction), while semantic retrieval improves classification. Results challenge assumptions that larger or domain-specific models are superior, emphasizing dynamic knowledge integration over brute-force scaling. This approach offers practical strategies for clinical NLP, enabling efficient EHR analysis without massive resources. Prioritizing retrieval strategies over model size advances tools for evidence-based healthcare, highlighting adaptability and cost-effectiveness in real-world medical applications.

Introduction

The rapid advancement of large language models (LLMs)^1,2 has significantly reshaped natural language processing (NLP), driving progress across various domains.^3,4 Models such as GPT-4,⁵ Llama-3,⁶ and Mistral⁷ have demonstrated remarkable capabilities in text comprehension,⁸ reasoning,⁹ and generation,¹⁰ setting new benchmarks in general-domain NLP. This progress has also propelled advancements in medical NLP,^11–13 where LLMs are increasingly applied to clinical text processing,^14,15 biomedical literature analysis,¹⁶ and knowledge discovery.¹⁷ One of the most critical applications in this domain is information extraction (IE),^18,19 which enables the transformation of unstructured electronic health records (EHRs) into structured, actionable knowledge. Given the vast volume of clinical data generated daily, accurate IE from EHRs is essential for clinical decision support,²⁰ drug safety monitoring,²¹ and biomedical research.^22,23

Despite their strong performance on public NLP benchmarks, LLMs still exhibit significant limitations in processing EHRs.^24,25 One major challenge arises from the private nature of EHRs,²⁶ as they are rarely publicly available due to strict patient privacy regulations. This prevents LLMs from being trained directly on large-scale clinical datasets, limiting their exposure to real-world medical narratives. Additionally, EHRs are primarily authored by medical professionals—such as doctors and nurses—who employ highly specialized terminology, abbreviations, and complex sentence structures that differ from general-domain text.²⁷ Moreover, medical knowledge is constantly discovering and accumulating, and the LLM can not always keep up with the latest. As a result, even powerful LLMs often fail to extract meaningful insights from EHRs, necessitating additional strategies to enhance their adaptability.

The emergence of retrieval-augmented generation (RAG)^28,29 has introduced a promising solution to this challenge by allowing LLMs to retrieve external knowledge dynamically during inference. Rather than relying solely on their pre-trained representations, RAG enables models to access relevant domain-specific information, example cases, or contextual cues that can significantly improve task performance. This paradigm shift—from direct problem-solving to knowledge-augmented reasoning³⁰—effectively reduces the burden on LLMs and enhances their ability to generalize to unseen medical data.³¹ In the context of EHR processing, where training data is scarce and medical text is highly specialized, RAG presents a powerful technique to supplement LLMs with real-time access to task-relevant information.

Recent advancements in retrieval-augmented generation (RAG) combined with large language models (LLMs) have shown promise in electronic health record (EHR) analysis, particularly for clinical document understanding,³² medical question answering,³³ and information extraction.^34,35. However, limited attention has been given to dietary supplement (DS)-related EHR processing. While our prior work RAMIE³⁶ built up a multi-task³⁷ framework in DS-EHR analysis, critical performance limitations persist in real-world clinical settings. Notably, surveys by the Council for Responsible Nutrition (CRN) indicate that 75% of Americans use dietary supplements,^38,39 underscoring their widespread impact on health behaviors, drug interactions, and therapeutic outcomes. Despite this significance, effective LLM-based methods for extracting DS-related information from EHRs are underdeveloped. This study bridges this gap by systematically evaluating RAG-enhanced LLMs for DS knowledge extraction, advancing foundational techniques for clinical text analysis.

To this end, we evaluate eight state-of-the-art LLMs on four dietary supplement datasets, utilizing retrieval-based augmentation techniques to enhance model performance. Our study systematically assesses how RAG influences LLMs’ ability to process DS-related EHRs, focusing on key information extraction tasks that are essential for structuring clinical knowledge. By analyzing the effectiveness of retrieval-augmented fine-tuning, we provide valuable insights into how LLMs can be optimized for EHR-based information extraction, offering guidance for future applications in medical NLP.

Methods

Tasks and datasets

This study focuses on the extraction of structured knowledge from clinical text. Specifically, we consider four fundamental tasks: named entity recognition (NER), relation extraction (RE), triple extraction (TE), and usage classification (UC).

Among these tasks, NER aims to identify dietary supplement mentions and their associated events, such as symptoms or diseases, within unstructured clinical narratives. RE involves identifying and categorizing relationships between dietary supplements and events. TE extends the combination of the NER and RE tasks by extracting structured triplets that encapsulate (supplement-relation-symptom) patterns, offering a more comprehensive evaluation of end-to-end information extraction ability. Finally, UC addresses the classification of dietary supplement usage, distinguishing between start, continue, discontinue, and uncertain. These tasks collectively contribute to structuring dietary supplement-related information within EHRs, enhancing their utility for downstream applications such as automated clinical documentation and evidence-based decision support.

To ensure a practical evaluation, we utilize four dietary supplement-related datasets derived from real-world EHR corpora, each curated to support a specific information extraction task. All datasets have undergone rigorous quality control procedures, including inter-annotator agreement validation, and have been de-identified to comply with patient privacy regulations. The details of data curation and examples are in previous works^40,41. The Institutional Review Board (IRB) approval was obtained for accessing these datasets. The dataset statistics are shown in Table. 1.

Table 1.

The statistics of datasets.

Dataset	Train	Validation	Test
Named entity recognition	2,365	292	292
Relation extraction	3,964	464	464
Triple extraction	2,365	292	292
Usage classification	2,000	230	230

Retrieval-augmented generation integration

As shown in Fig. 1, the retrieval mechanism operates by constructing a unified representation that concatenates the input sentence with its corresponding response, followed by a similarity computation with the input sentence embedding using cosine similarity. During the training phase, retrievers frequently identify instances where the retrieved sentence is identical to the input due to their high embedding similarity. This redundancy can lead the model to merely replicate the response from the example rather than engaging in substantive analytical reasoning. To prevent this undesirable reliance on direct copying, we enforce a constraint that prohibits the retriever from selecting the input sentence itself during training. Instead, it is configured to retrieve the most semantically similar sentence from the remaining dataset, thereby ensuring exposure to diverse yet contextually relevant examples. In contrast, during inference, the retriever is permitted to select the most relevant example from the entire training corpus, as there is no risk of overlap between the input and retrieved samples. The retrieved examples are then integrated into a structured prompt, which serves as additional context during instruction fine-tuning of the LLM. To maximize retrieval efficacy, we employ three state-of-the-art retrieval models—MedCPT,⁴² Contriever,⁴³ and BMRetriever.⁴⁴. Their information is summarised in Table 2.

Figure 1.

Retrieval-augmented generation integrated with large language models.

Table 2.

An overview of selected retrievers.

Retriever	Parameters	Pre-trained method	Validation
MedCPT	108M	Contrastive Pre-training	PubMed
Contriever	108M	Unsupervised contrastive learning	Wikipedia and CCNet
BMRetriever	410M	Unsupervised pre-training	PubMed, arXiv, MedRxiv, BioRxiv, etc

Large language models

To establish a robust foundation for our study, we systematically evaluate eight state-of-the-art LLMs: Mistral-7B,⁷ Llama-27B,⁴⁵ Llama-2-13B,⁴⁵ Llama-3-8B,⁴⁶ BioMistral-7B,⁴⁷ PMC-Llama-13B,⁴⁸ MedAlpaca-7B,⁴⁹ and MedAlpaca-13B.⁴⁹ The selected models encompass both general-domain and biomedical-domain LLMs, enabling a comprehensive assessment of their performance in dietary supplement-related information extraction.

Metrics

To evaluate the performance of the models across the dietary supplement-related information extraction tasks, we employ precision, recall, and F1-score as the primary evaluation metrics. Precision measures the proportion of correctly predicted instances among all predicted positive instances, reflecting the model’s ability to minimize false positives. Recall quantifies the proportion of correctly identified instances relative to the total number of ground truth instances, capturing the model’s effectiveness in identifying relevant information. F1-score, the harmonic mean of precision and recall, provides a balanced assessment of the model’s overall performance.

Experimental setting

All experiments were conducted on NVIDIA A100 GPUs with 80GB of memory, ensuring sufficient computational resources for large-scale model fine-tuning and inference. For fine-tuning, we adopted LoRA (Low-Rank Adaptation),⁵⁰ a parameter-efficient adaptation technique that enables effective specialization of large language models while significantly reducing memory overhead. The rank parameter was set to 64, with an alpha value of 32, and a dropout rate of 0.1, ensuring robust adaptation without overfitting. Fine-tuning was conducted using the AdamW optimizer,⁵¹ with a learning rate of 1e-5.

Results

The systematic evaluation of retrieval-augmented language models across four biomedical information extraction tasks reveals nuanced performance patterns tied to task complexity, model architecture, and retrieval methodology, as detailed in Tables 3-6.

Table 3.

Performance of retrieval-augmented LLMs on dietary supplements for named entity recognition task.

Model	Zero-shot			Random			MedCPT			Contriever			BMRetriever
	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1
BioMistral-7B	86.01	85.89	85.95	84.75	83.45	84.09	88.25	83.59	85.86	85.93	80.67	83.21	87.39	84.49	85.91
Llama-2-7B	85.55	81.66	83.56	85.46	79.78	82.52	85.17	81.16	83.12	83.86	80.95	82.38	84.99	83.45	84.21
Llama-2-13B	86.40	82.34	84.32	84.70	81.30	82.97	89.81	84.14	86.88	88.28	84.30	86.24	88.21	82.29	85.14
Llama-3-8B	86.18	84.38	85.27	86.64	85.67	86.15	87.37	83.73	85.51	87.68	84.14	85.87	88.09	83.78	85.88
MedAlpaca-7B	91.51	85.81	88.57	84.22	81.64	82.91	89.83	84.66	87.02	87.43	82.20	84.73	87.69	81.22	84.33
MedAlpaca-13B	85.01	82.90	83.94	82.68	78.83	80.71	82.60	81.34	81.96	78.55	78.32	78.93	82.39	81.90	82.14
Mistral-7B	84.01	84.83	84.42	85.63	85.51	85.57	86.63	84.35	85.47	85.73	85.73	85.73	86.34	85.26	85.79
PMC-Llama-13B	85.37	82.90	84.11	83.61	81.99	82.80	85.43	82.80	84.09	85.06	84.70	84.88	83.08	84.24	83.66

Table 6.

Performance of retrieval-augmented LLMs on dietary supplements for usage classification task.

Model	Zero-shot			Random			MedCPT			Contriever			BMRetriever
	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1
BioMistral-7B	90.79	90.79	90.79	89.96	89.96	89.96	89.52	89.52	89.52	89.08	89.08	89.08	91.27	91.27	91.27
Llama-2-7B	89.96	89.96	89.96	89.08	89.08	89.08	89.08	89.08	89.08	90.39	90.39	90.39	88.64	88.64	88.64
Llama-2-13B	91.70	91.70	91.70	89.96	89.96	89.96	90.39	90.39	90.39	94.76	94.76	94.76	92.14	92.14	92.14
Llama-3-8B	91.01	91.01	91.01	90.39	90.39	90.39	89.52	89.52	89.52	93.89	93.89	93.89	88.64	88.64	88.64
MedAlpaca-7B	92.57	92.57	92.57	87.77	87.77	87.77	89.96	89.96	89.96	93.45	93.45	93.45	89.95	89.95	89.95
MedAlpaca-13B	90.83	90.83	90.83	69.43	69.43	69.43	67.11	67.11	67.11	81.17	81.17	81.17	72.05	72.05	72.05
Mistral-7B	89.08	89.08	89.08	89.08	89.08	89.08	88.64	88.64	88.64	92.14	92.14	92.14	90.83	90.83	90.83
PMC-Llama-13B	89.08	89.08	89.08	89.52	89.52	89.52	91.70	91.70	91.70	92.14	92.14	92.14	90.39	90.39	90.39

Table 3 summarizes NER performance across models and retrieval strategies. MedAlpaca-7B achieved the highest baseline F1 score (88.57%), outperforming larger models like Llama-2-13B (84.32%) and PMC-Llama-13B (84.11%). Retrieval augmentation yielded mixed improvements: BMRetriever enhanced Mistral-7B (85.79% F1) and Llama-3-8B (85.88%), while MedCPT improved BioMistral-7B (85.86%) but degraded MedAlpaca-13B (81.96%). Notably, MedAlpaca-13B exhibited significant performance drops under retrieval-augmented settings, suggesting potential overfitting in domain-adapted models.

As shown in Table 4, BioMistral-7B with Contriever achieved the highest F1 score (93.95%), surpassing zero-shot baselines (93.09%) in RE task. General-purpose models like Mistral-7B and Llama-3-8B demonstrated robustness, achieving 93.30% and 91.79% F1 with BMRetriever and MedCPT, respectively. Conversely, domain-adapted models (MedAlpaca-7B/13B) underperformed, with F1 scores lower than 89.85% across retrievers. MedCPT retrieval paradoxically degraded performance for MedAlpaca-7B (87.26% vs. baseline 89.03%), highlighting misalignment between biomedical retrievers and specialized models.

Table 4.

Performance of retrieval-augmented LLMs on dietary supplements for relation extraction task.

Model	Zero-shot			Random			MedCPT			Contriever			BMRetriever
	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1
BioMistral-7B	93.09	93.09	93.09	93.30	93.30	93.30	92.44	92.44	92.44	93.95	93.95	93.95	91.36	91.36	91.36
Llama-2-7B	83.37	83.37	83.37	88.12	88.12	88.12	88.34	88.34	88.34	91.58	91.58	91.58	89.63	89.63	89.63
Llama-2-13B	92.66	92.66	92.66	93.09	93.09	93.09	91.79	91.79	91.79	92.87	92.87	92.87	92.44	92.44	92.44
Llama-3-8B	93.61	93.61	93.61	92.44	92.44	92.44	90.06	90.06	90.06	93.52	93.52	93.52	91.79	91.79	91.79
MedAlpaca-7B	89.03	89.03	89.03	88.77	88.77	88.77	87.26	87.26	87.26	89.85	89.85	89.85	86.39	86.39	86.39
MedAlpaca-13B	89.42	89.42	89.42	85.31	85.31	85.31	86.44	86.44	86.44	91.24	91.24	91.24	87.07	87.07	87.07
Mistral-7B	92.66	92.66	92.66	92.44	92.44	92.44	91.58	91.58	91.58	93.52	93.52	93.52	93.30	93.30	93.30
PMC-Llama-13B	90.50	90.50	90.50	91.36	91.36	91.36	90.28	90.28	90.28	92.44	92.44	92.44	91.36	91.36	91.36

Table 5 reveals that TE, the most complex task, exhibited the lowest F1 scores overall. Mistral-7B with BMRetriever achieved the highest F1 (77.12%), outperforming domain-adapted BioMistral-7B (75.74%). Model scale did not guarantee superiority: Llama-2-7B (71.39%) matched Llama-2-13B (75.47%) when paired with BMRetriever. MedAlpaca-13B showed severe degradation under MedCPT (64.52% F1), while random retrieval caused universal performance drops (e.g., MedAlpaca7B: 67.17% vs. baseline 75.36%). These results underscore TE’s sensitivity to compositional reasoning and retrieval relevance.

Table 5.

Performance of retrieval-augmented LLMs on dietary supplements for triple extraction task.

Model	Zero-shot			Random			MedCPT			Contriever			BMRetriever
	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1
BioMistral-7B	75.00	68.48	71.59	74.26	65.87	69.82	70.43	67.67	69.02	72.77	71.26	72.01	75.06	76.44	75.74
Llama-2-7B	72.09	64.57	68.12	69.66	63.36	66.36	69.71	63.46	66.44	70.07	62.88	66.28	71.90	70.90	71.39
Llama-2-13B	73.01	69.66	71.29	73.83	65.43	69.37	76.16	68.34	72.04	74.82	70.34	72.51	76.55	74.42	75.47
Llama-3-8B	78.29	70.39	74.13	73.60	66.06	69.63	74.10	67.91	70.87	72.95	69.78	71.33	77.75	73.95	75.80
MedAlpaca-7B	81.25	70.27	75.36	70.56	65.94	67.17	74.12	68.78	71.35	66.28	65.82	66.05	68.89	71.26	70.06
MedAlpaca-13B	76.05	67.78	71.68	68.42	67.82	68.12	65.42	63.63	64.52	69.97	62.87	66.23	65.37	64.48	64.92
Mistral-7B	74.88	73.33	74.09	75.13	67.13	70.90	75.12	67.03	70.85	73.70	69.07	71.31	77.30	76.94	77.12
PMC-Llama-13B	70.46	63.82	66.97	67.18	65.43	66.30	70.63	70.47	70.55	70.30	70.14	70.22	69.23	71.16	70.18

UC performance (Table 6) varied widely, with Llama-2-13B achieving state-of-the-art F1 (94.76%) using Contriever. Random retrieval severely impacted MedAlpaca-13B (69.43% vs. baseline 90.83%), while BMRetriever stabilized general models like Mistral-7B (90.83%). Domain adaptation again proved inconsistent: MedAlpaca-7B excelled in zero-shot settings (92.57%) but faltered with retrievers (89.95% for BMRetriever), whereas BioMistral-7B maintained robustness (91.27% F1 with BMRetriever). Task simplicity enabled high baselines, but retrieval strategies still provided critical gains for complex cases.

Retrieval augmentation strategies yielded divergent impacts across tasks. BMRetriever emerged as the most consistently beneficial approach, particularly for TE, where it improved Mistral-7B’s performance from 74.09% (zero-shot) to 77.12% (+3.03%). Contriever demonstrated task-specific superiority, achieving state-of-the-art UC results (94.76% F1 for Llama-2-13B) while underperforming in TE for domain-adapted models like MedAlpaca-7B (66.05% vs. zero-shot 75.36%). Notably, MedCPT retrieval paradoxically degraded performance for certain configurations, reducing MedAlpaca-13B’s UC accuracy by 23.72 percentage points compared to its zero-shot baseline.

Architectural analysis revealed a scale-performance decoupling: 13B parameter models failed to systematically outperform their 7B counterparts, with MedAlpaca-7B matching or exceeding MedAlpaca-13B’s TE performance across all retrieval conditions. Medical domain adaptation produced mixed outcomes—while MedAlpaca-7B achieved the highest zero-shot NER accuracy (88.57%), its 13B variant suffered severe performance degradation under retrieval-augmented conditions (64.52% TE F1 w/ MedCPT). Strikingly, the general-purpose Mistral-7B with BMRetriever outperformed all domain-adapted models in TE (77.12% vs. BioMistral-7B’s 75.74%), suggesting limitations in current medical adaptation methodologies for compositional tasks.

Task complexity inversely correlated with retrieval effectiveness, with TE showing maximum average gains (+4.23% F1 across methods) versus minimal RE improvements (+0.91%). This pattern held across model architectures, though with notable exceptions—Contriever enhanced UC performance by up to 12.59 percentage points (PMC-Llama-13B), while degrading MedAlpaca-13B’s TE accuracy by 5.45 points. Random example injection universally underperformed engineered retrieval, with particularly severe impacts on larger models (21.40% UC F1 drop for MedAlpaca-13B).

Discussion

The experimental findings provide critical insights into the interplay between RAG, model architecture, and task complexity in dietary supplement (DS)-related information extraction. Below, we contextualize these results within broader research trends, discuss their implications, and outline limitations and future directions.

Our results demonstrate that RAG’s effectiveness varies significantly across tasks, as shown in Fig. 2. While BMRetriever consistently improved performance for compositional tasks like triple extraction, Contriever excelled in usage classification, likely due to its ability to capture fine-grained semantic patterns. This indicates that retrieval mechanisms must align with task requirements—compositional reasoning benefits from structured context (e.g., BMRetriever’s example-based prompts), while classification tasks thrive on semantic similarity (e.g., Contriever’s dense embeddings). Notably, MedCPT underperformed in certain scenarios, particularly for domain-adapted models like MedAlpaca-13B, suggesting that biomedical-specific retrievers may inadvertently introduce noise when combined with already specialized models.

Figure 2.

Comparison of different retrievers for each model. The stars indicate the highest F1 score for each task within one model.

Contrary to expectations, medical domain adaptation did not universally enhance performance. While MedAlpaca-7B achieved superior zero-shot NER accuracy (88.57% F1), its 13B variant suffered degradation under retrieval-augmented settings. This highlights potential overfitting risks in domain-specific models when exposed to external knowledge. Strikingly, general-purpose models like Mistral-7B outperformed domain-adapted counterparts in TE when paired with BMRetriever (77.12% vs. 75.74%), indicating that current medical adaptation strategies may inadequately address compositional reasoning. Future work should explore hybrid approaches that balance domain specialization with flexible retrieval mechanisms.

The lack of systematic superiority of larger models (e.g., Llama-2-13B vs. Llama-2-7B) challenges the conventional assumption that model scale guarantees performance gains. This decoupling is most evident in TE, where task complexity (combining NER and RE) amplified variability across architectures. The inverse correlation between task complexity and retrieval efficacy (+4.23% F1 gain for TE vs. +0.91% for RE) underscores the need for task-aware retrieval strategies. For instance, UC’s high performance ceiling (94.76% F1) suggests that simpler tasks may saturate quickly, leaving limited room for improvement, while compositional tasks demand more sophisticated augmentation.

Finally, as shown in Fig. 3, the heat map of the average F1 scores across the four tasks indicates that Contriever and BMRetriever are more suitable for dietary supplement information extraction compared to other methods. Among the eight LLMs we used, Llama3-8B and Mistral-7B are the best-performing models for extracting dietary supplement information.

Figure 3.

Heat Map of average F1 score over four tasks for different retrievers.

Conclusion

This study highlights the efficacy of RAG in improving LLMs for extracting dietary supplement-related information from clinical narratives. The results reveal that the success of RAG hinges on aligning retrieval strategies with task requirements: certain retrievers excel in complex reasoning tasks, while others are better suited for classification-oriented objectives. Notably, smaller general-purpose models, when paired with tailored retrievers, often outperform larger or domain-specialized counterparts, challenging the assumption that model scale or domain adaptation alone guarantees superior performance. These findings underscore the importance of strategic retrieval integration over reliance on model size or pre-existing domain knowledge, offering a pathway to enhance structured clinical data analysis and support evidence-based healthcare decision-making.

Competing Interests

The authors declare no competing interests.