Addressing Generalizability in Clinical Named Entity Recognition: Federated Learning or Large Language Models?

Abstract

Clinical Named Entity Recognition (NER) is vital for extracting structured data from clinical text, but ensuring model generalizability across institutions remains challenging. This study compares two approaches: (1) Federated Learning (FL), a privacy-preserving decentralized method, and (2) Large Language Models (LLMs) trained on diverse corpora. We evaluate Visual Acuity (VA) extraction from ophthalmology notes at Stanford (USA) and Moorfields Eye Hospital (UK), using BERT-based models, FL strategies (FedAvg, STWT), and LLMs (Llama-3-70B, Mixtral-8x7B). Results show that FL significantly improves generalization, with STWT outperforming FedAvg in stability and accuracy. LLMs demonstrate strong performance on MEH data but struggle with structured Stanford notes. These findings highlight FL’s effectiveness for cross-institutional learning while revealing domain-specific limitations of LLMs, underscoring the need for tailored approaches to clinical NER.

1. Introduction

Clinical Named Entity Recognition (NER) has emerged as a crucial topic in Natural Language Processing (NLP), particularly due to its immense potential in digital health applications. From reducing manual labor of data entry and extraction to improving search functionalities and assisting other AI systems, clinical NER can significantly enhance the efficiency of healthcare workflows.^{1, 2, 3, 4, 5} As the demand for accurate clinical text processing increases, so does the need to address the generalizability of clinical NER models, ensuring that they can effectively handle data from diverse sources.

When facing the challenge of generalizing across different health institutions while maintaining privacy, there are generally two viable approaches: (1) federated learning (FL), (2) training a large model on vast amounts of diverse text from a single source. Federated learning,⁶ a decentralized machine learning approach, is designed to mitigate the generalization gap by collaboratively training models across multiple clients, without the need to transfer sensitive data. This enables models to learn from diverse datasets while preserving privacy.

An alternative approach to mitigate the generalization gap is to use massive models trained on a vast corpus of data, with the hope that these models will generalize well to new data and tasks. Recent studies^{7, 8} have shown that large language models (LLMs) are making rapid progress in clinical NER, sometimes outperforming BERT-based models in certain scenarios. For example, LLM could mitigate the generalization gap by excelling in low-resource settings and cross-institutional generalization. They outperform BERT when annotated data is limited, achieving up to 12.5% higher F1 in Relation Extraction (RE). Additionally, they adapt better to unseen datasets, with Llama-3-70B surpassing BERT by 7% (NER) and 4% (RE) on i2b2. Their ability to capture complex entity relationships (e.g., negation, temporal attributes) further enhances extraction accuracy.⁹

However, most clinical NER studies have been conducted on public datasets, which may overlap with the training data for these LLMs, potentially resulting in overestimated performance. To the best of our knowledge, there has been no study that directly compares the performance of LLMs in clinical NER on real-world clinical notes, nor has there been an investigation into how LLMs and federated learning compare in mitigating the generalization gap between data from different health institutions. This study provides a case study of such an investigation, focused on visual acuity extraction clinical notes originating from two eye institutes, one in the United States and one in the United Kingdom.

2. Method

2.1. Dataset

The primary NER task in this study is to extract Visual Acuity (VA) information from ophthalmic clinical notes. Visual Acuity refers to the clarity or sharpness of vision, typically measured using a standardized chart. It is the most important “vital sign” in ophthalmology that summarizes the functionality of an individual’s eyesight. Visual acuity can be measured under different circumstances, for example, with refractive correction (e.g. glasses), without correction, or gazing through a pinhole. In this study, we defined different VA entity types in the following format, consistent with our previous work:¹⁰

$$V\hspace{0.17em}A\hspace{0.17em}<\hspace{0.17em}\text{Laterality}\hspace{0.17em}>\hspace{0.17em}<\hspace{0.17em}\text{Testing}\hspace{0.17em}\text{Range}\hspace{0.17em}>\hspace{0.17em}<\hspace{0.17em}\text{Correction}\hspace{0.17em}>\hspace{0.17em}<\hspace{0.17em}\text{Pinhole}\hspace{0.17em}>$$

where < Pinhole > is optional. The components are defined as follows: Laterality = {OD (Right Eye), OS (Left Eye)}; Range = {DIST (measure at distance), NEAR (Measured near eye)} — with NEAR being rarely observed and therefore discarded; Correction = {CC (With correction, no pinhole), SC (Without correction), CCPH (With correction via pinhole), SCPP (Without correction via pinhole)}. Given these values, there are a total of 2×1×4 = 8 entity types. For example, “VAODDISTCC” translates to “The distance visual acuity measurement for the RIGHT eye with correction”, while “VAOSDISTSCPH” means “The distance visual acuity measurement for the LEFT eye without correction and WITH PINHOLE.”. The corpora of clinical notes used in this study were collected from two sources.

Stanford We used the same dataset collected in our previous study,¹⁰ which includes clinical notes and VA measurements of patients seen by the Stanford Ophthalmology Department since 2008. The VA section in the notes was labeled in a weakly super-vised manner using semistructured VA fields from the Epic EHR system. The dataset initially contained 333,958 notes from about 90,000 patients. After excluding notes without documented VA, 319,756 notes remained. Of these, 80% were used for training, 10% for validation, and a separate test set of 300 notes was independently manually annotated for ground truth.

MEH We collected another clinical notes dataset from Moor-fields Eye Hospital (MEH) in the UK. The dataset consists of 400 clinical notes collected before 2024 from the OpenEyes EHR system.¹¹ All VA entities were manually annotated. After excluding examples from the two eye institutes.

2.2. Named-Entity Recognition using BERT

VA extraction followed the approach in Bernstein et al.,¹⁰ using Bidirectional Encoder Representations from Transformers (BERT¹²) to perform token classification, i.e each token in the text is assigned a label indicating whether it is part of a VA entity. The BIO tagging scheme was used, where tokens are classified as ‘O’ (Outside any entity), ‘B-<type>‘ (Beginning of an entity), or I-<type>‘ (Inside an entity). Previous experiments in Bernstein et al.¹⁰ showed that BlueBERT¹³ performed best for VA extraction, so it was chosen for this study.

2.3. Federated Learning

Federated learning (FL) aims to minimize the global loss function $\mathcal{L}$(θ), which represents the overall model’s performance across all clients. Each client k holds local data D_k , and computes the local loss ${\mathcal{L}}_k$ (θ) on its dataset. The goal is to find the global θ that minimizes the global loss: $\mathcal{L}\left(\theta \right)=\frac{1}{K}{\displaystyle {\sum }_{k=1}^K{\mathcal{L}}_k\left(\theta \right)}$

where ${\mathcal{L}}_k$ (θ) is the local loss on client k’s data. The federated learning process minimizes $\mathcal{L}$(θ) by aggregating local updates in a privacy-preserving manner, thereby addressing both the generalization gap and data privacy concerns.

Federated learning (FL) algorithms can be categorized into two main groups: center-to-peer (or parallel) FL and peer-to-peer (or serial) FL. In center-to-peer FL, a central server coordinates the model updates by aggregating the local updates from clients, whereas in peer-to-peer FL, clients communicate directly with each other to share model updates.

Depending on the situation, one might find one setup more advantageous than the other. Center-to-peer FL often leads to more optimal, faster, and stable convergence,^{14, 15} but it can run into logistical problems, such as requiring training to take place in parallel across all institutions. This can be especially challenging if institutions have drastically different network connection speeds or deep learning hardware. Additionally, a central server must be established, raising concerns about which institution controls it.

The peer-to-peer setting can help mitigate these issues. While generally slower, peer-to-peer FL requires less communication overhead. In contrast, center-to-peer FL requires twice the communication, as model weights must be sent back and forth to complete a round. This study examined the most prominent algorithms of each group, FedAvg and Stochastic Weight Transfer (STWT), respectively.

FedAvg6 is a widely used algorithm in center-to-peer FL. In each round, clients compute local updates based on their local datasets and then send the updated model parameters θk to the central server. The server aggregates these updates by computing a weighted average: ${\theta }_{global}\hspace{0.17em}=\hspace{0.17em}{\sum }_{k\hspace{0.17em}=\hspace{0.17em}1}^K\frac{n_k}{n}{\theta }_k$.

where θk is the local model update on client k, nk is the number of data points on client k, and n is the total number of data points. The server then updates the global model θ and distributes it back to the clients for the next round of training. The detailed algorithm can be found in algorithm 1.

Algorithm 1:

Federated Averaging

In our experiment, we opted to use a simple average for model aggregation: ${\theta }_{global}\hspace{0.17em}=\hspace{0.17em}\hspace{0.17em}\frac{1}{K}{\sum }_{k\hspace{0.17em}=\hspace{0.17em}1}^K{\theta }_k$. This decision was made to ensure that each client contributes equally to the global model. The standard weighted averaging in FedAvg would result in the model being overly influenced by clients with larger datasets (Stanford), whereas a simple average provides a clearer view of FL dynamics without dataset size bias.

Stochastic weight transfer (STWT)^{16, 17} is a slight variation of Cyclical Weight Transferring (CWT),¹⁵ the peer-to-peer counterpart of FedAvg. In CWT, the global model θ is sequentially trained on each client using its local data for a fixed number of epochs before being transferred to the next client. This cycle continues until all clients have participated in training, and the process repeats for multiple rounds until convergence or a predefined number of communication rounds is reached. However, to ensure a fair comparison with other algorithms—where performance is evaluated after each round—and to prevent the model’s performance from being overly influenced by the last client in the sequence, we adopt Stochastic Weight Transfer. Unlike CWT, which follows a fixed client order, STWT randomly samples clients in each round, which, in our case, is implemented by shuffling the client order before each training cycle. Details and comparisons of both techniques are outlined in algorithm 2 and 3.

Algorithm 2:

Cyclic weight transfer with stopping criteria.

Algorithm 3:

Stochastic weight transfer

2.4. Large Language Model

We also tested the generalizability of large generative language models with two well-known models Llama-3-70B¹⁸ and Mixtral-8x7B.¹⁹ To reduce computational resources, we used the 4-quantized (Q4 for short) versions of these models. Quantization is a process that reduces the precision of the model’s parameters, converting them from floating-point numbers to lower-bit representations (e.g., from 32-bit to 4-bit), which significantly reduces memory usage and accelerates inference times, making the models more efficient for use with limited GPU resources. For our experiments, we utilized 5-shot in-context learning to assess model capabilities with a minimal number of examples. We used the prompt structure in Table 2 to extract VA from the clinical note.

Table 2:

Prompt template for LLM

2.5. Experimental setup

Evaluation metrics We used nervaluate²⁰ (initially proposed by Batista²¹), a Python module designed to evaluate Named Entity Recognition (NER) models, extending the SemEval 2013 Task 9 evaluation metrics. It addresses limitations of token-based evaluation by considering full named-entity matches, including scenarios where an entity’s type or boundary may be incorrect or partially identified. The primary metric is the strict F1 score, which evaluates exact boundary and entity type matches, serving as the upper bound. The secondary metric is the partial F1 score, which allows for partial boundary matches regardless of entity type, offering a lower bound for performance evaluation.

BERT Training at Client Each client trains a BERT model for token classification using the standard cross-entropy loss function. The Adam optimizer is employed with a learning rate of 5e⁻⁵ and a weight decay of 0.01. We use a batch size of 8 and perform 10 iterations per training round. Our preliminary experiments indicated no significant difference in performance across different iteration counts, and 10 was chosen as a reasonable balance between resolution and noise reduction.

Federated Learning (FL) The FL process was performed using the BlueBERT model (best model in VA extraction reported in Bernstein et al.¹⁰) in NER setting for 500 rounds. The best-performing model is selected based on the highest strict F1 score achieved during the communication rounds. FL was not conducted on large language models.

Baseline Evaluation To assess the individual client model performance, we conducted a Baseline (BL) experiment where each client c trained and tested its model solely on its own data. This serves as a reference point for evaluating improvements achieved through federated learning. Additionally, we performed Baseline Cross Evaluation (BL_CrossEval), where a model trained on the data of one client c was tested on the other client’s dataset. This evaluation highlights domain shifts and generalization capabilities across different data distributions. We use the notation $B{L}_{source}^{t\text{arget}}$ to indicate performance of a model trained on source and tested on target. For example, $B{L}_{SF}^{SF}$ represents a baseline model trained and tested on Stanford data, while $B{L}_{SF}^{MEH}$ denotes a cross-evaluation baseline where a model trained on Stanford data is tested on MEH data.

Ethics Statement This study was approved by the Stanford University institutional review board. The institutional review board granted a waiver of informed consent given the scale of the data and observational nature of the data set.

The study adhered to the tenets of the Declaration of Helsinki. The data from MEH was part of a registered audit at Moorfields Eye Hospital NHS Foundation Trust. Audit number: 948, version 1. Audit entitled: Using NLP model to extract VA terms from rare eye disease patient’s electronic health records in Moorfields Eye Hospital (MEH) Genetics Department, to increase diagnostic yield and provide a better genetic diagnostic service. Supervised by: Dr Nikolas Pontikos, Institute of Ophthalmology, UCL.

3. Results

3.1. Model Performance and Generalization

Performance of the FL algorithms, LLMs, and the two baseline models are summarized in Table 4. Baseline evaluation demonstrated that VA extraction using BERT is highly effective with Stanford data, reaching a micro F1 score of 0.943 (strict) and 0.985 (partial). However, the baseline MEH model performance achieved only F 1_strict = 0.427 and F 1_partial = 0.792. In addition, a large difference between F_strict and F_partial scores in the baseline model on MEH data suggests that while span detection was relatively easy, assigning the correct class of VA to the detected entity remained challenging.

Table 4:

Micro-F1 scores of Llama-3-70B-Q4, Mixtral-8x7B-Q4, FedAvg, STWT, and two baselines on 8 VA entity types for Stanford (left) and MEH (right) test sets. Strict and partial evaluation are shown in top and bottom sections, respectively. Micro-average F1 the is primary metrics; macro-average F1 is for reference. Best model per dataset is bolded, second-best underlined. Numbers below model names show best-F 1_strict samples (calculated as r_best × 20 (steps per round) × 8 (batch size). “N/A” = no examples in that category.

Strict F1 Score Evaluation
Evaluation on Stanford Notes								Evaluation on MEH Notes
		Llama-3 70B Q4 n=5	Mixtral 8x7B n=5	FedAvg n=21,120	STWT n=78,240	$B{L}_{SF}^{SF}$ n=48,480	$B{L}_{MEH}^{SF}$ (CrossEval) n=480	Llama-3 70B Q4 n=5	Mixtral 8x7B n=5	FedAvg n=44,800	STWT n=43,360	$B{L}_{SF}^{SF}$ n=79,840	$B{L}_{MEH}^{SF}$ (CrossEval) n=71,360
OD	CC	0.304	0.291	0.985	0.971	0.98	0.286	0.712	0.417	0.189	0.508	0.521	0.41
	SC	0.316	0.25	0.926	0.963	0.951	0.658	0.681	0.399	0.645	0.524	0.377	0
	CCPH	0.117	0	0.951	0.835	0.865	N/A	N/A	N/A	N/A	N/A	N/A	0
	SCPH	0.031	0.026	0.911	0.941	0.97	0.458	0.609	0.424	0.383	0.51	0.2	0
OS	CC	0.389	0.305	0.968	0.948	0.957	0.125	0.75	0.4	0.2	0.4	0.427	0.339
	SC	0.341	0.209	0.924	0.962	0.945	0.561	0.593	0.339	0.734	0.581	0.519	0
	CCPH	0.091	0	0.875	0.902	0.878	N/A	N/A	N/A	N/A	N/A	N/A	0
	SCPH	0.054	0.051	0.857	0.828	0.857	0.5	0.63	0.392	0.588	0.457	0.439	0.02
Micro Avg	0.375	0.288	0.942	0.94	0.943	0.502	0.66	0.423	0.544	0.519	0.427	0.204
Macro Avg	0.205	0.142	0.925	0.919	0.925	0.431	0.663	0.395	0.457	0.497	0.414	0.096

Partial F1 Score Evaluation
Evaluation on Stanford Notes								Evaluation on MEH Notes
		Llama-3 70B Q4 n=5	Mixtral 8x7B n=5	FedAvg n=21,120	STWT n=78,240	$B{L}_{SF}^{SF}$ n=48,480	$B{L}_{MEH}^{SF}$ (CrossEval) n=480	Llama-3 70B Q4 n=5	Mixtral 8x7B n=5	FedAvg n=44,800	STWT n=43,360	$B{L}_{SF}^{SF}$ n=79,840	$B{L}_{MEH}^{SF}$ (CrossEval) n=71,360
OD	CC	0.481	0.456	0.991	0.994	0.994	0.816	0.848	0.595	0.755	0.794	0.699	0.557
	SC	0.556	0.518	0.979	0.98	0.988	0.767	0.751	0.415	0.819	0.881	0.918	0.737
	CCPH	0.272	0.02	0.961	0.971	0.99	N/A	N/A	N/A	N/A	N/A	N/A	0.524
	SCPH	0.047	0.026	0.96	0.971	0.98	0.854	0.891	0.47	0.809	0.902	0.683	0.462
OS	CC	0.502	0.441	0.988	0.974	0.991	0.75	0.9	0.52	0.82	0.733	0.56	0.539
	SC	0.552	0.369	0.962	0.979	0.983	0.705	0.718	0.408	0.835	0.865	0.89	0.71
	CCPH	0.236	0.051	0.975	0.963	0.963	N/A	N/A	N/A	N/A	N/A	N/A	0.525
	SCPH	0.149	0.064	0.949	0.919	0.949	0.75	0.852	0.51	0.794	0.771	0.683	0.53
Micro Avg	0.543	0.467	0.977	0.976	0.985	0.76	0.777	0.497	0.814	0.844	0.792	0.583
Macro Avg	0.349	0.243	0.971	0.969	0.98	0.774	0.827	0.486	0.805	0.824	0.739	0.573

Baseline Cross Evaluation revealed a notable generalization gap for the baseline models. A model trained on Stanford data reached F 1_strict = 0.502 (ΔF 1_strict = 0.441) when tested on MEH data, and the MEH-trained model achieved only ΔF 1_strict = 0.223 on Stanford data.

Both FL algorithms significantly mitigated the performance drop due to data shifts between clients. FedAvg even outperformed the baseline on MEH, achieving F_strict = 0.544, and matched Stanford’s baseline with F_strict = 0.942. STWT achieved F_strict = 0.519 on MEH and 0.94 on Stanford. Surprisingly, STWT surpassed FedAvg on the partial score by 3%, attaining F_partial = 0.8444.

The Llama-3-70-Q4 model performed strongly on MEH, far exceeding BERT in F1-strict by accurately recognizing spans and classifying entities, with F_strict = 0.66 and F_partial = 0.777. However, it struggled with Stanford data, achieving only F_strict = 0.375 and F_partial = 0.534. Mixtral-8x7B-Q4 performed even worse, with F_strict = 0.288 and F_partial = 0.467. Although Mixtral showed a slight improvement on MEH, both F1 scores remained below 0.5.

3.2. Training Dynamics and Computational Considerations

Figure 2 shows the micro-F1 score progression across communication rounds. Overall, most models on both datasets converged efficiently and stabilized quickly after approximately 80 rounds. The only exception was FedAvg, which exhibited a much noisier F1 curve throughout the training process. Notably, STWT had consistent performance through-out that training process that generally outperformed FedAvg and was an improvement over cross-evaluation, when models trained on one institution were tested on another.

Figure 2:

Micro-F1 scores over 500 communication rounds for two clients c: Stanford (left) and MEH (right). The top row represents strict evaluation, while the bottom row corresponds to partial evaluation. Models include FedAvg, STWT, baselines (BL, BL_CrossEval), and Llama-3-70B-Q4 (dashed purple line).

BERT-based models required only 0.8 GB of GPU’s memory (VRAM) and achieved sub-6ms inference per note (batch size = 1) (See Table 3). In contrast, LLMs imposed substantial over-head despite being quantized to 4-bit. Llama-3-70B Q4 required 43.17 GB of VRAM and had an average inference time of 6.0±2.58 seconds on Stanford notes and 2.95±1.33 seconds on MEH notes. Mixtral-8x7B-Q4, though lighter at 27.2 GB VRAM, still took 1.79±1.15 seconds (Stanford) and 2.03±1.6 seconds (MEH). These models were run on two A100 (40GB) GPUs at Stanford and three Tesla P100s (16GB) at MEH.

Table 3:

Overhead cost. GPU’s memory required (VRAM in Gigabyte or GB) and inference time (seconds, mean ± SD) per note (batch size = 1) on Stanford and MEH datasets.

	VRAM (GB)	Inference time (s)
		Stanford	MEH
Llama-3-70B Q4	43.17	6.0±2.58	2.95±1.33
Mixtral-8x7B-Q4	27.2	1.79±1.15	2.03±1.6
BERT	0.8	0.00595	0.00495

4. Discussion

In this study, we developed pipelines for clinical named entity recognition of visual acuity from clinical progress notes from two institutions, comparing strategies utilizing FL, LLMs, and models trained from a single-institution’s data. We found that performance of each of these strategies varied between the two institutions. While the LLM performed well on MEH notes, it performed poorly on Stanford notes; conversely, baseline models trained on Stanford data performed well at Stanford and models trained on MEH data performed poorly at MEH. Models trained on a single institution’s notes did not generalize well when evaluated at the other institution, but some generalizability could be recovered through the use of FL strategies to train from data at both institutions. Of the FL strategies, STWT outperformed FedAvg.

The high performance of the baseline model trained on Stanford data aligns with previous findings,¹⁰ while the sharp decline on MEH data may stem from the complex and implicit assumptions in MEH notes. For instance, notes at MEH often assume that the first appearance of a VA is for the right eye and the latter for the left, or when no correction is mentioned, the VA is assumed to be unaided (without glasses). These nuanced assumptions and domain-specific rules present challenges for the model. On the other hand, the Stanford clinical notes, though sometimes unnatural to the human eye, often had a more consistent and structured format owing to the use of automated text formatting templates used in the electronic health record, potentially making it easier for the discriminative model like BERT to identify machine-like patterns.

This study highlights how FL can be an effective strategy for mitigating performance losses caused by data shifts without compromising data privacy. This contrasts with similar studies (FL on clinical NER using BERT) by Peng et al.²² using simulated public data, where performance was already saturated and the difference between the baseline at the client and the FL algorithm was negligible (< 5%), thus failing to highlight the role of FL. These results emphasize the importance of studies on real-world and unique health data, where privacy-preserving approaches like FL are critical and can yield substantial benefits.

We also found that for MEH clinical notes, Large Language Models (LLMs) showed very promising results, easily outperforming BERT when provided with just five examples, demonstrating their ability to capture complex language patterns in clinical notes written in natural language. However, the LLM struggled with Stanford notes, which were written with the aid of note language templates, where discriminative models like BERT performed much better. This demonstrates two important points: (1) data and concept drift can also heavily impact LLM performance, and (2) depending on the situation, BERT could still be a valuable tool for certain tasks, especially when dealing with domain-specific language and structured input.

A unique finding of this study was the superior performance of STWT compared to FedAvg. This contrasts with current literature,^{23, 24, 14} particularly in FL studies on image classification, where FedAvg is often regarded as the gold standard due to its stability and ability to quickly converge. On the other hand, STWT is often considered more volatile and much harder to converge. The discrepancy between our results and the literature could be attributed to several factors, including the number of clients in the experiment (2 vs. hundreds), as well as the nature of the task (token classification using BERT vs. image classification using Convolutional Neural Networks). This suggests that the choice of FL algorithm may be task-dependent, and further studies should explore these dynamics in different contexts. As most FL initiatives across health institutions are likely to include fewer rather than tens of clients, our results suggest that STWT should be considered as an option, given its relative ease to deploy without the need to establish the infrastructure of a data coordinating center.

The present study was limited to two institutions due to the difficulty of establishing collaborations across multiple healthcare sites who have aligned data types and data governance frameworks. This reflects a common constraint in real-world clinical FL studies. To our knowledge, no prior work has evaluated peer-to-peer FL methods like STWT on NLP tasks using transformer models. As a result, it is unclear how STWT would scale with more clients. Performance in FL is influenced by multiple factors such as model architecture, task type, and data heterogeneity, making generalizations difficult. The only reliable way to assess scalability is through future benchmark studies using public clinical NER datasets in simulated FL settings.

Finally, computation analysis highlights that while LLMs offer strong generalization in some settings, their high resource demands may limit feasibility in clinical environments lacking high-end infrastructure. BERT models, by contrast, remain well-suited for low-latency, resource-constrained deployment. Beyond computational constraints, it is important to recognize that despite their smaller size and faster inference times, FL methods face significant practical challenges in real-life deployment across institutions, whereas LLMs, though resource-intensive, can be easier to set up in certain environments. Deploying FL across institutions involves aligning software stacks, libraries, and secure APIs. Center-to-peer FL adds complexity through the need for synchronous coordination and central server governance. While peer-to-peer methods like STWT reduce centralization issues, they introduce challenges like handling asynchronous updates and dropouts. Regulatory concerns over model parameters as sensitive data further complicate cross-border setups. These barriers highlight the need for standardized FL toolkits tailored to clinical NLP.

This study has several limitations. While the study investigates a complex task with more than eight classes of entities, it focuses on one clinical variable: Visual Acuity (VA). Future work is planned to incorporate a wider range of clinical variables to offer a fuller picture of model performance in clinical entity extraction from progress notes. In addition, the number of clients in this study was significantly smaller than in typical Federated Learning studies, reflecting the realities of the challenges in establishing cross-institutional collaborations in the healthcare domain. Future work could include more institutions to provide further insights into the effect that number of participating institutions has on the performance of different FL algorithms for NER tasks. Lastly, a primary objective of this study was to establish a baseline for LLM performance on the task of VA extraction, to which FL methodologies could be compared. While LLM baseline performance itself offers valuable insights, further performance improvement on the visual acuity extraction task may be possible with the addition of chain-of-thought reasoning, multi-step processing, and/or fine-tuning.

In conclusion, we found that while LLMs show promise in clinical named entity recognition from progress notes, they do not always outperform models trained specifically to perform a particular NER task. Furthermore, incorporating federated learning techniques to train models with data from multiple sites can somewhat improve generalizablity of NER performance across sites. For our case study involving two institutions and a clinical NER task, the STWT algorithm performed better than the FedAvg algorithm. Future studies could extend this work to multiple additional sites and clinical named entities.

Figures & Tables

Figure 1:

Examples of VA entitiesfrom (A) Stanford Byers Eye Institute and (B) Moorfields Eye Hospital. The VA sections in Stanford notes, which were semi-automatically generated using structured EHR fields, appear more templated, whereas Moorfields notes, written entirely by clinicians, have a more natural, free-text style.

Table 1:

Comparison of entity counts between MEH and Stanford in train and test sets

	MEH		Stanford
	Train	Test	Train	Test
VAODDISTSC	162	80	11,136	121
VAOSDISTSC	161	74	11,008	115
VAODDISTCC	6	27	0	171
VAOSDISTCC	46	25	3,713	173
VAODDISTSCPH	45	25	3,818	50
VAOSNEARCC	0	1	0	0
VAOUDISTSC	38	4	10,503	0
VAOSDISTSCPH	3	17	0	48
VAODNEARCC	0	1	0	0
VAODDISTCCPH	0	0	3,106	52
VAOSDISTCCPH	0	0	3,192	40