Leveraging large language models for the deidentification and temporal normalization of sensitive health information in electronic health records

Abstract

Secondary use of electronic health record notes enhances clinical outcomes and personalized medicine, but risks sensitive health information (SHI) exposure. Inconsistent time formats hinder interpretation, necessitating deidentification and temporal normalization. The SREDH/AI CUP 2023 competition explored large language models (LLMs) for these tasks using 3,244 pathology reports with surrogated SHIs and normalized dates. The competition drew 291 teams; the top teams achieved macro-F1 scores >0.8. Results were presented at the IW-DMRN workshop in 2024. Notably, 77.2% used LLMs, highlighting their growing role in healthcare. This study compares competition results with in-context learning and fine-tuned LLMs. Findings show that fine-tuning, especially with lower-rank adaptation, boosts performance but plateaus or degrades in models over 6 B parameters due to overfitting. Our findings highlight the value of data augmentation, training strategies, and hybrid approaches. Effective LLM-based deidentification requires balancing performance with legal and ethical demands, ensuring privacy and interpretability in regulated healthcare settings.

Introduction

Healthcare providers generate large amounts of clinical data in the form of electronic health records (EHRs). The primary aim of EHRs is to deliver, manage, and document clinical care. These records have also been used extensively for secondary purposes, including clinical studies¹, public health surveillance², clinical audits for quality enhancement³, identification of social determinants of health (SDoH)⁴, and improvement of healthcare practices⁵. For example, Romanowski et al.⁴ proposed a sequence-to-sequence approach to extract SDoH, such as housing status, substance use, and employment, from unstructured EHRs using named entity recognition and relation extraction. These insights support population health management and personalized care strategies. However, using EHR data for secondary purposes, even after obtaining consent, raises privacy concerns. Furthermore, EHR data are accessible to various stakeholders, including government agencies, across different geographical locations via wired or wireless networks, making them vulnerable to cyberattacks⁶. Individual privacy can be significantly compromised when the sensitive health information (SHI) contained in EHRs is leaked or exposed.

To safeguard patients’ personal data from such threats, privacy regulations, such as the Health Insurance Portability and Accountability Act (HIPAA)^7,8, are in place alongside continually evolving techniques in machine learning and natural language processing (NLP). These include privacy-aware learning^9,10 and deidentification methods^11–13. Deidentified EHR notes are those from which SHIs, such as patient or doctor names and dates, are permanently removed or replaced with surrogates. Studies have shown that when private information is removed or deidentified, patients are more willing to share their medical records for study purposes^14,15. In addition, recent studies have highlighted that large language models (LLMs) have become increasingly embedded in medical training environments, and concerns surrounding its impact on academic integrity, learner dependency, clinical reasoning, and patient data confidentiality are becoming more pronounced¹⁶. Therefore, deidentification is required for data sharing to ensure compliance with regulations and to meet public expectations for privacy protection.

The standard process for generating deidentified EHR text prior to sharing involves replacing HIPAA-specified SHI entities with newly generated SHI surrogates¹⁷. Figure 1 illustrates a segment of such a deidentified report, where the SHIs that need to be recognized are highlighted in bold. The text in Fig. 1 contains only surrogated SHIs. For example, the male doctor's name “AADLAND ABRAHAN” shown in Fig. 1, is generated to preserve the original gender information of the actual doctor. Furthermore, date-related surrogated SHIs, such as the date of birth/collection dates or specific weekdays, such as “Friday,” are derived from the original real dates by adding a random number of days according to predefined constraints¹⁸. This process is crucial for maintaining longitudinal temporal integrity across events mentioned in a single patient’s EHRs, which is essential for enabling valid longitudinal analyses and maximizing the utility of deidentified data for secondary usage^19,20. In particular, studies focusing on SDoH often rely on accurate temporal sequencing to analyze trends in disease progression, treatment outcomes, and the influence of external factors—such as employment instability or limited healthcare access—on clinical trajectories over time²¹. However, date-related SHIs present a particular challenge in deidentifying real-world EHR data while preserving temporal integrity²². If this integrity is not maintained during the surrogate generation process, the interpretation of the released longitudinal data may become unreliable, potentially leading to inaccurate interpretations or biased conclusions in downstream analyses.

Fig. 1. Example of a deidentified text report highlighted with SHIs in bold.

This figure illustrates a segment of a deidentified report from the released dataset. All SHI mentions are highlighted in bold and have been replaced with surrogates to preserve privacy.

This level of deidentification requires the initial recognition of true SHIs, followed by the application of a surrogate generation algorithm to produce synthetic SHIs. For date-related SHIs, the surrogate generation algorithm relies on using the same randomly generated date shift for the same patient to avoid ambiguity and ensure temporal integrity^22,23. To facilitate this process, it is critical to normalize the data descriptions according to the ISO 8601²⁴. For example, in Fig. 1, “14/02/3014 at 11:42” and Friday are normalized to “3014-02-14T11:42” and “3014-02-18” respectively.

To explore state-of-the-art NLP techniques for the deidentification process, the SREDH/AI CUP 2023 deidentification competition was organized between September 18, 2023, and December 3, 2023. The competition featured two subtasks, SHI recognition and temporal information normalization, both of which were annotated on a deidentified dataset of 3244 pathology reports, whose date-related SHIs were surrogated following the rules to ensure data integrity. This study provides an in-depth overview of the competition and the observations of the results presented by the participants at the International Workshop on Deidentification of Electronic Medical Record Notes (IW-DMRN) in Kaohsiung, Taiwan, on January 15, 2024²⁵. Furthermore, in this study, we specifically present our approaches based on the fine-tuning of language models (LM), which serves as a baseline for comparison with participant results, aiding in the selection of teams for workshop presentations. The fine-tuned LM models are based on Pythia²⁶, ranging from the smallest LM model with 70 M parameters to the largest with 12B parameters, which are specifically designed to facilitate deidentification studies. In addition to fine-tuning, we employed in-context learning (ICL)²⁷ to evaluate the ability of the original Pythia model suite to make predictions solely based on context augmented with a few examples. The Pythia suite was selected because it is the only publicly available suite of LMs spanning several orders of magnitude on a model scale trained on the same data of the same order²⁶. This characteristic enabled us to study the relationship between the model scale and its effectiveness in deidentification. Furthermore, these models were used by most of the competition participants, and our results could serve as baselines for comparison with the participant results to understand the effectiveness of their proposed approaches.

The primary contributions of this study are as follows: (1) first-ever deidentification and temporal information normalization corpus release: we released the first-ever deidentification corpus annotated with both SHI spans and normalized temporal values for date SHIs, with date surrogates generated via consistent date shifts, ensuring temporal integrity in longitudinal data. (2) Comprehensive performance analysis of LLM fine-tuning: We conducted a detailed analysis of LLM fine-tuning methods for SHI deidentification and temporal normalization across various configurations, including lower-rank adaptation (LoRA)²⁸ and full-parameter tuning²⁹, across a range of model sizes. Our experiments demonstrate that while fine-tuning improves performance, it also reveals the inverse scaling issue³⁰, where models with more than 6B parameters tend to overfit, particularly due to the imbalanced and moderate size of the dataset—consistent with previous findings³⁰. (3) Effectiveness of hybrid approaches: Our evaluation shows that LLMs excel at SHI recognition through ICL, but benefit from hybrid approaches that combine with pattern-based methods for temporal normalization, where structured patterns and normalization rules are essential. This is also supported by analyses of top-performing team methods. However, we also identify persistent challenges even with hybrid approaches, such as handling typos, rarely temporal formats, and relative temporal expressions. (4) Data augmentation for imbalance issues: Based on an analysis of 30 submitted technical reports and proceeding papers²⁵, we emphasize the role of data augmentation in addressing dataset imbalance. Our findings show that LLM-based synthetic data generation can improve model performance for underrepresented SHI categories, as evidenced by comparisons of baseline models with the top-performing teams’ methods. (5) Future insights into LLM-based deidentification: Our findings suggest that hybrid methods that leverage LLMs for generalization while maintaining the control and interpretability of pattern-based approaches are highly effective. This need for fine-grained control was further highlighted by Ralevski et al.³¹, who compared the impact of using original versus de-identified clinical notes in the abstraction of SDoH. They found that the removal of certain SHIs, such as dates and locations, could hinder the accurate detection of the contextual clues needed for the LLM analysis of housing instability. Consequently, we emphasize that future research on LLM-based deidentification should focus on balancing model performance with the controllability and fidelity of surrogate outputs, ensuring both compliance and utility, especially in the age of generative AI.

Results

The OpenDeID v2 corpus

To evaluate the performance of various machine learning methods and hybrid approaches that combine machine learning and rule- or pattern-based methods developed by participants in the SREDH/AI CUP 2023 deidentification competition, the OpenDeID v1 corpus^32,33, which was originally sourced from the Health Science Alliance (HSA) biobank of the Lowy Cancer Research Center of the University of New South Wales, Australia³⁴, was extended with an additional 1144 pathology reports curated from the HSA biobank, resulting in 3244 reports (referred to as the OpenDeID v2 corpus). The dataset was divided into three sets: training (1734 reports), validation (560 reports), and testing (950 reports) for the competition, respectively.

The dataset includes six main SHI categories: NAME, LOCATION, AGE, DATE, CONTACT, and ID. Except for AGE, each category has subcategories, as shown in Fig. 2 and Supplementary Table 1. All date annotations in the OpenDeId v2 corpus were subdivided into the following subcategories: DATE, TIME, DURATION, and SET. Each annotation was assigned a normalized value according to the ISO 8601 standard to ensure temporal integrity. The updated guidelines and detailed annotation procedures are available elsewhere^25,35. A more detailed statistical summary of the compiled corpus is provided in Supplementary Table 2.

Fig. 2. Overview of the OpenDeId v1 and v2 corpora.

The figure provides an overview of the OpenDeId v1 and the v2 corpora. The numerical values displayed in the sunburst graph indicate the number of annotations in each category.

In the following subsections, we present the ability of the original Pythia model suite²⁶ to address two subtasks, wherein the context is augmented with a few examples. We then demonstrated the results of fine-tuning the same models without applying sophisticated postprocessing rules proposed by the participants to establish baselines and better understand the effectiveness of the participants’ approaches.

In-context learning performance

The first analysis assessed the few-shot learning capabilities of Pythia models across different scales. Unlike our previous study³⁶, which relied on the zero-shot ICL of ChatGPT, we used the k-nearest neighbor (kNN)-augmented in-context example selection approach³⁷ to select the five closest training instances as in-context examples for few-shot ICLs in the Pythia scaling suite.

Figure 3 presents the results, covering models of varying sizes: 70 M (million), 160 M, 410 M, 1 B (billion), 1.4 B, 2.8 B, 6.9 B, and 12 B parameters. The results indicate that pretrained Pythia models with more than 1 B parameters performed comparably to the average performance of all submissions for Subtask 1 in the SREDH/AI CUP 2023 Deidentification competition. However, these models underperformed relative to the average F1 scores in subtask 2. Notably, the model with only 160 M parameters can achieve a performance close to the average macro-F score for Subtask 1. In contrast, for Subtask 2, models with at least 1.4 B parameters were required to reach micro/macro-averaged F1 scores over 0.30/0.15. Furthermore, models with a minimum of 2.8 B parameters were necessary to achieve macro-averaged F1 scores above 0.32, approaching the median value of all submissions in Subtask 2. Overall, the best performing model for subtask 1 is the 12 B model, which demonstrated superior effectiveness in SHI recognition. For Subtask 2, the top-ranked model operated at a scale of 6.9 B. This result provides insight into the scalability of few-shot learning capabilities within LLM models, such as the Pythia model suite, demonstrating that larger models are particularly beneficial for more complex tasks, such as temporal information normalization.

Fig. 3. ICL performance (few-shot learning with five examples) comparison for Subtasks 1 and 2.

The figure shows the micro- and macro-F1 scores for Subtask 1 and 2 using ICL with five-shot prompts. Results are reported on the test sets for Pythia models of various sizes: 70 M, 160 M, 410 M, 1 B, 1.4 B, 2.8 B, 6.9 B, and 12 B parameters.

In Fig. 4, we further investigate the impact of the number of in-context examples on the performance of both subtasks. The 2.8 B model was chosen as the representative model in the experiment because of its superior performance over small models and comparable performance to larger models. Specifically, in the training set, we chose the number of in-context examples to be 1, 3, 5, and 7, selecting the closest training instances and arranging them in the prompt as demonstrations by their distances, in ascending order. Similar to the previous observations³⁸, we noticed that the recall rates for the two subtasks improve as the number of examples increases. However, the improvement diminishes when the number of samples exceeds five.

Fig. 4. Effect of varying in-context examples on Pythia-2.8B performance across training tasks.

a Performance impact of using 1, 3, 5, and 7 in-context examples for Subtasks 1 and 2 with the original training set. b–d Comparative analysis of Pythia-2.8 B performance on Subtasks 1 and 2 using 3, 5, 7 in-context examples, evaluated on both the original and deduplicated training sets.

Fine-tuning performance

In this experiment, we applied fine-tuning to the Pythia models via two widely used approaches: full-parameter and LoRA-based parameter-efficient fine-tuning³⁹. We observed that, except for the 70 M model trained with LoRA, the fine-tuned models demonstrated a notable improvement in recall rates for both tasks after fine-tuning, resulting in improved F1 scores over those achieved through ICLs (Fig. 5). The detailed results are summarized in Supplementary Table 3.

Fig. 5. Performance comparison between the ICL and fine-tuned models based on full and LoRA fine-tuning.

The figure illustrates that model performance generally improves with increasing parameters; however, gains from fine-tuning diminish beyond a certain point, with a crossover observed between the 70 M and 160 M models. Optimal performance is reached at 2.8 B parameters for both subtasks. Beyond this, the performance plateaus or declines, particularly for LoRA and fine-tuning approaches.

Compared with the ICL methods, the fine-tuned models with fewer than 2.8 B parameters showed a notable performance improvement. However, for Subtask 1, the advantage of conventional full fine-tuning over ICL was not significant in models with >6B parameters. Similarly, for subtask 2, the macro-averaged F1 scores were lower than those achieved via ICL in the models with more than 6B parameters. In contrast, our results demonstrate that LoRA is a particularly promising tuning method, outperforming the traditional full-parameter tuning approach because of its superior performance and high efficiency in the training phase, especially in models with more than 1B parameters. Given the moderate dataset size used in this study, our findings align with those of Dutt et al.⁴⁰, who demonstrated that parameter-efficient fine-tuning methods, such as LoRA, consistently outperform full fine-tuning when working with smaller datasets. Interestingly, the 70 M model trained with LoRA underperformed compared to its counterparts, including both the ICL and full-fine tuning approaches. We attribute this underperformance to the smaller base model, which results in a limited number of learnable parameters after applying LoRA. In such cases, the effectiveness of the adaptations may be restricted because there may not be sufficient interaction among the parameters to capture the complex relationships needed.

Increasing the model parameters generally improves the performance, but the performance gains from fine-tuning begin to decrease after a certain point, with a crossover occurring between the 70 and 160 M parameters (Fig. 5). Additionally, optimal performance is achieved with models of 2.8 B parameters for both subtasks. Beyond this point, the performance improvements plateau and even deteriorate for the LoRA and fine-tuning approaches, respectively. We believe that this is due to the size of the dataset. By limiting the number of trainable parameters, LoRA helps mitigate overfitting in larger models when working with smaller datasets⁴¹.

SREDH/AI CUP 2023 competition

The competition was hosted on CodaLab⁴², a platform that enables participants to download the training set for model development and upload predictions for the validation set to receive immediate performance feedback compared to other teams via a leaderboard. During the final testing phase, participants submitted their predictions for the test set in CodaLab, which were evaluated and ranked using the macro-averaged F1 measure as the primary evaluation metric. The final rankings were determined by averaging the individual ranks of both tasks for each team and sorting them based on the average values. The competition attracted 721 participants, who formed 291 teams. In the final testing phase, each team was allowed up to six submissions over the course of one day, resulting in 218 total submissions. In total, 103 teams submitted their prediction results during the final testing phase.

Table 1 presents the detailed performance of the top ten teams, comparing both micro- and macro-averaged F1 scores for the two subtasks, alongside team numbers. In subtask 1, the top-ranked system developed by Huang et al.⁴³ utilized an LLM-based method based on Qwen⁴⁴, thereby achieving micro- and macro-averaged F1 scores of 0.945 and 0.881, respectively. For subtask 2, the highest-ranking system was developed by Zhao et al.⁴⁵, which relies on pattern-based approaches and achieves micro- and macro-averaged F1 scores of 0.844 and 0.869, respectively. The overall performance showed average micro/macro-F scores for subtasks 1 and 2 of 0.666/0.496 and 0.6/0.394, respectively, with median micro/macro-F scores of 0.810/0.710 and 0.679/0.360, respectively. These results highlight significant disparities in performance, indicating that subtask 2 (normalization of temporal information) presents greater challenges than SHI recognition.

Table 1.

Top ten teams’ performances for subtasks 1 and 2 on the official test set in terms of micro- and macro-recall (R), precision (P), and F scores (F)

	Subtask 1 (Micro/Macro)			Subtask 2 (Micro/Macro)
Team	P	R	F	P	R	F
Team 145, Supplementary Table 4	0.963/0.900	0.921/0.835	0.941/0.866	0.872/0.932	0.817/0.813	0.844/0.869
Team 243, Supplementary Table 5	0.939/0.906	0.951/0.859	0.945/0.881	0.820/0.889	0.716/0.712	0.765/0.791
Team 3 Supplementary Table 6	0.924/0.870	0.942/0.817	0.933/0.842	0.832/0.920	0.760/0.775	0.795/0.841
Team 453, Supplementary Table 7	0.922/0.906	0.878/0.854	0.900/0.879	0.813/0.890	0.691/0.689	0.747/0.777
Team 5, Supplementary Table 8	0.887/0.825	0.905/0.795	0.896/0.809	0.823/0.913	0.733/0.705	0.775/0.796
Team 651, Supplementary Table 9	0.917/0.919	0.908/0.841	0.912/0.878	0.810/0.892	0.739/0.630	0.773/0.738
Team 7 Supplementary Table 10	0.897/0.775	0.936/0.782	0.916/0.778	0.824/0.914	0.742/0.747	0.781/0.822
Team 866, Supplementary Table 11	0.919/0.812	0.901/0.746	0.910/0.777	0.818/0.900	0.733/0.731	0.773/0.807
Team 947, Supplementary Table 12	0.914/0.868	0.931/0.861	0.922/0.865	0.825/0.814	0.682/0.655	0.747/0.726
Team 10 Supplementary Table 13	0.916/0.802	0.942/0.749	0.931/0.775	0.833/0.916	0.779/0.653	0.805/0.762
Pythia-410 M-LoRA	0.961/0.848	0.925/0.741	0.943/0.791	0.839/0.802	0.767/0.561	0.802/0.695
Pythia-2.8B-ICL	0.906/0.824	0.611/0.500	0.730/0.621	0.580/0.338	0.678/0.247	0.427/0.362
GPT-3.5-FT [Team 1]	0.949/0.771	0.943/0.733	0.946/0.752	0.862/0.927	0.811/0.702	0.836/0.799

The scores in bold indicate the highest performance among the top ten.

A notable trend in the competition was the widespread use of LLMs (Fig. 6). Specifically, 77.2% of the 57 teams integrated LLMs into their systems, and 63.9% of the top 30 teams used them. Decoder-only transformer architectures were the most prevalent, comprising 66.1 and 83.3% of the submissions for Subtasks 1 and 2, respectively. Encoder-only architectures were the next most common in subtask 1, accounting for 21.0%, whereas encoder-only and encoder-decoder architectures were equally utilized in subtask 2, each accounting for 8.3%. This distribution underscores the current trends in Clinical NLP.

Fig. 6. Overview of methods applied by the top 30 teams.

The figure presents a detailed overview of the methods applied by the top 30 teams in the SREDH/AI CUP 2023 competition. The results highlight the widespread adoption of LLMs. 77.2% of the 57 teams and 63.9% of the top 30 teams incorporated LLMs. Decoder-only transformer architectures were most common, comprising 66.1% and 83.3% of Subtask 1 and Subtask 2 submissions, respectively. Encoder-only architectures were the next most common in subtask 1, accounting for 21.0%, whereas encoder-only and encoder-decoder architectures were equally utilized in subtask 2.

Table 2 summarizes the techniques used by the top ten teams for Subtask 1. Pretrained transformer-based LLMs have been widely used, with nearly all top-performing teams incorporating them into their solutions. Most teams approached Subtask 1 as a standalone sequential labeling task, separate from Subtask 2. However, Teams 2, 3, 4, and 9 adopted an end-to-end text generation approach to simultaneously recognize SHIs and normalize temporal expressions. Ensembles of multiple models are typically employed. Almost all the top-performing teams incorporated rule-based methods to some degree. Only one team applied pure pattern and dictionary-based approaches. In addition, two teams explored methods to increase the size of the training sets, particularly for SHI types with limited training instances, and to enhance the context diversity by manipulating tokens within the context.

Table 2.

Brief summary of the techniques used by the top ten teams for subtask 1

Team number	Technique	Additional Dataset
2	Qwen-14b model trained with linear warmup, QLoRA (quantization-LoRA) and NEFTune (noisy embeddings improve instruction fine-tuning).	No
4	Ensemble the outputs of the two approaches:	Synthetic datasets generated by ChatGPT
	1. Pythia-410 m model trained with linear warmup.
	2. Pattern-based approach
6	Longformer-base with the conditional random field (CRF) layer.	No
1	Pattern and dictionary-based approaches.	No
9	Ensemble voting is performed using the FLAN-T5-large model trained with LoRA and the Pythia-160 m with intermediate layers frozen.	Generate new training instances by replacing annotated SHIs with new surrogates
		Randomly add or remove non-SHI tokens in approximately 15% of the source sentences
		Use three templates for generating instance in instruction tuning
3	Ensemble the outputs of three models:	No
	1. A FLAN-T5-large model trained on the original training set
	2. A FLAN-T5-large model trained on the training set excluding SHIs from the DATE, TIME, DURATION, and SET categories
	3. Pattern-based approach
	4. GPT-3.5-turbo using few-shot in-context learning
5	Transition-based chunking model based on the RoBERTa-base model trained with linear warmup and BILUO tags	No
7	RoBERTa-large model trained with linear warmup and BILUO tags	No
8	Ensemble outputs of two approaches:	No
	1. FLAN-T5-Efficient-BASE-DL2 model
	2. Pattern and dictionary-based approaches
10	Ensemble the outputs of the PERT (pretraining BERT with permuted LM) model and the CRF model with BIO tags.	No

The teams are listed in descending order based on their macro-F scores.

Table 3 summarizes the techniques used by the top-performing teams for Subtask 2. While ensemble and pretrained transformer-based LLMs are still widely used, seven out of ten teams have incorporated pattern-based approaches. Although several top-performing teams used commercial LLMs, such as OpenAI’s ChatGPT-3.5²⁷ and Google’s PaLM2⁴⁶ during experimentation, only Team 5 utilized PaLM2 for their final submission, achieving micro/macro-F1 scores of 0.747/0.777 (Supplementary Table 8). Although the ICL method applied by the team was simpler than the baseline models, when a fixed set of few-shot examples was used, the commercial PaLM2 model outperformed our ICL with the Pythia-6.9 B model.

Table 3.

Summary of techniques used by top-performing teams for the temporal information normalization subtask

Team number	Technique	Additional Dataset
1	Pattern-based approaches.	No
3	Ensemble the outputs of three models:	No
	1. A FLAN-T5-large model trained on the original training set
	2. Pattern-based approach
	3. GPT-3.5-turbo using few-shot in-context learning (ICL)
7	Ensemble the outputs of the RoBERTa-large model trained with linear warmup and BILUO tags and the pattern-based approach	No
8	Ensemble of the output of the FLAN-T5-Efficient-BASE-DL2 model and the pattern-based approach	No
5	PaLM2 model with few-shot ICL	No
2	Ensemble the outputs of the Qwen-7b model trained with linear warmup, QLoRA (quantization-LoRA) and NEFTune (noisy embeddings improve instruction fine-tuning) and chain-of-thought Few-shot ICL	No
4	Ensemble the outputs of the Pythia-410 m model trained with linear warmup and the pattern-based approach	Synthetic datasets generated by ChatGPT
10	Ensemble the outputs of the DeBERTa model and the pattern-based approaches	No
6	Pattern-based approach	No
9	The instruction tuned FLAN-T5-large model trained with LoRA.	Generate new training instances by replacing annotated SHIs with new surrogates.

The teams are listed in descending order based on their macro-F scores.

Compared with the performance achieved by the top ten teams, as shown in Supplementary Fig. 1, the models fine-tuned with LoRA achieved comparable micro-F₁ scores for both subtasks. However, most top-ranked teams presented better macro-F₁ scores, particularly for Subtask 2. This suggests that, while LoRA fine-tuning is effective, the enhanced methods proposed by the top ten teams could improve the ability of the pretrained models to handle variability across different SHI subcategories. We briefly summarize the enhanced methodologies employed by the top ten teams to provide insights learned from the competition. More details about the methods used by four of these teams can be found in the Supplementary Note 3.

The ensemble approach was found to be more effective. Team 9⁴⁷ compared the performance of BERT-based models^48,49 via a sequential labeling formulation with that of generative models based on Finetuned Language Net (FLAN)⁵⁰. Their experimental results showed the superior performance of the Pythia and FLAN ensembles compared to the BERT-based ensemble. Team 6⁵¹ employed an ensemble method to merge the prediction results of five Longformer-based⁵² models trained on distinct data splits. They observed that ensemble learning could significantly reduce the model prediction variability, particularly for categories with limited training instances (Supplementary Table 9).

Given the unbalanced nature of the released training set, the data augmentation method is another critical factor for improving the ability of fine-tuned models to identify all the SHI categories. Some teams^53,54 prompted ChatGPT to generate additional training instances to augment SHI types with limited training examples, or those where their fine-tuned model performed poorly. For example, the experimental results of Team 4 showed that LLM-based augmentation significantly improved the macro-averaged F₁-score of the 410 M Pythia fine-tuned model by 0.20 (Supplementary Table 7). Team 9 improved the model performance (Supplementary Table 12) for low-resource SHI categories by generating training samples with synthetic surrogates, introducing random noise‒ by inserting irrelevant words, and removing non-SHI words from ∼15% of the sentences‒to enhance the model’s robustness and adaptability⁴⁷. Finally, Gupta et al.⁵⁵ was the only team to use other openly available de-identification datasets beyond the released training set to improve the system performance. They augmented the training set with the 2014 i2b2/UTHealth⁵⁶ and 2016 CEGS N-GRID¹² deidentification corpora to train a BERT-based model⁵⁷ for subtask 1. However, they did not observe any performance improvements with this configuration.

In terms of training methods, most teams apply full-parameter fine-tuning to LLM models with fewer than 1B parameters owing to the limitations of the available computing resources. While full-parameter tuning is computationally intensive, the smaller model sizes make this approach feasible within the constraints of available computing resources. For larger models, teams have predominantly used parameter-efficient tuning techniques, such as LoRA or quantization-LoRA (QLoRA), to mitigate resource limitations. For example, Ru et al.⁵⁸ fine-tuned Pythia models (70 M, 160 M and 1B) with QLoRA and reported that more parameters in the pretrained model led to generally better prediction results. Chiu et al.⁵⁹ proposed combining fine-tuning and LoRA, a method similar to that of the Chain-of-LoRA⁶⁰, to enhance the performance of LLMs in the presented task. Two teams^43,51 proposed the application of sliding window methods to enable BERT-based models⁵² and LLMs⁴⁴ to aggregate training instances across sequences, thereby enhancing the generalizability and capability of the model to interpret and predict longer sequences. To minimize overfitting, two approaches were highlighted by the participants: Team 2 applied a noisy embedding instruction fine-tuning method⁶¹, and Team 9 applied parameter freezing in the intermediate layers and low-rank adaptation (LoRA). The detailed results of Team 2 and Team 9 are in Supplementary Table 5 and Supplementary Table 12, respectively.

Nearly all the top-ranked teams applied pattern-based methods to varying degrees, as shown in Fig. 6 and Tables 2, 3. Teams that focus on LLM-based approaches have highlighted the issue of named entity hallucinations⁶², which refers to out-of-report (OOR) or out-of-definition (OOD) SHIs. These teams used pattern-based methods to filter out OOR and OOD instances and duplicate instances generated by LLMs. For example, Li et al.⁶³ reported that postprocessing for hallucinations improved their micro- and macro-averaged F-scores by 0.018 and 0.170, respectively. Furthermore, many top-ranked teams^{43,52,53,58,63–66} have applied hybrid methods that combine LLMs and pattern-based approaches to increase recall rates, particularly for the recognition of age- and temporal-related SHIs, and for the normalization of temporal SHIs. In contrast, team 1⁴⁵ and Huang, et al.⁶⁷ presented two pure pattern-based approaches and demonstrated that heuristic patterns can achieve state-of-the-art performance (Supplementary Table 4). In the comparative study of GPT 3.5 fine-tuning and pattern-based approaches presented by Team 1, although pattern-based approaches require more human involvement to improve rule accuracy and coverage iteratively than data-driven LLM methods, they offer significantly better computational efficiency and lower computing power requirements.

Finally, regarding the use of commercial pretrained LLMs, GPT-3.5 was the most popular choice. We found that only one team⁶⁸ from the top 30 conducted a small-scale study to examine the feasibility of applying ICL with GPT-3.5 to both subtasks. Although they observed promising results in their study, they ultimately decided to shift to local LLM fine-tuning and used ChatGPT solely for data augmentation, owing to the cost associated with API calls for large-scale experiments. Only Team 1 applied fine-tuning based on GPT-3.5, and achieved F-scores of 0.752 and 0.799 for Subtasks 1 and 2, respectively (Supplementary Table 4). This performance is comparable to our results for LoRA fine-tuning of Pythia-2.8 B (0.764/0.650).

Discussion

In this study, we systematically analyzed the performance of LLMs for deidentifying SHI and performing temporal normalization within clinical notes. Our findings highlight key trade-offs across modeling strategies, including full fine-tuning, parameter-efficient adaptation, ICL, and pattern-based methods. We also observed a paradigm shift from encoder-only to decoder-only transformer architectures—driven by the latter’s strong zero- and few-shot generalization capabilities⁶⁹. Below, we summarize core techniques used by top-performing teams and identify areas that warrant further investigation.

One of the key techniques was the use of hybrid approaches that combine LLMs with pattern-based methods. These approaches have been widely recognized for improving deidentification performance⁷⁰ and have proven successful in broader information extraction tasks across clinical⁷¹ and business domains⁷². While pattern-based methods require manual rule engineering and lack generalizability, they can be highly effective when the context is relatively structured and normalization rules are well defined as demonstrated by Team 1. For instance, pattern-based methods achieved higher recall in SHI categories with consistent patterns, such as MEDICALRECORD, TIME, and PHONE, and exhibited superior precision when patterns were observable in the training set⁴⁵. However, these methods struggled with less frequent or irregular formats, such as “1745H on 18.02.66 (TIME: 2066-02-18T17:45)” and “15:55 h on the 22-Oct-2068 (TIME: 2068-10-22T15:55)”, or entries with typographic errors (e.g., “20.04.2070 at 12;45 pm (TIME: 2070-04-20T12:45)). Additionally, no team proposed integrated solutions for normalizing relative temporal expressions, such as “now,” “today,” or weekday names like “Friday,” as illustrated in Fig. 1. Furthermore, SHI categories requiring contextual understanding, such as NAME and ORGANIZATION, showed lower recall with pattern-based methods compared to LLM-based models. Larger models generally offered better generalization capabilities, as demonstrated by our Pythia-2.8 B ICL model and Team 5’s use of PaLM-2 in Subtask 2. These models, especially when used with prompt engineering in ICL, require no fine-tuning and still offer strong performance.

Fine-tuning further improves task-specific performance, particularly for Subtask 1, as shown by comparison between our ICL baselines and the fine-tuned models from Teams 2 and 4, as well as a comparison of Team 1 with other top-performing teams. However, for Subtask 2, fine-tuning offered no consistent advantage; Team 5’s ICL-based PaLM-2 model outperformed other fine-tuned systems like Teams 2 and 1’s GPT-3.5. This suggests that pattern-based methods remain optimal for highly rule-driven tasks like temporal normalization. Therefore, in the era of LLMs, we advocate for hybrid strategies that leverage LLMs or ensembles for more context-sensitive categories (e.g., NAME, ORGANIZATION) and pattern-based methods to well-structured categories (e.g., MEDICALRECORD, TIME, PHONE). This approach is particularly effective in resource-constrained settings, offering a balance of performance, cost-efficiency, and interpretability. In such contexts, the ability to trace and control specific rules triggered during deidentification is essential for compliance with healthcare regulations like HIPAA. Unlike end-to-end LLM approaches, hybrid systems provide greater transparency and auditability—allowing institutions to define interpretable, domain-specific rules while still benefiting from LLM generalization. Looking ahead, we believe that future deidentification research, especially in the era of generative AI, should emphasize not only model performance but also controllability and explainability of surrogate outputs. Techniques such as controllable text generation⁷³ can ensure that outputs remain compliant, interpretable, and reliable for downstream clinical or research use.

While hybrid approaches offer a compelling foundation, some scenarios demand more adaptable or targeted solutions like fine-tuning, particularly using LoRA, which can improve the performance of the pretrained LLM for both subtasks. However, our experiments revealed that the performance gains plateaued or even degraded for models with more than 6 B parameters, reflecting the inverse scaling issue observed by McKenzie et al.³⁰, in which larger models are more prone to memorizing data, which may lead to a drop in performance. In our cases, although micro-F scores peaked at a model size of 2.8 B parameters with full fine-tuning, macro-F scores declined as the model size increased, revealing overfitting on frequent SHI categories in a moderately sized and imbalanced dataset (Fig. 5). These results suggest a potential saturation point at which increasing the model complexity does not yield proportional performance gains⁷⁴. The 410 M Pythia model appears to strike a balance between capacity and generalization.

To mitigate class imbalance and improve data diversity, several teams used LLM-based data augmentation to synthesize rare SHI examples. Compared to our Pythia-410 M model (as shown in Table 2), Team 4’s fine-tuned Pythia-410 model outperformed larger models by training on augmented data, highlighting that data quantity and variety are often more impactful than model scale^74,75. This was further supported by the underperformance of Team 1’ fine-tuned GPT-3.5 model relative to smaller models. In line with this, Carlini et al.⁷⁶ showed that large models are more prone to memorizing duplicated content⁷⁷—a risk we identified in our dataset, which contained ~19.8% duplicate training sentences after preprocessing. While we used sampling to reduce duplication, this may still have contributed to overfitting, especially in larger models. This may also explain the results of the ICL experiment in Fig. 4a, which showed that increasing the number of examples beyond five did not improve the performance. Deduplication plays a critical role in improving the language model performance^78–80. For example, Hernandez et al.⁸¹ found that even minimal repetition in training data can lead to large degradation in the performance of LLMs. To further explore this, we evaluated ICL using the Pythia-2.8 B model with five non-duplicated instances, which led to macro-F scores improvement of 0.031 and 0.052 for Subtasks 1 and 2, respectively. As illustrated in Fig. 4b–e and Supplementary Fig. 2, quality and diversity of ICL examples were more influential than quantity of selected instances. Based on these findings, we recommend deduplication as an essential preprocessing step in preparing training datasets for LM-based deidentification tasks. This not only improves training efficiency but also helps reduce memorization-related privacy risks, as emphasized by Kandpal et al.⁷⁷. Additionally, the sliding window method proposed by Team 2 offers an effective alternative (Supplementary Table 5). By concatenating k surrounding sentences per training instance, they reduced duplication and enabled models to capture to inter-sentence context—vital for inferring normalized expressions like “Friday” in Fig. 1 with the absence of explicit dates.

Despite the advancements and paradigm shifts in deidentification and temporal normalization tasks demonstrated in this study, several limitations should be acknowledged. First, our evaluation focused solely on pathology reports, a specific type of clinical narrative, and focused on the subcategories of SHIs present in those reports, along with temporal normalization tasks. Given that clinical documents can vary considerably in structure, content, and linguistic style, the generalizability of the presented methods to other types of medical texts remains uncertain. Future studies should investigate the performance and adaptability of LLM-based deidentification and normalization techniques across more diverse datasets to assess cross-domain robustness and writing-style variation. Second, training and fine-tuning large LLMs require substantial computational resources, which can lead to bias toward resource-efficient solutions rather than optimal ones. Consequently, the presented results, including those from the participating teams, may reflect a bias toward models and methods that fit the limited available resources. Although methods, such as LoRA and QLoRA, mitigate some of these costs, the resource burden remains high. Our experiments also revealed performance sensitivity to dataset duplication. While sampling helped, more extensive deduplication and controlled evaluation are needed to better understand their impact on model scalability and performance.

Finally, recent evidence⁸² indicates that LLMs may exhibit unintended bias across demographic attributes. For example, models tend to perform significantly better on text associated with high-income countries compared to those from lower-income regions. Such disparities raise important fairness concerns when deidentifying SHIs tied to identity attributes like gender, nationality, or disability. Addressing these issues requires more diverse and representative training data and the implementation of fairness-aware evaluation protocols. We highlight this as a critical area for future research to ensure the equitable and trustworthy deployment of LLMs in clinical deidentification tasks.

Methods

The OpenDeID v2 corpus, evaluation format and metrics

The OpenDeID v2 corpus was constructed based on the original OpenDeID corpus by including an additional 1,144 pathology reports. The annotators annotated these new reports according to the annotation guidelines³⁵. In addition, for the original corpus, the annotators rechecked the annotations and updated the annotations for dates, including absolute and relative dates, setting them as DATE along with normalized values. For mentions of dates alongside time descriptions, the spans were updated to include time descriptions and the labels were updated to TIME. Additionally, the DURATION and SET labels were used to annotate durations (e.g., since last Friday) and expressions that indicate quantifiers and intervals (e.g., “two times weekly”). All the aforementioned SHIs belonging to the data subcategories were assigned normalized values following the ISO 8601 standard, as shown in Fig. 1 and Supplementary Table 1, and were double checked by the annotators to ensure that they met the criterion for temporal integrity.

Achieving this level of de-identification requires initially recognizing the true SHIs, followed by applying a surrogate generation algorithm²³ to produce synthetic SHIs. The normalization process for date SHIs involves standardizing the date description according to ISO 8601²⁴. For example, in Supplementary Fig. 2, “14/02/3014 at 11:42” is normalized to “3014-02-14T11:42”, and Friday is normalized to “3014-02-18”. The normalization process for date SHIs aids the surrogate generation procedure by reducing ambiguities associated with date mentioned in texts. Therefore, after the annotation process was completed, the annotated file was processed using the ISLab deidentification tool¹³ to automatically generate surrogates for all annotated SHIs. Surrogates for all annotations and their corresponding mentions in the source notes were generated randomly and replaced to prevent reidentification. When the replacement target type belonged to the DATEs, the normalized value provided by the annotator was used to facilitate the surrogate generation. Following the algorithm²³, we generated random date shifts within the range of 1–730 days. Our implementation ensured that the generated surrogates retained the written form of the original data. For example, a date shift of two years and two days would transform “at 10:30 am on 28/4/73” to “at 10:30am on “30/4/75” and “Dr C Kobe at 4:16am on the 11th of October 2713” to “Dr D Johnson at 4:16am on the 13th of October 2715”.

Two files were released for each annotated report: deidentified text with all SHIs replaced by surrogates and SHI annotations. Supplementary Figs. 3, 4 show the examples of these data files. The annotation format illustrated in Supplementary Fig. 4 serves as the standard submission format for the SREDH/AICUP 2023 competition. Precision, recall, and micro/macro-F1 measures were used (see Supplementary Note 1).

Development of baseline models

Before applying ICL or fine-tuning the pretrained Pythia models, each report was preprocessed to generate the training instances. First, we applied sentence segmentation⁸³ to divide the narrative text in each pathology report into separate sentences. Annotations for Subtasks 1 and 2 were collected for each sentence to form a training instance, as illustrated in Fig. 7. If a sentence contained temporal information requiring normalization, the ‘=>’ symbol followed by the normalized value was added to guide the model in generating the normalization results. If a sentence does not contain any SHI to be recognized, the output of the model is represented by ‘SHI: NULL’. For sentences containing more than one SHI, all annotations are concatenated by ‘\n’. This process can result in duplicate training instances. To address this, we randomly sampled up to three duplicates per sentence and excluded the remaining duplicates from the final training set to prevent overfitting and reduce the training time. Additionally, we extended Pythia’s built-in tokenizer vocabulary with special symbols defined in the Supplementary Table 14.

Fig. 7. Training instances used for ICL and fine-tuning.

The figure presents the training instance that was used for ICL and fine-tuning. The model input is shown on the left side, and the expected corresponding output is listed on the right. Each report was preprocessed to generate the training instances before applying ICL or fine-tuning the pre-trained Pythia models.

To develop the baseline models, we fine-tuned the pretrained Pythia suite models using both full-parameter fine-tuning and LoRA. Following the format shown in Fig. 7, the Pythia models at all scales were fine-tuned using the input-output pairs. The entire training set, including the validation set released during the SREDH/AICUP 2023 competition, as depicted in Fig. 2, was used and represented in the format shown in Supplementary Fig. 5 to fine-tune the models. Duplicated training instances were sampled and retained at most three times to avoid bias and overfitting caused by repeated examples, which could increase training costs and reduce the model’s ability to generalize effectively. The Pythia models were then fine-tuned on the converted training data using Adam and the Zero Redundancy Optimizer (ZeRO)⁸⁴ to enhance training efficiency, particularly for models with more than 6 B parameters. Detailed information on the model training process, fine-tuning templates, and hyperparameter settings is provided in the Supplementary Note 2. In addition to fine-tuning the Pythia models, we applied kNN-augmented ICL to directly prompt the Pythia models to learn both the SHI recognition and temporal normalization tasks from an analogy. The template shown in Fig. 8a was used to format the in-context demonstrations to instruct the model to complete the two subtasks simultaneously. For example, “D.O.B.: 09/08/2957”, after the template is applied, the prompt used to instruct the Pythia models to process the input text “Last edited: 17/7/2062 Page: 2” is illustrated in Fig. 8b.

Fig. 8. In-context Learning Setup and Prompt Structure for Model Evaluation.

The template includes task instructions, in-context demonstrations, and a target input for prediction. a Template for in-context learning. Each prompt included natural language instructions, k in-context examples, and unseen text as input. b Example of a prompt with one in-context example for instructing Pythia models. Figure presents the template that was used to format the in-context demonstrations to instruct the model to complete the two subtasks simultaneously. For example, “D.O.B.: 09/08/2957”, after the template is applied, the prompt used to instruct the Pythia models to process the input text “Last edited: 17/7/2062 Page: 2” is illustrated in (b).

Finally, to generate the output according to the desired format shown in Fig. 7, focusing only on the target SHI categories, the outputs of the fine-tuned models were postprocessed to avoid hallucinations, where the model generates text that is plausible sounding but factually incorrect. The predicted SHIs, whose types did not match any of the subcategory definitions listed in Table 3, were removed to avoid OODs. Additionally, if the output SHIs do not have corresponding appearances in the source input, the SHIs are ignored to eliminate OORs.

Author contributions

H.J.D., C.T.C., and J.J. contributed to the study conception, design, and funding. Data collection and analysis were performed by H.J.D., T.H.M., C.T.C., and J.J. The manuscript was written by H.J.D., T.H.M., C.T.C., and J.J. All authors have read, reviewed, and approved the final manuscript.

Data availability

The OpenDeID v2 corpus used in the SREDH/AICUP 2023 competition is publicly accessible and is subject to approval by signing a data-usage agreement. Access can be requested through the SREDH consortium GitHub page at https://github.com/SREDH-Consortium/OpenDeID-Corpus or https://www.sredhconsortium.org/sredh-datasets/opendeid-corpus-dataset.

Code availability

The implementations of the top-10 teams can be found at https://github.com/SREDH-Consortium/SREDH-AI-Cup-2023 to facilitate the understanding and reproduction of their results. The code used in this study for ICL and fine-tuning was developed using the PyTorch and HuggingFace APIs, as detailed in the tutorial on the official competition website at https://codalab.lisn.upsaclay.fr/competitions/15425#learn_the_details-add_info. Additionally, the fine-tuned models used to recognize SHIs are available upon request.

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at 10.1038/s41746-025-01921-7.