A machine-assisted framework for systematic error analysis in clinical concept extraction

Abstract

Error analysis is critical for evaluating and improving clinical natural language processing (NLP) models, yet no standardized framework exists to support systematic error analysis for clinical concept extraction. Here we present MedError, a machine-assisted, human-in-the-loop framework integrating a validated error taxonomy, large language model (LLM)-assisted classification, and a structured annotation workflow. MedError is developed from 1187 unique errors manually curated from 4227 clinical notes across three institutions, spanning 25 error types and 48 clinical concept categories. Here we show that MedError supports both single-site and federated multisite error analysis through three complementary evaluations. First, benchmarking of six proprietary and open-source LLMs shows that automated error classification alone is insufficient without human involvement. Second, MedError-derived error feedback improves NLP extraction performance by 34–62% over an unstructured baseline across 11 cognitive status concepts. Third, prospective external validation at two independent institutions demonstrates improved agreement with expert-adjudicated annotations by up to 30% (McNemar exact test, p = 0.035, φ = 0.38) and reduced annotation time by 26–33%, providing a systematic and practical framework for real-world clinical NLP deployment.

MedError, a machine-assisted human-in-the-loop framework, improves the standardization, validity, and efficiency of clinical NLP error analysis, supporting more transparent and trustworthy model deployment.

Introduction

Clinical concept extraction is a sub-task of natural language processing (NLP) that aims to extract pre-defined clinical concepts from unstructured text. The task includes concept mention detection and concept encoding¹. The primary source of information for clinical concept extraction is the electronic health record (EHR), a digital version of patients’ paper charts that provide a longitudinal, real-time record of an individual’s health information. Various NLP applications have been developed for different types of EHR notes, such as clinical notes, radiology reports, operative reports, and pathology reports, covering a wide range of clinical tasks and disease domains^1–3.

Error analysis is the systematic evaluation and documentation of errors produced by an NLP model, typically performed after the model has been applied^4,5. It plays an essential role across various phases of the model development, validation, and deployment cycle. During the training and validation phase, error analysis supports performance optimization by identifying failure patterns and informing model refinements. In the testing phase, it helps document limitations and improves model interpretability, providing actionable insights for future development. Another important application of error analysis arises during model deployment to new healthcare system environments or datasets. Internal model validity can shift substantially due to contextual changes across institutions⁶, and differences in EHR systems⁷, documentation patterns, clinical workflows, and data quality all contribute to heterogeneous declines in model performance. In such scenarios, researchers assess model portability, which refers to a model’s ability to maintain performance “out of the box” or after minimal refinement when applied to data from different institutions or EHR systems^8,9. Prior studies have reported F1-score (harmonic mean of precision and recall) degradation of up to 72% across multiple disciplines, such as mental health, cardiology, and orthopedics, underscoring the variability in generalizability^8,9. In these cases, detailed error analysis offers a mechanism to trace the sources of variability and identify the causes of performance degradation and further assists the development of new strategies to adapt, improve, or re-train the model for the new environment.

The process of conducting error analysis is typically performed manually by reviewing and categorizing model errors, followed by iterative refinement processes to optimize performance. Unlike corpus annotation, which is a standardized process for creating gold-standard datasets, error analysis requires evaluators to possess a strong understanding of both the clinical domain and NLP model behavior. Since various heterogeneous factors (e.g., documentation style, note structure, or terminology use) can significantly influence the form and format of clinical text, evaluators need to reachconsensus on how errors are interpreted and categorized. As clinical NLP models are increasingly deployed in real-world settings, yet systematic approaches for error analysis remain limited.

In this study, we present MedError, a machine-assisted, open-source framework that standardizes and enhances error classification and reasoning in clinical concept extraction tasks. Our approach leverages the contextual understanding and reasoning capabilities of large language models (LLMs), which have shown remarkable performance in complex NLP tasks requiring syntactic, semantic, and conceptual abstraction. We hypothesize that the cognitive demands of error classification, such as contextual interpretation and domain-specific clinical knowledge, are well-suited to the few-shot learning capabilities of LLMs. To test this hypothesis, we deployed four validated clinical NLP models (two symbolic and two transformer-based) on 4227 fully annotated clinical notes across three distinct hospital systems. Errors were manually labeled using a previously developed taxonomy⁵. We then evaluated six language models (three proprietary and three open-source) on their ability to classify clinical NLP errors across 25 and 15 predefined error classes. A corpus statistics and linguistic analyses was conducted to better understand model performance across heterogeneous settings, stratified by model types, error classes, and linguistic features. Based on our analysis, we derived an interactive framework, MedError, to guide the systematic error analysis assisted to facilitate adoption in real-world clinical NLP development and evaluation. Internal model refinement evaluation was conducted to assess whether MedError-derived error feedback improves downstream NLP performance by providing structured error feedback to a mechanistic NLP model across four experimental conditions. External validation was conducted at Mayo Clinic and the University of Pittsburgh through the ENACT NLP Working Group to evaluate generalizability.

Results

Corpus and error summary statistics

In total, we collected and manually curated 1187 unique errors (after de-duplication) from a total of 4227 notes. The corpus statistics indicate substantial variation in note length across institutions and concept extraction tasks (Fig. 1). For functional status notes, Memorial Hermann Health System (MHHS) had a mean word count of 1147 (95% CI: 1093–1201) compared with 465 (95% CI: 441–488) at UT Physicians (UTP). Similarly, cognitive status notes were longer at MHHS (mean 2053 words, 95% CI: 1931–2176) than at UTP (mean 1400 words, 95% CI: 1331–1470). Social status notes were generally shorter, with means of 316 words at UTP and 204 words at Texas Children’s Hospital (TCH). Wide ranges in word counts across datasets indicate substantial heterogeneity in documentation length.

Fig. 1. Characteristics of clinical notes across datasets, sites, and tasks.

a Word count distribution across six datasets. Bars represent mean word count per dataset. Error bars indicate 95% confidence intervals around the mean. Diamond markers indicate median word count. Vertical lines indicate minimum-maximum range. Each dataset is represented by a distinct color for visual differentiation. b Note type composition across datasets. Bars represent the percentage of clinical notes assigned to each note type category. Note types are color-coded as follows: Progress Notes (dark navy), Administrative/Export (medium blue), Communication (orange), Clinical/Assessment (light orange), Results/Reports (light blue), Discharge/Summary (very light blue), Social Work (purple), Instructions/Education (light gray), Procedures/Operations (medium gray), Nursing (dark gray), and Other (very light gray). n values indicate the total number of clinical notes per dataset. MHHS, Memorial Hermann Health System; UTP, UT Physicians; TCH, Texas Children’s Hospital. Source data are provided as a Source Data file.

Note type distributions further varied across institutions and tasks. At MHHS, functional notes were mainly progress (29.4%) and administrative/export notes (29.0%), while UTP functional notes were distributed across communication (24.1%), administrative/export (23.4%), and progress notes (22.1%). Cognitive status datasets at both MHHS and UTP consisted entirely of progress notes. Social status notes were task-specific, with all UTP notes from social work documentation, whereas TCH included mainly progress (56.0%) and clinical/assessment notes (23.0%). Additional corpus statistics can be found in Supplementary Table 12-13.

The total 1187 unique errors comprise 480 false negative and 707 false positive error types. Within this, we identified 25 error classes, with the three most frequent being Implied_Inference (i.e., a clinically meaningful concept not explicitly stated but directly supported by contextual or descriptive cues in the text, rather than general medical knowledge; e.g. Falls: “down on the floor”), Negation, and Exclusion. Across the four different tasks, 48 unique clinical concept categories were identified. The interactive Sankey diagram illustrates the top error classes associated with different concept extraction sub-tasks (Fig. 2).

Fig. 2. Distribution of clinical concept extraction errors across error classes and clinical concept categories.

The Sankey diagram illustrates the flow of 1187 errors from error type (left column) through error class (middle column) to clinical concept category (right column). Error types are shown as false positive (FP, n = 707, red) and false negative (FN, n = 480, dark blue). Error classes (gray) represent 25 distinct categories derived from the MedError taxonomy. Clinical concept categories are color-coded by task domain: cognitive status (blue), functional status (orange), and social status (green). Node width and flow thickness are proportional to error count.

The distribution of error classes by false negatives and false positives variant significantly across specific tasks. For example, in identifying whether a patient needs assistance with transferring, we observed that the model produced significantly more false negatives than false positives. Most of these false negatives reflected cases where the language used to describe transferring needs was semantically variable and non-standardized, relying on indirect or paraphrased expressions that deviate from explicit clinical terminology and are therefore more difficult for NLP models to capture reliably. In contrast, the falls event sub-task revealed a broader and more diverse range of false positive error types, primarily because the word “fall” is a common homonym. This highlights the need for word sense disambiguation to correctly interpret the semantic meaning of ambiguous terms within specific clinical contexts.

A total of 552 errors were associated with the functional status task, 481 with the cognitive status task, and 154 with the social status task. Common errors for the functional status task arose from clinical concepts expressed through indirect or action-oriented descriptions rather than explicit terminology, particularly for transferring, falls, and mobility, highlighting the need for models to infer underlying clinical meaning from contextual cues. For instance, in the sentence “the patient missed the last step while going down the stairs”, a fall event is not explicitly stated but is directly supported by the described action and outcome, making Fall an implied concept grounded in textual evidence. Cognitive status tasks, which involve identifying mentions of cognitive deficits, delirium, agitation, and related conditions, exhibited a mixture of negation, inference, and semantic errors. Language describing cognitive status can range from explicit statements (e.g., “patient is delirious”) to more nuanced or implied descriptions, often paired with standardized neurocognitive assessment documentation. This variability in expression likely contributes to typographical and morphological false negatives, where non-standard spellings, abbreviations, or inflectional variants of target terms (e.g., “encephalopthic,” “enceph.”) fall outside the model’s lexicon, an artifact of documentation style heterogeneity rather than a failure of linguistic understanding. On the other hand, semantic false negatives arose when cognitive concepts were conveyed through paraphrased or descriptive language that captures the clinical phenomenon without using standardized terminology. For example, “having trouble getting his thoughts out” does not explicitly mention disorganized thinking but provides a direct linguistic description of impaired thought organization, supporting its classification as an implied cognitive symptom. The primary error classes for social status tasks were Implied_Inference, Negation, and Exclusion. The most frequently affected concepts were food insecurity and smoking. Error types such as guideline error, absence of context, and future intent further underscore the challenges in detecting social status indicators, which are often embedded in non-clinical language, expressed implicitly, or associated with future-oriented statements.

LLM performance comparison

We evaluated six language models on their ability to classify clinical NLP errors across 25 and 15 predefined error classes. The F1 scores (harmonic mean of precision and recall) for GPT-4o, O1, O3mini, Llama-3.1-8B-Instruct (Llama-3.1 8B), Llama-3.1-70B-Instruct (Llama-3.1 70B), and Llama-3.1-405B-Instruct-FP8 (Llama-3.1 405B) on the 25-concept task were 0.505, 0.378, 0.338, 0.118, 0.297, and 0.379, respectively. On the 15-concept task, their F1-scores were 0.637, 0.531, 0.550, 0.205, 0.352, and 0.498, respectively. From the perspective of proprietary and open-source models, GPT-4o (proprietary model) achieved the highest F1-scores for both the 25-class (0.505) and 15-class (0.637) error classification tasks. Llama-3.1 405B, the largest open-source models, achieve an overall F1-scores of 0.379 (25-class) and 0.498 (15-class) and outperformed two proprietary models O1 and O3mini in 25-class task. The smallest Llama-3.1 8B model consistently underperformed across all tasks. This overall performance results serve as the benchmark result for comprehensive clinical NLP error classification across a diverse set of predefined error categories. The observed best performance is consistent with reported LLM performance on fine-grained clinical NLP classification benchmarks, where moderate effectiveness is common for multi-class tasks requiring nuanced clinical judgment. These results support the feasibility of LLM-assisted error analysis while confirming that fully automated classification remains insufficient for high-stakes clinical deployment¹⁰.

Table 1 presents the performance breakdown by three major clinical concept extraction tasks. For identifying cognitive status-related error classes, such as agitation, delirium, and disorganized thinking, where the language can be more specific and specialized compared to functional and social status tasks^11,12, GPT-4o yielded the strongest performance (F1 = 0.568 for 25 classes and 0.746 for 15 classes), followed by Llama-3.1 405B. The functional status task was dominated by the proprietary model GPT-4o and O1. This task contains the highest number of Implied_Inference error classes and rich linguistic error classes (Fig. 2), which requires the model to have both strong reasoning and computational linguistics capabilities to correctly identify errors like Homonyms and Orthographical errors. In the social status task, GPT-4o again outperformed others with an F1-scores of 0.487 and 0.506. Interestingly, Llama-3.1 70B achieved the second-highest performance on the social status task, with F1-scores of 0.370 and 0.403. Overall, we observed decreased performance across models when classifying errors related to functional status and social status, suggesting that these tasks likely involve more implicit, context-dependent, and semantic variable language when describing patient behaviors and social environments.

Table 1.

Overall performance comparison across six models

Task	Model	F1-score (25 error classes)	F1-score (15 error classes)
Cognitive Status (n = 481)	GPT-4o	0.568	0.746
	O1	0.412	0.530
	O3mini	0.443	0.576
	Llama-3.1 8B	0.183	0.304
	Llama-3.1 70B	0.366	0.437
	Llama-3.1 405B	0.511	0.684
Functional Status (n = 552)	GPT-4o	0.457	0.578
	O1	0.371	0.585
	O3mini	0.241	0.580
	Llama-3.1 8B	0.071	0.145
	Llama-3.1 70B	0.217	0.264
	Llama-3.1 405B	0.277	0.380
Social Status (n = 154)	GPT-4o	0.487	0.506
	O1	0.299	0.338
	O3mini	0.357	0.364
	Llama-3.1 8B	0.084	0.110
	Llama-3.1 70B	0.370	0.403
	Llama-3.1 405B	0.331	0.338

Bold values indicate the highest F1 score within each task. F1 score, the harmonic mean of precision and recall. GPT-4o Generative Pre-trained Transformer 4 omni, O1, OpenAI o1 reasoning model, O3mini OpenAI o3-mini reasoning model, Llama-3.1 8B/70B/405B Meta Llama 3.1 models with 8 billion, 70 billion, and 405 billion parameters, respectively, n number of annotated clinical notes.

We further analyzed the model performance across errors created by symbolic methods versus errors generated by transformer approaches (Fig. 3). Figure 3a present the top 10 most frequent error classes and suggest that there is high variation in the most frequent error classes between the two NLP approaches. Overall, all models yielded a better performance on symbolic generated error classes compared with transformer based. While symbolic models exhibited significantly higher error burden, their errors were more evenly distributed across structured categories such as Negation, Synonym, and Exclusion. In contrast, transformer-based models showed a highly concentrated error profile dominated by Implied_Inference, a context-dependent and semantically complex category (Fig. 3a). Consistent with this, error-type-specific analysis (Fig. 3b) presents a comparative analysis of performance across 25 error classes for six models, GPT-4o (blue), O1 (light blue), O3mini (lavender), Llama-3.1 8B (orange), Llama-3.1 70B (gray), and Llama-3.1 405B (dark gray). This analysis demonstrated that LLMs achieved higher F1 scores on structured, rule-aligned error types, but lower performance on context-heavy categories such as implied inference and hypothetical reasoning (Fig. 3b). These findings suggest that LLM-based error classification performance is influenced not only by error frequency but also by the underlying structure and complexity of error types.

Fig. 3. Comparative performance of large language models across error classes.

a Distribution of error categories across symbolic and transformer-based approaches. Bars represent the percentage of errors assigned to each category; the top 10 most frequent error categories are shown individually, with remaining categories grouped as Other. b Box plots show the distribution of micro F1 scores (harmonic mean of precision and recall) across error categories by method. Centre line indicates median; box bounds indicate 25th and 75th percentiles; whiskers extend to 1.5× the interquartile range. Individual data points represent F1 scores for each of the six models. c Heatmap of error-category-specific F1 scores by model. Cell values represent F1 micro scores; error categories are sorted by mean F1 score in descending order. Colors denote models in a, b: GPT-4o (blue), O1 (light blue), O3mini (lavender), Llama-3.1 8B (orange), Llama-3.1 70B (gray), and Llama-3.1 405B (dark gray). In c, color intensity reflects F1 score magnitude from white (0) to dark blue (1). Source data are provided as a Source Data file. GPT, Generative Pre-trained Transformer.

For model comparison, GPT-4o consistently outperforms the other four models in a majority of categories, particularly in linguistic and contextual tasks (e.g. Orthographic, Synonym and Negation). The O1 model achieved strong performance in the tasks that require higher reasoning, including Medical_Instruction, Medical_Question, Patient_Education, and Implied_Inference. These tasks were often associated with false positive cases and marked as Exclusion. The large open-source model, Llama-3.1 405B, demonstrated competitive performance in majority categories, which achieved the best performance in classifying error classes related to Morphological, Medical_Evaluation and Section. In summary, we found GPT-4o and O1 have broader generalization and contextualization strengths, and the largest open-source model can achieve competitive performance in specific subdomains. We also found a handful of error classes including Guideline, Future, Syntactic, and Typographical that yield poor agreements between model classification and human gold standard labels.

Corpus and linguistic analysis

Among the 91 linguistic features analyzed, 18 were statistically significant after FDR correction (Benjamini–Hochberg; all q < 0.05), with point-biserial correlations ranging from r = −0.250 (Clout) to r = 0.232 (Negations) (t-statistics ranging from −8.88 to 8.23, df = 1185) (Fig. 4). Features that were positively associated with performance included punctuation and syntactic markers, negation, affective cues, and clinical terminology. We believe that punctuation features help LLMs by enhancing the structural clarity of sentences, possibly because punctuation improves syntactic parsing and interpretability. The WC (word count) feature showed a negative correlation, suggesting that longer sentences are more difficult for models due to the challenge of capturing long-distance dependencies. Negation terms explicitly highlight the semantic polarity or certainty of events or clinical concepts, which correlates positively with LLM performance. Additionally, specific and explicit clinical terms (e.g., encephalopathy) tend to have more semantically constrained meaning than descriptive expressions (e.g., “status come and go”), which may further improve model accuracy. Conversely, several linguistic elements (e.g., clout [i.e., language associated with confidence, authority, and social dominance], verb, and pronoun) were negatively correlated with performance. These features may reflect more context-dependent or pragmatically nuanced language, which can introduce ambiguity and increase processing difficulty for current LLMs. In particular, inputs exhibiting more assertive or authoritative tone (as captured by clout) may be less explicit and therefore more challenging for models to interpret accurately.

Fig. 4. Point-biserial correlations between LIWC-derived linguistic features and model performance.

Features are grouped into three categories: linguistic (blue), psychological (orange), and social (purple) (n = 18 LIWC features). Positive correlations indicate features associated with improved model performance; negative correlations indicate features associated with reduced performance. Clout: the degree of confidence, authority, or social dominance expressed in language³⁴. Source data are provided as a Source Data file. LIWC, Linguistic Inquiry and Word Count.

The MedError framework

Based on the empirical patterns observed in this analysis, we derived the MedError framework to operationalize systematic error analysis. MedError is a machine-assisted, human-in-the-loop framework integrating a validated error taxonomy, large language model (LLM)-assisted classification, and a structured annotation workflow (Figs. 5, 6).

Fig. 5. User interface web application of large language model (LLM)-assisted error analysis.

The MedError interface accepts three user-provided files: annotation guidelines, error definitions, and model predictions, and generates a structured error analysis report. The left panel displays the active error taxonomy across dimensions. The right panel summarizes model performance and provides case-level details for each error. LLM, large language model; FN, false negative; FP, false positive; YAML, YAML Ain’t Markup Language; JSON, JavaScript Object Notation; NLP, natural language processing.

Fig. 6. The MedError framework for systematic error analysis and model refinement in clinical concept extraction.

The framework comprises four sequential stages: (1) Error Identification encompasses defining the target concept, electronic health record environment, and clinical context, with a structured multi-phase gold standard annotation workflow supported by MedTator (https://github.com/OHNLP/medtator); (2) Error Analysis involves applying a structured error taxonomy, sampling strategy, and LLM-assisted error classification in a HIPAA-compliant environment using MedError (https://github.com/OHNLP/MedError); (3) Error Interpretation and Reporting covers examining error distributions across dimensions, model types, and sites, with standardized reporting of corpus and model characteristics; and (4) Model Refinement involves iterative LLM-assisted rule generation and model re-evaluation on training and test sets. Abbreviations: EHR, electronic health record; ICU, intensive care unit; ED, emergency department; LLM, large language model; HIPAA, Health Insurance Portability and Accountability Act; NLP, natural language processing.

The framework contains an interface for organizing and supporting systematic error analysis (Fig. 5). This web application is a lightweight, serverless frontend built using the Vite build tool and the Vue 3 JavaScript framework. Designed for efficient client-side rendering and modular component loading, it requires no backend infrastructure, once built, it can be statically hosted on any CDN or file server. The app is optimized for performance with ES module preload and code-splitting, ensuring fast loading times and minimal runtime overhead. Its architecture is fully static and scalable, making it ideal for JAMstack-style deployments on platforms like Netlify, GitHub Pages, or Vercel. The input consists of annotation guidelines, an error analysis guideline, and a structured JSON file containing LLM outputs (error class and reasoning), gold standard annotations, and original model predictions for human-in-the-loop error analysis. Results are exported in a structured format compatible with MedTator, enabling seamless integration with existing in-browser annotation workflows for expert review, adjudication, and iterative refinement. The full framework, including the codebase, prompt templates, and schema, is available at https://github.com/OHNLP/MedError.

The conceptual framework operationalizes clinical NLP error analysis through four sequential stages (Fig. 6). In the Error Identification stage, the target clinical concept, EHR environment, and clinical context are formulated, and a gold standard corpus is developed through a structured multi-phase annotation workflow using MedTator (see Supplementary Fig. 1). Second, Error Analysis focuses on systematically identifying and categorizing errors using structured taxonomies (see Supplementary Note 2) and standardized workflows, with potential LLM assistance (e.g., GPT or Llama in secure settings) through the MedError interface. To conduct LLM-assisted error analysis, prompt design specifies input formats, reasoning expectations, and task-specific instructions, and LLM outputs are validated through a systematic human review process to adjudicate ambiguous cases. In the Error Interpretation and Reporting stage, error distributions are examined across dimensions and model types, cross-site heterogeneity is evaluated in multisite settings, and findings are reported against standardized criteria covering corpus characteristics, model methodology, and error patterns. Finally, Model Refinement leverages these insights through LLM-assisted generation of corrective rules or updates, followed by iterative retraining and rigorous re-evaluation, forming a continuous human- and LLM-in-the-loop cycle for robust and reproducible performance improvement in clinical NLP systems.

Internal evaluation of MedError derived error classes

A Friedman test revealed a significant overall difference in extraction performance across the three models (χ²(2) = 19.60, p = 0.0001) (Fig. 7a). Post-hoc pairwise Wilcoxon signed-rank tests with Bonferroni correction showed that GPT-5.2 significantly outperformed both GPT-4o (W = 199.0, p_adj = 0.039, r = 0.71) and GPT-4mini (W = 188.5, p_adj = 0.002, r = 0.73), with large effect sizes in both cases. The difference between GPT-4o and GPT-4mini did not reach significance after correction (W = 222.0, p_adj = 0.094, r = 0.68). GPT-5.2 was therefore selected for subsequent refinement experiments.

Fig. 7. Model refinement performance across experiments.

a Average F1 scores (harmonic mean of precision and recall) across concept categories for three models: GPT-5.2 (dark blue), GPT-4o (light blue), and GPT-4mini (orange). GPT denotes Generative Pre-trained Transformer. Bars represent mean F1 scores across 11 concept categories per model. Error bars represent 95% confidence intervals around the mean. Individual data points represent F1 scores for each concept category. b Performance across refinement stages: Baseline (gray, no refinement), E0 (lightest blue, error type only), E1 (light blue, error type and error class), E2 (medium blue, error reasoning only), and E3 (dark blue, error class and reasoning combined). Bars represent mean F1 scores across 11 concept categories per condition. Error bars represent 95% confidence intervals around the mean. Individual data points represent F1 scores for each concept category. Source data are provided as a Source Data file.

A Friedman test across refinement conditions revealed a significant overall effect of error feedback strategy on extraction performance (χ²(4) = 22.39, p = 0.0002). Compared with the unstructured baseline (macro F1 = 0.295), providing error type information alone (E0: F1 = 0.461; W = 0.0, p_adj = 0.034, r = 1.00) and LLM-generated reasoning without error class (E2: F1 = 0.479; W = 0.0, p_adj = 0.034, r = 1.00) both produced statistically significant improvements, with perfect effect sizes indicating consistent gains across all 11 concept categories (Fig. 7b). The improvement from E1 (F1 = 0.444; W = 0.0, p_adj = 0.051, r = 1.00) was marginal after Bonferroni correction. Combining error class and reasoning (E3: F1 = 0.394; W = 4.0, p_adj = 0.500, r = 0.91) did not reach significance. No significant pairwise differences were observed among E0-E3 conditions after correction (all p_adj ≥ 0.51), indicating broadly comparable performance across refinement strategies, with E2 yielding the numerically highest result.

External evaluation of MedError framework

MedError improved agreement with expert-adjudicated final annotations at both sites, non-significant improvement at Mayo Clinic (Mayo) and a statistically significant improvement at University of Pittsburgh (Pitt). At Mayo, agreement increased from 0.467 (14/30; 95% CI: 0.302–0.639) to 0.533 (16/30; 95% CI: 0.361–0.698), a gain of +6.7 percentage points (McNemar exact test, p = 0.754). At Pitt, agreement increased from 0.500 (15/30; 95% CI: 0.332–0.668) to 0.800 (24/30; 95% CI: 0.627–0.905), a gain of +30.0 percentage points (McNemar exact test, p = 0.035). The discordant pair breakdown showed 12 cases correctly classified only under MedError versus 3 only under Baseline relative to expert-adjudicated final annotations, a 4:1 ratio favouring MedError.

MedError reduced annotation time at both sites. At Mayo, total task time decreased from 30 to 20 minutes across 30 cases (1.00 to 0.67 min/case; −33.3%). At Pitt, where case-level timing was available for the Baseline condition (mean: 2.03 min/case, SD: 0.72, range: 1–3 min; total: 61 min), MedError reduced total task time to 45 minutes (1.50 min/case; −26.2%).

Discussion

Our study presents a set of methods in defining, collecting, modeling and analyzing real-world errors made by four existing and distinct NLP models in clinical concept extract tasks. In total, we collected and manually curated 1187 unique errors from three different EHR systems. The error corpus including 480 false negative and 707 false positive error types with the total of 25 error classes across 48 unique clinical concept categories. Systematic evaluation was performed on three proprietary LLMs and three open-source LLMs leveraged for automatic classification of clinical NLP errors across 25 and 15 predefined error classes.

We found that the task of error classification is inherently challenging due to the diverse error types, nuanced reasoning, domain knowledge, and context sensitivity required to accurately distinguish between fine grained error types. GPT-4o consistently leads across all three status (functional, cognitive and social) categories, suggesting that potential advantages such as longer context window inputs, multimodal capabilities, and the scale and coverage of training data contribute to its superior performance on nuanced classification tasks. We found that O1 shows interesting domain specific behavior by performing the best on Functional Status (15 class: O1 0.585 vs GPT-4o 0.578). This suggests that O1’s reasoning focused architecture may be particularly suited for certain types of status assessment but struggles with the complexity of cognitive evaluation. O3mini demonstrates the most dramatic improvement when moving from 25 to 15 error classes, especially in Functional Status (0.241 to 0.580). This suggests that the model has learned meaningful representations but is overwhelmed by fine grained distinctions. The Llama series reveals clear scaling effects, wherein Llama-3.1 405B significantly outperforms smaller variants. Llama-3.1 8B’s poor performance (F1 scores mostly under 0.2) indicates that these tasks likely require a minimum capability threshold that very small models cannot reach. Overall, the observed best performance (GPT-4o: 25-class [0.505] and 15-class [0.637]) indicates moderate effectiveness of LLM-based error classification. While these results support the feasibility of using LLMs to assist in systematic error analysis, they also highlight important limitations for practical deployment. We found the performance remains lower for context-dependent and semantically complex error types, suggesting that fully automated error classification is not yet sufficient for high-stakes clinical applications. Instead, these findings support a human-in-the-loop paradigm, where LLMs can efficiently prioritize and structure error analysis but require expert adjudication for nuanced or ambiguous cases.

Over the past two decades, numerous NLP models have been developed, yet only a small number have been successfully implemented in real-world settings or practically adopted for demonstration research¹³. This is largely due to persistent errors, lack of transparency in model decision-making, and insufficient validation across diverse clinical contexts, all of which hinder confidence in model outputs and limit adoption. Recognizing these barriers, the National Institutes of Health has recently issued several Notice of Special Interest (NOSI) to explicitly promote rigorous evaluation and transparent benchmarking of AI/NLP models in healthcare to advance real-world implementation and trustworthiness (e.g., NOT-CA-24-031). Our work directly addresses this challenge by introducing a systematic, machine-assisted framework for evaluating the clinical validity, portability, and explainability of NLP models used for clinical concept extraction. Example use cases of our framework include collaborative initiatives such as the Evolve to Next-Gen Accrual to Clinical Trials (ENACT) and the Observational Health Data Sciences and Informatics (OHDSI) networks, both of which increasingly require standardized NLP evaluation protocols and tools to ensure cross-institutional reproducibility¹⁴.

Error analysis is a crucial but often under reported, and under studied area due to the majority of NLP publications being largely based on measurement study design (evaluate the validity of the model) compared with demonstration design (apply the model to derive insights). Our effort aims to address current methodological and standardization gaps by proposing an informatics framework that supports efficient, user-friendly, and standardized error analysis. This framework is designed to facilitate more rigorous evaluation of NLP applications across diverse research settings and multi-institutional EHR environments. By improving the consistency and transparency of error analysis, our approach can enhance understanding of how to effectively implement and integrate NLP tools, ultimately supporting broader adoption and meaningful use of these technologies in clinical research and practice.

This study has several limitations. First, our work focuses exclusively on clinical concept extraction tasks. Clinical concept extraction was selected as the primary use case for MedError because it is a foundational and widely adopted task in clinical NLP, supporting downstream applications such as phenotyping, cohort identification, and clinical decision support. It captures many core linguistic challenges in real-world EHR documentation. However, it does not contain other important NLP tasks such as relation extraction or coreference resolution, which may limit the generalizability of the framework. While MedError is designed to be task-agnostic, extending it to other tasks will require adaptation of error definitions, prompt strategies, and review workflows. Second, the framework relies on high-quality annotations, structured taxonomies, and LLM-assisted classification, which may introduce variability related to annotators, model selection, and prompt design, particularly for rare or ambiguous error types. In addition, although our evaluation includes multiple institutions, all participating sites are large academic health systems; further validation in community-based, international, and non-English-language settings is needed to fully assess generalizability. Third, practical deployment of LLMs in real-world settings requires PHI-compliant environments, imposing technical and regulatory constraints. Fourth, the framework currently focuses on sentence- or span-level errors and does not explicitly model document-level or longitudinal reasoning across encounters. Furthermore, given the moderate performance of LLM-based error classification, particularly for complex and context-dependent error types, fully automated approaches remain insufficient for high-stakes clinical use. While MedError reduces manual burden, expert human review remains necessary, especially for nuanced or ambiguous cases. Finally, the curated error corpus, while substantial, reflects the characteristics of the evaluated datasets and may not capture the full diversity of error patterns in other clinical settings.

Future work should focus on validating the framework across diverse environments and developing human-AI collaborative approaches to enhance scalability and robustness. We will explore agentic frameworks to automate iterative error analysis and refinement. By enabling LLM agents to generate, evaluate, and update prompts based on structured error feedback, such systems could support more efficient human-in-the-loop model improvement. In addition, we plan to investigate methods for estimating the optimal number of refinement iterations, balancing performance gains with human annotation effort. This could help design practical workflows for real-world deployments.

Methods

This study involved retrospective analysis of existing electronic health record data. Written informed consent was waived for all study protocols by the University of Texas Health Science Center at Houston (UTHealth Houston) Committee for the Protection of Human Subjects (CPHS), the Institutional Review Board of record for this study, on the basis that the research posed minimal risk to subjects and could not practicably be carried out without the waiver. The study was approved by UT Physicians, Memorial Hermann Health System, and Harris County Psychiatric Center IRB and ethics committees (IRB#: HSC-MS-25-1090, HSC-SBMI-23-0765, HSC-SBMI-24-0403, HSC-SBMI-17-0354). The development of the machine-assisted error analysis framework involves a set of iterative processes including three major phases 1) error development, 2) error classification evaluation, and 3) error analysis framework development and evaluation (Fig. 8). The following sections describe each step in details.

Fig. 8. Methodology overview of the MedError development and evaluation process.

The process comprises three phases: Error Development (Steps 1–3), Error Classification Evaluation (Steps 4–6), and Framework Development and Evaluation (Steps 7–9). Abbreviations: EHR, electronic health record; LLM, large language model; GPT, Generative Pre-trained Transformer; LIWC, Linguistic Inquiry and Word Count; PHI, protected health information; NLP, natural language processing; TRUST, clinical text retrieval and use towards scientific rigor and transparent process; ENACT, Evolve to Next-Gen ACT; UTP, UT Physicians; MHHS, Memorial Hermann Health System; TCH, Texas Children’s Hospital.

Error generation

Despite the importance, large-scale, diverse error datasets from real-world EHRs remain scarce. Unlike public NLP datasets, EHR-derived corpora contain complex and institution-specific variations that LLMs have not been pre-exposed to, and when appropriately de-identified, they offer unique value for benchmarking LLM-assisted error analysis in novel clinical contexts. Due to potential variations in the EHR systems, documentation practices, and patient populations, the original NLP models may experience performance degradation when directly applied to a new environment. The process of collecting real-world NLP errors involves the following key steps: (1) clinical document collection, (2) gold standard development, (3) NLP model deployment, and (4) NLP model evaluation and error collection.

Clinical document collection was conducted across multiple healthcare systems, including UT Physicians, Memorial Hermann Health System, and Texas Children’s Hospital. A total of 4227 clinical notes were collected, comprising 2,891 notes from UT Physicians, 1,136 notes from Memorial Hermann Health System, and 200 notes from Texas Children’s Hospital. Corpus statistical analysis was conducted to compare site-specific heterogeneity in the information.

We then followed the TRUST annotation framework^6,15, which consists of task and annotation team formation, annotator training, annotation guideline development, annotation production, and post-annotation adjudication. The annotation team comprised a geriatrician (KZ), a trained senior clinical data abstractor (HL), and two medical students (GG, AC), and the process was overseen by two senior geriatrician (MK, NR) and a clinical informatician (SF). All annotators had prior experience in at least one clinical annotation study. Training activities included a detailed walkthrough of the annotation guidelines, pilot test annotations, and demonstrations of the annotation tool and data-sharing procedures. Annotation proceeded in two iterative consensus-building phases. Phase 1 involved systematic annotation of 25 randomly sampled clinical documents, with explicit evaluation of both false positive and false negative cases (where a positive case indicates documented impairment for a given geriatric care concept). Phase 2 focused on a stratified sample that ensured coverage of rare events by including at least three mentions per sub-task (120 mentions total). Each mention was independently annotated by four reviewers. Inter-annotator agreement (IAA) was assessed using the F1-score, calculated for each annotator against the majority vote, followed by structured adjudication discussions. The average IAA achieved was 0.845 in Phase 1 and 0.919 in Phase 2, indicating high annotation consistency and supporting the reliability of the fully annotated corpus.

We deployed and ran four previously validated and published NLP models originally developed at Mayo Clinic and the University of Kentucky (UK) on the UT Physicians (UTP) EHR systems. Four different clinical concept extraction tasks were considered, including cognitive status, functional status, and social status. Table 2 compares the F1-score performance of previously validated NLP models in their original development environments and after deployment in new institutional EHR systems. Across all tasks, model performance declined following cross-site deployment. Performance declined across all tasks following cross-site deployment. The cognitive status model decreased by 6.6%, the functional status ADL model by 5.3%, and the functional status fall model by 6.5%. The social status model showed larger declines of 15.4% at TCH and 8.6% at UTH.

Table 2.

Summary of concept extraction tasks, environment settings, and model types for error collection

Task Name	Model	Original Site	Applied Site	# of Notes	Model Type	Performance (F1-score before and after)
Cognitive Status	NLP-CAM28,31	Mayo Clinic	UT Physicians Memorial Hermann Health System	644 192	Symbolic	Before: 0.958; after: 0.895
Functional Status	Hybrid Transformer32, BERT17	Mayo Clinic	UT Physicians Memorial Hermann Health System	2047 944	Transformer, Hybrid	ADL: before 0.880, after 0.833 Fall: before 0.971, after 0.908
Social Status	MedTagger33	University of Kentucky	UT Physicians Texas Children’s Hospital	200 200	Symbolic, Transformer	Before: 0.985, after: 0.833 (TCH); after: 0.900 (UTP)

Bold text in the header indicates column labels. F1 score is the harmonic mean of precision and recall. NLP-CAM Natural Language Processing-based Confusion AssessmentMethod, BERT Bidirectional Encoder Representations from Transformers, MedTagger an open-source information extraction pipeline based on the Unstructured Information Management Architecture framework, ADL activities of daily living, TCH Texas Children’s Hospital, UTP UT Physicians.

Error definition

In our previous work, we developed a comprehensive error taxonomy comprising 43 distinct error classes organized into six error dimensions and four properties⁵. For the current study, we refined this taxonomy to align with the scope of machine-generated error analysis. Specifically, we removed (1) annotation error, as this study focuses exclusively on machine-generated errors; (2) logic error, which requires document- or patient-level reasoning beyond the sentence- or paragraph-level scope; and (3) rare or resource-dependent error classes (e.g., incomplete extraction, dictionary error, and normalization error) that require external knowledge bases for reliable assessment. These refinements resulted in a 25-class taxonomy, which we use as the primary fine-grained error schema.

During pilot annotation and model evaluation, we observed that both human annotators and LLM-based systems had difficulty reliably distinguishing between closely related fine-grained error types, limiting practical utility. In addition, reducing classification complexity is necessary to improve interpretability and real-world applicability. Therefore, in this study, we present a second, consolidated version of the taxonomy consisting of 15 classes. This reduced schema was derived by grouping conceptually related or low-frequency categories based on shared underlying constructs (e.g., uncertainty expression, semantic equivalence, or instructional context), following common practices in prior error taxonomies that adopt multi-level representations. The class-merging rules and full mappings between the two schemas are provided in the Supplementary Table 9. The finalized taxonomy definitions are publicly available on GitHub: https://github.com/OHNLP/MedError/tree/main/Taxonomy.

Error development

We formulate error analysis as a multi-class classification task operating on previously identified clinical concept extraction errors. Clinical concept extraction refers to the automated identification of predefined clinical concepts from unstructured text and typically consists of two components: concept mention detection and concept encoding. Concept mention detection is commonly implemented using named entity recognition (NER) techniques^2,16, which focus on identifying spans of text corresponding to clinical concepts. Concept encoding maps detected mentions to standardized terminologies or concept definitions required by downstream applications¹. Given an error instance produced by a clinical concept extraction system, the objective of the error classification task is to assign the error to a predefined category in the error taxonomy.

The error development workflow was guided by the TRUST process^6,15 and the National Center for Data to Health (CD2H) informatics playbook. The process involves a set of quality control iteration (e.g., annotation training, double annotation, IAA measurement) consensus development, and adjudication. All cases studies followed a standardized annotation guideline^11,17–19. We applied our previously established error definition to these newly generated errors through a formal annotation process performed by a team of trained abstractors (S.F., J.A., F.C., and Z.Y). The initial phase and second phase agreement ratios are 0.60 and 0.68, respectively. Quality check for all error cases was conducted an independent senior clinical NLP researcher. Example cases were described in Supplementary Table 10.

LLM implementation

We implemented GPT-4o, O3-mini, and O1 under the PHI-compliant Azure environment and Llama-3.1-8B-Instruct (Llama-3.1 8B), Llama-3.1-70B-Instruct (Llama-3.1 70B), and Llama-3.1-405B-Instruct-FP8 (Llama-3.1 405B) architectures within a local PHI environment. We employed in-context learning by providing error taxonomy, a task-specific guideline, examples, and structured prompts to guide model responses without requiring fine-tuning. Each error instance is represented as a structured record consisting of four elements:

1. a unique input identifier (ID)

2. the clinical concept type involved (e.g., falls, delirium, and food insecurity)

3. the input sentence providing the local textual context

4. the error class (false positive or false negative).

The recommended input length for LLM-based error classification is one sentence to approximately three short sentences, corresponding to roughly 30–150 tokens of clinical text. When additional context is necessary, the input may be extended to a single short paragraph, with an upper bound of approximately 300–400 tokens. No explicit named entity spans, assertion labels, or structured attributes are provided to the model beyond what is present in the sentence text and concept type.

Error classification evaluation

Model performance was evaluated against human annotations using standard metrics (See Statistics & Reproducibility section). To gain deeper insights into model behavior, we conducted a detailed error analysis by qualitatively reviewing both the misclassification outputs and the explanatory reasoning generated by LLMs. Our evaluation included: (1) overall performance comparison across models, (2) task-specific performance for different clinical concept extraction scenarios, and (3) error-specific performance stratified by predefined error classes.

Corpus and linguistic analysis

We employed the LIWC (Linguistic Inquiry and Word Count) text analysis tool²⁰ to extract well-validated psycholinguistic lexicons. LIWC leveraged a dictionary lookup approach to match the words that make up each text entry to sentiment-specific dictionaries^20,21. The tool was originally developed for psychology research. Due to its high effectiveness in emotion extraction and sentiment analysis, the tool has been widely adopted for various domains such as social computing and biomedical informatics^22–24. Based on previously established definitions, we examined a total of 91 linguistic and semantic features. A summary of LIWC feature definitions can be found in Supplementary Table 14.

Framework development

Framework development was guided by requirements gathered at two complementary levels. At the user level, the Nominal Group Technique was employed to systematically consolidate input from domain experts, existing literature, and established guideline and reporting standards, ensuring the framework reflects real-world clinical NLP workflows and evaluation needs. At the system level, the framework was designed to comply with EHR interoperability standards, support federated data infrastructure for multisite deployment, and operate within PHI-compliant environments to meet institutional data governance requirements.

Based on these requirements, we developed and released MedError, an open-source framework that aims to improve the validity, usability, efficiency and reproducibility of the error analysis process across diverse clinical NLP settings. To support transparency, reproducibility, and practical adoption, we provide three complementary supplementary materials that operationalize the framework. Supplementary Note 1 (Task Guideline) describes the annotation protocol and gold standard development process. Supplementary Note 2 (Error Guideline) defines the error taxonomy and provides detailed guidance for both prompt construction and human evaluation. Supplementary Note 3 (LLM Prompt) contains the full prompt templates used for LLM-assisted error analysis.

Internal evaluation of MedError derived error classes

The internal evaluation assessed whether the structured error information produced by the MedError framework could be used to guide automated model refinement of a symbolic NLP system. In contrast to prior tasks that focused on error identification, this task evaluates the capacity of LLMs to facilitate model refinement using previously identified error classes. In order to make the scope manageable, we only focus on a symbolic model and 11 cognitive status extraction tasks, agitation, altered mental status (AMS), confusion, delirium, disconnected thinking, disorganized thinking, disorientation, encephalopathy, fluctuation, hallucination, and inattention. For the symbolic framework, we adopted a rule-based framework, MedTagger (v1.3.16)^25,26, an open-source information extraction pipeline based on the UIMA (Unstructured Information Management Architecture) framework²⁷, to facilitate clinical text processing and evaluation of LLM generated lexicons against a gold standard. The study involved three experiments using OpenAI GPT 5.2:

• Baseline: model out of box performance.

• Experiment 0 (E0): provide only error type (FP and FN) information to the prompt

• Experiment 1 (E1): extended the prompt by including the LLM-predicted error class.

• Experiment 2 (E2): included the LLM-generated reasoning but excluded the error class.

• Experiment 3 (E3): included both the predicted error class and the generated reasoning.

The results from each experiment were then compared to the gold standard to assess MedTagger’s performance across different regex-generation methods.

External evaluation of MedError framework

To assess the generalizability and practical utility of MedError in independent, real-world settings, we conducted an external validation study across two institutions through the partnership within the national ENACT (Evolve to Next-Gen Accrual to Clinical Trials network) NLP Working Group. Two institutions participated the evaluation including Mayo Clinic and the University of Pittsburgh. The error cases used in this validation were drawn directly from the ENACT Delirium NLP-CAM multisite evaluation study²⁸. Specifically, each site reviewed real-world false positive (FP) and false negative (FN) errors produced by NLP-CAM^11,29, a rule-based NLP model applied to local EHR clinical notes as part of the ENACT delirium concept extraction task. Cases were retrospectively identified from hospitalized patients aged ≥65 years with delirium-related ICD-10 codes, consistent with the ENACT evaluation protocol. The errors span delirium-related clinical concepts including altered mental status, encephalopathy, disorientation, agitation, and inattention, the same concepts reported as the primary sources of NLP error in the ENACT delirium analysis.

We employed a within-subject, two-condition study design. Each evaluator reviewed the randomly selected two set of 30 error cases under: (1) a Baseline condition, in which errors were analyzed and labeled using free-form, unstructured review with only taxonomy guidance; and (2) a MedError-guided condition, in which evaluators followed the structured MedError workflow. Evaluators completed both conditions and recorded task time, enabling per-case efficiency comparison. All analysis was performed locally; no raw clinical text was shared.

The primary validity outcome was agreement with final adjudication: the proportion of cases in which the annotator’s assigned error class label matched the expert-adjudicated final annotation. Final adjudication was co-performed by two different senior NLP researchers. Agreement was computed separately for the Baseline and MedError conditions, yielding a directly comparable measure of classification quality across conditions. Wilson score 95% confidence intervals were calculated for each agreement proportion. To assess whether MedError produced a statistically significant improvement over Baseline within each site, a paired McNemar exact test was applied, using a binomial distribution under the null hypothesis that discordant pairs are equally likely to favour either condition. The secondary outcome was annotation efficiency, measured as total task time (minutes) per 30-case set and expressed as per-case annotation time. Percent reduction in per-case time from Baseline to MedError was computed for both sites.

Error de-identification

De-identification was conducted using a machine-assisted two-stage process integrated within the Open Health Natural Language Processing Toolkit (OHNLPTK) [https://www.github.com/OHNLP/PresidioDeidentificationforOHNLPTK], which incorporates the Microsoft Presidio de-identification framework³⁰. In the first stage, automated de-identification was performed using a RoBERTa model fine-tuned on data from the i2b2 2014 Track 1 de-identification challenge to identify and mask protected health information (PHI). In the second stage, all machine-processed notes underwent manual human review to ensure complete removal of residual identifiers and to verify de-identification quality. The entities targeted for de-identification followed the HIPAA Safe Harbor standard, which requires removal of 18 categories of identifiers, including names, geographic details smaller than a state, dates related to individuals, contact information (e.g., phone numbers and emails), identification numbers (e.g., SSNs, medical record numbers), device and vehicle identifiers, web and IP addresses, biometric identifiers, full-face images, and other unique identifying characteristics.

Statistics & reproducibility

No statistical method was used to predetermine sample size; the corpus of 1187 errors represents all unique errors identified after de-duplication across 4227 clinical notes from three institutions. No data were excluded from the analyses. Randomization was not applicable given the deterministic nature of NLP model deployment and error annotation. Investigator blinding was not employed, as error categorization required direct domain expertise and engagement with model outputs. LLM classification performance was evaluated using precision, recall, and macro-averaged F1-score computed against human-adjudicated gold standard labels. For the internal model comparison, overall differences across models were assessed using the Friedman test with post-hoc pairwise Wilcoxon signed-rank tests and Bonferroni correction. Rank-biserial r was reported as the effect size for all Wilcoxon tests. Point-biserial correlations between LIWC linguistic features and model performance were tested using a t-distribution (df = N − 2) and corrected for multiple comparisons using the Benjamini-Hochberg false discovery rate procedure (q < 0.05). For the external validation, McNemar exact tests were applied within each site to assess improvement over baseline, with Wilson score 95% confidence intervals calculated for each agreement proportion. All statistical analyses were conducted in Python using scipy.stats (v1.10.0) and statsmodels (v0.14.6).

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.

Author contributions

S.F. and H.L. conceptualized, designed, and wrote the study. S.F., Q.L., J.A., Z.Y., H.Y., and T.H. performed data analysis, model implementation, and framework development. F.C. and J.J. developed the visual analytics web application. J.W. conducted the error interpretation analysis. S.F., L.W., and A.W. contributed to taxonomy consensus development and revision. X.R. and M.H. facilitated data collection and use case design. M.K. and N.R. performed clinical gold standard development and validation. R.H. and Y.W. provided technical guidance on LLM implementation and design. T.H. and Y.W. conducted framework validation. All authors critically revised the manuscript and provided important intellectual content. All authors approved the final version of the study.

Peer review

Peer review information

Nature Communications thanks Thomas Ganslandt, who co-reviewed with Jakob Faller, Frank Kramer and the other, anonymous, reviewers for their contribution to the peer review of this work. A peer review file is available.

Funding

H.L. discloses support for the research of this work from the National Library of Medicine [grant number R01LM011934], the National Human Genome Research Institute [grant number R01HG012748], the National Institute on Aging [grant numbers R01AG072799 and U24AG098157], the National Institute of Biomedical Imaging and Bioengineering [grant number OT2OD038051], the National Heart, Lung, and Blood Institute [grant number R24HL180372], the Food and Drug Administration [grant number U01FD008720], and the Cancer Prevention Research Institute of Texas (CPRIT) [grant number RR230020]. S.F. discloses support for the research of this work from the National Institute on Aging [grant numbers K01AG088452, R33AG071744, P30AG073105, and U24AG098157]. Y.W. discloses support for the research of this work from the National Institutes of Health [grant numbers UL1TR001857, U24TR004111, R01LM014306, and R01LM014588]. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. All other authors declare no relevant funding.

Data availability

The de-identified clinical NLP error annotations generated in this study are available under restricted access due to patient privacy restrictions and institutional data governance requirements. Access can be obtained by contacting the Center for TEAM-AI at UTHealth Houston (https://sbmi.uth.edu/center-translational-ai/) or The University of Texas Health Intelligence Platform (huipeng.liu@austin.utexas.edu) and executing a Data Use Agreement; requests will be reviewed and are available to academic researchers for non-commercial purposes only. The raw clinical note data are protected and are not available due to patient privacy HIPAA laws and IRB restrictions. The processed error taxonomy definitions and annotation schemas generated in this study are available at https://github.com/OHNLP/MedError/tree/main/Taxonomy. Source data are provided with this paper.

Code availability

All code used in this study is openly available in the GitHub repository at https://github.com/OHNLP/Mederror. The finalized taxonomy definitions are publicly available on GitHub: https://github.com/OHNLP/MedError/tree/main/Taxonomy.

Competing interests

H.L. serves as an Editorial Board Member of NPJ Digital Medicine, which is unrelated to the subject matter of this manuscript. M.J.K. received consulting fees from Novo Nordisk and Endocrine and Diabetes Plus Clinic of Houston, and receives research funding from Medical AI, NIA, and UTHealth, none of which funded or influenced this study. All other authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at https://doi.org/10.1038/s41467-026-77067-1.