Prognostic data extraction harnessing a privacy-preserving large language model: a clinician-AI collaborative retrospective evaluation in head and neck oncology

Abstract

Privacy regulations and limited expert-validation constrain the deployment of large language models (LLMs) for electronic health record structuring. We evaluated locally deployed LLMs to extract 30 prognostic variables from 1,360 head and neck cancer reports (882 patients) using zero-shot prompting. A stratified 50-case subset was reviewed by three radiation oncologists (50 cases, 30 fields, 3 reviewers; 4,500 decisions) to form a majority-vote reference for Llama3.3-70B, which achieved 98.6% F1 with high clinician agreement and processed reports in 53 s/report. Among seven additional models (2.6B-70B) benchmarked against this reference, GPT-OSS-20.9B (F1 89.4%) and MedGemma-27B (F1 88.5%) performed best. Integrating LLM-extracted HPV status, smoking history, and Charlson Comorbidity Score into a multivariate Cox Proportional Hazards model (age, sex, T/N stage) improved disease-free survival (likelihood ratio test p = 0.014; ΔC-index + 0.071) and locoregional failure-free survival (p = 0.026; ΔC-index + 0.108) with 1,000-bootstrap internal validation. This clinician-AI collaborative evaluation shows that on-premises LLMs enable privacy-preserving and efficient tumour board support, longitudinal data curation, and outcome prediction.

Introduction

In radiation oncology, Electronic Health Records (EHR) contain rich but unstructured clinical documentation, particularly for head and neck (H&N) cancers, where diverse anatomical sites, risk factors, and staging criteria complicate data abstraction. Key prognostic features such as tumour-nodal-metastasis (TNM) staging, p16/human papillomavirus (HPV) status, and comorbidities are often buried in narrative notes, making manual extraction labour-intensive and time-consuming¹. Overcoming this manual burden bottleneck promises broad benefits in oncology, such as expediting tumour board preparations², enabling faster identification of trial-eligible patient screening^3,4, reinforcing oncology research database cancer registry curation and outcomes surveillance, and accelerating personalized clinical decision support⁵.

Traditional rule-based clinical natural language processing (NLP)⁶ methods relied on rule-based approaches such as regular expressions, dictionary matching, and lexicon lookups^6,7, have limited flexibility due to the variability of clinical language. These are interpretable and can achieve high precision on well-defined tasks⁸. However, these systems require extensive manual rule-crafting by domain experts, are brittle when faced with the heterogeneous and often ambiguous language of clinical narratives, and are poorly transferable across institutions or cancer types⁹. Supervised deep learning models, including token classification approaches using ClinicalBERT¹⁰, BioBERT¹¹, and Clinical-Longformer¹², have demonstrated strong performance on named entity recognition (NER) benchmarks¹³. Yet, these methods require large, manually annotated training corpora, typically hundreds to thousands of labelled examples per entity type, which are prohibitively expensive to produce in clinical settings where expert annotators have limited availability⁹. Moreover, fine-tuned encoder models are task-specific. For example, a model trained for TNM staging extraction cannot be reused for comorbidity or HPV status extraction without re-annotation and retraining.

Recently, large language models (LLMs) have demonstrated state-of-the-art performance in extracting clinical insights, including structured data and medical reasoning, from unstructured text input^14–16. Nevertheless, a primary concern for successful clinical workflow integration is protecting patient privacy. Several of the most powerful models, such as OpenAI’s GPT-4¹⁷ and Google’s Med-PaLM2¹⁸, are cloud-based and proprietary, meaning that routine use would necessitate transmitting patient data beyond secure hospital firewalls. More recently, open-weight models including Meta’s Llama family¹⁹, Google’s Gemma and MedGemma²⁰, and OpenAI’s GPT-OSS (released in August 2025)²¹ emerged, narrowing the performance gap while enabling fully local deployment. Nevertheless, transmitting protected health information to external servers remains subject to strict regulations such as the Personal Health Information Protection Act in Canada, and the majority of published clinical LLM studies still rely on proprietary API access^22–24.

Due to this reason, most prior studies have focused on simplified or simulated tasks, including question-answering or evaluation on public datasets such as MIMIC-IV^23,25, the United States Medical Licensing Examination (USMLE)²⁶, or fictional molecular tumour board cases²⁷, rather than on granular prognostic data extraction from real-world clinical cases. Henceforth, the curation of de-identified real-world clinical notes in oncology remains crucial. Recent efforts, exemplified by LLM-Anonymizer²⁸, demonstrated that a locally deployed, open-source pipeline could redact nearly 98–99% of patient identifiers in real clinical letters, highlighting the feasibility of privacy-preserving medical document processing. Despite its success, extracting prognostic data from real-world oncology notes remains largely underexplored, owing to the high cost of expert annotation and the limited scale of prior attempts¹⁵. Moreover, few studies have subjected LLM-extracted clinical variables to rigorous multi-clinician validation and their downstream prognostic utilities. A robust, privacy-preserving pipeline with thorough clinician evaluation is thus essential for integrating these methods into tumour board workflows, building research databases, and advancing outcome monitoring.

To address these gaps, we implemented and evaluated a locally deployed, privacy-preserving LLM pipeline to extract structured prognostic variables from a large corpus of H&N cancer reports, in collaboration with radiation oncologists who clinically validated the output to ensure real-world accuracy and hospital scalability. Beyond extraction accuracy, we demonstrate downstream clinical utility by showing that LLM-extracted features such as HPV status, smoking history, and Charlson comorbidity scores significantly improve prognostic survival predictions over standard clinical covariates alone, providing evidence that instruction-based LLM extraction can directly support outcome prediction in radiation oncology. To contextualize performance, we benchmark seven additional open-weight LLMs of varying size (2B-70B parameters), including medical domain-specific models, against the clinically validated ground truth.

Results

Overview

We curated a novel dataset of 1360 de-identified head and neck cancer clinical notes, including pathology and consultation reports, for structured extraction of key prognostic and treatment-related variables such as TNM staging, HPV/p16 status, comorbidities, and smoking history. To our knowledge, this represents the largest expert-validated corpus of unstructured clinical text in head and neck oncology. Using the locally hosted, open-source Llama 3.3 70B model as our primary model (Fig. 1), we extracted 30 fields per report using two prompting strategies: direct extraction (P1) and inference-based reasoning (P2), while ensuring that all processing remained within secure hospital infrastructure.

Fig. 1. Automatic extraction of prognostic information from H&N cancer clinical notes using an open-source large language model.

a From pathology reports and consultation notes; b two-stage prompting strategies where separate prompts were engineered to perform direct knowledge extraction, and inference-driven zero-shot prompting with chain-of-thought where medical reasoning was required. c Meta’s Llama3.3-70B LLM was locally deployed via the LangChain and Ollama Application Programming Interface to extract thirty fields. d Prognostic structured data extracted, enabling efficient data mining for tumour board meetings and large-scale research data curation. e Distribution of pathology reports (n = 574) and consultation notes (n = 786) available to 882 unique patients, of whom 478 had both report types; f Distribution of total tokens needed for each prompting step (either pure extraction or chain-of-thought inference). The vertical dashed line of 2048 tokens is the default context window of the OllamaChat API. g Clinician evaluation confusion matrix definition. h Clinician agreement workflow via majority-voting, versus individual response as ground truth.

Although extraction was performed on all 1360 reports, a stratified sample of 50 cases was selected for multi-rater clinician validation based on report availability and information richness. Three board-certified radiation oncologists independently reviewed each model-generated field (30 fields per case), yielding 1500 unique field-level outputs and 4500 total review decisions. Each field was marked as “Agree” or “Disagree.” We report the extraction performance of Llama3.3-70B as the primary reference model and inter-rater agreement analyses, and we use majority voting to derive a clinician-adjudicated reference that serves as the ground truth for subsequent benchmarking. Specifically, seven additional locally deployable LLMs (general-purpose and medical-domain, smaller-scale) were evaluated against this majority-vote reference using identical inputs and prompts. We then perform targeted error analyses of lower-agreement fields and apply post-hoc rule-based corrections to address systematic definition/criteria mismatches, including discrepancies attributable to prompt specification. These corrections are applied to the full cohort, enabling downstream characterization of prognostic variable distributions and multivariate outcome prediction modelling across all patients. Representative examples of clinical note inputs alongside their extracted 30 prognostic fields are shown in Section S1.

LLM extraction performance evaluated by three clinicians independently

Using the primary Llama3.3-70B model run on a 2× NVIDIA RTX 5000 32 GB GPUs workstation, each report was processed in 53.14 seconds on average for all 30 fields, averaging 1.77 seconds per field (Supplemental Fig. S4). In contrast, manual review by clinicians took at least 10–35 min per case, resulting in an efficiency gain of up to ten to twenty times. Across 4500 evaluations (30 fields × 50 cases × 3 evaluators) with majority voting consensus GTs, the model achieved 97.9% accuracy, 97.3% precision, 99.9% recall, 98.6% F1 score, and a 0.1% false negative rate (FNR), as shown in Table 1a. Per-field performance is detailed in Table 1b. FNR was 0% for all fields except Pathological TNM (0.45%).

Table 1.

Local large language model (Llama3.3-70B) performance to structured prognostic information evaluated by three radiation oncologist clinician evaluators, results ground truths were given by majority voting

(a) Overall performance metrics across all 1500 responses per evaluator for the 30 extracted fields (50 stratified cases, 3 evaluators).
Accuracy	Precision	Recall	F1	FNR
0.979	0.973	0.999	0.986	0.001

(b) Field-specific performance after majority voting from the three clinician evaluators, grouped by prompt type.
Structured Field	Accuracy	Precision	Recall	F1	FNR
Extraction-prompted fields
Sex	0.959	0.956	1.000	0.977	0.000
Anatomic Site of Lesion	1.000	1.000	1.000	1.000	0.000
Pathological TNM	0.980	1.000	0.955	0.977	0.045
Clinical TNM	0.960	0.947	1.000	0.973	0.000
Primary Tumor Size	0.980	0.977	1.000	0.989	0.000
Lymph Node Presence	1.000	1.000	1.000	1.000	0.000
Number of Positive Lymph Nodes	1.000	1.000	1.000	1.000	0.000
Extranodal Extension Presence	0.980	0.970	1.000	0.985	0.000
Pathology Details	0.980	0.980	1.000	0.990	0.000
Resection Margins	1.000	1.000	1.000	1.000	0.000
Immunohistochemical Profile	1.000	1.000	1.000	1.000	0.000
EBER Status	1.000	1.000	1.000	1.000	0.000
Lymphovascular Invasion Status	0.980	0.964	1.000	0.982	0.000
Perineural Invasion Status	0.960	0.920	1.000	0.958	0.000
Clinical Assessments Radiological Lesions	1.000	1.000	1.000	1.000	0.000
Clinical Assessments SUV from PET Scans	1.000	1.000	1.000	1.000	0.000
Inference-prompted fields
Tumor Type Differentiation	0.940	0.932	1.000	0.965	0.000
p16 Status	1.000	1.000	1.000	1.000	0.000
Smoking History	1.000	1.000	1.000	1.000	0.000
Alcohol Consumption	0.980	0.975	1.000	0.987	0.000
Pack Years	0.980	0.975	1.000	0.987	0.000
Patient Symptoms at Presentation	1.000	1.000	1.000	1.000	0.000
Treatment Recommendations	1.000	1.000	1.000	1.000	0.000
Follow-Up Plans	1.000	1.000	1.000	1.000	0.000
HPV Status	1.000	1.000	1.000	1.000	0.000
Patient History Status Prior Conditions	0.980	0.977	1.000	0.988	0.000
Patient History Status Previous Treatments	1.000	1.000	1.000	1.000	0.000
Karnofsky Performance Status	1.000	1.000	1.000	1.000	0.000
ECOG Performance Status	1.000	1.000	1.000	1.000	0.000
Charlson Comorbidity Score	0.694	0.694	1.000	0.819	0.000

(a) Overall performance across 4500 evaluator-field observations; (b) Field-specific performance, stratified by prompt type ("Extraction-prompted” vs. “Inference-prompted”).

Seventeen fields achieved perfect scores. Among the P1 (direct extraction) fields, these included lesion site, lymph node presence and count, resection margins, immunohistochemistry, EBER status, radiological lesion evaluations, and PET SUV values. Among P2 (inference-based) fields, perfect performance was reached for p16 and HPV status, smoking history, presenting symptoms, treatment recommendations, follow-up plans, prior treatments, and performance status (KPS and ECOG). An additional 12 fields exceeded 94% F1, including sex, TNM staging (clinical and pathological), tumour size, extranodal extension, pathology, lymphovascular invasion, and perineural invasion. Fields such as tumour differentiation, alcohol use, pack-years, and prior condition history also exceeded 94–98% accuracy with perfect recall.

The Charlson Comorbidity Score (CCS) remained an outlier with 69.4% accuracy, 69.4% precision, 100% recall, and an F1 score of 81.9%, as shown in Table 1. Examples of extractions can be seen in Supplemental Section S1.

Inter-rater reliability and variability analysis

Table 2 presents the inter-rater reliability test of Gwet’s AC1 statistics for each of the 30 structured fields, with a corresponding forest plot in Fig. S5. Majority-vote consensus confirmed 1468 of 1500 field-level extractions (98%) as correct (Table 1a), indicating strong agreement among the three evaluators. The Gwet’s AC1 for all fields combiend was 0.914 (95% CI: 0.901-0.927, p < 0.001), with a raw agreement rate of 92.2% (Table 2a). While raw agreement reflects direct concordance, Gwet’s AC1 adjusts for chance-level agreement, providing a more robust reliability measure.

Table 2.

Inter-rater reliability among three board-certified radiation oncologists quantified by Gwet’s AC1 for LLM-extracted prognostically relevant variables in head and neck cancer

(a) Overall evaluator agreement for 30 extracted fields (50 stratified cases, 3 evaluators).
Gwet AC1	95% CI	p value	Percent Agreement
0.914	(0.901, 0.927)	< 0.001	0.922

(b) Field-specific Gwet’s AC1 coefficients with 95% confidence intervals.
Structured Field	Gwet AC1	95% CI	p value	Percent Agreement
Extraction-prompted fields
Sex	0.955	(0.901, 1.000)	< 0.001	0.959
Anatomic Site of Lesion	0.973	(0.933, 1.000)	< 0.001	0.973
Pathological TNM	0.912	(0.838, 0.987)	< 0.001	0.920
Clinical TNM	0.743	(0.608, 0.879)	< 0.001	0.800
Primary Tumor Size	0.926	(0.858, 0.994)	< 0.001	0.933
Lymph Node Presence	0.898	(0.818, 0.977)	< 0.001	0.907
Number of Positive Lymph Nodes	0.775	(0.653, 0.898)	< 0.001	0.813
Extranodal Extension Presence	0.880	(0.791, 0.968)	< 0.001	0.893
Pathology Details	0.958	(0.907, 1.000)	< 0.001	0.960
Resection Margins	0.973	(0.933, 1.000)	< 0.001	0.973
Immunohistochemical Profile	0.973	(0.933, 1.000)	< 0.001	0.973
EBER Status	0.986	(0.959, 1.000)	< 0.001	0.987
Lymphovascular Invasion Status	0.972	(0.932, 1.000)	< 0.001	0.973
Perineural Invasion Status	0.956	(0.904, 1.000)	< 0.001	0.960
Clinical Assessments Radiological Lesions	0.986	(0.958, 1.000)	< 0.001	0.986
Clinical Assessments SUV from PET Scans	0.986	(0.958, 1.000)	< 0.001	0.986
Inference-prompted fields
Tumor Type Differentiation	0.877	(0.783, 0.970)	< 0.001	0.893
p16 Status	0.986	(0.959, 1.000)	< 0.001	0.987
Smoking History	0.986	(0.959, 1.000)	< 0.001	0.987
Alcohol Consumption	0.958	(0.907, 1.000)	< 0.001	0.960
Pack Years	0.928	(0.861, 0.995)	< 0.001	0.933
Patient Symptoms at Presentation	0.881	(0.795, 0.968)	< 0.001	0.893
Treatment Recommendations	0.958	(0.909, 1.000)	< 0.001	0.960
Follow-Up Plans	0.986	(0.959, 1.000)	< 0.001	0.987
HPV Status	0.865	(0.772, 0.957)	< 0.001	0.880
Patient History Status Prior Conditions	0.846	(0.744, 0.947)	< 0.001	0.867
Patient History Status Previous Treatments	0.879	(0.790, 0.967)	< 0.001	0.891
Charlson Comorbidity Score	0.167	(-0.061, 0.396)	0.147	0.537
Karnofsky Performance Status	0.943	(0.884, 1.000)	< 0.001	0.946
ECOG Performance Status	0.927	(0.860, 0.994)	< 0.001	0.932

(a) Overall agreement across all 4500 evaluator-field observations; (b) Field-specific agreements, grouped into “Extraction-prompted” and “Inference-prompted” categories.

Per-evaluator performance is illustrated in Fig. 2, showing F1 (Fig. 2a) and FNR (Fig. 2b) when each evaluator served as ground truth. These plots reflect inter-rater variability patterns consistent with AC1 and majority vote metrics. Additional details are provided in Section S3.

Fig. 2. Radar plots of individual clinician evaluators’ F1 and false negative rates.

a F1 score radar plot for all 3 evaluators, each treated as individual ground truths. F1 is the harmonic mean of precision and recall. b False negative rate radar plot for all 3 evaluators: each treated as an individual ground truth. A false negative is defined when the LLM outputted “Not inferred'', and the evaluator disagreed with it.

Field-level AC1 values (Table 2b) showed high reliability (AC1 ≥ 0.8) for most fields, with over half exceeding 0.95. Near-perfect consensus (p < 0.001) was seen for lesion site, pathology details, resection margins, immunohistochemistry, EBER status, lymphovascular invasion, radiology assessments, PET SUV, p16 status, smoking history, treatment plans, and follow-up. Performance scores for KPS (AC1 = 0.943) and ECOG (AC1 = 0.927) were similarly high.

Moderate-to-strong agreement (AC1 = 0.8-0.9) was observed for tumour type differentiation (AC1 = 0.877) and presenting symptoms (AC1 = 0.881). Clinical TNM (AC1 = 0.743) and number of positive lymph nodes (AC1 = 0.775) showed more rater variability. The CCS had the lowest reliability (AC1 = 0.167, 95% CI: -0.061-0.396, p = 0.147), consistent with its performance drop (Table 1a).

These reliability patterns are echoed in Fig. 2a. A clear drop appears for Charlson F1, particularly for Evaluators 2 and 3, while Evaluators 1 and 2 showed higher consistency for clinical TNM. Despite a slightly elevated FNR (>10%) for three fields flagged by Evaluator 3 (Fig. 2b), these were not corroborated by the others, preserving a low overall FNR of 0.1% (Table 1a).

Evaluator disagreement error analysis

Evaluators unanimously agreed in 4321 of 4500 judgments (96.0%). Of the 179 disagreements (4.0%), most (147) involved a single dissenter; 28 involved two dissenters, and only 4 had complete disagreement, representing a total absolute error rate of 0.089%. A breakdown of disagreement patterns appears in Fig. 3.

Fig. 3. Reasons Breakdown of clinician disagreement in field-level extraction from head and neck cancer reports.

Sankey diagram showing the map of responses without unanimous agreements with the LLM output (N=179/4500) to rater disagreement types, root causes of disagreement, and the specific extracted fields involved. Node size corresponds to the number of instances per category. Disagreements often stemmed from divergent interpretations of extraction guidelines or ambiguous source text, with fields such as the Charlson Comorbidity Score and pathological/clinical TNM staging appearing frequently across multiple root causes.

Among the disagreement cases, 83 stemmed from guideline interpretation differences, 50 from clinical reasoning variability, and 31 from source text ambiguity. Inconsistent application of staging or Charlson scoring criteria explained the rest. Notably, the Charlson Score accounted for 34 disagreements, the highest of any field, followed by clinical TNM (16), and lymph node fields. Patient history fields also showed multiple disagreements. Evaluator 1 agreed most frequently, while Evaluator 3 flagged the most contested outputs, suggesting greater strictness.

Field-specific error patterns in low-agreement variables

Disagreement patterns arose from clinician-evaluated Llama3.3-70B extracted prognostic variables were examined to identify systematic sources of error. This encompasses ambiguity in clinical documentation, implicit reasoning requirements, and inter-rater variability among clinicians. Here, we highlight three extracted fields with the most notable lower inter-rater reliability: clinical TNM staging, number of lymph nodes involved, and CCS as shown in Table 2.

Clinical TNM had relatively low reviewer agreement, with a Gwet’s AC1 of 0.743. We found 18 disagreement instances across 150 reviews for this field across 15 cases outlined in Table 3. Fourteen of these cases still had majority agreement, which contributed to the high field-specific performance in Table 1b. Most disagreements (7/18) occurred when the model extracted a stated clinical TNM stage while clinicians derived the stage from tumour measurements in the reports using AJCC criteria²⁹. Other disagreements (3/18) occurred when both clinical and pathological staging appeared in the same report, and the model extracted the pathological TNM stage instead of the clinical TNM stage. These findings indicate that clinical TNM should be inference-prompted rather than extraction-prompted, allowing explicit AJCC staging rules to be incorporated into the prompt.

Table 3.

Representative root causes for clinician comments for Clinical TNM disagreement cases

Case	LLM Output	Root Cause	Representative Comments
7	T1N2cM0	Clinical vs pathological confusion	Staging was pathological, not clinical
8	T1 N2a M0	Stage value disagreement	Clinical stage should be TxN2
37	T1 N0 M0	Stage value disagreement	Should be T2 based on 2.5 cm lesion
41	T4 N0 M0	Pathological instead of clinical	This is pathologic T4N0
44	T2, N0	Nodal status disagreement	Lymph nodes not mentioned, cannot be N0
47	Not inferred	Should have inferred	Indicated it’s metastatic, stage 4
50	T4N1M0	Stage value disagreement	Multiple ipsilateral nodes are N2

For Number of Positive Lymph Nodes (Gwet’s AC1 = 0.775), 69.2% (9/13) of disagreements stemmed from the LLM extracting clinically or radiologically identified nodes, while clinicians applied the AJCC/NCCN standard requiring pathological confirmation (Table 4). In these cases, for example, the LLM accurately described the imaging findings (e.g., “1 left level 3 lymph node measuring 1.5 cm"), but clinicians correctly noted this should be “Not Inferred" without surgical pathology confirmation. The LLM’s extraction was accurate based on our defined prompts; the disagreement reflected an ambiguity in our evaluation criteria regarding clinical versus pathological node counts, as it was defined as one of the extraction-driven prompts.

Table 4.

Representative cases with disagreements in number of positive lymph nodes extraction

Case	LLM Output	Root Cause	Explanation
6	0	pNX (Should Be Not Inferred)	Patient staged as pNX; no lymph nodes were surgically evaluated
19	1 (level 3, 1.5 cm)	No Pathological Confirmation	Node identified on CT imaging only; AJCC requires surgical pathology confirmation
30	1 (8 mm supraclavicular)	pNX	Patient staged as pNX; no neck dissection performed
48	2 (level 3 + level 2)	No Pathological Confirmation	Nodes identified radiologically on CT/MRI; no pathological confirmation available
50	1 (level 2, 2.4 cm)	Count Disagreement	LLM reported one node; clinicians identified multiple ipsilateral nodes in imaging

"Level” refers to the cervical lymph node level classification system (Levels I–VII), which describes anatomical regions of the neck used in head and neck oncology staging. Level II (upper jugular) and Level III (middle jugular) are common sites of nodal metastasis in head and neck cancers.

CCS showed the highest inter-rater disagreement, with Gwet’s AC1 of 0.167 (Table 2). As summarized in Table 5, the root cause of 51.5% (17/33) of disagreements was traced to hypertension being included in the inference-driven extraction prompt, but absent from the original 1987 Charlson Comorbidity Index³⁰, which the clinician reviewers followed. An additional 42.4% of disagreements stemmed from other conditions, such as hypothyroidism, epilepsy, and dyslipidemia, that similarly fall outside the Charlson criteria. Importantly, in more than 95% of disagreement cases, the LLM correctly identified the comorbidity conditions present in the clinical reports and applied the scoring rules specified in the prompt. The discrepancy arose not from model hallucination or fabrication, but from a systematic mismatch between the expanded criteria embedded in the prompt and the original Charlson criteria applied by clinician reviewers.

Table 5.

Root causes of Charlson Comorbidity Score disagreements organized by reviewer comment type

Root Cause	Instances	Representative Comments
Overestimated Score (general)	22	LLM-identified conditions not valid per 1987 Charlson Index
Hypertension Not Scored	13	Hypertension is a risk factor, not a Charlson condition
Hyperlipidemia/Hypercholesterolemia Not Scored	12	Not in original Charlson Index
Hypothyroidism Not Scored	12	Not in original Charlson Index
Other Conditions Not in Charlson	8	Conditions correctly identified but incorrectly assigned points: gout, glaucoma, anxiety, benign prostatic hyperplasia, sinusitis, tachycardia, smoking history, obstructive sleep apnea, basal cell carcinoma, thyroidectomy
Gastroesophageal Reflux Not Scored	4	Reflux is not peptic ulcer disease
Epilepsy Not Scored	4	Not in original Charlson Index
Deep Vein Thrombosis Not Scored	1	Venous condition, not arterial (peripheral vascular disease)

Instances represent the number of times each issue was cited in reviewer comments across 150 total reviews (50 cases × 3 reviewers). Categories are not mutually exclusive: a single disagreeing review may cite multiple issues, and multiple reviewers on the same case may cite the same issue. “Overestimated Score” represents cases where the LLM correctly identified conditions mentioned in the clinical report, but clinicians judged these conditions should not contribute points according to the original 1987 Charlson Comorbidity Index. The remaining categories specify which conditions were most frequently flagged as incorrectly scored.

Post-hoc hybrid LLM with rule-based post-processing for CCS

Since CCS had the lowest inter-rater agreement, we performed a post-hoc analysis to determine whether a lightweight rule-based post-processing step could correct the dominant, systematic error pattern identified in Table 5. Our inference prompt specified a structured CCS output from the Chain-of-thought (CoT) prompting technique that included both (i) a numeric score and (ii) bracketed reasoning (e.g., “3 [1 point for hypertension, 1 point for COPD, 1 point for diabetes]”). This design choice not only enabled reasoning interpretability but also produced a consistently parseable format that enabled downstream rule-based correction.

We implemented a regular expression (regex)-based parser to extract the score and the condition-point assignments from the bracketed explanation, then compared extracted conditions against a curated dictionary of the 1987 Charlson conditions³⁰ and recomputed CCS after removing non-Charlson conditions shown in Table 5 (e.g., hypertension, dyslipidemia, hypothyroidism, gastroesophageal reflux disease, deep vein thrombosis, epilepsy, anxiety/depression, and non-melanoma skin cancer).

This post-hoc correction substantially improved CCS agreement with the clinician-reviewed standard. Before post-processing, CCS achieved 69.4% accuracy (34/50 cases), with perfect recall but frequent score overestimation. After applying the correction rules, accuracy increased to 100% while maintaining perfect recall, correcting all 15 previously incorrect cases attributable to non-Charlson comorbidity conditions. These results suggest that a hybrid LLM+rules approach can mitigate criteria-mismatch errors for CCS, while preserving the model’s strength in comorbidity identification from unstructured text. This correction was then applied to the full dataset.

Comparative benchmarking of general-purpose and medical domain language models

For benchmarking purposes, the clinician-evaluated Llama3.3-70B model is hereafter referred to as the reference model. It was compared against seven open-source, instruction-tuned LLMs spanning both general-purpose and medical domain-adapted architectures using the same cohort of 50 cases with clinician-validated ground truth. Because model outputs may differ in wording from the ground truth while preserving the same clinical meaning, benchmarking was performed using a semantic similarity-based evaluation rather than exact string matching (see “Clinician Evaluation Case Sampling Strategy”). A summary of model characteristics, including parameter size, context window, and pretraining strategy, is provided in Table 9.

Table 9.

Technical specifications of benchmarked language models used for prognostic data extraction

Model	Provider	Architecture	Parameters	Type	Context Window	Training Orientation	Deployment Details
Llama 3.3 (Reference)	Meta	Llama-family	70.6B	General	131K	Instruction-tuned, general-purpose	Ollama (quantized), ~ 42 GB on-disk
GPT-OSS	OpenAI	GPT-OSS (MoE)	~ 20.9B	General	131K	Instruction-tuned, general-purpose	Ollama, ~ 13 GB on-disk
MedGemma 27B	Google	Gemma-family	27B	Medical	128K	Medical-domain fine-tuned	Ollama (quantized)
MedGemma 3.9B	Google	Gemma-family	3.9B	Medical	128K	Medical-domain fine-tuned	Ollama, ~ 2.5 GB on-disk
Mistral 7B	Mistral AI	Mistral family	7.2B	General	32K	Instruction-tuned, general-purpose	Ollama, ~ 4.4 GB on-disk
Gemma 2B	Google	Gemma-family	2.6B	General	8K	Instruction-tuned, general-purpose	Ollama, ~ 1.6 GB on-disk
Phi-3 Mini	Microsoft	Phi-family	3.8B	General	128K	Instruction-tuned, general-purpose	Ollama, ~ 2.2 GB on-disk
Meditron 70B	EPFL	Llama2-derived	70B	Medical	4K	Medical-domain fine-tuned (QA-focused)	Ollama (quantized)
Llama 2 7B	Meta	Llama-family	7B	General	4K	Instruction-tuned, general-purpose	Ollama, ~ 3.8 GB on-disk

All models were instruction-tuned and evaluated using identical prompting strategies. Model properties include architecture family, parameter count, context window, and deployment details. Context window refers to the maximum input token length supported by the model. Deployment configuration reflects the specific runtime environment used in this study (Ollama framework).

Benchmarking results are summarized in Table 6. The clinician-evaluated reference model (Llama3.3-70B) achieved an accuracy of 0.979 and an F1 score of 0.986 based on three-clinician majority voting. Across models, performance increased with model size and context window capacity. GPT-OSS-20.9B achieved the second-best performance with an overall F1 score of 0.895, demonstrating that competitive extraction ability is achievable with models substantially smaller than 70B parameters. In contrast, models with limited context windows (Meditron-70B: 4K, Llama2-2.6B: 4K) performed poorly (F1<0.30), as clinical reports frequently exceeded their input capacity.

Table 6.

Benchmarking performance of general-purpose and medical language models for prognostic data extraction

Model	Parameters	Type	Context	Accuracy	Precision	Recall	F1	FNR
Llama 3.3 (Clin. Eval.)	70.6B	Reference	131K	0.979	0.973	0.999	0.986	0.001
GPT-OSS	20.9B	General	131K	0.851	0.847	0.949	0.895	0.051
MedGemma 27B	27B	Medical	128K	0.835	0.833	0.943	0.885	0.057
MedGemma 4B	3.9B	Medical	128K	0.664	0.656	0.891	0.756	0.109
Mistral	7.2B	General	32.8K	0.632	0.652	0.842	0.735	0.158
Gemma 2	2.6B	General	8K	0.464	0.613	0.476	0.536	0.524
Phi-3	3.8B	General	128K	0.349	0.395	0.406	0.400	0.594
Meditron	70B	Medical	4K	0.305	0.373	0.157	0.221	0.843
Llama 2	7B	General	4K	0.279	0.436	0.092	0.151	0.908

Models compared include Llama3.3-70B (Reference), GPT-OSS-20.9B, MedGemma-27B and 4B, Mistral-7.2B, Gemma2-2.6B, Phi3-3.8B, Meditron-70B, Llama2-7B, number of model parameters, model type: general-purpose or medical-domain, context window length, accuracy, precision, recall, F1, and False Negative Rates.

Medical-domain fine-tuning showed scale-dependent effects (Fig. 4). At 4B parameters, MedGemma-3.9B outperformed general-purpose Phi3-3.8B (F1 = 0.756 vs 0.400), suggesting domain adaptation benefits smaller models (Fig. 4 <10B parameters). At larger scales, general-purpose models with sufficient context windows performed best, with Llama 3.3 70B achieving the highest overall performance, followed by GPT-OSS-20.9B and MedGemma-27B (medical-domain fine-tuned). This performance order is consistent with model size, and all three can be deployed in a privacy-preserving, open-source, on-premise manner. Meditron-70B exhibited substantially lower performance despite its size, coinciding with its limited context window relative to input report lengths, and it was trained focusing on question-answering rather than extraction tasks, resulting in high FNR, as “Not Inferred” field extractions when they did exist were high. Other performance metrics, including accuracy, precision, recall, and FNR across models, are visualized in Figure S8 Field-specific extraction performance varied according to the level of standardization and clinical interpretation required, shown in Fig. 5 for F1 scores, and Fig. S10 shows that of accuracy, precision, recall, and FNR. Highly standardized laboratory and pathology fields achieved the strongest performance across models. Variables such as p16 status, HPV status, and EBER status achieved perfect F1 scores (1.000) across all top-performing models, reflecting their explicit and consistent reporting in pathology reports. Similarly, lymph node status presence/absence and resection margins achieved F1 scores of 1.000 for the reference model.

Fig. 4. Benchmarking of clinician-evaluated Llama3.3-70b model (purple) compared with smaller-sized instruction-tuned general-purpose (green) and medical domain (yellow) language models.

The top panel include LLMs with less than 10 billion parameters, including Llama2-7B, Phi3-3.8B, Gemma2-2.6B, Mistral-7.2B, and MedGemma-3.9 B. The lower panel shows Meditron-70B, MedGemma-27B, GPT-OSS-20.9B, and Llama3.3-70B.

Fig. 5. Benchmarking of clinician-evaluated Llama3.3-70B model compared with smaller-sized instruction-tuned general-purpose and medical domain language models for each extracted field.

The top half (black) was extraction-prompting-based fields, and the lower (red) was the inference-prompting-based fields. Models compared include Llama3.3-70B, GPT-OSS-20.9B, MedGemma-27B, MedGemma-3.9B, Mistral-7.2B, Gemma2-2.6B, Phi3-3.8B, Meditron-70B, Llama2-7B.

Commonly documented clinical history fields also demonstrated high performance. Smoking history achieved an F1 score of 1.000 for the reference model and MedGemma 27B. Anatomic site of lesion and pathology details showed near-perfect extraction performance, with F1 scores of 0.990 and 0.980, respectively. Fields requiring greater clinical interpretation exhibited reduced performance. Pathological TNM staging showed lower F1 scores across models, with values of 0.977 for the reference model, 0.760 for GPT-OSS, and 0.588 for MedGemma 27B. Clinical assessment variables like SUV derived from PET imaging were also more challenging, with the reference model achieving an F1 score of 0.986, 0.67 for GPT-OSS-20.9B, and 0.75 for MedGemma-27B. The CCS was consistently difficult to extract across all evaluated models. The reference model achieved an F1 score of 0.819, while GPT-OSS and MedGemma 27B achieved F1 scores of 0.676 and 0.711, respectively. Smaller models showed substantially reduced performance in this field. Performance status variables demonstrated differential extraction difficulty. Karnofsky Performance Status achieved higher F1 scores than ECOG Performance Status across models. This disparity was more pronounced in smaller models.

Distributions of LLM-extracted prognostic variables

The distributions of each categorical LLM-extracted variable in the full patient cohort (n = 882) are shown in Fig. 6. This includes sex, anatomic site, clinical T stage (i.e., T0-1, T2, T3, T4), clinical N stage (i.e., N0, N2-3, N1), ECOG status (i.e., 0,1,2,3,4), lymph node involvement status, number of positive lymph nodes, extranodal extension, primary tumour size, resection margins, EBER status, lymphovascular invasion status, perineural invasion status, tumor differentiation (i.e, moderate, poor, well), HPV status, smoking history (i.e., former, never, and ever), alcohol consumption (i.e., yes, none), pack-years of smoking (i.e., never: 0, moderate: 21–40, heavy: >40), treatment type, and Charlson Comorbidity Score (i.e., 0, 1, 2, 3, 4, 5+). These structured prognostic variables could enable rapid cohort identification for focused retrospective studies, for example selecting HPV-negative patients¹, patients with moderate or heavy smoking exposure (i.e., pack-years)³¹, or patients with poor tumour differentiation at diagnosis³².

Fig. 6. Distributions of LLM-Extracted Categorical Prognostic Variables in Full Study Cohort.

a Sex; b Anatomic site; c T stage; d N stage; e ECOG; f Lymph node involvement status; g Number of positive lymph nodes; h Extranodal extension; i Primary tumour size; j Resection margins; k EBER status; l Lymphovascular invasion; m Perineural invasion; n Tumor differentiation; o HPV status; p Smoking history; q Alcohol consumption; r Pack-years of smoking; s Treatment type; t Charlson Comorbidity Score.

Univariate analyses for outcome prediction

The univariate survival analyses of LLM-extracted prognostic variables were first analyzed. Forest plots and representative Kaplan-Meier (KM) curves are illustrated for statistically significant associations. In the main paper, we present disease-free survival (DFS) because it is the most inclusive endpoint, capturing locoregional failure, distant metastasis, and death. Corresponding univariate results for locoregional failure-free survival (LFS), distant metastasis-free survival (DistFS), and overall survival (OS) are provided in the Supplementary Information (Figs. S12–S14).

Across 76 feature-outcome univariate associations tested (19 fields × 4 outcomes), 12 were statistically significant (p < 0.05). HPV status was significantly associated with all three disease-control outcomes, including LFS (HR=0.56, 95% CI: 0.37–0.83, p = 0.005), DFS (HR = 0.61, 95% CI: 0.43–0.88, p = 0.008), and OS (HR = 0.65, 95% CI: 0.43–0.98, p = 0.040), with HPV-positive patients demonstrating a 35–44% reduced risk. Charlson Comorbidity Score (CCS ≥2 vs 0–1) was significantly associated with increased failure risk for LFS (HR = 1.56, 95% CI: 1.14-2.15, p = 0.006) and DFS (HR=1.45, 95% CI: 1.09–1.92, p = 0.011). ECOG performance status (ECOG 2+ vs 0) demonstrated strong adverse prognostic associations across OS (HR = 2.90, 95% CI: 1.34–6.29, p = 0.007), DFS (HR = 2.17, 95% CI: 1.09–4.33, p = 0.028), and LFS (HR = 2.34, 95% CI: 1.05–5.18, p = 0.037). Heavy smoking exposure ( > 40 pack-years vs never) was significantly associated with worse OS (HR = 2.18, 95% CI: 1.15–4.13, p = 0.016) and DFS (HR = 1.81, 95% CI: 1.01–3.24, p = 0.047). For DistFS, only advanced T stage (T4 vs T0-1) reached statistical significance (HR = 1.87, 95% CI: 1.10–3.18, p = 0.020). These associations are summarized visually in forest plots (Fig. 7), with corresponding KM curves shown for significant associations to illustrate survival differences between patient groups defined by categorical prognostic variables with log-rank p statistics (Fig. 8).

Fig. 7. Univariate Cox regression forest plot for disease-free survival (DFS).

Using clinician-evaluated LLM-extracted prognostic variables.

Fig. 8. Disease-free survival representative Kaplan-Meier survival curves.

Clinically relevant prognostic variables identified in univariate analyses, including (a) HPV status, (b) smoking history, (c) pack-years of smoking, (d) Charlson Comorbidity Scores, and (e) ECOG status.

Multivariate analyses for outcome prediction

To evaluate whether LLM-extracted variables provide added prognostic value beyond established clinical factors, multivariate Cox Proportional Hazards (CPH) models were constructed for each outcome. A baseline (clinical-only) model included four established covariates (age, sex, T stage, and N stage). An enhanced model (clinical + LLM-extracted) additionally incorporated three a priori selected LLM-extracted variables: HPV status, smoking history, and Charlson Comorbidity Score, chosen based on established prognostic relevance and sufficient extraction yield to support multivariable modelling¹. The number of events per variable (EPV) for the enhanced models ranged from 3.3 to 7.9 across outcomes, limiting inclusion of additional predictors. Model improvement was assessed using a likelihood ratio test (LRT) comparing nested baseline and enhanced models (3 degrees of freedom, or df). Model discrimination was evaluated using Harrell’s concordance index (C-index), and the integrated Brier score (IBS) was computed for overall prediction error. Internal validation was performed using 1000 iterations of bootstrap resampling^33–35 to obtain optimism-corrected performance estimates.

Inclusion of the three LLM-derived variables significantly improved prognostic performance for disease-control outcome endpoints (Table 7). For DFS, the enhanced model demonstrated improved fit (LRT: χ² = 10.69, df = 3, p = 0.014) and increased the optimism-corrected C-index from 0.57 (95% CI 0.52–0.61) to 0.64 (95% CI 0.63–0.68), corresponding to a ΔC-index of + 0.07 (95% CI: + 0.003 to + 0.12). In adjusted analyses, HPV positivity remained independently associated with reduced recurrence risk (adjusted HR = 0.62, 95% CI: 0.40–0.95, p = 0.029). Similarly, for LFS, the enhanced model improved fit (LRT: χ² = 9.28, df = 3, p = 0.026) and increased the optimism-corrected C-index from 0.59 (95% CI 0.57–0.65) to 0.69 (95% CI 0.68–0.74), yielding a ΔC-index of + 0.11 (95% CI + 0.04 to + 0.16). HPV positivity retained independent prognostic significance (adjusted HR = 0.57, 95% CI: 0.35-0.92, p = 0.022).

Table 7.

Internal validation of multivariate Cox proportional hazards models using 1000 bootstrap resamples for the complete case cohort with both pathology and consultation notes modalities

Outcome	N	Events	EPV	C-index (95% CI)	C-index (95% CI)	IBS 2y (95% CI)	IBS 2y (95% CI)	LRT χ2	LRT p	Prop. p < 0.05	Median χ2
				Clinical only	Clinical + LLM-extracted	Clinical only	Clinical + LLM-extracted
LFS	355	36	5.1	0.59 (0.57, 0.65)	0.69 (0.68, 0.74)	0.022 (0.021, 0.022)	0.022 (0.021, 0.022)	9.28	0.026	69.9%	9.84
DistFS	355	23	3.3	0.76 (0.75, 0.79)	0.75 (0.75, 0.80)	0.025 (0.024, 0.025)	0.025 (0.024, 0.025)	1.41	0.703	1.7%	2.06
DFS	349	55	7.9	0.57 (0.52, 0.61)	0.64 (0.63, 0.68)	0.051 (0.050, 0.052)	0.051 (0.049, 0.051)	10.69	0.014	81.6%	11.81
OS	360	31	4.4	0.57 (0.52, 0.63)	0.66 (0.64, 0.71)	0.021 (0.020, 0.021)	0.021 (0.020, 0.021)	6.18	0.103	38.1%	6.65

For each outcome, optimism-corrected concordance index and two-year integrated Brier score are reported with 95% confidence intervals for the baseline clinical model and the enhanced model incorporating large language model-extracted variables. Improvement in model fit is assessed using the likelihood ratio test (χ², degrees of freedom = 3). Bold values indicate p < 0.05

In contrast, no statistically significant improvement was observed for OS (LRT: χ² = 6.18, df = 3, p = 0.103) likely due to low EPV, despite a numerical increase in the optimism-corrected C-index (from 0.57 to 0.66; Δ = + 0.09, 95% CI − 0.00 to + 0.15). No improvement was observed for DistFS (LRT: χ² = 1.41, df = 3, p = 0.703; ΔC-index = − 0.01, 95% CI − 0.09 to + 0.03). Optimism-corrected 24-month IBSs were similar between baseline and enhanced models across all outcomes, indicating minimal change in overall prediction error.

Discussion

This study demonstrated that an open-source, instruction-tuned LLM (Llama3.3-70B), deployed locally within hospital infrastructure, can accurately extract structured prognostic data from unstructured head and neck cancer notes using expert-guided zero-shot prompts. Without fine-tuning or annotated data, our two-stage prompt design separated direct extraction fields from inference-based reasoning. The model achieved an overall F1 score of 98.6%, with >95% F1 scores across most of the 30 fields, and a very low FNR of 0.1% (Table 1) through a multi-rater clinician evaluation framework in 4500 field-level review decisions in our de-identified corpus of 1360 real-world H&N oncology reports. The manual abstraction time was significantly reduced to 1.7 s per field. Its deployment was fully on-premise, ensuring strict privacy compliance and enabling its use in real-world settings where sensitive data must remain secure. In this study, Llama3.3-70B served as the clinician-evaluated reference model, disagreement patterns and error analyses were derived from clinician review of its outputs, and the resulting majority-vote reference standard was then used to benchmark seven additional models under identical prompts.

In the past, high-performance pipelines were often built on proprietary APIs or public datasets^15,36–38. For instance, RadOnc-GPT required institutional fine-tuning for treatment recommendations³⁹, and Med-PaLM2 achieved benchmark-leading results on medical Question-Answering tasks⁴⁰, yet they remain inaccessible to most hospital settings. In contrast, our method proved to be efficient, scalable, and adaptable. Our findings support the translational utility of open-source, zero-shot LLMs for clinical NLP, echoing previous successes with privacy-preserving approaches in cirrhosis care²³, clinical letters anonymization²⁸.

Our prompt design (Table 8) intentionally constrained outputs to reduce hallucinations and enforce field-specific interpretability. Direct extraction fields defaulted to “Not inferred” when no textual evidence was present, while inference-based fields, such as the CCS and ECOG/KPS, required stepwise reasoning. This allowed the model to recover implicit details, such as smoking status, from phrases like “quit nine years ago” and to generate structured explanations for complex fields, ensuring transparency. With a high Gwet’s AC1 statistic of inter-rater reliability of 0.914 (95% CI: 0.901–0.927, p < 0.001), a majority-vote consensus confirmed 98% of all LLM outputs as correct (Table 1). Figure 2 presents each evaluator’s F1 (Fig. 2a) and FNR (Fig. 2b), respectively, when each clinician’s response served as ground truth. These radar plots highlighted inter-rater variability aligning with Gwet’s AC1 coefficients and majority-vote decisions, rather than implying that any single evaluation was definitively superior. Although expert review showed high consensus, disagreement analysis revealed common causes such as ambiguous phrasing, differing interpretations of field criteria, and inferred diagnoses based on medications.

Table 8.

Prompt-engineering techniques in the direct extraction prompt (P1) and the inference-driven prompt (P2)

Prompt Technique	Implementation / Purpose	P1	P2	Illustrative Fields / Rules
Role conditioning (expert persona)	System-level instruction defines assistant as a domain-specific extractor for oncology reports.	✓	✓	Applies globally to all outputs.
Explicit instruction-following	Mandates line-by-line outputs with the format FieldName: Value, restricting output to predefined fields.	✓	✓	Prevents hallucinated or conversational outputs.
Field-constrained output schema	Fixed 30-field list with null policy: “Not Inferred” used when value is missing. P1 applies only to explicitly stated info; P2 allows reasoning before fallback.	✓	✓	All fields; P1 vs. P2 null-handling distinction.
Schema-constrained categorical prompting	Output constrained to fixed options (e.g., Present / Absent / Suspected / Not Inferred).	✓	✓	Lymph Node Status, Margin Status, HPV, p16.
Instructional zero-shot with in-line exemplars	Clarifying examples embedded within prompt instructions (zero-shot).	✓	✓	Smoking History, Pack Years, Treatment Recommendations. More frequent in P2.
Field-specific reasoning templates	Short field-level rules guide classification or inference (e.g., “quit nine years ago” ? Ex-smoker).	—	✓	Smoking History, HPV Status, Alcohol Consumption.
Chain-of-thought reasoning instructions	Prompts instruct step-by-step logic for computed or inferred values, with output explanations in brackets.	—	✓	Charlson Comorbidity Score, ECOG, KPS.
Deterministic cross-field consistency constraints	Enforces logical dependencies between fields via explicit rules.	—	✓	HPV Status mirrors p16; ECOG-KPS mapping.

Disagreement patterns for the reference model (Fig. 3) indicate where direct extraction prompting was insufficient and hybrid rule-based post-processing applied to the LLM-extracted outputs could be most beneficial. Clinical TNM staging and number of positive lymph nodes showed lower agreement (Table 2) because correctness often required deriving stage from tumour measurements and disambiguating clinical versus pathological staging, and enforcing pathology-confirmation before returning a numeric nodal count (Tables 3, 4). These findings motivate treating both variables as inference-driven tasks, using guideline-aware prompting or retrieval-augmented generation to inject AJCC/NCCN criteria into the prompt.

Charlson Comorbidity Score (CCS) showed the lowest clinician agreement (Gwet’s AC1 = 0.167; Table 2) due to a mismatch between the scoring criteria embedded in the inference-driven prompt and the scoring standard applied by clinician reviewers, rather than hallucination. In 51.5% (17/33) of disagreements, the inference-driven CCS prompt treated hypertension as score-contributing, whereas reviewers adhered to the original 1987 Charlson Comorbidity Index that excludes hypertension³⁰; an additional 42.4% arose from other prompt-included but non-Charlson conditions (e.g., dyslipidemia, hypothyroidism, epilepsy) (Table 5). In this setting, the model can correctly identify comorbidities in text, yet be evaluated as incorrect when the prompt and reviewer criteria are misaligned. To mitigate this, we applied a post-hoc hybrid correction that uses the CoT CCS output format, numeric score with an interpretable, bracketed scoring rationale, to enable deterministic rule-based post-processing that filters non-Charlson conditions and recomputes CCS under the 1987 criteria^22,41. On the 50-case clinician-evaluated subset, this resolved all CCS errors attributable to non-Charlson items, and the corrected CCS values were then propagated to the full cohort for downstream analyses.

The benchmarking analyses contextualized extraction performance across 7 additional instruction-tuned general-purpose and medical-domain LLMs (Table 6, Figs. 4 and 5), as compared to the clinician-validated reference model Llama3.3-70B. A central finding is that long, heterogeneous clinical narratives make context capacity a first-order constraint. Models with restricted context windows underperformed when inputs exceeded capacity, independent of model parameter count. In contrast, models with adequate context windows and sufficient scale achieved substantially stronger extraction performance. For example, the reference model Llama3.3-70B achieved 98.6% F1, while the best smaller open-source alternative achieved 89.6% F1 for GPT-OSS-20.9B (Table 6), at roughly one-third the parameter count of the reference model, illustrating that competitive extraction may be feasible under reduced compute budgets. Medical-domain fine-tuning showed scale-dependent benefits, improving performance among smaller models for MedGemma 27B and 4B, which achieved the third and fourth best performance, while larger general-purpose models with sufficient context capacity remained strongest overall (Fig. 4). This suggests that, for real-world oncology report extraction, architectural suitability to long-document processing may be as important as domain specialization.

Since clinically equivalent information can be expressed with varied wording, we scored benchmark outputs using embedding-based semantic similarity to avoid penalizing valid paraphrases with identical clinical meaning as the ground truths. Because semantic scoring requires a similarity threshold (τ), we selected τ = 0.7 by calibrating it on the clinician-evaluated reference model Llama3.3-70B to best match its originally reported performance (Fig. S7; Table 10); the same threshold was then used for all benchmarked models. In other datasets or institutions, τ may need recalibration depending on documentation style and the desired strictness of semantic matching.

Table 10.

Confusion matrix classification table for semantic-similarity evaluation (threshold τ = 0.7)

Scenario	Model Output	Reference	Semantic Match?	Classification
A	Inferred	Correct (clinician-validated)	✓ (≥0.7)	TP
B	Inferred	Incorrect	✓ (≥0.7)	FP
C	Not inferred	Not inferred (correct)	✓ (≥0.7)	TN
D	Inferred	Correct	× ( < 0.7)	FP (different answer)
E	Not inferred	Correct inference	× ( < 0.7)	FN (missed it)
F	Not inferred	Incorrect inference	× ( < 0.7)	TN (correctly abstained)

TP True Positive, FP False Positive, TN True Negative, FP False Positive.

Structured extraction enables downstream clinical and research workflows that are otherwise difficult to implement from unstructured documentation. The cohort-level distributions (Fig. 6) support rapid cohort identification and stratification (e.g., by biomarkers, smoking exposure, or comorbidity burden), converting routinely documented information into queryable variables. This reduces manual chart review and enables rapid cohort discovery, risk stratification, and outcomes analyses when key prognostic fields are absent or incomplete in structured registries or EHR databases.

Beyond cohort curation, our downstream survival analyses show that LLM-extracted variables improved prediction of disease-control endpoints compared with traditional clinical baselines. Using 1000 bootstrap resampling for internal validation, integrating three a priori selected LLM-extracted variables (HPV status, smoking history, Charlson Comorbidity Score¹) into baseline multivariate CPH models (age, sex, T/N stage) improved discrimination for DFS and LFS, with optimism-corrected ΔC-index values of 0.07 and 0.11, respectively (Table 7; see “Multivariate analyses for outcome prediction”). These gains were supported by nested-model LRT (Table 7), indicating that structured text-derived variables add prognostic information beyond staging and demographics. This strategy is readily extensible to other report types and endpoints (e.g., radiology and follow-up notes, toxicity or treatment-response outcomes) and to external cohorts. In contrast, the absence of statistically supported improvement for OS and DistFS is consistent with lower event counts and greater susceptibility to competing risks in this cohort. Minimal changes in the IBS suggest that the incremental gain in this setting was primarily improved discrimination rather than a reduction in 24-month overall prediction error, which also depends on calibration and may require more events and external validation to assess robustly.

Overall, these results suggest that LLM-based structuring can support rigorous prognostic research by converting unstructured clinical documentation into scalable, privacy-preserving variables that can be directly used in survival modelling pipelines^42,43. They also reinforce the value of augmenting conventional clinical variables with computationally derived, low-dimensional structured features from text or images in radiation oncology outcome modelling⁴⁴. In contrast to high-dimensional text embedding multimodal fusion with sparse clinical covariates, which in our prior experiments suggested performance degradation from over-parameterization and negative transfer^45,46, the present results support an intermediate strategy that integrates a focused set of clinically interpretable extracted variables into conventional survival models.

Nonetheless, this work has limitations. We evaluated eight locally deployed open-source LLMs at one academic hospital site using French-English-language notes, where French is mostly used in the note headers rather than the text body. Generalizability to other institutions will depend on documentation conventions, report templates, guideline adherence, and prompt adjustments. Although the model processed reports from 882 patients, the clinician validation subset included 50 stratified cases due to time constraints by clinician reviewers. However, this sample was rigorously selected to reflect the distribution of note types and report complexity, but may not capture rare field values or systemic documentation variability. Nevertheless, we implemented rule-based post-processing for the low-agreement fields and applied these corrections cohort-wide prior to downstream analyses, which led to downstream clinical characteristics (Fig. 6), and especially, improved outcome prediction for DFS and LFS (Table 7). Differences in guideline adherence, diagnostic thresholds, and phrasing conventions across institutions may affect generalizability. Still, the validation strategy aligns with TRIPOD-LLM guidance on multi-rater evaluation of LLMs in healthcare^47–49. Moreover, benchmarking used shared prompts optimized for the reference model; model-specific prompting might improve certain architectures. Lastly, downstream multivariate analyses were constrained by complete-case requirements and modest EPV; while internal bootstrap validation and regularization mitigated optimism and stabilized estimates, coefficient-level inference remains exploratory for event-sparse outcomes. Future work should evaluate external validation across institutions, investigate missing-data strategies (including multiple imputation where appropriate for survival settings), and compare structured extraction against advanced multimodal representation learning approaches with careful control for overfitting.

Accurate extraction of prognostic and treatment-related variables can support tumour board workflows by making key factors queryable at the point of care (Fig. 6). Extending extraction to longitudinal documentation would enable time-stamped tracking of disease status and treatment-related toxicity¹³. Deployment to streamline fragmented documentation across radiology, oncology, and pathology may further improve workflow efficiency⁵⁰, but will require sustained clinician-AI collaboration to ensure validity and interpretability across care settings^5,47. Future work will evaluate multi-center validation, live EHR integration, uncertainty-aware clinician interfaces, and practical considerations for locally hosted systems such as hardware, throughput, cost-effectiveness, and resource-constrained settings. We will also explore multi-agent frameworks for multi-step extraction and validation, and time-resolved variable extraction to support time-to-event analyses and multimodal alignment with imaging, genomics, and digital pathology.

In conclusion, a locally deployed, zero-shot LLM (Llama3.3-70B) can accurately and efficiently extract clinically meaningful prognostic variables from real-world head and neck oncology notes while preserving patient privacy. The model demonstrated strong agreement with clinician reviewers and achieved near-perfect accuracy across most fields, enabling substantial efficiency gains for both large-scale cohort curation and clinical decision support workflows. Benchmarking against smaller general-purpose and medical domain models identified practical open-source alternatives, including GPT-OSS-20.9B and MedGemma-27B or 3.9B, which achieved competitive performance with lower computational requirements. These results also further revealed the critical role of long-context capacity for processing real-world clinical reports. Downstream analyses showed that LLM-extracted variables can enable swift cohort identification and improve the prediction of disease-control outcomes beyond conventional clinical variables (e.g., age, sex, and stage). Overall, these findings establish prompt-based, privacy-preserving LLM extraction as a scalable foundation for real-world precision oncology data curation and prognostic modelling. While challenges remain in enforcing strict clinical reasoning, local LLM deployment offers a practical and secure pathway toward clinical NLP integration.

Methods

Study design and data collection

We conducted a retrospective study using 1360 de-identified pathology and consultation notes from 882 patients with biopsy-confirmed H&N cancer (2010–2023). The study was approved by the CIUSSS West-Central Montreal Research Ethics Board for research conducted at the Jewish General Hospital (protocol MP-05-2022-2469) and was performed in accordance with the Declaration of Helsinki. Notes were extracted from the Varian ARIA Oncology Information System, converted to text format, and de-identified according to institutional guidelines (i.e., name, birthdate, unique hospital ID). The requirement for informed consent was waived by the Research Ethics Board due to the retrospective nature of the study and the use of de-identified data. Reporting followed the LLM-TRIPOD checklist⁴⁷ (available at https://tripod-llm.vercel.app/).

Local LLM automation pipeline and deployment

We leveraged an open-source Llama3.3-70B on-premises (Fig. 1) using the Ollama Application Programming Interface (API; available at https://ollama.com/library/llama3.3)²⁴ backend on a 2 × NVIDIA RTX 5000 32 GB GPUs to support privacy-preserving deployment. A custom Python (v3.10) pipeline, built using LangChain, managed prompt construction and parsing of the 30 structured prognostic fields (Expert-Guided Definition and Extraction of Clinically Relevant Prognostic Data). The LangChain (v0.2.17) Ollama (v0.4.4) integration sends prompts to the local Llama3.3-70B instance and receives the generated outputs programmatically. A HuggingFace tokenizer (https://huggingface.co/baseten/Meta-Llama-3-tokenizer) was used to ensure input text and instructions remained within a defined 5000-token context window (see “Prompt Engineering Techniques and Prompt Templates for the Main Analysis”). Though the model supports up to 128 k tokens, the reduced window was chosen for efficiency, which still exceeded the maximum required across all cases, as verified through pre-computation of token lengths. All extractions were performed in a zero-shot instruction tuning prompting manner (see “Prompt Engineering Techniques and Prompt Templates for the Main Analysis”, and “Experimental Prompting Modes and Evaluation Strategy”), without fine-tuning, ensuring generalizability and resource-efficient deployment within hospital infrastructure.

Expert-guided definition and extraction of clinically relevant prognostic data

We defined 30 expert-guided structured fields based on clinical relevance in head and neck cancer. This encompassed patient demographics (i.e. age, sex), tumour characteristics (i.e., anatomical primary site, tumour type differentiation, HPV/p16 status, clinical and pathological TNM stage, lymph node (LN) status and the number of positive LN, extranodal extension, primary tumour size, pathology details, margin status, immunochemical profile including the Epstein-Barr virus-encoded small RNAs (EBER) status, lymphovascular invasion, perineural invasion, radiological lesion assessment such as SUV values from PET scans), lifestyle factors (i.e., smoking history, pack years number, alcohol consumption), patient presentation (i.e., symptoms at presentation, prior conditions, and previous treatments), treatment recommendations (i.e., follow up plans), and clinical performance indices (e.g., Eastern Cooperative Oncology Group (ECOG), Karnofsky Performance Status (KPS), and Charlson Comorbidity Score or CCS). Fields were grouped into direct-extraction or inference-based categories depending on whether their values were explicitly stated or required reasoning. For example, sex and margin status were directly extractable, while the CCS required clinical interpretation of narrative text. These field-level distinctions informed the design of downstream prompts (“Prompt Engineering Techniques and Prompt Templates for the Main Analysis”) and their evaluation.

Prompt engineering techniques and prompt templates for the main analysis

We designed two structured prompt templates, P1 and P2, used in combination. Prompts were written in JSON format and refined through a human-in-the-loop process with a senior radiation oncologist. The first was a Direct Extraction Prompt (P1) (Fig. 9), designed for straightforward lookup and transcription of explicitly stated information, which enforced strict outputs of Not inferred, or NI, for any field not explicitly stated in the input report. The second, an Inference-Driven Prompt (P2) (Fig. 10), focused on reasoning-based fields such as the CCS and ECOG/KPS performance status. Its additional instructions allowed the model to infer missing values from implicit information, even when those fields were not explicitly mentioned in the input text. Direct instruction, zero-shot prompting, role conditioning, and field-constrained output schema were used for both P1 and P2. CoT reasoning⁵¹, an internal, step-by-step reasoning path that links a question to its answer, was mainly applied to P2. Table 8 details how each prompt handled null value logic, applied explicit and reasoning prompting strategies, and enforced cross-field consistency constraints (e.g., HPV status must mirror p16 status). This framework was designed to be scalable and adaptable for extracting structured, expert-defined variables from unstructured clinical notes across diverse contexts.

Fig. 9. JSON prompt template of direct-extraction prompt (P1) field extractions.

`Not inferred' when information was not explicitly mentioned from input report.

Fig. 10. JSON Prompt template inference-driven prompt (P2) of field extractions.

`Not inferred' when information was not explicitly mentioned from the input report.

Iterative prompt refinement and human-in-the-loop design

Prompts were developed iteratively through a human-in-the-loop process. Initial outputs were reviewed for format adherence, semantic accuracy, and interpretability by the research team. Prompts were then refined with clarifying constraints, reworded instructions, or additional field-specific logic to minimize ambiguity and improve model reliability across all 30 fields.

Role conditioning, direct and explicit instructions-following, and field-constrained output schema

All prompts began with a role declaration assigning the model the task of acting as an extraction assistant specialized in oncology report interpretation. This primed domain-relevant completions and suppressed generic or conversational language, refraining from extraneous tokens. Both P1 and P2 included explicit formatting rules: line-by-line output with a fixed key-value schema FieldName: Value, strict adherence to the 30 extracted field list, and"Not Inferred” as the default null value. Without this reinforcement of the output format constraint, the results may have been more free-formed language. All outputs adhered to a consistent and parseable layout. This format constraint minimized hallucinated fields or unsolicited commentary, ensuring outputs remained clean and evaluable. Notably, in P1, fields were labelled “Not Inferred” if the relevant information was not explicitly stated in the input text, enforcing a conservative extraction policy. In contrast, in P2, the model was instructed to attempt reasoning or inference before resorting to ’Not inferred’. This distinction was the main reason why such prompts were written in two separate template JSON files, whose experimentation is detailed in “Experimental Prompting Modes and Evaluation Strategy”.

Zero-shot prompting for free-form outputs

Zero-shot prompting, which refers to querying the model without providing any in-context examples, was used in P1 and P2 for fields where outputs were more free-form or variable in structure. These included clinical descriptors such as Pathological TNM, Clinical TNM, Primary Tumour Size, Pathology Details, and Resection Margins.

Schema-constrained zero-shot prompting for categorical fields

For fields with explicitly available categorical values, such as Lymph Node Status Presence/Absence, Extranodal Extension, Lymphovascular Invasion, Perineural Invasion, and Margin Status, p16 or HPV status, we applied zero-shot prompts with explicit constraints on output categories. Each prompt listed only the permitted response options (e.g., “Present,” “Absent,” “Suspected,” “Not Inferred”) and included field-specific interpretation rules. For example, the prompt for Lymph Node Status Presence/Absence stated: “If the text explicitly says there is no neck lymphadenopathy, you may conclude “Absent”. Otherwise, return “Not Inferred’.”In these cases, direct examples were not required, and the prompt instead relied on instruction-based guidance to elicit the correct field content. This approach allowed the model to flexibly generate descriptive outputs while still conforming to a predefined field structure. This minimized semantic drift (e.g., avoiding “likely present” or “could be involved”) and facilitated categorical evaluation. This technique was implemented more frequently in P1, where conservative extraction was prioritized for clearly stated categorical findings.

Instructional zero-shot prompting with embedded exemplars

For semi-structured or descriptive fields such as Smoking History, Alcohol Consumption, Pack Years, Treatment Recommendations, and Patient Symptoms at Presentation, prompts combined zero-shot prompting with embedded in-line examples. These clarifying examples (e.g., “If the text says ‘quit nine years ago’, that is Ex-smoker”) were included as guidance, but not formatted as input-output demonstrations, maintaining zero-shot status. This approach helped standardize output format for list-style or interpretive fields without constraining them to rigid categories. This is more often used in P2.

Field-specific reasoning templates

In P2, certain fields included concise logic cues to support interpretation. For example, for Smoking History, the prompt instructed: “If the text says ‘quit nine years ago’, classify as Ex-smoker.” For HPV Status, the prompt noted: “Must mirror p16 Status exactly.” These field-specific guides helped map variable clinical phrasing into discrete categories, improving extraction robustness.

CoT reasoning for structured inference fields

Mainly used in P2, for fields requiring clinical computation or interpretive reasoning, we incorporated CoT prompting to elicit intermediate steps alongside final outputs. In the case of the CCS, the model was explicitly instructed to think step by step: identify relevant comorbidities, assign point values based on a defined rubric, and sum them to generate a total score. The prompt required that the final output include the score followed by a breakdown of the reasoning in brackets (e.g., “Charlson: 3 (Diabetes with end-organ damage = 2; Hypertension = 1)”). Similarly, for both ECOG and Karnofsky Performance Status (KPS), the model was directed to provide reasoning in brackets when converting between the two scales or when inferring performance scores based on available information. This structure encouraged the model to apply domain-specific logic and produce interpretable outputs that support clinician evaluation.

Cross-field consistency constraints

Logical dependencies between fields were enforced to ensure internal consistency. For example, the prompt specified that HPV Status must match p16 Status; if one was “Positive,” the other could not be “Negative.” Such constraints reduced contradictory outputs across interdependent fields. This was implemented in P2.

Experimental prompting modes and evaluation strategy

We evaluated four experimental prompting modes with two aspects to assess the impact of prompt structure and input strategy (Fig. 11). First, combined or separate clinical note inputs of pathology and consultation reports. In the combined approach, the pathology and consultation notes were concatenated and used as input to the model simultaneously, providing full context. This deliberate choice was due to occasions when a field that could be assumed to appear in a pathology note, such as perineural invasion status, could sometimes appear in the consultation notes instead, or vice versa. Therefore, we aimed for the model to have full context for all available information when prompted. In the separate approach, the two note types were processed in separate LLM calls, and their results were later merged. Second, a single-step or two-step prompt sequence of P1 and P2 was made. In the single-step prompt, P1 and P2 were merged into one prompt given to the input notes; whereas in the two-step prompt, the two prompts were presented separately. Collectively, these strategies resulted in four modes: Mode 1 used a single combined prompt on concatenated input; Mode 2 applied P1 and P2 sequentially on the combined input; Mode 3 used a single prompt on each note independently; and Mode 4 applied P1 and P2 separately to each note, requiring up to four LLM calls. Prompting configurations are visualized in Fig. 12, with corresponding token counts shown in Fig. 11. All prompts were run using the Llama3.3-70B LLM with a maximum context window of 5000 tokens within our GPU constraints, which safely exceeded the maximum number of tokens needed for all experimental modes.

Fig. 11. Token counts needed for four experimental prompting strategies.

Experimental mode 2 was selected as best performing from preliminary results and was therefore the strategy used in producing results for clinician evaluations and reported in the main text.

Fig. 12. Four experimental prompting modes for structured extraction from head and neck cancer clinical notes.

Mode 1 concatenates the pathology report and consultation note ( ⊕ ) and submits the combined text to a single prompt that performs both direct extraction (P1) and inference-driven reasoning (P2), yielding the final 30-field summary. Mode 2 uses the same combined input but sends it to two separate prompts (P1 and P2); the two outputs are merged ( ⊕ ) before producing the structured fields. Mode 3 processes the pathology and consultation notes independently: each document is passed once through a prompt that combines P1 + P2, and the two partial summaries are merged to form the final output. Mode 4 applies both P1 and P2 to each document separately,up to four sub-calls in total, followed by a merge of all outputs. The circled plus symbol ( ⊕ ) indicates concatenation of inputs or merging of prompt outputs, as noted in the legend.

Each mode was piloted on five of the longest notes (i.e., above 2000 tokens, when combined with the prompts), chosen to gage the model’s extraction performance under complex narrative conditions. The study team inspected the output of each mode to identify discrepancies, particularly the NI outputs. Modes 1 and 3 were simpler, but occasionally inferred or cross-document information was missed. This preliminary screening revealed that the combined-input, two-step prompt configuration (Mode 2) had the fewest evident omissions (fields incorrectly flagged as NI), making it the most promising strategy for further evaluation. Consequently, we adopted this approach for the entire dataset.

Clinician-AI evaluation and metrics

We conducted a multi-rater evaluation of the model’s extracted structured fields against expert knowledge. To ensure a representative and manageable evaluation set, a stratified random sample of 50 cases was selected, which reflected the cohort report availability distributions and complexities. They were grouped by pathology-only (n = 96, 11%), consultation-only (n = 308, 35%), and combined (n = 478, 54%). Thus, 5 cases with pathology-only reports, 18 with consultation-only notes, and 27 with both types were selected. The detailed sampling workflow is described in “Clinician Evaluation Case Sampling Strategy” and Fig. 13.

Fig. 13. Stratified random sampling for the 50 cases selection or clinician evaluations.

From all cases for each one of the three report availability types, we show the relationship between LLM outputted “Not inferred” counts and the number of input tokens. We denoted three levels of extraction difficulties based on the number of “Not inferred” fields into low (0–10), medium (10–20), and high (20–30).

Three board-certified radiation oncologists independently reviewed each LLM-extracted field alongside the input reports. Each evaluator responded “Agree" (i.e., correct extraction) or “Disagree" (i.e., incorrect extraction) for each field. A majority-vote consensus (≥2 of 3) was used as ground truth (GT) for subsequent performance evaluation. If a field was truly present in the text and the LLM extracted it correctly, as agreed by the evaluator, the result was recorded as a True Positive (TP). If the model incorrectly labelled a present field as “Not inferred,” it was a False Negative (FN). Conversely, if no relevant information existed for a field and the model correctly returned “Not inferred,” it was a True Negative (TN). Finally, if the model extracted a value for a field that was not supported by the text, it was classified as a False Positive (FP). The confusion matrix is outlined in Fig. 1g, with corresponding formulas shown in Equations (1) to (4).

The overall aggregate performance across all 4500 responses (50 patients × 30 fields × 3 evaluators) was computed, as well as individual field metrics, including accuracy, precision, recall, F1 score, and False Negative Rate (FNR). In addition, we assessed inter-rater reliability among the three evaluators using Gwet’s AC1 statistic. Inter-rater variability was further demonstrated by computing performance metrics of model field extractions with each rater’s evaluation as ground truth. The clinician evaluation workflow is summarized in Fig. 1h.

$$\mathrm{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN}$$ 1

$$\mathrm{Precision}=\frac{TP}{TP+FP}$$ 2

$$\mathrm{Recall}=\frac{TP}{TP+FN}$$ 3

$$F1=\frac{2\times \mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}$$ 4

$$\mathrm{False\; Negative\; Rate\; (FNR)}=\frac{FN}{TP+FN}$$ 5

Clinician evaluation case sampling strategy

We aimed to mirror the real-world distribution of report types while preserving a range of extraction difficulties. From the evaluation cohort (882 patients; 1360 reports), we first categorized patients as pathology-only (n = 96, 11%), consultation-only (n = 308, 35%), and combined (n = 478, 54%). In contrast to earlier works that relied on simple random sampling^15,52,53, we applied a proportional, stratified sample that matched these category frequencies. Each report varies depending on its length, whether both types are available, and the degree of detail; it is reasonable to expect certain structured fields queried to be extracted were legitimately absent. Correctly returning “Not inferred” (NI) when information is missing was therefore central to our evaluation; thus, we deliberately selected cases spanning a balanced range of NI counts across the three categories. For each patient, we recorded the maximum input token length, defined as the larger of the extraction- and inference-prompt added by the input report token counts, and the NI field count. After all reports were processed, NI counts were plotted against token budgets to visualize the extraction difficulty shown in Fig. 13. As expected, the NI counts decreased as the number of max input tokens, or the report length, increased.

Cases exceeding category-specific NI thresholds (i.e., pathology-only ≥ 20, consultation-only ≥10, combined ≥8) were excluded to remove extreme outliers. Applying this cut-off was also intended to prevent clinician evaluators from spending excessive time on cases dominated by NI outputs. Remaining candidates in each category were divided into three equal-width NI bins (e.g., 0–4, 5–9, 10–14), where k cases were required (k = 5, 18, and 27, respectively). We drew k/3 cases at random without replacement from each bin; any shortfall was filled by additional draws from the largest bin. Fixed random seeds ensured full reproducibility, yielding 5 pathology-only, 18 consultation-only, and 27 combined-report cases.

Comparative benchmarking of general-purpose and medical domain language models: evaluation design

Manual correction for cases with more than 2 disagreed reviews was performed. A ground truth of each of the 30 extracted fields for the 50 cases was extracted and corrected if it had two or more reviewer agreements (“Field-Specific Error Patterns in Low-Agreement Variables”). This ground truth was then used to benchmark seven additional open-sourced instruction-tuned LLM models against the clinician-validated Llama3.3-70B reference shown in Table 9, with varying model sizes and are either general-purpose or medical domain-tuned. They include GPT-OSS-20.9B, MedGemma-27B, 3.9B, Mistral-7B, Gemma-2B, Phi3-Mini, Meditron-70B, and Llama2-7B. For benchmarking, we manually reviewed and corrected 31/1500 extracted fields where more than two clinicians disagreed with the LLM output. These corrections ensure benchmark models are evaluated against clinically accurate ground truths.

Semantic evaluation of textual variability and threshold sensitivity analysis were conducted. The same prompting strategies used for the main model were applied to the benchmarked models. Clinically equivalent information may be expressed with different wording while retaining the same clinical meaning. Therefore, exact string matching between benchmark model outputs and reference outputs may incorrectly penalize valid responses. To accurately and fairly compare benchmark model outputs to the clinician-validated reference ground truths, we evaluated agreement using an embedding-based semantic similarity approach. Embeddings were generated for each benchmark model’s structured output using the nomic-embed-text model via the Ollama API, cosine similarity was computed to the corresponding ground-truth embedding, and treated outputs with similarity ≥τ as semantic matches (i.e., two outputs are considered semantically equivalent when their similarity exceeds τ). We assessed threshold sensitivity by sweeping τ from 0.5 to 0.9 (Fig. S7). A sensitivity threshold of τ = 0.7 was chosen because, when re-evaluating the clinician-evaluated Llama 3.3: 70B reference model against its own clinician-validated outputs, τ = 0.7 provided the closest agreement to best match the clinician-evaluated reference performance, indicating that it captures semantically equivalent clinical text while rejecting false matches; this τ was then used for all benchmarked models. Table 10 details the confusion matrix rationale of benchmarked models, thereby obtaining the benchmarked models’ performance metrics of accuracy, precision, recall, F1, and FNR as provided in equations (1) - (5). Table 10 summarizes the semantic-similarity confusion matrix logic.

Evaluation of downstream clinical tasks

To demonstrate downstream utility, we evaluated how the LLM-extracted prognostic variables could inform clinical decision support workflows (Fig. 1). We first show their distributions in the full head and neck cancer cohort to enable patient querying and cohort selection. We then assess their added value for outcome prediction models for survival and disease progression.

Univariate analyses for outcome prediction

Four clinical outcomes were evaluated for each LLM-extracted prognostic variable. Locoregional failure-free survival (LFS) was defined as time from diagnosis to locoregional recurrence or death from any cause, whichever occurred first. Distant metastasis-free survival (DistFS) was defined as time from diagnosis to the development of distant metastasis. Disease-free survival (DFS) was defined as time from diagnosis to the first occurrence of locoregional recurrence, distant metastasis, or death from any cause. Overall survival (OS) was defined as time from diagnosis to death from any cause. Patients without an event were censored at last follow-up.

Kaplan-Meier (KM) analysis with log-rank testing was used to assess the prognostic value of 19 LLM-extracted categorical variables across the four endpoints. For variables with more than two categories, omnibus log-rank tests were applied. Univariate Cox proportional hazards (CPH) regression was performed to estimate hazard ratios (HRs) with 95% confidence intervals (CIs). KM curves are shown for variables with statistically significant log-rank tests (p < 0.05), and forest plots summarize univariate HRs across all tested fields for each outcome. Multivariable CPH regression using complete-case analysis was conducted separately to evaluate the independent prognostic value after adjustment for confounders. Results for secondary endpoints (LFS, DistFS, and OS) are shown in Figs. S11–S14.

Multivariate analyses for outcome prediction

To evaluate whether LLM-extracted features provide independent prognostic value beyond established clinical factors, multivariate CPH models were constructed for each endpoint (DFS, LFS, OS, DistFS). Baseline clinical models included age, sex, T stage, and N stage. Enhanced models (i.e. clinical + LLM-extracted) incorporated three additional LLM-derived variables selected a priori: HPV status (positive/negative), smoking history (ever/never), and CCS (0-1 vs ≥2). Model comparisons were performed on complete-case patient cohorts for each outcome (n = 349–360 depending on outcome availability). Incremental prognostic value of the LLM-derived variables was assessed using likelihood ratio tests (LRTs) comparing nested baseline and enhanced models (3 additional degrees of freedom). Model discrimination was quantified using Harrell’s concordance index (C-index), and overall prediction error was evaluated using the integrated Brier score (IBS) at 24 months. All survival analyses were performed using the Python lifelines package (version 0.27.4).

Given the limited number of events, the 7-variable enhanced models had modest events-per-variable (EPV) ranging from 3.3-7.9 across outcomes shown in Table 7. To mitigate overfitting and stabilize coefficient estimation, L2 regularization (penalizer = 0.1) was applied. Internal validation was performed using 1000 bootstrap resamples per outcome^33–35. In each resample, both the baseline and enhanced Cox models were refitted. For each model and resample, performance (C-index and 24-month IBS) was evaluated in the bootstrap sample ("apparent" performance) and by applying the bootstrap-fitted model to the original dataset ("test" performance). Optimism was defined as the mean difference between apparent and test performance across resamples. Optimism-corrected performance was obtained by subtracting this mean optimism from the apparent performance in the original dataset for each model. The bootstrap percentile 95% CIs were obtained from the optimism-corrected differences between each enhanced and baseline model across bootstrap resamples: ΔC-index (enhanced − baseline) and ΔIBS (enhanced − baseline). As a descriptive measure of model stability, the LRT was recalculated within each bootstrap sample, and the proportion of significant tests (p < 0.05) was recorded. Complete multivariate model comparisons are summarized in Table 7.

Author contributions

Y.Z.: Conceptualization; Data curation; Formal analysis; Investigation; Methodology; Software; Validation; Visualization; Writing - Original Draft, Review & Editing; L.R.S., and F.F.: Conceptualization, Formal analysis; Investigation; Methodology; Software; Writing, Review & Editing. G.S., M.D., & T.Y.T.S.: Formal analysis; Investigation; Methodology; Validation; Review & Editing; contributed clinical expertise and interpretation of relevant fields. K.S. & P.B. Datacuration. F.M.: Conceptualization; Review & Editing. S.A.E: Conceptualization; Funding acquisition; Investigation; Methodology; Project administration; Resources; Supervision; Review & Editing.

Data availability

Data is provided within the manuscript or supplementary information files. The de-identified text data used in this study are not publicly available due to privacy constraints. The code is available on GitHub: https://github.com/engerlab/HNC_Reports_Agents.

Code availability

The de-identified text data used in this study are not publicly available due to privacy constraints. The code is available on GitHub (https://github.com/engerlab/HNC_Reports_Agents).

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at https://doi.org/10.1038/s41698-026-01521-y.