Automated extraction of temporalized tumor evolution from oncology EMRs using natural language processing

Abstract

Background

Extracting temporally sensitive outcomes such as tumor progression from unstructured electronic medical records (EMRs) remains a major challenge in oncology. This study evaluates a solution with a domain-adapted natural language processing (NLP) pipeline designed to extract structured, temporally anchored clinical outcomes from narrative EMR data.

Patients and methods

Patients with oncogene-addicted advanced or metastatic non-small-cell lung cancer (NSCLC) treated with oral targeted therapies between January 2020 and June 2023 at a French academic hospital were included. Extracted Facts were benchmarked against expert annotations. All outputs were mapped to Observational Medical Outcome Partnership vocabularies. F1-scores were calculated for the correct Concept detection without and with their Temporality. Real-world progression-free survival (rwPFS) was estimated based on retrieved clinical outcomes.

Results

Among 1030 NSCLC patients treated between 2020 and 2023, 112 were confirmed to have advanced or metastatic disease with an oncogenic driver mutation, primarily EGFR (n = 66), ALK (n = 23), and KRAS (n = 16). The NLP pipeline achieved high accuracy in extracting clinical concepts, with an F1-score of 79.7% for tumor evolution concepts and 62.0% when temporality was included. Overall performance across all domains reached F1-scores of 76.5% for concept extraction and 63.7% with temporality. Median rwPFS was 21.9 months for EGFR-mutated, 52.4 months for ALK-translocated, and 5.0 months for KRAS-mutant tumors, in line with published benchmarks. Reviewing automatically collected data was 5.8 times faster compared with manual collection.

Conclusions

Our solution demonstrates robust performance for extracting temporally structured tumor outcomes from EMRs and supports the reconstruction of real-world endpoints in oncology.

Highlights

• •Evaluation of clinical outcomes from EMR narrative using NLP for oncology.
• •F1-score of 79.7% for tumor evolution concepts and 62.0% with temporality.
• •Overall F1-scores were 76.5% for concept extraction and 63.7% with temporality.
• •The solution was able to recreate rwPFS data in line with published benchmarks.

Introduction

With ongoing advances in digital health and the widespread adoption of electronic medical records (EMRs), the volume of clinical data available for oncology research has grown substantially. However, much of this information remains unstructured, limiting its systematic use in real-world evidence generation. Structuring EMR data is essential to unlock its full research potential, particularly for retrospective cohort studies and endpoint reconstruction. In oncology, critical clinical concepts including treatment regimens, toxicities, co-medications, symptoms, and disease course, are often documented in narrative form where the temporal information is lacking, rather than captured through structured fields. This challenge is further amplified in the context of oral therapies, where treatment start dates, toxicity onset, and tolerability assessments are frequently underreported or inconsistently recorded. Yet, these parameters directly inform the tumor evolutions, which is essential for assessing therapeutic efficacy.

Natural language processing (NLP) approaches have emerged to facilitate the extraction of real-world clinical data from unstructured sources. Although these methods show promise, most existing solutions are not tailored to the complexity of oncology and often fail to incorporate temporality and the capacity to anchor clinical events to specific time points or intervals. Temporality is critical for reconstructing disease trajectories and deriving endpoints such as tumor progression and thus progression-free survival (PFS). Several studies have explored the capacities of NLP to collect data from an unstructured document, but the dates related to these data points were collected mostly from the document’s pre-structured information.1, 2, 3, 4 Extracting temporal information from unstructured clinical narratives is essential for inferring critical endpoints such as PFS. To our knowledge, no study has specifically focused on temporally anchored tumor evolution and real-world (rw)PFS, directly from the clinician’s narrative. Moreover, the temporal information described by physicians is not always accurately or consistently captured.5, 6, 7 Additionally, few tools leverage standardized data models such as Observational Medical Outcome Partnership (OMOP) or Fast Healthcare Interoperability Resources, which are essential for data interoperability and integration across systems. Although recent advances in large language models (LLMs) offer new opportunities for text-based data extraction, their limited alignment with standard terminologies currently constrain their applicability in structured oncology research on their own and require being combined in complex pipelines including classical NLP technologies. These limitations significantly hinder the adoption of current NLP systems in oncology and constrain their translational potential.

To address this gap, a domain-adapted NLP solution was developed to extract and structure oncology-specific clinical information from EMRs. It was developed to offer deterministic outputs, traceable to source documents, and natively mapped to standardized data models. The system combines rule-based and machine learning algorithms to identify key variables while preserving contextual attributes, including temporality, data granularity, and clinical certainty, such as hypothesis or uncertain events. Extracted data are traceable to their original source and can eventually be manually reviewed for quality control. The solution framework (hereafter referred to as ‘solution’) is described in Supplementary Figure S1, available at https://doi.org/10.1016/j.esmorw.2025.100660. The present study was carried out to assess the ability of the solution to extract temporally enriched tumor evolutions, particularly tumor evolution events, to support the reconstruction of rwPFS. To this end, the solution was evaluated on EMRs from patients with advanced non-small-cell lung cancer (NSCLC) receiving oral targeted therapies at a leading academic cancer center (University Hospital of Toulouse, France).

Materials and methods

Patient selection

All adult patients treated between January 2020 and June 2023 at the University Hospital of Toulouse for advanced or metastatic NSCLC harboring an oncogenic driver alteration (i.e. EGFR mutation, KRAS mutation, ALK translocation, etc.) and receiving at least one line of oral targeted therapy were eligible for inclusion. Patient identification was carried out through an institutional clinical registry. The study was approved by the Institutional Review Board and Ethics Committee of the University Hospital of Toulouse, and information notices were sent to all living patients, in accordance with French regulatory requirements.

Data sources

To assess the performance of the NLP system, we analyzed multiple types of clinical documents extracted from the hospital’s EMRs. For each identified patient, the data team from the University Hospital of Toulouse retrieved all discharge summaries (DSs) and external consultation letters (ECLs) dated between January 2020 and June 2023, even if the patient began their treatment before January 2020. Documents were extracted for each patient once inclusion criteria were confirmed. These document types represent the primary sources of longitudinal oncology information in the local clinical workflow. All documents were in French.

Variable extraction and structure

The NLP solution was designed to automatically extract and structure clinical data from unstructured medical reports, without requiring prior definition of the specific variables of interest. The system leverages the OMOP common data model and common standards of designation (e.g. ‘Facts,’ ‘Concept,’ ‘Temporality,’ etc.) to represent extracted data in a standardized and interoperable format. Each extracted data element, termed a ‘Concept,’ is a clinically meaningful unit [e.g. ‘EGFR mutation,’ ‘progressive disease,’ ‘Eastern Cooperative Oncology Group (ECOG) 2’], mapped to an OMOP vocabulary concept identifier (id) (hereafter referred to as a Concept). A concept id is a unique identifier set by the Observational Health Data Sciences and Informatics (OHDSI) that maps any clinical data point to a standardized identifier in several standard ontologies (e.g. SnomedCT, RxNorm, etc.).⁸ Concepts are paired with additional information within a ‘Fact:’ for temporality, dates can be absolute (e.g. June 2020) or relative (e.g. 3 months ago). Relative dates are estimated based on the document date, which is defined as ‘present.’ Dates may be expressed as single points in time or as intervals between two dates, as described in the clinical narrative. Dates are extracted either from the document date (retrieved from the ECL or DS) or from explicit mentions within the narrative text. If a section-specific date is identified, it takes precedence over the document’s present date. All Facts, including clinical outcomes, are associated with a date, either explicitly stated in the narrative or defaulting to ‘present’. Other information contained in a Fact are details describing numeric values (e.g. ECOG score), molecular variants (e.g. KRAS G12C), or contextual modifiers (e.g. ‘negative,’ ‘family history,’ ‘uncertain’) as described in Supplementary Figure S1, available at https://doi.org/10.1016/j.esmorw.2025.100660.

All terms used here are meant to align with the OHDSI common conventions and are not intended proprietary.

Data processing pipeline

Text from EMR documents is processed through a hybrid pipeline combining machine learning and rule-based natural language processing (Figure 1). Pre-trained models detect potential entities and facts, while expert-defined rules disambiguate meaning and map terms to standardized ontologies. Extracted data are referred to as Facts, consisting of a Concept and its associated Temporality, which are eventually presented to expert reviewers through a custom-built interface that enables manual validation and correction for a possible quality assessment.

Figure 1.

Data processing. OCR, optical character recognition; OMOP, observational medical outcomes partnership; QA, quality assurance.

Annotations comparison

To evaluate the accuracy of the automated extraction, two expert oncologists and one clinical research assistant reviewed and manually corrected all detected facts in the dataset, in a prelabeled display. This pre-annotated curated data served as the reference for performance comparison. The manual review was conducted directly within the NLP interface to ensure alignment with the context of original annotations.

Statistical analysis

Patients are described by sex, tumor histology, driver mutation, real-world follow-up, defined as the time from the start of targeted therapy, as documented in the physician’s narrative, even if therapy began before the document selection period, until the last reported date. To further evaluate the solution’s ability to infer PFS, we assessed rwPFS, defined as the period from the targeted therapy onset until tumor progression or death (Kaplan–Meier survival model), in patients with EGFR-mutated, ALK-translocated, and KRAS-mutant tumors. These cohorts were selected because the survival benchmarks are well-established.

Objectives

The primary objective was to assess the solution’s ability to extract temporally anchored tumor evolutions, with a focus on tumor evolution events, including radiologic progression, stable disease, and partial or complete responses. Events were matched to formal descriptors when available (e.g. RECIST 1.1 criteria) or inferred from clinician or radiologist statements. Secondary objectives included the evaluation of extraction performance for tumor diagnosis, tumor-specific treatments, ECOG performance status, molecular alterations, and metastatic sites, each of which plays a critical role in treatment stratification and real-world outcome analysis in NSCLC.

Performance metrics and evaluation

We evaluated performance by comparing the system’s automated output against curated data on a per-document basis. A Fact was considered a true positive if the extracted Concept matched the reference annotation, with an associated Temporality falling within a ±16-day window, reflecting the typical uncertainty of date attribution in oncologic documentation.

We computed standard performance metrics: ‘Recall’ was defined as the proportion of relevant Facts in the curated data that were correctly identified by the system; ‘Precision’ corresponded to the proportion of Facts identified by the system that were correct. The F1-score, the harmonic mean of Recall and Precision, provided a single metric balancing sensitivity and specificity. All metrics were reported as F1-scores in percentage.

We further reported these performance metrics across two levels: Concept only (e.g. detection of ‘progressive disease’ without time or context), and Concept + Temporality (e.g. ‘progressive disease on 12 June 2022’), defined as the correct identification of the Concept and its Temporality combined. Scores were calculated for the Concept alone, and the Concept with its Temporality. All metric definitions and illustrative examples are detailed in the Supplementary Methods section.

The performance of the system was assessed by comparing the automatically extracted facts with expert-curated annotations, which are considered the reference standard. Performance was reported using F1-scores, a widely accepted metric that combines Precision (proportion of correct predictions among all predictions) and Recall (proportion of correct predictions among all relevant cases), thereby offering a balanced measure of accuracy. We also compared the time required to complete 48 items from electronic case report form (eCRF) data manually with the time needed to review automated data using the solution, by evaluating the duration clinicians spent manually collecting patient data versus the time spent retrieving the same data through the solution.

Results

Patient cohort

Between January 2020 and June 2023, 1030 patients who were treated for a locally advanced or metastatic NSCLC, and 120 patients who were treated for suspected NSCLC with an oncogenic driver mutation were identified at the University Hospital of Toulouse. Of those 120 patients, a total of 1028 clinical documents were analyzed, including 891 ECLs and 137 DSs. Following data extraction and review using the NLP pipeline, 112 patients were confirmed as having advanced or metastatic NSCLC with a molecular alteration (Figure 2). Importantly, no patient with a driver NSCLC among the 120 patients identified by the University Hospital of Toulouse was missed (i.e. patient identification rate was 100%). Patient characteristics are summarized in Table 1. Among the 120 patients, 109 had adenocarcinoma, 2 had squamous-cell carcinoma, and 1 had sarcomatoid carcinoma; no cases of large-cell carcinoma were found. Driver mutations included EGFR (n = 66), ALK (n = 23), ROS1 (n = 6), KRAS (n = 16), BRAF (n = 5) and MET (n = 2). No patients were found to have HER2 or RET mutations.

Figure 2.

CONSORT diagram. DS, discharge summary; ECL, external consultation letter; GIST, gastrointestinal stromal tumor; LA, locally advanced; NET, neuroendocrine tumor; NSCLC, non-small-cell lung cancer.

Table 1.

Patients characteristics

	n (%) or median [IQR]
Age median (years)	66 [57-72]
Sex
Male	38 (33.9)
Female	74 (66.1)
Real-world follow-up duration (months)	20.8 [10.5-41.4]
Total patients with a NSCLC treated with an oral targeted therapy	112
Document types	1028
ECLs	891
DSs	137
Histology
Adenocarcinoma	109 (97.3)
Squamous-cell carcinoma	2 (1.8)
Sarcomatoid carcinoma	1 (0.9)
Oncogenic driver alteration
BRAF	5 (4.5)
EGFR	66 (58.9)
KRAS	16 (14.3)
ALK	22 (19.6)
ROS1	6 (5.4)
MET	2 (1.8)

DS, discharge summary; ECL, external consultation letter; NSCLC, non-small-cell lung cancer.

Algorithm performance

Regarding extraction of temporally anchored tumor evolution events, including mentions of progression, stability, or response, the system achieved an F1-score of 79.7% for Concept recognition alone, and 62.0% for both Concept and Temporality combined. For tumor diagnosis, the F1-score reached 94.0% for Concept detection and 80.0% for Concept + Temporality. Extraction of molecular alterations, including gene names and variants, achieved an F1-score of 94.0% for Concept identification alone. For ECOG performance status, the system reached an F1-score of 100% for Concept detection and 79.3% for Concept + Temporality. When evaluating the detection of NSCLC-specific treatments, such as oral targeted therapies and chemotherapies, the system achieved an F1-score of 91.3% for Concept and 69.8% for Concept + Temporality.

For metastatic sites, the system reached an F1-score of 83.5% for Concept and 67.1% when Temporality was considered. Finally, when considering all extracted Facts across all domains, the system achieved an overall F1-score of 76.5% for Concept alone, and 63.7% for Concept + Temporality. Detailed performance results per category for Concepts alone and combined with their Temporalities are shown in Figure 3.

Figure 3.

F1-scores per Fact at the document level for all facts of interest in percentage for Concepts alone and Concepts with their Temporalities. ECOG, Eastern Cooperative Oncology Group. ^aCR, complete response; PD, progressive disease; PR, partial response; SD, stable disease; defined either following RECIST 1.1 criteria or declared by physician.

When comparing the time to completion using a sample of 40 random patients, reviewing the 48 eCRF items from automatically collected data was 5.8 times faster compared with manual data collection. The mean completion times were 3.2 minutes (0.8-11.5) for automated review and 18.5 minutes (5.3-30.7) for manual entry (P < 0.001).

Real-world progression-free survival estimates

To evaluate the solution’s ability to infer PFS, rwPFS was assessed using a Kaplan–Meier model from targeted therapy initiation to progression or death in EGFR-mutated, ALK-translocated, and KRAS-mutant NSCLC, where survival benchmarks are well-established. The median real-world follow-up was 21.03 months for the whole cohort. The rwPFS for patients with EGFR-mutated (n = 66), ALK-translocated (n = 23), and KRAS-mutant tumors (n = 14) treated with the corresponding targeted therapies (anti-EGFR, anti-ALK, and anti-KRAS) were 21.9 months (CI 95% 14.6-29.6 months), 52.4 months (CI 95% 12.8-143 months), and 5.0 months (CI 95% 2.6-16.2 months), respectively (Figure 4). These survival estimates compare well with the recently published survival benchmarks for these molecularly-defined NSCLC cohorts: median PFS in EGFR-mutated NSCLC was 18.9 months^9,10; median PFS in ALK-translocated NSCLC was 34.8 months^11,12; median PFS in KRAS-mutant NSCLC was 5.0-6.5 months.13, 14, 15

Figure 4.

Real-world progression-free survival summaries. (A) PFS in patients with EGFR-driver mutation treated with anti-EGFR targeted therapy. (B) PFS in patients with ALK-driver translocation treated with anti-ALK targeted therapy. (C) PFS in patients with KRAS-driver mutation treated with anti-KRAS targeted therapy. CI, confidence interval; KM, Kaplan–Meyer; NSCLC, non-small-cell lung cancer; PFS, progression-free survival; rw, real-world.

Discussion

This study demonstrates the strong performance and practical utility of a domain-adapted NLP solution for extracting temporally anchored tumor evolutions from unstructured EMRs in thoracic oncology. Tumor evolution, defined through narrative descriptions of progression, response, or stability was the primary focus, given its central role in deriving endpoints such as rwPFS. The system achieved an F1-score of 79.7% for tumor evolution concept extraction and 62.0% for temporality alignment, representing one of the highest levels of performance described to date for this specific task, at a document level.

These findings compare favorably with previously published approaches for extracting oncologic outcomes from EMRs. Most existing solution frameworks rely either on structured data abstraction or clinician-reported summaries, with limited ability to systematically capture unstructured events or temporal dynamics. For instance, the multicenter study by McKelvey et al. assessed real-world tumor response using clinician-entered data from seven EMR-derived databases.¹⁶ Although they demonstrated reasonable consistency in response rates across vendors, the timing of response and progression events was not standardized and showed substantial variability. Notably, their method did not include automated extraction from unstructured text, and no formal metrics (e.g. precision, recall, or F1-score) were reported for temporality alignment. Similarly, the RUBY NLP platform, applied to breast cancer EMRs, reported F1-scores of 34.4%-100% across extracted concepts, but temporality and longitudinal outcome tracking were not evaluated.¹⁷ In contrast, our solution achieved an F1-score of 79.7% for tumor evolution structuration and 62.0% when combined with temporality, representing an interesting and clinically relevant performance for automated, temporally anchored extraction of tumor evolutions in oncology. Some studies that have evaluated temporal extraction from clinical narrative reported similar F1-scores, but did not focus on tumor evolution and did not involve this level of temporal details.5, 6, 7 Furthermore, unlike general-purpose LLM approaches or structured data pipelines, our system outputs are fully traceable, interpretable, and aligned with OMOP standards, supporting downstream use in research databases and regulatory-grade analytics. The rwPFS observed in this study aligns with outcomes reported in both clinical trials and real-world settings for EGFR-mutated, ALK-translocated, and KRAS-mutant NSCLC. Results are consistent with pivotal studies such as FLAURA, ALEX, and trials investigating KRAS G12C inhibitors, taking account of the large confidence intervals, supporting the reliability of real-world evidence in reflecting clinical trial outcomes and reinforcing the established benefit of targeted therapies across molecular subtypes of NSCLC.^9,11,13, 14, 15

Despite the solution’s promising performance, several limitations should be considered. Firstly, curated annotations are subject to inter-annotator variability, a known constraint in NLP evaluation, particularly for temporality, which may be interpreted differently based on clinical context. Secondly, the annotation interface, though designed for clinical usability, may introduce ‘lazy-annotator bias,’ especially for lower-impact concepts. Thirdly, performance was assessed at the document level, meaning that a missed fact in one document is counted as a false negative, even if it could be retrieved from another document in the same patient record. As with any automated extraction framework, this likely underestimates the system’s true performance at the patient level.

Technical challenges also emerged, including ambiguity introduced by copy-pasted text, conditional language, and misinterpretation of metadata sections (e.g. headers, footers). Temporality inference remains particularly complex. Although the system uses verb tense analysis and section tagging to classify time anchors, present tense is often used for past or future events, creating interpretative ambiguity. Further improvements may include tense-aware deep language models or document-type specific tuning.

Finally, generalizability remains to be proven across centers and disease types. The current evaluation was conducted on data from a single academic hospital with robust documentation practices. Broader deployment should include validation in community-based or regional care settings, where EMR completeness and linguistic patterns may differ significantly.

Conclusion

This study presents a robust, domain-specific NLP pipeline capable of extracting temporally structured oncology outcomes from unstructured EMRs. Performance on tumor evolution detection, the primary objective, exceeded previous benchmarks, and the system demonstrated strong potential for supporting real-world endpoint reconstruction, including rwPFS. By overcoming major limitations of structured-only or manual abstraction approaches, particularly in terms of temporality, interpretability, and scalability, this solution offers a valuable foundation for research-oriented and clinical applications in oncology. Further validation in diverse care settings and cancer types is warranted, as is continued exploration of integration with regulatory-grade real-world evidence frameworks.