Development and Portability of a Text Mining Algorithm for Capturing Disease Progression in Electronic Health Records of Patients With Stage IV Non–Small Cell Lung Cancer

Abstract

PURPOSE

The objective was to develop and evaluate the portability of a text mining algorithm for prospectively capturing disease progression in electronic health record (EHR) data of patients with metastatic non–small cell lung cancer (mNSCLC) treated with immunochemotherapy.

METHODS

This study used EHR data from patients with mNSCLC receiving immunochemotherapy (between October 1, 2018, and December 31, 2022) in four Dutch hospitals. A text mining algorithm for capturing disease progression was developed in hospitals 1 and 2 and then transferred to hospitals 3 and 4 to evaluate portability. Performance metrics were calculated by comparing its outcomes with manual chart review. In addition, data were simulated to come available over time to assess performance in real-time applications. Median progression-free survival (PFS) was calculated using the Kaplan-Meier method to compare text mining with manual chart review.

RESULTS

During development and portability, the text mining algorithm performed well in capturing disease progression, with all performance scores >90%. When real-time performance was simulated, the performance scores in all four hospitals exceeded 90% from week 15 after the start of follow-up. Although the exact progression dates varied in 46 patients of 157 patients with progressive disease, the number of patients labeled with progression too early (n = 24) and too late (n = 22) was well balanced with discrepancies ranging from –116 to 384 days. Nevertheless, the PFS curves constructed with text mining and manual chart review were highly similar for each hospital.

CONCLUSION

In this study, an accurate text mining algorithm for capturing disease progression in the EHR data of patients with mNSCLC was developed. The algorithm was portable across different hospitals, and the performance over time was good, making this an interesting approach for prospective follow-up of multicenter cohorts.

The text mining algorithm for capturing disease progression was accurate and portable across different hospitals.

INTRODUCTION

Real-world effectiveness studies of anticancer treatments commonly use progression-free survival (PFS) as an end point.^1,2 Most of these studies define PFS as the time from treatment initiation to the first documentation of disease progression or death, whichever comes first.³ In clinical practice, disease progression is evaluated through a variety of assessments, such as imaging, physical examination, and pathological outcomes, which findings are primarily documented by the thoracic oncologist and radiologist in the unstructured text fields within the electronic health records (EHRs). This multifactorial evaluation leads to a variety of terminology used to express disease progression (eg, tumor growth, disease worsening, or clinical deterioration). The lack of structured data requires time-consuming manual chart review of multiple notes for capturing disease progression end points. According to Griffith et al,⁴ the median time required to extract progression events from the EHR for one patient was 18 minutes. Therefore, manual review is not suitable for remote real-time detection of progression events in large cohorts under follow-up as it would require repeated EHR reviews per patient over time, which is even more time-consuming. Remote multicenter real-time evaluations of time to disease progression in real-world settings, however, could bring relevant early warning signals for possible efficacy-effectiveness gaps of novel oncology medicines. However, alternative strategies for real-time follow-up studies are needed for efficient and reliable capturing progression end points in EHRs.

CONTEXT

• Key Objective

• To develop and evaluate the portability of a text mining algorithm for prospectively capturing disease progression in electronic health record data.

• Knowledge Generated

• The text mining algorithm performed well in all hospitals, with all performance scores higher than 90%. The performance over time was also good, with scores above 90% from week 15 after the start of follow-up.

• Relevance

• The authors developed a text mining algorithm to detect disease progression in electronic health data—this is in itself very useful; the authors went one step further and demonstrated that the algorithm also worked in other hospitals, proving portability.

The challenges in collecting data from unstructured text fields have driven the exploration of alternative methods to identify and extract information. Text mining and natural language processing (NLP) techniques have been used to transform free text into structured data, enabling more efficient detection of data items in EHRs.^5,6 Cheng et al⁷ evaluated the performance of their custom-made NLP model for detecting tumor progression in neuroradiology reports of patients with brain tumors, showing promising results with 87.3% sensitivity, 93.1% specificity, 84.9% positive predictive value, and 94.3% negative predictive value. With the increasing success of custom-built NLP models, commercially available NLP is now used more frequently to extract information from EHR text fields. van Laar et al⁸ evaluated the performance of commercial NLP software in capturing patient characteristics and outcomes, including disease progression, in patients with metastatic renal cell carcinoma. The algorithm was able to capture disease progression, but the median PFS (mPFS) on the basis of its output was 17% longer compared with manually retrieved dates of progression (8.9 v 7.6 months), indicating challenges in accurately detecting disease progression events. Furthermore, the study was conducted in a single-center setting, preventing conclusions about portability of their algorithm toward different EHR systems and clinical settings. Moreover, the algorithm's performance was evaluated using historical data, leaving uncertainty about the performance of prospective use for detecting progression events.

The objective of this study was to develop and evaluate the portability of a text mining algorithm for prospectively capturing disease progression in EHRs of patients with metastatic non–small cell lung cancer (mNSCLC) and treated with immunochemotherapy in multicenter cohorts.

METHODS

Study Design, Population, and Setting

The performance of a text mining algorithm in capturing disease progression was assessed by comparing its results against manual chart review in a population of patients with mNSCLC treated with first-line pembrolizumab plus chemotherapy (immunochemotherapy). This population was chosen because the survival outcomes of mNSCLC are relatively short, and the treatment landscape is rapidly evolving, making it relevant to accelerate real-world evidence studies. Data were used from patients who received pembrolizumab plus chemotherapy between October 1, 2018, and December 31, 2022, in one of the following hospitals: Catharina Hospital Eindhoven (CZE; hospital 1), OLVG Hospital Amsterdam (OLVG; hospital 2), Leiden University Medical Centre (LUMC; hospital 3), and Haga Hospital Den Haag (HAGA; hospital 4). In hospitals 1 and 2 the Santeon Farmadatabase⁹ and in hospitals 3 and 4 the Clinical Data Collector (CTcue) were used to identify eligible patients. For this study, the workflow was built up in three steps. In the first step (the development) a text mining algorithm was developed in hospital 1 and then transferred to hospital 2 for optimization. In the second step (the portability), the algorithm was transferred to hospitals 3 and 4 for external evaluation. Finally, in step three (the real-time simulation) data were made available at incremental time points, and the algorithm was evaluated at each point. There are no standards for reporting text mining studies, so a modified version of the Standards for Reporting Diagnostic Accuracy Studies guidelines was used.¹⁰

Ethics Approval

In hospitals 1 and 2, approval for patient data was obtained by the Santeon Institutional Review Board (SDB 2019-008). In hospitals 3 and 4, approval was received by the local Medical Ethics Review Committees Leiden Den Haag Delft (P21-031). The need for informed consent was waived because of the retrospective nature of the study, and most patients were dead at the time of conducting the study.

Test Methods

Manual Chart Review (reference method)

As the reference method, manual chart review was applied. Reviewer 1 (M.V.V.) conducted the review in hospitals 1 and 2 while reviewer 2 (H.A.K.) performed the review in hospitals 3 and 4. Both reviewers had expertise in the treatment of patients with NSCLC. Manual chart review was conducted before text mining, with a minimum interval of 1 week. In addition, the start date of pembrolizumab, date of death, and date of last clinic visit were retrieved with manual chart review.

Text Mining (index test method)

As an index test method, the CTcue text mining tool (IQVIA Patient Finder Solution-CTcue B.V., Amsterdam, the Netherlands) was used. This Dutch language rule-based tool was used to search and extract data from structured and unstructured EHR data, excluding PDF files. The tool uses NLP techniques to transform unstructured text into structured data. Data features that match the criteria of a query are subsequently presented in a graphical interface, presenting the identified text item with 150 characters before and after it by default. Subsequently, these identified data features require manual verification to ensure correctness. The underlying architecture of the tool is described in detail by van Laar et al.⁸

Development

Data Source and Retrieval

For each patient, data were obtained from the EHR. Hospital 2 used EPIC software, and hospitals 1, 3, and 4 used Chipsoft Hix software as EHR systems. Information on disease progression was retrieved from all unstructured notes, such as medical letters and clinical or radiology notes. The period for retrieving information extended from the initial administration of pembrolizumab to the date of the last clinic visit for each patient.

Variables of Interest

The variables of interest in this study were disease progression and the corresponding date of progression. To minimize potential bias due to inconsistent criteria for disease progression, a decision tree was developed to classify the presence and the date of disease progression (Data Supplement, Fig S1). In patients with pseudoprogression, disease progression was only concluded when it was confirmed and reported in subsequent notes.

Text Mining Algorithm

First, an initial text mining algorithm was developed in hospital 1 by one researcher (M.V.V.) using clinical and radiological notes from 10 randomly selected patients of a total eligible population of 85 patients. Keywords related (eg, disease progression, tumor progression, or tumor recurrence) and unrelated to progression (eg, no progression, no signs of progression, or no recurrence) were identified through manual review. Additional words were formulated on the basis of spelling errors, jargon, and literature terms. The list of included and excluded keywords is provided in the Data Supplement (Table S1a). The performance of the text mining algorithm was evaluated in hospital 1 and afterward transferred to hospital 2 for optimization. To ensure adaptability to different clinical settings, the same researcher (M.V.V.) randomly selected 10 patients from the eligible population (N = 101) of hospital 2, and their clinical and radiological notes were also read entirely to identify new progression-related words, see the Data Supplement (Table S1b), to update the text mining algorithm. The performance of this updated algorithm was evaluated in hospital 2. Finally, the final version of the algorithm was developed by analyzing the misclassifications of hospitals 1 and 2, which were used as input to update the list of (key)words (Data Supplement, Table S1c).

Portability

The performance of the final version of the text mining algorithm was evaluated on a total of 50 patients for each hospital (3 and 4) to gain insight into the portability. A random selection of patients was made from the eligible population using the Clinical Data Collector CTcue, where patients are shuffled and pseudonymously assigned, ensuring a random sample. To assess the overall portability, the performance of both hospitals 3 and 4 was evaluated after 9 months of follow-up.

Real-Time Simulation

Through simulation of incremental data availability at weekly intervals, the algorithm's performance was assessed as if it were applied in real time. The reference point for each patient was the start date of pembrolizumab (t = 0), with EHR data made incrementally available with weekly intervals until the end of follow-up, that is, either the date of last clinic visit or death. The performance of the text mining algorithm for capturing disease progression was assessed for each interval (week 1; week 1 + 2; week 1 + 2 + 3; etc).

Analysis

The text mining results for disease progression were compared with manual chart review and classified as false positive (FP), false negative (FN), true positive (TP), and true negative. The performance of the text mining algorithm was evaluated using standard performance metrics (recall/sensitivity, precision/positive predictive value, F1 score, negative predictive value, and specificity); the details are in the Data Supplement (Tables S2 and S3).¹¹ Additionally, for TPs, a bar chart was constructed for the number of days between the date of disease progression extracted with text mining and manual chart review. PFS was calculated from the start of treatment until progression or death, whichever occurred first. Patients without an event were censored on the date of their last clinic visit. The Kaplan-Meier method was used to visualize the PFS curve, estimate the mPFS, and calculate the hazard ratio between the results captured using text mining and manual chart review. All analyses were performed using R software version 1.4.2.

RESULTS

Development

During development, the algorithm was applied on 75 patients in hospital 1 and 91 patients in hospital 2. In hospital 1, manual chart review identified 35 patients with and 40 patients without disease progression. Of these patients, the algorithm correctly captured 34 positives and 37 negatives, with one FN and three FPs (Fig 1A). In hospital 2, manual chart review identified 48 patients with and 43 without disease progression. Of these patients, the updated algorithm correctly captured 46 positives and 41 negatives, with two FNs and two FPs (Fig 1B). In both hospitals, the text mining algorithms scored high on all performance metrics, with all scores above 92% (Table 1).

FIG 1.

Confusion matrix for the performance of the text mining queries during (A) and (B) development and (C) and (D) portability. FN, false negative; FP, false positive; TP, true positive; TN, true negative.

TABLE 1.

Overall Performance During Development and Portability

Performance Measure	Development, %		Portability, %
	Hospital 1 (n = 75)	Hospital 2 (n = 91)	Hospital 3 (n = 50)	Hospital 4 (n = 50)
Recall/sensitivity	97	96	98	100
Precision/PPV	92	96	100	100
F1 score	94	96	99	100
NPV	97	95	91	100
Specificity	93	95	100	100

Abbreviations: NPV, negative predictive value; PPV, positive predictive value.

For patients correctly captured with progression by text mining the number of days between disease progression dates identified with manual chart review and text mining are presented in Figures 2A and 2B. In hospital 1, 29% of patients (10/34) had date discrepancies (range, –89 to 25 days), with six patients being too late and four too early. In hospital 2, 33% of patients (15/46) had discrepancies (range, –116 to 384 days), with seven too late and eight too early. The reasons for date disagreements (for those exceeding 14 days) are outlined for each patient in the Data Supplement (Table S4). Additionally, in hospital 1 (Fig 3A) and hospital 2 (Fig 3B), no difference in the mPFS between manual chart review and text mining was observed.

FIG 2.

Horizontal bar chart presenting the number of days between the manually extracted date and text mining date of progression for each patient correctly captured with progression during development in hospitals 1 (A) and 2 (B), and during portability in hospitals 3 (C) and 4 (D). TP, true positive.

FIG 3.

Kaplan-Meier curves of PFS for data extraction manual versus text mining during development in (A) hospitals 1 and (B) 2 and portability in (C) hospitals 3 and (D) 4. HR, hazard ratio; mPFS, median PFS; PFS, progression-free survival.

Portability

The final version of the algorithm was transferred to hospital 3 (N = 50) and hospital 4 (N = 50) to evaluate its portability, and the results are presented in Figures 1C and 1D, respectively. In hospital 3, manual chart review identified 40 patients with and 10 without progression. Of these patients, the algorithm correctly captured 39 positives and 10 negatives, with one FN identification. In hospital 4, manual chart review identified 38 patients with and 12 without progression. The algorithm correctly captured all these patients. The performance metrics in both hospitals were high ranging from 91% to 100% (Table 1).

In hospital 3, 23% (9/39) had date discrepancies (range, –6 to 77 days), with eight patients being too late and one being too early (Fig 2C). In hospital 4, 32% (12/38) had date discrepancies (range, –11 to 17 days), with one too late and 11 too early (Fig 2D). The reasons for date disagreements (>14 days) are outlined for each patient in the Data Supplement (Table S4). In hospital 3 (Fig 3C), the mPFS estimated with manual chart review was shorter than the mPFS estimated with text mining (5.3 v 6.4 months), with no difference in the PFS curves. In hospital 4 (Fig 3D), there was no difference between the mPFS and PFS curves estimated with manual chart review and text mining (11 v 11 months).

In summary, if the algorithm had been used to patients with mNSCLC treated with first-line immunochemotherapy in hospitals 3 and 4 and evaluated at 9 months of follow-up, it would have only missed one patient with progression (FN) with no difference between the mPFS estimated with manual chart review and text mining (6.7 v 6.7 months).

Real-Time Simulation

To simulate real-time application, data were provided incrementally every week (Data Supplement, Table S5). In hospital 1, one FN occurred at week 12 and three FPs occurred at weeks 39, 71, and 74. In hospital 2, two FNs occurred at weeks 7 and 32, and two FPs occurred at week 15. Hospital 3 had only one FN at week 5, and hospital 4 had no incorrect identifications. Performance metrics for these simulations at each time interval are presented in the Data Supplement (Fig S2).

DISCUSSION

This study showed that our text mining algorithm is feasible for capturing disease progression in the EHR of patients with mNSCLC treated with immunochemotherapy. The algorithm performed well in all hospitals, with all performance scores higher than 91%. The algorithm also performed well over time when repeatedly applied to newly available EHR data at weekly intervals. Although the exact progression dates varied in approximately 30% of patients, the number of patients labeled with progression too early (n = 24) and too late (n = 22) was well balanced with discrepancies ranging from –116 to 384 days. Nevertheless, the PFS curves constructed with text mining and manual chart review were highly similar for each hospital. These outcomes indicate that this algorithm can be used to investigate the real-world effectiveness of cancer treatments.

This study is the first to develop and evaluate the performance of a text mining algorithm for capturing disease progression in mNSCLC in multiple clinical settings. Previous studies on text mining models for disease progression were either in different malignancies or single-center studies. For example, Cheng et al⁷ developed a text mining algorithm for patients with brain tumor with comparable performance scores (all scores >87%) to our text mining algorithm. Ping et al¹¹ captured recurrent disease in patients with hepatocellular carcinoma with a comparable F1 score of 84%. Recently, Lee et al¹² developed an algorithm called the image-based-rule, which uses text mining to identify disease recurrence in radiological reports of patients with ovarian cancer. The performance of their algorithm was good, according to their reported numbers, and an F1 score of 81% could be calculated.¹² Furthermore, our models' overall performance was similar to the performance of van Laar et al,⁸ who evaluated the same commercial NLP tool (CTcue) for disease progression in renal cell carcinoma (F1 score 94.5%). However, our algorithm was more accurate in estimating PFS curves with no difference from manual review, whereas the PFS curves estimated by van Laar et al⁸ were less identical, with a mPFS difference of 17% between text mining and manual chart review. This difference may be due to our predefined decision tree for estimating progression, in contrast to their nonstandardized approach.

This study also evaluated the portability of the algorithm by testing its performance in multiple clinical settings. Overall, the algorithm performed well in different settings, with all performance scores above 90%. However, in 28% of all patients with progressive disease, discrepancies were observed between the calendar dates for progression. We observed that specific terms, such as growth of lesions or new nodular metastasis, were used to indicate progression, which underscores that health care providers may use specific terminology to indicate progression. One solution to address this issue and improve generalizability is to manually review 10 patients with and without progression to capture additional keywords before implementing the algorithm in a new setting and periodically after 1 year of deployment. Additionally, selecting patients with apparently short or long mPFS for manual review of their files can help users of the algorithm to identify potential FPs and FNs. Another explanation for the date discrepancies could be the lack of standardized response criteria, leading to varying interpretations of disease progression. This aligns with the findings of a previous study, demonstrating that relying only on standard response criteria to abstract information on cancer progression from EHRs is not possible as these criteria were not used in 75% of the cases.¹³ Discrete reporting of progression in standardized care pathways could overcome these disagreements.

Another relevant finding is the good performance of the algorithm over time, indicating its potential for prospective follow-up of patients to capture disease progression. Currently, prospective patient follow-up involves repetitive manual chart review of newly available EHR data. However, this process is time-consuming.⁴ Our algorithm could overcome this limitation by constructing an automated process for capturing progression events in multiple clinical settings parallel. The algorithm can run continuously, enabling real-time capture of progression and facilitating real-time evaluation of PFS. This real-time evaluation allows for incremental and timely comparison with the PFS reported in clinical trials to assess potential efficacy-effectiveness gaps, which could provide an early signal to health care providers regarding of the relative effectiveness of specific treatments in clinical practice, enabling them to make informed decisions on treatment strategies.¹⁴ However, sequentially evaluating this gap would require sophisticated statistical methodology to adjust for multiple testing to prevent FP conclusions, and the completeness and accuracy of mortality data should be validated for the calculation of PFS.^15,16 In the Netherlands, this validation is usually achieved by crosschecking it with the mortality data from the Personal Records Database.

The study's strengths include its multicenter approach for developing and evaluating a text mining algorithm and assessing its performance over time. However, certain limitations should be addressed. First, the use of a commercial text mining tool (CTcue) lacks transparency in the used NLP methods, as the underlying algorithms and processing pipelines are not openly accessible. This limits the generalizability to Dutch-language hospitals using this commercial software. However, the identified keywords could also be used by noncommercial NLP models. Second, the data were not subject to a manual chart review in duplicate to ensure its quality. Nevertheless, a decision tree (Data Supplement, Table S1) was used to minimize inconsistencies in the data, and in case of any ambiguities, a thoracic oncologist was consulted. Third, the mPFS observed in hospital 4 was around two times longer than the mPFS observed in the other hospitals, with no difference between manual chart review and text mining (11 v 11 months). As this study focused on the performance of our text mining algorithm, a detailed analysis of factors contributing to this prolonged mPFS was not conducted. Factors that could contribute to PFS outcomes in patients with mNSCLC treated with immunochemotherapy in clinical practice are described in more detail in our previous EE gap analysis.¹⁷ Finally, we only evaluated the performance of our algorithm in patients with mNSCLC treated with pembrolizumab plus chemotherapy in the Netherlands. This limits the generalizability of our findings to other patient populations, treatment regimens, and geographical locations.

In conclusion, in this study, an accurate text mining algorithm for capturing disease progression in EHR data of patients with mNSCLC was developed. The algorithm was portable across different hospitals, and the performance over time was good, making this an interesting approach for prospective follow-up of multicenter cohorts.

AUTHOR CONTRIBUTIONS

Conception and design: M.V. Verschueren, M. Deenen, L.T. Bloem, B.J.M. Peters, E.M.W. van de Garde

Provision of study materials or patients: M. Deenen, B.E.E.M. van den Borne, J. Zwaveling, L.E. Visser

Collection and assembly of data: H. Abedian Kalkhoran, B.E.E.M. van den Borne, E.M.W. van de Garde

Data analysis and interpretation: All authors

Manuscript writing: All authors

Final approval of manuscript: All authors

Accountable for all aspects of the work: All authors

AUTHORS' DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

The following represents disclosure information provided by authors of this manuscript. All relationships are considered compensated unless otherwise noted. Relationships are self-held unless noted. I = Immediate Family Member, Inst = My Institution. Relationships may not relate to the subject matter of this manuscript. For more information about ASCO's conflict of interest policy, please refer to www.asco.org/rwc or ascopubs.org/cci/author-center.

Open Payments is a public database containing information reported by companies about payments made to US-licensed physicians (Open Payments).