Automatic Detection of Distant Metastasis Mentions in Radiology Reports in Spanish

Abstract

PURPOSE

A critical task in oncology is extracting information related to cancer metastasis from electronic health records. Metastasis-related information is crucial for planning treatment, evaluating patient prognoses, and cancer research. However, the unstructured way in which findings of distant metastasis are often written in radiology reports makes it difficult to extract information automatically. The main aim of this study was to extract distant metastasis findings from free-text imaging and nuclear medicine reports to classify the patient status according to the presence or absence of distant metastasis.

MATERIALS AND METHODS

We created a distant metastasis annotated corpus using positron emission tomography-computed tomography and computed tomography reports of patients with prostate, colorectal, and breast cancers. Entities were labeled M1 or M0 according to affirmative or negative metastasis descriptions. We used a named entity recognition model on the basis of a bidirectional long short-term memory model and conditional random fields to identify entities. Mentions were subsequently used to classify whole reports into M1 or M0.

RESULTS

The model detected distant metastasis mentions with a weighted average F₁ score performance of 0.84. Whole reports were classified with an F₁ score of 0.92 for M0 documents and 0.90 for M1 documents.

CONCLUSION

These results show the usefulness of the model in detecting distant metastasis findings in three different types of cancer and the consequent classification of reports. The relevance of this study is to generate structured distant metastasis information from free-text imaging reports in Spanish. In addition, the manually annotated corpus, annotation guidelines, and code are freely released to the research community.

Detecting and classifying distant metastasis status in patients using radiology reports in Spanish.

INTRODUCTION

The latency and delay in starting treatment for patients with cancer in Chile have become a public health problem. More than 53,000 patients with cancer are waiting for treatment, keeping this pathology as the main cause of death on waiting lists.¹ Chile has a two-thirds deficiency of oncologists compared with Organization for Economic Cooperation and Development (OECD) countries. The average wait time for staging studies and oncology team re-evaluation varies between 30 and 60 days.

CONTEXT

• Key Objective

• To develop a named entity recognition model to detect and classify mentions and radiology reports in Spanish according to distant metastasis status in patients with breast, colorectal, and prostate cancers using positron emission tomography-computed tomography and computed tomography reports.

• Knowledge Generated

• A multicancer multireport model handles dependence of metastatic status on the primary tumor site and anatomical site of the secondary lesion, showing good performance in detecting distant metastasis mentions. Confounding from mixing cancer types and lesion locations showed that, in some cases, the approach of training models with one report type for a single cancer outperforms. A new corpus of manually annotated radiology reports contributes to closing the gap in language resources and systems for distant metastasis detection and classification in Spanish.

• Relevance

• Automated labeling of metastatic sites from free text is a critical step toward constructing a population-level understanding of metastatic patterns. This study demonstrates successful application of such a system to Spanish-language texts.

A feasible strategy is to strengthen the workforce for cancer and palliative care specialists while developing artificial intelligence models to assist clinicians in rapidly making decisions, ultimately expediting the prioritization of treatment pathways for patients, whether for curative or palliative purposes.¹ A key challenge is identifying high-need patient groups, like those with distant metastasis, in real time.

Detecting distant metastasis is essential in cancer management for clinical, epidemiological, and research purposes.² However, most of this information is found in radiology reports in free-text format,^3,4 which is rich in interpretive nuances and contextual variations,^5,6 making automatic information extraction highly challenging and complex.

Distant metastasis detection algorithms must face two additional challenges. First, the mentions of distant metastasis depend on the type of cancer and the anatomic site of the lesion. For instance, a secondary neoplastic lesion in the true pelvis is not metastatic for prostate cancer (PCa) but is metastatic for breast cancer (BCa) or colon cancer. Second, the technicality used in the descriptions may vary depending on the type of report analyzed (eg, computed tomography [CT], positron emission tomography with computed tomography [PET-CT]). Therefore, unless a substantial, diverse data set is available, it could make more sense to develop specialized systems to detect information using only one organ study or one type of report^7,8.

Natural language processing (NLP) can help address the challenges and difficulties of clinical text. NLP is a computer science and artificial intelligence subfield that aims to understand and produce human language.⁹ The main applications for clinical NLP are text classification, information retrieval, and named entity recognition (NER).⁴ NLP work on metastasis detection in imaging and radiology reports usually uses classification or NER methods and, to a lesser extent, both approaches. Most studies have focused primarily on reports in English^10-22 and more recently in Chinese.^23-25 These systems aimed to determine metastasis or recurrence status and other information, using reports of BCa,^12-15 lung cancer,^22,23 hepatocellular carcinoma,^14,24 PCa,^18,20 melanoma,¹¹ or colorectal cancer (CRC).²¹ Five studies did not select reports of patients affected with a cancer type but used single or multiple organ imaging reports of patients with different types of primary tumors to identify metastases and performed mostly binary classifications such as the presence of single versus multiple metastases or the presence versus absence of metastases across different organs.^{7,16,17,19,25}

Regarding the report type, the main approach was the use of notes and texts of diverse nature and specialty (discharge summaries, progress notes, radiology and pathology reports, etc) extracted from electronic health records^{13-15,18,24,25} and sometimes complemented with structured information.^11,12 Other studies used only one type of radiology report, such as bone scintigraphy,¹⁶ magnetic resonance imaging (MRI),¹⁷ CT,^7,19,21,23 PET-CT²² reports, or a combination of these.²⁰

By contrast, the development of systems using imaging reports in Spanish is scarce. Some work has focused on information extraction,²⁶ classification,^27,28 and anonymization²⁹; however, to the best of our knowledge, no work has focused on the detection of distant metastases. Pabón et al³⁰ extracted metastases mentions of patients with lung cancer in Spanish but were limited to the explicit TNM notation (eg, pT3N2M1), a different task addressed mainly using pathology reports.^31-34 Recently, Martín-Noguerol et al³⁵ developed a deep learning (DL)–based NLP model to support radiologists in differentiating high-grade glioma and brain metastases on the basis of MRI reports. One limitation was that probably to take advantage of the greater availability of state-of-the-art tools for English, the reports that were originally in Spanish were translated into English, which may have affected the performance of their model.

In this study, we contribute to the gap in linguistic resources and systems for distant metastasis detection in Spanish by developing a corpus of manually annotated radiology reports, together with a set of NER models to detect and classify mentions and radiology reports according to metastasis status.

Our study is different from previous work by considering three different cancer types (PCa, CRC, and BCa) and two different types of reports (PET-CT and CT) and by analyzing how a contextualized model handles the dependence of the metastatic status on the tumor primary site and the anatomical site of the secondary lesion in the identification of mentions.

MATERIALS AND METHODS

This study was conducted at the Oncology Institute Fundación Arturo López Pérez (FALP), a nonprofit oncology reference center,³⁶ under protocol number 2021-017-ITC-SIN-OTH approved by the Institutional Review Board.

Data Set

A total of 670 radiology reports of patients with PCa, CRC, and BCa cancers were collected from FALP. These cancer types were chosen because they had the highest incidence and prevalence in Chile in 2020.³⁷

Only patient reports that met the criteria for entry into the clinical pathway for each cancer were sampled (more details in the Data Supplement). PET-CT and CT reports were selected on the basis of the wide use of CT for metastasis detection and the higher specificity and sensitivity of PET-CT, according to the type of cancer and metastasis.^38-41 Data Supplement (Table S1) shows the distribution of reports according to cancer and report types. We will refer to this data set as the FALP radiology report (FALP-RR) corpus.

Corpus Annotation

The procedure and annotation scheme were established after a preanalysis of the PET-CT and CT reports and a detailed analysis of the mentions of distant metastases (full description provided in the Data Supplement).

The annotation team consisted of two general nursing technician, (GNT) experts in oncology records. A master of medical informatics worked as annotation manager, two radiologists as medical advisors, and an NLP scientist as methodology advisor. The GNT performed the manual annotation of the corpus using INCEpTION.⁴² They followed annotation guidelines to obtain quality and consistent annotations. The reviewed and final version of the annotation guidelines is shared with this publication.⁴³

The annotation scheme included the following two entities: M0 for negated distant metastatic lesion mentions and M1 for affirmative mentions (Data Supplement, Fig S1). The annotation quality was assessed by annotating 20% of the reports by both annotators, and the agreement between them was measured using the Strict and Relaxed F₁ score metrics.⁴⁴ The F₁ measure is preferred over others, such as Kappa statistic, when it is required to evaluate annotations with different token spans.^45,46 In the strict variant, only annotations that exactly match both the entity class and the selected tokens are considered correct. In the relaxed measure, cases where the annotations have the same class but have a partial match in entity length, with an overlap of tokens, are also considered correct.

The manager and methodology advisor reviewed and corrected each annotation to normalize the corpus (more details are in the Data Supplement). The FALP-RR corpus is freely available for noncommercial use.⁴⁷

DL Model

M0 and M1 mentions were recognized using the Multiple LSTM-CRF (MLC) architecture,⁴⁸ an adaptation of the model proposed by Lample et al⁴⁹ to solve the nested NER task.⁵⁰ Specifically, they used a Bi-LSTM and CRF layers to recognize each entity type and incorporated pretrained embeddings trained on the Chilean Waiting List corpus^44,51 and character-level contextualized embeddings.⁵² The code is freely available.⁵³

The data set was divided into training, validation, and testing subsets on the basis of 80:10:10 proportions. The training followed previously proposed settings⁴⁴ with the hyperparameters listed in the Data Supplement (Table S2). The model was trained for M1 and M0 labels at the same run.

A 10-fold cross-validation was performed to provide a more reliable estimate of model performance. The test subset contained 67 documents used to evaluate document-level classification on the basis of NER predictions. Performance was evaluated using precision, recall, and F₁ metrics.

To explore the effect on the performance of distant metastasis mentions recognition by including different types of radiology reports and various cancer types, 12 models for all possible combinations by cancer and report type were trained and evaluated, following the same parameters previously described.

The models for each cancer trained with the two report types (CT + PET-CT) were used in a cross-data subset evaluation to analyze how a contextualized model handles the dependence of the metastatic status on the primary tumor site and the anatomical site of the secondary lesion in the identification of the mentions.

Error Analysis

An error analysis⁵⁴ was performed to determine five types of errors that can occur in a NER task (Data Supplement, Fig S2).

Document Classification

A simple decision algorithm was implemented to assign a document-level class to each report in the test data set used for the NER task. The decision was taken on the basis of the presence on each report of at least one M1 label (M1 class) or absence of M1 labels (M0 class). To evaluate the automatic document-level classification method, annotators classified reports in the same way during the annotation process.

RESULTS

Interannotator Agreement

The interannotator agreement (IAA) evaluated in strict and relaxed F₁ score was 0.97 and 0.98 for M0, respectively, and 0.92 and 0.95 for M1, respectively. The strict and relaxed metrics analyzed by type of report were higher than 0.9, with less agreement on the labeling of PET-CT reports (Data Supplement, Table S3). The metric increased as the annotation rounds progressed (Data Supplement, Fig S3).

Annotated Corpus Composition

PET-CT reports, with an average of 546 tokens per document and almost 13 mentions per report, contributed 58.9% of the total annotations. An imbalance of annotated labels stands out, with an M0 proportion of 75.4% in PET-CT and 80.5% in CT (Table 1).

TABLE 1.

Number and Proportion of Annotations and Tokens by Type of Report

MLC Model

Entity recognition improved by including character-level and word-level embeddings versus a single representation. Using these settings, the entity recognition achieved a performance of 0.88 in M0 and 0.71 in M1 (Table 2). In the 10-fold cross-validation, the M0 models outperformed the M1 by almost 20%. The weighted average F₁ score between M0 and M1 was 0.83 (Data Supplement, Table S4).

TABLE 2.

NER Model Results for M0 and M1 Entities Using CT and PET-CT Reports for All Cancer Types

In addition to the previous multicancer multireport model, we used specific subsets of reports by cancer type to develop specific models. Six models with different report/cancer combinations, four of them using only CT reports, showed better performances with average weighted F₁ scores between 0.02 and 0.07 points above the initial model (Table 3). The results by entity type are shown in the Data Supplement (Table S5).

TABLE 3.

Average Weighted F₁ Score for Models Trained on All Possible Combinations by Cancer and Type of Report

When comparing the performance of the models trained and tested for the same cancer with their performance in the test sets of other cancer types (cross-data subset evaluation), it was observed that the performance in the recognition of M0 is reduced between 0.05 and 0.15 points, being greater the reduction in the evaluation of the BCa model tested on CRC (Table 4). As expected, the performance reduction is more evident in the recognition of M1 entities, with an F₁ score drop between 0.1 and 0.35 points being more evident in the comparison of CRC on BCa. Examples of entity label errors for this comparison are presented in the Data Supplement (Table S6).

TABLE 4.

NER Models Result From Cross-Data Subset Experiments

NER Error Analysis

Almost 20% of identified entities were distributed among the five types of errors. Types 1, 2, and 5 were distributed similarly, and type 3 was in smaller proportion, with only 1% of the errors. No type 4 errors were observed (Fig 1A). More false negatives were observed for M1 entities, whereas false positives and type 5 errors were more frequent in M0 entities (Fig 1B). The frequency of entities incorrectly labeled can be seen in the Data Supplement (Table S7).

FIG 1.

Distribution of error types. (A) The proportion of right and wrong predicted labels by error type. (B) The proportion of wrong predicted labels by the type of error.

Document Classification

Documents classified as M0 had no M1 mentions, whereas documents classified as M1 had a heterogeneous density of M0 and M1 entities per document (Data Supplement, Fig S4). Radiology reports were classified into M1 or M0 class with a high performance measured in F₁ score, mainly given by a high precision for class M1 and recall for M0 (Table 5). The confusion matrix (Data Supplement, Table S8) shows that five actual M1 class documents were classified as M0, whereas only one document was erroneously classified as M1.

TABLE 5.

Document-Level Classification Performance

DISCUSSION

The time taken to initiate cancer treatment is an unquestionably critical variable, which directly affects outcomes and health care costs. Since today not all patients can be managed simultaneously, we aim to design solutions that support and reduce turnaround times in current patient referral pathways for oncology care. Our solution focuses on metastasis staging, automatically extracting distant metastasis mentions from radiology findings descriptions. In the clinical setting, our goal is to promptly determine the M classification of patients to prioritize curative treatments for patients who have not yet developed distant metastasis while evaluating curative or palliative interventions for those who have as it is often a primary criterion for cancer chemotherapy trials.¹¹

We developed the FALP-RR corpus to support this task in Spanish. The corpus is a high-quality resource, with an average IAA close to 0.94. This is important because errors in clinical NER can occur because of modifiers, entity diversity, and manual annotation inconsistencies.⁵⁵ Accurate labeling of M0 and M1 entities is crucial to avoid errors affecting a prompt referral for evaluation of treatment for distant metastases or palliative care.

Our study demonstrates that a DL model for NER can effectively capture mentions of distant metastasis in different types of radiology reports and various cancer types, with an average F₁ score of 0.84. This value is even higher than that reported in similar works focused on metastases and tumor-related information extraction for a single type of cancer. Chen et al²⁴ achieved an F₁ score of 0.82 in recognizing mentions of suspected extrahepatic metastasis in CT reports of patients with hepatocellular carcinoma. Park et al²² obtained an F₁ score of 0.70 in the prediction of distant metastasis in PET-CT reports of patients with lung cancer.

However, the correct detection of M1 mentions was slightly lower than M0, potentially because of confounding from mixing cancer types and lesion locations. This observation is interesting to explore as some studies take the approach of developing a different model for each primary tumor or affected organ. Do et al⁷ developed 13 models, one for each organ to determine the presence or absence of metastases using CT reports. Our results also show that in some cases, it may be more favorable to use subsets of data (by report, cancer, or their combinations) to develop individual models than to develop a model that allows generalizing the recognition of distant metastases over multiple cancers and multiple reports at once. Despite this observation, we consider that our single model for three types of cancer and two report types provides remarkable results in recognizing metastatic mentions by taking advantage of the context provided in the reports.

This is supported by the cross-data subset evaluation, which shows that there is indeed a dependence of the metastatic status on the primary tumor site and the anatomical site of the secondary lesion. In one of the BCa reports, for example, the phrase “No mediastinal hypermetabolic adenopathies are identified” (Data Supplement, Table S6) does not contain any manual annotation since, for this cancer, it does not constitute a mention of distant metastases. The model trained for CRC, however, identifies as M0 the mention mediastinal hypermetabolic adenopathies, which would be correct because it is denied and because of the location if it were a CRC report. Still, it corresponds to a false positive because it is a report for BCa.

The imbalance of M0 and M1 entities in radiology reports (Data Supplement, Fig S4), where M0 entities are described multiple times as findings that are not observed, may also contribute to performance differences between entities. Increasing affirmative examples of distant metastasis could enhance M1 recognition. Classic techniques such as SMOTE, oversampling, and undersampling⁵⁶ address class imbalance. We did not use any of these methods because it is a small data set, with a binary classification, and where the results are quite promising.

Since clinicians describe the distant metastases findings through contextual cues, capturing this context is essential. We included an experimental attribute of uncertainty during manual annotation of M1 mentions, which was only used in the case of textually described uncertainty (eg, lesion of uncertain origin, undetermined lesion). It is important to note this because four of the five documents in which M1 misclassified as M0 had only one manually M1 annotated mention (Data Supplement, Table S8), all of which had the uncertainty attribute. The relationship between uncertainty and model confounding in document-level classification should be addressed further in future research, although it has been reported in other works in Spanish clinical text.^30,57 In the context of metastatic disease, it has been decided in some cases to remove the sentences with uncertain or vague information¹⁴ or to identify and categorize them as not recurrent/metastatic in others.¹³ Regarding the M0 document misclassified as M1, the NER model detected an actual M1 mention, which allowed us to establish that the document was manually misclassified as M0. The sentence omitted by the annotators read: en el mesorrecto medio y superior, se observan adenopatías hipercaptantes de carácter secundario (in the middle and upper mesorectum, there are secondary hypercatching lymphadenopathies). Incorporating contextual negation or uncertainty elements (eg, NUBES data set⁵⁷) will improve the performance of the metastasis detection.

In conclusion, this paper introduces a clinical corpus of radiology reports in Spanish manually annotated with distant metastasis mentions. The DL model trained on this corpus for the NER task demonstrated robust identification of distant metastasis mentions in CT and PET-CT reports from patients with PCa, CRC, and BCa. Our results also show that the distant metastasis NER output can be used to classify whole reports with significant performance, even with a F₁ score of 0.92 for M1 class.

The dependence of the metastatic status on the primary tumor site and the anatomical site of the secondary lesion was evident by using the multicancer multireport model. Although this effect is well tolerated, which is reflected in the performance metrics, the results of some specific cancer models show better performances, so we consider that the choice of approach in the use of the models should be analyzed in the light of the application to be implemented and the obvious limitations of using data sets as small as those used in the cancer-specific models.

Since uncertainty and negation play a crucial role in clinical NER tasks, incorporating these factors could improve M1 and M0 identification performance. Further research is needed to understand the impact of uncertainty and negation on distant metastasis detection. Expanding the FALP-RR corpus size and exploring data augmentation techniques are crucial steps in this direction.

In clinical settings, we propose including an M1 entity ratio alongside report classification to address potential errors in whole document classification. The M1 ratio allows clinicians to manually evaluate reports with fewer M1 mentions, where M1 entities may have been erroneously detected. Additionally, a higher M1 entity ratio can provide insights into the level of distant metastasis involvement, which could be evaluated in future work.

AUTHOR CONTRIBUTIONS

Conception and design: Jocelyn Dunstan, Inti Paredes, Pablo Báez

Financial support: Inti Paredes, Pablo Báez

Collection and assembly of data: Sergio Peñafiel, Inti Paredes

Data analysis and interpretation: Ricardo Ahumada, Matías Rojas, Inti Paredes, Pablo Báez

Manuscript writing: All authors

Final approval of manuscript: All authors

Accountable for all aspects of the work: All authors

AUTHORS' DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

The following represents disclosure information provided by authors of this manuscript. All relationships are considered compensated unless otherwise noted. Relationships are self-held unless noted. I = Immediate Family Member, Inst = My Institution. Relationships may not relate to the subject matter of this manuscript. For more information about ASCO's conflict of interest policy, please refer to www.asco.org/rwc or ascopubs.org/cci/author-center.

Open Payments is a public database containing information reported by companies about payments made to US-licensed physicians (Open Payments).