Natural Language Processing of Computed Tomography Reports to Label Metastatic Phenotypes With Prognostic Significance in Patients With Colorectal Cancer

PURPOSE

Natural language processing (NLP) applied to radiology reports can help identify clinically relevant M1 subcategories of patients with colorectal cancer (CRC). The primary purpose was to compare the overall survival (OS) of CRC according to American Joint Committee on Cancer TNM staging and explore an alternative classification. The secondary objective was to estimate the frequency of metastasis for each organ.

METHODS

Retrospective study of CRC who underwent computed tomography (CT) chest, abdomen, and pelvis between July 1, 2009, and March 26, 2019, at a tertiary cancer center, previously labeled for the presence or absence of metastasis by an NLP prediction model. Patients were classified in M0, M1a, M1b, and M1c (American Joint Committee on Cancer), or an alternative classification on the basis of the metastasis organ number: M1, single; M2, two; M3, three or more organs. Cox regression models were used to estimate hazard ratios; Kaplan-Meier curves were used to visualize survival curves using the two M1 subclassifications.

RESULTS

Nine thousand nine hundred twenty-eight patients with a total of 48,408 CT chest, abdomen, and pelvis reports were included. On the basis of NLP prediction, the median OS of M1a, M1b, and M1c was 4.47, 1.72, and 1.52 years, respectively. The median OS of M1, M2, and M3 was 4.24, 2.05, and 1.04 years, respectively. Metastases occurred most often in liver (35.8%), abdominopelvic lymph nodes (32.9%), lungs (29.3%), peritoneum (22.0%), thoracic nodes (19.9%), bones (9.2%), and pelvic organs (7.5%). Spleen and adrenal metastases occurred in < 5%.

CONCLUSION

NLP applied to a large radiology report database can identify clinically relevant metastatic phenotypes and be used to investigate new M1 substaging for CRC. Patients with three or more metastatic disease organs have the worst prognosis, with an OS of 1 year.

INTRODUCTION

Colorectal cancer (CRC) is the third most common cause of cancer-related mortality in the United States, with metastatic disease being the leading cause of death.^1,2 Metastatic CRC is increasingly recognized as a heterogeneous group of diseases with different biological and genetic features, with varying prognosis and survival outcomes.^3,4

CONTEXT

• Key Objective

• Can natural language processing be applied to a large database of radiology reports to extract relevant prognostic information from patients with colorectal cancer (CRC)?

• Knowledge Generated

• Overall survival of CRC decreases with an increasing number of metastatic organs, with a median survival of about 1 year when three or more organs are involved. The organs of most frequent metastases were the liver, abdominopelvic lymph nodes, and lungs (above 30%), followed by the peritoneum and thoracic nodes (above 20%), bones, and pelvic organs (above 8%), and infrequently to the spleen and adrenal metastases.

• Relevant

• Radiology reports contain imaging phenotypes that can inform the development of prognostic categories of CRC, such as new M1 substaging. Refined prognostic classifications can improve outcome analysis in clinical trials and patient series and be used to balance treatment arms appropriately.

In 2017, the eighth edition of the American Joint Committee on Cancer (AJCC) TNM staging expanded the M subgroup, adding the M1c category. M1a and M1b denote patients with metastases to one or more than one organ, respectively, while M1c corresponds to those with peritoneal metastases.^5,6 Evidence supporting the M1c includes results from randomized chemotherapy trials in metastatic CRC comparing those with and those without peritoneal involvement, showing a reduction of approximately 30% in overall survival (OS) and 20% in disease-free survival for the latter, even adjusting for other unfavorable prognostic factors.⁷ Other groups later validated these results.⁸

In CRC, metastatic disease patterns have been primarily obtained from autopsy series or clinical databases.^1,9-11 Because of the increased use of structured radiology reports,^12-14 natural language processing (NLP) can generate large databases of metastatic phenotypes in CRC and investigate their clinical relevance in the context of M staging.^15-20 NLP has been applied to radiology reports of CRC and other cancers to identify patterns of metastatic disease,²¹ but their prognostic significance has not been investigated to date.

We hypothesize that CRC metastatic phenotypes extracted from radiology reports by NLP have prognostic significance. The primary purpose of our study was to compare the OS of patients with CRC according to AJCC TNM staging and provide an alternative classification. The secondary outcome was to measure the frequency of metastatic disease for each organ in CRC.

METHODS

This retrospective study was Health Insurance Portability and Accountability Act–compliant and approved by the institutional review board at Memorial Sloan Kettering Cancer Center. The requirement for informed consent was waived as the study was determined to be at a low protocol risk level.

Consecutive radiology reports of computed tomography (CT) examinations of the chest, abdomen, and pelvis performed between July 1, 2009, and March 26, 2019, were collected by procedure codes from our institutional database. The initial date was chosen to coincide with the implementation of a departmental structured reporting template. In addition, the text from the findings and impression section of the report was included. The findings section was further divided by headers, which included the following organs: lungs, pleura, thoracic nodes, liver, spleen, adrenals, abdominopelvic nodes, pelvic organs, peritoneum, and bones. Only reports from CT of chest, abdomen, and pelvis were analyzed; other modalities such as bone scans or magnetic resonance imaging were not evaluated. Thus, metastases outside the CT field of view, such as the brain or extremities, were excluded.

Patients with CRC were identified on the basis of an algorithm developed to return the patient's most likely primary cancer type at the time of each CT report.²¹ Because individual patients could be assigned more than a single type of primary cancer if they had multiple CTs (eg, prostate cancer and CRC), only patients diagnosed with CRC throughout all their available reports were included in this study.

Report Annotation

Manual curation of the radiology reports was performed by three radiologists (R.D., a radiology attending physician with 11 years of experience; P.C.A., a radiology clinical fellow training in oncology imaging; and M.T., a radiology resident in training).²¹ Each curator was given instructions to label for the presence or absence of metastatic disease after reading the text sections from findings and impression, evaluating for every organ as positive or negative for metastatic disease. Findings were labeled as positive if the curator interpreted the findings as consistent with or suspicious for metastatic disease.²² Data curation was performed to obtain a minimum of 2,000 annotated reports. Because the three curators were independently working simultaneously, a total of 2,219 reports were annotated. In addition, the curators were blinded to all other clinical data, including additional imaging tests, and there was no overlap between curated reports. The inter-reader agreement among the annotators was not calculated.

NLP Model Development

The model²¹ was developed in Python 3.6.8 (open-source), with the following open-source libraries: Keras (v2.4.3), nltk (v3.5), pandas (v1.1.4), NumPy (v1.19.4), scikit-learn (v0.24), and Seaborn (v0.11.1).

The NLP model used term frequency (TF), or the number of times a word appears in a report divided by the total number of words in that report, and inverse document frequency (IDF), or the logarithm of the number of total reports divided by the number of reports in which the term appears.²¹

All text data were normalized to a format easiest to digest by models, and to ensure small lexical or stylistic differences in words (capital letters, apostrophes, etc) would not affect TF. This ensures the lexicon used does not grow too large, improving the effectiveness of the TF-IDF algorithm. Target values (ie, labels and y-values) were label-encoded from yes and no values to binary values for model digestion. These sets were processed using a TF-IDF method. TF is the number of times a word appears in a document (ie, a single radiology report) divided by the total number of words in that document. IDF is calculated as the logarithm of the number of reports divided by the number of reports in which the term appears. All TF-IDF methods and capabilities were assessed through the Tfidf Vectorizer class found in the feature_extraction.text submodule within the Python library scikit-learn.^23,24

All TF-IDF processed data were passed through an ensemble voting model built off inputs from a logistic regression model, support vector machine, a random forest model, and an extreme gradient boosting model. An ensemble model was chosen because the technique allows using a voting strategy to select the best result or prediction from several underlying statistical models.²³ Vote tallying can be done by using a hard vote counter, where the classification is made by a strict count of totals, or by a soft vote counter, where the classification is done by considering how certain each classifier performs by averaging the probabilities of each outcome. This ensemble model used a soft vote counter to leverage the confidence of each model in the final prediction.²³

A simple weighting algorithm was used to determine the relative importance of each of the input models. The algorithm compared the accuracy, precision, and recall results on the training set of each individual model with those of all other models and with the average accuracy, precision, and recall of all the models to assign a weight value to each model such that the best-performing classification model was given the largest weight value. This calculation was made once for each organ to better optimize model performance by location; for example, the weights assigned to the different models for predicting liver metastasis would be other than the weights assigned to the models for predicting lung metastasis.

The manually curated data were separated into training and testing sets with an 80%-20% split. NLP model performance was measured using accuracy, precision, and recall scores. The results of the manual data curation were used as the reference standard.

Once the model was fine-tuned to achieve the best performance on the training and testing data subsets, it was used to label the remaining unannotated reports for the presence or absence of metastatic disease. To determine the size of a validation set of reports, we performed a power calculation that assumed an estimated model accuracy of 90% and a goal of achieving a two-sided 95% CI with a width equal to 6%. This calculation yielded a sample size of 417 reports. To account for proper randomization, on the basis of demographic variables such as age, race, sex, and proportion of metastatic labels, this sample size was adjusted to 448. This validation set, which excluded reports used in the training and test sets, was manually annotated by radiology attending (R.D.) and used to determine the NLP model's accuracy, precision, and recall scores. The radiologist was blinded to all other clinical data during the annotation. During the curation process, short text strings such as unremarkable, were recognized as default or frequently occurring text strings that would be best addressed by creating rule-based labeling. Therefore, rule-based labeling for each of the 13 organs was later generated and applied to all reports to develop the final database of metastatic disease labels.

Estimating OS and Metastases Frequencies

The labels generated by the NLP model were used to calculate the frequency of metastatic disease for the CRC cohort at the patient level by dividing the number of all patients who have at least one positive metastasis label by the total number of patients. Next, each organ's cumulative incidence of metastases was calculated to describe their incidence over 10 years.

To investigate the prognostic significance of different metastatic disease patterns identified by NLP, patients were divided into those who did not have any metastasis (M0) and those who had at least one metastasis detected, with different follow-up starting points: the date of their first CT for the former, and the date of CT scan with the first metastasis detected for the latter. Survival time was measured from the date of the corresponding CT to the death date (uncensored) or last known follow-up date (censored at the last date of follow-up). Patients with metastases identified by NLP were categorized according to AJCC criteria as M1a, only an organ involved; M1b, two or more organs involved; and M1c, patients with peritoneal involvement, regardless of the total number of organs involved. An alternative model on the basis of the total number of organs involved was investigated and defined as: M1, M2, two organs; M3, three or more organs.

Statistical Analysis

Using M status as the stratifying variable, Kaplan-Meier curves were constructed using R 3.6.3 (R Core Team 2017). The log-rank test was used to assess the differences in OS between the M subcategories. Cox regression models were used to calculate hazard ratios with time to death as the dependent variable and M status as the stratifying variable using the mentioned classifications. Kaplan-Meier survival curves were used to visualize the differences in survival of the classifications. Type I error rate was set to .05 (α). Ten-year cumulative incidence rates for each metastatic organ were also estimated, with 95% CIs calculated using the Greenwood formula.²⁵

RESULTS

The study population consisted of 387,359 reports from 91,665 patients, with the most frequent primary cancers being colorectal, ovarian, renal, bladder, lung, soft tissue, breast, and prostate cancer.²¹ Two thousand two hundred nineteen reports from this cohort were used for manual annotation and for the training and test sets, and 448 reports were used for a validation set (Fig 1). The performance of the NLP model in the training, test, and validation sets is given in Appendix Table A1. The accuracies across all organs ranged from 90.2% to 99.8%.

FIG 1.

Flow diagram. AJCC, American Joint Committee on Cancer; CRC, colorectal cancer; CT, computed tomography; NLP, natural language processing.

From this larger cohort, we only evaluated the pattern of metastatic disease in patients with CRC, including 48,408 radiology reports from 9,928 patients with CRC (46.3% women). The mean [standard deviation] number of reports per patient was 4.88 [5.36]. The mean [standard deviation] age at the time of the first scan was 59.9 [13.7] years. The median follow-up time was 2.24 years, and the censoring rate was 62%. An example of a labeled CT report from a patient with CRC is shown in Figure 2.

FIG 2.

Annotated sample of a radiology report to show the terms targeted for labeling by natural language processing. CT, computed tomography.

On the basis of their initial CT, patients with CRC were subclassified into M0 (n = 3,880), M1a (n = 2,806), M1b (n = 1,681), and M1c (n = 1,258). Of note, the patient number analyzed was lower (n = 9,626) than the initial cohort (n = 9,928) because the suitable patients for analyses were only those with follow-up. Compared with M0, the OS of M1a, M1b, and M1c patients were lower, at 4.47, 1.72, and 1.52 years, respectively. However, given the overlapping OS between M1b and M1c, we explored an alternative subclassification on the basis of the total number of organs only, as described above. The OS of M1 (n = 3,197), M2 (n = 1,466), and M3 (n = 1,082) was 4.24, 2.05, 1.04 years, respectively. Median OS and hazard ratios for both existing and proposed methodologies are given in Table 1 and depicted in Figure 3.

TABLE 1.

Median OS Time and HRs Estimated Using Both the Grouping Methodologies (existing and proposed)

FIG 3.

Median survival time and hazard ratios estimated using both the grouping methodologies depicting Kaplan-Meier curves of CRC on the basis of metastatic stage: (A) American Joint Committee on Cancer and (B) alternative. CRC, colorectal cancer; KM, Kaplan-Meier.

In patients with CRC, metastases were most often in the liver (35.8%), abdominopelvic lymph nodes (32.9%), and lungs (29.3%), followed by the peritoneum (22.0%) and thoracic nodes (19.9%). Metastases to bones (9.2%) and pelvic organs (7.5%) were less common. Finally, pleura (4.9%), adrenal (4.4%), spleen (1.6%), and kidney (0.1%) metastases were rare, and occurred in < 5%. In the analysis of the cumulative incidence of metastases, we found that in some organs, such as the bone, the frequency at baseline scan may be low but it increased over time. By contrast, the frequency of metastases remained low throughout the 10 years of follow-up for some rare metastatic organs, such as the kidney or spleen. Table 2 summarizes the frequency of metastatic organ involvement at the patient level and the cumulative incidence of metastases over 10 years, also described in Figure 4.

TABLE 2.

Frequency of Metastatic Organ Involvement at the Patient Levels and Cumulative Incidence Over 10 Years

FIG 4.

Cumulative incidence of metastases over 10 years. Ad, adrenals; Ap, abdominopelvic lymph nodes.

DISCUSSION

The NLP model applied to a radiology report database of nearly 10,000 patients with 49,000 reports across 10 years depicted metastatic phenotypes of patients with CRC and provided evidence for a new and clinically relevant substaging classification.

We found through NLP of radiology reports that CRC most often metastasizes to the liver (35.9%) and lung (29.3%), in line with prior evidence on the basis of autopsy or clinical notes reporting 18%-48% in the liver^1,2,4,26,27 and 22%-47% in the lungs.^2,3,27,28 Metastases to the peritoneum were slightly less frequent than lungs and liver (22.2%), as previously reported in the 21%-28% range.²⁸ Finally, organs with low frequency such as adrenals, bones, and spleen were concordant with previously reported literature.^29-32 Unlike an autopsy series, our CT data with follow-up over 10 years allowed us to analyze each organ's cumulative incidence over time. For example, although the frequency of bone metastasis is lower at diagnosis, it increases over time, reaching a similar level to the peritoneum. Given the concordance of our results with the prior literature, we proceeded to investigate the prognostic significance of NLP-based metastatic patterns.

In the context of AJCC (eighth edition) M substages, we found that patients with M1b or M1c disease had significantly worse OS than M1a patients in our study. However, M1a patients had a higher OS (4.93 years/59.16 months) than previously reported. For example, Wang et al³ analyzed 26,170 patients from a SEER database and found an OS of 18 months for M1a patients. However, Miyoshi et al,²⁷ analyzing 83 M1a patients, found an OS of 43.7 months. The improved outcomes for M1a patients in our study may reflect a referral bias to our institution for patients with liver-limited disease responding to first-line chemotherapy for consideration of surgery or hepatic arterial infusion therapy. Interestingly, our study's OS of M1b patients was similar to that previously reported of 20.64 months.²⁷ In line with the eighth Edition of the AJCC TNM,^5-7 we found a decreased OS for M1c patients in our study; nevertheless, our OS for these patients was slightly higher (1.52 years/18.24 months) than the reported average of 13 months.^7,8 Although the difference between M1b and M1c patients was minimal (2.4 months) and similar to the range reported (eg, by Tong et al,⁸ who analyzed 1,090 cases from an institutional database field), the difference was still significant, probably because of our large sample size.

In our exploration of an alternative classification on the basis of the number of metastatic organs, patients with the alternative M3 substage, with three or more metastatic organs, had the shortest OS (11.9 months), in line with other investigators' results.^33,34 For example, Elias et al³³ analyzed 308 patients with liver and extrahepatic metastatic disease, and found that the strongest survival predictor is the number of metastatic organs and the suitability for surgical resection rather than the specific organ of metastasis.³³ Tanaka et al³⁴ proposed a subclassification of M1c and found that patients with the only peritoneal disease and no organ involvement had similar survival to M1b,³⁴ suggesting that a further subclassification of M1c is necessary.³⁴ Although our novel results need additional external validation, they include more than 1,000 patients in each substage.

This project has several limitations. As a single-site study from a tertiary cancer center, it may not be generalizable to other settings. Our ground truth was radiology reports, with inherent variability in imaging interpretation between radiologists. Furthermore, metastatic disease to abdominopelvic nodes was not discriminated between those with regional or distant nodal metastases. The use of radiology reports also ignores the clinical staging of the patient, which may also include pathologic and surgical records. For our analysis, we only considered OS, not disease-specific survival. Nevertheless, our data show the utility of analyzing a large database of radiology reports alone to gain insight into the survival of cohorts of CRC. Although we evaluated the role of multiorgan metastatic disease, we did not investigate the role of tumor burden as other authors did⁴ since it was not described in a standardized manner in radiology reports. Finally, we had a relatively short median follow-up of 2.24 years. Still, this time frame is expected since the largest fraction of the cohort was M1b and M1c, both with median survivals under two years.

In conclusion, NLP applied to an extensive radiology report database can identify clinically relevant metastatic phenotypes and can be used to investigate potential new M1 substaging for CRC. In addition, refined prognostic classifications can improve outcome analysis in clinical trials and patient series and be used to balance treatment arms appropriately. Future directions include correlating the survival curves with genomic data to help to tailor treatments in our growing era of personalized medicine.

APPENDIX

TABLE A1.

NLP Model Training: Accuracies for the NLP Model on the Training, Test, and Validation Sets

AUTHOR CONTRIBUTIONS

Conception and design: Pamela Causa Andrieu, Jennifer S. Golia Pernicka, Kaelan Lupton, Lior Gazit, Natalie Gangai, Varadan Sevilimedu, Richard Do

Financial support: Amber L. Simpson

Administrative support: Huy Nguyen, Natalie Gangai

Provision of study materials or patients: Richard Do

Collection and assembly of data: Pamela Causa Andrieu, Jennifer S. Golia Pernicka, Michio Taya, Lior Gazit, Huy Nguyen, Kevin Nicholas, Natalie Gangai, Shannan Dickinson, Viktoriya Paroder, David D.B. Bates, Richard Do

Data analysis and interpretation: Pamela Causa Andrieu, Jennifer S. Golia Pernicka, Rona Yaeger, Kaelan Lupton, Karen Batch, Farhana Zulkernine, Amber L. Simpson, Lior Gazit, Huy Nguyen, Natalie Gangai, Varadan Sevilimedu, Viktoriya Paroder, Richard Do

Manuscript writing: All authors

Final approval of manuscript: All authors

Accountable for all aspects of the work: All authors

AUTHORS' DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

The following represents disclosure information provided by authors of this manuscript. All relationships are considered compensated unless otherwise noted. Relationships are self-held unless noted. I = Immediate Family Member, Inst = My Institution. Relationships may not relate to the subject matter of this manuscript. For more information about ASCO's conflict of interest policy, please refer to www.asco.org/rwc or ascopubs.org/cci/author-center.

Open Payments is a public database containing information reported by companies about payments made to US-licensed physicians (Open Payments).