Natural Language Processing for the Identification of Incidental Lung Nodules in Computed Tomography Reports: A Quality Control Tool

NLP tool accurately identifies ILNs in chest CT reports, enabling early lung cancer detection and reducing patient loss to follow-up.

Abstract

PURPOSE

To evaluate the diagnostic performance of a natural language processing (NLP) model in detecting incidental lung nodules (ILNs) in unstructured chest computed tomography (CT) reports.

METHODS

All unstructured consecutive reports of chest CT scans performed at a tertiary hospital between 2020 and 2021 were retrospectively reviewed (n = 21,542) to train the NLP tool. Internal validation was performed using reference readings by two radiologists of both CT scans and reports, using a different external cohort of 300 chest CT scans. Second, external validation was performed in a cohort of all random unstructured chest CT reports from 57 different hospitals conducted in May 2022. A review by the same thoracic radiologists was used as the gold standard. The sensitivity, specificity, and accuracy were calculated.

RESULTS

Of 21,542 CT reports, 484 mentioned at least one ILN (mean age, 71 ± 17.6 [standard deviation] years; women, 52%) and were included in the training set. In the internal validation (n = 300), the NLP tool detected ILN with a sensitivity of 100.0% (95% CI, 97.6 to 100.0), a specificity of 95.9% (95% CI, 91.3 to 98.5), and an accuracy of 98.0% (95% CI, 95.7 to 99.3). In the external validation (n = 977), the NLP tool yielded a sensitivity of 98.4% (95% CI, 94.5 to 99.8), a specificity of 98.6% (95% CI, 97.5 to 99.3), and an accuracy of 98.6% (95% CI, 97.6 to 99.2). Twelve months after the initial reports, 8 (8.60%) patients had a final diagnosis of lung cancer, among which 2 (2.15%) would have been lost to follow-up without the NLP tool.

CONCLUSION

NLP can be used to identify ILNs in unstructured reports with high accuracy, allowing a timely recall of patients and a potential diagnosis of early-stage lung cancer that might have been lost to follow-up.

INTRODUCTION

An incidental lung nodule (ILN) is a pulmonary nodule detected during imaging studies performed for an unrelated reason.^1-3 Although most pulmonary nodules are benign, 1%-5% are malignant. Therefore, ILN should be strictly managed following international guidelines to identify potentially malignant nodules and improve the chances of diagnosing lung cancer at an early stage.⁴ However, ILN management is often neglected as it was not the main reason for the imaging study and lung nodule is a noncritical result, particularly in the emergency setting. In addition, the absence of clear recommendations in the imaging reports could also contribute to the loss of follow-up.

CONTEXT

• Key Objective

• It is possible to develop a natural language processing (NLP) tool to accurately detect incidental lung nodules (ILNs) in unstructured chest computed tomography (CT) reports.

• Knowledge Generated

• We developed a NLP algorithm to identify ILNs with sensitivity and specificity of 98.4% and 98.6%, respectively.

• Relevance

• NLP tools can accurately identify ILNs in chest CT reports, reducing loss to follow-up and allowing early lung cancer diagnosis.

Manual review of reports from every chest computed tomography (CT) scan can be excessively time-consuming and prohibitive, and there is increasing interest in the use of natural language processing (NLP) to automatically extract relevant information from electronic radiology reports. NLP is an artificial intelligence field in which computer science and linguistics are combined to enable machines to understand human language.^5,6 NLP enables the analysis of unstructured free text for the extraction of structured information.⁵ However, NLP analysis of radiologic reports is not a simple task because of variations in reporting styles among radiologists. To date, many NLP algorithms have shown high specificity but less-than-perfect sensitivity in terms of word variability.^7,8 Although NLP is useful for improving ILN follow-up and management in some settings, guidelines for radiology reports are required to enhance patient care.^7-12

In a previous study, an NLP algorithm had very high sensitivity (98.6%; 95% CI, 95.0 to 99.8) and specificity (100%; 95% CI, 93.9 to 100) to identify pulmonary nodules.⁴ This algorithm could even perform well for larger cohorts, identifying 217,771 cases of lung nodules among 717,304 chest CT reports (30.4%).⁴ To our knowledge, only one study used NLP to specifically identify ILN, reporting a sensitivity and specificity of 91.1% and 82.2%, respectively.¹³

This study aimed to develop an NLP algorithm for the identification of incidental nodules in radiology reports of chest CT scans with a high suspicion of malignancy, which would in turn lead to timely and optimal management of ILNs on the basis of the most recent society guidelines and recommendations.

METHODS

Data Sources and Study Sample

This study was performed with the approval of our institutional review board (institutional review board—approval number 55655922.9.0000.5249), waiving written consent and following Health Insurance Portability and Accountability Act compliance. The data collection steps are shown in the flow diagram in Figure 1. A group of six radiologists with three to 35 years of experience retrospectively reviewed all consecutive unstructured reports of chest CT scans performed in a tertiary hospital between January 2020 and December 2021 (n = 21,542). This cohort was used as a training set for the NLP system to automatically detect ILN, which was defined as a previously unknown pulmonary nodule larger than 4 mm in diameter with no clinical context of known neoplasia, pneumonia, or small airway disease. Therefore, reports were excluded when (1) the main CT indication was the evaluation of a previously known pulmonary nodule, (2) the nodule was reported as definitely benign, and (3) they followed any type of structured report model to describe lung nodules, including structured reports for lung cancer screening. Previously known pulmonary nodules were identified using the referring indication for the CT scan and description words in reports such as “stable,” “unchanged,” or “minimal change” or other related terms suggesting that the lesion was already described on a previous CT scan. Nodules were considered benign when they contained fat or nonmalignant calcification patterns or when they coexisted with features of inflammatory processes, such as diffuse centrilobular nodules, tree-in-bud pattern, and small airway disease.¹⁴ Any disagreements between the radiologists were resolved by consensus.

FIG 1.

Flow diagram. ^aFleischner Society guidelines published in 2017 by MacMahon et al.¹⁵ CT, computed tomography; ILN, incidental lung nodule; NLP, natural language processing.

After this initial training, the NLP tool underwent internal validation using a different external cohort of 300 chest CT scans. This sample consisted of 152 ILN cases and 148 controls, randomly selected from scans performed at the same initial institution between January 2022 and April 2022. Two thoracic radiologists with 6 and 8 years of experience in chest imaging analyzed both images and reports from the chest CT scans, evaluating the presence and characteristics of the pulmonary nodules. Positive cases consisted of ILN, defined as a previously unknown pulmonary nodule larger than 4 mm in diameter without a clinical context of known neoplasia, pneumonia, or small airway disease. Disagreements between the two readers were resolved by consensus.

In addition, we performed external validation of our NLP system, including all random unstructured chest CT reports from 57 different hospitals performed in May 2022. These reports were randomly selected and anonymized from the integrative picture archiving and communication systems records of these institutions. For external validation, we also considered as gold standard the review performed by the same two thoracic radiologists who once again analyzed both images and reports from the chest CT scans.

Finally, we evaluated how high-suspicion nodules were managed using the Fleischner Society guidelines as a reference.¹⁵ High-suspicion nodules were considered solid nodules larger than 8 mm or part-solid nodules with a solid component bigger than 6 mm.¹⁵ Although NLP was not clinically implemented, we manually monitored these high-suspicion cases up to December 2022 using their electronic medical records.

Development of the NLP Algorithm

The process of extracting information from medical reports was based on two steps: identification of sentences with nodule information and extraction of proper nodule information. The NLP tool analyzed the entire CT report, including the sections findings and impression. Stemming was used to preprocess the text. The selection consisted of identifying reports that contained mentions of the lung nodules and their sizes. For this task, we used an NLP technique called named entity recognition (NER). NER is an information extraction technique that involves recognizing entities whose description is made up of one or more words, such as 13.0 mm and apical segment of the upper lobe, which are treated as a single concept. We generated a set of regular expression rules to identify relevant information such as nodule characteristics, size, and body parts.

To extract information from each sentence highlighted in the previous step, we used another NLP technique, called dependency parser.¹⁶ The dependency parser technique generates a direct acyclic graph among words in the analyzed text on the basis of syntactic, grammatical, and morphologic language rules, which are called dependency trees. We devised a multistep algorithm to extract nodule information from these sentences.

In the first pass, starting from the root node, our algorithm recursively separated all the nodes that defined a subtree. The second pass concatenated the information chunks with their head nodes, keeping in mind that an information chunk can only have one concept of location, size, and nodule information. In the third pass, we identified the chunks with nodule information and associated all the linked information chunks. Figure 2 shows the process for the sentence “a solid, dense, regular contour nodule in the right lower lobe, measuring 1.0 cm of probable residual nature.”

FIG 2.

An example of our information extraction algorithm. (A) Initial dependency tree. (B) Final information chunks.

The algorithm was implemented in Python, using the spaCy¹⁷ and Turku neural parser¹⁸ packages. We needed to train new Turku models to generate dependency trees in Portuguese, as no pretrained model was available. To train these models, we used the Portuguese Bosque data set from the Universal Dependencies initiative.¹⁹

Statistical Analysis

Data are presented as frequencies (percentages) with 95% CI or mean ± standard deviation. Data were tested for normal distribution using the Shapiro-Wilk test. All tests were two-tailed with a level of significance set at <0.05. For the performance of our NLP system, sensitivity, specificity, positive likelihood ratio (PLR) and negative likelihood ratio (NLR), positive predictive value (PPV) and negative predictive value (NPV), and accuracy were calculated. Statistical analyses were performed using SPSS version v.22 (IBM Corp, Armonk, NY).

RESULTS

Study Sample

Among the initial 21,542 CT reports, 2.2% (n = 484) mentioned at least one lung nodule and were selected as a training set for the NLP system (female, n = 252; 52%; mean age, 71 ± 17.6 years). Among the 300 selected cases of ILN that underwent the validation test, most were men (n = 171; 57%), with a mean age of 62 ± 14.3 years. For the external validation, gathering data from 57 different centers, 977 chest CT scans were selected (men, 498; 51%; mean age, 60 ± 21 years).

Validation Set

Using the evaluation performed by thoracic radiologists as the gold standard in the internal validation set (n = 300), the NLP tool identified 154 ILNs. The analysis yielded 154 true positives, 140 true negatives, no false negatives, and six false positives (thyroid nodules, n = 3; pulmonary lymph nodes, n = 2; liver nodule, n = 1). Sensitivity and specificity were 100.0% (95% CI, 97.6 to 100.0) and 95.9% (95% CI, 91.3 to 98.5), respectively (Table 1). The PLR and NLR were 24.3 (95% CI, 11.1 to 53.3) and 0, respectively. The accuracy was 98.0 (95% CI, 95.7 to 99.3).

TABLE 1.

Diagnostic Performance of the Natural Language Processing Tool to Detect Incidental Lung Nodules

Test Cohort

For external validation (n = 977), the NLP tool identified 126 ILNs. The analysis yielded 837 true negatives, 126 true positives, two false negatives, and 12 false positives. The causes of false positive cases were the following: thyroid nodules (n = 6), pulmonary lymph nodes (n = 2), liver nodules (n = 2), and breast nodules (n = 2). The two false negatives were cases where the nonthoracic radiologist inappropriately described lung nodules as rounded opacity and rounded ground-glass opacity. Sensitivity and specificity were 98.4% (94.5 to 99.8) and 98.6% (97.5 to 99.3), respectively (Table 1). The PLR and NLR were 69.6 (39.7 to 122.2) and 0.1 (0.0 to 0.1), respectively. The accuracy was 98.6 (97.6 to 99.2). Considering only high-risk nodules (ie, a diameter of 8 mm or more) as true positives, there were no significant changes in sensitivity, specificity, or accuracy (Table 1).

Assessment of Nodules' Management

Among the ILN cases, 93 (19.21%) had a diameter of 8 mm or more (ie, considered as a high suspicion of malignancy) and were evaluated on their management based on the Fleischner Society guidelines.¹⁵ Twelve months after the initial reports, we identified eight (8.6%) patients through their electronic medical records who had a final diagnosis of primary lung cancer, among which two (2.2%) would have lost follow-up without the NLP tool and could be diagnosed in stage IA (Figs 3 and 4). Mortality at 12 months was 3.2% (n = 3), and all were cancer-related.

FIG 3.

A 64-year-old man presented in the emergency department with acute dyspnea. (A) Axial and (B) coronal enhanced chest CT images demonstrated a spiculated solid pulmonary nodule of 23 mm, located in the right upper lobe, and severe emphysema. The patient was discharged from the emergency department with a final diagnosis of COVID-19 infection and an orientation to arrange an appointment with a thoracic surgeon. However, the patient did not proceed with further investigation and the CT report was identified using our NLP system. The patient was recalled and, after investigation, the diagnosis of lung cancer was confirmed in an early-stage IA (T1cN0M0). CT, computed tomography; NLP, natural language processing.

FIG 4.

Images of a 58-year-old heavy smoker man who presented in the emergency department with a suspicion of pulmonary embolism. (A) Axial and (B) sagittal enhanced chest CT images demonstrated a spiculated solid pulmonary nodule with a diameter of 6 mm, located in the superior segment of the right lower lobe, and severe emphysema. Although a control scan at 6-12 months was suggested in the initial CT report, the patient lost follow-up, and the case was retrieved using our NLP system. In the control CT scan performed 18 months later, the nodule had doubled in size, as demonstrated in the (C) axial and (D) sagittal enhanced CT images. The final diagnosis was lung cancer, which was staged as T1AN0M0, IA. CT, computed tomography; NLP, natural language processing.

Most cases (n = 63; 67.7%) adhered to the Fleischner Society guidelines, classified as benign in 30 (47.6%) after a reduction in size or complete resolution. Some patients had no clinical condition to proceed with the investigation with biopsy or surgery (n = 4). Nodule management did not follow the guidelines in 30 (32.2%) patients. Some patients lost follow-up and were unable to be recalled (n = 18).

DISCUSSION

We have developed an NLP-based system trained to identify ILN, presenting sensitivity from 98.4% to 100%, specificity from 95.9% to 98.6%, and accuracy from 98% to 98.6%. When applied to a data set of 484 reports, the NLP identified 93 patients with pulmonary nodules with at least one ILN larger than 8 mm. The analysis of adequate follow-up according to the Fleischner recommendation guidelines showed that 30 patients (32.2%) did not have adequate follow-up after 1 year. The prevalence of new lung cancer diagnoses was 0.3% per 1,000 CT scans performed. In this context, the developed tool could identify a substantial percentage of nodules that did not require appropriate follow-up, improving patient care. The NLP tool promotes easier, large-scale identification of adequate follow-up and guidelines concordance, and rapid identification of potential malignancy.

In the peer-reviewed literature, only a few studies have evaluated the correct follow-up of ILN according to Fleischner guidelines.^20,21 A previous study reported that up to 54.8% of patients with lung nodules who required follow-up did not undergo imaging control in the recommended period.²⁰ Also, few reports have analyzed the use of NLP tools to aid proper nodule management.^13,22 To our knowledge, only Kang et al¹³ used NLP to specifically identify ILN, reporting a high sensitivity similar to that found in our study (91.1%; 95% CI, 83.8 to 95.6) and slightly smaller specificity (82.2%; 95% CI, 77.8 to 86.1). This difference in specificity from our algorithm could be explained by their tool including descriptions related to benign nodules, such as inflammatory ground glass opacity within both upper lobes and stable small nonspecific nodules.¹³ Lim et al²² significantly improved lung nodule follow-up by developing an analytical algorithm, mining data from radiology reports, and linking the identified patients and primary care physicians to close the loop with follow-up communication. However, that study included previously known or definitively benign nodules and did not analyze separately ILN.²² Considering the number of chest CT scans performed daily and the high prevalence of pulmonary nodules in these examinations, our algorithm would help the overcrowded health system by dismissing benign ILN, allowing nodules at high risk for lung cancer to be investigated and well managed. The NLP tool has enormous potential to positively affect lung cancer care, on the basis of early diagnosis, which would have a direct impact on patient survival.

One of the strengths of our research was demonstrating the reproducibility of the algorithm in an external validation cohort including reports from 57 different sites with high performance. This consistency indicates a resilient and robust model that performs well despite variations in linguistic nuances and reporting styles of unstructured reports, probably because of the standardized definition of lung nodule that was paved by the Fleischner Society.¹⁵ Moreover, the algorithm was implemented in Python, which is widely available with an abundance of free tools.²³ Therefore, our results support the application of this tool across different institutions or countries where Portuguese is the primary language.

Our study had some limitations. First, we only looked for the word “nodule” in the radiology reports, and some nonspecific terminology could have missed ILN. Second, we did not evaluate CT scans for nodules that were not detected by the radiologists. However, the objective of our study was not to aid in the diagnosis of nodules, but rather to replace the manual process of reviewing unstructured reports in which ILNs were described, to analyze follow-up recommendations that can help early lung cancer diagnosis. At this stage, our algorithm did not capture nodule descriptions included in more than one sentence or correctly identified mentions of multiple nodules in the same sentence.

In summary, we developed an automated NLP tool that enables the identification of high-risk ILNs in unstructured radiology reports with high accuracy. This represents an important and reliable quality and safety tool, ensuring optimal and timely follow-up and management of ILNs.

AUTHOR CONTRIBUTIONS

Conception and design: Rodrigo Basilio, Alysson Roncally Carvalho, Sephania Accorsi, Reza Forghani, Matheus Zanon, Bruno Hochhegger

Provision of study materials or patients: Rodrigo Basilio, Alysson Roncally Carvalho

Collection and assembly of data: Rodrigo Basilio, Rosana Rodrigues, Sephania Accorsi, Matheus Zanon, Bruno Hochhegger

Data analysis and interpretation: All authors

Manuscript writing: All authors

Final approval of manuscript: All authors

Accountable for all aspects of the work: All authors

AUTHORS' DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

The following represents disclosure information provided by authors of this manuscript. All relationships are considered compensated unless otherwise noted. Relationships are self-held unless noted. I = Immediate Family Member, Inst = My Institution. Relationships may not relate to the subject matter of this manuscript. For more information about ASCO's conflict of interest policy, please refer to www.asco.org/rwc or ascopubs.org/go/authors/author-center.

Open Payments is a public database containing information reported by companies about payments made to US-licensed physicians (Open Payments).