Undetected Lung Cancer at Posteroanterior Chest Radiography: Scratching the Surface of Deep Learning

See also the article by Nam et al in this issue.

Charles White, MD, FACR, FNASCI, is professor of radiology and internal medicine at the University of Maryland School of Medicine and the director of thoracic imaging and vice chairman for clinical affairs at the University of Maryland Medical Center. He received his training at Columbia-Presbyterian Medical Center in New York. Dr White is certified by both the American Board of Radiology and the American Board of Internal Medicine. He has published widely in the areas of cardiothoracic radiology, including lung cancer screening and cardiac imaging. His current roles include chairman of the Implementation Strategies Task Group of the National Lung Cancer Round Table, vice chairman of the Lung Cancer Screening Registry of the American College of Radiology, and membership chairman of the Fleischner Society.

Chest radiography remains the most widely used imaging technique in radiology, but its limitations are well-known. Foremost among these is that a pulmonary nodule representing lung cancer may be present on the image but overlooked by the interpreter. Over several decades, a variety of strategies have been employed in an attempt to mitigate this shortcoming, including double reading, comparison to prior radiographs, radiologist training to avoid satisfaction of search errors, and the use of computer-aided detection (CAD) (1). Most recently, artificial intelligence methods have been devised that use deep learning techniques and complex algorithms to optimize the available data and achieve a desired result. In this issue of Radiology: Cardiothoracic Imaging, Nam et al describe the application of a deep learning–based algorithm to detect subtle lung cancer on posteroanterior (PA) chest radiographs (2).

The authors used a commercially available deep learning algorithm (Lunit Insight) previously trained with more than 40 000 chest radiographs including more than 9000 with malignant nodules. The algorithm outputs a probability of between 0 and 1 that a lung nodule is present on the chest radiograph and localizes the nodule on the chest radiograph. Thresholding within this 0–1 range can be used to designate a chest radiograph as nodule present or nodule absent. The authors selected the manufacturer-recommended value of 0.15 as a cutoff.

Of 1001 chest radiographs of patients with lung cancer obtained prior to biopsy at their institution, a lung nodule was not reported by the interpreting radiologist on the most recent or an earlier chest radiograph in 168 patients. Each of the 168 patients underwent a prebiopsy chest CT scan which served as the ground truth, and a total of 187 nodules were detected (average size, 2.3 cm). The CT scan was correlated with the chest radiograph to determine the location of the overlooked nodules. Two thoracic radiologists graded the conspicuity of each of the potentially overlooked nodules on chest radiographs using a 0–4 scale (4 being most conspicuous). Forty-three percent of nodules were classified as category 3 or 4. The study was supplemented with 50 chest radiographs that were demonstrated to be nodule free with chest CT scan, resulting in a final study group consisting of 218 chest radiographs.

The stand-alone performance of the deep learning–based algorithm was measured both on a per-radiograph and per-nodule basis, and a reader performance test was undertaken. For the latter, four thoracic radiologists who were not aware of the ratio of the positive-to-negative cases initially reviewed the chest radiographs and evaluated for the presence of a pulmonary nodule using a five-point confidence scale (5 being highest confidence). In a second session, the readers reviewed their initial selection and modified or let stand their decisions based on input from the deep learning–based algorithm. Stand-alone and reader-based performance was assessed with area under the receiver operating characteristic curves (AUROCs) for chest radiographs and area under the jackknife free-response ROC curves (AUFROCs) for pulmonary nodules.

With respect to per-radiograph classification, the stand-alone performance of the algorithm was superior to that of the radiologist readers (AUROC of 0.899 vs range of 0.619–0.651). With use of the algorithm, there was significant improvement of the readers to an AUROC range (0.636 vs 0.688). It is noteworthy that even with the aid of the algorithm, readers fell short of the stand-alone performance of the algorithm. Relative to nodule detection performance the results were analogous, even for nodules partly obscured by overlapping shadows. Conspicuity and size of the nodule did not substantially affect the algorithm detection rate, whereas readers detected more conspicuous, larger nodules at a higher rate. However, even for more conspicuous nodules in the 2–4 range, the algorithm outperformed the readers. The false-positive rate of the algorithm was similar to that of the readers.

The findings of the study point to the potential for a dramatic advance in our ability to detect pulmonary nodules at chest radiography using deep learning methodology. This approach appears to substantially exceed the performance of previously available CAD systems. The improvement seems to be based not only on the ability of the algorithm to find nodules overlooked on chest radiographs but also to reduce the per-radiograph false-positive rate, which has been a limitation of CAD systems. Particularly valuable is that the algorithm outperformed the radiologists in detecting lesions designated as category 3, described as moderately visible, as these likely reflect an area with the greatest possibility of improvement for interpreters. Much of the improvement arises from the superior performance of the deep learning–based algorithm in detecting nodules in areas of known difficulty on chest radiographs, particularly in the upper lobes and near the hila.

There are several important items to consider in light of this study, some of which are discussed by the authors. The study was enriched with a large fraction of patients who had nodules (n = 168) as compared with those who did not (n = 50), which does not reflect a real-world situation but was necessary for a baseline assessment of the algorithm. It would be interesting and ultimately more important to repeat the study with a nodule-to-nonnodule chest radiographic ratio that more closely mimics the situation encountered in clinical practice. An additional feature of the study design is that the reading radiologists were specialists in thoracic imaging. Most chest radiographs are interpreted by nonspecialist radiologists. One might expect that they would underperform compared with thoracic radiologists, but this is not necessarily the case and would need to be investigated. Even if the performance of nonspecialists at baseline is inferior to that of thoracic radiologists at baseline, they would show a greater improvement than thoracic radiologists using the algorithm, which would be a gratifying result.

As the authors note, the study was performed using PA radiographs. While the majority of findings are noted on PA radiographs, the incremental role of lateral radiographs which typically accompany PA radiographs should be considered. Lateral radiographs may be particularly useful in the lower lung regions, which in this study were more likely to be neglected on PA radiographs by readers. The study also does not address the role of the deep learning–based algorithm for anteroposterior chest radiographs. Many of these are obtained in the intensive care setting where the relevance of nodule detection is limited. However, some anteroposterior chest radiographs are acquired in the emergency department setting where such an algorithm may be of benefit.

Beyond the study design, there are additional considerations that may play into the value of such a deep learning–based algorithm. One is the willingness of readers to accept the results of the algorithm. An intriguing finding of the study is that the radiologist readers often did not accept potential nodules marked by the algorithm that proved at CT to be actual nodules. This accounts for much of the superior performance of the stand-alone algorithm. Lack of radiologist acceptance was particularly evident for less conspicuous nodules and nodules in the lower lungs. To maximize the benefit of the deep learning–based algorithm, it would be critical to understand why the readers failed to accept these “real” findings and adjust their behavior. A second consideration is that only a single algorithm was evaluated in this study. It would be useful to do a side-by-side comparison of this algorithm with similar algorithms to determine their strengths and weaknesses in detecting nodules (3).

Assuming that more complete vetting of the deep learning–based algorithm approach demonstrates its worth, a final challenge is how to underwrite its cost. Although CAD-based systems did not achieve wide clinical use, in part because of performance issues as noted by the authors, the lack of reimbursement in most locales was certainly a contributory factor. If this algorithm indeed proves far superior to the capability of readers with low false-positive results, it would provide a clearer pathway to investing in impetus for further research to determine the precise value of this and other deep learning–based algorithms for nodule detection at chest radiography.