To the Editors:
Artificial intelligence (AI) radiomics-based tools demonstrate promise for indeterminate pulmonary nodule (PN) malignancy risk stratification.1 We performed a secondary analysis of a previous multi-reader, multi-case study2 to evaluate the effect of an AI tool on clinicians’ PN management decisions. The details of this study have been previously described.2 Briefly, 12 readers (6 radiologists, 6 pulmonologists) independently evaluated 300 indeterminate PN cases using solely axial CT chest scan imaging data. PNs were 5–30 mm in maximal diameter, and 50% were malignant. The AI tool assessed was the Lung Cancer Prediction Convolutional Neural Network (Virtual Nodule Clinic, version 2.0.0; Optellum Ltd, Oxford, UK).3,4 This tool calculates a Lung Cancer Prediction (LCP) score describing PN malignancy risk on a decile scale from 1 to 10 assuming a malignancy prevalence of 30%. For each case, each reader independently provided estimates of malignancy risk (0%–100%) and management decision (no follow-up, ≥6-month CT follow-up, 6-week to 6-month CT follow-up, immediate imaging follow-up, non-surgical biopsy, or surgical resection) before and after being shown the LCP score. We defined appropriate management of malignant PNs as non-surgical biopsy and surgical resection. For benign PNs, no follow-up or imaging follow-up were deemed appropriate. We classified immediate imaging as appropriate management for all PNs.
The median LCP score for malignant PNs was 9 (IQR, 8–10) and 5 (IQR, 2–7) for benign PNs (p < 0.001). Among malignant PNs, the average reader malignancy risk estimate was 60.2% (SD, 31.7%) without the AI tool compared to 69.0% (SD, 28.6%) with it (p < 0.001). Among benign PNs, the average reader malignancy risk estimate was 23.4% (SD, 28.1%) without the AI tool compared to 21.0% (SD, 26.9%) with it (p = 0.01). The distributions of management decisions are displayed in Figure 1. Overall, the proportion of cases with appropriate management decisions increased from 79.5% (SD, 5.7%) to 84.1% (SD, 6.6%) with AI (p = 0.008). Among malignant PNs, on average readers selected immediate imaging, biopsy, or surgical resection in 71.9% (SD, 14.0%) of cases without use of AI compared to 81.4% (SD, 13.7%) with the AI tool (p < 0.001). Among benign PNs, on average readers selected no action, short-term, long-term, or immediate follow-up imaging in 87.2% (SD, 10.4%) of cases without and 88.7% (SD, 11.1%) with the AI tool, respectively (p = 0.19).
FIGURE 1.
Distribution of management decisions with and without an artificial intelligence tool, stratified by pulmonary nodule diagnosis. In this multi-reader, multi-case study, 12 readers evaluated 300 CT chest scans with indeterminate pulmonary nodules and selected management decisions before and after computer-aided diagnosis with an artificial intelligence tool. Immediate imaging follow-up included contrast CT chest and positron emission tomography. Average reader management decisions are displayed by pulmonary nodule diagnosis: (A) benign; (B) malignant. AI, artificial intelligence.
We found that use of an AI tool was associated with an increase of the average proportion of cases with appropriate management decisions from 79.5% to 84.1%. This was largely driven by a 10 percentage point increase in malignant PNs appropriately managed with immediate imaging or tissue sampling. On the other hand, we did not observe a statistically significant difference in the management of benign PNs with use of the AI tool. Taken together, these results suggest that the previously demonstrated improvement in diagnostic accuracy with use of an AI tool may translate into meaningful changes in clinical management decisions and promote earlier diagnostic evaluation of malignant PNs, which may ultimately lead to increased timeliness of appropriate clinical treatment for thoracic malignancies.
ACKNOWLEDGEMENTS
Research funding:
Roger Y. Kim was supported by the National Cancer Institute of the National Institutes of Health (Award Number 5UM1CA221939). Jason L. Oke was partfunded by the NIHR Oxford Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust.
CONFLICTS OF INTEREST STATEMENT
This study was funded by Optellum Ltd. Roger Y. Kim reports research funding from Siemens outside of the submitted work. Anil Vachani reports research funding from MagArray, Inc., Broncus Medical, and PreCyte, Inc. and a consulting role with Novocure and Johnson & Johnson, outside of the submitted work. The remaining authors reported no relevant conflicts of interest.
Footnotes
HUMAN ETHICS APPROVAL DECLARATION
The use of deidentified imaging studies complied with Health Insurance Portability and Accountability Act guidelines, and the need for informed consent was waived by local institutional review boards.
DATA AVAILABILITY STATEMENT
Research data are not shared.
REFERENCES
- 1.Vliegenthart R, Fouras A, Jacobs C, Papanikolaou N. Innovations in thoracic imaging: CT, radiomics, AI and x-ray velocimetry. Respirology. 2022;27(10):818–33. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Kim RY, Oke JL, Pickup LC, Munden RF, Dotson TL, Bellinger CR, et al. Artificial intelligence tool for assessment of indeterminate pulmonary nodules detected with CT. Radiology. 2022;304(3):683–91. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Massion PP, Antic S, Ather S, Arteta C, Brabec J, Chen H, et al. Assessing the accuracy of a deep learning method to risk stratify indeterminate pulmonary nodules. Am J Respir Crit Care Med. 2020;202(2): 241–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Baldwin DR, Gustafson J, Pickup L, Arteta C, Novotny P, Declerck J, et al. External validation of a convolutional neural network artificial intelligence tool to predict malignancy in pulmonary nodules. Thorax. 2020;75(4):306–12. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
Research data are not shared.