Evaluating ChatGPT's Accuracy in Lung Cancer Prevention and Screening Recommendations

Editor:

The surge of worldwide interest in dialogue-based artificial intelligence (AI) large language models (LLMs) such as ChatGPT is prompting evaluation of information available on such platforms. Early studies found that ChatGPT provides generally appropriate clinical recommendations in areas such as cardiovascular disease (1), antibiotic selection (2), and breast cancer (3). Lung cancer remains a leading cause of death in the United States (4), and we evaluated how ChatGPT would respond to fundamental questions about lung cancer prevention, screening, and the Lung CT Screening Reporting and Data System (Lung-RADS).

In February 2023, we submitted 25 questions to ChatGPT and generated three answers, as inconsistencies among answers are reported (1–3). Three fellowship-trained cardiothoracic radiologists (C.T.L., A.S., J.J.) graded each set of responses based on clinical judgment as (a) appropriate, (b) inappropriate, or (c) inconsistent (Table 1). Final appropriateness for each set was determined by consensus and were summarized using descriptive statistics.

Table 1:

Evaluation of ChatGPT Responses to Fundamental Questions about Lung Cancer Prevention and Screening Based on Assessment by Board-certified Cardiothoracic Radiologists

ChatGPT's responses were 84% (21 of 25) appropriate, 8% (two of 25) inappropriate, and 8% (two of 25) inconsistent (Table 2). Inappropriate responses were about the Lung-RADS tool and specifically Lung-RADS 3. Inconsistent responses were about planning lung cancer screening and the COVID-19 vaccine and Lung-RADS 0.

Table 2:

Examples of ChatGPT Recommendations Graded as Inappropriate and Inconsistent and Examination of Incorrect Information

Developed in early 2022, the current version of ChatGPT provided appropriate information about lung cancer 84% of the time and included the most recent U.S. Preventive Services Taskforce guidelines for screening eligibility (5) and performed comparably to similar studies (1,3) about cardiovascular disease (84%) and breast cancer (88%). This difference could be due to a relative lack of information about lung cancer versus cardiovascular disease and breast cancer in the model's training data set. Inappropriate and inconsistent recommendations about Lung-RADS could have serious implications for patients who may be falsely reassured about their study results. For example, ChatGPT wrote that Lung-RADS 3 confers a “moderate probability” of cancer, which is not the correct terminology as described in the lexicon. Limitations of the present study include the rapid development of LLMs such as ChatGPT.

While these findings provide additional support for the potential of AI in providing patients with relevant educational material, gaps remain in specific radiology topics and physician oversight remains imperative.