ChatGPT for patient education: an evolving investigation

The authors would first like to thank Kleebayon et al¹ for taking the time to critically read our recent investigation.² Much of Kleebayon et al’s criticism stems from our investigation’s novelty. Namely, it is suggested that our prompts could be more varied, responses could be more rigorously graded, and comparisons could be drawn to additional, already-existing patient education resources. Indeed, we agree with such criticisms and accept the inherent limitations of our current investigation. Our study marks the first investigation into the utility of large language model chatbots for patient education in sleep medicine and surgery literature.² We acknowledge the recent expansion of this topic, and we hope that our publication can serve as a baseline study for future investigations to build upon by developing increasingly complex methodologies or chatbot comparisons.

Currently, ChatGPT (GPT-3.5) appears to be the most publicly used artificial intelligence (AI) chatbot, with over 100 million active users and 9 billion all-time page visits.³ Since data collection for our publication, multiple chatbots now exist in the public mainstream: GPT-4, Google Bard, and Bing Chat. Moreover, given the continuous software iterations of GPT-3.5, it is likely that the GPT-3.5 responses used in our publication may significantly differ at the time this response letter is published. Future investigations may seek to compare our GPT-3.5 responses (using the March 23, 2023, version) to newer GPT-3.5 versions as well as more recent AI chatbots.

Kleebayon et al notes that our use of four prompts may not “reflect the full spectrum of variables in real-world clinical situations.”¹ Indeed, our use of four prompts was an attempt to balance clinical utility with study simplicity. To our knowledge, our investigation was the first publication to demonstrate that “prompting” does significantly affect chatbot responses. Namely, prompting can decrease the response reading grade level (albeit still above recommended patient reading levels)⁴ and provide users with references from medical literature. Future investigations may seek to feed ChatGPT with increasingly complex prompts to generate responses even more clinically useful for patients, such as explicitly asking the chatbot to provide responses at or below the sixth-grade reading level.

Finally, we agree with Kleebayon et al’s sentiment that AI chatbots should be rigorously vetted for correctness and potential biases prior to recommending their use. Recent investigations have raised concern over the integrity and accuracy of ChatGPT specifically regarding scientific writing.⁵ OpenAI’s published internal reports do document a defined rate of AI hallucination, a concerning feature our publication discussed.² Future investigations may seek to critically assess the legitimacy of references provided by ChatGPT and compare these findings to that of future timepoints and other AI chatbots.

DISCLOSURE STATEMENT

All authors have seen and approved the manuscript. Institution where work was performed: Department of Otolaryngology–Head and Neck Surgery, Thomas Jefferson University Hospitals. This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. The authors report no conflicts of interest.

ABBREVIATIONS

AI

artificial intelligence