Exploring the Role of Artificial Intelligence Chatbots in Preoperative Counseling for Head and Neck Cancer Surgery

Abstract

Objective:

To evaluate the potential use of artificial intelligence (AI) chatbots, such as ChatGPT, in preoperative counseling for patients undergoing head and neck cancer surgery.

Study Design:

Cross-Sectional Survey Study.

Setting:

Single institution tertiary care center.

Methods:

ChatGPT was used to generate presurgical educational information including indications, risks, and recovery time for five common head and neck surgeries. Chatbot-generated information was compared with information gathered from a simple browser search (first publicly available website excluding scholarly articles). The accuracy of the information, readability, thoroughness, and number of errors were compared by five experienced head and neck surgeons in a blinded fashion. Each surgeon then chose a preference between the two information sources for each surgery.

Results:

With the exception of total word count, ChatGPT-generated pre-surgical information has similar readability, content of knowledge, accuracy, thoroughness, and numbers of medical errors when compared to publicly available websites. Additionally, ChatGPT was preferred 48% of the time by experienced head and neck surgeons.

Conclusion:

Head and neck surgeons rated ChatGPT-generated and readily available online educational materials similarly. Further refinement in AI technology may soon open more avenues for patient counseling. Future investigations into the medical safety of AI counseling and exploring patients’ perspectives would be of strong interest.

Level of evidence:

N/A.

INTRODUCTION

The treatment of head and neck cancer (HNC) has significant long-term physical, functional, and psychological effects on patients. Studies have shown that appropriate and timely patient education and counseling can improve patient satisfaction and overall quality of life in cancer survivors.¹ Unfortunately, up to 93% of cancer patients report a lack of information before and after cancer treatment,² which is correlated with post-treatment depression, anxiety, and reduced quality of life in HNC patients.³ Furthermore, a longitudinal study of HNC patients found that as many as four in five patients experienced decision regret 3 to 6 months after their cancer treatment.⁴ Despite its importance, preoperative counseling remains a significant challenge in the HNC population due to time constraints during encounters and the complexity of treatment.

Artificial intelligence (AI) has the potential to overcome critical barriers to preoperative counseling in HNC patients. AI is increasingly employed in medicine, ranging from diagnostics and treatment planning to drug development, thanks to its unparalleled capacity for pattern recognition using large volumes of data. In otolaryngology, for instance, AI and machine learning have been used for screening, diagnosis, and treatment decision in rhinology,⁵ otology,⁶ laryngology,⁷ and head and neck oncology.⁸ We have also previously demonstrated the utility of AI for predicting head and neck melanoma patients with a low risk of nodal metastasis⁹ and identifying patients with oral cavity squamous cell carcinoma at risk for occult nodal disease.¹⁰

Recently, OpenAI released its latest conversational AI chatbot, ChatGPT.¹¹ The software was developed using deep learning and natural language processing.¹² The platform was trained on a large amount of information from textbooks, websites, and scientific literature, totaling terabytes of data. The purpose of the platform is to develop a publicly available AI capable of understanding and generating human-like responses based on user prompts. Importantly, chatbots are freely available to the general public, and their interfaces are designed to be more user-friendly. In this study, we evaluated the potential and usefulness of AI chatbots, specifically ChatGPT, in pre-operative patient counseling for HNC procedures. We hypothesize that AI chatbots can provide timely and accurate information on surgical indications, risks, and recovery for various head and neck procedures.

METHODS

The following study consisted of surveying head and neck surgeons at our institution, and it was exempted from the Institutional Review Board at the University of Kansas Medical Center.

ChatGPT Bot

ChatGPT (https://chat.openai.com/chat) was used to generate presurgical patient educational information for five common head and neck surgeries, including glossectomy, parotidectomy, neck dissection, total laryngectomy, and free tissue transfer for head and neck reconstruction. Surgical information including indications, risks, and recovery time, was separately queried. The initial query prompt used was “explain the [surgical indications/risks and complications/recovery time] for [procedures]”. In addition, prompt modifiers such as “explain… to a medical professional/to a 6th grader” were added to examine the ability of ChatGPT to generate differential responses based on user input. Responses were recorded and used to construct individual excerpts for each separate procedure listed above for analysis and review (see supplemental material S1 for an example). For comparison, the most widely used search engine, Google, was used to directly search for patient educational information for each procedure separately. Key terms such as “total laryngectomy”, “parotidectomy”, “neck dissection”, “glossectomy”, and “free tissue transfer for the head and neck” were used. Surgical educational information from the first publicly available website, excluding any peer-reviewed scholarly articles, was extracted and formatted similarly to its corresponding ChatGPT counterpart. This was separately labeled as “Web”. Three of the five websites were from R1 Research Institution from (ex: Cleveland Clinic), while the other two were from other tertiary centers that also routinely perform such procedures. Together, four distinct groups were created: (1) Web, (2) ChatGPT-Native, (3) ChatGPT-Professional, and (4) ChatGPT-6th grade. To ensure uniformity in this process, the lead author J.C.L, independently extracted information, which also ensured that our head and neck surgeons were kept blind to the process to prevent any bias for downstream analysis.

Readability Scoring

To assess readability, Flesch Reading Ease Score¹³ and Gunning-Fog Index,¹⁴ two widely utilized readability scales;), were used to calculate the respective difficulty of reading each generated excerpts. The Flesch Reading Ease Score is a 100-point scale. The higher the score equates to easier comprehension. A reading score of 60 to 70 is roughly equivalent to 8–9^th grade reading comprehension level.¹⁵ Similarly, the Gunning-Fog index corresponds to US academic grade reading comprehension ability.¹⁶ It is calculated by using the following formula: 0.4 × [(word/sentences) + 100 × (complex words /total words)], where complex words are defined as words containing three or more syllables.¹⁶ Unlike the Flesch Reading Ease Score, the lower the score on the Gunning Fox Index equates to easier comprehension. Word count was also recorded from the raw, unblinded output of each source.

Assessment by Head and Neck Surgeon

Sample excerpts were delabeled and blinded for independent review by five head and neck surgeons. Specifically, only ChatGPT-6th grade excerpts were used for direct side-by-side comparison with “Web” excerpts during the review process. Author J.C.L independently extracted each excerpt without altering the content information. Excerpts were standardized to bullet point format to help with blinding. For instance, ChatGPT-generated output all included bullet points while some web sources did not. Additionally, ChatGPT-generated output all included filler phrases, such as “it is important to discuss risks and benefits with your health care professional”, that were removed without changing the content of the excerpt. All bullet points were also formatted to standard complete sentences. Each correspondent was then asked to grade each excerpt on a scale of 1 to 10 in terms of accuracy and thoroughness, with 1 being not very accurate or thorough and 10 being very accurate or thorough, respectively. In addition, each respondent was also asked to record the numbers of medical errors. Finally, respondents were asked to choose their preferred excerpt for each procedure.

Intraclass Correlation Coefficient

To determine the inter-rater reliability for more than two raters, an intraclass correlation coefficient was calculated as described by Koo and Li.¹⁷ A score of less than 0.5 indicates poor reliability. A score between 0.5 and 0.75 indicates good moderate reliability. A score between 0.75 and 0.9 indicates good reliability. A score greater than 0.9 indicates excellent reliability.

Statistical Analysis

Analysis was performed using GraphPad Prism using One-way ANOVA, multiple t-tests, Wilcoxon Mann Whitney U test, or McNemar test where appropriate. Statistical significance was defined a priori as p value less than 0.05.

RESULTS

Four distinct types of educational information were generated for each of the five common head and neck surgeries: (1) Web, (2) ChatGPT-Native, (3) ChatGPT-Professional, and (4) ChatGPT-6th grade. The total word count, Flesch Reading Ease Score, and Gunning-Fog index were then compared using One-way ANOVA. Total word count of unedited excerpts generated from publicly available website was significantly lower than those generated from ChatGPT, regardless of prompts (Fig. 1A, p < 0.01). “Web” and “ChatGPT-6th grade” have very similar Flesch Reading Ease Scores (Fig. 1B). Furthermore, both scores were significantly higher when compared to that of “ChatGPT-Native” and “ChatGPT-Professional,” suggesting that “Web” and “ChatGPT-6th grade” are easier to read (Fig. 1B, p < 0.05). This was also confirmed by a separate readability scale using the Gunning-Fog index (Fig. 1C, p < 0.01).

Fig. 1.

Comparison of ChatGPT-generated texts to web browsing from freely available websites. (A) ChatGPT-generated tests are significantly wordier. (B and C) addition of prompt modifiers, such as “explain… using 6th grade reading comprehension” resulted in improved readability. (D) ChatGPT-generated texts have similar content of knowledge regardless of prompt modifiers. ns, not significant with p > 0.05; * p ≤ 0.05; ** p ≤ 0.01; ***, p ≤ 0.001.

Next, we were interested to see if input modifier would affect response from ChatPGT. For instance, would “explain… to a medical professional” contain additional critical information as compared to “explain… at a 6th-grade level.” We specifically investigated the risks and complications output for each group to determine whether different prompts affect the detail of output. We felt that procedure-specific risks and complications would offer the most direct and in-depth differentiation for content of knowledge generated. To achieve this, an experienced head and neck surgeon reviewed and approved a list of procedure-specific risks (supplemental material 2). These procedure-specific risks were extracted initially from the National Institute of Health StatPearls for each procedure and further reviewed by a senior head and neck surgeon. We then determined the percentage of time procedure-specific risks that discussed by each output. On average, web resources covered 54.3% of the procedure-specific risks, while ChatGPT-Native, ChatGPT-Professional, and ChatGPT-6th grade covered 42.9%, 40.0%, and 40.0%, respectively (Fig. 1D, p = 0.23), suggesting that prompt modifiers did not affect the content of knowledge discussed by the chatbot. Importantly, “ChatGPT-6th grade” and “Web” showed similar content of knowledge as well as readability, prompting us to select them for further analysis in the following section.

“Web” and “ChatGPT-6th grade” generated comprehensive excerpts that included indication, risks, and recovery were then selected for independent review by five different head and neck surgeons. The review process was blinded so that each surgeon was unaware of whether surgical information was taken from publicly available websites or generated by ChatGPT. For the purpose of blinding, the “Web” and “ChatGPT-6thgrade” were edited as outlined in the method section. Four different parameters were assessed: accuracy, thoroughness, number of medical errors, and surgeon preference. We found that surgical information generated by ChatGPT has similar medical accuracy when compared to publicly available websites (mean of 7.7 and 8.1, respectively; p = 0.42; Fig. 2A). ChatGPT-generated information has similar thoroughness (mean of 7.5 and 7.3 respectively; p = 0.76; Fig. 2B). Additionally, both groups have limited numbers of medical errors (mean of 1.2 and 1.0, respectively; p = 0.30; Fig. 2C). To determine inter-rater reliability from individual head and neck surgeons, an intraclass correlation coefficient was calculated based on composite scoring from each surgeon. The coefficient was 0.85, indicating good inter-rater reliability. Each surgeon was then asked to select personal preference between the two outputs. Overall, there was an even split between “Web” (52%) and “ChatGPT-6th grade” (48%). Finally, three out of five “Web” excerpts were extracted from R1 Research Institution websites. There was no difference in surgeon preference for these specific excerpts over ChatGPT-generated responses based on the McNemar test (p = 0.68, odds ratio 0.5).

Fig. 2.

Assessment of medical accuracy, thoroughness, and number of medical errors by head and neck surgeons. ChatGPT-generated texts scored similarly to publicly available websites in terms of accuracy (A), thoroughness (B), and number of medical errors (C). ns, not significant with p > 0.05; * p ≤ 0.05; ** p ≤ 0.01; *** p ≤ 0.001.

DISCUSSION

Many HNC patients report inadequate pre-operative counseling about treatment side effects and long-term impact on quality of life.³

Here we demonstrated the feasibility of using AI chatbots, specifically ChatGPT, to provide pre-surgical information for common head and neck procedures. Our results showed that ChatGPTG could provide generally accurate and thorough surgical information without significant medical errors. In addition, ChatGPT-generated responses were rated similarly to available online resources by experienced head and neck surgeons. Importantly, we showed that ChatGPT prompts could be tailored to successfully explain relatively complex medical knowledge in simple terms. Together, these findings suggest that the use of AI chatbot in pre-operative counseling may help to address critical barriers that exist in HNC patient education and counseling.

We found that general operative risks such as pain, bleeding, infection, and damage to nearby structures were well discussed by chatbots. We further measured depth of knowledge by investigating procedure-specific risks and complications and found that both ChatGPT responses and online resources presented surgical information with similar content of knowledge.

Many prior studies have demonstrated a positive correlation between multimedia use in pre-operative patient counseling and overall patient satisfaction and learning.^18,19 Admittedly, an AI chatbot may fall short when compared to an intentionally designed quality educational video. There are a couple key advantages to AI chatbot. The first and foremost is the dynamic and interactive nature of AI chatbot. Patients can freely direct questions according to their own desires. Secondly, the flexibility of a chatbot is superior to the linear nature of a video. To apply AI chatbots to a clinical setting, clinicians could consider integrating private dedicated computer stations for patients to use after the clinic visit or at home. Voice-integrated software extension such as “VoiceGPT” could be utilized to obtain voice input and output from the chatbot, facilitating a truly conversation-like interaction with AI. Alternatively, mobile applications with chatbot voice integration, such as ChatGPT Voice, could be used by more technically adept HNC patients. It should also be noted that AI chatbots are becoming more advanced and that pictures and video editing are now possible using chatbots. It is foreseeable that a “multimedia” chatbot that incorporates images and/or videos while responding to a patient prompt could serve as a powerful tool for patient education. Nevertheless, the impact of pre-operative education using AI chatbot on patient quality of life and satisfaction remains to be investigated.

Our study is limited by surveying only head and neck surgeons from a single institution and lacks assessment of patients’ perspectives. However, in practice, employing an AI chatbot in the clinical setting may prove to be difficult. For instance, AI chatbots sometimes display “Artificial Hallucination”, in which AI appears to confidently provide erroneous information.²⁰ Such misinformation could potentially be dangerous or at worst life-threatening. The legality of using AI in medicine is just beginning to be scrutinized.²¹ Should an adverse event arise, who should ultimately be responsible for the misrepresentation? Another shortcoming of the methodology is that only a single web resource was used for evaluation per prompt. It is conceivable that some web resources may be more much more detailed than others. We chose the first publicly available website to best mimic its accessibility from a patient’s point-of-view. Another potential drawback of using AI is content bias and the quality of input data.^22,23 Indeed, many references cited by ChatGPT are often made-up and untraceable when queried.²⁰ This is due to the fact that the language platform was designed to predict the most likely text output without truly understanding the semantics or truthfulness of the information.²⁴

At present, chatbots should not be recommended as sources of educational information for patients undergoing head and neck surgery. Importantly, chatbots should not be viewed as a replacement for quality face-to-face time with surgeons. Instead, AI chatbot may serve as a supplemental tool that would augment patients’ understanding of the nature of the surgery. Nevertheless, AI chatbot technology is in its infancy. With the increasing popularity of AI chatbots, many technology companies are racing to develop the next generation of “smarter” chatbot platform. With further advancement and refinement in AI technology, we may soon witness a new era of hybrid medical care where physicians work synergistically with AI in both the clinical and surgical settings to enhance health care delivery for HNC patients and beyond.

CONCLUSION

Presurgical information generated by chatbots is equivalent to google searches in terms of accuracy, thoroughness, and errors when assessed by experienced head and neck surgeons. Furthermore, input modifiers to alter readability showed significant changes on the readability scale without omitting the content of knowledge. Future studies should investigate the familiarity and ease of access to chatbots and how supplementation of pre-operative education using chatbots affects the quality of life in HNC patients.

Supplementary Material

Additional supporting information may be found in the online version of this article.