Abstract
Background: The emergence of ChatGPT, a state-of-the-art language model developed by OpenAI, has introduced a novel avenue for patients to seek medically related information. This technology holds significant promise in terms of accessibility and convenience. However, the use of ChatGPT as a source of accurate information enhancing patient education and engagement requires careful consideration. The objective of this study was to assess the accuracy and reliability of ChatGPT in providing information on the indications and management of complications post-tympanostomy, the most common pediatric procedure in otolaryngology. Methods: We prompted ChatGPT-3.5 with questions and compared its generated responses with the recommendations provided by the latest American Academy of Otolaryngology–Head and Neck Surgery Foundation (AAO-HNSF) “Clinical Practice Guideline: Tympanostomy Tubes in Children (Update)”. Results: A total of 23 responses were generated by ChatGPT against the AAO-HNSF guidelines. Following a thorough review, it was determined that 22/23 (95.7%) responses exhibited a high level of reliability and accuracy, closely aligning with the gold standard. Conclusion: Our research study indicates that ChatGPT may be of assistance to parents in search of information regarding tympanostomy tube insertion and its clinical implications.
Keywords: ChatGPT, OpenAI, artificial intelligence, tympanostomy tube insertion, otolaryngology, myringotomy
1. Introduction
Chat GPT is a conversational artificial intelligence (AI) tool designed to be a resource for individuals seeking information on a wide range of topics, including medical issues. One of the tool’s fundamental strengths is its ability to elucidate complex medical terminology and describe complex physiology in a succinct manner [1,2,3,4]. However, concerns pertaining to the accuracy and reliability of the instruction provided by ChatGPT, specifically in matters relating to healthcare, have been raised [5,6,7]. As several parents of children undergoing myringotomy tube insertion in our practice have recently admitted using ChatGPT, the goal of our study was to evaluate the accuracy and efficacy of ChatGPT as a resource for patients seeking information on the most performed otolaryngology surgery: tympanostomy tube insertion.
We hypothesized that ChatGPT is capable of providing comprehensive answers to questions commonly asked by parents about tympanostomy tube insertion. In order to assess the performance of the application, we evaluated the responses generated by ChatGPT in comparison with the highest standards of healthcare practice.
The findings of this study can contribute significantly to the ongoing efforts to leverage AI in patient-centered care while also ensuring the provision of high-quality medical information to patients and caregivers. We hope to promote its responsible use as a valuable resource guiding parents to make informed decisions about the care of their child.
It is more important than ever to explore the incorporation of supplementary resources such as AI to enhance the effectiveness of patient–provider communication and the quality of services delivered in the healthcare system.
2. Materials and Methods
A list of predetermined questions regarding surgical indications and complications post-tympanostomy insertion was presented to ChatGPT by two users on separate devices using the ChatGPT-3.5 version, which is free of charge and readily available to the majority of parents. To prevent any biases from the prior question, a new ChatGPT session was started for each prompt. Responses collected by the two users were recorded in our database to be compared with the statements published in the latest “American Academy of Otolaryngology–Head and Neck Surgery Foundation Clinical Practice Guideline: Tympanostomy Tubes in Children (Update)” [8]. Two independent otolaryngologists evaluated if the answer provided by the chatbot was aligned with the latest guidelines, cited the verbatim guidelines, and explicitly referenced the AAO-HNSF, as well as communicated the importance of seeking a clinician’s input. The inter-rater reliability was assessed using Cohen’s Kappa test. To confirm the consensus on all 4 outputs, the responses of both parties were reviewed by the senior author (SJD). A visual representation of the workflow is depicted in Figure 1.
Figure 1.
Workflow diagram.
3. Results
A total of 23 responses generated by ChatGPT were assessed by two otolaryngologists from distinct clinical settings (Table 1). There was perfect agreement between the two judges (Cohen’s K = 100%) that ChatGPT provided comprehensive recommendations that adequately matched the information highlighted by the guidelines.
Table 1.
ChatGPT prompts.
| Action Point/Statement (S) | Question Asked on ChatGPT (Q) | Assessment of ChatGPT’s Response (AR) | |||||
|---|---|---|---|---|---|---|---|
| Closely Aligned with Guidelines | Cited AAOHNS * | Explicitly Referenced AAOHNSF * | Recommended Seeking a Clinician | Agreement between Evaluators | |||
| S 1: OME of short duration | Q 1: Should physicians perform a tympanostomy tube insertion in children with otitis media effusion (OME) of short duration? | AR 1: | ✓ | ✓ | Y | ||
| S 2: Hearing evaluation | Q 2: When should clinicians obtain a hearing evaluation for children with otitis media effusion? | AR 2: | ✓ | ✓ | ✓ | Y | |
| S 3: Chronic bilateral OME with hearing difficulty | Q 3: What should clinicians offer for children with chronic bilateral Otitis media effusion (OME) with hearing difficulty for more than 3 months? | AR 3: | ✓ | ✓ | ✓ | Y | |
| S 4: Chronic OME with symptoms | Q 4: What should an otolaryngologist offer to patients with chronic OME with symptoms (balance/vestibular problems, poor school performance, behavioural problems, ear discomfort, reduced quality of life)? | AR 4: | ✓ | ✓ | Y | ||
| S 5: Surveillance of chronic OME | Q 5: What is the surveillance period required for patients with chronic otitis media effusion who do not receive tympanostomy tubes? | AR 5: | ✓ | ✓ | ✓ | Y | |
| S 6: Recurrent AOM without MEE | Q 6: Should clinicians perform tympanostomy tube insertion for patients with recurrent acute otitis media (AOM) without middle ear effusion (MEE)? | AR 6: | ✓ | ✓ | Y | ||
| S 7: Recurrent AOM with MEE | Q 7: Should clinicians perform tympanostomy tube insertion for patients with recurrent acute otitis media AOM with unilateral or bilateral middle ear effusion? | AR 7: | ✓ | Y | |||
| S 8: At-risk children | Q 8: Should clinicians determine if a child with recurrent AOM or with OME of any duration is at increased risk for speech, language, or learning problems from otitis media because of baseline sensory, physical, cognitive, or behavioral factors? | AR 8: | ✓ | ✓ | Y | ||
| S 9: Long-term tubes | Q 9: Should clinicians insert long-term tubes as initial surgery for children who meet criteria for tube insertion? | AR 9: | ✓ | ✓ | Y | ||
| S 10: Adjuvant adenoidectomy | Q 10: Should clinicians perform adenoidectomy as an adjunct to tympanostomy tube insertion for children to potentially reduce future risk of recurrent otitis media? | AR 10: | ✓ | ✓ | Y | ||
| S 11: Perioperative education | Q 11: Should clinicians offer education to caregivers of children with tympanostomy in the perioperative section? | AR 11: | ✓ | ||||
| S 12: Perioperative ear drops | Q 12: Should clinicians prescribe peri-operative ear drops after tympanostomy tube placement? | AR 12: | ✓ | ✓ | Y | ||
| S 13: Acute tympanostomy tube otorrhea | Q 13: Should clinicians prescribe topical antibiotic ear drops for children with uncomplicated acute tympanostomy tube otorrhea? | AR 13: | ✓ | ✓ | ✓ | Y | |
| S 14: Acute tympanostomy tube otorrhea | Q 14: Should clinicians prescribe oral antibiotics for children with uncomplicated acute tympanostomy tube otorrhea? | AR 14: | ✓ | ✓ | Y | ||
| S 15: Water precautions | Q 15: Should clinicians encourage routine prophylactic water precautions (earplugs or headbands or avoidance of swimming) for children with tympanostomy tubes? | AR 15: | ✓ | ✓ | ✓ | Y | |
| S 16: Follow-up | Q 16: How long should a surgeon wait before examining the ears of a child who had tympanostomy tube insertion? | AR 16: | ✓ | ✓ | Y | ||
| S 17: Comparison of Short- vs. Long-term Tympanostomy Tubes | Q 17: Comparison of Short vs. Long-Term tympanostomy tubes in terms of duration of ventilation, indications, other common uses, advantages, disadvantages | AR 17: | ✓ | ✓ | Y | ||
| S 18: Ear tubes recommendation | Q 18: Why are ear tubes recommended? | AR 18: | ✓ | ✓ | Y | ||
| S 19: Ear tube longevity | Q 19: How long will my child’s ear tubes last? | AR 19: | ✓ | ✓ | Y | ||
| S 20: Post-op follow-up | Q 20: When does my child need to be seen again after the tubes are placed? | AR 20: | ✓ | ✓ | Y | ||
| S 21: Complications associated with ear tubes | Q 21: What are the possible complications or problems, or ear tubes? | AR 21: | ✓ | ✓ | Y | ||
| S 22: Water exposure | Q 22: Does my child need ear plugs when exposed to water? | AR 22: | ✓ | ✓ | Y | ||
| S 23: Calling ENT | Q 23: When should I call the otolaryngologist (ENT) for my children who had a tympanostomy with ear tubes? | AR 23: | ✓ | ✓ | Y | ||
* AAO-HNSF guideline: American Academy of Otolaryngology–Head and Neck Surgery Foundation “Clinical Practice Guideline: Tympanostomy Tubes in Children (Update)”; (✓) is used to indicate, “yes; this criteria has been met”; (Y) indicates “yes, there is a perfect agreement between the two evaluators” on the quality of the ChatGPT’s answer.
Both raters agreed that 22/23 responses (95.7%) presented high-quality information closely aligning with the guidelines. Recommendations of one response, pertaining to the need of tympanostomy tube insertion in children with recurrent acute otitis media (AOM) who have unilateral or bilateral middle ear effusion (MEE), differed from the guidelines. The AI tool was not committal as to the treatment provided, stating: “Decision for tympanostomy tube insertion in these cases may depend on factors such as the severity and duration of effusion, impact on hearing, associated symptoms, and overall clinical picture”, whereas the guidelines stipulated that children should be offered bilateral tympanostomy tubes.
On 2/23 occasions, ChatGPT cited the verbatim AAO-HNSF guideline in its response. Regarding the action point concerning acute tympanostomy tube otorrhea, the question asked on ChatGPT, “Should clinicians prescribe topical antibiotic ear drops for children with uncomplicated acute tympanostomy tube otorrhea?”, cited verbatim the guideline. According to the guidelines from the American Academy of Otolaryngology-Head and Neck Surgery (AAO-HNS), clinicians should consider topical antibiotic ear drops for children with uncomplicated acute tympanostomy tube otorrhea. On a second occasion, regarding the action point concerning water precautions, the question asked on ChatGPT, “Should clinicians encourage routine prophylactic water precautions (earplugs or headbands or avoidance of swimming) for children with tympanostomy tubes?”, cited verbatim the guideline. According to the guidelines from the American Academy of Otolaryngology-Head and Neck Surgery (AAO-HNS), routine prophylactic water precautions, such as the use of earplugs or headbands or avoidance of swimming, are generally not recommended for children with tympanostomy tubes.
In total, 3/23 of ChatGPT’s responses referenced the AAOHNS guidelines explicitly. ChatGPT’s responses made at least one direct reference to consult with a specialist or healthcare provider when considering all available treatment options and inherent risk factors. These references include the following key examples: “It’s important to have open and thorough discussions with your child’s healthcare provider, usually an ear, nose, and throat specialist (otolaryngologist), to determine the most appropriate course of action for your child”. Another frequently encountered recommendation goes as follows: “It is important for parents to have open communication with the child’s healthcare provider, discuss concerns, and ask questions to ensure a comprehensive understanding of the available options and make informed decisions about their child’s care”.
The evaluators found that the information provided by ChatGPT exceeded the depth of content found in the guidelines and was easily understandable and tailored to meet the needs of parents who may not be familiar with medical terminology.
A comprehensive breakdown of all responses generated by ChatGPT is available as Supplementary Data in Table S1.
4. Discussion
ChatGPT is being increasingly used by patients seeking medical opinions [9,10]. Despite the myriad of benefits offered by such technological advancements, it is imperative to ensure the veracity and accuracy of the information provided [9,11]. The fundamental responsibility remains in guaranteeing that the information delivered to patients is trustworthy and aligned with the highest standards of healthcare practice [12,13,14,15].
In this study, the chatbot differed from the guideline in only 1 question out of 23. While the answer provided by ChatGPT was non-committal as to the treatment provided, it remains accurate, and it does not impact safety.
It is important to note that ChatGPT’s answers are subject to modification over time as it acquires new knowledge through learning. The potential risk arises from the reliance on ChatGPT’s capacity to remain updated with the latest guidelines. Implementing follow-up prompts could have provided ChatGPT the opportunity to deliver more direct and accurate responses. A limitation of ChatGPT lies in its ability to capture the tone or context of a prompt and the nuances of a query. Future AI-based clinical decision-making tools must address these limitations.
The current trend toward close integration with search engines and web-based applications is already changing the face of healthcare. It is anticipated that more caregivers will adopt ChatGPT or start using comparable AI models attempting to gain a better understanding of the surgical indications, postoperative complications, and recovery time of various medical treatments and interventions.
In this study, we conducted an evaluation of ChatGPT’s capacity to deliver high-quality responses aimed at augmenting parental education concerning tympanostomy tube insertion in pediatric patients. Our research yields significant insights into its potential advantages for improving patient communication and underscores the importance of its responsible utilization. In light of these findings, it is imperative to consider future implications in medical practice, including the ongoing challenges faced by clinicians and the existing regulatory landscape in healthcare. To further explore the novel application of ChatGPT in otolaryngology practice, additional research should be conducted.
5. Conclusions
Our study suggests that ChatGPT can help parents seeking answers to tympanostomy tube insertion. The comprehensiveness of ChatGPT’s responses is evident in a comparison with the AAOHNSF guideline, where the tool provided predominantly reliable and extended answers, often including ancillary facts and references to support its responses. Although there was a slight discrepancy between the official guideline statement and ChatGPT response in one question, there would be no risk to the child if parents were to follow ChatGPT’s recommendation. In fact, it has helped us develop a pamphlet addressing frequently asked questions by parents. Further research is required to explore the potential of ChatGPT and similar AI tools in improving healthcare delivery and outcomes.
Supplementary Materials
The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/children10101634/s1. Table S1: Comprehensive breakdown of ChatGPT generated responses compared to AAO-HNSF guideline.
Author Contributions
Conceptualization and methodology, A.M. and S.J.D.; data curation and analysis, A.M. and A.C.-B.; evaluation, O.O. and M.K.A.; validation, S.J.D.; writing—original draft preparation, A.M.; writing—review and editing, A.M., A.C.-B., O.O., M.K.A. and S.J.D.; visualization, A.C.-B.; supervision, S.J.D. All authors have read and agreed to the published version of the manuscript.
Institutional Review Board Statement
Ethical review and approval were waived for this study by the McGill University Faculty of Medicine and Health Sciences Research Ethics Office on 18 April 2023, in accordance with the national legislation and the institutional requirements.
Informed Consent Statement
Not applicable.
Data Availability Statement
The authors confirm that the data supporting the findings of this study are available within the article and its Supplementary Data file.
Conflicts of Interest
The authors declare no conflict of interest.
Funding Statement
This research received no external funding.
Footnotes
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.
References
- 1.Liévin V., Egeberg Hother C., Winther O. Can large language models reason about medical questions? arXiv. 2022 doi: 10.48550/arXiv.2207.08143.2207.08143 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Hariri W. Unlocking the potential of ChatGPT: A comprehensive exploration of its applications, advantages, limitations, and future directions in natural language processing. arXiv. 2023 doi: 10.48550/arXiv.2304.02017.2304.02017 [DOI] [Google Scholar]
- 3.Carlbring P., Hadjistavropoulos H., Kleiboer A., Andersson G. A new era in Internet interventions: The advent of Chat-GPT and AI-assisted therapist guidance. Internet Interv. 2023;11:100621. doi: 10.1016/j.invent.2023.100621. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Sallam M. ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns. Healthcare. 2023;11:887. doi: 10.3390/healthcare11060887. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Májovský M., Černý M., Kasal M., Komarc M., Netuka D. Artificial Intelligence Can Generate Fraudulent but Authentic-Looking Scientific Medical Articles: Pandora’s Box Has Been Opened. J. Med. Internet Res. 2023;25:e46924. doi: 10.2196/46924. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Vaishya R., Misra A., Vaish A. ChatGPT: Is this version good for healthcare and research? Diabetes Metab. Syndr. 2023;17:102744. doi: 10.1016/j.dsx.2023.102744. [DOI] [PubMed] [Google Scholar]
- 7.Alkaissi H., McFarlane S.I. Artificial Hallucinations in ChatGPT: Implications in Scientific Writing. Cureus. 2023;15:e35179. doi: 10.7759/cureus.35179. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Rosenfeld R.M., Tunkel D.E., Schwartz S.R., Anne S., Bishop C.E., Chelius D.C., Hackell J., Hunter L.L., Keppel K.L., Kim A.H., et al. Clinical Practice Guideline: Tympanostomy Tubes in Children (Update) Otolaryngol. Head Neck Surg. 2022;166:S1–S55. doi: 10.1177/01945998211065662. [DOI] [PubMed] [Google Scholar]
- 9.Ortel C. The Rise of AI Chatbots in Healthcare: A Comparison of ChatGPT and Physicians in Responding to Patient Questions. LinkedIn. [(accessed on 10 August 2023)]. 1 May 2023. Available online: https://www.linkedin.com/pulse/rise-ai-chatbots-healthcare-comparison-chatgpt-physicians-ortel/
- 10.Nov O. Putting ChatGPT’s Medical Advice to the (Turing) Test. arXiv. 2023 doi: 10.2139/ssrn.4413305. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.American Medical Association. Unger T., Ehrenfeld J. ChatGPT, AI in Health Care and the Future of Medicine with AMA President Jesse Ehrenfeld, MD, MPH. AMA. [(accessed on 10 August 2023)]. 20 July 2023. Available online: https://www.ama-assn.org/practice-management/digital/chatgpt-ai-health-care-and-future-medicine-ama-president-jesse.
- 12.Nature Medicine Will ChatGPT transform healthcare? Nat. Med. 2023;29:505–506. doi: 10.1038/s41591-023-02289-5. [DOI] [PubMed] [Google Scholar]
- 13.Bates D.W., Levine D., Syrowatka A., Kuznetsova M., Craig K.J.T., Rui A., Jackson G.P., Rhee K. The potential of artificial intelligence to improve patient safety: A scoping review. NPJ Digit. Med. 2021;4:54. doi: 10.1038/s41746-021-00423-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Parviainen J., Rantala J. Chatbot breakthrough in the 2020s? An ethical reflection on the trend of automated consultations in health care. Med. Health Care Philos. 2022;25:61–71. doi: 10.1007/s11019-021-10049-w. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Minssen T., Vayena E., Cohen I.G. The Challenges for Regulating Medical Use of ChatGPT and Other Large Language Models. JAMA. 2023;330:315–316. doi: 10.1001/jama.2023.9651. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
The authors confirm that the data supporting the findings of this study are available within the article and its Supplementary Data file.