Abstract
Objective
This study investigates ChatGPT's accuracy, readability, understandability, and actionability in responding to patient queries on sudden sensorineural hearing loss (SSNHL) in English and Spanish, when compared to Google responses. The objective is to address concerns regarding its proficiency in addressing medical inquiries when presented in a language divergent from its primary programming.
Study Design
Observational.
Setting
Virtual environment.
Methods
Using ChatGPT 3.5 and Google, questions from the AAO‐HNSF guidelines were presented in English and Spanish. Responses were graded by 2 otolaryngologists proficient in both languages using a 4‐point Likert scale and the PEMAT‐P tool. To ensure uniform application of the Likert scale, a third independent evaluator reviewed the consistency in grading. Readability was evaluated using 3 different tools specific to each language. IBM SPSS Version 29 was used for statistical analysis using one‐way analysis of variance.
Results
Across both languages, the responses displayed a native‐level language proficiency. Accuracy was comparable between sources and languages. Google's Spanish responses had better readability (effect size 0.35, P < .001), while Google's English responses were more understandable (effect size 0.67, P = .018). ChatGPT's English responses demonstrated the highest level of actionability (60%), though not significantly different when compared to other sources (effect size 0.47, P = .14).
Conclusion
ChatGPT offers patients comprehensive and guideline‐conforming answers to SSNHL patient medical queries in the 2 most spoken languages in the United States. However, improvements in its readability and understandability are warranted for more accessible patient education.
Keywords: artificial intelligence, health education, large language model, Spanish, sudden sensorineural hearing loss
ChatGPT is a free‐to‐use, widely accessible, artificial intelligence (AI) based large language model. 1 There is much enthusiasm regarding its potential applications in healthcare practice. The potential benefits of ChatGPT in health care practice have been evidently outlined; from streamlining clinical workflow to producing efficient discharge summaries. 2 In the climate of widespread internet access but limited health care access, particularly in rural or low‐income settings, many patients rely on online resources as their primary source of health information. 3 Given the overload of online health information, varying in accuracy, reliability, and literacy levels, it can be challenging for patients and their family to obtain valid, accessible, and easy‐to‐understand health information.
Sudden sensorineural hearing loss (SSNHL) is a relatively common complaint in otologic practices. 4 With hearing loss having notable impacts on an individual's quality of life, cognition, and mental health, 5 public health‐wide methods to addressing prevention, early detection and prompt treatment of hearing loss is a worthwhile option. Rates of hearing recovery are inversely correlated with consultation with a physician, with better hearing outcomes associated with less time elapsed between the onset of hearing loss and the beginning of treatment. 6
Several studies have investigated the validity of ChatGPT as a patient information source in various medical disciplines. 2 In otolaryngology, ChatGPT has been shown to offer relevant and accurate medical information, though it lacks response depth and has certain knowledge restrictions. 7 , 8 , 9 , 10 , 11 , 12 Additionally, ChatGPT has a language bias, 9 , 13 as its primary programming language is English. Its performance in other languages remains to be thoroughly examined. It is worthwhile to investigate whether ChatGPT can provide patients with high‐quality health education, in non‐English languages.
This study investigates the effectiveness of ChatGPT in responding to common patient queries on SSNHL in both English and Spanish. The objective of this study is to address concerns regarding its proficiency in addressing medical inquiries when presented in a language divergent from its primary programming, as well as to assess the literacy level of its responses.
Methods
Frequently asked questions and their answers on sensorineural hearing loss were retrieved from the American Academy of Otolaryngology–Head and Neck Surgery (AAO) plain language summary 14 developed for adult patients and based on the “Clinical Practice Guideline: Sudden Hearing Loss (Update).” 15
ChatGPT
The ChatGPT‐3.5 version, available free of charge, was used by 2 separate users. A new session was created using different accounts and IP addresses. Only the first response generated by ChatGPT was considered for grading as no prompts were provided. The questions were individually submitted to ChatGPT by an English‐native otolaryngologist. For the Spanish evaluation, the questions were translated by a native Spanish‐speaking otolaryngologist and then submitted to ChatGPT. To guarantee linguistic accuracy, a 2‐stage translation approach was implemented. Forward translation from English into Spanish, following backward translation to compare with the original text and if necessary, make final editing.
Questions were submitted to the Google search engine in English and Spanish, individually. We ignored advertisement results. Information provided in the answer box, a short text snippet at the top of the page, were included. If no answer box was provided, the question was excluded.
Accuracy
The answers were compared to those in the guidelines and rated on a 4‐point Likert scale by 2 separate users (Table 1). To ensure uniform application of the Likert scale between the 2 reviewers, a third independent evaluator reviewed the consistency in grading for answers where discrepancies were observed between the first and second reviewers. The Likert scale was developed by the research team as no existing AI grading scale evaluates answers in comparison to guidelines, and other available scales, such as those for clinical scenario evaluations, are not applicable to our study. 16
Table 1.
Likert Scale
| Score | Description | |
|---|---|---|
| 1 | Completely Inadequate | The response does not match the guidelines at all. The information is incorrect or completely irrelevant. |
| 2 | Partially Adequate | The response contains some correct elements according to the guidelines but also includes significant errors or major omissions. |
| 3 | Mostly Adequate | The response is mostly correct and follows the guidelines with some minor inaccuracies or lack of details. |
| 4 | Fully Adequate | The response matches the guidelines perfectly. It is accurate, complete, and provides a clear and detailed explanation. |
Readability
The readability of the English text was assessed using Flesh Kincaid Grade level (FKGL) (range 0‐18), 17 Flesh Kincaid Reading Ease score (FRE) (range 0‐100) 18 and Simple Measure of Gobbledygook Index (SMOG) (based on syllable count) 19 tools. A validated online readability calculator (readble.com) was used. Their numerical scores were converted to their associated educational level. These formulas have been widely used to evaluate readability in otolaryngology and head and neck surgery literature 20 , 21 and are available in Supplemental S1, available online.
The readability of the Spanish text was assessed using Fernandez‐Huerta (FH) (range 0‐101), 22 Spaulding (SPL) (range 0 to over 141) 23 and Spanish Fry Graph (FG) (which utilizes a graph based on sentences and syllable count) 24 tools. A validated online readability calculator (spanishreadibility.com) was used. Their numerical scores were converted to their associated educational level. The formulas have previously been used to evaluate readability in Spanish in otolaryngology and head and neck surgery literature 25 and are available in Supplement S2, available online.
The mean readability scores were graded as follows: college graduate 0%, college 20%, high school 40%, middle school 60%, elementary 80%, and kindergarten 100%. The mean was calculated for each question using all 3 formulas. Then, an overall mean was calculated for all 7 questions.
Quality
The Patient Education Materials Assessment Tool‐printable (PEMAT‐P) 26 was used to assess the understandability and actionability of the information by 2 independent reviewers. Cohen's kappa was used to evaluate inter‐rater reliability. In cases of discrepancy, a discussion was held between the 2 reviewers to come to an agreement. The mean of the 3 scores was used.
Statistical Analysis
IBM SPSS Version 29 was used for statistical analysis. Frequencies and descriptive statistics for all responses were generated. Independent samples T‐test and one‐way analysis of variance (ANOVA) was used to determine differences in the means of groups with eta‐squared. If Levene's test indicated significant results (P ≤ .05), suggesting a violation of homogeneity of variances, Welch's ANOVA was applied. If ANOVA test indicated significant results (P ≤ .05), post hoc Tukey's test was used to identify specific group differences. A P ≤ .05 was considered statistically significant.
Ethics
This study was exempt from research ethics board review because it utilized publicly accessible information.
Results
Accuracy
Seven questions were submitted to ChatGPT and Google in both English and Spanish, and its scores are summarized in Table 2. Six out of the 7 ChatGPT responses were graded consistently between the English and Spanish responses, and all 4 questions were graded consistently between English and Spanish Google responses. In Spanish, Google did not provide an answer box for 3 questions (questions 3, 4, and 6), while for English questions, an answer box was presented for every single question.
Table 2.
Comparison of Health Information Tools
| Methods of evaluation | Information source | One‐way ANOVA η 2 (95% CI) | ||||
|---|---|---|---|---|---|---|
| English | Spanish | |||||
| Guidelinesa | ChatGPT | ChatGPT | ||||
| Accuracy, % (SD) | 100% | 82.14% (0.12) | 78.57% (0.38) | 78.57% (0.17) | 75% (0) | P = .003 0.44 (0.08‐0.58) |
| Readability, % (SD) | 33.33% (0.12) | 16.19% (0.08) | 35.2% (0.14) | 33.33% (0.10) | 40% (0.12) | P < .001 0.35 (0.18‐0.46) |
| PEMAT‐P Understandability, % (SD) | 68% (7.55) | 65.44% (6.56) | 89.17% (10.10) | 68.38% (1.48) | 68.15% (8.98) | P = .018 0.67 (0.03‐0.76) |
| PEMAT‐P Actionability, % (SD) | 60% (20) | 60% (20) | 20% (20) | 33.33% (28.87) | 33.33% (11.55) | P = .14 0.47 (0‐0.62) |
Note: Accuracy: The mean results for both English and Spanish responses were evaluated out of 100%. Readability: The mean readability scores were graded as follows: college graduate 0%, college 20%, high school 40%, middle school 60%, elementary 80% and kindergarten 100%. The mean was calculated for each question using all 3 formulas. Then, an overall mean was calculated for all 7 questions.
AAO Guidelines.
There was a statistically significant mean difference in accuracy scores between groups (P = .003) with a large effect size of 0.442 (95% CI 0.08‐0.58), when considering the AAO's guidelines as being 100% accurate. The mean accuracy scores of AAO's guidelines were statistically significantly better than every other health source, with the greatest mean difference when compared to Google's Spanish responses at 0.25 (95% CI 0.05‐0.45) (P = .009).
Readability
The readability of all information sources as defined by educational level is summarized in Figure 1. Scores for individual questions using each readability scale is available in the supplement.
Figure 1.
Readability. FG, Fry Graph; FH, Fernandez‐Huerta; FKGL, Flesh Kincaid Grade level; FRE, Flesh Kincaid Reading Ease score; SMOG, Simple Measure of Gobbledygook Index; SPL, Spaulding.
The readability level of ChatGPT's English responses was poor (16.19%, SD 0.081) (Supplemental S3, available online), while Spanish responses were relatively fair (33.3%, SD 0.097) (Supplemental S4, available online), suggesting that ChatGPT's proficiency in Spanish may be more basic. As for Google's English responses, the readability was better at 35.2% (SD 0.14) (Supplement S5, available online) and at 40% (SD 0.12) for Spanish (Supplement S6, available online). The readability level of the AAO guideline answers was of 33.33% (SD 0.12) (Supplement S7, available online). There was a statistically significant mean difference in readability scores between groups (P < .001) with a large effect size of 0.35 (95% CI 0.18‐0.46).
The mean readability scores for English Google responses were significantly (P < .001) better than ChatGPT's English responses with a mean difference of 0.19 (95% CI 0.095‐0.29). However, there was no significant (P = .47) difference with Spanish Google responses and ChatGPT's Spanish responses (mean difference 0.067, 95% CI −0.046‐0.18). The readability of the AAO guidelines was only significantly (P < .001) better than ChatGPT's English responses, with a mean difference of 0.171 (95% CI 0.076‐0.267).
PEMAT‐P
The kappa inter‐rater for the English version was 57.7% (P < .001), indicating moderate agreement, while for the Spanish version, it was of 79.1% (P < .001), indicating substantial agreement. The higher agreement for the Spanish ratings may be due to increased familiarity with the tool, as this evaluation was conducted second.
There was a statistically significant mean difference in understandability scores between groups (P = .018) with a large effect size of 0.67 (95% CI 0.03‐0.76). The mean understandability scores for English Google responses were significantly (P = .032) better than every other health source, with the greatest mean difference when compared to the AAO guidelines at 21.17 (95% CI 1.67‐40.66).
There was no statistically significant mean difference in actionability scores between groups (P = .14) with a large effect size of 0.47 (95% CI 0‐0.62).
Discussion
Web searches are widely used by patients seeking health information 3 , 27 with most users view only top results while searching on the internet 28 using ChatGPT even less effort has to be made to access the information and fulfill curiosity in the medical field that could shift search in the medical field toward ChatGPT and similar AI models that already widely accepted and integrated into the services of many large web companies worldwide. However, the reliability and accuracy of the information found online can vary 29 potentially jeopardizing patients' health and wellbeing. Given that large language models retrieve information from numerous sources, its potential for reliable, evidence‐based health information for patients merits consideration as a healthcare tool. Ensuring accessibility for all means addressing variances in language and education levels.
This study identified a high accuracy in ChatGPT's responses in both English and Spanish languages, with no statistically significant differences between their mean scores and when compared to Google responses. In Spanish, 1 question was graded as only partially adequate (2/4) as it omitted to include the administration methods of corticosteroids and the use of hyperbaric oxygen therapy, which were included in the English response for the same question. No answers were completely inadequate or contained information that may be harmful to patients. When comparing ChatGPT's to Google's answers, the most notable difference is ChatGPT's extensive amount of information. While ChatGPT rarely omits details, its comprehensive responses may be overwhelming for patients.
An important challenge with ChatGPT is its variability in responses based on the user and the context and phrasing of the prompt. While this variability limits the recommendation for patients to rely on it for health‐based inquiries, our results suggest that patients can obtain accurate and personalized information on their health queries. Due to their ability to continually integrate new information, chatbots could offer an automated and more up‐to‐date source of health information compared to traditional methods, which may easily become outdated. 30 Thus, ChatGPT should be considered as a potential tool to improve access to health information.
Health literacy, an important predictor of health, 31 is associated with reading level. The American Medical Association recommends that readability levels of patient education materials be written at or below a 6th‐grade reading level. 32 This study identified that ChatGPT generated English responses at a level consistent with college or college graduates, while its Spanish responses were comparable to a high school level, with this difference being statistically significant. The AI bot tends to be excessively verbose, providing overly comprehensive answers, a result of its training bias, 1 potentially explaining its poor readability performance. This verbosity and high reading level can be a significant barrier to making health information accessible to as many patients as possible.
The ChatGPT model is constantly evolving, an important limitation in studying its responses. It is crucial to consider the implications of the evolving nature of AI models like ChatGPT. As these models are continuously updated, there can be significant variations in responses over time. This dynamic nature necessitates ongoing monitoring and validation to ensure consistency and accuracy in the information provided. Moreover, incorporating user feedback mechanisms can help refine the AI's responses and better tailor them to patient needs. Additionally, comparing Spanish and English readability scores is limited due to inherent differences between the 2 languages, necessitating the use of different tools.
Another relevant aspect to explore is the performance of ChatGPT in languages that use completely different alphabets, such as Cyrillic, Mandarin, or Korean. Since the model is primarily trained on data in languages using the Latin alphabet, its proficiency in languages with distinct writing systems may vary significantly. Yeo et al identified that GPT‐3.5 performed poorly on questions related to cirrhosis when inquired in Mandarin or Korean, in comparison to English, 33 as it is mainly trained on English‐language data. When a question is asked in a language other than English, ChatGPT will try to translate the question to English to understand and provide the user with a response. This raises questions about the completeness and accuracy of responses in languages other than English, particularly those using non‐Latin alphabets, where the lack of extensive training data could result in less reliable information.
As with all online health information, patients and physicians alike must critically appraise the information presented. Physicians play a crucial role in guiding patients and families toward reliable and valid sources, helping them navigate the vast amount of available data. This patient‐physician collaboration not only allows for patient empowerment but also ensures the accuracy of information they use. Open discussions on chatbots, their strengths, and pitfalls, are essential to ensure patients access high‐quality, trustworthy data, enabling better‐informed healthcare decisions.
Our study revealed remarkable results in evaluating ChatGPT's ability to retrieve information from clinical guidelines. However, its performance in responding to clinical vignettes, which more closely mirrors the type of questions asked by patients, was not explored. In a study by Nastasi et. al published in Nature, while ChatGPT provided appropriate advice in response to advice‐seeking questions, it fell short in offering personalized responses or follow‐up questions. 34 This underscores ChatGPT's ineptitude in capturing the nuance and tone of a prompt, limiting its ability in offering patient‐centered medical advice.
Conclusion
Otolaryngologists treat patients with diverse educational backgrounds and languages. While ChatGPT is an accurate and reliable source of information for patients with SSNHL in both Spanish and English, its readability level far exceeds the recommended literacy levels to be considered an accessible health education resource.
Author Contributions
Emily Ajit‐Roger, study conception and design, data collection, statistical analysis, result interpretation, manuscript preparation; Alexander Moise, study conception and design, result interpretation, manuscript preparation; Ostap Orishchak, study conception and design, data collection, result interpretation, manuscript preparation; Carolina Peralta, data collection, manuscript preparation; Sam J. Daniel, study conception and design, manuscript preparation.
Disclosures
Competing interests
None.
Funding source
None.
Supporting information
Supporting information.
This article was presented at the AAO‐HNSF 2024 Annual Meeting & OTO EXPO; September 28‐October 1, 2024; Miami Beach, Florida
Data Availability Statement
Emily Ajit‐Roger has full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Additional data may be reasonably requested.
References
- 1. AI Open. Introducing ChatGPT; 2022. https://openai.com/blog/chatgpt
- 2. Sallam M. ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. Healthcare. 2023;11(6):887. 10.3390/healthcare11060887 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Pagedar NA, Schularick NM, Lee PC, Karnell LH. Health‐related internet use among otolaryngology patients. Ann Otol, Rhinol, Laryngol. 2018;127(8):551‐557. 10.1177/0003489418779414 [DOI] [PubMed] [Google Scholar]
- 4. Kuhn M, Heman‐Ackah SE, Shaikh JA, Roehm PC. Sudden sensorineural hearing loss: a review of diagnosis, treatment, and prognosis. Trends Amplif. 2011;15(3):91‐105. 10.1177/1084713811408349 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Committee on Accessible and Affordable Hearing Health Care for Adults; Board on Health Sciences Policy; Health and Medicine Division; National Academies of Sciences, Engineering, and Medicine ; In: Blazer DG, Domnitz S, Liverman CT, eds. Hearing Health Care for Adults: Priorities for Improving Access and Affordability. National Academies Press (US); 2016. [PubMed] [Google Scholar]
- 6. Xenellis J, Karapatsas I, Papadimitriou N, et al. Idiopathic sudden sensorineural hearing loss: prognostic factors. J Laryngol Otol. 2006;120(9):718‐724. 10.1017/S0022215106002362 [DOI] [PubMed] [Google Scholar]
- 7. Nielsen JPS, von Buchwald C, Grønhøj C. Validity of the large language model ChatGPT (GPT4) as a patient information source in otolaryngology by a variety of doctors in a tertiary otorhinolaryngology department. Acta Otolaryngol. 2023;143(9):779‐782. 10.1080/00016489.2023.2254809 [DOI] [PubMed] [Google Scholar]
- 8. Langlie J, Kamrava B, Pasick LJ, Mei C, Hoffer ME. Artificial intelligence and ChatGPT: an otolaryngology patient's ally or foe. Am J Otolaryngol. 2024;45(3):104220. 10.1016/j.amjoto.2024.104220 [DOI] [PubMed] [Google Scholar]
- 9. Zalzal HG, Abraham A, Cheng J, Shah RK. Can ChatGPT help patients answer their otolaryngology questions? Laryngoscope Investig Otolaryngol. 2023;9(1):e1193. 10.1002/lio2.1193 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Campbell DJ, Estephan LE, Sina EM, et al. Evaluating ChatGPT responses on thyroid nodules for patient education. Thyroid®. 2024;34(3):371‐377. 10.1089/thy.2023.0491 [DOI] [PubMed] [Google Scholar]
- 11. Ray PP. ChatGPT: a comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet Things Cyber Phys Syst. 2023;3:121‐154. 10.1016/j.iotcps.2023.04.003 [DOI] [Google Scholar]
- 12. Chandrasekhar SS, Hollingsworth DB, Monjur TM, Satterfield L. Plain language summary: sudden hearing loss. Otolaryngol Head Neck Surg. 2019;161:211‐217. 10.1177/0194599819859884 [DOI] [PubMed] [Google Scholar]
- 13. Moise A, Centomo‐Bozzo A, Orishchak O, Alnoury MK, Daniel SJ. Can ChatGPT guide parents on tympanostomy tube insertion. Children (Basel, Switzerland). 2023;10(10):1634. 10.3390/children10101634 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Chandrasekhar SS, Tsai Do BS, Schwartz SR, et al. Clinical practice guideline: sudden hearing loss (update). Otolaryngol Head Neck Surg. 2019;161(1_suppl):S1‐S45. 10.1177/0194599819859885 [DOI] [PubMed] [Google Scholar]
- 15. Lechien JR, Maniaci A, Gengler I, Hans S, Chiesa‐Estomba CM, Vaira LA. Validity and reliability of an instrument evaluating the performance of intelligent chatbot: the Artificial Intelligence Performance Instrument (AIPI. Eur Arch Otrhinolaryngol. 2024;281(4):2063‐2079. 10.1007/s00405-023-08219-y [DOI] [PubMed] [Google Scholar]
- 16. Flesch R. A new readability yardstick. J Appl Psychol. 1948;32(3):221‐233. 10.1037/h0057532 [DOI] [PubMed] [Google Scholar]
- 17. Kincaid JP, Fishburne RP, Rogers RL, Chissom BS. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Instit Simul Train. 1975;8(75):24. [Google Scholar]
- 18. McLaughlin GH. SMOG grading: a new readability formula. J Read. 1969;12(8):639‐646. [Google Scholar]
- 19. Kim JH, Grose E, Philteos J, et al. Readability of the American, Canadian, and British Otolaryngology‐Head and Neck Surgery Societies’ Patient Materials. Otolaryngol Head Neck Surg. 2022;166(5):862‐868. 10.1177/01945998211033254 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20. Kasabwala K, Agarwal N, Hansberry DR, Baredes S, Eloy JA. Readability assessment of patient education materials from the American Academy of Otolaryngology‐Head and Neck Surgery Foundation. Otolaryngol Head Neck Surg. 2012;147(3):466‐471. [DOI] [PubMed] [Google Scholar]
- 21. Wong K, Levi JR. Readability trends of online information by the American Academy of Otolaryngology–Head and Neck Surgery Foundation. Otolaryngol Head Neck Surg. 2017;156(1):96‐102. [DOI] [PubMed] [Google Scholar]
- 22. Fernández Huerta J. Medidas sencillas de lecturabilidad. Consigna. 1959;214:29‐32. [Google Scholar]
- 23. Spaulding S. A Spanish readability formula. Modern Lang J. 1956;40:433‐441. [Google Scholar]
- 24. Gilliam B, Peña SC, Mountain L. The fry graph applied to Spanish readability. Read Teach. 1980;33:426‐430. [Google Scholar]
- 25. Coco L, Colina S, Atcherson SR, Marrone N. Readability level of Spanish‐language patient‐reported outcome measures in audiology and otolaryngology. Am J Audiol. 2017;26(3):309‐317. 10.1044/2017_AJA-17-0018 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26. Shoemaker SJ, Wolf MS, Brach C. The Patient Education Materials Assessment Tool (PEMAT) and User's Guide. AHRQ Publication No. 14‐0002‐EF. Agency for Healthcare Research and Quality; 2013.
- 27. McMillan SJ, Avery EJ, Macias W. FROM HAVE NOTS TO WATCH DOGS: understanding internet health communication behaviors of online senior citizens. Inform Commun Soc. 2008;11(5):675‐697. 10.1080/13691180802126745 [DOI] [Google Scholar]
- 28. Mueller J, Jay C, Harper S, Davies A, Vega J, Todd C. Web use for symptom appraisal of physical health conditions: a systematic review. J Med Internet Res. 2017;19(6):e202. 10.2196/jmir.6755 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29. Keselman A, Arnott Smith C, Murcko AC, Kaufman DR. Evaluating the quality of health information in a changing digital ecosystem. J Med Internet Res. 2019;21(2):e11129. 10.2196/11129 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30. AI Open. Hello GPT‐4. 2024. https://openai.com/blog/chatgpt
- 31. Berkman ND, Sheridan SL, Donahue KE, Halpern DJ, Crotty K. Low health literacy and health outcomes: an updated systematic review. Ann Intern Med. 2011;155(2):97‐107. 10.7326/0003-4819-155-2-201107190-00005 [DOI] [PubMed] [Google Scholar]
- 32. Weis BD. Health Literacy: A Manual for Clinicians. American Medical Association, American Medical Foundation; 2003. [Google Scholar]
- 33. Yeo YH, Samaan JS, Ng WH, et al. GPT‐4 outperforms ChatGPT in answering non‐English questions related to cirrhosis. MedRxiv. Preprint posted online May 05, 2023. 10.1101/2023.05.04.23289482 [DOI]
- 34. Nastasi AJ, Courtright KR, Halpern SD, Weissman GE. A vignette‐based evaluation of ChatGPT's ability to provide appropriate and equitable medical advice across care contexts. Sci Rep. 2023;13(1):17885. 10.1038/s41598-023-45223-y [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Supporting information.
Data Availability Statement
Emily Ajit‐Roger has full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Additional data may be reasonably requested.