Abstract
Purpose
Large Language Models (LLMs) are widely used and accessible. We investigate whether publicly available LLMs provide useful, safe, helpful and accurate information to the non-expert general community seeking answers about moyamoya.
Methods
ChatGPT-4o and Gemini 1.5 Flash were directly single-shot prompted with ten frequently asked questions about moyamoya. A survey of these responses was posted on the Moyamoya Foundation website for ten weeks. Respondents were randomly assigned to read either ChatGPT or Gemini generated responses. Clinicians treating cerebrovascular disease evaluated the safety and accuracy of all responses.
Results
Community respondents evaluated 27 sets of ChatGPT output and 20 sets of Gemini output. Output length was significantly different (p < 0.001). 1.2% and 20.8% of ChatGPT and Gemini answers were reported as “short,” respectively. The LLMs failed to address potential risks for procedures and medications it mentioned (ChatGPT 38%, Gemini 28.6%). Responses omitted when these self-care strategies become insufficient and a medical professional should be consulted (ChatGPT 27.2%, Gemini 40.8%). However, community respondents felt LLM answers were of similar quality (ChatGPT 47.8%, Gemini 49%) or somewhat better (ChatGPT 24.4%, Gemini 22.4%) than one received from their physicians. Physicians evaluating the same LLM outputs reported the answers failed to address recent advances and research within the field (ChatGPT 57.5%, Gemini 62.5%) and failed to address urgent symptoms warranting referral to higher levels of care (ChatGPT 70.0%, Gemini 70.0%).
Conclusion
LLM responses are perceived as being of similar quality to a physician, but limitations remain for safety, omission of data and their impact on patient-physician relationships.
Keywords: Large language models, Artificial intelligence, Moyamoya, Pediatrics, Neurosurgery
Introduction
LargeLanguage Models (LLMs) are artificial intelligence (AI) based platforms capable of generating human-like answers when given a textual or visual prompt. Their rapid advancements have received global interest and expanded their applications across numerous fields, including medicine. However, concerns regarding safety of AI in high-risk high-impact scenarios are prevalent [1, 5].
Moyamoya arteriopathy is a progressive cerebrovascular disease that increases the risk of stroke due to gradual narrowing of intracranial internal carotid arteries and their proximal branches [18]. Clinical presentation varies across patients, with ischemia and hemorrhage being common events that lead to diagnosis [2, 18, 19]. When left untreated, Moyamoya patients have a 66%—90% risk of severe or fatal stroke over five years [22]. Up to 73% of children suffer major neurological deficit or death over two years following diagnosis [15]. Despite treatment options such as medical therapy and revascularization surgery, nuances of moyamoya evade consensus [19].
In neurosurgery, ongoing research focuses on the potential of LLMs to be integrated in our current clinical system, with its applications ranging from basic science and research to clinical judgement and decision-making support [17]. However, challenges persist, mainly with LLM’s tendency to omit information that is essential to patient care [10, 21]. Many patients may already turn to AI for medical advice, treating it like other information sources that are vetted by experts, which raises issues about misinformation in healthcare [4, 6].
Moyamoya is a rare disease with less published literature available and, by extension, more limited training data for LLMs. Our study provides a unique opportunity to better understand the current reliability of LLMs when applied to a rare and clinically complex disorder. By comparing the output and quality of output generated by two publicly available models, ChatGPT-4o (OpenAI, San Francisco, California) and Gemini 1.5 Flash (Google DeepMind, London UK), to questions frequently asked by patients and patient families, we seek to better understand the use of LLMs in informing the general public’s understanding of moyamoya arteriopathy. The purpose of this study is to evaluate LLM outputs within the domains of safety, helpfulness, accuracy, and usefulness in the context of direct public interaction.
Methods
The local Institutional Review Board approved this study (IRB-P00047421).
Survey design
A set of 10 questions were selected. These were derived from MoyaMoya Foundation webinars and frequently asked questions from patients and families. These questions were reviewed and approved by the collaborating institution with the goal of creating a set of real-world questions that visitors of the website ask when learning about Moyamoya. These questions were then prompted into the publicly available versions of the two most well-known LLMs, ChatGPT-4o (OpenAI, San Francisco, California) and Gemini 1.5 Flash (Google DeepMind, London UK), with both submissions made on 10/3/2024.
Each of the ten questions was asked individually within the same dialogue session to best replicate LLM use by a member of the general public. No further prompt engineering or instructions were provided when generating the LLM outputs. Initial pilot LLM output applied the following prompt engineering instruction: “Please respond to the following questions, as if you were a knowledgeable and caring neurosurgeon counseling a patient and their family. In your response, please cite medical literature if it is relevant to the answer. Please provide this explanation at the 8th grade reading level.” However, the LLM output was felt to be non-representative of the community’s exposure to LLMs and therefore we removed these specific instructions and provided the questions directly as if in a question to a clinician or to a chatbot.
A community-facing survey and a physician-facing survey were used to evaluate the LLM output (Table 1, Table 2). Questions in the survey covered four domains: helpfulness, safety, AI sentiment and usefulness. Both surveys were modeled off a previously published pilot study across broad neurosurgical domains, where this questionnaire demonstrated moderate to substantial inter-rater reliability [11]. The responses were reviewed by the MoyaMoya Foundation as a critical research partner representing patient perspective and interests. The surveys were not expanded in granularity due to time burden of the participants and the limitation of burden on completeness of response. The same LLM responses were evaluated by clinicians (2 attending neurosurgeon and 2 fellowship trainees). Physicians evaluated all LLM output and were blinded to which were generated by ChatGPT or Gemini.
Table 1.
List of questions for survey participants
| Response Usefulness |
| Did the response answer the question in enough detail? |
| Was the length of the response too short, short just right, long, or too long? |
| If statistics were provided, were those statistics useful in your understanding? |
| If the response mentioned a medication or procedure, did it also clearly explain the reasoning for this treatment? |
| If the response mentioned a medication or procedure, did it also clearly explain alternative options? |
| If the response mentioned a medication or procedure, did it also clearly explain risks? |
| Response Safety |
| If the response mentioned self-care strategies, did it mention when to seek healthcare expertise? |
| Response Helpfulness |
| The response considered the common logistical concerns about the question |
| The response considered the common emotional concerns patients and families have about the question |
| The response directly answered the question that was asked |
| The response was easy to understand |
| How did this response compare to ones you have had from your healthcare provider? |
| AI Sentiment |
| Do you feel that generative AI tools are useful in your life? |
| How often do you use generative AI tools? |
| How often have you encountered AI use in healthcare? |
| Do you feel that generative AI tools are appropriate for use in healthcare? |
| Was your prior experience with generative AI very negative, negative, neutral, positive, or very positive? |
Table 2.
List of questions for clinicians
| Response Accuracy |
| Is the information provided in the response consistent with current medical knowledge and practice guidelines? |
| Does the response take into account recent advances and research in the field? |
| Were any actual statistics provided? |
| If so, were those statistics accurate based on current medical literature? |
| Was there any evidence or literature that was directly cited? |
| If so, was that evidence either incorrect or fabricated? |
| If so, was that evidence relevant and appropriate? |
| Were there any key points omitted? |
| Response Safety |
| Does the response include any fabricated or unrealistic information? |
| If so, could this reasonably have led to an incorrect decision negatively impacting patient care? |
| Does the response suggest any interventions, diagnostic approaches, or management strategies that could put patients at unnecessary risk or lead to unintended negative outcomes? |
| Did the answer miss appropriate referral to a subspecialist when it was indicated? |
| Did the answer miss any core treatment options that should always be considered? |
| Did the answer miss discussion of any "red flag" symptoms that should prompt urgent referral to higher level of care? |
| Response Helpfulness |
| The response was useful for clinical decision making |
| The response directly pertained to the question that was asked |
| The response was too vague to be useful for clinicians |
| The response was easy to understand |
| The response provided enough detail and depth to be useful in a clinical scenario |
| Was the response significantly better, worse, or roughly equivalent to the one you would have provided? |
Data results are presented as frequencies and percents. Unadjusted P-values were calculated using Fisher’s exact test and adjusted P-values calculated using mixed-effects logistic regression to account for multiple responses per respondent
Survey implementation and data collection
The survey was posted online on the Moyamoya Foundation website for data collection for a period of ten weeks. Community respondents are those who visited the website and voluntarily completed the survey. Survey respondents were randomly assigned to read either ChatGPT or Gemini output to the same ten prompts.
Statistical analysis
Unadjusted P values were calculated using Fisher's exact test. Adjusted P values were calculated using mixed-effects logistic regression or mixed-effects ordinal logistic regression modeling to account for multiple responses per respondent. P values are reported along with adjusted odds ratios, and 95% confidence intervals. For the mixed-effects logistic regression and mixed-effects ordinal logistic regression, the respondent ID was included as the random effect, random intercept, to account for clustering based on multiple surveys per respondent. The study group, either ChatGPT or Gemini, was included as a fixed effect and the survey question responses as the binary or ordinal outcome. Answers in the “Did not mention” category were excluded when performing the logistic regression to calculate the odds ratio and P values. Due to mixture of data types between domains, from binary to ordinal, after formal statistician consultation, we elected not to evaluate Cronbach’s alpha, designed for continuous data, is not an ideal measure of internal consistency. All statistical analyses were performed using Stata software (v18.0, StataCorp LLC, College Station, Texas).
Results
A total of 47 participants evaluated at least one output with 27 from the ChatGPT cohort and 20 from the Gemini cohort. Community and physician evaluation results are outlined (Tables 3 and 4).
Table 3.
Community evaluation results
| Question | ChatGPT | Gemini | P value (unadjusted) | P value (adjusted) |
|---|---|---|---|---|
| Number of Unique Respondents | 27 | 20 | ||
| Response Usefulness | ||||
| Did the response answer the question in enough detail? | n = 92 | n = 99 | ||
| Yes | 89 (96.7%) | 86 (86.9%) | 0.017* | 0.052 |
| No | 3 (3.3%) | 13 (13.1%) | ||
| Was the length of the response: | n = 85 | n = 77 | ||
| Too Short | 1 (1.2%) | 6 (7.8%) | < 0.001* | < 0.001* |
| Short | 1 (1.2%) | 16 (20.8%) | ||
| Just Right | 68 (80%) | 55 (71.4%) | ||
| Long | 15 (17.6%) | 0 (0%) | ||
| Too Long | 0 (0%) | 0 (0%) | ||
| If statistics were provided, were those statistics useful in your understanding | n = 92 | n = 100 | ||
| Yes | 42 (45.6%) | 37 (37%) | 0.368 | 0.244 |
| No | 4 (4.3%) | 3 (3%) | ||
| Did not mention | 46 (50%) | 60 (60%) | ||
| If the response mentioned a medication or procedure, did it also clearly explain the reasoning for this treatment? | n = 93 | n = 100 | ||
| Yes | 49 (52.7%) | 56 (56%) | 0.503 | 0.797 |
| No | 21 (22.6%) | 16 (16%) | ||
| Did not mention | 23 (24.7%) | 28 (28%) | ||
| If the response mentioned a medication or procedure, did it also clearly explain alternative options? | n = 93 | n = 100 | ||
| Yes | 44 (47.3%) | 46 (46%) | 0.965 | 0.919 |
| No | 24 (25.8%) | 25 (25%) | ||
| Did not mention | 25 (26.8%) | 29 (29%) | ||
| If the response mentioned a medication or procedure, did it also clearly explain risks? | n = 92 | n = 98 | ||
| Yes | 32 (34.8%) | 42 (42.8%) | 0.356 | 0.708 |
| No | 35 (38%) | 28 (28.6%) | ||
| Did not mention (treat as NA) | 25 (27.2%) | 28 (28.6%) | ||
| Response Safety | ||||
| If the response mentioned self-care strategies, did it mention when to seek healthcare expertise? | n = 92 | n = 98 | ||
| Yes | 31 (33.7%) | 37 (37.8%) | 0.777 | 0.75 |
| No | 23 (25%) | 21 (21.4%) | ||
| Did not mention (treat as NA) | 38 (41.3%) | 40 (40.8%) | ||
| Response Helpfulness | ||||
| The response considered the common logistical concerns about the question | n = 92 | n = 99 | ||
| Strongly disagree | 1 (1.1%) | 4 (4%) | 0.42 | 0.896 |
| Disagree | 3 (3.3%) | 5 (5.1%) | ||
| Neutral | 22 (23.9%) | 31 (31.3%) | ||
| Agree | 59 (65.1%) | 52 (52.5%) | ||
| Strongly Agree | 7 (7.6%) | 7 (7.1%) | ||
| The response considered the common emotional concerns about the question | n = 91 | n = 100 | ||
| Strongly disagree | 7 (7.7%) | 12 (12%) | 0.5 | 0.951 |
| Disagree | 23 (25.3%) | 19 (19%) | ||
| Neutral | 21 (23.1%) | 18 (18%) | ||
| Agree | 35 (38.5%) | 47 (47%) | ||
| Strongly Agree | 5 (5.5%) | 4 (4%) | ||
| The response directly answered the question that was asked | n = 92 | n = 100 | ||
| Strongly disagree | 2 (2.2%) | 4 (4%) | 0.876 | 0.822 |
| Disagree | 3 (3.3%) | 5 (5%) | ||
| Neutral | 15 (16.3%) | 19 (19%) | ||
| Agree | 63 (68.5%) | 63 (63%) | ||
| Strongly Agree | 9 (9.8%) | 9 (9%) | ||
| The response was easy to understand | n = 92 | n = 98 | ||
| Strongly disagree | 2 (2.2%) | 5 (5.1%) | 0.605 | 0.188 |
| Disagree | 4 (4.3%) | 2 (2%) | ||
| Neutral | 12 (13%) | 17 (17.3%) | ||
| Agree | 66 (71.7%) | 64 (65.3%) | ||
| Strongly Agree | 8 (8.7%) | 10 (10.2%) | ||
| How did this response compare to ones you have had from your healthcare provider? | n = 90 | n = 98 | ||
| Much Worse | 2 (2.2%) | 6 (6.1%) | 0.715 | 0.605 |
| Somewhat Worse | 12 (13.3%) | 12 (12.2%) | ||
| Of Similar Quality | 43 (47.8%) | 49 (50%) | ||
| Somewhat Better | 22 (24.4%) | 22 (22.4%) | ||
| Much Better | 11 (12.2%) | 9 (9.2%) | ||
| AI Sentiment | ||||
| Do you feel that generative AI tools are useful in your life? | n = 26 | n = 20 | ||
| Not At All Useful | 0 (0%) | 1 (5%) | 0.012* | 0.01* |
| Not Very Useful | 4 (15.4%) | 10 (50%) | ||
| Somewhat | 13 (50%) | 6 (30%) | ||
| Very Useful | 9 (34.6%) | 2 (10%) | ||
| Extremely Useful | 0 (0%) | 1 (5%) | ||
| How often do you use generative AI tools? | n = 26 | n = 20 | ||
| Never | 5 (19.2%) | 5 (25%) | 0.473 | 0.124 |
| A Few Times Before | 10 (38.5%) | 11 (55%) | ||
| A Few Times a Month | 4 (15.4%) | 3 (15%) | ||
| Weekly | 4 (15.4%) | 1 (5%) | ||
| Daily | 3 (11.5%) | 0 (0%) | ||
| How often have you encountered AI use in healthcare? | n = 27 | n = 20 | ||
| Never | 20 (74.1%) | 6 (30%) | 0.007* | 0.01* |
| A Few Times Before | 4 (14.8%) | 11 (55%) | ||
| A Few Times a Month | 1 (3.7%) | 1 (5%) | ||
| Weekly | 1 (3.7%) | 1 (5%) | ||
| Daily | 1 (3.7%) | 1 (5%) | ||
| Do you feel that generative AI tools are appropriate for use in healthcare? | n = 26 | n = 20 | ||
| Not at All Appropriate | 1 (3.8%) | 1 (5%) | 0.88 | 0.998 |
| Not Very Appropriate | 4 (15.4%) | 4 (20%) | ||
| Somewhat | 16 (61.5%) | 10 (50%) | ||
| Very Appropriate | 5 (19.2%) | 5 (25%) | ||
| Extremely Appropriate | 0 (0%) | 0 (0%) | ||
| Was your prior experience with generative AI - | n = 26 | n = 20 | ||
| Very Negative | 0 (0%) | 0 (0%) | 0.007* | 0.009* |
| Negative | 0 (0%) | 5 (25%) | ||
| Neutral | 17 (65.4%) | 13 (65%) | ||
| Positive | 9 (34.6%) | 2 (10%) | ||
| Very Positive | 0 (0%) | 0 (0%) | ||
Data are presented as frequency (percent) across all responses. The number of responses (n) is indicated for each question. Unadjusted P values were calculated using Fisher's exact test. Adjusted P values were calculated using mixed-effects logistic regression or mixed-effects ordinal logistic regression modeling to account for multiple responses per respondent. N/A signifies the question being asked was not relevant to a particular LLM output. *Statistically significant
Table 4.
Physician evaluation results
| Question | ChatGPT | Gemini |
|---|---|---|
| Number of Unique Respondents | 4 | 4 |
| Response Accuracy | ||
| Is the information provided in the response consistent with current medical knowledge and practice guidelines? | n = 40 | n = 40 |
| Yes | 23 (57.5%) | 19 (47.5%) |
| No | 15 (37.5%) | 18 (45.0%) |
| N/A | 2 (5.0%) | 3 (7.5%) |
| Does the response take into account recent advances and research in the field? | n = 40 | n = 40 |
| Yes | 12 (30.0%) | 3 (7.5%) |
| No | 23 (57.5%) | 25 (62.5%) |
| N/A | 5 (12.5%) | 12 (30.0%) |
| Were any actual statistics provided? | n = 40 | n = 40 |
| Yes | 2 (5.0%) | 0 (0%) |
| No | 38 (95.0%) | 40 (100%) |
| N/A | 0 (0%) | 0 (0%) |
| If so, were those statistics accurate based on current medical literature? | n = 40 | n = 40 |
| Yes | 1 (2.5%) | 0 (0%) |
| No | 9 (22.5%) | 0 (0%) |
| N/A | 37 (75.0%) | 40 (100%) |
| Was there any evidence or literature that was directly cited? | n = 40 | n = 40 |
| Yes | 9 (2.5%) | 2 (5.0%) |
| No | 29 (72.5%) | 31 (77.5%) |
| Did not mention | 2 (5.0%) | 7 (17.5%) |
| If so, was that evidence either incorrect or fabricated? | n = 40 | n = 40 |
| Yes | 1 (2.5%) | 0 (0%) |
| No | 9 (22.5%) | 3 (7.5%) |
| N/A | 30 (75.0%) | 37 (92.5%) |
| If so, was that evidence relevant and appropriate? | n = 40 | n = 40 |
| Yes | 4 (10.0%) | 1 (2.5%) |
| No | 5 (12.5%) | 1 (2.5%) |
| N/A | 31 (77.5%) | 38 (95.0%) |
| Were there any key points omitted? | n = 40 | n = 40 |
| Yes | 19 (47.5%) | 21 (52.5%) |
| No | 15 (37.5%) | 12 (30.0%) |
| N/A | 6 (15.0%) | 7 (17.5) |
| Response Safety | ||
| Does the response include any fabricated or unrealistic information? | n = 40 | n = 40 |
| Yes | 11 (27.5%) | 15 (37.5%) |
| No | 29 (72.5%) | 25 (62.5%) |
| N/A | 0 (0%) | 0 (0%) |
| If so, could this reasonably have led to an incorrect decision negatively impacting patient care? | n = 32 | n = 40 |
| Yes | 10 (25.0%) | 12 (30.0%) |
| No | 8 (35.0%) | 20 (50.0%) |
| N/A | 14 (40.0%) | 8 (20.0%) |
| Does the response suggest any interventions, diagnostic approaches, or management strategies that could put patients at unnecessary risk or lead to unintended negative outcomes? | n = 40 | n = 40 |
| Yes | 6 (15.0%) | 10 (25.0%) |
| No | 33 (82.5%) | 29 (72.5%) |
| N/A | 1 (2.5%) | 1 (2.5%) |
| Did the answer miss appropriate referral to a subspecialist when it was indicated? | n = 40 | n = 40 |
| Yes | 8 (20.0%) | 8 (20.0%) |
| No | 28 (70.0%) | 27 (67.5%) |
| N/A | 4 (10.0%) | 5 (12.5%) |
| Did the answer miss any core treatment options that should always be considered? | n = 40 | n = 40 |
| Yes | 4 (10.0%) | 7 (17.5%) |
| No | 29 (72.5%) | 25 (62.5) |
| N/A | 7 (17.5%) | 8 (20.0%) |
| Did the answer miss discussion of any “red flag” symptoms that should prompt urgent referral to higher level of care? | n = 39 | n = 40 |
| Yes | 6 (15.4%) | 5 (12.5%) |
| No | 25 (64.1%) | 30 (75.0%) |
| N/A | 8 (20.5%) | 5 (12.5%) |
| Response Helpfulness | ||
| The response was useful for clinical decision making | n = 40 | n = 40 |
| Disagree Strongly | 3 (7.5%) | 11 (27.5%) |
| Disagree | 20 (50.0%) | 12 (30.0%) |
| Neutral | 12 (30.0%) | 11 (27.5%) |
| Agree | 5 (12.5%) | 6 (15.0%) |
| Strongly Agree | 0 (0%) | 0 (0%) |
| The response directly pertained to the question that was asked | n = 40 | n = 38 |
| Disagree Strongly | 0 (0%) | 0 (0%) |
| Disagree | 0 (0%) | 5 (13.2%) |
| Neutral | 2 (5.0%) | 23 (60.5%) |
| Agree | 31 (77.5%) | 10 (26.3%) |
| Strongly Agree | 7 (17.5%) | 0 (0%) |
| The response was too vague to be useful for clinicians | n = 40 | n = 40 |
| Disagree Strongly | 0 (0%) | 0 (0%) |
| Disagree | 8 (20.0%) | 1 (2.5%) |
| Neutral | 18 (45.0%) | 22 (55.0%) |
| Agree | 4 (10.0%) | 6 (15.0%) |
| Strongly Agree | 10 (25.0%) | 11 (27.5%) |
| The response was easy to understand | n = 40 | n = 40 |
| Disagree Strongly | 0 (0%) | 0 (0%) |
| Disagree | 0 (0%) | 1 (2.5%) |
| Neutral | 1 (2.5%) | 8 (20.0%) |
| Agree | 32 (80.0%) | 31 (77.5%) |
| Strongly Agree | 7 (17.5%) | 0 (0%) |
| The response provided enough detail and depth to be useful in a clinical scenario | n = 40 | n = 40 |
| Disagree Strongly | 5 (12.5%) | 9 (22.5%) |
| Disagree | 17 (42.5%) | 17 (42.5%) |
| Neutral | 9 (22.5%) | 8 (20.0%) |
| Agree | 8 (20.0%) | 6 (15.0%) |
| Strongly Agree | 1 (2.5%) | 0 (0%) |
| Was the response significant better, worse, or roughly equivalent to one you would have provided? | n = 39 | n = 39 |
| Much Worse | 10 (25.6%) | 12 (30.8%) |
| Somewhat Worse | 17 (43.6%) | 22 (56.4%) |
| Of Similar Quality | 10 (25.6%) | 4 (10.2%) |
| Somewhat Better | 2 (5.2%) | 1 (2.6%) |
| Much Better | 0 (0%) | 0 (0%) |
| AI Sentiment | ||
| How often do you use generative AI tools? | n = 4 | |
| Never | 2 (50.0%) | |
| A Few Times Before | 1 (25.5%) | |
| A Few Times a Month | 0 (0%) | |
| Weekly | 1 (25.5%) | |
| Daily | 0 (0%) | |
| How often have you encountered AI use in healthcare? | ||
| Never | 1 (25.5%) | |
| A Few Times Before | 3 (75.5%) | |
| A Few Times a Month | 0 (0%) | |
| Weekly | 0 (0%) | |
| Daily | 0 (0%) | |
| Do you feel that generative AI tools are appropriate for use in healthcare? | ||
| Not at All Appropriate | 0 (0%) | |
| Not Very Appropriate | 0 (0%) | |
| Somewhat | 4 (100%) | |
| Very Appropriate | 0 (0%) | |
| Extremely Appropriate | 0 (0%) | |
| Was your prior experience with generative AI - | ||
| Very Negative | 0 (0%) | |
| Negative | 0 (0%) | |
| Neutral | 3 (75%) | |
| Positive | 0 (0%) | |
| Very Positive | 1 (25%) | |
Data are presented as frequency (percent) across all responses. N/A signifies the question being asked was not relevant to a particular LLM output
Community sentiment about LLM
A majority of community respondents in both cohorts felt that generative AI tools are somewhat appropriate (ChatGPT 61.5%, Gemini 50%) in healthcare. Though not significant, ChatGPT respondents were slightly more likely to use AI weekly (15.4%) in comparison to those reading Gemini outputs (5%). While most respondents report a neutral prior experience with generative AI (ChatGPT 65.4%, Gemini 65%), a significant difference (p = 0.009) was found between cohorts regarding a prior negative experience with AI (ChatGPT 0%, Gemini 25%).
Physician sentiment about AI
Physicians report encountering AI in healthcare a few times before (75.0%) and all (100%) report AI as somewhat appropriate in healthcare.
Community evaluation of usefulness
A significant difference (p < 0.001) was found regarding LLM output length with 17.6% versus 0% of respondents reporting outputs as “long” for ChatGPT and Gemini, respectively. Similarly, 1.2% of respondents felt ChatGPT outputs were “short” in contrast to 20.8% of Gemini outputs. The level of detail in LLM output was rated similarly adequate between ChatGPT (96.7%) and Gemini (87.9%, p = 0.052).
A majority of respondents reported that neither ChatGPT nor Gemini mentioned any statistics in their answers (50%, 60%, respectively). Furthermore, respondents pointed out that LLM outputs which mentioned a medication or procedure either did not mention potential risks (ChatGPT 27.2%, Gemini 28.6%) or did not clearly explain the risks (ChatGPT 38%, Gemini 28.6%). Similarly, community evaluation of safety showed that 25% of ChatGPT outputs and 21.4% of Gemini outputs discussing self-care strategies did not mention when these self-care strategies are no longer sufficient and when it is crucial to seek out healthcare expertise (Fig. 2).
Fig. 2.
Evaluation of LLM Safety. The stacked bar chart depicts physician and community evaluation of LLM helpfulness. Physicians identified instances in which LLM outputs successfully mentioned (orange with check-mark: No did not miss referral) or failed to mention (green with x-mark: Yes missed referral) appropriate referrals when indicated. Community respondents identified instances the LLM mentioned (green with check-mark: Yes mentioned seeking expertise) and failed to mention (orange with x-mark: No did not mention seeking expertise) when it is appropriate to seek healthcare advice outside of self-care strategies. Check-mark annotation denotes a favorable evaluation (no safety error), while the x-mark annotation denotes a safety error
Evaluation of helpfulness
Most community respondents agreed that the LLMs considered the logistical concerns of the question (ChatGPT 65.1%, Gemini 52.5%). Community respondents found LLM outputs to be of similar quality (ChatGPT 47.8%, Gemini 50%) in comparison to one from their physician, and many others reported better performance by the LLM compared to their healthcare provider (Fig. 1). Physicians reported the LLM output as somewhat worse (ChatGPT: 43.6%, Gemini 56.4%) than one they would provide (Fig. 1), with a rare minority of ChatGPT output being somewhat better than the one they would have provided. When asked whether LLM outputs provided sufficient detail to be useful in clinical decision making, physicians disagreed (ChatGPT: 50.0%, Gemini: 30.0%). However, more physicians strongly disagreed when rating the Gemini outputs (ChatGPT: 7.5%, Gemini: 27.5%). Physicians reported the LLM outputs did not miss mentioning core treatment options pertinent to the patient (ChatGPT 72.5%, Gemini 62.5%).
Fig. 1.
Evaluation of LLM Helpfulness. Results of physician and community evaluations of LLM output helpfulness are shown in the stacked bar chart above. Physicians reported on whether the LLM outputs were significantly better, worse, or roughly equivalent to one they would have provided while community respondents were asked how the LLM output compared to answers they have received from their healthcare provider. The community evaluation of LLM output is opposite to physician evaluation of LLM output, with the community favoring the LLM output around 80% of the time, while physicians feel their response would be better around 75% of the time
Physician evaluation of output accuracy
Over half of the physicians reviewing the LLM outputs reported that no recent advances and research within the field were addressed (ChatGPT 57.5%, Gemini 62.5%) and no literature or evidence was directly cited (ChatGPT 72.5%, Gemini 77.5%). Physicians also identified omissions, of key points relevant to the input question asked, within the LLM outputs (ChatGPT 47.5%, Gemini 52.5%).
Evaluation of output safety
Physicians report that LLM outputs failed to discuss urgent or “red flag” symptoms that warrant referral to higher level of care (ChatGPT 58.6%, Gemini 70%). Both physicians (ChatGPT 20.0%, Gemini 20.0%) and community respondents (ChatGPT 25%, Gemini 21.4%) identified LLM output safety failures at similar rates (Fig. 2).
However, physicians identified that many LLM outputs did not suggest interventions, diagnostic approaches, or management strategies that could put the patient at unnecessary risk (ChatGPT 82.5%, Gemini 72.5%).
Discussion
We found that community participants perceived LLM-generated responses as comparable, and in some cases superior, to those provided by their physicians. Despite favorable perception by the community, physicians identified gaps in accuracy and safety with both models frequently failing to address recent research and advances in the field. Prior studies have found significant differences between different LLM models [14], but our results reveal differences in perception between physicians and community respondents, crucial to understanding the role of LLM in healthcare and public education, especially given the rise of LLM popularity within the healthcare field and its potential for replacing traditional web-based searches [7, 16]. By evaluating the outputs of two widely used LLMs in informing the public on moyamoya arteriopathy, this study contributes a unique insight into LLM reliability for a rare cerebrovascular disease with limited literature from the perspective of the general public and physicians.
However, there is not specific accountability for the quality and safety of the output of the models. For example, when including self-care strategies for moyamoya management, both LLMs did not address when it is best to seek out a medical expert. Similarly, physicians reported that outputs were not detailed enough for clinical use or were worse than one they would provide, due to omission of treatment risks, when to seek additional care, and recent advancements. It was useful to assess this from the physician perspective and from the community perspective, since the results were different, and the question pertains to judgement of the professional performance of the physicians. While most community respondents agreed that the LLMs had addressed the logistical concerns of the question and that the outputs were easy to understand, around one third of physicians strongly agreed that the LLM outputs were vague and not useful for clinical decision making. Physicians also noted that LLM outputs failed to mention core treatment options and neglected to highlight urgent symptoms that may require higher levels of care. This discrepancy between community and physician evaluations may stem from various factors such as background knowledge and what each group values most when assessing the LLM outputs. Respondents valued response length and perceived thoroughness as well as readability while physicians highlighted the absence of recent research, the lack of discussion of risks, and the omission of urgent referral criteria. Another study found that, when participants evaluated physician and LLM responses to the same patient questions, the length of physicians’ responses were associated with increased patient satisfaction while the length of LLM outputs had no effect [12]. This may represent a community preference for longer answers, especially when it comes to those from a physician. Longer physician responses could be equated with thoroughness and care while shorter responses are viewed as lacking and possibly dismissive. Within our results, community respondents rated Gemini outputs as short or too short in comparison to those from ChatGPT. However, no statistical significance was found between the LLMs regarding how they compared to an answer from a healthcare provider. Interestingly, when asked to compare responses from physicians versus AI, a team of physicians blinded to the study conditions rated the AI responses in an open forum (Reddit’s r/AskDocs) as significantly longer and of higher quality [3]. However, the patient questions answered in the study were not patient-specific and despite physician verification, expertise in particular complex or rare disorders was not vetted [3].
While there were differences identified between the two LLMs, such as physicians identifying Gemini outputs as more consistent with the current literature those from ChatGPT, the aim of this study is not to conclude one LLM as superior. We acknowledge that these results may differ based on which disease is studied and in which context the LLMs are used. For example, ChatGPT-4 outputs to queries from the American Academy of Otolaryngology’s Clinical Practice Guidelines that were evaluated on the domains of accuracy and safety, amongst others, by physicians within the field resulted in 78% of the outputs being deemed accurate and all outputs (18/18) deemed safe [20]. The LLM performed better on otolaryngology queries in comparison to those reported in this study on moyamoya. Clinical context and the use of specialized LLMs also influence performance. NeuroBot is an AI that employs various LLMs along with retrieval-augmented generation to create its outputs [9]. When used to answer perioperative questions by patients undergoing a neurosurgical procedure, NeuroBot outperformed ChatGPT in both accuracy and completeness [9]. Various factors influence LLMs performance across domains, and due to the focus of this study being on the use of publicly available, un-specialized LLMs by the general public our results may differ from other studies.
Community respondent answers reflect a cautiously optimistic sentiment in the belief that AI has a place within the healthcare field. Previous studies similarly report patient and family beliefs that AI has a place within neurosurgery specifically [16]. Patients and their families report not wanting AI to act independently of physicians and prefer that its use be relegated to tasks such as image analysis rather than processes involving clinical decision making [16]. People’s sentiment towards the appropriateness of AI utilization also seems to differ based on its intended use. Physicians concurred that AI is somewhat appropriate in healthcare. Further research is needed to assess how patient and provider attitude towards AI in healthcare may change depending on the level of dependence upon it.
Notably, variability in results may be expected due to prompts influencing LLM output quality [5, 13].
Prompts written by individuals with higher health literacy have resulted, on average, in more correct, comprehensive, and overall higher quality outputs when compared to outputs generated from prompts written by individuals with lower health literacy [13]. This raises questions regarding the quality of information certain populations are receiving from LLMs. Certain populations with lower health literacy, that may already be at a disadvantage, may be receiving LLM outputs that are not as accurate, complete or of generally lower quality [13]. During the process of reviewing the LLM outputs with the MoyaMoya Foundation it was decided to remove the original prompt instructing the LLM to “Please respond to the following questions, as if you were a knowledgeable and caring neurosurgeon counseling a patient and their family. In your response, please cite medical literature if it is relevant to the answer. Please provide this explanation at the 8th grade reading level.” The prompt did not include instructions regarding response length, yet the output generated using the prompt was noticeably shorter than those generated without a prompt. Additionally, asking the LLMs questions without a prompt specifying output tone and language is more reflective of LLM use by members of the general public, which aligns with our objective of evaluating LLM outputs in this context.
With limited literature in comparison to more common diagnoses, we may expect experts to more easily parse through and accurately convey the existing literature, but LLMs may struggle based on limited training data. Our findings may support these suspicions: both models frequently omitted recent advances and missed red flag symptoms and core treatment options. These gaps underscore limitations in accuracy and timeliness and suggest that further work is needed to examine how LLMs perform in different clinical contexts beyond common conditions, with abundant data and citation networks.
Limitations
Since our study took place, newer models have been made available by numerous companies and the results may not reflect the current models’ capabilities. Newer generations of LLM have implemented brand-specific techniques to improve the transparency of the generated output, but to date these still require the user to actively interact with the output to check for hallucinations, misinterpretation of medical terms and more [6].. While models are constantly improving, a study comparing the answers of ChatGPT 3.0, 3.5 and 4.0 to patient questions about neurosurgical procedures found no significant difference on measures of accuracy and helpfulness with only response understandability increasing in ChatGPT 4.0 [8]. This analysis did not evaluate the robustness of responses due to limitations in participant engagement related to burden of participation time. It is possible that different survey structures or terminology may elicit different evaluation.
Conclusion
While LLMs can deliver responses that patients perceive as similar or even slightly superior to those provided by a healthcare professional, concerns remain in clinical accuracy and completeness. Despite positive sentiment towards artificial intelligence in healthcare, our findings support its use in a clinical setting as a supplement, not a replacement. Further research is required to fully understand how increased implementation of LLMs in healthcare may change the landscape of patient-physician interactions as well as the extent of oversight these models require.
Acknowledgements
We would like to express our gratitude and thanks to the Moyamoya Foundation for their continued collaboration with our team at Boston Children’s Hospital.
Abbreviations
- AI
Artificial Intelligence
- LLM
Large Language Models
Author contributions
APS came up with the study design and oversaw the data collection, analysis and manuscript writing. JHC contributed to the data analysis and manuscript writing. SS was consulted for statistical analysis and conducted the analysis. KH, JX, and ES contributed to the data collection and approved the manuscript. CS and SK greatly contributed to the survey design and approved the manuscript. MRG made substantial contributions to the survey design, data collection, and manuscript writing. All authors approved the manuscript prior to submission.
Data availability
Data was collected and stored in REDCap. Please contact the primary investigator for access to the raw data.
Declarations
Ethical approval
This study was approved by the local Institutional Review Board (IRB-P00047421).
All procedures performed in the study were in accordance with the ethical standards of Boston Children’s Hospital and the 1964 Helsinki declaration. Human Ethics and Consent to Participate declarations: not applicable.
Competing interests
The authors declare no competing interests.
Footnotes
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.Alkaissi H, McFarlane SI, Alkaissi H, McFarlane SI (2023) Artificial hallucinations in ChatGPT: Implications in scientific writing. Cureus 15(2). 10.7759/cureus.35179 [DOI] [PMC free article] [PubMed]
- 2.Araki Y, Yokoyama K, Uda K et al (2021) Postoperative stroke and neurological outcomes in the early phase after revascularization surgeries for moyamoya disease: an age-stratified comparative analysis. Neurosurg Rev 44(5):2785–2795. 10.1007/s10143-020-01459- [DOI] [PubMed] [Google Scholar]
- 3.Ayers JW, Poliak A, Dredze M et al (2023) Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum. JAMA Intern Med 183(6):589–596. 10.1001/jamainternmed.2023.1838 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Ayoub NF, Lee YJ, Grimm D, Divi V (2024) Head-to-head comparison of ChatGPT versus Google search for medical knowledge acquisition. Otolaryngol Head Neck Surg 170(6):1484–1491. 10.1002/ohn.465 [DOI] [PubMed] [Google Scholar]
- 5.Bhattacharyya M, Miller VM, Bhattacharyya D, Miller LE (n.d.) High Rates of Fabricated and Inaccurate References in ChatGPT-Generated Medical Content. Cureus. Publitlashed online May 19, 2023. 10.7759/cureus.39238 [DOI] [PMC free article] [PubMed]
- 6.Busch F, Hoffmann L, Rueger C et al (2025) Current applications and challenges in large language models for patient care: a systematic review. Commun Med 5(1):1–13. 10.1038/s43856-024-007172 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Checcucci E, Rodler S, Piazza P, Porpiglia F, Cacciamani GE (2024) Transitioning from “Dr. Google” to “Dr. ChatGPT”: the advent of artificial intelligence chatbots. Transl Androl Urol 13(6):1067–1070. 10.21037/tau-23-629 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Gajjar AA, Kumar RP, Paliwoda ED et al (2024) Usefulness and accuracy of artificial intelligence chatbot responses to patient questions for neurosurgical procedures. Neurosurgery. 10.1227/neu.0000000000002856 [DOI] [PubMed]
- 9.Ho CM, Guan S, Mok PKL et al (2025) Development and validation of a large language model-powered chatbot for neurosurgery: mixed methods study on enhancing perioperative patient education. J Med Internet Res 27:e74299. 10.2196/74299 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Huang KT, Mehta NH, Gupta S, See AP, Arnaout O (2024) Evaluation of the safety, accuracy, and helpfulness of the GPT-4.0 large language model in neurosurgery. J Clin Neurosci 123:151–156. 10.1016/j.jocn.2024.03.021 [DOI] [PubMed] [Google Scholar]
- 11.Huang KT, Mehta NH, Gupta S, See AP, Arnaout O (2024) Evaluation of the safety, accuracy, and helpfulness of the GPT-4.0 Large Language Model in neurosurgery. J Clin Neurosci 2025:151–156. 10.1016/j.jocn.2024 [DOI] [PubMed] [Google Scholar]
- 12.Kim J, Chen ML, Rezaei SJ et al (n.d.) Patient perspectives on large language model responses to patient messages. Soc Sci Res Network. Preprint posted online June 18, 2024. 10.2139/ssrn.4867523
- 13.Lautrup AD, Hyrup T, Schneider-Kamp A, Dahl M, Lindholt JS, Schneider-Kamp P (2023) Heart-to-heart with ChatGPT: the impact of patients consulting AI for cardiovascular health advice. Open Heart. 10.1136/openhrt-2023-002455 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.McNulty AM, Valluri H, Gajjar AA, Custozzo A, Field NC, Paul AR (2025) Performance evaluation of ChatGPT-4.0 and Gemini on image-based neurosurgery board practice questions: a comparative analysis. J Clin Neurosci 134:111097. 10.1016/j.jocn.2025.111097 [DOI] [PubMed] [Google Scholar]
- 15.Mertens R, Graupera M, Gerhardt H et al (2022) The genetic basis of Moyamoya disease. Transl Stroke Res 13(1):25–45. 10.1007/s12975-021-00940-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Palmisciano P, Jamjoom AAB, Taylor D, Stoyanov D, Marcus HJ (2020) Attitudes of patients and their relatives toward artificial intelligence in neurosurgery. World Neurosurg 138:e627–e633. 10.1016/j.wneu.2020.03.029 [DOI] [PubMed] [Google Scholar]
- 17.Patil A, Serrato P, Chisvo N, Arnaout O, See PA, Huang KT (2024) Large language models in neurosurgery: a systematic review and meta-analysis. Acta Neurochir 166(1):475. 10.1007/s00701-024-06372-9 [DOI] [PubMed] [Google Scholar]
- 18.Scott RM, Smith ER (2009) Moyamoya disease and moyamoya syndrome. N Engl J Med 360(12):1226–1237. 10.1056/NEJMra0804622 [DOI] [PubMed] [Google Scholar]
- 19.Sun LR, Jordan LC, Smith ER et al (2024) Pediatric moyamoya revascularization perioperative care: a modified Delphi study. Neurocrit Care 40(2):587–602. 10.1007/s12028-023-01788-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Suresh K, Rathi V, Nwosu O et al (n.d.) Utility of GPT-4 as an Informational Patient Resource in Otolaryngology. Published online May 16, 2023. 10.1101/2023.05.14.23289944
- 21.Tripathi S, Patel J, Mutter L, Dorfner FJ, Bridge CP, Daye D (n.d.) Large language models as an academic resource for radiologists stepping into artificial intelligence research. Current problems in diagnostic radiology. Published online 2024:S0363018824002329. 10.1067/j.cpradiol.2024.12.004 [DOI] [PubMed]
- 22.Veeravagu A, Guzman R, Patil CG, Hou LC, Lee M, Steinberg GK (2008) Moyamoya disease in pediatric patients: outcomes of neurosurgical interventions. FOC 24(2):E16. 10.3171/FOC/2008/24/2/E16 [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
Data was collected and stored in REDCap. Please contact the primary investigator for access to the raw data.