Large language models’ performance in answering common patient questions about colonoscopy: an expert-based evaluation

Abstract

Background

This study aimed to compare the performance of two large language models (LLMs) the ChatGPT-4 and the Google Bard in delivering medical information about colonoscopy. It also sought to evaluate the models’ responses to common patient and caregiver questions in terms of accuracy, applicability, comprehensiveness, and communication quality, as assessed by experts using a rubric-based method.

Methods

This comparative, expert-based observational study analyzed the responses of both models to 11 colonoscopy-related questions written in Turkish, simulating real patient inquiries. To ensure consistency and fairness, questions were asked in separate sessions using a standardized zero-shot approach without prior examples or context. The gastroenterology experts from various institutions in Türkiye were recruited using snowball sampling and evaluated the responses using a rubric-based scale across four domains. Data were collected online.

Results

A total of 40 gastroenterologists participated (mean age 39.1 ± 6.4 years). For ChatGPT-4, more than 70% of responses were rated ≥ 4 across all domains, with median overall index scores ranging from 4.1 to 4.7. In contrast, Google Bard demonstrated lower performance, with median scores ranging from 3.6 to 4.0 and lower proportions of high ratings (≥ 4), particularly in comprehensiveness (55.7%) and applicability (58.4%). Comparative analysis showed that ChatGPT-4 significantly outperformed Google Bard across all domains: accuracy (p = 0.003, effect size = 0.47), applicability (p < 0.001, effect size = 0.64), comprehensiveness (p = 0.004, effect size = 0.46), and communication (p = 0.001, effect size = 0.52). No significant differences were observed across evaluator subgroups based on academic title, prior ChatGPT use, or age (all p-values > 0.05).

Conclusions

ChatGPT-4 outperformed Google Bard across all domains, providing more accurate, comprehensive, and patient-centered responses. Despite these findings, human oversight remains essential to ensure safety and reliability in clinical communication. It should also be noted that different versions of LLMs may yield different results.

Supplementary Information

The online version contains supplementary material available at https://doi.org/10.1186/s12876-026-05345-9.

Introduction

Colonoscopy remains the gold standard for the diagnosis, prevention, and follow-up of colorectal diseases, including colorectal cancer, the second leading cause of cancer-related mortality worldwide. However, despite its critical clinical utility, the entire patient experience from bowel preparation and the recovery phase is frequently complicated by significant levels of anxiety and a lack of information [1]. This lack of adequate patient comprehension frequently results in poor adherence to bowel preparation guidelines, subsequently causing increased rates of incomplete procedures and delays in diagnosis [2, 3]. Recent years have seen the widespread emergence of Large Language Models (LLMs), such as OpenAI’s ChatGPT and Google’s Gemini (formerly Bard). These Generative Artificial Intelligence (AI) tools are now widely adopted due to their unprecedented capability to generate near-human text in response to natural language queries [4] The utility of Large Language Models (LLMs) is being intensively explored across the healthcare sector, from streamlining clinical documentation and providing decision support to enhancing patient education [5]. Despite the ability of LLMs to offer easily accessible, conversational medical explanations, their use in delivering health-related information raises persistent concerns regarding their accuracy, completeness, and overall safety [6]. Evaluating these models within specific medical contexts, such as colonoscopy, is therefore essential to determine their reliability and potential role as educational aids. Several recent studies have investigated the performance of LLMs in gastroenterology. For instance, Lee et al. reported that ChatGPT produced accurate and comprehensible responses to common colonoscopy questions, often comparable to reputable hospital websites [7]. Laoveeravat et al. [8] and Mukherjee et al. [9] similarly demonstrated that LLM-based systems could provide guideline-consistent recommendations for colonoscopy surveillance intervals, though sometimes lacking nuance and personalization [8, 9]. In broader analyses, Gong et al. [10] and Berry et al. [11] emphasized the need for structured evaluation frameworks and expert validation before incorporating LLMs into clinical communication workflows [10, 11].

Furthermore, the Nature study by Yang et al. [12]. showed that advanced LLMs outperformed clinicians in complex gastrointestinal diagnostic reasoning, suggesting that these models may effectively supplement, but not replace, human expertise [12]. Similarly, Chen et al. highlighted the importance of human-centered evaluation frameworks in assessing both the accuracy and potential risks (e.g., hallucinations, misinformation) associated with LLMs in healthcare [13]. These findings collectively underscore the need for domain-specific evaluations guided by expert review to understand the strengths and limitations of LLMs in gastroenterology.

Given these considerations, the present study aims to evaluate the performance of two widely available LLMs, ChatGPT-4 (Plus version) and Google Bard (free version, later renamed Gemini), in answering common patient questions about colonoscopy. By employing an expert-based rubric evaluation, this study seeks to provide an evidence-based understanding of how effectively current AI models address real-world patient information needs within gastroenterology.

Materials and methods

This comparative, expert-based observational study evaluates the responses of two major LLMs, ChatGPT-4 and Google Bard (later renamed Gemini) to colonoscopy-related patient questions based on expert assessments. The questions were simulated as if they were asked by patients or their relatives and were submitted to LLMs in a manner typical of patient speech The ethical approval was obtained from the Akdeniz University Clinical Research Ethics Committee. All experts were informed about the purpose of the study, and informed consent was obtained.

This study was designed and reported in accordance with the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) guidelines to ensure transparency and completeness in reporting observational research.

Questions directed to LLMs

Two gastroenterologists with at least 5 years of experience independently listed the most common patient inquiries they encounter in clinical practice, including concerns about preparation, procedure discomfort, risks, and recovery. To ensure linguistic clarity and generalizability, the wording of each question was reviewed to reflect how patients naturally phrase their questions rather than medical jargon. Finally, consensus was reached on 11 core questions that represent the most typical concerns of patients before or after colonoscopy, which served as the basis for evaluating the performance of LLMs. The number of questions was limited to 11 to focus on the most frequently encountered patient concerns in routine clinical practice and to allow a structured and comparable evaluation of LLM responses.

We submitted the same set of 11 frequently asked questions about colonoscopy to two LLMs: the ChatGPT-4 (Plus version, OpenAI) and the free version of Google Bard (Renamed Gemini in February 2024), on September 9, 2023 (Table 1). We did not compare the free and paid versions of the same LLM (e.g., ChatGPT) because our aim was not to evaluate performance differences within a single platform or model. Instead, we intended to assess how two widely accessible, distinct LLMs; one subscription-based (ChatGPT-4) and one freely available at the time (Google Bard), perform when providing medical advice on common patient inquiries. This approach better reflects real-world usage scenarios where users might turn to different AI tools based on availability, cost, and platform preference.

Table 1.

Questions submitted to ChatGPT-4 and Google Bard

Q1	What foods should I eat before and after a colonoscopy?
Q2	Will I feel any pain or discomfort before or after the colonoscopy?
Q3	When can I return to my normal daily activities after the colonoscopy?
Q4	If anesthesia is used during the colonoscopy, are there any related risks?
Q5	Is it normal to experience gas or abdominal bloating after the colonoscopy? How long does it last?
Q6	Are there any changes in bowel movements after a colonoscopy? Is this normal?
Q7	Is it important to drink enough water before the colonoscopy? How much water should I drink?
Q8	Should I stop taking my blood thinner medications before the colonoscopy?
Q9	Should I take my diabetes medications before the colonoscopy? Do I need to change their dose or timing?
Q10	I have kidney failure, should I take the bowel preparation medicine prescribed for the colonoscopy? Are there any risks or special recommendations?
Q11	When should I perform the enema before the colonoscopy?

Each question was asked only once to each model, and no repeated queries were performed. The questions were simulated as if they were asked by patients or their relatives and were submitted to LLMs in a manner typical of patient speech. All questions were asked in Turkish, which reflects the language spoken by the majority of the patient population in the clinical context of this study.

The exact wording of all questions used in the study is provided in Table 1. No prompt engineering techniques were applied; instead, the models were directly presented with patient-like questions to simulate real-world usage. All questions were entered using a standardized zero-shot approach without prior context. Default model settings were used, and no parameter tuning was performed. To ensure consistency, all questions were entered by a single researcher in separate sessions. All questions were submitted to the ChatGPT-4 and Google Bard in (Supplementary File) september 9, 2023. In this study, the models were tested without being given any prior examples or sample answers. This method, known as zero shot prompting, was used to keep the evaluation consistent and fair across all questions. By not showing any examples beforehand, we ensured that the models’ answers reflected their general reasoning ability rather than learning from previous hints. To avoid the models changing their response style based on earlier questions in the same session (in-context adaptation), each question was asked in a new and separate chat session. The chat history was cleared before every question, so the models could not “remember” or adapt from earlier interactions. Each question was asked only once, and there was no word limit for the responses. This approach made sure that every answer represented a fresh, unbiased, and independent output from the model [14, 15].

Evaluator characteristics and data collection process

In this study, responses generated by ChatGPT-4 and Google’s Bard were evaluated by gastroenterology specialists. The assessment was conducted using the rubric-based evaluation method, which allows for structured and standardized comparison of model outputs.

The number of evaluators required for rubric-based studies typically depends on the study’s originality and scope, as there is no strict rule regarding the optimal number. In general, for content validity and expert evaluation studies, a minimum of five evaluators is recommended. However, when a more comprehensive or multi-dimensional evaluation is desired, larger expert panels may be preferred to enhance heterogeneity and the robustness of findings [16–20]. In line with this approach, we aimed to include a relatively large and diverse group of experts and therefore recruited 40 gastroenterology specialists from various institutions across Türkiye. Participants were recruited using a snowball sampling approach. Initial participants were identified among gastroenterology specialists from different institutions known to the research team and were invited to participate. These participants were then asked to refer other eligible experts within their professional networks. The recruitment process continued through approximately 2–3 referral waves until the target sample size was reached. To reduce potential network bias, efforts were made to include experts from different institutions and academic ranks. The data were collected via online evaluation forms prepared in Google Forms, which were distributed to participants through email and WhatsApp. Each expert independently rated the LLM-generated answers according to the defined rubric criteria, ensuring a unbiased assessment process.

Evaluation of ChatGPT responses

The primary objective of this study was to compare the overall performance of ChatGPT-4 and Google Bard in responding to common colonoscopy-related patient questions, based on expert evaluations across predefined domains including accuracy, applicability, comprehensiveness, and communication. The secondary objectives were to examine whether evaluator characteristics influenced scoring patterns. The responses were anonymized, and evaluators were blinded to the source of each response. Experts evaluated the model’s answers using a rubric-based scoring system, rating each response from 1 to 5 across predefined categories. Although rubric-based assessments are commonly used in educational settings to evaluate written work or projects, they have also been effectively applied in healthcare and research contexts to ensure objective and structured evaluations [21–24].

The evaluation domains for this study were determined through a comprehensive review of previous research on LLM performance in healthcare. These domains (accuracy, applicability, comprehensiveness, and communication quality) were selected based on conceptual frameworks and assessment criteria used in earlier studies evaluating AI-generated medical information [20, 24–27]. Prior to data collection, the rubric was reviewed by the research team to ensure clarity and relevance to the study context. However, no formal pilot testing or psychometric validation was performed.

Drawing from these studies, the expert panel assessed each model response according to four main criteria: accuracy, applicability, comprehensiveness, and communication quality (Table 2).

Table 2.

The rubric evaluation for physicians (Experts)

Criteria	1 Point	2 Points	3 Points	4 Points	5 Points
Accuracy	The information is completely incorrect or missing.	The information is mostly incorrect or missing.	The information is partially correct but incomplete.	The information is mostly correct but has some omissions.	The information is completely correct.
Applicability	The response is not applicable.	The response is barely applicable.	The response is somewhat applicable.	The response is mostly applicable.	The response is fully applicable.
Comprehensiveness	The response covers only one aspect.	The response covers very few aspects.	The response covers some important aspects.	The response covers most important aspects.	The response covers all important aspects.
Communication	The response is not understandable.	The response is difficult to understand.	The response is partially understandable.	The response is mostly understandable.	The response is completely understandable.

Each expert independently rated the model’s answers for all criteria. A Likert score of ≥ 4 indicated a highly satisfactory response. The “Overall Evaluation Percentage” represents the mean proportion of evaluators who rated responses as 4 or higher across all responses and criteria [24]. An overall index score for each domain was then computed by aggregating and averaging all ratings within that category, providing a single composite measure per domain [24].

For each evaluation domain (accuracy, applicability, comprehensiveness, and communication), an overall index score was calculated for each expert by averaging their ratings across all 11 questions. Specifically, if an expert rated the responses as x₁, x_2,… x₁₁ on a 5-point Likert scale, the overall index score was computed as:

This calculation was performed separately for each domain and for each model. The distribution of these index scores across all experts was then summarized using descriptive statistics.

All evaluators completed all ratings in full, and no domain-level data were missing. Therefore, no imputation or special handling of missing data was required.

Statistics

Statistical analyses were performed with IBM SPSS 26.0 (SPSS Inc., Chicago, USA). The distribution of continuous variables was assessed through both visual methods (such as histograms and probability plots) and statistical tests (Shapiro-Wilk tests). Descriptive statistics presented as number, percentage, mean±standard deviation and median (minimum; maximum). The Wilcoxon Signed Ranks Test was used for pairwise comparisons of the overall index scores of evaluation criteria. Effect sizes for Wilcoxon signed-rank tests were calculated as r=∣Z∣/N, where N represents the number of paired observations [28]. Inter-rater reliability metrics were not calculated, as the aim was to capture independent expert evaluations rather than agreement between raters.

A p-value of < 0.05 was considered statistically significant.

Results

Characteristics of experts

A total of 40 experts participated in the study. The mean age of the participants was 39.1 ± 6.4 years, ranging from 29 to 54 years, with a median of 37.0 years. Most participants were male (n = 25, 62.5%), and 15 were female (37.5%). The mean professional experience of the gastroenterology specialists participating in the study was 11.5 ± 6.1 years, with a median of 11.0 years. The minimum and maximum years of experience were 1.0 and 24.0 years, respectively. Regarding academic titles, 13 participants (32.5%) were associate professors or professors and 27 (67.5%) were specialist doctors. Among the 40 gastroenterologists, 47.5% (n = 19) had previously used ChatGPT and 15.0% (n = 6) had used Google Bard (all of whom had also used ChatGPT), including for various professional purposes.

Evaluation of LLMs responses

For each question, experts rated ChatGPT’s responses as ≥ 4 (highly satisfactory) across all four evaluation domains in more than 70% of cases (Table 3). The proportion of high ratings was highest for Accuracy (80.2%), followed by Applicability (75.0%), Comprehensiveness (74.5%), and Communication (72.9%). The overall mean scores were 4.3 ± 0.8, 4.2 ± 0.9, 4.1 ± 0.9, and 4.1 ± 0.8, respectively. Median overall index scores ranged from 4.1 to 4.7.

Table 3.

Evaluation of ChatGPT-4’s answers to the questions by the experts according to the scoring rubric in terms of accuracy, applicability, comprehensiveness, and communication

Experts, N = 40	Experts with Likert Response of ≥ 4, n (%)	Mean ± SD Score	Median (Min-max) Score
Question 1
Accuracy	34 (85.0)	4.3 ± 0.8	4.0 (2.0–5.0)
Applicability	31 (77.5)	4.1 ± 0.9	4.0 (2.0–5.0)
Comprehensiveness	28 (70.0)	4.0 ± 1.1	4.0 (1.0–5.0)
Communication	29 (72.5)	4.0 ± 1.0	4.0 (2.0–5.0)
Question 2
Accuracy	34 (85.0)	4.3 ± 0.8	4.0 (2.0–5.0)
Applicability	33 (82,5)	4.2 ± 0.9	4.0 (2.0–5.0)
Comprehensiveness	29 (72.5)	4.1 ± 0.8	4.0 (2.0–5.0)
Communication	32 (80.0)	4.2 ± 0.8	4.0 (2.0–5.0)
Question 3
Accuracy	32 (80.0)	4.3 ± 0.9	5.0 (2.0–5.0)
Applicability	32 (80.0)	4.3 ± 1.0	5.0 (2.0–5.0)
Comprehensiveness	32 (80.0)	4.2 ± 0.9	4.0 (2.0–5.0)
Communication	31 (77.5)	4.2 ± 0.9	4.0 (2.0–5.0)
Question 4
Accuracy	29 (72.5)	4.2 ± 1.0	5.0 (2.0–5.0)
Applicability	30 (75.0)	4.2 ± 1.0	4.5 (2.0–5.0)
Comprehensiveness	30 (75.0)	4.1 ± 0.9	4.0 (2.0–5.0)
Communication	29 (72.5)	4.2 ± 1.0	4.5 (2.0–5.0)
Question 5
Accuracy	33 (82,5)	4.4 ± 0.8	5.0 (2.0–5.0)
Applicability	31 (77.5)	4.3 ± 0.9	5.0 (2.0–5.0)
Comprehensiveness	32 (80.0)	4.2 ± 1.0	5.0 (1.0–5.0)
Communication	30 (75.0)	4.2 ± 0.9	4.0 (2.0–5.0)
Question 6
Accuracy	35 (87.5)	4.5 ± 0.8	5.0 (2.0–5.0)
Applicability	31 (77.5)	4.3 ± 0.9	5.0 (2.0–5.0)
Comprehensiveness	34 (85.0)	4.3 ± 0.9	5.0 (1.0–5.0)
Communication	29 (72.5)	4.2 ± 1.0	5.0 (2.0–5.0)
Question 7
Accuracy	32 (80.0)	4.4 ± 0.9	5.0 (2.0–5.0)
Applicability	28 (70.0)	4.2 ± 1.0	5.0 (2.0–5.0)
Comprehensiveness	30 (75.0)	4.1 ± 1.0	4.0 (1.0–5.0)
Communication	28 (70.0)	4.1 ± 1.0	4.0 (2.0–5.0)
Question 8
Accuracy	32 (80.0)	4.3 ± 1.1	5.0 (1.0–5.0)
Applicability	31 (77.5)	4.2 ± 1.1	5.0 (2.0–5.0)
Comprehensiveness	30 (75.0)	4.0 ± 1.2	4.0 (1.0–5.0)
Communication	30 (75.0)	4.0 ± 1.1	4.0 (1.0–5.0)
Question 9
Accuracy	30 (75.0)	4.1 ± 1.1	4.0 (1.0–5.0)
Applicability	28 (70.0)	4.0 ± 1.0	4.0 (2.0–5.0)
Comprehensiveness	28 (70.0)	3.8 ± 1.2	4.0 (1.0–5.0)
Communication	28 (70.0)	4.0 ± 1.0	4.0 (2.0–5.0)
Question 10
Accuracy	33 (82.5)	4.3 ± 0.9	4.5 (2.0–5.0)
Applicability	30 (75.0)	4.1 ± 1.0	4.0 (2.0–5.0)
Comprehensiveness	30 (75.0)	4.1 ± 1.1	4.0 (1.0–5.0)
Communication	29 (72.5)	4.1 ± 1.0	4.0 (2.0–5.0)
Question 11
Accuracy	32 (80.0)	4.3 ± 1.0	5.0 (2.0–5.0)
Applicability	28 (70.0)	4.1 ± 1.0	4.5 (2.0–5.0)
Comprehensiveness	29 (72.5)	4.1 ± 1.1	4.0 (1.0–5.0)
Communication	28 (70.0)	4.1 ± 1.0	4.0 (2.0–5.0)
All Questions	Overall Index Percentage of Responses Scoring ≥ 4, %*	Overall Index Score**	Overall Index Score**
Accuracy	80.2	4.3 ± 0.8	4.7 (1.9-5.0)
Applicability	75.0	4.2 ± 0.9	4.5 (2.0–5.0)
Comprehensiveness	74.5	4.1 ± 0.9	4.2 (1.4-5.0)
Communication	72.9	4.1 ± 0.8	4.1 (2.1-5.0)

*: The percentage of responses scoring ≥ 4 across all eleven questions are provided for each of the four evaluation areas **: The overall index scores for all questions are derived by aggregating and averaging the scores from all four areas across each question

In contrast, Google Bard’s responses received lower ratings in all domains (Table 4). The highest proportion of experts assigning scores ≥ 4 was again for accuracy (69.1%), followed by Communication (60.5%), Applicability (58.4%), and Comprehensiveness (55.7%). Mean domain scores ranged between 3.6 ± 1.0 and 4.0 ± 0.7.

Table 4.

Evaluation of Google Bard’s answers to the questions by the experts according to the scoring rubric in terms of accuracy, applicability, comprehensiveness, and communication

Experts, N = 40	Experts with Likert Response of ≥ 4, n (%)	Mean ± SD Score	Median (Min-max) Score
Question 1
Accuracy	20 (50.0)	3.5 ± 1.3	3.5 (1.0–5.0)
Applicability	16 (40.0)	3.3 ± 1.2	3.0 (1.0–5.0)
Comprehensiveness	23 (57.5)	3.5 ± 1.3	4.0 (1.0–5.0)
Communication	22 (55.0)	3.4 ± 1.2	4.0 (1.0–5.0)
Question 2
Accuracy	32 (80.0)	4.1 ± 0.8	4.0 (2.0–5.0)
Applicability	24 (60.0)	3.7 ± 1.0	4.0 (1.0–5.0)
Comprehensiveness	27 (67.5)	3.9 ± 1.2	4.0 (1.0–5.0)
Communication	26 (65.0)	3.7 ± 1.1	4.0 (1.0–5.0)
Question 3
Accuracy	26 (65.0)	3.9 ± 1.0	4.0 (1.0–5.0)
Applicability	26 (65.0)	3.7 ± 1.2	4.0 (1.0–5.0)
Comprehensiveness	22 (55.0)	3.7 ± 1.2	4.0 (1.0–5.0)
Communication	24 (60.0)	3.6 ± 1.1	4.0 (1.0–5.0)
Question 4
Accuracy	30 (75.0)	4.2 ± 0.9	4.0 (2.0–5.0)
Applicability	27 (67.5)	3.9 ± 1.0	4.0 (1.0–5.0)
Comprehensiveness	25 (62.5)	3.9 ± 1.0	4.0 (1.0–5.0)
Communication	27 (67.5)	3.8 ± 1.0	4.0 (1.0–5.0)
Question 5
Accuracy	29 (72.5)	4.0 ± 0.9	4.0 (2.0–5.0)
Applicability	25 (62.5)	3.9 ± 0.9	4.0 (2.0–5.0)
Comprehensiveness	24 (60.0)	3.8 ± 1.2	4.0 (1.0–5.0)
Communication	22 (55.0)	3.8 ± 1.1	4.0 (1.0–5.0)
Question 6
Accuracy	29 (72.5)	4.0 ± 0.9	4.0 (2.0–5.0)
Applicability	25 (62.5)	3.8 ± 1.0	4.0 (2.0–5.0)
Comprehensiveness	24 (60.0)	3.7 ± 1.1	4.0 (1.0–5.0)
Communication	22 (55.0)	3.7 ± 1.0	4.0 (1.0–5.0)
Question 7
Accuracy	30 (75.0)	4.1 ± 0.9	4.0 (2.0–5.0)
Applicability	26 (65.0)	3.9 ± 1.1	4.0 (2.0–5.0)
Comprehensiveness	24 (60.0)	3.8 ± 1.1	4.0 (2.0–5.0)
Communication	26 (65.0)	3.8 ± 1.0	4.0 (2.0–5.0)
Question 8
Accuracy	28 (70.0)	4.0 ± 0.9	4.0 (2.0–5.0)
Applicability	23 (57.5)	3.7 ± 1.1	4.0 (1.0–5.0)
Comprehensiveness	20 (50.0)	3.6 ± 1.2	3.5 (1.0–5.0)
Communication	23 (57.5)	3.6 ± 1.1	4.0 (1.0–5.0)
Question 9
Accuracy	29 (72.5)	4.1 ± 0.9	4.0 (2.0–5.0)
Applicability	25 (62.5)	3.7 ± 1.1	4.0 (2.0–5.0)
Comprehensiveness	21 (52.5)	3.6 ± 1.2	4.0 (1.0–5.0)
Communication	26 (65.0)	3.8 ± 1.1	4.0 (1.0–5.0)
Question 10
Accuracy	27 (67.5)	4.0 ± 0.9	4.0 (2.0–5.0)
Applicability	21 (52.5)	3.5 ± 1.1	4.0 (1.0–5.0)
Comprehensiveness	19 (47.5)	3.5 ± 1.1	3.0 (1.0–5.0)
Communication	23 (57.5)	3.6 ± 1.0	4.0 (1.0–5.0)
Question 11
Accuracy	26 (65.0)	3.9 ± 0.9	4.0 (2.0–5.0)
Applicability	18 (45.0)	3.4 ± 1.1	3.0 (1.0–5.0)
Comprehensiveness	17 (42.5)	3.4 ± 1.2	3.0 (1.0–5.0)
Communication	21 (52.5)	3.4 ± 1.2	4.0 (1.0–5.0)
All Questions	Overall Index Percentage of Responses Scoring ≥ 4, %*	Overall Index Score**	Overall Index Score**
Accuracy	69.1	4.0 ± 0.7	4.0 (2.6-5.0)
Applicability	58.4	3.7 ± 0.9	3.8 (2.0–5.0)
Comprehensiveness	55.7	3.6 ± 1.0	3.9 (1.4-5.0)
Communication	60.5	3.7 ± 0.9	3.8 (1.2-5.0)

*: The percentage of responses scoring ≥ 4 across all eleven questions are provided for each of the four evaluation areas **: The overall index scores for all questions are derived by aggregating and averaging the scores from all four areas across each question

Overall, ChatGPT demonstrated a consistently higher level of performance across all evaluation domains (Fig. 1).

Fig. 1.

Overall index percentage of experts’ responses scoring ≥ 4, %

No statistically significant differences in rubric scores were observed across evaluator subgroups stratified by academic title, prior LLM use, or age (Table 5).

Table 5.

Comparison of overall index scores by evaluator characteristics for ChatGPT-4 and Google Bard

	Overall Index Scores* for ChatGPT-4 Responses			Overall Index Scores* for Google Bard responses
	Median (Min-max) Score	Median (Min-max) Score	p value**	Median (Min-max) Score	Median (Min-max) Score	p value**
Groups	Specialist Doctor n  = 27	Associate Professor/Professor n  = 13		Specialist Doctor n  = 27	Associate Professor/Professor n  = 13
Accuracy	4.6 (1.9-5.0)	4.7 (3.0–5.0)	0.464	4.0 (2.6-5.0)	4.0 (2.9-5.0)	0.885
Applicability	4.5 (2.0–5.0)	4.5 (2.0–5.0)	0.611	3.9 (2.0–5.0)	3.5 (2.0–5.0)	0.470
Comprehensiveness	4.4 (1.4-5.0)	4.2 (2.6-5.0)	0.942	4.0 (1.4-5.0)	3.4 (1.5-5.0)	0.291
Communication	4.5 (2.6-5.0)	4.0 (2.0–5.0)	0.571	3.8 (1.6-5.0)	3.8 (1.2-5.0)	0.633
Groups	Not using LLM n  = 21	Using LLMs n  = 19		Not using LLM n  = 21	Using LLMs n  = 19
Accuracy	4.6 (2.9-5.0)	4.7 (1.9-5.0)	0.999	4.0 (2.9-5.0)	4.0 (2.6-5.0)	0.550
Applicability	4.5 (2.0–5.0)	4.5 (2.0–5.0)	0.437	3.8 (2.0–5.0)	3.6 (2.0–5.0)	0.818
Comprehensiveness	4.4 (2.6-5.0)	4.2 (1.4-5.0)	0.828	4.0 (1.4-5.0)	3.5 (1.5-5.0)	0.472
Communication	4.0 (2.0–5.0)	4.3 (2.6-5.0)	0.683	4.0 (1.6-5.0)	3.5 (1.2-5.0)	0.550
Groups	Age ≤ 40 years old n = 26	Age > 40 years old, n  = 14		Age ≤ 40 years old n= = 26	Age > 40 years old n  = 14
Accuracy	4.7 (1.9-5.0)	4.6 (3.0–5.0)	0.527	4.0 (2.6-5.0)	3.9 (2.9-5.0)	0.486
Applicability	4.6 (2.0–5.0)	4.5 (2.0–5.0)	0.887	3.9 (2.0–5.0)	3.7 (2.0–5.0)	0.496
Comprehensiveness	4.1 (1.4-5.0)	4.3 (2.6-5.0)	0.690	4.0 (1.4-5.0)	3.4 (1.5-5.0)	0.386
Communication	4.3 (2.6-5.0)	4.1 (2.0–5.0)	0.977	3.9 (1.6-5.0)	3.7 (1.2-5.0)	0.551

*: The overall index scores for all scenarios is derived by aggregating and averaging the scores from all four areas across each scenario **: Mann-Whitney-U test

Median overall index scores remained stable across all strata, suggesting that evaluators’ background characteristics did not influence scoring trends. When comparing the median overall index scores of ChatGPT and Google Bard based on expert evaluations, ChatGPT received significantly higher scores across all four evaluation categories: accuracy (p = 0.003, effect size = 0.47), applicability (p < 0.001, effect size = 0.64), comprehensiveness (p = 0.004, effect size = 0.46), and communication (p = 0.001, effect size = 0.52).

Discussion

The findings of this study demonstrate that LLMs can provide accurate, understandable, and clinically relevant information in response to common patient questions about colonoscopy. Among the evaluated domains, accuracy consistently received the highest scores, indicating that the outputs of both models were generally perceived by experts as factually correct and contextually appropriate. However, notable differences were observed between the models. While Google Bard’s responses were coherent and grammatically correct, they were often less detailed, less clinically specific, and occasionally omitted essential information, such as risk details or contraindications. These differences were also reflected in the quantitative results. The proportion of responses rated ≥ 4 was consistently higher for ChatGPT across all domains (accuracy: 80.2% vs. 69.1%, applicability: 75.0% vs. 58.4%, comprehensiveness: 74.5% vs. 55.7%, communication: 72.9% vs. 60.5%). This pattern suggests that ChatGPT not only provided accurate responses but also delivered more applicable, comprehensive, and better-communicated information.

However, these findings should be interpreted in light of the model configurations used in this study. ChatGPT-4 was accessed through a subscription-based (Plus) version, whereas Google Bard was evaluated as a freely available model at the time of data collection. Differences in access level and model version may have contributed to variations in response detail and clinical specificity. Therefore, the observed performance gap may not solely reflect intrinsic model capabilities. The rationale for this study design was to reflect real-world usage scenarios, in which patients and healthcare providers may interact with different AI tools depending on accessibility and cost. Several previous studies have adopted similar comparative approaches, emphasizing the importance of evaluating different commercial LLMs rather than focusing solely on version differences within a single platform [8, 29, 30].

These results align with findings from prior work in gastroenterology and medical AI. For instance, in Lee et al.’s landmark study, gastroenterologists rated ChatGPT’s responses to common colonoscopy questions as comparable or superior to those from major hospital websites, especially in understandability and scientific adequacy [7]. Similarly, Mukherjee et al. assessed ChatGPT’s ability to answer surveillance colonoscopy interval queries and found that it achieved high concordance with guideline-based recommendations, though occasional oversimplifications were noted [9]. Another study by Laoveeravat et al. investigated ChatGPT’s role in answering patient queries about surveillance colonoscopy, also showing promising performance in communicating interval recommendations [8]. Beyond colonoscopy-specific studies, a broader perspective emerges in the 2024 systematic review by Gong et al., which surveyed LLM applications in gastroenterology across diagnostics, patient education, and documentation. They concluded that while LLMs show real promise, variability in domain-specific accuracy and phrasing sensitivity remain limiting factors [10]. Likewise, Berry et al. proposed a clinical integration framework, highlighting that LLMs must be carefully validated in gastroenterology tasks such as hepatitis C management and endoscopic decision support, underscoring the importance of domain adaptation and oversight [11].

Taken together, these studies and our findings suggest that LLMs, particularly high-performing models like ChatGPT, can meaningfully support patient education and enhance access to trustworthy medical information in gastroenterology. The relatively high accuracy scores in our study align with prior reports, suggesting that ChatGPT provides reliable factual content when addressing patient-level questions. However, the lower but still substantial scores in comprehensiveness and communication indicate that while ChatGPT delivers relevant information, it sometimes lacks nuance or patient-centered phrasing that would improve empathy and engagement.

Responses related to clinically sensitive conditions such as anticoagulant use, diabetes management, and renal failure carry significant risks, including bleeding or thromboembolic complications, hypoglycemia, and electrolyte or volume imbalances. Upon review of the responses, no recommendations that were clearly unsafe or directly harmful were identified in either model. However, significant differences were observed in the depth and clinical specificity of the responses. While ChatGPT tended to provide more detailed explanations, including risk-benefit assessments, Google Bard’s responses were generally more superficial. In both models, the responses did not demonstrate consistent alignment with guideline-based, personalized management strategies. Although the frequently repeated recommendation to consult a doctor reflects a cautious approach, relying on these responses without professional guidance could still pose potential risks. These qualitative findings are also supported by the expert evaluation results. For clinically sensitive questions (Q8–Q10), both models achieved moderate-to-high accuracy scores; however, ChatGPT consistently demonstrated higher performance, particularly in comprehensiveness and applicability, whereas Google Bard showed lower scores in these domains.

Since this study was conducted in Turkish, differences in model performance may partly reflect differences in language proficiency, as large language models are typically trained on more extensive English-language datasets. Therefore, if the same questions had been asked in English or other high-resource languages, the performance gap between the models might have differed. On the other hand, the observed performance difference may not be solely due to language factors. The fact that the models used belong to different versions (e.g., Plus vs. Bard’s free version) may have played a role in this difference. However, since this study was not designed to compare different versions of the same model, the extent of this effect cannot be definitively determined. This highlights a significant limitation regarding generalizability and underscores the need for future studies to conduct comparisons both across different languages and between different versions of the same model.

A major strength of this study lies in the use of a large expert panel comprising 40 gastroenterologists from multiple institutions, ensuring diverse clinical perspectives and reducing individual bias. The use of a structured rubric evaluation system allowed for objective, reproducible, and domain-specific assessment across multiple dimensions, including accuracy, applicability, comprehensiveness, and communication. Additionally, the zero-shot prompting approach minimized the risk of contextual bias and ensured that each response represented the model’s true baseline reasoning capacity. The inclusion of both ChatGPT and Google Bard also allowed for comparative analysis between two major LLM architectures, enhancing the study’s external validity. Another strength of this study is the evaluation of LLM performance in a non-English language. Most existing studies on LLMs in healthcare have been conducted in English; however, patients worldwide access health information in their native languages. Therefore, it is crucial to reflect real-world usage and assess the applicability of these tools beyond English-speaking communities. Additionally, previous research has shown that LLM performance can vary across languages, particularly in settings with limited resources where training data may be scarce or in non-English environments. This can affect both the accuracy and clarity of the generated responses. Our study contributes to the literature by providing evidence on LLM performance in Turkish, a language relatively underrepresented in AI research. The multilingual performance of AI tools in health communication has been evaluated.

These findings suggest that AI models have significant potential not only in conveying information but also in supporting patient trust and understanding [31]. However, human aspects such as empathy, emotional awareness and personal approach still remain. This study also reminds us that the human touch is indispensable in the integration of AI into health communication.

Despite their promising performance, the potential risks of using LLMs in patient communication should be carefully considered. One major concern is the possibility of “hallucinations,” where models may produce misleading information. In addition, inconsistencies and the omission of clinically significant details may lead to patient misunderstandings. Over-reliance on AI-generated content without professional validation can also result in inappropriate health decisions [32].

Therefore, LLMs should not be used as standalone sources of medical advice but rather as supplementary tools for patient education [33, 34]. In gastroenterology, these models may be integrated into hospital websites or patient information platforms to provide standardized answers to frequently asked questions, such as those related to colonoscopy preparation and recovery. In such applications, clinician oversight remains essential to ensure that the information provided is accurate, up-to-date, and aligned with clinical guidelines [7]. Clear guidance should also be provided to patients, emphasizing that LLM-generated content serves as supportive information rather than authoritative medical advice.

Limitations of the study

This study has several limitations that should be considered when interpreting the findings. First, the use of snowball sampling may have introduced selection bias, as participating experts could belong to similar professional networks. In addition, a relatively high proportion of evaluators reported prior experience with ChatGPT, which may have influenced their perceptions and scoring of model responses. Second, although a structured rubric was used to standardize evaluations, it was not formally validated through psychometric testing or pilot calibration. Third, inter-rater reliability metrics such as ICC or kappa were not calculated, as the primary aim was not to assess agreement among experts but to capture individual expert judgments on LLM-generated responses. Fourth, the study was conducted using Turkish-language questions, which reflect real-world patient communication in our setting; however, this may limit the generalizability of the findings to other languages and cultural contexts. Fifth, we focused on comparing two widely used and accessible LLMs, one subscription-based and one freely available, to better reflect real-world usage scenarios in which users may choose different tools based on availability, cost, and platform preference. The comparison between a paid model (ChatGPT-4) and a free model (Google Bard) may introduce inherent performance differences, and the results reflect the specific versions available at the time of the study. Given the rapid evolution of LLMs, the findings may not be directly applicable to newer versions. Additionally, expert-based evaluations may not fully represent patient perspectives, as patients may prioritize factors such as empathy and simplicity differently. Additionally, the set of questions was developed by two gastroenterologists without direct involvement of patients, caregivers, or other healthcare professionals. Although efforts were made to reflect common patient concerns, the limited number of contributors and the restriction to 11 questions may not fully capture the diversity and breadth of colonoscopy-related information needs. Another important limitation is that LLM outputs are inherently non-deterministic and may vary across different sessions. In this study, each question was asked only once at a single time point (September 9, 2023). Therefore, the findings represent a snapshot of the specific model versions available at that time and may not fully reflect variability in responses or generalize to future model updates. No adjustment for multiple comparisons was performed, which may increase the risk of type I error. Finally, the study focused exclusively on colonoscopy-related questions, which may limit the applicability of the results to other gastrointestinal conditions or medical domains.

Conclusion

The findings of this study highlight that LLMs can serve as valuable educational tools for improving patient understanding of colonoscopy preparation, risks, and recovery. However, human oversight is essential to verify the accuracy and completeness of information before integration into clinical workflows or public-facing educational materials. In particular, the subscription-based version of ChatGPT-4 demonstrated superior performance across multiple domains, including accuracy, applicability, and comprehensiveness. However, this difference should be interpreted with caution, as variations in model access and version may have contributed to the observed performance gap. Future studies should aim to evaluate model performance in other procedural contexts, expand linguistic and cultural coverage, and investigate the impact of AI-generated content on patient anxiety, adherence, and informed consent. Collaborative frameworks involving clinicians, AI developers, and health communication experts will be critical to ensure that LLMs are implemented safely, ethically, and equitably within gastroenterology practice.

Supplementary Information

Below is the link to the electronic supplementary material.

Abbreviations

LLMs

Large Language Models

AI

Artificial Intelligence

Author contributions

All authors contributed to the study design. Data collection NGA and AMA and analysis were performed by DY, ETS and the manuscript was written by NGA, YDA and DY.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Data availability

The data set is available from the corresponding author on reasonable request.

Declarations

Ethics approval and consent to participate

In order to conduct the research, approval was obtained from the Akdeniz University Clinical Research Ethics Committee (Approval number: KAEK-682, Date:23.08.2023). And informed consent to participate was obtained from all of the participants in the study.

Patient consent statement

Institutional permission was also received while informed and verbal consent was collected from the experts.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.