Evaluating large language models for glaucoma counseling: A pilot study of ChatGPT and Google Gemini responses in Traditional Chinese

Abstract

PURPOSE:

Large language models (LLMs) such as ChatGPT and Google Gemini are increasingly used for patient education in various medical fields. However, their effectiveness in providing ophthalmology-related information in non-English languages remains underexplored. This study aimed to evaluate and compare the appropriateness of ChatGPT and Google Gemini responses to glaucoma-related questions presented in Traditional Chinese.

MATERIALS AND METHODS:

Twenty frequently asked glaucoma-related questions were translated into Traditional Chinese and categorized into four domains: disease understanding, diagnosis, treatment, and lifestyle. Responses were generated using ChatGPT (GPT-4) and Google Gemini. Three qualified ophthalmologists independently rated the appropriateness of each response using a 5-point Likert scale. Mean scores were calculated, and statistical comparisons were performed for overall and category-specific performance.

RESULTS:

ChatGPT achieved a higher overall appropriateness score (4.32 ± 0.65) than Google Gemini (4.03 ± 0.66), with a statistically significant difference (P = 0.004). Across all four content domains, ChatGPT consistently outperformed Gemini, although the differences did not reach statistical significance. Both models demonstrated relatively lower performance in lifestyle-related questions.

CONCLUSION:

Both ChatGPT and Google Gemini are capable of providing reasonably appropriate glaucoma-related information in Traditional Chinese. ChatGPT demonstrated superior overall performance. These findings support the potential role of LLMs in enhancing patient education in non-English settings, although further improvements in language localization and domain-specific accuracy are warranted.

Introduction

In recent years, large language models (LLMs), such as ChatGPT and Google Gemini (the successor of Google Bard), have emerged as powerful tools for natural language processing and expanded applications in health care.[1,2,3,4] For instance, in one study, Google Gemini and ChatGPT analyzed vitreoretinal patient records clearly and consistently, demonstrating strong alignment with expert surgeons’ assessments. Based on the global quality score, ChatGPT’s suggestions were more accurate and precise than Google Gemini. Gemini was not able to respond to some cases.[5] Another study compares Gemini artificial intelligence (AI) and ChatGPT against ophthalmology residents using real board exam data, revealing that Gemini advanced outperforms the others with notable improvements. ChatGPT-4 also shows meaningful progress, highlighting the growing potential of AI in medical education. The findings underscore AI’s effectiveness in ophthalmology while pointing to areas for further development.[6] Meanwhile, the general population increasingly uses these AI-powered chatbots to obtain health-related information, including chronic eye diseases such as glaucoma.[1,7] As a leading cause of irreversible blindness worldwide, glaucoma requires long-term management and patient adherence, both of which are significantly influenced by the quality and clarity of patient education.

With the increasingly widespread accessibility of AI chatbots, patients may use these platforms for quicker answers more frequently.[7,8,9,10] Nevertheless, LLMs are typically trained on broad, nonspecialized datasets and may produce diverse responses in terms of accuracy, comprehensiveness, and safety, especially when applied to subspecialty areas like ophthalmology. Previous studies have compared ChatGPT and other AI models in the context of glaucoma, indicating both their potentials and limitations. For example, Tan et al. assessed the performance of ChatGPT-3.5 in answering glaucoma-related questions and found that while many responses were appropriate, some lacked completeness or contained inaccuracies, particularly in complex pathophysiologic explanations.[8] Yalla et al. further compared the outputs of ChatGPT, Google Bard, and Bing, reporting variability in response quality and readability.[10] Ichhpujani et al. emphasized the trade-off between accuracy and readability among different chatbots when discussing glaucoma surgeries.[11]

Despite these growing efforts, there are few studies examining chatbot performance in Traditional Chinese, which is commonly used in regions such as Taiwan and Hong Kong. In addition, a comparison between ChatGPT and Google Gemini — two of the most widely used LLM chatbots – has not been conducted specifically for Traditional Chinese glaucoma-related patient education.

In this study, we aimed to evaluate and compare the appropriateness of glaucoma-related responses generated by ChatGPT and Google Gemini in Traditional Chinese. Twenty frequently asked questions were translated across four key domains: disease understanding, diagnosis, treatment, and lifestyle-related concerns. Responses were rated by three qualified ophthalmologists using a standardized Likert scale rubric. Our goal was to assess the relative quality of chatbot-generated answers and identify potential strengths and weaknesses of each system in the context of patient counseling.

Materials and Methods

Study design

This was a cross-sectional study evaluating the appropriateness of AI-generated responses to glaucoma-related questions presented in Traditional Chinese. The study compared two LLM chatbots – ChatGPT (GPT-4, OpenAI) and Google Gemini – using professional evaluations by ophthalmologists. The primary goal was to assess and compare the quality of chatbot responses across different categories of patient education.

Question selection

A total of 20 commonly asked glaucoma-related questions were collected based on clinical experience and frequently encountered patient concerns in the ophthalmic clinic. These questions were divided into four categories: (1) disease-related (n = 7), (2) diagnosis (n = 3), (3) treatment (n = 6), and (4) lifestyle and other concerns (n = 4). Each question was written in both English and Traditional Chinese to ensure accurate and culturally relevant translation [Table 1].

Table 1.

20 frequently asked questions from glaucoma patients in English and Traditional Chinese

Category	English questions	中文問題
Disease-related	What is glaucoma?	青光眼是什麼?
	Why did I get glaucoma?	為什麼我會得青光眼?
	Will glaucoma make me go blind?	青光眼會不會失明?
	Can glaucoma be cured?	青光眼可以治好嗎?
	Can younger people get glaucoma too?	我年紀不大也可能得青光眼嗎?
	Is glaucoma hereditary?	青光眼會遺傳嗎?
	Why do I have glaucoma if my eye pressure is normal?	我眼壓正常, 為什麼還說我是青光眼?
Diagnosis	What regular tests do I need for glaucoma?	青光眼需要定期做哪些檢查?
	What’s the difference between OCT and visual field tests?	光學斷層掃描和視野檢查有什麼不同?
	Do I still need follow-up visits if I have no symptoms?	沒有症狀也需要一直回診嗎?
Treatment	Why do I have to use eye drops every day?	為什麼要每天點眼藥水?
	What should I do if I forget to use my eye drops?	忘記點眼藥水怎麼辦?
	Do glaucoma eye drops have side effects?	眼藥水會有副作用嗎?
	What if the eye drops stop working?	如果眼藥水沒效怎麼辦?
	Will I definitely need surgery for glaucoma?	青光眼一定會需要手術嗎?
	Is glaucoma surgery risky?	手術會不會很危險?
Lifestyle and other concerns	Can I still exercise if I have glaucoma?	青光眼可以運動嗎?
	Can stress or staying up late worsen glaucoma?	壓力或熬夜會讓青光眼變嚴重嗎?
	Is it safe for me to drive with glaucoma?	我還可以安全開車嗎?
	Can people with glaucoma have LASIK or other laser eye surgery?	青光眼患者可以做近視雷射嗎?

OCT=Optical coherence tomography

Chatbot response generation

The 20 Traditional Chinese questions were entered into both ChatGPT (GTP-4) and Google Gemini (Gemini 2.0 Flash) using new chat sessions to minimize the influence from previous interactions. Every question was submitted once per chatbot using a standardized prompt format. Responses were created in March 2025 using the default settings of each chatbot system.

Grading and evaluation

Each AI-generated response was independently graded by three qualified ophthalmologists with experience and enough knowledge in glaucoma care. Graders were blinded to each other’s scores and the type of the chatbot. A five-point Likert scale was used to rate the appropriateness of each response [Table 2]:

Table 2.

Rubrics provided to the graders with corresponding numerical scores

Score	Description
5	Strongly agree
4	Agree
3	Neutral
2	Disagree
1	Strongly disagree

• 5 = Strongly agree (excellent and highly appropriate)

• 4 = Agree (appropriate with minor omissions)

• 3 = Neutral (acceptable but lacking important elements)

• 2 = Disagree (inappropriate or incomplete)

• 1 = Strongly disagree (inaccurate or potentially misleading).

Each grader reviewed all 40 responses (20 from ChatGPT and 20 from Gemini), resulting in 120 scores.

Data analysis

Descriptive statistics were calculated for each grader and chatbot, including mean scores and standard deviations [Table 3]. The overall performance and category-specific performance were compared between two chatbots using Mann–Whitney U-tests. P < 0.05 was considered statistically significant [Table 4].

Table 3.

Descriptive statistics from 3 graders

Type	Grader 1		Grader 2		Grader 3
	ChatGPT	Gemini	ChatGPT	Gemini	ChatGPT	Gemini
n	20	20	20	20	20	20
Mean±SD	4.65±0.49	4.45±0.60	4.1±0.72	3.6±0.60	4.2±0.62	4.05±0.51

SD=Standard deviation

Table 4.

Comparison of performance of different large language model chatbots

	ChatGPT	Gemini	P
Overall	4.32±0.65	4.03±0.66	0.004
Disease-related	4.48±0.60	4.19±0.68	0.211
Diagnosis	4.44±0.73	4.33±0.50	0.624
Treatment	4.33±0.59	4.00±0.69	0.190
Lifestyle and other concerns	3.92±0.67	3.58±0.51	0.285

According to the determination of the Institutional Review Board of Chiayi Chang Gung Memorial Hospital (IRB No. 202001816B0), this study did not include human participants or patient data; therefore, IRB review and informed consent were not required.

Results

A total of 120 responses, including 60 from ChatGPT and 60 from Google Gemini, were judged by three independent ophthalmologists using a five-point Likert scale. Overall, ChatGPT got higher scores than Gemini. The mean score for ChatGPT was 4.32 ± 0.65, compared to 4.03 ± 0.66 for Gemini. The difference was statistically significant (P = 0.004), indicating that ChatGPT responses were rated as more appropriate.

Every grader’s analysis showed consistent trends, with all three ophthalmologists assigning higher mean scores to ChatGPT than to Gemini. However, the magnitude of the difference varied between graders. Despite some variability, the direction of the difference was the same across all evaluators.

When analyzed by category, ChatGPT outperformed Gemini in all four domains. The largest difference was found in the disease-related category, whereas the smallest difference was noted in the diagnosis domain. However, none of the category-specific comparisons reached statistical significance.

Both models showed lower scores in the lifestyle and other concerns category compared to the disease-related and diagnosis categories. This trend was observed in both AI systems.

Discussion

In this study, we evaluated and compared the appropriateness of glaucoma-related responses generated by ChatGPT and Google Gemini in Traditional Chinese. Our findings find that while both LLMs could provide generally appropriate answers to common patient questions, ChatGPT demonstrated a statistically significant advantage in overall performance. Across all evaluated categories, ChatGPT consistently received higher average ratings, although category-specific differences did not reach statistical significance.

These results align with previous research evaluating English chatbot responses.[7,10,11,12] For instance, previous studies showed that ChatGPT responses to glaucoma questions were mostly accurate, but occasional incompleteness was also reported.[8] These prior findings are echoed in our Traditional Chinese dataset, suggesting consistency in performance across languages.

A key difference in our study is the use of Traditional Chinese, which is widely used in Taiwan, Hong Kong, and overseas Chinese communities. While AI models have demonstrated competence in English medical dialog, their capabilities in non-English contexts remain underexplored. Our findings suggest that ChatGPT and Gemini can also generate reasonably appropriate responses in Traditional Chinese. Nonetheless, subtle differences in linguistic structure, cultural phrasing, and patient expectations in non-English languages may affect how well these responses are received or understood by patients. Notably, both models performed lower in lifestyle-related domains, such as diet, exercise, and stress. These questions often involve cultural nuance and nonclinical context, which may not be well represented in LLM training data. As a result, responses in this category were more vulnerable to vagueness, culturally inappropriate phrasing, or hallucinations, underscoring the importance of clinician oversight when patients consult chatbots on lifestyle concerns. For instance, ChatGPT provided a generally accurate description of glaucoma management but omitted gonioscopy as a diagnostic component in one response. Gemini, on the other hand, offered advice such as “avoid staying up late” without citing medical evidence. While rare, we also identified occasional misleading simplifications, such as incomplete descriptions of medication side effects. These examples highlight the risks of hallucinations, omissions, and vague guidance in real-world patient counseling.

The potential for AI chatbots to serve as supplementary tools for patient education in Traditional Chinese is inevitable. Given the growing demand for accessible and timely medical information, especially in aging populations with high glaucoma prevalence, LLMs could play an important role in expanding health literacy and bridging communication gaps.[13,14] They may assist in reinforcing key messages provided during clinic visits, particularly for patients who may forget or misunderstand medical explanations.

Nonetheless, several limitations remain. First, AI-generated responses may vary in quality depending on the specific wording of input questions. Second, most LLMs, including ChatGPT and Gemini, are trained on general-domain data and are not specifically fine-tuned for Traditional Chinese ophthalmology content. As a result, occasional inaccuracies, omissions, or culturally inappropriate phrasing may occur. Moreover, although readability metrics were not evaluated in this study, previous research suggests that AI responses may be written at a level too complex for some patients, which could limit their utility, especially among individuals with lower health literacy.[13] Besides, this study used only 20 questions, which limits generalizability. Furthermore, the dataset was clinician-generated rather than patient-derived; real-world patient phrasing in Traditional Chinese may differ substantially, potentially influencing chatbot performance. Finally, our dataset focused on common questions, whereas more complex or technical pathophysiologic topics were not tested.

Future improvements in AI chatbot performance will likely require domain-specific fine-tuning, in addition to incorporating region-specific patient concerns and health communication styles, or different dialects and languages for immigrants. Integration with voice-based systems or mobile health platforms may further increase accessibility for elderly patients or those with visual impairment.

Finally, it is important to emphasize that AI chatbots are not a replacement for professional medical advice. Their role should be seen as serving to support, but not substitute, the physician–patient relationship. Hallucination in LLMs is a frequently discussed issue and has yet to be fully resolved. Therefore, the reliability of LLMs in providing accurate medical information remains a concern. For example, CommonHealth Magazine has developed a chatbot powered by an AI model that uses its proprietary database, with all content verified by medical professionals. This approach offers a useful reference. A potential recommendation is that hospitals should provide closed, internal databases to train AI models, making it more feasible to develop patient-facing chatbots that minimize the risk of hallucinations commonly associated with large LLMs.

Conclusion

This study demonstrates that both ChatGPT and Google Gemini are capable of generating generally appropriate responses to glaucoma-related questions in Traditional Chinese. Among the two, ChatGPT achieved higher overall appropriateness scores, with a statistically significant difference compared to Gemini. Although performance across specific content categories was similar, both models exhibited relative limitations in addressing lifestyle-related questions. These findings highlight the emerging potential of AI-powered chatbots as supplementary tools for patient education in ophthalmology, particularly in non-English language settings.

Data availability statements

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

Conflicts of interest

The authors declare that there are no conflicts of interests of this paper.

Funding Statement

Nil.