Evaluating the reliability of large language models in answering FAQs for cataract surgery

Abstract

Cataract surgery is one of the most common and effective surgeries performed worldwide, yet patient education remains a challenge due to limitations in health literacy among the general population. Our study evaluated the reliability of different large language models (LLMs) in providing accurate, complete, and clear responses to frequently asked questions (FAQs) related to cataract surgery. A comprehensive list of 20 FAQs about cataract surgery were submitted sequentially as a prompt to nine different LLMs. All 180 answers were recorded and scored by two expert ophthalmologists, blinded to the model type, on a 5-point scale measuring the degree of accuracy, completeness, and clarity. Interrater agreement was measured using a weighted kappa coefficient and model performances were compared using the Friedman test and post-hoc analysis. Our results showed all models performed well responding to FAQs (79% of responses scored “excellent”), serving as effective tools in answering patient FAQs. LLaMA 4 and Copilot scored lower on average relative to other models (p < .05), however, they remained effective at FAQ responses overall. Potential expansion of LLMs as patient education tools into clinical settings should be considered, as they exhibit effectiveness in providing clear, accurate, and complete responses to cataract surgery FAQs.

Introduction

Cataract surgery is one of the most common and effective surgeries performed in medicine.^1–3 In 2015, 3.7 million cataract surgeries were performed in the United States, and the incidence is expected to continue rising. ² As with any other surgery, patients seek answers to important questions about risks, benefits, expectations, and more, prior to consenting to this procedure.

Large language models (LLMs), a subset of artificial intelligence (AI) models, are being used for many different applications within ophthalmology, mainly within the realm of diagnostics.^4–6 Given the complexities of cataract surgery and the lack of health literacy surrounding the eye and aspects of vision health among the general population, patient education becomes a crucial component of perioperative counseling and care.^7–9 LLMs are particularly well-suited to aid in this task for providers due to their superiority in generating and understanding human texts. ¹⁰ Many providers are already using LLMs to help develop patient education leaflets and other materials.^3,11 Some are now wondering whether these LLMs can effectively assist in counseling patients on frequently asked questions (FAQs) as well.^12–15

Unfortunately, despite being a routine ophthalmic procedure, the complex nature of cataract surgery requires precise and clear answers. Furthermore, given the increased integration and widespread availability of LLMs on the internet, it is likely some patients are already using these models to answer their medical questions. Therefore, evaluating LLM performance in patient education becomes imperative to identify potential sources of health misinformation.

This study aims to assess the adequacy of various LLMs in providing accurate, complete, and clear answers to FAQs related to cataract surgery. Our findings are intended to inform on the safety and efficacy of AI tools in patient education. Evaluating these models is critical prior to wider adoption in the clinical setting.

Methods

Study design

In this single-blinded cross-sectional study, an internet search of several websites, including large academic medical centers and ophthalmology organizations was performed. These websites were browsed for educational pages on cataract surgery, specifically questions resembling patient FAQs about the surgery. An extensive list of questions was saved on a Word document along with the answers the educational pages provided. A set of 20 questions were finalized after screening out repetitive questions and questions not directly related to the surgery. The final list of FAQs, reference answers, and where they were sourced can be found in the supplementary material of this manuscript (Supplemental Document 1).

Data collection

Free accounts were created for nine different commonly used and well known LLMs: ChatGPT-4o, Claude 4 Sonnet, Gemini 2.5 Flash, LLaMA 4, Mixtral 8 × 22B, DeepSeek V3, Grok 3, Qwen 3, and Standard Copilot. Each of the 20 FAQs were entered into the models’ web-based interface as individual, sequential prompts within the same chat. The same question order was input into all model conversations for experimental consistency. Each model was accessed between 11 and 12 June 2025, and all answers were saved on a Word document. Questions were entered separately to allow the models to provide full answers to each prompt. Additionally, this was thought to resemble patient conversational behavior of asking one question at a time. Documentation of model answers resulted in a total of 180 answers (20 questions × 9 models).

Two expert ophthalmologists were assigned as graders to provide a score for each model's answers to the FAQs. Graders were assigned a duplicate copy of answers that were blinded to the model type (labeled “Models A-I”). Graders were also provided a reference sheet for the answers posted on the websites from where the FAQs were selected, and a scoring sheet (Excel sheet). Scoring was on a scale of 1–5 based on response accuracy, completeness, and clarity. Each response was also reviewed for potentially unsafe or misleading content (e.g. inaccurate medical advice). Only whole number scores were permitted, and providing rationale for scores was optional (per grader discretion). The scoring legend is provided in Table 1 of this article. The distribution of scores for each model's FAQ responses is illustrated in Supplemental Figure S1.

Table 1.

Scoring rubric.

Score	Grading	Description
5	Excellent	Fully accurate, complete, and clearly written. Matches expert-level answer. No misleading, biased, or unsafe content. Easy for patients to understand.
4	Good	Mostly accurate and complete. May omit minor detail or use less clear phrasing, but still safe and reliable. No major factual errors.
3	Acceptable	Generally accurate but includes small factual inaccuracies, vague phrasing, or lacks clarity. Generally safe but may confuse a layperson or omit an important point.
2	Poor	Contains clear factual inaccuracies, lacks important clinical detail, or is misleading. May sound plausible but would concern an expert reviewer. Potential safety risk.
1	Unacceptable	Factually incorrect, misleading, or unsafe. Fails to answer the question appropriately. Could cause harm if trusted.

Statistical analysis

Descriptive statistics summarized the performance of each model. Interrater agreement was also determined using the Weighted Cohen's Kappa Coefficient. Kappa was used instead of an interclass correlation coefficient due to the ordinal nature of the scoring data. A nonparametric test, the Friedman test, was used to compare model performance. Additionally, since the kappa indicated high levels of grader agreement, the scores from both graders were averaged for each answer and only one Friedman test was completed. Results were analyzed by Wilcoxon Signed-Rank test with False Discovery Rate (FDR) correction to make pairwise comparisons between models. The FDR correction was used to adjust for the multiple comparisons. This approach is suitable given the large number of pairwise comparisons (36), allowing for increased statistical power while also controlling the false positives.

Results

Scores for all individual AI model responses were primarily clustered at “4” and “5” on the scoring rubric. In fact, about 79% of responses received a score of 5, 20% a score of 4, and only 1% received a score of 3. Furthermore, no responses were deemed unsafe or contained medically harmful advice either. A Weighted Cohen's Kappa Coefficient measuring interrater reliability for our ordinal-level score data indicated a “substantial agreement” between graders (κ = 0.675; 95% CI [0.51–0.80]) and this finding was found to be statistically significant (p < .001).

Since grader scores were in substantial agreement, the scores of both graders were averaged to compare performance by model type. Model comparisons are illustrated in Figure 1, with error bars representing the standard deviation.

Figure 1.

This figure illustrates the differences in LLM performance by model type. Error bars represent standard deviation; *indicates Standard Copilot and LlaMa 4 have a statistically significant lower mean score compared to other models (p < .05).

LLM: large language model.

A Friedman test was performed thereafter to evaluate whether there was a difference in performance by model type. The results of this test indicated there is a statistically significant difference in performance by model types (χ² = 75.6, p-value = 3.778 × 10⁻¹³) with a moderate effect size (Kendall's W = 0.47). Therefore, post-hoc analysis using a Wilcoxon Signed-Rank test with FDR correction was used to make pairwise comparisons between the models.

Results of the post-hoc analysis showed LLaMA 4 and Copilot scored lower on average compared to all other models (p < .05), with pairwise effect sizes mostly ranging from small-to-moderate (r ≈ 0.03–0.26) with the exception of a large, poor performance effect size for LLaMA 4 performance against Mixtral 8 × 22B, DeepSeek V3, Grok 3, and Qwen 3 (r ≈ 0.84–0.88). However, no statistically significant difference was observed between LLaMA 4 and Copilot themselves (p = 1.000). There were no other significant findings between different model performances. Key findings of the pairwise comparisons are listed in Table 2, however, the complete list of results can be found in Supplemental Table S1.

Table 2.

Key model pairwise comparisons.

Pairwise comparison	Adjusted p-value	Effect size
ChatGPT-4o vs LLaMA 4	.003	0.259
ChatGPT-4o vs Copilot	.012	0.042
Claude 4 Sonnet vs LLaMA 4	.005	0.092
Claude 4 Sonnet vs Copilot	.011	0.058
Gemini 2.5 Flash vs LLaMA 4	.003	0.125
Gemini 2.5 Flash vs Copilot	.007	0.029
LLaMA 4 vs Mixtral 8 × 22B	.005	-0.847
LLaMA 4 vs DeepSeek V3	.003	-0.839
LLaMA 4 vs Grok 3	.003	-0.877
LLaMA 4 vs Qwen 3	.003	-0.877
LLaMA 4 vs Copilot	1.000	0.451
Mixtral 8 × 22B vs Copilot	.012	0.033
DeepSeek V3 vs Copilot	.011	0.042
Grok 3 vs Copilot	.005	0.117
Qwen 3 vs Copilot	.003	0.125

Discussion

Across the board, all models performed well, with an average of a “good” to “excellent” grading (Figure 1). In fact, none of the responses by any of the models scored below “acceptable” and the graders did not note any unsafe content in the responses. About 79% of responses among both graders scored a 5 (“excellent”), and only 20% scored a 4 (“good”). There was also a substantial degree of agreement between response graders (κ = 0.675, p < .001), adding validity to the findings.

When different LLMs were compared, it was found that most models performed at a similar level for cataract surgery FAQs. However, LLaMA 4 and Copilot underperformed compared to their peer models in providing fully accurate, complete, and clear responses to the FAQs (Table 2). Despite this, there was no statistically meaningful difference between LLaMA 4 and Copilot. Both models still performed in the “good” to “excellent” range, overall, for the 20 FAQs. This indicates the difference is not clinically meaningful.

This outcome, in itself, is a testament to the overall effectiveness of LLMs in providing high-quality responses to cataract surgery FAQs. The mean scores of best performing models are clustered closely, drawing concern for a potential “ceiling effect.” However, our statistical analysis was able to detect a significant difference in performance across the models. This demonstrates that our 5-point scale and statistical methods were sensitive enough to identify performance disparities, even with a narrow range of high scores. For a more detailed overview of the score distribution by AI model, Supplemental Figure S1 is available.

These findings demonstrate the ability of LLMs to function effectively in providing education to cataract patients. The FAQs evaluated in this study (Supplemental Document S1) were comprehensive, addressing a variety of questions patients typically ask pre-, intra-, and postoperatively for cataract surgery. This study supported our confidence in the increasing value of AI to address patient educational needs within ophthalmology. Patients already using LLMs to answer medically relevant questions benefit from responses that are clear, accurate, and easily accessible. Moreover, with the great efficacy LLMs present in ophthalmic patient education, there is potential to integrate LLMs into real-time ophthalmology clinics or virtual spaces.

Integration into the clinical workflow may alleviate time and staffing-associated burdens, while enhancing accessibility to cataract surgery information for patients. In practice, we could embed LLM-powered chat interfaces into patient portals, electronic health records, or clinic websites to address common questions. This would allow clinicians to focus on more complex concerns; however, several barriers still exist before such tools can be widely adopted. This includes rigorous validation of AI responses, proper regulatory oversight, and management of medical–legal liabilities.

This study has many factors contributing to its validity. For instance, question prompts were submitted individually, mimicking patient behavior and allowing the LLMs to formulate a full response. This study also incorporated a diverse set of widely used LLMs, making it more applicable to real-world scenarios. For instance, Gemini, one of the models evaluated, powers the Google “AI Mode” and “AI Overview” in Search, representing a tool that many patients may be inadvertently using. FAQs selected were clinically relevant and from patient-focused websites, ensuring real-world relevance. The use of kappa coefficient calculations and two expert ophthalmologists blinded to model types further strengthened the validity of our grading process and reduced bias in model scoring.

While prior studies have compared different LLMs in patient education tasks related to cataract surgery as well, they primarily focused on critical assessment of readability and understanding of the content.^3,11 Our study encompasses a larger variety of LLMs and greater breadth of factors being assessed by the graders. Some studies, such as those by Dihan et al. and Qiu et al., only evaluated three LLM models.^12,13 A study by Tailor et al. had a more coarse grading system compared to our 5-point scale. ¹⁴ On the other hand, our more granular grading scale allowed for a more statistically analytical comparison of the different models compared to prior studies.^15–19

Limitations

While this study has many strengths, there are a few noteworthy limitations as well. AI technology is rapidly developing and improving, therefore newer and better versions of the models assessed in this study will continue to emerge. Additionally, only unpaid versions of the models were evaluated in this study, whereas some patients may be using enhanced performance versions of these models. Paid versions of these models may produce different study results.

Question order and input temperature are other factors that may affect outcomes and introduce bias. Although the same set and sequence of questions were used for all models to maintain consistency, future research may benefit from randomizing question order or performing replicate runs to evaluate for context-related variability. Future studies can mitigate this bias by inputting each question multiple times to test for consistency in responses and evaluate how strongly question order prejudices the results.

Lastly, despite the objectivity of a quantitative scoring rubric by two ophthalmology experts, we concede that a certain degree of subjectivity remains. Assigning a particular score to a generated response is not free from subjectivity, however, a more structured, granular scoring rubric may reduce this subjectivity.

Future directions

Since only 20 FAQs are used and a single surgery studied, ceiling effects could have impacts on the outcomes. However, the questions were selected to represent the most commonly encountered patient concerns about cataract surgery, ensuring relevance and practical applicability. Future studies may consider expanding the question pool to include a broader range of topics or subspecialties to further assess model discriminative performance.

Future studies assessing LLM responses to FAQs should also consider an expanded scoring rubric with scores for different assessment domains, such as an ordinal subscale for unsafe or potentially harmful content, rather than a singular score for each response. This would allow a more detailed analysis for comparison of LLMs. This study should be replicated within other fields of ophthalmology, providing a more generalizable insight into the clinical scope of LLMs in patient education. Tailor et al., for instance, covered a broad range of ophthalmic subspecialties in their study, allowing for a comprehensive exploration of LLMs in ophthalmic patient education. ¹⁴

With the rapidly accelerating advancements in AI technology, we hope this study will contribute to their future clinical applications. Our findings suggest they show promise as effective patient education tools. An interesting direction for future studies would be to assess possible LLM integration into patient–physician communication portals.

Conclusions

LLMs demonstrate a strong ability to provide “good” to “excellent” quality complete, accurate, and comprehensible responses to cataract surgery FAQs. LLaMA 4 and Copilot performed worse relative to other models, yet this difference is not clinically significant. In summary, LLMs display effective delivery of patient education for this common ophthalmic surgical intervention. Further consideration should be given to the integration of LLMs in clinical practice, as it may reduce clinic burden and improve accessibility to patient education.

Abbreviations

The following abbreviations are used in this article:

LLM(s)

Large language model(s)

FAQ(s)

Frequently asked question(s)

FDR

False discovery rate

AI

Artificial intelligence