Beyond the hype: Unveiling the challenges of large language models in urology

Large language models (LLMs), like ChatGPT, have gained much attention in urology in recent years, with applications ranging from patient-facing chatbots to summarizing clinical notes.¹ In medical education, several groups have explored whether these LLMs can successfully pass urology certification exams, such as the American Urological association self-assessment study program and the European Board of Urology in-service assessment.

Performance of these models has varied widely, with scores ranging from 27–81%.^2–4 In this month’s CUAJ, Touma et al evaluated the performance of ChatGPT 4 on the 2022 Queen’s Urology Exam Skill Test (QUEST) multiple-choice questions (MCQ), which simulates the Canadian Royal College urology licensing exam.⁵ The authors found that, disappointingly, ChatGPT 4 achieved only a 46% accuracy rate, placing it in the sixth percentile, compared to a 63% average score among final-year Canadian residents, with the disparity in performance being most pronounced for oncology-related questions.

The wide range of performance across these studies is likely multifactorial. First, LLMs are heavily dependent on their training data, which grows with each model update; however, there is limited transparency regarding these information sources, and LLMs often lack access to content behind paywalls or restricted memberships. Moreover, outdated training data may contribute to lower performance on rapidly evolving topics like oncology. Similarly, there is no information regarding how LLMs handle questions with conflicting evidence or situations when recommendations differ across guidelines.

Second, although LLMs excel in questions that involve factual recall or basic clinical scenarios, they may struggle with more complex cases that require consideration of patient comorbidities, contraindications, or interpretation of clinical data. Kollitsch and colleagues found that ChatGPT 4 performed worse with increasing question difficulty, with the LLM achieving 100% accuracy on the easiest questions but only 32% on the most challenging cases.²

Third, LLMs are prone to generating hallucinations, which are fabricated and often incorrect responses produced without any supporting evidence. Unlike humans, who are capable of reasoning based on provided information, LLMs may generate hallucinations because they focus on predicting the next most likely sequence of words based on user prompts. In healthcare and medical education, these hallucinations are particularly dangerous because they can be sporadic yet delivered with the same level of confidence as factually correct responses. Therefore, careful design of the LLM prompts is vital to minimizing hallucinations and enhancing response accuracy.

This iterative process, known as prompt engineering, can range from simply submitting the question as-is to ChatGPT to employing more advanced techniques, such as chain-of-verification and degree of uncertainty (Figure 1).⁶ Chain-of-verification provides a structured approach for the LLM to crosscheck its output against trusted data sources, such as clinical practice guidelines. The degree of uncertainty enables the LLM to indicate its confidence level in its response, thereby building greater trust in its outputs.

Figure 1.

Example of prompt engineering strategies to improve accuracy of LLM responses. Additional strategies to mitigate hallucinations are described by Kwong et al.⁶

Last, transparent reporting remains a widespread cncern among artificial intelligence (AI) studies in urology.^7,8 Given the rapid advancements of LLM s, clear documentation of LLM versions, date of analysis, prompts used, and hallucination rates should be encouraged to enhance reproducibility. Recent guidelines, such as the CANGARU checklist and TRIPOD-LLM, offer valuable guidance on reporting key elements specific to LLMs in clinical applications.^9,10

Much work remains before LLMs are ready for prime time in urology. Their inconsistent performance emphasizes the need for regular updates to training data, careful prompt engineering, and transparent reporting. As the integration of AI into urology continues to grow, adhering to best practices in LLM reporting will be essential for ensuring their effective and responsible use in both clinical practice and medical education.