In Reply to Sengul I and Sengul D

We thank Prof Dr Ilker Sengul and Prof Dr Demet Sengul from Giresun University in Turkey for their valuable letter and discussion^1,2 on our work on evaluating large language models (LLMs) in radiation oncology.³ In their letter, they asked 4 excellent and important questions regarding the challenges of evaluating LLMs in radiation therapy and in medicine in general. Although the field continues to evolve at a rapid pace and many challenges are yet to be solved, we will try to present some answers or at least provide some ideas on these questions:

“(1) Given the laborious nature of studies involving real-world clinical questions and human expert comparisons, what methodologies can ensure the reproducibility and generalizability of such findings across diverse clinical settings, patient populations, and different LLM architectures?”

We believe that the evaluation of LLMs for clinical application will necessarily be a continuous effort. If used in clinical applications, LLMs (and other forms of generative artificial intelligence) will need to be constantly monitored and tested. To make these tasks feasible and as easy as possible, a systematic assessment within clinical workflows will be needed. Creation of dedicated benchmark data sets is important as well. For real-world clinical questions (without single correct answers), other metrics that are easier to measure than “subjective quality” might still be helpful (e.g., making sure an LLM does not give recommendations that are clearly incorrect or making sure that the answers are not unnecessarily long or complicated).

“(2) Although LLMs may achieve “expert doctor level” accuracy on medical examination questions and provide helpful answers, how do we systematically evaluate their performance on complex, nuanced, or atypical clinical cases where human reasoning, clinical experience, and ethical considerations extend beyond factual knowledge retrieval?”

We believe that real-world open-ended questions are fundamental for comprehensive clinical LLM evaluation. There are different ideas on how to systematically evaluate LLMs in clinical applications (e.g., Shool et al⁴ and Shah et al⁵). In any case, evaluations need to be multifaceted to encompass different factors. Although we have no definitive answer to this question, there is one other thing to consider regarding “atypical cases.” LLMs generate output based on their training data. If an LLM has not seen a similar “atypical case” in that training data, the LLM output will most likely be worse, and the risk of “hallucinations” will be higher than for a “typical case.” At the same time, “atypical cases” are the ones where clinicians will be in most need of support (e.g., by an LLM). This makes the issue even more complicated and important.

“(3) Because LLMs become more integrated into clinical practice, even in an assistive capacity, the "black box" nature of some models and the potential for “hallucinations”—generating incorrect statements that appear evidence-based—remain significant concerns for patient safety. What transparency mechanisms and safeguards are essential for building trust and ensuring responsible integration, especially because models are fine-tuned or combined with explicit knowledge bases?”

Indeed, the “black box” nature of LLMs makes it hard to identify and understand when and why a model may give a good or bad answer. Although “hallucinations” remain a considerable issue, it should also be noted that a lot of work has been done on this problem in recent months and years, and there have been some improvements (e.g., Janiak et al⁶ and Kalai et al⁷). Indeed, RAG and knowledge bases can be a strategy to mitigate this issue. Other strategies might lie in obtaining “certainty” metrics of the LLM when generating output for medical use (e.g., Wang et al,⁸ Likert scale output of model,⁹ and token output probability¹⁰). In any case, for responsible LLM integration, many different factors need to be considered.

“(4) Ultimately, the challenge of evaluating LLM performance without full transparency about their training data persists. Because LLMs are anticipated to have a significant impact on the practice and future of medicine and radiation oncology, collaborative efforts between LLM developers and medical professionals will be crucial to navigating these complex ethical, safety, and integration challenges responsibly.”

We believe that (if possible), LLMs for clinical applications should ideally be open-source with known training data and be deployed on-premise in the local hospital environment.^11,12 Knowledge about the training data will also be essential to ensure LLMs have not been trained on data sets one might want to use for LLM benchmarking. We absolutely agree that collaboration between different involved stakeholders is fundamentally important.

Finally, we want to mention that clinical trials that investigate whether the usage of generative artificial intelligence truly leads to better outcomes for patients or within clinical workflows are needed. Evaluating LLMs without looking at meaningful endpoints will not tell us if the technology helps us in treating patients, which is the most important question of all.