Real-world evaluation of large language model for patients medical and administrative queries in nuclear medicine

Abstract

Large language models (LLMs) may support patients by addressing their queries; however, their real-world clinical use remains uncertain. Patients’ queries prospectively collected and answered by nuclear medicine physicians, administrative staffs, and ChatGPT v4.1. Responses evaluated and Likert-scored by medical and administrative experts and two independent non-experts using 15 out of 17 dimensions of the proposed QUEST framework. LLM-generated responses classified as “better,” “equivalent,” or “worse” relative to human-generated responses on question-by-question basis; binomial tests assessed whether LLM performance exceeded 50%. Inter-rater agreement was assessed using Prevalence-Adjusted Bias-Adjusted Kappa (PABAK); statistical significance set at p < 0.05. 339 drug interaction, 42 medical, and 76 administrative queries were analysed. For medical queries, in 8 of 10 dimensions, 76–98% of LLM-generated responses rated equivalent or better than human-generated responses by medical expert (p < 0.001). For administrative queries, non-expert raters judged LLM-generated responses more informative (97%) and preferred (86%). For medical queries, LLM-generated responses rated more informative (67%), human-generated responses judged easier to understand (62%), with 60% disagreement on overall preference. PABAK showed higher agreement for LLM- than human-generated responses across medical (0.14–0.90 vs −0.90–0.52) and administrative (0.92–1.00 vs −0.63– − 0.13) queries. LLM-generated responses were consistently rated favourably, particularly for administrative queries, though further validation is required before clinical use.

Introduction

Large language models (LLMs) are poised to have a substantial impact on medicine owing to their outstanding capacity for complex reasoning and natural language understanding. In recent years, the emergence and rapid expansion of LLMs have heralded a new technological era in health care, including diagnostic imaging.

Recent studies have demonstrated that LLMs can identify and correct common errors in radiology reports¹, summarises radiology reports to improve comprehension when presented at a reading level appropriate for lay audiences², recommend differential diagnoses based on imaging findings³ and support clinical decision-making by generating concise clinical summaries to guide radiologists and assist with examination protocol selection⁴.

Within nuclear medicine, prior studies have explored LLM ability to answer nuclear medicine–related questions at a level appropriate for medical students and general practitioners⁵, to respond to patient-generated questions regarding nuclear stress testing and 18F-fluorodeoxyglucose positron emission tomography–computed tomography (18F-FDG PET-CT) for myocardial inflammation, prepare patients for 18 F FDG PET-CT and explain the scans’ reports^6,7. Additional applications include the generation of patient information sheets⁸ and the automated population of staging and interim response classifications based on scan reports, such as Lugano classification for malignant lymphoma⁹.

Despite these advances, public confidence in artificial intelligence (AI)–generated health information remains limited. In the United States, approximately one in six adults reports using AI chatbots at least once per month to seek health advice or information, increasing to one in four among adults younger than 30 years¹⁰. However, only one in three adults report being very or somewhat confident in the accuracy of health-related information encountered, and 56% of AI users indicate low confidence in the accuracy of AI-generated health information. Furthermore, while 21% of adults believe AI is offering more to help than harm in the context of health information, a similar proportion (23%) believe it is resulting in more harm than good¹⁰.

These findings underscore the complexity of integrating generative AI tools in routine clinical practice, particularly for patient-facing applications, despite their demonstrated performance. Moreover, beyond the accuracy of LLM-generated outputs, the linguistic and communicative aspects of health-related information produced by LLMs remain underrepresented in the existing literature.

To address these limitations, a new evaluation framework, QUEST, has been proposed to assess health-related AI-generated information using structured human evaluation, thereby accounting for dimensions not captured by traditional metrics¹¹.

In this study, applying the QUEST framework, we evaluate the application of LLMs to real-world questions posed by patients for diagnostic nuclear medicine examinations.

Results

Questions

A total of 339 medicinal queries concerning drug interaction, 42 patient’s non-medicinal general medical and 76 patient’s administrative queries were analysed (supplementary Tables 1 and 2 for list of medical and administrative questions).

Expert Evaluation of LLM- and human-generated responses

Table 1 summarises expert evaluation of human- and LLM-generated responses to administrative queries, whereas Table 2 and Fig. 1 summarise expert evaluations of responses to medical queries.

Table 1.

Expert evaluation of LLM vs human-generated responses for admin questions

Metric	LLM responses relative to human responses
	Better	Equivalent	Worse
Accuracy	0 (0%)	76 (100%)	0 (0%)
Agreement	76 (100%)	0 (0%)	0 (0%)
Bias	76 (100%)	0 (0%)	0 (0%)
Comprehensiveness	76 (100%)	0 (0%)	0 (0%)
Currency	0 (0%)	76 (100%)	0 (0%)
Fabrication/falsification	0 (0%)	76 (100%)	0 (0%)
Harm	0 (0%)	76 (100%)	0 (0%)
Relevance	0 (0%)	76 (100%)	0 (0%)
Self-awareness	76 (100%)	0 (0%)	0 (0%)
Usefulness	76 (100%)	0 (0%)	0 (0%)

LLM Large language model.

Table 2.

Expert evaluation of LLM vs human-generated responses for medical questions

Metric	LLM responses relative to human responses
	Better	Equivalent	Worse
Accuracy	2 (5%)	34 (81%)	6 (14%)
Agreement	4 (10%)	31 (74%)	7 (17%)
Bias	0 (0%)	41 (98%)	1 (2%)
Comprehensiveness	14 (33%)	25 (60%)	3 (7%)
Currency	1 (2%)	33 (79%)	8 (19%)
Fabrication/falsification	0 (0%)	28 (67%)	14 (33%)
Harm	2 (5%)	37 (88%)	3 (7%)
Relevance	1 (2%)	34 (81%)	7 (17%)
Self-awareness	5 (12%)	19 (45%)	18 (43%)
Usefulness	6 (14%)	26 (62%)	10 (24%)

LLM Large language model.

Fig. 1. Comparison of LLM and human-generated responses for medical questions evaluated by nuclear medicine expert.

Bar charts showing percentage of evaluated responses for each dimension indicating whether LLM-generated responses were better (green), same (grey) or worse (red) than human-generated responses. LLM: Large language model.

For administrative queries, all LLM-generated responses matched or exceeded expert performance across all evaluated metrics.

In contrast, performance for medical queries was more variable, with LLM-generated responses rated as worse than expert-generated responses for the self-awareness dimension in 18 (43%) of cases overall. However, for the comprehensiveness dimension, LLM-generated response were rated better than expert-generated responses in 14 (33%) of cases. LLM-generated responses in each dimension were rated equivalent or superior to human-generated responses (>50%). Statistical significance was observed in 9 dimensions, 8 with high agreement (76–98%; p < 0.001) with a lower agreement for fabrication/falsification (28 (67%); p = 0.044). Self-awareness did not reach statistical significance (24 (57%), p = 0.441) (Fig. 2). Although no increased risk of harm or fabrication/falsification was observed in LLM-generated responses to administrative queries, concerns regarding potential harm were identified in 3 (7%) of LLM-generated responses to medical queries. Additionally, 14 (33%) of LLM-generated medical responses were considered to contain falsified information.

Fig. 2. Forest plot of expert evaluation metrics for medical questions.

The p-values are derived from a binomial test evaluating % above 50. LLM: Large language model, N: Number.

Independent non-expert evaluation of LLM- and human-generated responses

Independent non-expert raters’ evaluations comparing LLM- and human-generated responses to administrative queries are summarised in Table 3. Overall, LLM-generated responses were most frequently rated as equivalent to or better than expert-generated responses. Observer 2 did not rate any of the responses worse than expert-generated responses, and Observer 1 did so in a small number of cases for clarity 2(3%), trust 1(1%) and satisfaction 1 (1%).

Table 3.

Independent non-expert raters assessment of LLM vs human-generated for administrative questions

Metric	LLM responses relative to human responses
	Better	Equivalent	Worse
Observer 1
Understanding	5 (7%)	71 (93%)	0 (0%)
Reasoning	3 (4%)	73 (96%)	0 (0%)
Clarity	15 (20%)	59 (78%)	2 (3%)
Trust	16 (21%)	59 (78%)	1 (1%)
Satisfaction	20 (26%)	55 (72%)	1 (1%)
Observer 2
Understanding	46 (61%)	30 (39%)	0 (0%)
Reasoning	57 (75%)	19 (25%)	0 (0%)
Clarity	49 (64%)	27 (36%)	0 (0%)
Trust	70 (92%)	6 (8%)	0 (0%)
Satisfaction	68 (89%)	8 (11%)	0 (0%)

LLM Large language model.

Concordance was markedly higher for LLM-generated than expert-generated responses (96% to 100% vs 18% to 43%; Fig. 3). Agreement among LLM ratings was consistently high across domains (PABAK 0.92 to 1.00), whereas human ratings showed poor agreement, with negative or near-zero PABAK values across metrics, particularly for trust, satisfaction, and reasoning (Tables 4 and 5).

Fig. 3. Comparison of independent non-expert raters concordance for admin questions answered by LLM versus human experts.

Bar charts showing percentages where the Likert scores were similar for both observers for LLM (red) and human-generated (blue) responses. LLM: Large language model.

Table 4.

Inter-rater agreement for non-expert raters assessment of LLM vs human-generated for administrative questions

Metric	LLM		Human
	Observed agreement (%)	PABAK [95% CI]	Observed agreement (%)	PABAK [95% CI]
Understanding	100%	1	43%	−0.13 [−0.36 to 0.11]
Reasoning	100%	1	29%	−0.42 [−0.62 to −0.19]
Clarity	96%	0.92 [0.78 to 0.98]	38%	−0.24 [−0.46 to 0.00]
Trust	97%	0.95 [0.82 to 0.99]	18%	−0.63 [−0.79 to −0.42]
Satisfaction	96%	0.92 [0.78 to 0.98]	28%	−0.45 [−0.64 to −0.22]

LLM Large language model, PABAK prevalence-adjusted and bias-adjusted kappa, CI Confidence interval.

Table 5.

Inter-rater agreement for non-expert raters assessment of LLM vs human-generated for medical questions

Metric	LLM		Human
	Observed agreement (%)	PABAK [95% CI]	Observed agreement (%)	PABAK [95% CI]
Understanding	95%	0.90 [0.68 to 0.99]	12%	−0.76 [−0.92 to −0.49]
Reasoning	90%	0.81 [0.55 to 0.95]	7%	−0.86 [−0.97 to −0.61]
Clarity	57%	0.14 [−0.18 to 0.45]	76%	0.52 [0.21 to 0.76]
Trust	83%	0.67 [0.37 to 0.86]	5%	−0.90 [−0.99 to −0.68]
Satisfaction	69%	0.38 [0.06 to 0.65]	12%	−0.76 [−0.92 to −0.49]

LLM Large language model, PABAK prevalence-adjusted and bias-adjusted kappa, CI Confidence interval.

For overall performance domains—including informativeness, ease of understanding, and overall preference—although both observers agreed that for most responses LLM-generated responses to admin question were more informative 74 (97%) and were preferred overall 65 (86%), they disagreed in 48 (63%) of cases regarding which response was easier to understand (Fig. 4).

Fig. 4. Overall Preference for LLM vs human expert generated responses on admin questions.

Bar charts of non-expert ratings for informativeness, clarity, and overall preference (LLM: red; human: green; disagreement: blue). LLM Large language model.

Independent non-expert raters agreed in nearly two-thirds of cases that LLM-generated responses to medical questions were more informative than human-generated responses, whereas over 60% (n = 26; 62%) agreed that human-generated responses were easier to understand. Concordance on overall preference was low, with disagreement in 25 cases (60%; Table 6, Figs. 5 and 6). Across understanding, reasoning, trust, and satisfaction, inter-rater concordance was higher for LLM-generated than human-generated responses (69% to 95% vs 5% to 12%), while clarity was the only domain with higher agreement for human-generated responses (76% vs 57%).

Table 6.

Independent non-expert raters assessment of LLM vs human-generated responses for medical questions

Metric	LLM responses relative to human responses
	Better	Equivalent	Worse
Observer 1
Understanding	0 (0%)	41 (98%)	1 (2%)
Reasoning	0 (0%)	38 (90%)	4 (10%)
Clarity	3 (7%)	23 (55%)	16 (38%)
Trust	1 (2%)	33 (79%)	8 (19%)
Satisfaction	6 (14%)	25 (60%)	11 (26%)
Observer 2
Understanding	36 (86%)	6 (14%)	0 (0%)
Reasoning	39 (93%)	3 (7%)	0 (0%)
Clarity	10 (24%)	32 (76%)	0 (0%)
Trust	41 (98%)	1 (2%)	0 (0%)
Satisfaction	41 (98%)	1 (2%)	0 (0%)

LLM Large language model.

Fig. 5. Comparison of independent non-expert raters concordance for medical questions answered by LLM versus human experts.

Bar charts showing percentages where the Likert scores were similar for both observers for LLM (red) and human-generated (blue) responses. LLM: Large language model.

Fig. 6. Overall Preference for LLM vs human expert generated responses on medical questions.

Bar charts of non-expert ratings for informativeness, clarity, and overall preference (LLM: red; human: green; disagreement: blue). LLM Large language model.

Medicinal drug–interaction queries

Among the 339 medicinal queries concerning drug–interaction queries analyzed, approximately 157 (46%) related to DaTScan examinations. Overall, LLM-generated responses were incorrect in 38 (11%) of cases. The highest proportion of incorrect responses was observed for drug–interaction queries associated with cardiac MIBG examinations 7 (17%) (Table 7).

Table 7.

Agreement between LLM and clinical guidelines for medicinal interaction

Modality		All	Cardiac MIBG	DaTScan	Gastric emptying study	Myocardial perfusion scan	Resting cardiac MIBI
Correct	LLM = No, Guidelines = No	295	31	139	78	29	18
	LLM=Yes, Guidelines=Yes	6	4	0	2	0	0
Incorrect	LLM = No, Guidelines=Yes	0	0	0	0	0	0
	LLM=Yes, Guidelines = No	38	7	18	11	2	0

LLM Large language model.

Discussion

In this study, we used selected components of a predefined evaluation framework to assess LLM–generated responses to real-world administrative and medical questions related to nuclear medicine diagnostic examinations. Our findings demonstrate that ChatGPT version 4.1 generally provides accurate and useful responses to patient inquiries, consistent with previously published work^6,12.

Our study however introduces several novelties compared with previously published work. First, we evaluated not only medical questions but also administrative queries and drug–interaction questions derived from real-world patient communications. Most prior studies have focused predominantly on researcher-generated or simulated medical questions^6,7,12, which may not fully capture the complexity and variability of everyday patient inquiries. By incorporating real-world patient questions, our study highlights the breadth of informational needs encountered in routine clinical practice.

Despite overall favourable performance, we observed important differences between administrative and medical queries. Although no increased risk of harm or fabrication/falsification was identified in LLM-generated responses to administrative questions, a higher proportion of potential harm and falsified statements was observed in responses to medical queries. These findings underscore the need for caution when deploying LLMs for patient-facing medical information, particularly in scenarios involving clinical decision-making or safety-sensitive content.

With respect to drug–interaction queries, our findings indicate that in approximately 11% of cases, LLM-generated responses overestimated drug interactions with nuclear medicine scans with no instances of underestimation observed. While overestimation may be considered safer than underestimation, it nonetheless has the potential to cause unnecessary patient anxiety or inappropriate delays in imaging and therefore warrants further refinement.

Second, we applied selected dimensions (15 out of 17 proposed dimensions) of the QUEST framework¹¹ to evaluate not only the accuracy of responses but also broader qualitative dimensions, including understanding, reasoning, expression style, safety, and trustworthiness. This multidimensional approach addresses a key gap in existing literature, where evaluations often focus primarily on correctness. Of particular relevance to lay non-expert end users are the linguistic and communicative aspects of LLM-generated responses. Using QUEST, we demonstrated that for administrative queries, non-expert raters consistently rated LLM-generated responses as equivalent or superior to human-generated responses across all evaluated domains. In contrast, for medical queries, substantial variability was observed. Although both non-expert raters preferred LLM-generated responses in 86% of administrative queries, they preferred LLM-generated responses in only 36% of medical queries.

Third, our study uniquely incorporates systematic assessments by independent non-expert raters with no medical or paramedical background. Given the rapid growth in public use of LLMs for health-related information, this perspective is notably underrepresented in prior work. For example, Matsutomo et al.¹² demonstrated superior accuracy and validity of ChatGPT-4 compared with ChatGPT-3.5 for generating explanations of common single photon emission computed tomography (SPECT) and PET applications; however, their evaluation relied exclusively on assessments by certified technologists. Similarly, Shinn et al.⁶ evaluated LLM-generated responses to questions on cardiac stress testing and 18F-FDG PET-CT for myocardial inflammation using expert reviewers only, concluding superior performance of newer model versions. Rogasch et al.⁷ also assessed ChatGPT-generated responses to patient questions regarding 18F-FDG PET-CT in Hodgkin’s lymphoma and lung cancer using expert-based criteria and reported high adequacy and usefulness. While these studies provide important insights, none included assessments from nonprofessional end users, limiting conclusions regarding patient comprehension and trust. Collectively, these studies highlight the absence of a consistent, guideline-recommended framework for evaluating LLM-generated medical responses. The QUEST framework offers a structured platform for human assessment of LLM-generated responses; however, there are potential areas for further refinement. For example, QUEST does not specify recommended scoring scales (eg, 3-point vs 5-point Likert scales), which may introduce inconsistencies across studies. Additionally, all principles and dimensions are currently equally weighted. From a clinical perspective, dimensions such as accuracy, harm, and fabrication/falsification arguably warrant greater weighting than attributes such as empathy. A weighted scoring approach may enhance the framework’s applicability to real-world healthcare deployment.

Several limitations of this study should be acknowledged. First, although our centre covers a diverse demographics, this was a single tertiary centre study, which may limit generalisability. Nevertheless, our findings suggest that LLMs can reliably support general and administrative patient inquiries and could potentially be adapted to local clinical workflows through targeted fine-tuning. Emerging approaches such as retrieval-augmented generation (RAG), parameter-efficient fine-tuning (PEFT), and the use of small language models (SLMs) have been proposed to enable safe and context-specific deployment of language models in healthcare settings^13–15.

Second, the study evaluated ChatGPT 4.1, selected because it was widely accessible to patients at the time of data collection. As LLM performance evolves rapidly, these findings should be interpreted as reflecting a specific technological time point rather than current state-of-the-art capability. Importantly, the selection of this model reflects the study’s primary objective of evaluating tools that were widely accessible to patients and the general public in real-world settings, rather than restricted or experimental systems.

We further acknowledge that domain-specific and newer-generation models, such as Google’s Med-PaLM^16,17 and Open AI’s newly released ChatGPT Health¹⁸, may exhibit different performance characteristics. Future studies incorporating these newer models will be essential to determine whether the safety concerns identified in this study persist as LLM technology continues to evolve.

Third, human-generated responses were produced by consensus between two clinicians or administrative staff members. Although this provided a standardised reference, it may represent an idealised benchmark relative to routine practice, where responses are often generated by a single time-constrained staff member. This may have reduced the apparent relative performance of LLM-generated responses in real-world settings. In addition, removal of generic safety disclaimers before evaluation improved comparability between response types, but may have altered the native safety framing of LLM outputs.

Fourth, although the overall dataset included a substantial number of real-world queries, the number of non-drug-interaction medical queries was relatively small. Findings for this subgroup should therefore be regarded as exploratory. Larger studies focusing on broader categories of medical queries will be needed to establish the robustness of these observations.

Fifth, In this study, we did not evaluate the consistency and empathy dimensions of the proposed QUEST framework.

Consistency, a component of the QUEST framework, was not formally assessed using repeated-query designs. This reflected our primary objective of evaluating single-response performance in a real-world patient-facing context. As such, our findings focus on the quality, safety, and interpretability of individual outputs rather than response stability across repeated prompts. Future work should incorporate repeated-query designs to assess reproducibility and technical robustness more directly. We did not evaluate the empathy dimension, as we considered it less directly relevant to diagnostic imaging–related queries. Notably, empathic language was observed primarily when prompts explicitly conveyed patient anxiety or concern, suggesting that empathy in LLM-generated responses may be context dependent.

Finally, although the inclusion of independent non-expert raters was a strength, only two were included. The observed variability in PABAK estimates, particularly for medical queries, likely reflects heterogeneity in lay interpretation of complex responses, but it also limits the generalisability of preference-based findings. Future studies including larger and more diverse lay samples would help to clarify how patients interpret and prioritise different response characteristics.

In conclusion, our findings demonstrate that LLM-generated responses provide accurate and useful information for general and administrative queries related to nuclear medicine diagnostic examinations and are consistently preferred by lay observers in these lower-risk contexts. In contrast, performance in medical queries remains variable, with clinically relevant inconsistencies across evaluated dimensions.

These results support a risk-stratified approach to LLM deployment, in which patient-facing medical applications are classified as higher-risk use cases. Consequently, implementation in hospital settings should be contingent upon rigorous domain-specific validation, defined human-in-the-loop oversight, and institutional governance structures that ensure accountability, auditability, and ongoing performance monitoring prior to clinical use, particularly as LLM capabilities continue to evolve.

Methods

Ethics statement

This study evaluated LLM performance using fully anonymised patient medical and administrative questions obtained via a departmental email account. No patient-identifiable data were processed, no patient contact or clinical intervention occurred, and outputs were not used to inform patient care. In accordance with UK Health Research Authority guidance, this work did not require Research Ethics Committee approval. Data handling been performed in accordance with the Declaration of Helsinki and complied with institutional information governance and data security policies.

Data collection

Figure 7 provides a flowchart overview of the data collection process. Patients’ medical queries forwarded to the departmental general email account were prospectively collected between January 2, 2024, and June 9, 2025. Queries submitted by medical professionals from specialties other than nuclear medicine and forwarded to the same general departmental email account were collected over the same period.

Fig. 7. Study flow chart.

LLM Large language model, SOP Standard operating procedure.

Patients’ administrative queries, were collected from patient emails and telephone calls by administrative team between November 4, 2024, and March 5, 2025 and transferred to a separate administrative database.

Data curation

531 medicinal queries concerning drug–interaction, 44 patients’ non-medicinal general medical, 13 doctors’ general medical queries, as well as 97 administrative queries were collected. Drug–interaction queries were further stratified by imaging modality and duplicate entries were removed (n = 192). For patients’ general medical queries, duplicate questions were removed (n = 2). Doctors’ general medical queries were excluded from further analysis (n = 13). Administrative queries were reviewed, and those deemed incomplete or incomprehensible as well as any duplicate questions were excluded (n = 21).

The final dataset comprised 339 medicinal queries concerning drug interactions, 42 non-medicinal general medical queries, and 76 administrative queries from patients.

Data analysis

Medical queries were answered by two nuclear medicine physicians (JB, MW), and administrative queries were answered by two administrative team members (KG, JW) in consensus.

All medical and administrative queries were also entered into a generative pretrained transformer model (ChatGPT version 4.1, OpenAI) as single prompts between June 26, 2025, and July 10, 2025. Each query was submitted in a separate chat, beginning with the phrase “I am a patient,” to ensure that responses were generated from a patient-facing perspective. To minimise bias during response evaluation, generic ChatGPT statements advising patients to discuss findings with medical or nuclear medicine teams were removed.

Consistent with the study objective of reflecting real-world patient use, analyses were conducted using single-response outputs for each query, and repeated-query assessments of response consistency were not undertaken.

Human-generated and LLM-generated responses were then organised into four booklets: medical answers (sets 1 and 2) and administrative answers (sets 1 and 2). All booklets were anonymized and blinded to the source of the responses. The responses were subsequently evaluated using selected components, 15 out of proposed 17 dimensions, of the QUEST framework¹¹.

Briefly, QUEST is a structured framework for human evaluation of health-related responses generated by large language models. It consists of five core principles—quality of information, understanding and reasoning, expression style and persona, safety and harm, and trust and confidence—encompassing 17 evaluation dimensions (supplementary Table 3 for detailed description of QUEST).

An information booklet describing the QUEST principles, dimensions, scoring system, and detailed instruction for expert and independent non-expert raters was provided.

Nuclear medicine expert (AO) and administrative experts (RO) independently and blindly assessed both human-generated and LLM-generated responses for quality of information (accuracy, relevance, currency, comprehensiveness, agreement, usefulness dimensions) and safety and harm (bias, harm, self-awareness, fabrication/falsification dimensions) principals. Predefined Likert scale scores were assigned for each dimension and each response.

Two non-expert raters (SMS, JMSF) independently and blindly assessed the same responses for understanding and reasoning (understanding and reasoning dimensions), expression style and persona (clarity dimension), and trust and confidence (trust and satisfaction dimensions) principals. In addition, non-expert raters indicated their preference for each question across three domains: (1) which answer they found more informative, (2) which they found easier to understand, and (3) which they preferred overall. We did not assess the consistency (from quality of information core principal) and empathy (from expression style and persona core principal) dimensions of the QUEST framework in this study.

For drug–interaction queries across all imaging modalities, LLM-generated yes/no drug–interaction responses were compared against the most up-to-date local standard operating procedures, as well as relevant national and international guidelines where applicable.

Statistical analysis

For drug–interaction queries, responses were classified as correct if they adhered to relevant guidelines. Proportions of correct responses were calculated for each imaging modality and overall.

For administrative and medical question ratings, ordinal variables were summarised using medians, whereas binary variables were reported as counts and percentages.

Performance comparisons between LLM-generated and human-generated responses were conducted on a question-by-question basis. Individual QUEST dimension scores, derived from Likert-scale ratings and categorical assessments, were first assigned to each response. For each dimension, LLM and human-generated response scores were directly compared, and a three-category comparative framework was applied:

For accuracy, relevance, currency, comprehensiveness, agreement, usefulness, and self‑awareness, the LLM was considered better when it received a higher score than the expert, “equivalent” when both answers received the same score, and “worse” when the LLM received a lower score. For the metrics bias, harm, and fabrication/falsification, the direction of desirability was reversed: the LLM was categorised as “better” when it had a lower value than the expert, “equivalent” when both values were identical, and “worse” when the LLM had a higher value. Proportions for each metric were calculated, and summarised in terms of a single proportion. Exact binomial tests were used to assess whether the proportion of responses classified as equivalent or better exceeded 50%.

Concordance between the two independent non-expert raters was assessed separately for administrative and medical questions on a question-by-question basis across five (understanding, reasoning, clarity, trust and satisfaction) evaluated dimensions. Concordance was defined as agreement between observers in the comparative classification across response type and evaluation domain. Proportions of concordant ratings were calculated and compared descriptively.

Independent non-expert raters preferences for informativeness, ease of understanding, and overall preference were also assessed. Responses were categorised as both prefer LLM, both prefer human expert, or disagreement.

Inter-rater agreement between non-expert raters was assessed using prevalence-adjusted and bias-adjusted kappa (PABAK), which accounts for potential distortions in agreement estimates arising from imbalanced category prevalence and rater bias. PABAK values were calculated for each evaluated dimension to quantify the degree of agreement beyond chance. Interpretation of agreement strength was guided by established benchmarks, with values closer to 1 indicating stronger agreement. This approach was selected in preference to conventional Cohen’s kappa due to the presence of skewed response distributions across several domains, which may underestimate agreement using unadjusted statistics^19–21.

All analyses were performed using R software, version 4.4.1 (R Foundation for Statistical Computing) and statistical significance was defined as a p < 0.05.

Author contributions

A.La.: Conceptualisation, data collection, data curation, formal analysis, methodology, project administration, original draft preparation, review and editing of manuscript, resources, supervision, visualisation. A.O.: Data collection, data curation, project administration, review and editing of manuscript. M.W.: Data collection, data curation, project administration, review and editing of manuscript. S.A.V.: Conceptualisation, methodology, review and editing of manuscript. I.K.: Conceptualisation, methodology, review and editing of manuscript. R.O.: Data collection, data curation, project administration. K.G.: Data collection, data curation, project administration. J.W.: Data collection, data curation, project administration. M.R.: Formal analysis, data curation, visualisation, review and editing of manuscript. A.Lo.: Formal analysis, data curation, visualisation, review and editing of manuscript. A.M.G.: Review and editing of manuscript. J.B.: Conceptualisation, data collection, data curation, methodology, project administration, original draft preparation, review and editing of manuscript, resources, supervision,

Data availability

The data generated and analysed during this study are available from the corresponding author upon reasonable request. The dataset contains de-identified patient queries and corresponding responses generated by human experts and the large language model used in this study. No identifiable patient information is included. The evaluation framework and analysis used to generate the results are also available from the corresponding author upon request to facilitate transparency and reproducibility.

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at https://doi.org/10.1038/s41746-026-02889-8.