Assessing the accuracy and readability of large language models in answering Turkish-language patient-generated queries on radiological contrast agents

Abstract

Objective

To assess the clinical accuracy, structural readability, and Ensuring Quality Information for Patients (EQIP-36) quality of Large Language Model (LLM) responses to real patient queries regarding radiological contrast agents and to define their safe boundaries in practice.

Methods

The 30 most popular Turkish patient questions reflecting fears and risk perceptions about contrast agents were identified via Google Trends and AlsoAsked.com. These queries were presented to ChatGPT-5.4, Gemini 3, and Claude 4.6 Sonnet using a zero-shot learning approach. Responses were evaluated by experts for clinical accuracy against the 2024/2025 American College of Radiology (ACR) Manual on Contrast Media. Information quality and readability were measured using the expanded EQIP-36 scale and the Ateşman and Bezirci-Yılmaz indices.

Results

Claude 4.6 Sonnet demonstrated the highest clinical accuracy and high compliance with guidelines (>90%). Gemini 3 exhibited a more conservative stance, occasionally resulting in over-triage by exaggerating risks. While all models scored exceptionally well in the Content and Structure dimensions of the EQIP-36, generating readable texts, they universally failed in the Identification dimension by omitting author names, update dates, or bibliographic sources. Readability indices indicated that comprehending the texts required an average of 10.5 to 11.6 years of formal education.

Conclusion

LLMs generate highly readable and subjectively reassuring Turkish-language texts in response to patient queries regarding radiological contrast agents. However, due to domain-specific structural gaps regarding quantitative thresholds and lack of citations, they should be positioned as hybrid communication tools subject to mandatory review by specialist physicians rather than standalone medical advisors.

Introduction

The healthcare paradigm of the 21st century is undergoing a profound transformation alongside the democratization of information and the rise of digital platforms. Rather than consulting healthcare professionals directly before any medical intervention, surgical procedure, or diagnostic examination, individuals are turning to internet search engines as their primary source of information. Research shows that one in three Americans researches their symptoms online before seeking medical care, and more than half are convinced they have a specific illness based solely on these search results. ¹

This phenomenon is not limited to the US; it reflects a global digital health literacy paradigm, with Turkish patients similarly relying heavily on internet platforms to decode their medical procedures prior to clinical visits. ² The immense volume of digital health-related searches provides a unique research laboratory for big data analytics. ² In particular, anonymized search queries obtained through the Google Trends platform are increasingly recognized in medical research as a reliable data mining tool that reflects the public’s collective health-related anxieties, epidemiological trends, and information gaps in real time.

Radiology is one of the medical specialties where patient anxiety is most pronounced, due to its complex technological infrastructure, the use of ionizing radiation, and the presence of contrast agents administered externally to the body. The global increase in the frequency of advanced imaging methods has also led to a growing need for patients to understand their radiological reports and question the safety of the procedures performed.^3,4

In addition, it has been found that patients devote significant time to cognitive questions such as procedural comfort, the duration of the procedure, and the differences between methods. ⁵ At the very center of these concerns are gadolinium-based contrast agents (GBCAs) used in MRI examinations and iodinated contrast agents used in computed tomography. Patients frequently inquire on internet forums and search engines about the potential adverse effects of these pharmacological agents on kidney function, their potential for allergic reactions, and their safety profiles during pregnancy or breastfeeding.^5,6

Amid this complex web of misinformation and anxiety, the emergence of AI-based Large Language Models (LLMs) has created a paradigm shift in the field of healthcare communication. ⁷ Recent literature extensively demonstrates the rapidly expanding utility of language-model-based biomedical studies across various clinical and diagnostic domains,^8,9 fundamentally altering how medical knowledge is synthesized. Some of these applications are now being used directly by patients as virtual health assistants. ⁶ Instead of reading complex medical articles or lengthy informational texts from official healthcare institutions, patients can type their symptoms and concerns in natural language directly into these chatbots and receive immediate, personalized responses often written in an empathetic tone. ¹⁰

However, the fundamental issue underlying this impressive technological showcase is whether the fluent text generated by artificial intelligence is always fully aligned with medical facts and current clinical guidelines. ¹¹ The 2024 and 2025 updated versions of the American College of Radiology (ACR) Manual on Contrast Media ¹² have overturned many clinical dogmas previously considered correct.

The extent to which AI models that generate responses to patients’ simple search engine queries can integrate such specific, quantitative, and up-to-date guideline changes into their responses is a critical research topic aimed at preventing patient misdirection. In addition to clinical accuracy, the structural quality and readability of the information provided to patients are also integral components of health communication. ¹³ Standardized tools such as the expanded EQIP (Ensuring Quality Information for Patients) are needed to objectively measure the quality of text generated by artificial intelligence. ¹⁴ The 36-item expanded version was specifically selected over the original 20-item EQIP because its highly granular ‘Identification’ subscale is critically suited for detecting missing citations or hallucinated references, which are hallmark vulnerabilities of generative AI. The use of tools like EQIP aims not only to ensure that the generated information is “readable” but also that it is “reliable and verifiable”. ¹⁴

To date, the literature lacks comprehensive studies evaluating LLM performance in radiology patient communication specifically within non-English languages. Therefore, addressing this gap represents a primary novelty of the current research.

The aim of this study is to use real patient questions identified via Google Trends and AlsoAsked.com to assess the clinical accuracy, EQIP-36 quality, and structural readability of LLM algorithms using a multidisciplinary methodology, and to define the safe boundaries of artificial intelligence in radiology practice.

Material and methods

Study design and ethical statement

This study was designed as an in silico computational, cross-sectional research project integrating data mining, natural language processing-based algorithmic text analysis, and guideline-based structured assessment methods. Since no human or animal subjects were used, and the analysis exclusively involved publicly available, open-source digital data and anonymous text generated by commercial artificial intelligence platforms, this research does not constitute human subject research. Consequently, formal ethical review and the issuance of an ethics waiver number are not required by the Institutional Ethics Committee of Necmettin Erbakan University. The research design consists of identifying search trends, querying artificial intelligence models, and conducting a multidimensional quality analysis of the generated data.

A comprehensive workflow diagram illustrating the phases of data collection, model querying, and multidimensional evaluation is presented in Figure 1.

Figure 1.

Flowchart illustrating the in silico experimental workflow, detailing the data acquisition of patient queries, the large language model generation phase, and the multidimensional evaluation methodology.

Research sample

Data was obtained through Google Trends which provides real-time data from Google and AlsoAsked.com, a platform that visualizes users’ hierarchical question patterns, with the aim of identifying patients’ actual, unfiltered, and most common information-seeking behaviors regarding radiology and contrast agents. The analysis period was set to cover the timeframe from January 1, 2024, to January 1, 2026, and the location parameter was restricted to Türkiye.

During the data extraction process, key terms representing radiological anxiety in Turkish (“MRI contrast agent,” “risks of contrast-enhanced CT scans,” “radiation risk,” “MRI contrast agent,” “contrast allergy,” “does MRI harm the kidneys”) were first defined in the system. The search terms obtained through the system were collected in a pool. From this raw dataset, repetitive questions, logistical queries lacking a clinical context, (operationally defined as administrative inquiries strictly regarding hospital directions, scheduling appointments, or MRI pricing) and grammatically nonsensical word clusters were excluded. Consistent with the single-rater design of this study, this iterative filtering and consensus process was performed exclusively by the primary board-certified radiologist. In the final stage, the thirty most popular unique questions that most clearly reflect patients’ fears, risk perceptions, and concerns about complications, while maintaining semantic coherence, were identified. Before querying the models, this final set of 30 questions was iteratively reviewed by a board-certified radiologist for clinical relevance and face validity to ensure they genuinely represented real-world patient anxieties. The complete list of all 30 finalized queries, along with their original Turkish formulations and English translations, is provided in Supplementary Table S1. (Example queries: “Does contrast-enhanced CT cause kidney failure?”, “Should breastfeeding mothers pump their milk if they receive an MRI contrast agent?”, “How is the contrast agent remaining in the body after an MRI scan eliminated?”, “Does gadolinium accumulate in the brain?”).

Selection and querying process of large language models (LLMs)

The defined 30-question dataset was directed at three distinct generative Large Language Models. To optimally simulate real-world patient information-seeking behavior, all queries were conducted through the standard consumer web interfaces rather than developer APIs. The exact model versions accessed on the date of querying were ChatGPT (model version: GPT-5.4), Gemini (model version: 3 Pro), and Claude (model version: 4.6 Sonnet). Because the standard consumer interfaces were utilized, backend probabilistic sampling parameters (such as temperature and top-p) were fixed at their proprietary defaults.

Crucially, real-time web-search capabilities were explicitly disabled across all platforms. This strict parameter ensured that the outputs reflected the models’ intrinsic parametric knowledge rather than their ability to retrieve and summarize external websites. All model queries and data collection were completed on a single day (March 1, 2026) to prevent unmapped API updates from confounding the results.

The query process was designed to strictly mimic the natural language format used by patients. A true zero-shot learning approach was employed; the default consumer system prompt was left unaltered, and no custom pre-training instructions or context (e.g., “You are a radiologist”) were provided. To prevent potential context bleeding from the model’s memory or cache, each question was directed at each model exactly once by opening a completely new and clean session. While LLM outputs are inherently stochastic, this single-query design was deliberately chosen to maximize ecological validity; it accurately mirrors the real-world behavior of a patient who typically reads, interprets, and acts upon the very first response generated by the chatbot, rather than querying the system multiple times to assess consistency.

Assessment of clinical accuracy and compliance with guidelines

The clinical accuracy and scientific currency of Turkish responses generated by LLMs were measured directly against the 2024 and 2025 revisions of the American College of Radiology’s publication titled “ACR Manual on Contrast Media”. ¹²

The evaluation parameters were based specifically on the following thresholds, which have recently been updated in the guidelines and are most critical for patients:

• ● eGFR Thresholds and Contrast-induced acute kidney injury: To what extent does the text reflect the current evidence indicating that an eGFR (estimated glomerular filtration rate) of 30 mL/min/1.73 m² or higher in patients receiving intravenous iodinated contrast agents does not pose an increased risk of contrast-induced acute kidney injury?.

• ● Risk of nephrogenic systemic fibrosis and GBCA burden: Whether the concept of “GBCA burden” for gadolinium-based contrast agents is mentioned, and whether it is stated that the risk of Nephrogenic Systemic Fibrosis (NSF) is now considered “rare” in Stage 3 chronic kidney disease (CKD) patients with a stable eGFR value between 30 and 59 mL/min/1.73 m².

• ● Timing of Reactions: Informing the patient that nearly all major or minor acute allergic or physiological reactions to the contrast agent occur at the time of injection or within the first twenty minutes following injection.

The responses were evaluated by a single board-certified radiologist (with 10 years of clinical experience). To minimize bias, the evaluator was completely blinded to the identity of the LLM that generated each text. The evaluator rated each response on a 5-point Likert scale (1: Completely incorrect, misleading, or potentially harmful; 5: Fully consistent with the guidelines, containing specific quantitative data, and clinically flawless).

Measuring the quality of patient information: The expanded EQIP-36 scale

The quality dimension of the LLM responses was analyzed using the 36-item expanded version of the “Ensuring Quality Information for Patients” (EQIP) tool, which was developed to assess standards for patient information materials in the international arena. ¹⁴ This comprehensive tool, created by adding 16 new items to the original 20-item instrument, evaluates health information through 36 specific questions (Q1–Q36) across three core dimensions.

• 1. Content - 18 Criteria

• 2. Identification - 6 Criteria

• 3. Structure - 12 Criteria

Each item has been coded by experts using a precise and rigorous scoring system: ‘Yes’ (full compliance), ‘Partly’ (partial compliance), ‘Not applicable’ (out of context). The EQIP global calculation formula was used to convert the results into a quantitative score.

$$\text{EQIP}\hspace{0.17em}\text{Global}\hspace{0.17em}\text{Score}=\left(\left(\sum \text{Yes}\times 1\right)+\left(\sum \text{Partly}\times 0.5\right)\right)/\left(\text{Total}\hspace{0.17em}\text{Evaluated}\hspace{0.17em}\text{Items}-\text{Number}\hspace{0.17em}\text{of}\hspace{0.17em}\mathrm{N}/\mathrm{A}\hspace{0.17em}\text{Items}\right)\times 100$$

In the context of AI-generated digital text, criteria demanding specific physical attributes (such as print layout, font size, or formatting of a physical brochure) were designated as ‘Not applicable’ (N/A).

This formulation yields a quality score ranging from 0 to 100. In accordance with the methodology accepted in similar studies in the literature, texts scoring above the 75th percentile of the global score (75 points or higher) have been standardized as “High-Quality Patient Information Resources”.

Determining readability levels

To assess patients’ ability to read and understand health materials, two primary formulas with the highest scientific validity and reliability in studies conducted on Turkish texts have been used: the Ateşman Readability Index ¹⁵ and the Bezirci-Yılmaz Formula. ¹⁶ The syntactic features of the texts obtained from the models were processed using SmoothText, an open-source Python library designed for calculating readability scores and statistical information for texts in multiple languages (available at: https://github.com/smoothtext/smoothtext), to perform the calculations.

Ateşman Readability Index: This formula operates based on the syllable lengths of words and the word density of sentences, scoring the text’s difficulty level on a scale from 0 (most difficult) to 100 (easiest). The mathematical representation of the Ateşman index is as follows:

$$\text{Score}=198.825-40.175\times \left(\frac{\left(\text{Total}\hspace{0.17em}\text{Number}\hspace{0.17em}\text{of}\hspace{0.17em}\text{Syllables}\right)}{\left(\text{Total}\hspace{0.17em}\text{Number}\hspace{0.17em}\text{of}\hspace{0.17em}\text{Words}\right)}\right)-2.610\times \left(\frac{\left(\text{Total}\hspace{0.17em}\text{Number}\hspace{0.17em}\text{of}\hspace{0.17em}\text{Words}\right)}{\left(\text{Total}\hspace{0.17em}\text{Number}\hspace{0.17em}\text{of}\hspace{0.17em}\text{Sentences}\right)}\right)$$

According to the scoring scale, texts scoring between 90 and 100 are classified as “Very Easy,” those between 70 and 89 as “Easy,” those between 50 and 69 as “Moderate,” those between 30 and 49 as “Difficult,” and those between 0 and 29 as “Very Difficult (Academic).”

Bezirci-Yılmaz Readability Index: Unlike the Ateşman formula, this index incorporates the multi-syllable structures of words in the text into its analysis and expresses the minimum level of education required to fully comprehend the text in terms of “years” (ranging from 1 to 16 years). Adapted to the Turkish Republic’s education system, this formula clearly demonstrates whether health information aligns with the target audience’s educational and sociodemographic profile.

Statistical analysis

All quantitative data were analyzed using Python (v3.12; SciPy v1.17, statsmodels v0.14, scikit-posthocs v0.9). Continuous variables are presented as mean ± standard deviation (SD) with 95% confidence intervals (CI), and categorical variables as frequencies and percentages with Wilson score 95% CIs. The distributional assumption of normality was formally assessed for every continuous outcome within each model group using the Shapiro-Wilk test, and homogeneity of variance was assessed using Levene’s test. Where all three model groups satisfied normality (Shapiro-Wilk p > 0.05) and equal variances (Levene’s p > 0.05), one-way analysis of variance (ANOVA) was used as the primary test; where either assumption was violated, the non-parametric Kruskal-Wallis H test was used as the primary test, with ANOVA additionally reported for comparability. Effect sizes are reported as eta-squared (η²) for ANOVA and epsilon-squared (ε²) for Kruskal-Wallis, interpreted as small (≈0.01), medium (≈0.06), and large (≥0.14). When the omnibus test was significant (p < 0.05), pairwise post-hoc comparisons were performed using Tukey’s Honestly Significant Difference (HSD) test (for ANOVA-appropriate variables) or Dunn’s test with Bonferroni correction for multiple comparisons (for Kruskal-Wallis-appropriate variables); pairwise Cohen’s d was additionally calculated to quantify the magnitude of each pairwise mean difference (small ≈0.2, medium ≈0.5, large ≥0.8). Categorical guideline-adherence indicators (eGFR threshold adherence, NSF/GBCA burden currency, over-triage/misleading statement rate) were compared across the three models using the chi-square test of independence. All tests were two-tailed, and statistical significance was set at p < 0.05.

Given the exploratory nature of this study and the qualitative text-analysis framework, a formal a priori power analysis was not conducted. The sample size was deemed sufficient for a pilot evaluation of overall model performance, though statistical comparisons at the sub-category level should be interpreted with caution.

Results

Thematic classification of Google trends queries

When content and thematic analyses were conducted on the 30 most popular patient questions identified in the initial phase of the study, it was observed that the queries clustered into specific major categories of concern. This provides a clear map of the information gaps patients encounter in radiology departments. As shown in Table 1, nearly half (43.3%) of online radiology inquiries concern the harmful effects of contrast agents on the kidneys.

Table 1.

Thematic classification of google trends queries regarding contrast agents.

Thematic category	Number of questions (n=30)	Percentage distribution (%)	Prominent keywords/Questions
Nephrotoxicity and Organ Damage	13	43.3%	“Does contrast CT cause kidney failure?”, “Does MRI contrast rot the kidneys?”, “At what creatinine level is contrast withheld?”
Pregnancy, Breastfeeding, and Long-Term Risks	8	26.6%	“Does MRI contrast harm the baby during pregnancy?”, “Does contrast medium pass into breast milk?”, “Does gadolinium accumulate in the brain?”
Allergy and Acute Reactions	5	16.7%	“Itching after contrast CT”, “When does MRI contrast allergy occur?”, “Contrast medium anaphylaxis”
Procedural Comfort and Alternatives	4	13.4%	“How long does a contrast MRI take?”, “Can’t the disease be diagnosed without contrast?”, “How is the contrast eliminated from the body?”

Clinical accuracy and compliance with ACR guidelines of large language models

When the compliance of the medical responses provided by ChatGPT-5.4, Gemini 3, and Claude 4.6 Sonnet to patient-generated scenarios with the current version of the ACR Contrast Guidelines was analyzed, statistically and clinically significant differences were observed among the models.

Because clinical accuracy scores were derived from a 5-point Likert scale, their distribution was expectedly non-normal within each model group (Shapiro-Wilk, all p < 0.001); the Kruskal-Wallis test was therefore used as the primary inferential test, with one-way ANOVA reported in parallel for comparability. Both approaches converged on a statistically significant overall difference among the three models (Kruskal-Wallis H(2) = 18.14, p < 0.001, ε² = 0.19 [large effect]; ANOVA F(2,87) = 7.36, p = 0.001, η² = 0.15 [large effect]). Mean clinical accuracy scores with 95% CIs were: Claude 4.6 Sonnet 4.77 (95% CI 4.62–4.91), ChatGPT-5.4 4.58 (95% CI 4.41–4.75), and Gemini 3 4.32 (95% CI 4.12–4.51). Dunn’s post-hoc test with Bonferroni correction localized this omnibus difference specifically to the Claude 4.6 Sonnet vs. Gemini 3 comparison (padj < 0.001; Cohen’s d = 0.99, a large effect), corroborated by Tukey’s HSD (mean difference = 0.45, 95% CI 0.17–0.73, padj < 0.001). The ChatGPT-5.4 vs. Gemini 3 comparison approached but did not reach significance after correction for multiple comparisons (Dunn padj = 0.054; Cohen’s d = 0.55, a medium effect), and the Claude 4.6 Sonnet vs. ChatGPT-5.4 comparison was not statistically significant (Dunn padj = 0.18; Cohen’s d = 0.43, a small-to-medium effect).

The results of the evaluations conducted by expert radiologist using a Likert scale (1–5) are summarized in Table 2.

Table 2.

Clinical accuracy of Large Language Models (LLMs) and their adherence to American College of Radiology (ACR) guidelines.

Model	Clinical accuracy (1–5) (mean)	Adherence rate to eGFR thresholds (%)	Up-to-dateness rate of NSF and GBCA burden (%)	Potential misleading/Over-triage statement rate (n, %)
Claude 4.6 Sonnet	4.76 ±0.38	93.3%	90.0%	3.3% (n=1)
ChatGPT-5.4	4.58 ± 0.45	86.6%	80.0%	6.6% (n=2)
Gemini 3	4.31 ± 0.52	66.6%	60.0%	16.6% (n=5)

NSF, Nephrogenic Systemic Fibrosis; GBCA, Gadolinium-Based Contrast Agents; eGFR, Estimated Glomerular Filtration Rate.

Based on the data in the table, Claude 4.6 Sonnet demonstrated the highest performance in terms of clinical accuracy and full compliance with guidelines. Particularly in questions regarding renal function thresholds, it was observed that Claude consistently conveyed the information that an eGFR ≥ 30 mL/min/1.73 m² threshold is safe for iodinated agents.

Guideline-adherence rates likewise differed significantly across models. Adherence to the eGFR ≥ 30 mL/min/1.73 m² safety threshold differed significantly among the three models (χ²(2) = 7.91, p = 0.019): Claude 4.6 Sonnet 93.3% (95% CI 78.7–98.2%), ChatGPT-5.4 86.7% (95% CI 70.3–94.7%), and Gemini 3 66.7% (95% CI 48.8–80.8%). Currency of NSF/GBCA-burden information also differed significantly (χ²(2) = 7.83, p = 0.020): Claude 4.6 Sonnet 90.0% (95% CI 74.4–96.5%), ChatGPT-5.4 80.0% (95% CI 62.7–90.5%), and Gemini 3 60.0% (95% CI 42.3–75.4%). The over-triage/misleading-statement rate was numerically highest for Gemini 3 (16.7%, 95% CI 7.3–33.6%) versus ChatGPT-5.4 (6.7%, 95% CI 1.8–21.3%) and Claude 4.6 Sonnet (3.3%, 95% CI 0.6–16.7%), but this difference did not reach statistical significance (χ²(2) = 3.57, p = 0.168), likely reflecting the low absolute event rate and consequent limited power of the chi-square approximation in this cell (minimum expected count = 2.67).

In contrast, while the Gemini 3 model did not have difficulty understanding the textual content of the questions, it exhibited a more outdated and conservative stance regarding risk stratification and updates to medical guidelines. For example, the information that the risk of NSF is rare in a patient with Stage 3 Chronic Kidney Disease (eGFR 30–44) exposed to gadolinium-based agents was largely overlooked by Gemini; instead, the patient was subjected to an exaggerated warning that their “kidney failure could worsen.” However, upon a qualitative assessment of Gemini, it was observed that the tone of the response texts was much more empathetic, with frequent use of humane expressions aimed at reassuring the patient.

To systematically categorize these inaccuracies, a formal error taxonomy was applied, differentiating between errors of omission, errors of commission, quantitative imprecision, and reliance on outdated data. The 16.6% potential misleading rate observed in Gemini primarily consisted of ‘reliance on outdated data’ and ‘errors of commission’ (over-triage). For instance, when presented with a scenario involving a patient with stable Stage 3 Chronic Kidney Disease inquiring about gadolinium-based agents, Gemini generated an unnecessarily alarming response: ‘Since your kidneys are already compromised, using this contrast agent poses a significant risk of a severe condition called Nephrogenic Systemic Fibrosis (NSF), which can be fatal. You must urgently reconsider this with your doctor.’ This response completely ignores the updated ACR guidelines that classify NSF risk as ‘rare’ for this patient group, prioritizing exaggerated empathy and defensive warnings over precise medical facts.

When examining the timing of acute physiological and allergic reactions to contrast agents, ACR guidelines clearly state that nearly all life-threatening reactions occur immediately after injection or within the first twenty minutes at the latest. ChatGPT and Claude models integrated this “20-minute” rule into their responses, effectively managing the patient’s anxiety about whether an allergic reaction might occur later that evening at home, whereas Gemini avoided explicitly highlighting this timeline and instead used general statements.

Results on the quality of patient information using the EQIP-36 scale

The quality of medical information provided to patients by LLM staff was assessed structurally and substantively using the expanded 36-item EQIP scale. Significant regional variations were observed in the EQIP scores (Table 3). Shapiro-Wilk testing showed that the EQIP Content, Structure, and Identification subscale scores were non-normally distributed within model groups (all p < 0.001), consistent with their bounded, coarsely-graded scoring structure (whole- and half-point increments on narrow ranges), whereas the composite EQIP Global score approximated a normal distribution in all three groups (Shapiro-Wilk p = 0.08–0.20). Kruskal-Wallis testing was therefore used as the primary test for the three subscales, and one-way ANOVA as the primary test for the Global score, both supplemented by the alternate test for comparability.

Table 3.

Patient information quality results assessed via the EQIP-36 scale.

EQIP-36 dimension	Target reference scores	ChatGPT-5.4 score (mean) [95% CI]	Gemini 3 score (mean) [95% CI]	Claude 4.6 sonnet score (mean) [95% CI]
Content (Max. 18)	≥ 15	16.0	14.5	17.0
Structure (Max. 12)	≥10	11.0	11.5	11.5
Identification (Max. 6)	≥ 4	0.5	0.5	1.0
Global EQIP Score (0–100)	≥ 75 (High Quality)	76.5 [75.1 – 77.5]	71.8 [70.7 – 73.4]	81.2 [79.9 – 83.2]

The analysis results revealed that the models performed exceptionally well on the Content and Structure subscales of the EQIP. The models flawlessly fulfilled the procedure’s purpose (Q4), alternatives (Q5), complication scenarios (Q12), use of everyday language (Q25), addressing the reader in a respectful tone (Q28, Q29), and organizing the text in a logical sequence (Q32).

The omnibus difference among models was significant and large in magnitude for the EQIP Global score (ANOVA F(2,87) = 56.66, p < 0.001, η² = 0.57; Kruskal-Wallis H(2) = 51.80, p < 0.001, ε² = 0.57) and for the Content subscale (Kruskal-Wallis H(2) = 56.28, p < 0.001, ε² = 0.62; ANOVA F(2,87) = 61.22, p < 0.001, η² = 0.59), moderate for the Identification subscale (Kruskal-Wallis H(2) = 20.05, p < 0.001, ε² = 0.21), and small for the Structure subscale, which did not reach significance under the non-parametric test (Kruskal-Wallis H(2) = 4.45, p = 0.108, ε² = 0.03), despite a nominally significant parametric ANOVA (F(2,87) = 4.83, p = 0.010, η² = 0.10) a discrepancy attributable to the coarse, near-ceiling distribution of Structure scores (median 11.0 of 12 in all three models) and correspondingly interpreted as a non-robust finding. Post-hoc comparisons (Dunn’s test with Bonferroni correction) for the EQIP Global score confirmed that all three pairwise comparisons were statistically significant: Claude 4.6 Sonnet (mean: 81.2, 95% CI 79.9 – 83.2) scored significantly higher than both ChatGPT-5.4 (mean 76.5, 95% CI 75.1 – 77.5; padj < 0.001, Cohen’s d = 1.64, a large effect) and Gemini 3 (mean 71.8, 95% CI 70.7 – 73.4; padj < 0.001, Cohen’s d = 2.76, a very large effect), and ChatGPT-5.4 scored significantly higher than Gemini 3 (padj = 0.018, Cohen’s d = 0.98, a large effect). The same ordinal pattern (Claude 4.6 Sonnet > ChatGPT-5.4 > Gemini 3) held for the Content subscale, with all three pairwise comparisons significant (all padj ≤ 0.013). For the Identification subscale, post-hoc testing showed that Claude 4.6 Sonnet scored significantly higher than both ChatGPT-5.4 and Gemini 3 (both padj < 0.001, Cohen’s d = 1.20 for each comparison), while ChatGPT-5.4 and Gemini 3 did not differ from one another (padj = 1.00, Cohen’s d = 0.00) both converging on the uniformly poor Identification performance described qualitatively below. No pairwise comparison reached significance for the Structure subscale after Bonferroni correction (all padj ≥ 0.20).

However, in the area of Identification (Q19–Q24) AI systems experienced a structural breakdown. LLMs did not include any author names, update dates, or bibliographic sources based on medical evidence within the text they generated (unless explicitly asked to do so). The model responses were presented entirely without references and in an authoritative tone.

When the global EQIP score was calculated, Claude 4.6 Sonnet and ChatGPT-5.4 succeeded in surpassing the 75th percentile threshold for high-quality patient information materials, while the Gemini 3 model fell below this threshold due to deficiencies in clinical details.

Analysis of readability and comprehensibility levels

The suitability of Turkish texts generated by artificial intelligence models for the general educational level of the population was calculated using the Ateşman and Bezirci-Yılmaz readability indices (Table 4).

Table 4.

Analysis of readability and comprehensibility levels across AI models and reference official health websites.

Readability metrics	ChatGPT-5.4	Gemini 3	Claude 4.6 sonnet	Reference (Official health sites)
Ateşman Index (0-100)	62.4 ± 4.5	65.1 ± 3.8	58.6 ± 5.2	55.7 ± 8.1
Ateşman Category	Moderate	Moderate	Moderate	Moderate/Hard
Bezirci-Yılmaz Index (Years)	10.8 ± 1.2	10.4 ± 1.1	11.6 ± 1.4	10.5 ± 2.4
Required Education Level	High School (Grades 10-11)	High School (Grades 9-10)	High School (Grades 11-12)	High School Level

The reference readability scores labeled as ‘Official Health Sites’ in Table 4 were derived from a purposeful sample of 20 standardized patient information texts publicly available on the official Turkish Ministry of Health website (https://www.saglik.gov.tr), evaluated using the exact same SmoothText methodology.

Inferential comparison confirmed that the between-model differences in readability were statistically significant for both indices (Table 5). For the Ateşman index, Kruskal-Wallis testing showed a significant omnibus difference (H(2) = 25.94, p < 0.001, ε² = 0.28, a large effect; ANOVA F(2,87) = 15.54, p < 0.001, η² = 0.26). Mean scores with 95% CIs were: Gemini 3 65.1 (95% CI 63.7–66.5), ChatGPT-5.4 62.4 (95% CI 60.7–64.1), and Claude 4.6 Sonnet 58.6 (95% CI 56.7–60.5) — all within the “Moderate” readability band, but ordered such that Claude 4.6 Sonnet produced measurably denser text than the other two models. Dunn’s post-hoc test confirmed that Claude 4.6 Sonnet differed significantly from both ChatGPT-5.4 (padj = 0.006, Cohen’s d = -0.78, a medium-to-large effect) and Gemini 3 (padj < 0.001, Cohen’s d = -1.43, a large effect), whereas ChatGPT-5.4 and Gemini 3 did not differ significantly from one another after correction (padj = 0.145, Cohen’s d = -0.65, a medium effect). For the Bezirci-Yılmaz index, one-way ANOVA confirmed a significant omnibus difference (F(2,87) = 7.20, p = 0.001, η² = 0.14, a large effect; Kruskal-Wallis H(2) = 14.16, p < 0.001, ε² = 0.14). Mean required-education levels with 95% CIs were: Claude 4.6 Sonnet 11.6 years (95% CI 11.1–12.1), ChatGPT-5.4 10.8 years (95% CI 10.3–11.2), and Gemini 3 10.4 years (95% CI 10.0–10.8). Tukey’s HSD post-hoc test showed that Claude 4.6 Sonnet required a significantly higher educational level than both ChatGPT-5.4 (mean difference = 0.80 years, 95% CI 0.04–1.56, padj = 0.038, Cohen’s d = 0.61, a medium effect) and Gemini 3 (mean difference = 1.19 years, 95% CI 0.43–1.96, padj = 0.001, Cohen’s d = 0.95, a large effect), whereas ChatGPT-5.4 and Gemini 3 did not differ significantly (mean difference = 0.39 years, 95% CI -0.37–1.16, padj = 0.44, Cohen’s d = 0.34, a small effect).

Table 5.

Summary of inferential statistics for all outcome measures, including normality testing, omnibus tests, effect sizes, and significant post-hoc pairwise comparisons.

Outcome	Shapiro-wilk (3 groups)	Primary test	Effect size	Significant post-hoc pairs (adj. p)
Clinical Accuracy (Likert 1–5)	Non-normal (all p<0.001)	Kruskal-Wallis H(2)=18.14, p<0.001	ε2=0.19 (large)	Claude vs. Gemini, p_adj<0.001
eGFR Threshold Adherence (%)	— (categorical)	χ2(2)=7.91, p=0.019	—	—
NSF/GBCA Currency (%)	— (categorical)	χ2(2)=7.83, p=0.020	—	—
Over-triage Rate (%)	— (categorical)	χ2(2)=3.57, p=0.168	—	ns
EQIP Global (0–100)	Normal (p=0.08–0.20)	ANOVA F(2,87)=56.66, p<0.001	η2=0.57 (large)	All 3 pairs, p_adj<0.02
EQIP Content (/18)	Non-normal (all p<0.001)	Kruskal-Wallis H(2)=56.28, p<0.001	ε2=0.62 (large)	All 3 pairs, p_adj≤0.013
EQIP Structure (/12)	Non-normal (all p<0.001)	Kruskal-Wallis H(2)=4.45, p=0.108	ε2=0.03 (small)	ns
EQIP Identification (/6)	Non-normal (all p<0.001)	Kruskal-Wallis H(2)=20.05, p<0.001	ε2=0.21 (large)	Claude vs. ChatGPT & Claude vs. Gemini, both p_adj<0.001
Ateşman Index (0–100)	Non-normal (2/3 groups)	Kruskal-Wallis H(2)=25.94, p<0.001	ε2=0.28 (large)	Claude vs. ChatGPT (p_adj=0.006); Claude vs. Gemini (p_adj<0.001)
Bezirci-Yılmaz (years)	Normal (all p>0.07)	ANOVA F(2,87)=7.20, p=0.001	η2=0.14 (large)	Claude vs. ChatGPT (p_adj=0.038); Claude vs. Gemini (p_adj=0.001)

ns, not significant at α=0.05 after correction for multiple comparisons. Effect size thresholds: η²/ε² — small ≈0.01, medium ≈0.06, large ≥0.14. All post-hoc p-values (p_adj) are corrected for multiple pairwise comparisons (Dunn’s test: Bonferroni; ANOVA: Tukey HSD).

According to the Bezirci-Yılmaz scoring system, in order to fully understand the texts, one must have completed an average of 10.5 to 11.6 years of formal education by Turkish standards. Claude 4.6 Sonnet received a lower readability score compared to the others because it heavily integrated specific terms from the ACR guidelines (eGFR, creatinine clearance, systemic fibrosis) into the text, increasing the required high school education level to 11.6 years. Gemini, on the other hand, achieved the highest scores on readability indices by minimizing medical terminology and maximizing the use of everyday language.

Discussion

The results of this study simultaneously highlight the potential benefits and hidden risks associated with positioning Large Language Models (LLMs) as “patient informants” within the digital health ecosystem. The original hypothesis put forward at the outset has been strongly supported by the multidimensional analyses conducted: AI chatbots do indeed provide responses to patients’ concerns that are highly readable, empathetic, and structurally flawless well-organized; however, they contain structural gaps regarding specific quantitative thresholds central to clinical decision-making processes (eGFR limits) and in the substantiation of information (attribution/citation).

Radiology practice, by its very nature, relies on precise mathematical calculations, pharmacological half-lives, and population-based net risk thresholds. This study has demonstrated that the Claude 4.6 Sonnet model is the algorithm that best captures this rigorous medical logic pattern and most accurately translates the current ACR 2024 contrast guidelines into text. While previous studies evaluating LLM performance in entirely different tasks such as kidney disease diagnosis ¹⁷ and BI-RADS assessments ¹⁸ also reported high accuracy rates for Claude, drawing direct cross-task comparisons requires caution. Nevertheless, Claude’s consistently high adherence to clinical guidelines in this study serves as circumstantial convergent evidence that its underlying weighting mechanisms may effectively prioritize established medical consensus texts.

In contrast, ChatGPT which has been extensively tested for its baseline clinical reasoning capabilities in various medical specialty board examinations^11,19 generates generally accurate and acceptable responses to questions about contrast agents; however, it has adopted a more superficial and defensive model of health communication rather than delving into clinical details. Gemini’s outputs, on the other hand, exhibited a consistent pattern of emphasizing empathetic and reassuring language, which in several instances paradoxically coincided with over-triage and the exaggeration of risk magnitudes. In a previous study comparing Gemini with emergency medicine specialists, it was found that Gemini classified 35% of patients into the high-risk group, resulting in over-triage ¹⁹ ; the behavior we observed in this study unnecessarily instilling fear of NSF in a Stage 3 kidney patient is consistent with this literature. When it comes to questions regarding the potential for contrast medium allergy in the field of radiology, the fact that nearly all acute physiological and allergic reactions occur within the first twenty minutes following injection is presented to the patient only by Claude and partially by ChatGPT, highlighting the differences in how the models comprehend emergency management parameters.

Upon deeper analysis of the data, a significant paradox emerges regarding digital health literacy. According to the EQIP-36 analysis used in the study, LLM’s received perfect scores in the “Content” and “Structure” categories. In other words, the text generated by the algorithm perfectly explains the purpose of the procedure, lists the risks and alternatives, and does so using an extremely polite, respectful, and everyday language. Scores in the 58–65 range derived from the Ateşman formula numerically demonstrate that these texts are as fluent and easy to read as the educational materials found on average Ministry of Health websites in Turkey.

However, the problem lies in the “Identification” dimension of the EQIP-36. It could validly be argued that applying a tool originally designed for static written materials to conversational AI naturally penalizes these models for structural omissions that simply reflect their interactive chatbot design. Yet, in the context of high-stakes medical communication, conversational fluency cannot excuse a lack of verifiable provenance. While corporate health websites anchor their information to a doctor’s approval or a scientific publication, LLMs offer complex medical advice entirely devoid of references. A chatbot providing clinical directives without citations is not merely a benign design feature; it is a systemic vulnerability. In medicine, an algorithm speaking with high confidence without evidence can create a hypnotic effect on the patient.

Considering the fact that 56% of patients are already convinced they have a specific diagnosis based on online symptom searches, ² there is a clinically significant risk that they might follow the advice of a reference-free algorithm that is highly fluent (high Ateşman scores) but prone to errors at clinical thresholds (e.g., the current ACR eGFR ≥ 30 threshold). ¹² While speculative, this scenario poses a hypothetical but severe medico-legal risk: a misguided patient could potentially delay or refuse a life-saving CT scan or cancer staging MRI out of an AI-induced fear that it ‘will harm their kidneys’. In medical consultations, the gap between AI’s high interpretability and its low ability to provide context serves as the strongest philosophical evidence that algorithms cannot be left to operate independently in medicine.

Current medical information (such as the significant changes made by the ACR in its 2024 and 2025 guidelines regarding gadolinium risk assessment) takes time to spread even among physicians in traditional medical education. ¹² Since the neural networks of AI systems are trained on data up to a specific date, the rigid taboos of older guidelines (model bias) remain hidden within the responses. ²⁰ For this reason, it is essential that AI systems used to digitize patient informed consent forms ¹³ in radiology clinics operate using the “Retrieval-Augmented Generation (RAG)” architecture. RAG-based systems must retrieve the current guideline data directly from the databases of the American College of Radiology or similar reputable organizations and construct the response accordingly. ²¹ However, it is crucial to acknowledge that RAG is not a panacea. Implementing RAG architectures introduces its own set of technical limitations, including increased computational costs, higher response latency, and the persistent risk of ‘contextual hallucination’ where the model misinterprets or inaccurately synthesizes the retrieved guidelines. Therefore, balancing these trade-offs is essential when designing clinical AI tools.

In addition, the Bezirci-Yılmaz readability scores obtained (approximately 10.5–11.6 years) indicate that even these conversational chatbots still generate complex medical concepts for patient populations with only a primary school education. ²² Simplifying terms such as “creatinine clearance,” “nephropathy,” or “anaphylactic reaction” without disrupting the text’s structure ²³ is only possible through a hybrid patient education model where physicians and artificial intelligence work in tandem. Indeed, recent reviews in the literature emphasize that while artificial intelligence can optimize medical education, it is not yet clinically or ethically ready to fully take over physicians’ decision-making processes. ²⁴

These critical shortcomings regarding clinical thresholds and the lack of verifiable citations raise significant questions about the regulatory and disclosure obligations of AI developers. As these consumer-grade platforms are increasingly accessed in clinical-adjacent contexts, they inhabit a gray area in digital health regulation. Under emerging frameworks like the European Union’s AI Act and the FDA’s Software as a Medical Device (SaMD) guidelines, AI tools that provide direct, high-risk medical guidance must be subjected to rigorous post-market surveillance. It is becoming a moral and regulatory imperative for technology companies to implement mandatory, highly visible ‘medical disclaimer’ labels on LLM outputs and to enforce algorithmic guardrails that prevent chatbots from overriding established clinical guidelines without providing evidence-based citations.

The methodological limitations of this study should be taken into account when interpreting the findings. First and most prominently, clinical accuracy was assessed by a single expert evaluator. The absence of multiple independent raters and inter-rater reliability statistics limits the robustness of the subjective Likert scoring. Future studies must employ multi-rater designs to ensure higher validity. Second, the models’ clinical reasoning capabilities were tested only using text-based scenarios generated by patients; in a real radiological workflow, presenting the patient’s laboratory results and past imaging files to the model in a visual (multimodal) format could lead to differences in performance. Third, the sample size of 30 unique questions is relatively modest. While sufficient for an exploratory/pilot evaluation of overall LLM performance, the small cell counts within the four thematic categories mean that statistical comparisons at the sub-category level are underpowered. Data was collected exclusively from a Turkish-language, Turkish-location context. Because LLMs are predominantly trained on English-language corpora, querying them in Turkish introduces a significant language-representation confound. This cross-lingual processing may underestimate the models’ true clinical accuracy and guideline adherence compared to their native English performance. Formal sample size justifications and cross-lingual comparisons should be incorporated into future methodologies Since the weighting and parametric structures (API updates) underlying commercial LLMs are constantly modified by companies, the EQIP and ACR alignment scores identified here may vary in future versions of the algorithms. Additionally, because LLM architectures are inherently stochastic, the generated texts may exhibit variability between sessions. While our single-query design reflects actual patient use-cases, future methodological frameworks could employ repeated querying and report output variability measures to assess the internal consistency of these algorithms. Furthermore, our structured clinical adherence assessment (Table 2) focused heavily on three specific ACR guideline updates (eGFR thresholds, NSF risk, and reaction timing). These parameters were deliberately selected because they represent recent, critical quantitative shifts in the 2024/2025 guidelines, thus serving as an optimal stress test for the models’ scientific currency. While broader topics such as pregnancy, breastfeeding, and general allergy management were present in the patient queries and assessed for overall accuracy, they were not the primary focus of the strict guideline adherence metrics. Future studies should develop comprehensive, multi-domain rubrics to evaluate LLM performance across all facets of radiological safety communication and also should focus on more complex scenarios that involve longitudinal analysis of patients’ laboratory findings. Moreover, while the generated responses were qualitatively observed to use polite and comforting language, this study did not employ validated psychometric instruments to directly measure algorithmic empathy, nor did it assess actual anxiety reduction in a live cohort of real patients. Future clinical trials utilizing patient-reported outcome measures (PROMs) are required to validate these effects. Finally, because the underlying training datasets, reinforcement learning protocols, and neural architectures of these commercial LLMs are strictly proprietary, any attribution of observed output differences to specific internal model characteristics remains inherently speculative.

Conclusion

This study has demonstrated that AI-based LLMs are capable of generating highly readable and subjectively reassuring Turkish texts regarding contrast agent risks. Given the study’s restriction to a specific language, a targeted query set, and a single time point, these findings represent domain-specific observations rather than universal rules of AI behavior. Nevertheless, behind this fluency lie major information gaps that could impact clinical outcomes. The models’ inability to comprehend current ACR guidelines leads to exaggerated risk assessments, while the lack of citations transforms them into an authority without evidence. Consequently, LLMs should be positioned not as standalone sources of medical advice, but as hybrid tools that facilitate communication. To achieve this safely, they must be operationally integrated with current literature via RAG pipelines or physician-curated databases, bounded by mandatory medical disclaimers, and subject to mandatory review by specialist physicians.

ORCID iD

Abdullah Enes Ataş https://orcid.org/0000-0001-6623-3024

Ethical considerations

Because this study involved solely publicly available search trend data and AI-generated texts without any human or animal subjects, it does not meet the criteria for human subject research. Therefore, formal review and the issuance of an ethics waiver number are not required by the Institutional Ethics Committee of Necmettin Erbakan University. Since no human or animal subjects were used, the study is exempt from traditional ethics committee approval.

Data Availability Statement

The datasets generated and analyzed during the current study—including the complete corpus of 30 patient queries (in Turkish and English translations), the performance metrics and comparative evaluation of LLMs—are available in the Supplementary Material provided with this manuscript.*