Reliability and readability of adenoid hypertrophy information generated by five publicly accessible LLM chatbots: A default-setting snapshot study

Abstract

Objective

This study evaluated the performance of five major large language model (LLM) chatbots in generating patient-oriented information on adenoid hypertrophy, focusing on content reliability and readability.

Methods

Sixty-three frequently asked questions (FAQs) on adenoid hypertrophy were collected, covering seven domains including etiology, symptoms, and treatment. From October 1, 2025, to January 10, 2026, questions were submitted in English to five LLMs via their official web interfaces. Reliability was assessed using DISCERN, EQIP, JAMA benchmarks, and the Global Quality Scale (GQS). Readability was measured by six standard indices (ARI, CLI, FKGL, GFI, SMOG, FRES). Three otorhinolaryngology clinicians blindly scored all responses.

Results

Significant differences in reliability were found among models (P <0.001). Perplexity scored highest on DISCERN (41.98±1.87) and EQIP (58.40±3.67), followed by Copilot; ChatGPT and DeepSeek scored lowest. Only Copilot and Perplexity scored 1 point on JAMA benchmarks. No model met the recommended sixth-grade reading level. Gemini had the best readability (FRES: 61.95±9.64), while Copilot scored poorest (FRES: 24.27±10.77). All models failed to meet the recommended sixth-grade readability thresholds. (P <0.001).

Conclusion

Current LLMs show a notable imbalance between reliability and readability in generating adenoid hypertrophy information, with none excelling in both. In this default-setting, product-level snapshot, Perplexity showed higher information-quality scores, whereas Gemini generated comparatively easier-to-read responses. These findings should not be interpreted as a controlled benchmark of underlying base models. Limitations include potential prompt sensitivity, single-response sampling, and the snapshot nature of the assessment given rapid model updates. Future improvements should focus on source transparency, text simplification, and condition-specific evaluation to enhance AI-assisted health communication for pediatric care.

Introduction

Background

Adenoid hypertrophy (AH) is common in children. It happens when the lymphoid tissue at the back of the nose grows too large. As an integral constituent of the Waldeyer’s ring, the adenoid plays a pivotal role in early-stage immune defense. Epidemiological evidence indicates that the prevalence of AH among children aged 3 to 8 years is as high as 34%–70%.^1,2 Persistent enlargement can cause nasal obstruction, sleep apnea, and ear infections, affecting both sleep and facial development. ³ For parents, finding accurate and understandable information is the first step toward informed decisions.

AI and patient education

Large language models (LLMs) like ChatGPT are increasingly used by the public as quick sources of health information. However, concerns remain about whether these models can be trusted for medical advice. ⁴ Leading health organizations recommend that patient materials be written at or below a sixth-grade reading level^5,6 -- a benchmark that LLMs may or may not meet.

What we know and Don’t know

Existing studies have examined LLM performance in areas like urology^7–9 and chronic disease management,^10,11 with mixed results on reliability and readability. However, to our knowledge, no study has systematically evaluated LLMs specifically for pediatric otorhinolaryngology conditions such as adenoid hypertrophy. Given the condition’s high prevalence and its impact on family decision-making, a focused evaluation is needed.

Objectives

This study conducted a comparative evaluation of five LLMs (ChatGPT, DeepSeek, Gemini, Perplexity, and Copilot) on their ability to generate reliable and readable patient-oriented information about adenoid hypertrophy for parent education. Our goal is to provide evidence-based guidance for clinicians and families navigating AI-generated health information.

Methods

Ethics

Our study did not require ethics committee approval because no real patients were involved. To maximize the transparency and reproducibility of our work, we applied the CHART (Chatbot Assessment Reporting Tool) reporting guideline throughout this study. ¹² This meant we systematically detailed our use of the large language model, including which specific version we employed, how we developed and refined our prompts through several rounds of testing, and the extent of researcher review applied to all model outputs. Following this checklist also prompted us to explicitly discuss the limitations and potential biases inherent in our AI-assisted approach.

Patient and public involvement

No direct patient or public involvement was conducted. Patient- and caregiver-oriented concerns were indirectly represented through publicly available search trends, social media questions, patient forums, and online Q&A platforms.

Study design and data collection

Question bank construction

Standardized nomenclature pertaining to “adenoid hypertrophy” was initially identified utilizing the Medical Subject Headings (MeSH) database, encompassing terms such as “Adenoid hypertrophy,” “Adenoidectomy,” and “Sleep apnea, obstructive/etiology.” These descriptors were subsequently consolidated into a unified theme within Google Trends to facilitate systematic tracking of global search dynamics.

The preliminary question repository was co-authored by an otorhinolaryngology resident and an associate chief physician, drawing upon both MeSH search outcomes and extensive clinical expertise. To ensure the authentic representation of patient concerns, this repository was further augmented and linguistically refined by incorporating high-frequency inquiries harvested from prominent social media platforms (e.g., Xiaohongshu and Zhihu) and specialized patient forums.

Prompt optimization was achieved through two iterative phases of pre-testing involving ten pilot questions. Observations during the pilot phase indicated a tendency for certain models to generate overly laconic responses to open-ended inquiries. However, to preserve the authentic nature of patient-sourced questions, no additional framing or structural instructions were added to the prompts. The 63 questions were submitted exactly as collected from public sources.

We stopped collecting questions when three consecutive sources gave us no new topics.^13,14 Inclusion and exclusion criteria. Questions were included if they met all three criteria: (a) directly related to adenoid hypertrophy (diagnosis, treatment, complications, or long-term management); (b) phrased as natural language questions (interrogative form); and (c) originated from patients, parents, or caregivers based on platform context and wording. Questions were excluded if they were duplicate or near-duplicate items; outside adenoid hypertrophy-specific concerns; administrative, financial, or lifestyle-only questions; overly technical clinician-facing prompts; likely non-human or promotional content; unsafe self-management prompts; or questions requiring individualized emergency triage or medication changes. For potentially safety-sensitive questions, items were excluded only when they required individualized management beyond the scope of a general patient-information evaluation, whereas general safety-related concerns were retained when they reflected common caregiver information needs.

Redundancy removal and representativeness assessment. After collection, two independent reviewers compared all candidate questions for duplication. A question was considered a duplicate if the exact wording matched after normalizing punctuation and spacing, or if the intended meaning was judged identical (e.g., “Does my child need surgery?” vs. “Is surgery necessary for my child?”). Disagreements were resolved by discussion with a third reviewer. For each set of duplicates, one representative version was retained. The final 63 questions were then independently classified by two clinical experts into seven predefined domains (etiology, symptoms, diagnosis, complications, treatment, surgery, and long-term care). Both experts confirmed that the final set adequately covered the clinical spectrum of adenoid hypertrophy, with no new domains proposed, supporting content representativeness.

A raw question pool screening log documenting the inclusion and exclusion decisions for all candidate questions is provided in Supplementary Table S1.

Model query and data acquisition

Five publicly accessible LLM chatbot products were evaluated: ChatGPT-5.2, DeepSeek, Gemini 3 Pro, Perplexity Pro, and Microsoft Copilot. Data were collected between October 1, 2025, and January 10, 2026, from Zhengzhou, China, using the official web-based interfaces of each product under default user-facing settings. No manual model switching, plug-in activation, or additional routing was selected by the investigators. When the exact base model or version was not displayed in the interface, it was recorded as “not disclosed.” Temperature, Top-P, and other generation parameters were not adjustable in the web interfaces used; therefore, the findings reflect real-world product-level performance rather than fully deterministic model benchmarking. No customized system prompts were implemented during the interaction. To eliminate potential contextual interference or “memory effects,” each inquiry was submitted individually in English through standalone chat sessions. For every model, only the initial response generated for each specific question was elicited and subsequently archived for comprehensive statistical analysis (Table 1).

Table 1.

Characteristics and access details of the LLMs evaluated in this study.

Model (Name)	Developer	Base model/Version identifier	Open source status	Key attributes/Knowledge cut-off	Access method in this study
ChatGPT	OpenAI	GPT-5.2	Closed-source (Proprietary)	Knowledge cut-off: August 31, 2025	OpenAI ChatGPT Plus web interface (https://chat.openai.com)
DeepSeek	DeepSeek-AI	DeepSeek-V3*	Open-source (Open-weights)	Latest version during testing period	DeepSeek official web chat interface (https://chat.deepseek.com)
Gemini	Google	Gemini 3 Pro†	Closed-source (Proprietary)	long-context window	Google AI Studio or Gemini Advanced web interface
Perplexity	Perplexity AI	Perplexity Pro (exact base model not disclosed)‡	Closed-source (Proprietary)	Real-time web search (default feature of Pro version)	Perplexity Pro subscription web interface (https://www.perplexity.ai)
Copilot	Microsoft	Microsoft Copilot (exact version not disclosed)	Closed-source (Proprietary)	Exact base model and knowledge cut-off not publicly disclosed in the interface used during testing.	Microsoft Edge built-in Copilot sidebar or https://copilot.microsoft.com

*DeepSeek-V3 was the version available during the testing period (October 2025–January 2026).

^†Accessed via Google Gemini Advanced subscription (Gemini 3 Pro).

^‡For Perplexity Pro and Microsoft Copilot, the web interfaces did not disclose the exact base model or version. Closed-source models may update without public notice; our results reflect a snapshot of the testing period.

Evaluation instruments and procedures

To ensure the robustness and validity of the findings, a multi-dimensional appraisal framework was implemented, utilizing cross-validation through multiple standardized instruments. ¹⁵

Reliability assessment

This study employed a systematic suite of indices to evaluate the reliability of AI-generated content. Specifically, DISCERN was utilized to assess the integrity of treatment-related decision-making, while the JAMA benchmark criteria focused on transparency. These were augmented by the EQIP and GQS frameworks, which addressed patient-oriented presentation and overall structural quality, respectively. These tools measure how well information is presented (structure, completeness, source transparency). These instruments assessed information quality, structure, transparency, and patient-oriented presentation rather than factual accuracy itself. Factual correctness and potential safety concerns were assessed separately against a predefined clinical reference set, primarily including the 2023 German S2k guideline and supporting systematic reviews on diagnosis, pharmacological treatment, surgical outcomes, and pathophysiology. No single regional consensus was treated as a universal gold standard. The specific instruments and criteria are detailed below.

DISCERN instrument

This validated tool is specifically designed to judge the quality of written health information, particularly concerning therapeutic options. ¹⁶ The instrument comprises 16 items that scrutinize various facets, including clarity of objectives, transparency of information sources, evidence reliability, acknowledgment of uncertainty, and balance of content. Utilizing a 5-point Likert scale (1–5), higher scores denote superior reliability. In accordance with established literature, total scores were categorized into five quality tiers: 63–75 (Excellent), 51–62 (Good), 39–50 (Fair), 27–38 (Poor), and 16–26 (Very Poor). The DISCERN scale has demonstrated high reliability and validity across numerous medical informatics studies. ¹⁷

Ensuring quality information for patients (EQIP)

To facilitate an in-depth examination of informational structure and linguistic quality, the 20-item EQIP framework was adopted. ¹⁸ Each item was rated as “Yes,” “No,” or “Not Applicable,” with weights of 1, 0, or excluded from the total, respectively. The standardized percentage score was derived as follows: “Standardized Score” = [ (“Number of ”“Yes”“ items”×1)/20 ]×100. Referring to the original grading standards of the tool developer, we set: ≥76 points as “Excellent”, 51-75 points as “Good (minor adjustments needed)”, 26-50 points as “Qualified (improvement needed)”, and ≤25 points as “Unqualified”. ¹⁹

Global Quality Scale (GQS)

The widely used five-level overall evaluation scale in clinical research is adopted, which comprehensively assesses from three aspects: information completeness, logical coherence, and user comprehension difficulty. ²⁰ The specific scoring criteria are as follows: 5 points indicate comprehensive and accurate information, clear organization, and easy understanding; 4 points indicate that the main information is complete but there are some details missing; 3 points indicate that the information framework is complete but some parts need further explanation; 2 points indicate that key information is missing or there is a risk of misleading; 1 point indicates that the information quality is seriously insufficient. This scale provides researchers with a direct basis for overall quality judgment.

JAMA benchmark criteria (JAMA)

Referring to the academic information quality standards formulated by the Journal of the American Medical Association, ⁵ we conduct quantitative evaluations from four key dimensions: (1) completeness of information disclosure; (2) author and responsible party identification; (3) citation of references and data sources; (4) explanation of content timeliness. Each dimension is marked with 1 point if it meets the standard and 0 points if it does not, with a total score ranging from 0 to 4. This framework is particularly suitable for assessing whether the information meets the normative requirements of professional medical publishing. ²¹

Error and safety coding

Before coding, three error categories were defined a priori: fact-related errors, key information omissions, and vague or incomplete advice. To identify common weaknesses, responses in the lowest 50% of DISCERN scores were reviewed. Two reviewers independently coded these responses. Disagreements were resolved by a third reviewer. Safety concerns were coded separately for direct harmful recommendations, incorrect medication or dosage advice, and indirect misleading risk.

Readability assessment

This study uses six internationally recognized readability indicators to evaluate the text generated by artificial intelligence, specifically including: Automated Readability Index, Flesch Reading Ease Score, Gunning Fog Index, Flesch-Kincaid Grade Level, Coleman-Liau Index, and SMOG Index. ²² All evaluations are conducted through a standardized online readability analysis platform (https://readabilityformulas.com).

During the assessment process, the scores obtained from each indicator are compared with the readability standards for public health information recommended by the American Medical Association and the National Institutes of Health. ⁶ According to this standard, text suitable for general public understanding should reach a sixth-grade reading level, specifically: a Flesch Reading Ease Score of no less than 80, and the other five index scores should all be below 6. The formulas are as follows.^23,24

Automated Readability Index (ARI).

ARI = 4.71 × (characters/words) + 0.5 × (words/sentences) − 21.43.

Flesch Reading Ease Score (FRES).

FRES = 206.835 − 1.015 × (words/sentences) − 84.6 × (syllables/words).

Gunning Fog Index (GFI).

GFI = 0.4 × [(words/sentences) + 100 × (complex words/words)]

Flesch-Kincaid Grade Level (FKGL).

FKGL = 0.39 × (words/sentences) + 11.8 × (syllables/words) − 15.59.

Coleman-Liau Index (CLI).

CLI = 5.88 × (characters/words) − 0.296 × (sentences/words) − 15.8.

Simple Measure of Gobbledygook (SMOG).

SMOG = 1.043 × √(polysyllabic words × 30/sentences) + 3.1291.

Evaluation protocol and quality assurance

All responses were independently evaluated by a panel of three clinical experts, comprising attending physicians and senior nurses, each possessing over ten years of specialized experience in otorhinolaryngology. To ensure the objectivity of the results, a double-blinded scoring methodology was implemented, wherein the evaluators remained unaware of the specific models originating each response. Before formal scoring, all three raters completed a 2-hour training session using 10 pilot responses not included in the final analysis. The training covered the scoring criteria for DISCERN, EQIP, JAMA, and GQS, with discussion of edge cases to ensure consistent interpretation. For DISCERN, EQIP, and GQS, the mean of the three independent raters’ scores was used for quantitative analysis. For categorical or item-level binary judgments, including JAMA items and safety/error coding, majority agreement was used. When no majority was reached, disagreements were resolved through discussion after completion of independent scoring. ICCs were calculated using the original independent ratings before consensus discussion.

The clinical reference set included the 2023 German S2k guideline, ²⁵ systematic reviews covering diagnostic accuracy, pharmacological efficacy, surgical outcomes, and pathophysiology,^3,26–29 and regional consensus documents where applicable. A question-to-guideline mapping table is provided in Supplementary Table S2.

Regarding the assessment of reliability and response consistency, a random subset comprising 10% of the initial inquiries was resubmitted to each model to generate three independent responses for longitudinal analysis.

Statistical analysis

Data analysis was conducted using R language (version 4.5.2). Descriptive statistics were calculated for DISCERN, EQIP, JAMA, GQS, and readability indicators. For between-model comparisons (each question answered by all five models, repeated-measures design), Friedman tests were used for overall comparisons, with Kendall’s W reported as the effect size. Paired Wilcoxon signed-rank tests with Benjamini-Hochberg correction were used for post-hoc pairwise comparisons.

Inter-rater consistency was evaluated using the intraclass correlation coefficient (ICC) with a two-way random-effects, absolute-agreement, single-measure model.

For readability indicators, one-sample Wilcoxon signed-rank tests were used to compare grade-level indices against the recommended sixth-grade standard (maximum of 6 for ARI, CL, FKGL, GFI, SMOG; minimum of 80 for FRES). All tests were two-tailed, with a significance level of α = 0.05.

Results

Reliability assessment

The LLMs varied widely in generating health information about adenoid hypertrophy (all reliability indicators P < 0.001, Friedman tests). As shown in Tables 2 and 3, Perplexity achieved the highest mean scores for DISCERN (41.98 ± 1.87)and EQIP(58.40 ± 3.67), followed by Copilot, whereas ChatGPT and DeepSeek showed lower scores. In the JAMA transparency assessment, only Copilot and Perplexity received 1 point out of 4, while the remaining models received 0 points. This indicates that all evaluated products had substantial limitations in source attribution, authorship disclosure, and timeliness reporting. GQS scores ranged from 3.11 to 3.51, suggesting that most responses provided a basic informational framework but still required further explanation or contextualization (Figure 1).

Table 2.

Reliability scores across LLMs.

Metric	ChatGPT	95% CI	DeepSeek	95% CI	Perplexity	95% CI	Copilot	95% CI	Gemini	95% CI	Overall P	Kendall’s W
DISCERN	31.19 ± 2.64	[30.54, 31.84]	32.97 ± 3.04	[32.22, 33.72]	41.98 ± 1.87	[41.52, 42.44]	39.37 ± 1.47	[39.01, 39.73]	38.67 ± 1.44	[38.31, 39.03]	<0.001	0.874
EQIP	49.21 ± 4.95	[47.99, 50.43]	48.67 ± 5.56	[47.31, 50.03]	58.40 ± 3.67	[57.50, 59.30]	52.35 ± 3.18	[51.57, 53.13]	51.02 ± 3.26	[50.22, 51.82]	<0.001	0.509
JAMA	0.00 ± 0.00	[0.00, 0.00]	0.00 ± 0.00	[0.00, 0.00]	1.00 ± 0.00	[1.00, 1.00]	1.00 ± 0.00	[1.00, 1.00]	0.00 ± 0.00	[0.00, 0.00]	<0.001	1.000
GQS	3.13 ± 0.34	[3.05, 3.21]	3.11 ± 0.32	[3.03, 3.19]	3.51 ± 0.50	[3.39, 3.63]	3.32 ± 0.47	[3.20, 3.44]	3.21 ± 0.41	[3.11, 3.31]	<0.001	0.146

Data are mean ± SD with 95% CI in brackets. Overall P values from Friedman tests; Kendall’s W as effect size. Pairwise comparisons: Wilcoxon signed-rank with Benjamini-Hochberg correction.

For GQS, the significant Friedman test (p < 0.001) with low Kendall’s W (0.146) indicates that while model differences were statistically significant due to the sample size, the magnitude of the difference was small, consistent with the overlapping confidence intervals.

Table 3.

Pairwise comparisons of reliability metrics (DISCERN, EQIP, GQS, JAMA) -- Adjusted p-values.

Pairwise comparison	DISCERN	EQIP	GQS	JAMA
ChatGPT -Copilot	<0.001	<0.001	0.024	<0.001
ChatGPT - DeepSeek	0.067	0.578	0.838	1
Copilot - DeepSeek	<0.001	<0.001	0.020	<0.001
ChatGPT - Gemini	<0.001	0.035	0.341	1
Copilot - Gemini	0.116	0.087	0.218	<0.001
DeepSeek - Gemini	<0.001	0.009	0.275	1
ChatGPT - Perplexity	<0.001	<0.001	<0.001	<0.001
Copilot - Perplexity	<0.001	<0.001	0.028	1
DeepSeek - Perplexity	<0.001	<0.001	<0.001	<0.001
Gemini - Perplexity	<0.001	<0.001	0.0905	<0.001

Pairwise comparisons were performed using paired Wilcoxon signed-rank tests with Benjamini-Hochberg correction. Adjusted p-values are shown. P < 0.05 was considered statistically significant.

Figure 1.

Mean reliability scores of the LLMs based on DISCERN, EQIP, GQS and JAMA indexes.

Readability assessment

The reading difficulty of the generated texts by all models did not reach the sixth-grade level recommended by medical institutions (FRES ≥ 80, other indicators ≤ 6). As shown in Table 4, Gemini had the best readability (FRES: 61.95 ± 9.64), close to the “standard difficulty” range; Copilot had the worst readability (FRES: 24.27 ± 10.77), within the “very difficult” comprehension range. The other models (ChatGPT, DeepSeek, Perplexity) had moderate readability, among which DeepSeek’s FKGL score standard deviation was relatively large (±7.60), suggesting significant fluctuations in the sentence lengths of its responses (Figure 2). One-sample Wilcoxon signed-rank tests confirmed that all readability scores differed significantly from the recommended sixth-grade thresholds (p < 0.001 for all comparisons).

Table 4.

readability scores across LLMs.

Metric	ChatGPT	DeepSeek	Perplexity	Copilot	Gemini	Overall P	Kendall’s W
ARI	10.73 ± 3.05; 9.65 [8.57, 12.34]	12.72 ± 1.76; 12.64 [11.57, 13.84]	12.19 ± 3.51; 12.43 [10.02, 14.82]	16.09 ± 2.19; 16.17 [14.57, 17.55]	9.74 ± 2.22; 9.67 [8.12, 11.38]	<0.001	0.461
CL	12.91 ± 2.10; 12.76 [11.38, 14.48]	15.00 ± 1.91; 14.87 [13.85, 16.02]	14.63 ± 4.10; 15.01 [12.17, 17.46]	17.32 ± 1.92; 17.45 [15.83, 18.68]	10.24 ± 1.89; 10.64 [9.04, 11.42]	<0.001	0.657
FKGL	9.73 ± 2.55; 9.43 [7.77, 10.98]	12.37 ± 7.60; 11.38 [10.13, 12.95]	9.76 ± 3.30; 9.81 [7.73, 12.04]	13.90 ± 1.98; 13.99 [12.66, 15.12]	7.61 ± 1.85; 7.63 [6.54, 9.08]	<0.001	0.592
GFI	11.92 ± 2.01; 11.80 [10.30, 13.10]	13.60 ± 2.17; 13.50 [12.20, 15.05]	12.08 ± 3.20; 12.00 [9.95, 14.35]	15.26 ± 1.88; 15.40 [14.15, 16.60]	9.96 ± 1.59; 9.80 [8.95, 11.40]	<0.001	0.587
SMOG	8.69 ± 2.34; 7.65 [7.02, 9.86]	10.27 ± 1.41; 10.26 [9.46, 11.11]	8.47 ± 1.89; 8.49 [7.07, 9.77]	11.95 ± 1.57; 11.90 [11.03, 13.16]	7.97 ± 1.39; 7.76 [6.86, 9.19]	<0.001	0.489
FRES	44.76 ± 11.14; 47.00 [36.00, 53.00]	36.24 ± 12.17; 38.00 [28.00, 43.50]	40.63 ± 20.99; 41.00 [25.00, 53.00]	24.27 ± 10.77; 23.00 [17.00, 30.00]	61.95 ± 9.64; 61.00 [55.50, 69.50]	<0.001	0.671

ARI: Automated Readability Index, FRES: Flesch Reading Ease Score, GFI: Gunning Fog Index, FKGL: Flesch-Kincaid Grade Level, CL: Coleman-Liau Index, SMOG: Simple Measure of Gobbledygook, SD: Standard Deviation.

Data are mean ± SD; median [Q1, Q3]. Overall comparisons: Friedman tests with Kendall’s W. Pairwise comparisons: paired Wilcoxon signed-rank tests with Benjamini-Hochberg correction. P < 0.05 was significant.

One-sample Wilcoxon tests showed all grade-level indices exceeded 6 (P < 0.001), and FRES fell below 80 (P < 0.001).

Figure 2.

Mean readability scores of the LLMs based on ARI, GFI, FKGL, CL, SMOG and FRES indexes. The horizontal red line marks the 6th-grade reading level, the maximum recommended by the American Medical Association for patient education materials. Scores below this line (for ARI, GFI, FKGL, CL, SMOG) are desirable; for FRES, higher scores are better and scores above 80 are recommended. ARI: Automated readability index, GFI: Gunning Fog index, FKGL: Flesch-Kincaid grade level, CL: Coleman-Liau index, SMOG: Simple measure of gobbledygook, FRES: Flesch reading ease score.

Inter-rater consistency

The three clinical evaluators showed generally consistent ratings across the four reliability tools (Table 5). The ICC for DISCERN was 0.879 (95% CI: 0.788-0.924); for EQIP it was 0.811 (95% CI: 0.777-0.841); for GQS it was 0.882 (95% CI: 0.860-0.902); and for JAMA items, all raters assigned identical binary judgments; therefore, inter-rater agreement was complete, but the ICC should be interpreted cautiously because of the absence of rating variability. All ICC values were above 0.60.

Table 5.

Intraclass Correlation Coefficient (ICC) analysis of the three independent.

Metric	Agreement index	Estimate	95% CI	P Value	Complete items	No. of raters
DISCERN	ICC(2,1)	0.879	0.788-0.924	<0.001	315	3
EQIP	ICC(2,1)	0.811	0.777-0.841	<0.001	315	3
JAMA	Perfect agreement/ICC not estimable	Not estimable	NA	NA	315	3
GQS	ICC(2,1)	0.882	0.860-0.902	<0.001	315	3

Note. ICC interpretation criteria (Cicchetti, 1994): <0.40 = Poor, 0.40 - 0.59 = Moderate, 0.60 - 0.74 = Good, ≥0.75 = Excellent. All P values are <0.001, indicating that the ICC values are significantly not equal to 0. ICCs were calculated from the original independent ratings before any discussion.

Error pattern analysis

Among the responses in the lowest 50% of DISCERN scores (n = 158), approximately 30% contained fact-related errors, 50% had key information omissions, and 20% provided vague or incomplete advice. Proportions were based on independent coding by two reviewers (disagreement rate <8%, resolved by a third reviewer).

Safety assessment

Three clinical healthcare professionals independently screened all 315 responses. No direct suggestions for dangerous behaviors or incorrect medication dosages were found. However, approximately 15% of the responses carried an indirect risk of misleading information, mainly manifested as: underestimating the severity of the condition (such as describing typical sleep apnea symptoms as “can be observed first”, providing outdated or incomplete medical information, and using absolute assertions to rule out certain possibilities. We did not formally conduct a demographic bias assessment; therefore, no conclusion can be drawn regarding the absence of gender-, race-, or ethnicity-related bias. However, reviewers noted that descriptions of “adenoid facies” were generally based on broad clinical characteristics and rarely addressed population-level variation. This observation suggests that future studies should include structured bias assessment using demographically diverse prompts and evaluation criteria.

Response consistency assessment

To evaluate output stability, six questions, representing approximately 10% of the final question set, were randomly selected and resubmitted to each model. For each of the 30 question–model combinations, three independent outputs were generated, resulting in 90 repeated outputs in total. Most repeated outputs showed only minor wording differences, and no semantic-level contradictions were identified in this subset. However, because only 10% of the question set was retested, these findings should be interpreted as preliminary evidence of short-term response consistency rather than definitive proof of model stability.

Discussion

We evaluated five LLM-based chatbot products -- ChatGPT, DeepSeek, Gemini, Perplexity, and Copilot -- on their ability to generate reliable and readable information about adenoid hypertrophy. Overall, none of the evaluated chatbot products achieved both high information quality and patient-friendly readability, although clear differences across products were observed. This is a comparison of publicly accessible products under default user-facing settings, not a controlled laboratory benchmark. Given rapid model updates and non-controlled backend changes, these findings reflect a time-specific product-level evaluation rather than intrinsic superiority of any particular LLM. It should be noted that Perplexity Pro had real-time web search enabled by default under its standard configuration, whereas the other products were used with their default settings without additional retrieval features explicitly activated. This asymmetry is inherent to a real-world product comparison and does not allow for conclusions about intrinsic model performance.

Variation in model performance

The reliability assessment revealed significant differences in the quality of adenoid hypertrophy information across products. Perplexity achieved the highest scores on DISCERN (41.98 ± 1.87) and EQIP (58.40 ± 3.67), indicating advantages in information structure, evidence support, and content breadth. A possible explanation is its default real-time web search feature. Thus, its higher scores may reflect access to current online information rather than superior intrinsic medical reasoning. These two factors cannot be separated; readers should consider this when interpreting the comparisons.

Copilot ranked second, particularly on the JAMA score (1.0 ± 0.0), matching Perplexity as the only products to receive any points. This indicates that Copilot provided source annotations to a limited extent, which is a crucial step for establishing information credibility. In contrast, ChatGPT and DeepSeek scored low across several reliability indicators. On the JAMA assessment, they received zero points, reflecting a lack of author information, source citations, timeliness statements, and conflict of interest disclosures. This lack of transparency limits their utility for health-related information.

Higher DISCERN and EQIP scores indicate that information was better structured and more complete. Factual correctness was assessed separately against the predefined clinical reference set and was not inferred from DISCERN, EQIP, GQS, or JAMA scores alone.

The readability assessment revealed differences in another dimension. Gemini performed best across all readability indicators, with its text closest to the recommended sixth-grade reading level (FRES: 61.95 ± 9.64), indicating the most concise and understandable language. In contrast, Copilot generated the most difficult text, requiring a university-level reading comprehension. Gemini was easier to read than the other products. One possible explanation is that its training data included a higher proportion of patient-friendly materials; however, this explanation is speculative and was not tested in this study.

Perplexity, which had the highest reliability scores, showed only moderate readability. We observed an imbalance -- products that were more reliable tended to be harder to read, and vice versa. This pattern was not tested statistically.

Influence of model architecture and training data

Because most products were closed-source, we could not determine the extent to which model architecture, retrieval capability, training data composition, or interface-level design contributed to the observed differences.

Information sources and transparency likely contributed to the observed differences. The higher reliability scores of Perplexity and Copilot may stem from stronger external information retrieval or citation mechanisms in their system design. ³⁰ ChatGPT and DeepSeek, which rely on static knowledge bases without enforced source citations, scored poorly on the JAMA transparency criteria.

Language style also appeared to differ across products. Because most evaluated products are closed-source, the mechanisms underlying the observed differences cannot be determined. Differences may reflect retrieval or citation features, interface-level defaults, response style, update timing, or other unobserved product-level factors. This study was not designed to isolate these mechanisms; therefore, mechanistic explanations should be considered speculative.

Third, domain-specific knowledge integration also appeared to differ. Although all products were trained on general corpora, their performance in handling medical information for a relatively specific disease like adenoid hypertrophy showed different knowledge integration and expression capabilities. This suggests that general-purpose models, without specific medical domain fine-tuning, may exhibit variability in knowledge accuracy and professional expression. These explanations remain hypotheses and were not directly tested in this study.

Comparison with similar studies

The results of this study are largely consistent with the conclusions of recent studies.^31–33 However, most prior studies tested different diseases (e.g., cancer or urology) or used different prompts, so direct comparisons should be made with caution. At the same time, poor readability is a common problem with AI-generated medical content, ³⁴ and the reading level of most model-generated text far exceeds the average level of public health literacy. This is consistent with the present observation that none of the products reached the “easy” level on the FRES scale. ³⁵ These common issues highlight the core challenges currently faced by LLMs in medical health communication ³⁶ : how to ensure the accuracy and verifiability of information while expressing it in plain and understandable language. ³⁷

What this means for practice

No product performed well enough to be trusted independently. Even Perplexity, the top performer, scored only “Fair” on the DISCERN reliability scale. Parents should not rely on any of these products for treatment decisions.

Different products have different strengths. Perplexity gave the most reliable answers. Gemini was the easiest to read. Copilot and Perplexity at least attempted to cite sources.

A major caveat is that most products are closed-source. The training data cannot be inspected, and it is possible that the test questions contaminated the training data, which would inflate performance estimates.

In addition, these products update constantly. The present results represent a snapshot from late 2025 to early 2026 -- not a permanent rating.

Therefore, these findings should be interpreted as product-level observations under default user-facing settings rather than evidence of intrinsic superiority of any underlying base model. These tools may support preliminary information seeking, but treatment-related information should be verified by qualified healthcare professionals.

Importantly, this study provides content-level validation against standardized information-quality tools and reference clinical materials, but it does not constitute clinical validation of any chatbot for diagnosis, treatment decision-making, or patient counseling. Future validation should include patient comprehension, clinician-supervised use, decision outcomes, and potential harms in real-world interactions.

Strengths and limitations

This study has several strengths. It is the first to systematically evaluate multiple mainstream LLMs for a specific pediatric disease (adenoid hypertrophy). It employed multi-dimensional standardized tools for reliability (DISCERN, EQIP, JAMA, GQS) and readability (six indices), providing a comprehensive performance profile.

Another strength is the consistency among the evaluators. The ICC for DISCERN was 0.879, for EQIP was 0.811, and for GQS was 0.882, all exceeding 0.60, indicating high reliability of the scoring process. This high consistency can be attributed to: (1) detailed scoring guidelines; (2) all assessors were attending physicians and head nurses with over 10 years of clinical experience; (3) unified training before assessment. Compared with previous studies evaluating AI-generated health information, the consistency among assessors in this study was higher. For example, Gibson et al. ³² reported an ICC of 0.65 for DISCERN, whereas the present study reached 0.879. This difference may stem from the stricter assessor training process and more explicit scoring criteria used here.

However, high inter-rater agreement does not guarantee the accuracy of the scores -- it only demonstrates consistency among raters.

This study also has several limitations.

First, model and timing limitations. For Perplexity Pro and Copilot, the web interfaces did not disclose which base model they were running. Although no additional instruction requiring comprehensive or structured responses was added, the natural wording and length of patient-sourced questions may still have influenced response length, depth, and readability. Therefore, the readability findings should be interpreted as reflecting responses to naturally phrased FAQ-style questions rather than responses to a fully standardized readability-controlled prompt. Only one response per query was analyzed (LLM outputs can vary; a consistency check on 10% of questions found no contradictions, but the sample size was small). The assessment was single-time (October 2025 – January 2026). Models update rapidly, so the results represent a snapshot. Default web interface settings were used (parameters not adjustable), so outputs may contain random variation -- this reflects real-world usage. ¹³ An asymmetric setup is also noted: Perplexity Pro had real-time web search enabled by default. Whether other models also have access to real-time information under different configurations cannot be confirmed. Thus, Perplexity’s higher reliability scores may partly reflect its access to current online information. The findings reflect product-level performance as users would experience it with default settings, not a pure model comparison.

Furthermore, model parameters such as temperature, Top-P, and backend versioning could not be controlled or retrieved, as we used only the default web interfaces without API access. This lack of parameter control inherently limits the reproducibility of our findings and should be considered when interpreting inter-model comparisons. Any replication attempt may yield different outputs even for the same product, given potential backend updates or stochastic generation settings.

Second, limitations of the assessment scenario. The models were tested in a standardized, single-round questioning environment, which differs from the complex, nonlinear, and ambiguous conversations that patients or parents may have with AI in real-world settings. Therefore, the results may overestimate model robustness and practicality in real interaction environments. Potential model biases are also acknowledged -- all closed-source models lack training data transparency, and the question sources (Chinese and English only) may not represent all populations. Language and contextual considerations. Although the questions were submitted in English to standardize model input and facilitate cross-model comparison, part of the question sources and reference materials reflected Chinese clinical and caregiver contexts. The clinical reference set included an international guideline and systematic reviews, but also regional consensus documents where applicable. This asymmetry may introduce contextual bias: some model responses may appear incomplete or discordant when judged against region-specific recommendations, while some locally relevant issues may not be fully generalizable to other healthcare systems. Therefore, our findings should be interpreted as an evaluation of English-language chatbot responses against a multi-source clinical reference set, rather than as a universally generalizable assessment across all languages, regions, or practice settings.

Third, evaluator subjectivity. Although standardized tools and blinding were used, judgments of “quality” and “harm” still partially relied on the subjective experience of clinical assessors. Limitations of readability metrics. Standard readability formulas rely on surface-level features (word length, sentence length) and may not fully capture the comprehensibility of AI-generated medical text, particularly when clinically necessary terminology is used. Thus, our readability findings represent one dimension of accessibility rather than a complete measure of patient comprehension. Future studies should consider patient comprehension testing alongside formula-based metrics.

Future recommendations

Based on the above findings, the following recommendations are proposed.

For model developers: prioritize integrating mandatory source reference functions for models used in medical and health applications, and establish enhanced retrieval links with authoritative and timely medical databases (e.g., UpToDate, clinical guidelines) to enhance information timeliness and credibility. ³⁸ A readability control module should also be developed to allow users or systems to dynamically adjust the language complexity of generated text based on the health literacy level of the target audience. ³⁹

For future research: longitudinal studies are needed to track and evaluate the impact of model updates on performance. A dedicated assessment framework for AI-generated medical information should be developed, going beyond static text evaluation to include dialogue interaction quality, personalized recommendation appropriateness, and potential bias detection. Finally, research on how to effectively incorporate patient feedback into the model optimization cycle is crucial for achieving a truly user-centered AI health assistant.

Conclusion

In summary, current LLMs show potential for generating health information about adenoid hypertrophy, but none performed well on both reliability and readability. In this snapshot evaluation, Perplexity scored highest on reliability; Gemini was easiest to read. Our results suggest an imbalance -- more reliable models tended to be harder to read, and vice versa. Future work should focus on improving source transparency and simplifying language. For now, these tools may help with background information, but they cannot replace professional medical advice.

ORCID iD

Xiaoming Qian https://orcid.org/0009-0009-2212-6003

Ethical considerations

Not applicable. This research did not involve human participants, human data, or human tissue requiring approval from an Ethics Committee or Institutional Review Board.

Data Availability Statement

The complete list of the 63 submitted questions, date-stamped raw chatbot outputs from all five evaluated chatbot products, scoring sheets, and the R analysis code are provided as Supplementary Files S1–S5. No patient-level data were used in this study.*