Are AI chatbots ready for chikungunya public education? Evidence on validity, reliability, and readability

Abstract

Background

Chikungunya continues to expand geographically, driving demand for trustworthy, easy-to-read public guidance. Conversational AI systems are increasingly used for health information, yet their medical validity, reliability, and readability remain uneven.

Methods

We evaluated four widely used chatbots (ChatGPT, Claude, DeepSeek, Gemini) on two task sets: (1) validity on a 50-item single-answer MCQ dataset about Chikungunya; and (2) reliability and readability on 13 core public-education questions derived from Google Trends “topics” and clinician. Reliability was scored with DISCERN, EQIP, GQS, and JAMA benchmarks by clinician raters; readability used ARI, CL, FKGL, FRES, GFI, and SMOG.

Results

Across three independent runs, Deepseek-V3.2 achieved the highest MCQ accuracy (86.7%); other models ranged near 72–78% accuracy. In paired brand-line comparisons, newer iterations outperformed predecessors: ChatGPT-5 vs. ChatGPT-4o showed a 5.3-percentage-point gain, and Deepseek-V3.2 vs. Deepseek-R1 showed an 8.7-point gain in accuracy. Reliability (DISCERN, EQIP, GQS, and JAMA) differed significantly across models (p < 0.05). Pairwise tests favored Deepseek-V3.2 on most instruments, though absolute JAMA scores were low for all systems, indicating limited transparency signals. Readability (ARI, CL, FKGL, FRES, GFI, and SMOG) also differed significantly (p < 0.05). Deepseek-V3.2 produced the easiest text on average, Gemini the most complex, and no model met the sixth-grade benchmark.

Conclusion

Under the benchmark conditions of this study, newer chatbot iterations showed higher factual performance, and DeepSeek-V3.2 achieved the strongest task-specific answer-quality metrics. However, these findings should not be interpreted as evidence of readiness for public-health deployment or as an endorsement of any platform. Transparency remained limited, readability exceeded recommended public-facing levels, and platform-level concerns (including privacy, data governance, content moderation, and sociopolitical acceptability) remain essential considerations. Chatbots should therefore be treated as supervised adjuncts to, not substitutes for, authoritative public-health communication.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12889-026-27713-6.

Introduction

Chikungunya is an arboviral illness caused by chikungunya virus (CHIKV), an alphavirus transmitted primarily by Aedes aegypti and Aedes albopictus mosquitoes. Patients typically present with acute high fever, severe arthralgia/arthritis, rash, and fatigue; although mortality is rare, the joint manifestations can be protracted and debilitating, creating substantial individual and public-health burdens [1–3]. Periodic resurgences continue to occur across regions where competent vectors are established, underscoring the need for accurate, timely, and accessible public education on prevention, diagnosis, and care. Recent reports indicate a marked expansion in 2025, with unprecedented outbreaks on Indian Ocean islands, ongoing activity in South Asia and East Africa, and sustained local transmission in temperate regions, including China (~ 8,000 cases in Guangdong), France (383), and Italy (167) [4–6]. This evolving footprint keeps chikungunya a continuing threat to travelers and residents alike. In this setting, high-quality public information is especially important because it may influence mosquito-avoidance behaviors, early symptom recognition, appropriate care-seeking, and public understanding of how chikungunya differs from other febrile mosquito-borne illnesses.

At the same time, the public increasingly turns to the internet and, more recently, to conversational AI systems for health information. AI chatbots can synthesize large corpora into fluent answers, potentially improving access to health guidance. These systems are typically powered by large language models (LLMs), which are capable of generating coherent, contextually plausible responses by learning statistical patterns from large-scale text corpora. LLM-based chatbots may be especially useful for summarizing information and answering common health questions, but they are also known to produce confident inaccuracies, uneven sourcing, limited transparency, and outputs that are not always well calibrated to the literacy needs of lay users. However, emerging evaluations show substantial variability in medical accuracy, reliability, and readability across models and use cases, raising concerns about misinformation, lack of sourcing/transparency, and the readability gap for lay audiences [7–10]. These concerns are particularly salient for chikungunya because public misunderstanding may have direct consequences for prevention and early recognition. Inaccurate or unclear information may weaken mosquito-bite prevention behaviors, delay recognition of clinically important symptoms, obscure distinctions from related arboviral infections such as dengue, and lead to inappropriate self-management or delayed medical evaluation. For these reasons, chikungunya represents a meaningful and policy-relevant test case for evaluating AI chatbot performance in public education. Although a growing literature has evaluated chatbot performance across multiple medical domains, less is known about how these systems perform in disease-specific public-education scenarios where prevention, early recognition, and outbreak communication are especially important.

A key barrier to effective public communication is readability. U.S. professional bodies recommend that patient-facing materials be written near the sixth-grade level, yet many online health resources exceed this threshold, limiting comprehension and actionability [11, 12]. Standard readability metrics include Automated Readability Index (ARI), Coleman-Liau Index (CL), Flesch-Kincaid Grade Level (FKGL), Flesch Reading Ease (FRES), Gunning Fog Index (GFI), and SMOG, which provide complementary estimates of linguistic complexity and reading grade level. In parallel, widely used instruments such as DISCERN, Ensuring Quality Information for Patients (EQIP), the Global Quality Scale (GQS), and the JAMA benchmark criteria enable appraisal of reliability, treatment-information quality, overall usefulness, and transparency (authorship, attribution, date, and ownership) [13–16].

Despite the prominence of chikungunya in the global health agenda, little is known about how contemporary AI chatbots perform when answering the specific questions that the public most often asks about this disease. To align our evaluation with real-world information needs, we used Google Trends “topics,” which aggregate conceptually equivalent queries across languages, to identify high-salience, patient-facing questions about chikungunya over the past five years [17–19]. We then assessed the validity (MCQ accuracy), reliability (DISCERN, EQIP, GQS, JAMA), and readability (ARI, CL, FKGL, FRES, GFI, and SMOG) of answers produced by four widely used chatbots (ChatGPT, DeepSeek, Gemini, Claude), which were selected because they were broadly accessible, widely used, and representative of major contemporary general-purpose AI systems available to the public during the study period. Unlike prior chatbot evaluations that have often focused on single dimensions, broad medical domains, or generic health prompts, our study integrates validity, reliability, and readability within a single framework anchored to real-world, disease-specific public-information demand during a period of renewed chikungunya activity. Our study aims to provide an end to end, instrument-based comparison that can inform safer deployment of AI systems for infectious-disease public education.

Materials and methods

Study design

This study evaluated the validity, reliability, and readability of answers generated by four widely used AI chatbots (ChatGPT, DeepSeek, Gemini, and Claude) on key topics related to chikungunya. These systems were selected for their broad adoption, accessibility, and capability to generate healthy and medicine-focused responses. The study consisted of two complementary components with distinct objectives: an MCQ-based validity assessment and a longer-form public-education assessment of reliability and readability. A schematic of the study workflow is provided in Fig. 1.

Fig. 1.

Workflow of study design and evaluation process

Chatbot selection and task allocation

For the MCQ-based validity assessment, we included six model versions: ChatGPT-5, ChatGPT-4o, DeepSeek-V3.2, DeepSeek-R1, Gemini 2.5 Pro, and Claude Sonnet 4.5. Two generations were included within the ChatGPT and DeepSeek product families to explore whether model iteration was associated with improved factual accuracy.

For the longer-form public-education assessment, our objective was different: rather than comparing multiple generations within the same product family, we aimed to provide a balanced cross-platform comparison of the most current and representative versions from each chatbot family. Therefore, we included ChatGPT-5, DeepSeek-V3.2, Gemini 2.5 Pro, and Claude Sonnet 4.5 in this phase, while ChatGPT-4o and DeepSeek-R1 were not evaluated further in the reliability and readability analyses in order to avoid redundant within-family comparisons.

Data collection and analysis of validity

We compiled a 50-item, single-answer multiple-choice (MCQ) dataset on chikungunya from two publicly available clinical question-answering datasets (MedQA and MedMCQA) as well as Chinese national medical examination items [20, 21] (Supplementary materials). Item selection was designed for comprehensive topical coverage of real-world information needs, spanning virology (pathogen), epidemiology, clinical manifestations, diagnosis and differential diagnosis, treatment, and prevention.

The final MCQ dataset comprised 10 items originally written in English and 40 items originally written in Chinese. All items had a single, prespecified correct option based on the source key. To minimize ambiguity introduced by secondary translation, each question was presented to the chatbots in its original source language, and identical wording and answer choices were used across all models. This study was not designed as a formal cross-lingual comparison of chatbot performance; rather, the use of source-language prompts was intended to preserve the fidelity of the original items.

To evaluate MCQ performance, each of the 50 items was presented to each AI chatbot (ChatGPT-5, ChatGPT-4o, Deepseek-V3.2, Deepseek-R1, Gemini 2.5 Pro, Claude Sonnet 4.5) under a standardized prompting protocol, in the same order and with no supplemental context, hyperlinks, or images. To account for stochastic variability, each item was asked three independent times per model; before each attempt, the conversation history was cleared to prevent memory effects. The primary endpoint was accuracy, defined as the proportion of correctly answered items.

Core public-education questions on chikungunya

To ensure comprehensive terminology coverage, we first retrieved the Medical Subject Headings (MeSH) entry for “Chikungunya Fever” and its entry terms: Chikungunya Fever; Chikungunya Fevers; Fever, Chikungunya; Fevers, Chikungunya; Chikungunya Virus Infection; Chikungunya Virus Infections; Infection, Chikungunya Virus; Infections, Chikungunya Virus. These terms were then used to query Google Trends. When entered, Google Trends consolidated them under the unified topic “Chikungunya Fever” which aggregates conceptually equivalent search terms across languages and is designed to capture the most relevant interest signals [22]. Using the topic rather than a single keyword allowed us to capture global, multilingual search behavior. We extracted the 25 top related queries associated with the “Chikungunya fever” topic worldwide over the most recent five-year window (2020–2025), as Google Trends reflects real-world information seeking and public interest over time [17, 18]. Data were accessed in October 2025 (Table S1 for the exported query list).

Because Google Trends reflects relative search interest rather than actual care-seeking behavior, clinical urgency, informational accuracy, or public-health priority, an infectious-disease clinician panel reviewed these queries alongside real-world patient encounters and distilled a set of 13 core, patient-facing questions that reflect public information needs and clinical safety priorities (Table 1). Selection criteria prioritized: (i) high public salience on Google Trends; (ii) clinical importance and safety relevance; (iii) clarity for non-expert readers; and (iv) non-overlap (merging semantically redundant or near-duplicate queries). The finalized questions explicitly cover: (1) definition (Q1), (2) transmission (Q2), (3) prevalence/endemicity (Q3), (4) clinical manifestations (Q4), (5) diagnosis (Q5), (6) treatment (Q6), (7) individual-level prevention (Q7), (8) population-level transmission control (Q8), (9) vaccine availability (Q9), (10) travel precautions (Q10), and (11–13) differential diagnosis versus dengue, Zika, and malaria (Q11–Q13). All 13 public-education questions were entered in English using identical wording across models, and the chatbot responses were collected for further reliability and readability analyses (Figure S1).

Table 1.

Thirteen core public-education questions on chikungunya

Number	Question
1	What is chikungunya fever?
2	How is chikungunya virus transmitted?
3	Where is chikungunya prevalent/endemic?
4	What are the clinical symptoms of chikungunya?
5	How is chikungunya diagnosed?
6	How is chikungunya treated?
7	How can chikungunya be prevented?
8	How can transmission of chikungunya virus be controlled?
9	Is there a vaccine against chikungunya?
10	What precautions should travelers take regarding chikungunya?
11	How can chikungunya be distinguished from dengue fever?
12	How can chikungunya be distinguished from Zika virus disease?
13	How can chikungunya be distinguished from malaria?

Analysis of reliability

We appraised the reliability of chatbot outputs using four established instruments: DISCERN, Ensuring Quality Information for Patients (EQIP), the Global Quality Scale (GQS), and the JAMA benchmark criteria. The core purpose, score range, and interpretation of these instruments are summarized in Table S2. Briefly, DISCERN evaluates the quality of written health information with an emphasis on treatment-related content [16, 23]; EQIP assesses clarity, completeness, and writing quality of patient-facing materials [15, 24]; GQS provides an overall user-oriented appraisal of health information quality [25]; and the JAMA benchmark criteria evaluate four transparency domains: authorship, attribution, currency, and disclosure/ownership [14].

All evaluations (DISCERN, EQIP, GQS, and JAMA) were performed independently by two infectious-disease clinicians who were blinded to chatbot identity during scoring. Any discrepancies were addressed through consensus discussion. If consensus could not be reached, a third senior clinician with 20 years of clinical experience adjudicated the rating, and the adjudicator’s score was taken as final.

Inter-rater reliability was quantified using the initial independent ratings before consensus adjudication. For total scores (DISCERN, EQIP, GQS, and JAMA), we calculated the intraclass correlation coefficient using a two-way random-effects model with absolute agreement for single measurements (ICC[A,1]). For the four binary JAMA benchmark items, agreement was additionally assessed using Cohen’s kappa. When an item showed no variability across targets, kappa was not estimable; in such cases, observed agreement was reported instead. Inter-rater reliability of the initial ratings was high across instruments: DISCERN, ICC = 0.918 (95% CI 0.859–0.952); EQIP, ICC = 0.942 (95% CI 0.899–0.967); GQS, ICC = 0.921 (95% CI 0.867–0.954); and JAMA total score, ICC = 1.000 (95% CI 1.000–1.000). For JAMA items with no variability, Cohen’s kappa was not estimable, but observed agreement was 100%.

Analysis of readability

We quantified the readability of chatbot responses using six widely cited indices: Automated Readability Index (ARI), Coleman-Liau Index (CL), Flesch-Kincaid Grade Level (FKGL), Flesch Reading Ease (FRES), Gunning Fog Index (GFI), and SMOG. Because these indices are primarily validated for English-language text, readability analyses were performed only on the English responses generated for the 13 public-education questions. Scores were computed with an online calculator (https://readabilityformulas.com/). In these metrics, FRES ranges from 0 to 100, with higher values indicating easier text, whereas FKGL, ARI, CL, GFI, and SMOG estimate the U.S. school grade level required to comprehend the material. This framework allowed direct comparison of linguistic complexity across chatbots and appraisal of their suitability for public-facing health communication. To benchmark accessibility, we compared scores against the sixth-grade readability target recommended by the AMA and NIH. Consistent with these guidelines, we considered FRES ≥ 80.0 as acceptable and required < 6 for each of the other five grade-level indices.

Statistical analysis

For each chatbot, responses to the 50 MCQs were recorded as correct or incorrect, and accuracy and error rates were calculated based on three independent trials per item. To enable cross-model comparisons, we summarized the percentage of correct answers across all items for each model and visualized results with scatter plots. Group differences in MCQ accuracy were assessed using one-way ANOVA, with p < 0.05 denoting statistical significance.

For reliability (DISCERN, EQIP, GQS, JAMA) and readability (ARI, CL, FKGL, FRES, GFI, SMOG) outcomes, we reported means and standard deviations for each program. Distributional assumptions were evaluated using the Shapiro-Wilk test. As many score distributions were non-normal, between-model comparisons for reliability and readability were conducted using the Kruskal-Wallis test.

To evaluate whether chatbot outputs met recommended 6th-grade readability targets (AMA/NIH), we used the Wilcoxon signed-rank test, comparing each metric against its reference threshold (FRES ≥ 80.0 and < 6 for the remaining five indices). Within each model we performed six tests (one per index) and controlled family-wise error using the Holm method.

Inter-rater reliability statistics are reported as ICCs with 95% confidence intervals and, where applicable, Cohen’s kappa values. All analyses were performed in R (version 4.5.0). Two-sided p-values < 0.05 were considered statistically significant. We denote significance as * p < 0.05, ** p < 0.01, and *** p < 0.001.

Results

Validity of AI chatbots on chikungunya multiple-choice questions

Across three independent runs of the 50-item MCQ set, performance differed across models (Fig. 2A-B). DeepSeek-V3.2 achieved the highest mean correct rate (86.7%±1.2%), whereas ChatGPT-4o showed the lowest accuracy (72.0%±2.0%). The remaining models clustered in an intermediate range with broadly similar performance. Full model-specific values are shown in Fig. 2A-B.

Fig. 2.

Validity of AI chatbots on chikungunya multiple-choice questions. A Correct rate (%) by model. Bars show mean correct rate across three independent runs of a 50-item MCQ set for each chatbot (ChatGPT-5, ChatGPT-4o, Deepseek-V3.2, Deepseek-R1, Gemini 2.5 Pro, Claude Sonnet 4.5). Error bars denote SD across runs. B Error rate (%) by model. Bars show mean incorrect rate with SD. * p < 0.05, ** p < 0.01, *** p < 0.001, **** p < 0.0001. C Item-level correctness across runs. For each of the 50 questions, colored dots show the number of correct responses obtained by each model across the three runs

Run-to-run variability was small for all systems (SD ≤ 2.3% points) and absent for Deepseek-R1, indicating stable outputs under repeated querying. Item-level distributions (Fig. 2C) further illustrate model differences across questions. Pairwise contrasts are annotated in Fig. 2A-B.

To address the potential influence of language composition, we performed a post hoc language-stratified sensitivity analysis of MCQ accuracy (Table S3). Because each item was tested across three independent runs, the denominators were 120 for the 40 Chinese-language MCQs, 30 for the 10 English-language MCQs, and 150 overall. All models showed higher accuracy on the English-language subset than on the Chinese-language subset. DeepSeek-V3.2 achieved the highest accuracy on the Chinese-language subset, with 83.3% correct answers (100/120), compared with 68.3%–73.3% for the other models. On the English-language subset, DeepSeek-V3.2 and DeepSeek-R1 both achieved 100.0% accuracy (30/30), followed by ChatGPT-5 at 96.7% (29/30). These findings suggest that DeepSeek-V3.2’s overall advantage was not restricted to Chinese-language items. However, because Chinese-language MCQs accounted for 80% of the MCQ dataset, the overall ranking was more strongly influenced by performance on Chinese-language items than by performance on English-language items.

To further examine error patterns, we reviewed MCQ items that were answered incorrectly across all models and repeated trials. Four items were never answered correctly by any of the six chatbots. These items concerned: (1) the hallmark clinical manifestation of chikungunya fever (question 6), (2) vector-control thresholds used to define the end of an outbreak (question 25), (3) distinctions between suspected-case criteria and confirmatory diagnostic criteria (question 27), and (4) management decisions in the setting of chikungunya-associated myocarditis or heart failure (question 47). These findings suggest that chatbot errors were not entirely random, but instead clustered in domains requiring exact guideline-based knowledge, exclusion-based diagnostic reasoning, or context-specific management decisions.

In paired brand-line comparisons, newer iterations outperformed their predecessors, supporting the view that product iteration improves answer correctness. ChatGPT-5 averaged 77.3% accuracy compared with 72.0% for ChatGPT-4o, an absolute increase of 5.3% points and a 7.4% relative improvement; the error rate decreased from 28.0% to 22.7%, an absolute reduction of 5.3% points (a 18.9% relative reduction). Similarly, Deepseek-V3.2 achieved 86.7% accuracy versus 78.0% for Deepseek-R1, an absolute gain of 8.7% points (a 11.2% relative improvement) and an error-rate reduction from 22.0% to 13.3% (an absolute reduction of 8.7% points, corresponding to a 39.5% relative reduction). These consistent improvements align with the groupwise statistics and pairwise annotations in Fig. 2A-B, indicating that successive releases deliver measurably better MCQ performance.

Reliability analysis

Reliability scores differed significantly across chatbots on all four instruments (Table 2; Fig. 3A-D). Overall, DeepSeek-V3.2 achieved the highest mean scores across DISCERN, EQIP, GQS, and JAMA, with statistically significant advantages in multiple pairwise comparisons (Fig. 3). Despite these differences, absolute JAMA scores remained low across all models, indicating generally limited transparency markers (authorship, attribution, date, and ownership) in the generated answers. In contrast, DISCERN, EQIP, and GQS captured broader aspects of informational quality and usability, suggesting that relatively acceptable content presentation did not necessarily coincide with adequate transparency disclosure.

Table 2.

Reliability scores across AI chatbots

	DISCERN (mean ± SD)	EQIP (mean ± SD)	GQS (mean ± SD)	JAMA (mean ± SD)
ChatGPT-5	35.38 ± 7.09	58.08 ± 13.00	3.08 ± 0.49	0.31 ± 0.63
Claude Sonnet 4.5	34.00 ± 3.76	60.77 ± 7.60	3.31 ± 0.63	0.08 ± 0.28
DeepSeek -V3.2	42.23 ± 6.46	69.23 ± 11.15	3.85 ± 0.38	0.54 ± 0.52
Gemini 2.5 Pro	35.54 ± 1.94	65.00 ± 5.00	3.54 ± 0.52	0.08 ± 0.28
P-value	0.004	0.036	0.004	0.021

Fig. 3.

Reliability of AI chatbot answers on 13 core public-education questions. A–D Mean reliability scores (bars) with SD (error bars) for ChatGPT-5, Claude Sonnet 4.5, Deepseek-V3.2, and Gemini 2.5 Pro on DISCERN (A), EQIP (B), GQS (C), and JAMA (D). Significance brackets indicate between-model contrasts: * p < 0.05, ** p < 0.01, *** p < 0.001

Collectively, these findings indicate that DeepSeek-V3.2 achieved the highest reliability scores under the predefined scoring instruments and benchmark conditions used in this study, while the other systems showed intermediate performance patterns consistent with Fig. 3. These score differences should be interpreted as task-specific output-quality differences rather than as evidence of broader platform suitability for public-health deployment.

Item-level heatmaps (Fig. 4A) show a consistent pattern across the 13 public-education questions: Deepseek-V3.2 exhibits warmer tiles (higher scores) on most items for DISCERN, EQIP, and GQS, whereas ChatGPT-5 and Claude display cooler tiles, with Gemini in between. Non-zero JAMA tiles are sparse and uneven across models, reinforcing the low absolute JAMA values observed in Table 2; Fig. 3. These per-question views align with the omnibus and pairwise statistics, indicating that Deepseek-V3.2 not only leads on average but does so across the majority of items rather than being driven by a small subset of questions.

Fig. 4.

Item-level heatmaps for reliability and readability across the 13 core public-education questions. A Reliability. For each question (rows 1–13), tiles display the clinician-assigned scores for DISCERN, EQIP, GQS, and JAMA for four chatbots (x-axis order within each block: ChatGPT-5, Claude Sonnet 4.5, Deepseek-V3.2, Gemini 2.5 Pro). Colors are mapped to the native range of each instrument as shown by the side color bars; warmer colors indicate higher values. B Readability. For the same questions, tiles show ARI, GFI, FKGL, CL, SMOG, and FRES computed from each chatbot’s answer (x-axis order as above). Warmer colors indicate higher numeric values. Note the interpretation: for ARI, CL, FKGL, GFI, and SMOG, higher values reflect greater reading grade level and lower accessibility; for FRES, higher values reflect easier text (0–100 scale). The heatmaps visualize per-item patterns that underlie the aggregate results in Figs. 3 and 5

Readability analysis

Readability differed significantly across chatbots on all six indices (Table 3; Fig. 5A-F). Overall, DeepSeek-V3.2 produced the most accessible text, with lower grade-level scores and higher FRES values than the other models, whereas Gemini 2.5 Pro generated the most complex text. ChatGPT-5 and Claude Sonnet 4.5 showed intermediate readability profiles. Pairwise comparisons were directionally consistent with these overall patterns (Fig. 5).

Table 3.

Readability scores across AI chatbots

Program	ARI (mean ± SD)	CL (mean ± SD)	FKGL (mean ± SD)	FRES (mean ± SD)	GFI (mean ± SD)	SMOG (mean ± SD)
ChatGPT-5	11.52 ± 1.45	14.03 ± 1.71	10.42 ± 1.32	40.77 ± 9.15	12.19 ± 1.99	9.50 ± 1.23
Claude Sonnet 4.5	12.64 ± 2.25	15.33 ± 2.11	11.21 ± 1.8	34.92 ± 10.32	12.65 ± 1.96	9.71 ± 1.61
DeepSeek -V3.2	10.45 ± 1.23	12.18 ± 1.05	9.40 ± 1.13	50.15 ± 5.74	11.58 ± 0.96	9.40 ± 1.02
Gemini 2.5 Pro	13.93 ± 2.27	14.36 ± 1.53	12.69 ± 1.84	36.31 ± 8.31	13.39 ± 1.55	11.7 ± 1.65
6th grade level score	6	6	6	80–90	6	6

Fig. 5.

Readability of AI chatbot answers on the 13 core public-education questions. A–F Mean readability scores (bars) with SD (error bars) for ChatGPT-5, Claude Sonnet 4.5, Deepseek-V3.2, and Gemini 2.5 Pro across six indices: ARI (A), CL (B), FKGL (C), GFI (D), SMOG (E), and FRES (F). Scores are averaged over the 13 questions. Red dotted lines denote AMA/NIH sixth-grade targets: 6 for grade-level indices (ARI, CL, FKGL, GFI, SMOG) and 80 for FRES. For interpretation, lower values on ARI/CL/FKGL/GFI/SMOG and higher values on FRES indicate easier text. For interpretation, lower values on ARI, CL, FKGL, GFI, and SMOG indicate easier text, whereas higher values on FRES indicate easier text. Thus, bars farther from the reference lines indicate poorer alignment with recommended public-facing readability targets. Significance brackets indicate between-model contrasts: * p < 0.05, ** p < 0.01, *** p < 0.001

Benchmarking against the AMA/NIH sixth-grade target showed that none of the chatbot outputs met recommended readability levels: mean FRES values were well below 80 for all models, and mean ARI, CL, FKGL, GFI, and SMOG values all exceeded 6 (Fig. 5; Table 3). Importantly, the readability metrics were directionally consistent in showing limited accessibility: lower scores indicate easier text for ARI, CL, FKGL, GFI, and SMOG, whereas higher scores indicate easier text for FRES; across all six indices, none of the chatbots approached the recommended threshold for public-facing materials. For each chatbot, all six indices deviated significantly from the benchmark based on Wilcoxon signed-rank tests with Holm correction (p < 0.05).Item-level heatmaps (Fig. 4B) mirrored these aggregate findings, with Deepseek-V3.2 displaying comparatively lighter grade-level tiles and higher FRES across most questions.

Overall, while Deepseek-V3.2 yielded the most accessible prose among the four systems, the absolute readability of all chatbots remained above the sixth-grade level, indicating room for improvement for AI-generated public-facing health communication.

Discussion

In this multi-instrument evaluation of four widely used AI chatbots answering chikungunya questions, we observed three consistent patterns. First, model iteration was associated with higher multiple-choice accuracy. Newer versions within the same product family outperformed predecessors, and DeepSeek-V3.2 achieved the highest overall MCQ accuracy under the source-language benchmark conditions used in this study. Second, reliability scores favored DeepSeek-V3.2 on most pairwise contrasts, but absolute JAMA transparency scores were low across all models, indicating limited authorship, attribution, date, and ownership disclosure in generated answers. Third, readability fell short of public-health recommendations for sixth-grade materials across all six indices, despite relative advantages for DeepSeek-V3.2. Collectively, these results suggest that contemporary chatbots can produce mostly correct and comparatively high-scoring responses on chikungunya, but these task-specific metrics should not be equated with readiness for autonomous public-health deployment.

The 2025 surge in chikungunya, spanning Indian Ocean islands and extending to temperate settings with sustained local transmission in China, France, and Italy, has heightened reliance on rapidly accessible digital information for prevention and care [1, 4–6]. WHO summaries and contemporaneous news analyses underscore that chikungunya remains a dynamic global threat, amplifying the stakes for accurate, transparent, and readable public guidance [1, 6, 26–29]. Our study aligns evaluation targets with real-world search interests by deriving core questions from Google Trends “topics,” which aggregate conceptually equivalent queries across languages, an approach recommended in infodemiology work to reflect authentic information-seeking behavior [17–19]. At the same time, Google Trends captures relative search interest rather than actual care-seeking behavior, clinical urgency, disease burden, or the completeness of public-health information needs; for this reason, trend-derived queries in our study were further filtered and consolidated by an infectious-disease clinician panel.

Our findings of high but variable accuracy, uneven reliability, and inadequate readability mirror broader evaluations of AI health advice. These findings are consistent with the known strengths and limitations of LLM-based systems, which may perform well in fluent answer generation while remaining vulnerable in transparency, exact factual precision, and accessibility for lay audiences. More broadly, relatively strong answer fluency or overall correctness does not necessarily translate into adequate transparency or patient-facing readability. The present study extends this literature by showing that these recurring concerns remain evident in a disease-specific public-health communication setting centered on chikungunya, where prevention messaging, differential recognition, and outbreak-sensitive communication are especially important. A closer review of the MCQ errors also suggested that chatbot inaccuracies were not entirely random. In particular, four items were never answered correctly by any of the six models across repeated trials. These items were concentrated in domains requiring precise knowledge of guideline-based thresholds, formal case-definition criteria, and clinically specific management decisions. This pattern suggests that even when overall factual accuracy is relatively high, chatbots may remain vulnerable in areas that depend on exact public-health standards, exclusion-based diagnostic reasoning, or nuanced treatment judgment. Reviews and cross-sectional studies report heterogeneous medical correctness, with risks of confident errors and inconsistently safe recommendations, especially for nuanced clinical tasks [30–34]. Recent work also highlights gaps in sourcing and transparency and documents that safeguards against health misinformation can be bypassed or enforced inconsistently across models and contexts [32, 35, 36]. These patterns reinforce the need for instrument-based appraisal, for example, DISCERN, EQIP, GQS, JAMA, and readability benchmarking before deploying chatbots for infectious-disease education.

An important interpretive point is that stronger benchmark performance should not be equated with readiness for unrestricted public-health deployment. This distinction is especially important for DeepSeek. The present study evaluated task-specific answer quality; it did not evaluate political neutrality, privacy safeguards, data-transfer practices, legal exposure, or institutional acceptability. Public and regulatory discussions have raised several DeepSeek-specific concerns. DeepSeek’s own privacy policy states that collected personal information may be stored on servers located in the People’s Republic of China and that user input may be used to improve and train its technology. The Personal Information Protection Commission of Korea reported that DeepSeek transferred users’ personal data to servers in China and the United States without separate consent or adequate disclosure at the time of launch in Korea, and also noted concerns about user-entered data for AI model development and training. In addition, some governments and public institutions have restricted or banned DeepSeek on official devices because of security or data-governance concerns. These issues are not peripheral to public-health communication. In outbreak-related education, public trust, political neutrality, transparency, accountability, and regulatory acceptability are core conditions for safe implementation. Therefore, DeepSeek-V3.2’s stronger performance in this benchmark should be interpreted only as evidence of higher task-specific answer quality under the present testing conditions, not as evidence that the platform is suitable for unrestricted public-health deployment.

For public health communication, the persistent gap above sixth-grade readability indicates that chatbot outputs, even when largely correct, may remain hard to act on for lay audiences, echoing longstanding observations that many patient-education materials exceed recommended reading levels [11, 12]. This distinction matters because textual readability is not merely stylistic; it affects comprehension, trust, and the likelihood that users can translate information into appropriate preventive or care-seeking behavior. Integrating automatic grade-level controls, plain-language rewriting, and layered explanations such as short take-home messages with optional detail could improve accessibility. This conclusion was consistent across all six readability indices, despite their differing scoring directions, strengthening the interpretation that limited accessibility was a general rather than metric-specific finding.

The reliability instruments used in this study capture related but not identical dimensions of response quality, which is important for interpreting the results. DISCERN, EQIP, and GQS primarily reflect informational quality, completeness, structure, and overall usefulness from the perspective of patient-facing health communication, whereas the JAMA benchmark focuses more narrowly on transparency markers such as authorship, attribution, currency, and ownership. Accordingly, a response may appear relatively strong in informational quality while still scoring poorly on JAMA if it lacks explicit sourcing or update metadata. This distinction helps explain why some chatbot outputs in our study achieved moderate-to-higher DISCERN, EQIP, or GQS scores despite persistently low JAMA values across models.

From a public-health perspective, differences in chatbot performance should not be interpreted merely as technical ranking differences, but as indicators of how safely these systems may function in risk communication and public education. A chatbot that provides relatively accurate but poorly readable information may still fail to support preventive behavior or timely care-seeking if users cannot readily understand or act on the message. Likewise, a chatbot that appears informative but lacks transparency markers such as citations, update dates, or source ownership may be less suitable for outbreak communication, where trust, accountability, and timely updating are critical. These findings suggest that AI tools used in public-health education should be evaluated not only for correctness, but also for accessibility, transparency, and fitness for context-specific communication tasks. In practice, this may mean using chatbots as supervised support tools for message drafting, FAQ development, or preliminary information access, while reserving final public-facing communication for clinician-reviewed or agency-reviewed materials. Accordingly, the practical value of this study lies less in directing individual users toward a specific chatbot than in informing researchers, clinicians, public-health professionals, and platform stakeholders about how differences in chatbot performance may affect the safe design, supervision, and governance of AI-assisted health communication.

For model developers and platforms, our JAMA results argue for default transparency features in health answers: visible authorship/ownership, time-stamped updates, explicit citations, and disclosure of limitations. Given evidence that product iteration improves accuracy, development roadmaps should explicitly include reliability and readability targets, with regression alarms when updates degrade these metrics. For healthcare organizations and public-health agencies, chatbot use should therefore remain supervised: outputs may assist with preliminary information access, but they should still undergo human review and contextual adaptation before being used in formal patient-education settings.

Strengths of this study include an end-to-end framework spanning validity, reliability, and readability; question selection anchored to real-world information demand; and rater adjudication using predefined instruments with blinded clinician assessment and adjudication. Several limitations should also be emphasized. First, limitations in this study include a restricted question set of 13 core public questions and 50 MCQs that may not cover all clinical edge cases. Second, the evaluation reflects specific model versions and access conditions in 2025; because chatbot behavior, training updates, guardrails, and product interfaces evolve rapidly, these findings should be interpreted as time-specific rather than fixed platform characteristics, and relative model performance may change as systems are updated. Third, prompts were standardized and did not include iterative clarification or user-specific context. Although this design may underestimate best-case performance, it likely better approximates typical one-shot public use. Fourth, the study was not designed as a formal cross-lingual comparison. MCQs were administered in their source language (English or Chinese) to preserve item fidelity, whereas the public-education questions and readability analyses were standardized in English. To further contextualize the MCQ findings, we performed a post hoc language-stratified sensitivity analysis. DeepSeek-V3.2 performed best on the Chinese-language subset and was tied for best performance on the English-language subset. However, because 40 of the 50 MCQs were originally written in Chinese and only 10 were originally written in English, the study cannot fully separate disease-specific medical knowledge from language-specific model capability, Chinese-language training exposure, or familiarity with Chinese medical-examination formats. Therefore, the overall MCQ advantage of DeepSeek-V3.2 should be interpreted as performance under the source-language benchmark conditions used in this study, rather than as definitive evidence of language-independent superiority. As a result, the findings should not be generalized to multilingual performance or to readability in non-English settings. Future studies should use balanced multilingual datasets, parallel professionally translated versions, and prespecified cross-lingual analyses to more rigorously evaluate language-independent medical validity. Fifth, readability indices estimate textual difficulty but do not directly measure actual patient comprehension, recall, trust calibration, or behavioral uptake. In addition, chatbot outputs were evaluated in a static benchmarking framework rather than in real interactive user journeys, and user behavior may differ when follow-up questioning, selective attention, or misunderstanding occurs. Sixth, while Google Trends helped identify high-salience public questions, it reflects online search activity rather than actual care-seeking behavior and cannot fully represent the needs of vulnerable populations, low-literacy users, or individuals with limited internet access. Seventh, although chikungunya was selected in part because it shares clinically relevant overlap with dengue while retaining important distinctions, the present study did not include a parallel dengue evaluation. We therefore could not test whether chatbot outputs would maintain disease-specific differences when responding to closely related arboviral conditions. Future comparative work should assess whether AI-generated content appropriately differentiates chikungunya from dengue, Zika, and other overlapping mosquito-borne infections. Future work should examine source-citation features, the temporal stability of answers during outbreaks, cross-lingual performance, and co-design with health agencies to e

Amid a year of unprecedented chikungunya activity, leading AI chatbots were able to provide largely accurate and comparatively reliable answers to high-salience public questions, yet they fall short on transparency and readability thresholds central to safe public communication. Their near-term value may therefore lie more in supported or supervised educational use than in autonomous substitution for established public-health messaging. Instrument-anchored evaluation, coupled with readability controls and default transparency, will be essential for responsibly integrating chatbots into infectious-disease education.

Conclusion

In conclusion, across four widely used chatbots, newer model iterations achieved higher multiple-choice accuracy, with Deepseek-V3.2 showing the strongest overall validity and reliability under the conditions tested in this study. However, transparency signals were low on JAMA benchmarks and none of the systems met the sixth-grade readability target, limiting suitability for direct, unsupervised public-facing chikungunya education. Chatbot outputs should therefore be treated as adjuncts to, rather than replacements for, established chikungunya-related public-health communication and clinician-reviewed educational materials. Even when answer quality appears strong, decisions about their use in chikungunya public education should also take into account transparency, governance, privacy, and ethical considerations that fall outside performance metrics alone. Improving default transparency, adding clear citations and time stamps, and enforcing plain-language controls should be priorities for safer use of chatbots in chikungunya-related public education.

Authors’ contributions

Qi Zhuang, Jiadong Zhao, and Yang Liu: Conceptualization, Data curation, Formal analysis, Investigation, Software, Writing—original draft. Anguo Zhao: Formal analysis, Software, Writing—original draft. Rui Liang and Lei Peng: Funding acquisition, Methodology, Project administration, Resources, Supervision, Writing – review and editing. Rongkang Li: Funding acquisition, Methodology, Project administration, Resources, Supervision, Writing – review and editing. All authors have read and agreed to the published version of the manuscript.

Funding

This research did not receive any research funding.

Data availability

Data will be made available on request.

Declarations

Ethics approval and consent to participate

Not applicable.

Competing interests

The authors declare no competing interests.