AI-generated patient information leaflets for oral anticoagulants: quality, usability, readability, and reproducibility compared with FDA-referenced materials

Abstract

Introduction

Oral anticoagulants are frequently associated with preventable hospital admissions, and patient education is essential to their safe and effective use. Patients increasingly use large language models (LLMs) for medication information, yet few studies have compared AI-generated outputs with regulatory materials or examined oral anticoagulants, a class in which misunderstanding can cause bleeding or thrombosis.

Aim

To assess the informational quality, usability, readability, and output reproducibility of AI-generated patient information leaflets (PILs) for oral anticoagulants compared with FDA-referenced patient materials.

Method

PILs for five oral anticoagulants (warfarin, apixaban, dabigatran, rivaroxaban, and edoxaban) were generated by ChatGPT, Gemini, and DeepSeek using a standardised zero-shot prompt based on FDA labelling templates; FDA-approved leaflets served as the comparator. Materials were anonymised and brand-blinded. Three clinical pharmacists independently evaluated each PIL using the Patient Education Materials Assessment Tool for Print Materials (PEMAT-P) and a modified DISCERN (mDISCERN), which assess informational quality, understandability, and actionability, but not pharmacological accuracy or clinical safety. Readability was assessed using seven validated indices. Output reproducibility was examined within the same day and at Day 1, 14, and 28.

Results

ChatGPT produced PILs of informational quality similar to FDA-referenced materials, both achieving a median mDISCERN score of (41/65); DeepSeek (34/65) and Gemini (33/65) scored significantly lower than ChatGPT; Gemini also scored significantly lower than the FDA-referenced materials. Understandability was acceptable for all sources, whereas actionability was limited, including in FDA materials, with no leaflet providing a patient summary or decision-support tool. All materials exceeded the recommended sixth- to eighth-grade range on most indices, with FDA leaflets the most complex. Output was stable across generations, with no significant within-model differences and the largest mean difference of 0.23 points.

Conclusion

Among the evaluated models, ChatGPT most closely matched FDA-referenced materials in the assessed informational domains and demonstrated stable output over 28 days. Shared deficiencies across AI-generated and FDA leaflets suggest broader limitations in written health information. This comparability was limited to the assessed informational domains and does not establish equivalence in pharmacological accuracy, clinical correctness, or patient safety. AI-generated PILs may serve as clinician-reviewed supplementary materials and should not be used as standalone tools without independent professional review and verification of clinical content.

Supplementary Information

The online version contains supplementary material available at https://doi.org/10.1007/s11096-026-02203-2.

Impact statements

• Large language models may support the development of patient information leaflets, but independent review of clinical accuracy and safety remains essential before patient use.

• Short-term output stability suggests that large language models could support standardised patient information development, although ongoing monitoring is needed as model performance may change following updates.

• The consistently high reading levels of both AI-generated and regulatory leaflets suggest that improving the accessibility of written medicine information remains an important priority for patient care.

• The limited actionability of both AI-generated and regulatory leaflets reinforces the importance of pharmacist-led counselling as a core component of patient education for high-risk medicines such as oral anticoagulants.

Introduction

Patient information leaflets (PILs) help patients use medications safely and effectively by providing guidance on indications, dosing, missed doses, adverse effects, precautions, drug interactions, and situations requiring urgent medical attention. Effective PILs improve health literacy, support medication adherence, reinforce verbal counselling, and reduce preventable medication-related harm, particularly for high-risk therapies. Oral anticoagulants are among the highest-risk medications used in routine clinical practice and account for a substantial proportion of preventable medication-related hospitalisations worldwide [1, 2]. Their safe use requires patients to understand complex instructions around dosing schedules, adherence, missed doses, bleeding recognition, monitoring requirements, and drug–drug and drug–food interactions. Even minor misunderstandings can contribute to poor anticoagulation management and serious complications such as major bleeding or thromboembolic events [3, 4]. Consequently, high-quality patient education is considered a critical component of anticoagulation care. However, many regulatory-approved anticoagulant PILs exceed recommended readability levels and provide limited practical guidance for patients with varying levels of health literacy [5–7].

At the same time, patients are increasingly turning to large language models (LLMs) such as ChatGPT, Gemini, and DeepSeek for medication-related information [8, 9]. These systems can produce conversational, accessible explanations of medication use, adverse effects, and self-management strategies, often in simpler language than conventional PILs. Unlike regulatory-approved materials, however, LLM outputs are not subject to formal clinical review or standardised quality assurance. This has raised concerns about inaccuracies, omission of safety–critical information, and hallucinations, defined as plausible but factually incorrect responses generated by AI systems [10–12]. In anticoagulation therapy, such errors may have direct and potentially life-threatening consequences. These concerns are not hypothetical. A recent benchmark of six large language models answering self-care medication questions found that, although accuracy was generally high, responses varied substantially with language, phrasing, and context, and some gave potentially unsafe advice [13].

Existing studies of AI-generated health information have applied instruments such as DISCERN and the Patient Education Materials Assessment Tool (PEMAT) or have focused on readability across different clinical conditions [14–20]. However, several important gaps remain. Most prior work has examined quality, usability, and readability separately, and few studies have directly compared AI outputs with FDA-approved patient materials. More broadly, evaluation of AI tools for medicines use has often prioritised usability over rigorous assessment of effectiveness or implementation, underlining the need for more robust evaluation [21]. In addition, the reproducibility of AI-generated medication guidance across repeated prompts and over time has rarely been examined, even though clinical use depends on output remaining consistent. The present study therefore evaluated the quality, usability, readability, and output reproducibility of PILs generated by ChatGPT, Gemini, and DeepSeek for oral anticoagulants, using FDA-referenced patient materials as the comparator. Assessing these four domains together provides a clinically relevant basis for judging whether AI-generated PILs can serve as supplementary tools in anticoagulation patient education.

Aim

To evaluate the quality, usability, readability, and output reproducibility of AI-generated PILs for oral anticoagulants, using FDA-referenced patient information materials as the comparator standard.

Method

Study design

This cross-sectional comparative study evaluated AI-generated PILs for five oral anticoagulants (warfarin, apixaban, dabigatran, rivaroxaban, and edoxaban) against FDA-referenced patient information materials and was reported in accordance with the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guideline [22] and informed by the MedinAI guidelines for artificial intelligence studies in medicines, pharmacotherapy, and pharmaceutical services [23, 24].

PIL development

AI model and comparator materials

Three publicly accessible LLMs were evaluated: ChatGPT (OpenAI, GPT-5.4), Gemini (Google DeepMind, 2.5 Pro), and DeepSeek (DeepSeek AI, V3). All models were accessed through their default web interfaces on 5 April 2026, with memory features and prior chat histories cleared before each session to minimise personalisation bias. FDA-approved PILs for Coumadin (warfarin), Eliquis (apixaban), Pradaxa (dabigatran), Xarelto (rivaroxaban), and Savaysa (edoxaban) served as comparator materials. Brand names were replaced with generic equivalents before evaluation to preserve blinding. The FDA-approved Medication Guides used as comparator materials are documented in Supplementary Table S8 (Supplementary Material S2).

Prompt design and formatting

The structure and content of the PILs were based on U.S. FDA patient labelling frameworks, specifically the Patient Information and Medication Guide templates [25–27]; the template used as the structural reference is provided in Supplementary Material S1, Table S2. All nine question domains outlined in these templates were adapted for this study, covering indications, contraindications, administration, precautions, adverse effects, storage, and general safety information (Supplementary Material S1, Table S1). The prompt was developed in accordance with established prompt engineering principles [28–31] and included clear task instructions, clinical context, alignment with FDA-style patient information, and safeguards against inappropriate or unsafe outputs. Preliminary versions were refined iteratively until all nine domains were consistently addressed without further prompting. A zero-shot approach was used, in which each LLM received a single instruction without example responses, to standardise inputs across models, minimise prompt-related variation, and ensure fair comparison between information sources while avoiding stylistic bias introduced by example-based prompting [28]. The final prompt is provided in the Supplementary Material S1, Supplementary Text S1. AI-generated and FDA-referenced PILs were then formatted using a uniform template aligned with health communication recommendations from the European Medicines Agency (EMA), Centres for Disease Control and Prevention (CDC), and U.S. Food and Drug Administration (FDA) [32–34]. This standardisation did not alter the underlying content of any material. All twenty evaluated patient information leaflets, comprising the AI-generated and FDA-referenced materials for each of the five anticoagulants, are provided in Supplementary Material S2.

Validation process

Expert panel and blinding

Three clinical pharmacists with at least five years of experience in anticoagulation management formed the validation panel. All reviewers were independent of the study design and the PIL generation process. Materials were anonymised using a computer-generated randomisation sequence and assigned unique alphanumeric identifiers, and reviewers remained blinded to both the content source and the model identity throughout the evaluation. Following a structured training session, reviewers independently rated all PILs using the PEMAT-P and DISCERN instruments. Disagreements were resolved through discussion until full consensus was reached.

Reproducibility assessment

Reproducibility was assessed by examining whether each model produced consistent outputs when the same prompt was repeated at different times. Two dimensions were examined: intra-day reproducibility, referring to consistency between three outputs generated within the same day, and temporal reproducibility, referring to consistency between outputs generated at Day 1, 14, and 28. Each output was then evaluated across five domains: accuracy, clarity, completeness, internal consistency, and temporal stability, each scored using a five-point Likert scale as shown in Supplementary Material S1, Table S3. The five-domain rubric used to rate reproducibility was developed specifically for this study. It has not undergone formal psychometric validation, and the reproducibility findings should therefore be read as exploratory.

Inter-rater reliability

Inter-rater reliability was calculated prior to consensus resolution. Fleiss’ kappa [35] was used for PEMAT-P scores, with agreement interpreted using the criteria of Landis and Koch [36], and intraclass correlation coefficients (ICC [2, k]) [37] together with Cronbach's alpha [38] were used for DISCERN. For PEMAT-P, agreement ranged from substantial to almost perfect across all sources (Fleiss' kappa = 0.76–0.82), with higher agreement for actionability (kappa = 0.81–1.00) than for understandability (kappa = 0.55–0.75). For DISCERN, reliability was good to excellent (ICC [2, k] = 0.93–0.96; Cronbach's alpha = 0.93–0.96). Full reliability statistics are provided in the Supplementary Material S1, Tables S6 and S7.

Outcome measures

Informational quality and usability assessment

PEMAT-P evaluates usability through two subscales: understandability (PEMAT-U), the extent to which patients can process the information presented, and actionability (PEMAT-A), the extent to which patients can identify and carry out the actions described [14]. DISCERN evaluates informational quality, including whether materials present balanced, evidence-informed content that supports informed treatment decision making [15]. Both instruments were selected because oral anticoagulant PILs must inform patients clearly, enable safety–critical actions, acknowledge uncertainty, and support patients as active participants in their own care.

For PEMAT-P, a score of 70% or higher on each subscale was considered acceptable. For DISCERN, items 4, 5, and 7, which assess source citations, were excluded because regulatory Medication Guides are intentionally designed without references in accordance with FDA patient labelling guidance; scoring FDA materials on these items would penalise a structural regulatory convention rather than reflect a genuine difference in informational quality. This modification is consistent with previous evaluations of LLM-generated health information [16, 39, 40]. DISCERN results were reported as overall scores, with higher scores indicating better informational quality and a maximum possible score of 65. The PEMAT-P and DISCERN instruments are provided in the Supplementary Material S1, Tables S4 and S5.

Readability assessment

Readability was assessed using seven validated indices: Flesch Reading Ease [41], Flesch–Kincaid Grade Level [42], Gunning Fog Index [43], Coleman–Liau Index [44], SMOG Index [45], Automated Readability Index [46], and FORCAST [47]. These indices estimate reading difficulty from different text features. Flesch Reading Ease and Flesch–Kincaid Grade Level are based on average sentence length and average syllables per word, the former reported as a 0–100 ease score and the latter as a US grade level. The Gunning Fog Index and the SMOG Index are based on sentence length and the proportion of multisyllabic words. The Coleman–Liau Index and the Automated Readability Index are character-based, using characters per word rather than syllable counts. FORCAST is based on the proportion of single-syllable words and does not use sentence length, making it suitable for materials that are not written in continuous prose, such as the partly list-formatted leaflets evaluated here. Applying these complementary indices together provides a more robust assessment of readability than any single measure. Results were interpreted against the recommended sixth- to eighth-grade reading level for patient education materials. All readability analyses were performed using Readable.com (Readable Software Ltd., UK).

Statistical analysis

PEMAT-P and DISCERN outcomes were summarised as medians with interquartile ranges, given their ordinal nature and small group sizes, while readability indices were summarised as means with standard deviations. Between-source differences for PEMAT-P, DISCERN, and readability outcomes were compared using the Kruskal–Wallis H test, with epsilon-squared reported as the effect size. Where significant differences were detected, pairwise comparisons were performed using the Mann–Whitney U test with Bonferroni correction. Reproducibility was analysed using repeated-measures ANOVA, with Greenhouse–Geisser corrections applied where sphericity was violated. Output stability was assessed from the observed mean differences across conditions and the presence or absence of statistically significant within-model variation, rather than against a formal equivalence margin. Given the small group sizes, a post-hoc sensitivity analysis was conducted to estimate the minimum detectable effect for the primary between-source comparisons (DISCERN, PEMAT-P, and readability) at five observations per group, using a significance level of 0.05 and 80% power. The analysis indicated that the study was powered to detect only large between-group effects (approximately η² = 0.41); smaller differences may not have been detected. All analyses were performed using IBM SPSS Statistics version 31.0.2.0 and RStudio Desktop version 2026.04.0.

Ethics approval

This study did not involve patients, clinical data, or identifiable personal information and was therefore exempt from formal ethics review in accordance with the Indian Council of Medical Research (ICMR) National Ethical Guidelines for Biomedical and Health Research Involving Human Participants (2017) and the Declaration of Helsinki. Written informed consent was obtained from all expert evaluators prior to participation.

Results

Quality and usability

All sources met the 70% acceptability threshold on both PEMAT-P subscales (Table 1). ChatGPT showed the highest median understandability (91.6%), followed by FDA comparator materials (83.3%), while DeepSeek and Gemini both reached 75%. Understandability differed significantly across sources (Kruskal–Wallis p = 0.007). ChatGPT and FDA comparator materials showed higher median understandability scores than DeepSeek and Gemini, although no pairwise comparison remained significant after Bonferroni correction, consistent with limited power at five observations per group. Actionability medians were 75% across all sources, with the only variation arising from a single FDA comparator leaflet; actionability scores were near-constant and did not differ significantly across sources (p = 0.392).

Table 1.

Quality and usability scores across sources

Source	DISCERN, Median [IQR]	PEMAT-U (%), Median [IQR]	PEMAT-A (%), Median [IQR]
ChatGPT	41 [39.5–45.0]	91.60 [83.30–91.60]	75 [75–75]
DeepSeek	34 [32.0–37.0]	75 [75–79.15]	75 [75–75]
Gemini	33 [32.5–34.5]	75 [75–83.30]	75 [75–75]
FDA comparator	41 [37.0–43.5]	83.30 [83.30–91.60]	75 [62.50–75.00]
Kruskal–Wallis	H (3) = 13.580, p = 0.004, ε2 = 0.715	H (3) = 12.053, p = 0.007, ε2 = 0.634	H (3) = 3.000, p = 0.392, ε2 = 0.158
Post hoc comparison	Significant: ChatGPT > DeepSeek (p = 0.008); ChatGPT > Gemini (p = 0.008); FDA > Gemini (p = 0.008) Not significant: ChatGPT vs. FDA (p = 0.690); DeepSeek vs. Gemini (p = 0.548); DeepSeek vs. FDA (p = 0.032)	Not significant (all comparisons): ChatGPT vs. DeepSeek (p = 0.016); ChatGPT vs. Gemini (p = 0.032); ChatGPT vs. FDA (p = 0.690); DeepSeek vs. Gemini (p = 0.690); DeepSeek vs. FDA (p = 0.016); Gemini vs. FDA (p = 0.056)	Not applicable—omnibus test not significant (p = 0.392); post hoc comparisons not conducted

Data are presented as median [IQR]. PEMAT-U = understandability; PEMAT-A = actionability. PEMAT-P scores ≥ 70% indicate acceptable understandability and actionability. The modified DISCERN maximum score was 65. Between-source differences were analysed using the Kruskal–Wallis H test, with effect size reported as epsilon-squared (ε²), calculated using H/(n − 1). Post hoc pairwise comparisons were performed using the Mann–Whitney U test with exact two-tailed p values, and statistical significance was assessed using a Bonferroni-adjusted threshold (α = 0.0083). For transparency, exact p values are reported for all pairwise comparisons, including those that did not remain significant after Bonferroni correction. For PEMAT-P understandability, although the omnibus test was significant, no pairwise comparison remained significant after Bonferroni correction, reflecting the limited statistical power associated with five observations per group. For PEMAT-P actionability, post hoc comparisons were not performed because the omnibus test was not statistically significant

DISCERN scores differed significantly across sources, with a large overall effect size (Table 1). ChatGPT and the FDA comparator materials achieved the highest median scores (both 41/65), whereas DeepSeek (34/65) and Gemini (33/65) scored lower. Post hoc analysis showed that ChatGPT scored significantly higher than both DeepSeek and Gemini, and the FDA comparator materials scored significantly higher than Gemini after Bonferroni correction. No statistically significant differences were observed between ChatGPT and the FDA comparator materials, DeepSeek and Gemini, or the FDA comparator materials and DeepSeek after Bonferroni correction (Table 1).

Readability

Readability indices mostly exceeded the recommended sixth- to eighth-grade reading level, and no significant between-source differences remained after Bonferroni correction; given the limited power at this sample size, this should not be taken as evidence of equivalent readability (Table 2).

Table 2.

Readability scores across AI-generated and FDA-referenced PILs

Metric	ChatGPT Mean ± SD	Gemini Mean ± SD	DeepSeek Mean ± SD	FDA Comparator Mean ± SD	H(3)	p	ε2
FRES	56.18 ± 3.26	62.48 ± 4.87	63.15 ± 4.21	60.18 ± 3.61	6.68	0.08	0.35
FKGL	7.28 ± 0.37	6.91 ± 0.83	6.94 ± 0.84	7.71 ± 0.90	2.49	0.47	0.13
GFI	8.21 ± 0.32	8.11 ± 0.68	8.96 ± 0.94	9.25 ± 0.95	3.59	0.30	0.18
CLI	9.93 ± 0.67	9.34 ± 0.99	9.33 ± 0.67	10.15 ± 0.93	3.02	0.38	0.15
SMOG	9.70 ± 0.32	10.04 ± 0.51	10.09 ± 0.69	11.25 ± 0.68	9.29	0.02	0.48
ARI	6.12 ± 0.53	6.02 ± 0.93	6.12 ± 0.80	7.21 ± 0.98	3.61	0.30	0.19
FORCAST	11.26 ± 0.41	10.39 ± 0.31	10.47 ± 0.28	10.58 ± 0.27	8.85	0.03	0.46

Data presented as mean ± standard deviation (n = 5 per source). For FRES, higher scores indicate easier reading; for all other indices, higher scores indicate greater complexity. Between-source differences were tested using the Kruskal–Wallis H test, with effect size reported as epsilon-squared (ε² = H/(n − 1)). After Bonferroni correction for multiple comparisons (adjusted α = 0.007), no index showed a statistically significant between-source difference. At five observations per group and a significance level of 0.05, the study was powered to detect only large effects (approximately η² = 0.41); after Bonferroni correction the detectable effect is larger still, so none of the observed readability differences should be interpreted as evidence of equivalence. Mean values exceeded the recommended sixth- to eighth-grade reading level on five of the seven indices. FRES = Flesch Reading Ease; FKGL = Flesch–Kincaid Grade Level; GFI = Gunning Fog Index; CLI = Coleman–Liau Index; SMOG = Simple Measure of Gobbledygook; ARI = Automated Readability Index; FORCAST = Forecasting Readability

Output reproducibility

The reproducibility rubric was developed for this study and was not formally validated before use, so the following findings describe output stability under standardised conditions rather than formal equivalence. All three models showed stable output across both intra-day and temporal assessments, with no statistically significant within-model differences identified across any repeated generation condition (all p > 0.05; Table 3, Fig. 1). The largest mean difference between any two conditions was 0.23 points, consistent with stable output within the same day and across the 28-day observation period.

Table 3.

Intra-day and temporal reproducibility of AI-generated PILs across repeated generations

LLM	Run 1 Mean ± SD	Run 2 Mean ± SD	Run 3 Mean ± SD	p	ηp2	Day 1 Mean ± SD	Day 14 Mean ± SD	Day 28 Mean ± SD	p	ηp2
ChatGPT	4.57 ± 0.50	4.67 ± 0.48	4.47 ± 0.51	0.065	0.090	4.47 ± 0.57	4.67 ± 0.61	4.63 ± 0.56	0.294	0.041
DeepSeek	4.10 ± 0.66	4.27 ± 0.52	4.07 ± 0.58	0.172	0.061	4.10 ± 0.66	3.93 ± 0.45	3.87 ± 0.68	0.213	0.052
Gemini	4.47 ± 0.57	4.33 ± 0.66	4.53 ± 0.57	0.058	0.093	4.47 ± 0.57	4.53 ± 0.57	4.37 ± 0.81	0.235	0.049

Composite mean ratings across five domains (accuracy, clarity, completeness, internal consistency, and temporal stability) on a five-point Likert scale where higher scores indicate better performance. Effect size (partial eta-squared): < 0.06 = small, 0.06–0.14 = medium, > 0.14 = large. Significance threshold p < 0.05. The largest observed mean difference between any two conditions across all models was 0.23 points, reported directly rather than against an equivalence margin. The reproducibility rubric was developed for this study and was not formally validated, so these findings describe output stability rather than formal equivalence

Fig. 1.

Composite expert ratings of AI-generated patient information leaflets across repeated generations. a intra-day reproducibility across three generations produced on the same day (Run 1 to Run 3); b temporal reproducibility across generations produced on Day 1, Day 14, and Day 28. Each point is the composite mean expert rating across five domains on a five-point scale, where higher values indicate better performance. Lines denote ChatGPT (solid line, circles), DeepSeek (dashed line, squares), and Gemini (dotted line, triangles). No statistically significant within-model differences were identified across runs or days (all p > 0.05, repeated-measures ANOVA); the Greenhouse–Geisser correction was applied for the DeepSeek intra-day analysis (Mauchly's W = 0.721, p = 0.010)

Discussion

This study evaluated the informational quality, usability, readability, and output reproducibility of AI-generated PILs for five oral anticoagulants against FDA-referenced materials. DISCERN quality varied between models: ChatGPT achieved DISCERN scores similar to those of FDA-referenced materials, whereas DeepSeek and Gemini achieved lower scores. Usability assessed by PEMAT-P was acceptable across all sources, although actionability was universally limited, with no source providing patient summaries or practical decision-support tools. Reading levels exceeded the recommended sixth- to eighth-grade range on most indices, with no significant between-source differences, although smaller differences in readability cannot be excluded. Output characteristics appeared stable across all models within the same day and over the 28-day observation period.

Understandability was acceptable across all sources, consistent with findings in other clinical areas [16, 48, 49]. However, models differed in how clearly they communicated medication information. DeepSeek failed the everyday language criterion for four of five anticoagulants and did not define medical terms in any of its outputs, and Gemini showed similar patterns across most drugs. The everyday language criterion assesses whether materials use plain, common words rather than unexplained medical or technical terms. Unclear medical terminology was not exclusively an AI limitation: FDA comparator materials also failed this criterion for three of five drugs. For example, patients who encounter unexplained terms such as thromboembolism or INR may fail to recognise warning signs or may not know what to report to their healthcare provider, with direct safety implications in anticoagulation management.

Actionability was limited across all sources. Two PEMAT-P items failed universally: no material provided a patient summary, and none offered a practical decision-support tool such as a missed-dose card, symptom checklist, or bleeding recognition guide. These failures were shared by ChatGPT and FDA comparator materials, suggesting a broader gap in patient education design rather than a model-specific limitation. The FDA comparator for rivaroxaban additionally failed to present recommended actions in a step-by-step format, further reducing its actionability score. In anticoagulation therapy, where the consequences of missed doses differ between medications and early bleeding recognition relies on patient action, the absence of structured decision support may create barriers to effective self-management [50]. Similar actionability ceilings have been reported in other disease areas [17, 18, 48, 51, 52], pointing to a broader weakness in patient education design. These findings support the continued use of verbal counselling and practical decision aids alongside written educational materials.

DISCERN scores suggested two broad performance groupings. ChatGPT and FDA comparator materials both reached fair-to-good quality, addressing most criteria but with identifiable gaps around treatment uncertainties and the basis for recommendations. DeepSeek and Gemini fell into a lower band, covering basic information but omitting risk communication and acknowledgement of evidence limitations. ChatGPT's performance is consistent with findings in other clinical areas [48, 53], while the lower DeepSeek scores contrast with one study reporting comparable performance between the two models for prostate cancer content [52], suggesting that performance differentials are sensitive to clinical context and prompting approach. A similar advantage for GPT-based models has been reported within clinical pharmacy; however, such findings should be interpreted within the context of rapidly evolving model architectures and periodic platform updates. In a benchmark of self-care medication questions, GPT-4.0 demonstrated the strongest performance among the evaluated models and was the only model reported to answer all items without a serious error [13], and an evaluation of the customised delirium model DELSTAR identified GPT-4o and GPT-3.5 as the most reliable and stable foundation models [54]. These findings align with the stronger performance of ChatGPT in the present study. The gap was most apparent in clarity of aims, balance and objectivity, and support for shared decision making, where DeepSeek and Gemini consistently scored lower across all five drugs. No source adequately acknowledged areas of uncertainty, such as variability in individual bleeding risk, and none addressed what would happen if treatment was stopped, gaps reported in other clinical areas [48, 52, 53] that extend beyond informational quality into whether these materials genuinely support patients as informed participants in their care. AI-generated materials did, however, explain how each medication works more clearly than FDA comparator materials across all five drugs. This observation suggests that AI-generated content may have potential to present mechanism-of-action information in more accessible language, although this should be confirmed across a broader range of medications and clinical contexts [55].

Most readability indices exceeded the recommended sixth- to eighth-grade reading level, although scores were generally more moderate than the college-level complexity reported in previous studies [16, 56, 57], possibly reflecting the influence of the structured prompting framework used. FDA comparator materials tended to show greater linguistic complexity across several readability indices, suggesting that readability challenges may reflect longstanding regulatory writing conventions rather than being unique to AI-generated materials [55]. Although this benchmark is widely used for patient education materials, literacy and health literacy vary substantially across populations and countries. Consequently, the accessibility challenges identified in this study may be even greater for individuals with limited health literacy, lower educational attainment, or those reading in a non-native language. No statistically significant within-model differences were identified either within the same day or across the 28-day observation period, indicating stable output under standardised conditions. Given the exploratory sample size and the absence of a formally validated reproducibility instrument, these findings should be interpreted as evidence of short-term output stability rather than proof of reproducibility. This contrasts with De Busser and colleagues, who observed day-to-day variability when the same question was posed daily over 60 days [13], and with the DELSTAR evaluation, which reported performance fluctuation across models and an instance of hallucination even within a controlled, domain-specific tool [54]. The difference may reflect their longer observation windows and single-question probes compared with our 28-day structured-prompt design, and it reinforces the need for longer-term reproducibility assessment noted in our limitations. Real-world use, however, involves variable prompting, platform updates, and model changes that may influence consistency over longer periods. These results should be interpreted as a time-stamped evaluation of the model versions available during the study period, and the findings may not fully generalise to future model versions following platform updates or changes in underlying architecture.

AI-generated PILs, particularly those produced by ChatGPT under structured prompting, may serve as clinician-reviewed supplementary resources in anticoagulation care. They should not be used as standalone patient education tools, given persistent actionability and readability limitations. This positioning is consistent with the broader direction of recent clinical pharmacy research; in an overview of this field, none of the reviewed studies pointed toward replacing pharmacists, AI tools being better understood as augmenting rather than substituting for pharmacists' expertise [58]. Clinical pharmacists should provide structured verbal counselling on missed-dose management, bleeding recognition, drug interactions, and when to seek urgent medical review, and should supplement the written material with practical decision aids such as missed-dose cards, symptom checklists, or bleeding-recognition guides to address the actionability gaps identified across both AI-generated and regulatory leaflets. Health systems should establish governance processes that include professional oversight, standardised prompting, and periodic re-evaluation following model updates [7, 59].

Several important questions remain open. No study has yet tested whether AI-generated PILs improve patient understanding or medication adherence in real-world anticoagulation care, and none has measured downstream safety outcomes such as bleeding events, missed-dose errors, or unscheduled healthcare contacts following exposure to AI-generated information. The performance of these models across patients with different health literacy levels, languages, and cultural contexts also remains unexamined, despite the global use of these tools. Standardised prompting frameworks for high-risk medication classes need to be developed, validated, and tested for transferability across institutions, and the reproducibility rubric used here requires formal psychometric validation. This need for standardisation is increasingly recognised in the field; the recently published MedinAI reporting guidelines identify transparency, reproducibility, and attention to model changes over time as central to the responsible reporting of AI studies in medicines and pharmacotherapy [23, 24]. Finally, the impact of platform updates and successive model versions on output quality should be assessed prospectively, since clinical reliance on outputs that change without notice introduces a governance challenge that the current literature has not addressed.

This study has several limitations. While exploratory in scope, it applied a structured and reproducible evaluation framework using standardised prompting, blinded expert assessment, and multiple validated instruments, and was intended to generate hypotheses rather than to provide a definitive comparison between sources. Importantly, the study evaluated the patient information leaflets themselves rather than patient outcomes, and no patient comprehension testing was performed. Consequently, the findings cannot determine how patients with differing levels of health literacy would understand, interpret, or act upon the information provided. The sample size of five drugs per source group limited statistical power, and findings were interpreted primarily using effect sizes and rank patterns. Consistent with this, the sensitivity analysis indicated that only large effects were detectable at this sample size; therefore, non-significant findings, particularly the readability comparisons, should not be interpreted as evidence of equivalence, and smaller but clinically meaningful differences cannot be excluded. The reproducibility framework was not formally validated prior to use. Its domains were derived from prior work rather than established through psychometric testing, so the reproducibility results should be regarded as preliminary and require confirmation with a validated instrument. Although this study evaluated output stability, as reflected by the consistency of expert ratings across repeated generations, it did not assess content-level reproducibility of the underlying medical information. Consequently, outputs receiving similar structural ratings may nonetheless differ in clinically meaningful details, particularly for high-risk medicines such as oral anticoagulants. Three DISCERN items assessing source citations were excluded to prevent systematic bias against FDA materials; future studies should consider reporting these items separately. The FDA-approved comparator materials also represent a fixed regulatory snapshot, with the corresponding version information documented in Supplementary Table S8. Because Medication Guides are periodically updated, the comparator materials reflect the FDA-approved versions available at the time of the study and may differ from those available subsequently, analogous to the timestamped evaluation of the LLM-generated outputs. Patient comprehension testing across different health literacy levels, evaluation using conversational and real-world prompts rather than a single standardised prompt, and longer reproducibility observation windows remain important next steps.

Conclusion

ChatGPT-generated PILs demonstrated informational quality similar to FDA-referenced materials, while DeepSeek and Gemini performed at lower levels. Actionability and readability limitations were consistent across both AI-generated and regulatory materials, suggesting that the challenges in anticoagulant patient education are not unique to AI but reflect broader gaps in how written health information is developed and evaluated. Output was stable across repeated generations, with no significant within-model differences; however, this finding alone is insufficient to establish clinical suitability.

These findings should be interpreted within the scope of the outcomes assessed. The study evaluated informational quality, understandability, actionability, readability, and output reproducibility, but did not assess pharmacological accuracy, clinical correctness, omission of safety–critical information, hallucinations, or patient safety outcomes. Importantly, PEMAT-P and mDISCERN assess how information is presented and communicated, rather than whether the underlying clinical content is accurate, complete, or safe for individual patients. Consequently, comparable performance on these instruments should not be interpreted as evidence of equivalent clinical accuracy or safety. Any clinical implementation of AI-generated patient information leaflets should therefore be subject to independent review by qualified healthcare professionals before dissemination to patients.

AI-generated PILs are best positioned as clinician-reviewed supplementary resources within structured anticoagulation care pathways, supported by governance frameworks that include professional review, standardised prompting protocols, and periodic re-evaluation. Future research should prioritise patient comprehension testing across health literacy levels, downstream patient safety outcomes, multi-language and cross-cultural evaluation, and formal psychometric validation of reproducibility instruments.

Supplementary Information

Below is the link to the electronic supplementary material.

Author contributions

Conceptualization: MAK, BP. Data curation: MAK, MM, BP. Formal analysis: MAK, MM, HAA, MQ. Investigation: MAK, MM, AA, IS, HAA, MQ, KS. Methodology: MAK, BP. Project administration: MAK, BP. Software: MAK, IS. Supervision: BP. Validation: MAK, AA, IS, KS, BP. Visualization: MAK, AA, IS, KS. Writing—original draft: MAK, MM. Writing—review and editing: MAK, AS, IS, KS, BP.

Funding

No funds, grants, or other support were received.

Data availability

No datasets were generated or analysed during the current study.

Declarations

Competing interests

The authors declare no competing interests.

Ethics approval

This study did not involve patients, clinical data, or identifiable personal information and was therefore exempt from formal ethics review in accordance with the Indian Council of Medical Research (ICMR) National Ethical Guidelines for Biomedical and Health Research Involving Human Participants (2017) and the Declaration of Helsinki. Written informed consent was obtained from all expert evaluators prior to participation.

Consent to participate

Written informed consent was obtained from all expert evaluators.

Consent for publication

All authors consent to publication.