Benchmarking freely accessible large language models for pediatric epilepsy care: a prospective mixed-methods study

Abstract

Background

Large language models (LLMs) may support patient education and clinical documentation, but their accuracy, safety, and usability in pediatric epilepsy remain insufficiently characterized. This study aimed to compare four freely accessible LLMs using standardized pediatric epilepsy scenarios and to evaluate the top-performing model that met the prespecified safety criterion in a supervised clinician workflow assessment.

Methods

We conducted a prospective, two-phase mixed-methods study at the University Hospital at King Abdulaziz University in Jeddah. In phase 1, four freely accessible LLMs (ChatGPT, Gemini, DeepSeek, and Perplexity) were evaluated using 10 standardized pediatric epilepsy scenarios. Outputs were assessed by pediatric neurologists across factual accuracy, comprehensiveness and relevance, safety, clinical appropriateness, readability and caregiver tailoring, and transparency and sources. Critical safety flags were recorded separately. In phase 2, the model with the highest observed composite score among systems meeting the prespecified safety criterion underwent a supervised clinician workflow assessment using the System Usability Scale (SUS), a caregiver-facing educational-material assessment, and semi-structured interviews.

Results

Forty LLM responses were evaluated. Observed composite scores differed significantly across the four model-specific rater groups (P<0.001); however, because each model was evaluated exclusively by a single rater, the observed differences cannot be attributed to model effects independently of evaluator-specific scoring tendencies. DeepSeek and Gemini generated no critical safety flags, whereas ChatGPT had flags in 3/10 responses (30%) and Perplexity in 8/10 (80%). DeepSeek had the highest observed composite score (4.94/5), compared with 4.42/5 for Gemini, 3.59/5 for ChatGPT, and 2.05/5 for Perplexity, and was therefore selected for phase 2. Clinician approval for caregiver use was higher for responses without safety flags than for flagged responses (82.8% vs. 9.1%).

Conclusions

In this specialty-specific benchmark, the evaluated LLMs showed substantial differences in observed accuracy and safety, with DeepSeek achieving the highest observed composite score and no critical safety flags. However, the single-rater-per-model design, small sample, model updates, and limited workflow assessment preclude definitive conclusions about comparative model superiority. These findings support specialty-specific evaluation of LLMs that prioritizes safety and clinical accuracy alongside usability. Larger, independently rated, multi-center studies with repeated model sampling and objective workflow and patient-centered outcomes are needed before routine clinical implementation can be considered.

Highlight box.

Key findings

• DeepSeek demonstrated the highest observed composite performance among the evaluated large language models (LLMs), with zero prespecified critical safety flags.

• Safety was strongly associated with clinician approval: only 1 of 11 responses with a safety flag (9.1%) was approved, compared with 24 of 29 responses without flags (82.8%).

• Perplexity generated the highest rate of critical safety flags (80%), characterized by clinically important vagueness in areas such as medication contraindications and emergency management.

• Clinicians perceived the selected model as potentially improving documentation efficiency, although formal timed measurements were not captured.

What is known and what is new?

• LLMs show promise in medicine but demonstrate variable accuracy and reliability; they have also been used to extract structured information from epilepsy clinic letters.

• This study provides a prospective, specialty-specific evaluation of freely accessible LLMs for patient-facing educational content in pediatric epilepsy, with clinician-rated assessment of factual accuracy, safety, usability, and clinical appropriateness.

What is the implication, and what should change now?

• LLM performance varies substantially across pediatric epilepsy tasks; rigorous, domain-specific benchmarking is essential before clinical adoption.

• The selected model may have potential as a supervised clinical assistant for drafting patient education materials, although objective efficiency gains remain to be confirmed.

• Clinicians should remain responsible for verifying LLM-generated content and applying clinical judgment; artificial intelligence should augment rather than replace clinician oversight.

• Institutions should develop model-specific, evidence-based policies for LLM use in epilepsy care, prioritizing safety, verification, and clinician oversight.

Introduction

Effective communication and accurate information management are important components of pediatric epilepsy care, which requires sustained family engagement, treatment adherence, and shared decision-making (1,2). Clinicians face several barriers to effective communication, including limited consultation time and differences in caregiver health literacy, which may affect understanding and treatment decisions (3). At the same time, important clinical information remains embedded in unstructured records such as clinic letters, making it difficult to use these data efficiently for research and clinical care (4).

Freely accessible large language models (LLMs) may offer potential approaches to supporting these communication and information-management challenges (5). For clinical communication, these tools can generate tailored educational materials and assist with documentation, which may improve accessibility and reduce some of the workload associated with these tasks (6). LLMs may also support clinical and research workflows by processing unstructured clinical information (7), although these applications were beyond the scope of the present study.

Early studies support the potential of LLMs for both applications. In information extraction, open-source models like Llama 2 have shown superior capability in accurately identifying key epilepsy details from clinic letters, outperforming other open-source LLMs evaluated for the same extraction tasks (7). In patient-facing contexts, initial evaluations of LLMs in medicine have demonstrated substantial variability in factual accuracy and clinical reliability (8,9). Such variability is particularly important in pediatric epileptology, where inaccurate or incomplete information could plausibly contribute to patient harm if presented without appropriate clinician review (9).

However, important gaps remain. There is limited prospective, specialty-specific benchmarking of freely accessible general-purpose LLMs for pediatric epilepsy, particularly using clinician-defined safety criteria and subsequent assessment in clinical workflows. Evaluation should therefore consider factual accuracy, safety, and usability rather than language quality alone (10). We therefore conducted a prospective, mixed-methods evaluation of freely accessible LLMs, benchmarking their performance in standardized pediatric epilepsy scenarios and subsequently assessing the system with the highest observed composite score in a supervised clinician workflow setting. We present this article in accordance with the STROBE reporting checklist (available at https://tp.amegroups.com/article/view/10.21037/tp-2026-0731/rc).

Methods

Study design and oversight

The study addressed two sequential research questions: phase 1, which freely accessible LLM demonstrates the highest observed factual accuracy and safety performance within standardized pediatric epilepsy patient-facing scenarios; and phase 2, how does the top-performing model from phase 1 perform in a supervised clinician workflow assessment regarding usability, perceived efficiency, and clinician-rated quality?

To address these separate aims, we employed a prospective, two-phase explanatory sequential design at the University Hospital at King Abdulaziz University in Jeddah, Saudi Arabia. Phase 1 is a controlled quantitative benchmarking experiment. Four freely accessible LLMs were evaluated against standardized clinical scenarios using a structured scoring instrument, with blinded expert neurologists assessing factual accuracy, safety, and clinical appropriateness. The primary purpose of phase 1 was to identify the model with the highest observed composite score for evaluation in phase 2.

Phase 2 is a mixed-methods supervised clinician workflow and human-factors assessment of this selected model. We incorporated the selected model into a supervised clinician workflow assessment to collect: (I) quantitative data on System Usability Scale (SUS) scores; and (II) qualitative data via semi-structured clinician interviews to explore trust, usability, and perceived clarity. This sequence allowed model safety and performance to be assessed before the supervised workflow assessment and separated benchmarking from the subsequent usability assessment. The Unit of Biomedical Ethics Research Ethics Committee is a registered committee at the National Committee of Bioethics (NCBE) of Saudi Arabia. This study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments, as well as the Good Clinical Practice guidelines. The study was approved by the Unit of Biomedical Ethics Research Ethics Committee at the Faculty of Medicine, King Abdulaziz University (No. 18-26). Informed consent was obtained from all clinician participants. Given the current absence of a reporting guideline specifically designed for LLM benchmarking studies, this study was reported in accordance with applicable items of the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis-Large Language Models (TRIPOD-LLM) guideline, particularly those addressing model identification and versioning, prompt/input transparency, and outcome definition. Relevant items from the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) checklist were also considered for the human-rater components of the study, including the phase 1 evaluation panel and phase 2 clinician cohort, where applicable. STROBE was not developed for artificial intelligence (AI) model benchmarking, and therefore its recommendations were applied selectively to the components relevant to the human participants (11). Additionally, five pediatric neurologists developed the scenarios and did not participate as evaluators, ensuring independence between scenario design and response scoring; a separate panel of seven pediatric neurologists was trained as evaluators, of whom four completed all phase 1 evaluations and were included in the quantitative analysis, and all seven participated in phase 2 usability and workflow integration assessments.

Phase 1: controlled benchmarking evaluation

A panel of five board-certified pediatric neurologists/epileptologists (the scenario development panel) developed 10 standardized clinical scenarios covering high-impact topics in pediatric epilepsy. These were designed to test diagnostic reasoning, medication safety, and patient counseling (Table 1).

Table 1. Clinical scenario domains for LLM benchmarking.

Domain	Example scenarios	Core competencies tested
New-onset epilepsy	Childhood absence epilepsy; benign epilepsy with centrotemporal spikes	Syndrome recognition, first-line treatment, prognosis counseling
Developmental & epileptic encephalopathies	Dravet syndrome; Lennox-Gastaut syndrome	Medication contraindications, emergency management, genetic counseling
Medication management	Polytherapy initiation; side effect explanation; valproate use in adolescents	Dosing accuracy, drug interaction awareness, teratogenicity risk communication
Non-pharmacological therapies	Ketogenic diet; VNS management	Dietary protocol, device safety, candidate selection criteria

LLM, large language model; VNS, vagus nerve stimulator.

LLM querying and response generation

The most recent publicly accessible versions of four LLMs were selected for evaluation. Perplexity is a retrieval-enabled, multi-model synthesis platform that may draw on multiple underlying models, whereas ChatGPT, Gemini, and DeepSeek provide conversational access to named LLM systems. This difference limits direct comparability and should be considered when interpreting the results. ChatGPT, Gemini, DeepSeek, and Perplexity were evaluated as publicly accessible LLMs and AI systems; exact model versions, interface specifications, and access dates are provided in Table 2. All queries were submitted through the respective web interfaces using default platform settings. Temperature and top-p were not manually modified, and no additional system prompts or external retrieval tools were introduced by the investigators. Platform-specific retrieval or other default functionality was retained where inherent to the interface. We used default settings to reflect the outputs that caregivers or clinicians would encounter when using these platforms as publicly available consumer tools. This approach may increase output variability and, combined with single-sample-per-prompt querying, limits the reproducibility of exact response text. A standardized prompt template was created for each scenario (e.g., “Explain Dravet syndrome to the parent of a newly diagnosed 2-year-old, including triggers and first-line medications”). Each prompt was submitted to each LLM in a new session to avoid context bias, and the first complete response was captured for evaluation. All prompts and raw texts have been deposited in Table S1 to support reproducibility and future benchmarking.

Table 2. LLMs evaluated in phase 1.

Model	Developer	Version/model ID	Access date	Access method
ChatGPT	OpenAI	GPT-5.4	April 2026	Web interface (chat.openai.com)
Gemini	Google	Gemini 3.1 Pro	April 2026	Web interface (gemini.google.com)
DeepSeek	DeepSeek AI	DeepSeek-V4 (V4-Pro)	April 2026	Web interface (chat.deepseek.com)
Perplexity	Perplexity AI	Perplexity Pro (multi-model synthesis)	April 2026	Web interface (perplexity.ai)

Model versions reflect the publicly available default models at the time of access. Given the continuous deployment nature of these platforms, exact underlying model versions are subject to change without notice. AI, artificial intelligence; LLM, large language model.

Blinded expert evaluation

A panel of seven pediatric neurologists (the evaluation panel) was trained to use the structured instrument. All seven were fully trained; however, due to clinical workload constraints, only four completed all 40 evaluations. These four provided the complete dataset for the quantitative phase 1 analysis. The remaining three contributed partial data and were excluded from the quantitative analysis per our prespecified completeness criterion. Each of the four complete raters was assigned to evaluate all responses generated by one designated model (10 scenarios per model), yielding a single-rater-per-response design with 40 total evaluations (one rating per response). Because each evaluator was assigned exclusively to one model, evaluator and model effects were fully confounded; consequently, observed between-model differences cannot be attributed to model effects independently of evaluator-specific scoring tendencies. The inferential comparison should therefore be interpreted as a comparison of model-specific rater groups rather than an unbiased estimate of model effects. It is further discussed as a limitation in the “Limitations and future directions” section. To minimize order effects within each rater’s assigned model, responses were randomized. All seven evaluators participated in phase 2 (usability and workflow integration assessments), ensuring that their clinical perspectives were retained for the qualitative and implementation components.

The instrument was developed in several stages. First, two pediatric neurologists conducted a literature review of existing LLM evaluation frameworks (including DISCERN, QAMAI, and the ARISE clinical rubrics) to identify relevant quality domains for patient-facing medical AI outputs. Second, they generated an initial pool of 45 candidate items across seven domains, which were refined through three iterative consensus rounds using a modified Delphi approach; during this process, two conceptually overlapping domains were merged, resulting in the final six-domain structure described below. Third, the instrument was pilot-tested using 10 LLM outputs not included in the study to assess clarity, feasibility, and face validity, resulting in the final 18-item instrument (Table S2).

The instrument comprises 18 items across six domains: (I) factual accuracy (items C1–C3), (II) comprehensiveness & relevance (items C4–C6), (III) safety & harm potential (items C7–C8), (IV) clinical appropriateness & actionability (items C9–C11), (V) readability & tailoring to caregiver (items C12–C15), and (VI) transparency & sources (items C16–C18). For each response, a composite domain score was calculated as the arithmetic mean of all 18 item scores, yielding an overall score from 1 to 5. For each model, the mean overall composite score was calculated across its 10 responses, representing the model-level mean; a condensed summary of composite scores, safety flags, SUS scores, and approval rates per model is provided in Table S3. The instrument also includes a binary critical safety checklist to flag dangerous recommendations (e.g., suggesting contraindicated medications, erroneous dosing more than 10 times the normal, or advising abrupt discontinuation of antiseizure drugs) (Table S4). In addition, each evaluator was asked to provide a global judgment of whether they would allow the output to be given to a caregiver after clinician review (‘clinician approval for caregiver use’), recorded as a binary (yes/no) response. This judgment reflected the evaluator’s overall assessment of safety, accuracy, clarity, and actionability regarding the suitability of the material for caregiver-facing communication. Dosing determinations were anchored to current evidence-based guidelines, including the 2023 International League Against Epilepsy (ILAE) neonatal seizure guidelines (12). The complete adjudication rubric with prespecified criteria for each safety flag type is provided in Table S4.

The instrument was developed specifically for this study and was not intended to undergo formal psychometric validation within this exploratory benchmark. Content validity was supported through literature review, modified Delphi consensus, and pilot testing. Because each response was evaluated by a single model-specific rater, inter-rater reliability could not be assessed.

Selection of model for phase 2 progression

The model advancing to phase 2 was selected from phase 1 results using a prespecified hierarchical decision rule: primary criterion—the model with the lowest (ideally zero) incidence of critical safety flags; secondary criterion—highest mean composite domain score among models meeting the primary safety threshold. DeepSeek and Gemini both had zero critical safety flags. Among these two models, DeepSeek achieved the higher composite domain score (4.94/5 vs. 4.42/5); therefore, DeepSeek was selected for the phase 2 supervised workflow assessment. ChatGPT, Gemini, and Perplexity were not evaluated in phase 2. This decision rule prioritized safety before other performance measures when selecting the model for the next phase.

Phase 2: clinician perception and usability assessment

The same seven pediatric neurology clinicians (attending pediatric neurologists and pediatric neurology fellows) from the pediatric epilepsy clinic who were trained as evaluators in phase 1 were enrolled in phase 2. The AI-generated educational materials were assessed by these clinicians using a tailored survey instrument adapted from the phase 1 evaluation rubric, with the materials designed to address common caregiver-facing clinical scenarios. No financial compensation was provided for participation; clinicians volunteered their time as part of their professional academic responsibilities. The arrangement was reviewed and approved by the Unit of Biomedical Ethics Research Ethics Committee.

Usability and perception assessment

During phase 1, following completion of the blinded evaluations, all seven clinician participants completed the SUS for each of the four LLMs. This assessed perceived interface usability (e.g., ease of navigation, clarity of interface, responsiveness) when using the respective web interfaces to generate responses for the standardized scenarios. These SUS scores reflect phase 1 interface perception only and are not measures of real-world clinical workflow integration.

During phase 2, following incorporation of the selected model (DeepSeek) into the supervised workflow assessment, the same clinicians completed the SUS again specifically for DeepSeek (to assess workflow-integrated usability) and participated in semi-structured interviews exploring themes of trust, workflow integration, and perceived impact on workload. Additionally, clinicians evaluated the quality of DeepSeek-generated educational materials using a tailored survey instrument adapted from the phase 1 evaluation rubric. The instrument comprised 12 items across three domains: clarity and readability (e.g., ‘The material is written in language appropriate for caregivers with low health literacy’), actionability and usefulness (e.g., ‘The material provides concrete, actionable steps that a caregiver can follow’), and appropriateness for caregiver-facing communication (e.g., ‘The tone is appropriately empathetic and supportive’), all scored on a 5-point Likert scale.

Following the workflow integration period, each clinician participant completed a 20–30-minute semi-structured interview. The interview guide explored: trust and confidence in AI-generated outputs; workflow integration compared to usual documentation processes; perceived impact on documentation burden and patient education time; necessary safeguards and clinician oversight; and recommendations for future AI tools in pediatric epilepsy care. All interviews were audio-recorded with participant consent, transcribed verbatim, and de-identified before analysis.

Statistical analysis

All analyses were performed using Statistical Package for the Social Sciences (SPSS) version 29.0. Observed composite scores were compared using a linear mixed-effects model with model as a fixed effect and scenario as a random intercept. Because model and rater assignment were completely confounded, this comparison should be interpreted as a comparison of model-specific rater groups rather than an unbiased estimate of model effects. Rater was not entered as a separate random effect because rater assignment was fixed to a single model throughout, making rater and model fully confounded in this design (see the “Limitations and future directions” section). Denominator degrees of freedom were estimated using Satterthwaite approximation. Pairwise comparisons between models were extracted from the model. Because each response was scored by only one rater, inter-rater reliability could not be assessed. Fisher’s exact test compared the incidence of critical safety flags across models, as expected cell counts fell below 5 for models with zero flags. An exploratory logistic regression analysis examined predictors of clinician approval for caregiver use (defined as the evaluator’s binary judgment of whether the output could be given to a caregiver after clinician review), with composite score and safety flag status as covariates. However, due to quasi-complete separation, collinearity between predictors, and small sample size, these results are presented as hypothesis-generating and should be interpreted with caution. Candidate covariates were prespecified based on clinical plausibility and bivariate associations (P<0.20). All variables were entered simultaneously using a forced-entry method. Interview transcripts from the clinician participants were analyzed using inductive semantic thematic analysis. Two researchers independently coded the transcripts, identified candidate themes related to barriers, facilitators, trust, and clinical utility, and resolved discrepancies through consensus discussion. No formal a priori sample-size calculation was performed because the exploratory benchmark was constrained by the prespecified number of scenarios and availability of trained clinical evaluators. The resulting sample size limits precision and generalizability, and inferential analyses should therefore be interpreted as exploratory.

Results

Baseline characteristics

A total of 40 unique LLM responses were evaluated, comprising 10 responses per model across 10 clinical scenarios (E1–E10). These 40 responses represent the unique outputs generated by the LLMs. Four expert raters completed all evaluations, with each rater assigned to evaluate all 10 responses from one designated model, yielding a single-rater-per-response design with 40 total evaluations (one rating per response). All seven clinicians participated in phase 2. The distribution of responses is summarized in Table 3.

Table 3. Baseline characteristics of evaluated LLM responses.

Characteristics	ChatGPT (n=10)	Gemini (n=10)	DeepSeek (n=10)	Perplexity (n=10)	Overall (n=40)
Word count	355.2±38.7	363.9±38.5	420.3±39.7	288.8±17.6	357.1±62.9
Critical safety flag	3 (30.0)	0 (0.0)	0 (0.0)	8 (80.0)	11 (27.5)
Composite domain score [1–5]	3.59±0.47	4.42±0.34	4.94±0.04	2.05±0.08	3.75±1.20
SUS score [0–100]	52.8±2.68	55.8±4.00	52.3±3.03	54.8±4.43	53.4±2.44
Clinician approval for use	5 (50.0)	9 (90.0)	10 (100.0)	1 (10.0)	25 (62.5)

Data are presented as mean ± SD or n (%). Fisher’s exact test used for safety flag comparisons due to small, expected frequencies. SUS scores reflect phase 1 interface usability perception; phase 2 workflow-integrated SUS for DeepSeek was 50.0. LLM, large language model; SD, standard deviation; SUS, System Usability Scale.

Primary safety and accuracy outcomes

The incidence of critical safety flags varied significantly across models (Fisher’s exact test, P<0.001). No prespecified critical safety flags were detected among the 10 responses evaluated for either DeepSeek or Gemini, while Perplexity had the highest rate (80%, 8/10 responses). ChatGPT had a 30% flag rate (3/10), associated with fabricated references (n=2) and a dangerous recommendation to abruptly discontinue levetiracetam (n=1) if attention-deficit/hyperactivity disorder (ADHD) symptoms worsened, a practice that could precipitate breakthrough seizures. A separate levetiracetam dosing example (30–50 mg/kg/day in a neonate) was reviewed against the 2023 ILAE neonatal seizure guidelines (12), which specify a maintenance dose of 40–60 mg/kg/day. Although the lower bound of the example (30 mg/kg/day) fell below this range, it did not meet the prespecified A2 threshold for an erroneous-dosing safety flag (>10× or <0.1× the expected mg/kg range), and this example was therefore not counted as a critical safety flag.

Observed composite scores differed significantly across the four model-specific rater groups (overall P<0.001). DeepSeek had the highest observed composite score [mean =4.94, standard deviation (SD) =0.04], followed by Gemini (mean =4.42, SD =0.34), ChatGPT (mean =3.59, SD =0.47), and Perplexity (mean =2.05, SD =0.08). Because each model was evaluated by a single dedicated rater, these between-model differences cannot be separated from potential rater-specific scoring tendencies. The detailed breakdown of domain performance is presented in Table 4.

Table 4. Comparative performance across evaluation domains (mean score, 1–5 scale).

Domain	ChatGPT	Gemini	DeepSeek	Perplexity	P value
1. Factual accuracy	3.60 (0.52)	4.50 (0.53)	4.92 (0.06)	2.10 (0.32)	<0.001
2. Comprehensiveness	3.70 (0.48)	4.30 (0.42)	4.95 (0.05)	2.05 (0.22)	<0.001
3. Safety & harm potential	3.60 (0.52)	4.40 (0.52)	4.98 (0.04)	2.00 (0.00)	<0.001
4. Clinical appropriateness	3.50 (0.53)	4.30 (0.42)	4.93 (0.06)	2.00 (0.00)	<0.001
5. Readability & tailoring	3.80 (0.42)	4.40 (0.52)	4.96 (0.05)	2.05 (0.22)	<0.001
6. Transparency & sources	3.30 (0.67)	4.60 (0.52)	4.88 (0.08)	2.00 (0.00)	<0.001
Overall mean domain score	3.59 (0.47)	4.42 (0.34)	4.94 (0.05)	2.05 (0.08)	<0.001

Data are presented as inter-rater reliability (P value). Formal inter-rater reliability could not be assessed, as each response was evaluated by a single, model-specific rater rather than by multiple raters scoring the same response. All P values are from linear mixed-effects models with scenario as a random intercept and model as a fixed effect. A condensed summary of scores by model is provided in Table S4.

DeepSeek achieved the highest overall composite score (4.94/5) and demonstrated zero critical safety flags. Based on the prespecified hierarchical selection rule prioritizing safety first, DeepSeek and Gemini both met the zero-safety-flag criterion; DeepSeek was advanced to phase 2 because it had the higher composite score.

Bivariate associations and predictors of clinician approval

Clinician approval was strongly associated with the presence of a critical safety flag for caregiver use (P<0.001). Only 1 of 11 responses with a safety flag (9.1%) was approved, compared to 24 of 29 responses without flags (82.8%), yielding a difference in approval rates of 73.7 percentage points. These estimates are based on a small number of flagged responses (n=11) and should be interpreted with caution.

To explore potential predictors of clinician approval, we conducted an exploratory binary logistic regression with approval as the dependent variable. Independent variables included the presence of a safety flag and the composite domain score. The full model was statistically significant (χ²=42.1, P<0.001, Nagelkerke R²=0.82). As shown in Table 5, the absence of a safety flag and a higher composite domain score were associated with approval. However, due to quasi-complete separation (DeepSeek: perfect scores, zero flags, 100% approval; Perplexity: low scores, 80% flags, 10% approval), collinearity between predictors, and a small sample size (n=40), these estimates are unstable and should be interpreted with caution. The wide confidence intervals reflect overfitting rather than stable effect estimates. These findings should therefore be considered exploratory rather than evidence of independent effects.

Table 5. Logistic regression predicting clinician approval for caregiver use.

Predictors†	B	S.E.	Wald	P value	Odds ratio (95% CI)
Constant	−14.82	4.21	12.38	<0.001	–
Critical safety flag (absent)	3.89	1.18	10.87	0.001	48.92 (4.86, 492.34)
Composite domain score (per 1-point increase)	3.12	0.82	14.47	<0.001	22.64 (4.54, 112.89)

^†, model identity was not entered as a predictor, as it is collinear with safety flag status and composite domain score (each model has a near-fixed combination of these two variables). Due to quasi-complete separation, collinearity between predictors, and small sample size (n=40), these estimates are unstable and should be interpreted with caution. The wide confidence intervals reflect overfitting rather than stable effect estimates. Hosmer-Lemeshow test: χ²=4.82, P=0.78. CI, confidence interval; S.E., standard error.

Usability and workflow integration potential

Phase 1 SUS scores showed little variation across the four models {F[3,24] =2.87, P=0.058}: ChatGPT 52.8 (SD =2.68), Gemini 55.8 (SD =4.00), DeepSeek 52.3 (SD =3.03), and Perplexity 54.8 (SD =4.43). These scores reflect perceived usability of the respective web interfaces rather than clinical workflow integration. In phase 2, the SUS score for DeepSeek was 50.0 (SD =3.53), similar to its phase 1 score.

No statistically significant differences were observed between models in pairwise comparisons (all P>0.05). A moderate positive correlation was observed between the composite domain score and SUS scores (r=0.45, P=0.021), suggesting a modest association between perceived usability and perceived clinical quality. The detailed breakdown of SUS item scores is presented in Table 6.

Table 6. SUS scores by model and item (mean, 1–5 scale).

SUS items	ChatGPT	Gemini	DeepSeek	Perplexity
1. I would use this tool frequently	3.0±0.82	4.1±0.38	4.9±0.38	2.4±0.79
2. I found the tool unnecessarily complex	2.9±0.69	2.0±0.58	1.3±0.49	3.9±0.38
3. I thought the tool was easy to use	3.4±0.79	4.4±0.53	4.9±0.38	2.7±0.76
4. I would need technical support	3.0±0.58	2.0±0.58	1.3±0.49	3.7±0.76
5. I found the various functions well integrated	3.6±0.79	4.4±0.53	4.9±0.38	2.6±0.79
6. Too much inconsistency in this tool	3.1±0.69	2.1±0.38	1.3±0.49	3.6±0.79
7. Most people would learn this tool quickly	3.6±0.79	4.4±0.53	4.9±0.38	2.7±0.76
8. I found the tool cumbersome to use	2.9±0.69	2.0±0.58	1.3±0.49	3.9±0.38
9. I felt very confident using the tool	3.3±0.76	4.3±0.49	4.9±0.38	2.4±0.79
10. I needed to learn a lot before using this tool	3.0±0.58	2.0±0.58	1.3±0.49	3.7±0.76
Raw SUS sum [10–50]	31.1±1.07	32.3±1.60	30.0±0.82	31.9±1.77
Final SUS score [0–100]	52.8±2.68	55.8±4.00	52.3±3.03	54.8±4.43

Data are presented as mean ± SD on the original 1–5 scale before reverse scoring. SUS scores reflect perceived interface usability assessed during phase 1 after clinicians interacted with each model’s web interface to generate responses for the standardized scenarios. These scores were collected for all four models during phase 1; they are not measures of clinical workflow integration. Phase 2 workflow-integrated SUS for DeepSeek (the only model integrated into clinical workflows) was 50.0 (SD =3.53). SD, standard deviation; SUS, System Usability Scale.

Supervised workflow assessment

Formal timed measurements of documentation efficiency were not collected; therefore, findings in this section reflect clinician perceptions rather than objectively measured time savings. Clinicians generally reported that supervised AI-assisted drafting was faster than standard template use or drafting from scratch. Efficiency-related SUS items were rated favorably, including ease of use, intention to use frequently, and low cumbersomeness. However, the overall phase 2 SUS score was 50.0, below the commonly cited benchmark of approximately 68. All clinicians approved the DeepSeek-generated caregiver materials and reported that faster first-draft generation could allow greater focus on personalization and direct patient interaction. These findings represent subjective perceptions and should not be interpreted as evidence of objectively improved efficiency. Formal time-motion and implementation studies are needed to determine whether these perceived benefits translate into measurable reductions in documentation or consultation time.

Scenario-specific and advanced analysis

Performance varied by clinical complexity. In high-stakes safety scenarios (E2: Dravet; E8: neonatal), DeepSeek maintained near-perfect safety and accuracy scores, while other models showed vulnerabilities, particularly in precise dosing and medication contraindications. Descriptive scenario-level comparisons are summarized in Table 7.

Table 7. Performance in high stakes vs. counseling scenarios.

Scenario type & examples	Scenarios	ChatGPT	Gemini	DeepSeek	Perplexity	Safety flag rate (all models)
High-stakes safety (Dravet, neonatal)	E2, E8	3.56±0.53	4.56±0.53	5.00±0.00	2.00±0.00	DeepSeek: 0% (0/2); others: 25% (2/8)
Complex counseling (valproate, pregnancy, PNES)	E4, E9, E10	3.43±0.53	4.33±0.58	4.96±0.03	2.00±0.00	DeepSeek: 0% (0/3); others: 17% (2/12)
Standard management (absence, febrile seizure)	E1, E6	3.75±0.42	4.50±0.53	5.00±0.00	2.00±0.00	DeepSeek: 0% (0/2); ChatGPT: 50% (1/2); Perplexity: 100% (2/2)
Surgical/device management (TLE surgery, VNS)	E3, E7	3.45±0.60	4.45±0.60	5.00±0.00	2.00±0.00	DeepSeek: 0% (0/2); others: 13% (1/8)
Comorbidities (ADHD and epilepsy)	E5	3.00±0.00	4.00±0.00	4.94±0.00	2.00±0.00	ChatGPT: 100% (1/1)

Data are presented as mean ± SD. Performance in high stakes vs. counseling scenarios (descriptive comparisons only). Data reflect mean composite scores across scenarios within each category. Formal interaction testing was not performed due to the single-rater-per-response design. ADHD, attention-deficit/hyperactivity disorder; PNES, psychogenic non-epileptic seizures; SD, standard deviation; TLE, temporal lobe epilepsy; VNS, vagus nerve stimulation.

Qualitative synthesis of major concerns

The safety results and analysis of major concerns identified distinct failure patterns across models. No prespecified critical safety flags were identified among the 10 responses evaluated for either DeepSeek or Gemini, while Perplexity had the highest rate (80%). ChatGPT had a 30% flag rate. Thematic analysis of the qualitative "major concerns" field revealed distinct failure patterns:

• ChatGPT (30% safety flag rate): a dangerous recommendation to abruptly discontinue levetiracetam if ADHD symptoms worsened (see the “Primary safety and accuracy outcomes” section).

• Gemini (0% safety flag rate): no critical safety flags were identified. Minor omissions included missing specific dosing ranges or timing details, but no responses contained dangerous or harmful recommendations.

• DeepSeek (0% safety flag rate): no critical safety flags were identified. Responses were consistently comprehensive, accurate, and appropriately nuanced. No responses required safety-related edits.

• Perplexity (80% safety flag rate): the most common and dangerous flaw was hazardous vagueness, particularly failing to name specific medications (e.g., not specifying “sodium channel blockers” in Dravet syndrome, not providing specific dosing for emergency medications) and omitting essential safety details [e.g., no red flags for febrile seizures, no specific magnetic resonance imaging (MRI) precautions for vagus nerve stimulation (VNS)]. This degree of vagueness substantially limited the clinical usefulness of the responses and could pose patient-safety concerns if used without clinician verification.

These findings were consistent with the quantitative safety results and showed that the models had different types of clinically relevant failure.

Exploratory supervised-use classification

For descriptive purposes, we categorized the evaluated systems according to their observed safety and composite scores to illustrate their potential suitability for supervised assistive use within the evaluated scenarios: highest observed performance for supervised assistive use (DeepSeek and Gemini)—indicating comparatively favorable observed performance within the evaluated scenarios, while still requiring real-time clinician verification; conditional acceptability with mandated verification (ChatGPT)—requiring independent validation of all factual claims, dosing, and source citations before any patient consideration; and not supported for supervised clinical use within the evaluated scenarios (Perplexity)—because of the high frequency of prespecified safety flags and clinically important omissions and vagueness. These study-derived categories are exploratory, have not been externally validated, and do not imply autonomous clinical readiness or safety. Any clinical use would require clinician supervision, verification of generated content, and adaptation to the individual clinical context.

Discussion

This study shows that the performance of general-purpose LLMs in pediatric epilepsy varies substantially, particularly in accuracy and safety, extending prior critical reviews of LLM potential in epilepsy care with direct, blinded, clinician-rated comparative benchmarking. These findings provide an initial basis for further evaluating the potential role of these tools in supervised clinical tasks, although further validation in broader clinical settings is needed. The principal finding was that DeepSeek had the highest observed composite score among the evaluated systems in this benchmark, although the study design does not permit definitive separation of model effects from rater effects. Notably, DeepSeek had the lowest numerical SUS score among the four models in phase 1, although the difference was not statistically significant. This did not affect its selection because the prespecified decision rule prioritized safety and composite performance over interface usability. These findings support evaluating accuracy, safety, and usability together when considering LLMs for clinical use. In pediatric epilepsy, fluent language alone is not sufficient if the underlying information is inaccurate or unsafe.

The failure modes identified in this study have direct implications for epilepsy care. For instance, a model prone to fabricating references (a key limitation of ChatGPT in our evaluation) erodes the foundational trust required for therapeutic alliances and could propagate misinformation. Perplexity frequently provided responses that lacked clinically important specificity, including failure to explicitly identify sodium channel blockers as contraindicated in Dravet syndrome. Such omissions could pose patient-safety concerns if the output were used without clinician verification. These findings reinforce the importance of minimizing harm when AI is used in healthcare, particularly in epilepsy, where medication errors or inappropriate advice can have serious consequences (13). Our findings support the use of safety as a primary gatekeeper when considering clinical utility: only 1 of 11 responses with a safety flag (9.1%) was approved for caregiver use, compared to 24 of 29 responses without flags (82.8%).

Our results both contrast and complement other applications of AI in epilepsy. While research has effectively demonstrated the capability of specialized LLMs like Llama 2 to extract structured data (e.g., seizure types, medications) from clinical notes (7), our work evaluates the inverse and equally critical function: generating comprehensible, safe explanations from structured input. This generative capability may be useful for patient education and counseling, which can be time-intensive in chronic disease management. The high performance in “actionable clinical guidance” and “specificity to pediatrics” shown by the top model suggests that, when properly vetted, LLMs can aid in drafting tailored seizure action plans or explaining complex concepts like ketogenic diet mechanisms, thereby extending the reach of the clinician’s expertise (14).

Clinicians identified potential benefits of using the selected model for supervised epilepsy workflows, particularly as a first-draft tool for patient education materials and documentation. They perceived that this could reduce the effort required for initial drafting and allow more time for personalization and discussion with families. However, these observations were subjective, and objective studies are needed to determine whether AI-assisted drafting reduces documentation or consultation time or improves caregiver understanding (14). Overall, these results favor an augmented intelligence approach in which AI assists with drafting while clinicians remain responsible for verification, clinical context, and communication with families. Outputs should be reviewed and adapted to the individual patient’s cognitive status, comorbidities, and family circumstances (15).

Limitations and future directions

This study has several limitations. First, the efficiency-related findings are based on clinician perceptions rather than objective time-motion measurements. Future implementation studies should measure documentation and consultation time directly. Second, although the scenarios covered several clinically important areas, they represent only a limited sample of pediatric epilepsy problems; the utility of LLMs in managing highly complex, refractory, or syndromic cases over time requires longitudinal assessment. Third, the evaluation involved seven clinician evaluators, of whom four completed the full phase 1 quantitative assessment. The small evaluator sample may limit generalizability to other healthcare settings, practice environments, and patient populations. The findings may not apply to primary care settings, resource-limited environments, or healthcare systems with different patient demographics and clinical workflows. Additionally, unlike the blinded phase 1 evaluation, clinicians in phase 2 were aware that DeepSeek had been selected for workflow integration based on the phase 1 results. This may have introduced expectation or confirmation bias in phase 2 usability ratings, approval judgments, and interview responses and should be considered when interpreting these findings. Fourth, while we specified the model versions and access dates (Table 2), because the evaluated models are proprietary and continuously updated, exact replication is difficult. Their training data, model architectures, and fine-tuning processes are also not fully disclosed, and these models are subject to continuous updates without notice. We have deposited all prompts, raw texts, and scoring data in Tables S1-S4 to support replication, but the underlying models themselves cannot be frozen. Future evaluations should specify exact versions and access dates, archive response sets where possible, and replicate assessments following major model updates. Fifth, we did not assess response variability to identical prompts across repeated queries, a known phenomenon in LLMs that can affect reliability in clinical applications. Sixth, the exploratory logistic regression analysis was limited by quasi-complete separation, collinearity between predictors, and a small sample size (n=40), resulting in unstable estimates with wide confidence intervals; these findings should be interpreted as hypothesis-generating rather than definitive evidence of independent predictor effects. Similarly, the safety-approval association is based on a small number of flagged responses (n=11), and the difference in approval rates (73.7 percentage points) should be interpreted cautiously given this small flagged-group sample size. Larger studies are needed to determine whether these associations persist. Seventh, the levetiracetam neonatal dosing example highlights the importance of anchoring safety determinations to cited guidelines. While we have now sourced this to the 2023 ILAE guidelines, future evaluations should prespecify guideline references for all dosing determinations to ensure consistency and reproducibility. The complete adjudication rubric (critical safety checklist, items A1–A6) is provided in Table S4. Eighth, each evaluator scored responses from only one designated model throughout phase 1, meaning that rater and model effects were fully confounded. Any systematic leniency or stringency of an individual rater therefore could not be distinguished from a genuine difference in model performance. Formal inter-rater reliability could not be assessed. Future confirmatory evaluations should use a fully crossed or otherwise independently replicated rating design in which multiple evaluators assess outputs from every model, allowing model and evaluator effects to be estimated separately and enabling formal assessment of inter-rater reliability. A confirmatory study should address these limitations by incorporating a prespecified reference standard, transparent item-level data, reproducible prompts and instruments, repeated model sampling, independent duplicate ratings, clustered statistical analyses that account for the nested data structure, and an independent usability cohort. This design would provide a more reliable estimate of comparative model performance of LLMs for pediatric epilepsy care. Epilepsy-specific prompts and computational guardrails should also be evaluated as potential approaches to improving reliability and safety. Additionally, we note that Perplexity differs from the other evaluated systems because it functions as a multi-model, retrieval-enabled platform rather than a single underlying LLM, which limits direct comparability. Furthermore, future implementation studies should examine caregiver perceptions and preferences regarding transparency about AI use in patient education, as caregivers may either perceive AI assistance as dismissive or, conversely, develop inappropriate trust in unverified AI tools. The results should be viewed as a time-specific assessment rather than a fixed ranking of the models.

Conclusions

In the specialized domain of pediatric epilepsy, the potential utility of freely accessible LLMs depends on their demonstrated performance in specialty-specific tasks. Among the models evaluated, DeepSeek had the highest observed clinician-rated composite score and shared a zero critical safety-flag rate with Gemini. This study provides a preliminary, specialty-specific framework for evaluating LLMs for supervised assistive tasks in epilepsy care, with safety and clinician oversight as central considerations. However, the framework requires prospective validation in larger, multi-center studies before it can be considered validated for broader clinical use. Nonetheless, these findings reflect expert benchmarking and perceived utility, not direct evidence of improved patient outcomes or objectively measured time savings. Future studies should progress from expert benchmarking to prospective evaluation of clinician workflow, caregiver understanding, adherence, and patient-centered outcomes, with clinician oversight and patient safety maintained throughout. Successful implementation will also depend on caregiver trust, transparency, and health literacy.

Supplementary

The article’s supplementary files as

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. This study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments. The study was approved by the Unit of Biomedical Ethics Research Ethics Committee at the Faculty of Medicine at King Abdulaziz University (No. 18-26). Informed consent was obtained from all clinician participants.

Data Sharing Statement

Available at https://tp.amegroups.com/article/view/10.21037/tp-2026-0731/dss

tp-15-10-409-dss.pdf ^{(63.5KB, pdf)}

DOI: 10.21037/tp-2026-0731