Language-dependent performance variation in large language models for dental trauma management: a comparative evaluation of ChatGPT-5.2, Gemini 3.0, and Claude 4.5 Sonnet

Abstract

Background

Large language models (LLMs) are increasingly evaluated for medical question answering and clinical information tasks, yet the impact of query language on their performance in specialized domains such as dental traumatology remains insufficiently studied. The primary objective was to evaluate whether query language (English vs. Turkish) affects LLM performance in a controlled scenario-based assessment of dental trauma management. Secondary objectives were to compare overall performance across three LLMs and to examine whether language effects are uniform across models or model-specific.

Methods

Twenty-seven clinical scenarios covering 13 dental trauma categories were presented to ChatGPT 5.2, Gemini 3.0, and Claude 4.5 Sonnet in both English and Turkish, generating 162 responses. Two blinded endodontists independently evaluated responses using a standardized rubric assessing accuracy (40%), completeness (35%), and safety (25%) against IADT 2020 Guidelines. Inter-rater reliability was assessed using intraclass correlation coefficient (ICC). Language effects were analyzed using Wilcoxon signed-rank tests; model comparisons employed Kruskal-Wallis and Mann-Whitney U tests with Bonferroni correction.

Results

Inter-rater reliability ranged from moderate to good across evaluation dimensions (ICC: 0.738–0.836). ChatGPT showed the strongest language effect with 9.14% higher performance in English (p < 0.001, r = 0.874). Gemini showed moderate English advantage (5.69%, p = 0.003, r = 0.572). Claude exhibited language independence with virtually identical performance in both languages (-0.02%, p = 0.220). In English, significant model differences emerged (H = 22.31, p < 0.001); however, model performance converged in Turkish (H = 2.89, p = 0.236).

Conclusions

This exploratory scenario-based study demonstrates model-specific, rather than universal, language effects on LLM performance in dental trauma management. ChatGPT 5.2 achieved the highest performance in English but exhibited the most pronounced Turkish-language degradation, including substantial safety score decline. Gemini 3.0 showed an intermediate pattern with moderate English advantage. Claude 4.5 Sonnet demonstrated language-independent performance across all evaluated dimensions. These findings are based on a standardized scenario-based assessment and should not be extrapolated to real clinical environments or patient care settings.

Supplementary Information

The online version contains supplementary material available at https://doi.org/10.1186/s12903-026-09191-9.

Introduction

Artificial intelligence has transformed healthcare delivery, with large language models (LLMs) emerging as promising clinical decision support tools [1, 2]. Models such as ChatGPT, Gemini, and Claude demonstrate notable capabilities in medical text processing, clinical decision support, medical education, and diagnostic assistance [3–8].

Traumatic dental injuries (TDIs) present a public health challenge requiring rapid clinical decisions. Approximately 25% of school-age children experience dental trauma, with luxation injuries predominating in primary dentition and crown fractures in permanent teeth [9]. The time-sensitive nature of dental trauma management, particularly in avulsion cases, makes this domain well-suited for AI-assisted decision support [10, 11].

The International Association of Dental Traumatology (IADT) 2020 guidelines serve as the gold standard for TDI management [9, 12]. These guidelines cover 13 distinct injury categories, each requiring specific diagnostic and treatment approaches [12, 13]. Treatment decisions depend on multiple interacting factors: root development stage, extra-alveolar dry time, storage medium, patient age, and injury characteristics [13, 14]. This multifactorial decision-making framework makes dental trauma appropriate for evaluating AI clinical reasoning capabilities.

Research on AI chatbot performance in dental trauma has yielded variable results [15–22]. Earlier studies reported moderate accuracy rates (57.5–76.7%) for models like ChatGPT 3.5 and Google Bard [15, 16]. Recent evaluations of advanced models show improved performance, with accuracy rates ranging from 85% to 100% [17, 18, 20–22]. However, most of these studies used direct questioning approaches where injury types were explicitly stated, which may not adequately assess the diagnostic reasoning capabilities required in clinical practice.

A critical consideration is the potential influence of query language on model performance. LLMs are predominantly trained on English-language corpora, which may create performance disparities when queried in other languages [23, 24]. Studies investigating this phenomenon in medical contexts have yielded mixed findings [25–30]. Several report significant English superiority in dental and medical examinations [25–28], while others found no significant language effect [29]. This inconsistency suggests that language effects may vary across clinical domains and warrants further investigation [30].

Despite growing literature examining AI performance in dentistry, gaps remain in understanding LLM capabilities in dental traumatology. Limited research has systematically compared multiple contemporary LLMs across different languages using clinically relevant scenario-based assessments. Most previous studies employed direct questioning methodologies where the diagnosis is provided, failing to evaluate whether AI systems can independently identify injury types from clinical presentations, a fundamental requirement for clinical decision support. Additionally, evaluation frameworks in prior studies often lacked comprehensive assessment of safety considerations.

The primary objective of this study was to evaluate whether query language (English vs. Turkish) affects LLM performance in dental trauma management. Secondary objectives were: (1) to compare overall performance across three LLMs (ChatGPT 5.2, Gemini 3.0, Claude 4.5 Sonnet) using a weighted rubric assessing accuracy, completeness, and safety; and (2) to examine whether language effects are uniform across models or model-specific. We hypothesized that AI models would perform better when queried in English compared to Turkish, given the predominance of English-language training data. Unlike previous studies that directly specified injury types, this study presented clinical scenarios describing patient symptoms, clinical findings, and radiographic features without explicitly naming the diagnosis. This approach assessed whether LLMs could independently identify traumatic injury types and generate appropriate treatment recommendations aligned with IADT 2020 Guidelines.

Methods

Study design

This comparative cross-sectional study evaluated the performance of three large language models in responding to scenario-based dental trauma questions presented in Turkish and English. The study employed a standardized evaluation framework based on IADT 2020 guidelines as the reference standard. The study protocol was exempt from ethical review as it involved analysis of AI-generated content without human participant involvement.

AI models evaluated

Three contemporary, publicly accessible LLMs were evaluated in their consumer-facing versions: ChatGPT-5.2 (OpenAI, San Francisco, CA, USA), Gemini-3.0 (Google, Mountain View, CA, USA), and Claude Sonnet 4.5 (Anthropic, San Francisco, CA, USA). Each model was accessed through its respective consumer-facing web interface between November 28 and December 10, 2025.

To ensure methodological rigor, a standardized protocol was applied to all model interactions. Each scenario was submitted as a single query in a new, independent conversation session (single-pass approach); no response regeneration, re-submission, or selection among multiple outputs was performed. The first and only response generated for each query was recorded verbatim. Memory and conversation history features were disabled prior to each session. Web search functionality was disabled to ensure responses were generated solely from training data. No system prompts, custom instructions, or pre-configured preferences were applied.

All three models were accessed through their respective consumer-facing web interfaces (chat.openai.com, gemini.google.com, claude.ai). All web interfaces were accessed using newly created accounts with no prior usage history, ensuring that no personalization, recommendation algorithms, or previously learned user preferences could influence the generated responses. In these interfaces, temperature and other sampling parameters are not user-configurable and are set internally by each provider; these values were not visible or accessible to the investigators through the consumer-facing interface. As such, exact temperature values cannot be independently verified or reported, a methodological limitation inherent to LLM evaluations conducted through consumer-facing interfaces. This constraint is shared by similar published studies in dental traumatology and medical AI evaluation that have likewise relied on consumer-facing interface access [15, 16, 18, 26].

The stochastic nature of LLM outputs constitutes a fundamental limitation of this study design: identical prompts may yield different responses across sessions due to non-deterministic generation. This means our findings reflect the performance of each model at the specific time of evaluation and cannot guarantee reproducibility of individual responses, although the patterns observed across 27 scenarios provide a more stable performance estimate than single-query assessments. Screenshots documenting interface settings and model version information are provided as Supplementary Material 2.

Scenario development

Clinical scenarios were developed by the primary investigator (H.Ö.), a board-certified specialist in Endodontics with 6 years of post-specialization experience, academic appointment at a dental faculty, and clinical and academic expertise in dental traumatology. Scenario content was reviewed by the second author (M.D.), also a board-certified endodontist with 6 years of post-specialization experience and academic appointment at a dental faculty, for clinical accuracy and alignment with IADT 2020 guidelines. Clinical scenarios were developed to reflect graduate-level specialist knowledge (endodontist/dental traumatologist level), as all correct management decisions require familiarity with IADT 2020 guidelines in their full complexity, including nuanced decisions such as differential splinting protocols, endodontic timing based on apex maturity, and management of avulsion cases with varying extra-alveolar conditions. Scenarios were not designed for predoctoral (undergraduate) level assessment; simplifications appropriate for dental students were deliberately avoided to provide a challenging benchmark for AI clinical reasoning evaluation.

An initial pool of 30 dental trauma clinical scenarios was developed based on IADT 2020 guidelines covering traumatic injuries to permanent teeth, avulsion, and injuries in primary dentition. Each scenario described a hypothetical patient presentation including demographic information (age, sex), mechanism of injury, time elapsed since trauma, clinical findings (tooth mobility, displacement, percussion sensitivity, pulp testing results), and radiographic features. Scenarios were developed from IADT 2020 guideline content to systematically sample all defined trauma categories and ensure comprehensive guideline coverage, a deliberate content validity strategy. This approach prioritized systematic guideline coverage over ecological validity; the integration of patient medical histories and clinical-experience-based scenario complexity represents the most important methodological advancement for future studies in this field. Importantly, while scenarios were structured around guideline content, each incorporated clinically realistic elements including injury mechanism, time elapsed since trauma, patient age, dentition stage, radiographic findings, and clinical examination details. All clinical scenarios were presented without patient medical history, a deliberate choice to isolate traumatology-specific clinical reasoning, though this constitutes an important design limitation discussed below.

A key feature of this study was the scenario-based approach: unlike previous studies that employed dichotomous questions (Ozden et al. [15]: yes/no format) or open-ended questions requiring single-dimension responses (Kuru et al. [16]; Sezer and Aydoğdu [18]), our scenarios were presented without explicit diagnostic statements and required models to independently identify the injury type from clinical descriptions and then generate comprehensive, multi-step management plans, thereby evaluating both diagnostic reasoning and treatment planning simultaneously.

All scenarios were initially developed in Turkish by the primary investigator with native-level proficiency in both languages. Forward translation into English was performed by the primary investigator. To ensure linguistic equivalence, a modified back-translation procedure was employed: an independent bilingual dental professional translated the English scenarios back into Turkish, and back-translated versions were compared with original Turkish scenarios by a second bilingual evaluator. Discrepancies were resolved through consensus discussion, and minor modifications were made to three scenarios to achieve semantic equivalence. Both reviewers confirmed that final translated scenarios maintained equivalent clinical meaning, difficulty level, and diagnostic complexity across languages.

This study was designed as an exploratory investigation; formal a priori power analysis was not conducted given the absence of established effect size estimates for cross-lingual LLM performance comparisons in dental traumatology at the time of study design. Thirty clinical scenarios were initially developed to ensure comprehensive coverage of all IADT 2020 trauma categories for both permanent and primary dentition; however, three scenarios were subsequently eliminated by the investigators during the pre-evaluation review because they were deemed to permit multiple clinically acceptable management pathways, which could introduce ambiguity into the scoring process. The final sample of 27 scenarios encompassed 13 distinct trauma categories: (1) enamel infraction, (2) enamel fracture, (3) enamel-dentin fracture, (4) enamel-dentin-pulp fracture, (5) crown-root fracture, (6) root fracture, (7) alveolar fracture, (8) concussion, (9) subluxation, (10) extrusive luxation, (11) lateral luxation, (12) intrusive luxation, and (13) avulsion. A minimum of two scenarios per category was ensured to capture intra-category clinical variability (e.g., varying extra-oral times for avulsion, different root development stages for luxation injuries). This approach prioritized content validity over statistical power optimization.

Avulsion was represented by 10 scenarios (37%), a proportion deliberately exceeding epidemiological prevalence, because avulsion management requires integration of multiple time-sensitive clinical variables (apex maturity, extra-alveolar time, storage medium) defined across distinct IADT protocol pathways [13, 14]. This design decision prioritized stress-testing LLM clinical reasoning on the most consequential and algorithmically complex trauma type rather than reflecting injury frequency distributions. The remaining 17 scenarios (63%) covered all 12 other IADT trauma categories with at least 1–2 scenarios each, ensuring comprehensive guideline coverage.

The resulting sample size of 27 scenarios is consistent with, and in many cases exceeds the item counts used in published LLM evaluation studies in dental traumatology: Ozden et al. [15] used 25 dichotomous questions; Kuru et al. [16] used 30 open-ended questions; Sezer and Aydoğdu [18] used 25 open-ended questions; Gökcek Taraç [17] used 31 questions. Readers should interpret our findings as preliminary and exploratory.

Data collection

Data collection was conducted between November 28 and December 10, 2025. Scenarios were submitted in a fixed sequence (Scenarios 1–27) across all models and both languages, with the same order maintained throughout data collection. Each scenario was submitted to all three LLMs in both Turkish and English, generating a total of 162 AI responses (27 scenarios × 3 models × 2 languages). To minimize potential order effects, each scenario was submitted as an independent conversation session with memory disabled, ensuring no carryover between scenarios. Responses were collected and verbatim stored for subsequent evaluation. Model outputs were de-identified and randomized prior to evaluation to ensure assessor blinding.

The 12-day data collection period was selected to balance practical constraints with methodological rigor. No major model updates were announced by any provider during the data collection window. To assess model stability during the data collection window, a subset of five scenarios (Scenarios 3, 9, 15, 21, 27) was re-submitted to all three models in both languages at two time points: at the start of data collection (November 28, 2025) and at the end of the collection period (December 10, 2025), representing a 12-day interval. The five scenarios were purposively selected to represent different trauma categories (complicated crown fracture, root fracture, lateral luxation, avulsion with immature tooth, and avulsion with mature tooth). Response patterns on these repeated submissions were compared with original responses by the primary investigator; results are reported in the Results section.

Evaluation framework

A comprehensive evaluation rubric was developed based on IADT 2020 guidelines and refined through iterative expert discussion. Model responses were evaluated across three dimensions:

Accuracy (weight: 40%)

Assessed concordance between AI responses and IADT 2020 guidelines. Evaluation criteria included correctness of diagnosis, appropriateness of treatment approach, splint type and duration, antibiotic selection and dosing (when indicated), endodontic timing recommendations, and follow-up protocols.

Completeness (weight: 35%)

Evaluated whether responses included all essential components specified in guidelines for each injury type. Required elements varied by trauma category and included immediate management steps, repositioning techniques (for luxation injuries), splinting protocols, pharmacological recommendations, pulp vitality monitoring protocols, and follow-up schedules.

Safety (weight: 25%)

Assessed absence of potentially harmful recommendations and presence of critical safety considerations. Critical violations included recommending replantation of primary teeth, prescribing tetracycline-class antibiotics to patients under 12 years of age, recommending rigid splints where flexible splints are indicated, or suggesting unnecessary endodontic treatment in immature teeth with revascularization potential.

Each dimension was scored on a 0–5 point scale (0 = unacceptable, 1 = poor, 2 = inadequate, 3 = acceptable, 4 = good, 5 = excellent). Specific point deductions were established for protocol violations categorized as critical (− 2 points), serious (− 1 point), moderate (− 0.5 points), or minor (− 0.25 points) based on clinical significance.

The total weighted score was calculated using the formula: Total Weighted Score = (Accuracy × 0.40) + (Completeness × 0.35) + (Safety × 0.25). The weighting scheme was established a priori through expert consensus among the authors, drawing on multidimensional evaluation frameworks employed in comparable LLM assessment studies in dentistry and medicine [7, 18, 19]. Accuracy was assigned the highest weight as correct identification of the injury type and selection of the appropriate treatment pathway is a prerequisite for all subsequent management decisions; errors in diagnosis render all other recommendations potentially harmful regardless of their completeness. Completeness was weighted second, reflecting that incomplete management, such as omitting splinting protocols, follow-up schedules, or pulp vitality monitoring, directly affects clinical outcomes. Safety received the lowest absolute weight (25%) because our scoring system included categorical penalties: critical safety violations (e.g., recommending replantation of primary teeth, prescribing tetracycline-class antibiotics to patients under 12 years) incurred − 2-point deductions, effectively creating floor effects that substantially reduced total scores independent of weighting. Thus, safety violations were weighted both explicitly through dimension scoring and implicitly through the penalty system.

Point deductions were categorized by clinical severity: critical violations (− 2 points) represented recommendations that could directly cause patient harm or contradict absolute IADT contraindications; serious violations (− 1 point) represented significant protocol deviations with likely clinical consequences; moderate violations (− 0.5 points) represented incomplete but not harmful recommendations; minor violations (− 0.25 points) represented suboptimal but acceptable deviations. This hierarchical classification was adapted from severity-graded scoring frameworks used in clinical AI evaluation research [19, 26] and tested during the rater calibration session. The complete scoring rubric with dimension-specific criteria, required elements per trauma category, and violation examples is provided as Supplementary Material 1 and is available from the corresponding author upon request.

Rater selection and training

Two endodontists, each with more than 5 years of clinical specialization experience, served as independent evaluators. Prior to the assessment phase, both raters participated in a calibration session where they independently evaluated a subset of five responses not included in final analysis. Discrepancies were discussed, and consensus was reached on scoring criteria interpretation. Raters were blinded to model identity and query language throughout evaluation.

Consensus process

Following independent evaluation, responses where rater scores differed by ≥ 0.5 points on any dimension were identified for consensus review. In these cases, both raters reviewed the response and relevant IADT guidelines together and reached a mutually agreed-upon consensus score. Consensus scores were used for all subsequent analyses.

Statistical analysis

Statistical analyses were performed using JASP software (Version 0.95.4). Statistical significance was set at α = 0.05 for all tests.

Inter-rater reliability was assessed using intraclass correlation coefficient calculated with a two-way random effects model for absolute agreement (ICC [1, 2]). Descriptive statistics including means, standard deviations, medians, and interquartile ranges were calculated for all performance metrics stratified by model and language. Given the ordinal nature of scoring data and non-normal distribution confirmed by Shapiro-Wilk tests, non-parametric methods were employed. Kruskal-Wallis H tests compared performance across models, with post-hoc pairwise comparisons using Mann-Whitney U tests with Bonferroni correction (adjusted α = 0.0167). Wilcoxon signed-rank tests compared paired English and Turkish responses within each model. Effect sizes were interpreted as small (r < 0.3), medium (0.3–0.5), or large (> 0.5). ICC values were interpreted according to Koo and Li criteria: <0.50 poor, 0.50–0.75 moderate, 0.75–0.90 good, > 0.90 excellent reliability.

Results

Inter-rater reliability

Analysis demonstrated adequate inter-rater reliability across all evaluation dimensions (Table 1). Per Koo and Li criteria, ICC values indicated good reliability for completeness (ICC = 0.836, 95% CI: 0.783–0.877) and accuracy (ICC = 0.783, 95% CI: 0.714–0.838), with moderate reliability for safety (ICC = 0.738, 95% CI: 0.659–0.802). While safety fell within the moderate range (0.50–0.75), this value remained above acceptable thresholds for clinical research and was consistent with inherent subjectivity in evaluating safety considerations in clinical recommendations. These reliability coefficients support the validity of the evaluation framework.

Table 1.

Inter-rater reliability assessment

Dimension	ICC	95% CI	Interpretation
Accuracy	0.783	0.714–0.838	Good
Completeness	0.836	0.783–0.877	Good
Safety	0.738	0.659–0.802	Moderate

Interpretation based on Koo and Li criteria

ICC Intraclass Correlation Coefficient

n = 162 evaluations

Language effect on model performance

The primary hypothesis examined whether LLM performance varies based on the language of clinical scenarios. Wilcoxon signed-rank tests revealed distinct language-dependent patterns across the three models (Fig. 1, Table 2).

Fig. 1.

Language effect on LLM performance in dental trauma management. Bar chart comparing total weighted scores between English and Turkish scenarios for each AI model. Error bars represent standard deviation. Statistical significance was determined by the Wilcoxon signed-rank test (***p < 0.001, **p < 0.01; ns = not significant)

Table 2.

Language effect analysis by model (expanded)

Model	English (Mean ± SD)	95% CI	Turkish (Mean ± SD)	95% CI	Diff (%)	p-value	Effect (r)
ChatGPT	4.856 ± 0.144	[4.798, 4.913]	4.431 ± 0.191	[4.355, 4.507]	+ 9.14	< 0.001***	0.874
Gemini	4.744 ± 0.364	[4.600, 4.888]	4.481 ± 0.420	[4.315, 4.648]	+ 5.69	0.003**	0.572
Claude	4.408 ± 0.459	[4.227, 4.590]	4.409 ± 0.407	[4.248, 4.570]	-0.02	0.220	0.236

Values presented as mean ± SD. Wilcoxon signed-rank test for paired comparisons. p < 0.001, p < 0.01. Effect size: small (r < 0.3), medium (r = 0.3–0.5), large (r > 0.5). η² interpretation: < 0.01 = trivial, 0.01–0.06 = small, 0.06–0.14 = medium, > 0.14 = large.*

ChatGPT demonstrated the most pronounced language effect, with higher performance in English (4.856 ± 0.144) compared to Turkish (4.431 ± 0.191), representing a 9.14% performance differential (W = 0.0, p < 0.001, r = 0.874). This large effect size indicates substantial language-dependent variation, with English responses consistently outperforming Turkish counterparts across all 27 scenarios.

Gemini exhibited a moderate language effect, scoring 4.744 ± 0.364 in English versus 4.481 ± 0.420 in Turkish, a 5.69% difference that reached statistical significance (W = 58.5, p = 0.003, r = 0.572). While less pronounced than ChatGPT, this pattern indicates meaningful performance variation attributable to language.

Claude demonstrated language independence, achieving virtually identical scores in English (4.408 ± 0.459) and Turkish (4.409 ± 0.407). The negligible difference of -0.02% was not statistically significant (W = 138.0, p = 0.220, r = 0.236), indicating that Claude’s clinical reasoning capabilities were unaffected by language of presentation (Fig. 2). This finding suggests fundamentally different multilingual processing architectures among the evaluated models.

Fig. 2.

Language effect magnitude expressed as percentage difference between English and Turkish performance. Positive values indicate English superiority; near-zero values indicate language independence

Overall model performance comparison

Kruskal-Wallis analysis revealed significant differences among models when combining both language conditions (H = 11.45, p = 0.003, η² = 0.071, medium effect) (Table 3). ChatGPT achieved highest overall mean score (4.643 ± 0.270), followed by Gemini (4.613 ± 0.407) and Claude (4.409 ± 0.426). Post-hoc pairwise comparisons with Bonferroni correction showed no significant difference between ChatGPT and Gemini (U = 1388.0, p = 0.667), while both outperformed Claude (ChatGPT vs. Claude: U = 1908.0, p = 0.006; Gemini vs. Claude: U = 1951.5, p = 0.002).

Table 3.

Model comparison results (Kruskal-Wallis H test)

Comparison	H	n	η²	Interpretation
Overall model comparison	11.45	162	0.071	Medium
Model comparison English	22.31	81	0.279	Large
Model comparison Turkish	2.89	81	0.036	Small

Language-stratified analysis revealed divergent patterns. In English, significant model differences were observed (H = 22.31, p < 0.001, η² = 0.279, large effect), whereas Turkish responses showed no significant differences among models (H = 2.89, p = 0.236, η² = 0.036, small effect) (Table 3). The large effect size for English model comparisons indicates that model selection accounts for approximately 28% of performance variance in English, a substantial proportion that underscores the practical importance of model choice for English-language dental trauma queries. In contrast, model selection explained only 3.6% of variance in Turkish, suggesting that in non-English contexts, language-related factors predominate over model-specific capabilities, effectively compressing inter-model performance differences.

Dimension-specific performance

Analysis of individual evaluation dimensions revealed consistent patterns across accuracy, completeness, and safety metrics (Fig. 3, Table 4). The language effect was most pronounced in completeness, where ChatGPT showed a 0.500-point decrease from English to Turkish, followed by Gemini (0.426-point decrease). Claude maintained stable completeness scores across languages (difference: 0.019 points).

Fig. 3.

Heatmap of performance across evaluation dimensions and languages. Color intensity represents consensus scores (green = higher, red = lower). Vertical white lines separate models

Table 4.

Performance by evaluation dimension, model, and language

Dimension	Model	English (Mean ± SD)	Turkish (Mean ± SD)	Difference
Accuracy	ChatGPT	4.833 ± 0.181	4.435 ± 0.224	+ 0.398
Accuracy	Gemini	4.657 ± 0.392	4.426 ± 0.446	+ 0.231
Accuracy	Claude	4.296 ± 0.488	4.315 ± 0.432	-0.019
Completeness	ChatGPT	4.778 ± 0.200	4.278 ± 0.253	+ 0.500
Completeness	Gemini	4.759 ± 0.380	4.333 ± 0.453	+ 0.426
Completeness	Claude	4.259 ± 0.489	4.241 ± 0.439	+ 0.019
Safety	ChatGPT	5.000 ± 0.000	4.639 ± 0.310	+ 0.361
Safety	Gemini	4.861 ± 0.291	4.778 ± 0.330	+ 0.083
Safety	Claude	4.796 ± 0.356	4.796 ± 0.356	0.000

All values represent consensus scores. Positive differences indicate higher English performance

Safety performance demonstrated a ceiling effect in English, particularly for ChatGPT which achieved perfect safety scores (5.000) across all 27 English scenarios. However, Turkish responses revealed more variability, with ChatGPT’s safety scores declining to 4.639 (only 25.9% of scenarios achieving perfect safety scores, compared to 100% in English). Claude exhibited the most consistent safety performance across languages (English: 4.796, Turkish: 4.796), with comparable rates of perfect safety scores (66.7% vs. 59.3%).

Response consistency

Analysis of response variability, measured by standard deviation of total weighted scores, revealed notable differences in consistency among models. ChatGPT demonstrated highest consistency with lowest variability (combined SD = 0.270), though this uniformity was partially attributable to a ceiling effect in English responses (SD = 0.144). Gemini and Claude showed comparable variability (combined SD = 0.407 and 0.426, respectively), indicating more heterogeneous performance across clinical scenarios.

Clinical acceptability

Clinical acceptability, defined as responses achieving total weighted scores ≥ 4.5, further illuminated performance patterns. In English, ChatGPT achieved 100% acceptability (27/27), compared to 81.5% for Gemini (22/27) and 51.9% for Claude (14/27). Claude’s acceptability rate improved in Turkish (59.3%, 16/27) while ChatGPT’s declined substantially (44.4%, 12/27), suggesting that language-related performance degradation disproportionately affects clinical utility for some models.

Model stability verification

Spot checks conducted by re-submitting five scenarios at the beginning and end of the data collection period yielded response patterns consistent with those obtained during the main collection phase, supporting model stability throughout the 12-day study window.

Discussion

This study provides evidence regarding language-dependent performance variation in large language models for dental trauma management. Our findings demonstrate that these variations are model-specific rather than universal, with ChatGPT 5.2 showing strong English superiority (9.14%, p < 0.001), Gemini 3.0 exhibiting moderate English advantage (5.69%, p = 0.003), and Claude 4.5 Sonnet demonstrating complete language independence (-0.02%, p = 0.220).

Our findings regarding ChatGPT’s English superiority align with recent literature. Studies have reported significant English advantage across dental and medical examinations [25–27]. Wójcik et al. found that Claude demonstrated the most consistent cross-lingual performance, paralleling our observation of Claude’s language independence [28]. However, Sozen Yanik et al. reported no significant language effect in maxillofacial prosthodontics, suggesting that language effects may vary across clinical domains [29]. The language-stratified analysis revealed that significant model differences observed in English (H = 22.31, p < 0.001) disappeared in Turkish (H = 2.89, p = 0.236). This convergence suggests that language-related degradation disproportionately affects high-performing models, effectively equalizing capabilities in non-English contexts. ChatGPT achieved the highest overall mean score (4.643 ± 0.270), consistent with recent benchmarks showing continued advancement in LLM capabilities from earlier reports of 57.5% accuracy [15] to current levels exceeding 85% [22, 31, 32]. Our lower diagnostic accuracy rates compared to studies reporting 100% [21] likely reflect our scenario-based approach requiring independent injury identification.

Safety performance in our study revealed clinically important patterns. ChatGPT achieved perfect safety scores (5.0/5.0) across all 27 English scenarios but showed marked safety degradation in Turkish (mean: 4.639, with only 25.9% of scenarios achieving perfect scores vs. 100% in English). This pattern warrants particular clinical attention, as it suggests that language-related degradation preferentially compromises safety-critical recommendations. Review of specific safety violations in Turkish ChatGPT responses revealed the following patterns: in avulsion scenarios involving primary teeth, Turkish responses showed reduced specificity regarding the contraindication of replantation; antibiotic prescribing recommendations in Turkish responses demonstrated greater variability, with some responses omitting age-specific dosing considerations; and splinting duration and type specifications showed greater imprecision in Turkish compared to English responses for lateral luxation scenarios. For example, in Scenario 18 (avulsion of an immature permanent incisor in an 8-year-old), the Turkish ChatGPT response recommended “Doksisiklin (yaş uygunluğu dikkate alınarak)” [doxycycline, considering age appropriateness] as an alternative antibiotic, using hedging language rather than explicitly stating the IADT contraindication for tetracycline-class antibiotics under age 12. The English response for the same scenario did not include this ambiguity. Similarly, in Scenario 21 (mature tooth avulsion, extended dry time), the Turkish response specified antibiotic dosing only as “erişkin dozunda” [adult dose] without numeric weight-based parameters present in the English version. Taken together, these examples confirm a systematic, language-dependent safety degradation rather than isolated instances. Claude demonstrated the most consistent safety performance across languages (English: 4.796, Turkish: 4.796, identical across conditions), suggesting that its multilingual training may preserve safety-critical information more uniformly than accuracy or completeness dimensions. These findings have direct implications for clinical implementation planning. A model that achieves high accuracy in its primary language but shows substantial safety degradation in a second language should not be used in multilingual settings without language-specific validation studies.

The evaluation of safety in our study was necessarily limited to contraindications defined by IADT 2020 guidelines independent of individual patient factors: replantation of primary teeth, tetracycline-class antibiotics in patients under 12 years (as specified in each scenario’s demographic information), rigid splinting in cases requiring flexible splinting, and unnecessary endodontic intervention in teeth with revascularization potential. Patient-specific medication contraindications (drug allergies, systemic disease interactions) could not be evaluated because medical histories were excluded from scenarios. This represents an important boundary condition of our safety findings.

The high performance observed in English scenarios warrants cautious interpretation. These scores reflect performance in a controlled evaluation against IADT guidelines using hypothetical clinical scenarios. Performance in actual clinical environments, involving real patients, time pressure, incomplete histories, and radiographic uncertainty, may differ substantially. Our findings should be regarded as a preliminary benchmark rather than evidence of clinical applicability or readiness for deployment. Haupt et al. demonstrated that ChatGPT significantly improved student accuracy in dental trauma assessments, supporting educational utility [33].

For Turkish-speaking dental practitioners, our findings suggest a nuanced model selection framework. Claude demonstrated statistically equivalent performance in Turkish and English (mean: 4.409 vs. 4.408, p = 0.220), making it the most language-consistent option. However, Claude’s lower absolute accuracy in English (4.408 vs. ChatGPT’s 4.856) means that Turkish-speaking practitioners who also work in multilingual settings face a trade-off between consistency and peak performance. Gemini represents an intermediate position with moderate English advantage and Turkish performance (4.481) comparable to Claude. ChatGPT’s substantially higher English performance does not translate reliably to Turkish contexts, as evidenced by both the 9.14% performance differential and the dramatic safety score degradation. Based solely on our scenario-based findings, Turkish-speaking clinicians would be best served by Claude for routine language consistency, or by ChatGPT only with explicit awareness that responses should be verified with particular care. These recommendations are preliminary and must be confirmed by studies incorporating practitioner validation and real patient case data.

The medicolegal implications of LLM-assisted clinical decision-making in dental trauma represent an emerging and unresolved area of healthcare policy and law. When a clinician uses an AI recommendation that results in patient harm, questions of responsibility involve multiple parties: the model developer, the platform operator, and the treating clinician. Regulatory frameworks in most jurisdictions are still adapting to AI-assisted clinical decisions, and inconsistency in LLM outputs, particularly across languages as demonstrated in our study, poses additional challenges for establishing clear standards of care [23]. The clinician retains professional and legal responsibility for all clinical decisions regardless of AI assistance, meaning that an incorrect LLM recommendation does not transfer liability away from the treating professional. Our findings, particularly ChatGPT’s safety degradation in Turkish, underscore that clinicians using LLMs in their non-primary-training-language settings may be exposed to a higher risk of AI-generated errors without necessarily recognizing the degradation. We recommend that institutional policies governing LLM use in clinical settings explicitly address: (a) language-specific validation requirements, (b) mandatory clinician oversight for all AI-generated recommendations, and (c) documentation practices for AI-assisted decisions.

This study offers several methodological strengths. The scenario-based approach requiring independent injury identification more closely approximates real-world clinical decision-making compared to studies using direct questioning where the diagnosis is provided. The blinded, dual-rater evaluation with consensus process enhances reliability and reduces individual assessor bias. The comprehensive evaluation framework incorporating accuracy, completeness, and safety dimensions provides a more holistic assessment of clinical utility than accuracy metrics alone. The direct language comparison within identical scenarios controls for content-related confounders that could affect cross-study comparisons. This is among the first studies to systematically evaluate Claude’s multilingual performance in dental traumatology, revealing its unique language-independent characteristics.

Several limitations should be acknowledged. First, the study evaluated only two languages; findings may not generalize to other language pairs. Second, evaluation was conducted at a single time point, and model performance may change with updates. Third, we conducted single-pass evaluations. Given the stochastic nature of LLM outputs, identical prompts may produce different responses across sessions. While this limitation is inherent to current LLM evaluation methodologies and is shared by similar published studies in the field [15, 16, 19], it means our results represent a snapshot of model performance rather than a stable absolute benchmark. Future studies should report test-retest reliability across multiple evaluation sessions. Fourth, the exploratory design and sample of 27 scenarios may limit statistical power for subgroup analyses (e.g., per-category or per-injury-type comparisons). Future confirmatory studies with formal a priori power calculations and larger scenario sets are warranted. Fifth, all clinical scenarios were presented without patient medical history. While this was a deliberate choice to isolate traumatology-specific clinical reasoning, it constitutes the most significant design limitation of this study. The absence of medical history precluded evaluation of patient-specific safety considerations, including antibiotic contraindications in patients with drug allergies, hepatic impairment, or concurrent medications, and age-appropriate dosing adjustments beyond those specified in each scenario’s demographic information. Future scenario-based evaluations should incorporate medical history variables as a priority, as this represents the most important avenue for strengthening the ecological validity of LLM dental trauma assessments. Sixth, the scenario distribution does not reflect epidemiological prevalence of dental trauma types; avulsion was intentionally overrepresented to evaluate model performance on the most complex IADT protocols. Future studies should consider scenario distributions weighted by epidemiological prevalence to assess whether LLM performance patterns differ when evaluated across injury types proportional to their clinical frequency. Seventh, although formal criterion validation (e.g., Delphi consensus) was not performed, the rubric demonstrates face validity (developed by two specialist endodontists from IADT guidelines), content validity (comprehensive coverage of all guideline-defined decision points), and inter-rater reliability (ICC 0.738–0.836), providing an adequate psychometric foundation for this exploratory study. Formal multi-expert Delphi validation represents a recommended next step. Eighth, the consistency assessment was based on 30 response-pair comparisons (5 scenarios × 3 models × 2 languages) spanning the full 12-day data collection window. While a complete test-retest evaluation of all 162 scenario-model-language combinations would provide more definitive evidence, no published dental traumatology LLM evaluation study has reported test-retest reliability data, making even limited temporal consistency verification a methodological contribution. We recommend that future studies incorporate full test-retest protocols with intraclass correlation analysis of repeated LLM responses as a standard methodological component of AI validation research. Multimodal capabilities and domain-specific models such as Dental Trauma Evo, which achieved 85.43% accuracy, were not evaluated [20].

The human-AI comparison represents the most important follow-up to this work. Future studies should incorporate responses from dental clinicians with varying expertise levels (undergraduate students, general practitioners, endodontic specialists) to contextualize LLM performance scores against human benchmarks. Such comparisons, as demonstrated by Haupt et al. [33] in a medical context, would clarify whether the performance levels observed in our study are clinically meaningful and would inform evidence-based recommendations for or against LLM use in dental trauma management. We are actively planning such a study building on the cross-lingual baseline established here. The performance convergence observed in Turkish across all three models raises an intriguing question: would retrieval-augmented generation (RAG) architectures, where models access a curated Turkish-language IADT 2020 knowledge base at inference time, reduce language-dependent performance gaps? If language effects are partially attributable to underrepresentation of Turkish clinical terminology in training corpora, RAG systems using Turkish-language dental trauma references could potentially equalize performance across language conditions. This represents a promising direction for applied LLM development in multilingual dental education and clinical decision support. Future research should also expand language comparisons, conduct longitudinal tracking of model updates, and integrate multimodal inputs.

Conclusions

This exploratory scenario-based study demonstrates model-specific, rather than universal, language effects on LLM performance in dental trauma management. In controlled scenario-based evaluations, the following patterns emerged: ChatGPT 5.2 achieved the highest performance in English scenarios but exhibited the most pronounced Turkish-language degradation, including substantial safety score decline; Gemini 3.0 showed an intermediate pattern with moderate English advantage and consistent safety performance; Claude 4.5 Sonnet demonstrated language-independent performance across all evaluated dimensions.

Based on these preliminary findings: for English-primary clinical settings, ChatGPT offers the highest performance ceiling; for Turkish-speaking or multilingual clinical contexts, Claude’s language consistency may offset its lower absolute performance. All models demonstrated clinically acceptable performance in English scenarios. These findings are based on a standardized scenario-based assessment and should not be extrapolated to real clinical environments or patient care settings. Further prospective validation studies involving practicing clinicians and real patient outcomes are necessary before any clinical deployment recommendations can be made. Model deployment in any clinical context requires institutional oversight, clinician validation of AI recommendations, and language-specific performance assessment beyond the scenario-based benchmark presented here.

Authors’ contributions

H.Ö. conceived and designed the study, developed the clinical scenarios, performed the translations, collected the data, conducted the statistical analysis, interpreted the results, and drafted the manuscript. M.D. contributed to data collection and served as an independent rater for response evaluation. Both authors read and approved the final manuscript.

Funding

This research received no specific funding.

Data availability

The raw scoring data, clinical scenarios (in both Turkish and English), and the complete evaluation rubric are available from the corresponding author upon reasonable request and will be deposited in an open-access repository upon acceptance. The complete evaluation rubric and clinical scenarios are provided as Supplementary Materials.

Declarations

Ethics approval and consent to participate

Not applicable. This study analyzed AI-generated responses to fictional clinical scenarios without human participant involvement or use of patient data.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.