Comparative assessment of the accuracy of different artificial intelligence models in answering analytical and knowledge-based questions in oral and maxillofacial radiology and oral and maxillofacial surgery

Abstract

Background

Artificial intelligence models are increasingly used in healthcare education; however, their ability to handle both factual knowledge and analytical clinical reasoning in dentistry remains unclear. This study aimed to compare the accuracy of different AIs in answering knowledge-based and analytical multiple-choice questions in Oral and Maxillofacial Radiology (OMFR) and Oral and Maxillofacial Surgery (OMFS), and to evaluate performance differences according to cognitive task type.

Methods

This cross-sectional comparative study analyzed 258 multiple-choice questions from the Turkish Dental Specialty Examination (DUS) conducted between 2012 and 2021 (202 knowledge-based, 56 analytical). Five AI models (ChatGPT-5.2 Go, ChatGPT-5.2 Plus, DeepSeek V3, Claude Sonnet 4.5, and Gemini 3 Flash) answered all questions under default settings in a single session. Accuracy rates were compared using Chi-square and Kruskal–Wallis tests with Bonferroni correction. Inter-model agreement and reliability were assessed using Cohen’s kappa and the intraclass correlation coefficient (ICC) (α = 0.05).

Results

Significant differences among models were observed in knowledge-based questions (p = 0.048), analytical questions (p = 0.032), and overall accuracy (p = 0.006). Gemini achieved the highest accuracy in knowledge-based questions, while Claude demonstrated the lowest performance. Although a general difference was detected in analytical questions, pairwise comparisons did not show clear model superiority. Overall performance largely reflected success in knowledge-based tasks. Agreement analysis showed low kappa values (κ = 0.226–0.339) but moderate ICC levels (0.597–0.728).

Conclusions

AIs demonstrate strong factual recall but remain limited in analytical clinical reasoning tasks. While these models may serve as supportive tools in dental education, their use as independent clinical decision-making systems is not yet reliable.

Introduction

Artificial intelligence (AI) is a rapidly evolving field that focuses on developing intelligent systems capable of performing tasks traditionally associated with human intelligence and has increasingly contributed to facilitating daily life over the past two decades [1]. AI technologies have also progressed significantly within the health sciences, particularly in medicine and dentistry, and AI systems have attracted growing interest due to their potential applications in academic assessment and clinical reasoning. This development has provided various advantages for students and educators, leading to an expanding role of AI in medical and dental education [2].

AI models and neural networks can process large datasets efficiently, enabling objective and comprehensive clinical and histopathological analyses, which may contribute to improved treatment strategies and prognostic outcomes [3, 4]. However, there is limited research evaluating how accurately these AI models interpret theoretical knowledge and how their performance in multiple-choice examinations compares with that of human candidates.

The present study aims to compare the answer accuracy of different AI models in responding to all Oral and Maxillofacial Radiology and Oral and Maxillofacial Surgery questions asked in the Turkish Dental Specialty Examination (DUS) between 2012 and 2021, a high-stakes national exam that assesses both theoretical and clinical knowledge. In addition, the study evaluates and compares the performance of these AI models in answering analytic-based and knowledge-based questions.

Oral and Maxillofacial Surgery (OMFS) is a specialized discipline concerned with the diagnosis, treatment, and surgical management of diseases affecting the oral cavity, teeth, jaws, facial skeleton, and associated soft tissues. This field encompasses a wide clinical spectrum, including odontogenic infections, jaw cysts and tumors, dentofacial deformities, temporomandibular joint (TMJ) disorders, facial trauma, and implant surgery [3]. In addition, congenital anomalies, orthognathic surgical procedures, and preprosthetic surgical interventions constitute important areas of practice within this specialty. OMFS adopts a multidisciplinary approach aimed at preserving functional, aesthetic, and biological integrity through comprehensive surgical care. Recent technological advancements, particularly in regenerative surgical techniques, have led to more predictable and successful outcomes in both diagnostic and therapeutic processes [3, 5, 6].

Oral and Maxillofacial Radiology (OMFR) is one of the fundamental disciplines of dentistry that contributes to diagnosis, treatment planning, and follow-up through imaging of the teeth, jaws, temporomandibular joint, and surrounding craniofacial structures [7]. In many situations where clinical examination is insufficient, radiographic investigations enable early detection of pathologies, assessment of anatomical variations, and prediction of potential complications [8]. In this respect, OMFR has an indispensable role in preventive dentistry as well as in restorative, endodontic, periodontal, orthodontic, and surgical procedures.

Key considerations in diagnostic imaging include selecting the appropriate imaging modality, maintaining radiation exposure at the lowest reasonably achievable level, accurately interpreting the acquired images, and correlating radiographic findings with clinical signs and symptoms [8, 9]. Furthermore, distinguishing artifacts and anatomical variations from true pathological findings is crucial to prevent misdiagnosis and unnecessary treatment. Therefore, oral and maxillofacial radiology represents a specialty that extends beyond image acquisition, requiring strong theoretical knowledge, clinical expertise, and a multidisciplinary approach [9, 10].

In this study, the performance of five different AI models (ChatGPT-5.2 Go, ChatGPT-5.2 Plus, DeepSeek V3 API, Claude Sonnet 4.5, and Gemini 3 Flash) was compared in answering multiple-choice questions related to Oral and Maxillofacial Radiology and Oral and Maxillofacial Surgery. These models belong to the category of Large Language Models (LLMs), which are capable of analyzing and responding to complex queries through natural language processing (NLP) algorithms [11].

In recent years, LLMs such as ChatGPT, Gemini, DeepSeek and Claude have advanced substantially, offering increasingly sophisticated capabilities in text generation, reasoning, and information retrieval. These models differ in terms of architecture, access options, and performance characteristics. For example, ChatGPT-4o, released by OpenAI in May 2025, is a multimodal model available through paid access. Gemini 2.5 Flash, introduced by Google in June 2025, is known for its speed and efficiency and is available free of charge. DeepSeek V3, launched in January 2024, has emerged as a powerful open-source alternative, while Claude Sonnet 4, released by Anthropic in May 2025, is optimized for advanced reasoning and is freely accessible. The increasing integration of such models into medical and dental practice, understanding their practical applications and limitations has become increasingly important [12]. Accordingly, the most recent versions of these models (ChatGPT-5.2 Go, ChatGPT-5.2 Plus, DeepSeek V3 API, Claude Sonnet 4.5, and Gemini 3 Flash) were used in this study.

Materials and methods

This study was conducted in Türkiye on January 27, 2026. The questions from the Turkish Dental Specialty Examination (DUS) were presented in their original language (Turkish) to various artificial intelligence models, including ChatGPT-5.2 Go, ChatGPT-5.2 Plus, DeepSeek V3 API, Claude Sonnet 4.5, and Gemini 3 Flash. The DUS exam is a national examination organized by the Assessment, Selection, and Placement Center (ÖSYM) to evaluate candidates seeking admission to dental specialty training programs at universities in Türkiye.

Between 2015 and 2021, the examination was administered once annually, whereas it was conducted twice yearly (spring and autumn) between 2012 and 2014. From 2023 onward, the DUS exam (Turkish Dental Specialty Examination) has again been administered twice per year. The examination consists of a total of 120 multiple-choice questions, including 40 basic science and 80 clinical science questions. Each year, the clinical sciences section includes a fixed number of 20 questions related to Oral and Maxillofacial Radiology and Oral and Maxillofacial Surgery, with 10 questions allocated to each discipline. Accordingly, 13 examinations and for each exam 20 specialty-related questions yielded an initial dataset of 260 multiple-choice questions. After review of the official examination archive, two Oral and Maxillofacial Surgery questions were excluded because they had been annulled by the examination authority following the examinations. Therefore, the final study sample consisted of 258 questions, including 130 Oral and Maxillofacial Radiology questions and 128 Oral and Maxillofacial Surgery questions.

In the present study, all Oral and Maxillofacial Radiology and Oral and Maxillofacial Surgery questions from the clinical sciences sections of the DUS examinations administered between 2012 and 2021 were analyzed to evaluate the answering performance of AI-based models. The questions were obtained from the official website of ÖSYM, where they are publicly available. All questions were provided in Turkish.

Each question was entered individually into the official web interfaces of the AI models during memory-disabled sessions. All inputs were performed independently to avoid cross-contamination between responses. A cross-sectional and comparative study design was employed. A total of 260 multiple-choice questions related to Oral and Maxillofacial Radiology and Oral and Maxillofacial Surgery from the clinical sciences sections of 13 DUS examinations administered between 2012 and 2021 were obtained from the official website of the Assessment, Selection, and Placement Center (ÖSYM) (https://www.osym.gov.tr/TR,15070/dus-cikmis-sorular.html) [13]. As this study did not involve human or animal subjects, ethical committee approval was not required.

The primary aim was to compare overall examination performance among AI systems under standardized conditions, analyses were conducted at the model level. Nevertheless, since all models responded to the same examination items, potential within-question dependency should be considered when interpreting inferential statistics.

Questions were independently classified by two researchers as either analytical-based (requiring clinical scenario interpretation and decision-making) or knowledge-based (requiring direct theoretical knowledge) according to predefined criteria. Questions were classified according to their primary cognitive demand. Knowledge-based questions were defined as items that could be answered predominantly through recall of factual information, theoretical concepts, anatomical knowledge, diagnostic criteria, radiographic characteristics, treatment indications, or guideline-based information without requiring interpretation of a clinical scenario. Analytical questions were defined as items requiring integration of multiple pieces of information, interpretation of clinical or radiographic findings, differential diagnosis, clinical decision-making, or multi-step reasoning to identify the correct answer. Questions were categorized based on the dominant cognitive process required for answering rather than on question length or complexity. Complete agreement (%100) was achieved between the researchers, with no discrepant classifications observed (Cohen’s kappa = 1.0).

The five large language models included GPT-5.2 accessed through ChatGPT Go and ChatGPT Plus subscription interfaces (OpenAI), DeepSeek V3 API (DeepSeek AI), Claude Sonnet 4.5 (Anthropic) and Gemini 3 Flash (Google DeepMind) were evaluated using a standardized single-attempt protocol in the Turkish language, whereby each question was entered only once per model without providing feedback or allowing iterative corrections.

Statistical analysis

All statistical analyses were performed using the Statistical Package for Social Sciences (SPSS) for Windows, version 27 (SPSS Inc., Chicago, IL, USA). Descriptive statistics were reported as frequencies, percentages, means, standard deviations, medians, and interquartile ranges (IQR). Statistical analyses were performed by evaluating the data both as categorical outcomes and as accuracy distributions. Categorical differences in correct and incorrect responses among the AI models were analyzed using the Chi-square test. Differences in overall accuracy distributions across models were assessed using the Kruskal–Wallis test because the binary scoring data (0 = incorrect, 1 = correct) did not meet normality assumptions. The assumption of normal distribution was assessed with the Shapiro–Wilk test, and homogeneity of variances was evaluated using Levene’s test. For comparisons of three or more independent groups that did not meet normality assumptions, the Kruskal–Wallis test was applied. Post hoc Bonferroni analyses were conducted to identify the groups responsible for significant differences. Measurement consistency was evaluated using Fisher’s kappa and the intraclass correlation coefficient (ICC). The level of statistical significance was set at α = 0.05 for all analyses.

Sample size determination

The study included the entire available sample of multiple-choice questions from the Oral and Maxillofacial Radiology and Oral and Maxillofacial Surgery sections of the Turkish Dental Specialty Examination (DUS) conducted between 2012 and 2021. A total of 260 questions were initially identified; however, two Oral and Maxillofacial Surgery questions were excluded due to post-examination cancellation. Therefore, the final sample consisted of 258 questions. Inclusion of the complete dataset allowed a comprehensive evaluation of AI model performance and minimized potential sampling bias.

Inclusion and exclusion criteria

A total of 258 multiple-choice questions selected from the Turkish Dental Specialty Examination (DUS) were included in this study. Of these, 130 questions were related to Oral and Maxillofacial Radiology, and 128 questions to Oral and Maxillofacial Surgery. Each question consisted of five answer options. Among the questions, 202 were classified as knowledge-based and 56 as analytical-based.

Inclusion criteria

All multiple-choice questions assessing Oral and Maxillofacial Radiology and Oral and Maxillofacial Surgery in the clinical sciences section of the DUS database were included.

Exclusion criteria

To ensure methodological uniformity in the analysis, questions that were not in a multiple-choice format (such as open-ended questions), questions with multiple correct answers, and questions that were annulled after the examination due to the identification of errors were excluded from the study.

All questions were obtained from the official website of the Assessment, Selection, and Placement Center. Two specialists (one oral radiologist and one oral surgeon) independently classified the questions as either analytical based (requiring clinical scenario interpretation and decision-making) or knowledge-based (requiring direct theoretical knowledge).

Execution of AI models

On January 27, 2026, new accounts were created for each AI model evaluated in the study. The evaluated large language models included GPT-5.2 accessed through ChatGPT Go and ChatGPT Plus subscription interfaces (OpenAI), DeepSeek V3 API (DeepSeek AI), Claude Sonnet 4.5 (Anthropic), and Gemini 3 Flash (Google DeepMind). All models were accessed on January 27, 2026, using their most recent publicly available versions at the time of evaluation. The official web-based user interfaces of each platform were used under default settings without parameter modification, additional prompting strategies, or external tools. Because commercial AI systems are continuously updated and detailed backend version identifiers are not always publicly disclosed by providers, the present study reflects the operational model versions available on the date of assessment.

GPT-5.2 accessed through the ChatGPT Go and ChatGPT Plus interfaces was evaluated separately because these subscription tiers may differ in response performance, access priority, processing capacity, and system-level optimization despite being based on the same underlying model family. Comparing both interfaces allowed assessment of whether platform-level access conditions could influence examination performance under real-world usage conditions.

Browser cookies and history were cleared prior to data entry. All questions were entered on the same day via the official web interfaces of each model under default settings, without any parameter adjustment or additional prompting. Questions were presented exactly as in the examination format, preserving original wording and order. Responses were carefully recorded by the principal investigator and verified for accuracy by a second researcher. Each question was submitted only once to each AI model within a single memory-disabled session in order to simulate a standardized real-world examination environment and to avoid potential response contamination arising from iterative prompting or contextual memory effects. This single-attempt approach was preferred because the primary aim of the study was to evaluate the immediate response performance of AI systems under conditions comparable to those faced by human candidates during high-stakes examinations.

Nevertheless, large language models are probabilistic systems, and repeated submissions of the same question may generate response variability due to stochastic token generation and context-dependent reasoning processes. Therefore, the present methodology reflects a cross-sectional snapshot of model performance rather than a measure of intra-model reproducibility across repeated attempts. The potential influence of response variability was considered a methodological limitation and should be taken into account when interpreting the findings.

The multiple-choice questions were entered verbatim without any modifications. Each AI model was presented with the same set of questions, preserving the original format, wording, and order used in the dental specialty examination. The responses generated by the AI models were carefully recorded by the principal investigator, and a second researcher independently verified the accuracy of the recorded answers. This procedure ensured reliable documentation of the AI-generated responses. Each question was submitted to each model only once to prevent repeated exposure or learning effects that could introduce bias.

Performance evaluation

All questions were submitted simultaneously to the AI models on January 27, 2026. Responses were classified as correct or incorrect according to the official answer keys. Responses generated by the AI systems were recorded by the principal investigator and independently verified by a second researcher to ensure data accuracy. Because commercial AI systems are continuously updated and detailed backend version identifiers are not always publicly disclosed by providers, the study reflects the operational model versions available on the date of assessment. Accuracy rates were calculated for each model, as well as separately for analytical and knowledge-based questions. To prevent learning bias, no feedback or follow-up prompts were provided to the AI models after their initial responses.

Results

A statistically significant association was found between AI models and performance on knowledge-based questions (χ² = 9.589, p = 0.048). Column percentage analysis showed that incorrect response rates were highest for Claude (27.0%) and lowest for Gemini (14.0%). Correct response rates were highest for Gemini (21.3%) and lowest for Claude (18.5%). According to Z-test results for column proportions, the false response rate of Claude and the true response rate of Gemini were significantly higher than those of certain other models (p < 0.05). Row percentage comparisons indicated that Claude had a significantly higher incorrect rate (23.8%) compared to Gemini (12.4%), whereas Gemini had a significantly higher correct rate (87.6%) than Claude (76.2%) (p < 0.05) (Fig. 1).

Fig. 1.

Distribution of correct and incorrect responses across models for knowledge-based questions

A significant relationship was also observed between AI models and analytical question performance (χ² = 10.619, p = 0.031). Column percentages demonstrated that incorrect analytical responses were distributed as follows: Claude 24.6%, DeepSeek 26.2%, Gemini 13.8%, GPTgo 26.2%, and GPTplus 9.2%. Correct analytical responses were highest for GPTplus (23.3%) and Gemini (21.9%). Although GPTplus showed a significantly higher true response rate compared to its false rate in Z-test analysis, row percentage comparisons indicated no statistically significant differences among models for analytical questions after multiple comparison adjustments (p > 0.05) (Fig. 2).

Fig. 2.

Distribution of correct and incorrect responses across models for analytical-based questions

For overall performance (combined knowledge and analytical questions), a significant association was found between model type and accuracy (χ² = 14.379, p = 0.006). Incorrect overall response rates were highest for Claude (26.3%) and lowest for Gemini (14.0%). Gemini achieved the highest overall correct response rate (21.4%), while Claude had the lowest (18.5%). Row percentage analysis revealed that Claude had a significantly higher incorrect rate (24.8%) than Gemini (13.2%), and Gemini had a significantly higher correct rate (86.8%) compared to Claude (75.2%) (p < 0.05) (Fig. 3).

Fig. 3.

Distribution of correct and incorrect responses across models for overall questions

Accuracy rates among AI groups for knowledge-based, analytical, and overall questions were compared using the Kruskal–Wallis test. The analysis revealed statistically significant differences among AI models in knowledge accuracy (KW = 9.579, p = 0.048), analytical accuracy (KW = 10.581, p = 0.032), and overall accuracy (KW = 14.368, p = 0.006) (p < 0.05). Bonferroni-adjusted pairwise comparisons showed that Claude’s accuracy was significantly lower than Gemini’s for both knowledge-based and overall accuracy rates (p < 0.05). However, no statistically significant differences were identified among models in analytical accuracy after Bonferroni correction (p > 0.05) (Table 1).

Table 1.

Comparison of accuracy rates according to AI models

	Knowledge	Analytical	Overall
	Mean ± S.d. (M., IQR)	Mean ± S.s. (M., IQR)	Mean.±S.d. (M., IQR)
Claude	0.76 ± 0.43 (1,0)	0.71 ± 0.46 (1,1)	0.75 ± 0.43 (1,0)
DeepSeek	0.81 ± 0.39 (1,0)	0.7 ± 0.46 (1,1)	0.79 ± 0.41 (1,0)
Gemini	0.88 ± 0.33 (1,0)	0.84 ± 0.37 (1,0)	0.87 ± 0.34 (1,0)
GPT Go	0.84 ± 0.37 (1,0)	0.7 ± 0.46 (1,1)	0.81 ± 0.4 (1,0)
GPT Plus	0.83 ± 0.38 (1,0)	0.89 ± 0.31 (1,0)	0.84 ± 0.36 (1,0)
Test statistic (KW)	9.579	10.581	14.368
p	0.048*	0.032*	0.006*
Bonferroni	Claude< Gemini (0.027*)	-	Claude< Gemini (0.007*)

KW  Kruskal–Wallis test, SD  Standard deviation, IQR  Interquartile range

*p < 0.05

Cohen’s kappa and the intraclass correlation coefficient (ICC) were calculated to evaluate the agreement, reliability, and internal consistency among the AI models (Claude, DeepSeek, Gemini, GPT Go, and GPT Plus). When agreement among models was examined, Cohen’s kappa coefficients for knowledge, analytical, and overall scores were found to be at a low level (κ = 0.226–0.339), although all kappa values were statistically significant (p < 0.001). This finding indicates low agreement among AI models in terms of exact classification and absolute scoring. In contrast, ICC results demonstrated moderate reliability for knowledge (ICC = 0.597), analytical (ICC = 0.728), and overall scores (ICC = 0.637), with all values reaching statistical significance (p < 0.001) (Table 2).

Table 2.

Agreement coefficients of AI models

	Kappa (p)	%95 GA	ICC (p)	%95 GA
Knowledge	0.226 (< 0.001*)	0.182, 0.270	0.597 (< 0.001*)	0.503, 0.678
Analytical	0.339 (< 0.001*)	0.256, 0.422	0.728 (< 0.001*)	0.599, 0.826
Overall	0.257 (< 0.001*)	0.219, 0.296	0.637 (< 0.001*)	0.563, 0.703

CI  Confidence interval, ICC  Intraclass Correlation Coefficient

*p < 0.05

These findings indicate that inter-model differences were more pronounced in knowledge-based questions than in analytical questions, and that overall performance patterns largely reflected success in knowledge recall tasks (Table 3).

Table 3.

The comparison of AI models in knowledge-based, analytical, and overall questions

	AI										X2	p
		False			True			Total
		n	%	%G	n	%	%G	n	%	%G
Knowledge	Claude	48	23.8a	27.0a	154	76.2a	18.5b	202	100.0	20.0	9.589	0.048*
	DeepSeek	38	18.8a, b	21.3a	164	81.2a, b	19.7a	202	100.0	20.0
	Gemini	25	12.4b	14.0a	177	87.6b	21.3b	202	100.0	20.0
	GPT Go	33	16.3a, b	18.5a	169	83.7a, b	20.3a	202	100.0	20.0
	GPT Plus	34	16.8a, b	19.1a	168	83.2a, b	20.2a	202	100.0	20.0
Analytical	Claude	16	28.6a	24.6a	40	71.4a	18.6a	56	100.0	20.0	10.619	0.031*
	DeepSeek	17	30.4a	26.2a	39	69.6a	18.1a	56	100.0	20.0
	Gemini	9	16.1a	13.8a	47	83.9a	21.9a	56	100.0	20.0
	GPT Go	17	30.4a	26.2a	39	69.6a	18.1a	56	100.0	20.0
	GPT Plus	6	10.7a	9.2a	50	89.3a	23.3b	56	100.0	20.0
Overall	Claude	64	24.8a	26.3a	194	75.2a	18.5b	258	100.0	20.0	14.379	0.006*
	DeepSeek	55	21.3a, b	22.6a	203	78.7a, b	19.4a	258	100.0	20.0
	Gemini	34	13.2b	14.0a	224	86.8b	21.4b	258	100.0	20.0
	GPT Go	50	19.4a, b	20.6a	208	80.6a, b	19.9a	258	100.0	20.0
	GPT Plus	40	15.5a, b	16.5a	218	84.5a, b	20.8a	258	100.0	20.0

Different lowercase letters in the same row/column indicate a persistently significant difference between groups according to Z-test representations (p < 0.05)

X²: Chi-square test statistic, %: Row percentage, %G: Column percentage

*p < 0.05

Gemini demonstrated the highest performance in knowledge-based questions, whereas ChatGPT-5.2 Plus achieved the best performance in analytical questions. Among the analytical questions, two questions were answered incorrectly by all five AI models (Fig. 4). In knowledge-based questions, one question was incorrectly answered by all five AI systems (Fig. 5).

Fig. 4.

Example of an analytical-based question answered incorrectly by all five AI models, illustrating the limitations of current LLMs in multi-step clinical reasoning and diagnostic interpretation. The figure presents the response generated by GPT-5.2 (ChatGPT Plus) as a representative example. (There was no translation from Turkish to English in the study method, but Fig. 4 was exemplified with a question translated from Turkish to English for the article’s readership)

Fig. 5.

Example of a knowledge-based question answered incorrectly by all five AI models, demonstrating that even high-performing LLMs may show limitations in specialized domain-specific factual knowledge retrieval. The figure presents the response generated by GPT-5.2 (ChatGPT Plus) as a representative example (There was no translation from Turkish to English in the study method, but Fig. 5 was exemplified with a question translated from Turkish to English for the article’s readership)

Discussion

The findings of the present study demonstrate that the performance of contemporary large language models (LLMs) varies according to the cognitive structure of examination questions. Although all evaluated AI systems achieved relatively high accuracy rates in knowledge-based questions, their performance was lower and less consistent in analytical questions requiring clinical interpretation and multi-step reasoning. These findings suggest that current LLMs perform better in factual recall tasks than in examination-based analytical tasks that involve reasoning-related processes.

The superior performance observed in knowledge-based questions may largely be attributed to differences in model architecture, training corpus composition, and optimization strategies. Knowledge-based questions primarily depend on retrieval of factual information and probabilistic matching of previously encountered textual patterns. Consequently, models trained on larger and more diverse biomedical datasets may demonstrate improved performance in these tasks. In the present study, Gemini achieved the highest accuracy in knowledge-based questions, whereas Claude demonstrated the lowest performance. This discrepancy may reflect differences in domain-specific exposure, reinforcement learning approaches, retrieval optimization, and the balance between generative fluency and factual precision among models. Previous studies evaluating LLMs in medical and dental examinations have similarly reported that performance in factual tasks is strongly associated with the scope and quality of healthcare-related training data [14–17].

In contrast, analytical questions require cognitive operations extending beyond information retrieval, including contextual integration, sequential reasoning, interpretation of clinical findings, exclusion of distractors, and maintenance of logical consistency throughout multi-step inference pathways. Unlike knowledge-based questions, analytical tasks depend on the model’s ability to approximate clinical reasoning rather than reproduce memorized information. Therefore, improvements in parameter size or training data volume alone may not proportionally enhance analytical performance. This interpretation may explain why statistically significant overall differences among models were identified in analytical questions, while pairwise comparisons did not demonstrate a clearly superior model after Bonferroni correction.

The discrepancy between factual recall and analytical reasoning has increasingly been emphasized in the recent literature. Urda et al. reported that AI models, despite achieving high factual accuracy, tend to make errors in complex reasoning tasks across scientific domains, showing lower performance in clinical reasoning compared with knowledge-focused questions [14]. Likewise, Dave et al. demonstrated that LLM accuracy in dental examinations varies substantially depending on whether assessments emphasize theoretical knowledge or clinical application, which parallels our findings that knowledge-based questions yield higher accuracy whereas performance declines in analytical tasks [15]. Similarly, Lin et al., who compared ChatGPT, Gemini, and Claude models in the Taiwanese National Dental Licensing Examination, reported that although some models demonstrated acceptable overall examination performance, no clear superiority was identified in analytically demanding questions [16]. These observations collectively suggest that current LLMs function more effectively as information retrieval systems than as autonomous clinical reasoning frameworks.

Recent studies in dentistry and healthcare education further reinforce this interpretation. Chau et al. demonstrated that advanced LLMs may achieve passing-level performance in dental licensing examinations; however, the authors also emphasized persistent limitations in clinically complex scenarios requiring contextual judgment and diagnostic synthesis [17]. Similarly, Gokkurt et al. reported that although LLMs can demonstrate strong examination performance, their reasoning pathways remain inconsistent and vulnerable to contextual errors in complex healthcare-related tasks [18]. Additional contemporary evidence also suggests that current LLMs should primarily be regarded as supportive educational and diagnostic adjuncts rather than autonomous clinical systems. Recent investigations evaluating AI applications in dental education and oral healthcare have shown that, despite rapid advances in generative capability, current models remain susceptible to contextual inconsistency, hallucination, and variability in diagnostic interpretation. These limitations are particularly important in oral and maxillofacial disciplines, where clinical decision-making often requires integration of radiographic findings, anatomical knowledge, patient-specific variables, and procedural risk assessment. Additional evidence from multimodal oral healthcare studies also supports this interpretation. Temur et al. compared several ChatGPT versions in the diagnosis of oral and maxillofacial pathology cases involving clinical, radiographic, and histopathological data and reported that, although advanced models achieved moderate-to-high diagnostic accuracy, experienced clinicians still outperformed LLMs in complex cases. Similarly, recent multimodal AI studies in oral lesion diagnosis emphasized that current systems are more suitable as decision-support and educational tools than as autonomous diagnostic systems because contextual interpretation and differential diagnosis remain challenging. Collectively, these studies reinforce the conclusion that current AI systems demonstrate promising potential in dental education and diagnostic assistance; however, clinically reliable reasoning and reproducibility remain important limitations [19].

Hassanein et al. evaluated the diagnostic accuracy and clinical applicability of large language models (LLMs) for oral lesion diagnosis and concluded that, although LLMs demonstrate promising diagnostic capabilities, significant limitations remain regarding diagnostic accuracy, clinical reliability, and potential bias. Consistent with these findings, our study also demonstrated that while contemporary AI models show considerable potential in dental applications, concerns related to reliability, consistency, and safe clinical implementation continue to represent important barriers to their autonomous use in clinical practice [20].

Khanagar et al. compared the performance of different LLMs in answering pediatric dentistry-related questions and reported that AI models can serve as useful tools for providing information and educational support; however, the quality and accuracy of their responses may vary considerably depending on the nature and complexity of the queries. Similarly, our findings revealed variability in model performance across different question types, suggesting that the effectiveness of LLMs is influenced by task characteristics and cognitive demands [21].

Zhao et al. demonstrated that LLM performance declines substantially as the level of clinical reasoning required increases. Specifically, they reported a marked reduction in accuracy across the transition from lower-level knowledge recall tasks (L1) to more advanced reasoning and individualized clinical decision-making tasks (L2–L3), highlighting persistent limitations in higher-order dental reasoning. In agreement with these observations, our results showed that the evaluated models generally performed better on knowledge-based questions than on analytical questions requiring clinical interpretation and reasoning, indicating that increasing reasoning complexity remains a significant challenge for current LLMs [22].

Helali et al., in a recent scoping review of 97 studies published between 2020 and 2026 on dental foundation models and LLMs, concluded that these models generally perform well in text-based educational, informational, and reasoning-related tasks. However, they also identified ongoing concerns related to hallucinations, clinical safety, and the lack of standardized evaluation frameworks. Consistent with these conclusions, our findings suggest that although LLMs may provide valuable support in dental education and clinical decision-making, further validation and careful clinical oversight are necessary before their routine integration into clinical practice [23].

The examples of questions answered incorrectly by all five AI systems provide additional qualitative insight into the limitations of current LLMs. The analytical question incorrectly answered by all models illustrates persistent challenges in multi-step clinical reasoning and diagnostic interpretation, whereas the universally incorrect knowledge-based question highlights that even highly advanced systems may still demonstrate deficiencies in specialized domain-specific factual retrieval. These findings suggest that increasing model scale alone may not fully address the limitations associated with clinical reasoning reliability and that future improvements may require integration of domain-specific training corpora, multimodal learning systems, and medically supervised reinforcement learning approaches.

Agreement analyses further supported this interpretation. Although Cohen’s kappa coefficients indicated relatively low inter-model agreement, ICC values demonstrated moderate reliability across models. This apparent discrepancy may be explained by the methodological differences between the two statistics: Cohen’s kappa evaluates exact item-level categorical agreement and is highly sensitive to imbalances in response distributions, whereas ICC reflects the consistency of overall scoring patterns across the dataset. Accordingly, the findings suggest that AI systems did not always generate identical responses to the same questions but still demonstrated moderate consistency in their overall scoring behavior and accuracy trends. These observations further support the probabilistic and context-dependent nature of large language model outputs.

From a practical perspective, these findings suggest that current LLMs may be useful as supplementary educational tools for theoretical learning and examination preparation in dentistry. However, the reduced performance observed in analytically demanding questions indicates that AI-generated outputs should still be interpreted under professional supervision, particularly in oral and maxillofacial disciplines where diagnostic decisions frequently require integration of radiographic, anatomical and clinical findings.

Conclusion

Large language models demonstrated strong performance in knowledge-based dental questions but showed more limited and inconsistent performance in analytical questions requiring clinical interpretation and multi-step analysis. These findings suggest that current AI systems may serve as valuable educational and knowledge-support tools in dentistry; however, their reliability for independent clinical decision-making remains insufficient. Although advances in model capability continue rapidly, human expertise and clinical judgment remain essential in complex diagnostic and treatment-related processes. Future studies should focus not only on examination accuracy but also on reasoning consistency, reproducibility, and clinical safety in real-world dental applications.

Study limitations

This study has several methodological limitations. An important limitation was the imbalance between knowledge-based and analytical questions. The dataset contained substantially more knowledge-based questions (202) than analytical questions (56), which may have reduced the statistical power to detect subtle inter-model differences in analytical performance. In addition, because overall accuracy calculations were largely driven by the larger number of knowledge-based questions, the general performance rankings may have more strongly reflected factual recall ability than performance on analytical tasks. This imbalance was primarily related to the structure of the ÖSYM examination, which does not specify a fixed distribution of analytical and knowledge-based questions across examination years. Consequently, the final dataset contained an unequal distribution of question types, and findings related to analytical performance should be interpreted with caution.

Model responses were obtained at a single time point using specific model versions; however, AI systems are continuously updated, and their performance may change over time. Therefore, the results reflect performance only at the time of evaluation. In addition, because each question was submitted only once, the study did not evaluate response reproducibility across repeated attempts. Since large language models may produce variable outputs for the same prompt, future studies should investigate intra-model consistency using repeated-query designs.

Another methodological limitation is that the same set of examination questions was presented to all AI models, resulting in a repeated-measures structure at the question level. Consequently, the observations were not completely independent, and the applied statistical methods may not have fully accounted for within-question dependency. Therefore, the reported statistical differences and p-values should be interpreted with appropriate caution and viewed as exploratory comparisons of model performance rather than definitive estimates of between-model superiority. Future studies involving larger datasets may employ mixed-effects models, generalized estimating equations, or other repeated-measures approaches to better account for clustering of responses within individual questions.

Furthermore, analytical examination questions should not be considered direct measures of real-world clinical reasoning, as they represent only a structured and indirect assessment of reasoning-related cognitive performance. Finally, the study measured only answer accuracy; qualitative aspects such as explanation quality, clinical justification accuracy, and the potential clinical impact of errors were not analyzed. Consequently, the findings reflect quantitative examination performance but do not fully address clinical decision safety.

Abbreviations

AI

Artificial intelligence

CI

Confidence interval

DUS

Turkish dental specialty examination

ICC

Intraclass correlation coefficient

IQR

Interquartile range

LLMs

Large language models

OMFR

Oral and maxillofacial radiology

OMFS

Oral and maxillofacial surgery

ÖSYM

Assessment, selection and placement center

SPSS

Statistical package for the social sciences

TMJ

Temporomandibular joint

Authors’ contributions

E.S. and A.Y. wrote the main manuscript text A.Y. prepared Figs. 1, 2, 3, 4 and 5. E.S. reviewed the manuscriptAll authors collected the data.

Funding

The authors declare that no funding was received for this study.

Data availability

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Declarations

Ethics approval and consent to participate

This study did not involve human participants, patient data, or animal subjects. Therefore, ethical approval was not required.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.