Abstract
Objective
This study was aimed to compare the accuracy of four state-of-the-art large language models (LLMs)(ChatGPT-5, ChatGPT-4o, DeepSeek, and Gemini 2.5 Pro) on Dentistry Specialization Exam (DUS) questions in Prosthodontics, and to assess the effect of querying method on performance.
Materials and methods
A total of 128 multiple-choice DUS questions (106 knowledge-based; 22 case-based) from 13 exams (2012–2021) were administered. Each model was tested within a single 24-hour window (September 9, 2025). Two protocols were used: (1) Independent Query (each item in a fresh chat) and (2) Sequential Query (exam-like blocks of 10 items). First responses only were recorded and scored against official answer keys. Statistical analyses were performed using the chi-square (χ²) test to compare categorical outcomes (correct/incorrect responses) across different large language models and query strategies. Inter-rater agreement for question classification was assessed using Cohen’s kappa coefficient, and all analyses were conducted at a significance level of α = 0.05.
Results
With Independent Query, the accuracy for knowledge questions was found to be 91% (GPT-5), 86% (GPT-4o), 71% (DeepSeek), and 88% (Gemini 2.5 Pro) while total accuracy was detected as 86%, 83%, 66%, and 85% respectively. In this context, statistically significant differences were reported across models, also due to the low scores of DeepSeek. Case-based accuracies (64%, 68%, 45%, and 77%) did not differ significantly. With Sequential Query, knowledge accuracies were 75%, 73%, 63%, and 82% and case accuracies were 77%, 68%, 64%, and 91%, respectively. Total accuracy still differed across models (75%, 72%, 63%, and 84%). Within-model comparisons showed significant drops for knowledge items from Independent to Sequential querying for GPT-5 (91%→75%) and GPT-4o (86%→73%). DeepSeek and Gemini 2.5 Pro showed no significant changes. Notably, Gemini 2.5 Pro yielded the highest performance in case-based questions (91%) with sequential query.
Conclusions
While the present study findings highlight current limitations in clinical reasoning, they support the conclusion that LLMs can be used as supplementary educational tools for DUS-style knowledge assessment but should not replace expert judgment or patient-specific decision-making.
Keywords: Artificial intelligence, Chatbots, ChatGPT, DeepSeek, Gemini
Introduction
Artificial intelligence (AI), as a field of science focused on developing systems that mimic human intelligence, has made a significant impact in healthcare, particularly through its subfields such as natural language processing (NLP), machine learning, and deep learning. AI technologies are increasingly being used effectively in numerous clinical applications, such as patient monitoring, diagnostic support systems, image analysis, and personalized treatment planning [1, 2] Dentistry has also been affected by this digital transformation and the use of AI-based applications has become widespread in various fields such as restorative dentistry, periodontology, orthodontics, and prosthetic dental treatment. At the same time, AI technologies are increasingly being integrated into dental education, offering new tools that facilitate teaching processes for both students and instructors [3, 4].
At the heart of developments related to AI are Large Language Models (LLMs), which stand out for their ability to understand and generate human language. Developed by technology companies such as OpenAI, Google, and DeepSeek AI, these systems can successfully perform complex tasks such as text generation, summarization, translation, and question answering by their architectures with billions of parameters [5, 6]. Contemporary large language models developed by companies such as OpenAI, Google, Anthropic, and DeepSeek differ in architecture, data sources, update mechanisms, and multimodal capabilities, which may result in variability in accuracy and response quality across disciplines and question types [7]. Therefore, the accuracy, contextual consistency, and ethical boundaries of the content produced by these models must be carefully and systematically evaluated, especially in fields requiring specialized knowledge such as medicine and dentistry [4].
While LLMs demonstrate transformative potential in healthcare, their reliability in specialized domains remains to be validated. Evaluating these models against rigorous benchmarks such as the Dentistry Specialization Exam (DUS) is essential to determine their readiness for dental education and clinical decision support. A successful and consistent model not only increases learning efficiency and equal opportunity by providing students with a personalized study companion, but also lays the foundation for clinical decision support systems that will serve physicians in the future [8, 9]. Although studies examining the performance of LLM models in various medical and dental examinations exist in the literature [7–17] there is still a need for original research comparing the competence of these models in specific areas such as prosthetic dental treatment with current models featuring different architectures.
The primary objective of this study is to comparatively evaluate the performance of ChatGPT-5 and ChatGPT-4o (OpenAI, San Francisco, CA, USA), DeepSeek-V3 (DeepSeek-AI, Beijing, China), and Gemini 2.5 Pro (Google DeepMind, Mountain View, CA, USA) representing the most current and advanced Large Language Models (LLMs) available—in answering Dental Specialization Examination (DUS) questions within the field of Prosthetic Dentistry. A secondary objective is to investigate whether the accuracy of the models’ responses varies according to the querying methodology. To this end, model performance was examined under two distinct scenarios: independent querying, where each question was evaluated individually without contextual influence, and sequential querying, where questions were presented consecutively to mimic an actual examination flow. This dual-pronged approach is critical as it reflects the two fundamental utilization patterns of LLMs in dental education. Independent questioning measures raw knowledge accuracy in isolation, whereas sequential questioning evaluates the model’s ability to maintain consistency throughout an extended session and assesses how performance fluctuates based on previous contextual prompts. Consequently, this study aims to reveal not only the theoretical knowledge levels of these models but also their reliability and consistency under practical application conditions. The models evaluated in this study represent the most up-to-date publicly accessible versions available at the time of data collection. Given the rapid and continuous evolution of large language models, including frequent version updates and limited disclosure of detailed technical documentation by developers, model architectures and performance characteristics may change over time. Therefore, the findings of this study should be interpreted within the temporal context in which the analyses were conducted.
The null hypotheses were that (1) there would be no significant difference among the four LLMs in answering Prosthetic Dentistry DUS questions correctly, and (2) the querying method (independent vs. sequential) would not significantly affect model performance.
Materials and methods
Large language models tested
Four different large language models were tested in this study. Chatbots were selected based on their wide accessibility, popularity, and different operational architectures. To ensure the stability and response consistency of the models, all queries were performed within a 24-hour timeframe on September 9, 2025.
For Method 1 (Independent Query), the same standardized initial instruction prompt (“avoid guessing and answer only when certain”) was provided at the beginning of each independent chat session for every question. Conversation history was cleared between the independent and sequential querying phases by initiating new chat sessions, ensuring that no prior contextual information influenced subsequent responses.
Large Language Models Used in the Study and Access Information,
ChatGPT-5, OpenAI, San Francisco, CA, USA (August 2025 version).
ChatGPT-4o, OpenAI, San Francisco, CA, USA (May 2024 version).
DeepSeek | DeepSeek AI, Beijing, China (March 2025 version).
Gemini 2.5 Pro, Google, Mountain View, CA, USA (June 2025 version).
GPT-5 extends this capacity to approximately 400,000 tokens, GPT-4o supports context windows of up to approximately 128,000 tokens with improved output efficiency, DeepSeek-V3 supports extended context windows of up to approximately 128,000 tokens based on its Mixture-of-Experts architecture, and Gemini 2.5 Pro enables very large input processing of up to approximately 1,000,000 tokens with output sizes of up to 64,000 tokens and a knowledge cutoff of January 2025 [18–21].
Data source and question selection
In this study, DUS questions from 13 periods conducted by the Student Selection and Placement Center (ÖSYM) in Turkey between 2012 and 2021 were used. In total, 130 questions (10 questions from each of 13 exams) were initially selected. However, two questions were canceled by the examination authority, resulting in 128 questions being included in the final analysis. The study included both questions containing only text and questions containing visuals (diagrams, radiographs, clinical photographs, etc.) in addition to text. The questions were used in their original Turkish form without any changes.
To analyze the nature of the questions, all questions were independently classified by two dentists specializing in Prosthetic Dentistry. The questions were divided into two categories: “case-based” (presenting a clinical scenario and requiring decision-making skills; n = 22) and “knowledge-based” (directly based on theoretical knowledge; n = 106). Classification disagreements between the two experts were resolved through consensus by consulting a third expert researcher. The inter-rater agreement analysis demonstrated a high level of consistency between the evaluators, as indicated by the Cohen’s kappa value. Disagreement occurred in only one item and was resolved by consensus, resulting in a Cohen’s kappa of 0.97.
Study design and data collection
Prompt protocol and data collection
A standardized protocol was followed throughout the data collection process. Prior to interaction with each model, a predefined initial instruction (prompt) was provided. This prompt is quoted in the manuscript and is also presented in full in Fig. 1. Only after the model explicitly acknowledged this instruction were the DUS questions administered.
Fig. 1.
Initial instruction (prompt) provided to the large language models prior to question administration. A: English version of the initial instruction. B: Turkish version of the same prompt
In the independent querying approach, the same initial instruction was repeated before each individual question, with each question evaluated in a separate session. The conversation history was completely cleared prior to the submission of every question to eliminate any potential contextual influence from previous interactions. In the sequential querying approach, the initial instruction was provided once at the beginning of the session, after which a block of 10 questions was presented consecutively within the same chat interface. This design aimed to simulate an examination-like workflow and to assess the potential impact of accumulated context on model performance.
Each model–question–method combination was administered only once, and only the first response generated by the model was recorded. The “Regenerate” or similar response renewal functions were not used. Leveraging the multilingual capabilities of the models, all interactions were conducted in Turkish. The responses were independently and manually entered into a Microsoft Excel spreadsheet (Microsoft Corporation, Redmond, WA, USA) by two researchers. For questions containing visual components (e.g., diagrams, radiographs, and clinical photographs), the corresponding images were explicitly uploaded and displayed within the chat interface, ensuring that both visual and textual information were simultaneously available to the models during evaluation.
Query methods
Method 1: Independent Query (Reset Chat): A new chat session was initiated with the relevant LLM for each of the 128 questions.
Method 2: Sequential Querying (Exam Simulation): A set of 10 questions for each DUS period was asked to the model consecutively within a single chat session.
Data analysis coding of responses
The model’s responses were evaluated by comparing them with ÖSYM’s official answer keys. The first valid letter of the model’s answer (A, B, C, D or E) was accepted as the final answer. Questions left blank or unanswered by artificial intelligence were coded as ‘blank’.
Statistical analysis
All statistical analyses were performed using non-parametric methods, as the study data consisted exclusively of categorical variables. Model type, querying strategy, and response outcomes (correct, incorrect, or unanswered) were treated as categorical variables; therefore, parametric assumptions such as normal distribution and homogeneity of variance were not applicable. Comparisons of response accuracy among different large language models were conducted using the chi-square (χ²) test. The same test was also applied to compare independent and sequential querying strategies within each model. For all inferential analyses, only observed absolute frequencies of correct and incorrect responses were used; percentage values presented in tables were provided solely for descriptive purposes and were not included in statistical calculations.
Prior to performing chi-square analyses, the adequacy of expected cell frequencies was evaluated to ensure compliance with test assumptions. All tests were conducted as two-tailed, and a significance level of α = 0.05 was adopted. Model performance was summarized using descriptive statistics, including frequencies and percentages of correct, incorrect, and unanswered responses. In addition to statistical significance testing, absolute percentage differences were calculated to facilitate interpretation of effect magnitude across models and querying strategies.
Inter-rater agreement for expert-based question classification was assessed using Cohen’s kappa coefficient, providing a measure of consistency beyond chance agreement. Statistical analyses were conducted using standard spreadsheet-based calculations and validated statistical procedures.
Results
In the study, a total of 128 DUS questions (106 knowledge-based, 22 case-based) from 2012 to 2021 were tested separately using four different artificial intelligence models (GPT-5, GPT-4o, DeepSeek, Gemini 2.5 Pro). The results obtained were evaluated using two separate methods; independent querying and sequential querying.
Independent query results
Using the independent query method, statistically significant differences were observed among the four artificial intelligence models in terms of knowledge-based questions and total questions (p < 0.05). These differences were primarily attributable to the lower performance of the DeepSeek model compared with the others; in contrast, the GPT-5, GPT-4o and Gemini 2.5 Pro models exhibited similar and high success rates (Table 1). Regarding case-based questions, however, no statistically significant difference was found between the models under independent querying (p > 0.05) (Table 1).
Table 1.
Performance of GPT-5, GPT-4o, DeepSeek, and gemini 2.5 pro on DUS questions under independent and sequential querying, expressed as accuracy (%) and correct response counts
| Query Type | Question Type (n) | GPT-5% (Correct n) | GPT-4o % (Correct n) | DeepSeek % (Correct n) | Gemini 2.5 Pro % (Correct n) | Chi-square (χ²) |
|---|---|---|---|---|---|---|
| Independent Query | Knowledge (106) | 91% (96) | 86% (91) | 71% (75) | 88% (93) | χ²(3) = 18.33, p < 0.05* |
| Case-based (22) | 64% (14) | 68% (15) | 45% (10) | 77% (17) | χ²(3) = 5.11, p = 0.164 | |
| Total (128) | 86% (110) | 83% (106) | 66% (85) | 85% (110) | χ²(3) = 21.25, p < 0.05* | |
| Sequential Query | Knowledge (106) | 75% (79) | 73% (77) | 63% (67) | 82% (87) | χ²(3) = 9.74, p < 0.05* |
| Case-based (22) | 77% (17) | 68% (15) | 64% (14) | 91% (20) | χ²(3) = 5.09, p = 0.165 | |
| Total (128) | 75% (96) | 72% (92) | 63% (81) | 84% (107) | χ²(3) = 13.86, p < 0.05* |
Statistical comparisons between models were performed using the chi-square (χ²) test based on observed frequencies of correct and incorrect responses; percentage values are presented for descriptive purposes only. A p-value < 0.05 was considered statistically significant
Sequential query results
When evaluations were conducted using the sequential query method, statistically significant differences were identified among the models for knowledge-based questions (p < 0.05). Likewise, analysis of total questions revealed a statistically significant difference across all models in sequential querying (p < 0.05) (Table 1). While sequential querying resulted in a significant decline in knowledge-question accuracy for GPT-5 and GPT-4o, the performance reduction observed for DeepSeek-V3 and Gemini 2.5 Pro was not statistically significant. In sequential querying, although a numerical increase was observed in the correct response counts for case-based questions across all models except GPT-4o (which remained stable), these increases were not statistically significant (p > 0.05).
GPT-5 results
The GPT-5 model achieved 91% success in knowledge questions using the independent inquiry method, while this success rate dropped to 75% using the sequential inquiry method; this difference was found to be statistically significant (p < 0.05). In contrast, for case questions, the difference in success rates between independent querying and sequential querying was not statistically significant (p > 0.05). These findings show that GPT-5 performed better with independent querying on knowledge questions, but that the change in querying method did not significantly affect model performance on case questions. (Table 2).
Table 2.
Comparison of AI models’ response performance on knowledge and case questions according to independent and sequential query methods
| Model | Knowledge Questions (n = 106) | Chi-square (χ²) | Case Questions (n = 22) | Chi-square (χ²) | ||
|---|---|---|---|---|---|---|
| Independent Query | Sequential Query | Independent Query | Sequential Query | |||
| GPT-5 | 91% (96) | 75% (79) |
χ²=8.38 p < 0.05* |
64% (14) | 77% (17) |
χ²=0.44 p = 0.509 |
| GPT-4o | 86% (91) | 73% (77) |
χ²=4.85 p < 0.05* |
68% (15) | 68% (15) |
χ²=0.00 p = 1.000 |
| DeepSeek | 71% (75) | 63% (67) |
χ²=1.05 p = 0.307 |
45% (10) | 64% (14) |
χ²=0.83 p = 0.364 |
| Gemini 2.5 Pro | 88% (93) | 82% (87) |
χ²=0.92 p = 0.338 |
77% (17) | 91% (20) |
χ²=0.68 p = 0.409 |
p values test the within-model difference between Independent and Sequential methods for the corresponding item type. p < 0.05 indicates a statistically significant difference; p > 0.05 indicates no significant difference
Examining the performance of the GPT-5 model over the years using independent and sequential querying methods, differences were observed in knowledge and case questions. The lowest success rate in knowledge questions in independent querying was 67% (2020), while in sequential querying it was 50% (2014). For case questions, the lowest success rate was found to be 50% in both query types (Table 3).
Table 3.
GPT-5 performance by year: success rates and Correct–Incorrect counts for knowledge, case, and total questions
| Gpt-5 | Knowledge | Case | Total | |
|---|---|---|---|---|
| Independent Query | 2021(5 Knowledge, 5 Case) | 100% (5/5) | 80% (4/5) | 90% (9/10) |
| 2020(6 Knowledge, 4 Case) | 67% (4/6) | 75% (3/4) | 70% (7/10) | |
| 2019(5 Knowledge, 5 Case) | 80% (4/5) | 60% (3/5) | 70% (7/10) | |
| 2018(7 Knowledge, 3 Case) | 71% (5/7) | 67% (2/3) | 70% (7/10) | |
| 2017(8 Knowledge, 0 Case) | 75% (6/8) | 0% (0/0) | 75% (6/8) | |
| 2016(9 Knowledge, 1 Case) | 89% (8/9) | 0% (0/1) | 80% (8/10) | |
| 2015(10 Knowledge, 0 Case) | 100% (10/10) | 0% (0/0) | 100% (10/10) | |
| 2014s(10 Knowledge, 0 Case) | 100% (10/10) | 0% (0/0) | 100% (10/10) | |
| 2014i(10 Knowledge, 0 Case) | 90% (9/10) | 0% (0/0) | 90% (9/10) | |
| 2013s(10 Knowledge, 0 Case) | 100% (10/10) | 0% (0/0) | 100% (10/10) | |
| 2013i(10 Knowledge, 0 Case) | 90% (9/10) | 0% (0/0) | 90% (9/10) | |
| 2012s(8 Knowledge, 2 Case) | 100% (8/8) | 50% (1/2) | 90% (9/10) | |
| 2012i(8 Knowledge, 2 Case) | 100% (8/8) | 50% (1/2) | 90% (9/10) | |
| Sequential Query | 2021(5 Knowledge, 5 Case) | 100% (5/5) | 100% (5/5) | 100% (10/10) |
| 2020(6 Knowledge, 4 Case) | 83% (5/6) | 100% (4/4) | 90% (9/10) | |
| 2019(5 Knowledge, 5 Case) | 60% (3/5) | 60% (3/5) | 60% (6/10) | |
| 2018(7 Knowledge, 3 Case) | 57% (4/7) | 67% (2/3) | 60% (6/10) | |
| 2017(8 Knowledge, 0 Case) | 75% (6/8) | 0% (0/0) | 75% (6/8) | |
| 2016(9 Knowledge, 1 Case) | 78% (7/9) | 0% (0/1) | 70% (7/10) | |
| 2015(10 Knowledge, 0 Case) | 90% (9/10) | 0% (0/0) | 90% (9/10) | |
| 2014s(10 Knowledge, 0 Case) | 70% (7/10) | 0% (0/0) | 70% (7/10) | |
| 2014i(10 Knowledge, 0 Case) | 50% (5/10) | 0% (0/0) | 50% (5/10) | |
| 2013s(10 Knowledge, 0 Case) | 80% (8/10) | 0% (0/0) | 80% (8/10) | |
| 2013i(10 Knowledge, 0 Case) | 70% (7/10) | 0% (0/0) | 70% (7/10) | |
| 2012s(8 Knowledge, 2 Case) | 75% (6/8) | 50% (1/2) | 70% (7/10) | |
| 2012i(8 Knowledge, 2 Case) | 88% (7/8) | 100% (2/2) | 90% (9/10) |
No statistically significant difference was found between independent and sequential query methods in the context of the GPT-5 model performance on knowledge and case questions based on exams (p > 0.05) (Table 4).
Table 4.
Year-Based performance of GPT-5 across knowledge and case questions using independent and sequential querying methods
| Query Type | N | Mean ± SD | p | |
|---|---|---|---|---|
| Knowledge Questions | Independent Query | 13 | 7.39 ± 2.29 | 0.16 |
| Sequential Query | 13 | 6.23 ± 1.74 | ||
| Case Questions | Independent Query | 13 | 1.08 ± 1.44 | 0.72 |
| Sequential Query | 13 | 1.31 ± 1.75 |
N Total number of exams, SD Standard deviation
Statistical significance was set at p < 0.05
GPT-4o results
The GPT-4o model showed significantly higher performance in knowledge questions when the independent query method was used compared with sequential querying (p < 0.05), indicating superior accuracy with independent queries (Table 2). In contrast, for case questions, no statistically significant difference was observed between independent and sequential query methods (p > 0.05). When model performance was analyzed by examination year, variations were observed in both knowledge and case questions under independent and sequential querying strategies. Although lower success rates were noted in certain years—particularly under sequential querying—these fluctuations did not alter the overall performance pattern of the model (Table 5).
Table 5.
GPT-4o’s performance by year: success rates and Correct–Incorrect counts for Knowledge, Case, and total questions
| GPT-4o | Knowledge | Case | Total | |
|---|---|---|---|---|
| Independent Query | 2021(5 Knowledge, 5 Case) | 80% (4/5) | 80% (4/5) | 80% (8/10) |
| 2020(6 Knowledge, 4 Case) | 67% (4/6) | 75% (3/4) | 70% (7/10) | |
| 2019(5 Knowledge, 5 Case) | 80% (4/5) | 80% (4/5) | 80% (8/10) | |
| 2018(7 Knowledge, 3 Case) | 71% (5/7) | 67% (2/3) | 70% (7/10) | |
| 2017(8 Knowledge, 0 Case) | 75% (6/8) | 0% (0/0) | 75% (6/8) | |
| 2016(9 Knowledge, 1 Case) | 89% (8/9) | 0% (0/1) | 80% (8/10) | |
| 2015(10 Knowledge, 0 Case) | 90% (9/10) | 0% (0/0) | 90% (9/10) | |
| 2014s(10 Knowledge, 0 Case) | 100% (10/10) | 0% (0/0) | 100% (10/10) | |
| 2014i(10 Knowledge, 0 Case) | 90% (9/10) | 0% (0/0) | 90% (9/10) | |
| 2013s(10 Knowledge, 0 Case) | 90% (9/10) | 0% (0/0) | 90% (9/10) | |
| 2013i(10 Knowledge, 0 Case) | 80% (8/10) | 0% (0/0) | 80% (8/10) | |
| 2012s(8 Knowledge, 2 Case) | 100% (8/8) | 50% (1/2) | 90% (9/10) | |
| 2012i(8 Knowledge, 2 Case) | 88% (7/8) | 50% (1/2) | 80% (8/10) | |
| Sequential Query | 2021(5 Knowledge, 5 Case) | 100% (5/5) | 80% (4/5) | 90% (9/10) |
| 2020(6 Knowledge, 4 Case) | 83% (5/6) | 100% (4/4) | 90% (9/10) | |
| 2019(5 Knowledge, 5 Case) | 60% (3/5) | 80% (4/5) | 70% (7/10) | |
| 2018(7 Knowledge, 3 Case) | 57% (4/7) | 33% (1/3) | 50% (5/10) | |
| 2017(8 Knowledge, 0 Case) | 75% (6/8) | 0% (0/0) | 75% (6/8) | |
| 2016(9 Knowledge, 1 Case) | 89% (8/9) | 0% (0/1) | 80% (8/10) | |
| 2015(10 Knowledge, 0 Case) | 80% (8/10) | 0% (0/0) | 80% (8/10) | |
| 2014s(10 Knowledge, 0 Case) | 70% (7/10) | 0% (0/0) | 70% (7/10) | |
| 2014i(10 Knowledge, 0 Case) | 50% (5/10) | 0% (0/0) | 50% (5/10) | |
| 2013s(10 Knowledge, 0 Case) | 70% (7/10) | 0% (0/0) | 70% (7/10) | |
| 2013i(10 Knowledge, 0 Case) | 70% (7/10) | 0% (0/0) | 70% (7/10) | |
| 2012s(8 Knowledge, 2 Case) | 75% (6/8) | 50% (1/2) | 70% (7/10) | |
| 2012i(8 Knowledge, 2 Case) | 75% (6/8) | 50% (1/2) | 70% (7/10) |
Statistical analyses show that there is no significant difference between independent and sequential querying methods for the GPT-4o model in terms of both knowledge-based and case-based questions (Table 6).
Table 6.
Year-Based performance of GPT-4o across knowledge and case questions using independent and sequential querying methods
| Query Type | N | Mean ± SD | p | |
|---|---|---|---|---|
| Knowledge Questions | Independent Query | 13 | 7.00 ± 2.16 | 0.23 |
| Sequential Query | 13 | 6.08 ± 1.61 | ||
| Case Questions | Independent Query | 13 | 1.15 ± 1.57 | 1.00 |
| Sequential Query | 13 | 1.15 ± 1.68 |
N Total number of exams, SD Standard deviation
Statistical significance was set at p < 0.05
DeepSeek results
Examining the performance of the DeepSeek model in information and case questions using independent and sequential querying methods, statistical analyses show no significant difference between independent and sequential querying for either question type (p > 0.05). The success rate for knowledge questions was 71% for independent querying and 63% for sequential querying, while for case questions, it was 45% for independent querying and 64% for sequential querying. These findings show that the DeepSeek model performed similarly on knowledge and case questions regardless of the independent or sequential querying method, but it is noteworthy that the success rates were lower compared to other models (Table 2).
In the performance of the DeepSeek model with independent and sequential query methods by year, the lowest success rates were observed in knowledge questions with 38% in independent queries (2017) and 50% in sequential queries; in case questions, the rates were 33% in independent queries and 60% in sequential queries (Table 7).
Table 7.
DeepSeek model’s performance by year: success rates and Correct–Incorrect counts for Knowledge, Case, and total questions
| DeepSeek | Knowledge | Case | Total | |
|---|---|---|---|---|
| Independent Query | 2021(5 Knowledge, 5 Case) | 80% (4/5) | 60% (3/5) | 70% (7/10) |
| 2020(6 Knowledge, 4 Case) | 50% (3/6) | 50% (2/4) | 50% (5/10) | |
| 2019(5 Knowledge, 5 Case) | 60% (3/5) | 60% (3/5) | 60% (6/10) | |
| 2018(7 Knowledge, 3 Case) | 57% (4/7) | 33% (1/3) | 50% (5/10) | |
| 2017(8 Knowledge, 0 Case) | 38% (3/8) | 0% (0/0) | 38% (3/8) | |
| 2016(9 Knowledge, 1 Case) | 78% (7/9) | 0% (0/1) | 70% (7/10) | |
| 2015(10 Knowledge, 0 Case) | 80% (8/10) | 0% (0/0) | 80% (8/10) | |
| 2014s(10 Knowledge, 0 Case) | 90% (9/10) | 0% (0/0) | 90% (9/10) | |
| 2014i(10 Knowledge, 0 Case) | 80% (8/10) | 0% (0/0) | 80% (8/10) | |
| 2013s(10 Knowledge, 0 Case) | 80% (8/10) | 0% (0/0) | 80% (8/10) | |
| 2013i(10 Knowledge, 0 Case) | 60% (6/10) | 0% (0/0) | 60% (6/10) | |
| 2012s(8 Knowledge, 2 Case) | 88% (7/8) | 0% (0/2) | 70% (7/10) | |
| 2012i(8 Knowledge, 2 Case) | 63% (5/8) | 50% (1/2) | 60% (6/10) | |
| Sequential Query | 2021(5 Knowledge, 5 Case) | 80% (4/5) | 60% (3/5) | 70% (7/10) |
| 2020(6 Knowledge, 4 Case) | 50% (3/6) | 75% (3/4) | 60% (6/10) | |
| 2019(5 Knowledge, 5 Case) | 60% (3/5) | 80% (4/5) | 70% (7/10) | |
| 2018(7 Knowledge, 3 Case) | 71% (5/7) | 67% (2/3) | 70% (7/10) | |
| 2017(8 Knowledge, 0 Case) | 38% (3/8) | 0% (0/0) | 38% (3/8) | |
| 2016(9 Knowledge, 1 Case) | 78% (7/9) | 0% (0/1) | 70% (7/10) | |
| 2015(10 Knowledge, 0 Case) | 70% (7/10) | 0% (0/0) | 70% (7/10) | |
| 2014s(10 Knowledge, 0 Case) | 80% (8/10) | 0% (0/0) | 80% (8/10) | |
| 2014i(10 Knowledge, 0 Case) | 60% (6/10) | 0% (0/0) | 60% (6/10) | |
| 2013s(10 Knowledge, 0 Case) | 50% (5/10) | 0% (0/0) | 50% (5/10) | |
| 2013i(10 Knowledge, 0 Case) | 60% (6/10) | 0% (0/0) | 60% (6/10) | |
| 2012s(8 Knowledge, 2 Case) | 63% (5/8) | 100% (2/2) | 70% (7/10) | |
| 2012i(8 Knowledge, 2 Case) | 63% (5/8) | 0% (0/2) | 50% (5/10) |
Statistical analyses show that there is no significant difference between independent and sequential querying in terms of both knowledge and case questions (Table 8).
Table 8.
Year-Based performance of deepseek across knowledge and case questions using independent and sequential querying methods
| Query Type | N | Mean ± SD | p | |
|---|---|---|---|---|
| Knowledge Questions | Independent Query | 13 | 5.77 ± 2.20 | 0.77 |
| Sequential Query | 13 | 5.54 ± 1.71 | ||
| Case Questions | Independent Query | 13 | 0.77 ± 1.17 | 0.56 |
| Sequential Query | 13 | 1.08 ± 1.50 |
N Total number of exams, SD Standard deviation
Statistical significance was set at p < 0.05
Gemini 2.5 pro results
In the performance of the Gemini 2.5 Pro model on knowledge and case questions using independent and sequential querying methods, the Gemini 2.5 Pro model showed similar performance in both knowledge and case questions in independent and sequential querying (p > 0.05) (Table 2). Although a numerical decrease was observed in knowledge-based questions and a numerical increase was observed in case-based questions under sequential querying compared to independent querying, these differences were not statistically significant.
In the year-based analysis, Gemini 2.5 Pro showed its lowest knowledge accuracy of 63% under independent querying in 2017 and under sequential querying in 2012, while the lowest case-based accuracy was 50% for both methods (Table 9).
Table 9.
Gemini 2.5 pro model’s performance by year: success rates and Correct–Incorrect counts for Knowledge, Case, and total questions
| Gemini 2.5 Pro | Knowledge | Case | Total | |
|---|---|---|---|---|
| Independent Query | 2021(5 Knowledge, 5 Case) | 100% (5/5) | 80% (4/5) | 90% (9/10) |
| 2020(6 Knowledge, 4 Case) | 83% (5/6) | 100% (4/4) | 90% (9/10) | |
| 2019(5 Knowledge, 5 Case) | 100% (5/5) | 80% (4/5) | 90% (9/10) | |
| 2018(7 Knowledge, 3 Case) | 86% (6/7) | 67% (2/3) | 80% (8/10) | |
| 2017(8 Knowledge, 0 Case) | 63% (5/8) | 0% (0/0) | 63% (5/8) | |
| 2016(9 Knowledge, 1 Case) | 89% (8/9) | 100% (1/1) | 80% (8/10) | |
| 2015(10 Knowledge, 0 Case) | 90% (9/10) | 0% (0/0) | 90% (9/10) | |
| 2014s(10 Knowledge, 0 Case) | 90% (9/10) | 0% (0/0) | 90% (9/10) | |
| 2014i(10 Knowledge, 0 Case) | 80% (8/10) | 0% (0/0) | 80% (8/10) | |
| 2013s(10 Knowledge, 0 Case) | 100% (10/10) | 0% (0/0) | 100% (10/10) | |
| 2013i(10 Knowledge, 0 Case) | 80% (8/10) | 0% (0/0) | 80% (8/10) | |
| 2012s(8 Knowledge, 2 Case) | 100% (8/8) | 50% (1/2) | 90% (9/10) | |
| 2012i(8 Knowledge, 2 Case) | 88% (7/8) | 50% (1/2) | 80% (8/10) | |
| Sequential Query | 2021(5 Knowledge, 5 Case) | 100% (5/5) | 100% (5/5) | 100% (10/10) |
| 2020(6 Knowledge, 4 Case) | 83% (5/6) | 100% (4/4) | 90% (9/10) | |
| 2019(5 Knowledge, 5 Case) | 100% (5/5) | 100% (5/5) | 100% (10/10) | |
| 2018(7 Knowledge, 3 Case) | 71% (5/7) | 67% (2/3) | 70% (7/10) | |
| 2017(8 Knowledge, 0 Case) | 75% (6/8) | 0% (0/0) | 75% (6/8) | |
| 2016(9 Knowledge, 1 Case) | 100% (9/9) | 100% (1/1) | 100% (10/10) | |
| 2015(10 Knowledge, 0 Case) | 80% (8/10) | 0% (0/0) | 80% (8/10) | |
| 2014s(10 Knowledge, 0 Case) | 90% (9/10) | 0% (0/0) | 90% (9/10) | |
| 2014i(10 Knowledge, 0 Case) | 70% (7/10) | 0% (0/0) | 70% (7/10) | |
| 2013s(10 Knowledge, 0 Case) | 80% (8/10) | 0% (0/0) | 80% (8/10) | |
| 2013i(10 Knowledge, 0 Case) | 90% (9/10) | 0% (0/0) | 90% (9/10) | |
| 2012s(8 Knowledge, 2 Case) | 75% (6/8) | 100% (2/2) | 80% (8/10) | |
| 2012i(8 Knowledge, 2 Case) | 63% (5/8) | 50% (1/2) | 60% (6/10) |
In the exam-based performance of the Gemini 2.5 Pro model, no statistically significant difference was observed between independent and sequential querying methods for either knowledge questions or case questions (Table 10).
Table 10.
Year-Based performance of gemini 2.5 pro across knowledge and case questions using independent and sequential querying methods
| Query Type | N | Mean ± SD | p | |
|---|---|---|---|---|
| Knowledge Questions | Independent Query | 13 | 7.15 ± 1.77 | 0.66 |
| Sequential Query | 13 | 6.85 ± 1.73 | ||
| Case Questions | Independent Query | 13 | 1.31 ± 1.65 | 0.75 |
| Sequential Query | 13 | 1.54 ± 1.94 |
N Total number of exams, SD Standard deviation
Statistical significance was set at p < 0.05
Discussion
In the present study, the accuracy of responses provided by different LLMs to DUS questions was compared, revealing a statistically significant difference among the four models, primarily due to the lower performance of DeepSeek; accordingly, the first null hypothesis was rejected. In contrast, the impact of querying strategies was model-dependent. While no significant performance difference was observed for DeepSeek and Gemini 2.5 Pro between independent and sequential strategies, a statistically significant decline in knowledge question accuracy was detected for GPT-5 and GPT-4o (p < 0.05). Consequently, the second null hypothesis was rejected for the GPT models regarding knowledge questions, whereas it was accepted for the other models.
AI is a rapidly advancing technological field that aims to develop intelligent systems capable of mimicking human intelligence and performing various cognitive tasks [22]. The impact of AI is increasingly being felt in health sciences such as medicine and dentistry, finding application in processes such as diagnosis, treatment planning, and patient follow-up [23]. In addition, with the proliferation of AI-based applications in health education, new opportunities are emerging in teaching processes for both students and educators [14]. There are a limited number of studies in the literature on the capacity of AI models to correctly process theoretical knowledge and their exam performance. This situation indicates that more research is needed on the place and reliability of artificial intelligence in dental education.
This study evaluated four widely used large language models— ChatGPT-5 and ChatGPT-4o (OpenAI), DeepSeek, and Gemini 2.5 Pro (Google)—which have been increasingly examined in recent comparative studies and healthcare-related applications [10, 12, 13, 24–28]. These models were selected to represent differing architectures and update dynamics, enabling a focused and meaningful comparative assessment.
Although the use of publicly available examination questions may raise concerns regarding potential data contamination, prior studies have shown that LLM performance varies substantially across model versions and prompting strategies even when identical question sets are used. This variability suggests that observed performance reflects dynamic model behavior and contextual reasoning rather than simple memorization of publicly accessible content. Recording only the first response generated by each model represents a limitation of this study, as large language models are inherently stochastic and may produce variable outputs across repeated attempts. However, previous research has shown that repeated or sequential answer generation does not necessarily improve accuracy; in some models, the highest accuracy was achieved in the first response, with no consistent performance gain observed even by the fifth attempt [16]. Nevertheless, we acknowledge that evaluating a single response captures only one realization of model behavior, and future studies incorporating repeated trials may provide additional insight into intra-model variability.
Overall, the evaluated LLMs demonstrated higher accuracy in knowledge-based questions than in case-based questions, indicating stronger performance in theoretical content than in clinically oriented reasoning. This pattern is consistent with previous studies reporting limitations of LLMs in case interpretation and applied decision-making despite high factual accuracy [29–31]. These findings highlight the need for domain-specific optimization, including the development of algorithms tailored to case-based dental questions and the use of high-quality, clinically annotated datasets.
Comparison of independent and sequential querying revealed distinct trends: while sequential querying led to a decline in knowledge-question performance across all models (statistically significant for GPT-5 and GPT-4o), it generally resulted in numerical improvements for case-based questions, with the exception of GPT-4o, which maintained stable performance. This pattern suggests differential sensitivity to contextual carryover across models, indicating that sequential querying does not uniformly induce learning bias but interacts with model architecture and question type. These findings provide novel evidence that LLM performance in dental examinations is shaped not only by knowledge representation but also by querying methodology and context management. Although there are studies in the literature measuring the performance of LLMs on dental exams [10, 13, 26, 28–32], no study has been found that compares independent and sequential questioning. Therefore, the findings reveal that artificial intelligence may exhibit performance differences not only in knowledge transfer but also in the context of questioning methods and question types.
Although the use of artificial intelligence–based language models in medicine is increasing, their application without evaluating consistency across different specialties and querying conditions may be misleading in clinical decision support processes [14]. In the present study, performance declined markedly in some models, with reduced accuracy for knowledge-based questions under sequential querying in ChatGPT-5, in contrast, the Gemini 2.5 Pro model showed a numerical decrease in performance on knowledge-based questions and a numerical increase on case-based questions; however, these changes did not reach statistical significance. While there are similar studies in the literature that find Gemini to be more successful than ChatGPT [13, 26, 32] there are also studies that publish opposite results [10, 29, 31]. The results reveal that the reliability of AI-based language models in medical applications can vary significantly according to the query method and question type. Tosun et al. [29] evaluated the performance of seven AI-based chatbots in correctly answering questions about DUS Prosthetic Dentistry treatment, reporting that ChatGPT-4o achieved the highest accuracy (75.8%), while Gemini AI had the lowest accuracy (46.1%). Although the current study used the same questions, we achieved success rates of 86% and 85% for ChatGPT-4o and Gemini 2.5 Pro, respectively. This is significant in that it demonstrates that AI models have shown a meaningful increase in performance in knowledge tests over time and that current versions can be used more reliably in dental education. Aşık et al. [24] directed questions from past DUS Pedodontics exams to the free version of Chat GPT 3.5 and achieved a 54.9% accuracy rate for all questions. This study found that responses to the 2014 DUS questions had the highest accuracy. The lowest success rate was 10% for questions from 2019 − 2015. While researchers claime that ChatGPT is not a reliable source, this conclusion is supported by the fact that ChatGPT versions achieved success rates of up to 100% in the current study, with the lowest success rate obtained from the spring 2014 DUS. Considering that even the latest versions available as of September 2025—most notably ChatGPT-5—have shown a decline to a success rate of approximately 50% in some examinations, it can be argued that the model is still not a fully reliable source of information.
In a recent study published in 2025, the performance of ChatGPT-4o (OpenAI), DeepSeek-v3 (DeepSeek), Gemini 2.0 (Google DeepMind), and Claude 3.7 Sonnet (Anthropic) in answering DUS Anatomy questions. ChatGPT-4o achieved the highest accuracy (98.6%), while other models achieved 89.2% accuracy. The fastest results were obtained from Gemini (average: 4.47 s). All questions were loaded in Turkish, and no translation was performed [12]. Literature findings also support the highest success rate of Chat GPT in this study, similar to other studies that solved anatomy questions using different LLMs [33]. In the present study, the questions were also asked in Turkish. The main reason for this is that the DUS is administered in Turkish in Türkiye and medical candidates prepare for the exam in their native language; it has also been observed that the Turkish language is used in similar studies [10, 12]. In similar studies in the literature, comparisons have been made between questions asked in Turkish and English, and it has been reported that the models achieved higher success with questions in English [25, 28]. However, the fact that artificial intelligence models are predominantly trained on English datasets is important for understanding performance variations across different languages. Therefore, not only the knowledge levels of the models but also their ability to generate responses in another language were evaluated. In a 2025 study, the responses of ChatGPT-4o, Gemini 2.0, and DeepSeek-R1 to DUS Endodontics questions were analyzed with respect to language. It was reported that all models achieved higher accuracy for questions asked in English, while DeepSeek-R1 and Gemini 2.0 outperformed ChatGPT-4o for Turkish questions [28]. Sarı et al. [25] evaluated the responses generated by ChatGPT-4o to DUS questions released between 2012 and 2021 and reported the lowest success rate in the field of Prosthetic Dental Treatment (20.1%). In the present study, however, higher accuracy rates were obtained for Prosthetic Dental Treatment questions posed to ChatGPT-4o in Turkish. Moreover, in Turkish-language questions, the ChatGPT-5 model achieved higher accuracy in knowledge-based questions compared with DeepSeek and Gemini 2.5 Pro. Accordingly, these findings suggest that ChatGPT models have shown substantial improvement over time and that multilingual artificial intelligence models are increasingly reducing performance differences across languages.
Previous studies in basic medical sciences, such as embryology, have shown that advanced LLMs can achieve high accuracy and good reliability in structured, knowledge-based examinations, particularly with models such as GPT-4o and Claude, when instructor verification is ensured [15]. Similarly, a study conducting a meta-analysis of studies evaluating the performance of ChatGPT-3. 5 or ChatGPT-4o in national licensing exams for medicine, pharmacy, dentistry, and nursing, calculated the average accuracy rate as 70.1% and reported that ChatGPT-4o showed statistically significant higher success compared to ChatGPT-3.5. The highest success rate across disciplines was observed in pharmacy, followed by medicine, dentistry, and nursing [9]. Studies in the literature have also found that the Chat GPT-4o version is more successful than the 3.5 version [11]. These results support our finding that the new versions perform better; however, the relatively lower accuracy rates in the field of dentistry suggest that this is due to the field requiring case interpretation and clinical reasoning skills rather than knowledge- and memorization-based questions.
Artificial intelligence models support the learning process in dental education through continuous availability and rapid feedback and may offer expanded training opportunities using clinical scenarios in the future. AI-based chatbots are increasingly used for academic support, educational guidance, and responding to questions from medical and dental students, as well as for assisting clinicians and informing patients [34]. However, models such as ChatGPT and Gemini are not specifically designed for medical or dental decision-making and may lack sufficient contextual understanding of complex disease–treatment relationships [35]. Nevertheless, several studies have demonstrated that these models can successfully answer expert-level questions [36–38]. In light of the present findings, artificial intelligence systems should be regarded as complementary tools, rather than primary resources, in dental clinical and theoretical education.
From a practical perspective, the implementation of large language models in dental education should consider cost-effectiveness, institutional access, and intended educational use. Models with higher standalone accuracy may be more suitable for individual self-study and knowledge reinforcement, whereas models demonstrating stable performance under sequential querying may be better suited for examination simulation and case-based learning environments. Given the variability in licensing models, computational requirements, and update frequency, dental schools should prioritize flexible, scalable solutions aligned with their curricular goals rather than a single “best” model.
The public accessibility of the examination material constitutes a limitation regarding potential data contamination. Despite this, our findings are consistent with recent literature indicating that large language models may produce variable outputs even when responding to familiar datasets, particularly following model updates or changes in prompting strategies. A key limitation of this study is that each model response was obtained from a single run per question and querying method. Given the stochastic nature of large language models, this approach reflects a single realization rather than a stable performance estimate; therefore, intra-model variability could not be assessed, and accuracy rates may under- or overestimate true performance. Nevertheless, this design mirrors real-world examination conditions, in which candidates are permitted only one response per question. Although artificial intelligence models demonstrated strong performance in structured, knowledge-based DUS questions, they lack the situational judgment and practical skills required for real clinical decision-making. Consequently, examination success cannot be directly extrapolated to clinical competence. In addition, the use of Turkish-language questions represents another limitation, as model performance may differ compared with English-language inputs due to training data imbalance.
To explore potential learning bias, questions were presented using sequential querying; although a numerical increase was noted in case-based questions, the method caused a significant decline in knowledge accuracy for GPT models in the aggregate dataset, which, however, did not retain statistical significance in the year-by-year analysis. Nevertheless, question order and presentation may still have exerted a minor influence on performance. Furthermore, performance differences among the evaluated language models indicate that the results are specific to the models tested and cannot be generalized to all artificial intelligence systems. Accordingly, artificial intelligence tools should be considered supportive aids rather than replacements for traditional dental education or clinical expertise.
Conclusion
The study results demonstrate that the question format affects the accuracy of the AI model used, suggesting that AI-based chatbots can be used as supplemental training material in knowledge-intensive exams such as the DUS. However, it should be noted that the models still have limitations in higher-level cognitive skills such as clinical reasoning, situational judgment, and patient-specific decision-making, and they should not replace clinical decision-making processes. Future studies are recommended to evaluate model performance in different languages, case-based scenarios, and situations requiring clinical decision-making.
Acknowledgements
The authors reviewed and edited all outputs and take full responsibility for the content.
Abbreviations
- LLMs
Large language models
- DUS
Dentistry Specialization Exam
- AI
Artificial Intelligence
- NLP
Natural Language Processing
- ÖSYM
Student Selection and Placement Center
Authors’ contributions
S.G.E.: Methodology; Investigation (data collection); Writing—original draft, A.B.Ç.G.: Methodology; Investigation (data collection); Writing—original draft. A.K.: Supervision; Validation; Writing—review & editing. All authors have read and approved the final manuscript.
Funding
This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.
Data availability
The datasets generated and/or analyzed during the current study are available from the corresponding author on reasonable request.
Declarations
Ethics approval and consent to participate
Not applicable.
Consent for publication
Not applicable.
Competing interests
The authors declare no competing interests.
Footnotes
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.Topol E. Deep medicine: how artificial intelligence can make healthcare human again. New York: Basic Books; 2019. [Google Scholar]
- 2.Jiang F, Jiang Y, Zhi H, Dong Y, Li H, Ma S, et al. Artificial intelligence in healthcare: past, present and future. Stroke Vasc Neurol. 2017;2(4):230–43. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Ding H, Wu J, Zhao W, Matinlinna JP, Burrow MF, Tsoi JKH. Artificial intelligence in dentistry-A review. Front Dent Med. 2023;4:1085251. [DOI] [PMC free article] [PubMed]
- 4.Dave M, Patel N. Artificial intelligence in healthcare and education. Br Dent J. 2023;234(10):761–4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.OpenAI. GPT-4 Technical Report. 2023. Cited 29 Oct 2025. https://openai.com/research/gpt-4.
- 6.Brown T, Mann B, Ryder N, Subbiah M, Kaplan JD, Dhariwal P. Language models are few-shot learners. Adv Neural Inf Process Syst. 2020;33:1877–901. [Google Scholar]
- 7.Eraslan R, Ayata M, Yagci F, Albayrak H. Exploring the potential of artificial intelligence chatbots in prosthodontics education. BMC Med Educ. 2025;25(1):321. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Dashti M, Khosraviani F, Azimi T, Hefzi D, Ghasemi S, Fahimipour A, et al. Assessing ChatGPT-4’s performance on the US prosthodontic exam: impact of fine-tuning and contextual prompting vs. base knowledge, a cross-sectional study. BMC Med Educ. 2025;25(1):761. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Jin HK, Lee HE, Kim E. Performance of ChatGPT-3.5 and GPT-4 in National licensing examinations for medicine, pharmacy, dentistry, and nursing: a systematic review and meta-analysis. BMC Med Educ. 2024;24(1):1013. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Sismanoglu S, Capan BS. Performance of artificial intelligence on Turkish dental specialization exam: can ChatGPT-4.0 and gemini advanced achieve comparable results to humans? BMC Med Educ. 2025;25(1):214. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Revilla-León M, Barmak BA, Sailer I, Kois JC, Att W. Performance of an artificial Intelligence–Based chatbot (ChatGPT) answering the European certification in implant dentistry exam. Int J Prosthodont. 2024;37(2):221–4. [DOI] [PubMed] [Google Scholar]
- 12.Tassoker M. Who knows anatomy best? A comparative study of ChatGPT-4o, DeepSeek, gemini and Claude. Clin Anat. 2025. 10.1002/ca.70012. Epub ahead of print. [DOI] [PubMed] [Google Scholar]
- 13.Taymour N, Fouda SM, Abdelrahaman HH, Hassan MG. Performance of the ChatGPT-3.5, ChatGPT-4, and Google gemini large Language models in responding to dental implantology inquiries. J Prosthet Dent. 2025;134(6):2427–34. [DOI] [PubMed] [Google Scholar]
- 14.Yilmaz BE, Gokkurt Yilmaz BN, Ozbey F. Artificial intelligence performance in answering multiple-choice oral pathology questions: a comparative analysis. BMC Oral Health. 2025;25(1):573. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Bolgova O, Ganguly P, Mavrych V. Comparative analysis of LLMs performance in medical embryology: A cross-platform study of ChatGPT, Claude, gemini and copilot. Anat Sci Educ. 2025;18(7):718–26. [DOI] [PubMed] [Google Scholar]
- 16.Mavrych V, Ganguly P, Bolgova O. Using large Language models (ChatGPT, Copilot, PaLM, bard and Gemini) in gross anatomy course: comparative analysis. Clin Anat. 2025;38(2):200–10. [DOI] [PubMed] [Google Scholar]
- 17.Gokkurt Yilmaz BN, Ozbey F, Yilmaz BE. Evaluation of the performance of different large Language models on head and neck anatomy questions in the dentistry specialization exam in Turkey. Surg Radiol Anat. 2025;47(1):211. [DOI] [PubMed] [Google Scholar]
- 18.DeepSeek-AI. DeepSeek-V3 Technical Report. 2024. Available from: https://arxiv.org/abs/2412.19437. Cited 2 Jan 2026.
- 19.OpenAI. GPT-4o System Card. 2024. Available from: https://cdn.openai.com/gpt-4o-system-card.pdf. Cited 2 Jan 2026.
- 20.Google, DeepMind. Gemini 2.5 Pro Technical Report: Advanced Reasoning and Long Context Capabilities. 2025. Available from: https://modelcards.withgoogle.com/assets/documents/gemini-2.5-pro.pdf. Cited 2 Jan 2026.
- 21.OpenAI. GPT-5 System Card. 2025. Available from: https://cdn.openai.com/gpt-5-system-card.pdf. Cited 2 Jan 2026.
- 22.Alowais SA, Alghamdi SS, Alsuhebany N, Alqahtani T, Alshaya AI, Almohareb SN, et al. Revolutionizing healthcare: the role of artificial intelligence in clinical practice. BMC Med Educ. 2023;23(1):689. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Bindra S, Jain R. Artificial intelligence in medical science: a review. Ir J Med Sci (1971 -). 2024;193(3):1419–29. [DOI] [PubMed] [Google Scholar]
- 24.Aşık A, Kuru E. Analysis of chatgpt’s answers to pedodontics questions asked in the dentistry specialization training entrance exam: Cross-Sectional study. Turkiye Klinikleri J Dent Sci. 2025;31(3):401–6. [Google Scholar]
- 25.Dündar Sarı MB, Sezer B. ChatGPT-4 omni’s accuracy in Multiple-Choice dentistry questions: A multidisciplinary and bilingual assessment. Essentials Dentistry. 2025;4(1):1–9. [Google Scholar]
- 26.Bilgin Avşar D, Ertan AA. A comparative study of ChatGPT-3.5 and gemini’s performance of answering the prosthetic dentistry questions in dentistry specialty exam: Cross-Sectional study. Turkiye Klinikleri J Dent Sci. 2024;30(4):668–73. [Google Scholar]
- 27.Lee S, Jung S, Park JH, Cho H, Moon S, Ahn S. Performance of ChatGPT, gemini and deepseek for non-critical triage support using real-world conversations in emergency department. BMC Emerg Med. 2025;25(1):176. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Şahin EG. Comparative performance of ChatGPT, Gemini, and deepseek on endodontic exam questions in Turkish and english. Preprint Res Square 2025. 2025. 10.21203/rs.3.rs-6738945/v1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Tosun B, Yilmaz Z, Sen. Comparison of artificial intelligence systems in answering prosthodontics questions from the dental specialty exam in Turkey. J Dent Sci. 2025;20(3):1454–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Temiz M, Güzel C. Assessing the performance of ChatGPT on dentistry specialization exam questions: A comparative study with DUS examinees. Med Records. 2025;7(1):162–6. [Google Scholar]
- 31.Kinikoglu I. Evaluating ChatGPT and Google gemini performance and implications in Turkish dental education. Cureus. 2025;17(1):e77292. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Deveci BA, Kocuk M, Şahin M, Yanikoglu F. The performance of AI on the dentistry specialization exam. Int Dent J. 2024;74:S114–5. [Google Scholar]
- 33.Al-Khater KMK. Comparative assessment of three AI platforms in answering USMLE step 1 anatomy questions or identifying anatomical structures on radiographs. Clin Anat. 2025;38(2):186–99. [DOI] [PubMed] [Google Scholar]
- 34.Eggmann F, Weiger R, Zitzmann NU, Blatz MB. Implications of large Language models such as ChatGPT for dental medicine. J Esthetic Restor Dentistry. 2023;35(7):1098–102. [DOI] [PubMed] [Google Scholar]
- 35.Cascella M, Montomoli J, Bellini V, Bignami E. Evaluating the feasibility of ChatGPT in healthcare: an analysis of multiple clinical and research scenarios. J Med Syst. 2023;47(1):33. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor RA, et al. How does ChatGPT perform on the united States medical licensing examination (USMLE)? The implications of large Language models for medical education and knowledge assessment. JMIR Med Educ. 2023;9:e45312. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Hoch CC, Wollenberg B, Lüers JC, Knoedler S, Knoedler L, Frank K, et al. ChatGPT’s quiz skills in different otolaryngology subspecialties: an analysis of 2576 single-choice and multiple-choice board certification Preparation questions. Eur Arch Otorhinolaryngol. 2023;280(9):4271–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Skalidis I, Cagnina A, Luangphiphat W, Mahendiran T, Muller O, Abbe E, et al. ChatGPT takes on the European exam in core cardiology: an artificial intelligence success story? Eur Heart J - Digit Health. 2023;4(3):279–81. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
The datasets generated and/or analyzed during the current study are available from the corresponding author on reasonable request.