Do question type and topic affect the performance of large language models in answering periodontology questions? A comparative study

Abstract

Background

There is increasing interest and ongoing debate about the integration of artificial intelligence and the large language models it supports in health science education. The performance of these models may vary depending on educational domains, clinical branches, complexity of questions, and question formats. This study aimed to compare the performance of ChatGPT-5, Gemini 2.5 Flash, and Copilot on periodontology questions and to determine whether their performance was affected by question type and topic.

Methods

One hundred thirty-four multiple-choice periodontology questions from the dental specialty exam in Türkiye were fed to these models. The responses were evaluated based on both their accuracy and the quality of their explanations. A Likert scale was used to examine the quality of explanation. The answers from the models were analyzed comparatively based on question type and topic. Regression analyses were performed to determine the effect of question type and topic.

Results

The total accuracy rates of ChatGPT-5, Gemini 2.5 Flash, and Copilot were similar. Overall, Gemini 2.5 Flash outperformed the others in terms of adequacy of response explanation (p < 0.05). Gemini provided more consistent explanations than others on basic science and clinical periodontology questions, as well as standardized and information-based questions (p < 0.05, for all). Regression analysis showed that question type and topic were related to explanation quality.

Conclusions

According to the findings of this study, question types and topics seem to affect the consistency and quality of large language models’ explanations more than the accuracy of the answers. This suggests that large language models are promising for periodontology education, but they still have limitations that require improvement. However, these findings should be interpreted cautiously within the limitations of the study.

Supplementary Information

The online version contains supplementary material available at https://doi.org/10.1186/s12909-026-10516-z.

Background

Large language models (LLMs) are high-performance artificial intelligence systems that understand and generate natural language using neural networks and deep learning [1]. LLMs are considered promising opportunities for application in medical education, patient care/clinical practice, and medical research [2–4]. Previous studies have shown that LLMs' performance on medical licensing, board, and specialty exams in medicine and dentistry is close to expert level [5–9]. This performance of LLMs has led them to be seen as potential educational tools. Previous studies have also reported that LLMs can achieve moderate to high accuracy in undergraduate and postgraduate dental education settings [10, 11]. However, there are implementation issues to be addressed and resolved such as ensuring reproducibility of results, transparency of reasoning, and domain-specific accuracy [12–14].

Due to the rapid proliferation of LLMs, further research is needed to assess decision-making processes in domain-specific clinical scenarios and to integrate them in domain-specific education [14, 15]. Furthermore, the datasets used to train LLMs generally lack domain-specific, in-depth content and consist of foundational medical knowledge. Current educational data and benchmarking methodologies may be insufficient for LLMs to obtain the necessary domain-specific knowledge and clinical competencies [3].

Multiple choice questions (MCQs) are frequently used to assess the knowledge levels of students and residents in medicine and dentistry [16]. Their ease of application and objective scoring make them a preferred tool in summative evaluations [17]. Depending on the MCQ type, direct knowledge recall or synthesis and contextual judgment abilities can be assessed [17]. LLMs have a high ability to answer MCQs [18], but the complex structure of the questions may affect their performance [19]. Standard MCQs primarily rely on direct recall, whereas knowledge-based questions require comprehensive knowledge integration, and case-based questions require both information integration and complex reasoning [20]. Educational tools such as Bloom's taxonomy are used to design high quality MCQs [21]. Bloom's Taxonomy provides a framework to distinguish recall-based knowledge from higher-level processes such as application and understanding [22, 23]; making it particularly relevant for assessing whether LLM performance in dentistry reflects simple recall or applied clinical reasoning. As a result, the depth and consistency of the model outputs may be affected. Therefore, understanding the effectiveness of LLMs on different question types and topics can contribute to understanding their depth of reasoning and potential for integration into education.

Dental specialization training has been officially implemented in Türkiye since 2012. Admission to dental specialty programs requires success in the Dental Specialization Examination (DUS), a nationwide, centralized examination coordinated and administered by the Student Selection and Placement Centre (ÖSYM). The DUS consists of MCQs covering both basic sciences and clinical dentistry disciplines, including periodontology. Periodontology is a unique branch of dentistry that requires basic medical knowledge as well as clinical reasoning and surgical knowledge and skills, thereby engaging multiple cognitive levels. Given its multidimensional structure, periodontology is an ideal testing ground to evaluate the capabilities of LLMs.

Previous studies evaluating LLM performance on periodontology-related questions have reported accuracy rates ranging approximately from 57 to 80% [24–26]. Additionally, studies have assessed LLM performance on DUS questions in other clinical branches, such as endodontics and prosthodontics [27, 28]. A recent study classified periodontology questions by topic areas and found that LLMs exhibited lower accuracy rates on clinical periodontology [29]. However, to the author’s knowledge, no study to date has examined LLMs performance on both different question types and topic areas in answering periodontology questions from the DUS in Türkiye. In this context, periodontology questions obtained from DUS were fed to three different LLMs, and their performances were comparatively evaluated. Additionally, the present study was aimed to compare the performance of the LLMs in answering periodontology questions and to evaluate whether question type and topic area affected their performance.

Methods

Study design

The present study was designed as a comparative, cross-sectional study. All data were obtained from a publicly accessible database. Ethics approval was not required because the study did not involve human participants or animal subjects.

In this study, periodontology questions from DUS were presented to three LLMs, ChatGPT-5 (OpenAI, San Francisco, CA, USA, access date: 16 October 2025), Gemini 2.5 Flash (Google LLC., Mountain View, CA, USA, access date: 16 October 2025), and Copilot (Microsoft Corporation, WA, USA, access date: 16 October 2025), and their ability to answer was evaluated. These LLMs were selected because they are widely documented in the medical and dental education literature. However, their evaluation according to question type and topic in periodontology is limited.

Data source and question selection

All periodontology questions used in this study were obtained from the publicly available database of the ÖSYM (https://www.osym.gov.tr), which openly publishes past DUS examinations [30]. Between 2012 and 2025, ÖSYM published 137 MCQs in periodontology, each containing five answer options with one correct answer. Three of these questions were excluded, including two questions that were canceled by ÖSYM in 2013 and 2014 and one question that contained an image. After these exclusions, 134 text-based questions were included in this study (Fig. 1).

Fig. 1.

Flowchart of the study design

Categorization of questions

Each question was categorized by an experienced periodontist according to both periodontology topic and question type.

Periodontology topics were divided into three categories: (i) Basic periodontology (n = 82), including anatomy, microbiology, histopathology, and etiology; (ii) Clinical periodontology (n = 46), including clinical diagnosis, treatment planning, and surgery; and (iii) Oral implantology (n = 6), including implant-related biological and surgical principles.

Question types were divided into three groups and aligned with Bloom’s taxonomy of cognitive domains [31]: (i) Standard MCQs (Remembering level) (n = 97, 72.4%), stem questions requiring direct recall. (ii) Information-based questions (Understanding level) (n = 17, 12.7%), premise-linked questions that require integration of supplementary data. (iii) Case-based (Appling level) (n = 20, 14.9%) questions, clinical scenario or patient vignette questions assessing applied reasoning. (Supplementary Fig. 1).

The final dataset contained of 134 questions distributed across these topic and question type categories.

Administration of questions

The original Turkish DUS periodontology questions (including answer options) were entered separately into ChatGPT-5, Gemini 2.5 Flash, and Copilot, and all the question-response phase were completed on 16 October 2025. All LLMs outputs were generated using the most current publicly accessible versions available through standard interfaces on the day of testing. ChatGPT-5 was accessed via a ChatGPT Plus subscription; Gemini 2.5 Flash via the respective free public interface, and Copilot via the free web interface. Before questioning each LLM, the following prompt was put: "Can you tell me the correct answer to the multiple-choice questions I will ask below?" to ensure standardization. No system instructions were modified, chain-of-thought prompting was not used, and all models operated with their default platform settings, including temperature and randomness parameters. All LLMs were accessed via the same computer using Google Chrome. Each question was entered in a separate, clean conversation window. Prior to testing, the system memory was cleared, and previous chats and cookies were completely deleted. Each question was presented only once to avoid potential learning or adaptation effects from repetition [32]. Since the selected LLMs provide multi-language support, all communication, including the prompts and questions, was conducted entirely in Turkish. No feedback, correction, or reinforcement was provided during the process. All evaluated LLMs spontaneously generated explanatory text alongside the selected answer as part of their default response behavior. The raw response comprised of selected answers and explanatory text of each model was recorded verbatim for subsequent evaluation. Representative screenshots illustrating the prompt, the multiple-choice question (including answer options), and the corresponding LLM outputs are provided in Supplementary Fig. 2.

Scoring of responses

Each LLM response was independently evaluated for accuracy and qualitative reasoning quality using two complementary measures:

• Accuracy: Each response was compared with the ÖSYM official answer key and recorded as correct or incorrect.

• Ratings of response explanations: An experienced periodontist (H.M.P.) with combined academic and clinical experience rated each explanation according to overall appropriateness and coherence to the correct answer using a predefined three-point Likert scale (0–2) [33]: 0-point: both response and explanation were incorrect, 1-point: correct response but the explanation was inaccurate and/or insufficient, and 2-point: the response was accurate and the explanation was appropriate and sufficiently articulated. A Likert scale assessment format has been frequently used in previous studies evaluating LLM performance, to provide a standardized and transparent scoring framework while maintaining reproducibility.

In addition, the total score was calculated by adding up the ratings of response explanations given to all questions.

Statistical analysis

SPSS v28.0 software (IBM Corp., Armonk, NY, USA) was used for data analysis. Categorical variables were shown as frequencies (n) and percentages (%), whereas numerical variables were expressed as mean and standard deviation (SD) values. Inter-group comparisons of the numerical data were made by using the Kruskal–Wallis H test, whereas the Chi-square test and Fisher’s Exact test were used for comparing the categorical data. The relationship between categorical data was investigated using the Phi Cramer-V test. Binary logistic regression analysis was applied to evaluate the effect of the variables on response accuracy. Multiple regression analysis was used to determine the effect of independent variables (LLMs type, topic, and question type) on the dependent variable (total score). To perform multiple regression analysis, outliers were removed and the data were normalized. In addition, linearity, multicollinearity, VIF, autocorrelation, and homoscedasticity checks were performed before multiple regression analysis. The level of statistical significance was set at p < 0.05.

A priori sample size calculation was not conducted. Post-hoc power analysis was performed to verify that the sample size of 134 questions was large enough to detect the significant differences. The study's power was 84.2% for a sample of 134 questions with a Type I error level of 0.05 (95% reliability), an effect size of 0.3 (medium), and 95% reliability, as calculated with G*Power 3.1.9.4 program.

Results

A total of 134 periodontology MCQs were asked to ChatGPT-5, Gemini 2.5 Flash, and Copilot. The answers of LLMs were comparatively analyzed.

Overall accuracy rates for each LLM were 89.6% for ChatGPT-5, 94% for Gemini 2.5 Flash, and 88.8% for Copilot. Although Gemini 2.5 Flash achieved the highest total accuracy rate, the difference among LLMs was not statistically significant (p = 0.278). Regarding the ratings of response explanations, Gemini 2.5 Flash had a larger proportion of 2-point responses (93.3%) compared with ChatGPT-5 (72.4%) and Copilot (68.7%) (p = 0.001). Gemini 2.5 Flash achieved 251 points, ChatGPT-5 217 points, and Copilot 211 points, with the total score differing significantly between LLMs (p = 0.018) (Table 1).

Table 1.

Comparison of the total accuracy rate, the distributions of ratings of response explanations, and total score of LLMs

	ChatGPT-5	Gemini 2.5 Flash	Copilot	p
Total Accuracy Ratea	120/134 (89.6%)	126/134 (94.0%)	119/134 (88.8%)	0.278
Ratings of response explanationsa n(%)				0.001*
0-point	14 (10.4)	8 (6.0)	15 (11.2)
1-point	23 (17.2)	1 (0.7)	27 (20.1)
2-point	97 (72.4)	125 (93.3)	92 (68.7)
Total Scoresb	217	251	211	0.018*

LLMs Large Language Models

^*p < 0.05

a: chi-square and Fisher Exact test

b: Kruskal Wallis-H test and Bonferroni test

For standard MCQs, accuracy was high across LLMs (88.7–93.8%, p = 0.435). For information-based questions, Gemini 2.5 Flash answered all items correctly (100%), followed by Copilot (94.1%) and ChatGPT-5 (82.4%) (p = 0.314). For case-based questions, the accuracy rates were similar among LLMs (85.0–90.0%, p = 0.886). Despite comparable accuracy rates, the ratings of response explanations varied significantly according to question type. Gemini 2.5 Flash provided the highest proportion of 2-point responses in both standard (p = 0.001) and information-based (p = 0.006) question categories compared to ChatGPT-5 and Copilot, no significant difference was observed for case-based questions (p = 0.106) (Table 2).

Table 2.

Comparison of accuracy rates and the distributions of ratings of response explanations of LLMs based on question type

Accuracy rates	ChatGPT-5	Gemini 2.5 Flash	Copilot	Total	p
	n (%)	n (%)	n (%)	n (%)
Standard MCQ
Correct	89 (91.8)	91 (93.8)	86 (88.7)	266 (91.4)	0.435a
Incorrect	8 (8.2)	6 (6.2)	11 (11.3)	25 (8.6)
Information-based MCQ
Correct	14 (82.4)	17 (100)	16 (94.1)	16 (94.1)	0.314b
Incorrect	3 (17.6)	0 (0)	1 (5.9)	4 (5.9)
Case-based MCQ
Correct	17 (85.0)	18 (90.0)	17 (85.0)	52 (86.7)	0.886a
Incorrect	3 (15.0)	2 (10.0)	3 (15.0)	8 (13.3)
Phi (Φ)Value	0.119	0.112	0.076	0.060
pᶜ	0.389	0.435	0.678	0.480
%95 CI (Lower–Upper)	0.033–0.359	0.070–0.306	0.022–0.275	0.017–0.185
Ratings of response explanations
Standard MCQ					0.001*a
0-point	8 (8.2)	6 (6.2)	1 (11.3)	25 (8.6)
1-point	21 (21.6)	1 (1.0)	17 (17.5)	39 (12.4)
2-point	68 (70.1)	90 (92.8)	69 (71.1)	227 (78.0)
Information-based MCQ					0.006*ᵇ
0-point	3 (17.6)	0 (0)	1 (5.9)	4 (7.8)
1-point	2 (11.8)	0 (0)	6 (35.3)	8 (15.7)
2-point	12 (70.6)	17 (100)	10 (58.8)	39 (76.5)
Case-based MCQ					0.106ᵇ
0-point	3 (15.0)	2 (10.0)	3 (15.0)	8 (13.3)
1-point	0 (0)	0 (0)	4 (20.0)	4 (6.7)
2-point	17 (85.0)	18 (90.0)	13 (65.0)	48 (80.0)
Phi (Φ)Value	0.230	0.124	0.157	0.094
pᶜ	0.132	0.725	0.506	0.467
%95 CI (Lower–Upper)	0.178–0.426	0.080–0.306	0.091–0.387	0.054–0.219

CI Confidence Interval, MCQ multiple-choice questions, LLMs Large Language Models

^*p < 0.05

^a: chi-square test

^b: Fisher Exact test

^c: Phi Cramer V test

When each model’s performance was analyzed by question type no significant differences were observed in either accuracy rate or the ratings of response explanations (p > 0.05, for all). Across different question types, ChatGPT-5 provided accuracy between 82.4% and 91.8% (Φ = 0.119, p = 0.389), while Gemini 2.5 Flash showed 90–100% (Φ = 0.112, p = 0.435) and Copilot 85–94.1% (Φ = 0.076, p = 0.678). For all three models, the distribution of 0-, 1-, and 2-point responses did not differ significantly among standard, information-based, and case-based MCQs (p > 0.05, for all) (Table 2).

When accuracy rates were analyzed across topics, none of the differences between LLMs reached statistical significance (p > 0.05, for all). For basic periodontology, the accuracy rates ranged from 91.5% to 95.1%, for clinical periodontology from 84.8% to 93.5%, and for oral implantology from 83.3% to 83.3%. However, the ratings of response explanations by topic showed significant differences for both basic (p = 0.001) and clinical (p = 0.009) domains. Gemini 2.5 Flash yielded the highest proportion of 2-point responses (95.1% in basic and 91.3% in clinical questions), compared to ChatGPT-5 (76.8% in basic and 67.4% in clinical questions) and Copilot (70.7% in basic and 63.0% in clinical questions). No significant differences were found for implantology questions (p = 0.625) (Table 3). When each model’s performance was analyzed by topic, no significant differences were observed in either accuracy rate or ratings of response explanations (p > 0.05, for all) (Table 3).

Table 3.

Comparison of accuracy rates and the distributions of ratings of response explanations of LLMs based on topics

Accuracy rates	ChatGPT-5	Gemini 2.5 Flash	Copilot	Total	p
	n (%)	n (%)	n (%)	n (%)
Basic periodontology
Correct	75 (91.5)	78 (95.1)	75 (91.5)	228 (92.7)	0.583ᵇ
Incorrect	7 (8.5)	4 (4.9)	7 (8.5)	18 (7.3)
Clinical periodontology
Correct	40 (87.0)	43 (93.5)	39 (84.8)	122 (88.4)	0.399a
Incorrect	6 (13.0)	3 (6.5)	7 (15.2)	16 (11.6)
Oral implantology
Correct	5 (83.3)	5 (83.3)	5 (83.3)	15 (83.3)	1.000a
Incorrect	1 (16.7)	1 (16.7)	1 (16.7)	3 (16.7)
Phi (Φ)Value	0.082	0.103	0.106	0.089
pᶜ	0.638	0.491	0.469	0.230
%95 CI (Lower–Upper)	0.035–0.315	0.038–0.403	0.028–0.336	0.024–0.214
Ratings of response explanations
Basic periodontology
0-point	7 (8.5)	4 (4.9)	7 (8.5)	18 (7.3)	0.001*ᵇ
1-point	12 (14.6)	0 (0)	17 (20.7)	29 (11.8)
2-point	63 (76.8)	78 (95.1)	58 (70.7)	199 (80.9)
Clinical periodontology
0-point	6 (13.0)	3 (6.5)	7 (15.2)	16 (11.6)	0.009*a
1-point	9 (19.6)	1 (2.2)	10 (21.7)	20 (14.5)
2-point	31 (67.4)	42 (91.3)	29 (63.0)	102 (73.9)
Oral implantology
0-point	1 (16.7)	1 (16.7)	1 (16.7)	3 (16.7)	0.625ᵇ
1-point	2 (33.3)	0 (0)	0 (0)	2 (11.1)
2-point	3 (50.0)	5 (83.3)	5 (83.3)	13 (72.2)
Phi (Φ)Value	0.149	0.158	0.151	0.101
pᶜ	0.562	0.499	0.545	0.395
%95 CI (Lower–Upper)	0.094–0.402	0.053–0.430	0.115–0.365	0.054–0.237

CI Confidence Interval, LLMs Large Language Models

^*p < 0.05

^a:chi-square test

^b:Fisher Exact test

^c:Phi Cramer V test

To further explore factors associated with the accuracy and total score of LLM responses, binary logistic regression and multiple regression analyses were performed, including LLM type, topic, and question type. None of the variables significantly affected the probability of a correct response (p > 0.05, Table 4). For the total score, the LLM type did not affect the total score obtained (β = −1.000, p = 0.788); the effect of question type and topic was significant (p = 0.008, β = −65.500, 95% CI, −106.932 to −24.069 and p = 0.001, β = −66.500, 95% CI, −75.216 to −57.784, respectively, Table 5).

Table 4.

The effect of variables on correct responses

	β	Sig	Exp	95% C.I
				Lower	Upper
LLMs
Gemini 2.5 Flash-ChatGPT-5	0.613	0.185	847	0.745	4.507
Copilot-ChatGPT-5	−0.078	0.843	0.925	0.426	2.008
Gemini 2.5 Flash -Copilot	0.691	0.131	1.997	0.814	4.899
Topic
Clinical periodontology-Basic periodontology	−0.368	0.419	0.692	0.284	1.690
Oral implantology-Basic periodontology	−0.936	0.170	0.392	0.103	1.492
Clinical periodontology-Oral implantology	0.569	0.443	1.766	0.413	7.558
Question Type
Information-based-Standard MCQs	0.033	0.954	1.033	0.335	3.184
Case-based- Standard MCQs	−0.298	0.577	0.742	0.260	2.118
Information-based-Case-based	0.331	0.667	1.393	0.309	6.286

Binary Logistic Regression was used

MCQ multiple-choice questions, CIConfidence Interval, Sig significance (p value), Exp exponential (Exp(B) = Odds Ratio (OR)), LLMs Large Language Models

^*p < 0.05

Table 5.

The effect of research variables on total scores

	β	Sig	95% C.I
			Lower	Upper
LLMs	−1.000	0.788	−9.716	7.716
Question Type	−65.500	0.008*	−106.932	−24.069
Topic	−66.500	0.001*	−75.216	−57.784

Multiple Regression was used

LLMs Large language models, CI Confidence Interval, Sig significance (p value)

^*p < 0.05

Discussion

The present study compared the performance of ChatGPT-5, Gemini 2.5 Flash, and Copilot, in answering periodontology questions and examined whether their performances were affected by question type and topic. All LLMs showed comparable overall accuracy performance, whereas they demonstrated differences in quality of response explanations. Explanation quality rather than accuracy, was mainly affected by question and topic types.

The results of this study showed that all models had high overall accuracy rates, with no significant differences. ChatGPT-5 achieved an accuracy rate of 89.6%, Gemini 2.5 Flash 94%, and Copilot 88.8%. These findings are consistent with previous studies emphasizing the role of LLMs in medical and dental education [7, 8, 25, 34]. In studies conducted across various specialties, ChatGPT, Gemini, and Copilot have reported accuracy rates between 61 and 91% [7, 35]. In periodontology studies, the accuracy rate was between 57 and 79% for ChatGPT (3.5 and 4) and 72.86% for Gemini [24, 25]. However, it should be noted that DUS questions are publicly available, and it remains unclear whether the observed high accuracy rates reflect genuine reasoning or memorization.

Gemini 2.5 Flash tended to provide more appropriate and consistent explanations compared to ChatGPT-5 and Copilot. As demonstrated in Tables 2 and 3, accuracy rates were similar among models, whereas differences in explanation quality were more pronounced. This highlights that observed LLM performance differences were driven by explanation quality rather than accuracy. Gemini's performance advantage can be attributed to differences in its response generation strategies [36]. Although LLMs have potential as educational tools when the information obtained is evaluated together, the reasoning skills required for advanced periodontology training are not sufficient. However, since the explanation quality scoring was performed by a single expert, the results should be interpreted with caution.

Regarding question types, all LLMs demonstrated similar accuracy on standard, information-based, and case-based MCQs. The ratings of response explanations varied significantly among question types. Gemini 2.5 Flash provided more consistent and adequate explanations for standard and information-based questions. In this context, determining the correct answer in model performance is independent of the question type. However, the adequacy and consistency of the answer explanation can vary with the complexity of the question. Unlike previous studies [28, 37], in this study, each LLM answered case-based questions with similar accuracy to standard and information-based questions. Moreover, there was no significant inter-group difference among LLMs regarding the response accuracy of case-based MCQs. The reason why the study results differ from previous studies may be that older versions of the models were used in prior studies or differences in the complexity of the clinical conditions examined in the studies. Case-based questions can pose challenges to LLMs because they require contextual understanding and clinical judgment. The results obtained in this study using more advanced, updated versions of the models may indicate that the response adequacy and consistency of these LLMs.

Performance by topic followed a similar trend with question type. Accuracy rates among basic, clinical, and implantology domains were similar, but the ratings of response explanations varied for basic and clinical periodontology, favoring Gemini 2.5 Flash (p = 0.001 and 0.009, respectively). Gemini 2.5 Flash achieved 95.1% of 2-point responses for basic and 91.3% for clinical periodontology. Copilot, in contrast, had the lowest proportion of high-quality responses (70.7% and 63% 2-point, respectively). All models produced similar ratings of response explanations on implantology questions, which may be attributable to their limited representation. Consistent with the results of the study, Sabri et al. [25] reported that ChatGPT-4 and Gemini models were reliable in implant dentistry and periodontal disease. They also reported that LLMs are potential tools for both patient and resident education.

Together with the present findings, topic and question type were associated with LLMs' reasoning quality, but not their accuracy rates, as confirmed by regression analyses. This indicates that question and topic characteristics primarily shape response depth. In this context, their variable reasoning and their tendency to produce superficially correct but conceptually weak explanations suggest that LLMs may rely on recall rather than reasoning, warranting cautious integration for education.

The main strength of this study is that it evaluates different question types and topics in different LLMs, and that the evaluation is not only based on right or wrong answers but also on the explanation quality. Additionally, the fact that periodontology questions involve both basic science and clinical judgment provided an ideal experimental field to evaluate these models. When interpreting the results, it should be kept in mind that the study has some important limitations. First, publicly available questions were used, and it is unknown whether these questions were used in training the models. This creates a potential risk of data leakage, and it remains unclear whether the models' performance reflects their actual reasoning abilities or their ability to memorize previously seen material. Moreover, the publication of these questions may contribute to future data leakage risks. Newer models may incorporate this material into training data, potentially affecting the interpretability of subsequent LLM assessments. Future studies should employ methods designed to detect or reduce such leakage. Second, LLMs have the potential to exhibit bias due to the differing volume and quality of available Turkish and English training data. Additionally, linguistic complexity in Turkish may have influenced the coherence and depth of LLM-generated explanations. This may limit the generalizability of findings and raise ethical concerns in dental education and practice. The imbalanced distribution of question types and topics -particularly the small number of implantology questions- could be considered another limitation. Although the overall study power was 84%, the results related to implantology should be interpreted with caution. Additionally, explanation scoring and the classification of questions were performed by a single experienced periodontist, which may introduce subjective bias. Using multiple calibrated raters in future studies will address this limitation. Last, although these models support multiple-language options, using only Turkish and periodontology questions may limit the generalizability of the findings to other languages and other health sciences education. While studies conducted in different languages have reported that these models perform well in languages other than English [33, 38, 39], further research is needed to establish this generalizability.

Conclusion

Within the limitations of this study, all three evaluated LLMs demonstrated high factual accuracy on periodontology questions. LLMs exhibited differences quality of response explanation; however, these findings should be interpreted cautiously due to single-expert rating. Question types and topics appeared to affect the consistency and quality of LLMs' explanations rather than their ability to answer questions accurately. The results demonstrate both the potential and current limits of using LLM in dental education. In order for these LLMs to be safely integrated into dental education, clinical reasoning skills require enhancement and domain-specific knowledge must be further deepened. LLMs should be used with caution and under expert supervision in dental education. Future research should focus on the creation of benchmark datasets, multilingual evaluation, evaluation in open-ended questions, or external validation.

Abbreviations

LLM

Large language model

MCQ

Multiple-choice question

ChatGPT

Chat Generative Pre-trained Transformer

DUS

Dental Specialization Examination

ÖSYM

Student Selection and Placement Centre

Author’s contributions

Hanife Merva PARLAK designed the study, wrote the main manuscript text, prepared tables and review the manuscript.

Funding

This study did not receive any financial support.

Data availability

The datasets generated and/or analyzed during the current study are not publicly available but are available from the corresponding author upon reasonable request.

Declarations

Ethics approval and consent to participate

Not applicable. This study involved no human participants or animal subjects, and all data were obtained from a publicly accessible database; therefore, ethics approval was not required.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.