Skip to main content
NCBI home page
Wiley Open Access Collection logoLink to Wiley Open Access Collection
. 2026 Feb 18;42(5):646–658. doi: 10.1111/edt.70063

Comparison of Artificial Intelligence Chatbots and Dental Students on Context of Dental Trauma

Şeyma Atlı 1,✉, Gamze Ebrar Nalbant 1, Tansu Merve Beşparmak 1, Ali Türkyılmaz 1, Ali Erdemir 1
PMCID: PMC13569050  PMID: 41708525

ABSTRACT

Aim

Daily use of chatbots has become widespread in dental education just as in every other field. Dental students consult them when performing tasks and studying for exams. This study aims to test comprehension abilities and knowledge of 5 different chatbots (Gemini 1.5/1.5Pro, ChatGPT 4.0/4.0o, and Deepseek) and compare them with students' at the topic of dental trauma.

Methodology

The questionnaire is designed as 25 multiple‐choice questions according to International Association of Dental Traumatology guideline and it's divided to three groups as ‘indication and description’, ‘intervention and treatment’ and ‘splint and follow up times’. The prepared questions in the survey were asked to five different chatbots (Gemini 1.5/1.5Pro, ChatGPT 4.0/4.0o and Deepseek) and 4th/5th (n = 134) year dental students. The accuracy percentages were calculated by the answers of the students and chatbots. While examining Skewness and Kurtosis test for data distribution, ANOVA test was used to assess the differences between the participant.

Results

After achievement test adjustments the highest mark of total questionnaire is received by the 5th graders (69.96%) and it's followed by the ChatGPT 4.0 and Gemini 1.5 Pro (68.75%) with the same score. Other chatbots couldn't be successful to reach out the level of knowledge of 5th or 4th graders (65.85%).

Conclusions

The speed and efficiency of artificial intelligence bots in providing information are undeniable. However, as the results of this study indicate that all the information they provide is not accurate, it is important to examine the sources of these responses and verify the information with multiple references.

Keywords: artificial intelligence, dental education, dental trauma

1. Introduction

In dentistry, researchers have been evaluating the highlight versatility of artificial intelligence (AI) and its potential to improve patient care and clinical outcomes. For instance, AI helps detect tooth decay earlier and more accurately, creates custom prosthetic designs faster, and estimates bone loss and periodontitis stages to guide treatment. It also assists in classifying radiolucent lesions for better diagnosis and prognosis [1, 2, 3, 4, 5, 6].

Dental injuries constitute a critical public health challenge, particularly among young individuals. These injuries demand immediate, expert intervention and require dentists to manage post‐treatment complications with precision [7]. Treatment timing proves as crucial as the intervention itself since most negative outcomes stem from inadequate or delayed emergency responses [8]. Consequently, parents, teachers, and sports coaches must possess fundamental knowledge of dental trauma emergency management alongside dental professionals.

People increasingly turn to online resources following dental trauma, especially when professional care remains inaccessible [9]. Public health policies in dental traumatology play a vital role in reducing injury incidence and impact through preventive initiatives, standardized treatment protocols, and heightened public awareness of immediate care importance. AI‐based chatbots offer distinct advantages over traditional search engines by delivering information in conversational formats that simplify complex dental topics [10]. This makes Large Language Models accessible resources when conventional dental services prove difficult to reach, providing users with instant, comprehensible guidance during critical moments.

At the same time, there are decision‐making and guidance applications also used in dental trauma as ToothSOS, and one of the aims of the ToothSOS mobile application, developed by IADT itself, is to meet this demand for situations that dentists, parents, and teachers encounter and need to manage [11].

The growing role of AI in dental education has sparked significant discussions about its various implications for teaching methodologies and student outcomes [12]. The numerous research initiatives are underway to investigate the integration of AI in educational settings, ensuring that the field of dentistry remains at the cutting edge of technological advancements exam preparation. Although limited studies have systematically evaluated these models' knowledge and understanding of clinical protocols in dental trauma [13, 14], no study evaluating the performance of chatbots and students, particularly in subjects based on strict protocols such as the IADT trauma guidelines, has been found in the literature. Hence, comparing AI models with student performance can address a critical need from both an educational and clinical perspective. It is important to evaluate the reliability of AI‐based tools as a complementary resource in education, test their compatibility with clinical protocols based on IADT guidelines, and determine the extent to which these tools can be used in student decision‐making processes. Therefore, this study aims to test comprehension abilities and knowledge of 5 different chatbots (Gemini 1.5/1.5Pro, ChatGPT 4.0/4.0o, and Deepseek) and compare them with students' at the topic of dental trauma.

2. Material Method

This study was approved by the Non‐Interventional Research Ethics Committee of Kırıkkale University. The sample size in our study was determined by looking at studies with similar methods in the literature [15, 16, 17, 18].

2.1. Questionnaire Design

The 25‐item multiple‐choice question set (Table 1) used in the study was prepared by three endodontists (Ş.A., G.E.N., T.M.B.) with at least 3 years of experience, based on the IADT Dental Trauma Guide. To ensure the content validity of the questions and to validate the content based on expert opinion, expert opinions were obtained from two experts (A.T., A.E.) experienced for 20 years in endodontics and dental trauma. The Davis technique was used to assess content validity, and the Content Validity Index (CVI) was calculated as 0.97. Since the Content Validity Ratio (CVR) of two questions was 0.66; these questions have been revised as necessary based on expert feedback. After revisions, the question set was finalized and made ready for implementation [19].

TABLE 1.

Questionnaire.

No Subgroups Question Options
1

Indication and description

Excluded

Which age group most commonly experiences traumatic dental injuries?

  1. It does not vary by age

  2. Infancy

  3. Childhood and early adulthood

  4. Late adulthood

  5. Terminal period
2 Indication and description Which traumatic dental injury is most commonly seen in primary teeth?

  1. Crown fracture

  2. Root fracture

  3. Subluxation

  4. Luxation

  5. Avulsion
3 Indication and description Which traumatic dental injury is most commonly seen in permanent teeth?

  1. Crown fracture

  2. Root fracture

  3. Subluxation

  4. Luxation

  5. Avulsion
4 Splint and follow‐up times
  1. Cooperation of the child and family
  2. Vitality of the periodontal ligament
  3. Vitality of cementum
  4. Root maturation level

In the avulsion of permanent teeth, prognosis depends on which of the following?

  1. Only II

  2. Only III

  3. II and III

  4. I, II and III

  5. I, II and IV
5

Splint and follow‐up times

Excluded

What is the follow‐up period for enamel fracture in primary dentition?

  1. 1st week

  2. 1st and 4th week

  3. 1st, 4th and 8th week

  4. 1st and 8th week

  5. No follow‐up required
6

Splint and follow‐up times

Excluded

Which traumatic dental injury in permanent teeth does NOT require splinting?

  1. Extrusion

  2. Lateral luxation

  3. Intrusion

  4. Avulsion

  5. Concussion
7 Splint and follow‐up times Which statement regarding tests after traumatic dental injuries is incorrect?

  1. Pulp sensibility tests measure pulp blood flow

  2. Early responses may be misleading

  3. Should be performed at every follow‐up

  4. Includes cold and electric tests

  5. The initial test predicts long‐term prognosis
8

Splint and follow‐up times

Excluded

What is the maximum diameter of stainless‐steel wire used for splinting?

  1. 0.2

  2. 0.3

  3. 0.4

  4. 0.5

  5. 0.6
9

Indication and description

Excluded

In which condition is the likelihood of displacement of the tooth within bone the lowest?

  1. Subluxation

  2. Lateral luxation

  3. Extrusion

  4. Intrusion

  5. Avulsion
10 Splint and follow‐up times Which statement regarding pulp canal obliteration (PCO) is incorrect?

  1. Crown fractures are common causes

  2. Frequent after root fractures

  3. More common in severe luxation injuries with open apex

  4. Lateral luxation, extrusion, intrusion often lead to PCO

  5. Low frequency after subluxation
11 Intervention and treatment Which statement regarding clinical findings and treatment of complicated crown fracture without periodontal injury is incorrect?

  1. Mobility is normal

  2. Sensitivity may require evaluation for luxation/root fracture

  3. Severe percussion/palpation sensitivity present

  4. Exposed pulp is sensitive to stimuli

  5. Conservative vital pulp therapy preferred
12 Indication and description

  1. Sensitive to touch/percussion.

  2. Bleeding from gingival sulcus.

  3. No early response due to transient pulp injury.

  4. PDL space widened apically/laterally.

  5. V. Repositioning by finger pressure under LA. Which are correct for subluxation?

  1. I and II

  2. II and III

  3. I, II and III

  4. II, III and IV

  5. I, II and IV
13 Intervention and treatment A 14‐year‐old boy presents with avulsion of tooth 21; extra‐oral dry time is 75 min. Which is incorrect?

  1. Delayed reimplantation has poor long‐term prognosis

  2. PDL is necrotic and will not regenerate

  3. Reimplantation is almost always correct even if dry > 60 min

  4. Internal resorption is most common post‐reimplantation

  5. Systemic antibiotics recommended
14

Intervention and treatment

Excluded

When is endodontic treatment indicated after reimplantation of an avulsed permanent tooth?

  1. Within 2 weeks

  2. Within 3 weeks

  3. Within 4 weeks

  4. Within 2 months

  5. Within 4 months
15

Indication and description

Excluded

Which statement regarding avulsion is incorrect?

  1. Immediate reimplantation is best

  2. If LA unavailable, reimplantation should be delayed

  3. Systemic antibiotics may be prescribed

  4. Topical antibiotics pre‐reimplantation do not help

  5. Tetanus risk must be evaluated
16 Indication and description Dental soft‐tissue injuries are most common in which period?

  1. Infancy

  2. Childhood and young adulthood

  3. Older adulthood

  4. Terminal period

  5. No period difference
17 Intervention and treatment Which does NOT affect success of vital pulp therapy in complicated crown fractures?

  1. Time to treatment

  2. MTA/CH as capping material

  3. Using the fractured fragment

  4. Extent of fracture line

  5. Restoration seal
18 Indication and description Which is incorrect regarding enamel crack?

  1. No percussion/palpation sensitivity

  2. Mobility normal

  3. Pulp test usually positive

  4. Radiograph shows thin radiopaque line

  5. Resin after etching is applied for significant cracks
19 Intervention and treatment Which is true for treatment of avulsed primary teeth?

  1. Reimplant and perform RCT

  2. Reimplant and splint 2 weeks

  3. Reimplant and splint 4 weeks

  4. Reimplant and follow

  5. Should NOT be reimplanted
20

Indication and description

Excluded

Which is incorrect for alveolar fractures in primary dentition?

  1. Involves alveolar bone and may extend to adjacent bone

  2. Segment moves with teeth

  3. Displaced segments should be repositioned

  4. Splint to adjacent teeth 4 weeks

  5. Rigid splint required
21

Intervention and treatment

Excluded

Which is incorrect regarding systemic antibiotics in avulsion?

  1. Efficacy debated

  2. May reduce inflammatory resorption

  3. Dose adjusted for age/weight

  4. Doxycycline suitable

  5. Best option < 12 years is tetracycline/doxycycline
22 Intervention and treatment An 8‐year‐old with 4 mm intrusion of tooth 21, no fractures. Correct approach?

  1. Surgical extrusion

  2. Surgical extrusion + 4‐week splint

  3. Surgical extrusion + RCT

  4. Orthodontic repositioning

  5. Allow spontaneous reeruption
23 Indication and description

I. Reposition displaced coronal fragment promptly

II. Confirm with radiograph

III. Cervical fractures can heal; do not remove stable fragment

IV. Start RCT during emergency visit

Which are correct regarding root fractures in permanent teeth?

  1. I, II, IV

  2. I, II, III

  3. I and III

  4. II and III

  5. I, II, III, IV
24 Intervention and treatment Which are ideal storage media before reimplantation?

  1. HBSS—Milk

  2. Milk—Saline

  3. Water—Saline

  4. Saliva—Milk

  5. Saliva—HBSS
25 Splint and follow‐up times Splinting duration for root fractures in cervical third?

  1. 2 weeks

  2. 4 weeks

  3. 6 weeks

  4. 2 months

  5. 4 months
Open in a new tab

Since the language of instruction for the sample group in which the study was conducted was Turkish, the questions were prepared in Turkish. Referencing studies that found no significant difference in responses across sufficiently widely used AI models in different languages [20, 21], the questions were posed to all participants, including chatbots, in the same language.

The 25 questions were evaluated in three subgroups, aiming to test inclusiveness, diversity, and different abilities simultaneously. These subgroups are ‘indication and description (1st)’, ‘intervention and treatment (2nd)’, and ‘splint and follow‐up times (3rd)’.

The questions in the first group demanded a detailed analysis and interpretation of the data, requiring respondents to examine patterns and connections to understand their interrelationships. This involved critical thinking and the ability to draw insights from the provided information. In contrast, the questions in the third group were designed to elicit specific information, asking for direct answers to well‐defined queries. The second group of questions served as a bridge between the two, incorporating cascading inquiries that merged elements of both analytical reasoning and precise factual recall. This layering allowed for a more comprehensive exploration of the topic, leading to a deeper understanding.

The quality of assessment and evaluation processes in education is critical for accurately determining students' learning levels [22]. Achievement tests are widely used to measure students' knowledge and skills in a particular subject area, and ensuring the reliability and validity of these tests directly affects the quality of the educational process [23]. In the test development process, examining the psychometric properties of test items through item analysis methods is an important step in increasing the measuring power of the test [24]. The achievement test developed in this study aims to create an effective and reliable measurement tool in the light of item discrimination indices and independent sample t‐test results [25], which analyze the differences between the item responses of the upper and lower 27% groups. Especially, as Crocker and Algina stated, questions with item discrimination levels of 0.30 and above play an important role in the creation of a valid test. In addition, for the test items to accurately reflect student achievement, the item writing principles suggested by Haladyna were taken into consideration [22, 26].

The achievement test consisting of 25 multiple‐choice questions created in this study aims to objectively measure students' knowledge and skill levels in the specified subject area. In the application, the responses of 134 participants were coded in a binary format (true = 1, false = 0). The data obtained were analyzed to evaluate the discrimination and reliability of each item. Especially in the item‐level analyses, it was aimed to increase the measurement adequacy of the test by comparing the performances of the upper and lower 27% groups.

2.2. Collecting Answers

The consent forms and questionnaires were distributed to voluntary 4th (n = 67) and 5th (n = 67) grade dental students at Kırıkkale University, both in paper format and as an online link. Additionally, five chatbots, two of which are paid versions, were asked in sequence, and to prevent answers from being dependent on each other, no user input was required, a new conversation tab was opened for each new question, and no introductory or guiding sentence was used before the questions. Cookies and user history were cleared between each question. Since the questions are multiple‐choice, the temperature is kept at its lowest. All results were collected, and a database was created.

2.3. Statistical Analysis

SPSS 30.0 (Statistical Package for Social Sciences) software was used for statistical analysis of the data obtained in the study. Frequency and percentage values were calculated for categorical data. Before starting the analysis, the suitability of the numerical data for normal distribution was examined with Skewness and Kurtosis tests, Histogram and Q‐Q Plot graphs. As a result of the analyses, it was seen that the data came from a normal distribution. Achievement test development stages were applied, and item discrimination indices were calculated with the lower and upper 27% groups. The difference between the lower and upper 27% groups was analyzed by independent sample t‐test. Quantitative data are shown with mean and standard deviation. ANOVA test was used to analyze the difference between 4th and 5th graders and AI applications and achievement test, and significance level p < 0.05 was accepted.

3. Results

When the answers to the 25 questions in this achievement test are analyzed, the test results of 20 students, whose success rates were either very high or very low, were considered outliers and removed from the evaluation. The z‐score was calculated for each item, and 20 participants were excluded from the analysis due to outlier data falling outside the range of −3 to +3. The ±3 z‐score limit has been reported as a commonly recommended threshold for identifying extreme values under the assumption of normal distribution [27, 28]. As seen in Table 2, it was observed that students answered most of the questions correctly at a high rate; however, the level of success decreased in some questions. The most straightforward question was question 1, which 93.9% of the students answered correctly, while the most challenging item was question 7, which had a correct response rate of only 32.5%. In general, the correct answer rate was above 60% for most questions; it was indicating that students have a general command of the knowledge and skills measured in the test.

TABLE 2.

Descriptive statistics of achievement test.

Incorrect Correct
n % n %
1Q 7 6.1 107 93.9
2Q 42 36.8 72 63.2
3Q 20 17.5 94 82.5
4Q 51 44.7 63 55.3
5Q 32 28.1 82 71.9
6Q 21 18.4 93 81.6
7Q 77 67.5 37 32.5
8Q 28 24.6 86 75.4
9Q 30 26.3 84 73.7
10Q 54 47.4 60 52.6
11Q 40 35.1 74 64.9
12Q 47 41.2 67 58.8
13Q 42 36.8 72 63.2
14Q 18 15.8 96 84.2
15Q 38 33.3 76 66.7
16Q 61 53.5 53 46.5
17Q 21 18.4 93 81.6
18Q 30 26.3 84 73.7
19Q 34 29.8 80 70.2
20Q 22 19.3 92 80.7
21Q 21 18.4 93 81.6
22Q 38 33.3 76 66.7
23Q 31 27.2 83 72.8
24Q 16 14.0 98 86.0
25Q 59 51.8 55 48.2
Open in a new tab

3.1. Inclusion and Exclusion of Questions

Table 3 presents the correct response rates for the upper and lower 27% groups for each item in the achievement test, along with the item discrimination index (P j ) and the recommended item treatments. The item discrimination index reflects the power of an item to distinguish between high and low achievers. It was typically calculated based on the difference between the correct answer rates of the upper and lower groups [22, 25]. Such analyses provide critical information about the reliability and measurement power of the items. In the literature, it was stated that items with a P j value above 0.30 are considered “good discriminators,” those between 0.20 and 0.29 should be revised, and those with a P j value of 0.19 and below should be removed from the test [29].

TABLE 3.

Lower and upper group item discrimination values.

Lower group % Upper group % P j Category
1Q 94.4 94.4 0 Exclude
2Q 52.8 83.3 0.306 Include
3Q 66.7 97.2 0.306 Include
4Q 22.2 77.8 0.556 Include
5Q 69.4 91.7 0.222 Exclude
6Q 77.8 86.1 0.083 Exclude
7Q 11.1 80.6 0.694 Include
8Q 75 72.2 −0.028 Exclude
9Q 72.2 100 0.278 Exclude
10Q 30.6 86.1 0.556 Include
11Q 47.2 91.7 0.444 Include
12Q 38.9 83.3 0.444 Include
13Q 50 88.9 0.389 Include
14Q 75 83.3 0.083 Exclude
15Q 61.1 88.9 0.278 Exclude
16Q 38.9 69.4 0.306 Include
17Q 61.1 91.7 0.306 Include
18Q 55.6 88.9 0.333 Include
19Q 58.3 91.7 0.333 Include
20Q 72.2 94.4 0.222 Exclude
21Q 75 91.7 0.167 Exclude
22Q 36.1 94.4 0.583 Include
23Q 44.4 97.2 0.528 Include
24Q 61.1 97.2 0.361 Include
25Q 22.2 91.7 0.694 Include
Open in a new tab

According to Table 3, many items in the test have high discrimination values, and especially questions 4, 7, 10, 11, 12, 22, 23, and 25 make a significant difference between the upper and lower groups and have high measurement power. On the other hand, some questions, such as items 1, 6, 14, and 21, were recommended to be removed from the test because they have low discrimination power and were answered correctly similarly in the upper and lower groups. As item 8 has a negative discrimination value (the lower group is more successful), the inclusion of this question in the test may be inconvenient in terms of measurement. Questions 5, 9, 15, and 20 should be removed from the achievement test since their item discrimination value is below 0.30. The distribution of the questions, categorized according to the findings obtained from Table 3, was visualized in Figure 1.

FIGURE 1.

FIGURE 1

Open in a new tab

P j values and categories.

Table 4 shows the mean scores of the upper and lower 27% groups for the achievement test items, the independent sample t‐test results (t‐value, p‐value) between these groups and whether the results are statistically significant. Testing whether the mean score difference between the two groups is statistically significant is a classic approach in item discrimination analyses. Items with a p‐value below 0.05 are significantly discriminating between students at the upper and lower achievement levels; this finding is an important indicator to show that the test consists of reliable and functional items in accordance with the measurement purpose [23].

TABLE 4.

Independent sample t‐test results between upper and lower 27% groups.

Item Upper group, average Lower group, average t p Significance
1Q 0.944 0.944 0.000 1.000 Not a
2Q 0.833 0.528 2.902 0.005 a
3Q 0.972 0.667 3.621 0.001 a
4Q 0.778 0.222 5.590 0.000 a
5Q 0.917 0.694 2.447 0.017 a
6Q 0.861 0.778 0.912 0.365 Not a
7Q 0.806 0.111 8.129 0.000 a
8Q 0.722 0.750 −0.264 0.793 Not a
9Q 1.000 0.722 3.669 0.001 a
10Q 0.861 0.306 5.706 0.000 a
11Q 0.917 0.472 4.608 0.000 a
12Q 0.833 0.389 4.285 0.000 a
13Q 0.889 0.500 3.896 0.000 a
14Q 0.833 0.750 0.863 0.391 Not a
15Q 0.889 0.611 2.833 0.006 a
16Q 0.694 0.389 2.695 0.009 a
17Q 0.917 0.611 3.226 0.002 a
18Q 0.889 0.556 3.354 0.001 a
19Q 0.917 0.583 3.489 0.001 a
20Q 0.944 0.722 2.613 0.012 a
21Q 0.917 0.750 1.919 0.060 Not a
22Q 0.944 0.361 6.485 0.000 a
23Q 0.972 0.444 5.966 0.000 a
24Q 0.972 0.611 4.153 0.000 a
25Q 0.917 0.222 8.229 0.000 a
Open in a new tab
a

Significant.

The results show that most items in the test created a significant performance difference between the upper and lower groups. In particular, items 2, 3, 4, 7, 10, 11, 12, 13, 16, 17, 18, 19, 20, 22, 23, 24 and 25 exhibited strong discrimination as their p‐values were well below 0.05. On the other hand, no significant difference was found in items 1, 6, 8, 14 and 21; since the discrimination of these questions is not sufficient, it may be suggested that they could be removed from the test. In general, the table reveals that most of the test contains items with high discrimination levels and supports the validity of the measurement tool. As a result, questions 1, 5, 6, 8, 9, 14, 15, 20 and 21, which gave negative results as a result of both methods, were removed from the achievement test.

Statistical information about the 16‐item achievement test is given in Table 5. There are 6 items in the ‘indication and description’ group (2, 3, 12, 16, 18, 23), 6 items in the ‘intervention and treatment’ group (11, 13, 17, 19, 22, 24) and 4 items in the ‘splint and follow‐up’ group (4, 7, 10, 25). The mean score of all students was 10.81. The KR‐20 of the achievement test was calculated and found to be 0.702. This value shows that the achievement test was reliable. Skewness and Kurtosis values between −1.5 and +1.5 indicated that the data were normally distributed [27]. When the table was examined, it was seen that the results of the achievement test meet the normality assumption.

TABLE 5.

Statistics on achievement test.

Average Standard deviation Median Minimum Maximum Skewness Kurtosis
Total 10.81 3.06 11.00 3.00 16.00 −0.369 −0.577
Indication and description 4.11 1.32 4.00 0.00 6.00 −0.430 −0.225
Intervention and treatment 4.52 1.28 5.00 1.00 6.00 −0.753 −0.173
Splint and follow‐up times 2.18 1.30 2.00 0.00 4.00 −0.141 −1.096
Open in a new tab

3.2. Comparison of Performance

According to the results in Table 6, 5th grade students had the highest average total score (11.19 ± 2.86), while free artificial intelligence systems showed the lowest performance (8.67 ± 2.08); however, these differences are not statistically significant (p = 0.366). Looking at the subgroups, paid AI systems had the highest average (2.50 ± 0.71) in the “Splint and Follow‐up” area, while 4th grade students showed the lowest performance (1.94 ± 1.40). In the “Intervention and Treatment” category, 5th year students scored the highest (4.70 ± 1.28), while free AI systems scored the lowest (3.67 ± 1.53). In the “Indication And Description” area, 4th grade students achieved the highest average (4.22 ± 1.42), while free artificial intelligence systems again showed the lowest performance (2.67 ± 0.58). Differences between groups were not statistically significant in all subcategories (p > 0.05).

TABLE 6.

Analysis of the difference between class levels and artificial intelligence based on achievement test results.

Mean value Standard deviation F p
Total
4th grade (n = 67) 10.54 3.29 1.065 0.366
5th grade (n = 67) 11.19 2.86
Chatbots (free) (n = 3) 8.67 2.08
Chatbots (paid) (n = 2) 10.00 1.41
Indication and description
4th grade (n = 67) 4.22 1.42 1.866 0.138
5th grade (n = 67) 4.09 1.20
Chatbots (free) (n = 3) 2.67 0.58
Chatbots (paid) (n = 2) 3.00 1.41
Intervention and treatment
4th grade (n = 67) 4.37 1.28 1.189 0.316
5th grade (n = 67) 4.70 1.28
Chatbots (free) (n = 3) 3.67 1.53
Chatbots (paid) (n = 2) 4.50 0.71
Splint and follow‐up times
4th grade (n = 67) 1.94 1.40 1.478 0.223
5th grade (n = 67) 2.40 1.19
Chatbots (free) (n = 3) 2.33 1.15
Chatbots (paid) (n = 2) 2.50 0.71
Open in a new tab

The results were evaluated out of 100 points, with each question receiving equal points. For students, the average score was calculated. Figure 2 presents the percentages of correct answers on total questionnaire. The 5th graders scored the highest, closely followed by the 4.0 free version of ChatGPT and the 1.5 Pro paid version of Gemini. Although the 4th grade ranked lower, it still outperformed other chatbots.

FIGURE 2.

FIGURE 2

Open in a new tab

Total answers.

The distribution of scores by subgroups is given in the Figure 3. In the ‘splint and follow‐up times’ group, all chatbots except Deepseek outperformed the students' scores. However, no chatbot surpassed the students in the ‘intervention and treatment’ group.

FIGURE 3.

FIGURE 3

Open in a new tab

Distribution of scores by subgroups.

4. Discussion

Dental trauma falls under emergency treatments and is essential knowledge for every undergraduate student, even those who have not pursued specialty training [30]. Although it is not commonly encountered in daily practice, this topic risks being overshadowed and reduced to mere theoretical knowledge. Today, the widespread trust in and use of AI means that both dental professionals and patients are likely to turn to chatbots as quick sources of information when faced with a trauma case. The IADT guide, which is based on the most accurate and up‐to‐date information, was chosen in this study because it was quite detailed and descriptive on this subject and accepted worldwide [31, 32, 33, 34].

On the other hand, it is worth mentioning that the reason students prefer to work with chatbots rather than existing trauma decision support tools such as Tooth SOS may be that this mobile application is less well known in Turkey, or that they tend to use a single general application that they can consult on a variety of topics rather than an application specifically designed for trauma [35].

A comprehensive multiple‐choice question test was developed to rigorously evaluate the application of AI in dental education. This approach was selected to clearly differentiate between correct and incorrect responses, thereby eliminating any potential for ambiguity. Although open‐ended questions could provide deeper insights, they often involve subjective interpretation, leading to variability in evaluator assessments. Additionally, it was chosen not to include Yes/No questions, as they offer an equal probability of correct answers, which could compromise the validity of the results if either the AI or students possess incomplete knowledge. This deliberate methodology could enable significant advances in dental education by making the assessment both more reliable and accurate.

Another factor to consider when evaluating students' results is the differences in their educational levels. Dental trauma requires a multifaceted assessment and a blend of theoretical and practical knowledge. At our faculty, theoretical training in dental traumatology is provided in the 3rd grade, in accordance with the “national core curriculum for undergraduate dental education.” All 4th and 5th grade students evaluated in this study received the same theoretical education in their 3rd year and passed the relevant exams. The main difference between the 4th and 5th grade students evaluated is that 4th grade students observe trauma cases, while 5th grade students perform treatments under supervision. The internship periods also differ; 5th grade students complete internships twice as long as 4th grade students.

In this study, it was used both the free versions, which are available from the outset, and the paid versions, which are said to be more advanced, were used to add depth to the comparison. Although it was predicted that the paid versions would yield better results than the free ones, the outcomes did not completely support this assumption. Also, ScholarGPT, a specialized version of ChatGPT focused on academic studies, was intentionally excluded from the study's planning process because two other versions were already included, which is a clear limitation of the study. However, there are studies in the field of dentistry that show significantly better performance [36, 37].

The architectural distinctions among artificial intelligence systems constitute a critical analytical dimension that demands rigorous examination within this comparative framework. The technical specifications and underlying computational paradigms of these systems directly determine their response capabilities and performance characteristics. ChatGPT's Reinforcement Learning from Human Feedback (RLHF) based model architecture unequivocally positions it as the superior platform for intensive dialogical engagement and conversational depth. Google Gemini's multimodal architecture, operating within an integrated framework, establishes its definitive advantage for complex contextual tasks requiring the synthesis and interpretation of diverse data modalities, including audio and visual information. Deepseek's Mixture of Experts (MoE) approach, which strategically activates specialized expert parameters, demonstrates clear optimization for structured analytical problems, particularly in domains such as logical reasoning and software development [38].

These informations demand a comprehensive evaluation of ChatGPT 4.0 and Gemini 1.5 Pro, which clearly dominate the assessment. ChatGPT's architectural features render both versions highly suitable for this investigation. Moreover, its dialogue‐optimized infrastructure has decisively positioned the free version, ChatGPT 4.0, at the forefront, driven by its substantially larger and more active user base. Conversely, Gemini's infrastructure has been deliberately engineered for multifaceted evaluation tasks. The paid version, Gemini 1.5 Pro, delivers demonstrably more sophisticated analytical capabilities, which directly accounts for its leadership position in the overall assessment.

Deepseek's architecture, while strategically optimized for specialized logical operations, failed to achieve competitive rankings because of its inferior natural language processing capabilities. This fundamental limitation definitively restricts its operational efficacy to narrow technical domains, specifically software engineering and mathematical computation, where linguistic sophistication remains secondary to algorithmic precision.

The main reason, divided into subgroups of the questions in this study, was to evaluate the effect of questions with different characteristics on the percentage of participants, and this was confirmed. As expected, the success level of the bots increased in the ‘splint and follow‐up times’ group, which requires more memorization and precise handling of limited information. The observation that AI systems demonstrate greater consistency in delivering precise, source‐specific responses may be interpreted as a substantive indicator of their proficiency in information retrieval and pattern recognition. Consequently, these technological instruments prove particularly reliable when addressing inquiries pertaining to subject matter that is readily accessible within publicly available repositories and databases. This enhanced reliability stems from their capacity to systematically scan extensive datasets, identify relevant information with considerable accuracy, and extract pertinent details without the variability inherent in human‐mediated research processes. Furthermore, the algorithmic nature of these systems ensures reproducibility of results, thereby establishing them as dependable resources for evidence‐based inquiry within domains characterized by well‐documented, openly accessible information sources.

In contrast, the students were more successful in the ‘indication and description’ group, which involves analyzing data and understanding connections. Within this particular category of inquiry, wherein questions exhibit substantially greater complexity and necessitate sophisticated interpretation alongside comprehensive synthesis of disparate information sources, one may discern compelling evidence suggesting that human cognitive capabilities maintain superior consistency and continue to surpass artificial intelligence systems with respect to multidimensional analytical reasoning. The human capacity for nuanced understanding, contextual awareness, and integrative thinking across multiple domains represents a distinctive advantage that remains largely unmatched by contemporary algorithmic approaches. This cognitive superiority manifests particularly in scenarios demanding creative problem‐solving, ethical consideration, emotional intelligence, and the ability to navigate ambiguous or contradictory information while maintaining coherent analytical frameworks. Consequently, these observations underscore the enduring primacy of human intellectual faculties in addressing complex, multifaceted challenges that transcend the pattern‐recognition capabilities inherent in current artificial intelligence architectures. This result also provides essential insight into the area and form in which AI bots can be utilized.

A recent study in the literature evaluating the reliability of AI in dental trauma indicates that these tools have significant potential for improvement [14]. Importantly, to the best of our knowledge, this is the first study that directly compares AI bots with students on this specific topic, particularly regarding students as a source of knowledge. One of the strengths of this study was the statistical analyses performed to compare the questions and ensure they were sufficiently discriminative. As a result, any questions that might influence the inferences were excluded from the test.

AI demonstrates systematic limitations in processing nuanced dental terminology and clinical distinctions. Chatbots struggle to differentiate between quantitative expressions such as ‘frequent’ and ‘often’ as evidenced by their errors in question 10. Therefore LLM's may exhibit difficulty recognizing subtle linguistic cues that signal diagnostic certainty, often misclassifying conditionally accurate statements as incorrect due to their lack of absolute certainty, leading to inappropriate clinical recommendations.

Question 12, which all chatbots answered incorrectly, reveals significant confusion when distinguishing between closely related pathological conditions. In this question, they confused subluxation and luxation injuries and selected option D. Similarly, chatbots conflated internal resorption and cervical resorption, leading to mistakes in question 13. Once again, in question 18, it misinterprets radiographic terminology by treating radio‐opaque and radiolucent as simple opposites rather than descriptive characteristics.

In the answers given to question 14, particularly evident regarding root fractures within the cervical third, systems provide incorrect splinting duration recommendations. This could be because of failing to account for fracture line location. Another noticeable mistake has been made in question 19, exemplified by inappropriate recommendations such as reimplanting avulsed primary teeth—a contraindicated procedure that reflects inadequate attention to age‐specific treatment protocols.

Despite these limitations, in such scenarios, the models consistently recognize the medical nature of dental trauma and reliably incorporate referrals to dental professionals, suggesting some capacity for appropriate scope‐of‐practice awareness within the context of these presentations.

In our study, students were seen as more advanced compared to chatbots. These findings align closely with the results of Li et al. who compared the performance of students and chatbots using multiple‐choice and open‐ended questions on periodontal surgery and reported that students had higher accuracy [16]. Ullah et al. created queries related to dental anatomy and reported a difference in favor of students between the scores they received and those from ChatGPT. Therefore, they expressed the opinion that it should not be used as a sole learning resource [39]. Again, in a recent study, Ozturk et al. provided students with an 8‐week training program on bone lesions and then compared their answers to the questionnaire with those of chatbots, reporting that student scores were significantly high [40]. These results are consistent with the results of this study.

In contrast, two studies based on AAE terminology and guidelines reported that AI models demonstrated superior endodontic diagnostic capabilities [13, 15]. A definitive conclusion regarding its effectiveness may not be appropriate at this time. Further research is essential. Additionally, it is important to consider that the specific topics addressed may influence the outcomes.

The limitations of this study include a small sample size and the fact that the participants were exclusively from Kırıkkale University. This restriction significantly limits the generalizability of findings to a broader population. To achieve a more comprehensive understanding of this issue, it could suggest expanding the sample size and including students from a diverse range of educational institutions across different regions. Additionally, further research is essential to explore the role of AI bots in this context. This future research should involve actively practicing dentists, both those with specialized training and those without, to provide a more nuanced view of how these technologies can be applied effectively in dental practice. Such studies could yield valuable insights that enhance understanding of the integration of AI in healthcare education and practice.

5. Conclusion

Considering that AI bots are learning and evolving models, it is likely that they will become a more reliable source in the coming years as the number of references on specific topics increases and they encounter more complex questions, thereby advancing themselves. Although their adequacy has not yet been fully proven, they show promise for the future. Taking all this into account, it can be considered that their use, when verified by various sources, increases its potential to be an essential tool for disseminating and accessing accurate information.

Author Contributions

Şeyma Atlı: conceptualization, methodology, investigation, visualization, writing – original draft. Gamze Ebrar Nalbant: conceptualization, methodology, investigation, visualization, writing – original draft. Tansu Merve Beşparmak: conceptualization, methodology, investigation, visualization, writing – original draft. Ali Türkyılmaz: conceptualization, formal analysis, writing – review and editing, validation, resources. Ali Erdemir: conceptualization, formal analysis, writing – review and editing, validation, supervision.

Funding

The authors have nothing to report.

Conflicts of Interest

The authors declare no conflicts of interest.

Data Availability Statement

The data that support the findings of this study are available from the corresponding author upon reasonable request.

References

  • 1. Ariji Y., Yanashita Y., Kutsuna S., et al., “Automatic Detection and Classification of Radiolucent Lesions in the Mandible on Panoramic Radiographs Using a Deep Learning Object Detection Technique,” Oral Surgery, Oral Medicine, Oral Pathology, Oral Radiology 128, no. 4 (2019): 424–430. [DOI] [PubMed] [Google Scholar]
  • 2. Ekert T., Krois J., Meinhold L., et al., “Deep Learning for the Radiographic Detection of Apical Lesions,” Journal of Endodontics 45, no. 7 (2019): 917–922. [DOI] [PubMed] [Google Scholar]
  • 3. Mertens S., Krois J., Cantu A. G., Arsiwala L. T., and Schwendicke F., “Artificial Intelligence for Caries Detection: Randomized Trial,” Journal of Dentistry 115 (2021): 103849. [DOI] [PubMed] [Google Scholar]
  • 4. Takahashi T., Nozaki K., Gonda T., and Ikebe K., “A System for Designing Removable Partial Dentures Using Artificial Intelligence. Part 1. Classification of Partially Edentulous Arches Using a Convolutional Neural Network,” Journal of Prosthodontic Research 65, no. 1 (2021): 115–118. [DOI] [PubMed] [Google Scholar]
  • 5. Chang H. J., Lee S. J., Yong T. H., et al., “Deep Learning Hybrid Method to Automatically Diagnose Periodontal Bone Loss and Stage Periodontitis,” Scientific Reports 10, no. 1 (2020): 7531. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6. Tsoromokos N., Parinussa S., Claessen F., Moin D. A., and Loos B. G., “Estimation of Alveolar Bone Loss in Periodontitis Using Machine Learning,” International Dental Journal 72, no. 5 (2022): 621–627. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7. Petti S., Glendor U., and Andersson L., “World Traumatic Dental Injury Prevalence and Incidence, a Meta‐Analysis—One Billion Living People Have Had Traumatic Dental Injuries,” Dental Traumatology 34, no. 2 (2018): 71–86. [DOI] [PubMed] [Google Scholar]
  • 8. Needleman H. L., Stucenski K., Forbes P. W., Chen Q., and Stack A. M., “Massachusetts Emergency Departments' Resources and Physicians' Knowledge of Management of Traumatic Dental Injuries,” Dental Traumatology 29, no. 4 (2013): 272–279. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9. Erwin J., Horrell J., Wheat H., et al., “Access to Dental Care for Children and Young People in Care and Care Leavers: A Global Scoping Review,” Dental Journal 12, no. 2 (2024): 37. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10. Pandey S. and Sharma S., “A Comparative Study of Retrieval‐Based and Generative‐Based Chatbots Using Deep Learning and Machine Learning,” Healthcare Analytics 3 (2023): 100198. [Google Scholar]
  • 11. Khehra A., Cohenca N., Cehreli Z. C., and Levin L., “The International Association of Dental Traumatology ToothSOS Mobile App: A 2‐Year Report,” Dental Traumatology 37, no. 1 (2021): 145–150. [DOI] [PubMed] [Google Scholar]
  • 12. Bisdas S., Topriceanu C. C., Zakrzewska Z., et al., “Artificial Intelligence in Medicine: A Multinational Multi‐Center Survey on the Medical and Dental Students' Perception,” Frontiers in Public Health 9 (2021): 795284, https://www.frontiersin.org/journals/public‐health/articles/10.3389/fpubh.2021.795284/full. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13. Durmazpinar P. M. and Ekmekci E., “Comparing Diagnostic Skills in Endodontic Cases: Dental Students Versus ChatGPT‐4o,” BMC Oral Health 25, no. 1 (2025): 457. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14. Ozden I., Gokyar M., Ozden M. E., and Sazak Ovecoglu H., “Assessment of Artificial Intelligence Applications in Responding to Dental Trauma,” Dental Traumatology 40, no. 6 (2024): 722–729. [DOI] [PubMed] [Google Scholar]
  • 15. Qutieshat A., Rusheidi A. A., Ghammari S. A., Alarabi A., Salem A., and Zelihic M., “Comparative Analysis of Diagnostic Accuracy in Endodontic Assessments: Dental Students vs. Artificial Intelligence,” Diagnosis 11, no. 3 (2024): 259–265. [DOI] [PubMed] [Google Scholar]
  • 16. Li C., Zhang J., Abdul‐Masih J., Zhang S., and Yang J., “Performance of ChatGPT and Dental Students on Concepts of Periodontal Surgery,” European Journal of Dental Education 29, no. 1 (2025): 36–43. [DOI] [PubMed] [Google Scholar]
  • 17. Shoufan A., “Can Students Without Prior Knowledge Use ChatGPT to Answer Test Questions? An Empirical Study,” ACM Transactions on Computing Education 23, no. 4 (2023): 45. [Google Scholar]
  • 18. Taesotikul S., Singhan W., and Taesotikul T., “ChatGPT vs Pharmacy Students in the Pharmacotherapy Time‐Limit Test: A Comparative Study in Thailand,” Currents in Pharmacy Teaching & Learning 16, no. 6 (2024): 404–410. [DOI] [PubMed] [Google Scholar]
  • 19. Davis L. L., “Instrument Review: Getting the Most From a Panel of Experts,” Applied Nursing Research 5, no. 4 (1992): 194–197. [Google Scholar]
  • 20. Yao Z., Duan L., Xu S., Chi L., and Sheng D., “Performance of Large Language Models in the Non‐English Context: Qualitative Study of Models Trained on Different Languages in Chinese Medical Examinations,” JMIR Medical Informatics 13 (2025): e69485. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21. de Mendonça Moura J. D., Fontana C. E., da Reis Silva Lima V. H., Souza Alves I., de André Melo Santos P., and Almeida Rodrigues P., “Comparative Accuracy of Artificial Intelligence Chatbots in Pulpal and Periradicular Diagnosis: A Cross‐Sectional Study,” Computers in Biology and Medicine 183 (2024): 109332. [DOI] [PubMed] [Google Scholar]
  • 22. Crocker L. and Algina J., Introduction to Classical and Modern Test Theory (Holt, Rinehart and Winston, 1986), https://eric.ed.gov/?id=ed312281. [Google Scholar]
  • 23. Nitko A. J., Educational Assessment of Students, 2nd ed. (Prentice‐Hall Order Processing Center, 1996). [Google Scholar]
  • 24. Bright I., “ITEMS. Instructional Topics in Educational Measurement Comparison of Classical Test Theory and Item Response Theory and Their Applications to Test Development,” https://www.academia.edu/27340976/ITEMS_Instructional_Topics_in_Educational_Measurement_Comparison_of_Classical_Test_Theory_and_Item_Response_Theory_and_Their_Applications_to_Test_Development.
  • 25. Ebel R. L. and Frisbie D. A., Essentials of Educational Measurement, 5th ed. (Prentice Hall, 1991). [Google Scholar]
  • 26. Haladyna T. M., Developing and Validating Multiple‐Choice Test Items (Routledge, 2004). [Google Scholar]
  • 27. Tabachnick B. G. F., “Using Multivariate Statistics,” in Using Multivariate Statistics, 7th ed. (Pearson, 2019). [Google Scholar]
  • 28. Field A., Discovering Statistics Using IBM SPSS Statistics (SAGE Publications, 2024), 1114. [Google Scholar]
  • 29. Çokluk Ö., Şekercioğlu G., and Büyüköztürk Ş., Sosyal Bilimler için çok Değişkenli Istatistik: SPSS ve LISREL Uygulamaları (Pegem Akademi, 2016). [Google Scholar]
  • 30. Berlin‐Broner Y. and Levin L., “Enhancing, Targeting, and Improving Dental Trauma Education: Engaging Generations Y and Z,” Dental Traumatology 41, no. S1 (2025): 90–96. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31. Levin L., Day P. F., Hicks L., et al., “International Association of Dental Traumatology Guidelines for the Management of Traumatic Dental Injuries: General Introduction,” Dental Traumatology 36, no. 4 (2020): 309–313. [DOI] [PubMed] [Google Scholar]
  • 32. Bourguignon C., Cohenca N., Lauridsen E., et al., “International Association of Dental Traumatology Guidelines for the Management of Traumatic Dental Injuries: 1. Fractures and Luxations,” Dental Traumatology 36, no. 4 (2020): 314–330. [DOI] [PubMed] [Google Scholar]
  • 33. Fouad A. F., Abbott P. V., Tsilingaridis G., et al., “International Association of Dental Traumatology Guidelines for the Management of Traumatic Dental Injuries: 2. Avulsion of Permanent Teeth,” Dental Traumatology 36, no. 4 (2020): 331–342. [DOI] [PubMed] [Google Scholar]
  • 34. Day P. F., Flores M. T., O'Connell A. C., et al., “International Association of Dental Traumatology Guidelines for the Management of Traumatic Dental Injuries: 3. Injuries in the Primary Dentition,” Dental Traumatology 36, no. 4 (2020): 343–359. [DOI] [PubMed] [Google Scholar]
  • 35. Sari M. B. D., Sari E., Bal C., and Aksoy M., “Evaluation of the Knowledge Level of Pediatricians on Dental Trauma and Their Awareness of the ToothSOS Mobile Application: A Cross Sectional Study,” Dental Traumatology 40, no. 2 (2024): 195–203. [DOI] [PubMed] [Google Scholar]
  • 36. Balel Y., “Scholargpt's Performance in Oral and Maxillofacial Surgery,” Journal of Stomatology, Oral and Maxillofacial Surgery 126, no. 4 (2025): 102114. [DOI] [PubMed] [Google Scholar]
  • 37. Baris S. D. and Baris K., “Assessment of Various Artificial Intelligence Applications in Responding to Technical Questions in Endodontic Surgery,” BMC Oral Health 25, no. 1 (2025): 763. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38. Rahman A., Mahir S. H., Tashrif MdTA M. T. A., et al., “Comparative Analysis Based on DeepSeek, ChatGPT, and Google Gemini: Features, Techniques, Performance, Future Prospects,” Systems and Soft Computing 7 (2025): 200396. [Google Scholar]
  • 39. Ullah R., Shaikh M. S., Shahani N., Lone M. A., Fareed M. A., and Zafar M. S., “Comparing ChatGPT and Dental Students' Performance in an Introduction to Dental Anatomy Examination: A Cross‐Sectional Study,” European Journal of Dentistry 4 (2025): 869. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40. Öztürk K., Akkoca F., and Ilhan G., “ChatGPT vs. Dental Students: Bone Lesion Knowledge Comparison,” International Dental Journal 74 (2024): S29. [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Data Availability Statement

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Articles from Dental Traumatology are provided here courtesy of Wiley

RESOURCES