Artificial intelligence chatbots versus dentists: a comparative knowledge assessment on traumatic dental injury management

Abstract

Background

The aim of this study is to conduct a comparative analysis of the guideline-based knowledge levels of dentists and artificial intelligence(AI)-powered chatbots (ChatGPT-4o and Gemini) regarding the emergency management of traumatic dental injuries (TDIs).

Methods

A 20-item multiple-choice questionnaire, developed based on the trauma guidelines recommended by the American Association of Endodontists (AAE), was administered to both AI-powered chatbots (ChatGPT-4o and Gemini) and practicing dentists. The guideline-based knowledge level and consistency of the AI responses were evaluated based on the collected data. Furthermore, the knowledge levels of the AI systems were statistically compared to those of the dentists, using a significance level of p < 0.05 and a 95% confidence interval.

Results

Upon analysis of the questionnaire responses, ChatGPT-4o provided significantly more correct answers than both dentists and Gemini in 17 out of the 20 questions (p < 0.05). There was a statistically significant difference in guideline-based knowledge levels among the groups (p = 0.001; p < 0.05). The rate of high-level knowledge demonstrated by ChatGPT-4o (100%) was statistically significantly greater than that of both dentists (12.6%) and Gemini (3.4%) (p < 0.05). ChatGPT-4o exhibited similar internal consistency score to Gemini in terms of reliability.

Conclusions

ChatGPT-4o and Gemini may be considered potential sources of information in the context of TDIs. Although ChatGPT-4o provided significantly more accurate and consistent responses compared to Gemini, it is not entirely sufficient. Further research involving AI models specifically developed for the field of endodontics is necessary to address current limitations.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12903-026-07728-6.

Background

Artificial intelligence (AI), a branch of computer science, focuses on creating systems that can mimic human cognitive functions, including learning, reasoning, and solving problems [1]. Chatbots offer various services, such as delivering information on specific topics, responding to user inquiries, and creating an artificial environment for human-like interaction [2, 3]. AI is driving significant advancements across various domains such as the economy, industry, legal systems, and social media, and is also emerging as a powerful tool in the healthcare sector by enhancing diagnostic accuracy, improving treatment planning, and supporting medical research [4]. This technology is designed to alleviate the workload of healthcare professionals and improve overall efficiency, with notable applications in the field of dentistry [5]. AI has made substantial contributions to dental literature by improving diagnostic precision, streamlining treatment processes, and enabling novel clinical strategies through the analysis of large-scale data [6].

Large Language Models (LLMs) are advanced AI systems trained on extensive datasets to emulate human language processing capabilities. LLMs are trained using large-scale data through reinforcement and supervised learning, employing deep learning algorithms and advanced modeling based on human language patterns to generate coherent and contextually appropriate word sequences [7]. Several LLMs have emerged in recent years. In late 2022, OpenAI launched the Generative Pre-trained Transformer (ChatGPT), a chatbot powered by LLMs [8]. It reached a record-breaking 100 million new users within the first three months of its release [9]. Initially based on the GPT-3.5 language model, the first version was offered for free. As of February 2023, access to the upgraded GPT-4 version has been restricted to subscribers of the paid ChatGPT Plus service. In March 2024, OpenAI introduced ChatGPT-4.0, followed shortly by ChatGPT-4o in May 2024. These updated versions reportedly provide significant advancements in operational speed, response latency, and multilingual performance, particularly in non-English text and code generation [10]. Additionally, ChatGPT-4o features multimodal capabilities, enabling the processing and production of text, audio, and image data. Unlike its predecessor, whose training knowledge was limited to data available up to 2021, ChatGPT-4o incorporates information up to 2023, thus offering a broader and more current knowledge base [10]. Currently, several applications with comparable capabilities exist, trained on linguistic data and equipped with natural language processing (NLP), translation, and text generation functionalities. One prominent example is Google Bard, an AI-driven chatbot developed as an alternative to ChatGPT. Like ChatGPT, Google Bard leverages artificial intelligence to respond to user inquiries, facilitate interactive conversations, and produce creative content [11]. Google Bard, initially introduced on March 21, 2023, was rebranded as Gemini in February 2024. Gemini is an AI-driven information retrieval system and advanced chatbot, utilizing a ‘native multimodal’ model to effectively process and adapt to diverse data types, including text, audio, and video. It is freely accessible, allowing users to ask unlimited questions and engage in continuous dialogue. Additionally, Gemini provides free dialog-based AI services powered by a series of deep learning algorithms designed to generate responses to surveys and inquiries [12, 13]. Similar to their impact in other domains, LLMs are AI technologies that significantly contribute to advancements in the healthcare sector. The performance of LLMs in national healthcare exams suggests their capability to serve as educational support tools [14–16].

AI technologies have diverse applications in dentistry, including assisting in diagnosis, forecasting treatment outcomes, enhancing workflow efficiency, supporting academic research, and advancing educational practices [17]. Numerous studies have explored the integration of AI into various dental specialties, including periodontology, oral and maxillofacial surgery, prosthodontics and pediatric dentistry [18–21]. Similarly, AI has been the focus of numerous studies in the field of endodontics within a relatively short period. Karobari et al. [17] investigated the performance of AI in various aspects of clinical endodontic practice, including the detection of root fractures, identification of periapical pathologies, determination of working length, assessment of root morphology, and diagnostic processes. In a study conducted by Qutieshat et al. [22], the responses of dental students and ChatGPT were compared in the diagnosis of pulpal and apical diseases.

TDIs affecting permanent teeth represent a major public health issue worldwide, with a prevalence estimated between 18% and 33%, ranking as the fifth most prevalent human disease or condition [23]. In children and adolescents, dental trauma mainly arises from traffic accidents, falls, assaults, and sports injuries [24]. In cases of trauma, appropriate treatment can be achieved through timely intervention and the correct clinical approach.

Although there are studies investigating the reliability and validity of AI chatbots as sources of information in various fields of dentistry [10, 17, 18], no research to date has compared the guideline-based knowledge level of AI chatbots with that of dentists specifically in the context of TDIs. Therefore, the aim of the present study is to compare the guideline-based knowledge level of AI chatbots (ChatGPT-4o and Gemini) regarding TDIs with that of dentists, and to evaluate the accuracy and consistency of the chatbots themselves. The null hypothesis of the study is that there is no significant difference between the knowledge levels of AI chatbots and dentists.

Materials and methods

Participation of dentists in this survey study was entirely anonymous and voluntary. Informed consent to participate was obtained from all of the participants in the study. The study fully adhered to the Declaration of Helsinki. Power analysis was performed using G*Power (Version 3.1.9.7) to estimate the required sample size. A minimum of 85 samples per group was found to be sufficient based on a power analysis with an alpha level of 0.05, a statistical power of 0.95, and a medium effect size (Cohen’s d: 0.34).

A total of 20 multiple-choice questions (Table 1) were developed using Google Forms based on the TDI guidelines provided by the American Association of Endodontists (AAE) [25]. The AAE guidelines were selected as the reference standard because they provide a comprehensive, evidence-based, and internationally recognized framework for the diagnosis and management of dental trauma. The questions consist of case diagnosis and management, clinical findings, treatment protocols, radiographic findings, and follow-up procedures. The development of the survey questions was carried out by two endodontists with expertise in the field. For questionnaire development, the relevant recommendations were translated into Turkish by two bilingual endodontists. The two translations were compared side by side, and discrepancies in wording were resolved by discussion and consensus. Additionally, the questions were carefully selected from topics commonly encountered in real-life cases of TDIs and were meticulously prepared. This questionnaire was designed as a criterion-referenced performance assessment based on AAE Trauma Guidelines [25]. Each scenario has an objective, guideline-defined correct answer; therefore, psychometric validation procedures such as construct validity, internal consistency, or test–retest reliability were not applied. Content validity was ensured through expert review. As a result, the survey retained the characteristics of an assessment tool designed to evaluate the guideline-based knowledge levels, rather than resembling a test-style examination. The questionnaire was directed to dentists working in Türkiye via both an e-mail link and social media platforms. Dentists who had completed their undergraduate dental education in Türkiye were included in the study using a convenience sampling method. Dentists were included only as a reference comparison group to contextualize AI performance. This approach also minimized unnecessary personal data collection and maintained participant anonymity. A total of 87 dentists participated in and completed the survey. Meanwhile, one of the researchers presented the same survey questions to ChatGPT-4o and Gemini 87 times on different days and times. For Gemini, internet browsing and real-time data retrieval functions were disabled. All responses were generated using the standard non-browsing mode to ensure repeatability and to prevent external information sources from influencing the output. Responses from Gemini were obtained using Gemini Advanced (powered by Gemini 1.5 Pro), which was the publicly available version during the data collection period (April 2025). Each time, a new conversation was created on different days (between April 10, 2025, and April 25, 2025) (Fig. 1). To ensure consistency, a standard prompt was presented to the chatbot prior to administering the survey: “Can you answer the following survey questions related to traumatic dental injuries from the perspective of a dentist?” Each scenario was submitted to the AI systems in an independent chat session to avoid contextual influence from prior interactions. Since the evaluated chatbots maintain conversational memory only within a single active session, initiating a new session for every scenario ensured that no previous dialogue history could affect the responses. This procedure was followed consistently for both models to ensure reliability and standardization. The same browser and standardized prompt format were used throughout data collection. The responses were compiled into an Excel spreadsheet (Microsoft, Redmond, WA, USA) by the same researcher; however, no subjective analysis was performed at this stage.

Table 1.

Survey questions with answers

Questions	Answer Options
1- Which of the following is the most commonly observed type of traumatic dental injuries (TDIs) in permanent teeth?	□ A. Luxation ■ B. Crown fracture □ C. Avulsion □ D. Root fracture □ E. Enamel crack
2- Which of the following permanent tooth groups is most frequently affected by TDIs?	■ A. Maxillary central incisors □ B. Mandibular central incisors □ C. Maxillary canines □ D. Mandibular canines □ E. Maxillary first premolars
3- Which of the following statements regarding the evaluation of pulp status in TDIs is incorrect?	□ A. Sensibility testing assesses neural activity, not vascular supply. □ B. Temporary loss of pulp sensibility is common finding during pulp healing after truma. ■ C. Lack of response to pulp sensibility testing in teeth with TDI is a definitive diagnostic criterion for pulp necrosis. □ D. Pulp sensibility testing should be performed at the initial visit and at every follow-up visits for teeth with TDI. □ E.In teeth affected by TDI, sensibility testing may be unreliable due to transient neural response loss or incomplete differentiation of A-delta nerve fibers in immature teeth.
4-Which of the following types of TDIs does not require follow-up?	□ A. Enamel fracture ■ B. Enamel infraction □ C. Enamel-dentin fracture □ D. Crown fracture □ E. Concussion
5- Which of the following is not a possible clinical finding in an uncomplicated crown fracture (enamel-dentin fracture)?	□ A. Normal mobility □ B. Generally positive response to pulp sensibility testing. □ C. Negative sensitivity to percussion and palpation □ D. Sensitivity to percussion and palpation may indicate possible luxation injury or root fracture ■ E. Widening of periodontal ligament
6- Which of the following is not one of the applicable treatment protocol for a complicated crown fracture?	□ A. In cases with immature roots and open apices, partial pulpotomy or pulp capping is recommended to promote continued root development. □ B. In teeth with completed root development, conservative pulp therapies (e.g., partial pulpotomy) are recommended. □ C. In a mature tooth with completed root development, root canal treatment is the treatment of choice if post placement is required for crown retention. ■ D. If the fractured tooth fragment is available, it can be reattached directly to the tooth following rehydration. □ E. In the absence of the fractured fragment, exposed dentin should be covered with composite resin.
7- Which of the following outlines the appropriate clinical and radiographic follow-up intervals for an uncomplicated crown-root fracture?	□ A. 1 week, 1 month, 3 months, 6 months, 1 year □ B. 10 days, 3 weeks, 3 months, 6 months, 1 year □ C. 2 weeks, 1 month, 2 months, 5 months, 1 year ■ D. 1 week, 6–8 weeks, 3 months, 6 months, 1 year □ E. 1 week, 6–8 weeks, 4 months, 1 year, 2 years
8- Which of the following is not among possible future treatment options for a complicated crown-root fracture?	□ A.Completion of root canal treatment and coronal restoration □ B. Surgical intrusion □ C. Intentional replantation with or without tooth rotation ■ D. Tooth extraction □ E. Leaving the root embedded
9- Which of the following statements regarding the radiographic evaluation and imaging techniques of root fractures is incorrect?	□ A. A root fracture can occur at any level of the root. □ B. Parallel periapical radiography is among the recommended imaging techniques. □ C. Occlusal radiography is one of the suggested imaging methods. □ D. In some cases, Cone Beam Computed Tomography imaging may be beneficial. ■ E. Radiographs taken at different horizontal angulations generally do not provide additional information.
10- Which of the following statements regarding alveolar fractures is correct?	□ A. Clinical and radiographic follow-up should continue for at least 2 years after the first year. ■ B. Initiating root canal treatment during emergency management is contraindicated. □ C. Occlusal radiography is not among the recommended imaging techniques. □ D. Teeth within the fractured segment show exaggerated responses to pulp sensibility tests. □ E. Teeth should be rigidly splinted for 4 weeks.
11- Which of the following options consists entirely of examples of periodontal tissue injuries?	□ A. Root fracture, concussion, subluxation, extrusion ■ B. Extrusion, intrusion, lateral luxation, subluxation □ C. Alveolar fracture, avulsion, concussion, root fracture □ D. Complicated crown-root fracture, concussion, intrusion □ E. Avulsion, extrusion, alveolar fracture, subluxation
12- Which of the following is not among the initial emergency interventions that can be performed at the scene in cases of avulsion injuries?	□ A. The avulsed tooth should be located and immediately repositioned in the socket. □ B. If the tooth is dirty, it should be rinsed with milk, saline, or the patient’s saliva before reinsertion. □ C. After placing the tooth in the socket, biting on gauze can help stabilize it. □ D. The root surface of the tooth should be avoided when handling it. ■ E. If replantation is not possible, the avulsed tooth can primarily be stored in water until dental care is available.
13- Which of the following is not a factor that influences treatment decisions in tooth avulsion cases?	□ A. Stage of root development □ B. Condition of the periodontal ligament cells □ C. Extraoral time (duration the tooth remains out of the socket) □ D. Storage conditions of the tooth before reaching the dentist ■ E. Pulpal status of the avulsed tooth
14- Which of the following types of TDIs has the poorest prognosis?	□ A. Lateral luxation □ B. Concussion □ C. Extrusion ■ D. Intrusion □ E. Root fracture
15- Which of the following statements regarding the use of systemic antibiotics in avulsion injuries is incorrect?	□ A. Systemic antibiotics are recommended after replantation to prevent infection-related complications. □ B. The appropriate antibiotic dosage should be calculated based on the patient’s age and weight. □ C. The first-line antibiotics are amoxicillin or penicillin. ■ D. Tetracycline is an antibiotic that can be used in all age groups. □ E. Alternative antibiotics should be considered for patients with penicillin allergy.
16- Which of the following statements about splinting of replanted teeth is/are correct?  I. The splint should be placed on the labial surface.  II. Wire or composite splints can be used for splinting.  III. In certain cases, the splinting period may be extended.  IV. If the avulsed tooth cannot remain in the correct position, the splint should be removed.  V. The bonding agent used to attach the splint should be placed as close as possible to the gingival margin.	■ A. I, II, III □ B. I, II, IV □ C. I, III, V □ D. II, III □ E. II, III, IV
17- Which of the following is not one of the conditions to be assessed during clinical examination in cases of TDIs?	□ A. Tissue lacerations □ B. Edema □ C. Asymmetry ■ D. Presence of periapical lesion □ E. Ecchymosis
18- Which of the following can be evaluated through joint examination in cases of TDIs?  I. Joint pain  II. Muscle pain  III. Gingival sulcus bleeding  IV. Deviation	□ A. I, II, III □ B. II, III, IV □ C. I, III, IV ■ D. I, II, IV □ E. I and IV
19- In which of the following TDIs is splinting not required?	□ A. Alveolar fracture □ B. Root fracture ■ C. Concussion □ D. Extrusion □ E. Lateral luxation
20- Which of the following statements about intrusion injuries is true?	□ A. In immature teeth (with incomplete root development), if reeruption does not occur within 2 weeks regardless of the degree of intrusion, surgical extrusion should be performed. □ B. In mature teeth (with complete root development) intruded less than 3 mm, immediate surgical extrusion should be performed. ■ C. In mature teeth intruded more than 7 mm, surgical extrusion should be performed. □ D. In mature teeth intruded between 3–7 mm, only orthodontic extrusion should be attempted. □ E. In immature teeth, root canal treatment should be initiated immediately.

‘’■’’ sign presents right answer according to the AAE Trauma Guideline [25]

Fig. 1.

Screenshots of ChatGPT-4o and Gemini’s responses to questions

The survey questions were scored using a Likert-type scale, with 1 point for each correct answer and 0 points for each incorrect answer, resulting in a maximum total score of 20 points. According to this, the guideline-based knowledge levels were categorized as ‘’0 point uninformed’’, ‘’1–7 points low knowledge level’’, ‘’8–14 points moderate knowledge level’’ and ‘’15–20 points high knowledge level’’. Data analysis was conducted separately for each of the 3 groups and the responses were evaluated in terms of the knowledge level, knowledge mean score and also the reliability of the chatbots.

Statistical analysis

Statistical analyses were performed using IBM SPSS 22nd version (IBM Corp., Armonk, NY, USA). The normality of distribution for the parameters was assessed using the Kolmogorov-Smirnov test, which indicated that the data did not follow a normal distribution. In the evaluation of study data, descriptive statistical methods (minimum, maximum, mean, standard deviation, median, and frequency) were used. For comparisons of quantitative data between groups, the Kruskal-Wallis test was applied, and Dunn’s test was used to identify the group(s) responsible for significant differences. For the comparison of qualitative data, the Chi-square test and the Fisher-Freeman-Halton exact Chi-square test were employed. Internal consistency was assessed using Cronbach’s alpha coefficient. Significance was evaluated at p < 0.05 level.

Results

An analysis of the survey responses revealed that for 17 out of 20 questions, the correct answers provided by ChatGPT-4o were significantly more accurate than those given by dentists and Gemini (p < 0.05). An analysis of the responses to Question (Q) 6 showed that dentists provided significantly more correct answers than both ChatGPT-4o and Gemini (p < 0.05). The accuracy rate of Gemini’s responses was significantly higher than that of ChatGPT-4o (p < 0.05). Notably, ChatGPT-4o failed to provide a correct response to this question on all occasions. Dentists outperformed Gemini in 14 of the 20 questions, with a significantly higher rate of correct responses (p < 0.05). No statistically significant difference was found between dentists and Gemini in terms of correct response rates for 5 of the questions (Q2, Q3, Q7, Q17, Q19) (p > 0.05). Gemini outperformed dentists in only one question, with a significantly higher accuracy rate (p < 0.05). Regarding Q19, no statistically significant differences were observed in the accuracy rates among the three groups (p > 0.05).

The distribution of the correct response rates to the questions across the groups is presented in Fig. 2.

Fig. 2.

Distribution of groups in terms of correct response rates

There was a statistically significant difference in the guideline-based knowledge levels among the groups (p = 0.001; p < 0.05). Gemini exhibited the highest rate of low knowledge level, dentists demonstrated the highest rate of moderate knowledge level, and ChatGPT-4o showed the highest rate of high knowledge level. Accordingly, ChatGPT-4o demonstrated a significantly higher rate of high knowledge level classification (100%) than dentists (12.6%) and Gemini (3.4%) (p < 0.05). The proportion of dentists with a high knowledge level (12.6%) was also significantly higher than that of Gemini (3.4%) (p < 0.05) (Table 2).

Table 2.

Evaluation of groups in terms of guideline-based knowledge levels

Knowledge	Dentists (n = 87)	ChatGPT-4o(n = 87)	Gemini (n = 87)	p
Level	n (%)	n (%)	n (%)
Low	7 (%8)	0 (%0)	54 (%62.1)
Moderate	69 (%79.3)	0 (%0)	30 (%34.5)	0.001*
High	11 (%12.6)	87 (%100)	3 (%3.4)

Chi-square test *p < 0.05

Analysis of the question scores revealed a statistically significant difference in the knowledge level scores between the groups (p = 0.001; p < 0.05). ChatGPT-4o’s average total score was significantly greater compared to both dentists (p = 0.001) and Gemini (p = 0.001) (p < 0.05). The mean score of dentists was also significantly higher than that of Gemini (p = 0.001; p < 0.05) (Table 3).

Table 3.

Evaluation of groups in terms of knowledge level mean scores

	Knowledge Score				p
	Minimum	Maximum	Ort ± SD	Median
Dentists	1	17	11.46 ± 3.06	12
ChatGPT-4o	15	18	17.14 ± 0.61	17	0.001*
Gemini	2	16	7.29 ± 2.58	7

Kruskal Wallis test, *p < 0.05, SD: Standart Deviation

Table 4 presents the results regarding the internal consistency of the responses given by the AI chatbots, and they showed similar consistency rates.

Table 4.

Evaluation of reliability across chatbot groups

	Cronbach’s alpha	95%CI	p
ChatGPT-4o	0.607	0.477–0.718	0.001*
Gemini	0.580	0.440–0.698	0.001*

*p < 0.05

Discussion

AI-based chatbots are software applications capable of interacting with humans through natural language and typically provide various services via text-based data [26–28]. Chatbots have started to be used in almost every field, including medicine and dentistry [3, 29].

Clinical decision-making and treatment implementation in dentistry can be difficult in certain cases, particularly in complex scenarios that demand advanced experience and knowledge, such as TDIs. In the present study, the responses of dentists, ChatGPT-4o, and Gemini to 20 multiple-choice questions related to TDIs were compared. According to the findings, ChatGPT-4o predominantly demonstrated a “high level of knowledge,” while dentists mostly exhibited a “moderate level of knowledge,” and Gemini showed a “low level of knowledge” to a large extent. Based on these findings, the null hypothesis of the study was rejected.

Although several studies have been conducted in the field of dentistry involving AI models such as ChatGPT-4o and Gemini, these studies have generally focused on broader diagnostic or educational applications. However, no research to date has specifically addressed TDIs—a clinical area that requires careful patient evaluation and a detailed treatment protocol. Moreover, there is a lack of studies comparing the performance of these AI models with that of dentists in managing TDI-related scenarios. In this context, the present study represents the first comprehensive analysis directly comparing the performance of chatbots and dentists in the domain of TDIs.

Johnson et al. [30] compared the validity and reliability of various AI chatbots as public information sources on dental trauma. According to the findings, ChatGPT-3.5 provided more detailed responses compared to the other chatbots evaluated (Bing, Claude). In terms of the reliability of the responses, no significant difference was found between ChatGPT and Gemini. However, in the present study, ChatGPT-4o demonstrated higher reliability than Gemini. In the study conducted by Özden et al. [31], dichotomous (yes/no) questions related to dental trauma were posed to ChatGPT-3.5 and Gemini over a period of 10 days. The findings showed that ChatGPT-3.5 correctly answered 51% of the questions, compared to 64% for Gemini. Conversely, in the current study, ChatGPT-4o demonstrated a significantly higher overall accuracy rate of 85.68%, while Gemini’s accuracy was notably lower at 36.43%. The inconsistency between the studies may be explained by the use of ChatGPT-4o, a more advanced version of the chatbot, in the current study. Furthermore, variations in the formats and types of questions employed may have played a role in these differing outcomes.

Kaplan et al. [32] evaluated ChatGPT-3.5 and Gemini using various types of questions related to avulsion injuries that are commonly asked by patients and may serve educational purposes.

The findings indicated that Gemini outperformed ChatGPT-3.5 in most scoring metrics and achieved a higher overall accuracy rate. In the present study, ChatGPT-4o provided correct answers at a significantly higher rate than Gemini. This difference may be attributed to the use of a different version of ChatGPT in the current study. Additionally, while Kaplan et al. focused specifically on avulsion injuries, the present study encompassed the full spectrum of TDIs. The variation in the topics examined may have contributed to these differing results. Accordingly, it can be suggested that ChatGPT demonstrates a higher capacity for accurate self-updating compared to Gemini.

Suarez et al. [33] reported an accuracy rate of 57.33% in a study utilizing ChatGPT-4o to diagnose pulp and periapical conditions. Additionally, a different investigation assessing the diagnostic performance of different AI chatbots in identifying malignant oral lesions [34] found that Gemini exhibited lower accuracy than ChatGPT-4o. In a study comparing different chatbots in restorative dentistry [35], ChatGPT-4 was shown to have “good” reliability, whereas Gemini demonstrated “moderate” reliability. The same study also reported that ChatGPT-4o exhibited the highest treatment knowledge and quality scores. These findings are consistent with the current study. Gemini’s real-time internet access may be unreliable, which could lead to the retrieval of unverified or conflicting information and increase the likelihood of inaccurate responses.

Rapid advancements in the field of AI may cause older versions to become outdated over time. This dynamic evolution necessitates the use of the most current AI models to ensure optimal performance and reliability. Moura et al. [36] also demonstrated that ChatGPT-4o has significantly higher diagnostic accuracy compared to ChatGPT-3.5. Furthermore, while the ChatGPT-3.5 version is limited to data up to 2021, ChatGPT-4o is more up-to-date, with access to more recent information. To the best of our knowledge, no study in the literature has evaluated the performance of ChatGPT-4o and Gemini in the context of TDIs. Therefore, ChatGPT-4o and Gemini were selected as the chatbot models in the present study.

The findings of the present study indicate that ChatGPT-4o achieved the highest mean guideline-based knowledge score, whereas Gemini obtained the lowest. Dentists achieved a moderate score in the knowledge assessment. ChatGPT-4o achieved a 100% accuracy rate on 15 out of 20 questions by consistently providing correct answers across all attempts. In contrast, neither dentists nor Gemini reached a 100% repeated accuracy rate on any question. Interestingly, for the remaining 5 questions, ChatGPT-4o failed to provide a correct answer in any instance for 2 of them—specifically, the questions related to complicated crown fractures and splint applications. ChatGPT-4o demonstrated inconsistent performance on Q6, which may be attributed to the probabilistic nature of LLM outputs. Because these models do not apply rule-based clinical logic but instead generate responses based on learned linguistic patterns, scenarios requiring multiple sequential steps, precise trauma management algorithms, or fine-grained guideline knowledge may lead to variability in responses. This is a known limitation of current LLM architecture and reflects model behavior rather than study design issues. These differing outcomes among AI chatbots may be attributed to variations in their underlying algorithms.

The tendency of LLMs to generate inaccurate information and fabricated references has been observed in various studies [33, 37, 38] across different fields, and this phenomenon has been referred to as “hallucination” [39]. To mitigate hallucinations and other known limitations of AI systems, several practical strategies can be implemented. These include the use of structured and standardized prompts, explicit anchoring of queries to established clinical guidelines, and cross-validation of AI-generated recommendations by clinicians. Importantly, AI tools should be used as decision-support systems rather than autonomous decision-makers, with final clinical judgment remaining the responsibility of the clinician. Such measures may enhance the safe and effective integration of AI into clinical workflows. In the present study, in order to eliminate hallucinations and ensure access to accurate and up-to-date information, the questions and answers were prepared based on the guidelines recommended by AAE, and the scoring and evaluation were conducted accordingly.

In the present study, the performance of AI applications were compared with that of dentists who had completed dental education. This comparison allowed for an objective assessment of the theoretical knowledge level of AI, rather than its capabilities in a clinical context.

Furthermore, the most up-to-date version of ChatGPT available at the time - ChatGPT-4o – was utilized in the present study. This methodological choice contributes to the study’s originality and scientific rigor, while offering a contemporary perspective that underscores its strenghts.

Nevertheless, it is important to acknowledge the limitations of the present study. Although the prepared questions touched on all topics related to TDIs, they may not include all possible clinical TDIs scenarios. Additionally, the exclusive use of multiple-choice questions constitutes another limitation, as this question format may not adequately reflect the complex and multifactorial nature of clinical decision-making in practice. The AI chatbots evaluated in the present study were not specificially trained in the field of endodontics; rather, they were designed to respond to general questions from the public. Therefore, the results may be subject to certain biases. Developing specialized versions of these language models for domains such as healthcare and law could enhance their reliability. In addition, in the present study, dentists were considered as a general group within the framework of the current healthcare system in Türkiye. Participants were not stratified according to trauma-specific experience levels or areas of specialization, which may be considered one of the limitations of the study.

Conclusion

The language models Gemini and ChatGPT-4o show promising potential in answering multiple-choice questions based on the AAE’s dental trauma guidelines. Among them, ChatGPT-4o demonstrated a significantly higher level of guideline-based knowledge compared to Gemini. This study highlights the potential of language models to be used as informational tools in the context of dental trauma. However, their misuse may lead to incorrect answers and inappropriate healthcare decisions. Therefore, dentists should critically evaluate the suggestions provided by such models and remain cautious, as inaccurate outputs could negatively impact patient care. Future studies should consider incorporating a broader range of question types, supported by open-ended items, to enhance the applicability of the results to clinical scenarios and to include larger and more diverse samples. In addition, further research stratifying dentists according to their levels of clinical experience and area of specialization may help clarify the role of artificial intelligence in this context.

Authors’ contributions

Study conceptualisation: HS, GPS. Study protocol and design: HS, GPS. Data collection: HS, GPS, SSK. Data analysis and interpretation: HS, GPS, TK. Writing original manuscript: HS, GPS, TK. Reviewing and editing of manuscript: HS, GPS, SSK. Reading and approval of the manuscript: HS, GPS, TK, SSK.

Funding

This study was funded by the authors. There was no financial support from any funding organization.

Data availability

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Declarations

Ethics approval and consent to participate

Ethic committee approval was obtained from the Ethics Committee of Biruni University (2024-BİAEK/09–47). Ethical approval document is attached as a supporting file. The study fully adhered to the Declaration of Helsinki. Informed consent to participate was obtained from all of the participants in the study.

Consent for publication

Not Applicable.

Competing interests

The authors declare no competing interests.