Skip to main content
BMC Oral Health logoLink to BMC Oral Health
. 2026 Mar 4;26:492. doi: 10.1186/s12903-026-08026-x

Accuracy, quality, and readability analyses of responses from large language models to questions on pediatric dental sedation

Merve Hayriye Kocaoğlu 1, Akif Demirel 1,✉, İlhan Kaya 2
PMCID: PMC12994223  PMID: 41781947

Abstract

Background

Large language models (LLMs) have become increasingly integrated into healthcare communication, including dentistry. However, the extent to which they are capable of providing accurate, guideline-based information on critical topics such as pediatric dental sedation remains unclear. The aim of this study was to evaluate and compare the accuracy, content quality, and readability of responses provided by five widely used LLM-based chatbots -ChatGPT-4o, ChatGPT-3.5, Google Gemini, Microsoft Copilot, and Anthropic Claude- to clinical questions related to pediatric dental sedation.

Methods

A total of 32 clinically relevant questions covering preoperative, intraoperative, and postoperative aspects of pediatric dental sedation were presented to each chatbot. Responses were assessed independently by two blinded experts using an evidence-based grading system for accuracy, DISCERN tool for content quality. Also, readability levels were calculated using Flesch-Kincaid Grade Level formula. Data were analysed using Kruskal–Wallis tests to evaluate overall differences among chatbot groups, followed by pairwise comparisons for multiple testing. Inter-reviewer reliability for DISCERN scores was assessed using intraclass correlation coefficients. Descriptive statistics were calculated for each metric. The statistical significance level was set at 0.05.

Results

Gemini and ChatGPT-4o achieved the highest accuracy, providing the majority of their responses in full compliance with the guidelines. ChatGPT-3.5 and Claude performed moderately, while Copilot showed the lowest accuracy and highest rate of guideline deviation. In content quality, ChatGPT-4o recorded the highest mean DISCERN score (57.77), closely followed by Gemini (57.56), however no significance was detected between them (p > 0.05). Readability analysis revealed that ChatGPT-3.5 produced the most accessible content, while Claude’s responses were the most complex. Inter-rater reliability for DISCERN scoring was excellent (> 0.85) for all bots, supporting the robustness of the evaluations.

Conclusions

Despite the fact that ChatGPT-4o and Gemini exhibited superior performance in general, none of the evaluated chatbots fully aligned with clinical guidelines or consistently achieved high accuracy across all phases. These findings underscore the imperative for expert oversight when employing AI chatbots for pediatric dental sedation information. It is recommended that future research concentrate on multilingual testing, iterative dialogue-based evaluations, and domain-specific fine-tuning with a view to enhancing clinical applicability and patient safety.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12903-026-08026-x.

Keywords: Accuracy, Anesthesia, Artificial intelligence, Chatbots, Large language models, Readability, Sedation

Background

Significant applications have been brought about in several sectors, including medicine, by generative artificial intelligence (AI), which is basically a broad collection of technologies that mimic human intelligence [1–3]. Machine learning is a kind of AI that finds out statistical patterns in data to predict new data. Large Language Models (LLM) are a type of machine learning tool that can handle a wide range of Natural Language Processing (NLP) tasks, including text generation and classification, conversational question answering, and text translation from one language to another [4]. Also, a subbranch of machine learning called “deep learning” uses several layered mathematical calculations to examine and extrapolate meaning from complex data sources [2, 5]. Popular chatbots of LLMs are ChatGPT, Google’s Gemini, Microsoft’s Copilot and Anthropic’s Claude, which are designed to make conversations with the user [3, 5]. LLMs are trained on massive data sets, which are constantly changing. They use articles, textbooks, and webpages, and they rely on deep learning and neural network designs as data source [6].

Recently, AI is used in many different fields of medicine and dentistry, including diagnostics, forensics, prediction-making and treatment planning [7, 8]. In pediatric dentistry, deep learning is increasingly being used for early childhood caries and oral lesion detection, predicting future disease occurrence or progression, and age estimation [7, 9, 10]. As today’s society becomes more information-driven, patients increasingly have opportunities to learn about medical issues from sources other than healthcare providers [11, 12]. A similar trend has been observed among students and health professionals. They also use AI applications for many different reasons (learning, accelerating workflow, easy access, etc.) [1, 7, 13]. While internet searches provide easy access to large amounts of information, they can also lead to potential information pollution and malpractice due to insufficient verification of consistency between information sources.

However, although they make use of large data sets and up-to-date information, such software still lacks sufficient competence and licenses/permits for medical use [14, 15]. The reasons for this include the fact that the medical field is very specific, some of the medical literature is not open source, and these learning models target a more general population when selecting data sets. The main expectation from AI at this stage is to analyze complex clinical cases and recommend correct and reliable applications. In addition, since we still do not know exactly how these systems learn, the results and recommendations produced are not fully reliable and reproducible. In this context, it was stated that AI applications cannot fully address the recommendations in the most recent American Academy of Pediatrics (AAP) /American Academy of Pediatric Dentistry (AAPD) [16] and American Society of Anesthesiologists (ASA) [17] clinical guidelines on pediatric dental sedation in terms of scientific accuracy and comprehensiveness. In the field of dentistry, the majority of published evaluations of large language models have concentrated on general oral health information or relatively low-risk subjects such as preventive care, restorative materials, or the management of common conditions. Conversely, the domain of pediatric dental sedation is characterised by its high stakes nature and strict adherence to guidelines. The dissemination of misinformation in this context can have immediate and significant consequences for patient safety. Decisions regarding the timing of fasting intervals, the selection of patients, and the stratification of risk, in addition to the selection and dosage of sedative agents, airway management, monitoring requirements, and discharge criteria, are all subject to stringent regulation by recommendations put forth by the AAP/AAPD and ASA. Pediatric dental sedation is a field that intersects with both pediatric dentistry and anaesthesiology, necessitating the integration of knowledge from these two disciplines. Consequently, there is a necessity for a focused evaluation of the manner in which general-purpose LLM-based chatbots respond to inquiries concerning pediatric dental sedation. This evaluation would serve to complement broader studies that explore the utility of LLMs in the domain of dental information. Based on this, the main research question of the current study is to evaluate the accuracy and comprehensiveness of the responses of different AI-based chatbots to questions related to procedural sedation applied in the field of pediatric dentistry. There are many recent studies in the literature testing the accuracy of medical information interpreted by AI. However, to our knowledge, testing the accuracy, quality and readability of dental sedation questions posed to chatbots with their guideline knowledge has never been studied before. Therefore, the aim of this study is to evaluate and compare the accuracy, medical information quality, and readability levels of responses from five general-purpose large language model chatbots (ChatGPT-4o, ChatGPT-3.5, Google Gemini, Microsoft Copilot, and Anthropic Claude) to questions on pediatric dental sedation.

Methods

Study design and procedure

Ethical approval was not required as no human/animal participants were included in the study. This study involves posing a total of 32 questions (Table 1) about pediatric procedural dental sedation to different AI-based chatbots and then analyzing the accuracy-reliability, quality and readability of the responses. This study was also designed and reported in accordance with the CHART (Chatbot Assessment Reporting Tool) guideline for chatbot evaluation studies.

Table 1.

Questions

Phase Questions

Preoperative

(n = 11)

1. What is the purpose of pediatric dental sedation?

2. How can pediatric patients be prepared for dental procedures with less need for deep sedation or general anesthesia?

3. How many hours before elective dental sedation should food and fluid intake be stopped?

4. Can the infant be breastfed before dental sedation?

5. In the event of a patient not fasting adequately and having an indication for emergency dental sedation, what should be done?

6. Are there any medications that should be discontinued or taken before pediatric sedation?

7. Do enlarged tonsils pose a risk for deep dental sedation?

8. Who should assess the medical condition prior to pediatric dental sedation?

9. How can pediatric patients be identified as suitable candidates for sedation? Are there any patient types that may not be suitable for deep sedation?

10. Is there an age group for which dental sedation carries a higher risk?

11. Which patient group is at higher risk of complications for pediatric sedation?

Intraoperative

(n = 13)

12. Who is qualified to perform dental sedation? Which procedures should they be competent in?

13. What preparations and equipment are necessary to minimize dental sedation-related complications?

14. What complications may develop during the dental sedation procedure?

15. Which anesthesia methods can be chosen for dental procedures?

16. What algorithm is recommended if airway obstruction develops during dental sedation? / (how to manage airway obstruction during dental sedation? )

17. What are the different types and depths of sedation?

18. Can the child tolerate a dental procedure with less depth of sedation?

19. Can the depth of sedation be increased involuntarily? What should be done in such a case?

20. How many people should be present during deep dental sedation? What are their duties?

21. Which anesthetic agents can be used during dental sedation?

22. What is the mandatory monitoring equipment during deep dental sedation?

23. Is there anything to be considered in drug dosing during sedation for obese children?

24. What is minimal sedation? What are the requirements for this procedure?

Postoperative

(n = 8)

25. Where and how long should a child be monitored after deep sedation?

26. Which pediatric patients should be monitored for longer after deep sedation?

27. What are the requirements for discharge after pediatric sedation?

28. Are there accepted assessment scales that define discharge criteria after pediatric sedation?

29. What are the complications that may be observed after discharge following pediatric dental sedation?

30. Which drugs are more likely to cause airway obstruction in young children during transportation home after sedation?

31. How many people should be present to accompany the pediatric patient after sedation?

32. Are there any considerations after sedation in pediatric patients with obstructive sleep apnea?

For the procedural sedation applications applied in the field of pediatric dentistry, 11, 13 and 8 questions were selected for the preoperative, intraoperative and postoperative phases of dental sedation, respectively. Based on previous studies in the literature [18–21], details about the preparation of the questions were presented below.

Source of question

A search was conducted using the Google search engine (www.google.com) with the term “frequently asked questions about pediatric dental sedation”. Because many private clinics advertise sedation services in this field, the first 50 websites that appeared in the search were examined. From these, a comprehensive list of 140 candidate questions was compiled after excluding irrelevant, non-clinical, duplicate, promotional or non-English items. The remaining questions were then reviewed by a multidisciplinary expert panel consisting of a board-certified anesthesiologist (M.H.K.), a pediatric dentistry specialist (A.D.) and an oral and maxillofacial surgeon (İ.K.), each with at least 10 years of clinical experience. Structured criteria were used to decide which questions would be retained: (i) overlap with key domains of the AAP/AAPD and ASA pediatric sedation guidelines, (ii) importance for clinical decision making in the preoperative, intraoperative or postoperative phase, (iii) potential for patient harm or malpractice if misunderstood, and (iv) relevance to high-risk or medically complex pediatric patients. In this study, a question was considered to have clinical nuance when it required expert judgement beyond simple factual recall and demanded the integration of several clinical parameters (such as age, ASA status, comorbidities, fasting status, choice and dosing of sedative agents, airway risk and monitoring needs), where incorrect guidance could directly affect sedation planning or patient safety. Each panel member independently rated the questions against these criteria, and any disagreements were resolved by discussion until consensus was reached. Through this process, the list was reduced to 32 questions with the highest clinical nuance, which were then grouped into preoperative, intraoperative and postoperative categories (Table 1).

Focused approach

Rather than addressing the full scope of procedural sedation in pediatric dentistry, this study focused on key aspects of the clinical management of speciality or high-risk patient groups, as well as critical details that could result in malpractice if not performed as recommended in the guidelines. The clinical relevance of the questions is detailed below.

Clinical relevance

The questions were divided into 3 groups according to the preoperative, intraoperative and postoperative stages of procedural sedation in pediatric dentistry. The selected questions consist of topics that are critical in clinical decision making and make a difference in practice such as patient selection, patient preparation, preoperative risk management in critically ill patients, intraoperative anesthesia management, intraoperative complications, postoperative care and postoperative complications.

Chatbots included

A total of 32 questions included in the study were presented to five different artificial intelligence based chatbots [ChatGPT-4o (OpenAI), ChatGPT-3.5 (OpenAI), Google Gemini, Microsoft Copilot, and Anthropic’s Claude] on the same day (17th May 2025, Ankara, Turkey). In each case, we used the public web version identified in the interface by these model names, analogous to software version identifiers (e.g., “ChatGPT-4o”). All outputs were generated on 17 May 2025 and therefore reflect the model configurations and updates active on that date. All generative AI models and chatbots evaluated in this study (ChatGPT-4o, ChatGPT-3.5, Gemini, Microsoft Copilot and Claude) are closed-source/proprietary systems. The underlying model architectures, training data and parameter weights are not publicly available. In this study, all chatbots were used as deployed, instruction-tuned conversational systems, without any additional task-specific fine-tuning by the authors. Also, no standalone base model was trained or fine-tuned in this study; we evaluated only deployed proprietary chatbots (ChatGPT-4o, ChatGPT-3.5, Gemini, Microsoft Copilot and Claude) via their public web interfaces. Additionally, no novel base model or additional fine-tuning was developed for this study, and the authors did not modify the training data or parameters of any chatbot. All chatbots were accessed in a standard desktop web browser; no API access, local deployment or custom integration was used.

All questions are asked in English, and answers are evaluated in English. For each chatbot, after each question was presented, the history of web browser was cleared, and the next question was asked to prevent the machine learning from the previous questions and answers. A total of 160 responses given by five chatbots to 32 questions were recorded for later evaluation. After the responses had been received, they were assessed for accuracy, quality and readability as detailed below.

Accuracy evaluation

Two authors (M.H.K. and A.D.), who were unaware of each other’s assessments, evaluated the responses independently using an evidence-based approach and the ASA/AAPD guidelines. The accuracy was evaluated with response accuracy system developed by Mika et al. The quality of the responses as a medical piece of information was scored with DISCERN tool which was developed and verified for quality assessment. The grading system first proposed by Mika et al. [21] previously used to assess ChatGPT responses related to hip arthroplasty and subsequently used for similar purposes in numerous studies. Each response was rated by both authors according to the grading system of Mika et al. [21] using a Likert scale described below.

Score 1 (Excellent response not requiring clarification)

Responses did not include any information that contradicted current literature.

Score 2 (Satisfactory requiring minor clarification)

Responses requiring minimal clarification lacked additional information or detail.

Score 3 (Satisfactory requiring moderate clarification)

Responses requiring moderate clarification had outdated or irrelevant information.

Score 4 (Unsatisfactory requiring substantial clarification)

Responses had either incorrect or irrelevant information that could be misinterpreted.

Quality evaluation

The overall quality of chatbot responses was assessed using the DISCERN instrument, which is a standardized and validated tool designed to evaluate the reliability and quality of health-related written information. DISCERN consists of 16 items, each scored from 1 (lowest) to 5 (highest), covering aspects such as clarity of aims, relevance, accuracy of information sources, description of treatment options, risks, benefits, and shared decision-making considerations. The total DISCERN score ranges from 16 to 80, and final overall scores were classified into five quality categories: Excellent (63–80), good (51–62), fair (39–50), poor (27–38), and very poor (16–26). In the current study, an anesthesiologist (M.H.K.) and a pediatric dentist (A.D.) evaluated all chatbot outputs using the DISCERN criteria (Table 2) [22].

Table 2.

DISCERN score categories and quality interpretation

DISCERN Overall Scores Quality Category
63–80 Excellent
51–62 Good
39–50 Fair
27–38 Poor
16–26 Very poor

Readability evaluation

To evaluate readability, the Flesch–Kincaid Grade Level (FKGL) was used. The readability of chatbot responses was evaluated using the FKGL formula. This is a widely used metric that estimates the school grade level required to understand a given text. FKGL is calculated using the average sentence length and the average number of syllables per word, providing a numerical score that corresponds to an educational grade (for example, a score of 8 indicates that an eighth grade reading level is required). Scores of lower values are indicative of greater readability and are generally recommended for patient-facing health information, with the aim of improving accessibility and comprehension [23, 24]. In this study, FKGL scores were calculated for each chatbot response using the formula below. This methodological approach facilitated an objective comparison of the linguistic complexity of the outputs and an assessment of their suitability for a general audience seeking information on pediatric dental sedation.

graphic file with name d33e540.gif

Inter-reviewer reliability

Intraclass correlation coefficients (ICC) were used to assess inter-reviewer reliability. Values below 0.5 indicate poor reliability, values between 0.5 and 0.75 indicate moderate reliability, values between 0.75 and 0.9 indicate acceptable reliability, and any number above 0.9 indicates excellent reliability. The ICC values range from 0 to 1.

Statistical analysis

In this study, the responses provided by five different chatbots (ChatGPT-4o, ChatGPT-3.5, Gemini, Copilot, Claude) to questions related to pediatric dental sedation were evaluated by two independent evaluators according to accuracy (1–4 score range) and DISCERN (16–80 score range) criteria. The analyses were performed using SPSS 22 software (IBM Corporation, Armonk, NY, USA). The mean, median, minimum-maximum values, and standard deviations (SD) of accuracy and DISCERN scores were calculated for each chatbot. Similar descriptive statistics were also reported for FKGL. For the reviewer agreement between accuracy and DISCERN scores, Cohen’s Kappa coefficient was calculated for evaluations. In addition to the Kappa coefficient, the agreement level (%) was also specified. The Intraclass Correlation Coefficient (ICC) value was calculated between the two reviewers for the DISCERN score (Two-Way Mixed Model, Absolute Agreement). The results were reported with 95% confidence intervals. Due to the fact that the data did not meet the normal distribution assumption in the comparison of models in terms of DISCERN averages and FKGL, the Kruskal-Wallis H test was applied. Bonferroni-corrected post-hoc analyses were performed for comparisons found to be significant. The relationship between the quality of the chatbots’ responses (DISCERN average) and FKGL was evaluated using Spearman’s rho correlation analysis due to the non-parametric data structure. For the comparison of chatbot accuracy scores across different perioperative phases (preoperative, intraoperative, and postoperative), the Kruskal-Wallis H test was used due to the ordinal nature and non-normal distribution of the data, as assessed by the Shapiro-Wilk test. In cases where the Kruskal-Wallis test indicated significant differences (p < 0.05), post hoc pairwise comparisons were conducted using the Mann-Whitney U test with Bonferroni correction to adjust for multiple comparisons. The statistical significance level was set at 0.05.

Results

According to the accuracy analysis, a fair level of agreement was observed between the reviewers for ChatGPT-4o (κ = 0.404, p = 0.009). In contrast, ChatGPT-3.5 demonstrated a substantial agreement (κ = 0.653, p = 0.0001). Gemini also showed a substantial level of agreement (κ = 0.611, p = 0.0001), while Copilot exhibited a fair agreement (κ = 0.393, p = 0.002). Lastly, Claude achieved a moderate agreement between reviewers (κ = 0.481, p = 0.0001) (Table 3). In all cases, the agreements were statistically significant (Table 3). These findings suggested that ChatGPT-3.5 and Gemini demonstrated more consistent and reliable performance across reviewers. In contrast, Copilot and Claude showed comparatively lower inter-reviewer agreement, indicating potential variability in response accuracy (Table 3). These agreement results complement the accuracy findings and further support the comparative evaluation of chatbot performance in providing reliable information (Table 3; Figs. 1 and 2).

Table 3.

Accuracy ratings and inter-reviewer agreement levels (κ) for each chatbot

ChatBots Reviewer (R) Accuracy Levels R2 Kappa Test Inter-reviewer
Agreement Level (%)
Score 1 Score 2 Score 3 Score 4 Total
n % n % n % n % n % Kappa p
ChatGPT-4o R1 Score 1 14 77.8 4 22.2 0 0.0 0 0.0 18 100.0 0.404 0.009 68.75
Score 2 5 38.5 7 53.8 1 7.7 0 0.0 13 100.0
Score 3 0 0.0 0 0.0 1 100.0 0 0.0 1 100.0
Score 4 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0
Total 19 59.4 11 34.4 2 6.3 0 0.0 32 100.0
ChatGPT-3.5 R1 Score 1 8 80.0 2 20.0 0 0.0 0 0.0 10 100.0 0.653 0.0001 78.13
Score 2 1 6.3 12 75.0 3 18.8 0 0.0 16 100.0
Score 3 0 0.0 1 16.7 5 83.3 0 0.0 6 100.0
Score 4 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0
Total 9 28.1 15 46.9 8 25.0 0 0.0 32 100.0
Gemini R1 Score 1 23 92.0 2 8.0 0 0.0 0 0.0 25 100.0 0.611 0.0001 84.38
Score 2 0 0.0 4 66.7 2 33.3 0 0.0 6 100.0
Score 3 0 0.0 0 0.0 0 0.0 1 100.0 1 100.0
Score 4 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0
Total 23 71.9 6 18.8 2 6.3 1 3.1 32 100.0
Copilot R1 Score 1 1 50.0 1 50.0 0 0.0 0 0.0 2 100.0 0.393 0.002 65.63
Score 2 0 0.0 5 62.5 3 37.5 0 0.0 8 100.0
Score 3 0 0.0 3 15.8 14 73.7 2 10.5 19 100.0
Score 4 0 0.0 0 0.0 2 66.7 1 33.3 3 100.0
Total 1 3.1 9 28.1 19 59.4 3 9.4 32 100.0
Claude R1 Score 1 5 71.4 2 28.6 0 0.0 0 0.0 7 100.0 0.481 0.0001 65.63
Score 2 4 26.7 9 60.0 2 13.3 0 0.0 15 100.0
Score 3 0 0.0 3 33.3 6 66.7 0 0.0 9 100.0
Score 4 0 0.0 0 0.0 0 0.0 1 100.0 1 100.0
Total 9 28.1 14 43.8 8 25.0 1 3.1 32 100.0

Fig. 1.

Fig. 1

Accuracy score distribution of each chatbot for reviewer 1

Fig. 2.

Fig. 2

Accuracy score distribution of each chatbot for reviewer 2

Among the evaluated chatbots, ChatGPT-4o yielded the highest mean DISCERN score (57.77), whereas Copilot had the lowest (44.59) (Table 4). A Kruskal–Wallis test indicated a statistically significant overall difference (H = 70.90, p < 0.001; ε² ≈ 0.42, large effect). Post-hoc analyses showed that ChatGPT-4o scored significantly higher than Copilot (p < 0.001), Claude (p < 0.001), and ChatGPT-3.5 (p = 0.024). Gemini likewise outperformed Copilot (p < 0.001), Claude (p < 0.001), and ChatGPT-3.5 (p = 0.021). Additional contrasts revealed that ChatGPT-3.5 exceeded Copilot (p = 0.001) and Claude surpassed Copilot (p = 0.047). No significant difference emerged between ChatGPT-4o and Gemini (p > 0.05). ChatGPT-4o and Gemini delivered the highest-quality and most reliable content, ChatGPT-3.5 and Claude occupied a mid-quality tier, and Copilot lagged substantially behind (Table 4).

Table 4.

Discern scoring of each chatbot and statistical comparisons

ChatBots DISCERN Scores Kruskal-Wallis H test
n Mean Median Min Max SD Mean Rank H p Post-hoc Tests
ChatGPT-4o 32 57.77 57.00 46.00 67.00 4.29 4.16 70.90 0.0001

ChatGPT-4o and Copilot: p < 0.001

ChatGPT-3.5 and Copilot: p = 0.001

Gemini and Copilot: p < 0.001

Copilot and Claude: p = 0.047

ChatGPT-4o and Claude: p < 0.001

Gemini and Claude: p < 0.001

ChatGPT-4o and ChatGPT-3.5: p = 0.024

ChatGPT-3.5 and Gemini: p = 0.021

ChatGPT-3.5 32 52.88 53.00 41.00 63.00 5.23 2.84
Gemini 32 57.56 59.50 40.00 66.00 6.39 4.01
Copilot 32 44.59 44.00 32.00 57.00 5.43 1.46
Claude 32 50.34 51.00 35.00 64.00 7.04 2.53
Total 160 52.63 54.00 32.00 67.00 7.54 -

n Number, Min Minimum, Max Maximum, SD Standard Deviation

Among the evaluated chatbots, Claude exhibited the highest mean Flesch-Kincaid score (mean = 13.96), indicating the highest reading complexity, while ChatGPT-3.5 had the lowest (mean = 11.94). A statistically significant difference was found among the chatbots (H = 25.30, p = 0.0001) (Table 5). Post-hoc analyses revealed that Claude’s scores were significantly higher than those of Copilot (p = 0.042), ChatGPT-4o (p = 0.001) and ChatGPT-3.5 (p < 0.001). These results suggest that Claude generates content with a higher reading level compared to most of the other chatbots evaluated (Table 5). In terms of readability, ChatGPT-3.5 was rated as the easiest to read, whereas Claude’s content was considered more complex due to its higher Flesch-Kincaid score. ChatGPT-4o achieved a balance of high content quality and easy readability, making it potentially more user-friendly. Overall, these findings highlight distinct differences in both information quality and accessibility among the evaluated chatbots (Table 5).

Table 5.

Flesch-Kincaid scoring of each chatbot and statistical comparisons

ChatBots Flesch-Kincaid Scores Kruskal-Wallis H test
n Mean Median Min Max SD Mean Rank H p Post-hoc Tests
ChatGPT-4o 32 12.19 12.00 8.50 17.00 1.92 67.02 25.30 0.0001

ChatGPT-4o and Claude: p = 0.001

ChatGPT-3.5 and Claude: p < 0.001

Copilot and Claude: p = 0.042

ChatGPT-3.5 32 11.94 11.90 8.40 14.80 1.59 59.11
Gemini 32 12.82 12.40 9.40 16.80 1.51 84.38
Copilot 32 12.91 12.00 9.40 17.20 2.22 79.48
Claude 32 13.96 13.90 12.00 17.80 1.56 112.52
Total 160 12.76 12.30 8.40 17.80 1.89 -

n Number, Min Minimum, Max Maximum, SD Standard Deviation

The ICC values for all chatbots were found to be above 0.85, indicating a very high level of consistency among evaluators in their DISCERN scale ratings. All p values were < 0.001, confirming that evaluator consistency was statistically significant (Table 6).

Table 6.

Inter-rater reliability based on ICC-DISCERN assessment

Chatbots ICC Confidence Interval (95% CI) p Agreement Level
ChatGPT 4.0 0.899 [0.792–0.951] < 0.001 Excellent
ChatGPT-3.5 0.852 [0.697–0.928] < 0.001 Excellent
Gemini 0.950 [0.898–0.976] < 0.001 Almost Perfect
Copilot 0.918 [0.832–0.960] < 0.001 Excellent
Claude 0.913 [0.822–0.958] < 0.001 Excellent

Table 7 examines whether there is a significant relationship between the readability level of chatbot responses and information quality (DISCERN score). According to this, there is no statistically significant relationship between DISCERN and F-K (readability) levels in any model. Correlation coefficients are also generally between − 0.30 and + 0.20, i.e. weak. A negative (but insignificant) correlation was observed in Copilot and Claude. However, positive but insignificant relationships were found in the Gemini, ChatGPT-4o, and 3.5 models (Table 7).

Table 7.

Spearman correlation results

Chatbots Correlation Coefficient (r) p Interpretation
ChatGPT-4o 0.059 0.748 Not significant, very weak
ChatGPT-3.5 0.042 0.818 Not significant, very weak
Gemini 0.179 0.327 Not significant, weak
Copilot -0.285 0.114 Not significant, weak negative
Claude -0.235 0.195 Not significant, weak negative

In the preoperative phase, significant differences were found among chatbots (H = 15.18, p = 0.004). Copilot showed the highest mean accuracy score (2.55 ± 0.82), followed by Claude (2.09 ± 1.04), ChatGPT-3.5 (1.82 ± 0.75), ChatGPT-4o (1.64 ± 0.67), and Gemini (1.18 ± 0.40). Post hoc pairwise comparisons indicated that Gemini was significantly more accurate than Copilot (p = 0.005) (Table 8).

Table 8.

Descriptive statistics and comparative analyses of chatbot performances across different phases (pre-op, intra-op, and post-op), as evaluated by Reviewer 1

Phase Chatbot n Mean ± SD Median Min Max Mean Rank Kruskal-Wallis H Post-hoc tests (Mann-Whitney U test with Bonferroni correction)
H p
Pre-op ChatGPT-4o 11 1.64 ± 0.67 2.0 1 3 24.82 15.18 0.004* Gemini vs. Copilot: p = 0.005*
ChatGPT-3.5 11 1.82 ± 0.75 2.0 1 3 28.09
Gemini 11 1.18 ± 0.40 1.0 1 2 15.82
Copilot 11 2.55 ± 0.82 3.0 1 4 39.86
Claude 11 2.09 ± 1.04 2.0 1 4 31.41
Intra-op ChatGPT-4o 13 1.31 ± 0.48 1.0 1 2 20.54 33.44 < 0.001*

ChatGPT-4o vs. Copilot: p = 0.000*

ChatGPT-3.5 vs. Copilot: p = 0.002*

Gemini vs. Copilot: p = 0.000*

Copilot vs. Claude: p = 0.000*

ChatGPT-3.5 13 1.85 ± 0.69 2.0 1 3 32.96
Gemini 13 1.39 ± 0.65 1.0 1 3 22.04
Copilot 13 3.00 ± 0.41 3.0 2 4 56.12
Claude 13 1.85 ± 0.55 2.0 1 3 33.35
Post-op ChatGPT-4o 8 1.50 ± 0.53 1.5 1 2 14.75 19.08 0.001*

ChatGPT-4o vs. Claude: p = 0.042*

Gemini vs. Copilot: p = 0.038*

Gemini vs. Claude: p = 0.007*

ChatGPT-3.5 8 2.00 ± 0.76 2.0 1 3 21.5
Gemini 8 1.13 ± 0.35 1.0 1 2 9.31
Copilot 8 2.50 ± 0.93 2.5 1 4 27.12
Claude 8 2.63 ± 0.52 3.0 2 3 29.81

n Number, Min Minimum, Max Maximum, SD Standard Deviation

*statistically significant

In the intraoperative phase, the Kruskal-Wallis test indicated significant differences among chatbots (H = 33.44, p < 0.001). Copilot achieved the highest mean accuracy (3.00 ± 0.41), followed by Claude (1.87 ± 0.55), ChatGPT-3.5 (1.85 ± 0.69), Gemini (1.39 ± 0.65), and ChatGPT-4o (1.31 ± 0.48). Post hoc analyses showed that Claude, ChatGPT-3.5, ChatGPT-4o, and Gemini all had significantly lower (better) accuracy scores than Copilot (p = 0.000, p = 0.002, p = 0.000, and p = 0.000, respectively) (Table 8).

In the postoperative phase, significant differences were also observed among chatbots (H = 19.08, p = 0.001). Claude had the highest mean score (2.63 ± 0.52), followed by Copilot (2.50 ± 0.93), ChatGPT-3.5 (2.00 ± 0.76), ChatGPT-4o (1.50 ± 0.53), and Gemini (1.13 ± 0.35). Post hoc comparisons revealed significant differences between Claude and ChatGPT-4o (p = 0.042), Claude and Gemini (p = 0.007), and Copilot and Gemini (p = 0.038) (Table 8).

It should be noted that in this scoring system, lower mean accuracy scores indicate higher accuracy performance. Therefore, although Copilot and Claude exhibited higher numerical mean scores, this actually reflects lower accuracy compared to models with lower mean scores such as Gemini and ChatGPT-4o in some phases (Table 8).

In the preoperative phase, significant differences were found among chatbots (H = 15.34, p = 0.004). Copilot had the highest mean score (2.55 ± 0.52), followed by Claude (2.18 ± 0.98), ChatGPT-3.5 (1.82 ± 0.87), ChatGPT-4o (1.46 ± 0.82), and Gemini (1.36 ± 0.51). Post hoc pairwise comparisons indicated that ChatGPT 4o significantly outperformed Copilot (p = 0.038), and Gemini significantly outperformed Copilot (p = 0.005) (Table 9).

Table 9.

Descriptive statistics and comparative analyses of chatbot performances across different phases (pre-op, intra-op, and post-op), as evaluated by Reviewer 2

Phase Chatbot n Mean ± SD Median Min Max Mean Rank Kruskal-Wallis H Post-hoc tests (Mann-Whitney U test with Bonferroni correction)
H p
Pre-op ChatGPT-4o 11 1.46 ± 0.82 1.0 1 3 20.27 15.34 0.004*

ChatGPT-4o vs. Copilot: p = 0.038*

Gemini vs. Copilot: p = 0.005*

ChatGPT-3.5 11 1.82 ± 0.87 2.0 1 3 27.14
Gemini 11 1.36 ± 0.51 1.0 1 2 19.27
Copilot 11 2.55 ± 0.52 3.0 2 3 40.45
Claude 11 2.18 ± 0.98 2.0 1 4 32.86
Intra-op ChatGPT-4o 13 1.46 ± 0.52 1.0 1 2 24.08 26.64 < 0.000*

ChatGPT-4o vs. Copilot: p = 0.000*

ChatGPT-3.5 vs. Copilot: p = 0.013*

Gemini vs. Copilot: p = 0.005*

Copilot vs. Claude: p = 0.001*

ChatGPT-3.5 13 2.0 ± 0.71 2.0 1 3 35.62
Gemini 13 1.54 ± 0.97 1.0 1 4 23.81
Copilot 13 3.08 ± 0.64 3.0 2 4 54.19
Claude 13 1.62 ± 0.65 2.0 1 3 27.31
Post-op ChatGPT-4o 8 1.50 ± 0.54 1.5 1 2 14.0 16.66 0.002* Gemini vs. Claude: p = 0.042*
ChatGPT-3.5 8 2.13 ± 0.64 2.0 1 3 22.62
Gemini 8 1.25 ± 0.71 1.0 1 3 10.44
Copilot 8 2.50 ± 0.76 3.0 1 3 27.69
Claude 8 2.50 ± 0.54 2.5 2 3 27.75

n Number, Min Minimum, Max Maximum, SD Standard Deviation

*statistically significant

In the intraoperative phase, the Kruskal-Wallis test revealed significant differences among chatbots (H = 26.64, p < 0.001). Copilot had the highest mean score (3.08 ± 0.64), followed by ChatGPT-3.5 (2.00 ± 0.71), Claude (1.62 ± 0.65), ChatGPT-4o (1.46 ± 0.52), and Gemini (1.54 ± 0.97). Post hoc analyses showed that ChatGPT-4o, ChatGPT-3.5, Gemini, and Claude all had significantly lower (better) accuracy scores than Copilot (p = 0.000, p = 0.013, p = 0.005, and p = 0.001, respectively) (Table 9).

In the postoperative phase, significant differences were also observed among chatbots (H = 16.66, p = 0.002). Copilot and Claude shared the highest mean score (2.50 ± 0.76 and 2.50 ± 0.54, respectively), followed by ChatGPT-3.5 (2.13 ± 0.64), ChatGPT-4o (1.50 ± 0.54), and Gemini (1.25 ± 0.71). Post hoc comparisons revealed a significant difference between Gemini and Claude (p = 0.042) (Table 9).

As lower mean accuracy scores indicate higher accuracy performance, although Copilot and Claude exhibited higher numerical mean scores, this actually reflects lower accuracy compared to models with lower mean scores such as Gemini and ChatGPT-4o in some phases (Table 9).

Discussion

This study is, to the best of our knowledge, the first to examine the responses of general-purpose LLM-based chatbots to frequently asked questions about pediatric dental procedure sedation, using the AAPD/ASA guidelines as a reference. Gemini and ChatGPT-4o produced the most guideline-concordant answers, while Claude, ChatGPT-3.5 and most notably Copilot generated progressively more clinically relevant errors. Crucially, even the top two models still required expert clarification in roughly one-third of their responses, underscoring that current accuracy gains remain insufficient for unsupervised clinical use. Model size is not the only consideration; there are other factors to take into account. In a 2025 content analysis of 4,900 LLM research summaries, the authors showed that the latest GPT-4-class chatbots exhibited a fivefold increase in the propensity to produce over-generalized statements, even when explicitly prompted to avoid inaccuracies [25]. Despite the occurrence of incremental performance gains, the persistent presence of these lends further support to the calls from guideline-adherence frameworks such as AMEGA and reporting standards like TRIPOD-LLM for continuous, transparent re-evaluation of clinical LLMs [26, 27]. Therefore, the present study was planned based on the importance of evaluating the accuracy of the answers provided by LLM from a clinical perspective.

In the present study, the questions concerning pediatric dental sedation were divided into three subheadings: preoperative, intraoperative and postoperative. The rationale behind this categorization is to ascertain the inquiries posed by individuals seeking information regarding pediatric dental sedation, and the degree of accuracy with which these inquiries can be addressed. Indeed, the public could direct their attention more profoundly towards the domain of pre-sedation preparations and the subsequent management of complications, along with the requisite precautions. Conversely, healthcare professionals may find greater utility in leveraging AI-powered chatbots to access contemporary guidelines for decision-making and ensuring optimal clinical practices. In the current study, an extensive review was conducted to ascertain the extent to which various AI-based chatbots provide precise responses to frequently asked questions concerning pediatric dental sedation. The investigation drew upon current clinical practice guidelines formulated by experts in the fields of pediatrics, anesthesia, and pediatric dentistry, serving as a reference point for the evaluation.

In the present study, a 32-question test based on sedation guidelines in pediatric dentistry was administered to five general-purpose LLMs. The results demonstrated that Gemini exhibited the highest accuracy rate, with ChatGPT-4o ranking second. Notably, both models were able to provide the majority of their responses in full compliance with the guidelines. ChatGPT-3.5 and Claude demonstrated intermediate performance levels, producing a combination of accurate and partially incorrect responses. Notably, Copilot exhibited the lowest accuracy and was identified as the model most susceptible to deviations from the established guidelines. Whilst the performance profiles across phases (pre-, intra-, and postoperative) were found to be generally similar, additional errors were identified in the postoperative questions for some models (particularly Claude). These results underscore the critical importance of selecting an LLM for clinical information retrieval and the need for expert validation, even with the most sophisticated models.

Recent evaluations focusing on anesthesia reflect the performance hierarchy we have observed. Fujimoto et al. [28] compared ChatGPT-4, Claude-3 Opus, and Gemini 1.0 with questions from the Japanese Dental Society of Anesthesiology Board Certification exam and found that GPT-4 achieved the highest correct response rate (51.2%), followed closely by Claude (47.4%), while Gemini lagged behind with 30.3% [28]. Although the ranking differs slightly from our Gemini, both studies agree that newer, larger-parameter models cut error rates yet still leave many answers in need of expert correction, particularly on drug-dose and airway-management items. This is further reinforced by broader peri-operative research. In a 2025 study of Corpman et al. [29], the authors tested four popular LLM-based chatbots with basic and complex regional-anesthesia scenarios and they conclude that, at present, LLM chatbots are not ready to provide reliable clinical decision support for regional anesthesia [29]. Also, Mondillo et al. [30] evaluated six mainstream LLMs (ChatGPT-3.5, ChatGPT-4, Claude 3-OPUS, Gemini 1.0, Mixtral 8 × 7B, and Llama 2 70B) on 227 multiple-choice pediatrics questions, first “out of the box” and then after feeding each model the source journal articles as a bespoke training corpus. Targeted training boosted performance with Claude rising to 95.6%, ChatGPT-4 to 91.6% The authors conclude that there is a need for specific training datasets that reflect the complexities of pediatric conditions to tailor the models accurately, and they advocate open-source models plus curated datasets as a practical route to decision-support tools that reflect the unique age-dependent nuances of child health. This result similarly indicates that the performance of LLMs in answering medical/dental questions needs to be evaluated. In this context, these findings suggest that raw model size is less decisive than targeted training and rigorous, context-specific validation—aligning with our recommendation that even the best off-the-shelf systems require ongoing expert oversight before clinical reliance.

ChatGPT-4o and Gemini consistently outperformed the others across multiple evaluation criteria. ChatGPT-4o showed the highest DISCERN score (mean = 57.77), suggesting that it delivers medically reliable and well-structured information. Gemini closely followed with comparable quality scores. However, there was no statistically significant difference between ChatGPT-4o and Gemini in terms of DISCERN scores. These findings align with previous research showing that GPT-4-based models often provide more evidence-based and context-aware responses in clinical queries. In contrast, Copilot scored the lowest in both accuracy and quality, raising concerns about its suitability as a source of medical guidance, particularly in high-stakes fields like sedation. An experiment that fed ten frequently-asked anesthesia questions to ChatGPT-4, Google Bard (also known as Gemini) and Bing Chat found a comparable pattern: GPT-4 and Bard showed median content-accuracy ratings of 4/5, while Bing Chat trailed at 3/5, and experts flagged the latter’s answers as least comprehensive and least safe for patients [31]. This finding was consistent with the current study, and since Claude was not used in the study of Nguyen et al. [31], the differences in obtained results can be attributed to the differences of methodological designs. Taken together, these anesthesia-focused evaluations confirm that newer, larger models tend such as ChatGPT-4o to provide more evidence-aligned and well-structured content—closely matching the high DISCERN score (≈ 58/75) we recorded for ChatGPT-4o and the near-parity achieved by Gemini in pediatric dental-sedation queries. Similar concerns surface in pediatric-oriented work. A case-control study that asked ChatGPT-based (version 4.0) models to assess pre-anesthesia risk in 150 simulated day-surgery patients reported that ChatGPT had extensive knowledge related to day-surgery anesthesia with 94.0% correctness. Authors also stated that this performance of ChatGPT was also better in the domains of peri-anesthesia concerns, lifestyle, and emotional support [32]. In this study, the high success rate of the ChatGPT-v4.0 bot was consistent with our findings. The convergence of evidence supports the conclusion that, while GPT-4o and Gemini currently provide the most reliable sedation guidance available from general-purpose chatbots, the necessity for rigorous expert oversight remains paramount, particularly given the continued accessibility of lower-performing systems such as Copilot to caregivers and families seeking pediatric dental sedation information.

The concept of readability analysis can be defined as the examination of how easily a target audience can comprehend a given text. This examination is usually undertaken by applying formula-based indices such as the FKGL or the Flesch Reading Ease (FRE). Health-literacy organisations such as the American Medical Association have issued guidelines recommending that patient education materials be written at approximately sixth-grade reading level [23, 24, 33]. However, ease of reading does not equate to factual accuracy. In a study of oral cancer FAQs, Gemini’s answers were more accurate than ChatGPT-4’s, but required a higher FKGL (approximately 12.3 versus 10.3). This demonstrates an inverse relationship between readability and accuracy [34]. Similarly, a comparative evaluation of GPT models for anesthesia education before total-hip arthroplasty told a complementary story. In this context, GPT-4P produced the clearest prose while GPT-3.5 delivered the best accuracy score, underscoring that the two metrics can diverge [35]. Similar patterns emerge in ophthalmic local anesthesia leaflets, where Gemini outputs are more readable than traditional materials, yet still omit key safety details Gondode et al. [36]. Following these statements, readability analysis using the Flesch-Kincaid scoring revealed further differences in the current study. While Claude exhibited the highest readability level (the most difficult to read), indicating more complex language, ChatGPT-3.5 was the easiest to read. This highlights a critical trade-off between content depth and accessibility. Ideally, medical information -especially that intended for patient or caregiver use- should balance clarity with clinical precision. Notably, ChatGPT-4o achieved this balance, combining strong DISCERN performance with moderate readability, making it the most user-friendly option among those tested.

Two reviewers participated in the present work, which was a strong point of the current study. The inter-reviewer reliability for DISCERN scoring was excellent across all chatbots, with ICC values exceeding 0.85, reinforcing the consistency of evaluation methods. However, kappa values for accuracy varied more widely. While Gemini and ChatGPT-3.5 demonstrated substantial agreement between reviewers, Copilot and Claude achieved only fair to moderate agreement. These discrepancies suggest not only variation in information quality, but also potential ambiguity or inconsistency in how these tools frame responses. Interestingly, correlation analyses did not show any significant monotonic relationship between chatbot performance metrics. This suggests that high scores in one domain (e.g., readability) do not necessarily predict strong performance in another (e.g., accuracy), underlining the importance of multi-dimensional evaluation in AI-based applications.

This study is one of the first to simultaneously analyze accuracy, content quality, and readability together in the context of pediatric dental sedation, offering a comprehensive perspective rarely addressed in previous literature. In addition to this, the strengths of the current work lie in its rigorous, guideline-anchored gold standard, derived from the most recent AAPD/ASA recommendations. This provides a clinically relevant benchmark, as opposed to abstract knowledge questions. A comparative evaluation of five widely used, general-purpose LLMs was conducted, with identical prompts employed to ensure a level playing field. This approach enabled the identification of a transparent performance hierarchy. The 32-item test bank was stratified across the pre-, intra- and postoperative phases, with the aim of reflecting real-world information needs of both caregivers and clinicians. All responses were scored by two pediatric-dentistry experts who were blinded to the responses, and who used validated accuracy and DISCERN rubrics. This process yielded excellent inter-rater reliability and mitigated individual rater bias. Finally, by reporting readability, content quality and accuracy in parallel, a multidimensional view of chatbot utility is offered, which provides more information about patient-facing deployment than single-metric evaluations.

It is important to note that this study is subject to several noteworthy constraints. Firstly, it should be noted that since all prompts and evaluations were conducted in English, the findings may not be generalisable to other languages. The linguistic nuances, guideline translations and local terminology of, for example, Turkish or Spanish, could alter both readability and accuracy when the same chatbots are queried. Secondly, the output of a chatbot is highly sensitive to the design of the prompts and the conversational context. The method of posing each question in isolation that was utilised in this study facilitated a clean comparison; however, it does not capture the iterative, follow-up style that is characteristic of real clinical consultations. In such consultations, additional patient details or clarifications can significantly influence responses. Thirdly, despite the high inter-rater reliability, the scoring process involved human judgement, and subtle differences in how reviewers interpret “partially correct” versus “misleading” content can never be fully eliminated. In conclusion, the evaluation provides merely a time-specific overview. Due to the frequent updating of large language models, the accuracy, tone and safety features may be subject to alteration over a period of weeks or months. Consequently, the current rankings may be subject to change as the underlying models evolve. On the other hand, although real-world clinical decision-making generally involves an iterative dialogue, where follow-up questions, the patient’s medical history and contextual clarification shape the final recommendation, our study employed a single-turn, isolated-question design. This approach was chosen to establish a standardized framework that allows for direct comparison among different LLMs, independent of the impact of multi-turn correction, retrospective self-adjustment or user-driven prompting. Single-turn evaluation provides insight into the baseline reliability and patient safety orientation of each model when confronted with complex pediatric sedation questions. This is important because many non-professional users, including parents seeking guidance, tend to ask LLMs one-off clinical questions without providing detailed context. Although this design does not fully replicate the dynamic nature of clinician–patient communication, it provides a reliable and clinically relevant estimate of model performance during the initial stage of interaction. This is an important consideration, as misinformation can impact parental expectations and decision-making prior to professional consultation. At the same time, we acknowledge that LLMs may perform differently in multi-turn conversations that allow for refinement and correction. Therefore, our findings should be interpreted as reflecting the foundational, unprompted capacities of the models rather than their optimized performance during extended dialogue. Future research should incorporate multi-turn, context-rich clinical scenarios to evaluate how effectively LLMs can approximate real clinical reasoning processes, and whether iterative conversations can improve or potentially reinforce inaccurate assumptions.

In order to address the limitations of this study, future research could focus on several axes. Firstly, multilingual evaluations should be conducted, with the same set of questions applied in different languages to test the sensitivity of LLMs to terminology and local guideline translations. This will increase the universality of the findings. Secondly, rather than employing single-question prompts, multi-turn, dialogue-based scenarios incorporating patient information and follow-up questions should be designed to examine how prompt sensitivity changes in clinical reality. Thirdly, in order to reduce subjective elements in the evaluation process, automatic metrics (e.g. fact-checking algorithms or large expert panel consensus) should be combined with human ratings in order to develop a more objective accuracy metric. Fourthly, in order to observe the evolution of models’ performance with version updates, it is necessary to repeat the same protocol at three- to six-month intervals. This will enable the quantification of any ‘model drift’. Finally, a comparison should be made between the impact of domain-specific fine-tuning or retrieval-augmented systems with real-time access to pediatric dentistry and sedation guidelines and general-purpose models. This will allow for the determination of the effect of domain-specificity on accuracy and safety. These steps will serve to strengthen the evidence base, which is necessary for large language models to become reliable clinical support tools in the field of pediatric dental sedation.

The overall findings of this study underscore the current limitations of prevailing AI tools in terms of their capacity to furnish comprehensive and guideline-compliant information on the subject of pediatric dental sedation. Despite their sophisticated language capabilities, none of the evaluated chatbots consistently reflected current ASA and AAPD clinical guidelines. This underscores a fundamental challenge in the field of medical AI: the difference between linguistic fluency and clinical reliability. It is evident that conversational AI systems such as ChatGPT and Claude, which are constructed on non-specialised datasets, may encounter challenges in accessing the latest or domain-specific evidence necessary for the accurate formulation of medical recommendations.

Conclusions

In conclusion, while AI-based chatbots show promise in delivering basic medical information, significant variation exists in their accuracy, quality, and accessibility when addressing complex, guideline-sensitive topics such as pediatric dental sedation. ChatGPT-4o and Gemini were the most consistent and reliable tools in this study. However, until these systems integrate more structured clinical knowledge and transparency in source attribution, they should not be relied upon for clinical decision-making in pediatric dental settings.

Supplementary Information

Supplementary Material 1. (45.7KB, docx)

Acknowledgements

Not applicable.

Clinical trial number

Not applicable.

Abbreviations

AI

Artificial Intelligence

LLM

Large Language Models

AAP

American Academy of Pediatrics

AAPD

American Academy of Pediatric Dentistry

ASA

American Society of Anesthesiologists

FKGL

Flesch–Kincaid Grade Level

ICC

Intraclass Correlation Coefficient

n

number

Min

Minimum

Max

Maximum

SD

Standard Deviation

p

Probability Value

FRE

Flesch Reading Ease

Authors’ contributions

M.H.K. and A.D. conceived and designed the study; M.H.K., A.D. and İ.K. performed the study procedure; M.H.K. and A.D. analyzed the data; M.H.K. and A.D. prepared the tables; M.H.K., A.D. and İ.K. authored and reviewed drafts of the article; M.H.K., A.D. and İ.K. approved the final draft.

Funding

No external support was received for this study.

Data availability

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Declarations

Ethics approval and consent to participate

Not required.

Consent for publication

Not required.

Competing interests

The authors declare no competing interests.

Footnotes

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

References

  • 1.Ríos-Hoyo A, Shan NL, Li A, Pearson AT, Pusztai L, Howard FM. Evaluation of large language models as a diagnostic aid for complex medical cases. Front Med (Lausanne). 2024;11:1380148. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Schwendicke F, Samek W, Krois J. Artificial Intelligence in Dentistry: Chances and Challenges. J Dent Res. 2020;99:769–74. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Thirunavukarasu AJ, Ting DSJ, Elangovan K, Gutierrez L, Tan TF, Ting DSW. Large language models in medicine. Nat Med. 2023;29:1930–40. [DOI] [PubMed] [Google Scholar]
  • 4.Eggmann F, Weiger R, Zitzmann NU, Blatz MB. Implications of large language models such as ChatGPT for dental medicine. J Esthet Restor Dent. 2023;35:1098–102. [DOI] [PubMed] [Google Scholar]
  • 5.Haman M, Školník M. Using ChatGPT to conduct a literature review. Acc Res. 2024;31:1244–6. [DOI] [PubMed] [Google Scholar]
  • 6.Cascella M, Montomoli J, Bellini V, Bignami E. Evaluating the Feasibility of ChatGPT in Healthcare: An Analysis of Multiple Clinical and Research Scenarios. J Med Syst. 2023;47:33. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Thurzo A, Urbanová W, Novák B, Czako L, Siebert T, Stano P, Mareková S, Fountoulaki G, Kosnáčová H, Varga I. Where Is the Artificial Intelligence Applied in Dentistry? Systematic Review and Literature Analysis. Healthc (Basel). 2022;10:1269. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Ahmed N, Abbasi MS, Zuberi F, Qamar W, Halim MSB, Maqsood A, Alam MK. Artificial Intelligence Techniques: Analysis, Application, and Outcome in Dentistry-A Systematic Review. Biomed Res Int. 2021;2021:9751564. [DOI] [PMC free article] [PubMed]
  • 9.Kaya E, Gunec HG, Gokyay SS, Kutal S, Gulum S, Ates HF. Proposing a CNN Method for Primary and Permanent Tooth Detection and Enumeration on Pediatric Dental Radiographs. J Clin Pediatr Dent. 2022;46:293–8. [DOI] [PubMed] [Google Scholar]
  • 10.Koopaie M, Salamati M, Montazeri R, Davoudi M, Kolahdooz S. Salivary cystatin S levels in children with early childhood caries in comparison with caries-free children; statistical analysis and machine learning. BMC Oral Health. 2021;21:650. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Buldur M, Sezer B. Evaluating the accuracy of Chat Generative Pre-trained Transformer version 4 (ChatGPT-4) responses to United States Food and Drug Administration (FDA) frequently asked questions about dental amalgam. BMC Oral Health. 2024;24:605. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Kooraki S, Hosseiny M, Jalili MH, Rahsepar AA, Imanzadeh A, Kim GH, Hassani C, Abtin F, Moriarty JM, Bedayat A. Evaluation of ChatGPT-Generated Educational Patient Pamphlets for Common Interventional Radiology Procedures. Acad Radiol. 2024;31:4548–53. [DOI] [PubMed] [Google Scholar]
  • 13.Chonde DB, Pourvaziri A, Williams J, McGowan J, Moskos M, Alvarez C, Narayan AK, Daye D, Flores EJ, Succi MD. RadTranslate: An Artificial Intelligence-Powered Intervention for Urgent Imaging to Enhance Care Equity for Patients With Limited English Proficiency During the COVID-19 Pandemic. J Am Coll Radiol. 2021;18:1000–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Haze T, Kawano R, Takase H, Suzuki S, Hirawa N, Tamura K. Influence on the accuracy in ChatGPT: Differences in the amount of information per medical field. Int J Med Inf. 2023;180:105283. [DOI] [PubMed] [Google Scholar]
  • 15.Fujimoto M, Kuroda H, Katayama T, Yamaguchi A, Katagiri N, Kagawa K, Tsukimoto S, Nakano A, Imaizumi U, Sato-Boku A, Kishimoto N, Itamiya T, Kido K, Sanuki T. Evaluating Large Language Models in Dental Anesthesiology: A Comparative Analysis of ChatGPT-4, Claude 3 Opus, and Gemini 1.0 on the Japanese Dental Society of Anesthesiology Board Certification Exam. Cureus. 2024;16:e70302. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Coté CJ, Wilson S. Guidelines for Monitoring and Management of Pediatric Patients Before, During, and After Sedation for Diagnostic and Therapeutic Procedures. Pediatr Dent. 2019;41:259–60. [PubMed] [Google Scholar]
  • 17.Practice Guidelines for Moderate Procedural Sedation and Analgesia. 2018: A Report by the American Society of Anesthesiologists Task Force on Moderate Procedural Sedation and Analgesia, the American Association of Oral and Maxillofacial Surgeons, American College of Radiology, American Dental Association, American Society of Dentist Anesthesiologists, and Society of Interventional Radiology. Anesthesiology. 2018;128:437–79. [DOI] [PubMed]
  • 18.Ekmekci E, Durmazpinar PM. Evaluation of different artificial intelligence applications in responding to regenerative endodontic procedures. BMC Oral Health. 2025;25:53. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Özbek EA, Ertan MB, Kından P, Karaca MO, Gürsoy S, Chahla J. ChatGPT can offer at least satisfactory responses to common patient questions regarding Hip Arthroscopy. Arthroscopy. 2025;41(6):1806–27. [DOI] [PubMed]
  • 20.Dursun D, Bilici Geçer R. Can artificial intelligence models serve as patient information consultants in orthodontics? BMC Med Inf Decis Mak. 2024;24:211. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Mika AP, Martin JR, Engstrom SM, Polkowski GG, Wilson JM. Assessing ChatGPT Responses to Common Patient Questions Regarding Total Hip Arthroplasty. J Bone Joint Surg Am. 2023;105:1519–26. [DOI] [PubMed] [Google Scholar]
  • 22.Gökcek Taraç M, Nale T. Artificial intelligence in pediatric dental trauma: do artificial intelligence chatbots address parental concerns effectively? BMC Oral Health. 2025;25:736. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Badarudeen S, Sabharwal S. Assessing readability of patient education materials: current role in orthopaedics. Clin Orthop Relat Res. 2010;468:2572–80. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Wang LW, Miller MJ, Schmitt MR, Wen FK. Assessing readability formula differences with written health information materials: application, results, and recommendations. Res Social Adm Pharm. 2013;9:503–16. [DOI] [PubMed] [Google Scholar]
  • 25.Peters U, Chin-Yee B. Generalization bias in large language model summarization of scientific research. R Soc Open Sci. 2025;12:241776. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Fast D, Adams LC, Busch F, Fallon C, Huppertz M, Siepmann R, Prucker P, Bayerl N, Truhn D, Makowski M, Löser A, Bressem KK. Autonomous medical evaluation for guideline adherence of large language models. NPJ Digit Med. 2024;7:358. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Gallifant J, Afshar M, Ameen S, Aphinyanaphongs Y, Chen S, Cacciamani G, Demner-Fushman D, Dligach D, Daneshjou R, Fernandes C, Hansen LH, Landman A, Lehmann L, McCoy LG, Miller T, Moreno A, Munch N, Restrepo D, Savova G, Umeton R, Gichoya JW, Collins GS, Moons KGM, Celi LA, Bitterman DS. The TRIPOD-LLM reporting guideline for studies using large language models. Nat Med. 2025;31:60–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Fujimoto M, Kuroda H, Katayama T, Yamaguchi A, Katagiri N, Kagawa K, Tsukimoto S, Nakano A, Imaizumi U, Sato-Boku A, Kishimoto N, Itamiya T, Kido K, Sanuki T. Evaluating Large Language Models in Dental Anesthesiology: A Comparative Analysis of ChatGPT-4, Claude 3 Opus, and Gemini 1.0 on the Japanese Dental Society of Anesthesiology Board Certification Exam. Cureus. 2024;16:e70302. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Corpman D, Auyong DB, Walters D, Gray A. Using AI chatbots as clinical decision support for regional anesthesia: is the new era of "e-Consults" ready for primetime? Reg Anesth Pain Med. 2025:rapm-2025-106850. [DOI] [PubMed]
  • 30.Mondillo G, Perrotta A, Frattolillo V, Colosimo S, Indolfi C, Miraglia del Giudice M, Rossi F. Large language models performance on pediatrics question: a new challenge. J Med Artif Intell. 2025;8:14. [Google Scholar]
  • 31.Nguyen TP, Carvalho B, Sukhdeo H, Joudi K, Guo N, Chen M, Wolpaw JT, Kiefer JJ, Byrne M, Jamroz T, Mootz AA, Reale SC, Zou J, Sultan P. Comparison of artificial intelligence large language model chatbots in answering frequently asked questions in anesthesia. BJA Open. 2024;10:100280. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Cheng T, Li Y, Gu J, He Y, He G, Zhou P, Li S, Xu H, Bao Y, Wang X. The performance of ChatGPT in day surgery and pre-anesthesia risk assessment: a case-control study of 150 simulated patient presentations. Perioper Med (Lond). 2024;13:111. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33.Roster K, Kann RB, Farabi B, Gronbeck C, Brownstone N, Lipner SR. Readability and Health Literacy Scores for ChatGPT-Generated Dermatology Public Education Materials: Cross-Sectional Analysis of Sunscreen and Melanoma Questions. JMIR Dermatol. 2024;7:e50163. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Diniz-Freitas M, López-Pintor RM, Santos-Silva AR, Warnakulasuriya S, Diz-Dios P. Assessing the accuracy and readability of ChatGPT-4 and Gemini in answering oral cancer queries—an exploratory study. Explor Digit Health Technol. 2024;2:334–45. [Google Scholar]
  • 35.Fisher AD, Fisher G, George R, Hay E, Wolla CD, Wolf BJ. Comparative evaluation of ChatGPT versions in anesthesia patient education for total hip arthroplasty. J Med Artif Intell. 2024;7:19. [Google Scholar]
  • 36.Gondode P, Duggal S, Garg N, Lohakare P, Jakhar J, Bharti S, Dewangan S. Comparative Analysis of Accuracy, Readability, Sentiment, and Actionability: Artificial Intelligence Chatbots (ChatGPT and Google Gemini) versus Traditional Patient Information Leaflets for Local Anesthesia in Eye Surgery. Br Ir Orthopt J. 2024;20:183–92. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplementary Material 1. (45.7KB, docx)

Data Availability Statement

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.


Articles from BMC Oral Health are provided here courtesy of BMC

RESOURCES