Comparative evaluation of ChatGPT, Gemini, and Grok in clinical decision-making and general knowledge assessment for impacted maxillary canines

Abstract

Objective

This study aimed to compare extraction versus orthodontic eruption decisions for impacted maxillary canines made by three artificial intelligence-based chatbots (ChatGPT, Gemini, and Grok) with those made by orthodontist raters, and to evaluate the overall accuracy of these artificial intelligence-generated recommendations.

Methods

Thirty-three patients with impacted maxillary canines were selected, and standardized case scenarios incorporating key diagnostic parameters were presented to the three chatbots. Their treatment decisions were recorded and compared with orthodontists’ consensus decisions. Additionally, 10 general queries regarding impacted maxillary canines were submitted to the chatbots. The responses were rated by three orthodontists using a modified 5-point Global Quality Score.

Results

The chatbots and orthodontists showed moderate agreement regarding treatment decisions (κ = 0.411–0.524, P < 0.05). Gemini produced significantly more discordant responses, frequently over-recommending orthodontic eruptions (P = 0.002), whereas Grok and ChatGPT received significantly higher scores than Gemini in the case-based scenarios (P < 0.001). Grok outperformed both ChatGPT and Gemini for general queries (P = 0.006).

Conclusions

While Gemini showed lower clinical alignment with orthodontists for treatment decisions regarding impacted canines, ChatGPT and Grok demonstrated moderate agreement with orthodontists and produced relatively accurate responses. These findings highlight the potential of chatbots as supportive tools for orthodontic decision-making. However, their use requires careful supervision to avoid the risks associated with inaccurate or misleading recommendations.

INTRODUCTION

Deviations in the physiological eruption pathways of maxillary canines frequently result in impaction.¹ This phenomenon can be largely attributed to the fact that the maxillary canines have the longest developmental period and eruption path, which starts from their formation site lateral to the piriform fossa and ends at their final occlusal position.² These teeth are the second-most frequently impacted teeth after third molars, and the prevalence of impaction in these teeth is approximately 2%.³ Due to the complex nature of impacted maxillary canines, their management requires a multidisciplinary approach, with treatment options that include extraction or surgical exposure with subsequent orthodontic guidance.

The decision to extract an impacted maxillary canine or pursue orthodontic eruption is influenced by multiple factors and remains a matter of clinical judgment. Patient considerations include age, motivation, cooperation, and oral hygiene, whereas clinical characteristics guiding the treatment decision include skeletal growth, occlusal relationships, and arch space. Radiographic assessments are essential for evaluating the severity and prognosis of impactions. Orthopantomograms (OPGs) are the gold standard for the initial diagnostic evaluation, allowing clinicians to assess the degree of overlap between the impacted canine and adjacent teeth, its angulation relative to the midline, and the vertical distance from the cusp tip to the occlusal plane.³ Nevertheless, OPGs are two-dimensional radiographic images with limitations, including magnification, distortion, and anatomical superimposition.⁴

Cone-beam computed tomography (CBCT) provides a comprehensive three-dimensional (3D) evaluation, enabling the assessment of the buccopalatal position and depth of the impacted canine, the amount of surrounding bone coverage, and any root resorption affecting adjacent teeth.⁴ Recent studies have confirmed the superior diagnostic accuracy of CBCT in comparison with conventional radiography in the assessment of impacted canines.^4,5 The Kau-Pan-Gallerano index is a classification system that can quantify the severity of impaction on CBCT scans by assigning grid-based scores to the positions of both the cusp and root tip along the x, y, and z axes, with the cumulative score indicating treatment difficulty.⁶ Impacted canines with greater lateral incisor overlap,⁷ angulations over 31 degrees,⁸ and vertical distances over 14 mm from the occlusal plane⁹ tend to have poor prognoses and longer treatment times and are more likely to be extracted, especially when the KPG total scores fall in the “Extremely difficult” range.^6,10 However, despite these guidelines, the choice between extraction and orthodontic eruption of impacted canines is not always clearly defined and remains subject to clinical judgment.

Recent technological breakthroughs have increased interest in the application of artificial intelligence (AI), particularly large language models (LLMs), in healthcare.¹¹ By analyzing vast amounts of data, AI-powered chatbot models offer substantial potential for understanding clinical expertise and delivering relevant information, thereby enabling healthcare providers to improve diagnostic accuracy, develop personalized treatment plans, and refine clinical decision-making, ultimately enhancing patient care efficiency and treatment outcomes.^11,12

Recent studies have evaluated the accuracy of chatbots in dentistry,^12-18 including assessments of their performance on orthodontic topics such as general orthodontic information,^11,19-23 clear aligners,^24,25 interceptive orthodontics,²⁶ orthognathic surgery,^27,28 and temporomandibular joint disorders.²⁹ As chatbots have gained attention for addressing dental queries, evaluating their performance is crucial because they can generate plausible-sounding but incorrect answers from potentially biased or outdated data, posing serious concerns in healthcare.^13,30

Despite the substantial potential of chatbots in orthodontics, studies on their reliability in making complex clinical decisions,³¹ such as those involving maxillary canine impactions, are lacking. Since treatment decisions for impacted canines rely heavily on clinician judgment, AI-powered chatbots could be a valuable tool for providing more objective data-driven support that could enhance orthodontic planning for impacted maxillary canines.

Therefore, the primary aim of this study was to compare treatment decisions regarding extraction versus orthodontic eruption for impacted maxillary canines that were made by three AI chatbots with those made by orthodontist raters, and to evaluate the overall accuracy of these AI-generated recommendations. The secondary aim was to evaluate the accuracy of AI-generated responses to general queries regarding impacted maxillary canines. The null hypothesis was that the treatment decisions (extraction versus orthodontic eruption) for impacted maxillary canines made by the AI chatbots did not differ from those made by the orthodontist raters and that the accuracy of the chatbots’ recommendations did not differ from that of the orthodontists.

MATERIALS AND METHODS

This study was approved by the Ethics Committee of Izmir Tinaztepe University (protocol number: MUKCE2025/33) and written informed consent was obtained from all participants. Using G*Power (version 3.1.9.7; Kiel University, Kiel, Germany), an a priori power analysis was conducted employing the repeated-measures analysis of variance (ANOVA) test with a significance level (α) of 0.05, statistical power of 0.80, and a medium effect size (f = 0.25).³² In the absence of prior studies comparing treatment decisions made by chatbots with those made by orthodontists, a medium effect size was selected on the basis of similar studies in the literature.^25,33 According to this calculation, the minimum required sample size was determined to be 30 patients. Therefore, clinical and radiographic records (OPGs and CBCTs) of 33 patients diagnosed with impacted maxillary canines were randomly selected from the archive of the Orthodontics Department at Izmir Tinaztepe University, Izmir, Türkiye. The records were reviewed in reverse chronological order until the required sample size was obtained. All the patients were of Caucasian ethnicity. Of these, 21 were female (mean age: 19.76 ± 10.11 years) and 12 were male (mean age: 24.09 ± 11.21 years).

Detailed case-based scenarios of the selected patients with impacted maxillary canines were formatted into standardized prompts and provided to three chatbots: ChatGPT (o3-mini; OpenAI, San Francisco, CA, USA), Gemini Advanced (2.0 Pro Experimental; Google, Mountain View, CA, USA), and Supergrok (model 3; xAI, San Francisco, CA, USA). The treatment decisions (extraction versus orthodontic eruption) recommended by the chatbots and the accuracy of their responses were evaluated. Comprehensive clinical scenarios based on written case descriptions were used, with specific parameters measured and converted into numerical data for chatbot analysis.¹⁶ The information used to construct each scenario was derived from clinical records and radiographs, as follows:

Information regarding the following parameters was obtained from the patient records:

• -Chronological age of the patients
• -Available space (mm) in the dental arch for the impacted tooth

The following parameters were assessed using OPGs:

• -Angulation (degrees) of the impacted canine, which was defined as the angle between the long axis of the impacted canine and the midline constructed from the anterior nasal spine and the intermaxillary suture
• -Distance (mm) from the cusp tip of the impacted canine to the occlusal plane
• -The amount of overlap with the adjacent lateral teeth (obtained by dividing the root of the lateral into mesial and distal halves)

The following parameters were assessed using CBCT:

• -Buccopalatal position of the impacted canine (palatal, buccal, or middle of the alveolar crest)
• -Distance from the cusp tip to the mid-alveolar crest (mm), as observed on the axial plane on CBCT
• -Presence or absence of root resorption in adjacent teeth. In cases showing root resorption, the amount of resorption was classified as follows: (1) no resorption (intact root surfaces, but the cementum layer may be lost); (2) slight resorption (resorption of up to half of the dentin thickness from the pulp); (3) moderate resorption (resorption midway to the pulp or more, with the pulp lining remaining unbroken); (4) severe resorption (pulp exposed owing to resorption).³⁴

Each case was entered as an open-ended question with a separate prompt, and the responses were recorded verbatim without giving the chatbots further prompts or context beyond that provided within the initial presentation of the case (Figure 1). Additionally, ten general queries related to impacted maxillary canines were compiled and presented to the chatbots in the same standardized manner (Figure 2 and Supplementary Table 1). The chat window was refreshed for every question, and the search history was cleared between each query. One author (MGK) compiled the chatbot responses into randomized word files for the raters’ evaluations.

Figure 1.

Example of a case-based scenario provided to the chatbots along with the corresponding output.

CBCT, cone-beam computed tomography.

Figure 2.

Example of a general query related to impacted maxillary canines along with the corresponding chatbot output.

The same detailed case-based scenarios, including the radiographic data of the patients, were shown to three orthodontic raters (GAS, CS, and HS). The raters first decided on treatment (extraction versus orthodontic eruption) and then used a modified 5-scale Global Quality Score (GQS) to blindly evaluate the chatbots’ responses for both case scenarios and general queries using the best available evidence and their clinical expertise.³⁵ The participating orthodontists were from both private practice and academic settings and had between 8 and 10 years of professional experience, thus ensuring that the assessments were standardized and based on similar current knowledge in the field of orthodontics.

For assessment of intrarater reliability, 30% of the chatbot responses were randomly and equally selected from each chatbot group to be reassessed by the raters after two weeks.¹¹

Statistical analysis

Descriptive statistics (median and interquartile range) were summarized for all variables. Intrarater reliability among orthodontists was assessed using Cohen’s kappa values for treatment decisions and weighted kappa values for case-based scenarios and general queries. For treatment decisions, the agreement among orthodontist raters and that among the chatbots was evaluated separately using Fleiss’ kappa values.

Cohen’s kappa values were used to assess the overall level of agreement between the chatbot decisions and orthodontist-aggregated treatment decisions, which allowed us to determine whether the chatbots were generally making treatment decisions that were consistent with expert clinical judgment. McNemar’s test was used to detect directional discordance and examine the pattern of disagreement between chatbots and orthodontists.

The Kruskal–Wallis test with a post-hoc Dunn’s test was employed to compare the mean ranks of the chatbot groups on the basis of both individual and pooled rater scores for case-based scenarios and general queries. Additionally, the Friedman test, followed by pairwise comparisons using the Wilcoxon signed-rank test with Bonferroni adjustment (with a significance threshold of P < 0.017), was used to evaluate the mean ranks of chatbot response accuracy in case-based scenarios and general queries across both pooled and individual rater scores (Supplementary Tables 2–9).

All tests were 2-tailed, and statistical significance was set at P < 0.05. Calculations were performed using SPSS (version 26.0; IBM, Armonk, NY, USA).

RESULTS

In the assessment of within-rater consistency, intrarater reliability was high across all categories (κ > 0.767), indicating substantial to almost perfect agreement (Table 1).³⁶ The frequencies and distributions of treatment decisions and GQSs for chatbot responses are shown in Tables 2 and 3.

Table 1.

Intrarater reliability of orthodontist raters

	Rater	κ	P value	95% CI
				Lower	Upper
Treatment decisions	Rater 1	1.000†	0.002**	1.000	1.000
	Rater 2	1.000†	0.002**	1.000	1.000
	Rater 3	1.000†	0.002**	1.000	1.000
Case-based scenarios	Rater 1	0.869‡	< 0.001***	0.757	0.981
	Rater 2	0.892‡	< 0.001***	0.783	1.000
	Rater 3	0.822‡	< 0.001***	0.677	0.967
General queries	Rater 1	0.886‡	< 0.001***	0.661	1.111
	Rater 2	0.804‡	< 0.001***	0.565	1.043
	Rater 3	0.767‡	< 0.001***	0.518	1.017

CI, confidence interval; κ, kappa.

**P < 0.01; ***P < 0.001.

^†Cohen’s kappa; ^‡weighted kappa.

Table 2.

Treatment decisions of chatbots and orthodontist raters

	Orthodontic eruption	Extraction
ChatGPT	25 (75.8)	8 (24.2)
Gemini	10 (30.3)	23 (69.7)
Grok	17 (51.5)	16 (48.5)
Rater 1	21 (63.6)	12 (36.4)
Rater 2	19 (57.6)	14 (42.4)
Rater 3	19 (57.6)	14 (42.4)

Values are presented as number (%).

Table 3.

GQS distribution of chatbot responses

		ChatGPT	Gemini	Grok
GQS of case-based scenarios	1. Very poor quality	6 (6.1)	12 (12.1)	2 (2.0)
	2. Poor quality	5 (5.1)	22 (22.2)	1 (1.0)
	3. Moderate quality	16 (16.2)	32 (32.3)	15 (15.2)
	4. Good quality	32 (32.3)	19 (19.2)	34 (34.3)
	5. Excellent quality	40 (40.3)	14 (14.2)	47 (47.5)
GQS of general queries	1. Very poor quality	0 (0.0)	2 (6.7)	0 (0.0)
	2. Poor quality	4 (13.3)	5 (16.7)	0 (0.0)
	3. Moderate quality	14 (46.7)	7 (23.3)	7 (23.3)
	4. Good quality	11 (36.7)	15 (50.0)	19 (63.3)
	5. Excellent quality	1 (3.3)	1 (3.3)	4 (13.4)

Values are presented as number (%).

GQS, Global Quality Score.

Regarding treatment decisions, orthodontist raters demonstrated substantial agreement (κ = 0.706, P < 0.001), whereas chatbots showed only fair agreement (κ = 0.352, P < 0.001; Table 4).³⁶ Given the substantial agreement, orthodontists’ treatment decisions were aggregated through majority vote (2/3 or 3/3) to form a consensus for subsequent comparison with each chatbot’s decisions. Moderate agreement was observed between the chatbots and orthodontists (κ = 0.411–0.524, P < 0.05), although McNemar’s test showed that Gemini significantly over-recommended orthodontic eruption in comparison with orthodontist decisions in 10 discordant cases (P = 0.002; Table 5).

Table 4.

Agreement levels for orthodontist raters and chatbots regarding treatment decisions

	κ	P value	95% CI
			Lower	Upper
Rater 1	0.706	< 0.001***	0.700	0.713
Rater 2
Rater 3
ChatGPT	0.352	< 0.001***	0.346	0.358
Gemini
Grok

CI, confidence interval; κ, Fleiss’ kappa.

***P < 0.001.

Table 5.

Discordance and agreement between orthodontists’ aggregated treatment decisions and chatbots’ recommendations

	P value†	κ	P value‡
Orthodontists vs. ChatGPT	0.125	0.524	0.001***
Orthodontists vs. Gemini	0.002**	0.441	0.002**
Orthodontists vs. Grok	0.508	0.451	0.008**

κ, kappa.

**P < 0.01; ***P ≤ 0.001.

^†McNemar’s P value; ^‡Cohen’s kappa P value.

In the case-based scenarios, a consistent pattern was observed across the individual and pooled rater scores for chatbot response accuracy, with Gemini scoring significantly lower than ChatGPT and Grok (P < 0.001). No significant difference was observed in the scores between ChatGPT and Grok (P > 0.05). Additionally, differences were found among the raters in how they scored the responses from each chatbot (P < 0.05; Table 6 and Figure 3).

Table 6.

Evaluation of the accuracy of chatbot responses regarding case-based scenarios by raters

Chatbot	Raters								P value†	P value† (pooled scores)
	Rater 1AB			Rater 2A			Rater 3B
	Median	IQR		Median	IQR		Median	IQR
ChatGPTa	4.0a	2.0		5.0a	2.0		3.0a	2.0	0.018*	< 0.001***
Geminib	3.0b	2.0		3.0b	2.0		4.0b	2.0	0.007**
Groka	4.0ABa	1.0		5.0Aa	1.0		4.0Ba	1.0	0.001***
P value‡	< 0.001***			< 0.001***			< 0.001***
P value‡ (pooled scores)	< 0.001***

IQR, interquartile range.

*P < 0.05; **P < 0.01; ***P ≤ 0.001.

^†Friedman test with pairwise comparisons using Wilcoxon signed-rank test and Bonferroni adjustment (significant at P < 0.017); ^‡Kruskal–Wallis test.

Different lowercase letters within a column indicate statistically significant differences between chatbots, as rated by each individual rater. Different uppercase letters within a row indicate statistically significant differences between raters for each chatbot.

Different lowercase letters in the “Chatbot” column and different uppercase letters in the “Raters” row indicate significant differences for pooled scores across all raters and chatbots, respectively.

Figure 3.

Case-based Global Quality Score (GQS) ratings of chatbots by the raters.

For general queries, the pooled rater scores indicated that Grok achieved higher scores than ChatGPT and Gemini (P = 0.006), whereas no difference was observed between ChatGPT and Gemini (P > 0.05). Moreover, variations were observed among raters in their scoring of ChatGPT responses (P = 0.020; Table 7 and Figure 4).

Table 7.

Evaluation of the accuracy of chatbot responses regarding general queries by raters

Chatbot	Raters								P value†	P value† (pooled scores)
	Rater 1A			Rater 2B			Rater 3AB
	Median	IQR		Median	IQR		Median	IQR
ChatGPTa	4.0	1.0		3.0	1.0		3.0	1.0	0.020*	0.008**
Geminia	3.5	1.0		3.5	2.0		4.0	1.0	0.368
Grokb	4.0	1.0		4.0	1.0		4.0	0.0	0.289
P value‡	0.117			0.129			0.076
P value‡ (pooled scores)	0.006**

IQR, interquartile range.

*P < 0.05; **P < 0.01.

^†Friedman test with pairwise comparisons using Wilcoxon signed-rank test and Bonferroni adjustment (significant at P < 0.017); ^‡Kruskal–Wallis test.

Different lowercase letters in the “Chatbot” column and different uppercase letters in the “Raters” row indicate significant differences for pooled scores across all raters and chatbots, respectively.

Figure 4.

Global Quality Score (GQS) assigned by the raters to the chatbots’ responses to general queries.

DISCUSSION

Impacted maxillary canines present a major clinical challenge that often requires treatment decisions guided by clinician expertise. To support this process, AI algorithms, such as supervised machine learning and deep learning methods, have been extensively researched in recent years and have demonstrated diagnostic potential by classifying impacted canines from panoramic radiographs and predicting palatal impactions on the basis of maxillary dimensions estimated from CBCT scans.^37,38 However, while such AI models can facilitate treatment planning, they are often limited by the need for large annotated datasets, high computational demands, and limited interpretability for clinical end-users.³⁹ In contrast, chatbot-based LLMs offer the advantages of accessibility and real-time interaction, making them more practical for point-of-care applications. Although their diagnostic accuracy is still evolving, their ability to synthesize complex information and respond to clinical queries can enable dynamic support for decision-making.

Considering these recent advancements, this study aimed to evaluate the accuracy of treatment decisions regarding impacted maxillary canines made by three AI-driven chatbots in comparison with experienced orthodontists, and to assess the overall quality of the chatbot responses. Although previous studies have evaluated the diagnostic and treatment planning abilities of chatbots in other areas of dentistry,^14,16,26,40 to the best of our knowledge, this is the first study to directly compare AI language models with orthodontists in managing impacted canines. Gemini showed greater disagreement with the orthodontists’ decisions, often favoring eruption when extraction was indicated, whereas ChatGPT and Grok were more aligned with the orthodontists’ clinical consensus. Among the three chatbots, Grok generally produced the highest-rated responses for both case-based scenarios and general queries. In contrast, Gemini scored significantly lower in both areas despite being a search engine-based model that leverages real-time Internet access to incorporate up-to-date information and offers exploration of topics from multiple angles.^11,27

Differences in network architectures and training data cause LLMs to yield unique responses to identical queries, resulting in distinct strengths, weaknesses, capabilities, and limitations.^21,41 ChatGPT utilizes a transformer architecture that employs a self-attention mechanism to learn the context of the dialogue and provide more accurate responses.¹⁹ Google’s Gemini Advanced is a multimodal model that processes text, images, audio, and code, thereby enhancing logical reasoning and coding.^12,42 Grok, developed by xAI, stands out for its integration with the X platform (formerly Twitter) and focuses on social interactions and trend analyses. Although its detailed architecture is not disclosed, the model is based on the transformer framework adopted by most modern LLMs.⁴²

Previous studies have focused on earlier models such as Google Bard (the predecessor of Google Gemini and Gemini Advanced)^{11,13,14,19,21,41} and various versions of ChatGPT (including ChatGPT-3.5^13,14,16,43 and ChatGPT-4^{12-14,18,21,23,25,27,29,40,41}). Consequently, limited data are available on the performance of the newer models used in this study, such as ChatGPT o3-mini and Supergrok model 3. Additionally, only a small number of studies have evaluated Gemini Advanced in dentistry.^12,44 As chatbots continue to evolve rapidly, caution is needed when comparing findings with earlier literature or interpreting results because the capabilities of these models can change substantially over time.

Previous studies have reported inconsistent findings regarding the accuracy of chatbots in dentistry. Some investigations found no difference between ChatGPT and Gemini (or its predecessor, Bard) in domains such as orthodontic questions,^11,21 peri-implantitis,⁴⁴ and dental trauma,¹⁸ a pattern that was also observed in this study for general queries on impacted maxillary canines. In contrast, other studies have indicated that ChatGPT outperforms Bard in areas such as orthodontics,¹⁹ periodontology,¹² pediatric dentistry,¹³ and evidence-based dentistry,⁴¹ with one study even excluding Bard because it could not answer clinical queries because of its text-based limitations,¹³ raising concerns regarding its clinical utility. However, several studies have also highlighted the limitations of ChatGPT, noting that it lacks clinical reasoning, at times producing redundant or plausible-sounding, yet incorrect information.^24,45 Contrary to the results of this study, some studies have shown that Gemini outperforms ChatGPT in answering questions regarding orthognathic surgery²⁷ and dental implantology,⁴³ particularly because of its ability to include references and multimedia content.

Few studies have directly compared the diagnostic capabilities of chatbots or their ability to accurately respond to complex real-life clinical cases, which was the primary focus of the present study. In dental implantology, although ChatGPT has been shown to perform better than Gemini in handling complex clinical scenarios, the difference between the two was not significant.⁴⁰ Similarly, in one study, ChatGPT accurately staged and graded periodontitis on the basis of detailed case descriptions in over half of the cases.¹⁶ In the only other study involving orthodontic clinical scenarios related to interceptive treatment, ChatGPT’s responses were rated as entirely correct in 46% of the cases.²⁶ Moreover, in pulpal and periradicular diagnosis, ChatGPT demonstrated the highest accuracy and Bard the lowest,¹⁴ which coincides with the results of this study, where ChatGPT received higher ratings than Gemini in case-based clinical scenarios.

To our knowledge, this is the first study to evaluate Grok in the field of dental research. Originally designed for trend analysis and real-time interactions on social media, concerns exist regarding Grok’s suitability for more generalized or technical clinical tasks.⁴² Nonetheless, Grok demonstrated the best overall performance in this study by outperforming Gemini in case-based scenarios, and both ChatGPT and Gemini in general queries regarding impacted maxillary canines. Recent studies in the medical field have reported mixed results regarding Grok’s capabilities, with one study showing that it outperformed ChatGPT in generating novel systematic review ideas in aesthetic plastic surgery,⁴⁶ but another reporting that it showed lower performance than ChatGPT-4 in responding to clinical guideline-based queries related to implant-based breast reconstruction.⁴⁷ Nevertheless, the findings of this study suggest that Grok may have the potential to support clinical decision-making in dentistry.

Although chatbots are not substitutes for clinical expertise, they may serve as valuable tools for orthodontic treatment planning. In this study, ChatGPT and Grok demonstrated potential for supporting clinical decision-making, although their outputs must be interpreted with caution because of the risk of AI-generated misinformation. In addition, chatbots currently lack the ability to autonomously interpret complex radiographic data, such as 3D CBCT imaging scans or multifactorial clinical variables used in diagnosis, treatment planning, and decision-making.⁴⁴ Therefore, they should be considered adjuncts rather than replacements for clinical judgment.¹⁴ Moreover, the integration of AI into clinical practice raises important ethical considerations, particularly regarding data privacy and patient confidentiality.²⁹ To address this, since real case-based scenarios were used in this study, meticulous care was taken to exclude any confidential patient information when presenting the cases to the chatbots, and written consent was obtained from the patients for the use of their clinical and radiographic data.

One of the main strengths of this study was its focus on applying AI to treatment decision-making for impacted maxillary canines, which is characterized by diagnostic complexity and subjectivity. Furthermore, the use of real case scenarios based on actual patients rather than fictional cases enhanced the clinical relevance and applicability of the findings. Nonetheless, this study had several limitations that warrant consideration. This study was conducted in English, which may restrict the generalizability of its findings to non-English-speaking populations and overlook valuable insights into the performance of AI tools in other languages. The findings of this study are based on the chatbot versions available as of April 2025. Because these systems are periodically updated, and their datasets may change, their performance may differ at future time points. Additionally, the case details were provided as text-based clinical scenarios rather than radiographic images, potentially limiting the interpretation of clinical complexity. Furthermore, while efforts were made to standardize the input, responses were collected only once without follow-up prompts, which may have limited the accuracy of the responses. Finally, although the orthodontist raters had similar levels of experience and were blinded to the chatbot identities, the limited number of evaluators may have introduced subjectivity despite their substantial agreement, which is consistent with findings in the literature.^48-50

Future studies should explore the integration of multimodal data, such as radiographs and CBCT scans, into AI models to enhance their diagnostic accuracy. Although chatbots have been explored in general dental contexts, more studies are required to assess their performance in scenarios requiring clinical judgment because of their multifactorial nature. Repeating queries over time and expanding case diversity could also help assess the consistency of chatbot performance, further enhancing their generalizability.

CONCLUSIONS

This study showed moderate agreement between the AI chatbots and orthodontists in treatment decisions for impacted maxillary canines, with Grok achieving the highest scores and Gemini the lowest. The recent rise of generative AI chatbots capable of producing seemingly evidence-based responses highlights their potential to serve as “chairside consultants” for dental professionals. However, current chatbots should be viewed strictly as supportive tools rather than replacements for clinical judgment, since their use requires careful oversight to avoid the risks associated with inaccurate or misleading recommendations.

SUPPLEMENTARY MATERIAL

Supplementary data is available at https://doi.org/10.4041/kjod25.174