Evaluating the performance of AI chatbots in responding to dental implant FAQs: A comparative study

Abstract

Background

This study aims to evaluate and compare the performance of five publicly accessible large-language-model (LLM) based chatbots—ChatGPT-o1, Deepseek-R1, Google-Gemini-Advanced, Claude-3.5-Sonnet, and Perplexity-Pro—in providing responses to frequently asked questions (FAQs) about dental implant treatment. The primary goal was to assess the accuracy, completeness, clarity, relevance, and consistency of chatbot-generated answers.

Methods

A total of 45 FAQs commonly encountered in clinical practice and online patient forums regarding dental implants were selected and categorized into nine thematic domains. Each question was submitted to the chatbots individually using a standardized protocol. Responses were independently assessed by a panel of four dental experts and one layperson using a 5-point Likert-scale. Python with Google-Colab was used for statistical analysis.

Results

ChatGPT-o1 achieved the highest overall performance, particularly in relevance (M = 4.99), consistency (M = 4.97), and accuracy (M = 4.96). Deepseek-R1 followed closely, with strong scores in completeness and relevance. Claude-3.5-Sonnet ranked moderately, while Gemini-Advanced and Perplexity-Pro showed lower performance in completeness and clarity. Significant differences were observed among chatbots across all criteria (p < 0.001). Inter-rater reliability was high (α = 0.87), confirming consistency among evaluators.

Conclusions

AI-driven chatbots demonstrated strong potential in delivering accurate and patient-friendly information about dental implant treatment. However, performance varied considerably across platforms, with ChatGPT-o1 and Deepseek-R1 showing the highest reliability. These findings highlight the emerging role of AI chatbots as supplementary tools in dental education and patient communication, while also underscoring the need for continued validation and ethical oversight in clinical applications.

Background

Artificial intelligence (AI) chatbots, powered by advanced Large Language Models (LLMs), utilize natural language processing (NLP) to engage in human-like conversations [1–4]. These AI tools leverage deep learning techniques, such as artificial neural networks, to process and learn from vast amounts of language data across various sources. Extensive training has substantially enhanced their ability to comprehend and generate coherent, realistic responses, with the later version demonstrating superior performance compared to its predecessors across multiple domains of knowledge [1, 5–7]. These advancements have enabled wide-ranging applications, including customer support, education, healthcare, and technical troubleshooting, making AI chatbots indispensable tools in numerous domains [8, 9].

AI chatbots are increasingly employed in healthcare to provide information, triage patient concerns, and guide individuals through common medical and dental procedures [10, 11]. However, their effectiveness depends on factors such as the complexity of the topic, the design of the underlying algorithms, and the quality of the training data used to enhance their responses [12].

In recent years, patients have increasingly used online community forums, websites, and social media platforms such as YouTube, Facebook, Instagram, and Twitter, as well as AI chatbots, to obtain information and answer questions about their dental treatments [13]. These chatbots have emerged as a new alternative informational resource, fundamentally reshaping the mechanisms through which online information is accessed [5]. This challenge is further compounded by the democratization of knowledge, where advancements in technology enable patients to independently access and educate themselves about medical information, reducing sole reliance on healthcare professionals [14].

Dental implant treatment, a subject of growing interest among patients exploring tooth replacement options, encompasses various aspects, including procedural details, post-operative care, and long-term maintenance. Delivering accurate and comprehensive answers to questions about dental implants is essential for effective patient education and informed decision-making [15, 16].

Moreover, in addition to the presence of incorrect information on the web that can exacerbate dental implant treatment hesitancy, the overwhelming volume of conflicting data can lead to confusion and misinformed decisions [17–19]. Patients may struggle to discern credible sources from unreliable ones, further complicating their ability to make informed choices about their care [20, 21]. This underscores the critical role of AI chatbots and other digital tools in providing trustworthy information. When properly designed and trained, these tools can help bridge the gap between professional guidance and patient self-education, fostering greater confidence in dental implant treatments and other healthcare decisions [22]. But it remains unknown [23].

Recent advancements in generative artificial intelligence have prompted an increasing number of studies evaluating chatbot performance in various domains of dentistry, beyond patient-facing FAQs. For instance, Chau et al. investigated the capability of AI chatbots in answering multiple-choice questions within prosthodontics and restorative dentistry, highlighting notable variations in accuracy and clinical relevance among different platforms [24]. Similarly, Chau et al. also explored the efficacy of generative AI models in dental licensing examinations, reporting that while these models could answer factual questions adequately, their performance declined in scenarios requiring clinical reasoning [25]. Brozović et al. assessed Bing Chat’s responses to dental exams and patients’ common questions, revealing critical limitations in depth and guideline adherence, thus emphasizing the need for content validation before clinical application [26].

Beyond assessment-based evaluations, AI chatbots have also been examined in dental education contexts. Danesh et al. evaluated ChatGPT’s performance on periodontic in-service examinations, reporting a moderate success rate, albeit with limitations in providing evidence-based rationale [27]. Additionally, Fang et al. examined chatbot integration into clinical learning environments, demonstrating that while AI-driven interactions enhanced student engagement and preliminary knowledge, they still lacked the depth and interactivity of human-facilitated education [28]. Collectively, these findings underline the dual potential and current limitations of AI chatbots in dentistry, not only as tools for patient communication but also as adjuncts in professional education and standardized assessments. However, studies focusing on their role in patient-directed implantology information, especially in a comparative multi-platform design, remain scarce. Therefore, this study aims to address this gap by systematically evaluating chatbot-generated responses to frequently asked questions about dental implants, across multiple advanced LLMs, with both expert and layperson assessments.

Recognizing the critical need for accurate information about dental implant treatment, this study aimed to evaluate the accuracy, completeness, clarity, relevance, and consistency of chatbots’ responses to questions about implant surgery and prosthodontics. The objectives were to: (i) evaluate the ability of well-known chatbots to deliver relevant and accurate information about dental implant treatment; (ii) identify significant differences in performance among various chatbots; and (iii) examine whether the use of AI-driven chatbots can enhance health literacy and mitigate hesitancy toward dental implant treatment The research hypothesis posited that significant performance variability would exist among chatbots across key response quality dimensions, specifically accuracy, completeness, and clarity, due to differences in their language models and training datasets. While previous studies have evaluated AI chatbots in implantology [29–32], to our knowledge, this study is among the first to comprehensively compare multiple new chatbot versions using both expert and layperson assessments across various performance dimensions.

Methods

Chatbot selection and study period

Five well-known large language model (LLM)-based chatbots were selected for evaluation: ChatGPT o1 (OpenAI, San Francisco, CA, USA), Deepseek R1 (DeepSeek Inc., Zhejiang, China), Google Gemini Advanced (Google LLC, Mountain View, CA, USA), Claude 3.5 Sonnet (Anthropic Inc., San Francisco, CA, USA), and Perplexity Pro (Perplexity AI, Cork, Ireland). All chatbot versions were accessed and tested as of February 1, 2025 (Fig. 1). The selection was based on their widespread popularity, unrestricted public accessibility, and their growing relevance in healthcare-related AI applications, as evidenced by recent literature up to January 2025. These models represent a cross-section of AI chatbots commonly used in patient information-seeking contexts.

Fig. 1.

Graphical abstract summarizing the study

Question design and categorization

The 45 frequently asked questions (FAQs) were curated through a combination of content analysis of patient inquiries in clinical settings, online forums, and social media platforms (such as Reddit, YouTube comments, and dental patient groups). An initial pool of over 100 questions was compiled and subsequently refined through an expert consensus process involving two prosthodontists and two periodontists. The final set of 45 questions was selected based on the consensus agreement among these specialists, ensuring clinical relevance, diversity of topics, and alignment with common patient concerns.

Questions were categorized into nine thematic domains: [1] procedure overview [2], eligibility and patient factors [3], safety and risks [4], cost and insurance [5], appearance and functionality [6], care and maintenance [7], longevity and effectiveness [8], post-operative considerations, and [9] comparison to other treatment options (Table 1).

Table 1.

Frequently asked questions (FAQs) based on common patient inquiries about dental implant treatment

• Procedure Overview: Focused on the dental implant process and its general attributes.

1. What is a dental implant?

2. How does a dental implant work?

3. Are dental implants permanent?

4. Do dental implants require surgery?

5. Can dental implants replace multiple teeth?

6. Can dental implants be placed in the front teeth?

7. Are dental implants a one-day procedure?

8. Can dental implants be placed on the same day as a tooth extraction?

9. Are dental implants a common procedure?

• Eligibility and Patient Factors: Addressed qualifications for implants and relevant health conditions.

10. Can anyone get dental implants?

11. Do dental implants work for older adults?

12. Can my child get dental implants?

13. Is bone grafting always required for dental implants?

14. Can smokers get dental implants?

15. Can dental implants be done if there is insufficient bone?

16. Can dental implants be placed in patients with diabetes?

17. Can pregnant women get dental implants?

18. Can you get a dental implant with high blood pressure?

19. Can allergies to metals affect eligibility for dental implants?

20. Are dental implants the best option for everyone?

• Safety and Risks: Examined implant safety and potential complications.

21. Are dental implants safe?

22. Can dental implants fail?

23. Are dental implants affected by gum disease?

24. Do dental implants increase the risk of cavities?

25. Do dental implants cause headaches?

• Cost and Insurance: Explored financial considerations and coverage options.

26. Does insurance cover dental implants?

27. Is dental implant treatment expensive?

28. Are dental implants more expensive than other tooth replacement options?

• Appearance and Functionality: Highlighted aesthetic and functional outcomes of dental implants.

29. Do dental implants look like natural teeth?

30. Are dental implants noticeable when speaking or smiling?

31. Do dental implants improve chewing ability?

• Care and Maintenance: Covered post-procedure upkeep and required hygiene practices.

32. Do dental implants need special care?

33. Do dental implants require a special toothbrush?

34. Are there any dietary restrictions after getting dental implants?

35. Can dental implants be removed?

36. Do dental implants prevent facial sagging?

• Longevity and Effectiveness: Assessed the durability and overall success rate of implants.

37. Do dental implants last a lifetime?

38. Do dental implants require replacement?

39. Do dental implants help prevent bone loss?

40. Do dental implants fuse with the bone?

• Post-operative Considerations: Discussed recovery, follow-up care, and aftercare guidelines.

41. Are dental implants painful?

42. Do dental implants require a hospital stay?

43. Can you eat normally after dental implant surgery?

• Comparison to other Solutions: Contrasted implants with alternatives, such as dentures or bridges.

44. Are dental implants better than dentures?

45. Can dental implants be used to support dentures?

To ensure relevance and clarity, questions were excluded if they were: (i) vague or ambiguous without clinical context, (ii) subjective or opinion-based, (iii) not phrased in grammatically correct English, or (iv) redundant with overlapping content. This process ensured the final set of 45 FAQs comprised clear, objective, and clinically relevant patient inquiries. To minimize bias, browser data was cleared, and the “New Chat” feature was used for each input session with distinct user logins for each chatbot. Each chatbot was prompted with the same set of questions across all categories. Questions were entered sequentially but no follow-ups were allowed into the chatbot’s user interface.

Evaluation framework

Responses were blindly and independently evaluated by a panel of five raters: two prosthodontists (M.T., N.B.), two periodontists (C.A.A., K.A.), and one layperson (E.B.). Each response was scored independently using a 5-point Likert scale based on five criteria:

• Accuracy: Alignment with evidence-based clinical knowledge.

• Completeness: Inclusion of all relevant components of the answer.

• Clarity: Ease of understanding for a general audience.

• Relevance: Direct response to the question posed.

• Consistency: Reproducibility and stability of answers to repeated queries.

In addition to their clinical experience, the expert raters consulted reputable, evidence-based scientific sources—including the PUBMED, the ITI Treatment Guides [33], consensus statements from the International Team for Implantology (ITI) [34–38], guidelines from the European Federation of Periodontology (EFP) [39], and the American Dental Association (ADA) [40] to ensure objectivity and consistency throughout the evaluation process. The layperson rater was included to reflect the patient perspective and assess the clarity and accessibility of chatbot responses from a non-expert viewpoint, helping to simulate real-world understanding by typical end users.

Data collection and scoring

A total of 45 unique responses were generated per chatbot, with each of these responses independently evaluated by a panel of five raters (5 raters × 45 responses = 225 evaluations per chatbot). Responses were scored on a scale from 1 (Poor) to 5 (Excellent), where the scale categories were defined as [1] Poor [2], Fair [3], Good [4], Very Good, and [5] Excellent. Average scores per criterion and chatbot were calculated and converted into percentages. Final scores were organized in Microsoft Excel (version 16.0, Microsoft Corp., Redmond, WA).

Statistical analysis

Descriptive statistics were calculated for each chatbot and criterion. Inter-rater reliability was assessed using Cronbach’s alpha. One-way analysis of variance (ANOVA) was conducted to determine significant differences in chatbot performance across criteria, followed by Tukey’s HSD post-hoc tests. Correlation between evaluation domains (e.g., accuracy vs. completeness) was assessed using Spearman correlation coefficients. Statistical analysis was performed using Python (Google Colab environment), employing appropriate libraries and packages. Assumptions of normality and homogeneity of variances were tested.

Results

Overview

This study analyzed the performance of five advanced AI chatbots—ChatGPT o1, Deepseek R1, Claude 3.5 Sonnet, Google Gemini Advanced, and Perplexity Pro—in responding to 45 frequently asked questions (FAQs) related to dental implants. Responses were evaluated across five core criteria: accuracy, completeness, clarity, relevance, and consistency, using a structured 5-point Likert scale. A total of 5625 individual ratings were generated by a panel comprising four dental specialists and one layperson.

Descriptive statistics

Descriptive performance of chatbots

Across all evaluation criteria—accuracy, completeness, clarity, relevance, and consistency—performance varied significantly among the five AI chatbots. The scores, based on the criteria provided by the evaluators, were summarized using the mean, standard error of the mean, median, minimum, maximum, standard deviation, variance, and count (Table 2). The evaluation results revealed notable differences in chatbot performance across various assessment criteria. Based on the evaluators’ scores, ChatGPT o1 demonstrated the highest overall performance among the chatbots, with a mean score of 4.90, the lowest standard error (0.007), and minimal variability (SD = 0.243). It consistently achieved top scores across all evaluation criteria, particularly in Relevance (M = 4.99) and Consistency (M = 4.97), reflecting its ability to generate highly relevant and coherent responses. Deepseek R1 also performed strongly (M = 4.87), with solid scores in Accuracy, Completeness, and Consistency, and relatively low variance. On the other hand, Claude 3.5 Sonnet and Perplexity received moderate scores overall, with notable variability, especially in Completeness and Consistency, where standard deviations exceeded 0.4. Google Gemini Advanced scored the lowest across the board (M = 4.38), particularly underperforming in Completeness (M = 3.87) and showing the widest range of responses (Min = 2.5, Max = 5). These results indicate significant performance differences among the chatbots, with ChatGPT o1 emerging as the most reliable across key quality dimensions. (Fig. 2).

Table 2.

Descriptive statistics for the average scores of answers provided by ChatGPT-01, Claude 3.5 sonnet, deepseek R1, Google gemini advanced, and perplexity pro, based on the evaluated criteria: accuracy, clarity, completeness, consistency, and relevance

Chatbot	Criterion	Mean	Standard Error of Mean	Median	Minimum	Maximum	Standard Deviation	Variance	Count
Chatgpt o1	Accuracy	4.96	0.010	5	4	5	0.148	0.022	225
	Clarity	4.87	0.017	5	4	5	0.255	0.065	225
	Completeness	4.72	0.025	5	3.5	5	0.371	0.137	225
	Consistency	4.97	0.009	5	4.25	5	0.130	0.017	225
	Relevance	4.99	0.005	5	4.5	5	0.076	0.006	225
	Total	4.90	0.007	5	3.5	5	0.243	0.059	1125
Claude 3.5 Sonnet	Accuracy	4.84	0.020	5	4	5	0.300	0.090	225
	Clarity	4.64	0.026	4.5	4	5	0.386	0.149	225
	Completeness	4.15	0.026	4	3	5	0.388	0.151	225
	Consistency	4.68	0.027	5	3	5	0.409	0.167	225
	Relevance	4.72	0.028	5	3	5	0.419	0.176	225
	Total	4.61	0.013	5	3	5	0.450	0.202	1125
Deepseek R1	Accuracy	4.95	0.012	5	4	5	0.175	0.031	225
	Clarity	4.70	0.021	4.5	4	5	0.317	0.100	225
	Completeness	4.85	0.020	5	4	5	0.296	0.087	225
	Consistency	4.90	0.015	5	4	5	0.223	0.050	225
	Relevance	4.95	0.012	5	4	5	0.182	0.033	225
	Total	4.87	0.008	5	4	5	0.261	0.068	1125
Google Gemini Advanced	Accuracy	4.45	0.023	4.5	3.75	5	0.350	0.123	225
	Clarity	4.33	0.024	4.5	3.5	5	0.360	0.130	225
	Completeness	3.87	0.026	4	2.5	5	0.397	0.158	225
	Consistency	4.48	0.026	4.5	3.5	5	0.385	0.148	225
	Relevance	4.77	0.023	5	3.5	5	0.345	0.119	225
	Total	4.38	0.014	4.5	2.5	5	0.470	0.221	1125
Perplexity Pro	Accuracy	4.79	0.025	5	4	5	0.378	0.143	225
	Clarity	4.46	0.029	4.5	3	5	0.435	0.189	225
	Completeness	4.15	0.030	4	3	5	0.447	0.200	225
	Consistency	4.54	0.032	4.5	3	5	0.484	0.234	225
	Relevance	4.64	0.032	5	3	5	0.473	0.224	225
	Total	4.51	0.015	4.5	3	5	0.493	0.243	1125

Fig. 2.

Mean evaluation scores by chatbot and criterion

In addition to mean and standard error values, 95% confidence intervals (CIs) were calculated for each chatbot’s performance scores. For example, ChatGPT o1 demonstrated a mean accuracy score of 4.96 (95% CI: 4.940–4.980), while its relevance score reached 4.99 (95% CI: 4.980–5.000). Across all chatbots and evaluation criteria, the CIs were found to be narrow, indicating high precision in the observed mean values.

Reliability assessment

Inter-rater reliability

Evaluation scores demonstrated high reliability among the five raters. Cronbach’s α = 0.87, indicating excellent internal consistency. This suggests that the evaluation framework and scoring rubric were well understood and applied consistently across raters, regardless of professional or lay status.

Comparative analysis

One-way ANOVA: inter-chatbot comparison

One-way analysis of variance (ANOVA) was conducted separately for each evaluation criterion to determine whether there were statistically significant differences in chatbot performance (Table 3; Fig. 3). The results showed significant main effects of chatbot type on all five criteria: accuracy (F(4,1120) = 115.58, p < 0.001), completeness (F(4,1120) = 266.56, p < 0.001), clarity (F(4,1120) = 79.67, p < 0.001), relevance (F(4,1120) = 45.26, p < 0.001), and consistency (F(4,1120) = 82.38, p < 0.001).

Table 3.

Table provides a clear overview of the statistical significance of chatbot type on each evaluation criterion

Fig. 3.

Bar chart visualizing the ANOVA results for each evaluation criterion (accuracy, completeness, clarity, relevance, and consistency). The chart displays the F-statistic values, all of which exceeded the critical F-value of 2.38 (p < 0.05), indicating statistically significant effects of chatbot type on all criteria

Post-hoc analysis: Tukey HSD comparisons

Tukey’s Honest Significant Difference (HSD) test was used to determine which chatbot pairs had significant differences in performance across the evaluation criteria (Fig. 4). The results showed that ChatGPT o1 performed significantly better than Claude 3.5 Sonnet, Google Gemini Advanced, and Perplexity Pro on all five criteria (p < 0.001). In terms of accuracy and relevance, ChatGPT o1 and Deepseek R1 had statistically similar performance (p > 0.05), indicating they were both top-tier models in these areas. However, Deepseek R1 significantly outperformed Claude, Gemini, and Perplexity in completeness, consistency, and clarity (p < 0.001). Claude 3.5 Sonnet ranked in the mid-tier, performing significantly better than Gemini and Perplexity (p < 0.05) but falling short compared to ChatGPT and Deepseek. Among all models, Google Gemini Advanced and Perplexity Pro had the lowest performance, with significantly lower scores in clarity, completeness, and consistency than the other chatbots.

Fig. 4.

Tukey’s Honest Significant Difference (HSD) test

In the post-hoc comparisons, ChatGPT o1 achieved significantly higher scores than Deepseek R1 in clarity (p < 0.05), while no significant differences were found between the two models in completeness and consistency (p > 0.05). For completeness, both ChatGPT o1 and Deepseek R1 significantly outperformed Claude 3.5 Sonnet, Google Gemini Advanced, and Perplexity Pro (p < 0.001). Similarly, in consistency, ChatGPT o1 and Deepseek R1 scored significantly higher than all other chatbots (p < 0.001). These findings indicate that while ChatGPT o1 maintains a slight advantage in clarity, both top-performing models demonstrate comparable and superior performance in completeness and consistency compared to other evaluated platforms.

These findings highlight that ChatGPT o1 consistently exceeds user expectations across evaluative domains, followed closely by Deepseek R1. Performance of Claude 3.5 Sonnet, while functional, lacked depth and reliability in some responses. In contrast, Gemini and Perplexity struggled with maintaining response quality, particularly in technical and nuanced queries.

Correlation analysis

Correlation among evaluation criteria

Spearman correlation coefficients were calculated to analyze the relationships between different evaluation criteria (Fig. 5; Table 4). The results showed moderate to strong positive associations between several dimensions. Accuracy and completeness had a correlation of 0.47, while completeness and consistency were correlated at 0.42. Clarity and consistency showed a correlation of 0.39, and accuracy and clarity had a weaker but still positive correlation of 0.30. Similarly, completeness and relevance were also moderately correlated at 0.30.

Fig. 5.

Correlation among evaluation criteria

Table 4.

Spearman correlation coefficients (r) and p-values for significant associations between evaluation criteria presented in Fig. 5

Criteria 1	Criteria 2	Spearman r	p-value
Accuracy	Completeness	0.47	<0.001
Completeness	Consistency	0.42	<0.001
Clarity	Consistency	0.39	<0.001
Accuracy	Clarity	0.30	<0.001
Completeness	Relevance	0.30	<0.001

These correlations suggest that while each evaluation domain captures a distinct aspect of response quality, high-performing chatbots tend to score well across multiple domains, indicating interdependence among linguistic fluency, factual completeness, and clarity of information.

Discussion

The results of this study provide sufficient statistical evidence to reject the null hypothesis, which stated that there are no significant differences in performance among the evaluated chatbots across the measured criteria. The one-way ANOVA revealed significant differences for all evaluation domains (p < 0.001), thereby supporting our research hypothesis that performance variability exists among chatbots due to differences in their underlying language models and training datasets [24, 31].

This study evaluated the performance of five advanced AI chatbots—ChatGPT o1, Deepseek R1, Claude 3.5 Sonnet, Gemini Advanced, and Perplexity Pro—in answering frequently asked questions (FAQs) related to dental implant treatment. The results revealed statistically significant differences in chatbot performance across all five evaluation domains: accuracy, completeness, clarity, relevance, and consistency. These findings underscore the growing potential of AI-driven chatbots in implantology as supplementary tools in dental education and patient engagement. When used appropriately, chatbots may help bridge the knowledge gap between clinicians and patients, offering timely and accurate responses to common concerns while reducing the burden on healthcare providers.

The findings of this study highlight important implications for the use of AI chatbots in responding to dental implant FAQs. ChatGPT o1’s superior performance, particularly in relevance and consistency, suggests that it may offer more reliable and user-centered responses compared to its counterparts [24, 30, 31]. However, the relatively lower scores observed in the completeness and clarity criteria among some chatbots point to ongoing challenges in generating thorough and easily understandable content in complex medical domains. These discrepancies underscore the need for continued refinement of language models to ensure not only factual accuracy but also the delivery of comprehensive and coherent information.

The observed correlations among evaluation criteria provide further insights into chatbot performance dynamics. The moderate positive correlation between accuracy and completeness (r = 0.47) indicates that chatbots capable of delivering factually correct responses are also more likely to provide comprehensive answers [7]. Similarly, the correlation between completeness and consistency (r = 0.42) suggests that detailed responses tend to be more stable and reproducible across repeated queries. These interrelationships highlight the interconnected nature of response quality dimensions, where improvements in one criterion often enhance others.

Domain-specific analysis revealed notable strengths and weaknesses among the evaluated chatbots. ChatGPT o1 exhibited high alignment between relevance and completeness, particularly excelling in procedural and post-operative care categories. Deepseek R1 demonstrated exceptional consistency in responses related to eligibility and patient factors, reflecting its robustness in addressing clinical variations. Similar domain-specific strengths of LLM-based chatbots have been reported in dental education contexts [7]. On the other hand, Claude 3.5 Sonnet, while performing adequately in general knowledge domains, struggled to provide detailed and contextually rich responses in cost and insurance-related inquiries, echoing limitations noted in prosthodontic applications [41]. Google Gemini Advanced and Perplexity Pro exhibited broader inconsistencies, with variable performance in safety and risk domains, often lacking depth and clarity in nuanced clinical scenarios [42].

These findings underscore the importance of evaluating chatbot responses not only through aggregate scores but also by examining performance within specific thematic contexts. Understanding such domain-specific strengths and limitations is crucial for guiding the development of more reliable AI-driven patient communication tools.

Additionally, the variability in scores across evaluators and questions indicates the subjective nature of human judgment, which should be considered when interpreting performance metrics. These findings align with earlier research indicating that LLM-based chatbots can generate high-quality, user-friendly information across various healthcare topics [1, 4, 7, 17]. Notably, ChatGPT o1 and Deepseek R1 achieved the highest ratings in most criteria, with ChatGPT o1 consistently leading in relevance and accuracy. These outcomes corroborate prior studies where OpenAI’s ChatGPT demonstrated superior performance in providing reliable responses to clinical inquiries [17, 43, 44]. In addition, other studies also show high reliability of AI chatbots in addressing clinical queries in other fields, such as vaccination misconceptions [1] and methotrexate use [5].

Interestingly, the strong clarity scores observed across all chatbots reflect ongoing advances in natural language generation and user experience optimization. This is consistent with prior work suggesting AI chatbots can present complex medical content in an understandable format for laypersons [4, 45, 46]. Furthermore, none of the chatbots delivered diagnostic or treatment-specific recommendations, instead appropriately emphasizing the necessity of consulting healthcare professionals—an important ethical safeguard [5, 47].

However, despite promising results, some chatbots produced incomplete or clinically superficial responses, especially for questions requiring detailed context. Similar limitations have been reported in domains such as prosthodontics [41], oral surgery [42], and scientific writing [47], reinforcing the current inability of LLMs to fully replace expert-guided communication in clinical practice. For instance, Freire et al. (2024) found ChatGPT’s accuracy in prosthodontics to be as low as 25.6%, and Dashti et al. (2023) reported its failure to generate verifiable references—limitations echoed in our study.

A study evaluated the accuracy and bias of ophthalmologist recommendations by ChatGPT 3.5, Bing Chat, and Google Bard across 20 U.S. cities, found high rates of inaccuracy (62.5%−73.8%), underrepresentation of female ophthalmologists (particularly by Bing Chat at 1.61% and Google Bard at 8.0%), and a disproportionate preference for ophthalmologists in academic medicine compared to the national average [48].

Some chatbots demonstrated incomplete responses or missed nuanced details. This limitation reflects findings from studies on chatbot responses in dental education, which revealed inconsistencies in accuracy and depth [7].

Additionally, the observation that paid versions (e.g., ChatGPT o1) outperformed free or less developed models raises concerns about health equity. As noted by Oca et al. (2023), unequal access to reliable AI-generated health information may exacerbate disparities in patient knowledge and decision-making [48, 49]. Thus, as AI tools grow in capability and popularity, ensuring equitable access and ethical use is paramount.

Although our evaluation design was informed by existing studies on the use of artificial intelligence (AI) in dentistry [50], certain limitations remain. The single-turn nature of chatbot interactions limited the realism of conversational flow. Additionally, our FAQ set, while comprehensive, may not fully reflect the diversity of patient queries encountered in real clinical settings.

The role of AI chatbots in healthcare extends beyond information dissemination to supporting patient decision-making and addressing health literacy gaps. For example, chatbots have successfully educated patients on vaccination and endodontics, similar to their performance in this study on dental implants [10]. However, the inability to provide personalized advice or navigate complex medical conditions remains a challenge.

The improved performance of paid chatbot versions, as noted in this study, raises equity concerns. Subscription-based access to advanced AI models, such as ChatGPT Plus or professional API versions of Deepseek, creates an informational divide where patients with financial means can access higher-quality, more accurate responses, while others relying on free versions may receive less reliable or incomplete information. This disparity poses a risk of exacerbating existing inequalities in health literacy and informed decision-making, particularly among socioeconomically disadvantaged populations [48]. This finding resonates with prior research on disparities in access to reliable online medical information and its implications for decision-making [17]. Moreover, ethical concerns regarding reliance on AI for critical health decisions without expert guidance remain pertinent [4].

Furthermore, beyond financial barriers, technological literacy represents an additional layer of inequality. Patients who lack digital proficiency may struggle to effectively use AI chatbots, misinterpret responses, or fail to navigate platform interfaces, limiting the potential benefits of these tools. Di Battista et al. (2023) emphasize that the efficacy of AI-driven patient communication tools is inherently linked to users’ ability to critically engage with digital health content [49].

Addressing these accessibility challenges requires a multifaceted approach, including the development of user-friendly chatbot interfaces, implementation of equitable access models (e.g., public healthcare-sponsored AI platforms), and proactive education initiatives aimed at improving digital health literacy across diverse patient demographics. Establishing transparent guidelines on AI usage in patient education and enforcing content quality standards will also be pivotal to mitigate health information disparities driven by technological advancement.

Future studies should explore dynamic multi-turn interactions, include broader patient populations, and evaluate chatbot integration into dental electronic health record (EHR) systems. Investigating the use of AI tools in shared decision-making and personalized treatment communication will also be essential. As emphasized by Danesh et al. [7] and Balel [42], AI’s greatest utility lies in its potential as an adjunct, not a substitute, for professional expertise [17, 42].

In terms of practical application, integrating chatbot technology into dental practice, especially through electronic health records or patient portals, could improve patient education and satisfaction. However, to ensure safe and equitable use, it is essential to optimize chatbot performance and establish regulatory frameworks that address content accuracy, data security, and ethical considerations.

Limitations

This study has several limitations that should be acknowledged. First, the single-turn interaction design, where each question was entered individually without allowing follow-up prompts or clarification, inherently restricts the conversational depth and realism of chatbot interactions. While this approach ensured methodological consistency across all chatbot sessions, it does not fully capture the dynamic and iterative nature of real-world patient inquiries. As a result, the chatbots’ ability to refine, contextualize, and elaborate on responses through multi-turn dialogues could not be assessed. Future research should explore multi-turn conversational simulations that allow chatbots to engage in extended exchanges, thereby providing a more comprehensive evaluation of their adaptive reasoning, context retention, and response refinement capabilities.

Second, the subjective nature of human judgment presents a potential source of assessment bias, as individual interpretations of criteria such as clarity, completeness, and relevance can vary—particularly in nuanced clinical scenarios. Although our study employed a structured 5-point Likert scale and achieved high inter-rater reliability (Cronbach’s α = 0.87), some degree of variability is inevitable in human evaluations. To further minimize subjectivity in future research, automated text analysis tools or natural language processing (NLP)-based scoring algorithms could be integrated to provide objective, reproducible measures of linguistic quality and factual accuracy. For example, sentiment analysis, readability indices (e.g., Flesch–Kincaid Grade Level), and domain-specific AI models trained on verified dental literature could complement human ratings. Additionally, hybrid evaluation frameworks combining expert judgment with AI-assisted content scoring software (such as Coh-Metrix, Grammarly analytics, or custom-trained BERT/ChatGPT models) may enhance consistency, reduce bias, and improve scalability of chatbot performance assessment. Additionally, the subjective nature of the assessment process, which relied on human judgment to rate chatbot responses, may have introduced some degree of variability despite the use of a structured evaluation rubric and high inter-rater reliability.

Third, while the study’s FAQ set was curated through an expert consensus process, it may not encompass the full spectrum of patient concerns encountered in diverse clinical settings. Broader and more heterogeneous question pools derived from large-scale patient data repositories or real-time consultation transcripts could enhance the generalizability of future findings.

Furthermore, the study only evaluated five publicly available chatbots as of February 2025. As the AI landscape evolves rapidly, the performance and capabilities of chatbots are expected to change with subsequent model updates and fine-tuning iterations. Therefore, periodic reassessment of chatbot performance will be necessary to ensure the continued relevance of such evaluations.

Lastly, the study highlighted potential ethical concerns, including unequal access to high-performing chatbot versions (e.g., paid vs. free models), which could exacerbate health information disparities. Additionally, reliance on AI-generated information without professional oversight raises risks of misinformation and patient misguidance. Furthermore, privacy protection and data security represent critical ethical considerations for AI applications in dental care. Although the chatbots in this study did not collect or store patient-identifiable information, real-world deployment in clinical environments often involves the exchange of sensitive personal health data. Improper handling, storage, or transmission of such data could violate regulations such as the General Data Protection Regulation (GDPR) in Europe or the Health Insurance Portability and Accountability Act (HIPAA) in the United States. Potential risks include unauthorized access, data breaches, and secondary use of patient information without consent. To mitigate these risks, developers and healthcare providers should implement strong encryption protocols, secure authentication systems, and transparent privacy policies. In addition, patients should be informed about how their data will be processed, stored, and potentially shared, ensuring that informed consent is obtained prior to any chatbot-mediated interaction involving personal health information. These ethical dimensions warrant the establishment of regulatory frameworks and best practice guidelines for AI-assisted patient communication in dentistry.

Future guidelines should also focus on transparency in chatbot response generation, ensuring users are aware of the limitations and appropriate contexts for AI-assisted information.

Ethical concerns and sources of bias in AI applications in dentistry

The application of AI in dentistry presents several ethical challenges that must be addressed to ensure safe, equitable, and effective patient care. Unequal access to high-performing AI models (e.g., paid vs. free versions) may exacerbate health information disparities, while reliance on AI-generated content without professional oversight poses risks of misinformation and patient misguidance. Privacy protection and data security are critical, especially when handling sensitive patient information, requiring compliance with regulations such as the GDPR in Europe and HIPAA in the United States.

Potential sources of bias include limitations in training datasets (e.g., underrepresentation of specific populations, languages, or clinical scenarios), algorithmic bias arising from model design, and performance disparities between AI systems due to proprietary fine-tuning. These biases can lead to uneven accuracy, inappropriate recommendations, or exclusion of minority groups. Mitigation strategies include the use of diverse and representative training datasets, continuous performance monitoring across demographic subgroups, transparency in algorithm development, and the integration of human oversight in AI-assisted decision-making.

Addressing these ethical and bias-related concerns through clear regulatory frameworks, standardized evaluation protocols, and professional guidelines will be essential for building trust and ensuring the responsible adoption of AI technologies in dental practice.

Future directions and challenges

Beyond patient communication, AI has significant potential to transform multiple aspects of dental practice. In tele-dentistry, AI-driven chatbots and virtual assistants could enhance remote consultations by triaging patient concerns, providing preliminary education, and supporting follow-up care, especially for patients in underserved or rural areas [10]. In advanced imaging analysis, AI algorithms integrated with cone-beam computed tomography (CBCT) and intraoral scanners can assist in detecting pathologies, planning implant placement, and monitoring treatment outcomes with high precision [11]. Robotic assistance in surgical procedures, guided by AI-driven planning systems, may improve accuracy, reduce operative time, and enhance patient safety [6]. However, these advancements also bring challenges, including the need for rigorous clinical validation, integration with existing dental workflows, high implementation costs, and regulatory approval processes. Ensuring interoperability across platforms, maintaining clinician oversight, and addressing ethical considerations—such as data privacy, algorithmic bias, and equitable access—will be essential for safe and effective adoption of AI technologies in dentistry.

Conclusion

This study demonstrated that advanced large language model-based chatbots, particularly ChatGPT o1 and Deepseek R1, can deliver accurate, relevant, and understandable answers to common patient questions on dental implants. However, significant performance differences exist across platforms, especially in completeness and consistency. While AI chatbots show promise as supplementary tools for patient education, they cannot replace professional consultation. Future efforts should focus on improving content completeness, integrating validated dental knowledge, and ensuring equitable and ethical access to AI-assisted communication.

Clinical significance

AI-driven chatbots have the potential to improve patient education and health literacy by providing timely, accessible, and understandable information on dental implants. When integrated into clinical workflows under professional supervision, they can help bridge the information gap between dentists and patients and support informed decision-making. Their application should remain supplementary, with careful attention to accuracy, completeness, and ethical considerations.

Authors’ contributions

M.T. and N.B. conceived and designed the project, M.T.; N.B. and E.B. acquired the data; M.T., C.A.A. and E.B. analyzed and interpreted the data; M.T. and K.A. wrote the paper.

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Data availability

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Declarations

Ethics approval and consent to participate

Since the study only involved a data audit of publicly available chatbots with no human or animal participants, no ethics committee approval, consent, or review was required.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.