Abstract
Purpose
This study aimed to evaluate and compare the responses of an artificial intelligence (AI)–powered chatbot and professional periodontists to patient queries in periodontology and implantology, using the Korean Academy of Periodontology’s (KAP) online question and answer (Q&A) section.
Methods
In this comparative cross-sectional study, we analyzed 219 patient-submitted periodontal and implant knowledge questions from the KAP online Q&A section. A panel of 10 evaluators—5 periodontists and 5 laypersons—rated both the periodontist’s and the AI chatbot’s responses using standardized scales. We applied the t-test and Spearman correlation coefficients to compare response quality, empathy, consistency, and evaluator preferences.
Results
Ten evaluators judged the AI chatbot’s responses to be significantly superior in quality and empathy compared to periodontist replies. A higher proportion of periodontist responses fell below acceptable quality (“very poor” or “poor”) than chatbot responses (28.7% vs. 15.0%; P<0.001), and more chatbot replies were rated “empathetic” or “very empathetic” (62.5% vs. 42.8%; P<0.001). Overall response consistency was deemed satisfactory at 64.2%, with no significant difference in consistency or preference between periodontist and lay evaluators.
Conclusions
AI-powered chatbots can deliver more accurate and empathetic answers than human periodontists, suggesting their potential role as consultation assistants merits further investigation. The high intraclass correlation coefficient values (0.79–0.93) indicate a high level of agreement among evaluators in both the periodontist and lay evaluator groups, thus confirming the reliability and robustness of the study’s assessment methodology.
Keywords: Artificial intelligence, Deep learning, Dentists, Natural language processing, Patients
Graphical Abstract
INTRODUCTION
Despite healthcare advances, many patients continue to encounter obstacles—such as high costs and limited infrastructure—to accessing services [1]. Simultaneously, escalating demand has intensified shortages of qualified medical and dental providers, which are among the most current and significant challenges facing healthcare providers and patients [2]. Periodontitis, in particular, remains a prevalent yet underdiagnosed chronic condition that leads to pain, impaired chewing, tooth loss, and reduced quality of life [3,4,5,6].
Over the past decade, artificial intelligence (AI) has advanced substantially, especially in computer vision applications for medical image analysis [7,8]. In dentistry, deep convolutional neural networks have matched or surpassed clinician performance in diagnosing oral conditions, including periodontal disease, dental caries, and oral cysts, and in detecting implant-related complications [9,10,11,12,13,14,15,16,17].
More recently, large language models (LLMs) have demonstrated the ability to generate accurate responses to diverse medical queries [18,19]. Platforms such as OpenAI’s ChatGPT (2022), Google’s PaLM, and Meta’s LLaMA are leading natural language processing (NLP) tools that are being evaluated to determine their ability to improve healthcare efficiency [20]. Nonetheless, validating the accuracy, reliability, and clinical suitability of LLM-generated medical information remains critical.
Since 2012, the Korean Academy of Periodontology (KAP) has operated a complementary online question and answer (Q&A) section, where patients anonymously post personalized questions about periodontal disease and public health issues, and board-certified periodontists provide expert answers. While fields outside dentistry have compared professional and AI chatbot responses, there remains a gap in periodontics research contrasting periodontist and chatbot replies [21,22]. Therefore, the aim of this study was to evaluate and compare the quality, empathy, consistency, and preference of responses provided by a professional periodontist and an AI-powered chatbot to patient questions on the KAP online Q&A section.
MATERIALS AND METHODS
Ethics
The study protocols and procedures were approved by the Institutional Review Board (IRB) of Jeonbuk National University Hospital (Approval No. CUH 2023-12-055), and a waiver of informed consent was obtained from the IRB as no personal information was collected. This comparative study was conducted according to the Strengthening the Reporting of Observational Studies in Epidemiology guideline [23].
KAP online Q&A section
This study retrospectively analyzed periodontal and implant knowledge questions submitted anonymously by users of the official KAP online Q&A section between May 2012 and June 2023, reflecting the KAP’s role as a recognized authority on patient questions about periodontal disease and related public health issues. All submissions were encrypted to protect patient confidentiality and were accessible only to designated KAP board members. A board-certified periodontist—KAP’s director of information technology and communications—provided a single expert response to each question, which was likewise encrypted and written in layperson’s terms. All answers underwent review by KAP board members to ensure quality and accuracy.
Patient questions
Of the 324 questions submitted, 219 were included in the study. The remaining 105 questions were excluded for the following reasons: 1) requests for dental clinic recommendations (62 questions), 2) requests for treatment costs (34 questions), and 3) the presence of detailed and specific personal information deemed inappropriate for the scope of the study (9 questions). The 219 questions were first classified by topic: periodontal disease (n=154; 70.3%), dental implants and related complications (n=18; 8.2%), esthetic concerns such as gingival recession and pigmentation (n=24; 11.0%), and mechanical prophylaxis devices (n=20; 9.1%). They were then grouped by nature: knowledge-based queries (n=126; 57.5%), treatment options (n=64; 29.2%), diagnosis (n=14; 6.4%), and other topics (n=15; 6.8%). Table 1 presents representative questions from each category, along with periodontist and chatbot responses.
Table 1. Sample questions and responses related to periodontal disease from the periodontist and the AI-powered chatbot.
| Questiona) | Periodontist responsea) | AI-powered chatbot responseb) | |
|---|---|---|---|
| I was told in a private clinic that I need gum grafting or gum regeneration due to localized severe gingival recession in the region of my lower front teeth. Please tell me about gum grafting or gum regeneration treatment. | To briefly summarize your question, it sounds like you're asking whether severe gingival recession requires regenerative or gingival grafting treatment. | When you mention your gums being “localized severe gingival recession,” it suggests significant gum issues, possibly advanced periodontal disease. This condition can lead to gum recession and loose teeth. | |
| 1. Purpose of gingival grafting: Primarily used to treat gingival recession, in cases with a lack of soft tissue or missing flesh in the gums. | 1. Regeneration capabilities: The human body has limited ability to regenerate gum tissue naturally. Once the gums have receded or been damaged to a certain extent, they typically do not grow back on their own. | ||
| 2. Conditions for transplantation: The success of gingival grafting depends on the recipient area’s ability to accept transplanted tissue. | 2. Treatment options for gum regeneration: | ||
| 3. Limitations with general recession: Grafting is challenging in cases of general gum recession, and is not often performed. | A. Gum grafting: This surgical procedure involves taking tissue from another part of your mouth (like the palate) and attaching it to the areas where the gums have receded. | ||
| 4. Diagnostic procedures: Determining the feasibility of grafting requires an X-ray and examination of the depth and thickness of the gums. | B. Guided tissue regeneration: This involves placing a special membrane between the bone and gum tissue, which encourages the growth of new bone and gum tissue. | ||
| 5. Consideration of systemic diseases: Additional tests might be necessary in cases with a history of systemic diseases. | 3. Managing loose teeth: If your teeth are loose, this might be due to the loss of supporting bone and gum tissue. Treatment could involve splinting (joining teeth together for stability), bone grafts, or even tooth extraction if the teeth are too loose to save. | ||
| 6. Recommendation for consultation: A second consultation, preferably at a University Hospital Periodontology Department, is recommended for a thorough evaluation. | 4. Prevent further damage: Good oral hygiene, regular dental visits, and possibly lifestyle changes (like quitting smoking if applicable) are crucial for preventing further damage to your gums. | ||
| 5. Early intervention: The earlier you address gum issues, the better the outcomes. Advanced gum disease can lead to tooth loss and requires more complex treatments. | |||
| 6. Consultation and personalized treatment: It’s vital to have a detailed consultation with a periodontist. They can assess your specific situation and suggest a personalized treatment plan, which might include surgical and non-surgical options. | |||
| Remember, each case is unique, and the effectiveness of treatment depends on various factors, including the extent of the disease, overall health, and adherence to prescribed treatment and oral hygiene practices. | |||
The patient’s questions and the periodontist’s responses have been translated into English, while the AI-powered chatbot’s responses are presented in their original English format.
AI: artificial intelligence.
a)Patient questions and periodontist answers were posted on the Korean Academy of Periodontology online question and answer section between May 2012 and June 2023.
b)An AI-powered chatbot (GPT-4, OpenAI, San Francisco, CA, USA) was used to generate AI-based responses.
AI-powered chatbot responses
GPT-4 (OpenAI, San Francisco, CA, USA) was used to generate AI-based responses from 1 August to 30 August 2023. The collection and management of patient questions and the responses from the periodontist and the AI-powered chatbot were handled by a single board-certified periodontist (Jae-Hong Lee), who was not involved in the evaluation process of this study. To ensure semantic consistency with the original text, a single periodontist (Jae-Hong Lee) meticulously translated the questions into English, while the AI-powered chatbot’s responses were translated from English to Korean with equal accuracy. Each question was posed in a separate chat session to prevent context carryover and ensure unbiased responses.
Evaluators
Five board-certified periodontists from Jeonbuk National University Dental Hospital and 5 university-educated lay evaluators—each without financial ties to AI technologies—independently reviewed and scored the Korean-language responses to the original patient questions. Due to characteristic stylistic differences, all 10 reviewers could correctly distinguish periodontist from chatbot answers, despite source blinding. Each evaluator provided written consent to maintain confidentiality. Inclusion and exclusion criteria for both rater groups are detailed in Table 2.
Table 2. Inclusion criteria for periodontist and lay evaluators.
| Periodontist evaluators | |
| 1. Must be a practicing periodontist affiliated with the Department of Periodontology. | |
| 2. Must have formal education and training in periodontology. | |
| 3. Must have no financial conflicts of interest related to the study. | |
| Lay evaluators | |
| 1. Must have been recommended by the principal investigator or co-investigators. | |
| 2. Must not have any professional background in health-related fields, including medicine and dentistry. | |
| 3. Must have a formal university education. | |
| 4. Must have attended a calibration session and received full training in the assessment guidelines prior to the study. | |
| 5. Must have no financial conflicts of interest related to the study. | |
Calibration
Before data collection, evaluators participated in a calibration session to align scoring criteria. The evaluators assessed the following aspects of the Q&A responses using Likert scales: quality (subjective and overall assessment of the effectiveness, clarity, and relevance of the responses, ranging from “very poor” to “very good” on a 5-point scale), empathy (ability to understand and share the feelings or emotional state, ranging from “not empathic” to “very empathic” on a 5-point scale), consistency (comparing the content of the 2 responses between the periodontist and the AI-powered chatbot, rating from “strongly disagree” to “strongly agree” on a 5-point scale), and the evaluator’s subjective preference between 2 responses (a binary choice between the periodontist’s and the chatbot’s response) (Table 3).
Table 3. Definitions of quality, empathy, consistency, and preference.
| Evaluating parameters | Scale | Definition |
|---|---|---|
| Quality | Very poor to very good on a 5-point scale | Quality refers to the overall effectiveness, clarity, and relevance of the responses, assessing how well the response addresses the question by providing accurate and informative content that meets the questioner’s needs. Good quality responses tend to be clear, detailed and directly relevant to the questions asked. |
| Empathy | Not empathic to very empathic on a 5-point scale | Empathy is defined as the ability to understand and share the feelings of the person asking the question. It involves recognizing the patient's emotional state, acknowledging their concerns, and responding in a way that reflects an understanding and sensitivity to their emotional state. Responses that are empathetic tend to be more personable, and may connect better with the individual, providing comfort or reassurance. |
| Consistency | Strongly disagree to Strongly agree on a 5-point scale | Consistency compares the content of 2 responses to a patient's question: one from a periodontist and one from an AI-powered chatbot. It reflects the reliability and predictability of the responses. High consistency ensures that the advice or information provided is not contradictory, and that the same standards are maintained across all interactions. |
| Preference | Binary choice between the periodontist and the AI-powered chatbot | This is an evaluation based purely on the personal preference of the evaluator. It involves choosing between 2 responses (e.g., from a human and an AI-powered chatbot), based on which one the evaluator personally finds more convincing, appropriate, or preferable. This measure is inherently subjective, and can vary significantly between different evaluators based on their personal biases, experiences, and expectations. |
AI: artificial intelligence.
Statistical analysis
Power analysis based on prior studies indicated that 219 question-response pairs would yield >80% power for detecting a 10% difference between periodontist and chatbot responses [21]. For the statistical analysis, the evaluator scores were set on a consistent scale. We used various statistical measures, including mean, median, standard deviation, and interquartile range (IQR), to compare the distribution of scores for each response obtained from both the periodontist and the AI-powered chatbot. The normality of the data and homogeneity of variances were assessed using the Shapiro–Wilk test and the Levene test, respectively. The χ2 test and the 2-tailed t-test were subsequently conducted to examine whether there were significant differences in the mean scores between the 2 groups of responses. In addition, Spearman correlation coefficients (r) were calculated to assess the relationships between response quality, empathy, preference, and consistency scores. Statistical significance was set at P<0.05. All statistical analyses were performed using statistical software (SAS version 9.4 and JMP version 17.1, both products of the SAS Institute, Cary, NC, USA).
RESULTS
Quality and empathy of responses
Evaluators rated the chatbot’s answers significantly higher in both quality and empathy than periodontist responses (Table 4). A larger proportion of periodontist replies fell below acceptable quality (very poor or poor) compared to the chatbot (periodontist: n=628 [28.7%] vs. chatbot: n=329 [15.0%]) (P<0.001). Similarly, chatbot responses were judged more empathetic, with a higher share rated “empathetic” or “very empathetic” (periodontists: n=930 [42.75%] vs. chatbot: n=1,370 [62.5%]; P<0.001) (Figure 1). Strong correlations between quality and empathy scores were observed for both periodontists (r=0.838; P<0.001) and the chatbot (r=0.811; P<0.001) (Figure 2).
Table 4. Quality and empathy of responses from periodontists and the AI-powered chatbot.
| Respondent | Periodontists | AI-powered chatbot | P value | |||||
|---|---|---|---|---|---|---|---|---|
| Periodontist evaluators | Lay evaluators | Total | Periodontist evaluators | Lay evaluators | Total | |||
| Quality | ||||||||
| Very good | 192 (17.5%) | 152 (13.9%) | 344 (15.7%) | 189 (17.3%) | 324 (29.6%) | 513 (23.4%) | 0.678a) | |
| Good | 194 (17.7%) | 422 (38.5%) | 616 (28.1%) | 288 (26.3%) | 598 (54.6%) | 886 (40.5%) | ||
| Acceptable | 263 (24.0%) | 339 (31.0%) | 602 (27.5%) | 345 (31.5%) | 117 (10.7%) | 462 (21.1%) | ||
| Poor | 349 (31.9%) | 150 (13.7%) | 499 (22.8%) | 265 (24.2%) | 50 (4.6%) | 315 (14.4%) | <0.001b) | |
| Very poor | 97 (8.9%) | 32 (2.9%) | 129 (5.9%) | 8 (0.7%) | 6 (0.5%) | 14 (0.6%) | ||
| Empathy | ||||||||
| Very empathic | 203 (18.5%) | 233 (21.3%) | 436 (19.9%) | 194 (17.7%) | 262 (23.9%) | 456 (20.8%) | <0.001a) | |
| Empathic | 138 (12.6%) | 356 (32.5%) | 494 (22.6%) | 311 (28.4%) | 603 (55.1%) | 914 (41.7%) | ||
| Acceptable | 266 (24.3%) | 303 (27.7%) | 569 (26.0%) | 289 (26.4%) | 163 (14.9%) | 452 (20.6%) | ||
| Slightly empathic | 413 (37.7%) | 144 (13.2%) | 557 (25.4%) | 294 (26.8%) | 56 (5.1%) | 350 (16.0%) | <0.001b) | |
| Not very empathic | 75 (6.8%) | 59 (5.4%) | 134 (6.1%) | 7 (0.6%) | 11 (1.0%) | 18 (0.8%) | ||
AI: artificial intelligence.
a)Result from the χ2 test after merging very good/very empathic/strongly agree and good/empathic/agree.
b)Result from the χ2 test after merging very poor/not very empathic/strongly disagree and poor.
Figure 1. Distribution of quality (A-C) and empathy (D-F) ratings for periodontist and AI-powered chatbot responses to patient questions.
AI: artificial intelligence.
Figure 2. Correlation of (A) quality and (B) empathy ratings for periodontist and AI-powered chatbot responses to patient questions.
AI: artificial intelligence.
Consistency and preference of responses
Overall consistency between periodontist and chatbot replies exceeded the acceptable threshold, at 64.2% (Table 5). Only 56.5% of the periodontist evaluators and 71.9% of the lay evaluators rated the response consistency as acceptable or better (Figure 3A). Ten evaluators preferred the chatbot’s responses (n=1,371 [62.6%]), and preference rates did not differ significantly between periodontists (n=692 [63.2%]) and lay evaluators (n=679 [62.0%]; P=0.501) (Figure 3B). A detailed breakdown by rater group is presented in Table 6.
Table 5. Consistency between periodontists and the artificial intelligence-powered chatbot.
| Respondent | Periodontist evaluators | Lay evaluators | Total | P value |
|---|---|---|---|---|
| Strongly agree | 197 (18.0%) | 50 (4.6%) | 247 (11.3%) | <0.001a) |
| Agree | 105 (9.6%) | 307 (28.0%) | 412 (18.8%) | |
| Acceptable | 317 (28.9%) | 430 (39.3%) | 747 (34.1%) | |
| Disagree | 459 (41.9%) | 236 (28.0%) | 695 (31.7%) | <0.001b) |
| Strongly disagree | 17 (1.6%) | 72 (6.6%) | 89 (4.1%) |
a)Result from the χ2 test after merging very good/very empathic/strongly agree and good/empathic/agree.
b)Result from the χ2 test after merging very poor/not very empathic/strongly disagree and poor.
Figure 3. (A) Preference and (B) consistency ratings for periodontist and artificial intelligence-powered chatbot responses to patient questions.
AI: artificial intelligence.
Table 6. Comparative analysis of quality, empathy, consistency, and preferences of the responses from periodontists and AI-powered chatbots, as rated by 2 groups of evaluators.
| Scale | Periodontist evaluators | Lay evaluators | P value | |
|---|---|---|---|---|
| Human quality | 3.0±1.2 | 3.5±1.0 | <0.001 | |
| 1: Very poor | ||||
| 5: Very good | ||||
| AI quality | 3.4±1.1 | 4.1±0.8 | <0.001 | |
| 1: Very poor | ||||
| 5: Very good | ||||
| Human empathy | 3.0±1.2 | 3.5±1.1 | <0.001 | |
| 1: Not empathic | ||||
| 5: Very empathic | ||||
| AI empathy | 3.4±1.1 | 4.0±0.8 | <0.001 | |
| 1: Not empathic | ||||
| 5: Very empathic | ||||
| Consistency | 3.0±1.1 | 3.0±1.0 | 0.888 | |
| 1: Strongly disagree | ||||
| 5: Strongly agree | ||||
| Preference | 1.6±0.5 | 1.6±0.5 | 0.537 | |
| 1: Periodontists | ||||
| 2: AI-powered chatbot | ||||
AI: artificial intelligence.
Lengths of responses
Patient questions averaged 72±62 words (median [IQR], 52 [46–62]). Periodontist answers averaged 61±37 words (52 [46–59]), whereas chatbot replies averaged 265±50 words (261 [249–274]). The chatbot’s responses were significantly longer (P<0.001), and there was no significant correlation between the 2 groups’ response lengths (r=0.108; P=0.108). Both periodontist and chatbot response lengths increased weakly but significantly with question length (periodontist: r=0.242, P<0.001; chatbot: r=0.264, P<0.001 [Spearman correlation coefficients]) (Figure 4).
Figure 4. Correlation between the mean length of patient questions and AI-powered chatbot and periodontist responses.
AI: artificial intelligence.
Inter-rater reliability
Using a 2-way random-effects intraclass correlation coefficient (ICC) model for absolute agreement, ratings for quality, empathy, and consistency showed robust reliability (ICC range: 0.79–0.93). Among periodontist evaluators, ICCs were 0.913 (human quality), 0.936 (AI quality), 0.915 (human empathy), 0.933 (AI empathy), and 0.926 (consistency). Lay evaluators yielded ICCs of 0.825, 0.861, 0.795, 0.855, and 0.829, respectively, indicating strong agreement, albeit slightly lower than that of the specialists.
DISCUSSION
AI-powered chatbots have emerged as valuable tools in healthcare, leveraging NLP and machine learning to simulate human conversation and deliver tailored health advice or specialist guidance based on user inputs [24]. They can predict diseases, manage patient information, and support emergency care, thereby reducing clinicians’ workload [25]. Additionally, chatbots aid patients in interpreting symptoms and identifying appropriate specialists, improving access to care—particularly during pandemics [24,25]. As AI advances, these systems are poised to play an increasingly critical role in personalized healthcare, from patient monitoring and diagnosis to treatment recommendations.
A recent internal medicine study using similar methods found that chatbot responses were significantly longer and rated higher in quality and empathy than physicians’ answers, aligning with the findings of this study [21]. In our study, 63.9% of chatbot responses were rated “good” or “very good” vs. 43.8% for periodontists, and empathy ratings were 62.6% vs. 42.5%, respectively. In the physician study, 78.5% of chatbot replies were rated “good” or “very good” and 45.1% as empathetic, compared with 22.1% and 4.6% for physicians [21]. Despite the differences in specialization, chatbots consistently outperformed experts on these metrics, suggesting their potential to complement clinicians and enhance patient-centered care.
Many people frequently access a wide range of medical information online with regard to both common and rare diseases. However, the potential inaccuracy and commercial bias of online sources, coupled with a lack of personalization, can result in misdiagnosis and unreliable information [26,27]. Despite these concerns, our data show that chatbot replies in periodontal contexts matched or exceeded expert responses in quality, empathy, and user preference. Moreover, 65.9% of evaluations rated consistency between periodontist and chatbot answers as acceptable or better, underscoring chatbots’ promise as consultation assistants.
Most previous comparisons of human experts and AI-powered chatbots have used publicly available datasets, which may overlap with the training data of LLMs, potentially biasing chatbot performance [21,28]. In contrast, the patient questions and periodontist answers in the KAP online Q&A are private and not publicly accessible, making it unlikely that these exchanges were included in the model’s training. This suggests that the chatbot’s responses in our study stem from independent inference rather than memorization of expert answers, and supports the suitability of the KAP dataset for an unbiased comparison.
While the AI-powered chatbot showed promising results in this study, significant limitations and potential biases challenge direct comparisons with human experts or claims of its superiority. First, the periodontists who contributed to the KAP Q&A did so as volunteers and may not represent the full spectrum of clinical practitioners; their written advice may not capture the nuance of in-person consultations [29]. Although their responses are subjective, the participants in this study, who were members of the KAP board of directors, provided the answers on behalf of KAP, adding significant credibility and reliability to their statements. Further, if the KAP board members had known that their responses would be subjected to comparative analysis, they may have provided more comprehensive responses. Nevertheless, using unmodified, unaware responses provides a more authentic benchmark for comparison.
Second, consistent with a previous study in which chatbot responses were significantly longer than physician responses (52 vs. 211 words, P<0.001), we found that the chatbot responses were longer than periodontist responses (61 vs. 265 words, P<0.001), and were perceived as superior in both quality and empathy [20]. We also observed a significant positive correlation between response length and perceived quality and empathy (P<0.001), indicating a potential rater bias, in which longer responses are perceived as more comprehensive and empathetic. The findings of this study should be interpreted cautiously in light of this possibility. In addition, the design of the quality and empathy scale in this study may be a limitation, as it appeared to be biased toward positive responses, potentially conflicting with the principle of balanced positive and negative response options on Likert scales.
Third, in periodontal consultations, the capacity for empathy—defined as the complex interplay of emotional and cognitive processes—differs markedly between human experts and AI-powered chatbots, highlighting AI’s limitations in replicating the depth of human empathy in clinical settings [30]. Periodontists responding in the KAP Q&A section often employed emoticons and special characters to mirror patients’ emotional states and underscore the seriousness of their concerns. This practice demonstrates active listening, genuine concern, and responsiveness to the emotional dimensions of patient care. In contrast, while AI-powered chatbots generated long, detailed, and professionally phrased replies, they lack inherent empathic qualities and thus cannot capture the nuanced understanding of human emotions that healthcare professionals provide [31].
Furthermore, since most of our dataset comprised Korean-language content, performance differences could arise because LLMs such as GPT-4 are predominantly trained on English-language data [32]. To mitigate this, patient queries originally written in Korean were translated into English and reviewed by a qualified periodontist before being used as prompts for GPT-4, ensuring that our performance assessment was not adversely affected by language-related limitations.
A notable limitation of LLM-based AI chatbots is hallucination, where factual medical or scientific information may be subtly distorted [22,33]. Ensuring the accuracy of specialized content—such as periodontics and dental implant information—is crucial to prevent misinformation when disseminated to the public. In this study, 5 periodontists reviewed 219 questionnaires and identified few or no instances of hallucinated or distorted dental information in the chatbot’s periodontal responses. This suggests that, as AI algorithms continue to improve, hallucination frequency and severity should decline. Moreover, advanced domain-specific LLMs such as Med-PaLM and MedPrompt, which are designed to enhance the quality and efficiency of medical consultations, are expected to outperform the model used here [34].
The findings of this study highlight the potential utility of AI-powered chatbots as adjuncts in periodontal consultations and their ability to disseminate relevant health information quickly and consistently. However, several significant caveats remain: chatbots may lack personalization, harbor potential inaccuracies, exhibit implicit bias or misinterpretation, and raise privacy and ethical concerns [34]. Consequently, although GPT-4 and other medical-specific LLMs can increase the efficiency of information delivery in periodontal practice, they must not be misconstrued as replacements for professional consultations. As novel AI chatbots emerge in dental healthcare, ongoing scrutiny and evaluation will be imperative to determine their true efficacy in enriching clinical practice and improving patient outcomes. Future research should replicate this study across diverse populations and settings to validate the broader applicability of our results.
Although subject to certain limitations and potential biases, our results indicate that the AI-powered chatbot generated higher-quality and more empathetic responses to patient questions in the KAP online Q&A section than did periodontists. The role of AI-powered chatbots as periodontist assistants merits further investigation, with a focus on their tangible benefits for both clinicians and patients.
ACKNOWLEDGEMENTS
We would like to express our gratitude to the board members of the Korean Academy of Periodontology (KAP) who participated in maintaining the KAP online question and answer (Q&A) section and providing answers from 2012 to 2023.
Footnotes
Funding: This research was supported by a grant of the Korea Health Technology R&D Project through the Patient-Doctor Shared Decisi Research Center (PDSDM), funded by the Ministry of Health & Welfare, Republic of Korea (grant number: RS-2023-KH142251).
Conflict of Interest: No potential conflict of interest relevant to this article was reported.
- Conceptualization: Jae-Hong Lee, Young-Taek Kim.
- Formal analysis: Jae-Hong Lee, So-Hae Oh, Young-Taek Kim, Falk Schwendicke, Akhilanand Chaurasia.
- Investigation: Jae-Hong Lee, So-Hae Oh, Young-Taek Kim, Falk Schwendicke, Akhilanand Chaurasia.
- Methodology: Jae-Hong Lee, So-Hae Oh, Young-Taek Kim, Falk Schwendicke, Akhilanand Chaurasia.
- Project administration: Jae-Hong Lee, So-Hae Oh, Young-Taek Kim.
- Writing - original draft: Jae-Hong Lee, So-Hae Oh, Young-Taek Kim, Falk Schwendicke, Akhilanand Chaurasia.
- Writing - review & editing: Jae-Hong Lee, So-Hae Oh, Young-Taek Kim, Falk Schwendicke, Akhilanand Chaurasia.
References
- 1.Dawkins B, Renwick C, Ensor T, Shinkins B, Jayne D, Meads D. What factors affect patients’ ability to access healthcare? An overview of systematic reviews. Trop Med Int Health. 2021;26:1177–1188. doi: 10.1111/tmi.13651. [DOI] [PubMed] [Google Scholar]
- 2.Meskó B, Hetényi G, Győrffy Z. Will artificial intelligence solve the human resource crisis in healthcare? BMC Health Serv Res. 2018;18:545. doi: 10.1186/s12913-018-3359-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Petersen PE, Ogawa H. The global burden of periodontal disease: towards integration with chronic disease prevention and control. Periodontol 2000. 2012;60:15–39. doi: 10.1111/j.1600-0757.2011.00425.x. [DOI] [PubMed] [Google Scholar]
- 4.Albandar JM. Aggressive and acute periodontal diseases. Periodontol 2000. 2014;65:7–12. doi: 10.1111/prd.12013. [DOI] [PubMed] [Google Scholar]
- 5.Lee JH, Choi JK, Jeong SN, Choi SH. Charlson comorbidity index as a predictor of periodontal disease in elderly participants. J Periodontal Implant Sci. 2018;48:92–102. doi: 10.5051/jpis.2018.48.2.92. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Lee JH, Jeong SN, Choi SH. Predictive data mining for diagnosing periodontal disease: the Korea National Health and Nutrition Examination Surveys (KNHANES V and VI) from 2010 to 2015. J Public Health Dent. 2019;79:44–52. doi: 10.1111/jphd.12293. [DOI] [PubMed] [Google Scholar]
- 7.Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA. 2016;316:2402–2410. doi: 10.1001/jama.2016.17216. [DOI] [PubMed] [Google Scholar]
- 8.Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017;542:115–118. doi: 10.1038/nature21056. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Lee JH, Kim DH, Jeong SN, Choi SH. Diagnosis and prediction of periodontally compromised teeth using a deep learning-based convolutional neural network algorithm. J Periodontal Implant Sci. 2018;48:114–123. doi: 10.5051/jpis.2018.48.2.114. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Lee JH, Kim DH, Jeong SN, Choi SH. Detection and diagnosis of dental caries using a deep learning-based convolutional neural network algorithm. J Dent. 2018;77:106–111. doi: 10.1016/j.jdent.2018.07.015. [DOI] [PubMed] [Google Scholar]
- 11.Lee JH, Kim DH, Jeong SN. Diagnosis of cystic lesions using panoramic and cone beam computed tomographic images based on deep learning neural network. Oral Dis. 2020;26:152–158. doi: 10.1111/odi.13223. [DOI] [PubMed] [Google Scholar]
- 12.Lee DW, Kim SY, Jeong SN, Lee JH. Artificial intelligence in fractured dental implant detection and classification: evaluation using dataset from two dental hospitals. Diagnostics (Basel) 2021;11:233. doi: 10.3390/diagnostics11020233. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Lee JH, Kim YT, Lee JB, Jeong SN. Deep learning improves implant classification by dental professionals: a multi-center evaluation of accuracy and efficiency. J Periodontal Implant Sci. 2022;52:220–229. doi: 10.5051/jpis.2104080204. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Park W, Schwendicke F, Krois J, Huh JK, Lee JH. Identification of dental implant systems using a large-scale multicenter data set. J Dent Res. 2023;102:727–733. doi: 10.1177/00220345231160750. [DOI] [PubMed] [Google Scholar]
- 15.Chaurasia A, Namachivayam A, Koca-Ünsal RB, Lee JH. Deep-learning performance in identifying and classifying dental implant systems from dental imaging: a systematic review and meta-analysis. J Periodontal Implant Sci. 2024;54:3–12. doi: 10.5051/jpis.2300160008. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Lee JH, Kim YT, Lee JB. Identification of dental implant systems from low-quality and distorted dental radiographs using AI trained on a large multi-center dataset. Sci Rep. 2024;14:12606. doi: 10.1038/s41598-024-63422-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Lee JH, Yun JH, Kim YT. Deep learning to assess bone quality from panoramic radiographs: the feasibility of clinical application through comparison with an implant surgeon and cone-beam computed tomography. J Periodontal Implant Sci. 2024;54:349–358. doi: 10.5051/jpis.2302880144. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Thirunavukarasu AJ, Ting DSJ, Elangovan K, Gutierrez L, Tan TF, Ting DSW. Large language models in medicine. Nat Med. 2023;29:1930–1940. doi: 10.1038/s41591-023-02448-8. [DOI] [PubMed] [Google Scholar]
- 19.Peng C, Yang X, Chen A, Smith KE, PourNejatian N, Costa AB, et al. A study of generative large language model for medical research and healthcare. NPJ Digit Med. 2023;6:210. doi: 10.1038/s41746-023-00958-w. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Cascella M, Montomoli J, Bellini V, Bignami E. Evaluating the feasibility of ChatGPT in healthcare: an analysis of multiple clinical and research scenarios. J Med Syst. 2023;47:33. doi: 10.1007/s10916-023-01925-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Ayers JW, Poliak A, Dredze M, Leas EC, Zhu Z, Kelley JB, et al. Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum. JAMA Intern Med. 2023;183:589–596. doi: 10.1001/jamainternmed.2023.1838. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Lee P, Bubeck S, Petro J. Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine. N Engl J Med. 2023;388:1233–1239. doi: 10.1056/NEJMsr2214184. [DOI] [PubMed] [Google Scholar]
- 23.von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP, et al. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. J Clin Epidemiol. 2008;61:344–349. doi: 10.1016/j.jclinepi.2007.11.008. [DOI] [PubMed] [Google Scholar]
- 24.Lee H, Kang J, Yeo J. Medical specialty recommendations by an artificial intelligence chatbot on a smartphone: development and deployment. J Med Internet Res. 2021;23:e27460. doi: 10.2196/27460. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Chakraborty S, Paul H, Ghatak S, Kumar Pandey S, Kumar A, Singh KU, et al. An AI-based medical chatbot model for infectious disease prediction. IEEE Access. 2022;10:128469–128483. [Google Scholar]
- 26.Clarke MA, Moore JL, Steege LM, Koopman RJ, Belden JL, Canfield SM, et al. Health information needs, sources, and barriers of primary care patients to achieve patient-centered care: A literature review. Health Informatics J. 2016;22:992–1016. doi: 10.1177/1460458215602939. [DOI] [PubMed] [Google Scholar]
- 27.Bovi AM Council on Ethical and Judicial Affairs of the American Medical Association. Use of health-related online sites. Am J Bioeth. 2003;3:W-IF3. doi: 10.1162/152651603322874780. [DOI] [PubMed] [Google Scholar]
- 28.Balel Y. Can ChatGPT be used in oral and maxillofacial surgery? J Stomatol Oral Maxillofac Surg. 2023;124:101471. doi: 10.1016/j.jormas.2023.101471. [DOI] [PubMed] [Google Scholar]
- 29.Davies L, LeClair KL, Bagley P, Blunt H, Hinton L, Ryan S, et al. Face-to-face compared with online collected accounts of health and illness experiences: a scoping review. Qual Health Res. 2020;30:2092–2102. doi: 10.1177/1049732320935835. [DOI] [PubMed] [Google Scholar]
- 30.Decety J, Meyer M. From emotion resonance to empathic understanding: a social developmental neuroscience account. Dev Psychopathol. 2008;20:1053–1080. doi: 10.1017/S0954579408000503. [DOI] [PubMed] [Google Scholar]
- 31.Powell J. Trust me, i’m a chatbot: how artificial intelligence in health care fails the turing test. J Med Internet Res. 2019;21:e16222. doi: 10.2196/16222. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Navigli R, Conia S, Ross B. Biases in large language models: origins, inventory, and discussion. J Data and Information Quality. 2023;15:1–21. [Google Scholar]
- 33.Gilbert S, Harvey H, Melvin T, Vollebregt E, Wicks P. Large language model AI chatbots require approval as medical devices. Nat Med. 2023;29:2396–2398. doi: 10.1038/s41591-023-02412-6. [DOI] [PubMed] [Google Scholar]
- 34.Singhal K, Azizi S, Tu T, Mahdavi SS, Wei J, Chung HW, et al. Large language models encode clinical knowledge. Nature. 2023;620:172–180. doi: 10.1038/s41586-023-06291-2. [DOI] [PMC free article] [PubMed] [Google Scholar]