A comparative analysis of embedded chatbot models and ChatGPT-4 for answering orthodontic treatment queries

Abstract

This study evaluates the effectiveness of an embedded chatbot model in delivering orthodontic treatment-related information and compares its performance with the fourth generation of the Chat Generative Pre-trained Transformer (ChatGPT-4, OpenAI, USA), focusing on accuracy, clarity, relevance, and up-to-date knowledge. An embedded chatbot model, designed specifically for orthodontic treatment-related queries, was integrated into a web-based platform. It was compared with ChatGPT-4 (OpenAI, USA) using standardized orthodontic questions. Responses from both models were evaluated by six orthodontic consultants (≥ 5 years of experience) using a 5-point Likert scale. Content validity was assessed using item-level (I-CVI) and scale-level (S-CVI) content validity indices. Statistical analysis was performed using the Wilcoxon signed-rank test. Normality was assessed using the Shapiro-Wilk test. The embedded chatbot demonstrated higher content validity and received better scores in accuracy and clarity compared to ChatGPT-4 (OpenAI, USA). A greater number of its responses achieved acceptable I-CVI (≥ 0.78) and S-CVI (≥ 0.8) values. Although the embedded model showed numerical trends favoring its performance over ChatGPT-4 (OpenAI, USA), the differences were not statistically significant (p = 0.173). The embedded chatbot model showed improved clarity and validity in orthodontic responses, suggesting potential advantages of domain-specific embedding and prompt engineering. While performance differences were not statistically significant, the findings support the growing role of tailored artificial intelligence (AI) tools in enhancing patient communication and clinical support in orthodontics.

Supplementary Information

The online version contains supplementary material available at 10.1038/s41598-026-39263-3.

Introduction

The healthcare landscape is undergoing a transformative shift with the advent of artificial intelligence (AI) and machine learning technologies¹. Orthodontics, a specialized branch of dentistry focused on correcting teeth and jaw misalignment, plays a crucial role in enhancing both aesthetic appeal and oral health². With the advent of digital technologies, the field has seen a remarkable transformation in patient care, particularly in the way information is disseminated and queries are handled³. Traditionally, orthodontic consultations have been conducted through in-person visits, but the integration of AI and large language models (LLMs) has opened new avenues for patient interaction and education^4,5.

In recent years, large language models like OpenAI’s Chat Generative Pre-trained Transformer (ChatGPT, OpenAI, USA) have emerged as powerful tools capable of understanding and generating human-like text⁶. These models have been trained on vast amounts of data, allowing them to provide detailed and accurate responses to a wide range of queries⁷. This capability presents a unique opportunity for orthodontic practices to enhance patient engagement and streamline the delivery of information related to pre-treatment, during-treatment, and post-treatment phases.

The utilization of AI-driven chatbots embedded within orthodontic platforms offers a tailored solution that can address common patient inquiries throughout the treatment journey⁸. These embedded chatbots are typically designed to handle specific scenarios based on predefined rules and datasets⁹. On the other hand, ChatGPT (OpenAI, USA), with its more generalized training, can offer a broader and potentially more nuanced understanding of patient queries, albeit with some limitations in specialization¹⁰.

The impetus for this study stemmed from the increasing demand for effective communication tools in orthodontics. Patients frequently have a multitude of questions and concerns that arise throughout the entire treatment process. While traditional methods of patient education, such as in-person consultations and printed brochures, are effective, they have limitations regarding accessibility, immediacy, and scalability. Digital solutions, especially those powered by artificial intelligence, hold the potential to overcome these limitations by providing continuous support and personalized information. However, a thorough evaluation of large language models’ effectiveness in providing orthodontic treatment-related queries in the field of orthodontics has not yet been thoroughly conducted.

This comparative study aimed to assess the performance of an embedded chatbot model in responding to orthodontic treatment-related queries and to compare its effectiveness with that of ChatGPT-4 (OpenAI, USA). By analyzing their performance in handling patient queries related to orthodontic care, the study sought to determine which approach offered superior patient support and education.

Materials and methods

After obtaining an exemption from the institutional ethical review committee, a cross-sectional comparative study was conducted at a tertiary care hospital. To build the embedded chatbot model, we utilized Anthropic’s Claude 3.7 Sonnet (USA) through Google Colab (Google LLC, USA). The model was deployed on a local machine running Windows 11 Pro (Microsoft Corporation, USA), specifically an HP EliteBook with an Intel Core i7-8550U processor, 16 GB of RAM, and a 64-bit operating system. The model was accessed via Google Colab (Google LLC, USA), utilizing its cloud-based capabilities for scalability, while the local machine was used for the development and testing phases. The hardware specifications of the local machine used for development included an Intel Core i7-8550U processor (1.80 GHz, boost up to 1.99 GHz) with 16 GB of RAM (15.8 GB usable). This system provided sufficient resources for handling the local development environment, while the cloud-based deployment was leveraged for processing the Retrieval-Augmented Generation (RAG)-based embeddings and interacting with the Claude 3.7 Sonnet (Anthropic, USA) Application Programming Interface (API). The model was accessed via API integration, and prompt engineering was employed to define a strict role for the chatbot in addressing orthodontic treatment-related queries. The model was not fine-tuned specifically for orthodontic content but was augmented using RAG techniques. This involved embedding curated orthodontic-related reference documents into the model, improving the contextual relevance of its responses. This approach enabled the chatbot to provide more targeted and accurate orthodontic information, even without fine-tuning the model for this domain. A contextual knowledge base, consisting of curated orthodontic care instructions, was uploaded as a reference document. Using Retrieval-Augmented Generation (RAG) techniques, relevant embeddings were created and stored in a vector database to enable precise and contextually accurate responses. The contextual knowledge base referred to as “orthodontic care instructions” consisted of patient-focused educational materials and clinically grounded explanatory content provided in Supplementary Appendix A for transparency and reproducibility. The primary source comprised publicly available British Orthodontic Society (BOS) patient information leaflets, which provide standardized guidance routinely used in orthodontic practice. These leaflets cover oral hygiene during orthodontic treatment, dietary advice, appliance care, management of common problems (e.g., pain, ulcers, loose brackets), elastic wear, retainer use, and post-treatment maintenance.

To ensure clinical completeness and conceptual clarity when responding to patient queries, the BOS materials were supplemented with content derived from established orthodontics textbooks, which were used to support brief explanatory concepts underlying treatment decisions (e.g., rationale for extractions, anchorage requirements, retention protocols). The following textbooks were referenced:

1. Contemporary Orthodontics (Proffit, W. R., Fields, H. W., Larson, B. E., Sarver, D. M., 6th Edition).

2. Orthodontics: Current Principles and Techniques (Graber, T. M., Vig, K. W. L., Huang, G., Fleming, P. S., 7th Edition).

3. Esthetics and Biomechanics in Orthodontics (Nanda, R., 2nd Edition).

4. Handbook of Orthodontics (Cobourne, M. T., DiBiase, A. T., 3rd Edition).

The knowledge base was not intended to function as a comprehensive academic reference but rather to reflect the type of information typically communicated during chairside counseling and written take-home instructions. Document selection and curation were performed by a postgraduate trainee in orthodontics. The decision to involve a postgraduate orthodontic trainee in the document selection and curation was made because of their current clinical exposure and familiarity with patient care protocols, ensuring the materials were highly relevant to common orthodontic queries. The trainee’s work was closely supervised and guided by senior clinicians to maintain accuracy and alignment with established orthodontic practices. Inclusion criteria were (i) relevance to common orthodontic patient questions, (ii) consistency with routine clinical practice, and (iii) alignment with guidance from recognized professional bodies such as BOS or standard orthodontic textbooks. Materials were excluded if they were redundant, institution-specific without general applicability, or beyond the scope of patient education. For orthodontic topics with variable clinical approaches or debated evidence (e.g., extraction therapy and facial aesthetics), the knowledge base retained balanced, explanatory information drawn from orthodontics textbook sources. Where differing viewpoints existed, contextual qualifiers (such as skeletal pattern, soft-tissue profile, and patient-specific factors) were preserved. During inference, if retrieved passages reflected disagreement, the model was instructed to summarize alternative perspectives, avoid definitive recommendations, and emphasize individualized clinical judgment. The temperature parameter was set to 0.7 to encourage coherent and informative responses, striking a balance between creativity and factual accuracy. No other significant parameters (e.g., max tokens, frequency penalty) were altered from the default settings for the model (Figure 1).

Fig. 1.

Workflow for developing the Embedded Chatbot Model using Claude 3.7 Sonnet (Anthropic, USA).

A purposive sampling method was used to recruit participants. To be eligible, participants must be practicing orthodontists holding a postgraduate qualification in Orthodontics and willing to sign the informed consent form. Professionals from other dental specialities, such as Endodontics or Periodontics, and Orthodontists who are not actively practicing were excluded from the study sample. Considering the recommendations, the number of experts for content validation should be at least six and not exceed ten¹¹. A total of six orthodontic consultants from various institutions were invited to participate in the evaluation process. A joint meeting with all assessors was held before data collection to ensure consistency in scoring and alignment on evaluation criteria. A formal calibration exercise was conducted to ensure scoring consistency. Assessors participated in a training session to review sample responses, align on evaluation criteria, and discuss scoring standards.

Evaluation criteria

The evaluation of the model’s performance was based on the following criteria:

Accuracy

The degree to which the response provided by the model was factually correct and relevant to the query.

Clarity

How clearly the model’s response conveyed the information, including language use and structure.

Relevance

The degree to which the model’s response addressed the query’s core issue, avoiding irrelevant or tangential content.

Current knowledge

The extent to which the model’s response reflected up-to-date information, particularly in relation to recent clinical guidelines and standards.

A scoring rubric with clear criteria was provided to ensure consistency in evaluation. The scoring criteria were developed by expert orthodontic consultants and validated through internal review to ensure relevance and clarity. Assessors were familiarized with the reference materials used by the model to ensure consistent evaluation.

A questionnaire comprising 30 real-world orthodontic treatment-related queries, categorized by pre-treatment, mid-treatment, and post-treatment, is provided in Supplementary Appendix B. These questions were subjected to validation by a panel of domain-specific experts to ensure relevance and comprehensiveness. To ensure the relevance and accuracy of the questions, we conducted a structured validation process involving a panel of expert orthodontists. These experts reviewed each question for relevance and clarity. Unfortunately, patient representatives were not involved in the validation process. The questions were primarily intended to address patient-level queries, focusing on common concerns that patients typically have about orthodontic treatments, including treatment options, oral hygiene practices, and post-treatment care. Each question was assessed on a scale of relevance (e.g., from “not relevant” to “highly relevant”) to ensure it addressed the key issues faced by patients. Feedback was gathered from the experts, and any questions deemed inadequate or unclear were revised or removed. Following validation, the same set of questions was processed to generate responses via two sources: the embedded chatbot model and ChatGPT-4 (OpenAI, USA). We used ChatGPT-4 (specifically, the GPT-4 model accessible via ChatGPT Plus) for generating responses to orthodontic queries. For each query, both models were provided with a structured prompt tailored to the specific question asked. A sample of the typical prompt structure used is as follows: “You are an orthodontic expert. Provide a clear, concise, and accurate response to the following patient query: [INSERT QUERY]. Please ensure the explanation is understandable for a layperson while maintaining clinical accuracy.” For example, the query, “Are there any dietary restrictions I should prepare for?” was prompted as, “You are an orthodontic expert. Provide a clear, concise, and accurate response to the following patient query: Are there any dietary restrictions I should prepare for? Please explain in simple terms for a patient considering orthodontic treatment.” Each question was submitted once to the model for response generation. The generated responses from both models were anonymized using unique identifiers “A” and “B” and disseminated to evaluators via Google Forms. Each participant was asked to assess the responses using a 5-point Likert scale: 1 = Strongly disagree; 2 = Disagree; 3 = Neutral; 4 = Agree; 5 = Strongly agree across four dimensions: accuracy, clarity, up-to-date knowledge, and relevance (Figure 2). Consent was obtained prior to the completion of the questionnaire by the dental professionals. The assessors remained blinded to the origin of each response to reduce potential bias. Since the assessors were orthodontic consultants with expertise in the field, no gold-standard reference answers were provided. The evaluation was based on professional judgment regarding accuracy, clarity, up-to-date knowledge, and relevance.

Fig. 2.

Study design for response evaluation.

Statistical analysis and content validation

The analysis of the data was performed using SPSS (version 26.0). Shapiro-Wilk test was performed to examine the normality of data. Inter-rater reliability was assessed using the modified kappa statistic (K*) to ensure consistency among expert ratings. The Wilcoxon signed-rank test was applied to compare the content validity scores between the two models. To evaluate the quality of responses produced by both models, a quantitative Content Validity Index (CVI) was employed. This index was assessed at two levels: the item level (I-CVI) and the scale level (S-CVI/Ave). Experts were asked to assess each response based on four key indicators, i.e., relevance, clarity, accuracy, and current knowledge, using a 5-point Likert scale: 1 = Strongly disagree; 2 = Disagree; 3 = Neutral; 4 = Agree; 5 = Strongly agree. Assessors were instructed to rate each criterion independently based on the response’s merit for that particular aspect. This meant that even when contradictions occurred between criteria (e.g., agreeing that the response reflects current knowledge but disagreeing with its accuracy), each criterion was assessed and scored separately. No weighting was given to one criterion over another; each was evaluated on its own.

For analysis purposes, the scale was dichotomized into two categories: agreement (ratings of 4 or 5) and disagreement (ratings of 1, 2, or 3). The I-CVI for each item was determined by dividing the number of experts in agreement by the total number of raters. To account for the possibility of agreement occurring by chance, a modified kappa coefficient was also calculated. To evaluate the overall validity across all items, the Scale-Level Content Validity Index using the averaging method (S-CVI/Ave) was computed by taking the mean of all I-CVI scores (i.e., the total sum of I-CVI values divided by the number of items).

Results

Inter-rater reliability was assessed using the modified kappa (K)* statistic, which measures the agreement between evaluators across the four domains: accuracy, clarity, relevance, and current knowledge. While acceptable responses in each of the four domains showed an inter-rater reliability ranging from 0.75 to 1, indicating excellent agreement amongst evaluators, the overall K value* for the 30 queries was below 0.6, indicating weak inter-rater agreement.

As shown in Table 1, there were numerical trends in favor of the embedded chatbot model across various parameters, but the differences were not statistically significant.

Table 1.

Comparison between embedded chatbot and ChatGPT-4.

Variable	Embedded chatbot Median (IQR)	ChatGPT-4 Median (IQR)	p-value
Accuracy	4.01 (2.84, 4.68)	3.71 (3.06, 3.97)	0.116
Clarity	3.96 (2.99, 4.63)	3.50 (2.99, 3.99)	0.115
Relevance	4.03 (2.96, 4.65)	3.66 (3.10, 4.04)	0.173
Up-to-date knowledge	4.01 (2.83, 4.67)	3.61 (3.05, 3.87)	0.138
Overall content validity	4.00 (2.90, 4.66)	3.60 (3.07, 3.97)	0.173

N = 6, IQR = Interquartile range, Wilcoxon Signed Rank Test, p < 0.05*.

Embedded chatbot model

The assessment of content validity across parameters such as relevance, accuracy, clarity, and up-to-date knowledge indicated a validity rate of 74.16%, meaning that 89 out of 120 responses achieved an acceptable item-level content validity index (I-CVI) within the range of 0.78–1 (Table 2). Overall, the responses demonstrated satisfactory validity for relevance (S-CVI = 0.844) and accuracy (S-CVI = 0.811). In contrast, clarity (S-CVI = 0.794) and up-to-date knowledge (S-CVI = 0.778) were slightly below the generally accepted benchmark of 0.8 (Table 2).

Table 2.

Responses generated via ChatGPT-4 and embedded model in terms of I-CVI.

Variable	Embedded chatbot	ChatGPT-4
Accuracy	23 (76.66%)	16 (53.33%)
Clarity	21 (70.00%)	14 (46.66%)
Relevance	26 (86.66%)	14 (46.66%)
Up-to-date knowledge	19 (63.33%)	17 (56.66%)
Overall	89 (74.16%)	61 (50.83%)

N = 30 I-CVI = (Item-level content validity index) Number of responses with acceptable I-CVI (between 0.78–1).

ChatGPT-4 (OpenAI, USA)

For ChatGPT-4, the evaluation of content validity across the same four domains—relevance, accuracy, clarity, and currency of knowledge—showed a validity rate of 50.83%, with 61 out of 120 responses attaining an acceptable I-CVI score within the 0.78–1 interval (Table 3). The overall findings reflected suboptimal validity across all domains, with S-CVI = 0.739 for relevance, S-CVI = 0.711 for accuracy, S-CVI = 0.728 for clarity, and S-CVI = 0.644 for up-to-date knowledge (Table 3).

Table 3.

Responses generated via ChatGPT-4 and embedded model in terms of S-CVI.

Variable	Embedded chatbot	ChatGPT-4
Accuracy	0.811a	0.711b
Clarity	0.794b	0.728b
Relevance	0.844a	0.739b
Up-to-date knowledge	0.778b	0.644b

S-CVI = (scale-level content validity index). ^aAcceptable content validity (S-CVI = 0.8 or greater). ^bUnacceptable content validity (S-CVI < 0.8).

To enhance interpretability, we present several sample orthodontic questions along with corresponding responses generated.

Question #1: Will I experience pain or discomfort during braces treatment?

Embedded chatbot model

Yes, you may experience some discomfort or pain during the initial adjustment of your fixed brace. This is normal and should go away within a few days. If you experience any discomfort, take what you normally have for a headache, and be sure to ask your orthodontist for advice.

ChatGPT-4 (OpenAI, USA)

Yes, it’s common to feel some discomfort, especially after braces are first applied or adjusted. This usually lasts a few days and can be managed with over-the-counter pain relievers and soft foods.

Question #2: Do braces affect my singing or speaking?

Embedded chatbot model

No, braces do not affect your singing or speaking ability. Your speech will be different at first, but it will return to normal within a couple of days. Practice speaking with the brace in place, e.g., read out aloud at home on your own. In this way, your speech will return to normal within a couple of days.

ChatGPT-4 (OpenAI, USA)

Possibly at first. You might notice slight changes, especially if you sing or speak professionally. However, most people adjust quickly, and any initial difficulty usually fades within a couple of weeks.

Question #3: Will I experience pain or discomfort during braces treatment?

Embedded chatbot model

As per your orthodontist’s recommendation, you will need regular follow-up appointments during orthodontic treatment. These appointments are necessary to monitor the progress of your treatment, check for any potential problems, and adjust your braces as needed. The frequency of follow-up appointments may vary, but it is best to stick to a regular schedule to ensure optimal results.

ChatGPT-4 (OpenAI, USA)

Typically, follow-up visits are scheduled every 4–8 weeks. These allow your orthodontist to make adjustments and monitor progress.

Question #4: Will brace treatment help with jaw pain or TMJ?

Embedded chatbot model

Brace treatment is designed to help improve the alignment of your teeth and jaws. It is not a remedy for jaw pain or TMJ. Braces are not designed to treat these conditions. However, they can help in the short term with the improvement of your jaw and jaw joint alignment, which can help with pain relief and prevent further damage.

ChatGPT-4 (OpenAI, USA)

In some cases, yes. Correcting a misaligned bite may relieve TMJ-related symptoms. However, orthodontics isn’t a guaranteed cure for TMJ disorders and should be evaluated on a case-by-case basis.

Discussion

The rise of generative AI in healthcare has prompted intense interest in its role in patient education and communication. Our study focused on orthodontic patient queries, an area where streamlined information delivery could significantly reduce clinician workload and patient anxiety. Prior reviews note that ChatGPT and similar LLMs “can significantly enhance dental practices” by answering patient questions and aiding in education, provided their limitations (accuracy, ethics, privacy) are addressed¹². Indeed, clinicians and patients alike are exploring AI chatbots as accessible information tools. Our findings suggest that domain-specific augmentation of LLMs can enhance their utility. In our study, the embedded chatbot model showed trends favoring improved accuracy and clarity compared to the generic ChatGPT (OpenAI, USA). This aligns with recent reports; for example, a dental chatbot using RAG achieved markedly higher factual accuracy than its non-augmented counterpart (correctly answering 10/15 guideline-based facts vs. 3/15).¹³ In another study on medication-related osteonecrosis of the jaw (MRONJ), a custom “GuideGPT” (using RAG and GPT-4) scored significantly higher than a generic GPT in content accuracy and explanation¹⁴. These results underscore that embedding domain-specific knowledge yields more reliable AI responses, supporting our approach.

Moreover, our emphasis on accuracy, clarity, relevance, and currency mirrors earlier work. In orthodontics, Praveen et al.¹⁵ found that 95% of ChatGPT’s responses were clear and 87.6% scientifically accurate and relevant to patient queries. Similarly, Mesude¹⁶ reported that ChatGPT-3.5 provided “reliable information” on oral cancer frequently asked questions (FAQs) with no incorrect answers. We likewise observed no blatantly false claims in either model’s answers, likely due to careful prompt design and a conservative (low) temperature setting. These findings echo broader evidence; a systematic evaluation of dental aligner FAQs found ChatGPT often gives correct or partly correct answers, though completeness varied¹⁷. ChatGPT-4 (OpenAI, USA) has shown high accuracy on specialized topics (e.g., orthodontic impacted canines) when assessed by experts¹⁸. Thus, our results that both models performed reasonably well on routine orthodontic questions are consistent with others’ experiences that LLMs can handle common clinical questions fairly competently¹⁹. In fact, one study comparing ChatGPT (OpenAI, USA) and Google Bard on orthodontic FAQs concluded that both provided “satisfactory responses,” with ChatGPT (OpenAI, USA) only slightly better¹⁹.

This suggests that modern LLMs generally capture broad dental knowledge, which explains why our embedded chatbot system and plain ChatGPT were both adequate in many cases. On the other hand, our study revealed that the embedded chatbot model surpassed ChatGPT (OpenAI, USA) in key metrics. The MRONJ study explicitly demonstrated that incorporating up-to-date research via RAG significantly improved content quality and reliability¹⁴. Similarly, the “Dental Loop Chatbot” project found that fine-tuning and embedding domain text made factual responses more accurate¹³. These similarities arise because our embedding likely provided additional recent orthodontic guidelines and textbooks that the base ChatGPT (OpenAI, USA) (with a 2021 cutoff) lacked. In a related study, Buldur and Sezer²⁰ showed ChatGPT-4’s answers matched FDA guidance on dental amalgam, but only after their direct comparison—without context, ChatGPT (OpenAI, USA) might have missed nuances. By contrast, our embedded model effectively gave itself context, explaining its higher accuracy. In essence, our findings suggest that a specialized, context-aware model may show trends favoring better performance over a generic chatbot, which aligns with existing literature on retrieval-augmented systems^13,14.

Several factors may account for why our results sometimes differ from others. Notably, model version and tuning play a large role. Studies showing superior performance often used GPT-4 Turbo or GPT-4o rather than the ChatGPT-4 Plus API we employed. For instance, Zhou et al.²¹ found that ChatGPT-4 answers actually outperformed human orthodontists on common FAQs. In their study, 50.8% of cases had ChatGPT ranked first. This contrasts with our finding that ChatGPT-4 lagged behind our embedded model. The difference likely stems from ChatGPT-4’s more advanced reasoning abilities and enhanced knowledge base²². Makrygiannakis et al.²² also found that Bing Chat (GPT-4) scored higher than ChatGPT-4 and Bard on orthodontic questions, whereas our focus on a tuned model (Claude-based) led to its advantage. These contradictions suggest that when using the latest LLMs with larger context or internet access, generic models can approach or exceed specialist knowledge. Our work, by contrast, emphasizes customization via embedding to bridge the gap with older models. Other contradictions arise from how questions are sourced and evaluated. For example, in a study by Abu Arqub et al.²³ questions answered by ChatGPT had a substantial error rate; only 58% of ChatGPT’s statements were fully true and 15% false. In our study, no answers were deemed outright incorrect, likely because we selected and vetted our question set carefully. Abu Arqub et al.²³ also did not use embedding or retrained contexts, which may explain the higher error rate. Likewise, some studies comparing chatbots found that Google Gemini (GPT-4 based) achieved the highest user-rated quality, surpassing ChatGPT and Copilot on FAQs²⁴. This suggests that not all LLMs are equal; Gemini’s more natural, readable output scored better than ChatGPT’s more complex language. We observed a similar nuance; our ChatGPT responses were often verbose and sometimes too technical, which may have hurt clarity scores. In sum, differences in the underlying model (3.5 vs. 4.0 vs. Bard vs. Gemini) and in evaluation (expert vs. patient ratings) can lead to divergent findings. Interestingly, while some reports flag concerns about LLMs, we did not observe any harmful misinformation. Studies have warned that “misleading information spreads rapidly” and that hallucinations in chatbots could mislead patients¹². We mitigated this by using low temperature and specific prompts, which appears to have limited fabrications. This contrasts with Guillen-Grima et al.¹⁰ who noted that ChatGPT’s references were often fabricated. In our study, we did not examine citations, but our content remained factual. Thus, our work aligns with studies reporting reliable outputs under constrained conditions and suggests that careful engineering can prevent “garbage” responses noted elsewhere¹⁵.

The inter-rater reliability assessed using modified kappa (K*) showed weak agreement, with the overall K* value below 0.6. This suggests variability in how assessors rated the responses across the domains of accuracy, clarity, relevance, and current knowledge. The low K* values likely reflect the subjectivity involved in evaluating clarity and relevance, where expectations may differ between assessors.

The limitations of our work must be acknowledged. First, we assessed only two specific AI systems (ChatGPT-4 and our embedded Claude-based RAG). Results may not generalize to other models (e.g., Bard, Gemini, or other Claude versions). Second, the content validity survey, while robust, was inherently subjective. Different expert panels might rate responses differently. Third, our embedded model’s knowledge was limited to what it was trained/fed from the internet and textbook snippets. Fourth, our evaluation did not measure actual patient satisfaction. It remains possible that patients would perceive the generic ChatGPT or Google more favorably, given novelty or accessibility. Despite these limitations, our study has notable strengths. It is (to our knowledge) the first to apply RAG embedding to assess orthodontic patient education. The robust masking and use of six orthodontic consultants lend credibility to the findings. In contrast to most prior work that relied on non-quantitative ratings, we provided both qualitative and quantitative content validity data. These methodological choices, combined with having a sufficiently large question set covering the spectrum of orthodontic care, strengthened our conclusions.

Conclusions

• The embedded chatbot model showed numerical trends favoring its performance over ChatGPT-4 in the context of orthodontic care, although the difference was not statistically significant.

• Nonetheless, the findings highlight the advantages of embedding and prompt engineering, suggesting promising potential for future advancements in healthcare applications, particularly in orthodontics.

Supplementary Information

Below is the link to the electronic supplementary material.

Abbreviations

API

Application Programming Interface

BOS

British Orthodontic Society

AI

Artificial intelligence

I-CVI

Item-level content validity index

IQR

Interquartile range

GPT

Generative Pre-trained Transformer

LLMs

Large language models

RAG

Retrieval-Augmented Generation

S-CVI

Scale-level content validity index

Author contributions

R.K: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Software; Writing – original draftL.A: Data curation, Software; Writing – original draftR.H.S: Supervision, Validation, Statistical analysis and Proof reading of manuscript, Writing - Review & EditingM.F: Supervision, Validation, Proof reading of manuscript.

Funding

There was no funding for this project.

Data availability

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Declarations

Competing interests

The authors declare no competing interests.

Human Ethics and consent to participate

The study was approved by the Institutional Review Board Aga Khan University, Pakistan under number (2024-10522-31446) and conforms to the declaration of Helsinki. Patients’ consent was not required and informed consent was waived by the Institutional Review Board of Aga Khan University, Pakistan.