Abstract
This study collected and screened the 50 most common pre-treatment consultation questions from adult orthodontic patients through clinical practice. Responses to these questions were generated using three large language models: Ernie Bot, ChatGPT, and Gemini. The responses were evaluated across six dimensions: Professional Accuracy (PA), Accuracy of Content(AC), Clarity and Comprehensibility (CC), Personalization and Relevance (PR), Information Completeness (IC), and Empathy and Patient-Centeredness (EHC). Results indicated that scores for each group in various dimensions primarily fell within the range of 3–4 points, with relatively few high-quality scores (5 points). While large language models demonstrate some capability in addressing open-ended questions, their use in medical consultation, particularly in orthodontic medicine, requires caution and further integration with professional guidance and verification. Future research and technological improvements should focus on enhancing AI(Artificial Intelligence) performance in accuracy, information completeness, and humanistic care to better meet the needs of diverse clinical scenarios.
Supplementary Information
The online version contains supplementary material available at 10.1186/s12903-025-06246-1.
Keywords: Large language models, Pre-treatment information, Orthodontic, Ernie bot, Chatgpt, Gemini
Background
With the increasing awareness of health and advancements in medical technology, demand for orthodontic treatment has surged, particularly among adults. Unlike adolescent patients, adult patients often exhibit heightened concerns about facial aesthetics during the decision-making phase before orthodontic treatment, leading to numerous apprehensions. As a result, this population frequently relies on online resources for orthodontic consultation [1–3]. However, the current online environment is overwhelmed with advertisements and misinformation, making it increasingly complex to obtain accurate and targeted answers to specific orthodontic questions [4].
In recent years, advancements in artificial intelligence have integrated large language models into everyday life, with growing numbers of adults seeking advice on real-world issues from intelligent systems. Studies suggest that compared to traditional search engines, large language models can provide more precise and personalized responses, potentially alleviating shortages in medical resources, uneven distribution, and inefficiencies within the healthcare system [5, 6]. However, other studies have raised concerns, arguing that these intelligent models may generate entirely fabricated scientific conclusions [7, 8]. While these answers may initially appear complex and error-free, experts often identify semantic inaccuracies and errors upon closer examination [9].
Given the high level of specialization in orthodontics and significant variations among patients, this study aims to address this situation by evaluating the accuracy of responses generated by various large language models through open-ended questions. The objective is to provide data and theoretical support for the potential use of large language models as consultation tools for adult orthodontic patients.
Method
Language model selection
The selected large language models are publicly accessible and have garnered extensive international user bases. Ernie Bot, developed by Chinese technology company Baidu, is a general-purpose Chinese language model that excels in tasks related to language processing. It is primarily utilized for search engines, dialogue systems, and content recommendations. ChatGPT, introduced by American company OpenAI in November 2022, is an artificial intelligence chatbot that operates through text-based interactions. It employs deep learning to generate responses based on patterns and statistical rules from pre-training data. Gemini, developed by Google DeepMind, is a multimodal large language model capable of understanding and generating both text and images. The model adopts an approach similar to OpenAI’s DALL-E, integrating natural language processing with image generation capabilities to produce visually relevant content while interpreting text.
Considering the scope and effectiveness of large language models, this study selected three leading intelligent models for analysis. Since these models were used solely for text-based processing in this research, Gemini’s image-generation functionality was not utilized. As shown in Table 1, the following provides an overview of the large language models and their corresponding basic information used in this experiment.
Table 1.
Large-scale Language model
| language model | Corporate and National |
|---|---|
| Ernie Bot4.0(EB) | (Baidu, China) |
| Chatgpt 4.0(C) | (OpenAI, United States) |
| Gemini 1.5 Pro(G) | (Google, United States) |
Evaluation criteria
To strengthen the evaluation of responses provided by large language models during pre-treatment consultations for adult orthodontic patients, our research team expanded the assessment criteria to six aspects based on prior studies: Professionalism(PA)、Accuracy of Content (AC)、Clarity and Comprehensibility(CC)、Personalization and Relevance(PR)、Information Completeness(IC)、Empathy and Human-Centeredness(EHC) [6, 7]. PA refers to whether the content aligns with professional standards, clinical guidelines, and industry consensus in the field of orthodontics. AC ensures that the content is accurate, free from factual errors or vague statements.CC assess whether the information is presented in a logical and patient-friendly manner, avoiding ambiguity or overly specialized language.PR evaluate whether responses are tailored to individual patient needs rather than relying on generic templates.IC measures whether the information comprehensively addresses patient concerns without critical omissions or vagueness.EHC gauge whether the responses demonstrate sensitivity to patients’ emotional states and provide appropriate emotional support. All evaluations are conducted using a 1-to-5 rating scale, with scoring criteria developed based on: (1) orthodontic clinical guidelines; (2) patient information readability studies; and (3) quality assessments of AI-generated medical texts [10–14]. Table 2 provides a systematic overview of the specific details of these scoring standards. This expanded framework aims to provide a more detailed and nuanced understanding of how these AI models perform in delivering specialized and patient-centered consultations.
Table 2.
Evaluation criteria
| Professionalism (PA) | Score |
| Fully aligns with orthodontic professional standards; no deviation from clinical guidelines or consensus. | 5 |
| Generally conforms to professional standards but contains minor, non-principal deviations. | 4 |
| Partially meets standards; key steps or principles conflict with established guidelines. | 3 |
| Most content deviates from professional consensus and may mislead clinical decision-making. | 2 |
| Severely violates professional norms. | 1 |
| Accuracy of Content (AC) | Score |
| Precise, comprehensive expression with no errors in data, terminology, or processes. | 5 |
| Core content is accurate; minor deviations exist in secondary details. | 4 |
| Some content is accurate, but critical information contains errors. | 3 |
| Multiple factual errors that may affect patient understanding or procedural accuracy. | 2 |
| Core information is completely erroneous. | 1 |
| Clarity and Comprehensibility(CC) | Score |
| Extremely clear and logically coherent; patients can fully understand without follow-up questions. | 5 |
| Clear expression but some terms or sentence structures are slightly complex, requiring minimal patient effort to understand. | 4 |
| Generally smooth expression but contains ambiguities or logical gaps, necessitating patient clarification. | 3 |
| Overly specialized or vague language; patients must ask multiple questions to grasp key information. | 2 |
| Disjointed and disorganized presentation; patients cannot identify critical content. | 1 |
| Personalization and Relevance(PR) | Score |
| The response is highly personalized, directly addressing the specific needs of the patient. | 5 |
| There is good personalization, but some content appears somewhat generic. | 4 |
| Some personalized content is present, but much of the information is general. | 3 |
| The response has low personalization and is relatively template-like. | 2 |
| There is no personalization at all, and the information is very general. | 1 |
| Information Completeness(IC) | Score |
| The information is very comprehensive, covering all the details of concern to the patient. | 5 |
| Most of the information is complete, with only minor, non-essential details missing. | 4 |
| The main information is covered, but some important content is missing or insufficiently explained. | 3 |
| The information is incomplete, with key issues not adequately addressed. | 2 |
| There is a significant lack of information, and the patient’s primary concerns remain unanswered. | 1 |
| Empathy and Human-Centeredness(EHC) | Score |
| The response frequently addresses the patient’s emotions and concerns, using language that clearly alleviates their anxiety or doubts | 5 |
| The response explicitly acknowledges the patient’s emotions or needs at least once, but falls slightly short in establishing a deeper emotional connection | 4 |
| Although the response considers the patient’s emotional needs, the language used is rather detached or formulaic | 3 |
| The response lacks empathy, with language that feels rigid and only provides objective information | 2 |
| The response entirely lacks empathy, with a tone that is cold or unfriendly, potentially leading to negative emotions for the patient. | 1 |
Source and handling of questions
This study collected a total of 1,762 real orthodontic consultation questions from January 1, 2024, to July 1, 2024. The data sources included the following: (1) Outpatient consultation records: written records of the patient questions were collected with the patient’s consent and stored anonymously, yielding a total of 923 questions. (2) Electronic Health Record (EHR) text analysis: Using natural language processing (NLP), questions related to preoperative consultations were extracted from the chief complaint section of medical records. A total of 512 questions met the inclusion criteria. (3) Patient online questionnaires**: Anonymous questionnaires were distributed to 320 adult orthodontic patients who had recently visited for treatment, with 272 completed and returned. This process yielded a total of 327 questions. All duplicate questions were removed, resulting in the final selection of 50 most frequently asked questions, which were used for testing large language models (LLMs).
Scoring Process.
Four experts with master’s degrees in orthodontics were invited to evaluate and score the PA and AC of responses generated by the language model. Prior to scoring, training was conducted to ensure that all evaluation criteria were based on standards from authoritative orthodontic textbooks [11]. Additionally, four adult patients were randomly selected from a case repository to score dimensions such as CC, PR, EHC, and IC. These patients were representative in terms of demographic factors (e.g., age, gender) and clinical characteristics. Participants were required to meet the following criteria: (1) Adequate understanding of medical-related issues. (2) Ability to independently complete the scoring task. Training for patient scorers: Patients were provided with detailed definitions and scoring criteria for each dimension (e.g., a 1–5 scale) along with examples to ensure consistent understanding of the rating rules. To guarantee objectivity and consistency in scoring, all evaluators underwent an inter-rater reliability analysis after training. The Cohen’s kappa coefficient was used to assess consistency across dimensions; retraining was conducted until all scorers achieved satisfactory levels of agreement(Table 3).
Table 3.
Rater consistency analysis
| PA | AC | CC | PR | IC | EHC | |
|---|---|---|---|---|---|---|
| Cronbach’s alpha | 0.81 | 0.89 | 0.91 | 0.85 | 0.83 | 0.84 |
PA, AC, CC, PR, IC, AndEHC represent professionalism, accuracy of Conten, clarity and comprehensibility, personalization and relevance, information completeness, and empathy and Human-Centeredness, Respectively.Cronbach’s alpha values ≥ 0.7 indicate acceptable agreement
All evaluators independently scored the LLM’s responses according to the specified criteria, with individual scores collected for each dimension. If the four experts provided identical scores, those scores were adopted. In cases of significant discrepancies, a structured resolution mechanism was applied to ensure accuracy and consistency:1. For each dimension, mean scores and the difference between maximum and minimum scores were calculated. 2. If the score variance exceeded a predefined threshold (e.g.,1 point), a second round of discussion and reassessment was conducted. 3.To address discrepancies, evaluators were summoned for online or offline discussions to clarify reasons for differing scores. Patients were reminded of the scoring criteria and instructed to revisit the LLM’s responses before independently completing a new set of evaluations. 4. If variances persisted after the second round of scoring, an arbitration mechanism was invoked. A senior coordinator (an experienced patient representative) or expert in the field was invited to provide a final score based on established standards and professional judgment. To eliminate carryover effects from prior interactions, each question was submitted to the intelligent model within a new chat session with explicit instructions to ignore historical context. For example: “Forget our previous conversation and generate an answer based solely on the current question [15, 16].
Statistical analysis
The data were analyzed using SPSS version 27. Normality and variance homogeneity were assessed using the Shapiro-Wilk test and Levene’s test, respectively. If the data met the assumptions of normality, results were reported as means with standard deviations. For data that did not satisfy normality criteria, results were presented as means, standard deviations, and medians (P25, P75). Additionally, for non-normally distributed data, group differences were compared using the non-parametric Wilcoxon rank-sum test. Categorical variables were reported in terms of actual counts and percentages to clearly indicate the distribution across categories (Fig. 1).
Fig. 1.
Consensus process for scoring responses from large language models
Results
As shown in Table 4, the means across groups exhibited some variability but generally fell within the range of 3.52 to 4.08. Specifically, EB-IC and EB-PR demonstrated higher mean scores of 4.08 and 3.98, respectively, indicating superior performance in ratings for these groups. In contrast, G-EHC and C-CC exhibited lower means (3.44 and 3.46, respectively). Furthermore, most groups showed a maximum value of 5, suggesting that individual samples performed exceptionally well in actual ratings. Notably, EB-PR had the largest standard deviation (0.71), indicating greater variability in scores among samples within this group.
Table 4.
Statistical descriptions of each group
| N | Mean | Mean±SD | Min | MAX | P 25 | M | P 75 | |
|---|---|---|---|---|---|---|---|---|
| EB-PA | 50 | 3.8 | 0.67 | 3 | 5 | 3 | 4 | 4 |
| G-PA | 50 | 3.8 | 0.7 | 3 | 5 | 3 | 4 | 4 |
| EB-AC | 50 | 3.52 | 0.68 | 3 | 5 | 3 | 3 | 4 |
| G-AC | 50 | 3.82 | 0.69 | 3 | 5 | 3 | 4 | 4 |
| EB-CC | 50 | 3.82 | 0.52 | 3 | 5 | 3.75 | 4 | 4 |
| G-CC | 50 | 3.86 | 0.57 | 3 | 5 | 3.75 | 4 | 4 |
| EB-PR | 50 | 3.98 | 0.71 | 3 | 5 | 3 | 4 | 4.25 |
| G-PR | 50 | 3.78 | 0.65 | 3 | 5 | 3 | 4 | 4 |
| EB-IC | 50 | 4.08 | 0.67 | 3 | 5 | 4 | 4 | 5 |
| G-IC | 50 | 3.8 | 0.50 | 3 | 5 | 3.75 | 4 | 4 |
| EB-EHC | 50 | 3.66 | 0.63 | 3 | 5 | 3 | 4 | 4 |
| G-EHC | 50 | 3.44 | 0.50 | 3 | 4 | 3 | 3 | 4 |
| C-PA | 50 | 3.64 | 0.63 | 3 | 5 | 3 | 4 | 4 |
| C-AC | 50 | 3.6 | 0.57 | 3 | 5 | 3 | 4 | 4 |
| C-CC | 50 | 3.46 | 0.54 | 3 | 5 | 3 | 3 | 4 |
| C-PR | 50 | 3.8 | 0.45 | 3 | 5 | 4 | 4 | 4 |
| C-IC | 50 | 3.58 | 0.54 | 3 | 5 | 3 | 4 | 4 |
| C-EHC | 50 | 3.6 | 0.67 | 3 | 5 | 3 | 3.5 | 4 |
EB, C, and G represent ernie bot, ChatGPT, and Gemini, respectively. PA, AC, CC, PR, IC, AndEHC represent professionalism, accuracy of Conten, clarity and comprehensibility, personalization and relevance, information completeness, and empathy and Human-Centeredness, respectively
To compare the differences among different groups, we conducted a non-parametric analysis using the Wilcoxon rank-sum test (Mann-Whitney U test). The significance level was set at α = 0.05. As shown in Table 5, significant differences were observed between G-AC and EB-AC, C-CC and EB-CC, as well as G-IC and EB-IC. Additionally, highly significant differences were found between C-CC and G-CC, and between C-IC and EB-IC. In other comparisons, such as G-PA vs. EB-PA and C-PA vs. EB-PA, no statistically significant differences were observed (P > 0.05). Furthermore, some comparisons exhibited P-values approaching significance, for example, C-IC vs. G-IC: P = 0.065; however, these results did not reach statistical significance.
Table 5.
Post-Hoc pairwise comparisons of scores for AI chatbots
| Wilcoxon Rank-Sum Test for Group Comparisons | Z | P |
|---|---|---|
| G-PA vs. EB-PA | 0.0 | 1.0 |
| C-PA vs. EB-PA | 1.3 | 0.2 |
| C-PA vs. G-PA | 1.1 | 0.3 |
| G-AC vs. EB-AC | 2.2 | 0.0# |
| C-AC vs. G-AC | 1.6 | 0.1 |
| C-AC vs. EB-AC | 0.6 | 0.6 |
| G-CC vs. EB-CC | 0.4 | 0.7 |
| C-CC vs. EB-CC | 3.1 | 0.0# |
| C-CC vs. G-CC | 3.5 | 0.0# |
| G-PR vs. EB-PR | 1.3 | 0.2 |
| C-PR vs. EB-PR | 1.5 | 0.1 |
| C-PR vs. G-PR | 0.2 | 0.9 |
| G-IC vs. EB-IC | 2.3 | 0.0# |
| C-IC vs. EB-IC | 3.6 | 0.0# |
| C-IC vs. G-IC | 1.8 | 0.1 |
| G-EHC vs. EB-EHC | 1.9 | 0.1 |
| C-EHC vs. EB-EHC | 0.5 | 0.7 |
| C-EHC vs. G-EHC | 1.4 | 0.2 |
EB, C, and G represent ernie bot, ChatGPT, and Gemini, respectively. PA, AC, CC, PR, IC, AndEHC represent professionalism, accuracy of Conten, clarity and comprehensibility, personalization and relevance, information completeness, and empathy and Human-Centeredness, respectively. # indicates P < 0.05
As shown in Table 6, the results indicate that within the Ernie Bot group, its distribution is relatively concentrated across the PR, IC, EHC, and PA metrics, with most of its scores clustered in the 4-point segment. Notably, in the CC metric, Ernie Bot demonstrated superior performance, with 70.0% of its scores concentrated in the 4-point interval. However, in the AC metric, only 32.0% of its scores fell within the 4-point segment, while 58.0% were distributed across the 1–3 point segments.For the ChatGPT group, its performance was more concentrated in the PR metric, with a significant majority (76.0%) of its capabilities clustered in the 4-point segment. The proportions for the 1–3 point interval and the 5-point segment were relatively low at 22.0% and 2.0%, respectively. In contrast, ChatGPT’s performance across the other metrics was more balanced, with approximately similar proportions observed for the 1–3 point, 4-point, and 5-point segments.In the Gemini group, its distribution within the IC metric was relatively concentrated, with most of its capabilities (72.0%) clustered in the 4-point interval. A smaller proportion (24.0%) fell within the 1–2 point segment, while only 4.0% were allocated to the 5-point segment. Similarly, in the CC metric, Gemini’s distribution was concentrated, with a majority (66.0%) of its capabilities clustered in the 4-point interval.
Table 6.
Response score distribution of each chatbot based on the scoring scale and model classification
| PA | AC | CC | PR | IC | EHC | ||
|---|---|---|---|---|---|---|---|
| EB | 1–3 | (17)34.0% | (29)58.0% | (12)24.0% | (13)26.0% | (9)18.0% | (21)42.0% |
| 4 | (26)52.0% | (16)32.0% | (35)70.0% | (25)50.0% | (28)56.0% | (25)50.0% | |
| 5 | (7)14.0% | (5)10.0% | (12)24.0% | (12)24.0% | (13)26.0% | (4)8.0% | |
| C | 0–3 | (22)44.0% | (22)44.0% | (28)56.0% | (11)22.0% | (22)44.0% | (25)50.0% |
| 4 | (24)48.0% | (26)52.0% | (21)42.0% | (38)76.0% | (27)54.0% | (20)40.0% | |
| 5 | (4)8.0% | (2)4.0% | (1)2.0% | (1)2.0% | (1)2.0% | (5)10.0% | |
| G | 1–3 | (18)36.0% | (17)34.0% | (12)24.0% | (17)34.0% | (12)24.0% | (28)56.0% |
| 4 | (24)48.0% | (25)50.0% | (33)66.0% | (27)54.0% | (36)72.0% | (22)44.0% | |
| 5 | (8)16.0% | (8)16.0% | (5)10.0% | (6)12.0% | (2)4.0% | (0)0.0% |
EB, C, and G represent ernie bot, ChatGPT, and Gemini, respectively. PA, AC, CC, PR, IC, AndEHC represent professionalism, accuracy of Conten, clarity and comprehensibility, personalization and relevance, information completeness, and empathy and Human-Centeredness, respectively
Discussion
The advanced language models utilized in this study leverage deep learning techniques to generate human-like responses to natural language inputs. These models are proficient at capturing the subtle nuances and complexities of human language, enabling them to produce contextually relevant replies across diverse prompts [8]. While previous studies have employed various scoring systems such as the Global Quality Scale (GQS), Flesch-Kincaid Grade Level (FKGL), Coleman-Liau Index (CLI), and SMOG readability formula, the unique nature of medical consultation necessitates the development of a customized scoring system in this study, building upon prior research. This system evaluates responses across six dimensions: PA、AC、CC、PR、IC、EHC [17–20].
Although there were discrepancies in scores among the three large language models when addressing open-ended questions, the overall trends across six dimensions were largely consistent. All models scored within a range of 3 to 4 points, with relatively few high-quality scores (5). This indicates that none of the models demonstrated consistently reliable performance across key areas such as professionalism and accuracy, clarity and comprehensibility, personalization and relevance, and completeness and thoroughness of information (Table 6). Previous research has suggested that in 40.5% of cases, artificial intelligence models were rated as fully accurate in their responses to open-ended questions, while in 50.5% of cases, the completeness of answers was deemed entirely correct, highlighting the accuracy and completeness of AI-generated responses [21–26]. Furthermore, Makrygiannakis M.A. and colleagues found that Bing Chat achieved the highest scores, followed by ChatGPT-4, Google-Bard, and ChatGPT-3.5.However, these findings do not necessarily reflect the overall reliability of large language models in addressing various open-ended questions accurately, as their performance is heavily dependent on extensive training datasets and complex neural network architectures with parameters far exceeding the number of neurons in the human brain[5, 27].
In reality, large language models are prone to inaccuracies and biases when generating responses. The phenomenon of AI “hallucination” refers to instances where models produce seemingly plausible but erroneous or fabricated information in the absence of factual data support. This issue is particularly critical in the field of medical AI [28–29]. Therefore, it is imperative not to rely solely on AI-generated responses in clinical applications but to seek validation and confirmation from qualified healthcare professionals when necessary [28–30].
The limitations of large-scale language models are evident when applied to specialized areas such as orthodontic consultation. These limitations include semantic translation errors (e.g., discrepancies between direct translations of English terminology and Chinese medical expressions) and culturally influenced contextual understanding biases (e.g., differences in cultural perceptions of “smile aesthetics”). Future research should incorporate multilingual testing, focusing on systematically comparing AI performance in terms of response accuracy, information completeness, and patient acceptance in different linguistic settings, and the weighting and complexity of the different scoring dimensions should be thoroughly investigated. In addition, the temporal context of the model has not been specified during this study, i.e., the exact timestamps cannot be tracked, which results in possible differences in the model’s analytical capabilities. Therefore, there is a need to develop large-scale language models tailored specifically for medical applications, with possible avenues including specialized training using web-based consultation data or creating new data-driven models in collaboration with specialized orthodontic teams.
Conclusion
In summary, while large language models demonstrate certain capabilities in handling open-ended questions, their application in the demanding field of orthodontic medical consultation requires caution and should be complemented by professional guidance and verification. Future research and technological advancements should focus on enhancing AI performance in terms of accuracy, information completeness, and empathy to better meet the needs of the medical field.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Acknowledgements
We thank Z.W.Y, Department of Stomatology, Jiaxing No. 2 Hospital, for supporting the work of this study.
Abbreviations
- AI
Artificial Intelligence
- PA
Professionalism
- AC
Accuracy of Content
- CC
Clarity and Comprehensibility
- PR
Personalization and Relevance
- IC
Information Completeness
- EHC
Empathy and Human-Centeredness
- NLP
Using natural language processing
- LLMs
large language models
Author contributions
Jingcheng Chen: Experimental design, data analysis and paper writing; Xiangyu Ge: Data analysis; Data collection, Chenyang Yuan: Data collection, Data analysis; Yanan Chen: Data collection, Data analysis, Xiangyu Li: Data verification; Xi Zhang: Data collection, Statistical analyses; Shixiang Chen: Data collection, Statistical analyses; WeiYing Zheng: Experimental design guidance and technical support; Chunqin Miao: Experimental design guidance and technical support.
Funding
No funding.
Data availability
Due to patient information privacy concerns, the datasets generated and/or analyzed during this study are not publicly available. However, they can be obtained from the corresponding author upon reasonable request.
Declarations
Ethics approval and consent to participate
This study affirms that it was conducted in accordance with the Declaration of Helsinki and was approved by the Human Research Ethics Committee of Jiaxing No. 2 Hospital(JXEY-2022JYT006). All participants and/or guardians gave informed consent to participate and signed an informed consent form.
Consent for publication
Not applicable.
Competing interests
The authors declare no competing interests.
Footnotes
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.Fenton GD, Cazaly MHM, Rolland SL, Vernazza CR. Eliciting preferences for adult orthodontic treatment: A discrete choice experiment. JDR Clin Trans Res. 2022;7(2):118–26. [DOI] [PubMed] [Google Scholar]
- 2.Chow L, Goonewardene MS, Cook R, Firth MJ. Adult orthodontic retreatment: A survey of patient profiles and original treatment failings. Am J Orthod Dentofac Orthop. 2020;158(3):371–82. [DOI] [PubMed] [Google Scholar]
- 3.Gao J, Feng Y, Xu S, et al. Appearance anxiety and social anxiety: A mediated model of self-compassion. Front Public Health. 2023;11:1105428. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Chen S, Makhoul AT, Janis JE, Perdikis G, Drolet BC. Board certification in cosmetic surgery: an examination of online advertising practices. Ann Plast Surg. 2022;88(5):S461–5. [DOI] [PubMed] [Google Scholar]
- 5.Makrygiannakis MA, Giannakopoulos K, Kaklamanos EG. Evidence-based potential of generative artificial intelligence large Language models in orthodontics: a comparative study of ChatGPT, Google bard, and Microsoft Bing. Eur J Orthod. 2024;13. [DOI] [PMC free article] [PubMed]
- 6.Kılınç DD, Mansız D. Examination of the reliability and readability of chatbot generative pretrained Transformer’s (ChatGPT) responses to questions about orthodontics and the evolution of these responses in an updated version. AJODO. 2024;165(5):546–55. [DOI] [PubMed] [Google Scholar]
- 7.Kurt Demirsoy K, Buyuk SK, Bicer T. How reliable is the artificial intelligence product large Language model ChatGPT in orthodontics? Angle Orthod. 2024;94(6):602–7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Dave T, Athaluri SA, Singh S. ChatGPT in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations. Front Artif Intell. 2023;6:1169595. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Májovský M, Černý M, Kasal M, Komarc M, Netuka D. Artificial intelligence can generate fraudulent but Authentic-Looking scientific medical articles: Pandora’s box has been opened. J Med Internet Res. 2023;25:e46924. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Naureen S, Kiani HG. Assessing the accuracy of AI models in orthodontic knowledge: A comparative study between ChatGPT-4 and Google bard. J Coll Physicians Surg Pak. 2024;34(7):761–6. [DOI] [PubMed] [Google Scholar]
- 11.Vågenes H, Pranić SM. Analysis of the quality, accuracy, and readability of patient information on polycystic ovarian syndrome (PCOS) on the internet available in english: a cross-sectional study. Reprod Biol Endocrinol. 2023;21(1):44. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Lavin CV, Fahy EJ, Abbas DB, et al. Readability of online patient information relating to cleft palate surgery. Cleft Palate Craniofac J. 2022;59(3):330–5. [DOI] [PubMed] [Google Scholar]
- 13.Ghanem YK, Rouhi AD, Al-Houssan A, et al. Dr. Google to dr. ChatGPT: assessing the content and quality of artificial intelligence-generated medical information on appendicitis. Surg Endosc. 2024;38(5):2887–93. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Graber LW, Vanarsdall RL, Vig KWL et al. Orthodontics: Curr Principles Techniques 2016(6). 1–1027.
- 15.Onder CE, Koc G, Gokbulut P, Taskaldiran I, Kuskonmaz SM. Evaluation of the reliability and readability of ChatGPT-4 responses regarding hypothyroidism during pregnancy. Sci Rep. 2024;14(1):243. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Erkan A, Koc A, Barali D, et al. Can patients with urogenital Cancer rely on artificial intelligence chatbots for treatment decisions?? Clin Genitourin Cancer. 2024;22(6):102206. [DOI] [PubMed] [Google Scholar]
- 17.Matsuoka M, Onodera T, Fukuda R et al. Evaluating the alignment of artificial Intelligence-Generated recommendations with clinical guidelines focused on soft tissue tumors. J Surg Oncol. 2024;(5). [DOI] [PubMed]
- 18.García-Rudolph A, Sanchez-Pinsach D, Opisso E, Evaluating AI. Models: performance validation using formal Multiple-Choice questions in neuropsychology. Arch Clin Neuropsychol.2024;(4). [DOI] [PubMed]
- 19.Lee JE, Park KS, Kim YH, Song HC, Park B, Jeong YJ. Lung Cancer staging using chest CT and FDG PET/CT Free-Text reports: comparison among three ChatGPT Large-Language models and six human readers of varying experience. AJR Am J Roentgenol. 2024;(4). [DOI] [PubMed]
- 20.Hatia A, Doldo T, Parrini S, et al. Accuracy and completeness of ChatGPT-Generated information on interceptive orthodontics: A multicenter collaborative study. J Clin Med. 2024;13(3):735. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Abu Arqub S, Al-Moghrabi D, Allareddy V, Upadhyay M, Vaid N, Yadav S. Content analysis of AI-generated (ChatGPT) responses concerning orthodontic clear aligners. Angle Orthod. 2024;94(3):263–72. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Desai P, Wang H, Davis L, Ullmann TM, DiBrito SR. Bias perpetuates bias: ChatGPT learns gender inequities in academic surgery promotions. J Surg Educ. 2024;81(11):1553–7. [DOI] [PubMed] [Google Scholar]
- 23.Molena KF, Macedo AP, Ijaz A, et al. Assessing the accuracy, completeness, and reliability of artificial Intelligence-Generated responses in dentistry: A pilot study evaluating the ChatGPT model. Cureus. 2024;16(7):e65658. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Labrague LJ. Utilizing artificial Intelligence-Based tools for addressing clinical queries: ChatGPT versus Google gemini. J Nurs Educ. 2024;63(8):556–9. [DOI] [PubMed] [Google Scholar]
- 25.Motoki K, Spence C, Velasco C. Colour/shape-taste correspondences across three languages in ChatGPT. Cognition. 2024;253:105936. [DOI] [PubMed] [Google Scholar]
- 26.Urbina JT, Vu PD, Nguyen MV. Disability ethics and education in the age of artificial intelligence: identifying ability Bias in ChatGPT and gemini. Arch Phys Med Rehabil.2024; 30. [DOI] [PubMed]
- 27.Aljamaan F, Temsah MH, Altamimi I, et al. Reference hallucination score for medical artificial intelligence chatbots: development and usability study. JMIR Med Inf. 2024;12:e54345. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Templin T, Perez MW, Sylvia S, et al. Addressing 6 challenges in generative AI for digital health: A scoping review. PLOS Digit Health. 2024;3(5):e0000503. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Dashti M, Ghasemi S, Ghadimi N, et al. Performance of ChatGPT 3.5 and 4 on U.S. Dental examinations: the INBDE, ADAT, and DAT. Imaging Sci Dent. 2024;54(3):271–5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Armbruster J, Bussmann F, Rothhaas C, Titze N, Grützner PA, Freischmidt H. Doctor ChatGPT, can you help me?? The patient’s perspective: Cross-Sectional study. J Med Internet Res. 2024;26:e58831. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
Due to patient information privacy concerns, the datasets generated and/or analyzed during this study are not publicly available. However, they can be obtained from the corresponding author upon reasonable request.