Abstract
Introduction
The aim of this study was to compare the response capabilities of generative pre-trained transformers (GPTs) for identical lateral cephalograms (LCs) based on text- and image-based prompts, using the newly developed Artificial Intelligence Based Diagnosis and Treatment Planning Index (AIDTI).
Methods
A total of 90 LCs from 30 cases each with skeletal Class I, II, and III malocclusions were included. The LCs were presented to GPT-4o, GPT-o3 pro, GPT-5, and GPT-5 pro in two different formats: text-based (numerical data including cephalometric analysis measurements) and image-based (direct image upload) prompts. The responses of the GPTs were evaluated using the newly developed AIDTI, scored on a 0–10 scale and consisting of five criteria: diagnostic accuracy, capability for differential diagnosis, clinical appropriateness of the proposed treatment plan, disclosure of risks and complications associated with the treatment plan, and ability to offer alternative treatment options.
Results
Higher performance was observed in text-based prompts, with the highest score achieved by GPT-5 pro (9.62 ± 1.13). By contrast, the performance of all GPTs on image-based prompts was notably lower, with the highest score being 4.16 ± 4.12 for GPT-o3 pro. Additionally, all models showed a tendency to randomly categorize malocclusions as Class II, suggesting systematic bias in their predictions.
Conclusion
AIDTI provides a structured and multidimensional framework for the concise, balanced, and clinically meaningful interpretation of GPT performance. Because GPTs are unable to directly analyze LCs, their use as independent or reliable supportive tools in orthodontics remains limited at this stage. GPTs may contribute more reliably to orthodontic practice only when orthodontists prompt the model with text-based cephalometric measurement data.
Clinical trial number
Not applicable.
Supplementary information
The online version contains supplementary material available at https://doi.org/10.1186/s12911-026-03587-x.
Keywords: Large language model, Generative pre-trained transformer, Artificial intelligence, AIDTI (Artificial Intelligence Based Diagnosis and Treatment Planning Index), Cephalometric analysis
Introductıon
The treatment of orthodontic malocclusions involves three main stages: diagnosis, treatment planning, and treatment implementation [1]. By integrating lateral cephalograms (LCs) into the initial diagnostic phase, it has become possible to perform various analyses for the qualitative and quantitative assessment of malocclusions [1–3]. Due to its capacity for easy analysis of hard tissue structures, performance of measurements, and comparison through superimposition, cephalometric analysis has become the gold standard [1]. However, certain limitations in cephalometric diagnosis have also been highlighted. Proponents of this view argue that reliance on different measurement methodologies for cephalometric analyses can lead to divergent diagnoses, treatment plans, and outcomes from the same image. This complicates determining a single treatment option [1, 2, 4]. Accordingly, a study evaluating treatment plans formulated using different cephalometric analyses has reported only a 40% concordance rate [5].
The variability in treatment approaches resulting from different cephalometric analysis systems can complicate clinical decision-making. This poses significant challenges for clinicians, particularly those with limited clinical experience, in providing accurate diagnostic support and guiding patients toward appropriate treatment. Consequently, artificial intelligence (AI)-based applications show potential for assisting clinicians in diagnostic and treatment planning. A substantial portion of these AI technologies consists of machine learning and large language models (LLMs), which are increasingly being utilized across various fields, including dentistry [6].
Among the increasingly utilized LLMs in recent years, ChatGPT models (OpenAI, San Francisco, Calif) have ushered in a significant shift in the field of language AI, owing to their extensive databases and pattern recognition capabilities [7, 8]. GPTs are based on intelligent human-computer interactions and are designed to simulate conversations with human users over the internet [9]. These models can generate fluent and coherent texts, answer questions in dialogues, perform visual analyses, translate languages, and accomplish numerous other language-related tasks [6, 10]. However, the potential for GPTs to provide incorrect answers, generate nonsensical content, or present misinformation as fact raises substantial concerns in the healthcare domain [6].
Most existing assessment scales primarily use standardized scoring approaches and lack a structured, itemized framework that can comprehensively evaluate diagnostic processes and treatment planning. In an era where AI platforms are expected to contribute to clinical decision support systems, this limitation has become more apparent. Diagnostic processes involve the simultaneous assessment of multiple clinical and contextual parameters, creating a complex, multidimensional structure. Similarly, treatment planning cannot be adequately evaluated using a single criterion or limited numerical score because it requires multifaceted clinical appraisal. Consequently, relying on a single aggregate score derived from an isolated scale does not provide a robust or reliable clinical evaluation. These considerations underscore the critical need for a comprehensive, purpose-specific index, particularly in contemporary clinical settings where AI-based GPTs are being evaluated for diagnostic and treatment support applications [11–13]. In this context, the Artificial Intelligence Based Diagnosis and Treatment Index (AIDTI) was developed to evaluate not only the diagnostic accuracy of GPTs but also their capacity for differential diagnosis, the clinical appropriateness of proposed treatment plans, the disclosure of potential risks and complications, and the ability to provide alternative treatment options. In doing so, AIDTI addresses the limitations of existing scales and enables a more comprehensive and systematic evaluation of GPT outputs in clinical contexts.
The reliability and quality of web-based information are of paramount importance because they influence patient cooperation, adherence to treatment, communication with physicians, and overall doctor–patient trust [14]. When used for diagnostic purposes, unreliable information can lead to erroneous diagnoses. Therefore, GPT-mediated diagnostic support tools should be thoroughly investigated. The aim of this study was to compare the responses of AI-based GPTs to text- and image-based prompts from LCs of skeletal Class I, II, and III malocclusions using the newly developed AIDTI.
Material and methods
For this study, ethical approval was obtained from the Non-Interventional Clinical Research Ethics Committee of Zonguldak Bülent Ecevit University (approval number 2025/12–4, dated June 11, 2025). This study was conducted in accordance with the principles of the Declaration of Helsinki. The study utilized 90 LCs obtained from patients who presented to the Department of Orthodontics at the Faculty of Dentistry, Zonguldak Bülent Ecevit University for clinical examination. All LCs were anonymized prior to analysis by removing all personal identifiers. Lateral cephalometric radiographs used in this study were obtained during routine clinical practice using the Veraviewepocs 2D system (J. Morita Mfg. Corp., Kyoto, Japan) and were retrospectively retrieved from archival records. During image acquisition, it is assumed that patients were positioned in centric occlusion with relaxed lips and in a natural head position in accordance with the device’s standard protocol. The radiographic images were analyzed in JPEG format and exhibited variability in pixel dimensions due to their retrieval from clinical archives. Nevertheless, all images were evaluated under standardized viewing and assessment conditions to ensure consistency across analyses.
Skeletal classification was based on the ANB angle, with Class I defined as 0–4°, Class II as greater than 4°, and Class III as less than 0°. For each class, 30 LCs were selected. Class I LCs corresponded to individuals with normal maxillomandibular sagittal relationships, Class II LCs to patients with mandibular retrognathism, and Class III LCs to patients with maxillary retrognathism. The distribution of female and male patients was balanced across the groups. The mean and standard deviation values of the demographic characteristics and cephalometric measurements of the study groups are presented in Table 1.
Table 1.
Demographic characteristics and cephalometric measurements according to skeletal classifications (mean ± SD)
| Variables | Skeletal classifications | ||
|---|---|---|---|
| Class I (n = 30) | Class II (n = 30) | Class III (n = 30) | |
| Demographic variables | |||
| Female, n | 15 | 15 | 15 |
| Male, n | 15 | 15 | 15 |
| Age (years) | 16.30 ± 2.65 | 12.81 ± 1.63 | 11.68 ± 1.92 |
| Cephalometric variables | |||
| SNA (°) | 81.70 ± 1.22 | 82.03 ± 1.51 | 76.90 ± 2.21 |
| SNB (°) | 79.63 ± 1.28 | 75.50 ± 1.16 | 80.90 ± 1.96 |
| ANB (°) | 2.03 ± 1.30 | 6.47 ± 1.04 | −3.37 ± 2.32 |
| U1–NA (°) | 22.43 ± 6.24 | 33.87 ± 9.53 | 24.73 ± 6.24 |
| L1–NB (°) | 25.00 ± 7.05 | 25.13 ± 5.92 | 22.84 ± 6.45 |
| GoGn/SN (°) | 31.30 ± 4.28 | 30.30 ± 6.02 | 30.23 ± 5.14 |
| U1–NA (mm) | 5.28 ± 3.03 | 7.16 ± 2.66 | 5.34 ± 3.95 |
| L1–NB (mm) | 4.56 ± 2.36 | 4.49 ± 2.54 | 3.93 ± 1.80 |
SNA: sella–nasion–A point angle, SNB: sella–nasion–B point angle, ANB: Apoint–nasion–B point angle, U1–NA: angle between upper incisor and NA line, L1–NB: angle between lower incisor and NB line, GoGn/SN: angle between mandibular plane (GoGn) and SN plane, U1–NA (mm): linear distance from upper incisor to NA line, L1–NB (mm): linear distance from lower incisor to NB line. n: sample size, SD: standard deviation, °: degrees, mm: millimeters
The selection of the GPT models used in this study was based on their frequent evaluation in recent literature and their representation as widely used examples of large language models in clinical and dental research contexts [10, 15–17]. In the first phase of the study, all LCs were directly uploaded sequentially and randomly submitted to the ChatGPT-4o, ChatGPT-o3 Pro, ChatGPT-5, and ChatGPT-5 Pro platforms using the prompt: “Provide the malocclusion diagnosis and formulate an appropriate orthodontic treatment plan for the patient represented in this lateral cephalometric radiograph.” Following each radiographic evaluation, the chat session was reset, and after initiating a new instance, the subsequent radiograph was uploaded to the respective GPT using the same standardized prompt. This study utilized all models through publicly accessible web-based platform interfaces with their default settings. The generated outputs reflect each model’s platform-defined default inference behavior.
In the second phase of the study, digital cephalometric analyses of all LCs were performed by a single researcher (A.Y.) using NemoCeph software (Nemotec, Madrid, Spain). The SNA, SNB, ANB, U1/NA, L1/NB, and GoGn/SN angles, along with the linear distances of U1/NA and L1/NB, obtained from these analyses were provided as text-based prompts to each GPT with the prompt: “Provide the malocclusion diagnosis and formulate an appropriate orthodontic treatment plan for the patient whose cephalometric measurements are given.” After each evaluation, the chat session was reset, and following the initiation of a new instance, the skeletal measurements of the next LC were submitted to the respective model using the same standardized prompt.
For each prompt presented in the form of either a radiograph or cephalometric measurement data, the responses generated by the respective AI platforms were analyzed, and the malocclusion diagnoses provided as text outputs were categorized as Class I, Class II, or Class III. In all responses, the classification was explicitly stated by the platforms; therefore, no additional clarification or further prompting was required.
All prompts were delivered to the respective LLMs using the same laptop (MacBook Air M3, 16 GB RAM; Apple, Cupertino, Calif), a 4.5 G internet connection, and a virtual private network (VPN) server (version 3.9; Astrill Systems Corp, Santa Clara, Calif). Each text- and image-based prompt was submitted to every GPT only once. When a model failed to provide a response, no additional clarification was given, and no rephrasing of the answer was requested. Furthermore, the prompts were not repeated by any other researcher. All responses obtained from the GPTs were individually recorded in a Microsoft Excel (Microsoft 365 Apps, Microsoft Corporation, 2025) spreadsheet.
Development process of the Artificial Intelligence Based Diagnosis and Treatment Planning Index (AIDTI)
This study employed the Artificial Intelligence Based Diagnosis and Treatment Planning Index (AIDTI), a five-item evaluation tool developed by the researchers to assess the responses of AI-based LLM platforms to both direct image-based prompts involving LC uploads and text-based prompts containing cephalometric analysis measurements.
The index items were determined through content analysis conducted by a three-expert panel. A scoring system with a maximum of 10 points was established, where each item was rated using the options 0 (inadequate), 1 (partially adequate), and 2 (fully adequate). Validity and reliability analyses of the scale were performed using content validity index (CVI) [18], inter-rater reliability (Intraclass Correlation Coefficient, ICC) [19] and criterion validity (Pearson r) [20]. The content validity of the items was calculated based on evaluations by seven specialists, yielding I-CVI ≥ 0.78 for each item and S-CVI ≥ 0.90 at the scale level. Inter-rater reliability was assessed through independent scoring of the same 30 cases by two different experts and analyzed using the ICC(2,k) method, a two-way random-effects mean measurement model. The analysis results for both indices exceeded the acceptability threshold of ≥0.75, demonstrating a good level of agreement among the expert evaluations.
For criterion validity, two experts scored the responses of 30 cases using a classic 5-point Likert scale, and the relationship between these scores and those of the developed index was analyzed using Pearson correlation coefficients. The results exceeded the acceptance threshold of r ≥ 0.78, demonstrating a strong and statistically significant correlation between both scales and the reference criteria. Gold standard responses were established to enable the comparative analysis of GPTs’ competencies in clinical decision-making processes through the developed index. In this context, for each case, gold standard responses for diagnosis and treatment planning were independently developed by two expert clinicians based on a synthesis of current scientific literature and established clinical knowledge, and were subsequently finalized through consensus.
The responses generated by the GPTs were scored by two independent researchers using the developed AIDTI. In cases of disagreement between the evaluators, a third evaluator was consulted and a consensus score was established. Following the consensus procedure, a high level of agreement between the two evaluators was observed (ICC = 0.87). In addition, intra-examiner reliability was high (ICC = 0.83). Therefore, the scores obtained from a single evaluator were used as the reference scores. The evaluation criteria of the AIDTI and their corresponding descriptions are presented in Table 2.
Table 2.
Artificial intelligence based diagnosis and treatment planning index (AIDTI)
| Evaluation Criteria | Explanations of Index Items | |
|---|---|---|
| 1 | Diagnostic Accuracy | The extent to which the AI system’s diagnosis aligns with the actual clinical presentation and expert consensus. |
| 2 | Capability for Differential Diagnosis | The AI system’s ability to distinguish between similar clinical presentations, consider alternative diagnoses and justify exclusions. |
| 3 | Clinical Appropriateness of the Proposed Treatment Plan | Evaluates the relevance and suitability of the proposed treatment plan by the AI system for the given diagnosis and the patient’s overall clinical context. |
| 4 | Disclosure of Risks and Complications Associated with the Treatment Plan | Determines whether the AI system adequately informs about potential adverse effects, risks of failure, and other complications related to the recommended treatment. |
| 5 | Ability to Offer Alternative Treatment Options | Assesses the AI system’s capacity to present multiple, evidence-based alternative options suitable for the clinical scenario, rather than a single approach. |
Table 3 presents the ten-point rating system used to interpret the total AIDTI scores according to clinical quality levels [21].
Table 3.
Description of the AIDTI Ten-point scale used to evaluate the clinical quality of AI-generated diagnostic and therapeutic outputs
| Total Score | Verbal Label | Explanation |
|---|---|---|
| 0–2 | Very poor quality | The AI outputs are not clinically reliable and exhibit significant deficiencies. |
| 3–4 | Poor quality | The AI outputs demonstrate limited accuracy and appropriateness, with evident shortcomings. |
| 5–6 | Moderate quality | The AI outputs show partial accuracy and appropriateness; however, overall performance remains limited. |
| 7–8 | Good quality | The AI outputs are generally consistent and satisfactory, despite minor deficiencies. |
| 9–10 | Excellent quality | The AI outputs are clinically robust, demonstrating high accuracy and concordance. |
Statistical analysis
The obtained data were statistically analyzed using the Statistical Package for the Social Sciences (SPSS, version 26; IBM Corporation, New York, NY, USA). The normality of the data distribution was assessed using the Kolmogorov–Smirnov test. The Kruskal–Wallis test was used for comparisons between groups, while the Mann–Whitney U test was employed for pairwise comparisons. Within-group comparisons between text-based and image-based evaluations were conducted using the Wilcoxon signed-rank test. A significance level of p < 0.05 was considered statistically significant. Additionally, based on the text and image outputs of the models, the ability to accurately classify each malocclusion category was assessed. Accuracy = (TP + TN)/(TP + TN + FP + FN) (TP, true positives; TN, true negatives; FP, false positives; FN, false negatives). Sensitivity = TP/(TP + FN). Precision = TP/(TP + FP). Specificity = TN/(TN + FP). F1 score = 2 × (Precision × Sensitivity)/(Precision + Sensitivity).
Results
Results for the AIDTI
For diagnostic accuracy
In text-based prompts, the models’ mean scores ranged from 1.94 to 2.00, while in image-based prompts, the means ranged from 0.73 to 0.88. The Kruskal–Wallis test indicated no statistically significant differences between the models in either condition (text: p = 0.15; image: p = 0.716).
For capability for differential diagnosis
In text-based prompts, significant differences were observed between the models (p < 0.001), with GPT-5 pro achieving the highest mean score (1.90 ± 0.36) and GPT-4o the lowest (1.51 ± 0.56). In image-based prompts, the mean scores ranged from 0.40 to 0.67, with no statistically significant differences detected (p = 0.108).
Clinical appropriateness of the proposed treatment plan
In text-based prompts, the models’ mean scores ranged from 1.78 to 1.90, while in image-based prompts, the means ranged from 0.72 to 0.87. The Kruskal–Wallis test indicated no statistically significant differences between the models in text (p = 0.06) or image-based prompts (p = 0.664).
Disclosure of risks and complications associated with the treatment plan
In text-based prompts, significant differences were observed between the models (p < 0.001), with GPT-5 pro achieving the highest mean score (1.95 ± 0.20) and GPT-5 and GPT-4o the lowest (0.57 ± 0.82 and 0.60 ± 0.69, respectively). Significant differences were also found in image-based prompts (p < 0.001), where GPT-o3 pro achieved the highest mean score (1.05 ± 0.94) and GPT-5 the lowest (0.03 ± 0.18).
Ability to offer alternative treatment options
In text-based prompts, the mean scores ranged from 1.87 to 1.97, with no statistically significant differences observed between the models (p = 0.149). In image-based prompts, the means ranged from 0.64 to 0.87, also showing no significant differences (p = 0.32).
Total AIDTI score
Significant differences were observed in text-based prompts (p < 0.001), with GPT-5 pro achieving the highest total score (9.62 ± 1.13) and GPT-4o the lowest (7.83 ± 1.26). Significant differences were also found in image-based prompts (p = 0.007), where GPT-o3 pro obtained the highest mean score (4.16 ± 4.12) and GPT-5 the lowest (2.53 ± 3.19). Figure 1 illustrates the distribution of response quality levels based on total AIDTI scores for text and image prompts across the evaluated GPT models.
Fig. 1.
Distribution of response quality categories based on total AIDTI scores for text-based and image-based prompts across GPT-4o, GPT-o3 pro, GPT-5, and GPT-5 pro models. The bars represent the number of responses classified as excellent, good, moderate, poor, and very poor quality according to predefined AIDTI scale ranges
Results for the performance of GPT models
The detailed statistical analysis of all parameters, including means, standard deviations, and significance levels for text- and image-based prompts across all GPT models, is presented in Table 4. Additionally, a clustered three-dimensional column chart showing the mean scores of different GPT models across all parameters and prompt types is presented in Fig. 2. Representative examples of the responses generated by the GPT-o3 pro model to prompts based on cephalometric measurements and lateral cephalometric radiographs are presented in Supplementary Figures 1 and 2.
Table 4.
Statistical results of GPT models’ scored responses to all text and image prompts
| Parameters | Prompt Type | GPT models | p | |||
|---|---|---|---|---|---|---|
| GPT-4o a | GPT-o3 pro b | GPT-5 c | GPT-5 pro d | |||
| Mean ± SD | Mean ± SD | Mean ± SD | Mean ± SD | |||
| Diagnostic Accuracy | For text | 1.94 ± 0.23 | 1.97 ± 0.14 | 2 ± 0 | 1.95 ± 0.25 | 0.15K |
| For image | 0.88 ± 0.94 | 0.83 ± 0.96 | 0.73 ± 0.93 | 0.81 ± 0.95 | 0.716K | |
| p | <0.001* W | <0.001* W | <0.001* W | <0.001* W | ||
| Capability for Differential Diagnosis | For text | 1.51 ± 0.56b,d | 1.77 ± 0.44d | 1.65 ± 0.5d | 1.9 ± 0.36 | <0.001* K |
| For image | 0.67 ± 0.81 | 0.52 ± 0.73 | 0.4 ± 0.68 | 0.52 ± 0.72 | 0.108K | |
| p | <0.001* W | <0.001* W | <0.001* W | <0.001* W | ||
| Clinical Appropriateness of the Proposed Treatment Plan | For text | 1.9 ± 0.33 | 1.78 ± 0.41 | 1.88 ± 0.31 | 1.88 ± 0.34 | 0.06 K |
| For image | 0.84 ± 0.89 | 0.87 ± 0.93 | 0.72 ± 0.88 | 0.77 ± 0.92 | 0.664 K | |
| p | <0.001* W | <0.001* W | <0.001* W | <0.001* W | ||
| Disclosure of Risks and Complications Associated with the Treatment Plan | For text | 0.6 ± 0.69b,d | 1.54 ± 0.62c,d | 0.57 ± 0.82d | 1.95 ± 0.2 | <0.001* K |
| For image | 0.22 ± 0.57b,c | 1.05 ± 0.94c,d | 0.03 ± 0.18d | 0.16 ± 0.47 | <0.001* K | |
| p | <0.001* W | <0.001* W | <0.001* W | <0.001* W | ||
| Ability to Offer Alternative Treatment Options | For text | 1.87 ± 0.39 | 1.97 ± 1.14 | 1.87 ± 0.39 | 1.92 ± 0.26 | 0.149 K |
| For image | 0.81 ± 0.91 | 0.87 ± 0.94 | 0.64 ± 0.91 | 0.74 ± 0.93 | 0.32 K | |
| p | <0.001* W | <0.001* W | <0.001* W | <0.001* W | ||
| Total AIDTI score | For text | 7.83 ± 1.26b,d | 9.06 ± 0.99c,d | 8 ± 1.19d | 9.62 ± 1.13 | <0.001* K |
| For image | 3.44 ± 3.65 | 4.16 ± 4.12c,d | 2.53 ± 3.19 | 3.02 ± 3.57 | 0.007* K | |
| p | <0.001* W | <0.001* W | <0.001* W | <0.001* W | ||
K: Kruskal–Wallis (Mann–Whitney U test), W: Wilcoxon test, SD: Standard deviation, p: significance value, *: p < 0.05
a: Difference with GPT-4o p < 0.05, b: Difference with GPT-o3 pro p < 0.05, c: Difference with GPT-5 p < 0.05, d: Difference with GPT-5 pro p < 0.05
Fig. 2.
Clustered 3D column chart illustrating the mean scores of GPT-4o, GPT-o3 pro, GPT-5, and GPT-5 pro across all evaluation parameters for text- and image-based prompts
As a result of the analyses related to malocclusion classification performance, in the Class I malocclusion group, the ‘pro’ models achieved higher total scores in the text-based analysis. For image-based total scores, GPT-4o and GPT-o3 pro received higher scores compared to the other models. In the Class II group, GPT-5 pro achieved the highest mean score for text-based total performance, followed by GPT-o3 pro. For image-based total scores, GPT-o3 pro ranked first, followed by GPT-4o, although the difference between these two models did not reach statistical significance. In the Class III group, GPT-5 pro obtained the highest score in the text-based total, followed by GPT-o3 pro. However, no statistically significant difference was found among the models in terms of image-based total scores. The statistical analysis results regarding within-group and between-group comparisons of text-based and image-based scores for each malocclusion group across GPT models are presented in Table 5.
Table 5.
Comparison of GPT models’ scored responses for text- and image-based prompts in each malocclusion
| For Class I | Prompt Type | GPT models | p | |||
|---|---|---|---|---|---|---|
| GPT-4o | GPT-o3 pro b | GPT-5 c | GPT-5 pro d | |||
| Mean ± SD | Mean ± SD | Mean ± SD | Mean ± SD | |||
| Diagnostic Accuracy | For text | 1.96 ± 0.18 | 2 ± 0 | 2 ± 0 | 1.86 ± 0.43 | 0.103 |
| For image | 0.66 ± 0.75c,d | 0.36 ± 0.66 | 0.16 ± 0.46 | 0.1 ± 0.3 | 0.001* K | |
| p | <0.001* W | <0.001* W | <0.001* W | <0.001* W | ||
| Capability for Differential Diagnosis | For text | 1.43 ± 0.62b,d | 1.86 ± 0.43 | 1.7 ± 0.46 | 1.73 ± 0.58 | 0.007* K |
| For image | 0.43 ± 0.62c,d | 0.2 ± 0.55 | 0.1 ± 0.4 | 0.13 ± 0.43 | 0.011* K | |
| p | <0.001* W | <0.001* W | <0.001* W | <0.001* W | ||
| Clinical Appropriateness of the Proposed Treatment Plan | For text | 1.7 ± 0.53 | 1.66 ± 0.47 | 1.8 ± 0.4 | 1.7 ± 0.53 | 0.726K |
| For image | 0.63 ± 0.66c,d | 0.53 ± 0.68d | 0.23 ± 0.43 | 0.2 ± 0.55 | 0.005* K | |
| p | <0.001* W | <0.001* W | <0.001* W | <0.001* W | ||
| Disclosure of Risks and Complications Associated with the Treatment Plan | For text | 1.03 ± 0.76b,c,d | 1.5 ± 0.62c,d | 0.5 ± 0.77d | 1.9 ± 0.3 | <0.001* K |
| For image | 0.26 ± 0.63b | 1.33 ± 0.84c,d | 0.03 ± 0.18 | 0.03 ± 0.18 | <0.001* K | |
| p | <0.001* W | <0.001* W | <0.001* W | <0.001* W | ||
| Ability to Offer Alternative Treatment Options | For text | 1.63 ± 0.61 | 1.93 ± 0.25 | 1.8 ± 0.48 | 1.83 ± 0.37 | 0.114K |
| For image | 0.5 ± 0.62c,d | 0.5 ± 0.68c,d | 0.1 ± 0.4 | 0.13 ± 0.46 | 0.001* K | |
| p | <0.001* W | <0.001* W | <0.001* W | <0.001* W | ||
| Total AIDTI score | For text | 7.76 ± 1.88b,d | 8.96 ± 1.06c | 7.8 ± 1.24d | 9.03 ± 1.79 | <0.001* K |
| For image | 2.5 ± 2.75c,d | 2.93 ± 2.74c,d | 0.63 ± 1.21 | 0.6 ± 1.56 | <0.001* K | |
| p | <0.001* W | <0.001* W | <0.001* W | <0.001* W | ||
| For Class II | ||||||
| Diagnostic Accuracy | For text | 1.93 ± 0.25 | 1.93 ± 0.25 | 2 ± 0 | 2 ± 0 | 0.251K |
| For image | 1.2 ± 0.99 | 1.26 ± 0.98 | 1.1 ± 0.95 | 1.2 ± 0.96 | 0.904K | |
| p | 0.001* W | 0.002* W | <0.001* W | 0.001* W | ||
| Capability for Differential Diagnosis | For text | 1.6 ± 0.56d | 1.66 ± 0.47d | 1.6 ± 0.56d | 2 ± 0 | 0.002* K |
| For image | 1.06 ± 0.94 | 0.83 ± 0.79 | 0.66 ± 0.84 | 0.73 ± 0.78 | 0.323K | |
| p | 0.008* | <0.001* W | <0.001* W | <0.001* W | ||
| Clinical Appropriateness of the Proposed Treatment Plan | For text | 2 ± 0b,c | 1.76 ± 0.43d | 1.86 ± 0.34 | 1.96 ± 0.18 | 0.012* K |
| For image | 1.16 ± 0.98 | 1.23 ± 0.97 | 1.1 ± 0.95 | 1.1 ± 0.92 | 0.918K | |
| p | <0.001* W | 0.008* W | 0.001* W | <0.001* W | ||
| Disclosure of Risks and Complications Associated with the Treatment Plan | For text | 0.3 ± 0.46b,d | 1.43 ± 0.67c,d | 0.6 ± 0.85d | 1.96 ± 0.18 | <0.001* K |
| For image | 0.26 ± 0.63b | 1.16 ± 0.94c,d | 0.03 ± 0.18d | 0.33 ± 0.66 | <0.001* K | |
| p | 0.796W | 0.175W | 0.004* W | <0.001* W | ||
| Ability to Offer Alternative Treatment Options | For text | 2 ± 0 | 2 ± 0 | 1.9 ± 0.4 | 2 ± 0 | 0.109K |
| For image | 1.16 ± 0.98 | 1.26 ± 0.98 | 1.13 ± 0.97 | 1.13 ± 0.93 | 0.921K | |
| p | <0.001* W | 0.001* W | 0.002* W | <0.001* W | ||
| Total AIDTI score | For text | 7.83 ± 0.79b,d | 8.8 ± 1.09c,d | 7.96 ± 1.27d | 9.93 ± 0.36 | <0.001* K |
| For image | 4.86 ± 4 | 5.76 ± 4.5c,d | 4.03 ± 3.56 | 4.5 ± 3.74 | 0.049* K | |
| p | 0.002* W | 0.007* W | <0.001* W | <0.001* W | ||
| For Class III | ||||||
| Diagnostic Accuracy | For text | 1.93 ± 0.25 | 2 ± 0 | 2 ± 0 | 2 ± 0 | 0.109K |
| For image | 0.8 ± 0.99 | 0.86 ± 1 | 0.93 ± 1.01 | 1.13 ± 1 | 0.602K | |
| p | <0.001* W | <0.001* W | <0.001* W | <0.001* W | ||
| Capability for Differential Diagnosis | For text | 1.5 ± 0.5b,d | 1.8 ± 0.4d | 1.66 ± 0.47d | 1.96 ± 0.18 | <0.001* K |
| For image | 0.53 ± 0.73 | 0.53 ± 0.73 | 0.43 ± 0.62 | 0.7 ± 0.74 | 0.536K | |
| p | <0.001* W | <0.001* W | <0.001* W | <0.001* W | ||
| Clinical Appropriateness of the Proposed Treatment Plan | For text | 2 ± 0 | 1.93 ± 0.25 | 2 ± 0 | 2 ± 0 | 0.109K |
| For image | 0.73 ± 0.94 | 0.86 ± 1 | 0.83 ± 0.94 | 1.03 ± 0.96 | 0.671K | |
| p | <0.001* W | <0.001* W | <0.001* W | <0.001* W | ||
| Disclosure of Risks and Complications Associated with the Treatment Plan | For text | 0.46 ± 0.62b,d | 1.7 ± 0.53c,d | 0.63 ± 0.85d | 2 ± 0 | <0.001* K |
| For image | 0.13 ± 0.43b | 0.66 ± 0.92c,d | 0.03 ± 0.18 | 0.13 ± 0.43 | 0.001* K | |
| p | <0.001* W | <0.001* W | <0.001* W | <0.001* W | ||
| Ability to Offer Alternative Treatment Options | For text | 2 ± 0 | 2 ± 0 | 1.93 ± 0.25 | 1.93 ± 0.25 | 0.251K |
| For image | 0.76 ± 0.97 | 0.86 ± 1 | 0.7 ± 0.95 | 0.96 ± 0.99 | 0.736K | |
| p | <0.001* W | <0.001* W | <0.001* W | <0.001* W | ||
| Total AIDTI score | For text | 7.9 ± 0.84b,d | 9.43 ± 0.67c,d | 8.23 ± 1.07d | 9.9 ± 0.3 | <0.001* K |
| For image | 2.96 ± 3.74 | 3.8 ± 4.47 | 2.93 ± 3.3 | 3.96 ± 3.66 | 0.497K | |
| p | <0.001* W | <0.001* W | <0.001* W | <0.001* W | ||
K: Kruskal–Wallis (Mann–Whitney U test), W: Wilcoxon test, SD: Standard deviation, p: significance value, *: p < 0.05
a: Difference with GPT-4o p < 0.05, b: Difference with GPT-o3 pro p < 0.05, c: Difference with GPT-5 p < 0.05, d: Difference with GPT-5 pro p < 0.05
In the image-based analyses, although all GPTs achieved higher total scores in the Class II group, the difference did not reach statistical significance. On the other hand, in the Class I group, the total scores of all models in the image-based analyses were found to be lower, and this difference reached statistical significance for the GPT-5 and GPT-5 Pro models. The statistical analysis results for both between-group and within-group comparisons of text-based and image-based scores for each malocclusion type across all GPT models are presented in Table 6.
Table 6.
Comparison of GPT models’ scored responses between malocclusions using text- and image-based prompts
| Prompt Type | Malocclusion Groups | p | |||
|---|---|---|---|---|---|
| Class I a | Class II b | Class III c | |||
| Mean ± SD | Mean ± SD | Mean ± SD | |||
| For GPT-4o | |||||
| Diagnostic Accuracy | For text | 1.96 ± 0.18 | 1.93 ± 0.25 | 1.93 ± 0.25 | 0.811K |
| For image | 0.66 ± 0.75 | 1.2 ± 0.99 | 0.8 ± 0.99 | 0.094K | |
| p | <0.001* W | 0.001* W | <0.001* W | ||
| Capability for Differential Diagnosis | For text | 1.43 ± 0.62 | 1.6 ± 0.56 | 1.5 ± 0.5 | 0.510K |
| For image | 0.43 ± 0.62 | 1.06 ± 0.94a,c | 0.53 ± 0.73 | 0.015* K | |
| p | <0.001* W | 0.008* W | <0.001* W | ||
| Clinical Appropriateness of the Proposed Treatment Plan | For text | 1.7 ± 0.53b,c | 2 ± 0 | 2 ± 0 | <0.001* K |
| For image | 0.63 ± 0.66 | 1.16 ± 0.98 | 0.73 ± 0.94 | 0.078K | |
| p | <0.001* W | <0.001* W | <0.001* W | ||
| Disclosure of Risks and Complications Associated with the Treatment Plan | For text | 1.03 ± 0.76b,c | 0.3 ± 0.46 | 0.46 ± 0.62 | <0.001* K |
| For image | 0.26 ± 0.63 | 0.26 ± 0.63 | 0.13 ± 0.43 | 0.669K | |
| p | 0.001* W | 0.796W | 0.046* W | ||
| Ability to Offer Alternative Treatment Options | For text | 1.63 ± 0.61b,c | 2 ± 0 | 2 ± 0 | <0.001* K |
| For image | 0.5 ± 0.62b | 1.16 ± 0.98 | 0.76 ± 0.97 | 0.032* K | |
| p | <0.001* W | <0.001* W | <0.001* W | ||
| Total AIDTI score | For text | 7.76 ± 1.88 | 7.83 ± 0.79 | 7.9 ± 0.84 | 0.867K |
| For image | 2.5 ± 2.75 | 4.86 ± 4 | 2.96 ± 3.74 | 0.069K | |
| p | <0.001* W | 0.002* W | <0.001* W | ||
| For GPT-o3 pro | |||||
| Diagnostic Accuracy | For text | 2 ± 0 | 1.93 ± 0.25 | 2 ± 0 | 0.132K |
| For image | 0.36 ± 0.66b | 1.26 ± 0.98 | 0.86 ± 1 | 0.002* K | |
| p | <0.001* W | 0.002* W | <0.001* W | ||
| Capability for Differential Diagnosis | For text | 1.86 ± 0.43 | 1.66 ± 0.47 | 1.8 ± 0.4 | 0.106K |
| For image | 0.2 ± 0.55b,c | 0.83 ± 0.79 | 0.53 ± 0.73 | 0.002* K | |
| p | <0.001* W | <0.001* W | <0.001* W | ||
| Clinical Appropriateness of the Proposed Treatment Plan | For text | 1.66 ± 0.47c | 1.76 ± 0.43 | 1.93 ± 0.25 | 0.039* K |
| For image | 0.53 ± 0.68 | 1.23 ± 0.97a | 0.86 ± 1 | 0.022* K | |
| p | <0.001* W | 0.008* W | <0.001* W | ||
| Disclosure of Risks and Complications Associated with the Treatment Plan | For text | 1.5 ± 0.62 | 1.43 ± 0.67 | 1.7 ± 0.53 | 0.219K |
| For image | 1.33 ± 0.84c | 1.16 ± 0.94c | 0.66 ± 0.92 | 0.018* K | |
| p | 0.325W | 0.175W | <0.001* W | ||
| Ability to Offer Alternative Treatment Options | For text | 1.93 ± 0.25 | 2 ± 0 | 2 ± 0 | 0.132K |
| For image | 0.5 ± 0.68b | 1.26 ± 0.98 | 0.86 ± 1 | 0.011* K | |
| p | <0.001* W | 0.001* W | <0.001* W | ||
| Total AIDTI score | For text | 8.96 ± 1.06 | 8.8 ± 1.09 | 9.43 ± 0.67 | 0.056K |
| For image | 2.93 ± 2.74 | 5.76 ± 4.5 | 3.8 ± 4.47 | 0.119K | |
| p | <0.001* W | 0.007* W | <0.001* W | ||
| For GPT-5 | |||||
| Diagnostic Accuracy | For text | 2 ± 0 | 2 ± 0 | 2 ± 0 | 1.000K |
| For image | 0.16 ± 0.46b,c | 1.1 ± 0.95 | 0.93 ± 1.01 | <0.001* K | |
| p | <0.001* W | <0.001* W | <0.001* W | ||
| Capability for Differential Diagnosis | For text | 1.7 ± 0.46 | 1.6 ± 0.56 | 1.66 ± 0.47 | 0.820K |
| For image | 0.1 ± 0.4b,c | 0.66 ± 0.84 | 0.43 ± 0.62 | 0.004* K | |
| p | <0.001* W | <0.001* W | <0.001* W | ||
| Clinical Appropriateness of the Proposed Treatment Plan | For text | 1.8 ± 0.4 | 1.86 ± 0.34 | 2 ± 0a,b | 0.044* K |
| For image | 0.23 ± 0.43b,c | 1.1 ± 0.95 | 0.83 ± 0.94 | 0.001* K | |
| p | <0.001* W | 0.001* W | <0.001* W | ||
| Disclosure of Risks and Complications Associated with the Treatment Plan | For text | 0.5 ± 0.77 | 0.6 ± 0.85 | 0.63 ± 0.85 | 0.830K |
| For image | 0.03 ± 0.18 | 0.03 ± 0.18 | 0.03 ± 0.18 | 1.000K | |
| p | 0.007* W | 0.004* W | 0.003* W | ||
| Ability to Offer Alternative Treatment Options | For text | 1.8 ± 0.48 | 1.9 ± 0.4 | 1.93 ± 0.25 | 0.338K |
| For image | 0.1 ± 0.4b,c | 1.13 ± 0.97 | 0.7 ± 0.95 | <0.001* K | |
| p | <0.001* W | 0.002* W | <0.001* W | ||
| Total AIDTI score | For text | 7.8 ± 1.24 | 7.96 ± 1.27 | 8.23 ± 1.07 | 0.606K |
| For image | 0.63 ± 1.21b,c | 4.03 ± 3.56 | 2.93 ± 3.3 | 0.002* K | |
| p | <0.001* W | <0.001* W | <0.001* W | ||
| For GPT-5 pro | |||||
| Diagnostic Accuracy | For text | 1.86 ± 0.43 | 2 ± 0 | 2 ± 0 | 0.065K |
| For image | 0.1 ± 0.3b,c | 1.2 ± 0.96 | 1.13 ± 1 | <0.001* K | |
| p | <0.001* W | 0.001* W | <0.001* W | ||
| Capability for Differential Diagnosis | For text | 1.73 ± 0.58b,c | 2 ± 0 | 1.96 ± 0.18 | 0.008* K |
| For image | 0.13 ± 0.43b,c | 0.73 ± 0.78 | 0.7 ± 0.74 | 0.001* K | |
| p | <0.001* W | <0.001* W | <0.001* W | ||
| Clinical Appropriateness of the Proposed Treatment Plan | For text | 1.7 ± 0.53b,c | 1.96 ± 0.18 | 2 ± 0 | 0.001* K |
| For image | 0.2 ± 0.55b,c | 1.1 ± 0.92 | 1.03 ± 0.96 | <0.001* K | |
| p | <0.001* W | <0.001* W | <0.001* W | ||
| Disclosure of Risks and Complications Associated with the Treatment Plan | For text | 1.9 ± 0.3 | 1.96 ± 0.18 | 2 ± 0 | 0.163K |
| For image | 0.03 ± 0.18 | 0.33 ± 0.66 | 0.13 ± 0.43 | 0.052K | |
| p | <0.001* W | <0.001* W | <0.001* W | ||
| Ability to Offer Alternative Treatment Options | For text | 1.83 ± 0.37 | 2 ± 0 | 1.93 ± 0.25 | 0.054K |
| For image | 0.13 ± 0.43b,c | 1.13 ± 0.93 | 0.96 ± 0.99 | <0.001* K | |
| p | <0.001* W | <0.001* W | <0.001* W | ||
| Total AIDTI score | For text | 9.03 ± 1.79b,c | 9.93 ± 0.36 | 9.9 ± 0.3 | 0.001* K |
| For image | 0.6 ± 1.56b,c | 4.5 ± 3.74 | 3.96 ± 3.66 | <0.001* K | |
| p | <0.001* W | <0.001* W | <0.001* W | ||
K: Kruskal–Wallis (Mann–Whitney U test), W: Wilcoxon test, SD: Standard deviation, p: significance value, *: p < 0.05
a: Difference with Class I p < 0.05, b: Difference with Class II p < 0.05, c: Difference with Class III p < 0.05
Based on the text and image outputs of the models, key performance metrics including sensitivity, specificity, accuracy, precision, and F1 score were calculated according to the correct and incorrect classifications within Class I, II, and III malocclusion groups. These results are presented in Table 7. While the text-based performance parameters yielded notably high values, the image-based performance parameters were found to be low, and consistent results could not be achieved.
Table 7.
Performance metrics of GPT models for Class I, II, and III malocclusion classification based on text and image-based prompts
| Accuracy | Malocclusion | Sensitivity | Specificity | Precision | F1 Score | |
|---|---|---|---|---|---|---|
| Response for text | ||||||
| GPT-4o | 0.96 | Class I | 0.93 | 0.97 | 0.93 | 0.93 |
| Class II | 0.93 | 0.97 | 0.93 | 0.93 | ||
| Class III | 0.97 | 0.98 | 0.97 | 0.97 | ||
| Overall | 0.94 | 0.97 | 0.94 | 0.94 | ||
| GPT-o3 pro | 0.99 | Class I | 0.93 | 1.00 | 1.00 | 0.97 |
| Class II | 1.00 | 1.00 | 1.00 | 1.00 | ||
| Class III | 1.00 | 1.00 | 1.00 | 1.00 | ||
| Overall | 0.98 | 1.00 | 1.00 | 0.99 | ||
| GPT-5 | 1.00 | Class I | 1.00 | 1.00 | 1.00 | 1.00 |
| Class II | 1.00 | 1.00 | 1.00 | 1.00 | ||
| Class III | 1.00 | 1.00 | 1.00 | 1.00 | ||
| Overall | 1.00 | 1.00 | 1.00 | 1.00 | ||
| GPT-5 pro | 0.98 | Class I | 1.00 | 0.98 | 0.97 | 0.98 |
| Class II | 1.00 | 0.97 | 0.94 | 0.97 | ||
| Class III | 0.90 | 1.00 | 1.00 | 0.95 | ||
| Overall | 0.97 | 0.99 | 0.97 | 0.97 | ||
| Response for image | ||||||
| GPT-4o | 0.333 | Class I | 0.00 | 1.00 | 0.00 | 0.00 |
| Class II | 0.97 | 0.03 | 0.33 | 0.50 | ||
| Class III | 0.03 | 0.97 | 0.33 | 0.06 | ||
| Overall | 0.333 | 0.667 | 0.222 | 0.185 | ||
| GPT-o3 pro | 0.322 | Class I | 0.00 | 0.98 | 0.00 | 0.00 |
| Class II | 0.97 | 0.00 | 0.33 | 0.49 | ||
| Class III | 0.00 | 1.00 | 0.00 | 0.00 | ||
| Overall | 0.322 | 0.661 | 0.109 | 0.162 | ||
| GPT-5 | 0.333 | Class I | 0.00 | 1.00 | 0.00 | 0.00 |
| Class II | 0.97 | 0.02 | 0.33 | 0.49 | ||
| Class III | 0.03 | 0.98 | 0.50 | 0.06 | ||
| Overall | 0.333 | 0.667 | 0.277 | 0.185 | ||
| GPT-5 pro | 0.389 | Class I | 0.00 | 1.00 | 0.00 | 0.00 |
| Class II | 1.00 | 0.10 | 0.36 | 0.53 | ||
| Class III | 0.17 | 0.98 | 0.83 | 0.28 | ||
| Overall | 0.389 | 0.694 | 0.397 | 0.268 |
Discussion
Various AI-assisted systems have been developed for orthodontic diagnosis and treatment planning. In this context, Li and Wang [22] developed a system that combines multi-task reinforcement learning and explainable AI to provide personalized and data-driven recommendations for orthodontic treatment planning. Although this system demonstrated high accuracy, its technical infrastructure requirements and need for user training have limited its clinical integration. Therefore, this study evaluated the competency of more accessible GPTs that require no technical setup and can process both text- and image-based prompts in orthodontic clinical scenarios.
In the field of orthodontics, LLMs have been able to provide accurate responses to questions involving specific topics such as clear aligners, temporary anchorage devices, and digital imaging [23], and have demonstrated moderate success in assessing bone age directly from hand-wrist radiographs without a text-based prompt [15]. These studies in the literature indicate that the models’ potential is not limited to information transfer but also extends to functions such as diagnosis and treatment planning. In this context, Rakhmatulla et al. [24] tested LLMs on virtual orthodontic cases via an API and reported that the models showed significant progress in diagnostic and treatment planning. However, since the model evaluations were conducted through direct programming rather than via the user-facing web interface, the obtained results may have limited applicability in reflecting real-world usage conditions.
Diagnosis and treatment planning are processes that require accurate interpretation of the clinical context and consideration of patient-specific variables [25]. In order to reliably assess the competencies of LLMs in these areas, studies should be conducted under conditions that align with clinical scenarios and utilize real user interfaces [26]. For such multidimensional analyses, it is necessary to develop a comprehensive and systematic evaluation tool. The AIDTI, developed for this purpose, enables the assessment of GPT performance using more realistic criteria. The index addresses overly simplified or context-insensitive inferences and clinically inappropriate treatment recommendations [6, 26], which are frequently highlighted in the literature, within a broader evaluative framework.
Recent studies have demonstrated the utility of LLMs in risk assessment, diagnosis, anomaly detection, appointment scheduling, and scientific research [27]. Within this scope, LLMs have been evaluated across various clinical domains including restorative treatment planning [28], oral lesion identification [29], and implantology support [30]. Similarly, in orthodontics, model performance has been assessed through responses to questions derived from clinical scenarios [17].
Beyond text-based research, the multimodal reasoning capacity of LLMs has also led to an increase in the number of analyses performed directly on radiographic or clinical images [15, 31, 32]. In a study evaluating oral mucosa lesions from photographs using ChatGPT-4o, Suárez et al. [32] reported a diagnostic accuracy of 58.2%, but noted high accuracy in treatment recommendations for correctly diagnosed cases. Hasei et al. [31], in their study using questions from the Japanese Orthopaedic Association specialization exam, compared the diagnostic accuracy of GPT-4o and o1 pro, reporting that GPT o1 pro achieved higher accuracy, particularly on image-based questions. Parallel to these findings, the present study demonstrated that the ‘pro’ models exhibited better performance in both image-based and text-based prompts. Another noteworthy finding was that GPT-o3 pro outperformed GPT-5 pro in image-based prompts. This result indicates that newer models do not always yield better outcomes and that model performance is not solely dependent on version updates. It is hypothesized that earlier versions, potentially exposed to broader datasets for longer durations, might have an advantage in pattern recognition and inference capabilities.
In the AIDTI evaluation, the high scores obtained by models when using text-based inputs may create a misleading perception of success. Prompts in which cephalometric measurement data acquired through conventional methods are presented in a text-based format may give the impression that GPTs possess the capability to establish diagnoses or formulate treatment plans, as they are able to accurately and consistently reproduce literature-based and textbook knowledge contained within their training data. However, this apparent performance should not be interpreted beyond the models’ capacity to present information and should be approached with caution. Indeed, the present study clearly demonstrates that GPTs are unable to directly extract measurement data by interpreting cephalometric radiographs. Moreover, diagnosis and treatment planning are not limited to the interpretation of a subset of cephalometric measurements, as applied in the present study, nor can they be reliably performed based solely on cephalometric evaluations. The processes of diagnosis and treatment planning are inherently complex and require the comprehensive and integrated assessment of numerous clinical parameters.
LLMs process LCs, which are composed of thousands of pixels with varying brightness levels, at the pixel level by converting them into high-dimensional embeddings and transforming these digitally encoded representations into computational language to generate verbal responses [33, 34]. During this process, attention mechanisms may focus on prominent deviations, leading to their overrepresentation while causing normative structures to remain less distinct within visual representations. Indeed, when evaluating GPT performance in malocclusion classification, all four models produced lower total scores for image-based prompts of skeletal Class I cases compared to Class II and Class III. This finding suggests that the models are prone to detecting orthodontic anomalies in image-based inputs and are less capable of adequately distinguishing Class I relationships, which lack significant sagittal skeletal discrepancies. In line with this observation, Grinberg et al. [35] similarly reported in their study evaluating ChatGPT-4’s differential diagnosis competency for oral mucosal lesions based on text-based inputs that the model exhibited higher sensitivity in detecting malignant lesions.
Regarding model performance in malocclusion classification, all models achieved higher total scores for image-based prompts in the Class II group. However, the applied performance metrics analysis revealed that this did not reflect true classification success. Extremely low specificity values observed in the Class II group indicate that the models tended to label the majority of radiographs as Class II, creating an artificial impression of success in this group. Therefore, evaluations based solely on total scores or a single metric may overlook such erroneous tendencies and lead to misleading conclusions [36]. In this context, model performance should be assessed from a multidimensional perspective, and success should be interpreted based on consistent results derived from multiple performance metrics and statistical analyses.
CNN-based AI models have demonstrated successful outcomes in cephalometric landmark detection and diagnostic assessment in orthodontics. The literature reports that models such as YOLOv3 and multi-stage convolutional neural networks operate with low error rates, comparable to inter-examiner variability [37–39]. In addition, AI systems have been shown to outperform human evaluators in certain challenging landmark identifications [37, 38]. Comparative studies further indicate that AI systems can achieve diagnostic accuracy comparable to orthodontists, while semi-automated approaches significantly reduce analysis time [40]. The findings of the present study suggest that, although LLMs have the potential to provide useful outcomes in this domain, they have not yet demonstrated the same level of consistent and reliable performance as CNN-based models. Furthermore, considering the relatively high access costs of certain models and the energy demands associated with LLMs [41], it is important to critically evaluate whether the marginal benefits they provide justify these costs in all contexts.
Visual data are typically represented using Contrastive Language–Image Pretraining (CLIP)-based Vision Transformer (ViT) encoders. However, examination of current results indicates that these encoders are not semantically sufficient in medical contexts [42]. To improve visual analysis performance, training with higher-resolution and contextually rich images [43], using advanced encoders such as DINOv2, SigLIP, or CLIP-ConvNeXt, and employing projection layers like Q-former or Perceiver Resampler instead of simple linear projections for transferring visual information to the language model should be considered. Additionally, techniques such as Instruction Tuning and Direct Preference Optimization during training can enhance the transfer of visual context into the language model. When applied in combination with Multimodal In-Context Learning strategies, these approaches may increase the influence of visual inputs on model outputs and support more reliable clinical analyses [44].
The limitations of this study include the evaluation of GPT performance using only selected cephalometric measurement data, without incorporating additional clinical information such as intraoral and extraoral photographs, digital model analyses, or three-dimensional imaging data. Furthermore, given the stochastic nature of LLM outputs, model responses may exhibit substantial variability across different runs, which should be taken into account when interpreting the results. Additionally, the GPTs evaluated were limited to specific model versions, and their performance may vary with newer or alternative architectures. Validation with larger, multi-center sample sizes is also required to improve the generalizability of the findings. Furthermore, because the AIDTI consists of only five items, relevant parameters such as patient satisfaction, treatment duration, and long-term clinical outcomes could not be assessed. In addition, the AIDTI assigns equal weights to all five subdimensions, which may not fully reflect their relative clinical importance in orthodontic decision-making. Future studies are recommended to directly compare LLM models with CNN-based models specifically developed for orthodontic applications in order to enable more comprehensive conclusions.
Conclusion
In text-based inputs, GPTs have the potential to convey information to a limited extent based on the data provided to them. However, this potential is evident only when numerically expressed cephalometric analysis data derived from LCs are directly supplied to the model. The reliable acquisition of such cephalometric measurement values requires that these analyses be performed by an orthodontist. Indeed, the findings of the present study demonstrate that LLMs are not capable of independently performing cephalometric analyses and lack the capacity for autonomous inference or clinical interpretation. Accordingly, they should not be regarded as independent tools for diagnosis or treatment planning.
Another important implication of this study is that the use of AIDTI enabled a systematic evaluation of the parameters assessed across the models. The multidimensional assessment framework provided by this index prevented performance evaluation from relying on a single metric and allowed for a more balanced and clinically relevant interpretation of the findings. Accordingly, AIDTI may serve as an appropriate evaluation index for future AI studies involving both text-based and image-based prompts.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Acknowledgements
The authors have no acknowledgements to declare.
Abbreviations
- AIDTI
Artificial Intelligence Based Diagnosis and Treatment Planning Index
- CVI
Content Validity Index
- FN
False Negatives
- FP
False Positives
- GPTs
Generative Pre-trained Transformers
- ICC
Intraclass Correlation Coefficient
- LCs
Lateral Cephalograms
- LLMs
Large Language Models
- TN
True Negatives
- TP
True Positives
- VPN
Virtual Private Network
Author contributions
Ahmet Yıldırım: Resources, Investigation, Validation, Writing - original draft, Conceptualization, Methodology, Writing - review & editing. Orhan Cicek: Software, Data curation, Formal analysis, Visualization, Writing - review & editing. Yavuz Selim Genç: Conceptualization, Methodology, Data curation, Investigation, Writing - review & editing. Ruşen Erdem: Conceptualization, Methodology, Data curation, Formal analysis, Investigation, Supervision.
Funding
Not applicable.
Data availability
The datasets analyzed during the current study are available from the corresponding author on reasonable request.
Declarations
Ethics approval and consent to participate
For this study, ethical approval was obtained from the Non-Interventional Clinical Research Ethics Committee of Zonguldak Bülent Ecevit University (approval number 2025/12–4, dated June 11, 2025). This study was conducted in accordance with the principles of the Declaration of Helsinki. Written informed consent was obtained from all patients for the use of retrospectively collected radiographs in scientific research following anonymization.
Consent for publication
Not applicable.
Competing interests
The authors declare no competing interests.
Footnotes
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.Arnett GW, Bergman RT. Facial keys to orthodontic diagnosis and treatment planning. Part I. Am J Orthod Dentofac Orthopedics. 1993;103(4):299–312. 10.1016/0889-5406(93)70010-L. [DOI] [PubMed] [Google Scholar]
- 2.Holdaway RA. A soft-tissue cephalometric analysis and its use in orthodontic treatment planning. Part I. Am J Orthod. 1983;84(1):1–28. 10.1016/0002-9416(83)90144-6. [DOI] [PubMed] [Google Scholar]
- 3.Ricketts RM. Esthetics, environment, and the law of lip relation. Am J Orthod. 1968;54(4):272–89. 10.1016/S0002-9416(68)90278-9. [DOI] [PubMed] [Google Scholar]
- 4.Jacobson A. Planning for orthognathic surgery:art or science? Int J Adult Orthodon Orthognath Surg. 1990;5(4):217–24. [PubMed] [Google Scholar]
- 5.Wylie G, Fish L, Epker B. Cephalometrics: a comparison of five analyses currently used in the diagnosis of dentofacial deformities. Int J Adult Orthodon Orthognath Surg. 1987;2(1):15–36. [PubMed] [Google Scholar]
- 6.Eggmann F, Weiger R, Zitzmann NU, Blatz MB. Implications of large language models such as ChatGPT for dental medicine. J Esthet Restor Dent. 2023;35(7):1098–102. 10.1111/jerd.13046. [DOI] [PubMed] [Google Scholar]
- 7.Makrygiannakis MA, Giannakopoulos K, Kaklamanos EG. Evidence-based potential of generative artificial intelligence large language models in orthodontics: a comparative study of ChatGPT, Google Bard, and Microsoft Bing. Eur J Orthod. 2025;48(1):cjae017. 10.1093/ejo/cjae017. [DOI] [PMC free article] [PubMed]
- 8.Brown T, Mann B, Ryder N, Subbiah M, Kaplan JD, Dhariwal P, et al. Language models are few-shot learners. Adv Neural Inf Process Syst. 2020;33:1877–901. [Google Scholar]
- 9.Adamopoulou E, Moussiades L. An overview of chatbot technology. In: Artificial intelligence applications and innovations. Springer; 2020. p. 373–83.
- 10.Vaishya R, Misra A, Vaish A. ChatGPT: is this version good for healthcare and research? Diabetes Metabolic Syndr: Clin Res Rev. 2023;17(4):102744. 10.1016/j.dsx.2023.102744. [DOI] [PubMed] [Google Scholar]
- 11.Hasson D, Arnetz BB. Validation and findings comparing VAS vs. Likert scales for psychosocial measurements. Int Electron J Health Educ. 2005;8:178–92. [Google Scholar]
- 12.Marzuki HN, Zahirah I, Lau MN, Kuppusamy E, Mustapha NMN, Ashari A. Likert scale versus the visual analogue scale in evaluating dentofacial aesthetics: a systematic review. Australas Orthodontic J. 2024;40(1):158–68. 10.2478/aoj-2024-0010. [DOI] [Google Scholar]
- 13.Yusof NADM, Jamil PASM, Hashim NM, Karuppiah K, Rasdi I, Tamrin SBM, et al. Likert scale vs. visual analogue scale on vehicle seat discomfort questionnaire: a review. Malays J Med Health Sci. 2019;15:204. [Google Scholar]
- 14.Lu X, Zhang R, Wu W, Shang X, Liu M. Relationship between internet health information and patient compliance based on trust: empirical study. J Med Internet Res. 2018;20(8):e253. 10.2196/jmir.9364. [DOI] [PMC free article] [PubMed]
- 15.Yıldırım A, Cicek O, Genç YS. Can AI-Based ChatGPT models accurately analyze hand–wrist radiographs? A comparative study. Diagnostics. 2025;15(12):1513. 10.3390/diagnostics15121513. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Cicek O, Yıldırım A. Evaluating the reliability of artificial intelligence in clinical decisions on early maxillary expansion: a comparative study of eight large language models. BMC Oral Health. 2026;26(1):715. 10.1186/s12903-026-08111-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Hatia A, Doldo T, Parrini S, Chisci E, Cipriani L, Montagna L, et al. Accuracy and completeness of ChatGPT-generated information on interceptive orthodontics: a multicenter collaborative study. JCM. 2024;13(3):735. 10.3390/jcm13030735. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Polit DF, Beck CT. The content validity index: are you sure you know what’s being reported? Critique and recommendations. Res Nurs Health. 2006;29(5):489–97. 10.1002/nur.20147. [DOI] [PubMed] [Google Scholar]
- 19.Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. 2016;15(2):155–63. 10.1016/j.jcm.2016.02.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Akoglu H. User’s guide to correlation coefficients. Turk J Emerg Med. 2018;18(3):91–93. 10.1016/j.tjem.2018.08.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Bernard A, Langille M, Hughes S, Rose C, Leddin D, Van Zanten SV. A systematic review of patient inflammatory bowel disease information resources on the World Wide Web. Am J Gastroenterol. 2007;102(9):2070–77. 10.1111/j.1572-0241.2007.01325.x. [DOI] [PubMed] [Google Scholar]
- 22.Li Z, Wang L. Multi-task reinforcement learning and explainable AI-driven platform for personalized planning and clinical decision support in orthodontic-orthognathic treatment. Sci Rep. 2025;15(1):24502. 10.1038/s41598-025-09236-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Tanaka OM, Gasparello GG, Hartmann GC, Casagrande FA, Pithon MM. Assessing the reliability of ChatGPT: a content analysis of self-generated and self-answered questions on clear aligners, TADs and digital imaging. Dent Press J Orthod. 2023;28(5):e2323183. 10.1590/2177-6709.28.5.e2323183.oar. [DOI] [PMC free article] [PubMed]
- 24.Rakhmatulla N, Iroda N, Kakhramonjon A, Rustambek R, Bekhzodbek R, Ruziev S. AI-powered orthodontics: revolutionizing diagnosis, planning, and education with DeepSeek, Grok 3, and ChatGPT. 2025. 10.20944/preprints202503.1686.v1. [DOI]
- 25.Guo Z, Lai A, Thygesen JH, Farrington J, Keen T, Li K. Large language models for mental health applications: systematic review. JMIR Ment Health. 2024;11:e57400. 10.2196/57400. [DOI] [PMC free article] [PubMed]
- 26.Hasal M, Nowaková J, Ahmed Saghair K, Abdulla H, Snášel V, Ogiela L. Chatbots: security, privacy, data protection, and social aspects. Concurrency Computation. 2021;33(19):e 6426. 10.1002/cpe.6426. [DOI]
- 27.Thurzo A, Urbanova W, Novak B, Czako L, Siebert T, Stano P, et al. Where is the artificial intelligence applied in dentistry? Systematic review and literature analysis. Healthcare (Basel). 2022;10(7):1269. 10.3390/healthcare10071269. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Shirani M, Emami M. Performance comparison of large language models in treatment planning for the restoration of endodontically treated teeth over time. J Dent. 2025;161:105998. 10.1016/j.jdent.2025.105998. [DOI] [PubMed] [Google Scholar]
- 29.Hajibagheri P, Sani SK, Samami M, Tabari-Khomeiran R, Azadpeyma K, Sani MK. ChatGPT’s accuracy in the diagnosis of oral lesions. BMC Oral Health. 2025;25(1):1229. 10.1186/s12903-025-06582-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Wu X, Cai G, Guo B, Ma L, Shao S, Yu J, et al. A multi-dimensional performance evaluation of large language models in dental implantology: comparison of ChatGPT, DeepSeek, Grok, Gemini and qwen across diverse clinical scenarios. BMC Oral Health. 2025;25(1):1272. 10.1186/s12903-025-06619-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Hasei J, Nakahara R, Takeuchi K, Yoshida A, Itano T, Fujiwara T, et al. Comparative analysis of a standard (GPT-4o) and reasoning-enhanced (o1 pro) large language model on complex clinical questions from the Japanese orthopaedic board examination. J Orthopaedic Sci. 2025;30(3):565–68. 10.1016/j.jos.2025.03.005. [DOI] [PubMed] [Google Scholar]
- 32.Suárez A, Freire Y, Suárez M, Díaz-Flores García V, Andreu-Vázquez C, Thuissard Vasallo IJ, et al. Diagnostic performance of multimodal large language models in the analysis of oral pathology. Oral Dis. 2025;31(12):3344–54. 10.1111/odi.70009. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Chen YW, Stanley K, Att W. Artificial intelligence in dentistry: current applications and future perspectives. Quintessence Int. 2020;51(3):248–57. 10.3290/j.qi.a43952. [DOI] [PubMed] [Google Scholar]
- 34.Lee JH, Kim DH, Jeong SN. Diagnosis of cystic lesions using panoramic and cone beam computed tomographic images based on deep learning neural network. Oral Dis. 2020;26(1):152–58. 10.1111/odi.13223. [DOI] [PubMed] [Google Scholar]
- 35.Grinberg N, Whitefield S, Kleinman S, Ianculovici C, Wasserman G, Peleg O. Assessing the performance of an artificial intelligence based chatbot in the differential diagnosis of oral mucosal lesions: clinical validation study. Clin Oral Investig. 2025;29(4):188. 10.1007/s00784-025-06268-7. [DOI] [PubMed] [Google Scholar]
- 36.Shirani M. Comparing the performance of ChatGPT 4o, DeepSeek R1, and Gemini 2 pro in answering fixed prosthodontics questions over time. J Prosthet Dent. 2026;135(3):571–76. 10.1016/j.prosdent.2025.04.038. [DOI] [PubMed] [Google Scholar]
- 37.Hwang HW, Park JH, Moon JH, Yu Y, Kim H, Her SB, et al. Automated identification of cephalometric landmarks: part 2- might it Be better than human? Angle Orthodontist. 2020;90(1):69–76. 10.2319/022019-129.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Lee JH, Yu HJ, Kim MJ, Kim JW, Choi J. Automated cephalometric landmark detection with confidence regions using Bayesian convolutional neural networks. BMC Oral Health. 2020;20(1):270. 10.1186/s12903-020-01256-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Kim MJ, Liu Y, Oh SH, Ahn HW, Kim SH, Nelson G. Automatic cephalometric landmark identification system based on the multi-stage convolutional neural networks with CBCT combination images. Sensors (Basel). 2021;21(2):505. 10.3390/s21020505. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Dot G, Rafflenbeul F, Kerbrat A, Rouch P, Gajny L, Schouman T. Three-dimensional cephalometric landmarking and Frankfort horizontal plane construction: reproducibility of conventional and novel landmarks. JCM. 2021;10(22):5303. 10.3390/jcm10225303. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Jegham N, Abdelatti M, Koh CY, Elmoubarki L, Hendawi A. How hungry is AI? Benchmarking energy, water, and carbon footprint of LLM inference. arXiv. 2025;2505.09598.
- 42.Liu H, Li C, Wu Q, Lee YJ. Visual instruction tuning. Adv Neural Inf Process Syst. 2023;36:34892–916. [Google Scholar]
- 43.Sarangi PK, Datta S, Swarup MS, Panda S, Nayak DSK, Malik A, et al. Radiologic decision-making for imaging in pulmonary embolism: accuracy and reliability of large language models—Bing, Claude, ChatGPT, and perplexity. Indian J Radiol Imag. 2024;34(04):653–60. 10.1055/s-0044-1787974. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Liu S, Pu W, Xu C, Huang Z, Li Q, Wang H, et al. A comprehensive survey of multimodal large language models: concept, application and safety. 2024. 10.21203/rs.3.rs-5270567/v1. [DOI]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
The datasets analyzed during the current study are available from the corresponding author on reasonable request.