Abstract
Although many studies have examined large language models (LLMs) for dental diagnosis, most focus on a single topic, and few have investigated their broader usefulness for the variety of patients. Twenty-five cases meeting the eligibility criteria were extracted from paediatric dentistry questions in the National Examination. Age, sex, and chief complaint were collected and presented to Copilot and Gemini along with intraoral photographs and radiographic examinations, instructing them to identify abnormal areas and provide a diagnosis. Both LLMs showed >70% accuracy in identifying pathological locations and approximately 60% accuracy in making diagnoses. However, accuracy decreased markedly when the chief complaint was omitted. In incorrect location responses, models often selected adjacent teeth when a chief complaint was present, whereas entirely different sites were more common when no chief complaint was provided. Although diagnostic accuracy for apical periodontitis was high under all four conditions, accuracy dropped substantially for paediatric-specific topics such as abnormalities in tooth number and abnormal eruption when the chief complaint was absent. These findings suggest the potential for using LLMs in paediatric dentistry; however, obtaining an accurate chief complaint is essential. Moreover, LLMs still show limitations in areas unique to paediatric dentistry, highlighting the need for further development.
Key words: Artificial intelligence, Dental education, Diagnosis, Paediatric dentistry
Introduction
Artificial intelligence (AI) has advanced rapidly in recent years and is now used in a wide range of settings. Generative AI refers to models that produce synthetic outputs based on patterns learned from the datasets used to train them.1 Large language models (LLMs) such as ChatGPT and Gemini, in particular, are already influencing healthcare by supporting patient care, medical research, and administrative tasks.2 In dentistry, studies using qualification examinations and actual patient cases have shown the potential for future clinical application.3, 4, 5 However, because certification examinations rely on multiple-choice questions, candidates only need to select the most appropriate option from a limited set, which differs from real clinical practice. In paediatric dentistry, clinicians routinely encounter a wide range of patient complaints – not only dental caries but also fused teeth, supernumerary teeth, congenitally missing teeth, and eruption abnormalities. However, the use of AI in pedodontics has thus far focused primarily on detection and prediction models, and prior work has largely assessed LLM performance for single conditions, such as trauma or supernumerary teeth.4, 5, 6 Only a few studies have examined whether LLMs can accurately identify pathological findings from patient information and generate correct diagnoses from the broad range of possibilities. Given this gap, we saw a need to assess whether LLMs can reliably identify abnormal sites and provide accurate diagnoses when presented with intraoral photographs and periapical radiographs. This study evaluated the ability of LLMs to interpret test results and make diagnoses; it also examined whether their focus changes depending on whether a chief complaint is provided.
Methods
This study designed its methodology based on the latest AI reporting framework.7 The requirement for ethical approval was waived because this study evaluated the usefulness of AI and did not involve human or animal subjects or any patient data. The flowchart for case selection is shown in Figure 1. The Japanese National Dental Examination consists of compulsory questions, general dentistry, and topics specific to each dental field; this study focused on general dentistry and the field-specific sections.3 Examination questions were obtained from the official website of the Ministry of Health, Labour and Welfare. From the 103rd to 117th examinations, we extracted 393 questions related to paediatric dentistry over a 15-year period, excluding compulsory questions. Of these, 249 contained an image, and from that group we selected those that included only intraoral photographs and periapical radiographs (n = 93). As a result of strictly excluding questions requiring facial findings or additional tests, the final 25 questions were extracted in this study. The extraction was performed by two dentists (T.A. and S.K.), each with more than 5 years of clinical experience in a university hospital’s paediatric dentistry department, based on the following criteria.
Fig. 1.
Flowchart of case selection.
Inclusion criteria
- Location and diagnosis can be confirmed through intraoral and periapical photographs.
- Diagnosis is not affected by the presence or absence of a chief complaint.
Exclusion criteria
- Facial or other physical findings are necessary for diagnosis.
- Additional examinations are necessary for diagnosis.
Although dental caries is common, it was excluded because diagnosis can be challenging when pulpitis is involved; only cases of apical periodontitis were included. The questions were adjusted to avoid overrepresentation of any specific subject area, and the final set of 25 was determined through discussion between the two dentists. From each selected question, we extracted age, sex, chief complaint, intraoral photographs, and periapical radiographs, and these were provided to the LLMs. To assess differences related to the presence or absence of a chief complaint, we also evaluated cases in which the chief complaint was removed.
We used the free versions of Microsoft Copilot (Microsoft, Redmond, Washington, D.C., USA) and Gemini 2.5 (Google, Mountain View, CA, USA) from 7th to 10th November 2025. After presenting each patient’s age, sex, chief complaint, intraoral photographs, and periapical radiograph, we asked the LLMs in Japanese, 'Please identify the location and provide a diagnosis.' Each question was presented three times on different days to evaluate the reproducibility of LLMs’ responses, totalling 75 responses per each condition for analysis. All responses were evaluated by one examiner (T.A.), with 9 years of clinical experience and a paediatric dental specialist certification from the Japanese Society of Paediatric Dentistry, according to two criteria: location and diagnosis.
When findings were present in multiple teeth, responses were considered correct if they appropriately addressed any one of them. Unless the problem involved several teeth, answers were counted as correct only when they included the maxillary or mandibular arch and the left or right side in addition to the tooth type; responses that mentioned only the tooth type, such as 'central incisor,' were marked incorrect. For cases involving supernumerary teeth or eruption hematoma, answers referring to areas such as the midline were accepted. Accuracy rates were recorded and compared between the two LLMs. We also analysed incorrect answers by location and made a more detailed comparison of diagnostic accuracy across individual topics.
Statistical analyses were performed using GraphPad Prism 9 (GraphPad Software Inc., La Jolla, CA, USA). Comparisons between two groups were conducted using Fisher’s exact test, and differences were considered statistically significant at P < .05.
Results
Table 1 shows the accuracy rates for location and diagnosis across the 25 questions. Based on the averages of the three trials, Gemini with a chief complaint achieved the highest overall score, followed by Copilot with a chief complaint. For both LLMs, accuracy for location was higher than for diagnosis. By contrast, removing the chief complaint markedly reduced performance in both categories, with accuracy dropping by roughly 20% to 30% compared with the condition in which the chief complaint was provided.
Table 1.
Accuracy scores for location and diagnosis by generative AI models.
| First |
Second |
Third |
Average |
|||||
|---|---|---|---|---|---|---|---|---|
| Location | Diagnosis | Location | Diagnosis | Location | Diagnosis | Location | Diagnosis | |
| Copilot with chief complaint |
19 (76.0%)* |
14 (56.0%) |
19 (76.0%) |
14 (56.0%) |
18 (72.0%)⁎⁎ |
15 (60.0%)* |
18.7 (74.7%)* |
14.3 (57.3%) |
| Copilot without chief complaint |
10 (40.0%) |
9 (36.0%) |
12 (48.0%) |
9 (36.0%) |
7 (28.0%) |
7 (28.0%) |
9.7 (38.7%) |
8.3 (33.3%) |
| Gemini with chief complaint |
20 (80.0%)# |
17 (68.0%) |
19 (76.0%)# |
15 (60.0%) |
19 (76.0%)## |
14 (56.0%) |
19.3 (77.3%)# |
15.3 (61.3%) |
| Gemini without chief complaint |
11 (44.0%) |
13 (52.0%) |
11 (44.0%) |
8 (32.0%) |
8 (32.0%) |
10 (40.0%) |
10.0 (40.0%) |
10.3 (41.3%) |
Statistically significant difference compared with Copilot without a chief complaint: P < .05.
Statistically significant difference compared with Copilot without a chief complaint: P < .01.
Statistically significant difference compared with Gemini without a chief complaint: P < .05.
Statistically significant difference compared with Gemini without a chief complaint: P < .01.
Table 2 summarizes how consistently each model answered the same question across the three trials. For location, Copilot with a chief complaint produced the same response in 96% of cases: 72% were all correct and 24% were all incorrect. Gemini with a chief complaint also showed reproducibility in 60% of cases: 56% were all correct and 4% were all incorrect. However, when the chief complaint was omitted, both Copilot and Gemini showed a substantial increase in the proportion of all-incorrect answers. The number of questions with one or two correct responses also increased relative to those with all correct or all incorrect answers. Diagnostic scores followed a similar pattern, with lower all-correct rates and higher all-incorrect rates compared with the location scores.
Table 2.
Number of correct responses for the same question across three trials.
| Location | Three times | Two times | One time | All incorrect |
|---|---|---|---|---|
| Copilot with chief complaint |
18 (72.0%)⁎⁎ |
1 (4.0%) |
0 (0.0%) |
6 (24.0%) |
| Copilot without chief complaint |
6 (24.0%) |
3 (12.0%) |
5 (20.0%) |
11 (44.0%) |
| Gemini with chief complaint |
14 (56.0%)# |
6 (24.0%) |
4 (16.0%) |
1 (4.0%)# |
| Gemini without chief complaint |
5 (20.0%) |
3 (12.0%) |
9 (36.0%) |
8 (32.0%) |
| Diagnosis | Three times | Two times | One time | All incorrect |
| Copilot with chief complaint |
12 (48.0%) |
3 (12.0%) |
1 (4.0%) |
9 (36.0%) |
| Copilot without chief complaint |
5 (20.0%) |
4 (16.0%) |
2 (8.0%) |
14 (56.0%) |
| Gemini with chief complaint |
11 (44.0%) |
5 (20.0%) |
3 (12.0%) |
6 (24.0%) |
| Gemini without chief complaint |
5 (20.0%) |
6 (24.0%) |
4 (16.0%) |
10 (40.0%) |
Statistically significant difference compared with Copilot without a chief complaint: P < .01.
Statistically significant difference compared with Gemini without a chief complaint: P < .05.
Across the 75 total trials for each LLM (25 questions × 3 tests), the number of incorrectly identified locations was 19 for Copilot with a chief complaint, 46 for Copilot without a chief complaint, 17 for Gemini with a chief complaint, and 45 for Gemini without a chief complaint, for a total of 127 incorrect responses. We analysed the causes of these errors (Table 3). Overall, 33.9% involved the correct tooth type but incorrect arch or side. When chief complaint information was available, misidentification often involved adjacent teeth; without it, errors were more variable, increasing the proportion categorized as 'Others.'
Table 3.
Reasons for incorrect location identification.
| Differences in arch or side | Differences between adjacent teeth | Differences between primary and permanent teeth | Others | Total | |
|---|---|---|---|---|---|
| Copilot with chief complaint |
7 (36.8%) |
9 (47.4%)* |
1 (5.3%) |
2 (10.5%)⁎⁎ |
19 (100.0%) |
| Copilot without chief complaint |
14 (30.4%) |
9 (19.6%) |
2 (4.3%) |
21 (45.7%) |
46 (100.0%) |
| Gemini with chief complaint |
6 (35.3%) |
6 (35.3%) |
2 (11.8%) |
3 (17.6%)# |
17 (100.0%) |
| Gemini without chief complaint |
16 (35.6%) |
5 (11.1%) |
2 (4.4%) |
22 (48.9%) |
45 (100.0%) |
| Total | 43 (33.9%) |
29 (22.8%) |
7 (5.5%) |
48 (37.8%) |
127 (100.0%) |
Statistically significant difference compared with Copilot without a chief complaint: P < .05.
Statistically significant difference compared with Copilot without a chief complaint: P < .01.
Statistically significant difference compared with Gemini without a chief complaint: P < .05.
When diagnostic accuracy was compared across case types, high performance was observed for 'apical periodontitis' and 'malformation' when chief complaint information was available (Table 4). Even without a chief complaint, 'apical periodontitis' remained relatively accurate, whereas the accuracy for 'abnormalities in number of teeth' and 'abnormal eruption' declined markedly.
Table 4.
Comparison of accuracy rates by subject.
| Apical periodontitis (6 × 3 tests = 18) | Dental trauma (6 × 3 tests = 18) | Abnormalities in number of teeth (4 × 3 tests = 12) | Abnormal eruption (4 × 3 tests = 12) | Malformation (3 × 3 tests = 9) | Others (2 × 3 tests = 6) | |
|---|---|---|---|---|---|---|
| Copilot with chief complaint |
16 (88.9%) |
6 (33.3%) |
7 (58.3%) |
5 (41.7%)* |
6 (66.7%)* |
3 (50.0%) |
| Copilot without chief complaint |
11 (61.1%) |
8 (44.4%) |
2 (16.7%) |
0 (0.0%) |
1 (11.1%) |
3 (50.0%) |
| Gemini with chief complaint |
17 (94.4%) |
11 (61.1%) |
3 (25.0%) |
5 (41.7%)# |
9 (100.0%) |
1 (16.7%) |
| Gemini without chief complaint |
16 (88.9%) |
6 (33.3%) |
0 (0.0%) |
0 (0.0%) |
6 (66.7%) |
3 (50.0%) |
| Total | 60 (83.3%) |
31 (43.1%) |
12 (25.0%) |
10 (20.8%) |
22 (61.1%) |
10 (41.7%) |
Statistically significant difference compared with Copilot without a chief complaint: P < .05.
Statistically significant difference compared with Gemini without a chief complaint: P < .05.
Discussion
Both Copilot and Gemini correctly identified abnormal areas in more than 70% of cases across all three trials when the chief complaint, intraoral photographs, and radiographs were provided. Diagnostic accuracy was lower (approximately 50%-60%), indicating that pinpointing the affected area was easier for the models than determining the correct diagnosis. By contrast, when cases were presented without a chief complaint, accuracy for both location and diagnosis declined across all trials in both LLMs. This suggests that the chief complaint serves as an important starting point for identifying the correct site and diagnosis and that providing it is essential when using LLMs. It also suggests that current multimodal LLMs function more as 'verification tools' (confirming what the text says) rather than independent 'diagnostic tools.' Therefore, LLMs have a risk of misdiagnosis when the patient (or the parent) provides a vague or incorrect chief complaint, and dental professionals need to understand this when using LLMs. However, it is known that the diagnosis suggested by the chief complaint does not always match the final diagnosis.8 Moreover, nearly half of the lesions detected on panoramic radiographs have been reported to be unrelated to the patient’s chief complaint, underscoring the need for radiographic interpretation that goes beyond it.9 Taken together, LLMs may have potential clinical utility only when the chief complaint is sufficiently accurate and aligned with the eventual diagnosis. Nonetheless, identifying abnormalities from available clinical data will require further improvements in performance.
Assessment of reproducibility across the three trials showed that Copilot produced the same answer – either correct or incorrect – 96% of the time for location and 84% for diagnosis. Gemini showed reproducibility of 60% for location and 68% for diagnosis. When the chief complaint was removed, however, the proportion of cases with one or two correct answers increased, indicating that LLM responses varied from trial to trial. Kamminga et al10 also reported reduced reproducibility in LLM responses to patient questions about melanoma. Such variability is problematic for both clinicians and patients, and further investigations assessing reproducibility over time are needed.
Analysis of incorrect answers for location showed that many errors involved confusion between upper and lower or left and right, regardless of whether a chief complaint was provided. When the chief complaint was present, incorrect answers often involved adjacent teeth; without it, errors more frequently fell into the 'Others' category. These findings suggest that LLMs can narrow the abnormal area more effectively when guided by the chief complaint.
When accuracy was compared across conditions, apical periodontitis showed the highest scores in all four testing scenarios. AI models have demonstrated strong performance in detecting apical periodontitis on periapical radiographs, with reported sensitivity and specificity exceeding those of human experts.11 This may explain why apical periodontitis appears relatively easy for LLMs to diagnose. Although AI has also been reported to be effective for dental trauma, trauma cases in the present study – despite being presented with the same number of problems as apical periodontitis – achieved only about half the accuracy.12,13 Previous studies typically presented images of dental trauma and evaluated classification or the presence and absence of fractures. In our study, although the chief complaint was provided, it did not necessarily point toward trauma, making it harder for LLMs to identify tooth fractures among many possible diagnoses. Accuracy for abnormalities in the number of teeth and abnormal eruption varied markedly depending on whether a chief complaint was provided. These are topics more specific to paediatric dentistry, and the results indicate that diagnosis in this area was particularly difficult without a chief complaint. Mixed dentition continues to pose challenges for LLMs in radiographic interpretation.14 Determining whether the teeth in the image are primary or permanent or whether the findings reflect normal eruption patterns adds complexity, and LLMs still have substantial limitations in this domain. In addition, in this study, the questions requiring additional tests were excluded. However, panoramic radiographs are the standard examination for diagnosing tooth number anomalies and eruption sequences in paediatric dentistry. The reliance solely on periapical radiographs might have disadvantaged the LLMs in these specific categories.
Although AI has rapidly spread throughout the dental field in recent years, it is reported that dental students lack sufficient understanding of AI applications in dentistry.15 This study also suggests that although LLMs have the potential to be useful tools, the final diagnosis requires the judgment of dental professionals. It is necessary to provide an educational environment that informs dental students about the current state of AI clinical applications and enables them to use AI appropriately.
Limitations
This study has several limitations. First, this study is preliminary investigation, and the sample size was small. Additionally, Furthermore, the AI-generated responses were evaluated by only one dentist, although he is an experienced dentist with a paediatric dental specialist certification by the Japanese Society of Paediatric Dentistry. Further research into a larger sample size and different people who grade the answers independently to make sure the scoring is fair are needed in the future. Second, we used a free version of LLMs, and it has concerns about updating all the time without informing users. On the other hand, the paid version has the drawback that its system varies with the price, making comparisons difficult.16 Considering the point that it is accessible to everyone, this study used the free version, but we will consider changes of over time or comparisons with the paid version in the future. Third, we used the questions from the Japanese National Dental Examination, and some questions had oral disease with multiple teeth. In this study, even if the AI identified only one of them, it was considered correct. However, missing the other cavities would be a failure in clinical situations. One primary ethical concern surrounding the integration of AI into dentistry revolves around patient privacy and data security.17 While considering ethical concerns, it is desirable to conduct similar research again using problems suitable for evaluation.
Conclusions
This study showed that the patient’s chief complaint serves as an important clue for LLMs in identifying pathological areas and guiding them toward appropriate diagnoses. Even when no specific symptoms are present, LLMs might still be used as a form of screening; however, their performance in such situations was limited, especially in areas characteristic of paediatric dentistry, such as abnormalities in the number of teeth and abnormal eruption. To apply LLMs effectively as diagnostic aids in paediatric dentistry, it will be necessary to incorporate diverse datasets and strengthen knowledge specific to this field.
Author contributions
Conceptualization, T.A. and R.N.; methodology, T.A. and S.K.; investigation, T.A., A.K., T.N., M.H. and S.K., data curation, T.A.; writing – original draft, T.A. and R.N.; writing – review and editing, T.A. and R.N. All authors have read and agreed to the published version of the manuscript.
Conflict of interest
None disclosed.
Acknowledgements
The authors thank Ms. Asami Aoya (Department of Paediatric Dentistry, Graduate School of Biomedical and Health Sciences, Hiroshima University) for her assistance with manuscript editing.
References
- 1.Kunze K.N. Generative artificial intelligence and musculoskeletal health care. HSS J. 2025;21(3):248–256. doi: 10.1177/15563316251335334. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Feretzakis G., Anastasiou A., Pitoglou S., et al. Securing a generative AI-powered healthcare chatbot. stud health technol inform. Stud Health Technol Inform. 2024;321:195–199. doi: 10.3233/SHTI241091. [DOI] [PubMed] [Google Scholar]
- 3.Akitomo T., Hamada M., Tsuge Y., et al. Artificial intelligence's performance on the Japanese National Dental Examination. Cureus. 2024;16(11) doi: 10.7759/cureus.73103. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Kusaka S., Akitomo T., Hamada M., et al. Usefulness of generative artificial intelligence (AI) tools in pediatric dentistry. Diagnostics (Basel) 2024;14(24):2818. doi: 10.3390/diagnostics14242818. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Çege E.E., Cömert H., Akal N., Ölmez A. Evaluation of the performance of artificial intelligence based chatbots in providing first aid information on dental trauma according to the ToothSOS application. Dent Traumatol. 2025;41(6):696–705. doi: 10.1111/edt.13078. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Samaranayake L., Tuygunov N., Schwendicke F., et al. The transformative role of artificial intelligence in dentistry: a comprehensive overview. Part 1: fundamentals of AI, and its contemporary applications in dentistry. Int Dent J. 2025;75:383–396. doi: 10.1016/j.identj.2025.02.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Khurshid Z., Osathanon T., Shire M.A., Schwendicke F., Samaranayake L. Artificial intelligence in dentistry: a concise review of reporting checklists and guidelines. Int Dent J. 2026;76 doi: 10.1016/j.identj.2025.109322. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Begier E.M., Sockwell D., Branch L.M., et al. The National Capitol Region's Emergency Department syndromic surveillance system: do chief complaint and discharge diagnosis yield different results? Emerg Infect Dis. 2003;9(3):393–396. doi: 10.3201/eid0903.020363. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Asaumi J.I., Hisatomi M., Yanagi Y., et al. Evaluation of panoramic radiographs taken at the initial visit at a department of paediatric dentistry. Dentomaxillofac Radiol. 2008;37(6):340–343. doi: 10.1259/dmfr/31214423. [DOI] [PubMed] [Google Scholar]
- 10.Kamminga N.C.W., Kievits J.E.C., Plaisier P.W., et al. Do large language model chatbots perform better than established patient information resources in answering patient questions? A comparative study on melanoma. Br J Dermatol. 2025;192(2):306–315. doi: 10.1093/bjd/ljae377. [DOI] [PubMed] [Google Scholar]
- 11.Basso Á, Salas F., Hernández M., Fernández A., Sierra A., Jiménez C. Machine learning and deep learning models for the diagnosis of apical periodontitis: a scoping review. Clin Oral Investig. 2024;28(11):600. doi: 10.1007/s00784-024-05989-5. [DOI] [PubMed] [Google Scholar]
- 12.Abdelazim R., Fouad E.M. Artificial intelligent-driven decision-making for automating root fracture detection in periapical radiographs. BDJ Open. 2024;10(1):76. doi: 10.1038/s41405-024-00260-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Bani-Hani T., Wedyan M., Al-Fodeh R., Shuqeir R., Al Jundi S., Tewari N. Artificial intelligence model for application in dental traumatology. Eur Arch Paediatr Dent. 2025;26(6):1117–1124. doi: 10.1007/s40368-025-01063-0. [DOI] [PubMed] [Google Scholar]
- 14.Mine Y., Iwamoto Y., Okazaki S., et al. Challenges and limitations of multimodal large language models in interpreting pediatric panoramic radiographs. Int J Paediatr Dent. 2025;36(1):74–80. doi: 10.1111/ipd.70029. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Dashti M., Londono J., Ghasemi S., et al. Attitudes, knowledge, and perceptions of dentists and dental students toward artificial intelligence: a systematic review. J Taibah Univ Med Sci. 2024;19(2):327–337. doi: 10.1016/j.jtumed.2023.12.010. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Hamaguchi S, Hamada M, Ikeda S, Kusaka S, Akitomo T, Nomura R. Comparison of the performances of different updated generative artificial intelligence models on the Japanese National Dental Examination. J Dent Sci n.d.. https://www.sciencedirect.com/science/article/pii/S1991790225003617. [DOI] [PMC free article] [PubMed]
- 17.Tuygunov N., Samaranayake L., Khurshid Z., et al. The transformative role of artificial intelligence in dentistry: a comprehensive overview Part 2: the promise and perils, and the International Dental Federation communique. Int Dent J. 2025;75(2):397–404. doi: 10.1016/j.identj.2025.02.006. [DOI] [PMC free article] [PubMed] [Google Scholar]