Comparison of ChatGPT, Gemini, and DeepSeek responses to questions on medication-related osteonecrosis of the jaw (MRONJ): a comparative analysis of expert-rated clinical quality, readability, response time, and clinical safety risk

Abstract

Background

Medication-related osteonecrosis of the jaw (MRONJ) is a clinically significant complication associated with antiresorptive and antiangiogenic therapies, and its prevention and management depend on reliable, guideline-concordant information. Large language models (LLMs) are increasingly used to generate medical information, but their performance in MRONJ remains insufficiently investigated. This study compared the responses generated by ChatGPT- 5.4 Thinking, Gemini 3 Deep Think, and DeepSeek-V3.2 in thinking mode to MRONJ-related open-ended questions in terms of clinical quality, readability, and response time.

Methods

In this cross-sectional comparative study, a standardized 30-item question set was developed from the MASCC/ISOO/ASCO Clinical Practice Guideline and the Italian Consensus Update. Each question was entered separately into each model under standardized conditions, yielding 90 responses. Outputs were anonymized and independently evaluated by three blinded oral and maxillofacial surgeons using a modified Global Quality Scale on a five-point Likert scale. A post hoc Clinical Risk Classification was additionally performed to distinguish minor informational deficiencies from omissions or inaccuracies with greater potential clinical consequences. Readability was assessed using five established English-language indices, and response time was measured with an online digital stopwatch. Appropriate statistical tests were applied, and inter-evaluator reliability was assessed using the intraclass correlation coefficient.

Results

Inter-evaluator agreement was high across all models (all p < 0.001). Mean expert quality scores differed significantly among models (p < 0.001). Post hoc comparisons showed that ChatGPT-5.4 Thinking received significantly lower expert ratings than both DeepSeek-V3.2 and Gemini 3 Deep Think, whereas no statistically significant difference was detected between DeepSeek-V3.2 and Gemini 3 Deep Think. Response time also differed significantly (p < 0.001): DeepSeek-V3.2 produced the fastest responses, Gemini 3 Deep Think showed intermediate latency, and ChatGPT-5.4 Thinking had the longest response times. Readability analyses revealed significant inter-model differences across all indices, with Gemini 3 Deep Think generating the most complex outputs and DeepSeek-V3.2 producing comparatively more readable responses. Clinical Risk Classification showed that most responses were categorized as having no clinically relevant risk or low clinical risk, while moderate-risk classifications were uncommon and no high-risk responses were identified.

Conclusions

Under single-prompt, single-session benchmark conditions, potentially clinically relevant differences were observed among the models in MRONJ-related response quality, readability, response time, and clinical risk classification. DeepSeek-V3.2 and Gemini 3 Deep Think achieved higher expert-rated quality scores than ChatGPT-5.4 Thinking, while DeepSeek-V3.2 also showed advantages in speed and readability. These findings do not indicate general model superiority and support using LLMs only as adjunctive informational tools, not substitutes for specialist and guideline-based clinical decision-making.

Supplementary Information

The online version contains supplementary material available at https://doi.org/10.1186/s12903-026-09421-0.

Background

Medication-related osteonecrosis of the jaw (MRONJ) is a potentially serious adverse event associated with antiresorptive and antiangiogenic therapies and is typically defined as exposed bone, or bone that can be probed through an intraoral or extraoral fistula in the maxillofacial region, persisting for more than 8 weeks in the absence of previous radiation therapy to the jaws or metastatic disease to the jaws [1–3]. MRONJ is most commonly encountered in patients receiving bisphosphonates, denosumab, and certain antiangiogenic agents for osteoporosis, multiple myeloma, and solid malignancies with skeletal involvement [1–4]. Although the absolute risk is relatively low in patients treated for osteoporosis, the risk is substantially higher in oncology populations receiving high-dose therapy, and the clinical consequences may include chronic pain, secondary infection, impaired oral intake, functional limitation, and reduced quality of life [1, 3, 4]. Because MRONJ prevention and management depend heavily on guideline-concordant dental assessment, risk stratification, timely intervention, and careful communication among oncologists, dentists, oral and maxillofacial surgeons, and patients, the quality of information provided in this field is clinically important [1–4]. International recommendations, particularly the Multinational Association of Supportive Care in Cancer (MASCC)/International Society of Oral Oncology (ISOO)/American Society of Clinical Oncology (ASCO) Clinical Practice Guideline and the Italian Consensus Update, emphasize pretreatment dental evaluation, elimination of infectious foci, individualized risk assessment, and patient education as key components of care [1, 2]. Despite the availability of evidence-based recommendations, implementation in routine practice remains heterogeneous, and interdisciplinary coordination continues to represent a practical challenge in many settings [2–4].

Large language models (LLMs) have rapidly emerged as widely accessible tools for retrieving, summarizing, and generating medical information [5–9]. Previous work in medicine has shown that artificial intelligence (AI) tools can perform strongly in licensing-style examinations, while studies in dentistry have reported promising but variable performance across knowledge assessment, clinical reasoning, and patient-information tasks [6–14]. Within oral and maxillofacial surgery and dentomaxillofacial radiology, recently published benchmarking studies have reported mixed findings, with model performance varying according to specialty, question format, and the scoring framework used [10–14].

Despite their promise, the use of LLMs in clinically sensitive areas remains limited by concerns regarding hallucinations, omission of key recommendations, inconsistent risk communication, and substantial variation in readability and practical usability [5, 9–14]. These concerns are particularly relevant in MRONJ, where incorrect advice regarding tooth extraction, implant placement, medication exposure, preventive strategies, or stage-based management may have direct clinical consequences [1–4, 9]. Notably, a recent MRONJ-specific study evaluating a context-aware retrieval-augmented chatbot suggested that source-grounded systems may provide more dependable specialty-level guidance than generic generative systems alone [15].

At present, direct comparative evidence focusing specifically on contemporary LLM performance in MRONJ remains limited [10–15]. In particular, there is a lack of structured benchmarking studies evaluating whether different models can provide guideline-concordant responses to open-ended MRONJ questions across core clinical domains while also examining readability and response latency [10–15]. Addressing this gap is important because LLMs are increasingly being used not only for general information seeking, but also as informal decision-support aids and patient-facing educational tools.

Therefore, the aim of the present study was to compare the responses generated by ChatGPT-5.4 Thinking, Gemini 3 Deep Think, and DeepSeek-V3.2 in thinking mode to a standardized set of 30 open-ended MRONJ questions derived from internationally recognized guideline documents [1, 2]. The primary objective was to evaluate the overall expert-rated clinical quality of the generated responses. The secondary objectives were to compare the models with respect to readability, response latency, and post hoc clinical risk classification. We hypothesized that potentially clinically relevant inter-model differences would be observed across these domains and that stronger expert-rated clinical quality would not necessarily be associated with better readability, shorter response time, or lower clinical risk classification.

Materials and methods

Study design

The performance of three publicly accessible LLMs in answering open-ended questions related to MRONJ was evaluated in this cross-sectional, comparative, observational study. The study benchmarked model responses against contemporary evidence-based MRONJ guidance and compared the systems in terms of clinical quality, readability, and response time. The study was described as cross-sectional because all model outputs were collected within a predefined testing period and no longitudinal follow-up was performed. It was considered comparative because the same question set was applied across three different LLM platforms. Although the study did not involve human participants, clinical interventions, or randomized clinical outcomes, the AI platform was deliberately assigned by the investigators for each query. Therefore, the design may also be interpreted as a quasi-experimental benchmarking study. In this manuscript, the term observational is used to indicate that no patient-level intervention, clinical outcome manipulation, or human-subject experimentation was performed.

Ethics statement

In this study, responses generated by publicly accessible artificial intelligence systems were analyzed, and no human participants, patient data, human tissue, biological samples, or identifiable personal information were involved. Accordingly, ethics committee approval and informed consent were not required. Consent for publication was not applicable.

AI systems and access conditions

The performance of three web-accessible LLMs in answering open-ended questions related to MRONJ was evaluated: ChatGPT-5.4 Thinking (OpenAI, San Francisco, CA, USA), DeepSeek-V3.2 in thinking mode (DeepSeek AI, Beijing, China), and Gemini 3 Deep Think (Google LLC, Mountain View, CA, USA). All systems were accessed through their publicly available web interfaces using the same hardware configuration, the same internet connection, and newly initiated user sessions. Testing was conducted on March 6 and 7, 2026, in matched morning sessions beginning at approximately 08:00 local time (Fig. 1).

Fig. 1.

Example of prompt submission and response generation in ChatGPT-5.4 Thinking

The model names were recorded exactly as displayed in the respective platform interfaces at the time of testing. ChatGPT-5.4 Thinking and Gemini 3 Deep Think were accessed through the authors’ paid Plus subscription accounts, whereas DeepSeek-V3.2 was accessed through its publicly available web interface using the default user-facing access conditions. All models were used through their web-based interfaces with default user-facing settings. No temperature, top-p, seed, system-level sampling parameter, or backend inference setting was manually adjustable or modified by the investigators. Because these platforms operate as server-side systems, the exact backend model snapshot, load-balancing configuration, and model update timing could not be independently verified. Therefore, the testing dates, access conditions, standardized prompt wording, and response archiving protocol were documented to improve reproducibility.

To improve reproducibility and reduce avoidable variability, a standardized interaction protocol was used for all three systems. Each question was submitted in a de novo chat window. Browser cache and conversation history were cleared between queries, and no follow-up prompts, clarification prompts, or regeneration requests were used. All questions were entered in English. In ChatGPT, the memory function was turned off throughout the testing period. Responses were archived verbatim without editorial modification. The exact wording of the system prompt and the full list of questions and responses are provided in the Supplementary Material.

"Act as an expert Oral and Maxillofacial Surgeon with extensive experience in bone pathology and medication-related osteonecrosis of the jaw (MRONJ). Base your answer primarily on the Multinational Association of Supportive Care in Cancer (MASCC)/International Society of Oral Oncology (ISOO)/American Society of Clinical Oncology (ASCO) Clinical Practice Guideline and the Italian MRONJ Consensus Update. Provide a clinically applicable, evidence-based and clearly structured answer. Use a formal medical tone." The question is….”

Development of the MRONJ question set

A 30-item benchmark question set was developed to assess both factual knowledge retrieval and applied clinical reasoning in the context of MRONJ (Table 1). The question pool was derived from two reference documents used as the primary evidence standards for the study: the MASCC/ISOO/ASCO Clinical Practice Guideline and the Italian Consensus Update [1, 2]. Initial questions were drafted by the study team to reflect clinically relevant scenarios encountered in oral and maxillofacial surgery practice and to cover domains considered important for both clinician-facing and patient-facing information needs.

Table 1.

Standardized benchmark question set used in the study

No	Category	Question type	Benchmark question
Q1	General MRONJ management	Factual/guideline knowledge	What is MRONJ?
Q2	General MRONJ management	Factual/guideline knowledge	Which patient group is at greater risk of developing MRONJ?
Q3	General MRONJ management	Factual/guideline knowledge	What is the half-life of drugs in the bisphosphonate class?
Q4	General MRONJ management	Factual/guideline knowledge	Why does MRONJ most commonly occur in the jaw bones?
Q5	General MRONJ management	Factual/guideline knowledge	What is the half-life of denosumab, and how does it differ from drugs in the bisphosphonate class?
Q6	General MRONJ management	Factual/guideline knowledge	What is the mechanism of action of drugs in the bisphosphonate class?
Q7	General MRONJ management	Factual/guideline knowledge	Does the duration of bisphosphonate therapy affect the risk of MRONJ?
Q8	General MRONJ management	Factual/guideline knowledge	Does the route of administration (oral vs intravenous/intramuscular) of drugs in the bisphosphonate class affect the risk of MRONJ?
Q9	General MRONJ management	Applied clinical reasoning	Does denosumab administration following prior exposure to bisphosphonates increase the risk of MRONJ?
Q10	General MRONJ management	Applied clinical reasoning	Is the serum C-terminal cross-linked telopeptide of type I collagen (CTX) value significant in the diagnosis of MRONJ? What is its role in diagnosis and treatment?
Q11	General MRONJ management	Factual/guideline knowledge	Is tooth extraction an initiating factor for MRONJ?
Q12	General MRONJ management	Factual/guideline knowledge	Is traumatic occlusion within the oral cavity an initiating factor for MRONJ?
Q13	General MRONJ management	Factual/guideline knowledge	Is chronic inflammation within the oral cavity an initiating factor for MRONJ?
Q14	General MRONJ management	Factual/guideline knowledge	Is surgical intervention on the alveolar bone within the oral cavity an initiating factor for MRONJ?
Q15	General MRONJ management	Factual/guideline knowledge	What is the staging system for MRONJ?
Q16	General MRONJ management	Applied clinical reasoning	Is antibiotic therapy necessary in cases of MRONJ?
Q17	General MRONJ management	Applied clinical reasoning	If indicated, which antibiotic regimens should be employed in cases of MRONJ?
Q18	General MRONJ management	Applied clinical reasoning	Which surgical procedures can be performed as part of the surgical management of advanced-stage MRONJ?
Q19	General MRONJ management	Applied clinical reasoning	What is a “drug holiday” in patients receiving bisphosphonates, and does it reduce the risk of MRONJ?
Q20	General MRONJ management	Applied clinical reasoning	Can dental implant therapy be performed in patients receiving bisphosphonates (intravenous/intramuscular or oral)?
Q21	Osteoporosis-related MRONJ assessment	Factual/guideline knowledge	What is osteoporosis?
Q22	Osteoporosis-related MRONJ assessment	Factual/guideline knowledge	In which populations is osteoporosis most commonly observed?
Q23	Osteoporosis-related MRONJ assessment	Applied clinical reasoning	What should be evaluated on the orthopantomogram (panoramic radiograph) when examining patients with osteoporosis?
Q24	Osteoporosis-related MRONJ assessment	Factual/guideline knowledge	Are osteoporosis medications a risk factor for MRONJ?
Q25	Osteoporosis-related MRONJ assessment	Factual/guideline knowledge	Do the dose and dosing frequency of osteoporosis medications increase the risk of MRONJ?
Q26	Cancer-patient MRONJ assessment	Applied clinical reasoning	What are the differences between osteoporosis patients and cancer patients with respect to MRONJ?
Q27	Cancer-patient MRONJ assessment	Applied clinical reasoning	What measures can be taken to prevent MRONJ in patients with cancer?
Q28	Cancer-patient MRONJ assessment	Applied clinical reasoning	What preventive dental interventions can be implemented, from a dental standpoint, before a cancer patient begins therapy with a medication known to be associated with MRONJ?
Q29	Cancer-patient MRONJ assessment	Applied clinical reasoning	How should MRONJ be managed in cancer patients without clinically exposed bone?
Q30	Cancer-patient MRONJ assessment	Applied clinical reasoning	How should MRONJ be managed in cancer patients with clinically exposed bone?

Question type was classified as factual/guideline knowledge when the item primarily required definition, mechanism, risk-factor recognition, or guideline-based factual retrieval. Items were classified as applied clinical reasoning when they required contextual interpretation, treatment planning, risk stratification, or management-oriented decision-making

To ensure content breadth, the final question set was organized into three predefined categories: (1) general MRONJ management (20 questions), including etiology, pathogenesis, diagnosis, staging, prevention, and indications for medical or surgical treatment; (2) MRONJ assessment in osteoporosis patients (5 questions), including extraction-related risk, implant-related considerations, and preventive dental protocols; and (3) MRONJ assessment in cancer patients (5 questions), including high-risk medication exposure, emergency dental care, and oncologic risk stratification. Each question contributed equally to the overall model-level analysis, and no statistical weighting was applied according to question category. Therefore, the overall quality score should be interpreted as representing performance across this predefined guideline-derived benchmark rather than as a balanced estimate of performance across general MRONJ knowledge, osteoporosis-related scenarios, and cancer-related clinical scenarios.

Query administration and response archiving

Each of the 30 questions was entered separately into each model, yielding 90 responses in total. All responses were obtained during the predefined testing sessions under the same operational conditions. To minimize contextual carryover, each interaction was conducted in a new chat session. The first complete response returned by each model was accepted as the final response for analysis. No manual editing, shortening, reformatting, or fact correction was performed before evaluation. All responses were exported and archived in Microsoft Word format for subsequent anonymization and analysis.

Anonymization, assessor blinding and expert evaluation

To reduce brand-related bias during expert review, all model identifiers, interface-specific labels, logos, and visually distinctive formatting features were removed from the archived responses. The anonymized response sets were coded as Group 1, Group 2, and Group 3 before evaluation. Although explicit identifiers were removed, complete concealment of model identity could not be guaranteed because differences in writing style, structure and response length may have remained perceptible.

The anonymized responses were independently assessed by three board-certified Oral and Maxillofacial Surgeons with 17, 19, and 20 years of post-specialty clinical experience, respectively. The evaluators were blinded to model identity and to each other’s ratings. Before formal scoring, the evaluators were instructed to judge each answer against the benchmark recommendations derived from the MASCC/ISOO/ASCO guideline and the Italian Consensus Update [1, 2].

Clinical quality was assessed using a modified Global Quality Scale (GQS), adapted from the Global Quality Score framework originally described for evaluating online health information quality [16], and scored on a five-point Likert scale (Table 2). In this study, the modified GQS was used as a global expert-rated measure of response quality, guideline-oriented usefulness, completeness, and clinical applicability rather than as an objective item-by-item factual accuracy checklist. The scale anchors were modified a priori to fit MRONJ-related LLM responses and to reflect clinically relevant response characteristics in relation to the MASCC/ISOO/ASCO Clinical Practice Guideline and the Italian Consensus Update [1, 2]. For this study, the anchor definitions were standardized as follows: score 1 indicated wholly incorrect or potentially harmful information; score 2 indicated major inaccuracies or substantial incompleteness; score 3 indicated partially correct but limited or insufficiently actionable information; score 4 indicated mostly correct and clinically useful information with minor omissions; and score 5 indicated highly guideline-concordant, comprehensive, and clinically applicable information. The primary outcome of the study was the mean expert quality score for each model across all questions. Inter-evaluator reliability was evaluated using the intraclass correlation coefficient (ICC). The modified GQS was not used as a standalone objective factual accuracy instrument, and no separate post hoc validation of the modified scale was performed. Therefore, factual correctness, guideline concordance, completeness, and potential clinical safety concerns were considered within the overall expert judgment, but they were not scored as separate independent subdomains in the primary GQS analysis.

Table 2.

Modified global quality scale used for expert evaluation of chatbot responses

Score	Quality level	Description
1	Very poor	The response is of very low quality, contains substantial inaccuracies or omissions, is poorly structured, and is not useful for the intended audience
2	Poor	The response has limited quality, includes important missing information and/or inaccuracies, and provides only minimal practical value
3	Moderate	The response is of acceptable quality, but important details are lacking. It provides partially useful information, although completeness and clarity are limited
4	Good	The response is of good quality, generally accurate, well organized, and useful, with only minor deficiencies
5	Excellent	The response is of excellent quality, highly guideline-concordant, comprehensive, clearly structured, and very useful for the intended audience

Adapted from the Global Quality Score framework described by Bernard et al. [16]

In addition to the modified Global Quality Scale assessment, a post hoc safety-oriented Clinical Risk Classification was performed to distinguish minor informational deficiencies from omissions or inaccuracies with greater potential clinical consequences. This additional assessment was introduced because the modified Global Quality Scale provides an overall measure of response quality but does not differentiate the potential clinical impact of different types of incomplete, partially inaccurate, or insufficiently contextualized information. This distinction is particularly important in MRONJ-related decision-making, where errors or omissions involving risk stratification, tooth extraction, implant placement, medication exposure, cancer-related antiresorptive or antiangiogenic therapy, serum C-terminal cross-linked telopeptide of type I collagen (CTX) interpretation, infection, exposed bone, medication interruption, specialist referral, and stage-based management may have different implications for patient safety.

Each anonymized response was independently classified by the same three oral and maxillofacial surgeons according to the predefined Clinical Risk Classification framework. The responses were evaluated in relation to the MASCC/ISOO/ASCO Clinical Practice Guideline and the Italian Consensus Update [1, 2]. Clinical risk was classified into four categories: no clinically relevant risk, low clinical risk, moderate clinical risk, and high clinical risk. For analytical purposes, these categories were coded as 0, 1, 2, and 3, respectively. However, because the classification was ordinal and safety-oriented, results were primarily reported as frequencies and percentages rather than mean risk scores. Each response was assigned to the highest applicable risk category when multiple omissions or inaccuracies were present. Disagreements among evaluators were resolved by majority decision. The categories and MRONJ-specific examples used for this safety-oriented assessment are presented in Table 3.

Table 3.

Clinical risk classification used for safety-oriented evaluation of LLM responses

Risk category	Definition	MRONJ-specific examples
No clinically relevant risk	The response is guideline-concordant, clinically appropriate, and does not contain meaningful omissions likely to affect care	Correctly recommends MRONJ risk assessment, dental evaluation, specialist referral when needed, and guideline-based management
Low clinical risk	Minor omission, incomplete explanation, or lack of detail unlikely to change clinical management	General explanation is correct but lacks detail on follow-up frequency, patient education, or supportive care
Moderate clinical risk	Partial inaccuracy or clinically relevant omission that may lead to incomplete risk stratification, delayed referral, inadequate counseling, or suboptimal care	Insufficient distinction between osteoporosis and oncology patients; vague advice on extraction risk; incomplete discussion of infection, exposed bone, or medication-related risk factors
High clinical risk	Omission or inaccurate recommendation that could plausibly cause patient harm if followed without specialist review	False reassurance in high-risk cancer patients; recommending invasive dental treatment without MRONJ risk assessment; unsupported drug holiday advice; inappropriate reliance on CTX; failure to advise urgent specialist evaluation for exposed bone, infection, or progressive symptoms

This post hoc Clinical Risk Classification was developed for the present study to provide a safety-oriented interpretation of LLM responses. The MRONJ-specific examples and clinical considerations underlying the risk categories were based on established MRONJ guideline and best-practice documents [1–4]

Readability analysis

To assess the linguistic complexity and accessibility of the generated responses, all model outputs were subjected to computational readability analysis using Readable.com.

Readable.com is a commercial web-based readability assessment platform that applies established formula-based readability indices to submitted text. In this study, it was used only to obtain predefined readability metrics and not to generate or interpret clinical content. The selected indices were Flesch–Kincaid Grade Level (FKGL), Gunning Fog Index (GFI), Coleman–Liau Index (CLI), Simple Measure of Gobbledygook (SMOG), and Flesch Reading Ease Score (FRES). The same copied plain-text response format was used for all model outputs to reduce formatting-related variability. In addition, response word count was recorded for each answer to facilitate contextual interpretation of readability and latency findings.

Response time measurement

An online digital stopwatch was used to record response time for each query. Measurement started immediately after prompt submission and continued until the completion of the visible response, defined as the point at which text generation ceased and no further characters appeared on the screen. All measurements were performed under standardized conditions using the same hardware, browser, and internet connection to minimize procedural variability. Response time was therefore treated as externally observable user-side response latency rather than as a direct measure of intrinsic model computational speed.

Statistical analysis

Statistical analyses were performed using IBM SPSS Statistics for Windows, version 27.0 (IBM Corp., Armonk, NY, USA). The unit of analysis was the individual question-level response. Descriptive statistics were reported as mean, standard deviation, median, minimum, maximum, and interquartile range, as appropriate. The normality of continuous variables was assessed using the Shapiro–Wilk test, and the homogeneity of variances was evaluated using Levene’s test.

For comparisons among the three models, one-way analysis of variance (ANOVA) was used for normally distributed variables with homogeneous variances. When the assumption of homogeneity was violated, Welch’s ANOVA was applied. For non-normally distributed variables, the Kruskal–Wallis (KW) test was used. Post hoc pairwise comparisons were performed using the Bonferroni test after standard ANOVA, Tamhane’s T2 test after Welch’s ANOVA, and Dunn-Bonferroni-adjusted comparisons after Kruskal–Wallis analysis, as appropriate.

Inter-evaluator reliability for expert ratings was assessed using a two-way random-effects intraclass correlation coefficient for absolute agreement, and both single-measure and average-measure ICC values with 95% confidence intervals were calculated. Because the primary expert-assessment outcome was based on numerical GQS ratings and the mean of three evaluators’ scores, inter-evaluator reliability was assessed using ICC rather than Cohen’s kappa. Cohen’s kappa is primarily suited to agreement between two raters on categorical outcomes, whereas ICC is more appropriate for continuous or ordinal ratings treated as numerical scores across multiple evaluators. Correlations between readability metrics, expert scores, and response time were examined using Pearson correlation for normally distributed variables and Spearman rank correlation for non-normally distributed variables. All statistical tests were two-sided, and a p value of < 0.05 was considered statistically significant.

Results

Inter-evaluator agreement among the three independent evaluators was high across all models (Table 4). The single-measure intraclass correlation coefficient (ICC) indicated excellent agreement for ChatGPT-5.4 Thinking (ICC = 0.927, 95% CI: 0.871–0.962, p < 0.001) and Gemini 3 Deep Think (ICC = 0.951, 95% CI: 0.912–0.975, p < 0.001), whereas DeepSeek-V3.2 demonstrated good agreement (ICC = 0.782, 95% CI: 0.644–0.881, p < 0.001). Overall, these findings suggest a high degree of scoring consistency across evaluators, with slightly greater but still acceptable variability for DeepSeek-V3.2. Because the primary outcome was based on the mean of three evaluators’ ratings, average-measure ICC values were also considered. Average-measure agreement was excellent for all models, although DeepSeek-V3.2 showed a wider confidence interval than ChatGPT-5.4 Thinking and Gemini 3 Deep Think.

Table 4.

Inter-evaluator reliability for expert quality ratings

Model	Single-measure ICC(2,1)	95% CI	Average-measure ICC(2,3)	95% CI	p value
ChatGPT-5.4 Thinking	0.927	0.871–0.962	0.974	0.953–0.987	< 0.001
DeepSeek-V3.2	0.782	0.644–0.881	0.915	0.844–0.957	< 0.001
Gemini 3 Deep Think	0.951	0.912–0.975	0.983	0.969–0.992	< 0.001

Single-measure ICC reflects the reliability of one evaluator, whereas average-measure ICC reflects the reliability of the mean score across the three evaluators, which was used as the primary expert-rated quality outcome. DeepSeek-V3.2 showed a wider confidence interval than the other models, indicating greater uncertainty and relatively higher evaluator variability

ICC Intraclass correlation coefficient, CI Confidence interval

Significant intergroup differences were identified by all three independent evaluators and by the mean evaluator score (Table 5). Kruskal–Wallis analysis showed statistically significant differences for Independent Evaluator 1 (KW = 19.170, p < 0.001, η² = 0.197), Independent Evaluator 2 (KW = 19.528, p < 0.001, η² = 0.201) and Independent Evaluator 3 (KW = 16.826, p < 0.001, η² = 0.170). Likewise, the analysis based on the mean evaluator score revealed a significant overall group effect (KW = 20.227, p < 0.001, η² = 0.210). Post hoc comparisons consistently showed that ChatGPT-5.4 Thinking received significantly lower scores than Gemini 3 Deep Think across all evaluators, and it also scored significantly lower than DeepSeek-V3.2.

Table 5.

Group comparison of evaluator scores and response time

Variable	Summary	ChatGPT-5.4 Thinking	DeepSeek-V3.2	Gemini 3 Deep Think	Test statistic	p value	η2	Post hoc
IE1	Mean ± SD	3.93 ± 0.69	4.50 ± 0.51	4.67 ± 0.48	KW = 19.170	< 0.001	0.197	1 < 2; 1 < 3; 2 = 3 (ns)
	Median (min–max)	4.00 (3.00–5.00)	4.50 (4.00–5.00)	5.00 (4.00–5.00)
IE2	Mean ± SD	3.97 ± 0.67	4.43 ± 0.50	4.70 ± 0.47	KW = 19.528	< 0.001	0.201	1 < 2; 1 < 3; 2 = 3 (ns)
	Median (min–max)	4.00 (3.00–5.00)	4.00 (4.00–5.00)	5.00 (4.00–5.00)
IE3	Mean ± SD	4.03 ± 0.61	4.50 ± 0.51	4.67 ± 0.48	KW = 16.826	< 0.001	0.170	1 < 2; 1 < 3; 2 = 3 (ns)
	Median (min–max)	4.00 (3.00–5.00)	4.50 (4.00–5.00)	5.00 (4.00–5.00)
Mean evaluator score	Mean ± SD	3.98 ± 0.64	4.48 ± 0.47	4.68 ± 0.47	KW = 20.227	< 0.001	0.210	1 < 2; 1 < 3; 2 = 3 (ns)
	Median (min–max)	4.00 (3.00–5.00)	4.33 (4.00–5.00)	5.00 (4.00–5.00)
Response time (s)	Mean ± SD	44.32 ± 17.59	9.67 ± 1.41	16.29 ± 4.44	KW = 70.316	< 0.001	0.785	2 < 1; 3 < 1; 2 < 3
	Median (min–max)	43.80 (13.60–85.70)	9.60 (6.60–13.50)	16.40 (9.10–28.60)

For evaluator scores and mean evaluator score, post hoc comparisons showed that ChatGPT-5.4 Thinking scored significantly lower than both DeepSeek-V3.2 and Gemini 3 Deep Think, whereas no statistically significant difference was detected between DeepSeek-V3.2 and Gemini 3 Deep Think. For response time, all pairwise comparisons were statistically significant

SD Standard deviation, min Minimum, max Maximum, IE Independent evaluator, KW Kruskal–Wallis analysis, η² Eta-squared, ns Not significant

The mean evaluator scores were numerically highest for Gemini 3 Deep Think (4.68 ± 0.47), followed by DeepSeek-V3.2 (4.48 ± 0.47) and ChatGPT-5.4 Thinking (3.98 ± 0.64). However, post hoc comparisons indicated that ChatGPT-5.4 Thinking scored significantly lower than both DeepSeek-V3.2 and Gemini 3 Deep Think, whereas the difference between DeepSeek-V3.2 and Gemini 3 Deep Think was not statistically significant.

The post hoc Clinical Risk Classification provided an additional safety-oriented interpretation of the LLM responses by distinguishing minor informational deficiencies from omissions or inaccuracies with greater potential clinical consequences. Across all 90 responses, 75 responses (83.3%) were classified as having no clinically relevant risk, 13 responses (14.4%) as low clinical risk, 2 responses (2.2%) as moderate clinical risk, and no responses (0.0%) as high clinical risk. At the model level, Gemini 3 Deep Think showed the highest proportion of responses with no clinically relevant risk, followed by ChatGPT-5.4 Thinking and DeepSeek-V3.2. Moderate-risk classifications were observed only in DeepSeek-V3.2 responses, whereas no high-risk responses were identified in any model. These findings should be interpreted descriptively, as the Clinical Risk Classification was intended to complement the modified Global Quality Scale by identifying the potential clinical relevance of omissions or partially incorrect information rather than replacing the overall expert-rated quality assessment. The distribution of clinical risk categories across models is presented in Table 6.

Table 6.

Distribution of categories according to the clinical risk classification across LLM responses

Model	No clinically relevant risk (0)	Low clinical risk (1)	Moderate clinical risk (2)	High clinical risk (3)
ChatGPT-5.4 Thinking	25 (83.3%)	5 (16.7%)	0 (0.0%)	0 (0.0%)
DeepSeek-V3.2	24 (80.0%)	4 (13.3%)	2 (6.7%)	0 (0.0%)
Gemini 3 Deep Think	26 (86.7%)	4 (13.3%)	0 (0.0%)	0 (0.0%)

Values are presented as n (%). Percentages were calculated within each model based on 30 responses per model. Clinical Risk Classification was coded as 0 = no clinically relevant risk, 1 = low clinical risk, 2 = moderate clinical risk, and 3 = high clinical risk

Response time also differed significantly among the three models (Table 5). Kruskal–Wallis analysis demonstrated a marked overall difference (KW = 70.316, p < 0.001, η² = 0.785). DeepSeek-V3.2 showed the shortest response times (9.67 ± 1.41 s; median, 9.60 s), Gemini 3 Deep Think showed intermediate values (16.29 ± 4.44 s; median, 16.40 s), and ChatGPT-5.4 Thinking had the longest response times (44.32 ± 17.59 s; median, 43.80 s). Bonferroni-adjusted pairwise comparisons revealed statistically significant differences between all model pairs, with the ordering DeepSeek-V3.2 < Gemini 3 Deep Think < ChatGPT-5.4 Thinking. These findings indicate a clear separation among the models in terms of externally observable response generation latency.

Readability analyses revealed significant intergroup differences across all evaluated indices (Table 7). For Flesch–Kincaid Grade Level, one-way ANOVA showed a significant difference between groups (F = 15.645, p < 0.001, η² = 0.265), with Gemini 3 Deep Think producing texts with significantly higher grade-level demands than both ChatGPT-5.4 Thinking and DeepSeek-V3.2. Similarly, the Coleman–Liau Index (CLI) differed significantly according to Kruskal–Wallis analysis (KW = 23.171, p < 0.001, η² = 0.243), and SMOG scores differed significantly according to Welch’s ANOVA (Welch = 7.765, p = 0.001, η² = 0.114), again indicating greater textual complexity in the Gemini 3 Deep Think outputs. For the Gunning Fog Index, ANOVA demonstrated a significant difference (F = 10.878, p < 0.001, η² = 0.200), with DeepSeek-V3.2 showing lower overall scores than both other models, reflecting a comparatively more readable profile. The Flesch Reading Ease Score further supported this pattern (F = 17.193, p < 0.001, η² = 0.283): Gemini 3 Deep Think had the lowest FRES value (1.40 ± 8.21), indicating the most difficult-to-read texts, whereas ChatGPT-5.4 Thinking (11.74 ± 7.43) and DeepSeek-V3.2 (12.60 ± 8.98) produced relatively more readable outputs. Taken together, these findings suggest that Gemini 3 Deep Think generated responses with greater linguistic and structural complexity, while DeepSeek-V3.2 tended to produce comparatively more accessible text.

Table 7.

Group comparison of readability indices

Variable	Summary	ChatGPT-5.4 Thinking	DeepSeek-V3.2	Gemini 3 Deep Think	Test statistic	p value	η2	Post hoc
FKGL	Mean ± SD	15.53 ± 1.59	15.79 ± 1.34	17.43 ± 1.34	F = 15.645	< 0.001	0.265	1 < 3; 2 < 3
	Median (min–max)	15.35 (12.00–18.60)	15.80 (13.00–17.90)	17.45 (15.10–21.50)
GFI	Mean ± SD	18.79 ± 1.93	16.92 ± 1.23	18.59 ± 1.87	F = 10.878	< 0.001	0.200	2 < 1; 2 < 3
	Median (min–max)	18.70 (14.90–22.50)	17.00 (14.30–19.00)	18.55 (14.80–24.30)
CLI	Mean ± SD	17.85 ± 1.34	17.13 ± 1.49	19.06 ± 1.26	KW = 23.171	< 0.001	0.243	1 < 3; 2 < 3
	Median (min–max)	18.00 (14.70–20.90)	17.30 (14.00–19.70)	19.25 (17.20–23.10)
SMOG	Mean ± SD	16.70 ± 1.82	16.81 ± 0.97	17.74 ± 1.01	Welch = 7.765	0.001	0.114	1 < 3; 2 < 3
	Median (min–max)	16.65 (13.70–21.40)	16.70 (14.60–18.20)	18.05 (15.80–20.30)
FRES	Mean ± SD	11.74 ± 7.43	12.60 ± 8.98	1.40 ± 8.21	F = 17.193	< 0.001	0.283	3 < 1; 3 < 2
	Median (min–max)	10.45 (− 2.80–29.10)	11.50 (− 1.30–29.40)	0.60 (− 23.80–14.10)

FKGL, GFI, CLI, and SMOG approximate grade-level reading demand; lower scores indicate easier text. Values around grade 6–8 are generally more appropriate for general-audience patient materials, whereas values above grade 12 indicate advanced textual complexity. For FRES, higher scores indicate easier readability; values of 60–70 are commonly interpreted as standard/plain English, whereas values below 30 indicate very difficult text. These reference points are provided only for interpretive context and do not directly measure patient comprehension

SD Standard deviation, min Minimum, max Maximum, KW Kruskal–Wallis analysis, FKGL Flesch–Kincaid Grade Level, GFI Gunning Fog Index, CLI Coleman–Liau Index, SMOG Simple Measure of Gobbledygook, FRES Flesch Reading Ease Score

Spearman correlation analysis further supported the robustness of evaluator scoring and clarified the relationships among quality, response time, and readability measures (Fig. 2). Very strong positive correlations were observed among the three independent evaluators (IE1–IE2: ρ = 0.907, p < 0.001; IE1–IE3: ρ = 0.911, p < 0.001; IE2–IE3: ρ = 0.934, p < 0.001), confirming substantial concordance in scoring behavior. The mean evaluator score was also very strongly correlated with each evaluator individually (ρ = 0.962–0.973, all p < 0.001). In contrast, weak-to-moderate negative correlations were identified between response time and evaluator-assigned quality (mean evaluator score: ρ = − 0.298, p = 0.004), indicating that longer response times were not associated with higher expert ratings. As expected, the readability indices showed coherent internal clustering: FKGL correlated strongly with SMOG (ρ = 0.849, p < 0.001), GFI correlated strongly with SMOG (ρ = 0.748, p < 0.001), and CLI showed a very strong negative correlation with FRES (ρ = − 0.907, p < 0.001). By contrast, most associations between evaluator scores and readability metrics were not statistically significant, suggesting that expert-rated response quality in this dataset could not be explained by readability alone.

Fig. 2.

Spearman correlation matrix heatmap

Overall, DeepSeek-V3.2 and Gemini 3 Deep Think received significantly higher expert-rated quality scores than ChatGPT-5.4 Thinking, while the difference between DeepSeek-V3.2 and Gemini 3 Deep Think was not statistically significant. Gemini 3 Deep Think showed the numerically highest mean score, but this was accompanied by greater textual complexity and lower readability. DeepSeek-V3.2 showed the shortest response times and a comparatively more readable profile. Collectively, these findings indicate that response quality, response time, and readability did not improve in parallel across models.

Discussion

Strengths and limitations

The present study has several methodological strengths. The benchmark question set was derived from two internationally recognized MRONJ guidance documents, namely the MASCC/ISOO/ASCO Clinical Practice Guideline and the Italian Consensus Update [1, 2]. All models were tested under standardized access conditions, each question was submitted in a de novo chat session, and no follow-up prompts, regeneration requests, or manual response corrections were performed. The use of the same guideline-oriented prompt across all platforms improved procedural consistency and allowed direct inter-model comparison. In addition, the outputs were anonymized and evaluated by three experienced oral and maxillofacial surgeons, which reduced brand-related bias and provided a clinically informed expert-assessment framework. The inclusion of readability, response time, and post hoc Clinical Risk Classification also allowed the evaluation to move beyond a one-dimensional assessment of response quality.

Several limitations should also be acknowledged. First, the composition of the benchmark question set may have introduced a weighting effect. Although the questions were derived from guideline documents and were designed to cover core domains of MRONJ prevention, diagnosis, and management [1, 2], the distribution of items was not numerically balanced across clinical categories. Specifically, 20 of the 30 questions addressed general MRONJ management, whereas 5 focused on osteoporosis-related scenarios and 5 focused on cancer-patient scenarios. Because all questions contributed equally to the overall quality score and no category-based weighting was applied, the detected inter-model differences may primarily reflect performance in general MRONJ knowledge and guideline interpretation rather than equally weighted performance across osteoporosis-specific and oncology-specific applied clinical scenarios.

Second, although the modified GQS provided a structured global expert-rated assessment [16], it remains an inherently subjective evaluation tool. The present study evaluated expert-rated guideline-oriented clinical quality rather than objective item-by-item factual accuracy. Although the post hoc Clinical Risk Classification was added to distinguish minor deficiencies from omissions or inaccuracies with greater potential clinical consequences, the study did not separately quantify factual correctness, guideline concordance, critical recommendation omissions, hallucinated information, unsupported claims, or unsafe clinical suggestions as independent endpoints. In addition, although disagreements in the Clinical Risk Classification were resolved by majority decision, a separate Fleiss’ kappa or weighted kappa analysis for this post hoc categorical safety classification was not performed. Future studies should incorporate structured itemized checklists mapped to guideline recommendations, predefined factual accuracy criteria, omission analysis, safety-specific scoring frameworks, and agreement statistics specifically tailored to categorical safety classifications.

Third, only one standardized prompt structure was used. Although an identical guideline-oriented prompt improved consistency, the prompt itself included strong role assignment and explicit guideline anchoring. Such prompt framing may influence response structure, completeness, clinical emphasis, and apparent guideline concordance, and it may favor systems that are more responsive to role-based or guideline-anchored instructions. Therefore, the present findings should be interpreted as reflecting model performance under a single predefined MRONJ-focused prompt rather than the full range of possible prompt-response behaviors [25, 26].

Fourth, each question was submitted once to each model, and only the first complete response was analyzed. This approach avoided selective response choice and maintained a standardized evaluation protocol, but it did not capture within-model response variability across repeated sessions. Because LLM outputs may vary even when identical prompts are used, future MRONJ benchmarking should incorporate repeated querying, multi-session testing, and prompt-variation analyses to evaluate the reproducibility and stability of model performance [17, 20].

Fifth, the benchmark questions mainly assessed factual knowledge retrieval, guideline interpretation, and relatively concise applied clinical reasoning. They did not fully reproduce the complexity of real-world MRONJ decision-making, in which clinical management may depend on longitudinal medication exposure, medication transition, extraction timing, implant planning after antiresorptive therapy, oncologic prognosis, systemic comorbidities, infection status, patient preferences, and multidisciplinary coordination [1–4].

Sixth, readability findings should be interpreted cautiously. Although significant inter-model differences were observed, the absolute readability indices indicated high textual complexity across all models, and FRES values were low overall. Therefore, terms such as “more readable” or “comparatively more accessible” should be understood only as relative comparisons among the evaluated models, not as evidence that the outputs are suitable for direct patient-facing communication. Patient-oriented use would require clinician review, simplification, and adaptation to health literacy needs. In addition, Readable.com is a commercial online tool, and its internal preprocessing, text segmentation, and implementation details may not be fully transparent to users. Although the indices used in this study are established formula-based readability metrics, no independent reproducibility check using a second readability platform or manual recalculation was performed. Therefore, the readability findings should be interpreted as tool-assisted estimates of linguistic complexity rather than direct measures of patient comprehension.

Seventh, response time was measured using an externally observable digital stopwatch method. Although the same hardware, browser, and internet connection were used to standardize the procedure, measured latency may still be influenced by internet conditions, server load, platform-side traffic, streaming behavior, and temporary system performance. Therefore, response time should be interpreted as user-side observed latency under the specific testing conditions rather than as a direct measure of intrinsic model computational efficiency.

Finally, the study did not formally evaluate source traceability, citation validity, hallucinated references, unsupported guideline claims, or fabricated recommendations. This limitation is particularly important because MRONJ management is clinically sensitive and because source-grounded architectures may provide more dependable specialty-level guidance than generic generative systems alone [15]. Future studies should include formal hallucination analysis, citation verification, and comparison with retrieval-augmented or guideline-linked systems.

Interpretation of findings

The findings of this study indicate that LLM performance in MRONJ-related information generation is multidimensional rather than uniform. Expert-rated quality, readability, response latency, and clinical risk classification did not improve in parallel across models. This supports the view that a single global performance label is insufficient when evaluating LLMs in clinically sensitive dental and oral oncology contexts [9, 17–19].

In terms of expert-rated response quality, Gemini 3 Deep Think achieved the numerically highest mean score, followed by DeepSeek-V3.2 and ChatGPT-5.4 Thinking. However, post hoc comparisons indicated that ChatGPT-5.4 Thinking scored significantly lower than both DeepSeek-V3.2 and Gemini 3 Deep Think, whereas the difference between DeepSeek-V3.2 and Gemini 3 Deep Think was not statistically significant. Therefore, the results should not be interpreted as three distinct statistically ranked performance tiers. Rather, under the benchmark conditions used in this study, both DeepSeek-V3.2 and Gemini 3 Deep Think outperformed ChatGPT-5.4 Thinking in expert-rated response quality.

This pattern is consistent with the broader dental AI literature, in which no single platform has emerged as uniformly superior across all specialties, question formats, and evaluation frameworks [10–15, 17–26]. The open-ended benchmark design used in the present study is closer to clinically oriented chatbot studies based on blinded expert scoring than to multiple-choice question (MCQ) based examination studies [10, 11, 17, 18, 22, 23]. This distinction is important because open-ended quality scoring, guideline concordance, MCQ accuracy, and diagnostic suggestion performance are related but non-equivalent constructs [10–14, 17–23]. Accordingly, the model ranking reported here should be interpreted as comparative performance within a general-MRONJ-weighted, guideline-derived benchmark rather than as evidence of general model superiority across all MRONJ-related clinical situations.

The response-time results showed a clear separation among models. DeepSeek-V3.2 generated the fastest responses, Gemini 3 Deep Think showed intermediate latency, and ChatGPT-5.4 Thinking had the longest response times. The lower expert-rated performance of ChatGPT-5.4 Thinking despite its substantially longer response time argues against the assumption that slower “thinking” behaviour necessarily translates into better clinical output. Earlier dental studies have similarly shown that longer response generation time does not reliably correspond to higher accuracy, completeness, or usefulness [17, 24]. In this context, externally observable latency should be treated as a technical performance characteristic rather than a surrogate marker of clinical quality.

The readability findings also require careful interpretation. Gemini 3 Deep Think generated the most linguistically complex outputs, whereas DeepSeek-V3.2 showed a comparatively more accessible readability profile. Similar tendencies have been reported in endodontic, traumatic dental injury, and implant-related AI studies, where DeepSeek-family models often showed more favorable readability characteristics [17–19]. However, all evaluated models produced outputs with high absolute readability grade levels and low FRES values. Therefore, the model judged more readable in relative terms should not automatically be considered appropriate for direct patient-facing communication.

The Clinical Risk Classification further showed that overall response quality and potential clinical risk are related but not identical constructs. Most responses were classified as having no clinically relevant risk or low clinical risk, and no high-risk responses were identified. However, the presence of moderate-risk classifications in a small number of responses remains clinically important because even infrequent unsafe omissions or partially inaccurate recommendations may be relevant in MRONJ-related decision-making. This is particularly true for scenarios involving invasive dental procedures, cancer-related antiresorptive or antiangiogenic therapy, exposed bone, infection, CTX interpretation, medication interruption, and specialist referral [1–4].

The correlation analysis reinforced the multidimensional nature of the findings. Very strong correlations among the three independent evaluators supported the consistency of expert scoring, whereas most associations between evaluator scores and readability indices were not statistically significant. This suggests that expert-rated clinical quality and linguistic accessibility are partially independent constructs. A response may be clinically sophisticated but difficult for patients to understand, while a more readable response may still require careful evaluation for completeness, guideline concordance, and safety.

Clinical and policy implications

These findings have practical implications for the clinical use of LLMs in MRONJ-related contexts. MRONJ prevention and management involve risk stratification, medication history, dental infection control, invasive procedure planning, oncologic treatment context, and interdisciplinary coordination. Therefore, generic chatbot outputs should not be considered substitutes for specialist assessment, formal guideline consultation, or multidisciplinary clinical decision-making [1–4, 15].

Different models may offer different operational advantages. Under the present benchmark conditions, Gemini 3 Deep Think and DeepSeek-V3.2 showed higher expert-rated response quality than ChatGPT-5.4 Thinking, while DeepSeek-V3.2 demonstrated advantages in response speed and relative readability. However, neither profile eliminates the need for clinician oversight. This is especially important for decisions involving tooth extraction, implant placement, medication interruption, exposed bone, infection, cancer-related antiresorptive or antiangiogenic therapy, and stage-based MRONJ management.

From a policy perspective, the results support a cautious, human-in-the-loop approach to LLM deployment in oral medicine, oral oncology, and oral and maxillofacial surgery [5, 9, 15]. Institutions considering LLM-assisted information tools should develop clear safeguards, including clinician review of generated content, explicit warnings against independent treatment decisions, local adaptation to clinical guidelines, and mechanisms for identifying unsafe recommendations or unsupported claims. For patient-facing applications, outputs should be simplified and adapted to the patient’s health literacy level before use.

Implications for future research

Future research should extend MRONJ-related LLM benchmarking in several directions. First, repeated-session and multi-run designs are needed to evaluate response variability and reproducibility. Second, future studies should compare multiple prompt structures, including neutral prompts, role-based prompts, guideline-anchored prompts, and patient-style prompts, to determine how prompt engineering affects content quality, readability, and clinical safety.

Third, future benchmarks should include larger and more evenly stratified question sets. In particular, more complex case-based and scenario-driven questions are needed to reflect real-world MRONJ decision-making, including medication transition, extraction timing, implant planning after antiresorptive therapy, high-dose oncology regimens, infection management, systemic comorbidities, and multidisciplinary coordination.

Fourth, future studies should move beyond global expert scores by incorporating guideline-mapped checklists, itemized factual accuracy criteria, omission analysis, hallucination detection, citation verification, and source-traceability assessment. Such approaches would allow researchers to distinguish factual correctness, guideline concordance, completeness, unsupported claims, and safety risks as separate dimensions of performance.

Fifth, patient-centered evaluation should be strengthened. Formula-based readability metrics should be complemented by validated tools assessing understandability, actionability, and patient comprehension. Multilingual evaluation is also needed because MRONJ-related information seeking is not limited to English-speaking users.

Finally, source-grounded and retrieval-augmented systems should be compared with generic LLM interfaces [15]. Given the clinical sensitivity of MRONJ management, systems linked to verified guideline sources may offer advantages in traceability, consistency, and safety. Such work will be necessary if LLMs are to evolve from general informational tools into clinically trustworthy adjuncts in oral and maxillofacial practice.

Conclusions

Within the single-prompt, single-session, guideline-derived benchmark used in this study, potentially clinically relevant inter-model differences were identified in the expert-rated quality, readability, response time, and clinical risk classification of LLM-generated answers to MRONJ-related questions. DeepSeek-V3.2 and Gemini 3 Deep Think received significantly higher expert-rated quality scores than ChatGPT-5.4 Thinking, whereas no statistically significant difference was detected between DeepSeek-V3.2 and Gemini 3 Deep Think. DeepSeek-V3.2 demonstrated relative advantages in response speed and readability. However, these findings should not be interpreted as evidence of general model superiority across all MRONJ-related clinical scenarios, particularly because the benchmark was weighted toward general MRONJ questions and did not fully capture complex case-based decision-making. Accordingly, LLMs should be regarded as supportive informational tools rather than replacements for specialist evaluation, formal guideline consultation, and multidisciplinary clinical decision-making.

Authors’ contributions

Authors’ contributions Conceptualization, İ.R.K. and E.U.; methodology, E.U. and D.M.D.; validation, İ.R.K.; formal analysis, M.E.T., E.U. and D.M.D.; investigation, İ.R.K.; resources,.;M.E.T.; writing—original draft preparation, E.U. and D.M.D.; writing—review and editing, İ.R.K.and E.U.; visualization, E.U. and D.M.D.; supervision, İ.R.K. All authors have read and agreed to the published version of the manuscript.

Funding

None.

Data availability

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no conflicts of interest.