LLM-assisted multidisciplinary decision-making in lung cancer: efficacy of Gemini-2.5-Pro and GPT-5 in a real-world MDT setting

Abstract

Background

Multidisciplinary team (MDT) review is essential for individualized treatment planning in lung cancer, where decisions often require integration of staging, pathology, molecular findings, performance status, and multimodal treatment options. However, the effective implementation of MDT review remains constrained by limited specialist availability in resource-limited hospitals and by high patient volume, heavy physician workload, and tight clinical schedules in tertiary care centers. Large language models (LLMs) may support MDT workflows by synthesizing clinical information and generating preliminary treatment recommendations for clinician review. This study compared GPT-5 and Gemini-2.5-Pro with expert MDT recommendations in real-world lung cancer cases and evaluated how information completeness and input format influenced LLM-generated treatment recommendations.

Methods

We conducted a head-to-head evaluation of GPT-5 and Gemini-2.5-Pro for lung-cancer multidisciplinary team decision support. For each of 150 consecutively collected cases, structured vignettes were entered using three information-completeness levels in both natural-language and isomorphic JavaScript Object Notation (JSON) formats. Each model produced a single-sentence recommendation. Outputs were compared with the expert multidisciplinary team plan, which was treated as a clinical reference standard for concordance analysis rather than an objective guideline-only gold standard, using predefined plan-level categories. To better quantify model performance, we applied keyword extraction and semantic matching to normalize both multidisciplinary team recommendations and model outputs into ten treatment categories with subsequent manual review and correction.

Results

Stratifying by information level and format, Information Level III produced the best overall performance for Gemini-2.5-Pro under natural-language input, with F1 0.7603, accuracy 0.8647, precision 0.7031, and recall 0.8278. GPT-5 achieved its highest recall under Information Level III with JSON input (recall 0.9132), but this was accompanied by lower precision and F1. Under the same natural-language Information Level III, GPT-5 reached recall 0.8770 with F1 0.6826 and accuracy 0.7967.

Conclusions

Gemini-2.5-Pro show promise as adjuncts to MDT decision-making in lung cancer, offering high agreement with expert plans under controlled conditions. With appropriate clinician oversight, LLM-based tools may help expand access to standardized, high-quality recommendations where specialist availability is limited.

Supplementary Information

The online version contains supplementary material available at https://doi.org/10.1186/s12911-026-03723-7.

Introduction

Lung cancer remains the leading cause of cancer-related mortality worldwide [1]. It comprises various histological subtypes, including but not limited to non-small cell lung cancer, small cell lung cancer, and several rarer forms [2]. Effective management of lung cancer requires the integration of diverse clinical expertise, including oncology, radiology, pathology, thoracic surgery, and radiotherapy, to formulate individualized and evidence-based treatment strategies. Multidisciplinary team (MDT) meetings are now recognized as a cornerstone of contemporary oncologic care [3, 4]. Beyond facilitating the formulation of optimal treatment recommendations, MDT provide essential oversight to ensure the appropriateness and consistency of therapeutic strategies [5]. Nevertheless, the implementation of MDT remains challenging in resource-limited hospitals. Even in tertiary care centers with multiple specialists, MDT delivery can be constrained by the number of cases requiring review, the specialists scheduling availability, and the need to maintain consistent guideline-based reasoning across meetings. Under these circumstances, there is an urgent need for innovative approaches that can support and augment clinical decision-making, thereby guaranteeing equitable access to high-quality, standardized cancer care irrespective of institutional constraints.

In recent years, large language models (LLMs) have undergone remarkable advancements, propelled by progress in natural language processing algorithms and deep learning architectures [6]. These models now demonstrate an impressive ability to comprehend complex medical literature, synthesize diverse sources of clinical evidence, and generate contextually relevant recommendations across a broad spectrum of healthcare scenarios [7–10]. By harnessing these capabilities, LLMs such as Gemini-2.5-Pro and GPT-5 present novel opportunities to enhance MDT decision-making in lung cancer management [11]. By leveraging the advanced LLMs, we can improve the quality and consistency of care for lung cancer patients, regardless of institutional resources or geographic location.

To date, real-world evidence regarding the effectiveness and reliability of LLM-based tools in MDT settings for lung cancer remains limited [12, 13]. In this study, we performed an evaluation of Gemini-2.5-Pro and GPT-5 against expert MDT consensus recommendations, and assessed how information completeness and input format influenced LLM-generated lung cancer treatment recommendations. By employing the consensus outcomes of experienced multidisciplinary teams as the reference standard, we evaluate the consistency, efficiency, and potential clinical utility of LLM-assisted MDT discussions. This comparative analysis aims to elucidate the strengths and limitations associated with integrating LLMs into the MDT workflow. Notably, such technologies hold particular promise in regions with limited access to specialist expertise, where LLMs may help bridge gaps in clinical knowledge and facilitate broader access to high-quality, standardized cancer care. Our findings may inform the adoption of advanced LLMs in routine oncologic practice and contribute to the optimization of multidisciplinary cancer management.

Materials and methods

Patient cohort

Cases were consecutively collected from patients with lung cancer who firstly presented for care between October 13 and October 17, 2025 in the Shandong Cancer Hospital. Exclusion criteria included missing essential clinical or staging information, absence of a verifiable MDT consensus recommendation, duplicate records, or inadequate de-identification for analysis. This study included 150 patients who underwent institutional MDT review, with sufficient de-identified clinical information for constructing standardized case vignettes and a documented final MDT consensus recommendation. The data was approved by the Shandong Cancer Hospital Institutional Review Board (SDTHEC2024007020) in accordance with the Declaration of Helsinki, and the requirement for informed consent was waived due to the retrospective anonymized study.

The lung cancer MDT disciplines include respiratory medicine, thoracic surgery, thoracic radiation oncology, radiology, and pathology. In a typical meeting, one to three senior experts from each core discipline attended according to the clinical schedule. The core participating disciplines remained consistent during the study period. MDT recommendations were generated through case presentation, cross-disciplinary discussion, and final consensus led by the attending MDT chair. When differing opinions arose during MDT meetings, they were resolved through multidisciplinary discussion, and only the final consensus recommendation was used as the clinical reference standard. Tumor characteristics and relevant clinical data were extracted from the on-site MDT presentation materials used by the experts during the MDT meeting and anonymized before LLM querying to ensure confidentiality. Tumor stage was assigned according to the AJCC/UICC TNM classification system. Overall stages I–IV were determined by predefined combinations of the primary tumor extent (T), regional lymph node involvement (N), and distant metastasis status (M). Cases with M1 disease were classified as stage IV irrespective of T or N category [14].

Large language model

Large Language Models (LLMs) are complex neural networks based on the Transformer architecture, which enables them to process and understand language through self-attention mechanisms. Through training on extensive and diverse text corpora, they develop a profound ability to recognize linguistic patterns, generate coherent, human-like text, and respond to prompts with a high degree of contextual awareness. In the clinical realm, this translates to a powerful capacity to analyze and synthesize vast amounts of unstructured data—from medical literature to clinical guidelines—thereby aiding healthcare professionals by providing relevant, summarized information for enhanced clinical decision support.

GPT-5 is an instruction-tuned model built on a scalable transformer architecture and is regarded as one of the leading general-purpose LLMs for complex clinical reasoning and safety-aligned text generation, with strong performance reported across professional and biomedical benchmarks.

Gemini-2.5-Pro is a publicly accessible chatbot based on an advanced transformer model capable of producing context-aware, human-like responses; it is trained on a broad and up-to-date corpus that includes open-access web sources up to early 2024 [11, 15]. Prior reports have noted notable performance of Gemini models in several medical subfields, including ophthalmology, laryngoscopic examination, and breast reconstruction [16–18].

When constrained by structured prompts, LLMs can summarize complex patient data and propose treatment options in a concise, reproducible manner, making them suitable for benchmarking against multidisciplinary team (MDT) decisions. In this study, we evaluated two contemporary LLMs, GPT-5 (OpenAI) and Gemini-2.5-Pro (Google) as question–answering systems for lung cancer MDT decision support, comparing their recommendations with expert MDT plans.

Methods

Figure 1 presents the overall framework for constructing the LLM–assisted MDT recommendation system for lung cancer management. The pipeline integrates clinical data processing, multi-format prompt construction, LLM-based inference, and structured postprocessing. By combining both narrative and structured inputs across multiple levels of clinical completeness, the framework enables a systematic assessment of how different prompting strategies influence model-generated therapeutic suggestions.

Fig. 1.

Overall workflow of the LLM-assisted lung cancer MDT recommendation system

To approximate the information available in real MDT decision-making, the LLM inputs were constructed from the de-identified clinical materials available to MDT experts at the time of discussion rather than from artificially simplified case descriptions.

The workflow comprises three major steps. First, prompt generation was performed by consolidating multiple clinical data sources, including demographic information, functional status (KPS), pathology reports, radiological interpretations, TNM staging, etc. These data elements were standardized and synthesized into question–answer (QA) pairs, forming the foundation for prompt design. The prompt were formulated into two distinct information styles. We constructed two types of prompts: an narrative-style prompt that mimics real world MDT case presentations, and a JSON-style structured format that provides machine-readable clinical attributes. JSON refers to JavaScript Object Notation, a structured key-value data format. Detailed examples in JSON format are provided in Supplementary. Each case was further organized into three information levels (I–III) to examine the influence of information completeness on model recommendations. As shown in Fig. 1, Information Level I included core clinical variables required for preliminary treatment recommendation, including age, sex, KPS score, previous treatment history, histologic subtype, clinical stage, metastatic status, PD-L1 expression, and genomic testing results. Information Level II added information from additional clinical examinations, and Information Level III further incorporated chest contrast-enhanced CT imaging diagnosis, thereby representing progressively increasing levels of clinical completeness. These three levels were constructed in parallel for both narrative-style and JSON-style prompts. Each case was represented at all three information levels, enabling within-case comparisons across increasing information completeness. The JSON-style prompts contained the same clinical information as the corresponding narrative-style prompts at each information level; JSON was used only as an alternative structured input format rather than as an additional source of clinical content. The full narrative and JSON prompt templates for Information Levels I–III are provided in the Supplementary Appendix.

Second, the constructed prompts were integrated with two advanced large language models (GPT-5 and Gemini-2.5-pro) to generate model-derived MDT recommendations. To systematically evaluate the influence of information completeness and prompt format on LLM-generated recommendations, each case was converted into six standardized input conditions. For each patient, six input conditions were generated: Information Levels I, II, and III in narrative format, and the same three information levels in JSON format. Each input was submitted independently to both LLMs. The models were instructed to act as specialists and to return one single-sentence, guideline-consistent treatment recommendation [19, 20]. The one-sentence constraint was chosen to standardize response length across models and input conditions, reduce ambiguity during mapping of free-text recommendations to predefined treatment categories, and make comparisons across the six standardized input conditions reproducible. All model queries were performed through the API between October and November 2025 with temperature set to 0. The OpenAI model was GPT-5 using the API model identifier gpt-5-2025-08-07. The Google model was Gemini-2.5-Pro using the Gemini API stable model identifier Gemini-2.5-Pro. After each response, no further dialogue was conducted; the conversation history was cleared, and a new session was initiated for the next case to prevent any potential influence from previous interactions.

Finally, a postprocessing and evaluation module was applied. All LLM-generated outputs and ground-truth MDT recommendations were normalized via automated keyword extraction and semantic matching algorithms, which mapped free-text recommendations into ten predefined therapeutic categories (Surgery; Neoadjuvant Therapy; Stereotactic Body Radiation Therapy (SBRT); Thoracic Radiotherapy; Chemotherapy; Immunotherapy; Targeted Therapy; Palliative Care; Brain Metastasis Radiotherapy; Prophylactic Cranial Irradiation(PCI)). Normalization proceeded in three steps. We applied rule-based keyword extraction with a curated clinical lexicon (drug classes, procedure names, common abbreviations, and guideline terms) expanded by synonym lists and regular expressions (for example, “concurrent chemoradiation,” “cCRT,” and “thoracic RT” were mapped to thoracic radiotherapy; “TKI,” “osimertinib,” and “EGFR-targeted therapy” to targeted therapy). Second, we performed sentence-level semantic matching using embedding-based similarity to capture paraphrases and implicit intents not covered by exact keywords (for example, “post-chemotherapy consolidation in the chest” mapped to thoracic radiotherapy; “best supportive care” to palliative care). Third, we applied decision rules for scope and polarity: negations, contraindications, or explicitly non-recommended options (for example, “immunotherapy not indicated”) were excluded from positive labels [21, 22]. Manual review by clinical experts was performed to ensure accuracy and resolve ambiguous classifications. The resulting structured label enabled quantitative comparison of model performance across different prompting strategies.

Calculating metrics

Outputs were compared with the expert multidisciplinary team (MDT) plan as the clinical reference standard for concordance analysis using predefined plan-level categories. To enable consistent, fine-grained scoring, we further normalized both the MDT recommendations and the large language model (LLM) outputs into a fixed, multi-label schema comprising ten treatment categories: surgery, neoadjuvant therapy, stereotactic body radiotherapy (SBRT), thoracic radiotherapy, chemotherapy, immunotherapy, targeted therapy, palliative care, brain radiotherapy, and prophylactic cranial irradiation (PCI).

The resulting ten-label vectors for the MDT and for each LLM output were used for all downstream metrics. The evaluation indicators computed included accuracy (Acc), precision (Prec), recall (Rec), and F1-score (F1). Accuracy quantifies the overall proportion of correctly classified labels, precision quantifies the proportion of LLM-recommended labels that were also present in the MDT reference, and recall quantifies the proportion of MDT-recommended labels recovered by the LLM. False-positive and false-negative labels were defined relative to the MDT reference categories and do not necessarily indicate definitive clinical errors. The F1-score is the harmonic mean of precision and recall. The values of F1-score closer to 1 indicate a better balance between avoiding non-indicated recommendations and avoiding omission of MDT-recommended treatments, which is particularly relevant for multi-label treatment recommendation tasks. The associated functions are shown as follows:

1

2

3

where TP, FP, and FN denote the number of true positives, false positives, and false negatives, respectively.

4

Statistical analysis

Each case was evaluated by both LLMs under matched input conditions, model comparisons were treated as paired analyses at the patient level. The primary comparative analysis focused on narrative-style Information Level III, which was the highest-completeness natural-language condition and the best-performing Gemini-2.5-Pro setting. For this primary comparison, 95% confidence intervals (CIs) for micro-Acc, micro-Prec, micro-Rec, and micro-F1 were estimated using patient-level nonparametric bootstrap resampling with 10,000 replicates. Paired bootstrap resampling was also used to estimate the CI for the difference between Gemini-2.5-Pro and GPT-5 and to calculate two-sided bootstrap p values.

Results

The cohort encompassed individuals with metastatic disease, recurrent tumors, newly diagnosed cases, and complex follow-up scenarios. Patient ages ranged from 32 to 83 years, with a mean age of 63.6 years. Tumor characteristics and relevant clinical data are summarized in Table 1.

Table 1.

Baseline characteristics stratified by TNM stage

Variable	Stage					All	p value
	I	II	III	IV	Unknown
Total cases	10	12	38	85	5	150
Sex							0.76
Male	8	10	28	58	4	108
Female	2	2	10	27	1	42
Age							0.027
Mean	69.6	61.8	66.7	61.5	66.8	63.6
Range	[57, 77]	[38, 74]	[45, 79]	[32, 83]	[61, 76]	[32, 83]
Histological type							0.009
Small cell lung cancer	1	1	13	23	1	39
Non-small cell lung cancer	6	10	25	59	3	103
Other/unspecified	3	1	0	3	1	8
Metastasis status							< 0.001
Yes	0	0	0	85	0	85
No	10	12	38	0	4	64
Unknown	0	0	0	0	1	1

Overall comparison

As shown in the Table 2, Gemini-2.5-Pro achieved consistently higher precision and F1-score than GPT-5 across all information-level variants and information formats. In contrast, GPT-5 demonstrated consistently higher recall. Under the narrative-style prompt, the best results were obtained by Gemini-2.5-Pro with Information Level III (micro-F1 0.7603; micro-Acc 0.8647), outperforming GPT-5 on the same prompt (micro-F1 0.6826; micro-Acc 0.7967). A similar advantage for Gemini-2.5-Pro was also observed by Information Level I and Information Level II. The Table 3 showed the primary paired bootstrap analysis of narrative-style Information Level III. Gemini-2.5-Pro achieved micro-Acc 0.8647 (95% CI, 0.8467–0.8827), micro-Prec 0.7031 (95% CI, 0.6562–0.7505), micro-Rec 0.8278 (95% CI, 0.7862–0.8668), and micro-F1 0.7603 (95% CI, 0.7238–0.7953). GPT-5 achieved micro-Acc 0.7967 (95% CI, 0.7753–0.8180), micro-Prec 0.5588 (95% CI, 0.5149–0.6044), micro-Rec 0.8770 (95% CI, 0.8431–0.9093), and micro-F1 0.6826 (95% CI, 0.6454–0.7194). The paired bootstrap difference (Gemini-2.5-Pro minus GPT-5) favored Gemini-2.5-Pro for micro-Acc (difference 0.0680; 95% CI, 0.0467–0.0893; P < 0.0001), micro-Prec (difference 0.1443; 95% CI, 0.1064–0.1826; P < 0.0001), and micro-F1 (difference 0.0777; 95% CI, 0.0417–0.1129; P < 0.0001), whereas GPT-5 had higher micro-Rec (difference −0.0492; 95% CI, −0.0903 to −0.0103; p = 0.0150).

Table 2.

The indicator statistics of LLMs’ answer under various information levels and input formats. Acc, accuracy; Prec, precision; Rec, recall; F1, F1-score. The color scale indicates the relative intensity of heatmap-style cell shading and does not represent an additional metric

Table 3.

Primary paired bootstrap analysis of narrative-style Information Level III performance

Metric	Gemini-2.5-Pro	GPT-5	Differencea	p value
Micro-Acc	0.8647 (0.8467–0.8827)	0.7967 (0.7753–0.8180)	0.0680 (0.0467–0.0893)	< 0.0001
Micro-Prec	0.7031 (0.6562–0.7505)	0.5588 (0.5149–0.6044)	0.1443 (0.1064–0.1826)	< 0.0001
Micro-Rec	0.8278 (0.7862–0.8668)	0.8770 (0.8431–0.9093)	−0.0492 (−0.0903 to −0.0103)	0.0150
Micro-F1	0.7603 (0.7238–0.7953)	0.6826 (0.6454–0.7194)	0.0777 (0.0417–0.1129)	< 0.0001

Values are shown as estimates with 95% confidence intervals in parentheses. ^a Difference was calculated as Gemini-2.5-Pro minus GPT-5. CI, confidence interval.

Increasing clinical granularity from Information Level I to Information Level III generally improved performance in the natural-language setting for both models. For Gemini-2.5-Pro, micro-F1 rose from 0.7365 (Information Level I) to 0.7603 (Information Level III), alongside higher precision (micro-Prec 0.6612 to 0.7031). GPT-5 exhibited a smaller but consistent gain in micro-F1 (0.6821 to 0.6826) and persistently higher recall (micro-Rec 0.8760 to 0.8770). Notably, Gemini-2.5-Pro showed a slight dip on Information Level II versus Information Level I in macro-F1 (0.6579 vs. 0.7081), suggesting that partial but not fully detailed context can introduce ambiguity for precision-oriented behavior.

The result also reveals that format effects diverged by models. For Gemini-2.5-Pro, natural-language inputs yielded the strongest results. Information Level III sentences (micro-F1 0.7603) exceeded Information Level III JSON (0.7244). In contrast, GPT-5 benefited from structured JSON at higher granularity. For Information Level III, JSON format improved micro-F1 from 0.6826 to 0.6961, driven by a very high recall score (micro-Rec 0.9132) while maintaining modest precision (micro-Prec 0.5624). Across all JSON information levels, Gemini-2.5-Pro still led in precision (micro-Prec 0.6993 vs. 0.5624) and delivered higher micro-F1 (0.7244 vs. 0.6961), whereas GPT-5 led recall (0.9132 vs. 0.7513).

Differences between macro- and micro-metrics were informative. Both models showed lower macro-F1 than micro-F1 across most conditions, indicating that treatment options with lower occurrence frequencies were also more difficult to predict. Gemini-2.5-Pro’s macro-F1 advantage over GPT-5 in narrative-style prompt suggests better balancing across frequent and infrequent treatment categories.

Across information levels and input formats, Gemini-2.5-Pro generally demonstrated a more balanced overall performance profile, with higher precision and F1-scores and treatment recommendations that were more closely aligned with expert MDT opinions. This suggests that Gemini-2.5-Pro was less likely to generate additional treatment categories not adopted by the MDT, which may be advantageous when avoiding over-generation is clinically important. In contrast, GPT-5 showed a tendency toward higher recall, capturing a broader set of possible treatment options but with a greater likelihood of including additional non-adopted recommendations. These metrics should therefore be interpreted jointly rather than in isolation. Low precision indicates that a model generated more treatment categories that were absent from the MDT reference plan, reflecting over-generation or false-positive recommendations. In contrast, high recall indicates that fewer MDT-recommended treatment categories were omitted. Therefore, a model with low precision but high recall may provide broad coverage of possible treatments but at the cost of additional non-adopted suggestions requiring clinician filtering. F1-score provides a more balanced measure by jointly reflecting both omission and over-generation. Meanwhile, accuracy should be interpreted cautiously in sparse multi-label settings because it can be strongly influenced by the large number of correctly predicted absent categories.

Aggregate performance across clinical stages

Figure 2 shows the comparison of Information Level III performance between Gemini-2.5-Pro and GPT-5 across TNM stage categories. Across stages, Gemini-2.5-Pro achieved higher average accuracy and precision, whereas GPT-5 achieved higher average recall. These results indicate that Gemini-2.5-Pro exhibits a precision-leaning performance profile. GPT-5, in contrast, shows a sensitivity-leaning pattern. Overall, Gemini-2.5-Pro maintains a modest advantage in the F1 score.

Fig. 2.

Stage-stratified comparison of narrative-style information Level III performance between Gemini-2.5-Pro and GPT-5

In Stage I, Gemini-2.5-Pro showed higher accuracy (0.860 vs 0.830) and precision (0.700 vs 0.568), while GPT-5 achieved substantially higher recall (0.955 vs 0.636), resulting in a higher F1 for GPT-5 (0.712 vs 0.667). In Stage II, Gemini-2.5-Pro outperformed on accuracy (0.885 vs 0.800), precision (0.757 vs 0.569), and F1 (0.789 vs 0.690), whereas recall remained slightly higher for GPT-5 (0.879 vs 0.824). In Stage III, accuracy was identical for the two models (0.887 vs 0.887), with Gemini-2.5-Pro showing a precision edge (0.789 vs 0.774) and GPT-5 maintaining higher recall (0.898 vs 0.863); F1 values were close, marginally favoring GPT-5 (0.831 vs 0.824). In Stage IV, Gemini-2.5-Pro delivered clear gains in accuracy (0.860 vs 0.755), precision (0.672 vs 0.478), and F1 (0.745 vs 0.614), while GPT-5 again retained a recall advantage (0.859 vs 0.835). For the ×stage group, both models had equal accuracy (0.740 vs 0.740); GPT-5 showed higher recall (0.800 vs 0.700) and F1 (0.552 vs 0.519), whereas precision remained low for both (0.421 for GPT-5; 0.412 for Gemini-2.5-Pro).

The plots derived from the four subgroups show that Gemini-2.5-Pro’s higher precision and accuracy translate into superior F1 in most stage groups, particularly Stage II and Stage IV. While GPT-5’s higher recall can lead to competitive or better F1 when sensitivity is paramount, as in Stage I and Stage III. These different operating tendencies suggest that model outputs should be interpreted according to clinical priority: Gemini-2.5-Pro may be preferable when minimizing non-indicated treatments is critical, whereas GPT-5 may be more useful when the chief concern is omission of potentially relevant options.

The relatively high proportion of Stage IV cases reflected the real-world MDT case mix during consecutive patient collection, as some patients were referred after prior treatment at external hospitals with subsequent disease progression, rather than intentional stage enrichment. Nevertheless, the small number of Stage I–II cases limited the precision and universality of stage-specific performance estimates. Because geographic location and socioeconomic information were not included in the de-identified prompts, the models could not account for these contextual factors when generating recommendations.

Aggregate performance across local and metastatic stages

To examine the impact of intrinsic differences between non-metastatic and metastatic lung cancer, we performed a subgroup analysis according to metastasis status. After excluding one patient with unclear metastasis status, this subgroup analysis included 149 patients, comprising 64 non-metastatic and 85 metastatic cases. Overall micro-level performance was calculated by pooling TP, FP, TN, and FN across the ten treatment categories within each metastasis subgroup.

As shown in Table 4 and Fig. 3, both models performed better in non-metastatic cases than in metastatic cases. For Gemini-2.5-Pro, F1 decreased from 0.7914 in non-metastatic cases to 0.7366 in metastatic cases, with a corresponding reduction in precision from 0.7551 to 0.6667. For GPT-5, the decline was more pronounced, with F1 decreasing from 0.7711 to 0.6137 and precision decreasing from 0.6751 to 0.4784. GPT-5 maintained high recall in both subgroups, whereas Gemini-2.5-Pro retained a more balanced precision–recall profile. These findings support the clinical distinction between non-metastatic and metastatic workflows: metastatic cases, which often require integration of molecular status, prior treatment exposure, systemic therapy sequencing, symptom burden, and local-control needs, appeared more challenging for LLM-generated treatment recommendations.

Table 4.

Metastasis-stratified overall performance under narrative-style Information Level III

Model	Metastasis status	Patients	Accuracy	Precision	Recall	F1
Gemini-2.5-Pro	No metastasis	64	0.8781	0.7551	0.8315	0.7914
Gemini-2.5-Pro	Metastasis	85	0.8553	0.6667	0.8230	0.7366
GPT-5	No metastasis	64	0.8516	0.6751	0.8989	0.7711
GPT-5	Metastasis	85	0.7541	0.4784	0.8557	0.6137

Fig. 3.

Metastatic-stratified comparison of narrative-style information Level III performance between Gemini-2.5-Pro and GPT-5

Aggregate performance across 10 treatment categories

Figure 4 presents the output distribution of Gemini-2.5-Pro across the three narrative-style information levels. Among the ten treatment labels, Information Level III achieved the highest accuracy, precision, and F1-score, whereas recall was highest with Information Level I (0.808 vs. 0.795). This pattern suggests that adding granular clinical details (Information Level I to Information Level III) helps models make more specific, better-calibrated recommendations (higher precision and F1), with a small trade-off in sensitivity relative to the most concise information level.

Fig. 4.

The indicators of 10 statistical subcategories under three narrative-style information levels of Gemini-2.5-pro

At the label level, several consistent trends emerged. For PCI, both discrimination and sensitivity improved with granularity: F1 increased from 0.722 (Information Level I) to 0.850 (Information Level III) and recall from 0.765 to 0.944, indicating that structured clinical context particularly aids prophylactic cranial irradiation decisions. Palliative care also benefited (F1 0.667 to 0.753; precision 0.560 to 0.707), suggesting clearer model recognition of supportive-care indications when more context is provided. Thoracic radiotherapy showed steady precision gains (0.836 to 0.893) with modest recall stability, yielding a net F1 improvement (0.797 to 0.820). In contrast, neoadjuvant therapy exhibited a recall drop with added details (0.833 to 0.583) and a slight F1 decline (0.833 to 0.667), implying that additional fields sometimes introduce conservative thresholds for neoadjuvant selection. SBRT remained the most challenging category by F1 (mean 0.452 across information levels), driven by low precision and recall despite high accuracy.

Across categories, chemotherapy and thoracic radiotherapy achieved the highest mean F1 (0.818 and 0.791, respectively), reflecting robust, balanced identification of these common treatments. Surgery, TKI, and PCI clustered at intermediate F1 levels (0.737, 0.736, 0.740), while brain radiotherapy and SBRT lagged (0.591 and 0.452). Notably, SBRT showed high average accuracy (0.920) but poor F1, underscoring that micro-averaged metrics or accuracy can mask systematic under- or over-prediction on rare treatments. By contrast, brain radiotherapy achieved the highest mean recall (0.944) with only moderate precision.

In addition, the narrative-style Information Level III generally improved metrics (accuracy, precision, F1) for PCI, palliative care, and thoracic radiotherapy. These findings align with a clinically intuitive trade-off: richer context enables more selective, guideline-concordant recommendations. Overall, Information Level III provided the best aggregate balance in this study, but category-specific tuning remains important for neoadjuvant therapy and SBRT, where targeted prompt refinements may be required to mitigate under- or over-selection.

Discussions

This study indicates that contemporary advanced LLMs have potential to support structured pre-MDT treatment recommendation generation in lung cancer within a controlled evaluation framework based on real-world MDT cases. When provided with standardized, de-identified clinical summaries, both models generated treatment categories broadly consistent with expert MDT plans, supporting the feasibility of LLM-assisted structured recommendation generation for clinician review. Information Level III consistently yielded the best performance, suggesting that LLM concordance with MDT recommendations depends not only on model reasoning capability, but also on the completeness, clinical relevance, and organization of the input information. More complete clinical context likely reduces ambiguity in staging, molecular interpretation, treatment history, and eligibility for local or systemic therapies, thereby enabling closer alignment with MDT decision patterns. Across models, Gemini-2.5-Pro showed a more precision-oriented and overall balanced performance profile, whereas GPT-5 tended toward higher recall with a greater likelihood of including additional treatment categories. These differences reflect distinct output tendencies rather than a strict hierarchy of performance, and they highlight the importance of evaluating both omission and over-generation in clinical recommendation tasks.

Recent oncology-related LLM studies have shown that model performance is highly dependent on clinical task definition, input structure, disease context, and evaluation endpoint [23]. Existing studies have evaluated LLMs in diverse cancer-related settings, including personalized oncology decision support, cancer treatment information, multimodal prognostication, and comparison with human board recommendations [9, 12, 24, 25]. For example, Yap et al. externally validated pre-trained LLMs for multimodal prognostication in laryngeal and hypopharyngeal cancer, highlighting both the potential of LLM-based clinical reasoning and the need for interpretable, context-aware evaluation frameworks [25]. In lung cancer, prior work has examined the agreement between artificial intelligence-based treatment decisions and multidisciplinary tumor board recommendations [13], while MDT-focused studies have emphasized the importance of structured clinical data, multidisciplinary consensus, and treatment equality in lung cancer care [3, 5, 26–28]. Our study extends this literature by directly comparing two contemporary advanced LLMs with expert MDT recommendations in real-world lung cancer cases and by systematically evaluating how model type, information completeness, and input format influence recommendation concordance. A key strength of this study is its clinically anchored, paired comparison design, in which each real-world MDT case was evaluated under standardized input conditions. This approach allowed direct within-case assessment of model behavior while enabling evaluation of both omission and over-generation through precision, recall, and F1 score.

The observed performance differences across clinical stages further suggest that model concordance is strongly influenced by decision-space complexity rather than disease stage alone. In non-metastatic lung cancer, treatment decisions are relatively structured and guided by well-established curative-intent pathways defined in major guidelines [19, 22, 26, 27, 29], including surgical resectability, perioperative systemic therapy, and radiotherapy sequencing. This constrained decision space likely contributes to higher concordance between LLM outputs and MDT recommendations. In contrast, metastatic or recurrent disease involves substantially higher combinatorial complexity, requiring integration of prior treatment exposure, molecular profiles, disease dynamics, and symptom burden, which expands the space of plausible but clinically non-equivalent options. Under these conditions, precision degradation and increased variability in recommended categories become more evident, particularly for models with higher recall-oriented behavior.

To illustrate these differences, two representative cases highlight model behavior in clinically distinct contexts. In a Stage IV EGFR-mutant patient with bone metastasis and good performance status, LLM recommendations were largely consistent with MDT decisions, correctly prioritizing continued EGFR-TKI therapy combined with local radiotherapy and bone-modifying agents. This reflects strong alignment in low-entropy, guideline-deterministic scenarios. In contrast, in a heavily pretreated metastatic patient with multiple metastatic sites, prior immunotherapy-related toxicity, and ongoing targeted therapy, LLMs tended to suggest broader management strategies including treatment switching, SBRT for oligoprogression, or clinical trial enrollment. While individually guideline-consistent, these recommendations were not fully aligned with MDT judgment, reflecting limitations in trajectory-dependent reasoning and context integration. The representative cases discussed above are intended to illustrate model behaviors rather than to capture the full spectrum of complex MDT scenarios.

Beyond performance metrics, LLMs demonstrate a clinically relevant capability in structuring heterogeneous patient information into standardized, guideline-oriented formats. This function is particularly valuable in MDT workflows, where clinicians must synthesize multimodal clinical data into concise case presentations under time constraints. In this setting, LLMs may serve as a pre-MDT cognitive support layer that organizes information, generates preliminary treatment option sets, and highlights guideline-consistent pathways for expert review. Importantly, their role should remain assistive rather than autonomous, functioning as a structured reference to support, rather than replace, multidisciplinary decision-making [30]. In resource-limited settings, LLMs may have potential to support standardized case preparation and guideline-oriented preliminary review, but such use would require adaptation to available resources. In high-volume tertiary centers, where large patient volumes and intensive clinical workloads limit the timely organization of MDT discussions, the primary value of LLMs lies not in replacing specialists, but in enhancing pre-MDT preparation. By streamlining clinical information synthesis, standardizing preliminary treatment considerations, and providing a reproducible starting framework for discussion, LLMs may improve the efficiency and consistency of expert review while preserving the central role of MDT deliberation.

Several limitations and safety considerations should be acknowledged. First, the use of curated clinical summaries rather than raw longitudinal records limits evaluation of end-to-end information extraction, uncertainty handling, and contradiction resolution. Second, MDT consensus was used as the reference standard, which may not fully capture inter-expert variability or alternative clinically valid opinions [28]. Third, deployment in real-world settings raises important issues related to privacy, auditability, data governance, and institutional variability in treatment resources [31, 32]. Future systems should incorporate structured input pipelines, uncertainty-aware decision escalation, version-controlled models, and retrieval-augmented integration of institution-specific guidelines to ensure clinically safe and applicable outputs.

Conclusion

In this study, we evaluated the performance of Gemini-2.5-Pro and GPT-5 as a decision-support tool for the management of lung cancer patients within a MDT framework. Our findings demonstrate that Gemini-2.5-Pro with Information Level III can reliably provide guideline-concordant recommendations for patients with common presentations. The model’s ability to synthesize large volumes of clinical information and suggest standardized treatment options highlights its potential as a supportive resource for clinicians. However, for patients with complex clinical courses, Gemini-2.5-Pro’s recommendations were often less nuanced and failed to capture the individualized considerations integral to expert MDT decision-making. These limitations underscore the necessity of human oversight in the management of challenging cases. Future improvements should aim to help the model better handle detailed and evolving clinical situations, making it more useful in complex, real-world cases. Make the LLM generate more reliable treatment recommendations and help resource-limited regions benefit from technological advancements.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Abbreviations

LLM

Large Language Model

MDT

Multidisciplinary Decision Making

QA

Question Answer

SBRT

Stereotactic Body Radiation Therapy

PCI

Prophylactic Cranial Irradiation

RT

Radiotherapy

cCRT

Concurrent Chemoradiation

TKI

Tyrosine Kinase Inhibitor

JSON

JavaScript Object Notation

CIs

Confidence intervals

Author contributions

Z.Z. and J.L. designed the study and developed the methodology. M.W. collected and preprocessed the imaging data. M.W. and Z.W. performed the experiments and statistical analysis. J.L., Z.W., and H.Lv. supervised the project and provided critical revisions to the manuscript. Z.Z. drafted the main manuscript text, M.W., H.Lv. and J.L. prepared the figures. All authors reviewed and approved the final version of the manuscript.

Funding

This work was supported by the Beijing Natural Science Foundation [Grant Nos. 7254539, L242024 and 7242267]; the National Natural Science Foundation of China [Grant No. 62522119]; the Beijing High-Level Innovation and Entrepreneurship Talent Support Program [Grant No. G202522110]; the Open Projects of Sichuan Province Clinical Medical Research Center for Imaging Medicine [Grant No. YXYX2409]; Beijing Friendship Hospital, Capital Medical University [Grant No. YYZZ202334]; and Capital Medical University [Grant No. B2408].

Data availability

The data can be shared up on request.

Declarations

Ethics approval and consent to participate

The data from Shandong Cancer Hospital was approved by the Institutional Review Board (SDTHEC2024007020) in accordance with the Declaration of Helsinki, and the requirement for informed consent was waived due to the retrospective anonymized study. The anonymized data posed no risk to patient privacy. The personal identifiers were removed, and the study did not submit identifiable patient information to commercial LLM platforms.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.