Predicting efficacy in patients with locally advanced/metastatic urothelial carcinoma (mUC) treated with immunotherapy using explainable machine learning approaches: the SamUR-AI trial on behalf of the Meet-URO group

Abstract

Background

Immune checkpoint inhibitors (ICIs) have reshaped the treatment landscape for metastatic urothelial carcinoma (mUC), yet reliable predictive biomarkers remain limited. The SamUR-AI study was designed to evaluate whether machine learning (ML) and explainable artificial intelligence (XAI) approaches could improve prediction of clinical outcomes in patients with mUC treated with ICIs.

Materials and methods

We conducted a multicenter retrospective analysis including 438 patients treated with ICIs across 34 Italian institutions from the Meet-URO network. Baseline clinical and laboratory features were analyzed using ML and XAI methodologies to predict objective response rate (ORR), progression-free survival (PFS), and overall survival (OS).

Results

Classification models showed suboptimal performance in predicting ORR (best test F1-score: 0.61), likely due to class imbalance and overfitting. In contrast, survival models achieved moderate predictive accuracy, with the extra survival trees model yielding a concordance index (C-index) of 0.67 for OS. SHapley Additive exPlanations-based explainability identified key prognostic factors, including Eastern Cooperative Oncology Group performance status, line of immunotherapy and treatment combinations, liver and lung metastases, neutrophil count, and hemoglobin level.

Conclusions

Although further validation is needed, our findings highlight the potential of XAI-enhanced ML to identify clinically relevant features and to support personalized treatment strategies in patients with mUC.

Highlights

• •ML was applied to predict IO response in mUC.
• •Survival analysis models showed acceptable performance, with a C-index of 0.67 on test data.
• •Poor ECOG PS and liver or lung metastases are the main negative prognostic factors.
• •Combination IO as early therapy improved patient outcomes.

Introduction

Bladder cancer represents the 10th most common cancer type globally, with ∼573 000 new diagnoses in 2020¹; in Italy, urothelial carcinoma (UC) accounts for >29 200 new cases each year, with 5840 muscle invasive bladder cancer and 1460 new diagnoses of metastatic UC (mUC).² Non-muscle invasive bladder cancer patients typically exhibit a favorable prognosis, showing a 5-year overall survival (OS) rate nearing 90%; unfortunately, this percentage drops to 6% in cases of metastatic disease.³ Historically, platinum-based chemotherapy has been the most effective treatment for mUC, representing the standard of care in first-line therapy.⁴ However, the emergence of immune checkpoint inhibitors (ICIs) has reshaped the treatment paradigm for UC; moreover, together with antibody–drug conjugates, immunotherapy (IO) now stands as one of the most prominent drug classes under investigation in ongoing trials.⁵

Only 10%-40% of patients with advanced solid tumors respond to ICI treatment,⁶ underscoring the critical need to identify biomarkers that can predict response to IO.⁷ Programmed death-ligand 1 and tumor mutational burden stand out as the most extensively investigated biomarkers for IO,^8,9 but the clinical usefulness of these biomarkers in advanced UC is an area of uncertainty. Different trials conducted in the last few years have shown controversial results,^10,11 highlighting the importance of delving deeper into this topic. Tertiary lymphoid structure, tumor microenvironment, circulating tumor DNA, T effector signatures, and transforming growth factor-β expression seem to be potentially predictive factors among different cancer histologies, and their use is currently under investigation in mUC.^12,13

Artificial intelligence (AI) showed¹⁴ promise in facing these limitations, particularly leveraging machine learning (ML) and deep learning (DL) algorithms.^15,16 The main reason for the growing importance of ML and DL is their ability to analyze complex, non-linear data, facilitating integration among different data types. The use of these methodologies in clinical research is currently a rapidly expanding field. Evidence indicates that AI technologies can accelerate the diagnosis process,^17,18 enhance accuracy in predicting treatment outcomes,^19,20 and streamline health care by reducing time and costs.²¹

These achievements come at the expense of interpretability, since most of the ML models are characterized by complex mathematical functions and representations in space, which make it difficult to be aware of the logical flow behind the prediction. Especially in the medical field, it is of utmost importance to understand how the result has been achieved. Thus, explainable AI (XAI) methods have been developed to enhance model transparency.²²

Our research focuses on using state-of-the-art ML and XAI methods to identify predictive biomarkers of response or resistance to IO in UC, thereby informing personalized treatment strategies.

Materials and methods

The methodological workflow is outlined in Figure 1.

Figure 1.

Methodological workflow. AUC, area under the curve; SHAP, SHapley Additive exPlanations.

Dataset collection

SamUR-AI is a retrospective, observational, multicenter study involving patients with UC from the Meet-URO 1 databases, who were treated with IO between January 2015 and June 2022 at 34 Italian institutions belonging to the Meet-URO network. A total of 438 patients were enrolled. The main eligibility criteria included diagnosis of metastatic or locally advanced UC of the bladder or the upper urinary tract in patients who received an IO-based treatment for advanced disease (alone or in combination with other drugs). The study was approved by the regional ethical committees.

Clinical data were retrospectively collected at each participating center by trained oncologists or data managers from electronic medical records and institutional databases. Data were entered into a centralized electronic case report form specifically designed for the study. To ensure harmonization across centers, outcome definitions and variable coding were shared before data extraction. Objective response was assessed according to RECIST version 1.1 whenever radiological evaluations were available in the clinical records, reflecting routine clinical practice in a real-world setting.

The study is reported in accordance with the European Society for Medical Oncology (ESMO) Guidelines for Reporting Oncology Real-World Evidence (GROW); the completed checklist is provided in Supplementary Material, available at https://doi.org/10.1016/j.esmorw.2025.100681.

Data curation

Data collected in the database included demographic, histologic, clinical, and laboratory variables, in addition to treatment details and survival information. According to clinical expertise, an initial selection of features was carried out, leading to 31 features collected at baseline of IO treatment. In addition, the inflammatory index neutrophil-to-lymphocyte ratio was derived at baseline, for a total of 32 features, listed in Table 1. Basic data quality checks were carried out to assess internal consistency, temporal plausibility, and completeness. In case of inconsistencies or missing critical information, source medical records were re-reviewed locally to resolve discrepancies.

Table 1.

List of features initially considered for the analysis with corresponding percentage of missing values

Feature name	Missing data (%)	Feature name	Missing data (%)
Demographic		IO type	0
Age	1.42	ECOG PS	1.14
Sex	0.29	BMI	21.4
Clinical		Steroids	0.57
Site	0.86	Antibiotics	0.57
Previous neoadjuvant CHT	0
Radical surgery	0.29	Laboratory
Previous adjuvant CHT	0	WBC	5.43
Node metastases	0.57	Neutrophils	4.57
Lung metastases	1.14	Lymphocytes	5.42
Bone metastases	0.57	NLR	6.00
Liver metastases	0.57	Eosinophils	6.86
Soft tissue metastases	0.29	Hemoglobin	4.00
Pancreas metastases	0.29	Platelets	5.14
Kidney metastases	0.29	LDH	37.14
Adrenal metastases	0.29	Albumin	57.71
Brain metastases	0.57	Calcium	21.71
Line of IO	0

BMI, body mass index; CHT, chemotherapy; ECOG PS, Eastern Cooperative Oncology Group performance status; IO, immunotherapy; LDH, lactate dehydrogenase; NLR, neutrophil-to-lymphocytes ratio; WBC, white blood cells.

Outcomes

Regarding classification task, objective response rate (ORR) was analyzed as the binary outcome, using RECIST,²³ labeling patients with progressive disease (PD) or stable disease as class 0, and those with partial or complete response as class 1.

For the survival analysis task, we tested OS, defined as the time expressed in months between the start of IO and patient death or last follow-up, and progression-free survival (PFS), defined as the time expressed in months between the start of IO and date of PD or death/last follow-up.

In case of discordance between radiological evidence and clinical assessment (e.g. radiological progression with ongoing treatment due to perceived clinical benefit), the treating physician’s final documented clinical judgment was used to define progression, reflecting real-world practice. This approach was consistently applied across centers for the retrospective definition of endpoints.

Model development

Preprocessing

All analyses were conducted in Python v3.11. The data preparation was common to all outcomes to have the same adjustments for subsequent analyses. The original dataset was divided into training and test sets with an 80%–20% ratio, employing a stratification with respect to the binary indicator of OS, keeping the same percentage of censored patients in the training and test sets. The percentage of missing values was calculated for each feature (Table 1), and those exceeding 25% were eliminated. The remaining missing values were imputed using IterativeImputer, with Bayesian Ridge linear regressor as the estimator.²⁴ After imputation, categorical and numerical features were separately treated. Barplots for categorical features were plotted, and VarianceThreshold by scikit-learn²⁵ was employed to exclude from analysis the categorical features whose variance was below the 10% threshold. For numerical features, logarithmic transformation was applied to highly skewed distributions. Next, standardization was used to scale continuous features through mean removal and scaling to unit variance. Subsequently, the correlation among features was investigated using Pearson’s coefficient and a threshold of 80% was established, with feature removal guided by clinicians’ opinion. The dataset thereby obtained was employed in the following analyses after excluding patients with missing data for each outcome.

Feature selection

For the classification task, feature selection was carried out using minimum redundancy maximum relevance (MRMR),²⁶ a filter method, and three wrapper methods: forward feature selection, backward feature selection, and recursive feature elimination.²⁷ For survival analysis, a univariate feature selector, a model-specific technique, was employed. Each feature was evaluated independently by training a model and computing the chosen performance metric, concordance index (C-index). Features associated with a C-index above a chosen threshold of 0.55 were kept, while the others were discarded, with the aim of maintaining features with individual predictive capability for the specific model.

Model training and evaluation

In the classification task, eight models were trained: XGBoost (XGB),²⁸ logistic regression (LR),²⁹ random forest (RF),³⁰ multilayer perceptron (MLP),³¹ support vector machines (SVM),³² AdaBoost (AB),³³ Extra Trees (ET),³⁴ LightGBM (LGBM).³⁵ XGB and LGBM required external packages for implementation,^28,35 whereas all the others were contained in the scikit-learn package.²⁵ For the optimization of hyperparameters, GridSearch with 10-fold cross-validation was used, stratified by outcome, and whose target was to maximize the weighted F1-score. During the training phase, class weighting was used to compensate for over-representation of the majority class in the dataset. The main metric chosen to evaluate the models’ performance was the F1-score; in addition, accuracy, recall, precision, and area under the curve (AUC) were computed.

Six models were evaluated for survival analysis: survival trees (ST),³⁶ gradient boosting survival (GBS),³⁷ random survival forest (RSF),³⁸ extra survival trees (EST),³⁴ survival support vector machine (SSVM),³⁹ and Cox proportional hazards (CPH)⁴⁰ as a baseline model. All of them are included in the scikit-survival Python package.⁴¹ GridSearch was employed to optimize the hyperparameters, using five-fold cross-validation, where splitting was stratified based on the censoring label, and its target was to maximize the C-index. The assessment of the models’ performance was mainly based on the C-index⁴²; mean AUC and Brier score were also computed.

Explainability

After selecting the optimal combination of feature selector and model for each ML task, SHapley Additive exPlanations (SHAP)⁴³ values were applied to gain insight into the prediction process. SHAP is a state-of-the-art explainability method that assigns each feature a ‘Shapley value’, representing an index of the feature importance for the model’s prediction. The explainer was trained on the selected model, and SHAP values were subsequently computed for the test set using the SHAP Python library.⁴⁴ As tools for global explainability, beeswarm plots were used to understand the overall behavior of the model and to identify the most important features driving its prediction capability.

Results

Dataset description

The SamUR-AI cohort includes 438 patients with mUC who had received IO for metastatic disease. Table 2 reports the list of the main baseline patient characteristics and the types of IO used. Of the 32 features initially considered, lactate dehydrogenase and albumin were excluded, according to the 25% missing value threshold. Soft tissue metastases, pancreas metastases, kidney metastases, adrenal metastases, brain metastases, steroids, and antibiotics were also removed, as they had the same value in >90% of the cohort. Logarithmic transformations were applied to neutrophils and platelets. White blood cells were discarded after feature correlation analysis due to high positive correlation with logNeutrophils (Pearson’s coefficient = 90%). In the end, 22 features remained for the analysis.

Table 2.

Baseline patient and treatment characteristics

Patient characteristics (n = 438)	n (%)
Median age, years (range)	66 (37-87)
Sex
Male	340 (77.6)
Female	98 (22.4)
Site
Bladder	319 (72.8)
Upper tract	109 (24.9)
Urethra	10 (2.3)
Prior radical surgery
Yes	300 (68.5)
No	138 (31.5)
Prior neoadjuvant chemotherapy
Yes	54 (12.3)
No	384 (87.7)
Prior adjuvant chemotherapy
Yes	68 (15.5)
No	370 (84.5)
Site of metastases (at IO start)
Nodes	326 (74.4)
Lung	165 (37.7)
Bone	130 (29.7)
Liver	101 (23.1)
ECOG PS (at IO start)
0	267 (60.1)
1	146 (33.3)
>1	21 (4.8)
Immunotherapy characteristics	n (%)
IO line
1	152 (34.7)
2	226 (51.6)
3	39 (8.9)
>3	21 (4.8)
Immunotherapy combinations
ICI alone	277 (63.2)
ICI + ICI	36 (8.2)
ICI + chemotherapy	61 (13.9)
ICI + other	64 (14.6)

ICI alone includes anti-PD-1/PD-L1 monotherapy (mainly pembrolizumab). ICI + chemotherapy refers predominantly to platinum–gemcitabine or taxane-based regimens. ICI + ICI includes dual checkpoint blockade. ICI + other comprises combinations with targeted agents such as VEGF-TKIs or HER2-directed therapies.

ECOG PS, Eastern Cooperative Oncology Group performance status; HER2, human epidermal growth factor receptor 2; ICI, immune checkpoint inhibitors; IO, immunotherapy; PD-1, programmed cell death protein 1; PD-L1, programmed death-ligand 1; TKI, tyrosine kinase inhibitor; VEGF, vascular endothelial growth factor.

After preprocessing and feature correlation, for each outcome separately, patients who did not have the response of interest were excluded from the analysis: for ORR, 16 patients were excluded; for PFS, 12 patients; for OS, only 1 patient.

Classification task

For the classification task, the models’ performance was far from optimal with a tendency toward overfitting, as shown by the heatmaps for each set (Figure 2), containing the value assumed by the F1-score for all the combinations of feature selector and model. The model that returned the best results was SVM with no feature selection, with F1-scores of 0.88, 0.66, and 0.61 on the training, cross-validation, and test sets, respectively, as well as an AUC of 0.56 and a balanced accuracy of 0.61 on the test set.

Figure 2.

Heatmaps of the F1-score values returned by the combinations of models and feature selectors for training, cross-validation, and test sets for the outcome ORR. AB, AdaBoost; BFS, backward feature selection; ET, Extra Trees; FFS, forward feature selection; LGBM, LightGBM; LR, logistic regression; MLP, multilayer perceptron; MRMR, minimum redundancy maximum relevance; RF, random forest; RFE, recursive feature elimination; SVM, support vector machine; XGB, XGBoost.

Survival analysis task

Compared with the classification task, survival analysis models returned better results, and in particular, models predicting the outcome OS were slightly more efficient than those built to predict PFS. Table 3 presents the results obtained with the employed ML models for OS and PFS. The model showing the best performance for the prediction of OS was EST with all the features, which achieved C-indexes of 0.76, 0.75, and 0.67 on the training, cross-validation, and test sets, respectively. On the test set, the mean AUC was 0.70 and the Brier score was 0.20. The best model for PFS was identified as SSVM fed by the subset of features maintained after feature selection [Eastern Cooperative Oncology Group (ECOG) performance status (PS), hemoglobin, line of IO, liver metastases, neutrophils, platelets], which achieved C-indexes of 0.67, 0.66, and 0.64 on the training, cross-validation, and test sets, respectively, as well as a mean AUC of 0.69 on the test set. Regarding the comparison between the models fed by all features and those to which univariate selection was applied, they yielded comparable results on average, showing that a smaller set of features can be effective as well in predicting the outcome of interest.

Table 3.

Outcomes OS and PFS

Model	Feature selection	C-index training	C-index CV	C-index test	Mean AUC test	Brier score test
OS
CPH	None	0.76	0.73	0.65	0.67	0.22
	Univariate (9)	0.76	0.74	0.66	0.72	0.21
ST	None	0.69	0.66	0.64	0.69	0.21
	Univariate (6)	0.72	0.69	0.62	0.63	0.23
GBS	None	0.82	0.70	0.64	0.67	0.20
	Univariate (10)	0.78	0.70	0.63	0.64	0.20
RSF	None	0.76	0.73	0.64	0.68	0.20
	Univariate (8)	0.76	0.74	0.65	0.71	0.20
EST	None	0.76	0.75	0.67	0.70	0.20
	Univariate (9)	0.75	0.74	0.65	0.71	0.20
SSVM	None	0.80	0.74	0.65	0.69	—
	Univariate (10)	0.78	0.75	0.65	0.71	—
PFS
CPH	None	0.70	0.67	0.61	0.64	0.20
	Univariate (7)	0.67	0.66	0.65	0.69	0.18
ST	None	0.64	0.63	0.56	0.60	0.18
	Univariate (4)	0.69	0.66	0.60	0.65	0.20
GBS	None	0.72	0.64	0.58	0.63	0.18
	Univariate (4)	0.69	0.67	0.59	0.63	0.18
RSF	None	0.70	0.68	0.61	0.66	0.17
	Univariate (4)	0.70	0.67	0.60	0.64	0.19
EST	None	0.71	0.67	0.62	0.66	0.17
	Univariate (6)	0.68	0.67	0.63	0.68	0.17
SSVM	None	0.70	0.67	0.63	0.64	—
	Univariate (6)	0.67	0.66	0.64	0.69	—

The table presents the outcomes OS and PFS, containing the performance metrics computed for each model trained both with the entire set of features (22) and with those selected by the univariate method (number contained in brackets). The best performing model, highlighted in bold, is SSVM fed by the six features selected with the univariate method.

CPH, Cox proportional hazards model; EST, extra survival trees; GBS, gradient boosted survival; RSF, random survival forest; SSVM, survival support vector machines; ST, survival tree.

Moreover, it is worth noting that ML models performed similarly to the CPH model, which is the gold standard for the analysis of medical data that pertain to the survival analysis task.⁴⁰ This is supported by the cross-validation C-index values, which were closely aligned across methods: the CPH model with feature selection yielded a C-index of 0.74, and the highest value achieved by any ML model was 0.75.

Explainability

Regarding the classification task, Figure 3A shows the beeswarm plot displaying the six features that were identified as most important by SVM for predicting the ORR of the instances contained in the test set. IO combination has been evaluated as the most important feature for the prediction of ORR. As reported in Table 2, it is a categorical variable with four levels: ICI alone (encoded as 0), ICI + chemotherapy (encoded as 1), ICI + other targeted therapies (encoded as 2), and ICI + ICI (encoded as 3). Blue dots, corresponding to patients who underwent IO monotherapy, are mainly associated with negative SHAP values, meaning that the model tends to classify them as non-responders. On the other hand, medium-high values of the feature (i.e. patients who underwent IO in combination with IO or other treatments) are associated with positive SHAP values and, hence, identified as responders.

Figure 3.

The beeswarm plot allows to identify the most important features driving the model’s prediction capability. The horizontal axis contains SHAP values; in classification, negative SHAP values are associated with non-responders, while in survival analysis, they correspond to a lower risk, i.e. better prognosis, and vice versa for positive values. The vertical axis holds the most important features, ranked starting from the most relevant at the top, with importance determined by the average of the SHAP values. Within a line, each dot corresponds to one patient. In addition, the color map represents the feature values. (A) Beeswarm plot of the SVM model predicting ORR applied to the test set. (B) Beeswarm plot of the SSVM model predicting PFS applied to the test set. (C) Beeswarm plot of the EST model predicting OS applied to the test set. ORR, objective response rate; OS, overall survival; PFS, progression-free survival; SHAP, SHapley Additive exPlanations; SVM, support vector machines.

For the survival analysis, the output of the models is a risk score. SHAP values are computed with respect to the model’s predicted risk score, such that each value reflects the contribution of a feature to the model’s prediction for a given patient. Consequently, features with high positive SHAP values contribute strongly to increasing the patient’s risk of death or progression, depending on whether the analysis focuses on OS or PFS, respectively. Figure 3C displays the beeswarm plot of the EST model regarding the outcome OS for the test set. Among all the features given as input to the model, worse ECOG PS, high line of IO, and neutrophils, as well as the presence of liver and lung metastases, are identified as the most relevant negative prognostic factors.

Figure 3B displays the same type of graph built for the SSVM model fed by the subset of features remaining after univariate feature selection regarding the outcome PFS for the test set. Similar to the outcome OS, ECOG PS, liver metastases, line of IO, and neutrophils appear to be among the five most important features driving the prediction. The fifth is hemoglobin, with high values associated with a prolonged survival time.

Discussion

In oncology, AI has demonstrated significant potential in recent years, and in UC, where response rates to IO remain suboptimal, AI could serve as a promising tool to improve treatment outcomes. Our study utilized state-of-the-art ML and XAI methodologies to predict response and survival outcomes in a large Italian cohort of patients with mUC treated with ICIs.

The classification task yielded suboptimal results: the chosen model achieved an F1-score of 0.61, which is considered weak for predictive tasks, indicating the limited reliability of these models in their current form. Several factors likely contributed to this suboptimal performance, with class imbalance being the leading one: the responder-to-non-responder ratio was 1 : 2 in the training set, and 1 : 3 in the test set. This imbalance skewed model predictions, despite the application of class balancing techniques during model training. Another significant challenge was overfitting, particularly in boosting algorithms, as can be seen in Figure 2.

The survival analysis task produced more promising results: metric values approaching 0.7 are generally considered acceptable for survival models,⁴⁵ although there is still room for improvement.

The findings following XAI analysis confirmed previously established prognostic factors,⁴⁶ such as the negative impact of poor ECOG PS and the presence of metastases, particularly liver and lung metastases. Additionally, our results indicated possible trends toward improved outcomes when IO was administered in combination with other agents or in earlier treatment lines. However, these observations should be interpreted with caution, as the classification model demonstrated limited predictive performance, warranting further investigation. Nevertheless, these findings align with emerging evidence supporting a synergistic effect of combination strategies in UC.^47,48

Despite some promising results, our study has several limitations. Firstly, as a retrospective, observational study, inherent selection biases may affect the generalizability of our findings. The dataset, being rather small, may not fully capture real-world variability, and the absence of an external validation dataset prevents an independent evaluation on new data. Moreover, while ML models can uncover complex relationships among variables, their performance remains contingent upon the quality and completeness of the dataset. The relatively modest predictive power observed in our classification task underscores the challenges in developing highly accurate models for clinical use. This highlights the importance of refining feature selection techniques, incorporating additional variables, and increasing sample size to improve model performance.

A key direction for future studies is the integration of multi-omics data, including genomic, transcriptomic, and proteomic profiles, to enhance the robustness of predictive models. Equally important is the improvement of real-world data quality through structured digital collection—such as standardized electronic health records—and the inclusion of patient-reported outcomes such as quality-of-life and toxicity assessments. Furthermore, advancements in DL methodologies, such as convolutional neural networks and transformer-based architectures, could be leveraged to capture intricate patterns in imaging and histopathological data.⁴⁹ Combining structured clinical data with unstructured data sources, such as radiology and pathology reports, may provide a more comprehensive approach to predicting treatment responses. Together, these strategies may increase data accuracy and model generalizability, paving the way for prospective validation in larger, multicentric cohorts and ultimately facilitating AI-driven precision oncology.

Conclusion

In conclusion, the results obtained were not satisfactory for classification tasks and demonstrated lower performance compared with standard methods in survival analysis. Nonetheless, the outputs aligned with clinical practice, suggesting that no implausible predictions were generated: low ECOG PS scores and combination treatments were associated with improved survival, and presence of metastases was identified as a negative prognostic factor. These findings underscore the challenges of applying ML in clinical contexts, where limited sample sizes and data quality constrain model performance, while also highlighting the potential of XAI to enhance interpretability and transparency. Further studies are needed to better understand IO efficacy in patients with mUC, ideally involving larger cohorts and the integration of multi-omics data to improve model robustness and predictive power.