Prediction model for additional procedure requirement in flexible ureterorenoscopy using explainable artificial intelligence

Abstract

A significant proportion of patients require additional intervention following flexible ureterorenoscopy (f-URS), a commonly performed procedure in stone surgery. This study aimed to develop machine learning (ML)–based prediction models for additional intervention after f-URS and to enhance their clinical applicability through explainable artificial intelligence (XAI) methods. A retrospective analysis was performed on 656 patients who underwent f-URS between 2015 and 2025. Demographic, clinical, anatomical, and operative variables were collected. Feature selection was conducted using Boruta, LASSO, and ElasticNet, and fourteen ML algorithms, along with ensemble approaches, were evaluated. Model explainability was assessed using SHAP, LIME, model-based importance, and permutation importance analyses.Additional intervention was required in 180 patients (27.4%), including repeat f-URS (n = 94), ureteroscopy (n = 61), extracorporeal shock wave lithotripsy (n = 22), and percutaneous nephrolithotomy (n = 3). The ureteropelvic junction–pelvis angle (UPJ–PA) emerged as the strongest predictor of additional intervention, with intervention rates of 84.3% below the 110° threshold compared with 2.8% above it (OR: 29.6; p < 0.001). Logistic regression demonstrated excellent discriminative performance (AUC = 0.987), while ridge classifier showed high clinical sensitivity with a low false-negative rate. Comparative explainability analyses consistently identified UPJ–PA as the dominant predictor across all methods (normalized importance score: 1.000), followed by access sheath diameter and use of flexible and navigable suction ureteral access sheaths (FANS-UAS). Robustness analysis confirmed the stability of UPJ–PA against measurement variability.ML and explainable XAI provide accurate, transparent, and clinically meaningful predictions for additional intervention after f-URS. A UPJ–PA below 110° represents a critical anatomical risk marker across all renal stone locations, while operative factors such as access sheath diameter and FANS-UAS use further influence clinical outcomes. These findings support the role of AI-based decision support systems in improving patient selection and surgical planning in routine f-URS practice.

Supplementary Information

The online version contains supplementary material available at 10.1038/s41598-026-46898-9.

Introduction

Urolithiasis is a prevalent disease with increasing global prevalence, leading to significant morbidity and healthcare expenditures¹. In recent years, due to technological advances, flexible ureterorenoscopy (f-URS) has become one of the first-line treatment modalities for kidney stones, owing to its minimal invasiveness, high success rate, and low complication profile². However, complete stone-free status cannot be achieved in all cases following f-URS, and a considerable proportion of patients require additional intervention (repeat URS, percutaneous nephrolithotomy, or extracorporeal shock wave lithotripsy). The need for additional intervention represents a significant clinical problem that both negatively impacts patient satisfaction and imposes additional costs on the healthcare system.

In the current literature, various scoring systems (Guy’s Stone Score, CROES nomogram, etc.) have been developed to predict f-URS success. However, most of these systems remain limited due to their consideration of restricted variables, performance degradation in heterogeneous patient populations, and limited applicability in clinical practice^3,4. In recent years, machine learning (ML)-based approaches have emerged as promising alternatives for surgical outcome prediction, demonstrating higher predictive power compared to classical scoring methods. Nevertheless, the generally ‘black box’ nature of ML model decision mechanisms leads to trust issues among clinicians⁵. At this juncture, explainable artificial intelligence (XAI) techniques come into play, transparently revealing which variables ML models rely upon for their predictions. This not only enhances prediction accuracy but also strengthens the adoptability of clinical decision support systems⁶.

While ML-based studies exist in the literature for predicting stone-free rates after f-URS, there are no studies directly predicting the need for additional intervention. This gap constitutes a significant research void in terms of personalizing patient management, optimizing surgical planning, and ensuring efficient utilization of healthcare resources⁷.

The originality of this study lies in several key aspects. First, a rigorous nested stratified cross-validation approach was employed to prevent overfitting and to ensure unbiased estimates of model performance. Second, a comprehensive model evaluation framework was used, including bootstrap-based confidence intervals, calibration analysis using Brier scores, and Decision Curve Analysis to assess clinical benefit. Third, multi-method explainability analyses including SHAP, LIME, permutation importance, and model-based importance measures were integrated to obtain robust and consistent feature importance rankings. Finally, a detailed robustness analysis was conducted to evaluate model stability under measurement uncertainty. In contrast to standard ML approaches, which may exhibit a tendency toward overfitting in small and heterogeneous datasets, the methodology applied in this study provides more reliable and generalizable estimates of real-world clinical performance.

In this context, the aim of the study was to use ML algorithms to predict the need for additional intervention following f-URS and to develop a clinically applicable, reliable, and innovative decision-support tool by rendering these models transparent through explainable XAI techniques.

Materials and methods

Study design and participants

This retrospective study included data from 656 patients who underwent f-URS between 01/01/2015 and 01/07/2025. During the study period, a total of 812 patients treated for kidney stones were initially screened. Among all patients treated for kidney stones, only those who underwent f-URS and had complete clinical data and preoperative computed tomography (CT) imaging were included in the study.

Of the initially screened patients, 51 patients in whom a second procedure was intentionally planned preoperatively (planned staged ureterorenoscopy) were excluded to ensure that the outcome reflected only unplanned additional interventions. In addition, 64 patients with missing or inadequate preoperative CT imaging and 41 patients with incomplete clinical records were excluded.

Accordingly, all anatomical measurements, including the ureteropelvic junction–pelvis angle (UPJ-PA), were derived exclusively from preoperative CT scans. Local ethics committee approval was obtained for the study (Protocol no: 2025/7 − 2). All methods were conducted in accordance with the relevant guidelines and regulations. Informed consent was obtained from all patients and/or their legal guardians.

Surgical technique and equipment

All procedures were performed using a standardized technique. Throughout the study period, a Holmium: YAG laser system (35 W) was used for stone fragmentation. The primary lithotripsy strategy consisted of fragmentation with active extraction, while the dusting technique was selectively applied in stones with favorable characteristics (low Hounsfield unit values and < 15 mm in size). Flexible ureteroscopes were used in all cases. Ureteral access sheaths were placed at the discretion of the operating surgeon when deemed necessary.

Flexible and Navigable Aspiration Suction Ureteral Access Sheaths (FANS-UAS) were introduced at our institution in January 2023. To address potential temporal bias, a subgroup analysis comparing outcomes before and after the introduction of FANS-UAS was performed. The primary analysis included all patients, and a sensitivity analysis restricted to patients treated after 2023 was performed to assess the robustness of the findings in the era following the introduction of FANS-UAS.

Data collection and variables

Demographic characteristics (age, sex, body mass index), stone characteristics (stone dimensions, area, volume, localization, and Hounsfield units [HU]), anatomical features (UPJ-PA), operative parameters (operation time, access sheath use, access sheath diameter, and FANS-UAS use), and postoperative outcomes were retrospectively collected from the database. Stone dimensions (maximum length, width, and depth) were measured on preoperative non-contrast CT images. Stone volume was then calculated using the ellipsoid formula (length × width × depth × π / 6).

The UPJ–PA angle was measured on preoperative non-contrast CT images in the coronal plane. A line extending from the midpoint of the widest portion of the renal pelvis toward the ureteropelvic junction (UPJ) was defined as the pelvic axis. A second line, extending caudally from the same UPJ point along the central axis of the ureter, was defined as the ureteral axis. The angle formed between these two lines at the UPJ point was defined as the UPJ–PA (Fig. 1). All angles were measured independently by two experienced urologists, and the mean values were used for analysis. Interobserver agreement was assessed using the intraclass correlation coefficient (ICC = 0.94, 95% CI: 0.92–0.96).

Fig. 1.

Schematic illustration of the ureteropelvic angle.

The final dataset consisted of 656 patients who underwent f-URS during the study period. The outcome variable was defined as the need for additional surgical intervention. Among these patients, 180 (27.4%) required an additional intervention, including re-f-URS in 94 patients, ureteroscopy in 61 patients, extracorporeal shock wave lithotripsy in 22 patients, and percutaneous nephrolithotomy in 3 patients. Rigid ureteroscopy was classified as an additional intervention only when performed for residual stone fragments or access failure following f-URS. Procedures performed solely for planned postoperative stent removal were not considered additional interventions.

Feature selection

Three different feature selection methods were comparatively evaluated: Boruta (using a random forest classifier with 500 trees over 100 iterations), LASSO regression (with the optimal regularization parameter α = 0.008, determined via 5-fold cross-validation), and Elastic Net (with optimal parameters α = 0.009 and L1 ratio = 0.9)^8–10. To ensure robust feature selection, only variables selected by at least two methods were included in the final model. This process yielded 16 features: UPJ–PA, stone volume, stone surface area, stone width, single versus multiple stones, access sheath size, use of an access sheath, use of FANS-UAS, sex, modified Charlson Comorbidity Index, preoperative hydronephrosis, preoperative serum creatinine, urinary leukocyte count, hemoglobin, platelet count, and PLR (Table 1).

Table 1.

Baseline Characteristics of Patients Undergoing Flexible Ureterorenoscopy (N  = 656).

Variable	Overall (N = 656)	No Additional Surgery (n = 476)	Additional Surgery (n = 180)	p	Effect Size
Demographic Characteristics
Age, years	43 (33–57)	42 (33–56)	44 (34–60)	0.204	r = 0.11
Sex				0.662	φ = 0.02
Female	288 (43.9)	206 (43.3)	82 (45.6)
Male	368 (56.1)	270 (56.7)	98 (54.4)
BMI, kg/m²	24.97 (23.60–27.41)	24.94 (23.51–27.42)	24.99 (23.70–27.00)	0.656	r = 0.00
Stone Characteristics
Stone side				0.449	φ = 0.03
Right	331 (50.5)	245 (51.5)	86 (47.8)
Left	325 (49.5)	231 (48.5)	94 (52.2)
Stone localization				0.016*	V = 0.13
Ureter	25 (3.8)	20 (4.2)	5 (2.8)
Pelvis	347 (52.9)	236 (49.6)	111 (61.7)
Lower pole	82 (12.5)	62 (13.0)	20 (11.1)
Middle pole	172 (26.2)	139 (29.2)	33 (18.3)
Upper pole	29 (4.4)	19 (4.0)	10 (5.6)
Multiple locations	1 (0.2)	0 (0.0)	1 (0.6)
Stone multiplicity				< 0.001***	φ = 0.37
Single	571 (87.0)	453 (95.2)	118 (65.6)
Multiple	85 (13.0)	23 (4.8)	62 (34.4)
Stone length, mm	11 (8–16)	10 (8–14)	16 (10–18)	< 0.001***	r = 0.92
Stone width, mm	13 (10–16)	12 (9–15)	15 (13–17)	< 0.001***	r = 0.82
Stone depth, mm	14 (11–16)	13 (10–15)	15 (13–18)	< 0.001***	r = 0.65
Stone surface area, mm²	106.76 (70.65–160.92)	94.20 (65.35–125.60)	178.59 (117.75–226.28)	< 0.001***	r = 1.28
Stone volume, mm³	922.91 (566.33–1604.60)	784.47 (503.40–1208.96)	1910.84 (991.08–2553.21)	< 0.001***	r = 1.35
Hounsfield units	859.73 (628.92–1161.16)	852.00 (623.62–1150.67)	907.00 (650.00–1200.00)	0.342	r = 0.09
UPJ-PA, degrees	127.85 (100.78–142.38)	135.11 (124.97–147.87)	90.55 (78.86–99.29)	< 0.001***	d = 2.60
Preoperative Clinical Parameters
Preoperative hydronephrosis				< 0.001***	V = 0.17
None	72 (11.0)	53 (11.1)	19 (10.6)
Grade 1	113 (17.2)	87 (18.3)	26 (14.4)
Grade 2	301 (45.9)	233 (48.9)	68 (37.8)
Grade 3	170 (25.9)	103 (21.6)	67 (37.2)
Modified CCI				0.292	V = 0.10
1	375 (57.2)	282 (59.2)	93 (51.7)
2	171 (26.1)	124 (26.1)	47 (26.1)
3	64 (9.8)	42 (8.8)	22 (12.2)
4	34 (5.2)	21 (4.4)	13 (7.2)
≥ 5	11 (1.7)	6 (1.3)	5 (2.8)
Preoperative nephrostomy	11 (1.7)	4 (0.8)	7 (3.9)	0.018*	φ = 0.09
Preoperative DJ stent	57 (8.7)	44 (9.2)	13 (7.2)	0.506	φ = 0.03
Urine leukocyte ≥ 10/hpf	364 (55.5)	264 (55.5)	100 (55.6)	1.000	φ = 0.00
Serum creatinine, mg/dL	0.87 (0.71–1.09)	0.88 (0.73–1.10)	0.85 (0.70–1.09)	0.203	r = 0.06
Operative Parameters
Access sheath use	565 (86.1)	408 (85.7)	157 (87.2)	0.710	φ = 0.01
Access sheath size				< 0.001***	V = 0.47
No sheath	91 (13.9)	68 (14.3)	23 (12.8)
9.5/11.5 Fr	192 (29.3)	79 (16.6)	113 (62.8)
10/12 Fr	240 (36.6)	204 (42.9)	36 (20.0)
11/13 Fr	133 (20.3)	125 (26.3)	8 (4.4)
FANS-UAS use	312 (47.6)	280 (58.8)	32 (17.8)	< 0.001***	φ = 0.36
Inflammatory Markers
Hemoglobin, g/dL	14.29 (12.73–15.39)	14.30 (12.79–15.40)	14.17 (12.70–15.30)	0.323	r = 0.09
Leukocyte, ×10³/µL	8.51 (7.50–9.61)	8.53 (7.50–9.60)	8.47 (7.57–9.63)	0.874	r = 0.02
Neutrophil, ×10³/µL	4.50 (3.81–5.45)	4.52 (3.83–5.40)	4.50 (3.75–5.59)	0.796	r = 0.00
Lymphocyte, ×10³/µL	2.55 (2.04–3.00)	2.53 (2.07–3.00)	2.59 (2.00–2.98)	0.833	r = 0.00
Monocyte, ×10³/µL	0.75 (0.68–0.83)	0.75 (0.68–0.82)	0.75 (0.70–0.84)	0.040*	r = 0.25
Platelet, ×10³/µL	253.75 (209.93–302.85)	257.35 (209.55–302.85)	247.00 (210.00–301.80)	0.539	r = 0.02
NLR	1.82 (1.47–2.27)	1.83 (1.48–2.29)	1.80 (1.44–2.25)	0.784	r = 0.04
PLR	104.06 (82.49–124.64)	103.75 (82.35–124.37)	104.66 (84.32–125.42)	0.823	r = 0.01
LMR	3.38 (2.67–4.14)	3.39 (2.69–4.16)	3.33 (2.56–4.05)	0.210	r = 0.11
PIV	345.83 (268.08–440.41)	352.75 (268.44–431.64)	335.25 (267.17–462.96)	0.820	r = 0.08
SII	464.33 (359.69–581.94)	465.47 (363.18–581.82)	452.03 (349.61–584.02)	0.558	r = 0.02

Note. Continuous variables are presented as median (IQR). Categorical variables are presented as n (%). BMI = body mass index; UPJ-PA = ureteropelvic junction angle; CCI = Charlson Comorbidity Index; FANS-UAS = Flexible and Navigable Suction Ureteral Access Sheath; NLR = neutrophil-to-lymphocyte ratio; PLR = platelet-to-lymphocyte ratio; LMR = lymphocyte-to-monocyte ratio; PIV = pan-immune-inflammation value; SII = systemic immune-inflammation index.

Effect sizes d = Cohen’s d for t-tests; r for Mann-Whitney U tests; φ (phi) or V (Cramér’s V) for χ² tests.

*p < 0.05. ***p < 0.001.

Machine learning models and validation strategy

A total of fourteen ML algorithms were evaluated: Logistic Regression, Ridge Classifier, Support Vector Machine (SVM), Random Forest, Extra Trees, Gradient Boosting, AdaBoost, XGBoost, CatBoost, LightGBM, k-Nearest Neighbours (KNN), Naive Bayes, Decision Tree, and Neural Network (Multilayer Perceptron). In addition, ensemble methods, including soft voting and stacking with a logistic regression meta-learner, were implemented^11–13.

Nested Stratified Cross-Validation

To prevent overfitting and to obtain unbiased performance estimates, a nested stratified 5-fold cross-validation framework was used. The outer loop (5 folds) was employed for performance estimation, whereas the inner loop (5 folds) was used for hyperparameter tuning. This design ensures that performance estimates are not influenced by data leakage arising from hyperparameter selection.

Handling Class Imbalance

The dataset exhibited class imbalance (72.6% negative, 27.4% positive; imbalance ratio = 2.64). Multiple resampling techniques were evaluated, including SMOTE, Borderline-SMOTE, ADASYN, random over-/under-sampling, Tomek links, and Edited Nearest Neighbours (ENN). Based on comprehensive evaluation using AUC, F1 score, and calibration metrics, ENN was selected as it provided the optimal balance between discrimination and calibration without introducing synthetic data that could distort real-world patterns. Importantly, resampling was applied only within the training folds, while the test data were retained in their original distribution to ensure realistic performance estimation^14–16.

Model performance evaluation

Discrimination metrics

The area under the receiver operating characteristic curve (AUC–ROC), sensitivity, specificity, precision, F1 score, and Matthews correlation coefficient (MCC) were calculated. Bootstrap resampling (n = 1,000) was used to generate 95% confidence intervals for all metrics.

Calibration assessment

Model calibration was evaluated using the Brier score (overall accuracy of probability estimates), Expected Calibration Error (ECE) (mean calibration error across probability bins), and Maximum Calibration Error (MCE) (worst-case calibration error). A Brier score < 0.10 and ECE < 0.10 were considered indicative of excellent calibration.

Clinical utility: Decision Curve Analysis (DCA) was performed to assess the clinical usefulness of the models across a range of threshold probabilities (1–60%). Net benefit was calculated as: Net Benefit = (True Positives / n) – (False Positives / n) × (pt / (1 – pt)),

Where pt denotes the threshold probability. The clinically relevant threshold range was defined as 10–40%, based on the clinical context in which the cost of missing a case requiring intervention is approximately 1.5–4 times higher than the cost of unnecessary evaluation.

Model discrimination was evaluated using the area under the receiver operating characteristic curve (AUC-ROC) with nested stratified 5-fold cross-validation. The outer loop assessed test performance on held out data, while the inner loop validated model training. Mean ROC curves with 95% confidence intervals were computed across all folds. In DCA and ROC analysis, the Modified Nearest Neighbours (ENN) subsampling method was applied only to the training folds veya training sets, ensuring that the test sets remained 100% original and unchanged.

Model explainability

Four complementary methods were used to achieve model explainability:

SHAP (SHapley Additive exPlanations)

Based on principles from game theory, SHAP provides both global feature importance rankings and local explanations for individual predictions.

LIME (Local Interpretable Model-agnostic Explanations)

LIME constructs local surrogate models to explain individual predictions.

Permutation Importance

Feature importance was quantified by measuring the decrease in model performance when each feature was randomly permuted.

Model-based Importance

For logistic regression, absolute coefficient values after standardization were used as indicators of feature importance.

Feature importance scores were normalized to a 0–1 scale to enable cross-method comparison. Inter-method agreement was assessed using Spearman correlation coefficients.

Robustness analysis

A robustness analysis was conducted to evaluate the stability of model predictions under measurement uncertainty, particularly for UPJ–PA, which is prone to interobserver variability. Gaussian noise with standard deviations of 0°, 3°, 5°, 7°, and 10° was added to the UPJ–PA measurements, and model performance was re-evaluated at each noise level. The model was considered robust if the decrease in AUC was less than 1% at a noise level of σ = 10°.

Statistical analysis

Normality was assessed using the Kolmogorov–Smirnov test. Normally distributed continuous variables were presented as mean ± standard deviation and compared using the Student’s t-test. Non-normally distributed variables were presented as median (interquartile range [IQR]) and compared using the Mann–Whitney U test. For variables exhibiting mixed distribution patterns between groups, IQRs were reported for both groups to maintain consistency. Categorical variables were expressed as counts (percentages) and compared using the chi-square test. Effect sizes were calculated as Cohen’s d for t-tests, r for Mann–Whitney U tests, and Cramér’s V or phi coefficient for chi-square tests. Baseline characteristics were additionally compared between patients treated with and without FANS-UAS to assess potential confounding related to technological adoption, with results reported in the Supplementary Material. DeLong’s test was used to compare AUC values between models. A P value < 0.05 was considered statistically significant. All analyses were performed using Python 3.12.3 with the scikit-learn, SHAP, and statsmodels libraries.

Code availability

The custom Python code developed for this study, including data preprocessing, class balancing (EditedNearestNeighbours), feature selection (Boruta, RFE), nested stratified cross-validation, machine learning model training, explainability analyses (SHAP, LIME, permutation importance), decision curve analysis, calibration analysis, robustness testing, and the clinical decision support system, is publicly available at: https://github.com/huseyinKutlu/fURS-additional-procedure-prediction (DOI: 10.5281/zenodo.19225028).

Results

Patient characteristics

A total of 656 patients were included in the analysis. The study cohort consisted of 368 men (56.1%), with a median age of 43 years (IQR: 33–57). Additional surgical intervention was required in 180 patients (27.4%), including repeat f-URS (n = 94, 52.2%), ureteroscopy (n = 61, 33.9%), extracorporeal shock wave lithotripsy (n = 22, 12.2%), and percutaneous nephrolithotomy (n = 3, 1.7%).

The most striking finding was related to UPJ–PA measurements. The mean UPJ–PA angle was 88.89° ± 17.66 in patients requiring additional surgery, compared with 135.99° ± 18.33 in those not requiring additional intervention (t(654) = 15.23, p < 0.001, Cohen’s d = 2.60). Stone-related characteristics, including stone volume, stone surface area, and the presence of multiple stones, differed significantly between the groups (all p < 0.001). The distribution of access sheath sizes showed marked differences between groups (χ²(3) = 145.61, p < 0.001, Cramér’s V = 0.47), with smaller sheath sizes being more prevalent in the additional intervention group. FANS-UAS use was significantly lower in patients requiring additional surgery (17.8% vs. 58.8%, χ²(1) = 87.23, p < 0.001, φ = 0.36). Complete baseline characteristics are presented in Table 1.

Baseline characteristics stratified by FANS-UAS utilization

Baseline characteristics stratified by FANS-UAS utilization are presented in Supplementary Table S1. The two groups were comparable in terms of age (45.5 ± 15.2 vs. 44.5 ± 15.8 years, p = 0.29), body mass index (25.5 ± 3.3 vs. 25.2 ± 3.2 kg/m², p = 0.37), Hounsfield unit values (883.9 ± 329.2 vs. 895.8 ± 336.3 HU, p = 0.57), and preoperative creatinine levels (1.07 ± 0.86 vs. 1.01 ± 0.58 mg/dL, p = 0.81).

In contrast, significant differences were observed in stone-related parameters. Patients treated with FANS-UAS had smaller stone length (11.3 ± 4.1 vs. 12.5 ± 4.9 mm, p = 0.003), width (12.1 ± 3.6 vs. 13.6 ± 4.0 mm, p < 0.001), depth (13.0 ± 3.4 vs. 14.0 ± 3.8 mm, p < 0.001), surface area (107.8 ± 53.8 vs. 137.7 ± 78.5 mm², p < 0.001), and volume (982.6 ± 669.4 vs. 1349.5 ± 969.5 mm³, p < 0.001) compared with those treated without FANS-UAS. In addition, the UPJ–PA was significantly wider in the FANS-UAS group (131.3 ± 23.2° vs. 115.6 ± 29.4°, p < 0.001; Cohen’s d = 0.59). These findings demonstrate a non-random distribution of FANS-UAS utilization with respect to baseline stone burden and ureteropelvic anatomy.

Feature selection results

Boruta selected 9 features, whereas LASSO and Elastic Net each selected 16 features. Seven features were consistently selected by all three methods: UPJ–PA, stone volume, stone surface area, stone width, single versus multiple stones, access sheath size, and FANS-UAS use. The final model included 16 features selected by at least two methods. Performance comparison demonstrated that the 16-feature model (AUC: 0.986, 95% CI: 0.973–1.000) achieved performance comparable to the full 43-feature model, while providing a 63% reduction in model complexity (Table 2).

Table 2.

Feature Selection Method Comparison and Model Performance Panel A: Features Selected by Each Method.

Feature	Boruta	LASSO	ElasticNet	Final Model
UPJ-PA	✓	✓	✓	✓
Stone volume	✓	✓	✓	✓
Stone surface area	✓	✓	✓	✓
Stone width	✓	✓	✓	✓
Single/multiple stone	✓	✓	✓	✓
Access sheath size	✓	✓	✓	✓
FANS-UAS use	✓	✓	✓	✓
Sex	—	✓	✓	✓
Modified CCI	—	✓	✓	✓
Preoperative hydronephrosis	—	✓	✓	✓
Serum creatinine	—	✓	✓	✓
Urine leukocyte	—	✓	✓	✓
Access sheath use	—	✓	✓	✓
Hemoglobin	—	✓	✓	✓
Platelet	—	✓	✓	✓
PLR	—	✓	✓	✓
Stone length	✓	—	—	—
Stone depth	✓	—	—	—

Panel B: Model Performance by Feature Set.

Feature Set	n Features	AUC (SD)	F1 Score (SD)
All features	34	0.981 (0.015)	0.892 (0.042)
Boruta only	9	0.984 (0.011)	0.889 (0.034)
LASSO only	16	0.982 (0.014)	0.897 (0.039)
ElasticNet only	16	0.982 (0.014)	0.897 (0.039)
Common (all 3 methods)	7	0.985 (0.010)	0.892 (0.036)
Final model (≥ 2 methods)	16	0.986 (0.009)	0.898 (0.031)

Note. Boruta was run with a 500-tree random forest classifier for 100 iterations. LASSO used optimal regularization parameter α = 0.008 via 5-fold cross-validation. ElasticNet used α = 0.009 with L1 ratio = 0.9. The final model includes 16 features selected by at least two methods, achieving 63% complexity reduction compared to the full 34-feature model while maintaining comparable performance. UPJ-PA = ureteropelvic junction angle; CCI = Charlson Comorbidity Index; FANS-UAS = Flexible and Navigable Suction Ureteral Access Sheath; PLR = platelet-to-lymphocyte ratio.

✓ = selected by method; — = not selected. Performance metrics are from nested 5-fold cross-validation with Logistic Regression.

Handling class imbalance

Comparison of resampling methods demonstrated that Edited Nearest Neighbours (ENN) provided the optimal balance between discrimination and calibration (Table 3). While SMOTE and Borderline-SMOTE achieved marginally higher AUC values (0.981), ENN exhibited superior calibration, reflected by lower Brier scores. The ENN approach improves class separability by removing noisy majority-class samples near decision boundaries, thereby avoiding the introduction of synthetic samples that could distort real-world patterns.

Table 3.

Comparison of Class Imbalance Handling Methods.

Method	Distribution	AUC	F1	Precision	Recall	Bal. Acc.	MCC
Baseline	380:144	0.979 (0.016)	0.857 (0.066)	0.889 (0.049)	0.833 (0.101)	0.897 (0.052)	0.810 (0.082)
Random Under	144:144	0.980 (0.014)	0.882 (0.045)	0.823 (0.079)	0.956 (0.028)	0.937 (0.024)	0.838 (0.062)
Random Over	380:380	0.979 (0.015)	0.887 (0.056)	0.883 (0.065)	0.894 (0.073)	0.924 (0.042)	0.846 (0.077)
SMOTE	380:304	0.981 (0.012)	0.895 (0.034)	0.889 (0.048)	0.906 (0.054)	0.931 (0.026)	0.857 (0.045)
BorderlineSMOTE	380:304	0.981 (0.012)	0.888 (0.053)	0.885 (0.074)	0.894 (0.054)	0.924 (0.035)	0.846 (0.073)
SVMSMOTE	380:304	0.980 (0.014)	0.880 (0.052)	0.873 (0.070)	0.889 (0.050)	0.919 (0.035)	0.835 (0.074)
ADASYN	380:280	0.977 (0.017)	0.885 (0.053)	0.877 (0.061)	0.894 (0.062)	0.923 (0.038)	0.841 (0.074)
Tomek Links	363:144	0.980 (0.014)	0.871 (0.044)	0.878 (0.045)	0.867 (0.069)	0.910 (0.036)	0.824 (0.060)
ENN a	281:144	0.979 (0.018)	0.891 (0.049)	0.860 (0.066)	0.928 (0.048)	0.934 (0.033)	0.850 (0.069)
SMOTE + ENN	236:338	0.979 (0.015)	0.866 (0.058)	0.827 (0.090)	0.917 (0.068)	0.919 (0.040)	0.816 (0.082)
SMOTE + Tomek	375:375	0.978 (0.016)	0.873 (0.060)	0.883 (0.064)	0.867 (0.075)	0.911 (0.044)	0.828 (0.082)

Note. Data source: Table 2_Balancing_Methods_Comparison.xlsx. Performance metrics are presented as mean (SD) from nested 5-fold cross-validation with Logistic Regression. Distribution shows the ratio of majority to minority class after balancing. Original distribution: 476:180 (ratio: 2.64). AUC = area under the ROC curve; Bal. Acc. = balanced accuracy; MCC = Matthews correlation coefficient; ENN = Edited Nearest Neighbours; SMOTE = Synthetic Minority Over-sampling Technique; ADASYN = Adaptive Synthetic Sampling.

^aSelected method. ENN provided the optimal balance between discrimination and calibration without introducing synthetic data.

UPJ–PA angle and risk stratification

Decile analysis of UPJ–PA values identified 110° as a critical threshold (Fig. 2). The rate of additional intervention was 84.3% (167/198) in patients with UPJ–PA < 110° (deciles 1–3), compared with 2.8% (13/458) in those with UPJ–PA ≥ 110° (deciles 4–10). The risk ratio was 29.7 (95% CI: 17.1–51.2, p < 0.001). A particularly sharp transition was observed between the third decile (96–110°; intervention rate: 72.7%) and the fourth decile (110–122°; intervention rate: 7.7%), confirming the threshold-based nature of this anatomical predictor.

Fig. 2.

Risk stratification by UPJ-PA deciles demonstrating a sharp transition at the 110° threshold. Deciles 1–3 (UPJ-PA < 110°) exhibited an 84.3% intervention rate compared to 2.8% for Deciles 4–10 (UPJ-PA ≥ 110°). Risk Ratio: 29.7.

Based on this analysis, a four-tier clinical risk stratification system was developed (Fig. 3): Very High Risk (UPJ–PA < 90°, n = 94, intervention rate: 92.6%), High Risk (90–110°, n = 101, intervention rate: 78.2%), Intermediate Risk (110–130°, n = 176, intervention rate: 6.2%), and Low Risk (> 130°, n = 285, intervention rate: 1.1%).

Fig. 3.

Clinical Decision Support System showing four-tiered risk stratification based on UPJ-PA categories: Very High Risk (< 90°, 92.6%), High Risk (90–110°, 78.2%), Moderate Risk (110–130°, 6.2%), and Low Risk (> 130°, 1.1%).

Stone location–stratified analysis according to UPJ–PA

To evaluate whether the predictive effect of UPJ–PA varied according to stone location, a stratified analysis was performed across five anatomical subgroups: ureter, renal pelvis, lower pole, middle pole, and upper pole stones. Within each subgroup, patients with a UPJ–PA ≤ 110° consistently demonstrated higher rates of additional intervention compared with those with a UPJ–PA > 110°. Although the magnitude of the association was most pronounced in lower pole stones, clinically relevant differences were also observed in renal pelvis and upper pole stones.

These findings suggest that UPJ–PA reflects a global anatomical constraint affecting intrarenal access and fragment clearance rather than a parameter limited to a specific stone location. Detailed stratified results are provided in Supplementary Table S2.

Machine learning model performance

Performance metrics obtained from nested stratified 5-fold cross-validation are summarized in Table 4. The highest AUC–ROC values were achieved by the Ridge Classifier (0.987, 95% CI: 0.978–0.997) and Logistic Regression (0.987, 95% CI: 0.973–1.000), followed by the ensemble methods (Stacking: 0.986; Soft Voting: 0.986). All top-performing models demonstrated excellent discrimination, with AUC values exceeding 0.98.

Table 4.

Machine Learning Model Performance Metrics With 95% Bootstrap Confidence Intervals.

Model	AUC [95% CI]	F1 [95% CI]	Precision [95% CI]	Recall [95% CI]	MCC [95% CI]
Ridge Classifier	0.987 [0.978–0.997]	0.894 [0.851–0.937]	0.841 [0.775–0.907]	0.956 [0.925–0.986]	0.854 [0.794–0.914]
Logistic Regression a	0.987 [0.973–1.00]	0.889 [0.837–0.941]	0.837 [0.764–0.910]	0.950 [0.905–0.995]	0.847 [0.774–0.920]
Stacking	0.986 [0.972–1.00]	0.895 [0.832–0.957]	0.861 [0.771–0.951]	0.933 [0.881–0.986]	0.854 [0.767–0.941]
Soft Voting	0.986 [0.972–1.00]	0.895 [0.842–0.948]	0.857 [0.778–0.936]	0.939 [0.894–0.984]	0.855 [0.781–0.928]
CatBoost	0.985 [0.971–0.998]	0.884 [0.832–0.937]	0.847 [0.765–0.930]	0.928 [0.875–0.980]	0.840 [0.768–0.913]
Random Forest	0.983 [0.963–1.00]	0.903 [0.838–0.967]	0.858 [0.753–0.962]	0.956 [0.937–0.974]	0.866 [0.777–0.954]
SVM	0.982 [0.965–0.999]	0.876 [0.808–0.944]	0.820 [0.715–0.926]	0.944 [0.920–0.969]	0.829 [0.735–0.923]
AdaBoost	0.982 [0.965–0.998]	0.888 [0.831–0.944]	0.859 [0.785–0.932]	0.922 [0.844–1.00]	0.846 [0.767–0.925]
LightGBM	0.978 [0.960–0.996]	0.896 [0.849–0.943]	0.852 [0.766–0.938]	0.950 [0.899–1.00]	0.858 [0.793–0.923]
XGBoost	0.978 [0.963–0.993]	0.887 [0.851–0.924]	0.844 [0.781–0.907]	0.939 [0.873–1.00]	0.845 [0.793–0.897]
Extra Trees	0.978 [0.964–0.991]	0.868 [0.814–0.921]	0.828 [0.730–0.927]	0.917 [0.862–0.971]	0.818 [0.742–0.893]
Gradient Boosting	0.977 [0.950–1.00]	0.898 [0.835–0.960]	0.863 [0.767–0.959]	0.939 [0.882–0.996]	0.859 [0.771–0.947]
KNN	0.958 [0.936–0.980]	0.820 [0.734–0.906]	0.879 [0.763–0.995]	0.772 [0.670–0.874]	0.762 [0.644–0.881]
Naive Bayes	0.955 [0.924–0.987]	0.822 [0.734–0.910]	0.794 [0.721–0.868]	0.856 [0.730–0.981]	0.755 [0.631–0.878]
Decision Tree	0.940 [0.903–0.976]	0.865 [0.807–0.923]	0.789 [0.692–0.886]	0.961 [0.942–0.980]	0.815 [0.736–0.894]
Neural Network	0.919 [0.848–0.990]	0.779 [0.679–0.879]	0.846 [0.707–0.986]	0.739 [0.574–0.904]	0.714 [0.591–0.838]

Note. Data source: Table_Detailed_CI.xlsx. Performance metrics from nested stratified 5-fold cross-validation with Edited Nearest Neighbours (ENN) balancing. Bootstrap resampling (n = 1,000) was used to generate 95% confidence intervals. Models are sorted by AUC in descending order. AUC = area under the receiver operating characteristic curve; MCC = Matthews correlation coefficient; SVM = support vector machine; KNN = k-nearest neighbours.

^aRecommended model for clinical implementation based on optimal balance of discrimination (AUC: 0.987), calibration (Brier score: 0.050), interpretability, and minimal overfitting (train-test AUC gap: 1.1%).

Overfitting Assessment: The difference between training and test AUC was minimal for Logistic Regression (1.1%) and the Ridge Classifier (1.0%), indicating good generalization performance. In contrast, ensemble methods exhibited slightly larger differences (Stacking: 1.4%; CatBoost: 1.5%), although these remained within acceptable limits (< 5%). The Decision Tree demonstrated the highest risk of overfitting (difference: 5.0%).

Note on the Ridge Classifier

The Ridge Classifier does not inherently produce probability estimates required for standard AUC computation, which can result in AUC values being reported as zero. Therefore, Platt scaling was applied to calibrate the Ridge Classifier outputs into probability estimates, enabling the calculation of the correct AUC value (0.987).

Model calibration

Calibration analysis demonstrated excellent performance for most models (Table 5). CatBoost achieved the lowest Brier score (0.048) along with excellent ECE (0.029). Logistic Regression maintained the highest AUC while exhibiting good calibration (Brier: 0.050; ECE: 0.063). Among the ensemble approaches, soft voting showed the best overall calibration (Brier: 0.045; ECE: 0.037). Despite its high discriminative performance, the Ridge Classifier demonstrated moderate calibration, attributable to its lack of inherent probability estimation (Brier: 0.106; ECE: 0.225).

Table 5.

Model Calibration Metrics Including Brier Score, Expected Calibration Error, and Maximum Calibration Error.

Model	Brier Score	ECE	MCE
Soft Voting	0.045	0.037	0.211
Stacking	0.047	0.041	0.631
CatBoost	0.048	0.029	0.345
Random Forest	0.050	0.066	0.274
Logistic Regression a	0.050	0.063	0.341
SVM	0.052	0.036	0.350
XGBoost	0.054	0.035	0.266
Gradient Boosting	0.054	0.045	0.481
LightGBM	0.055	0.047	0.447
KNN	0.070	0.023	0.121
Extra Trees	0.071	0.111	0.300
Decision Tree	0.075	0.069	0.444
Naive Bayes	0.086	0.076	0.453
AdaBoost	0.097	0.216	0.307
Ridge Classifier	0.106	0.225	0.342
Neural Network	0.112	0.138	0.182

Note. Data source: Table_Calibration.xlsx. Models are sorted by Brier score in ascending order (lower is better). Brier score measures overall accuracy of probability predictions (range: 0–1; <0.10 indicates excellent calibration). ECE = expected calibration error, representing average calibration error across probability bins (< 0.10 indicates excellent calibration). MCE = maximum calibration error, representing worst-case calibration error. Calibration status: Excellent = Brier < 0.10 and ECE < 0.10; Good = Brier < 0.15 or ECE < 0.25. SVM = support vector machine; KNN = k-nearest neighbours.

^aRecommended model for clinical implementation. Logistic Regression combines excellent calibration (Brier: 0.050, ECE: 0.063) with the highest discrimination (AUC: 0.987) and full interpretability through coefficient analysis.

ROC and decision curve analysis

Receiver operating characteristic analysis demonstrated excellent discriminative performance across all models (Fig. 4A-B). Ridge Classifier achieved the highest AUC of 0.987 (95% CI: 0.973–1.000), followed by Logistic Regression (0.986; 95% CI: 0.968–1.000) and CatBoost (0.984; 95% CI: 0.959–1.000). All ensemble and linear models achieved AUC values exceeding 0.97, indicating robust discrimination between patients requiring additional surgery and those who did not. Decision Curve Analysis demonstrated clinical benefit across a wide range of threshold probabilities (Fig. 4C-D). All top-performing models showed a positive net benefit compared with the “treat-all” and “treat-none” strategies across threshold probabilities ranging from 3% to 60%. CatBoost achieved the highest mean net benefit within the clinically relevant range (10–40%), with an average net benefit of 0.245. Logistic Regression demonstrated competitive clinical utility while maintaining interpretability (mean net benefit: 0.240). Considering both discriminative accuracy (AUC = 0.986) and clinical utility, Logistic Regression was selected as the optimal model for clinical implementation due to its excellent performance combined with superior interpretability for clinical decision-making.

Fig. 4.

ROC and Decision Curve Analysis of prediction models. (A) ROC curves for top 6 models (mean ± SD). (B) ROC comparison of all models. (C) Decision curve analysis across threshold probabilities 1–60%; green shading indicates clinical range (10–40%). (D) DCA in clinical decision range. (E) AUC-ROC comparison with error bars. (F) Performance summary. Nested 5-fold cross-validation with ENN applied only to training folds.

Clinical Interpretation

The developed prediction model provides clinicians with a quantitative tool to estimate the probability of additional surgery requirement in individual patients. At the recommended decision threshold of 20%, the model demonstrated a net benefit of approximately 0.24, indicating that for every 100 patients evaluated, the model would correctly identify 24 additional patients requiring intervention compared to a strategy of treating no patients, without increasing unnecessary interventions.

The clinical utility of the model was validated across a wide range of threshold probabilities (10–40%), allowing clinicians to adjust the decision threshold based on institutional resources, patient preferences, and the relative costs of false-positive versus false-negative predictions. In settings where the consequences of missing a patient requiring additional surgery are severe, a lower threshold (e.g., 10–15%) may be preferred to maximize sensitivity. Conversely, in resource-limited settings, a higher threshold (e.g., 30–40%) may be appropriate to minimize unnecessary interventions.

The model’s predictions should be interpreted as decision-support tools rather than definitive diagnoses. Clinical judgment remains essential, and the predicted probabilities should be integrated with patient-specific factors, clinical examination findings, and shared decision-making processes. The high discriminative accuracy (AUC = 0.986) combined with positive net benefit across clinically relevant thresholds supports the implementation of this model in routine clinical practice to facilitate early identification of high-risk patients and optimize surgical planning.

Model explainability

Normalized feature importance scores across the four explainability methods are presented in Table 6. UPJ–PA emerged as the dominant predictor across all methods, achieving perfect normalized scores (1.000) and demonstrating exceptional inter-method agreement. The second most important feature was FANS-UAS use (mean: 0.464), followed by access sheath size (0.396), single versus multiple stones (0.256), and stone volume (0.246).

Table 6.

Normalized Feature Importance Scores Across Four Explainability Methods With Inter-Method Correlations.

Feature	SHAP	Permutation	LR Coefficient	LIME	Average
UPJ-PA	1.000	1.000	1.000	1.000	1.000
FANS-UAS use	0.903	0.122	0.276	0.553	0.464
Access sheath size	0.573	0.101	0.312	0.598	0.396
Single/multiple stone	0.358	0.025	0.135	0.507	0.256
Stone volume	0.308	0.018	0.218	0.439	0.246
Stone surface area	0.297	0.012	0.127	0.378	0.203
Access sheath use	0.096	0.006	0.293	0.000	0.099
Stone width	0.159	−0.004	0.094	0.111	0.090
Modified CCI	0.031	−0.005	0.105	0.032	0.041
Serum creatinine	0.013	−0.001	0.115	0.020	0.037
Hydronephrosis	0.040	0.002	0.029	0.034	0.026
Urine leukocyte	0.021	−0.001	0.054	0.019	0.023
Sex	0.016	−0.001	0.035	0.014	0.016
Platelet	0.010	−0.003	0.034	0.019	0.015
PLR	0.007	−0.000	0.036	0.016	0.015
Hemoglobin	0.013	−0.001	0.024	0.018	0.013

Note. Data source: Table_XAI_Normalized_Comparison.xlsx. Normalized feature importance scores across methods are presented for comparison. Features are sorted by average importance in descending order. Negative permutation importance values indicate that shuffling the feature improved model performance, suggesting minimal predictive contribution. SHAP = SHapley Additive exPlanations; LR = logistic regression; LIME = Local Interpretable Model-agnostic Explanations; UPJ-PA = ureteropelvic junction angle; CCI = Charlson Comorbidity Index; PLR = platelet-to-lymphocyte ratio.

Inter-method Spearman correlations: SHAP–LIME r = 0.84; SHAP–LR r = 0.79; SHAP–Permutation r = 0.77; LR–Permutation r = 0.72; LR–LIME r = 0.62; Permutation–LIME r = 0.63. All correlations significant at p < 0.05.

SHAP Analysis (Fig. 5): The SHAP beeswarm plot visualized the individual effects of features on model output. Lower UPJ–PA values (blue points on the right side of the plot) were strongly associated with positive SHAP values, indicating an increased probability of requiring additional surgical intervention. The distinct separation pattern observed for UPJ–PA confirms its threshold-based rather than linear predictive role. FANS-UAS use exhibited protective effects (negative SHAP values when FANS-UAS = 1), whereas larger stone volumes were associated with increased predicted risk.

Fig. 5.

SHAP beeswarm plot for the Logistic Regression model showing feature effects on model output. Low UPJ-PA values (blue points) strongly push predictions toward additional surgery requirement.

Clinical Interpretation of Feature Effects

The clinical relevance of the identified predictors can be understood as follows.

(1) The UPJ–PA angle reflects the anatomical difficulty of flexible ureteroscope navigation—narrow angles (< 110°) create a more acute turn at the ureteropelvic junction, limiting scope deflection and hindering complete stone clearance;

(2) FANS-UAS provides continuous aspiration during the procedure, improving visualization and facilitating fragment removal, thereby reducing residual stone burden;

(3) Larger access sheaths (> 10 Fr) allow improved irrigation flow and fragment extraction, decreasing the need for repeat procedures; and.

(4) Multiple stones and larger stone volumes inherently require longer operative times and are associated with a higher likelihood of residual fragments.

LIME Case Analysis (Fig. 6): Correctly classified true-positive cases (Cases 3–6) were examined to evaluate local feature contributions. UPJ–PA consistently provided the strongest positive contribution to predicting additional surgery (+ 0.68 to + 0.71). In false-positive cases (Cases 1–2), low UPJ–PA values drove the model toward a surgical prediction despite the absence of a subsequent intervention. These cases likely represent patients in whom other protective factors (e.g., surgeon experience or optimal equipment) compensated for unfavorable anatomy. This analysis illustrates the model’s clinically plausible decision-making process and highlights potential areas for clinical override.

Fig. 6.

LIME explanations for representative cases showing local feature contributions. Cases 1–2: False positives where low UPJ-PA drove predictions despite no surgery required. Cases 3–6: True positives correctly identified by the model.

Robustness analysis

The robustness analysis demonstrated excellent model stability under measurement uncertainty (Fig. 7; Table 7). At a noise level of σ = 10°, representing substantial measurement variability, logistic regression showed only a 0.26% decrease in AUC (from 0.986 to 0.984), well below the 1% threshold for clinical acceptability. All top-performing models maintained AUC values above 0.97, even at the highest noise level. UPJ–PA remained the most important feature across all noise levels, confirming the robustness of the findings.

Fig. 7.

Robustness analysis showing model performance under UPJ-PA measurement noise (σ = 0° to 10°). All top-performing models maintained AUC > 0.97 with < 0.4% degradation at σ = 10°.

Table 7.

Robustness Analysis Results Showing AUC and F1 Degradation Under Increasing UPJ-PA Measurement Noise Levels.

Model	σ = 0°	σ = 3°	σ = 5°	σ = 7°	σ = 10°	AUC Δ%	F1 Δ%
	AUC	AUC	AUC	AUC	AUC
Naive Bayes	0.955	0.955	0.955	0.955	0.954	0.17	0.53
Extra Trees	0.978	0.977	0.977	0.976	0.976	0.21	1.77
Random Forest	0.982	0.982	0.982	0.981	0.979	0.25	1.62
Logistic Regression a	0.986	0.986	0.986	0.985	0.984	0.26	1.73
LightGBM	0.977	0.977	0.977	0.976	0.974	0.30	1.89
SVM	0.979	0.979	0.979	0.978	0.976	0.30	0.93
Ridge Classifier	0.987	0.987	0.986	0.986	0.984	0.30	1.99
CatBoost	0.984	0.983	0.983	0.982	0.981	0.31	1.58
Gradient Boosting	0.977	0.977	0.976	0.975	0.974	0.37	1.61
XGBoost	0.977	0.977	0.976	0.975	0.973	0.39	0.69
AdaBoost	0.976	0.976	0.975	0.973	0.972	0.44	1.34
Decision Tree	0.909	0.908	0.907	0.907	0.904	0.56	0.76
KNN	0.932	0.938	0.939	0.939	0.938	−0.62	0.73

Note. Data source: Table_Robustness_Detailed.xlsx and Table_Robustness_Summary.xlsx. Gaussian noise with standard deviations (σ) of 0°, 3°, 5°, 7°, and 10° was added to UPJ-PA measurements to simulate inter-observer variability. Models are sorted by AUC degradation percentage (ascending). AUC Δ% = percentage decrease in AUC from baseline (σ = 0°) to σ = 10°; F1 Δ% = percentage decrease in F1 score. Negative degradation indicates improved performance with noise (likely due to regularization effect). A model was considered robust if AUC degradation remained < 1% at σ = 10°. SVM = support vector machine; KNN = k-nearest neighbours.

^aRecommended model. Logistic Regression demonstrated excellent robustness with only 0.26% AUC degradation at σ = 10°, maintaining AUC > 0.98 across all noise levels while preserving full interpretability.

Bootstrap analysis of SHAP importance (n = 1,000 iterations) confirmed the dominant role of UPJ–PA, with a mean importance of 3.27 (95% CI: 2.70–3.90), representing 2.7 times the importance of the second-ranked feature.

Recommended model for clinical use

Based on the comprehensive evaluation, logistic regression is recommended for clinical implementation for the following reasons:

(1) Competitive discrimination comparable to more complex models (AUC: 0.987);

(2) Competitive calibration providing reliable probability estimates (Brier score: 0.050);

(3) High clinical utility within the relevant threshold range (DCA net benefit: 0.240);

(4) Minimal risk of overfitting (training–test AUC difference: 1.1%);

(5) Full interpretability through coefficient analysis;

(6) Computational efficiency, enabling real-time predictions; and.

(7) An established regulatory pathway for clinical deployment.

Although CatBoost demonstrated a slightly higher net benefit in DCA (0.245 vs. 0.240), the superior interpretability and regulatory considerations of logistic regression make it the preferred choice for initial clinical implementation. The model may be deployed as a web-based calculator, allowing clinicians to input patient-specific variables (e.g., UPJ–PA, stone characteristics, and planned equipment) and receive probability estimates with confidence intervals.

Discussion

In this study, UPJ–PA emerged as the strongest predictor of additional intervention after f-URS. The intervention rate was 84.3% for angles below 110° and decreased sharply to 2.8% for angles above this threshold. This pronounced threshold behavior suggests that UPJ–PA is a parameter applicable to all renal stone locations, globally reflecting renal pelvic anatomy rather than being limited to a specific calyceal subgroup. Consistently high model performance was observed (e.g., Logistic Regression AUC = 0.987, Soft Voting AUC = 0.986, Random Forest AUC = 0.983), and among the selected predictors, access sheath diameter and FANS-UAS use emerged alongside UPJ–PA as core determinants of additional intervention risk.

Although previous studies have defined specific cut-off points for the infundibulopelvic angle (IPA) and infundibular length (IL) as anatomical factors affecting surgical success—particularly in isolated lower pole stones—a single anatomical parameter reflecting global collecting system geometry has not been adequately discussed in the literature. IPA has been shown to be useful in predicting stone-free rates in lower pole stones; however, this predictive value appears limited to anatomically restricted subgroups^17–19. In contrast, the present study demonstrates that UPJ–PA has predictive value across all stone locations within the kidney, supporting its role as a more universally applicable anatomical marker.

The rate of additional intervention was significantly higher in patients treated with smaller-diameter ureteral access sheaths. While the literature suggests that larger sheaths may improve irrigation efficiency and visualization, reported findings remain heterogeneous^20,21.

Our results indicate that access sheath diameter plays a particularly important role in patients with a narrow UPJ–PA, where altered uretero-pelvic geometry is associated with impaired intrarenal drainage and fragment outflow, and anatomical constraints may limit scope deflection and fragment clearance. From an explainable XAI perspective, SHAP interaction analyses demonstrated that the protective effect of larger sheath diameter becomes more pronounced as UPJ–PA decreases, suggesting that enhanced irrigation and fragment evacuation may partially compensate for unfavorable anatomy. Similarly, FANS-UAS use emerged as a strong protective factor, ranking among the second most influential predictors in the model^22,23.

Further explainability analyses clarified the mechanistic contribution of these procedural factors. SHAP dependence and force plots demonstrated that the protective effects of larger access sheath diameter and suction-assisted access were particularly evident in patients with narrow UPJ–PA values. Directionality analyses showed that increasing sheath diameter and suction uniformly reduced individual patient risk across the predicted probability spectrum. These findings suggest that modifiable intraoperative strategies may partially offset fixed anatomical limitations and highlight the value of explainable AI outputs not only for post-hoc interpretation but also for personalized surgical planning and real-time decision support.

Several algorithms, including logistic regression and the ridge classifier, demonstrated excellent performance with AUC values exceeding 0.98. Explainability analyses using SHAP, LIME, model-based feature importance, and permutation importance consistently confirmed the dominant role of UPJ–PA in model predictions. Removal of UPJ–PA from the model resulted in a 0.038 (3.8%) decrease in AUC, underscoring its critical contribution to predictive performance. Robustness analyses further demonstrated that UPJ–PA maintained its ranking superiority even under substantial measurement perturbations, supporting the stability of this anatomical parameter. These findings are concordant with studies by Nedbal et al., who reported high stone-free rates and provided explainable AI–based insights into feature contributions at both individualized and global levels in f-URS outcomes⁶.

The proposed model can be integrated into the clinical workflow as follows: (1) preoperative CT imaging is reviewed and the UPJ–PA is measured (2–3 min); (2) stone characteristics and the planned equipment are entered into a web-based calculator; (3) the model provides the probability of requiring an additional intervention together with a confidence interval; and (4) for high-risk patients (predicted probability > 50%), clinicians may consider alternative approaches (e.g., percutaneous nephrolithotomy or preoperative stenting), enhanced patient counseling, or a modified surgical technique. Notably, a high predicted risk does not mandate a specific surgical approach (e.g., PCNL) but instead underscores the need for individualized decision making based on stone size, location, patient comorbidities, and surgeon expertise. The model is designed to support rather than replace clinical decision making.

This study has several strengths:

(1) the use of rigorous nested cross-validation to prevent overfitting;

(2) comprehensive performance evaluation encompassing discrimination, calibration, and clinical utility;

(3) the application of multiple explainability methods to ensure robust and consistent feature importance rankings;

(4) a dedicated robustness analysis confirming model stability under measurement uncertainty;

(5) clear clinical actionability enabled by the identification of explicit threshold values; and.

(6) transparent methodological reporting, allowing reproducibility.

Indeed, several studies by Nedbal et al., in which XAI –based algorithm training and validation were performed on large databases of patients treated for urolithiasis, support our findings by demonstrating near-perfect accuracy in predicting surgical outcomes and complications^7,24,25.

Among the limitations of this study are the restricted generalizability inherent to its retrospective, single-center design. In addition, the 110° UPJ–PA threshold was derived using a data-driven decile-based approach rather than a physiologically validated anatomical cut-off and should therefore be interpreted cautiously. UPJ–PA was not directly compared with established parameters such as IPA or IL and should be considered an exploratory anatomical metric, with a residual risk of inter-observer variability despite excellent agreement. Therefore, prospective studies including sufficient numbers of lower pole stones and direct, head-to-head comparisons between UPJ–PA and established lower pole anatomical parameters (e.g., IPA and infundibular length) are required to clarify their relative predictive value. Although stratified analyses were performed according to stone location, these findings should be interpreted as exploratory and hypothesis-generating, given the limited sample size within certain subgroups. Model performance may also be influenced by variability in CT imaging protocols and measurement precision. Moreover, all UPJ–PA measurements were performed by experienced urologists; thus, the reproducibility of this measurement in less experienced observers, radiologists, or automated algorithms remains uncertain, and noise-robustness analyses cannot fully substitute for real-world inter-observer variability. Furthermore, as FANS-UAS was introduced later in the study period, its association with outcomes should be interpreted cautiously. Although overfitting was addressed through nested validation and minimal training–test AUC differences (≈ 1%), the absence of external validation remains a significant limitation. Consequently, the reported thresholds and model performance warrant confirmation in multicenter prospective studies with independent validation cohorts.

Conclusion

This study demonstrates that machine learning and explainable XAI can reliably predict the need for additional intervention after f-URS with high accuracy and clinical interpretability. A UPJ–PA angle below 110° was associated with a sharply increased risk of additional intervention across all stone locations, establishing this parameter as a key anatomical determinant. Procedural factors such as access sheath diameter and FANS-UAS use also emerged as strong predictors. The developed models achieved AUC values above 98%, and their decision-making mechanisms were transparently elucidated through multiple explainability techniques. These findings support the potential of personalized risk stratification to improve patient selection, optimize surgical planning, and enhance resource utilization in routine f-URS practice.

Supplementary Information

Below is the link to the electronic supplementary material.

Author contributions

Conseption and design: F.Ç., B.K., H.K.; Data acquisition: F.Ç., B.K.;Data analysis and interpretation: F.Ç., H.K.; Drafting the manuscript: F.Ç., B.K., H.K.; Critical revision of the manuscript for scientific and factual content: F.Ç., B.K.; Statistical analysis: H.K.; All authors have reviewed the manuscript.

Funding/Support

This study received no funding.

Data availability

The data sets generated during and/or analyzed during the current study are available from the corresponding author upon reasonable request.

Declarations

Competing interests

The authors declare no competing interests.

Conflict of interest Disclosures

The authors declare that they have no competing interests.

Ethics Statement: This study received approval from the Non-Interventional Clinical Research Ethics Committee of Adıyaman University (decision date: September16, 2025, meeting number: 7, decision number: 2025/7 − 2).