Development of Explainable Machine Learning Models to Identify Patients at Risk for 1-Year Mortality and New Distant Metastases Postendoprosthetic Reconstruction for Lower Extremity Bone Tumors

Abstract

Background:

Accurate prediction of postoperative metastasis and mortality risks in patients undergoing lower-limb oncological resection and endoprosthetic reconstruction is essential for guiding adjuvant therapies and managing patient expectations. Current prediction methods are limited by variability in patient-specific factors. This study aims to develop and internally validate explainable machine learning (ML) models to predict the 1-year risk of new distant metastases and mortality in these patients.

Methods:

We performed a secondary analysis of data from the Prophylactic Antibiotic Regimens in Tumor Surgery trial, which included 604 patients. Candidate features were selected based on availability and clinical relevance and then narrowed using Least Absolute Shrinkage and Selection Operator (LASSO) regression and Boruta algorithms. Six ML classification algorithms were tuned and calibrated: logistic regression, support vector machines, random forest, Light Gradient Boosting Machine (LightGBM), eXtreme Gradient Boosting (XGBoost), and neural networks. Models were developed with and without including percent tumor necrosis due to its high missing data rate (>30%). Hyperparameters were tuned using Bayesian optimization. Internal validation was conducted using a 30% hold-out set. Model explainability was assessed using permutation-based feature importance and SHapley Additive exPlanations.

Results:

LightGBM was identified as the best-performing algorithm for both outcomes. For 1-year mortality prediction without percent necrosis, LightGBM achieved an area under the receiver operating characteristic curve (AUC-ROC) of 0.78 (95% confidence interval [CI] 0.70-0.86) during cross-validation and 0.72 on internal validation. For distant metastasis prediction, the LightGBM model without percent necrosis achieved an AUC-ROC of 0.77 (95% CI 0.71-0.84) during cross-validation and 0.77 on internal validation. Including percent necrosis did not significantly improve model performance. The top predictors identified were patient age, largest tumor dimension, and tumor stage.

Conclusions:

Explainable ML models can effectively predict the 1-year risk of mortality and new distant metastases in patients undergoing lower-limb oncological resection and endoprosthetic reconstruction. Further external validation and consideration of other data modalities are required before integrating these ML-driven risk assessments into routine clinical practice.

Level of Evidence:

Level II, Prognostic Study. See Instructions for Authors for a complete description of levels of evidence.

Introduction

The long bones of the lower limbs are frequent anatomical sites for primary and metastatic bone tumors¹. These tumors are typically treated with either amputations or limb-sparing resections combined with endoprosthetic reconstruction. Limb-sparing procedures are preferred in over 80% of cases due to their superior functional, cosmetic, and psychological outcomes^2,3. Given that bone tumors—particularly malignant neoplasms like osteosarcomas—are most commonly found in younger, more active patients^4,5, accurate prognostication of disease progression following limb-sparing surgeries is essential for guiding adjuvant therapies to reduce the risk of distant metastases and for managing patient expectations.

Current practices for predicting postsurgical metastasis and mortality risks rely on historical data and clinician expertise. However, these approaches are limited by variability in patient-specific demographic, tumor, and surgical characteristics. The increasing availability of data from patients undergoing limb-sparing resections, such as those from the Prophylactic Antibiotic Regimens in Tumor Surgery (PARITY) trial⁶, combined with advancements in machine learning (ML) techniques, offers new opportunities to provide accurate and individualized surgical outcome prognostication. ML-based risk stratification tools can recognize and utilize complex, nonlinear trends that clinicians may not readily identify with traditional statistical methods^7,8. While many “black-box” ML algorithms have historically faced skepticism in clinical applications due to their lack of transparency^9,10, modern explainability techniques—such as permutation-based feature importance and SHapley Additive exPlanations (SHAP)—are able to highlight major predictors driving ML predictions. This increase in model interpretability can foster clinician trust in these tools and contribute to current understandings of key indicators in surgical outcomes. To our knowledge, there are currently no explainable ML-based prognostic tools that can predict the risk of metastases or overall survival in patients undergoing lower extremity endoprosthetic reconstruction.

Therefore, the objective of this study was to develop and internally validate explainable ML models using the PARITY trial data set to predict the 1-year risk of new distant metastases and 1-year mortality following oncological resection and endoprosthetic reconstruction for bone tumors of the lower extremities. By building a transparent, ML-powered predictive tool, we aim to provide orthopaedic surgeons with a novel method for accurate metastasis and mortality risk stratification.

Materials and Methods

This study was conducted and reported in accordance with the TRIPOD + AI guidelines (Supplementary Table S1)¹¹.

Data Sources

We performed a secondary analysis of data from the PARITY trial⁶. PARITY is a blinded randomized controlled trial that compared 1-day versus 5-day antibiotic regimens in reducing surgical site infection rates in patients undergoing lower-limb oncological resection and complex endoprosthetic reconstruction for primary bone tumors or soft tissue sarcomas. The trial was performed across 48 clinical sites in 12 countries from January 1, 2013, to October 29, 2019. Detailed eligibility criteria and trial protocol have been described previously¹². Patients who died before developing new distant metastases were excluded from the distant metastasis modeling, as their outcome status could not be determined.

Target Outcomes

The goal of our modeling efforts was to predict the probability of 1-year mortality and 1-year distant metastasis for patients receiving lower-limb oncological resection and endoprosthetic reconstruction surgery. The predicted probabilities were transformed into a binary classification using a decision threshold to highlight patients who are at high risks for developing these outcomes.

Feature Selection

Candidate features were first selected from the PARITY data set based on availability (≤30% of missing data) and expert domain (see Supplementary Table S2 for the list of candidate features). The candidate feature set was further narrowed using a union of Least Absolute Shrinkage and Selection Operator (LASSO)–penalized logistic regression^13,14 and Boruta^14,15. Boruta is a random forest-based feature selection algorithm that selects relevant features by iteratively testing the importance of candidate features against random noise.

Modeling Including Tumor Necrosis

Previous literature has shown that tumor necrosis is a highly predictive factor for survival and distant metastases in specific bone tumor subtypes, such as osteosarcoma^16,17 and Ewing sarcoma¹⁸. However, percent necrosis was not included as a selected feature as it had over 30% of missing data in the data set. Based on the expert domain, we created additional sets of ML models that included percent necrosis (with imputed missing values) for clinicians who may wish to incorporate percent necrosis into the model predictions.

Data Preprocessing

The PARITY data set was split into a 70% derivation set and a 30% internal validation set for each outcome, stratified by the outcome classifications. The derivation set was further split into different training and test sets during 10-fold cross-validation for model tuning and selection.

Categorical features were encoded as integers, or through one-hot encoding¹⁹ if there was no clear ordinality. Missing data were imputed, separately for the train, test, and internal validation sets, using Multivariate Imputation by Chained Equations (MICE)²⁰. The number of MICE iterations was guided by the maximum percentage of missing data in the data set^14,21. If a ML algorithm has native missing data handling methods, those methods were trialed as alternatives to using MICE.

The presence of multicollinearity in the data set was assessed using variance inflation factors (VIFs)²², and principal component analysis (PCA) was trialed if VIF exceeds 5 for any of the selected features. The lowest number of principal components needed to explain at least 95% of the variance was kept.^14,23

Before the data set is entered into PCA and the ML algorithms, the normality of continuous features in the data set was assessed using the Shapiro-Wilk test²⁴ and through visual inspection of the feature's histogram²⁵. If the continuous features were normally distributed, then the continuous feature columns were Z-score normalized to center the feature columns around their means and to scale the columns according to their standard deviations^14,26. If the continuous features were skewed, then the feature columns were transformed using robust normalization, which centers the columns around the median and scales the columns based on their interquartile ranges^14,26.

Machine Learning Algorithms

We tuned 6 commonly used ML classification algorithms to identify the best performing model configurations. These included

1. Two classical algorithms: (1) penalized logistic regression and (2) support vector machine (SVM).

2. Three ensemble algorithms: (1) random decision forest, (2) Light Gradient-Boosting Machine²⁷ (LightGBM), and (3) eXtreme Gradient Boosting²⁸ (XGBoost).

3. Multilayer perceptron neural network consisting of 2, 3, or 4 hidden layers. The network structure is illustrated in Supplementary Figure S1. The neural networks were configured to behave deterministically for reproducibility and consistency of prediction outputs.

Hyperparameter Tuning

Hyperparameter tuning was conducted using Bayesian optimization²⁹ to minimize cross-entropy loss during cross-validation. Bayesian optimization starts with a random parameter search to gather initial performance data. It then uses these results to build a surrogate model that predicts the performance of different hyperparameter combinations. Using this model, it identifies promising parameters to try next and then uses the results of these new trials to update its surrogate model. This process repeats until a prespecified number of iterations is reached¹⁴. Because random hyperparameter search, which is more primitive than Bayesian optimization, can reliably identify the top 5% most performant hyperparameter combinations with 60 iterations²⁹, we aimed to perform Bayesian optimization at or beyond this budget for every algorithm tuned.

Calibration

After hyperparameter tuning, all models underwent calibration. Calibration refers to the process of adjusting a model’s predicted probability estimates to better reflect the true likelihood of the outcome in real-life. Three calibration approaches (Platt scaling³⁰, isotonic regression³¹, and no further calibration) were trialed. The calibration approach that resulted in the lowest average cross-entropy loss during cross-validation was selected¹⁴.

Threshold Tuning

Following calibration, each model underwent threshold tuning to maximize their binary classification performance. We first identified thresholds that best balance the sensitivity and specificity of the model (the “balanced” threshold). The Youden Index³², which is an aggregate assessment of the model's sensitivity and specificity, was calculated for every classification threshold between 0.01 and 0.99 at an interval of 0.01. The threshold with the highest average Youden Index across 10-fold cross-validation was selected as the balanced threshold.

Given that clinicians may prefer more conservative risk classifications during patient counseling or treatment planning, we also created a “high-sensitivity” threshold which keeps the average sensitivity closest to and above 0.80 during cross-validation¹⁴.

Model Evaluation and Selection

We followed a previously published framework to rank and select the most optimal ML model configurations³³. The classification performance of each tuned configuration was evaluated using area under the curve of the receiver operating characteristic curve (AUC-ROC), Youden Index, sensitivity, and specificity. The threshold-dependent metrics (i.e., Youden Index, sensitivity, and specificity) were calculated for both the balanced and high-sensitivity thresholds. Calibration performance was assessed using Brier score, calibration curve slope, and calibration curve intercept.

The Brier score represents how far the model’s predicted probabilities (e.g., 0.70) are compared with the patient’s actual outcome (i.e., whether the event occurred [1.00] or did not [0.00]). A lower Brier score means that the predictions are closer to the actual outcome and thus represents better model calibration. It ranges from 0 (perfect predictions) to 1 (poor predictions)³⁴. Calibration curves (examples of which are shown in Figs. 1-B and 1-D and 2-B and 2-D) plot the model’s predicted probabilities against actual outcome probabilities in the data set. In a perfectly calibrated model, the predicted probabilities would equal the actual probabilities, resulting in a diagonal line with a slope of 1 and an intercept of 0. Deviations of the slope and the intercept from these values indicate that the model may be overestimating or underestimating outcome probabilities, resulting in a worse calibrated model³⁵.

Fig. 1.

Plots showing the discriminative performance and calibration of the mortality and distant metastasis models without percent tumor necrosis during 10-fold cross-validation. Metrics shown on receiver operating characteristic curves are the area under the curve with 95% CIs. Metrics shown on calibration curves are Brier scores with 95% CIs. Fig. 1-A Averaged receiver operating characteristic curves for mortality modeling. Fig. 1-B Averaged calibration curves for mortality modeling. Fig. 1-C Averaged receiver operating characteristic curves for distant metastasis modeling. Fig. 1-D Averaged calibration curves for distant metastasis modeling. SVM = support vector machine, LightGBM = light gradient-boosting machine, MLP = multilayer perceptron, and XGBoost = eXtreme gradient boosting.

Fig. 2.

Plots showing the discriminative performance and calibration of the optimal LightGBM model configuration without percent tumor necrosis for mortality and distant metastasis prediction during internal validation. Metrics shown on receiver operating characteristic curves are the area under the curve. Metrics shown on calibration curves are Brier scores. Fig. 2-A Receiver operating characteristic curve for mortality prediction. Fig. 2-B Calibration curve for mortality prediction. Fig. 2-C Receiver operating characteristic curve for distant metastasis prediction. Fig. 2-D Calibration curve for distant metastasis prediction. LightGBM = light gradient-boosting machine.

All metrics were averaged across stratified 10-fold cross-validation and reported with 95% confidence intervals (CIs) during model selection.

For visualization of performance during cross-validation, we presented averaged ROC plots through vertical averaging³⁶ and averaged calibration curves. The averaged calibration curves were generated by aligning calibration curves from cross-validation to 5 common bin centers using linear interpolation.

Given that we expect high-sensitivity thresholds to be most commonly used in clinical settings, we selected the best performing model by choosing the model with the highest Youden Index under the high-sensitivity threshold. Calibration metrics, including Brier score and calibration curve slope and intercept, were used as tiebreakers.

Internal Validation

Once the optimal model configurations were identified through cross-validation, we retrained the configurations with the full derivation set. The retrained models were then internally validated using the internal validation set using the same metrics described above.

During internal validation, we also performed decision curve analysis (DCA) for the retrained models on the internal validation set to assess the models’ clinical usefulness in real-world settings^37,38. DCA relies on a metric called net benefit, which weighs the value of correctly identifying patients who truly need outcome interventions against the harm of incorrectly treating those who do not. DCA compares the predictive models’ net benefit against 2 alternative approaches—one where all patients are treated regardless of their outcome risks (i.e., “treat all”) and the other where all patients are not treated (i.e., “treat none”). The net benefit values of the 3 approaches are plotted across all possible model thresholds to create a “decision curve,” which allows clinicians to see at what risk levels (threshold probabilities) the model confers a greater advantage—by identifying more patients who truly need treatment and avoiding overtreating those who do not—compared with the “treat all” or “treat none” strategies³⁹.

Model Explainability

Following internal validation, the optimal model configuration was retrained with the full PARITY data set. To better understand how the final models make their predictions, we used permutation-based feature importance as a global explainability technique to assess which feature had the greatest impact on the model's cross-entropy loss when the feature is replaced with noise⁴⁰. We performed 100 permutations for each assessment.

To understand how the models make individual predictions, we used SHAP as a local explainability technique^41,42. Beeswarm plots were used to illustrate SHAP feature attributions for each patient in the PARITY data set. We also leveraged SHAP as an additional global explainability method accounting for feature interactions by averaging the SHAP values for each feature across all patients and comparing the resulting importance rankings with those produced by permutation importance⁴².

Clinical Risk Calculator

The final models were integrated into a web-based clinical risk calculator based on shiny. A SHAP waterfall plot was provided for each prediction to improve the models’ transparency.

SHAP requires access to the original training data set at runtime, which could represent a data security concern in an online environment. To facilitate SHAP implementation in the calculator, we generated a synthetic data set that mimics the characteristics of the PARITY data set using a Differentially Private Conditional Tabular Generative Adversarial Network (DP-CTGAN)⁴³.

Software

Descriptive statistics were generated using arsenal, and DCA was conducted using rmda in R. The modeling efforts were completed using scikit-learn, BorutaPy, lightgbm, xgboost, tensorflow, keras, scikit-optimize, eli5, shap, and smartnoise-synth in Python.

Results

We included 604 patients developing the mortality models. Within the data set, 18 patients experienced mortality before developing new distant metastases. Thus, 586 patients were used for developing the distant metastasis models. The characteristics of patients included for both outcomes are given in Table I. Characteristics of patients excluded from distant metastasis modeling are given in Supplemental Table S3.

TABLE I.

Characteristics of Patients from the Parity Data Set Included in Machine Learning Modeling

	Cohort Used for Mortality Modelinga			Cohort Used for Distant Metastasis Modeling
	Did Not Experience Mortality (N = 527)	Experienced Mortality (N = 77)	Total (N = 604)	Did Not Develop New Distant Metastasis (N = 438)	Developed New Distant Metastasis (N = 148)	Total (N = 586)
Age	34.2 (18.6, 58.0)	63.4 (42.4, 71.7)	39.7 (19.1, 60.5)	35.3 (19.0, 56.9)	53.0 (19.7, 67.3)	38.9 (19.0, 60.2)
Sex (%)
Male	319 (60.5)	42 (54.5)	361 (59.8)	264 (60.3)	88 (59.5)	352 (60.1)
Female	208 (39.5)	35 (45.5)	243 (40.2)	174 (39.7)	60 (40.5)	234 (39.9)
Race (%)b
Native American	12 (2.3)	3 (3.9)	15 (2.5)	9 (2.1)	5 (3.4)	14 (2.4)
Asian	102 (19.4)	11 (14.3)	113 (18.7)	90 (20.5)	20 (13.5)	110 (18.8)
Black	36 (6.8)	7 (9.1)	43 (7.1)	31 (7.1)	9 (6.1)	40 (6.8)
Hispanic	34 (6.5)	0 (0.0)	34 (5.6)	30 (6.8)	4 (2.7)	34 (5.8)
White	328 (62.2)	56 (72.7)	384 (63.6)	265 (60.5)	108 (73.0)	373 (63.7)
Other	15 (2.8)	0 (0.0)	15 (2.5)	13 (3.0)	2 (1.4)	15 (2.6)
Tumor stage (%)c
Stage I	61 (12.7)	2 (3.0)	63 (11.6)	60 (15.2)	1 (0.7)	61 (11.5)
Stage II	108 (22.5)	6 (9.1)	114 (20.9)	98 (24.7)	15 (11.1)	113 (21.3)
Stage III	218 (45.5)	36 (54.5)	254 (46.6)	175 (44.2)	70 (51.9)	245 (46.1)
Stage IV	92 (19.2)	22 (33.3)	114 (20.9)	63 (15.9)	49 (36.3)	112 (21.1)
Tumor type (%)
Osteosarcoma	254 (48.2)	20 (26.0)	274 (45.4)	201 (45.9)	67 (45.3)	268 (45.7)
Ewing sarcoma	34 (6.5)	5 (6.5)	39 (6.5)	31 (7.1)	7 (4.7)	38 (6.5)
Chondrosarcoma	84 (15.9)	15 (19.5)	99 (16.4)	78 (17.8)	18 (12.2)	96 (16.4)
Oligometastatic bone disease	35 (6.6)	21 (27.3)	56 (9.3)	25 (5.7)	27 (18.2)	52 (8.9)
Soft tissue sarcoma	51 (9.7)	11 (14.3)	62 (10.3)	39 (8.9)	21 (14.2)	60 (10.2)
Benign bone tumors	46 (8.7)	0 (0.0)	46 (7.6)	45 (10.3)	1 (0.7)	46 (7.8)
Other malignant bone tumors	23 (4.4)	5 (6.5)	28 (4.6)	19 (4.3)	7 (4.7)	26 (4.4)
Tumor necrosis (%)d	61.0 (20.0, 90.0)	40.0 (16.2, 67.5)	60.0 (20.0, 90.0)	70.0 (20.0, 90.0)	50.0 (20.0, 80.0)	60.0 (20.0, 90.0)
Largest tumor dimension (cm)e	8.3 (5.7, 12.0)	11.3 (7.0, 15.5)	9.0 (6.0, 12.6)	8.0 (5.0, 12.0)	10.0 (7.5, 13.2)	8.9 (6.0, 12.6)
Femoral involvement	431 (81.8)	67 (87.0)	498 (82.5)	352 (80.4)	132 (89.2)	484 (82.6)
Tibial involvement	100 (19.0)	9 (11.7)	109 (18.0)	89 (20.3)	16 (10.8)	105 (17.9)
Number of neoadjuvant chemotherapy cycles (%)f
No chemotherapy	266 (50.6)	48 (62.3)	314 (52.1)	229 (52.4)	74 (50.0)	303 (51.8)
1 cycle	8 (1.5)	0 (0.0)	8 (1.3)	5 (1.1)	3 (2.0)	8 (1.4)
2 cycles	81 (15.4)	4 (5.2)	85 (14.1)	61 (14.0)	23 (15.5)	84 (14.4)
3 cycles	56 (10.6)	4 (5.2)	60 (10.0)	44 (10.1)	15 (10.1)	59 (10.1)
4 cycles	37 (7.0)	4 (5.2)	41 (6.8)	30 (6.9)	10 (6.8)	40 (6.8)
5 cycles	23 (4.4)	4 (5.2)	27 (4.5)	21 (4.8)	5 (3.4)	26 (4.4)
6 cycles	29 (5.5)	5 (6.5)	34 (5.6)	25 (5.7)	9 (6.1)	34 (5.8)
7 cycles	7 (1.3)	1 (1.3)	8 (1.3)	6 (1.4)	2 (1.4)	8 (1.4)
>7 cycles	19 (3.6)	7 (9.1)	26 (4.3)	16 (3.7)	7 (4.7)	23 (3.9)
Cumulative neoadjuvant radiotherapy dose (%)g
No radiotherapy	515 (97.7)	67 (89.3)	582 (96.7)	427 (97.5)	139 (95.2)	566 (96.9)
≤20 Gy	4 (0.8)	1 (1.3)	5 (0.8)	3 (0.7)	2 (1.4)	5 (0.9)
21-50 Gy	5 (0.9)	7 (9.3)	12 (2.0)	5 (1.1)	5 (3.4)	10 (1.7)
51-60 Gy	2 (0.4)	0 (0.0)	2 (0.3)	2 (0.5)	0 (0.0)	2 (0.3)
>60 Gy	1 (0.2)	0 (0.0)	1 (0.2)	1 (0.2)	0 (0.0)	1 (0.2)
Total operative time (hours)	4.5 (3.5, 6.2)	4.5 (3.3, 6.5)	4.5 (3.4, 6.3)	4.5 (3.4, 6.0)	4.7 (3.4, 6.7)	4.5 (3.4, 6.3)
Overall surgical margin (%)h
Negative	490 (93.7)	57 (76.0)	547 (91.5)	407 (93.8)	127 (87.0)	534 (92.1)
Microscopically positive	22 (4.2)	11 (14.7)	33 (5.5)	20 (4.6)	9 (6.2)	29 (5.0)
Grossly positive	11 (2.1)	7 (9.3)	18 (3.0)	7 (1.6)	10 (6.8)	17 (2.9)

Categorical/binary characteristics are presented as N (%), and continuous characteristics are presented as median (interquartile range).

^aThe entire PARITY data set was used for mortality modeling, while patients who experienced mortality before developing new distant metastases were removed from distant metastasis modeling due to unknown outcomes.

^bTwo patients had missing race data in the nonmortality group for mortality modeling. Two patients had missing race data in the nonmetastasis group for distant metastasis modeling.

^cForty-eight patients had missing tumor stage data in the nonmortality group, and 11 patients had missing tumor stage data in the mortality group for mortality modeling. Forty-two patients had missing tumor stage data in the nonmetastasis group, and 13 patients had missing tumor stage data in the metastasis group for distant metastasis modeling.

^dTwo hundred one patients had missing percent tumor necrosis data in the nonmortality group, and 39 patients had missing percent tumor necrosis data in the mortality group for mortality modeling. One hundred seventy-eight patients had missing percent tumor necrosis data in the nonmetastasis group, and 52 patients had missing percent tumor necrosis data in the metastasis group for distant metastasis modeling.

^eOne hundred fifty-two patients had missing tumor dimension data in the nonmortality group, and 23 patients had missing tumor dimension data in the mortality group for mortality modeling. One hundred thirty-seven patients had missing tumor dimension data in the nonmetastasis group, and 32 patients had tumor dimension data in the metastasis group for distant metastasis modeling.

^fOne patient had missing neoadjuvant chemotherapy data in the nonmortality group for mortality modeling. One patient had missing neoadjuvant chemotherapy data in the nonmetastasis group for distant metastasis modeling.

^gTwo patients had missing neoadjuvant radiotherapy data in the mortality group for mortality modeling. Two patients had missing neoadjuvant radiotherapy data in the metastasis group for distant metastasis modeling.

^hFour patients had missing surgical margin data in the nonmortality group, and 2 patients had missing surgical margin data in the mortality group for mortality modeling. Four patients had missing surgical margin data in the nonmetastasis group, and 2 patients had missing surgical margin data in the metastasis group for distant metastasis modeling.

Performance of Mortality Models During Cross-Validation

The model pipeline, which includes robust scaling and trial of PCA, is illustrated in Supplemental Figure S2. Eleven features were included in the modeling (see Supplemental Table S4). The best hyperparameter combinations and threshold values identified for each ML algorithm are tabulated in Supplemental Tables S5-S6.

The performance metrics of the tuned and calibrated models with and without the percent necrosis feature are given in Table II and illustrated in Figs. 1-A and 1-B, and Supplementary Figs. S3-S4. LightGBM was identified as the best ML algorithm for both models including and excluding percent necrosis. Modeling including percent necrosis did not yield substantially improved performance compared with modeling without the additional feature.

TABLE II.

Average Performance of Machine Learning Algorithms for Predicting 1-Year Mortality During 10-Fold Cross-Validation

Machine Learning Algorithm	Threshold-Independent Metrics				Threshold-Dependent Metrics (Balanced Threshold)			Threshold-Dependent Metrics (High-Sensitivity Threshold)
	AUC-ROC	Brier Score	Calibration Curve Slope	Calibration Curve Intercept	Youden Index	Sensitivity	Specificity	Youden Index	Sensitivity	Specificity
Models without percentage necrosis
Penalized logistic regression	0.74 (0.66 to 0.83)	0.10 (0.09 to 0.11)	1.64 (0.74 to 2.54)	−0.08 (−0.19 to 0.04)	0.39 (0.25 to 0.53)	0.65 (0.53 to 0.77)	0.74 (0.68 to 0.80)	0.29 (0.16 to 0.42)	0.82 (0.71 to 0.92)	0.48 (0.42 to 0.53)
Support vector machine	0.74 (0.67 to 0.82)	0.10 (0.09 to 0.11)	1.42 (0.62 to 2.22)	−0.06 (−0.17 to 0.05)	0.40 (0.24 to 0.57)	0.66 (0.54 to 0.79)	0.74 (0.67 to 0.81)	0.26 (0.13 to 0.39)	0.82 (0.74 to 0.90)	0.44 (0.36 to 0.52)
Random decision forest	0.75 (0.66 to 0.84)	0.10 (0.09 to 0.11)	0.91 (0.40 to 1.43)	0.02 (-0.06 to 0.09)	0.43 (0.23 to 0.63)	0.63 (0.46 to 0.81)	0.80 (0.75 to 0.85)	0.33 (0.17 to 0.49)	0.82 (0.71 to 0.93)	0.52 (0.45 to 0.58)
LightGBM	0.78 (0.70 to 0.86)	0.10 (0.09 to 0.11)	1.04 (0.20 to 1.88)	0.01 (-0.12 to 0.14)	0.47 (0.32 to 0.62)	0.73 (0.60 to 0.86)	0.74 (0.70 to 0.79)	0.35 (0.23 to 0.46)	0.84 (0.74 to 0.93)	0.51 (0.46 to 0.56)
XGBoost	0.75 (0.66 to 0.84)	0.10 (0.09 to 0.11)	0.97 (0.24 to 1.69)	0.00 (-0.16 to 0.16)	0.42 (0.27 to 0.58)	0.66 (0.52 to 0.81)	0.76 (0.70 to 0.82)	0.25 (0.16 to 0.33)	0.86 (0.76 to 0.95)	0.39 (0.32 to 0.46)
MLP neural network	0.72 (0.62 to 0.82)	0.11 (0.09 to 0.12)	2.19 (0.47 to 3.91)	−0.13 (−0.32 to 0.07)	0.30 (0.15 to 0.46)	0.54 (0.38 to 0.70)	0.76 (0.66 to 0.87)	0.25 (0.07 to 0.43)	0.83 (0.70 to 0.96)	0.42 (0.29 to 0.54)
Models with percentage necrosis
Penalized logistic regression	0.75 (0.66 to 0.83)	0.10 (0.09 to 0.11)	1.15 (0.35 to 1.96)	−0.03 (−0.14 to 0.07)	0.43 (0.28 to 0.57)	0.61 (0.49 to 0.73)	0.81 (0.77 to 0.86)	0.28 (0.13 to 0.43)	0.81 (0.68 to 0.94)	0.46 (0.40 to 0.53)
Support vector machine	0.76 (0.70 to 0.82)	0.10 (0.09 to 0.11)	0.98 (0.30 to 1.65)	0.01 (−0.09 to 0.11)	0.41 (0.28 to 0.53)	0.74 (0.64 to 0.84)	0.67 (0.60 to 0.73)	0.38 (0.29 to 0.48)	0.83 (0.76 to 0.90)	0.55 (0.50 to 0.60)
Random decision forest	0.73 (0.64 to 0.83)	0.10 (0.09 to 0.11)	1.24 (0.30 to 2.18)	−0.05 (−0.19 to 0.09)	0.35 (0.17 to 0.54)	0.67 (0.52 to 0.82)	0.68 (0.63 to 0.74)	0.26 (0.15 to 0.37)	0.84 (0.71 to 0.96)	0.42 (0.39 to 0.46)
LightGBM	0.78 (0.69 to 0.86)	0.10 (0.09 to 0.11)	1.46 (0.87 to 2.05)	−0.06 (−0.14 to 0.02)	0.43 (0.26 to 0.61)	0.72 (0.58 to 0.87)	0.71 (0.64 to 0.78)	0.42 (0.29 to 0.55)	0.86 (0.76 to 0.95)	0.57 (0.50 to 0.63)
XGBoost	0.78 (0.69 to 0.87)	0.10 (0.09 to 0.10)	1.56 (0.59 to 2.54)	−0.07 (−0.20 to 0.06)	0.43 (0.25 to 0.61)	0.70 (0.54 to 0.86)	0.73 (0.67 to 0.78)	0.35 (0.24 to 0.46)	0.86 (0.76 to 0.95)	0.49 (0.44 to 0.54)
MLP neural network	0.72 (0.62 to 0.83)	0.10 (0.09 to 0.12)	0.62 (−0.11 to 1.35)	0.06 (−0.06 to 0.18)	0.32 (0.13 to 0.50)	0.78 (0.62 to 0.95)	0.53 (0.46 to 0.60)	0.20 (0.05 to 0.35)	0.84 (0.70 to 0.98)	0.36 (0.24 to 0.48)

Performance metrics are shown as mean (95% confidence interval).

AUC-ROC = area under the curve of the receiver operating characteristic curve, LightGBM = light gradient-boosting machine, MLP = multilayer perceptron, and XGBoost eXtreme = gradient boosting.

Performance of Distant Metastasis Models During Cross-Validation

The model pipeline for predicting distant metastasis is the same as the pipeline for mortality modeling, which is illustrated in Supplemental Figure S2. Eleven features were included in the modeling (see Supplemental Table S7). The best hyperparameter combinations and threshold values identified for each ML algorithm are tabulated in Supplemental Tables S7-S9.

The performance metrics of the tuned and calibrated models with and without the percent necrosis feature are given in Table III and illustrated in Figs. 1-C and 1-D, and Supplementary Figs. S5-S6. Similar to the mortality modeling, LightGBM was identified as the best ML algorithm for both models including and excluding percent necrosis. Modeling including percent necrosis did not yield substantially improved performance compared with modeling without the additional feature.

TABLE III.

Average Performance of Machine Learning Algorithms for Predicting 1-Year New Distant Metastasis During 10-Fold Cross-Validation

Machine Learning Algorithm	Threshold-Independent Metrics				Threshold-Dependent Metrics (Balanced Threshold)			Threshold-Dependent Metrics (High-Sensitivity Threshold)
	AUC-ROC	Brier Score	Calibration Curve Slope	Calibration Curve Intercept	Youden Index	Sensitivity	Specificity	Youden Index	Sensitivity	Specificity
Models without percentage necrosis
Penalized logistic regression	0.73 (0.65 to 0.81)	0.17 (0.15 to 0.18)	0.99 (0.61 to 1.37)	−0.00 (−0.09 to 0.09)	0.34 (0.26 to 0.43)	0.89 (0.81 to 0.97)	0.45 (0.40 to 0.51)	0.33 (0.21 to 0.44)	0.83 (0.71 to 0.95)	0.49 (0.43 to 0.56)
Support vector machine	0.74 (0.67 to 0.81)	0.16 (0.15 to 0.18)	1.12 (0.59 to 1.65)	−0.05 (−0.17 to 0.07)	0.38 (0.27 to 0.49)	0.79 (0.68 to 0.90)	0.59 (0.53 to 0.66)	0.37 (0.25 to 0.50)	0.83 (0.71 to 0.94)	0.55 (0.48 to 0.61)
Random decision forest	0.74 (0.66 to 0.82)	0.17 (0.15 to 0.18)	0.97 (0.51 to 1.42)	−0.00 (−0.11 to 0.11)	0.36 (0.21 to 0.51)	0.67 (0.53 to 0.81)	0.69 (0.63 to 0.75)	0.34 (0.23 to 0.45)	0.81 (0.70 to 0.93)	0.53 (0.47 to 0.58)
LightGBM	0.77 (0.71 to 0.84)	0.16 (0.14 to 0.17)	0.87 (0.52 to 1.23)	0.04 (−0.05 to 0.13)	0.45 (0.34 to 0.56)	0.72 (0.61 to 0.82)	0.73 (0.68 to 0.78)	0.42 (0.29 to 0.55)	0.83 (0.73 to 0.92)	0.59 (0.53 to 0.66)
XGBoost	0.76 (0.69 to 0.83)	0.17 (0.15 to 0.18)	0.64 (-0.02 to 1.30)	0.12 (−0.06 to 0.30)	0.44 (0.31 to 0.56)	0.77 (0.63 to 0.90)	0.67 (0.62 to 0.72)	0.42 (0.34 to 0.51)	0.82 (0.70 to 0.93)	0.61 (0.54 to 0.67)
MLP neural network	0.77 (0.70 to 0.83)	0.16 (0.14 to 0.18)	0.83 (0.47 to 1.19)	0.02 (−0.07 to 0.11)	0.36 (0.21 to 0.52)	0.54 (0.39 to 0.69)	0.82 (0.77 to 0.87)	0.31 (0.18 to 0.44)	0.82 (0.69 to 0.96)	0.48 (0.42 to 0.54)
Models with percentage necrosis
Penalized logistic regression	0.73 (0.65 to 0.81)	0.17 (0.15 to 0.18)	0.99 (0.56 to 1.42)	0.01 (−0.12 to 0.13)	0.34 (0.20 to 0.48)	0.62 (0.49 to 0.76)	0.72 (0.66 to 0.78)	0.31 (0.21 to 0.42)	0.83 (0.72 to 0.93)	0.49 (0.42 to 0.56)
Support vector machine	0.73 (0.64 to 0.82)	0.17 (0.15 to 0.18)	0.73 (0.26 to 1.21)	0.07 (−0.06 to 0.21)	0.34 (0.20 to 0.49)	0.72 (0.56 to 0.87)	0.63 (0.57 to 0.69)	0.32 (0.18 to 0.46)	0.80 (0.68 to 0.93)	0.52 (0.45 to 0.58)
Random decision forest	0.74 (0.66 to 0.82)	0.17 (0.15 to 0.18)	1.06 (0.70 to 1.41)	−0.04 (−0.12 to 0.05)	0.35 (0.26 to 0.45)	0.86 (0.79 to 0.93)	0.49 (0.41 to 0.57)	0.31 (0.22 to 0.41)	0.81 (0.74 to 0.89)	0.50 (0.41 to 0.59)
LightGBM	0.79 (0.73 to 0.86)	0.15 (0.14 to 0.17)	1.05 (0.63 to 1.47)	−0.01 (−0.12 to 0.10)	0.45 (0.34 to 0.55)	0.79 (0.72 to 0.87)	0.65 (0.58 to 0.72)	0.44 (0.35 to 0.54)	0.81 (0.74 to 0.89)	0.63 (0.56 to 0.70)
XGBoost	0.79 (0.72 to 0.85)	0.16 (0.14 to 0.17)	0.97 (0.40 to 1.54)	0.01 (−0.14 to 0.16)	0.45 (0.34 to 0.56)	0.79 (0.69 to 0.88)	0.67 (0.60 to 0.74)	0.39 (0.28 to 0.50)	0.81 (0.72 to 0.89)	0.58 (0.51 to 0.65)
MLP neural network	0.75 (0.68 to 0.83)	0.16 (0.15 to 0.18)	0.97 (0.37 to 1.57)	0.00 (−0.14 to 0.15)	0.35 (0.23 to 0.47)	0.66 (0.49 to 0.83)	0.69 (0.61 to 0.77)	0.32 (0.22 to 0.42)	0.80 (0.67 to 0.92)	0.53 (0.46 to 0.59)

Performance metrics are shown as mean (95% confidence interval).

AUC-ROC = area under the curve of the receiver operating characteristic curve, LightGBM = light gradient-boosting machine, and XGBoost eXtreme = gradient boosting, and MLP = multilayer perceptron.

Performance of the Optimal Model Configurations During Internal Validation

Internal validation performance metrics for the mortality and distant metastasis LightGBM model configurations, with and without the percent necrosis feature, are tabulated in Table IV and illustrated in Fig. 2 and Supplementary Figs. S7-S12. Compared with performance during cross-validation, the sensitivity of the mortality model without percent necrosis is decreased at both the balanced and high-sensitivity threshold. By contrast, the sensitivity and specificity of the model with percent necrosis is similar, if not higher, during internal validation compared with cross-validation. The opposite is seen in the distant metastasis models, where the sensitivity of the model is preserved in the model without percent necrosis, but the sensitivity is decreased in the model including percent necrosis.

TABLE IV.

Internal Validation Performance of Best-Performing Machine Learning Model Configurations

Machine Learning Algorithm	Threshold-Independent Metrics				Threshold-Dependent Metrics (Balanced Threshold)			Threshold-Dependent Metrics (High-Sensitivity Threshold)
	AUC-ROC	Brier Score	Calibration Curve Slope	Calibration Curve Intercept	Youden Index	Sensitivity	Specificity	Youden Index	Sensitivity	Specificity
1-year mortality prediction
LightGBM (without percent necrosis)	0.72	0.10	1.17	-0.02	0.28	0.52	0.76	0.26	0.74	0.52
LightGBM (with percent necrosis)	0.83	0.09	1.07	0.11	0.41	0.65	0.76	0.45	0.83	0.62
1-year distant metastasis prediction
LightGBM (without percent necrosis)	0.77	0.15	0.71	0.06	0.36	0.68	0.68	0.36	0.84	0.52
LightGBM (with percent necrosis)	0.73	0.16	0.77	0.07	0.35	0.73	0.62	0.33	0.73	0.60

AUC-ROC = area under the curve of the receiver operating characteristic curve, and LightGBM = light gradient-boosting machine.

Based on DCA (illustrated in Fig. 3 and Supplementary Figs. S9 and S12), our models generally confer greater net benefit than the “treat all” or “treat none” approaches at the balanced thresholds. However, at the high-sensitivity threshold for the mortality model without percent necrosis, the model’s net benefit was comparable with “treat all,” which reflects the poor trade-off between sensitivity and specificity at that threshold.

Fig. 3.

Decision curve analysis plots showing the change in net benefit of the optimal LightGBM model configuration without percent tumor necrosis for mortality and distant metastasis prediction across different decision thresholds during internal validation, as compared with the “treat all” and “treat none” strategies. Fig. 3-A Decision curve analysis for mortality prediction. Fig. 3-B Decision curve analysis for distant metastasis prediction. LightGBM = light gradient-boosting machine.

Global and Local Feature Importance

The permutation importance and SHAP results based on the final models are illustrated in Figs. 4 and 5, and Supplementary Figs. S13-S18. Global explainability through both permutation importance and SHAP consistently showed age, largest tumor dimension, and tumor stage as the top 3 most important features for both mortality and distant metastasis models, both with and without percent necrosis. The SHAP beeswarm plot (Figs. 5-C and 5-D) showed good separability in these top 3 features between high and low values. In general, higher age values, larger tumor dimensions, and more advanced AJCC stages are associated with increased predicted probabilities of 1-year mortality and distant metastasis.

Fig. 4.

Permutation importance plots illustrating the global feature importance for the final LightGBM models without percent tumor necrosis for mortality and distant metastasis predictions across the training data sets. Fig. 4-A Permutation importance plot for the mortality LightGBM model. Fig. 4-B Permutation importance plot for the distant metastasis LightGBM model. LightGBM = light gradient-boosting machine.

Fig. 5.

SHAP plots illustrating summaries of the local and global feature importance for the final LightGBM models without percent tumor necrosis for mortality and distant metastasis predictions across the training data sets. Local feature importance is represented using beeswarm plots. On the beeswarm plots, each dot represents a patient's data. Dots to the right for a particular feature row indicate that the feature increases the patients' predicted probability, and vice versa. The color of the dot shows whether a feature's value is high or low for that patient. Clusters of dots on the beeswarm plot can demonstrate trends in both the direction and magnitude of features' impact on patient risk predictions across the entire training data set. Fig. 5-A SHAP global feature importance plot for the mortality LightGBM model. Fig. 5-B SHAP global feature importance plot for the distant metastasis LightGBM model. Fig. 5-C SHAP beeswarm plot for the mortality LightGBM model. Fig. 5-D SHAP beeswarm plot for the distant metastasis LightGBM model. LightGBM = light gradient-boosting machine, and SHAP = SHapley Additive exPlanations.

Model Deployment

The final models were incorporated into shiny-based clinical risk calculators with SHAP for demonstration purposes. The calculators are available at https://parity-ml.shinyapps.io/distmet-estimator/ and https://parity-ml.shinyapps.io/mortality-estimator/.

Examples of SHAP waterfall plots produced based on the synthetic data set (using the same implementation as the online calculators) are shown in Fig. 6. Our calculators automatically produce an associated waterfall plot for every model prediction. SHAP waterfall plots provide a visual explanation of how individual features contribute to a probability prediction for an individual patient. The plot starts with the model’s base value (labelled E[f(X)]) at the bottom of the plot. This base value represents the average probability prediction across all samples in the training/synthetic data set and serves as the same starting point for every model prediction.

Fig. 6.

Examples of SHAP waterfall plots for individual mortality predictions using the final LightGBM models without percent tumor necrosis. These plots are generated using the same procedure as the implementation on the clinical risk calculators. Note that the feature values shown on the y-axis may not appear intuitive as they have been transformed by the robust scaler before SHAP applications. Fig. 6-A Waterfall plot for a hypothetical, low mortality risk patient case with nonmetastatic benign bone tumors, with negative resection margins. Fig. 6-B Waterfall plot for a hypothetical, high mortality risk patient case with stage 4 metastatic bone disease, with microscopically positive resection margins. In both cases, the patients’ predictions start at the SHAP base value (the average predicted mortality risk in the training/synthetic data set, denoted E[f(X)]) of 0.08. Each feature then either increases or decreases this probability to reach the final predicted probability produced by the model, denoted f(x) at the top of the plots. LightGBM = light gradient-boosting machine, and SHAP = SHapley Additive exPlanations.

Moving up the plot, each individual feature sequentially adds (shown by the red arrows) or subtracts (shown by the blue arrows) probabilities from the base value. As an example in Fig. 6, having AJCC stage 1 decreased the predicted probability of 1-year mortality by 0.01 in the first patient, while having AJCC stage 4 increased the predicted probability by 0.10 in the second patient. These positive and negative feature contributions collectively sum to the final predicted probability, denoted f(x), at the top of the waterfall plot.

Discussion

In this study, we developed and internally validated explainable machine learning (ML) models to predict the 1-year risk of mortality and new distant metastases in patients undergoing lower-limb oncological resections and endoprosthetic reconstructions. Utilizing data from the PARITY trial—a comprehensive, multicenter data set—we demonstrated that ML algorithms, particularly LightGBM, may effectively provide individualized mortality and metastasis risk assessments using a combination of patient demographics, tumor characteristics, and operative parameters.

Accurate predictions of 1-year mortality and distant metastasis enable clinicians to identify high-risk patients early in the treatment process and adjust their surveillance protocols accordingly. For example, surgeons can schedule more frequent follow-up visits or leverage more advanced imaging techniques (e.g., whole-body MRIs⁴⁴) in high-risk cases to monitor for metastases. Starting these patients on more aggressive postoperative chemotherapy and radiotherapy may also reduce local recurrence rates and improve overall survival^45-48. While some predictive features, such as surgical margins or operative time, cannot be definitively established preoperatively, our model may still enable surgeons to assess the impact of different anticipated operative complexities on patient outcomes, which could aid in preoperative patient counseling and education, as well as proactive decision-making regarding postoperative care.

While previous studies have developed similar ML-based tools, such as SORG-MLAs^49,50, these models largely function as “black boxes” with limited insight into why particular features drive their predictions, which can hinder clinical acceptance. By contrast, our study focuses on explainable algorithms—incorporating permutation-based feature importance and SHAP analyses—to illuminate how each patient’s data contribute to their estimated risk. This transparency is crucial for building trust among surgeons and oncologists, who must understand a model's rationale before applying it in practice.

In addition to our emphasis on interpretability, our study is also strengthened by its use of the PARITY data set—one of the largest prospective cohorts of patients undergoing lower-limb oncological resection and endoprosthetic reconstruction⁵¹. The multicentered nature of the data set can ensure our models capture a diverse patient population and reflect real-world clinical variability, which enhances the potential generalizability of our findings.

Study Limitations

Despite the promising results, our study has several limitations. The first is our model’s reliance on data from predominantly White and Asian patients, and the inclusion of race variables by Boruta and LASSO. Because of limited sample sizes, with non-White, non-Asian racial subgroups having fewer than 20 patients in the internal validation cohorts, we were unable to assess model fairness based on different racial subgroups. We acknowledge that the inclusion of race in predictive healthcare modeling is contentious^52-54; however, we note that these variables did not emerge as major contributors to model predictions as indicated by our explainability techniques. We aim to further assess model fairness characteristics in future external validation studies using a more diverse set of patient data.

Another limitation is our focus on 1-year endpoints, which may not capture longer-term disease progression or survival trends. Although the 1-year timeframe is clinically relevant for guiding initial postoperative management and was defined by the PARITY trial’s design, we acknowledge that mortality and distant metastases can occur well beyond this period. Future extensions of our model could explore multiyear endpoints to provide more comprehensive assessments of patient outcomes.

Finally, model overfitting on the PARITY data set remains a substantial concern despite cross-validation and internal validation. External validation using an unrelated, diverse data set would be needed before the models are introduced into clinical environments.

A potential avenue to improving the performance of our model involves incorporating imaging data to create multimodal ML models. Preoperative and postoperative MRI images, for instance, may allow ML models to identify features such as the presence of critical neurovascular structures or perilesional edema, which could indicate close or positive margins implicating higher chances of recurrence and metastasis⁵⁵. While imaging data are usually more difficult to collect and analyze, the inclusion of these data has been associated with improved classification performance in previous ML modeling efforts involving patients with sarcoma⁵⁶.

Conclusion

Our study provides a proof-of-concept for using explainable ML models to predict the risk of 1-year mortality and new distant metastases in patients undergoing lower limb oncological resections and endoprosthetic reconstructions. By identifying high-risk surgical candidates, clinicians can implement early interventions and more frequent surveillance to improve patient outcomes and quality of life. While our models show promise, further external validation and consideration of other data modalities are needed to integrate ML-driven risk assessments into routine clinical care.

Data Availability Statement

Supporting data is not directly available from the authors of this study. To request access to deidentified PARITY trial data, contact the corresponding author(s) of the original trial publication. Statistical codes used in this study are available upon reasonable request if the requesting individual has access to the PARITY data set.

Appendix

Supporting material provided by the author is posted with the online version of this article as a data supplement at jbjs.org (http://links.lww.com/JBJSOA/A823). This content has not been copyedited or verified.