Building and validating machine learning models to predict appendiceal perforation during conservative treatment of fecalith-associated appendicitis: a 20-algorithm multicenter retrospective analysis

Abstract

Fecalith-associated appendicitis presents unique challenges in conservative management due to increased perforation risk. Early identification of patients at high risk for appendiceal perforation during conservative treatment remains crucial for optimal clinical decision-making. To develop and validate machine learning-based prediction models for appendiceal perforation risk assessment during conservative treatment of fecalith-associated appendicitis. This retrospective cohort study analyzed 1247 patients with fecalith-associated appendicitis who underwent initial conservative treatment across four tertiary care centers between January 2018 and December 2023. The study design and reporting adhere to the TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis) guidelines for model development and external validation. Twenty machine learning algorithms were systematically trained and validated using clinical, laboratory, and imaging parameters. LASSO regularization identified eight optimal predictive features from 34 candidate variables. The final nominated model for clinical deployment is the Gradient Boosting classifier, trained on all eight LASSO-selected features. Primary outcome was appendiceal perforation within 72 h of conservative treatment initiation. Of 1247 patients, 186 (14.9%) developed appendiceal perforation during conservative treatment. The ensemble gradient boosting model achieved the highest performance with an AUC of 0.892 (95% CI 0.871–0.913), sensitivity of 84.4% (95% CI 79.2–89.6%), and specificity of 81.7% (95% CI 77.8–85.6%). External validation in an independent cohort (n = 225; The People’s Hospital of Sishui, January 2023–December 2024) confirmed model generalisability: AUC = 0.909 (95% CI 0.859–0.951), sensitivity = 73.7%, specificity = 93.0%, PPV = 68.3%, and NPV = 94.6%. SHAP analysis identified key predictive features: fecalith size (importance: 0.234), C-reactive protein (0.186), white blood cell count (0.162), appendiceal wall thickness (0.143), and patient age (0.121). Risk stratification classified patients into low-risk (58.9%, 3.8% perforation rate), moderate-risk (31.9%, 24.6% perforation rate), and high-risk (9.2%, 71.3% perforation rate) categories. Decision curve analysis demonstrated significant clinical utility with net benefit of 0.08 at 15% threshold probability. Machine learning models, particularly ensemble gradient boosting methods, demonstrate excellent accuracy in predicting appendiceal perforation risk during conservative treatment of fecalith-associated appendicitis, with performance confirmed in an external validation cohort. These validated models provide clinically actionable risk stratification that may assist in treatment decision-making and patient monitoring strategies, potentially preventing unnecessary surgeries while identifying high-risk patients requiring enhanced surveillance or early surgical intervention.

Introduction

Acute appendicitis remains the most frequent surgical emergency worldwide, affecting approximately 7–8% of the population during their lifetime, with an annual incidence of 96.5–100 cases per 100,000 individuals in developed countrie^1,2. Historically, immediate surgical appendectomy has been the definitive treatment since McBurney’s first description in 1889³. Over the past two decades, however, mounting evidence has supported conservative antibiotic therapy as a viable alternative for carefully selected patients with uncomplicated acute appendicitis^4,5. Landmark studies including the APPAC trial⁶ and the NOTA trial⁷ demonstrated that antibiotic therapy achieves success rates of 70–95% in uncomplicated appendicitis, leading to widespread adoption of conservative management protocols. Within this evolving paradigm, the presence of an appendicolith has emerged as a critical determinant of treatment success.

The appendicolith, also known as a fecalith or appendiceal concretion, is identified in approximately 10–20% of all appendicitis cases^8,9. Its pathophysiological significance lies in the mechanical obstruction it creates within the appendiceal lumen, leading to increased intraluminal pressure, compromised vascular drainage, and progressive bacterial overgrowth — a cascade that substantially elevates the risk of appendiceal perforation even during appropriately administered antibiotic therapy^10–12. Current evidence confirms that fecalith-associated appendicitis represents a clinically distinct and more complicated disease phenotype¹³, with higher treatment failure rates¹⁴, stronger associations with perforation¹⁵, and specific imaging characteristics that differentiate it from uncomplicated cases¹⁶.

Current clinical decision-making tools, including the Alvarado score¹⁷ and the Appendicitis Inflammatory Response (AIR) score¹⁸, were primarily developed in general appendicitis populations without specific consideration of appendicolith presence, leaving their predictive accuracy for perforation risk in fecalith-associated cases suboptimal (AUC 0.69–0.83)^19,20. The multifactorial nature of perforation risk — encompassing patient-specific factors, fecalith characteristics, inflammatory parameters, and imaging findings — involves complex non-linear interactions that conventional statistical approaches are inherently limited in capturing^21,22. Machine learning methodologies offer distinct advantages in this context, as algorithms such as XGBoost²³ and Random Forests²⁴ can automatically identify high-order interactions within multidimensional datasets. While machine learning has shown promise in emergency surgery broadly^25–27, specific applications to fecalith-associated appendicitis remain absent from the literature²⁸.

This evidence gap carries direct clinical consequences: patients who perforate during conservative treatment face significantly increased morbidity^29,30, while unnecessary surgery incurs avoidable operative risks and healthcare costs³¹. Accurate, individualized risk stratification is therefore essential to guide treatment decisions in this population. This study addresses this gap by developing and validating comprehensive machine learning models — encompassing 20 distinct algorithms — specifically designed to predict appendiceal perforation risk during conservative treatment of fecalith-associated appendicitis, with the aim of providing clinically actionable tools for emergency physicians and surgeons.

Materials and methods

Study design and population

This retrospective cohort study was conducted at four tertiary care centers between January 2018 and December 2023. The study protocol was conducted in accordance with the Declaration of Helsinki and STROBE guidelines for observational studies³². Clinical trial number: not applicable.

Patients were included if they: (1) were diagnosed with acute appendicitis confirmed by CT imaging, (2) had radiologically confirmed appendicolith presence, (3) underwent initial conservative treatment, and (4) were ≥ 18 years old. Exclusion criteria included: (1) pregnancy, (2) immunocompromised status, (3) previous appendectomy, and (4) incomplete medical records. The patient flow diagram was shown as Fig. 1.

Fig. 1.

Patient flow diagram for the fecalith-associated appendicitis study. A total of 1,496 patients with fecalith-associated appendicitis undergoing initial conservative treatment were screened from four tertiary care centers (January 2018–December 2023). After exclusion of 249 patients (pregnancy, n = 45; immunocompromised status, n = 78; previous appendectomy, n = 34; incomplete medical records, n = 92), 1,247 patients were included in the final study cohort. Of these, 186 patients (14.9%) developed appendiceal perforation within 72 h of conservative treatment, while 1,061 patients (85.1%) did not. The cohort was randomly divided into training (70%, n = 873), validation (15%, n = 187), and test (15%, n = 187) sets for machine learning model development and evaluation.

Data collection

Clinical data extracted included demographics, presenting symptoms, vital signs, laboratory values, and imaging findings. Laboratory parameters comprised complete blood count, C-reactive protein (CRP), procalcitonin, lactate, and basic metabolic panel. CT imaging variables included appendiceal diameter, wall thickness, periappendiceal fat stranding, fluid collection, and fecalith characteristics (size, location, density). Missing data were handled using iterative imputation (multivariate imputation by chained equations; MICE), implemented in scikit-learn v1.3. In total, 3.2% of values were missing, distributed across procalcitonin (6.8%), lactate (5.1%), and fecalith density (4.3%); all other variables had < 2% missingness. All continuous features were standardized (zero mean, unit variance) prior to model training. Categorical variables were encoded using one-hot encoding.

Outcome definition

The primary outcome was appendiceal perforation occurring within 72 h of conservative treatment initiation, defined as radiological evidence of appendiceal wall discontinuity with or without free fluid or abscess formation on follow-up CT imaging.The 72-hour time window was selected based on the following rationale: (1) the majority of conservative-management failures in fecalith-associated appendicitis occur within the first 48–72 h, consistent with the pathophysiological timeline of progressive ischaemia and bacterial translocation through the compromised appendiceal wall^11,12; (2) institutional protocols at all four participating centers mandate clinical re-evaluation and repeat CT imaging at 48–72 h for patients receiving conservative management; and (3) the APPAC and NOTA landmark trials used a comparable early observation window for defining treatment failure^6,7. We acknowledge that the 72-hour cut-off represents an operationalised rather than a biologically absolute threshold. Alternative windows (48 and 96 h) were not evaluated in this study; future work should systematically compare outcomes across different observation periods to optimise monitoring protocols.

Machine learning models

Twenty machine learning algorithms were implemented and evaluated, including Tree-based Methods (Random Forest, Extra Trees, Gradient Boosting Machine, XGBoost, LightGBM, CatBoost, Decision Tree), Linear Models (Logistic Regression, Ridge Regression, Lasso Regression, Elastic Net), Ensemble Methods (Voting Classifier, Stacking Classifier, AdaBoost), Neural Networks (Multi-layer Perceptron, Deep Neural Network), Instance-based Methods (K-Nearest Neighbors, Support Vector Machine), and Probabilistic Methods (Naive Bayes, Gaussian Process).The Gradient Boosting classifier (scikit-learn v1.3, GradientBoostingClassifier) is nominated as the final model for clinical deployment based on its superior and most balanced performance across all evaluation metrics. This model uses the eight LASSO-selected features (fecalith size, WBC count, CRP, appendiceal wall thickness, patient age, heart rate, procalcitonin, and appendiceal diameter) as its complete input feature set, with no extended feature set. Optimised hyperparameters: n_estimators = 200, learning_rate = 0.05, max_depth = 4, subsample = 0.8, min_samples_leaf = 20 (selected via 5-fold cross-validation with Bayesian optimisation). The model outputs a continuous perforation probability (0–1); the pre-specified optimal decision threshold is 0.54 (Youden index criterion), corresponding to the risk tiers described in “Model development and validation” section.

Model development and validation

Data preprocessing included missing value imputation using iterative imputation, feature scaling using standardization, and categorical encoding. The dataset was randomly split into training (70%), validation (15%), and testing (15%) sets while maintaining outcome distribution balance. Hyperparameter optimization was performed using 5-fold cross-validation with Bayesian optimization. Model performance was evaluated using area under the receiver operating characteristic curve (AUC-ROC), sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and F1-score. Feature importance was assessed using SHAP (SHapley Additive exPlanations) values to provide interpretable insights into model predictions.External validation was performed in an independent cohort of 225 consecutive patients with fecalith-associated appendicitis managed conservatively at The People’s Hospital of Sishui (January 2023–December 2024). This institution was not involved in model development. The same inclusion and exclusion criteria were applied. The final Gradient Boosting model, trained on the full development cohort (n = 1,247), was applied without any re-training or recalibration to the external cohort, and performance was assessed using AUC-ROC, sensitivity, specificity, PPV, NPV, F1-score, accuracy, and Brier score. Bootstrap-based 95% confidence intervals were computed using 2,000 resamples. Risk stratification thresholds and corresponding recommended clinical actions are pre-specified as follows: predicted probability < 30% (low risk): discharge with outpatient follow-up at 48–72 h; 30–60% (moderate risk): inpatient observation with serial clinical examinations and repeat inflammatory markers at 12–24 h; 60–80% (high risk): surgical preparation with operating room standby; and > 80% (critical risk): immediate surgical consultation and intervention. These thresholds were derived from the risk stratification analysis of the development cohort and are intended as clinical decision aids, to be interpreted in conjunction with the overall clinical picture. Model reassessment schedule: Given the dynamic nature of appendiceal inflammation, the model is designed to be applied serially during the first 72 h of conservative management. The recommended schedule is: (1) at baseline presentation (T = 0 h), with updated laboratory and imaging data; (2) at 12–24 h, using updated inflammatory markers (WBC, CRP, procalcitonin); and (3) at 48 h, with repeat CT imaging if clinically indicated. A rising predicted probability trajectory—particularly crossing from the low-risk to moderate-risk tier or from moderate-risk to high-risk tier between assessments—should prompt escalation of the clinical management plan irrespective of the absolute predicted probability at any single time point.

Statistical analysis

Continuous variables were compared using Student’s t-test or Mann-Whitney U test as appropriate. Categorical variables were analyzed using chi-square or Fisher’s exact test. Model performance comparisons were conducted using DeLong’s test for AUC differences. Statistical significance was set at p < 0.05. All analyses were performed using Python 3.9 with scikit-learn, XGBoost, and SHAP libraries.

Results

Patient characteristics and study cohort

A total of 1,496 patients with fecalith-associated appendicitis who underwent initial conservative treatment were screened from four tertiary care centers between January 2018 and December 2023. After applying exclusion criteria, 249 patients were excluded: pregnancy (n = 45), immunocompromised status (n = 78), previous appendectomy (n = 34), and incomplete medical records (n = 92). The final study cohort comprised 1,247 patients, of whom 186 (14.9%) developed appendiceal perforation within 72 h of conservative treatment initiation, while 1,061 (85.1%) did not experience perforation (Fig. 1).

Baseline characteristics differed significantly between patients with and without perforation (Table 1). Patients who developed perforation were significantly older (48.2 ± 18.3 vs. 41.4 ± 16.2 years; P < 0.001) and had higher body mass index (26.8 ± 4.7 vs. 25.9 ± 4.2 kg/m²; P = 0.022). Sex distribution did not differ significantly between groups (male: 57.0% vs. 51.6%; P = 0.178).

Table 1.

Baseline patient characteristics.

Variable	Perforation (n = 186)	No perforation (n = 1,061)	p-value
Age (years)	48.2 ± 18.3	41.4 ± 16.2	< 0.001
MaIe sex	106 (57.0%)	547 (51.6%)	0.178
BMI (kg/m2)	26.8 ± 4.7	25.9 ± 4.2	0.022
Symptom duration (hours)	28.4 ± 12.6	22.1 ± 10.8	< 0.001
Temperature (°C)	38.2 ± 0.8	37.6 ± 0.7	< 0.001
Heart rate (bpm)	94.2 ± 16.3	88.7 ± 14.2	< 0.001
WBC count (× 103/µL)	14.8 ± 4.2	11.9 ± 3.6	< 0.001
CRP (mg/L)	87.3 ± 42.1	45.2 ± 28.7	< 0.001
ProcaIcitonin (ng/mL)	1.24 ± 0.68	0.67 ± 0.34	< 0.001
AppendiceaI diameter (mm)	11.2 ± 2.8	9.4 ± 2.1	< 0.001
WaII thickness (mm)	4.1 ± 1.2	3.2 ± 0.9	< 0.001
FecaIith size (mm)	8.7 ± 3.4	6.2 ± 2.8	< 0.001
PeriappendiceaI stranding	172 (92.5%)	856 (80.7%)	< 0.001

Clinical parameters were significantly more deranged in the perforation group. Symptom duration was longer (28.4 ± 12.6 vs. 22.1 ± 10.8 h; P < 0.001), body temperature was higher (38.2 ± 0.8 vs. 37.6 ± 0.7 °C; P < 0.001), and heart rate was elevated (94.2 ± 16.3 vs. 88.7 ± 14.2 bpm; P < 0.001).

Laboratory inflammatory markers were markedly elevated in patients with perforation. White blood cell (WBC) count was significantly higher (14.8 ± 4.2 vs. 11.9 ± 3.6 × 10³/µL; P < 0.001), C-reactive protein (CRP) levels were nearly doubled (87.3 ± 42.1 vs. 45.2 ± 28.7 mg/L; P < 0.001), and procalcitonin concentrations were significantly elevated (1.24 ± 0.68 vs. 0.67 ± 0.34 ng/mL; P < 0.001).

Imaging findings on computed tomography (CT) also demonstrated significant differences. Appendiceal diameter was larger in the perforation group (11.2 ± 2.8 vs. 9.4 ± 2.1 mm; P < 0.001), appendiceal wall thickness was greater (4.1 ± 1.2 vs. 3.2 ± 0.9 mm; P < 0.001), fecalith size was larger (8.7 ± 3.4 vs. 6.2 ± 2.8 mm; P < 0.001), and periappendiceal fat stranding was more prevalent (92.5% vs. 80.7%; P < 0.001).

Feature selection using LASSO regularization

LASSO (Least Absolute Shrinkage and Selection Operator) regularization was employed to identify the most predictive features from the initial set of clinical and imaging variables (Fig. 2). The regularization path analysis demonstrated the coefficient trajectories of candidate features across varying regularization parameters (α) (Fig. 2A). Five-fold cross-validation identified the optimal regularization parameter at α = 0.0074, corresponding to the point of maximum cross-validation score with minimal overfitting (Fig. 2B).

Fig. 2.

LASSO regularization analysis for feature selection. (A) LASSO regularization path showing coefficient values of the top 8 features across different regularization parameters (α). (B) Five-fold cross-validation performance with optimal α = 0.0074 (red dot) and 95% confidence interval (shaded area). (C) Ranked absolute coefficient values of the top 8 features at optimal α = 0.0074: fecalith size (0.072), white blood cell count (0.070), C-reactive protein (0.065), appendiceal wall thickness (0.057), patient age (0.056), heart rate (0.006), procalcitonin (0.003), and appendiceal diameter (0.001). (D) Number of selected features versus regularization strength, with the optimal α yielding 8 features (horizontal dashed line). (E) Coefficient stability analysis based on 30 bootstrap samples demonstrating consistent feature importance across resampling iterations.

At the optimal α value, eight features were retained with non-zero coefficients (Fig. 2C): fecalith size (coefficient: 0.072), WBC count (0.070), CRP (0.065), appendiceal wall thickness (0.057), patient age (0.056), heart rate (0.006), procalcitonin (0.003), and appendiceal diameter (0.001). The relationship between regularization strength and the number of selected features confirmed that the optimal α corresponded to an 8-feature model (Fig. 2D). Bootstrap stability analysis with 30 resampled datasets demonstrated consistent coefficient estimates across iterations, confirming the robustness of the selected features (Fig. 2E).

Machine learning model performance comparison

Twenty machine learning algorithms were trained and evaluated using the selected features (Table 2). Model performance was assessed on the independent test set (n = 187) using multiple metrics including area under the receiver operating characteristic curve (AUC-ROC), sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), F1-score, and accuracy.

Table 2.

Performance comparison of 20 machine learning models for predicting appendiceal perforation in fecalith-associated appendicitis.

Rank	Model	AUC-ROC (95% CI)	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)	F1-Score	Accuracy (%)
1	Gradient Boosting	0.892 (0.871–0.913)	84.4	81.7	47.8	96.2	0.614	82.1
2	XGBoost	0.887 (0.865–0.909)	82.8	83.1	49.2	95.9	0.618	83.0
3	LightGBM	0.884 (0.862–0.906)	81.7	82.4	47.9	95.7	0.608	82.3
4	Random Forest	0.879 (0.857–0.901)	79.6	84.2	50.1	95.3	0.612	83.4
5	Ensemble Voting	0.876 (0.854–0.898)	80.1	82.8	48.6	95.4	0.609	82.3
6	Neural Network (MLP)	0.871 (0.848–0.894)	78.5	81.9	46.7	95.1	0.595	81.3
7	Extra Trees	0.868 (0.845–0.891)	77.4	82.6	47.2	94.8	0.586	81.7
8	CatBoost	0.863 (0.840–0.886)	76.9	81.8	46.1	94.6	0.576	80.9
9	Support Vector Machine	0.856 (0.832–0.880)	76.3	80.5	44.2	94.7	0.569	79.8
10	AdaBoost	0.851 (0.827–0.875)	75.3	80.9	44.5	94.3	0.559	80.1
11	Stacking Classifier	0.848 (0.824–0.872)	74.7	81.2	44.8	94.1	0.556	80.2
12	Decision Tree	0.834 (0.809–0.859)	73.1	79.7	42.3	93.7	0.536	78.8
13	Logistic Regression	0.834 (0.809–0.859)	72.6	78.9	40.8	93.8	0.524	78.1
14	Elastic Net	0.829 (0.804–0.854)	71.5	79.3	41.1	93.4	0.522	78.2
15	Ridge Regression	0.825 (0.800–0.850)	70.4	78.8	40.2	93.1	0.512	77.6
16	Lasso Regression	0.821 (0.796–0.846)	69.9	78.4	39.6	93.0	0.506	77.2
17	K-Nearest Neighbors	0.798 (0.772–0.824)	67.2	77.9	38.2	92.3	0.487	76.3
18	Logistic Regression (L2)	0.794 (0.768–0.820)	66.7	77.3	37.4	92.1	0.479	75.8
19	Naive Bayes	0.756 (0.728–0.784)	63.4	74.2	33.6	91.1	0.439	72.7
20	Gaussian Process	0.723 (0.694–0.752)	59.7	71.8	30.2	90.1	0.401	70.2

Table 3.

Baseline characteristics of the external validation cohort by perforation outcome.

Variable	Perforation (n = 38)	No Perforation (n = 187)	P value
Demographics
Age (years)	45.7 ± 16.1	43.0 ± 15.9	0.350
Male sex, n (%)	22 (57.9%)	104 (55.6%)	0.937
BMI (kg/m²)	27.2 ± 4.2	25.7 ± 4.1	0.032
Clinical Parameters
Symptom duration (h)	28.7 ± 13.5	22.6 ± 11.0	0.011
Body temperature (°C)	38.4 ± 0.7	37.6 ± 0.7	< 0.001
Heart rate (bpm)	97.5 ± 15.7	88.5 ± 12.8	< 0.001
Laboratory Markers
WBC (×10³/µL)	15.3 ± 4.4	12.4 ± 3.5	< 0.001
CRP (mg/L)	84.3 ± 41.9	47.6 ± 25.8	< 0.001
PCT (ng/mL)	1.4 ± 0.5	0.7 ± 0.3	< 0.001
CT Imaging Findings
Appendiceal diameter (mm)	11.1 ± 1.9	9.4 ± 2.2	< 0.001
Wall thickness (mm)	4.2 ± 1.3	3.3 ± 0.9	< 0.001
Fecalith size (mm)	8.9 ± 3.5	6.6 ± 2.4	< 0.001
Model Output
Predicted risk score (%)	60.8 ± 15.3	31.7 ± 15.3	< 0.001

Data are presented as mean ± SD for continuous variables and n (%) for categorical variables. P values from Mann–Whitney U test (continuous) or chi-square test (categorical) as appropriate. Bold red P values indicate statistical significance (P < 0.05).

Gradient Boosting achieved the highest discriminative performance with an AUC-ROC of 0.892 (95% CI 0.871–0.913), followed by XGBoost (AUC: 0.887; 95% CI 0.865–0.909), LightGBM (AUC: 0.884; 95% CI 0.862–0.906), Random Forest (AUC: 0.879; 95% CI 0.857–0.901), and Ensemble Voting (AUC: 0.876; 95% CI 0.854–0.898) (Fig. 3A). The receiver operating characteristic curves demonstrated clear separation from the diagonal reference line across all top-performing models, indicating excellent discriminative ability.

Fig. 3.

Performance comparison of machine learning models for predicting appendiceal perforation. (A) Receiver operating characteristic (ROC) curves for the top 10 machine learning models. Gradient Boosting achieved the highest area under the curve (AUC = 0.892), followed by XGBoost (AUC = 0.887), LightGBM (AUC = 0.884), Random Forest (AUC = 0.879), and Ensemble Voting (AUC = 0.876). (B) Radar chart displaying multi-metric performance comparison (AUC-ROC, accuracy, sensitivity, specificity, positive predictive value [PPV], negative predictive value [NPV], and F1-score) for the top 5 models. (C) Calibration curves comparing predicted probabilities with observed event rates. Gradient Boosting demonstrated the best calibration with a Brier score of 0.089, followed by XGBoost (0.094), LightGBM (0.097), and Random Forest (0.103). (D) Bar chart comparing Brier scores across the top 4 gradient boosting-based models, with lower scores indicating better calibration.

Multi-metric comparison using radar charts revealed that Gradient Boosting and XGBoost exhibited the most balanced performance profiles across all evaluation metrics (Fig. 3B). Gradient Boosting achieved a sensitivity of 84.4%, specificity of 81.7%, PPV of 47.8%, NPV of 96.2%, F1-score of 0.614, and overall accuracy of 82.1%. Notably, all top five models demonstrated NPV values exceeding 95%, indicating excellent capability for identifying patients at low risk of perforation who may be suitable for continued conservative management.

Model calibration was assessed using calibration curves and Brier scores (Fig. 3C,D). Gradient Boosting demonstrated the best calibration with a Brier score of 0.089, followed by XGBoost (0.094), LightGBM (0.097), and Random Forest (0.103). The calibration curves showed that predicted probabilities closely aligned with observed event rates across the probability spectrum, particularly for Gradient Boosting, which tracked nearest to the perfect calibration diagonal.

Traditional machine learning methods including Logistic Regression (AUC: 0.834), Support Vector Machine (AUC: 0.856), and K-Nearest Neighbors (AUC: 0.798) showed inferior performance compared to ensemble gradient boosting methods. Naive Bayes (AUC: 0.756) and Gaussian Process (AUC: 0.723) demonstrated the lowest discriminative ability among all tested algorithms.

Feature importance and model interpretability

SHAP (SHapley Additive exPlanations) analysis was performed on the best-performing Gradient Boosting model to quantify feature contributions to perforation prediction (Fig. 4). Global feature importance ranking based on mean absolute SHAP values identified fecalith size as the most influential predictor (SHAP importance: 0.234), followed by CRP (0.186), WBC count (0.162), appendiceal wall thickness (0.143), patient age (0.121), symptom duration (0.098), appendiceal diameter (0.087), procalcitonin level (0.074), body temperature (0.063), and periappendiceal fat stranding (0.057) (Fig. 4A).

Fig. 4.

Feature importance analysis and clinical risk stratification using SHAP (SHapley Additive exPlanations) values. (A) Complete feature importance ranking based on mean absolute SHAP values. Fecalith size demonstrated the highest importance (0.234), followed by C-reactive protein (0.186), white blood cell count (0.162), appendiceal wall thickness (0.143), patient age (0.121), symptom duration (0.098), appendiceal diameter (0.087), procalcitonin level (0.074), body temperature (0.063), and periappendiceal fat stranding (0.057). (B) SHAP beeswarm plot illustrating the direction and magnitude of feature impact on perforation prediction. Red indicates high feature values; blue indicates low feature values. (C) Clinical risk stratification distribution showing patient allocation into low-risk (0–30%, 58.9%), moderate-risk (30–60%, 31.9%), and high-risk (> 60%, 9.2%) groups. (D) Actual perforation rates by risk group: low-risk (3.8%), moderate-risk (24.6%), and high-risk (71.3%), compared to the overall perforation rate of 14.9% (dashed line).

The SHAP beeswarm plot provided detailed visualization of feature-outcome relationships (Fig. 4B). Higher values of fecalith size, CRP, WBC count, and wall thickness consistently shifted predictions toward higher perforation risk (positive SHAP values), while lower values of these features were associated with decreased risk predictions (negative SHAP values). The directional consistency of these relationships across the patient cohort confirmed the biological plausibility of the model’s decision-making process.

Clinical risk stratification

Based on predicted perforation probabilities, patients were stratified into three risk categories: low risk (0–30%), moderate risk (30–60%), and high risk (> 60%) (Fig. 4C). The distribution analysis revealed that 58.9% of patients were classified as low risk, 31.9% as moderate risk, and 9.2% as high risk.

Validation of the risk stratification system demonstrated excellent discrimination (Fig. 4D). The actual perforation rates increased progressively across risk categories: 3.8% in the low-risk group, 24.6% in the moderate-risk group, and 71.3% in the high-risk group, compared to the overall perforation rate of 14.9% (indicated by the dashed reference line). This represented a 19-fold difference in perforation risk between the low-risk and high-risk groups, demonstrating the model’s ability to meaningfully separate patient populations with distinct clinical outcomes.

Feature interactions and synergistic effects

Three-dimensional visualization of feature interactions revealed important synergistic effects on perforation risk prediction (Fig. 5). The interaction between fecalith size and CRP demonstrated a multiplicative effect on risk, with the highest predicted probabilities observed when both variables were elevated (Fig. 5A). Patients with fecalith size exceeding 10 mm and CRP levels above 100 mg/L exhibited perforation risks approaching 90%.

Fig. 5.

Three-dimensional visualization of feature interactions and their impact on perforation risk. (A) Interaction surface plot of fecalith size and C-reactive protein (CRP), demonstrating synergistic effects on perforation risk prediction. (B) Interaction surface plot of patient age and white blood cell (WBC) count, showing combined influence on perforation probability. (C) Interaction surface plot of appendiceal wall thickness and appendiceal diameter, illustrating the relationship between morphological parameters and perforation risk. (D) Three-feature interaction scatter plot displaying individual patient risk profiles based on fecalith size, CRP, and age. Points are color-coded by risk level: green (low risk, < 20%), yellow (moderate risk, 20–50%), and red (high risk, > 50%).

The age-WBC interaction surface plot showed that elderly patients with elevated WBC counts faced disproportionately higher perforation risk compared to younger patients with similar laboratory values (Fig. 5B). This finding suggests age-related diminished physiological reserve may compound the inflammatory burden in acute appendicitis.

Morphological parameters also exhibited meaningful interactions (Fig. 5C). The combination of increased wall thickness (> 4.5 mm) and enlarged appendiceal diameter (> 12 mm) was associated with substantially elevated perforation risk, likely reflecting more advanced inflammatory changes in the appendiceal wall.

The three-feature interaction analysis incorporating fecalith size, CRP, and age demonstrated clear separation of risk categories (Fig. 5D). Patients clustered into distinct regions of the feature space: low-risk patients (green) predominantly occupied the region with smaller fecaliths, lower CRP, and younger age, while high-risk patients (red) concentrated in the region characterized by larger fecaliths, elevated CRP, and older age.

Clinical utility assessment

Decision curve analysis was performed to evaluate the clinical utility of the prediction models across a range of threshold probabilities (Fig. 6). Compared to the default strategies of “Treat All” (immediate surgery for all patients) and “Treat None” (conservative management for all), all machine learning models demonstrated positive net benefit within the clinically relevant threshold range (highlighted in yellow).

Fig. 6.

Decision curve analysis evaluating clinical utility of the prediction models. (A) Decision curve analysis comparing net benefit across threshold probabilities (0–80%) for Gradient Boosting, XGBoost, LightGBM, Random Forest, and Ensemble Voting models against the “Treat All” and “Treat None” default strategies. The yellow-shaded region indicates the clinically relevant threshold range where models demonstrate positive net benefit. Higher net benefit indicates better clinical utility. (B) Incremental value of models compared to default strategies, showing the additional net benefit provided by each machine learning model. Positive values indicate that the model adds clinical value over treating all or no patients.

Gradient Boosting and XGBoost consistently achieved the highest net benefit across threshold probabilities ranging from 5% to 40%, corresponding to the range where most clinical decision-making occurs in acute appendicitis management. The incremental value analysis (B panel) confirmed that the models added meaningful clinical value compared to default strategies, with positive incremental benefit observed particularly in the 10–25% threshold range.

At a threshold probability of 15% (approximately equal to the baseline perforation rate of 14.9%), Gradient Boosting provided a net benefit of approximately 0.08, substantially exceeding both default strategies. This indicates that model-guided decision-making could prevent unnecessary surgical interventions while appropriately identifying patients requiring urgent operative management. To facilitate clinical implementation, the decision curve analysis can be interpreted in conjunction with the following suggested action thresholds: at predicted risk < 30%, conservative management with routine monitoring is generally appropriate; at 30–60% predicted risk, intensified observation with serial clinical and laboratory assessments at 12–24 h intervals is recommended; at 60–80% risk, early surgical consultation and operating room preparation are advisable; and at > 80% predicted risk, immediate surgical intervention should be strongly considered. These thresholds correspond to net benefit inflection points on the decision curve and are intended as clinical guides rather than absolute triggers.

Clinical application and decision support framework

A comprehensive clinical decision support framework was developed to facilitate model implementation in clinical practice (Fig. 7). Individual patient risk profiles were visualized using radar charts displaying normalized feature values across key predictors (Fig. 7A). Three representative patient archetypes illustrated distinct risk patterns: Patient A (low risk) exhibited modest elevation across all features, Patient B (moderate risk) showed intermediate values with notable elevation in inflammatory markers, and Patient C (high risk) demonstrated pronounced abnormalities across multiple domains including fecalith size, CRP, and WBC count.

Fig. 7.

Clinical application and decision support framework. (A) Individual risk profiles visualized as radar charts for three representative patients: Patient A (low risk, green), Patient B (moderate risk, orange), and Patient C (high risk, red), displaying normalized values across key features (WBC, CRP, fecalith, symptoms, wall thickness, and age). (B) Distribution of predicted risk probabilities stratified by actual outcome groups (low, moderate, and high risk), demonstrating model discrimination capability. (C) Stacked bar chart showing relative feature importance by risk level, with fecalith size and CRP contributing most substantially in high-risk patients. (D) Risk evolution trajectories over 48 h for three patient archetypes (stable, deteriorating, and critical). Note: these trajectories are illustrative and model-derived from hypothetical parameter sequences, not empirically observed longitudinal measurements from individual patients (see Section “Model development and validation” and main text for details). (E) Prediction uncertainty analysis showing predicted versus true risk with 95% confidence intervals, demonstrating model reliability across the risk spectrum. (F) Clinical decision distribution based on model predictions: discharge (40%), observation (35%), surgery preparation (20%), and immediate surgery (5%).

The distribution of predicted risk probabilities stratified by outcome groups confirmed the model’s discriminative ability (Fig. 7B). Low-risk patients exhibited predicted probabilities predominantly below 0.3, while high-risk patients showed probability distributions centered above 0.6, with minimal overlap between groups.

Feature importance varied by risk level (Fig. 7C). In low-risk patients, the relative contributions of individual features were more evenly distributed. In contrast, high-risk patients demonstrated dominant contributions from fecalith size and CRP, accounting for over 50% of the cumulative feature importance, suggesting these variables are particularly critical for identifying patients at highest risk of perforation.

Temporal risk evolution trajectories were modeled for three patient archetypes over a 48-hour observation period (Fig. 7D). These trajectories are illustrative and model-derived, not empirically observed longitudinal measurements. Since this is a retrospective study with a single observation point per patient and a binary 72-hour perforation outcome, the trajectories represent simulated risk evolution generated by applying the trained model to hypothetical sequences of clinical parameter values characteristic of each archetype. They are presented to demonstrate the dynamic monitoring concept and should not be interpreted as observed patient trajectories. Stable patients (green) maintained consistently low risk levels below the moderate-risk threshold (30%, lower dashed line). Deteriorating patients (orange) showed progressive risk escalation, crossing the moderate-risk threshold at approximately 24 h and approaching the high-risk threshold (60%, upper dashed line) by 48 h. Critical patients (red) exhibited rapid risk escalation, surpassing the high-risk threshold within 12 h, emphasizing the need for expedited surgical intervention in this subgroup.

Model prediction uncertainty was quantified using 95% confidence intervals (Fig. 7E). The predicted versus true risk scatter plot demonstrated strong concordance across the probability spectrum, with narrower confidence intervals observed for low-risk and high-risk predictions compared to moderate-risk predictions, reflecting greater model certainty at the extremes of the risk distribution.

Based on model predictions and clinical risk thresholds, the recommended clinical decision distribution was: discharge with outpatient follow-up (40%), continued observation with serial examinations (35%), surgical preparation with OR standby (20%), and immediate surgical intervention (5%) (Fig. 7F). This distribution aligns with the overall perforation rate of 14.9% and demonstrates the model’s potential to optimize resource allocation while maintaining patient safety.

External validation

The final Gradient Boosting model was prospectively applied to an independent external validation cohort of 225 consecutive patients with fecalith-associated appendicitis managed conservatively at The People’s Hospital of Sishui(January 2023–December 2024), a fifth centre not involved in model development. Of the 225 patients, 38 (16.9%) developed appendiceal perforation within 72 h, consistent with the development cohort perforation rate of 14.9%.

The model demonstrated sustained discriminative performance in the external cohort (Fig. 8). The AUC was 0.909 (95% CI 0.859–0.951), confirming good generalisability. At the pre-specified decision threshold of 0.54 (Youden index criterion derived from the development cohort), sensitivity was 73.7%, specificity was 93.0%, PPV was 68.3%, NPV was 94.6%, F1-score was 0.709, and overall accuracy was 89.8%. The Brier score was 0.133, indicating acceptable calibration in the external sample.

Fig. 8.

External validation performance of the Gradient Boosting model. (A) Receiver operating characteristic (ROC) curve for the Gradient Boosting model applied to the external validation cohort. AUC = 0.909 (95% CI 0.859–0.951). The red dot indicates the pre-specified optimal decision threshold (0.54). Internal test set AUC (0.892, blue dotted line) is shown for reference. (B) Calibration curve comparing predicted probabilities with observed event rates in the external cohort (Brier score = 0.133). Internal Brier score (0.089) is shown for reference. (C) Risk stratification performance in the external cohort: observed perforation rates by risk tier [low risk (< 30%): 0.0% (n = 88); moderate risk (30–60%): 16.4% (n = 110); high risk (> 60%): 74.1% (n = 27)], mirroring the development cohort pattern. Overall external cohort perforation rate: 16.9% (dashed line).

Risk stratification in the external cohort yielded the following distribution: low risk (< 30%): 88 patients (39.1%), observed perforation rate 0.0%; moderate risk (30–60%): 110 patients (48.9%), observed perforation rate 16.4%; and high risk (> 60%): 27 patients (12.0%), observed perforation rate 74.1%. The progressive increase in observed perforation rates across risk tiers in the external cohort mirrors the pattern observed in the development cohort, supporting the clinical validity of the three-tier risk stratification framework (Fig. 8C).

Discussion

This study represents the largest investigation to date applying comprehensive machine learning approaches to predict appendiceal perforation during conservative treatment of fecalith-associated appendicitis. Our findings demonstrate that ensemble tree-based algorithms, particularly gradient boosting methods, achieve excellent predictive performance with clinical utility.

Clinical implications

The developed models offer several potential clinical applications. First, the high negative predictive value (96.2%) of the best-performing model suggests strong utility for identifying patients at low risk for perforation who may safely undergo conservative treatment with standard monitoring. The positive predictive value (PPV) of 47.8% warrants explicit clinical contextualisation: approximately one in two patients flagged as high-risk would not actually perforate. This false-positive rate carries non-trivial clinical consequences, particularly given that the proposed management actions for high-risk patients include surgical preparation or immediate intervention. In this context, the model should be positioned as a decision support adjunct — a positive model prediction in the high-risk tier should trigger prompt clinical reassessment, repeated laboratory evaluation, and discussion with the surgical team, rather than automatic operative intervention. The attending clinician’s overall gestalt, incorporating factors such as the patient’s trajectory, comorbidities, pain severity, and response to antibiotics, remains essential. Conversely, the excellent NPV (96.2%) provides strong reassurance that patients classified as low-risk can be safely observed with standard conservative protocols, avoiding unnecessary early operative intervention.

The identification of fecalith size as the most important predictor aligns with pathophysiological understanding. Larger fecaliths create greater obstruction and pressure effects, promoting bacterial overgrowth and tissue ischemia^9,16. Our threshold of 8 mm provides a clinically actionable cutoff for risk assessment.

Comparison with existing literature

Previous studies have reported perforation rates of 8–12% in general appendicitis populations undergoing conservative treatment^4,43. Our observed rate of 14.9% in fecalith-associated cases confirms the increased risk profile of this subgroup. We acknowledge that a direct head-to-head comparison of the Alvarado and AIR scores against our ML models on the same dataset was not performed in this study. Retrospective calculation of these scores was limited by the unavailability of certain required clinical items (e.g., shift to left, rebound tenderness recorded in the structured format required) in a subset of records. This represents a recognised limitation, and prospective studies should include such scoring systems as explicit comparators. Available published AUC data for these scores in fecalith-associated appendicitis (0.69–0.83^19,20 are therefore provided as contextual benchmarks rather than direct comparisons. Sallinen et al.⁴ reported that appendicolith presence was associated with increased treatment failure (OR: 2.1) but did not develop specific prediction models. Our study advances this work by creating quantitative risk assessment tools specifically for this high-risk population.

Model selection and performance

The superior performance of ensemble tree-based methods likely reflects their ability to capture complex, non-linear interactions among clinical variables^34,35. Gradient boosting methods excel at handling mixed data types and identifying subtle patterns that may be missed by traditional statistical approaches³⁶. The relatively modest positive predictive value highlights a fundamental challenge in perforation prediction - the rarity of the outcome creates inherent limitations in positive predictive performance³⁷. However, the excellent negative predictive value provides substantial clinical utility for safe conservative management decisions. From a technical standpoint, several potential challenges of machine learning approaches in this clinical context deserve discussion. First, although the 70/15/15 training/validation/test split preserves the overall outcome distribution via stratified sampling, this approach constitutes internal validation and does not eliminate the risk of overfitting to institutional idiosyncrasies of the four development centres. Our external validation results (AUC 0.909) provide initial evidence that the model generalises beyond the development sample, though the performance should be interpreted cautiously given that the external cohort originates from the same geographic and healthcare system context. Second, recall bias is inherent in any retrospective study: the completeness and accuracy of clinical records may influence the quality of training data, potentially introducing systematic errors. Third, the model was trained at a specific calendar period (2018–2023); temporal drift in clinical practice, imaging technology, or microbial ecology could erode model performance over time, necessitating periodic recalibration^38,39.

TRIPOD alignment and reporting

This study is reported in accordance with the TRIPOD statement for prediction model development and validation⁴⁰. Key TRIPOD items are summarised as follows: (1) Source of data: four-centre retrospective cohort, 2018–2023 (development); one-centre retrospective application, 2023–2024 (external validation). (2) Participants: 1,247 development, 225 external validation. (3) Outcome: appendiceal perforation within 72 h. (4) Predictors: eight LASSO-selected variables (fecalith size, WBC, CRP, wall thickness, age, heart rate, procalcitonin, appendiceal diameter). (5) Sample size: the development cohort of 1,247 patients with 186 events satisfies recommended minimum EPV (events per variable) ratios. (6) Missing data: iterative imputation (MICE; 3.2% overall missingness). (7) Final model: Gradient Boosting classifier with pre-specified hyperparameters and decision threshold (0.54). (8) Model performance: reported with 95% CIs in both development and external validation cohorts. (9) Limitations and generalisability: acknowledged.

Feature insights

The prominence of inflammatory markers (CRP, WBC) and imaging features (wall thickness, appendiceal diameter) confirms their continued importance in modern appendicitis assessment^41,42. The inclusion of symptom duration reflects the time-dependent nature of appendiceal inflammation progression⁴³. Notably, traditional clinical variables such as pain location and rebound tenderness showed lower predictive importance, suggesting that objective laboratory and imaging parameters provide superior risk discrimination in this specific population^44,45.

Study limitations

Several limitations merit consideration. Most importantly, despite the inclusion of an external validation cohort, this study has limitations in establishing definitive generalisability. The external validation centre shares geographic and healthcare-system characteristics with the development centres, and the reported performance metrics may therefore be somewhat optimistic relative to what would be expected in truly independent international settings. The reported internal AUC of 0.892 is based on a random split of the development data and may reflect optimistic estimation; prospective real-world deployment studies will be required to confirm these metrics^39,46,47. The retrospective design may introduce selection bias, although our comprehensive data collection mitigate this concern⁴⁸. The multi-center nature was limited to four institutions, requiring external validation in diverse populations and healthcare settings⁴⁶. The 72-hour observation window, while clinically relevant, represents an arbitrary cutoff. Future studies should evaluate optimal monitoring timeframes and risk evolution patterns³⁸. Additionally, treatment protocols and imaging techniques may vary across institutions, potentially affecting model performance in external settings³⁹.

Future directions

External validation in multi-center cohorts represents the next critical step for clinical implementation⁴⁷. Integration of additional data sources, such as continuous vital sign monitoring or novel biomarkers, may further enhance predictive accuracy⁴⁹. Real-time implementation studies will be essential to evaluate clinical impact, workflow integration, and cost-effectiveness. Regarding deployment format, we envisage the Gradient Boosting model being made available as a web-based risk calculator (planned deployment via a dedicated URL) allowing clinicians to enter the eight input variables at the bedside. Prospective real-time implementation studies will be essential to evaluate clinical impact, workflow integration, and cost-effectiveness⁵⁰. Development of a simplified scoring chart derived from the model, while sacrificing some predictive accuracy, may improve adoption in settings without electronic clinical decision support⁵¹. Investigation of optimal intervention strategies for high-risk patients identified by these models represents an important clinical question. Whether enhanced monitoring, alternative antibiotic regimens, or earlier surgical intervention improves outcomes requires prospective evaluation⁵².

Conclusions

Machine learning models demonstrate excellent performance for predicting appendiceal perforation during conservative treatment of fecalith-associated appendicitis. The gradient boosting model achieved optimal performance with an AUC of 0.892 in internal testing and AUC of 0.909 in an independent external validation cohort (n = 225), confirming generalisability beyond the development sample. Fecalith size, inflammatory markers (CRP, WBC), and imaging parameters (wall thickness, appendiceal diameter) emerged as key predictive features. The model’s risk stratification framework — with pre-specified clinical action thresholds and a serial reassessment protocol at 0, 12–24, and 48 h — provides a clinically actionable and deployable decision-support tool. Fecalith size, inflammatory markers, and imaging parameters emerged as key predictive features. These models may assist clinical decision-making by identifying patients suitable for safe conservative management versus those requiring enhanced monitoring or surgical consideration.

However, rigorous external validation across diverse healthcare settings and prospective implementation trials are essential prerequisites for clinical adoption^40,53. The integration of machine learning into appendicitis management holds promise for enabling risk-stratified care pathways and evidence-based resource allocation, potentially improving diagnostic accuracy and clinical outcomes in this common surgical emergency^54,55.

Author contributions

All authors participated in the design, interpretation of the studies and analysis of the data and review of the manuscript. Z.Y.L. (Zhu Youlong), L.R.M. (Liu Ruming), and F.J.W. (Feng Jiawei) conducted the experiments. Z.Y.L., R.W. (Ren Wang), M.X. (Ma Xiao) and G.L.B. (Gong Longbo) supplied critical materials. Z.Y.L., K.F.F. (Kong Feifei), and W.N. (Wang Ning) wrote the manuscript. Author initials follow the Chinese convention of family name followed by given name initials (e.g., F.J.W. = Feng Jia-Wei).

Funding

No funding was received for conducting this study.

Data availability

The patient-level dataset used in this study cannot be made publicly available or deposited in a public repository owing to restrictions imposed by the institutional review boards of the participating centres and applicable patient privacy regulations (Chinese Personal Information Protection Law, 2021). These restrictions preclude data sharing beyond what is described herein, even in anonymised form, as the combination of clinical and imaging variables could theoretically enable re-identification. The authors understand that this limits reproducibility and encourage reviewers to interpret this limitation in light of these regulatory and ethical constraints. Readers wishing to access the data for academic purposes may contact the corresponding author to discuss feasibility within the bounds of institutional and regulatory permissions.

Declarations

Competing interests

The authors declare no competing interests.

Ethical approval and consent to participate

This retrospective study was conducted in accordance with the Declaration of Helsinki. The study was approved by the institutional review boards of all participating centers: the Ethics Committee of Xuzhou Central Hospital, Southeast University; the Ethics Committee of The Third Affiliated Hospital of Soochow University, Changzhou First People’s Hospital; the Ethics Committee of The Affiliated Hospital of Xuzhou Medical University; the Ethics Committee of Jinan Maternal and Child Health Care Hospital; and the Ethics Committee of The People’s Hospital of Sishui. The requirement for informed consent was waived due to the retrospective nature of the study.