Accurate prediction of mortality in children with sepsis: development and validation of an explainable model based on real-world data

Abstract

Background

Sepsis remains the leading cause of in-hospital deaths among children, and there is currently a lack of precise early prediction models. This study aimed to develop an interpretable machine learning (IML) model to predict in-hospital mortality in pediatric patients diagnosed with sepsis according to the 2024 Phoenix Sepsis Diagnostic Criteria.

Methods

This single-center retrospective cohort study analyzed 464,459 children who were hospitalized in the Children’s Hospital Affiliated to Nanjing Medical University from January 2018 to December 2023 were screened. Based on the Phoenix Sepsis Score, 2,477 cases of childhood sepsis were finally diagnosed. The features were filtered using three methods the minimum absolute contraction and selection operator (LASSO) regression, the Boruta algorithm, and random forest importance ranking, ultimately identifying 12 features. Eight machine learning algorithms were trained on 70% of the data and evaluated on a 30% test set. The optimal model was screened based on the accuracy of the test set and the area under the receiver operating characteristic curve (AUC). The interpretability of the model is enhanced by leveraging SHapley Additive exPlanations (SHAP) summary graphs, individual SHAP force maps, and partial dependency plots (PDPs).

Results

A total of 2,477 patients with sepsis met the participation criteria, with a median age of 26 months (IQR, 5–78 months). Among the 2,477 participants, 1,448 (58.5%) were boys. The CatBoost model demonstrated the best performance among the 8 constructed models, achieving an AUC of 0.889 and an accuracy of 92.9% in the test set. Notably, at the default decision threshold, the model demonstrated high specificity but relatively low sensitivity. However, by optimizing the decision threshold using the Youden Index, the model’s sensitivity was substantially improved to 83.05%, effectively mitigating the potential risk of missed diagnoses for high-risk patients in clinical practice. Feature importance analysis indicated that invasive mechanical ventilation, Glasgow Coma Scale score, platelet count, uric acid, and SpO₂/FiO₂ ratio were the top five features that had the greatest impact on the CatBoost model. SHAP analysis provided both global feature importance and individualised risk explanations, and the model has been deployed as a freely accessible web calculator.

Conclusions

The IML model developed in this study provides a highly interpretable and accurate tool for the early prediction of in-hospital mortality in pediatric sepsis patients. The SHAP approach improves model interpretability and helps clinicians understand the factors driving those predictions. However, as a single-center retrospective study, the generalizability of our findings to other clinical settings may be limited. Future multi-center prospective studies are necessary to further validate the model and address its current limitations in sensitivity to minimize the risk of missing high-risk cases.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13052-026-02287-6.

Background

Sepsis remains a leading cause of pediatric morbidity and mortality worldwide. Globally, it is estimated that there are approximately 50 million sepsis cases annually, of which about 20 million involve children under five years of age, resulting in 2.9 million deaths in this age group; an additional 454,000 deaths occur among children and adolescents aged 5 to 19 years [1]. In recent decades, the incidence of pediatric sepsis has been on a continuous upward trend, particularly evident in low-income and middle-income countries where limited medical resources exacerbate the risk of sepsis occurrence and mortality [2].Even in high-resource settings, sepsis accounts for a significant portion of admissions to pediatric intensive care units (PICUs), often leading to prolonged hospital stays, long-term neurocognitive effects, and high medical costs [3, 4].

Substantial advances have been made in recognizing, diagnosing, and managing pediatric sepsis. Early goal-directed therapy, timely broad-spectrum antibiotics, fluid resuscitation, and organ support have collectively improved patient survival [5, 6]. Several scoring systems are widely used for risk stratification and prognosis, including the Pediatric Risk of Mortality (PRISM), the Pediatric Index of Mortality (PIM), the Pediatric Logistic Organ Dysfunction (PELOD) score, and sepsis-specific tools such as the 2005 International Pediatric Sepsis Consensus Conference (IPSCC) criteria and newer organ dysfunction scores [7]. In addition, biomarkers such as procalcitonin (PCT), C-reactive protein (CRP), lactate, and interleukin-6 have also been incorporated into the clinical decision-making process to facilitate the early identification of sepsis and the monitoring of disease severity [8].

Despite the above progress, there are still several key limitations in the diagnosis and treatment of sepsis in children. traditional scoring systems often exhibit poor discrimination ability and insufficient calibration in different pediatric populations, which is particularly prominent in regions with limited medical resources where patients in these regions often experience delayed medical consultations [9].The 2005 IPSCC criteria relied heavily on parameters related to systemic inflammatory response syndrome (SIRS), which yielded low specificity and failed to capture organ dysfunction comprehensively [10].Adult-focused diagnostic and therapeutic frameworks, such as the Sepsis-3 criteria and the Sequential Organ Failure Assessment (SOFA) score, have limited applicability to pediatric patients because of age-specific physiological differences between children and adults [11–13]. Moreover, many existing prognostic models depend on two problematic variable types, namely those not uniformly obtainable across institutions within the first 24 h of admission and those requiring serial measurements. This results in a delay in risk stratification during the early stage, which is crucial for intervention effectiveness [14]. Although machine learning methods have shown potential in adult sepsis populations, there is still a lack of models for the pediatric population that are interpretable, can utilize routinely collected early clinical data, and meet the latest diagnostic criteria [15, 16].

To address these gaps, this retrospective cohort study used the newly released 2024 Phoenix Sepsis Criteria to identify cases and developed an interpretable machine-learning model based on clinical and laboratory variables obtained within the first 24 h of admission. The study compared 8 advanced algorithms and finally selected the optimal model. Meanwhile, by introducing the SHAP method, the interpretability of the model at both the overall and individual levels was achieved. The findings of this study provide a practical and high performance prognostic tool for early prediction of in - hospital mortality in pediatric sepsis patients. The model has been deployed as a freely accessible web calculator, aiming to support bedside clinical decision making and potentially improve patient prognosis through timely risk stratification.

Methods

Study population

This retrospective study enrolled children with suspected infection who were consecutively admitted to Children’s Hospital of Nanjing Medical University from January 1, 2018 to December 31, 2023. Patients who met the sepsis diagnostic criteria published in 2024 were included in the analysis. Using R (v4.4.3), the cohort was randomly split into a training set (70%) for model development and a test set (30%) for independent validation. To maintain a consistent distribution of the primary outcome between the two sets, we employed a stratified random sampling method based on in-hospital mortality. Model performance was assessed on the test set to evaluate generalizability. The study conformed to the ethical principles of the Declaration of Helsinki and was reviewed and approved by the Research Ethics Committee of Nanjing Children’s Hospital Affiliated to Nanjing Medical University (ethics number: 202407019-1) on August 22, 2024, for the study titled “Validation of the Phoenix Pediatric Sepsis Criteria (2024) in Chinese Children and Sepsis Prognosis Study”. This retrospective cohort study used data from Children’s Hospital Affiliated to Nanjing Medical University and adhered to TRIPOD for prediction models and STROBE for observational studies (Supplemental Table S1 and Table S2).

Definition of sepsis

According to the 2024 Phoenix Sepsis Criteria, pediatric sepsis is defined as a Phoenix Sepsis Score (PSS) of ≥ 2 points on the basis of suspected infection. Suspected infection is defined as concurrent systemic antibiotic treatment and completion of microbiological sampling (such as blood, urine or cerebrospinal fluid culture) within 24 h after admission. PSS is calculated based on the worst values of functional disorders in the cardiovascular, respiratory, nervous and coagulation organ systems (with a maximum total score of 13 points) [5, 6].

Data collection and processing

We extracted demographic information, vital signs, and laboratory results recorded within 24 h of admission from electronic medical records to develop a predictive model. Patients were eligible if they met all of the following criteria: (1) clinical diagnosis of infection or suspected infection; (2) age between 28 days and 18 years; and (3) pediatric sepsis as defined by the 2024 Phoenix Sepsis Criteria (PSS ≥ 2) [5]. Exclusion criteria were: (1) age ≤ 28 days or ≥ 18 years; and (2) hospitalization shorter than 24 h. The detailed patient selection process is illustrated in Fig. 1.

Fig. 1.

Study design and patient selection Flowchart. Flowchart illustrating the study design and patient selection process for developing and validating the machine learning model to predict in-hospital mortality in pediatric sepsis patients. The diagram outlines the steps from patient inclusion based on the 2024 Phoenix Sepsis Criteria, data preprocessing, feature selection, model development, and evaluation. ML: machine learning; SHAP: SHapley Additive exPlanations; CatBoost: Categorical Boosting

We initially considered 58 candidate variables, all obtainable within the first 24 h of admission. These included Age, Gender, Height, Weight, Body Mass Index (BMI), Intensive Care Unit (ICU), Invasive Mechanical Ventilation (IMV), Arterial Partial Pressure of Oxygen (PaO₂), Pulse Oxygen Saturation (SpO₂), Fraction of Inspired Oxygen (FiO₂), Oxygenation Index (P/F), Oxygen Saturation Index (S/F), Vasoactive Drugs, Lactic Acid (LAC), Systolic Blood Pressure (SBP), Diastolic Blood Pressure (DBP), Mean Arterial Pressure (MAP), Platelet Count (PLT), International Normalized Ratio (INR), D-Dimer (DDimer), Fibrinogen (FIB), Glasgow Coma Scale (GCS), Continuous Renal Replacement Therapy (CRRT), Temperature (Tem), Heart Rate (HR), Respiratory Rate (RR), Base Excess (BE), Urine Output (UO), Arterial Partial Pressure of Carbon Dioxide (PaCO₂), Serum Creatinine (SCr), Total Bilirubin (TBIL), C-Reactive Protein (CRP), Procalcitonin (PCT), Erythrocyte Sedimentation Rate (ESR), Hypertension (HBP), Absolute Neutrophil Count (ANC), White Blood Cell Count (WBC), Absolute Eosinophil Count (AEC), Red Blood Cell Distribution Width Standard Deviation (RDWSD), Red Blood Cell Distribution Width Coefficient of Variation (RDWCV), Mean Platelet Volume (MPV), Prothrombin Time (PT), Activated Partial Thromboplastin Time (APTT), Thrombin Time (TT), Alanine Aminotransferase (ALT), Retinol-Binding Protein (RBP), Creatine Kinase (CK), Creatine Kinase-MB (CK-MB), Uric Acid (UA), Urea, Cystatin C (CysC), Lactate Dehydrogenase (LDH), Potassium (K), Sodium (Na), Chloride (Cl), Calcium (Ca), Magnesium (Mg), and Phosphorus (P). Variables with more than 30% missing values were excluded, highly correlated pairs (|Spearman correlation coefficient| > 0.7) were pruned by retaining the more clinically relevant feature. Remaining missing data were imputed using multiple imputation by chained equations (MICE). Considering the large age span (from 28 days to 18 years) in the research cohort and the strong correlation between multiple physiological parameters and age, we performed age correction on HR, RR, SBP, DBP, MAP, Height, Weight, and BMI before feature screening and model training. (Supplemental Text S1) Correction was performed using the residual method by establishing a robust linear regression model with age as the independent variable, following the exclusion of abnormal residual data beyond 2.5 times the standard deviation. The age-corrected value was obtained by adding the residual of each variable’s observed value relative to the predicted value of this model to the original overall mean of the variable. This method removed the linear effect of age while preserving the original measurement scale and the variable’s overall mean (see Appendix Text S11). To avoid overfitting and identify robust prognostic predictors, we used a consistent feature-selection strategy that combined three methods, including ten-fold cross-validated LASSO regression with the λ.min criterion, the Boruta algorithm based on random forest importance, and the intrinsic feature-importance score from random forests. The final variable set for model construction was the intersection of the features selected by all three methods, yielding 12 highly stable predictors.

Model development and comparison

To prevent overfitting, the dataset was randomly partitioned into a training set (70%) for model development and an independent test set (30%) for evaluation. Twelve selected features were used to develop prediction models with eight machine-learning algorithms: boosted regression trees (BRT), support vector machine (SVM), extreme gradient boosting (XGBoost), multilayer perceptron (MLP), AdaBoost, CatBoost, artificial neural network (ANN), and gradient boosting machine (GBM). Each model used the chosen independent risk factors to predict in-hospital mortality among pediatric patients with sepsis. In order to optimize the prediction model, a grid search method combined with manual fine-tuning was used to obtain the final hyperparameters. The performance of the model was assessed using the receiver operating characteristic (ROC) curve and the AUC, along with metrics such as accuracy, sensitivity, specificity, F1 score, positive predictive value (PPV), and negative predictive value (NPV). Calibration was assessed using calibration curves, and the clinical net benefit was measured using decision curve analysis (DCA). The best-performing model was subjected to further validation through 10-fold cross-validation on the training data to reduce the risk of overfitting, with the ultimate performance evaluation conducted on the test set.

Feature selection and model explanation

To tackle the challenges posed by high-dimensional feature spaces, we utilize three complementary approaches for feature screening, including the LASSO method [14], the Boruta algorithm, and the RF-RFE technique. In cases of high collinearity, we did not allow the algorithm to drop variables blindly; instead, we manually retained the feature with higher clinical priority or a stronger association with the outcome to ensure the retention of clinically meaningful predictors. For example, MAP was prioritized over SBP due to its superior clinical relevance in pediatric sepsis resuscitation. The Boruta algorithm, built on the random forest framework, distinguishes between actual features and shadow features by assessing their importance levels, thereby identifying features with practical correlation significance [15]. RF-RFE iteratively removes the least important feature based on random forest feature importance scores. To enhance the robustness and reliability of the feature screening process, we only retained the features that were simultaneously screened out by these three methods. This strict consistency screening strategy ultimately yields 12 key features, which are used for subsequent machine learning modeling.

During the interpretation phase of the model, we utilized the SHAP, a cohesive analytical framework rooted in cooperative game theory. SHAP assesses the impact of specific features on both individual prediction outcomes and the overall model behavior by effectively approximating the Shapley value computationally. This approach offers a clear and reliable rationale for the model’s decision-making process [17].

Web page deployment tool based on shiny framework

To facilitate clinical translation and promote bedside applications, this study deploys the optimized CatBoost model as a freely accessible interactive web-based calculator based on the R-language Shiny framework [18]. This user-friendly tool allows clinicians to enter values for 12 selected predictors obtained within the first 24 h of a patient’s admission and then instantly generates patient-specific predictions of risk of death during hospitalization, as well as providing interpretable SHAP force plot and SHAP waterfall plot to visually illustrate the risk of death during hospitalization. The tool offers interpretable SHAP force and waterfall plots to visually show how each feature impacts the prediction. It is publicly accessible and aims to aid quick risk assessment and evidence-based decision-making in diverse clinical settings.

Statistical analysis

Data processing and analysis were performed in R 4.4.3 (https://www.r-project.org). We adhered to TRIPOD + AI guidance [19] and STROBE Statement [20]. Continuous variables with skewed distributions are reported as medians and interquartile ranges and were compared using the Mann–Whitney U test or the Kruskal–Wallis H test. Categorical variables are reported as counts and percentages and were compared using the chi-square test or Fisher’s exact test. Analysis of covariance (ANCOVA) was used to adjust for confounders. To strictly evaluate the reliability of the MICE method, this study conducted a comprehensive two-level sensitivity analysis. A non-parametric test (Wilcoxon rank sum test) was used to compare the statistical distribution of each variable before imputation (only using true observations) and after imputation (using complete imputation values), in order to verify that the original data structure was effectively retained [21]. AUC was used to evaluate predictive performance and to select the optimal cutoff by maximizing the Youden index (sensitivity + specificity − 1). DCA and precision–recall (P–R) curve analysis were also performed. Two-tailed P values < 0.05 were considered statistically significant.

Role of the funding source

Supported by the National Natural Science Foundation of China, General Program, Grant No. 82,170,733, this study received no funder involvement in design, data collection, analysis, interpretation, or manuscript preparation. Authors independently decided to submit the manuscript, having full access to all study data.

Results

Patient characteristics

During the research period, a total of 464,459 hospitalized children were recorded in the Children’s Hospital Affiliated to Nanjing Medical University. Among them, 106,041 cases were identified as suspected infections because they received systemic antibiotic treatment simultaneously within 24 h after admission and completed microbiological sampling. After excluding 13,409 individuals who did not meet the criteria, a total of 92,632 cases were ultimately included in the suspected infection cohort. Researchers retrospectively calculated the Phoenix Sepsis Score (PSS) of the above-mentioned children based on the 2024 Phoenix Sepsis Diagnostic Criteria. Among them, 2,477 cases with a Phoenix sepsis score of ≥ 2 were diagnosed as childhood sepsis and included in the main cohort of this study (Supplemental Table S3 and S4). The overall research process and algorithm demonstration are illustrated in Fig. 1 and Supplemental Figure S1.

The median age of this cohort was 26.0 months (IQR 5–78 months), 58.5% (1,448/2477) were male, and the average body mass index (BMI) was 16.48 kg/m². The in-hospital mortality rate was 8.0% (199/2,477), and the median length of hospital stay was 5 days (IQR 2 to 13.5 days) (Supplemental Table S5). Compared with children suspected of non-sepsis infection, there was no statistically significant difference in gender and PaCO₂ between the two groups (P > 0.05), while there were statistically significant differences in the remaining clinical indicators (P < 0.05) (Supplemental Table S5).

To evaluate the diagnostic performance of the 2024 Phoenix standard, we simultaneously applied the 2005 IPSCC standard for comparative analysis. A total of 19,801 children met the IPSCC criteria or their combined criteria, among which only 1,289 cases simultaneously met the dual criteria of Phoenix and IPSCC. A total of 1,188 cases unique to the Phoenix standard and 18,512 cases unique to the IPSCC standard were identified respectively. The distribution of mortality rates further highlights the differences between the two criteria, showing that among the 2,477 confirmed cases by the Phoenix criterion, 199 patients (8.0%) died, whereas among the 19,801 cases confirmed by the IPSCC criteria, 175 cases (0.9%) died. Among the 1,289 cases with overlapping double criteria, 144 cases (11.2%) died (Supplementary Table S6).

Feature selection

The data set was randomly divided into 2 parts, with 70% (n = 1735) of the data allocated for model training and the remaining 30% (n = 742) used for model testing. (Supplementary Figure S5) In order to avoid multicollinearity between features affecting prediction accuracy, when the correlation coefficient of two features in Spearman correlation analysis was > 0.7, the features with low correlation with outcome were excluded, and characteristics with missing values greater than 30% were excluded to reduce bias due to missing data (Supplementary Table S7 and S8). Finally, 37 characteristics were analyzed, including Age, Gender, Height, BMI, IMV, SpO₂, S/F, Vasoactive Drugs, DBP, MAP, PLT, INR, FIB, GCS, CRRT, Tem, HR, RR, SCr, WBC, TBIL, RBP, PCT, RDWSD, MPV, APTT, ALT, CK, UA, Urea, CysC, LDH, K, Na, Cl, Mg and P.

To ensure the robustness of the screening results of predictors, three complementary methods were adopted in this study for feature screening. LASSO regression screened out 14 variables with non-zero coefficients. The Boruta algorithm, a wrapped algorithm based on random forests, confirmed that 29 variables were important. Meanwhile, based on the random forest importance index, the top 19 variables that exceed the preset threshold are screened out. The intersection of the screening results from the three methods ultimately yielded 12 relatively robust predictors, namely IMV, S/F, MAP, PLT, INR, FIB, GCS, CRRT, HR, UA, LDH, and Mg (Table 1, Supplemental Figure S6 and Table S11).

Table 1.

Main variables for patients with sepsis in Train set (N = 1735)

Variables	Overall (n = 1735)	Survivors (n = 1595)	Non-survivors (n = 140)	P value
Demographic data
Age, month	25.00 (4.50, 77.00)	24.00 (4.00, 77.00)	34.50 (9.00, 72.50)	0.08
≤ 1	210 (12.1)	200 (12.5)	10 (7.1)	0.244
1–12	456 (26.3)	422 (26.5)	34 (24.3)
12–60	525 (30.3)	474 (29.7)	51 (36.4)
60–144	413 (23.8)	380 (23.8)	33 (23.6)
≥ 144	131 (7.6)	119 (7.5)	12 (8.6)
Gender (male), n (%)	1022 (58.9)	935 (58.6)	87 (62.1)	0.474
BMI, kg/m2	16.53 (14.94, 18.51)	16.49 (14.86, 18.47)	16.94 (15.37, 18.89)	0.076
Selected variables
MAP, mmHg	62.00 (54.00, 70.00)	63.00 (55.00, 71.00)	53.00 (43.25, 62.00)	< 0.001
FIB, mg/dL	228.00 (146.00, 333.00)	236.50 (154.75, 340.00)	138.00 (81.00, 256.00)	< 0.001
PLT, 109/L	210.00 (80.00, 346.00)	226.50 (85.00, 352.75)	114.50 (49.25, 207.50)	< 0.001
S/F	285.71 (211.11, 360.00)	303.18 (230.00, 368.00)	179.00 (110.50, 252.01)	< 0.001
INR	1.25 (1.09, 1.47)	1.24 (1.09, 1.44)	1.46 (1.21, 2.01)	< 0.001
HR, bpm	157.00 (142.00, 174.00)	156.00 (142.00, 173.00)	168.00 (149.00, 193.00)	< 0.001
LDH, U/L	395.00 (291.00, 655.00)	380.50 (286.00, 599.00)	778.00 (450.00, 1621.00)	< 0.001
UA, µmol/L	260.00 (180.00, 380.00)	435.00 (307.50, 616.50)	250.50 (175.00, 357.25)	< 0.001
Mg, mmol/L	0.92 (0.83, 1.01)	0.91 (0.83, 1.00)	1.02 (0.91, 1.15)	< 0.001
IMV, n (%)	674 (38.8)	546 (34.2)	128 (91.4)	< 0.001
CRRT, n (%)	118 (6.8)	75 (4.7)	43 (30.7)	< 0.001
GCS, points	15.00 (13.00, 15.00)	15.00 (13.00, 15.00)	6.00 (4.00, 13.00)	< 0.001

Demographic and clinical variables for 1,735 pediatric sepsis patients in the training cohort, stratified by survival status (survivors: n = 1,595; non-survivors: n = 140)

Model construction and evaluation

Data collected within the first 24 h of admission were used to screen 12 variables in the training set, which were then applied to develop 8 machine learning models to predict in-hospital mortality in children with sepsis.

In the evaluation of eight machine learning architectures within the training cohort, the CatBoost algorithm emerged as the best-performing model for predicting in-hospital mortality in pediatric patients with sepsis. This model achieved a superior area under the receiver operating characteristic curve of 0.889 and a global accuracy of 92.9% when evaluated on the independent test set. At the default decision threshold of 0.5, the model exhibited a pronounced specificity of 98.8%, which serves a vital clinical function by ensuring high reliability in positive predictions—reflected by a positive predictive value of 63.6%—and potentially mitigating the risk of alarm fatigue within intensive care environments. Nevertheless, this stringent threshold resulted in a relatively low sensitivity of 23.7%, which creates an inherent clinical risk of failing to identify a significant proportion of high-risk patients during the critical early stages of admission. As described in Table 2, the sensitivity, specificity, PPV, and NPV of the CatBoost model were 23.7%, 98.8%,63.6% and 93.8%, respectively.To enhance clinical utility and prioritize patient safety through improved risk stratification, the decision threshold was recalibrated using the maximum Youden Index. By adjusting the cutoff value to 0.0519, the sensitivity of the CatBoost model improved substantially to 83.05% while maintaining a balanced specificity of 79.65%. (Supplemental Figure S8, Table S9, Table S10 and Figure S13) This optimized operating point provides a broader early warning window, enabling clinicians to implement timely interventions and intensive monitoring for the majority of high-risk cases, which is essential for reducing the risk of in-hospital death in pediatric sepsis management.

Table 2.

Performance metrics of machine learning models for predicting in-hospital mortality in pediatric sepsis

Model	AUC	Accuracy	Sensitivity	Specificity	PPV	NPV	F1	Youden	Brier
Train cohort
CatBoost	0.942	0.95	0.479	0.992	0.838	0.956	0.609	0.47	0.039
LightGBM	0.924	0.821	0.871	0.817	0.295	0.986	0.44	0.688	0.12
BRT	0.912	0.858	0.757	0.867	0.333	0.976	0.463	0.624	0.095
XGBoost	0.915	0.871	0.757	0.882	0.359	0.976	0.487	0.639	0.088
AdaBoost	0.912	0.867	0.743	0.878	0.348	0.975	0.474	0.621	0.093
SVM	0.91	0.833	0.821	0.834	0.303	0.982	0.442	0.655	0.115
MLP	0.914	0.816	0.85	0.813	0.285	0.984	0.427	0.663	0.123
ANN	0.941	0.867	0.871	0.867	0.365	0.987	0.515	0.739	0.096
Test cohort
CatBoost	0.889	0.929	0.237	0.988	0.636	0.938	0.346	0.226	0.054
LightGBM	0.885	0.829	0.746	0.836	0.282	0.974	0.409	0.582	0.118
BRT	0.882	0.864	0.678	0.88	0.328	0.969	0.442	0.558	0.095
XGBoost	0.882	0.872	0.661	0.89	0.342	0.968	0.451	0.551	0.091
AdaBoost	0.882	0.849	0.644	0.867	0.295	0.966	0.404	0.511	0.099
SVM	0.881	0.823	0.763	0.829	0.278	0.976	0.407	0.591	0.117
MLP	0.879	0.818	0.729	0.826	0.265	0.972	0.389	0.555	0.122
ANN	0.878	0.845	0.712	0.857	0.3	0.972	0.422	0.568	0.107

Performance metrics of ten machine learning models evaluated on the training (n = 1,735) and testing (n = 742) cohorts for predicting in-hospital mortality in pediatric sepsis patients. Metrics include area under the receiver operating characteristic curve (AUROC), accuracy, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), F1 score, Youden index, and Brier score

The ROC curves for the top 12 features of all ML models are shown in Fig. 2 and Supplemental Figure S7. The ROC curves for both training and test sets are presented in Fig. 2A and B, with CatBoost and LightGBM demonstrating the highest AUC values, indicating their superior classification ability. The precision-recall curves (test set) in Fig. 2C reveal that while all models exhibit a decline in precision with increasing recall, CatBoost maintains notably high precision across a broad range of recall levels, suggesting robust performance on imbalanced data. The calibration curves of the test set (Fig. 2D) show that CatBoost aligns most closely with the diagonal, reflecting excellent calibration reliability, whereas other models such as MLP deviate noticeably, indicating systematic overestimation or underestimation of probabilities. Furthermore, in the decision curve analysis for both training (Fig. 2E) and test (Fig. 2F) sets, CatBoost consistently achieves the highest net benefit across the majority of clinically relevant threshold probabilities, supporting its strong potential for practical application.

Fig. 2.

Performance evaluation of machine learning models for predicting pediatric sepsis mortality. This figure presents the receiver operating characteristic (ROC) curves for eight machine learning models—CatBoost, LightGBM, Boosted Regression Trees (BRT), eXtreme Gradient Boosting (XGBoost), AdaBoost, Support Vector Machine (SVM), Multi-Layer Perceptron (MLP), and Artificial Neural Network (ANN)—in predicting mortality among pediatric sepsis patients

Furthermore, a comprehensive sensitivity analysis confirmed the reliability of the data imputation strategy in this study. The Wilcoxon rank sum test results show that the median and interquartile range of the imputation values are completely consistent with the natural distribution of the observed values, successfully preserving the original data structure without introducing significant statistical bias. Crucially, when evaluating the impact of imputed data on model performance on the complete case test set, the performance of the data-imputed model was comparable to that of the complete data model (AUC under the curve 0.844 vs. 0.820), and the DeLong test results showed no statistically significant difference between the two (P = 0.1118). This result confirms that the MICE method does not introduce artificial noise, and the model constructed based on imputed data still has good generalization ability in strictly true and fully observed data. (Supplemental Figure S3 and S4)

Model explanation and application

To address clinicians’challenges with opaque prediction models, we employed the SHAP method to enhance the interpretability of the CatBoost model by quantifying each feature’s contribution to individual predictions. The feature importance analysis, based on mean absolute SHAP values, revealed that IMV was the most influential predictor, followed by GCS, PLT, UA, S/F and Mg in descending order of importance (Fig. 3A(a)). Furthermore, the SHAP beeswarm plot (Fig. 3A(b)) visualized the direction and magnitude of each feature’s impact on model output, demonstrating that high values of IMV (= 1), GCS (lower scores) were associated with increased mortality risk, while elevated MAP values appeared protective.

Fig. 3.

Local SHAP analysis for individual patient predictions. SHAP waterfall plots for two representative patients in the test cohort, illustrating the contribution of each feature to the CatBoost model’s prediction of in-hospital mortality. B (a) Survival case: the model predicted low mortality probability with f(x) = -1.72, mainly driven by favorable features including IMV = 0, UA = 132 µmol/L, PLT = 229 × 10⁹/L, LDH = 254 U/L, HR = 138 bpm, and GCS = 15. The patient truly survived. B (b) Death case: the model predicted high mortality probability with f(x) = 5.4, primarily driven by adverse features including GCS = 4, HR = 201 bpm, IMV = 1, Mg = 1.2 mmol/L, CRRT = 1, FIB = 88 g/L, and LDH = 2056umol/L. The patient truly died

For individual case interpretation, we generated SHAP waterfall plots to illustrate how key features drove specific predictions for two representative cases (Fig. 3B). In the survival case (Fig. 3B(a)), favorable clinical indicators including IMV = 0, UA = 132 umol/L, and PLT = 229 × 10⁹/L collectively shifted the prediction toward survival. Conversely, in the mortality case (Fig. 3B(b)), critical conditions including GCS = 4, HR = 201 bpm, IMV = 1, Mg = 1.2mmol/L and CRRT = 1 synergistically contributed to the high mortality prediction.

To further elucidate the functional relationships between the 12 selected predictors and the risk of in-hospital mortality, PDPs were generated for each feature (Supplemental Figure S11). These plots demonstrate that most variables exhibit predominantly nonlinear associations, underscoring the advantage of the tree-based CatBoost model over conventional linear models. Of note, some PDPs show wide confidence intervals or irregular fluctuations in sparse regions. These regions contain limited training data, and the risk estimates therein should be interpreted with caution. These visualization results confirm that most clinical predictors do not have a simple linear relationship with mortality. In neurological assessment, the risk of death remains relatively stable when the GCS score is high, but there is a sharp and non-linear increase when the score drops below 9, which may reflect a shift towards severe brain dysfunction. Hemodynamic parameters also show a similar pattern, when the MAP drops below 50 mmHg, the risk of death increases sharply, highlighting the critical window period for intervention in the management of septic shock. Metabolic and laboratory indicators also showed significant non-proportional effects. The uric acid level shows a gradually increasing risk trend between 200 and 400 mmol/L, and then reaches a risk plateau period. The magnesium level shows a clear positive correlation. When the value exceeds 1.0 mmol/L, the predicted probability of death will rise rapidly. Regarding coagulation function, platelet count shows a protective effect and tends to stabilize when the value exceeds 200 × 10⁹/L, while the increase in the INR value indicates a stepwise rise in risk after exceeding the 1.6 threshold.

To facilitate clinical translation and point-of-care application, we operationalized the CatBoost model into a publicly accessible, web-based risk calculator (https://sepsis-prediction2-njch.ndemo.net/) (Supplemental Figure S12). This tool provides clinicians with an immediate, individualized mortality risk estimation for pediatric sepsis patients by inputting key clinical parameters, thereby offering valuable, data-driven guidance for early risk stratification and clinical decision-making.

Discussion

This study diagnosed children with sepsis based on the 2024 Phoenix Sepsis Criteria, developed and completed the internal validation of a machine learning model based on CatBoost, which can be used for the early prediction of in-hospital mortality in pediatric sepsis patients [6]. The model only uses clinical and laboratory variables collected within the first 24 h after the child’s admission. It demonstrated excellent discriminatory performance in an independent test set of 742 patients, with the AUC reached 0.889, and the accuracy was 92.9%, outperforming 7 other advanced machine learning algorithms [14]. Through SHAP analysis, the study clarified the contributions of features at both the overall and individual levels, further enhancing the interpretability of the model, and identified IMV, GCS, PLT, UA, and S/F as the most influential predictive indicators. Currently, this optimized model has been deployed as a publicly accessible web calculator to support bedside risk stratification.

Traditional sepsis scoring systems (such as PRISM Ⅲ, PIM2/3 and PELOD-2) have been proven to have certain prognostic differentiating abilities in clinical applications, with the area under the curve (AUC) ranging from 0.75 to 0.93 [22–25]. However, these systems often rely on parameters that consume a large amount of resources (such as vasoactive-inotropic score and oxygenation index) or require serial measurements over an extended period [25, 26]. Early studies utilizing machine learning for pediatric sepsis predominantly relied on the 2005 International Pediatric Sepsis Consensus Conference (IPSCC) criteria, yielding AUC values ranging from 0.82 to 0.92 [27, 28]. Nevertheless, these studies often included variables with high missing rates or features derived from adults, thereby limiting the early applicability and generalizability of the models [26, 28]. With the update of the diagnostic Criteria for Sepsis, in recent years, related studies have gradually shifted to constructing prognostic assessment models based on the 2024 Phoenix Sepsis Criteria [5, 6, 29]. Moore and colleagues [14] validated several machine learning algorithms for sepsis in Phoenix in a PICU cohort. The results showed that their AUC values ranged from 0.81 to 0.88, confirming the feasibility of model construction under this standard. It should be noted that the deep learning models such as convolutional neural networks proposed by Huang et al., although reported to have AUC values ranging from 0.89 to 0.97, demonstrating higher discriminative efficiency, have high requirements for data and rely on continuous and high-frequency streaming data of EHR for support [30].

Several key methodological optimizations enable our CatBoost model to achieve consistent accuracy while maintaining practical applicability. Firstly, this study strictly adhered to the 2024 Phoenix Sepsis criteria to define the research cohort, ensuring the diagnostic accuracy of the research subjects, making the model results more in line with the current clinical diagnosis and treatment status of pediatric sepsis, and enhancing its clinical applicability. Secondly, in the feature screening stage, this study abandon-the traditional “black box” modeling approach of machine learning and adopts a strict screening strategy that integrates the LASSO algorithm, Boruta algorithm, and random forest algorithm. This effectively eliminates redundant variables and data noise, reduces the risk of model overfitting, and enhances model stability. Ultimately, the CatBoost algorithm can fully exert its prognostic assessment value by merely utilizing the 12 clinical variables routinely collected within 24 h of the child’s admission. Meanwhile, through SHAP analysis, the intrinsic prediction logic of the model is visually interpreted, making the prediction mechanism of the model completely transparent to clinicians and further enhancing the clinical acceptability of the model.

The key predictors identified in this study are both consistent with well-defined pathophysiologic mechanisms and highlight some markers that may be underestimated. IMV = 1 and a decreased GCS reflect severe respiratory and neurological dysfunctions, these are not only core elements of the Phoenix Sepsis Criteria but also recognized independent risk factors for death [6].A decrease in mean arterial pressure (MAP) and thrombocytopenia indicate impairments in the cardiovascular and hematological systems [5, 6]. Notably, elevated UA and Mg have been identified as significant prognostic factors in our model. In critically ill patients, hyperuricemia may suggest tissue hypoperfusion, oxidative stress, and abnormal purine metabolism, and previous studies have shown its association with poor prognosis in pediatric cohorts [31, 32].Basic research has found that when the level of soluble uric acid abnormally rises, it can act as a damage-associated molecular pattern (DAMP), gradually amplifying the typical systemic inflammatory response of sepsis by activating the NLRP3 inflammasome [33, 34]. Although hyperuricemia has been closely associated with poor prognosis in previous studies on critically ill children and adults with severe sepsis, considering the current situation that international guidelines have not yet included UA as a routine monitoring indicator for pediatric sepsis [35], its role in the machine learning model of this study is primarily exploratory, warranting further prospective validation. Hypermagnesemia has been found to be associated with increased mortality in adult sepsis datasets (e.g., the MIMIC - IV database, Chinese multicenter studies) and in pediatric populations with acute kidney injury, and its mechanism of action may be related to neuromuscular blockade, vasodilation, or renal function impairment [36, 37]. These findings suggest that beyond conventional organ dysfunction scores, routinely measured but often overlooked metabolic and electrolyte parameters may further refine early risk stratification.

The clinical utility of these predictors is further refined by SHAP partial dependence plots, which demonstrate that the majority of variables exhibit predominantly non-linear associations with in-hospital mortality. These plots reveal critical clinical thresholds, such as a sharp escalation in mortality risk when the GCS score falls below 9 or when MAP drops below 50 mmHg. Recognizing these non-linear inflection points allows the CatBoost model to outperform traditional linear models by capturing actionable physiological boundaries.

However, the core challenge faced in the process of transforming this model into clinical practice lies in the clinical balance between sensitivity and specificity. At the default decision threshold (0.5), although the CatBoost model can achieve a high specificity of 98.8%, its sensitivity is only 23.7%. High specificity can effectively ensure the accuracy of positive early warnings and reduce the ineffective alarms that are prone to occur in the clinical work of PICUs. However, it is important to note that too low sensitivity can easily lead to false negatives and missed diagnoses in high-risk children. If such potential high-risk children are not identified in time, it will directly delay the early intervention opportunity [38]. To maximize the safety of the children patients and give full play to the early warning value of the model, we combined the actual clinical needs and optimized and adjusted the decision threshold of the model based on the maximum Youden Index, and finally determined it to be 0.0519. After adjustment, the model maintained a good specificity of 79.65% while significantly increasing its sensitivity to 83.05%. This ensures that the vast majority of children at risk of death are effectively identified within the golden intervention window in the early stage of admission, providing reliable support for the timely implementation of targeted treatment. In subsequent research, we plan to further introduce cost-sensitive learning strategies, set higher penalty weights for false negative results during the model training stage, continuously optimize the model’s recognition efficiency for high-risk children from the algorithmic level [39], and further enhance its clinical applicability.

Despite the advances described above, this study still has several limitations that need to be considered. First, as a single-center retrospective study without external validation, the generalizability of our findings to other clinical settings and patient populations may be limited, and the consistency of predictive performance across diverse scenarios remains to be verified. Second, variables with a missing rate of more than 30% (such as arterial blood gas and lactate) were excluded from the study, which may have missed valuable physiological information. Third, the decision to use the worst values within the first 24 h, while pragmatic and aligned with clinical convention, may introduce potential bias. This strategy is susceptible to extreme outliers and fails to capture dynamic physiological trends. To mitigate this, we applied robust regression residual correction for age, excluded outliers beyond 2.5 standard deviations, and employed the CatBoost algorithm, which inherently exhibits greater robustness to outliers compared with linear models. However, the lack of longitudinal data remains a limitation, as it prevents the model from reflecting disease progression or early treatment response. Fourth, our model treats in-hospital mortality as a static binary outcome and does not explicitly account for time dependency. This simplification fails to distinguish between early death and late mortality, which likely involve different pathophysiological processes. Furthermore, in a time-to-event framework, discharge alive acts as a competing risk for in-hospital death, and our binary approach does not adjust for the differential exposure window created by varying lengths of stay. Consequently, the model may conflate distinct mortality phenotypes and could introduce bias if certain predictors are differentially associated with the likelihood of early discharge. Future work should explore survival-based extensions to generate individualized temporal risk profiles that can distinguish patients at immediate risk of early death from those whose risk accumulates more gradually. To overcome the inherent limitations of this single-center retrospective study and further promote the clinical transformation of research results, we plan to conduct follow-up research from the following four aspects. Firstly, independent datasets from different regions or medical institutions at different levels should be given priority for external validation of the machine learning model constructed in this study to further clarify its extrapolation and universality. Secondly, conduct multi-center prospective cohort studies to evaluate the real-time predictive efficacy of the model in real-world clinical practice, as well as its practical intervention value in guiding clinical decisions and improving the prognosis of children. Thirdly, to address the limitations of static 24-hour data, we plan to introduce longitudinal data such as continuous vital signs and serial laboratory measurements collected at multiple time points to develop a real-time early warning model that can dynamically reflect disease progression and treatment response. This will allow for progressively refined risk estimates as the clinical picture unfolds. Finally, Cost-sensitive learning strategies are explored at the algorithmic level. By assigning higher penalty weights to false negative results during the model training stage, the risk of missed diagnosis of high-risk children is minimized from the underlying algorithmic logic.

Conclusion

The CatBoost model, enhanced by SHAP analysis, provides accurate and interpretable predictions of in-hospital mortality in pediatric sepsis. Its deployment as a web-based tool supports clinical decision-making, with potential to improve patient outcomes through early risk identification.Subsequent investigations should emphasize prospective external validation across multiple centers to ensure the model’s reliability in broader patient cohorts and to explore its effective implementation into bedside clinical practice.

Supplementary Information

Below is the link to the electronic supplementary material.

Abbreviations

AUC

Area Under the Curve

CatBoost

Categorical Boosting

GCS

Glasgow Coma Scale

IMV

Invasive Mechanical Ventilation

IPSCC

International Pediatric Sepsis Consensus Conference

LASSO

Least Absolute Shrinkage and Selection Operator

PELOD

Pediatric Logistic Organ Dysfunction

PICU

Pediatric Intensive Care Unit

PLT

Platelet Count

PRISM

Pediatric Risk of Mortality

PSS

Phoenix Sepsis Score

ROC

Receiver Operating Characteristic

SHAP

Shapley Additive exPlanations

SOFA

Sequential Organ Failure Assessment

SpO₂

Pulse Oxygen Saturation

S/F

Oxygen Saturation Index

UA

Uric Acid

XGBoost

Extreme Gradient Boosting

Author contributions

Xue-Jing Yang: Conceptualization, Methodology, Software, Formal analysis, Writing-original draft, Visualization. Wei Gu: Conceptualization, Resources, Supervision, Writing-review and editing. Hong-Jun Miao: Investigation, Data curation, Validation. Dan Shi: Formal analysis, Writing-review and editing. Ya-Li Qian: Investigation. Meng-Yuan Wu, Xiao-He Chen: Software, Validation. Yan-Qun Sun and Xu-Hua Ge (Corresponding authors): Research supervision, methodological guidance, manuscript revision and finalization. All authors have read and approved the final manuscript.

Funding

This research was supported by the National Natural Science Foundation of China (Grant No. 82170733). The funder was not involved in the research design, data collection, analysis, interpretation or manuscript writing.

Data availability

The datasets generated and analyzed during the current study are available from the corresponding author upon reasonable request. To facilitate external validation and model optimization, we welcome academic exchange and collaboration. Researchers with relevant data may contact the corresponding author, Prof. Xuhua Ge (email: gexuhua@njmu.edu.cn) or Dr. Yanqun Sun (email: yanq_sun@163.com). We are prepared to share model parameters, training procedures, and analysis codes to support multi-center validation. Furthermore, we are actively seeking partnerships to initiate a prospective multi-center study.

Declarations

Ethics approval and consent to participate

This study adheres to the relevant ethical principles of the Declaration of Helsinki and was approved by the Research Ethics Committee of the Children’s Hospital of Nanjing Medical University on August 22, 2024 (Ethics Approval Number: 202407019-1). As this study was a retrospective analysis and the ethics committee waived the requirement for informed consent.

Consent for publication

Not applicable.

Competing interests

The authors have no competing interests to declare.