Diagnostic performance of machine learning in systemic infection following percutaneous nephrolithotomy and identification of associated risk factors

Abstract

Objective

This study aims to investigate the predictive performance of machine learning in predicting the occurrence of systemic inflammatory response syndrome (SIRS) and urosepsis after percutaneous nephrolithotomy (PCNL).

Methods

A retrospective analysis was conducted on patients who underwent PCNL treatment between January 2016 and July 2022. Machine learning techniques were employed to establish and select the best predictive model for postoperative systemic infection. The feasibility of using relevant risk factors as predictive markers was explored through interpretability with Machine Learning.

Results

A total of 1067 PCNL patients were included in this study, with 111 (10.4 %) patients developing SIRS and 49 (4.5 %) patients developing urosepsis. In the validation set, the risk model based on the GBM protocol demonstrated a predictive power of 0.871 for SIRS and 0.854 for urosepsis. Preoperative and postoperative platelet changes were identified as the most significant predictors. Both thrombocytopenia and thrombocytosis were found to be risk factors for SIRS or urosepsis after PCNL. Furthermore, it was observed that when the change in platelet count before and after PCNL surgery exceeded 30*109/L (whether an increase or decrease), the risk of developing SIRS or urosepsis significantly increased.

Conclusion

Machine learning can be effectively utilized for predicting the occurrence of SIRS or urosepsis after PCNL. The changes in platelet count before and after PCNL surgery serve as important predictors.

1. Introduction

With the advancements in medical technology, minimally invasive surgery has become a preferred alternative to traditional open surgery in many cases due to its advantages of reduced trauma, minimal bleeding, and faster recovery. Currently, PCNL is recommended as the primary treatment for large or complex kidney stones [1]. However, compared to other minimally invasive procedures like ureteroscopy, PCNL is associated with a higher risk of complications, including bleeding, tissue damage, infection, and thromboembolism [2,3]. Urinary tract infection is one of the most common complications following PCNL, with reported incidence rates as high as 37 % [4]. Even with postoperative antibiotic therapy, some cases may progress to urosepsis [5].

Sepsis is a systemic inflammatory response of the body that can lead to high mortality rates. Studies have shown that the mortality rate for severe sepsis can reach 50 %, and the mortality rate for urosepsis ranges from 20 % to 40 % [6,7]. Early clinical symptoms of sepsis are often subtle, and if corresponding pathophysiological changes occur, the optimal window for early intervention may be missed. Even with intensified treatment later on, the outcomes may not be satisfactory [8].

Therefore, it is crucial to identify specific predictive markers and enable early diagnosis of sepsis in order to guide the perioperative management of PCNL. This study aims to develop and evaluate a clinical risk model for early diagnosis of SIRS and urosepsis after PCNL using machine learning techniques. Additionally, the study will assess the accuracy and feasibility of perioperative clinical indicators as predictive markers.

2. Methods and materias

2.1. Research design and study subjects

This study collected 1067 patients with kidney stones who underwent PCNL in January 2016 and July 2022. Exclusion criteria included: lack of preoperative CT imaging data; combined with other surgical options; congenital malformations such as polycystic kidney disease; patients with other serious diseases such as tumors, blood system or immune system diseases.

2.2. Data collection

Patient general and clinical information was collected.

General information includes: gender, age, date of admission, height, weight, body mass index, etc.

Preoperative data includes: peripheral blood leukocytes, lymphocytes, neutrophils, platelets, hemoglobin, uric acid, urine leukocytes, urine nitrite, preoperative antibiotic use, stone score, etc.

Intraoperative data includes: operation time, intraoperative blood loss, etc.

Postoperative information is uniformly measured at 6 o'clock in the morning on the first day after surgery no matter before or during initial symptoms of SIRS appear, including: peripheral blood leukocytes, platelets, lymphocytes, neutrophils, blood pressure, heart rate, oxygenation, respiration, body temperature, Glasgow score, etc.

Diagnostic criteria for SIRS: white blood cell count <4000 or >12,000 cells/ul, body temperature >38 or <36 °C, heart rate >90 beats/min, respiratory rate >20 beats/min or PaCO 2 < 32 mmHg. Urosepsis was defined as qSOFA (Quick Sepsis-Associated Organ Failure Assessment) score ≥2 criteria: respiratory rate ≥22 breaths/min; altered mental status (GCS score <13); systolic blood pressure ≤100 mmHg [9,10].

2.3. Model development

We use R language to build the H2O machine learning platform for implementing machine learning [11]. To minimize the adverse impact of variable scale differences on model accuracy when using variable weighting and fitting, we performed (0–1) standardization on all values. We used the train_test_split function to randomly divide the data into a train set (70 %) and a valid set (30 %). During model training, we again split the train set (70 %) into a train set (50 %) and a test set (20 %), selecting the best model based on its fit to the test set. We used machine learning model methods based on five algorithms: generalized linear regression (glm), random forest (RF), gradient boosting (GBM), deep learning (deep learning), and Xgboost. The glm model used a grid search method for hyperparameter tuning, while other models used random search. We used K-fold cross-validation to evaluate each combination of hyperparameter values in the train set. In this study, we performed 5-fold cross-validation to help reduce overfitting and selection bias in the valid set. K-fold cross-validation is a method that randomly divides a given dataset into K folds, where each fold is used as a validation set and the other folds are used to train the model. This process is repeated K times until all folds have been used as validation sets. The hyperparameter combination that produces the best performance on the test set is then tested separately on the valid set.

2.4. Evaluation of model performance

On each dataset, we presented the AUC values of glm, GBM, Xgboost, RF, and deep learning for urinary sepsis and SIRS to demonstrate their performance on each dataset. Select the model with the best performance in both the test and valid sets for subsequent analysis. The model performance in valid set is displayed by the ROC curve.

2.5. Feature selection

We used the built-in function “feature importance” in H2O and analyzed the top-ranked features. This provides information about the relative contribution of the corresponding features to the model, by calculating the contribution of each feature to the model. To examine the individual effects of important parameters and their derived features, we used a widely used tool called SHapley Additive exPlanations (SHAP) to study the contribution of a feature to model predictions when it interacts with other features.

2.6. Statistical analysis

Statistical analysis was performed using R version 4.0.4 and SPSS. The model is visualized by h2o explain. A p < 0.05 was considered statistically significant [12].

3. Results

The number of cases with SIRS after PCNL surgery was 111, accounting for 10.4 % of all cases. The number of cases with urosepsis after PCNL was 49, accounting for 4.5 % of all cases.

The clinical characteristics of the patients are summarized and presented in Table 1. The characteristics of patients who developed SIRS included higher Preoperative white blood cell (WBC) (p = 0.002), higher Preoperative neutrophils(N)(p < 0.001), higher Preoperative neutrophil to lymphocyte ratio (NLR)(p = 0.001), operation time (p = 0.001), Platelet. difference.before.and.after.surgery (p = 0.045), height (p < 0.001),weight (p < 0.001), diabetes history (p = 0.01), positive Urine. nitrite (p < 0.001), positive Urine. culture (p < 0.001) and female gender (p < 0.001). The characteristics of patients who developed urosepsis included higher Preoperative. N (p = 0.01), higher Preoperative. NLR(p = 0.006), height (p < 0.001),weight (p = 0.014), positive Urine. nitrite (p < 0.001), positive Urine. culture (p < 0.001) and female gender (p = 0.001). Five methods have been applied to the construction of prediction models of SIRS or urosepsis after PCNL，they were deep learning, GBM, Generalized Linear Model (glm), random forest (RF), and extreme gradient boosting (Xgboost). We divided 1067 patients into a train set, a test set, and a valid set according to the ratio of 5:2:3. Through fitting the test set, we trained and constructed the model in the training set, and finally verified the model in the valid set. The predicted AUC for SIRS in train set, test set and valid set is 0.958,0.772,0.607 (deep learning),1,0.899,0.871 (GBM),0.85,0.776,0.669 (glm),1,0.791,0.784 (RF) and 1,0.807,0.8 (Xgboost) respectively. The predicted AUC for urosepsis in train set, test set and valid set is 1,0.866,0.782 (deep learning),1,0.937,0.854 (GBM),0.818,0.702,0.684 (glm),1,0.895,0.824 (RF) and 1,0.771,0.759 (Xgboost) respectively (Fig. 1 A-B). We selected a risk model based on the GBM protocol for interpretable analysis (Fig. 1 C-D). The analysis of variable importance indicated that the top ten risk factors related to SIRS were Platelet. difference.before.and.after.surgery, Preoperative platelets (PLT), Preoperative. NLR, Preoperative. serum.creatinine, Operation. time, Age, Preoperative hemoglobin (HB), body mass index (BMI), Preoperative lymphocytes(L), Preoperative. uricacid (Fig. 2A). The top ten risk factors related to urosepsis were Platelet. difference.before.and.after.surgery, Preoperative. HB, Preoperative. L, Preoperative. N, Preoperative. NLR, Preoperative. uricacid, Preoperative. serum.creatinine, Urine. nitrite, Operation. time, Age (Fig. 2B). The sharp diagram suggested that platelet changes before and after PCNL were crucial for the prediction of SIRS and urosepsis. Platelet changes were not linearly related to SIRS and Urosepsis, and both thrombocytosis and thrombocytopenia were risk factors for SIRS and urosepsis (Fig. 3A–B). In order to further explore the important role of platelet changes, we stratified the changes and conducted relevant analysis. When the absolute value of platelet change was greater than 30*10⁹/L, the risk of SIRS was highly increased (28 % vs 6 %, P < 0.001)(Fig. 4A), as was the risk of urosepsis (11%vs3%,p < 0.001)(Fig. 4C). The risk of SIRS (21%vs34%vs6%,p < 0.001)(Fig. 4B) or urosepsis (13%vs10%vs3%, p < 0.001)(Fig. 4D) was significantly increased when the platelets were decreased by more than 30*10⁹/L or increased by more than 30*10⁹/L.

Table 1.

Clinical characteristics in patients with SIRS and sepsis after PCNL.

	non-SIRS	SIRS	P.value	non-Urosepsis	Urosepsis	P.value
	(N = 956)	(N = 111)		(N = 1018)	(N = 49)
Age (years)
Mean (SD)	52.6 (12.5)	51.8 (12.3)	0.528	52.5 (12.4)	52.4 (14.1)	0.959
Median [Min, Max]	53.0 [21.0, 86.0]	52.0 [21.0, 86.0]		53.0 [21.0, 86.0]	52.0 [21.0, 86.0]
Preoperative.WBC (10^9/L)
Mean (SD)	6.79 (1.77)	7.56 (2.49)	0.002	6.86 (1.86)	7.23 (2.03)	0.218
Median [Min, Max]	6.58 [2.99, 15.3]	6.89 [3.13, 19.8]		6.62 [2.99, 19.8]	6.87 [3.13, 13.4]
Preoperative.L(10^9/L)
Mean (SD)	2.10 (0.713)	2.09 (0.948)	0.981	2.10 (0.737)	1.95 (0.815)	0.21
Median [Min, Max]	2.03 [0.270, 5.35]	1.97 [0.830, 5.32]		2.03 [0.270, 5.35]	1.79 [0.830, 3.92]
Preoperative.N(10^9/L)
Mean (SD)	3.93 (1.39)	4.57 (1.63)	< 0.001	3.97 (1.43)	4.48 (1.30)	0.01
Median [Min, Max]	3.69 [1.23, 12.6]	4.29 [2.01, 12.9]		3.74 [1.23, 12.9]	4.31 [2.01, 8.60]
Preoperative.NLR(10^9/L)
Mean (SD)	2.10 (1.25)	2.48 (1.11)	0.001	2.12 (1.24)	2.58 (1.08)	0.006
Median [Min, Max]	1.81 [0.523, 19.1]	2.27 [0.694, 7.91]		1.82 [0.523, 19.1]	2.27 [1.11, 5.61]
Preoperative.eosinophils(10^9/L)
Mean (SD)	0.190 (0.156)	0.201 (0.176)	0.516	0.191 (0.157)	0.188 (0.186)	0.897
Median [Min, Max]	0.150 [0, 1.39]	0.160 [0, 1.12]		0.150 [0, 1.39]	0.140 [0, 1.12]
Preoperative.basophil(10^9/L)
Mean (SD)	0.0471 (0.0505)	0.0524 (0.0520)	0.31	0.0473 (0.0507)	0.0551 (0.0494)	0.287
Median [Min, Max]	0.0300 [0, 0.600]	0.0300 [0, 0.260]		0.0300 [0, 0.600]	0.0400 [0, 0.170]
Preoperative.PLT(10^9/L)
Mean (SD)	274 (74.7)	289 (86.3)	0.07	275 (76.1)	278 (76.2)	0.788
Median [Min, Max]	269 [55.0, 596]	285 [101, 592]		271 [55.0, 596]	275 [110, 499]
Preoperative.HB(g/L)
Mean (SD)	136 (18.7)	133 (17.3)	0.09	135 (18.8)	134 (13.8)	0.424
Median [Min, Max]	137 [62.0, 394]	133 [66.0, 177]		137 [62.0, 394]	133 [98.0, 158]
Preoperative.serum.creatinine(umol/L)
Mean (SD)	109 (85.6)	104 (72.0)	0.542	109 (85.3)	92.6 (57.8)	0.065
Median [Min, Max]	87.0 [39.0, 1010]	83.0 [36.0, 461]		87.0 [36.0, 1010]	81.0 [50.0, 434]
Preoperative.uricacid(umol/L)
Mean (SD)	413 (103)	400 (110)	0.215	413 (103)	397 (112)	0.348
Median [Min, Max]	411 [147, 828]	382 [166, 749]		409 [147, 828]	375 [182, 749]
operation time(min)
Mean (SD)	113 (36.6)	127 (43.2)	0.001	113 (37.3)	125 (43.1)	0.083
Median [Min, Max]	115 [30.0, 305]	120 [50.0, 300]		115 [30.0, 305]	120 [55.0, 225]
Platelet.difference.before.and.after.surgery(10^9/L)
Mean (SD)	−0.964 (29.0)	−14.9 (72.0)	0.045	−2.68 (33.1)	3.04 (76.0)	0.602
Median [Min, Max]	−5.00 [-220, 197]	−15.0 [-230, 210]		−5.00 [-230, 197]	−2.00 [-181, 210]
height(m)
Mean (SD)	1.65 (0.0911)	1.60 (0.0758)	< 0.001	1.64 (0.0906)	1.59 (0.0798)	< 0.001
Median [Min, Max]	1.66 [0.650, 1.83]	1.58 [1.42, 1.78]		1.65 [0.650, 1.83]	1.58 [1.42, 1.76]
weight(Kg)
Mean (SD)	63.5 (12.2)	59.2 (11.0)	< 0.001	63.2 (12.2)	59.3 (10.5)	0.014
Median [Min, Max]	64.0 [31.5, 110]	59.0 [37.0, 86.0]		63.3 [31.5, 110]	60.0 [40.0, 80.0]
BMI(Kg/m2)
Mean (SD)	23.4 (5.01)	22.8 (3.20)	0.078	23.4 (4.93)	23.0 (3.16)	0.513
Median [Min, Max]	23.0 [12.6, 137]	22.8 [15.8, 33.3]		23.0 [12.6, 137]	23.3 [17.5, 33.3]
Diabetes(n,%)
no	831 (86.9 %)	86 (77.5 %)	0.01	879 (86.3 %)	38 (77.6 %)	0.129
yes	125 (13.1 %)	25 (22.5 %)		139 (13.7 %)	11 (22.4 %)
Previous.DJ.indwelled(n,%)
no	888 (92.9 %)	107 (96.4 %)	0.232	948 (93.1 %)	47 (95.9 %)	0.638
yes	68 (7.1 %)	4 (3.6 %)		70 (6.9 %)	2 (4.1 %)
Previous.nephrostomy(n,%)
no	884 (92.5 %)	108 (97.3 %)	0.09	945 (92.8 %)	47 (95.9 %)	0.589
yes	72 (7.5 %)	3 (2.7 %)		73 (7.2 %)	2 (4.1 %)
Urine.WBC(n,%)
≥50 cells/ul	673 (70.4 %)	70 (63.1 %)	0.138	714 (70.1 %)	29 (59.2 %)	0.142
<50 cells/ul	283 (29.6 %)	41 (36.9 %)		304 (29.9 %)	20 (40.8 %)
Urine.nitrite(n,%)
negative	844 (88.3 %)	75 (67.6 %)	< 0.001	890 (87.4 %)	29 (59.2 %)	< 0.001
positive	112 (11.7 %)	36 (32.4 %)		128 (12.6 %)	20 (40.8 %)
Urine.culture(n,%)
negative	775 (81.1 %)	66 (59.5 %)	< 0.001	817 (80.3 %)	24 (49.0 %)	< 0.001
positive	181 (18.9 %)	45 (40.5 %)		201 (19.7 %)	25 (51.0 %)
Antibiotics.before.surgery(n,%)
no	775 (81.1 %)	83 (74.8 %)	0.26	822 (80.7 %)	36 (73.5 %)	0.436
yes	181 (18.9 %)	28 (25.2 %)		196 (19.3 %)	13 (26.5 %)
Number.of.channels(n,%)
1	881 (92.2 %)	97 (87.4 %)	0.198	935 (91.8 %)	43 (87.8 %)	0.639
2	69 (7.2 %)	14 (12.6 %)		77 (7.6 %)	6 (12.2 %)
3	5 (0.5 %)	0 (0 %)		5 (0.5 %)	0 (0 %)
4	1 (0.1 %)	0 (0 %)		1 (0.1 %)	0 (0 %)
Gender(n,%)
male	591 (61.8 %)	47 (42.3 %)	< 0.001	620 (60.9 %)	18 (36.7 %)	0.001
female	365 (38.2 %)	64 (57.7 %)		398 (39.1 %)	31 (63.3 %)

SIRS, systemic inflammatory response syndrome; PCNL, percutaneous nephrolithotomy; SD, standard deviation; BMI, body mass index; WBC, white blood cell; L, lymphocytes; N, neutrophils; NLR, neutrophil to lymphocyte ratio; PLT, platelet; HB, hemoglobin; DJ, double J tube.

Fig. 1.

The predicted AUC for SIRS(A) and Urosepsis(B) in train set, test set and valid set based on five methods. ROC curves for SIRS(C) and urosepsis(D) in valid set based on GBM.

Fig. 2.

Variable importance for SIRS(A) and Urosepsis(B) in valid set based on GBM.

Fig. 3.

Sharp diagram for SIRS(A) and Urosepsis(B) in valid set based on GBM.

Fig. 4.

Chi-square test analysis of four-panel table based on platelet change stratification. A: Difference in incidence of SIRS based on absolute value of platelet change. B: Difference in incidence of SIRS based on true value of platelet change. C: Difference in incidence of Urosepsis based on absolute value of platelet change. D: Difference in incidence of Urosepsis based on true value of platelet change.

4. Discussion

The first report of the minimally invasive stone removal operation of PCNL was published by I Fernström and B Johansson in 1976 [13]. Over time, PCNL has become the recommended first-line surgical option for the treatment of large and complex kidney stones due to its expanding indications. However, urinary tract infection is a common complication of PCNL, which if not detected and treated promptly, can progress rapidly to SIRS or sepsis, and in severe cases, even lead to death [14,15]. The incidence of SIRS after PCNL ranges from 10 % to 30 %, while the incidence of urosepsis is approximately 2 %–7 % [16,17]. Our findings are consistent with previous reports, as we observed an incidence of 10.4 % (111/1067) for SIRS and 4.5 % (49/1067) for urosepsis in our study.

Early diagnosis and timely intervention play a crucial role in the management and outcome of urosepsis [18]. In recent years, interpretable machine learning has gained significant attention in the field of biology, as its powerful learning ability has proven effective in predicting disease outcomes and screening markers [[19], [20], [21]]. Various machine learning methods, such as logistic regression, artificial neural networks, and random forest, have been employed in research studies [22]. In our study, we utilized multiple machine learning methods and found that the GBM and Xgboost protocols yielded better diagnostic efficiency compared to the glm protocol. This could be attributed to the characteristics of platelet changes, which may not be adequately captured by linear correlation regression methods.

In our interpretable machine model, we heavily emphasize the changes in platelet levels before and after surgery. Numerous studies have investigated the role of platelets in sepsis. Thrombocytopenia (platelet count <100,000/μl) has even been used as a diagnostic criterion for sepsis and severe sepsis [9]. In sepsis, activated platelets aggregate on endothelial cells and interact with other factors to form clots [23]. This process appears to be a significant cause of thrombocytopenia in sepsis. Additionally, some scholars have suggested that myelosuppression caused by sepsis may contribute to thrombocytopenia [24]. Therefore, thrombocytopenia often indicates microcirculatory thrombosis, inadequate organ perfusion, and poor prognosis [[25], [26], [27]]. Studies have shown that the prognostic value of thrombocytopenia for sepsis is even superior to that of procalcitonin (PCT) [28]. In our study, we found that the difference in platelet levels before and after surgery, rather than the specific platelet count, is particularly noteworthy. When the decrease in platelet count reaches a certain threshold, the risks of systemic inflammatory response syndrome (SIRS) and urosepsis increase significantly, even if the true platelet count is still within the normal range (100–300 * 109/L). Our results also revealed another interesting phenomenon: thrombocytosis is also a significant risk factor for sepsis. Similarly, when the increase in platelet count reaches a certain threshold, the risks of SIRS and urosepsis increase significantly, even if the true platelet count is still within the normal range. Contrary to our general perception, studies have found that increased circulating immature platelets are positively correlated with mortality and severity of sepsis, and platelet production is increased in sepsis [29,30].

Although there is a lack of high-quality clinical research, some scholars have proposed that thrombocytosis may serve as an early indicator of sepsis [31,32]. In this study, we found that when the absolute change in platelet count exceeded 30*10^9/L, the risk of SIRS significantly increased (28 % vs 6 %, P < 0.001), as did the risk of urosepsis (11 % vs 3 %, p < 0.001). Furthermore, a decrease or increase in platelet count exceeding 30*10^9/L was found to significantly elevate the risk of SIRS (21 % vs 34 % vs 6 %, p < 0.001) or urosepsis (13 % vs 10 % vs 3 %, p < 0.001), respectively.

5. Study limitations

As a large single-center retrospective study, our data is inevitably subject to internal bias. Therefore, it is important to note that these results may not be applicable to other specific environments or populations, as the data primarily spanned from 2016 to 2022, potentially limiting the generalizability of our findings. Additionally, we did not conduct subgroup analysis, which prevents us from verifying the universality of these results among specific patient subgroups. Although our predictive model demonstrates high accuracy and discrimination, large-scale randomized controlled trials are still necessary to further validate our findings. While our machine learning approach has provided explainability through built-in functions, the specific mechanism remains somewhat unclear, and the full exploration of feature selection and its importance cannot be achieved through machine explainability alone. Moreover, our analysis of platelets as a marker is somewhat preliminary, and further clinical trials should be designed to explore platelet changes at different time points after PCNL. Additionally, in vivo and in vitro experiments should be conducted to investigate the role of platelets in the mechanism of sepsis.

6. Conclusion

In conclusion, we have confirmed the powerful predictive ability of machine learning and have developed a risk model to aid in the identification of high-risk individuals, which is beneficial for early recognition and diagnosis of SIRS and urosepsis. Furthermore, we consider postoperative changes in platelet count to be a potential predictive marker, either increased or decreased.

Ethics statement

The parts of the study that include human participants were reviewed and approved by Ethic Committee of the first affiliated hospital of sun yat sen university. Informed consent was not required for this study because this study does not disclose the privacy and personal identity information of the subjects, and the exemption from informed consent will not adversely affect the rights and health of the subjects.

Funding

This study was supported by grants from National Natural Science Foundation of China (Award Number: 81725016, 81872094, 81772718, 81602219, 81972376), Guangdong Provincial Science and Technology Foundation of China (Award Number: 2017B020227004, 2017A030313538).

Data availability statement

Data will be made available on request.

CRediT authorship contribution statement

Pengju Li: Writing – review & editing, Writing – original draft, Formal analysis, Data curation. Yiming Tang: Formal analysis, Data curation. Qinsong Zeng: Project administration, Data curation. Chengqiang Mo: Writing – review & editing, Writing – original draft. Nur Ali: Data curation. Baohua Bai: Data curation. Song Ji: Data curation. Yubing Zhang: Data curation. Junhang Luo: Writing – review & editing, Conceptualization. Hui Liang: Writing – review & editing, Formal analysis. Rongpei Wu: Writing – review & editing, Writing – original draft, Supervision, Project administration, Conceptualization.

Declaration of generative AI and AI-assisted technologies in the writing process

During the preparation of this work the author used chatgpt in order to improve language and redability. After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the content of the publication.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.