Comparing machine learning and logistic regression for acute kidney injury prediction in trauma patients: A retrospective observational study at a single tertiary medical center

Abstract

Acute kidney injury (AKI) is common in patients with trauma and is associated with poor outcomes. Therefore, early prediction of AKI in patients with trauma is important for risk stratification and the provision of optimal intensive care unit treatment. This study aimed to compare 2 models, machine learning (ML) techniques and logistic regression, in predicting AKI in patients with trauma. We retrospectively reviewed the charts of 400 patients who sustained torso injuries between January 2016 and June 2020. Patients were included if they were aged > 15 years, admitted to the intensive care unit, survived for > 48 hours, had thoracic and/or abdominal injuries, had no end-stage renal disease, and had no missing data. AKI was defined in accordance with the Kidney Disease Improving Global Outcomes definition and staging system. The patients were divided into 2 groups: AKI (n = 78) and non-AKI (n = 322). We divided the original dataset into a training (80%) and a test set (20%), and the logistic regression with stepwise selection and ML (decision tree with hyperparameter optimization using grid search and cross-validation) was used to build a model for predicting AKI. The models established using the training dataset were evaluated using a confusion matrix receiver operating characteristic curve with the test dataset. We included 400 patients with torso injury, of whom 78 (19.5%) progressed to AKI. Age, intestinal injury, cumulative fluid balance within 24 hours, and the use of vasopressors were independent risk factors for AKI in the logistic regression model. In the ML model, vasopressors were the most important feature, followed by cumulative fluid balance within 24 hours and packed red blood cell transfusion within 4 hours. The accuracy score showed no differences between the 2 groups; however, the recall and F1 score were significantly higher in the ML model (.94 vs 56 and.75 vs 64, respectively). The ML model performed better than the logistic regression model in predicting AKI in patients with trauma. ML techniques can aid in risk stratification and the provision of optimal care.

1. Introduction

Acute kidney injury (AKI) is common in critically ill patients with trauma and is associated with a high mortality rate, prolonged hospital stay, and increased costs.^[1–3] Furthermore, patients with AKI can develop chronic kidney disease and end-stage renal disease, leading to the need for renal replacement therapy.^[4] Early recognition of AKI in critically ill patients with trauma could be helpful in fluid resuscitation and avoiding nephrotoxic drugs.^[3,5] According to the quantitative increase in serum creatinine levels or decrease in urine output, the International Kidney Disease: Improving global outcomes organization has classified AKI into 3 stages.^[6] However, these classical biomarkers are not applicable for early diagnosis of AKI.^[4,7,8]

With recent advancements in computer technology, studies have reported that artificial intelligence (AI) can improve the performance of existing tests for diagnosing various diseases.^[9–11] Machine learning (ML) is a type of AI involving a set of algorithms that are not explicitly programmed but instead, learn and improve through a set of predefined objectives.^[11] Numerous AI algorithms have been developed to support predictive models that demonstrate improved accuracy compared to traditional multivariate regression models.^{[2,5,10–12]}

Using logistic regression, we previously analyzed the risk factors of AKI in major patients with trauma with torso injuries.^[3] We hypothesized that the prediction model using ML would have a better predictive performance in this patient population than the traditional logistic regression model. In this study, we aimed to investigate whether an ML-based prediction model can provide better accuracy in predicting AKI in critically ill patients with trauma with torso injuries compared to traditional logistic regression models, potentially leading to improved early recognition and management of AKI in this population.

2. Methods

2.1. Study population

This retrospective observational cohort study was conducted at a single-level I trauma center at the Chungbuk National University Hospital, South Korea. Overall, 2020 patients with trauma were admitted to the intensive care unit between January 2016 and June 2020. The inclusion criteria were patients with thoracic and/or abdominal injuries, age > 15 years, and survival > 48 hours. Patients with end-stage renal disease or with missing data were excluded.

This study was approved by the Institutional Review Board of the Chungbuk University Hospital (2023-05-036). The requirement for informed consent was waived by the review board owing to the retrospective nature of the study.

3. Study design

The patients were divided into 2 groups: AKI and non-AKI groups. We compared the baseline characteristics, clinical outcomes, and laboratory results between the 2 groups to identify potential risk factors. Finally, we selected 11 potential risk factors after the univariate analysis through least absolute shrinkage and selection operator regression.^[13] A logistic regression model to predict the development of AKI was established using stepwise selection, and an ML model was established using a decision tree with hyperparameter optimization using grid search and cross-validation (Fig. 1).

Figure 1.

Fow chart of data processing and analysis.

4. Data preprocessing and model establishment

We divided the original dataset into a training (80%) and a test set (20%). Both the ML and logistic regression models were trained on the training set and assessed on the test set. The imbalanced nature of the dataset leads to classifiers being biased toward the majority class.^[14] To overcome this issue, we used synthetic minority oversampling technique (SMOTE) preprocessing to balance the data and then conducted ML training.^[15]

In the grid search and cross-validation, the data were randomly divided into 10 equal-sized subsamples for training the model and then validated in the remaining subsamples. The hyperparameters were tuned, including “splitter,” “min_impurity_decrease,” “max_depth,” “min_samples_split,” and “max_features.” The best estimator model was evaluated using the test set. All the analyses were performed using Python version 3.10.5.

5. Model performance and comparison

The model was evaluated using a confusion matrix, area under receiver operating characteristic curve, area under the precision-recall curve, sensitivity (recall), positive predictive value (precision), accuracy, and F1 score. The F1 score is the harmonic mean of sensitivity and precision.^[16]

6. Study variables and definitions

Clinical and laboratory results were collected upon arrival in the emergency room. Hypotension was defined as a systolic blood pressure < 90 mm Hg. Thoracic and/or abdominal injuries such as hemothorax, pneumothorax, rib fractures, abdominal solid organ injury, and intestinal injuries were identified by a formal reading of computed tomography scans. During the first 24 hours after arrival at the hospital, the quantity of any fluid administered was obtained from the medical records, excluding the number of blood transfusions. The cumulative fluid balance for 24 hours was calculated by subtracting all outputs from the total volume of the infused fluid for 24 hours. AKI diagnosis was made based on the current improving global outcomes criteria,^[6] using serum creatinine (SCr) and urine output criteria during the initial 7 days. Baseline SCr level was used as the initial laboratory test result. In most patients, the baseline SCr level was unknown, but an SCr increase ≥ .3 mg/dL within 48 hours was the first criterion for diagnosis.^[17] Therefore, we studied patients who survived for > 48 hours during the observation period for AKI diagnosis.

7. Results

7.1. Clinical characteristics of study population

In total, 400 patients with trauma who met the inclusion criteria were included in this study. Of the 400 patients, 78 (19.5%) developed AKI (AKI group), and 322 (80.5%) did not develop AKI (non-AKI group). The differences in baseline characteristics between the 2 groups are shown in Table 1. The patients in the AKI group were significantly older than those in the non-AKI group (60 vs 53 years, P < .001). The injury severity score (ISS) showed that patients were more severely injured in the AKI group than in the non-AKI group (29.1 vs 24.5, P < .001). The abbreviated ISS for the chest showed significant differences between the 2 groups (Table 1).

Table 1.

Baseline characteristics of patients with torso injury.

		Total (n = 400)	AKI (n = 78)	Non-AKI (n = 322)	P value
Age (yr), mean ± SD		54.4 ± 17.2	60.0 ± 18.8	53.1 ± 16.5	.0001
Male, n (%)		285 (71)	55 (70)	230 (71)	.983
Underlying disease, n (%)
HTN		118 (30)	27 (35)	91 (28)	.334
DM		46 (12)	13 (17)	33 (10)	.163
CAoD		12 (3)	3 (4)	9 (3)	.906
CVA		14 (4)	4 (5)	10 (3)	.597
Injury mechanism, n (%)					.156
Pedestrian TA		72 (18)	18 (23)	54 (17)
In car TA		113 (30)	23 (32)	90 (29)
Motorcycle TA		45 (12)	9 (13)	36 (12)
Fall		97 (26)	11 (15)	86 (28)
Others		56 (15)	14 (19)	42 (14)
ISS, mean ± SD		25.4 ± 10.2	29.1 ± 11.2	24.5 ± 9.7	<.001
AIS score, n (%)
Chest	1	14 (4)	2 (3)	12 (4)	.044
	2	79 (20)	7 (9)	72 (22)
	3	253 (63)	53 (68)	200 (62)
	4	47 (12)	14 (18)	33 (10)
	5	7 (2)	2 (3)	5 (2)
Abdomen	1	1 (.2)	0 (0)	1 (.3)	.963
	2	195 (49)	36 (46)	159 (49)
	3	141 (35)	29 (37)	112 (35)
	4	57 (14)	12 (15)	45 (14)
	5	6 (2)	1 (1)	5 (2)

Values are presented as mean ± standard deviation or n (%).

AIS = abbreviated injury scale, AKI = acute kidney injury, CAoD = coronary artery occlusive disease, CVA = cerebrovascular accident, DM = diabetes mellitus, HTN = hypertension, ISS = injury severity score, TA = traffic accident.

The clinical and laboratory features are presented in Tables 2 and 3, respectively. The AKI group had greater proportions of patients with hypotension, hemothorax, and intestinal injury than the non-AKI group (50% vs 25%, P < .001; 50% vs 35%, P = .016; 32% vs 8%, P < .001, respectively). The AKI group was more likely to receive blood transfusions in the initial 4 hours and a greater volume of fluid infusion (Table 2).

Table 2.

Comparison of clinical parameters between patients with and without acute kidney injury.

	Total (n = 400)	AKI (n = 78)	Non-AKI (n = 322)	P value
Hemothorax, n (%)	150 (38)	39 (50)	111 (35)	.016
Pneumothorax, n (%)	174 (43)	41 (53)	133 (41)	.094
Rib fracture (s), n (%)	314 (79)	66 (85)	248 (77)	.442
Abdominal solid organ injury, n (%)	210 (53)	37 (47)	173 (54)	.405
Intestinal injury, n (%)	52 (13)	25 (32)	27 (8)	<.001
Transfusion 4 hours, n (%)	213 (53)	59 (76)	154 (48)	<.001
pRBC transfusions (U) 4 h, median (IQR)	0 (0–3)	4 (0–10)	0 (0–2)	<.001
Total volume of infused fluid 24 h (L), median (IQR)	4.7 (3.4–6.6)	6.4 (4.2–8.3)	4.3 (3.3–6.0)	<.001
Cumulative fluid balance at 24 hours (L), median (IQR)	1.9 (.7–3.6)	3.6 (2.2–5.6)	1.6 (.5–2.9)	<.001
Hypotension, n (%)	120 (30)	39 (50)	81 (25)	<.001
Use of vasopressor, n (%)	129 (32)	53 (68)	76 (24)	<.001
In-hospital mortality, n (%)	52 (13)	26 (33)	26 (8)	<.001

Values are presented as n (%) or median [interquartile range].

AKI = acute kidney injury, pRBC = packed red blood cell.

Table 3.

Comparison of laboratory findings between patients with and without acute kidney injury.

	Total (n = 400)	AKI (n = 78)	Non-AKI (n = 322)	P value
Hemoglobin (g/dL), Median (IQR)	12.9 (11.1 to 14.1)	12.2 (10.3 to 13.6)	12.9 (11.1 to 14.2)	.040
Mean corpuscular volume (fL), median (IQR)	94.3 (91.5 to 98.1)	95.6 (92.1 to 99.4)	94.1 (91.4 to 97.5)	<.039
Creatinine (μmol/L), median (IQR)	.9 (.74 to 1.10)	1.0 (.76 to 1.21)	.9 (.73 to 1.01)	.002
CPK (g/dL), median (IQR)	377 (222 to 670)	421 (230 to 683)	357 (219 to 662)	.361
pH, median (IQR)	7.4 (7.3 to 7.4)	7.3 (7.2 to 7.4)	7.4 (7.3 to 7.4)	<.001
Base deficit (mmol/L), median (IQR)	−4.0 (−7.6 to − 1.8)	−7.5 (−10.2 to − 2.7)	−3.8 (−6.8 to − 1.7)	<.001
Lactate (mmol/L), median (IQR)	3.0 (1.9 to 4.5)	4.2 (2.6 to 6.7)	2.9 (1.9 to 4.1)	<.001

Values are presented as median [interquartile range].

AKI = acute kidney injury, CPK = creatine phosphokinase.

The 2 groups had significant differences in hemoglobin, creatinine, base deficit, and lactate levels (Table 3).

7.2. Logistic regression model

We selected 11 potential risk factors, and the logistic regression analysis identified intestinal injury, cumulative fluid balance for 24 hours, and use of vasopressors as independent risk factors for AKI [adjusted odds ratio (OR), 3.03; 95% confidence interval (CI), 1.373–6.683; P = .006; OR, 1.000; 95% CI, 1.000–1.003; P = .004; OR, 2.781; 95% CI, 1.352–5.718; P = .005, respectively; Table 4].

Table 4.

Logistic regression model with stepwise variable selection

Variables	Univariate analysis		Multivariate analysis
	OR (95% CI)	P value	OR (95% CI)	P value
Age (yr)	1.018 (1.003–1.034)	.021
ISS	1.052 (1.026–1.079)	<.001
AIS_Chest	1.671 (1.160–2.402)	.005
Hypotension (SBP < 90 mm Hg, yes/no)	3.345 (1.954–5.726)	<.001
Hemothorax (yes/no)	1.917 (1.139–3.229)	.014
Intestinal injury (yes/no)	5.204 (2.773–9.764)	<.001	3.030 (1.373–6.683)	.006
pRBC transfusions (U) 4 h	1.234 (1.152–1.322)	<.001
Mean corpuscular volume (fL)	1.051 (1.000–1.100)	.0315
Cumulative fluid balance 24 h	5.742 (3.249–10.146)	<.001	1.000 (1.000–1.003)	.004
Lactate (mmol/L)	1.311 (1.171–1.467)	<.001
Use of vasopressors (yes/no)	7.612 (4.347–13.331)	<.001	2.781 (1.352–5.718)	.005

AIS_chest = abbreviated injury scale score of the chest, CI = confidence interval, ISS = injury severity score, OR = odds ratio, pRBC = packed red blood cell, SBP = systolic blood pressure.

7.3. ML model

The results of the best model of the decision tree with hyperparameter optimization using grid search and cross-validation are shown in Supplemental Digital Content 1, http://links.lww.com/MD/J530, and the feature importance using the Shapley additive explanations (SHAP) method is shown in Figure 2. The features are ordered according to their effects on the prediction. The colors of the points indicate whether a certain observation has a higher or lower value than that of other observations.^[18] Higher cumulative fluid balance within 24 hours, pRBC transfusion within 4 hours, lactate levels, and age had a positive impact on AKI prediction. The learning curves showed that the training scores decreased and the validation scores increased as the number of training examples increased (Fig. 3). By repeatedly training and validating the training set, the difference between the training and validation sets decreased as the training increased, indicating that the model became more generalized and robust.

Figure 2.

SHAP value of the machine learning model output. SHAP = shapley additive exPlanations, pRBC = packed red blood cells, ISS = injury severity score, AIS_chest = abbreviated injury scale score (chest).

Figure 3.

Learning curve of the decision tree model after hyperparameter tuning.

7.4. Model performance and comparison

The accuracy, precision, recall, F1 score, and confusion matrix for the 2 models validated by the test dataset are shown in Figure 3. The accuracy score showed no differences between the 2 groups; however, the recall and F1 score were significantly higher in the ML model (.94 vs 56 and.75 vs 64, respectively; Fig. 4). The area under receiver operating characteristic curve of the ML model was significantly higher than that of the logistic regression model (.912 vs840; Fig. 5).

Figure 4.

Confusion matrices of both models and accuracy, precision, recall, and F1 scores using the test dataset.

Figure 5.

Receiver operating characteristic curve for estimating the discrimination between the Logistic regression model and the machine learning model.

8. Discussion

In this study, we analyzed patients with torso injuries who were admitted to the intensive care unit, and the results showed a 19.5% incidence of AKI and a 4-fold increase in in-hospital mortality compared with non-AKI patients. Therefore, predicting AKI is important for risk stratification, and the ML model showed excellent performance in predicting AKI compared with the logistic regression model. In patients with torso injury with age 45 years, ISS of 25, abbreviated injury scale_chest score of 3, cumulative fluid balance for 24 hours of 2000 mL, lactate level of 2.5, mean corpuscular volume of 90 fL, pRBC transfusion within 4 hours of 1 U, positive hemothorax, intestinal injury, hypotension, and use of vasopressor, the AKI prediction rate was 66.7% (Supplemental Digital Content 2, http://links.lww.com/MD/J531). Based on these results, a system that sends electronic alerts and identifies at-risk patients through AI applications could be developed to monitor renal function.^[19,20]

Decision tree algorithms have been a basic and major field used since the early days of ML and are widely used in various fields, such as classification and regression.^[21] However, a simple decision tree model may have a low classification performance for other ML models, and it is difficult to process noisy datasets.^[10,21–23] To address these challenges, we used a dataset with no missing data, and feature selection was aided by the least absolute shrinkage, and selection operator regression method.^[13] In the unbalanced data, the specificity or local accuracy of the majority class exceeded that of the minority class. The SMOTE method is not just a simple oversampling approach but can also increase the number of minority class instances by introducing new minority class examples in the neighborhood, thus overcoming overfitting and helping the classifier improve its generalization on the testing data.^[15] Finally, we used hyperparameter optimization techniques such as grid search and cross-validation (Grid Search CV) from scikit-learn. To overcome the overfitting constraint in an ordinary grid search, stratified cross-validation is applied, where samples are randomly divided into k-folds, causing an increase in accuracy.^[22] The learning curves represent the model’s generalization performance as a function of the training set size.^[24]

We used the SHAP method to analyze the feature importance in the ML model. The use of vasopressors was the most important feature, followed by cumulative fluid balance within 24 hours and pRBC transfusion within 4 hours. The SHAP method intuitively shows the contribution of each feature and is effective in determining the importance of individual features.^[11,25] The confusion matrix was categorized into 4 groups based on the classification outcome and is most commonly used for evaluating binary classifiers.^[11] The F1 score is the harmonic mean of precision and recall. In general, the F1 score tends to be closer to smaller measures, and a high value of F1 scores indicates that both measures are reasonably high and convincing.^[16]

This study provides additional evidence to the finding of previous studies that ML models outperform logistic regression in predicting AKI.^[10,11,26] Although the XG Boost algorithm is known to exhibit better performance than other algorithms, it requires more time for model training.^[25,27] Our study demonstrated a good classification performance with only hyperparameter tuning through Grid Search CV, without using XG Boost or gradient boosting tree algorithms.

Our study had some limitations. First, this was a retrospective, single-center study. Our cohort was specific to patients with thoracic and/or abdominal trauma. Thus, they were not free from selection bias. The second limitation is the unbalanced nature of the dataset. We used the SMOTE technique to overcome this issue; however, it may affect model generalization and reliability. Third, our sample size was relatively small. The ML model requires additional data and external validation.

9. Conclusion

In this study, the ML model showed excellent performance in predicting AKI compared with the logistic regression model. ML will help realize the future of improved healthcare by better analyzing complex nonlinear relationships^[28] and enabling predictions, which in turn provide immediate results in personalized patient care.

Author contributions

Conceptualization: Hanlim Choi, Jin Young Lee.

Data curation: Jin Young Lee, Younghoon Sul, Seheon Kim, Jin Bong Ye, Jin Suk Lee, Suyoung Yoon, Junepill Seok, Jonghee Han, Jung Hee Choi, Hong Rye Kim.

Formal analysis: Hanlim Choi, Jin Young Lee.

Methodology: Jin Young Lee

Supervision: Jin Suk Lee, Suyoung Yoon, Junepill Seok.

Validation: Seheon Kim, Junepill Seok, Jonghee Han.

Writing – original draft: Hanlim Choi, Jin Young Lee, Younghoon Sul.

Writing – review & editing: Jin Young Lee, Younghoon Sul, Seheon Kim, Jin Bong Ye, Jin Suk Lee, Suyoung Yoon, Junepill Seok, Jonghee Han, Jung Hee Choi, Hong Rye Kim.