Construction and evaluation of a mortality prediction model for patients with acute kidney injury undergoing continuous renal replacement therapy based on machine learning algorithms

Abstract

Background

To construct and evaluate a predictive model for in-hospital mortality among critically ill patients with acute kidney injury (AKI) undergoing continuous renal replacement therapy (CRRT), based on nine machine learning (ML) algorithm.

Methods

The study retrospectively included patients with AKI who underwent CRRT during their initial hospitalization in the United States using the medical information mart for intensive care (MIMIC) database IV (version 2.0), as well as in the intensive care unit (ICU) of Huzhou Central Hospital. Patients from the MIMIC database were used as the training cohort to construct the models (from 2008 to 2019, n = 1068). Patients from Huzhou Central Hospital were utilized as the external validation cohort to evaluate the models (from June 2019 to December 2022, n = 327). In the training cohort, least absolute shrinkage and selection operator (LASSO) regression with cross-validation was employed to select features for constructing the model and subsequently established nine ML predictive models. The performance of these nine models on the external validation cohort dataset was comprehensively evaluated based on the area under the receiver operating characteristic curve (AUROC) and the optimal model was selected. A static nomogram and a web-based dynamic nomogram were presented, with a comprehensive evaluation from the perspectives of discrimination (AUROC), calibration (calibration curve) and clinical practicability (DCA curves).

Results

Finally, 1395 eligible patients were enrolled, including 1068 patients in the training cohort and 327 patients in the external validation cohort. In the training cohort, LASSO regression with cross-validation was employed to select features and nine models were individually constructed. Compared to the other eight models, the Lasso regularized logistic regression (Lasso-LR) model exhibited the highest AUROC (0.756) and the optimal calibration curve. The DCA curve suggested a certain clinical utility in predicting in-hospital mortality among critically ill patients with AKI undergoing CRRT. Consequently, the Lasso-LR model was the optimal model and it was visualized as a common nomogram (static nomogram) and a web-based dynamic nomogram (https://chsyh2006.shinyapps.io/dynnomapp/). Discrimination, calibration and DCA curves were employed to assess the performance of the nomogram. The AUROC for the training and external validation cohorts in the nomogram model was 0.771 (95%CI: 0.743, 0.799) and 0.756 (95%CI: 0.702, 0.809), respectively. The calibration slope and Brier score for the training cohort were 1.000 and 0.195, while for the external validation cohort, they were 0.849 and 0.197, respectively. The DCA indicated that the model had a certain clinical application value.

Conclusions

Our study selected the optimal model and visualized it as a static and dynamic nomogram integrating clinical predictors, so that clinicians can personalized predict the in-hospital outcome of critically ill patients with AKI undergoing CRRT upon ICU admission.

Introduction

Acute kidney injury (AKI) is one of the most common and severe complications in the intensive care unit (ICU) [1–3]. Despite some advances in treatment and management, the mortality rate remains high among critically ill patients with AKI, estimated at around 28% [4,5]. In the ICU, approximately 33–67% of critically ill adult patients experience AKI, and about 10% of patients with require renal replacement treatment (RRT) [6,7]. Research has shown that critically ill patients with AKI face a higher risk of mortality compared to those without AKI [8]. Furthermore, patients receiving continuous renal replacement therapy (CRRT) exhibit a relatively higher mortality rate, with short-term mortality exceeding 50% [9–11], posing a significant threat to public health. Therefore, investigating how to early predict the survival of patients with AKI undergoing CRRT is crucial for timely identification of high-risk patients and merits further study.

In recent years, the exploration of factors contributing to the mortality risk in such patients has become one of the focal points of research. In 2021, a cohort study including 626 critically ill patients with AKI undergoing RRT [12], concluded that sequential organ failure assessment (SOFA) score, Charlson score and inter-hospital transfer were independent predictors of in-hospital mortality in such patients. In addition, researchers have explored a series of factors, including albumin-corrected anion gap [13], hyperphosphataemia [14], hypernatremia [15], hypoalbuminaemia [16], growth differentiation factor-15 [17] and neutrophil gelatinase-associated lipocalin [18] as independent predictors of mortality in critically ill patients with AKI undergoing CRRT. However, these factors have not yet gained widespread clinical application.

In recent years, with the widespread application of machine learning (ML) algorithms in the medical field, constructing clinical prediction models to forecast prognosis of patients using information data from electronic medical record systems or large clinical databases has become a current research focus. Scholars are currently focusing on building prognostic models for critically ill patients with AKI [19,20]. However, there is a relative scarcity of research on clinical prognosis prediction models specifically tailored for the subgroup of critically ill patients with AKI undergoing CRRT, and the research also has existed certain limitations, such as the data come from a single centre and lack external verification. One study used ML methods to predict RRT-free survival in critically ill patients with AKI requiring CRRT. The optimal prediction model was finally obtained. However, the study population was only patients in the medical information mart for intensive care (MIMIC) database and no external validation was performed [21]. In another study that used ML algorithms to construct and verify a prognostic model for patients with AKI undergoing CRRT, the final ML model was screened out and concluded that the ML model was better at predicting in-hospital mortality than acute physiologic assessment and chronic health evaluation II (APACHE II), SOFA, etc., but the training cohort and external validation cohort of this study were both from a single centre, and the model was not visualized or aligned [22]. It can be seen that this area is worthy of further exploration.

Therefore, the study retrospectively analysed clinical data of patients with AKI undergoing CRRT from the MIMIC database IV (Version 2.0) in the United States. Nine ML algorithms were applied to construct a risk prediction model for in-hospital mortality during ICU admission for such patients. The model was validated using a separate cohort of patients from Huzhou Central Hospital, serving as an external validation population. The optimal model was selected, and its generalizability and applicability were further verified.

Materials and methods

Data source and study population

The study retrospectively collected critically ill patients with AKI receiving CRRT during their first hospitalization. These patients were sourced from two hospitals: (1) the MIMIC IV database (Version 2.0 [23]); (2) Huzhou Central Hospital. Following the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) statement, we developed and validated the model.

We selected critically ill patients with AKI undergoing CRRT from the MIMIC IV database (Version 2.0) between 2008 and 2019 as the training cohort, comprising a total of 1068 cases. The database was approved by the Institutional Review Boards of Beth Israel Deaconess Medical Center and the Massachusetts Institute of Technology, and one author of the study was granted access to the database (ID number: 42303155 and 53446653).

For external validation cohort, we chose critically ill patients with AKI undergoing CRRT who were admitted to the ICU at Huzhou Central Hospital. The validation cohort consisted of 327 cases, and data collection occurred during the period from June 2019 to December 2022.

Clinical and laboratory data collection

PostgreSQL software (Version 14.5-1) and Navicat Premium 15 software were used to access the MIMIC-IV database, and structured query language (SQL) was employed to extract the data for the training cohort. Data of external validation cohort were manually collected and extracted from the electronic medical record system of Huzhou Central Hospital. Data for all study subjects were collected at a time point within the 48 h preceding the initiation of CRRT, and as close as possible to the actual start time of the first CRRT treatment.

Clinic data of the patients included age, sex, marital status, Kidney Disease: Improving Global Outcomes (KDIGO) stage, SOFA score, anion gap, white blood cell (WBC) count, red blood cell (RBC) count, haemoglobin, platelet, red blood cell distribution width (RDW), mean corpuscular volume (MCV), mean corpuscular haemoglobin (MCH), haematocrit, alanine aminotransferase (ALT), aspartate aminotransferase (AST), total bilirubin, albumin, glucose, creatinine, blood urea nitrogen (BUN), prothrombin time, magnesium, total calcium, sodium, potassium, therapy of albumin infusion, norepinephrine, furosemide, mechanical ventilation, complications/comorbidities, ICU length of stay (LOS) and in-hospital mortality.

Complications/comorbidities included hypertension, diabetes, congestive heart failure, atrial fibrillation, cirrhosis, chronic obstructive pulmonary disease (COPD), cerebral infarction, malignant tumour, sepsis, septic shock, acute pancreatitis, acute respiratory failure, cardiac arrest, cardiogenic shock and acute myocardial infarction.

Inclusion and exclusion criteria

The inclusion criteria for this study were as follows:

1. Only patients admitted to the ICU for the first time were included.

2. Adults (≥18 years old) with AKI meeting the diagnostic criteria of the KDIGO clinic practice guideline.

3. Patients who received therapy of CRRT.

4. ICU LOS ≥24 hours.

The exclusion criteria were as follows:

1. Repeat admissions to the ICU.

2. End-stage renal disease.

3. Discharge from the ICU within 24 hours of admission.

4. Patients with missing critical data or insufficient information.

Outcome definition

The primary endpoint was the in-hospital mortality.

Statistical analysis

Data were processed and analysed using Stata version 14.0 (StataCorp, College Station, TX) and R software version 4.2.3 (R Foundation for Statistical Computing, Vienna, Austria). Variables with missing values exceeding 30% were excluded, while those with missing data were subjected to multiple imputation using the mice package in R software. If quantitative data followed a normal distribution, it was represented using the mean ± standard deviation ($\overline{x}$ ± s), and a t-test was employed. If quantitative data did not follow a normal distribution, it was represented using the median (interquartile range) (M (QL, QU)), and the Mann–Whitney U-test was used. Additionally, a heatmap of the feature association matrix was plotted for correlation analysis. Categorical data were represented as percentages (%), and the Chi-square (χ²) test was utilized for comparing between groups.

Missing data were imputed using multiple imputation. To avoid overfitting, the least absolute shrinkage and selection operator (LASSO) with 10-fold cross-validation analysis was performed to select variables to be included in the model. To ensure the evaluation performance of the model, fivefold cross-validation with grid search were used on the training cohort to select the optimal parameters for the model. The selected variables were used to construct nine predictive models, including Lasso regularized logistic regression (Lasso-LR), decision tree (DT), ridge regression (RR), k-nearest neighbor (KNN), light gradient boosting machine (LightGBM), RF, extreme gradient boosting (XGBoost), support vector machines (SVMs) and neural network (NN).

The performance of these nine models on the external validation cohort dataset was comprehensively evaluated based on metrics such as the area under the receiver operating characteristic curve (AUROC), sensitivity and specificity and the best one was selected. Also, we utilized the method of SHAP to illustrate how these variables predict the in-hospital mortality of patients with AKI undergoing CRRT in the best model. Finally, the selected predictive model was visualized as a static nomogram and a web-based dynamic nomogram, with a comprehensive evaluation from the perspectives of discrimination, calibration and clinical practicability. Discrimination was primarily assessed using the AUROC, while calibration was evaluated by plotting a calibration curve to assess the accuracy of the predictive model. Decision curve analysis (DCA) curves were generated to evaluate the clinical utility of the predictive model. Statistical significance was considered when p < .05.

Results

Study population

A total of 43,032 patients with AKI meeting the diagnostic criteria of the KDIGO clinic practice guideline in MIMIC-IV database were selected. Ultimately, based on the inclusion and exclusion criteria, 1068 individuals were included in the training cohort. At the same time, 698 patients from Huzhou Central Hospital were screened and 327 patients were finally left as the external validation cohort, as shown in Figure 1.

Figure 1.

Flowchart of the study.

Table 1 demonstrates the demographic baseline characteristics between the training cohort and the external validation cohort. Significant differences were observed between the two groups in various parameters, including age, gender, marital status, SOFA score, anion gap, RBC count, MCV, haematocrit, ALT, AST, total bilirubin, albumin, blood glucose, creatinine, prothrombin time, blood magnesium, total calcium, sodium, chloride, administered treatments (albumin infusion, epinephrine, mechanical ventilation), complications/comorbidities (hypertension, congestive heart failure, atrial fibrillation, cirrhosis, malignant tumour, sepsis, septic shock, acute respiratory failure, cardiogenic shock and acute myocardial infarction) and ICU LOS (p < .05).

Table 1.

The characteristics of adult patients in the training cohort and external validation cohort.

Characteristics	Training cohort (n = 1068)				External validation cohort (n = 327)				t/z/χ2	All p value
	Total cohort (n = 1068)	Survivors (n = 527)	Non-survivors (n = 541)	p value	Total cohort (n = 327)	Survivors (n = 198)	Non-survivors (n = 129)	p value
Age (years)	62.60 ± 15.39	61.06 ± 15.67	64.11 ± 14.98	.001	66.16 ± 17.18	65.24 ± 16.65	67.56 ± 17.94	.234	−3.551	<.001
Gender (n (%))
Female	430 (40.26)	206 (39.09)	224 (41.40)		106 (32.42)	63 (31.82)	43 (33.33)
Male	638 (56.74)	321 (60.91)	317 (58.60)	.440	221 (67.58)	135 (68.18)	86 (66.67)	.775	6.514	.011
Marital status (n (%))
Othera	606 (27.53)	286 (54.27)	320 (59.15)		79 (24.16)	52 (26.26)	27 (20.93)
Married	462 (43.26)	241 (45.73)	221 (40.85)	.108	248 (75.84)	146 (73.74)	102 (79.07)	.271	106.344	<.001
KDIGO stage (n (%))
Stage 1 + 2	294 (27.53)	157 (29.79)	137 (25.32)		108 (33.03)	67 (33.84)	41 (31.78)
Stage 3	774 (72.47)	370 (70.21)	404 (74.68)	.102	219 (66.97)	131 (66.16)	88 (68.22)	.699	3.691	.055
SOFA score (score)	12.50 ± 4.08	12.02 ± 3.91	12.97 ± 4.19	<.001	10.03 ± 4.27	9.13 ± 3.96	11.42 ± 4.38	<.001	9.468	<.001
Anion gap (mmol/L)	20.99 ± 6.43	19.59 ± 5.55	22.37 ± 6.91	<.001	19.48 ± 8.30	18.27 ± 8.29	21.33 ± 7.99	.001	3.481	.001
WBC (×109/L)	13.30 (9.00, 19.70)	12.60 (8.50, 18.30)	14.40 (9.50, 20.70)	.001	13.30 (7.80, 18.90)	12.60 (7.60, 18.10)	15.10 (8.10,22.50)	.083	1.298	.194
RBC (×1012/L)	3.11 ± 0.66	3.15 ± 0.64	3.06 ± 0.69	.035	3.48 ± 1.04	3.55 ± 0.98	3.38 ± 1.13	.140	−7.755	<.001
Haemoglobin (g/L)	94.07 ± 18.59	95.03 ± 18.49	93.13 ± 18.66	.095	106.60 ± 32.08	109.49 ± 30.46	102.15 ± 34.06	.043	−8.818	<.001
Platelet (×109/L)	119.00 (70.00, 192.00)	135.00 (85.00, 206.00)	103.00 (63.00, 173.00)	<.001	125.00 (76.00, 192.00)	124.50 (77.00, 197.00)	125.00 (67.00, 180.00)	.643	0.035	.973
RDW (%)	16.94 ± 2.84	16.43 ± 2.43	17.44 ± 3.11	<.001	14.85 ± 2.70	14.54 ± 2.35	15.34 ± 3.112	.009	11.771	<.001
MCV (fL)	92.82 ± 8.11	91.72 ± 7.65	93.90 ± 8.41	<.001	94.45 ± 8.23	94.10 ± 7.48	94.98 ± 9.26	.344	−3.163	.002
MCH (pg)	30.52 ± 2.80	30.31 ± 2.63	30.72 ± 2.94	.162	30.74 ± 2.93	30.94 ± 2.74	30.43 ± 3.19	.122	−1.238	.216
Haematocrit (%)	28.59 ± 5.55	28.71 ± 5.50	28.48 ± 5.60	.503	32.68 ± 9.54	33.27 ± 8.90	31.77 ± 10.30	.164	−9.646	<.001
ALT (U/L)	55.00 (23.00, 256.50)	45.00 (20.00, 204.00)	70.00 (25.00, 320.00)	<.001	44.00 (18.90, 144.00)	29.10 (17.90, 77.70)	73.80 (25.00, 303.10)	<.001	3.024	.003
AST (U/L)	115.00 (45.50, 477.00)	90.00 (37.00, 293.00)	144.00 (52.00, 645.00)	<.001	76.50 (32.10, 264.90)	55.50 (27.70, 119.00)	162.00 (51.00, 717.00)	<.001	4.207	<.001
Total bilirubin (μmol/L)	28.22 (10.26, 90.63)	22.23 (8.55, 66.69)	35.91 (13.68, 131.67)	<.001	18.00 (10.20, 34.50)	16.85 (10.60, 32.40)	19.60 (9.80, 36.50)	.608	5.358	<.001
Albumin (g/L)	32.68 ± 9.54	29.10 ± 7.26	28.30 ± 7.66	.080	29.55 ± 6.92	31.03 ± 5.93	29.72 ± 5.99	.053	−4.008	<.001
Glucose (mmol/L)	7.33 (5.89, 13.33)	7.22 (5.83, 9.39)	7.56 (5.89, 10.00)	.226	8.61 (5.43, 12.65)	8.13 (5.42, 12.34)	9.24 (5.52, 12.91)	.639	−2.942	.003
Creatinine (μmol/L)	344.76 (238.68, 459.68)	371.28 (265.2, 503.88)	318.24 (221.00, 415.48)	<.001	286.70 (194.20, 429.70)	320.45 (207.30, 458.70)	247.90 (176.30, 355.50)	.002	4.336	<.001
BUN (mmol/L)	21.00 (13.53, 31.51)	20.65 (14.24, 30.97)	21.00 (12.82, 31.68)	.366	19.42 (12.82, 31.55)	20.45 (14.19, 31.39)	18.51 (11.45, 31.55)	.223	0.763	.445
Prothrombin time (s)	17.10 (14.00, 23.555)	15.70 (13.30, 20.10)	19.50 (15.20, 26.60)	<.001	15.20 (13.30, 18.50)	14.45 (12.90, 17.30)	16.50 (13.90, 19.60)	<.001	5.803	<.001
Magnesium (mmol/L)	0.96 ± 0.20	0.96 ± 0.21	0.97 ± 0.19	.241	0.90 ± 0.26	0.88 ± 0.28	0.91 ± 0.24	.371	4.940	<.001
Total calcium (mmol/L)	2.02 ± 0.26	2.02 ± 0.26	2.03 ± 0.27	.636	1.97 ± 0.22	1.98 ± 0.24	1.96 ± 0.20	.298	3.042	.002
Sodium (mmol/L)	136.91 ± 6.30	136.42 ± 5.65	137.38 ± 6.85	.013	139.99 ± 8.86	137.61 ± 8.57	143.63 ± 8.05	<.001	−6.977	<.001
Chlorine (mmol/L)	100.73 ± 7.27	100.74 ± 7.15	100.72 ± 7.40	.966	107.61 ± 9.60	106.32 ± 9.78	109.59 ± 9.00	.003	−13.810	<.001
Potassium (mmol/L)	4.69 ± 0.93	4.60 ± 0.92	4.78 ± 0.93	.002	4.66 ± 1.08	4.70 ± 1.06	4.61 ± 1.11	.454	0.432	.651
Therapy (n (%))
Albumin infusion	359 (33.61)	157 (29.79)	202 (25.32)	.009	207 (63.30)	108 (54.55)	99 (76.74)	<.001	91.517	<.001
Norepinephrine	676 (63.30)	269 (51.04)	407 (75.23)	<.001	157 (48.01)	70 (35.35)	87 (76.74)	<.001	24.309	<.001
Furosemide	488 (45.69)	242 (45.92)	246 (45.47)	.883	138 (42.20)	80 (40.40)	58 (44.96)	.415	1.233	.267
Mechanical ventilation	939 (87.92)	438 (83.11)	501 (92.60)	<.001	204 (62.39)	93 (46.97)	111 (86.05)	<.001	110.294	<.001
Complications/comorbidities (n (%))
Hypertension	323 (30.24)	149 (28.27)	174 (32.16)	.166	134 (40.98)	80 (40.40)	54 (41.86)	.794	13.098	<.001
Diabetes	348 (32.58)	188 (35.67)	160 (29.57)	.033	89 (27.22)	56 (28.28)	33 (25.58)	.592	3.352	.067
Congestive heart failure	395 (36.99)	189 (35.86)	206 (38.08)	.454	137 (41.90)	75 (37.88)	62 (48.06)	.068	2.559	.110
Atrial fibrillation	404 (37.83)	203 (38.52)	201 (37.15)	.645	60 (18.35)	34 (17.17)	26 (20.16)	.496	42.792	<.001
Cirrhosis	259 (24.25)	110 (20.88)	149 (27.54)	.011	14 (4.28)	6 (3.03)	8 (6.20)	.166	63.427	<.001
COPD	154 (14.42)	74 (14.04)	80 (14.79)	.729	36 (11.01)	21 (12.12)	15 (10.85)	.773	2.475	.116
Cerebral infarction	96 (8.99)	46 (8.73)	50 (9.24)	.769	38 (11.62)	24 (12.12)	14 (10.82)	.726	1.997	.158
Malignant tumour	161 (15.07)	76 (14.42)	85 (15.71)	.556	25 (7.65)	14 (7.07)	11 (8.53)	.628	11.959	.001
Sepsis	998 (93.45)	485 (92.03)	513 (94.82)	.065	120 (36.70)	74 (37.37)	46 (35.66)	.753	506.617	<.001
Septic shock	482 (45.13)	178 (33.77)	304 (56.19)	<.001	103 (31.50)	62 (31.31)	41 (31.78)	.929	19.108	<.001
Acute pancreatitis	82 (7.68)	45 (8.54)	37 (6.84)	.297	27 (8.26)	24 (12.12)	3 (2.33)	.002	0.117	.733
Acute respiratory failure	681 (63.76)	317 (60.15)	364 (67.28)	.015	172 (52.60)	73 (36.87)	99 (76.74)	<.001	13.135	<.001
Cardiac arrest	99 (9.27)	33 (6.26)	66 (12.20)	.001	40 (12.23)	8 (4.04)	32 (24.81)	<.001	2.450	.118
Cardiogenic shock	166 (15.54)	64 (12.14)	102 (18.85)	.002	20 (6.12)	6 (3.03)	14 (10.85)	.004	19.253	<.001
Acute myocardial infarction	135 (12.64)	58 (11.01)	77 (14.23)	.113	15 (4.59)	3 (1.52)	12 (9.30)	.001	16.919	<.001
ICU LOS	17.75 (8.63, 29.94)	25.00 (17.29, 38.88)	9.92 (4.04, 18.29)	<.001	12.00 (6.00, 22.00)	15.00 (8.00, 24.00)	8.00 (3.00, 18.00)	<.001	5.091	<.001

AKI: acute kidney injury; CRRT: continuous renal replacement treatment; KDIGO: Kidney Disease Improving Global Outcomes; WBC: white blood cell; RBC: red blood cell; SOFA: score sequential organ failure assessment score; RDW: red cell distribution width; MCV: mean corpuscular volume; MCH: mean corpuscular haemoglobin; ALT: alanine aminotransferase; AST: aspartate transaminase; BUN: blood urea nitrogen; COPD: chronic obstructive pulmonary disease; ICU: intensive care unit; LOS: length of stay.

^a Unmarried, widowed, divorced status or unknown.

Feature extraction

Factors predictive of in-hospital mortality

A heatmap of the feature correlation matrix was generated, revealing significant correlations among many features (Supplementary Figure A1). For example, there was a positive correlation between urea nitrogen and creatinine, and a negative correlation between RBC count and mean RBC volume (The larger the area of the small square in the picture and the darker the colour, the stronger the correlation.) Therefore, the in-hospital mortality of critically ill patients with AKI undergoing CRRT was used as the dependent variable. Missing data were imputed using multiple imputation as shown in Figure 2. Then, the 46 features of the training cohort were incorporated into LASSO regression analysis with cross-validation, yielding a λ value at one standard error (λ.1se) of 0.074. Finally, 10 features were selected including age, platelet count, serum creatinine, prothrombin time, anion gap, potassium, cardiac arrest, septic shock, administration of epinephrine, and receipt of mechanical ventilation as shown in Figure 2. Subsequently, Lasso-LR, DT, RR, KNN, LightGBM, RF, XGBoost, SVM and NN models were individually constructed.

Figure 2.

Selection of clinical features. (a, b) Missing data of the training cohort were imputed using multiple imputation. (c) Selection process of the value of lambda by LASSO regression with cross validation.

Comprehensive analysis of the mortality prediction models

Each of the 10 features was individually incorporated into Lasso-LR, DT, RR, KNN, LightGBM, RF, XGBoost, SVM and NN models, with parameter optimization conducted through fivefold cross-validation. Multiple iterations of model training were performed to identify the optimal model. The performance metrics for the aforementioned nine models on the training and external validation cohorts are presented in Table 2 and Figures 3 and 4.

Table 2.

Performance of nine different in-hospital mortality prediction models in training set and external validation set.

Models	AUROC (95%CI)	Sensitivity	Specificity	Accuracy	Kappa	NPV	PPV	Recall
Training cohort
Lasso-LR	0.771	0.771	0.649	0.711	0.420	0.734	0.693	0.771
Decision tree	0.833	0.795	0.744	0.770	0.539	0.779	0.761	0.795
Ridge regression	0.764	0.742	0.673	0.707	0.414	0.688	0.728	0.673
k-nearest neighbor	0.919	0.806	0.867	0.836	0.673	0.813	0.862	0.806
LightGBM	0.954	0.863	0.905	0.884	0.768	0.866	0.903	0.863
Random forest	0.875	0.828	0.770	0.800	0.599	0.814	0.787	0.828
XGBoost	0.806	0.797	0.679	0.739	0.477	0.765	0.718	0.797
Support vector machines	0.773	0.617	0.799	0.707	0.415	0.670	0.759	0.617
Neural network	0.761	0.664	0.757	0.710	0.420	0.687	0.737	0.664
External validation cohort
Lasso-LR	0.756	0.721	0.702	0.709	0.410	0.794	0.612	0.721
Decision tree	0.630	0.535	0.667	0.615	0.200	0.687	0.511	0.535
Ridge regression	0.755	0.543	0.788	0.691	0.338	0.726	0.625	0.543
k-nearest neighbor	0.733	0.605	0.758	0.697	0.364	0.746	0.619	0.605
LightGBM	0.713	0.558	0.758	0.679	0.320	0.725	0.600	0.558
Random forest	0.730	0.612	0.722	0.679	0.332	0.741	0.590	0.612
XGBoost	0.731	0.597	0.742	0.685	0.340	0.739	0.602	0.597
Support vector machines	0.747	0.504	0.813	0.691	0.329	0.716	0.637	0.504
Neural network	0.745	0.519	0.793	0.685	0.321	0.717	0.620	0.519

AUROC: area under the receiver operating curve; AUROC: area under the receiver operating curve; NPV: negative predictive value; PPV: positive predictive value; Lasso-LR: LASSO-logistic regression; DT: decision tree; RR: ridge regression; KNN: k-nearest neighbor; LightGBM: light gradient boosting machine; RF: random forest; XGBoost: extreme gradient boosting; SVM: support vector machine; NN: neural network.

Figure 3.

Comparison of ROC curves of nine models in the external validation cohort.

Figure 4.

Calibration curves for different algorithm models in the external validation cohort.

In the training cohort, the AUROC for each model exceeded 0.70, with LightGBM achieving the highest AUROC of 0.954, while NN exhibited the lowest at 0.761. In the external validation cohort, the Lasso-LR model achieved the highest AUROC (0.756), followed by the ENet model (0.755), SVM model (0.747), NN model (0.745), KNN model (0.733), XGBoost model (0.731), RF model (0.730), LightGBM model (0.713) and DT model (0.630), as illustrated in Figure 3 and detailed in Table 2.

The calibration curves for different algorithmic models in the external validation are illustrated in Figure 4, indicating good calibration across all models. Among them, the Lasso-LR model exhibited superior calibration with a Brier score of 0.197 compared to the other eight models. In addition, except for the DT model, which performed relatively poorly, DCA curves suggested that the remaining eight models possess some clinical utility (Supplementary Figure A2).

Optimal model selection and evaluation

Compared to the other eight models, the Lasso-LR model exhibited the highest AUROC (0.756) and the optimal calibration curve. The DCA curve suggested a certain clinical utility in predicting in-hospital mortality among critically ill patients with AKI undergoing CRRT. Consequently, the Lasso-LR model was the best choice. We utilized the confusion matrix and precision–recall (PR) curve to further validate the performance of the Lasso-LR model, as shown in Supplementary Figures A3 and A4.

The SHAP to model interpretation

To intuitively explain the selected variables, we utilized SHAP to illustrate how these variables predict the in-hospital mortality of patients with AKI undergoing CRRT in the Lasso-LR model as shown in Figure 5. Supplementary Figure A5 shows the importance of 10 features, ranked from top to bottom as septic shock, anion gap, creatine, prothrombin time, age, platelet, norepinephrine, potassium, cardiac arrest and mechanical ventilation, with the x-axis SHAP value indicating the importance of the forecast model. Additionally, we provided two typical examples to illustrate the interpretability of the model. One was a patient with severe AKI undergoing CRRT who died during hospitalization, with a high SHAP predictive score (0.899), while another surviving patient had a lower SHAP predictive score (0.332) (Supplementary Figures A6 and A7).

Figure 5.

The impact of relevant features on prediction results (a: continuous variable, b: categorical variable).

Nomogram for in-hospital mortality

As mentioned above, the Lasso-LR model was the optimal model, incorporating 10 features, namely age, platelet count, serum creatinine, prothrombin time, anion gap, blood potassium, cardiac arrest, septic shock, administration of epinephrine bitartrate, and receipt of mechanical ventilation. We visualized it as a common nomogram (static nomogram) and a web-based dynamic nomogram (https://chsyh2006.shinyapps.io/dynnomapp/), for clinical applicability. Clinical practitioners could access it freely and conveniently through devices such as smartphones and computers, facilitating research, learning and assisting in clinical decision-making, as shown in Figure 6.

Figure 6.

Nomogram for predicting in-hospital mortality in patients with AKI undergoing CRRT.

For instance, when an AKI patient undergoing CRRT met the conditions (age 65 years old, a history of cardiac arrest, platelet count of 60 × 10⁹/L, prothrombin time of 20 s, creatinine level of 300 μmol/L, anion gap of 20 mmol/L, serum potassium of 4.7 mmol/L, and received vasoactive drugs and mechanical ventilation treatment), then the nomogram model showed that the patient’s probability of death during hospitalization is 72%, as shown in Supplementary Figure A8.

Performance evaluation of nomograms

ROC curve of training cohort and external validation cohort (discrimination)

In the Lasso-LR model, the AUROC of the training cohort and the external validation cohort were 0.771 (95%CI: 0.743–0.799) and 0.756 (95%CI: 0.702–0.809), respectively. The results suggested that the Lasso-LR model demonstrated good discriminative performance in predicting mortality among critically ill patients with AKI undergoing CRRT, as illustrated in Figure 7.

Figure 7.

Receiver operating characteristic curve of the nomogram model.

Calibration and DCA curves for the training and external validation cohort

The calibration curves for the training cohort and external validation cohort are depicted in Figure 8. The calibration slopes and Brier scores were 1.000 and 0.195 for the training cohort, and 0.849 and 0.197 for the external validation cohort, respectively, as shown in Figure 8. The predicted values aligned closely with the observed values, and both the calibration and ideal curves exhibited good fitting, indicating excellent calibration accuracy. The DCA curves showed that using the model to predict in-hospital mortality provided more benefit than either the ‘all-treatment strategy’ or the ‘no-treatment strategy’ over a wide range of thresholds, and demonstrated a favourable net benefit over a range of mortality risks for both the training cohort and the validation cohort, which indicated that the model had a certain clinical application value, as shown in Figure 8.

Figure 8.

Calibration and DCA curves for the training and external validation cohort. (a) Calibration curves for the training cohort. (b) Calibration for the external validation cohort. (c) DCA curves for the training. (d) DCA for the external validation cohort.

Discussion

It was a real-world clinical research that utilized data from the MIMIC-IV database and critically ill patients with AKI undergoing CRRT in the ICU at Huzhou Central Hospital. Nine ML algorithm prediction models were developed and the optimal one was selected to forecast in-hospital mortality for critically ill patients with AKI undergoing CRRT. Consequently, the Lasso-LR model was the optimal model and it was visualized as a static nomogram and a web-based dynamic nomogram (https://chsyh2006.shinyapps.io/dynnomapp/). Discrimination, calibration and DCA curves were demonstrated excellent performance of the nomogram. From the constructed nomograms, it was evident that the features associated with mortality during ICU hospitalization for those patients included age, blood potassium, platelet count, creatinine, prothrombin time, anion gap, concomitant or subsequent cardiac arrest, septic shock and the administration of epinephrine and mechanical ventilation. This provided clinical professionals with the ability to predict the in-hospital mortality of such patients based on these clinical features and the nomogram.

For critical care physicians, accurately assessing the severity of illness and predicting prognosis in critically ill patients is paramount. In clinical settings, patients and their families are primarily concerned about the prognosis of the illness, particularly in ICU cases where survival and quality of life are at the forefront. As healthcare professionals and researchers in intensive care, precise prediction of outcomes and prognosis not only alleviates anxiety and fosters better doctor-patient communication but also enables tailored patient management, ultimately improving outcomes.

Currently, widely used scoring systems such as the SOFA score, APACHE II score and simplified acute physiology score (SAPS) aid in assessing illness severity and prognosis. However, these scores are generalized and may not suit specific categories of critical illnesses [22]. Given the seriousness and high mortality rate of patients with AKI undergoing CRRT, it is crucial to develop tailored clinical risk prediction tools or scoring systems to accurately forecast prognosis, thus holding significant theoretical and clinical value.

Certainly, some scholars have constructed clinical prediction models related to the mortality of critically ill patients with AKI undergoing CRRT. In 2019, a retrospective study [24] included 828 adult critically ill patients with AKI undergoing CRRT as the modelling group, with the primary outcome being a seven-day mortality rate (34.42%, 285 cases). Utilizing multifactorial logistic regression analysis, the study identified the following seven variables as independent predictors of mortality in these patients: serum bilirubin, arterial blood pH, blood sodium, WBC count, Glasgow Coma Scale score, alveolar-arterial oxygen difference, and whether there was hypotension. Based on this, a simplified scoring system was constructed, with an AUROC of 0.772. In this study, an external validation group of 497 critically ill patients with AKI undergoing CRRT from another hospital was selected, yielding an AUROC of 0.720. In 2020, a retrospective study [25] utilized 594 adult critically ill patients with AKI undergoing RRT as the modelling group, with 229 patients as the external validation group. Using LASSO regression, the study selected the following five variables: urine output before RRT initiation, serum creatinine, platelet count, serum bilirubin and cumulative fluid balance. Based on this, logistic regression prediction models for 90-day and 1-year mortality were constructed, with internal validation AUROCs of 0.66 and 0.63, respectively. External validation AUROCs were also suboptimal, at 0.64 and 0.63. Of course, in recent years, there have been other studies constructing prediction models for the mortality of critically ill patients with AKI undergoing CRRT [10,22,26–28].

Currently, clinical indicators to achieve early individualized warnings for the prognosis of critically ill patients with AKI undergoing CRRT remains a challenge. Despite the construction of some predictive models for the mortality of these patients, previous studies have limitations such as small sample sizes, single-centre designs, mostly model construction based on traditional statistical methods, and lack of external validation. The extrapolation of research results is limited, warranting further exploration and research. To this end, we conducted this clinical study from the real-world setting.

The strengths of the study were as follows: first, nine ML algorithms were employed to construct corresponding prediction models, ultimately optimal clinical risk prediction model was selected. Second, the research data were obtained from two hospitals, both domestic and international, with a large sample size (1395 cases). Third, the optimal model underwent external validation, and was visualized as a nomogram which facilitated early prognosis assessment by physicians.

Certainly, there existed some limitations. First, it was a retrospective study, and inherent bias was challenging to avoid. Second, due to the nature of retrospective studies, this section of the research only constructed a short-term (mortality during ICU hospitalization) prognosis prediction model for critically ill patients with AKI undergoing CRRT. Therefore, it was necessary to further investigate the long-term prognosis of these patients in the future.

Conclusions

Our study selected the optimal model and visualized it as a static and dynamic nomogram integrating clinical predictors, so that clinicians can personalized predict the in-hospital outcome of critically ill patients with AKI undergoing CRRT upon ICU admission. In the future, it was necessary to conduct larger-scale multicentre studies to validate the applicability of this model.

Funding Statement

No funding was received.

Author contributions

Zhenzhen Yang and Jianhong Lu contributed to design the study. All the authors were responsible for the data collection (the external validation cohort). Yongbin Wang and Jianhong Lu wrote the manuscript. Jianhong Lu and Lei Zhong were responsible for the data collection (the training cohort). Jianhong Lu and Lei Zhong also processed the data and did the analyses. Lei Zhong and Xu Sun contributed to review and edit the manuscript. All the authors read and approved the final manuscript.

Ethical approval

The database was approved by the institutional review boards of the BIDMC and the Massachusetts Institute of Technology. The study (external validation data set) was approved by the Institutional Review Board of Huzhou Central Hospital (approval numbers: 202212010-01, 202203021-01).

Consent form

The study (external validation data set) was approved by the Institutional Review Board of Huzhou Central Hospital (approval numbers: 202212010-01, 202203021-01) with a waiver of informed consent because of the anonymous nature of this study.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The datasets analysed during the current study are available from the corresponding author upon reasonable request.