Predicting In-Hospital Acute Kidney Injury after Cardiac Surgery Using Machine Learning

Abstract

Objective:

We aimed to develop and evaluate CSA-AKI machine learning prediction models which included clinical variables, biomarkers, and high frequency hemodynamic measurements.

Design:

A prospective, observational study.

Setting:

Single-center university hospital.

Participants:

667 adult patients undergoing elective and urgent cardiac surgery between 05/2015 and 03/2023.

Measurements and Main Results:

After excluding patients for clinical and administrative reasons and those with missing data, 602 patients were included in the predictive models. Random Forest (RF) and Long Short-Term Memory (LSTM) models were trained on data from 492 patients which included hemodynamic data from throughout the perioperative period, and BNP and [TIMP-2]x[IGFBP7] biomarkers. These models were evaluated in a test set of 110 patients to predict CSA-AKI, which was defined as serum creatinine rise of 0.3 mg/dL in the first 48 hours after surgery or 1.5 times baseline creatinine between postoperative day 1 to day 7. The RF model achieved an AUC of 0.73 (0.72 – 0.74) and accuracy of 0.77 (0.76 - 0.79) on the test set. The RF model achieved a sensitivity and specificity of 0.59 (0.57 - 0.62) and 0.81 (0.79 – 0.84), respectively. The LSTM model achieved an AUC of 0.68 (0.67 - 0.70) and accuracy of 0.72 (0.69 – 0.74) on the test set. The sensitivity of the LSTM model was 0.60 (0.57 – 0.64) and the specificity was 0.74 (0.71 – 0.78).

Conclusions:

We developed two machine learning models which incorporated perioperative biomarkers and hemodynamics as input data to predict CSA-AKI. The models exhibited competitive performance for prediction of the outcome and lay the foundation for further advancement in this critical area of research in kidney injury prevention.

Introduction

Acute Kidney Injury (AKI) is a common complication after cardiac surgery and affects approximately 20-30% of patients.^1,2. The high incidence of AKI reflects both comorbid conditions which predispose patients to develop AKI as well as intra-operative and post-operative factors including hypotension, hemodilution, blood transfusion, exposure to cardiopulmonary bypass (CPB), and the use of vasopressors and nephrotoxic medications. Because CSA-AKI is associated with both short and long term mortality and morbidity^3,4 and increased hospital length of stay⁵, there is a critical need to predict this complication and implement interventions to prevent or mitigate it. A number of preventive strategies such as optimizing tissue oxygen delivery and implementing a nephroprotective ‘bundle of care’ have proven effective in reducing the incidence of CSA-AKI in randomized controlled trials.^6,7 Identifying high risk patients early in their post-operative course using the Nephrocheck AKI risk score and randomizing them to receive nephroprotective interventions reduced the incidence of moderate-severe AKI in one international RCT.⁷ However, there are few reliable tools to accurately predict CSA-AKI early in the perioperative period. Predicting AKI using time dependent features of the electronic health record can potentially 1) improve the effectiveness of AKI prevention strategies, and 2) enable novel therapies in patients at high risk for developing AKI.

Several studies have developed and evaluated CSA-AKI risk prediction models. Most of these are limited to prediction of severe CSA-AKI, including AKI requiring renal replacement therapy (RRT)⁸. However, given that the majority of CSA-AKI is mild⁹, a substantial portion of outcomes are not evaluated despite the profound clinical and financial impact (e.g. CKD development, long term RRT) associated with them. Studies which evaluate all stages of AKI^3,10,11 have been limited by their retrospective design¹⁰, inclusion of overlapping input and outcome timelines¹¹, and use of algorithms with misclassification rates too high for clinical use, such as the 20% sensitivity threshold used in Karkouti et al.³ All of the models thus far developed lack newer tools and more detailed features such as biomarker data and high frequency perioperative hemodynamics. Given the dynamic nature of multimodal data in the critical care setting, machine learning models are well suited to capture and recognize patterns for kidney injury prediction beyond standard statistical methods¹². Here, we aim to develop and evaluate two machine learning models which incorporate key perioperative features including high frequency hemodynamic data to predict in-hospital CSA-AKI risk.

Methods

Study Setting and Data

667 adult patients (age > 18) undergoing elective and urgent cardiac surgery between 05/2015 and 03/2023 were recruited at an urban university hospital for this study. The study was approved by the Institutional Review Board, and all subjects provided written informed consent. We included coronary artery bypass graft (CABG) surgery, valvular surgery, aortic surgery, septal myectomy, and combinations of these procedures performed using CPB. Patient characteristics, medications, comorbidities, surgical characteristics, and postoperative events during the primary surgical hospitalization were collected using a standardized case report form. Serum and urine samples for biomarkers BNP, and [TIMP-2]x[IGFBP7], respectively, were obtained preoperatively, at hour 6 after CPB, and daily on PODs 1 to 4. However, our model only incorporated biomarkers from the preoperative, hour 6 after CPB and POD 1 serum and urine samples. Serum samples were centrifuged at 1227g for 15 minutes at 4C. Urine samples were centrifuged at 626g for 30 minutes at 4 Celsius. Serum and urine aliquots were stored at −80 Celsius until thawed for batch analysis. Urine creatinine concentrations were measured by capillary electrophoresis. High frequency hemodynamic data, which was measured as part of the standard of care for these patients, was extracted from the electronic medical record. (Figure 1). This included vital signs and hemodynamic data measured every minute in the operating room and every fifteen minutes in the ICU. For purposes of the model prediction, the hemodynamic data were divided into three categories: non-CPB time, CPB time, and 12 hours of postoperative time in the intensive care unit (ICU). Study data from our EHR was requested with ClinDEN and collected and managed with REDCap.

Figure 1 –

Data collected from pre-operative time up to 12 hours post-operative, except biomarkers which also included Post-operative day (POD) 1 blood samples. Outcome observation window began at POD 1

Primary Objective

The primary objective of this study was to determine if machine learning models which include clinical variables, established biomarkers and high frequency hemodynamic data improve prediction of in-hospital CSA-AKI compared to conventional models using clinical data alone. This task is illustrated in Figure 2. The data is multimodal, including both constant and time-series dependent data. The modeling used includes decision-tree based modeling and a deep learning approach which is intended for sequence or time-series data. In-hospital CSA-AKI was defined using the KDIGO serum creatinine criteria¹³, which was serum creatinine rise of 0.3 mg/dL in the first 48 hours or 1.5 times baseline creatinine between postoperative day 1 to day 7.

Figure 2 –

Variety of inputs particular to cohort for two different machine learning models for determining kidney outcomes; Created in BioRender. (2024) https://BioRender.com/t58c338

Model Development and Evaluation

We developed both a random forest model and long short-term memory model for in-hospital CSA-AKI prediction. To train and test these models, of 667 eligible patients, 38 patients were excluded for clinical or administrative reasons such as requiring pre-operative dialysis, cancelled or aborted surgery, or withdrawn consent and 14 patients were excluded for missing either high frequency hemodynamic data or biomarkers (Figure 3). Of the remaining 602 patients, 110 (18.2%) developed in-hospital CSA-AKI. We divided the cohort into training (481 patients) and testing (121 patients) sets.

Figure 3 -.

Flow Diagram of Patient Selection into Study Cohort

Random forest inputs included all categories of variables mentioned above and these were engineered into 181 distinct features: 44 categorical, 2 ordinal, and 135 continuous variables. The 44 categorical variables were a result of encodings spanning 24 original features (e.g. ethnicity spanning 5 features). The continuous variables included time series hemodynamics data, which were collated as summary statistics (e.g. mean pulse oximetry, standard deviation of mean arterial blood pressure, etc.) within each of the temporal categories (non-CPB, CPB and ICU). Time series features were also preprocessed to remove physiologically implausible values and artifacts, and to impute others. Protocol for smoothing of datapoints included removing spikes in measurements which was above neighboring measurements (within 2 minutes) of spikes when changed by 50% or more. For exact details, please see Supplementary Materials for information. Hyperparameters were found by grid search with 5-fold cross validation.

We also trained a deep learning model with long short-term memory architecture, which is intended to handle sequential data such as hemodynamics measured over time. This model was used to leverage the high frequency perioperative data and received either the same inputs as our random forest model or an analogous version for hemodynamic data which better suited the long short-term memory architecture. Our long short-term memory model contained the exact same 44 categorical features, 2 ordinal, and 23 of the same continuous features. The model also included analogous versions of the feature engineered time series hemodynamics data gathered as summary statistics within various temporal windows (non-CPB, CPB, and ICU). The advantage to this architecture was the preservation of sequential nature of this data with reasonable resolution, via what is known as step counts. We implemented long short-term memory units for each temporal window independently, and along with a dense feed forward network applied to both categorical and constant numeric variables, we concatenated all 4 units (3 long short-term memory units, 1 feed forward neural network) to an overall fully connected component and finally a classification head. Training was performed for 10000 epochs, with batch size of 32, with early stoppage of 50 based on maximum validation set AUC (validation split from training – 25% of training set for validation). Optimization was via Adam optimizer on binary cross entropy loss.

Statistical analysis was performed using both R (version 4.2.1) and Python (version 3.10). The random forest model was trained on scikit-learn (version 1.3.2) library. The long short-term memory model was trained on Keras/TensorFlow framework/library (version 2.15.0). Additional methods, including all our imputation and preprocessing rules, and all features for the random forest and long short-term memory models and long short-term memory step counts are detailed in Supplementary Materials.

Results

Study Cohort

The study cohort included 602 patients, among whom 110 (18.3%) developed in-hospital CSA-AKI. The demographic and clinical characteristics of the cohort are summarized in Table 1.

Table 1.

Cohort characteristics.

		Grouped by AKI or No
		Missing	Overall	AKI	No	P-Value
n			602	110	492
AKI stage	Stage I			84
	Stage II			13
	Stage III			13
Age, mean [min,max]		0	61 [23,87]	63 [38,84]	61 [23,87]	0.043
BMI, median [Q1,Q3]		0	29.4 [25.7,33.8]	31.7 [27.7,36.0]	28.8 [25.4,33.1]	<0.001
Ethnicity, n (%)	Hispanic or Latino White	0	153 (25.4)	26 (23.6)	127 (25.8)	0.212
	Not Hispanic or Latino Asian		25 (4.2)	3 (2.7)	22 (4.5)
	Not Hispanic or Latino Black		87 (14.5)	20 (18.2)	67 (13.6)
	Not Hispanic or Latino Other		3 (0.5)	2 (1.8)	1 (0.2)
	Not Hispanic or Latino White		333 (55.3)	59 (53.6)	274 (55.7)
	Hispanic or Latino Black		1 (0.2)		1 (0.2)
Gender, n (%)	Female	0	170 (28.2)	25 (22.7)	145 (29.5)	0.192
	Male		432 (71.8)	85 (77.3)	347 (70.5)
eGFR, mean (SD)		0	78.7 (21.7)	68.8 (25.4)	80.9 (20.1)	<0.001
Hypertension, n (%)	No	1	96 (16.0)	10 (9.1)	86 (17.5)	0.042
	Yes		505 (84.0)	100 (90.9)	405 (82.5)
Reoperation, n (%)	No	0	542 (90.0)	95 (86.4)	447 (90.9)	0.213
	Yes		60 (10.0)	15 (13.6)	45 (9.1)
Preoperative Hemoglobin, mean (SD)		0	13.3 (1.9)	12.6 (1.9)	13.5 (1.9)	<0.001
Left Ventricular Ejection Fraction, mean (SD)		2	53.5 (12.5)	51.6 (13.2)	54.0 (12.3)	0.080
ACE Inhibitor Use, n (%)	No	3	385 (64.3)	69 (63.3)	316 (64.5)	0.902
	Yes		214 (35.7)	40 (36.7)	174 (35.5)
ARB Use, n (%)	No	3	444 (74.1)	79 (72.5)	365 (74.5)	0.754
	Yes		155 (25.9)	30 (27.5)	125 (25.5)
Cross Clamp Time (Min), mean (SD)		0	97.0 (42.6)	106.4 (52.6)	94.9 (39.7)	0.032
Perfusion Time (Min), mean (SD)		0	140.1 (54.7)	159.4 (69.7)	135.8 (49.8)	0.001
Max Number of Concurrent Ionotrops or Vasoconstrictors, mean (SD)		0	1.9 (1.0)	2.1 (1.0)	1.8 (1.0)	0.002
Surgery Type, n (%)	Isolated CABG	0	300 (49.8)	50 (45.5)	250 (50.8)	0.498
	Isolated valve		201 (33.4)	37 (33.6)	164 (33.3)
	CABG + valve		54 (9.0)	13 (11.8)	41 (8.3)
	Aortic surgery		19 (3.2)	6 (5.5)	13 (2.6)
	Pulmonary arterial surgery w/wo pulmonary valve		12 (2.0)	2 (1.8)	10 (2.0)
	Septal myectomy w/wo valve		16 (2.7)	2 (1.8)	14 (2.8)
IABP Preoperative or Intraoperative, n (%)	No	0	556 (92.4)	98 (89.1)	458 (93.1)	0.219
	Yes		46 (7.6)	12 (10.9)	34 (6.9)
DDAVP Given, n (%)	No	1	488 (81.2)	80 (72.7)	408 (83.1)	0.017
	Yes		113 (18.8)	30 (27.3)	83 (16.9)
Intraoperative PRBC transfusion, n (%)	No	0	405 (67.3)	67 (60.9)	338 (68.7)	0.115
	Yes		197 (32.7)	43 (39.1)	154 (31.3)
Intraoperative + Post-Operative First 24 hours PRBC transfusion, n (%)	No	0	407 (67.6)	63 (57.3)	344 (69.9)	0.010
	Yes		195 (32.4)	47 (42.7)	148 (30.1)
Number of Bypasses, mean (SD)		0	1.1 (0.3)	1.1 (0.2)	1.1 (0.3)	0.740
BUN, mean (SD)		1	16.6 (7.1)	21.3 (10.4)	15.5 (5.6)	<0.001
OR Time (Hr), mean (SD)		0	7.3 (1.7)	7.9 (2.0)	7.2 (1.7)	0.001
Preoperative [TIMP-2]x[IGFBP7], mean (SD)		8	0.4 (0.5)	0.4 (0.5)	0.4 (0.6)	0.526
Preoperative BNP, mean (SD)		2	195.0 (360.3)	317.2 (463.3)	167.6 (327.2)	0.002

^*P-values are calculated between the training and test sets using a Mann-Whitney U test for continuous variables and a chi-square test for categorical variables. Created with use of open source Python package tableone.²⁵

Prediction Performance

We split the data randomly into a training (n = 481) and test (n = 121) set and report the performance results on the test set. We performed this procedure 100 times to acquire average performance results with 95% confidence intervals.

Random forest:

In predicting in-hospital CSA-AKI within our test set, the random forest model had a mean accuracy of 0.77 (0.76 - 0.79) and mean AUC of 0.73 (0.72 - 0.74). Sensitivity and specificity were 0.59 (0.57 - 0.62) and 0.81 (0.79 - 0.84), respectively. ROC curves including the mean ROC curve are shown in Figure 4. The confusion matrix, which includes proportions of true negatives, false positives, and additional metrics (F1 score, precision) is shown in Figure 5. The threshold selected for calculating sensitivity, specificity, precision and accuracy was the threshold which resulted in the maximum F1 score within the test set. Please see Supplementary Materials for information on the summary metrics and derivation of the mean ROCs. Variable importances for the random Forest model are provided in Figure 6. Variables of greatest importance to the model were derived from throughout the perioperative period and included modifiable variables such as hemodynamics and medications. Notably, neither BNP nor [TIMP-2]x[IGFBP7], the biomarkers included in the model, contributed significantly its predictive power.

Figure 4 –

100 Test set ROC curves with mean curve and AUC for Random Forest

Figure 5 –

Performances including confusion matrix for Random Forest model on the Test set

Figure 6 –

Feature Importance of Random Forest Model

Long short-term memory:

The long short-term memory model had a mean accuracy of 0.72 (0.69 - 0.74), sensitivity 0.60 (0.57 - 0.64), specificity 0.74 (0.71 - 0.78) (confusion matrix in Figure 7), and mean AUC of 0.68 (0.67 - 0.70) (ROC curves in Figure 8) for the prediction task of in-hospital CSA-AKI within our test set. Mean AUC, accuracy, precision and F1 score across the 100, independent random splits of the data were significantly higher for random forest model than the long short-term memory metrics (each with p < 0.01), based on paired Wilcox test (aka paired Mann-Whitney-U test), except for sensitivity (p = 0.52), where there was no statistical difference between the models.

Figure 7 –

Performances including confusion matrix for Long Short Term Memory model on the Test set

Figure 8 –

100 Test set ROC curves with mean curve and AUC for LSTM

Discussion

In this study, we used two different machine learning approaches – random forest and long short-term memory, both of which were able to accurately predict in-hospital CSA-AKI. These machine learning approaches allowed utilization of the granular features of high frequency perioperative hemodynamic data in ways which could not be leveraged using traditional multi-variable regression modeling. (Figure 1).

Previously published logistic regression-based models accurately predicted moderate to severe CSA-AKI^11,14-17. A few studies which did evaluate mild CSA-AKI had competitive results with regression modeling but were limited by their retrospective design¹⁰ or had clinically impractical sensitivity thresholds for detection³. For example, Vogt et al.¹⁰ achieved an AUC of 0.84 (0.81-0.86) using a retrospective cohort of 7633 patients but used post-operative GFR as a predictor. Karkouti et al.³ achieved an AUC of 0.78 (0.75-0.80) while having a sensitivity threshold of only 20% and using post-operative creatinine as a predictor. Because both of these models use post-operative renal function to predict CSA-AKI, they have limited utility in pre-operative risk stratification or planning early interventions. These studies also limited their predictive window to be within only the first two days post-operative.

Previous machine learning approaches used to predict in-hospital CSA-AKI incorporated perioperative variables and achieved comparable AUCs (0.74¹⁸, 0.78¹⁹, 0.84²⁰) but did not incorporate AKI related biomarker data²⁰, peri-operative hemodynamics¹⁸, or limited their evaluation to severe AKI outcomes¹⁹. While Tseng et al.¹⁹ used similar model inputs and evaluated similar outcomes as our study, their hemodynamic data was limited to heart rate and blood pressure which was chosen from a 6-hour observation window and excluded extracorporeal hemodynamics. Our model included 8 time-series hemodynamic variables including temperature, central venous pressure and mean arterial pressure during CPB. We also used an extended observation window which included the entire intra-operative period and 12 post-operative hours in the ICU.

Key features of our study were 1) the use of high frequency hemodynamic data from throughout the perioperative period, 2) inclusion of extracorporeal hemodynamic measurements during CPB, and 3) the incorporation of established biomarker data.

We were able to partially address some issues with black box phenomena typically encountered in complex machine learning models¹⁹ via variable importance rankings. Here the data for a given variable (e.g. surgery type) is permuted or scrambled across patients in the cohort. The model’s subsequent drop in prediction accuracy as a result is compared to the true accuracy of the unscrambled version of the data. This was reported for the top 20 variables by their change in accuracy (Figure 6). This list included known predictors of CSA-AKI such as demographics, comorbidities, surgery type, left ventricular ejection fraction; time-series hemodynamic variables such as heart rate and invasive pressures.

Race and ethnicity, which reflected the interplay between race and Hispanic versus non-Hispanic ethnicity, was the most important contributor of accuracy to our model. Race has been found to be an important predictor of CSA-AKI with a recent registry study of 34,520 cardiac surgery patients reported that Black patients had a 50% increased odds for having moderate-severe CSA-AKI compared with White patients²¹. This may reflect genetic predisposition as well as factors related to the social determinants of health including access to care, nutrition, social support and pre-operative medical optimization. The largest epidemiological studies evaluating CSA-AKI have enrolled predominantly Caucasian patients^15,16 or did not collect information about race and ethnicity.^17,22 Not surprisingly, race is only included in a single clinical model used to predict post-operative AKI¹⁵ and Hispanic ethnicity has not yet been incorporated into any models. Further analysis of our racially more diverse cohort of cardiac surgical patients may provide novel insights about unique risk factors and therapies relevant to these groups.

We believe that the variation in frequency of hemodynamic measurements during the perioperative period hindered our ability to enhance resolution of the data, and ultimately made it infeasible to provide raw, individual hemodynamic measurements directly from the EHR to the long short-term memory model. This likely could have strengthened the deep learning approach further. Instead, the long short-term memory model had to receive summary statistics from a given interval in the perioperative timeline sequence as input. In this iteration of our work, random forests showed to be more accurate and overall outperformed long short-term memory. Future studies with consistent hemodynamic measurement frequencies in the timeline could very well make a long short-term memory or other attention-based model such as a transformer-based architecture stand out.

The sensitivity of our models for detection of AKI was low (0.59-0.60), which limits their clinical utility. However, the mean AUC of 0.73 (0.72 - 0.74) of our random forests model exceeds the performance of the early post-operative [TIMP-2]x[IGFBP7] (pooled AUC 0.66²³) and BNP (AUC 0.63²⁴) biomarkers alone for prediction of CSA-AKI. The performance of our random forest model is less robust but comparable to other machine-learning based models developed to predict all-stage AKI using early postoperative measurements (AUC 0.77-0.78).¹⁹ The accuracy of our two models (0.77 and 0.72) was also similar to the accuracy of other machine learning models (pooled accuracy 0.73) in a meta-analysis of 38 studies which included 60 machine learning models.²⁵

Key limitations of our study include lack of external validation, algorithmic bias, selection bias, and chronology bias. While our dataset has not been externally validated, we hope to test our model on large external datasets to ensure that it performs robustly across other populations of cardiac surgical patients. Since AKI is associated with CKD development and other morbid outcomes, we also plan to create models using multi-modal perioperative inputs to predict longer term outcomes. Finally, in the future with more observation time gained on this cohort, we intend to extend this model’s inputs to span a timeline which includes the outpatient setting. This can potentially include more post-operative data and measures, such as mortality and Major-Adverse Kidney Events (MAKEs)²⁶. A second limitation to our modeling is algorithmic bias, the lack of clarity regarding how the machine learning inputs are analyzed to make predictions. We also attempted to mitigate these black box phenomena via variable importance evaluation. However, it is difficult to assess how the model determines patterns on the perioperative input data, particularly for the deep learning long short-term memory method. There is the potential of selection bias and chronology bias, as the parent study enrolled only a fraction of patients scheduled to undergo cardiac surgery over an 8-year period. Enrollment in the parent study was limited by the availability of funding and personnel, an inability to co-enroll patients with other ongoing studies, and willingness of patients to provide informed consent for the parent study which requires biosample collection and long-term follow-up. We attempted to overcome selection bias by standardizing patient recruitment interviews and utilizing Spanish language questionnaires. We attempted to overcome chronology bias by training our algorithms with key variables which may account for any variation in clinical practice that may have occurred during the study.

Conclusions

We developed a perioperative predictive model for in-hospital CSA-AKI from a cohort of cardiac surgery patients. Our machine learning models utilize a wide range of multimodal perioperative data including demographics, comorbidities, surgical characteristics, high frequency perioperative hemodynamic measurements, conventional laboratory tests including BNP and [TIMP-2]x[IGFBP7] biomarkers collected at fixed time perioperative time points (Figure 2). While both having comparable results, our random forest decision-tree based approach outperformed the deep learning long short-term memory model. The results of our study can be used as preliminary data to create a clinical decision tool for early and accurate prediction of in-hospital CSA-AKI. Ultimately, the goal of predictive modeling in this setting is to reduce the common but costly outcome of CSA-AKI.