Including urinary output to define AKI enhances the performance of machine learning models to predict AKI at admission

Abstract

Purpose:

Acute kidney injury (AKI) is a prevalent and detrimental condition in intensive care unit patients. Most AKI predictive models only predict creatinine-triggered AKI (AKI_Cr) and might underperform when predicting urine-output-triggered AKI (AKI_UO). We aimed to describe how admission AKI_Cr prediction models perform in all AKI patients.

Materials and methods:

Three types of models were trained: 1) pAKI_any, predicting AKI based on creatinine or urine output, 2) pAKI_UO, predicting AKI based only on urine output, and 3) pAKI_Cr, predicting AKI based only on creatinine. We compared model performance and predictive features.

Results:

The pAKI_any models had the best overall performance (AUROC 0.673–0.716) and the most consistent performance across three patient cohorts grouped by type of AKI trigger (min AUROC of 0.636). The pAKI_Cr models had fair performance in predicting AKI_Cr (AUROCs 0.702–0.748) but poor performance predicting AKI_UO (AUROCs 0.581–0.695). The predictive features for the pAKI_Cr models and pAKI_UO models were distinct, while top features for the pAKI_any models were consistently a combination of those for the pAKI_Cr and pAKI_UO models.

Conclusion:

Ignoring urine output in the outcome during model training resulted in models that are unlikely to predict AKI_UO adequately and may miss a substantial proportion of patients in practice.

1. Introduction

Acute kidney injury (AKI) commonly affects critically ill patients and adversely impacts patient outcomes such as mortality, ICU and hospital length of stay, hospital cost, and post-discharge quality of life [1,2]. Based on the Kidney Disease Improving Global Outcomes (KDIGO) guidelines, AKI is classified into three stages based on decreases in urine output (oliguria/anuria) or increases in serum creatinine levels [3]. AKI at higher stages is associated with worse outcomes. Since AKI is often initially asymptomatic, its diagnosis is commonly delayed or missed entirely [4]. Tremendous research efforts focusing on predicting AKI using data from hospital or ICU stay have been implemented to avoid missed opportunities for appropriate preventive care [5–9]. The majority of previous models do not include urine-output-triggered AKI (AKI_UO) as an outcome, focusing on the more-readily available measured creatinine to define AKI. This is a startling omission since AKI_UO is more prevalent and still associated with higher mortality, particularly at higher stages [10,11].

In this study, we aimed to evaluate the impact of using either or both criteria of the AKI definition in the development and performance of prediction models [12]. Using multiple machine learning techniques, we built models to predict three groups of AKI patients: 1) pAKI_any to predict AKI_any (triggered by either creatinine or urine output), 2) pAKI_Cr to predict AKI_Cr (creatinine-triggered AKI), and 3) pAKI_UO to predict AKI_UO (urine output-triggered AKI). We compared the performance and important features of these models. Additionally, we conducted sensitivity analyses to assess how well the pAKICr models performed in identifying patients whose AKI was detected by a decline in urine output, an increase in serum creatinine, or both.

2. Materials and methods

2.1. Study population

We screened all adult intensive care unit (ICU) admissions to the Mayo Clinic Hospital (Rochester, MN) from January 1, 2005, to December 31, 2017. All adult, non-pregnant subjects who provided research authorization were included. We combined admissions from a single patient <10 h apart into a single encounter. Encounters that 1) received renal replacement therapy before or during ICU stay, 2) had a baseline creatinine >5 mg/dL, 3) had insufficient creatinine measurements or urine chartings (no measurements of creatinine or fewer than two non-zero urine output measurements), or 4) had community-acquired AKI were excluded.

This study was reviewed and approved by the Mayo Clinic institutional review board (IRB# 07–001380) and the Philips Research Internal Committee for Biomedical Experiments. The need for informed consent was waived due to the minimal risk of this retrospective study.

2.2. Data extraction and definitions

Data extracted from the ICU database included both predictor features and outcome variables. Predictor variables consisted of patient demographics and comorbidities, history of nephrotoxic medication exposure within the six months before the index hospital admission, admission and diagnosis information, and baseline creatinine when available (Appendix Table 2). The outcome variables included hourly urine outputs, serum creatinine, and weight measurements over the length of ICU stay. Appendix A provides definitions of various machine learning terms [13].

We standardized continuous features such as age and height using Z-score standardization. Missing values were imputed using mean-imputation, though few values were missing (Appendix Table 3). Nephrotoxic drug history was converted to counts of drugs prescribed within 15 pre-defined categories; drugs not mentioned in the patient history were assumed not to have been prescribed (see Appendix B.2). We considered primary diagnoses (defined by ICD-9/10 codes) when they were prevalent in >0.01% of all patients and were either admission diagnoses or reflected comorbid conditions. Further details are provided in Appendix B.

AKI stages throughout ICU stay were calculated using an existing electronic implementation of the KDIGO criteria [14]. Baseline creatinine was defined as the average of all serum creatinine measurements during the 180 to 7 days before the index hospital admission or estimated using the MDRD equation back-calculation [3,15,16], assuming an eGFR of 60 mL/min per 1.73 m² [14]. Each encounter was assigned three distinct maximum stages, which were assessed throughout the ICU stay (Fig. 1). There were three distinct labels based on the maximum stages: 1) creatinine-triggered AKI stage > 0 (AKI_Cr), 2) urine-output-triggered AKI stage > 0 (AKI_UO), and 3) overall AKI stage by either criterion > 0 (AKI_any). See Appendix C for additional details.

Fig. 1.

Overview of model training. The original 131,873 ICU admissions are combined into encounters and filtered (see Section Data Extraction and Definitions). Predictive features are extracted for each encounter; then, the AKI staging by urine output and by creatinine is calculated over the entire ICU stay for each encounter. The AKI staging is simplified into three labels based on whether the stage was ever above 0: a urine-output-triggered AKI label (AKI_UO), a creatinine-triggered AKI label (AKI_Cr), and an AKI_any label (triggered by either urine output or creatinine). Lighter colors indicate stage 0 (No AKI), and darker colors stages greater than 0. These labels are then combined with the predictive features to build three separate models (pAKI_Cr, pAKI_UO, and pAKI_any) using each of four architectures (neural network, gradient boosting, logistic regression, and random forest).

2.3. Model training and evaluation

The data were split randomly into a training cohort (80% of the samples) and a testing cohort (20%) (non-stratified). The train-test split was constant across all models so that no test data were ever used during model training. All models were fit in Python version 3.7.3. Four architectures were employed: gradient boosting, logistic regression, random forest (all implemented in scikit-learn version 0.20.3 [17]), and a PyTorch neural network (torch version 1.2.0 [18]). Each architecture was used to build three models, one to predict each of the three labels, for a total of twelve models. The full details for fitting these architectures are outlined in Appendix D.

Overall model performance was quantified using the area under the receiver operating characteristic (AUROC) and area under the precision-recall curve (AUPRC) and evaluated for detecting AKI_any or detecting three disjoint subsets of AKI_any patients (AKI_UO only, AKI_Cr only, both AKI_Cr and AKI_UO; see Appendix E). To quantify false positive/negative rates, thresholds were also selected for each model (Appendix F).

Feature importance was evaluated using SHAP (SHapley Additive exPlanations) values [19], calculated using the python package shap, version 0.30.0.

3. Results

3.1. Cohort characteristics

Among 131,873 screened ICU admissions, 98,472 encounters met all eligibility criteria for the final analyses (Fig. 1). The incidence of AKI at any time during ICU stay was 40%. Patients with AKI were older and had higher baseline creatinine levels, a higher mortality rate, and longer ICU stays (Table 1). Of patients with AKI, most (61%) had a maximum stage one (Appendix Table 4). As expected, mortality rates (both in-hospital and ICU) and ICU length of stay increased with increasing AKI stage. Hospital mortality was 6.4% in stage 1 and 14.5% in stage 3. Median ICU length of stay was 2.0 days in stage 1 and 4.0 days in stage 3. Patients with AKI stage 3 tended to be slightly younger, were less likely to have baseline creatinine measured, and had lower Charlson Comorbidity Index scores (Appendix Table 4). However, patients with stage 3 had generally more severe illness, evidenced by being more likely to have a previous ICU stay (Appendix Table 4); and more likely to be admitted with kidney dysfunction, sepsis, and respiratory failure (Appendix Table 5).

Table 1.

Baseline characteristics of patients with and without AKI. A summary of selected demographic and outcome variables in encounters with and without AKI (by either criterion) in our cohort.

Feature	No AKI	Any AKI	p-value
ICU stays#	59,165 (60.1%)	39,307 (39.9%)
Age (years)*	61.96 (17.12)	65.43 (16.3)	<0.001
Length of ICU stay (hours)+	25.83 (23.5)	60.55 (85.62)	<0.001
Charlson Comorbidity Index*	4.02 (2.43)	4.67 (2.47)	<0.001
Readmit#	3686 (6.2%)	3098 (7.9%)	<0.001
Gender (female)#	24,735 (41.8%)	16,372 (41.7%)	0.63
Race (black)#	806 (1.4%)	567 (1.4%)	0.292
Baseline creatinine (mg/dL)*	1.04 (0.43)	1.13 (0.52)	<0.001
Death in Hospital#	1585 (2.7%)	3588 (9.1%)	<0.001
Death in ICU#	583 (1.0%)	1961 (5.0%)	<0.001
Baseline creatinine, when available	33,741 (57.0%)	23,717 (60.3%)	<0.001

^#indicates features described by “count (%),” and tested using Fisher’s exact test;

⁺indicates features described by “median (IQR)” and tested using a Wilcoxon rank-sum test;

^*indicates features described by “mean (SD)” and tested using a two-sample t-test with unequal variances.

P-values test the difference between AKI and non-AKI encounters. Across all metrics except the proportion of black encounters and the proportion of female encounters, AKI patients are significantly different from non-AKI patients.

In comparison with AKI_Cr, AKI_UO was substantially more common. Among all AKI patients, 46% never triggered the creatinine criterion, and only 28% had AKI triggered by both AKI criteria (Appendix Table 6). Patients who triggered both criteria tended to have a substantially longer length of stay and higher mortality rate than patients with AKI_UO only or AKI_Cr only (Appendix Table 6). The staging by urine output had a substantial effect on mortality, independent of staging by creatinine. Among patients with AKI_Cr stage 3, those with AKI_UO stage 0 had a mortality rate of 3%, while those with AKI_UO stage 3 had a mortality rate of 28% (p-value <0.001) (Appendix Table 7).

3.2. Model performance

We evaluated the performance of all models on the AKI_any label using the samples from the test set. Of all the architectures evaluated, the neural network architecture had the best performance measured by both AUROC and AUPRC (Fig. 2). For each architecture, pAKI_any models had the best performance in predicting AKI_any both by AUROC and AUPRC. The pAKI_Cr models had slightly lower performance than pAKI_any models (a drop in AUC ranging from 0.02–0.03).

Fig. 2.

Model performance predicting any AKI. The AUROC (top) and AUPRC (bottom) values were calculated for each of the twelve models when they were used to predict any AKI in the held-out test set.

The pAKI_Cr models had much higher false-positive rates and slightly lower false-negative rates than pAKI_UO or pAKI_any models (Appendix Fig. 1). Positive and negative predictive values were relatively consistent independent of the AKI label, though both tended to be slightly higher for pAKI_any models (Appendix Fig. 1). The proportion of missed AKI patients who only had AKI_UO was 63.7%–73.9%, 56.3%–60.5%, and 47.3%–51.8% for the pAKI_Cr, pAKI_any, and pAKI_UO models, respectively (Appendix Fig. 2).

The performance of the models on subsets of patients grouped by AKI defining criteria also revealed performance differences. In identifying patients with only AKI_UO, AUROCs of pAKI_Cr models were 0.571 to 0.6 (Fig. 3). This is while the pAKI_Cr models performed with higher AUROCs in identifying patients with only AKI_Cr (increased AUROCs 0.113–0.148). The pAKI_any models AUROCs in detecting patients with only AKI_UO was just 0.03–0.05 lower than identifying patients with only AKI_Cr. For patients who reached the AKI definition by both criteria, all models uniformly had the highest AUROCs.

Fig. 3.

Model performance predicting disjoint subsets of AKI patients. The AUROC (top) and prevalence-corrected AUPRC (bottom) values were calculated for each of the twelve models when distinguishing each of three cohorts and controls. See Section Model Training and Evaluation and Appendix E for details.

3.3. Feature importance

We evaluated feature importance on the best-performing overall model architecture, the neural network (Fig. 3), though similar results were obtained for the other architectures (Appendix Figs. 3–5) The most important features were quite different for the pAKI_Cr (Fig. 4A) and pAKI_UO (Fig. 4B) models. When predicting AKI_UO, demographics such as weight, body mass index (BMI), age, and gender were all highly influential. Higher weights and BMI values, as well as greater age or female gender, were predictive of developing AKI_UO during ICU stay. By contrast, for the pAKI_Cr model, comorbidities, including chronic kidney disease (CKD), coagulation disorders, congestive heart failure (CHF), and hypertension, were among the most important features. The presence of these comorbidities, as well as higher values for the Charlson Comorbidity Index, were predictive of developing AKI_Cr during ICU stay.

Fig. 4.

Feature importance for the neural network models evaluated by SHAP values. The SHAP values for all encounters in the held-out test set (see Section Model Training and Evaluation) for the top 10 features for the pAKI_Cr model (A), pAKI_UO model (B), and pAKI_any model (C). The mean absolute SHAP value determines the feature ranking across all encounters for that feature. The x-axis is the SHAP value, measuring the impact of a particular feature on the prediction for a particular encounter: negative values indicate a decreased risk of AKI, while positive values indicate an increased risk. The colors indicate the feature value, shifting from bright blue (the minimum value for a feature) to bright red (the maximum value for a feature). Binary features, which have two values, thus have only bright blue and bright red points in this plot.

The most important features of the pAKI_any model were a combination of the critical features for the pAKI_UO and pAKI_Cr models (Fig. 4C). The top feature for predicting AKI_any was weight, which was also the most predictive feature for AKI_UO. Features among the top ten for both AKI_UO and AKI_Cr were also quite predictive of AKI_any, including the presence of coagulation disorders, cardiac dysrhythmias or congestive heart failure, and not being admitted in the neurosurgery unit.

4. Discussion and conclusions

We found that regardless of model architecture used, including the urine output in the model training resulted in improved performance, particularly in detecting patients with AKI_UO only. While pAKI_any models did exhibit higher performance, the gains were modest. Further, the negative predictive values of pAKI_any were similar regardless of the criteria used for the AKI definition.

While serum creatinine levels and urine output are both indicators of kidney function, they provide slightly different information about kidney physiology. Serum creatinine level is a function of glomerular clearance of creatinine with a supposedly constant rate of production [20]. This is while urine output is a function of glomerular filtration and tubular processing. Therefore, it is not surprising to note differences between factors that can predict an increase in serum creatinine versus a decrease in urine output when machine learning tools identify the variables without supervision. In agreement with previous studies [10], we found that AKI_UO was more prevalent than AKI_Cr. AKI_UO often existed in the absence of AKI_Cr, and both were associated with high mortality rates. Both AKI_UO and AKI_Cr were associated with higher mortality rates and longer ICU lengths of stay (Appendix Table 6). The high prevalence of AKI patients who experience only a decline in urine output raises the possibility that training pAKI_Cr models might lead to missing a substantial portion of AKI patients. Dissecting the AKI patients missed by each model revealed essential differences. The pAKI_any models performed considerably more consistently in predicting AKI_UO and AKI_Cr. Interestingly, all models typically missed more AKI_UO, suggesting prediction and detection of AKI_UO is more difficult at ICU admission than AKI_Cr.

We demonstrated that the predictive features of AKI_Cr and AKI_UO are different, highlighting the variabilities between these two criteria when identifying patients with AKI. The presence of comorbidities, such as CKD, coagulation disorders, hypertension, and CHF, was important for predicting AKI_Cr. The presence of CKD, in particular, might be important. Generally, CKD patients have a higher availability of measured baseline serum creatinine. Estimating baseline creatinine by back-calculating from the MDRD equation would likely result in lower baseline creatinine among CKD patients. Conversely, demographics such as weight, BMI, and age were the most important for predicting AKI_UO. Weight might be important because urine output is scaled by weight when determining the AKI stage for the label. AKI is defined by an increase in serum creatinine level and a weight-adjusted decline in urine output [12]. It appears that the size of organs in different individuals is based on their height. For instance, lung size should be assessed based on the ideal body weight, which is calculated using height [21]. While serum creatinine production in larger individuals is higher, as their kidney sizes are bigger as well, the blood serum creatinine levels among individuals are comparable, yet to evaluate the estimated GFR adjustments for race, age, and gender need to be considered [20]. In the urine output case, larger individuals generate more osmoles each day (average 5–10 mosmol/kg/day with American diet); therefore, given comparable urinary concentrations, they need to generate more urine to excrete all generated osmoles. Thus, the urine output criterion of AKI is adjusted based on weight (i.e., ml/kg/h) [10,22–25]. The predictive features of AKI_any were a combination of the top features of pAKI_Cr and pAKI_UO. Higher weights and BMI values were highly predictive for AKIUO and AKIany, while coagulation disorders and CHF were highly predictive for AKI_Cr and AKI_any. Although CKD was a significant predictor of AKI_Cr, it was not as important in predicting AKI_any or AKI_UO. Interestingly, admission in the neurosurgery unit was associated with a decreased AKI risk regardless of AKI definition, likely reflecting that such patients are mostly younger and healthier, with a consequently lower risk of AKI.

Similar to retrospective studies, our research is not immune to limitations, including limited ability to remove community-acquired AKI patients and the single-center nature of the dataset used to train the models. While we excluded all patients with known community-acquired AKI, we had no access to urine output measurements before ICU admission. Further, measured creatinine levels before ICU admission were only available in a subset of patients. Thus, we could only exclude a subset of patients who had community-acquired AKI_Cr. This may result in selection bias in our findings. Having mentioned this, a large proportion (90%) of AKI_UO patients had normal urine output and creatinine measurements before the development of AKI. Obviously, there may be some differences among those with previously measured serum creatinine with those who never had their serum creatinine assessed. The differences could be in their comorbidity profile (e.g., individuals with other comorbidities with potentially lower kidney function reserve are more likely to have measured serum creatinine) or healthcare access. While assessing these differences is not in the scope of this manuscript, it should be considered for prospective studies. Although our models were trained using a large data set over multiple years, the data did come from a single-center, which may impact the generalizability of our results. As the dataset used for this analysis is one of the few datasets with hourly urine output information, the validation of our results in other datasets was not feasible. By contrast, the strengths of our study include a large dataset of ICU patients, availability of urine output data, using several architectures to develop higher-performing models and internal validation of our models.

In conclusion, we demonstrated that models trained to predict ICU-acquired AKI_Cr at ICU admission are ill-suited to predict AKI_UO. This work highlights the importance of accounting for AKI_UO during model training.