Predicting acute kidney injury in critically ill patients using comorbid conditions utilizing machine learning

Abstract

Background

Acute kidney injury (AKI) carries a poor prognosis. Its incidence is increasing in the intensive care unit (ICU). Our purpose in this study is to develop and externally validate a model for predicting AKI in the ICU using patient data present prior to ICU admission.

Methods

We used data of 98 472 adult ICU admissions at Mayo Clinic between 1 January 2005 and 31 December 2017 and 51 801 encounters from Medical Information Mart for Intensive Care III (MIMIC-III) cohort. A gradient-boosting model was trained on 80% of the Mayo Clinic cohort using a set of features to predict AKI acquired in the ICU.

Results

AKI was identified in 39 307 (39.9%) encounters in the Mayo Clinic cohort. Patients who developed AKI in the ICU were older and had higher ICU and in-hospital mortality compared to patients without AKI. A 30-feature model yielded an area under the receiver operating curve of 0.690 [95% confidence interval (CI) 0.682–0.697] in the Mayo Clinic cohort set and 0.656 (95% CI 0.648–0.664) in the MIMIC-III cohort.

Conclusions

Using machine learning, AKI among ICU patients can be predicted using information available prior to admission. This model is independent of ICU information, making it valuable for stratifying patients at admission.

INTRODUCTION

The prevalence of acute kidney injury (AKI) among patients admitted to intensive care units (ICUs) continues to increase [1]. The incidence of AKI requiring dialysis has increased by 10%/year over a decade [2, 3]. Being recognized as a serious event, AKI continued to correlate with adverse outcomes [4–8]. These include increased length of stay in the ICU or hospital, chronic and end-stage kidney disease, need for renal replacement therapy (RRT) at discharge or death [6, 9, 10]. While there are no effective treatments available for AKI, prediction and early detection can potentially help with primary and secondary prevention of AKI, respectively. In the Prevention of AKI Study, patients at high risk of AKI were identified using urinary biomarkers and were randomized to receive either standard care or implementation of the Kidney Disease: Improving Global outcomes (KDIGO) bundle [11]. The occurrence of AKI in the intervention group was significantly lower (55.1% versus 71.7%; P = 0.004) compared with the control group. In patients who underwent major elective noncardiac surgery, those randomized to the KDIGO care bundle had a lower incidence of moderate or severe AKI compared with the control group (6.7% versus 19.7%; P = 0.04) [12].

There are several risk factors for the development of AKI, including older age [13], exposure to nephrotoxins, need for mechanical ventilation or vasopressor support, sepsis, volume overload and presence of comorbid medical conditions [14–16]. There have been efforts to identify patients at high risk of developing AKI through both designing simple clinical scores that can inform clinicians of potential risk and the use of data available in electronic health records (EHRs) to generate automated alerts for use by treating clinicians [17–23]. Although several AKI prediction systems have been published, none of these risk scores has been universally adopted in clinical practice [24]. The use of machine learning in predicting AKI has been increasing in recent publications. Kate et al. [25] developed two models to predict AKI within 24 h of admission and later during the hospital stay. In another study using fuzzy logic systems, Argyropoulos et al. [26] developed a model to predict AKI within 7 days of admission. A more recent study by Koyner et al. used a gradient-boosting model to predict the risk of Stage 2 AKI, with an area under the receiver operating curve (AUROC) of 0.90 for predicting Stage 2 AKI within 24 h and 0.87 within 48 h [27].

Most of the above-named models rely heavily on ICU data that are unlikely to be available on admission. Frequently, the performance of the models improves over the course of ICU admission, but is suboptimal within the first few hours of admission when the volume and availability of data are lowest.

In this study, we aimed to utilize machine learning to develop a model that only relies on comorbid medical conditions available before ICU admission to predict the risk of ICU-acquired AKI.

MATERIALS AND METHODS

Study design

This was a retrospective analysis of adult patients admitted to ICU at Mayo Clinic in Rochester, MN from 1 January 2005 to 31 December 2017. All adult (age >18 years), nonpregnant subjects who provided research authorization were included. This study was reviewed and approved by the Mayo Clinic Institutional Review Board and the Philips Internal ethical review board, and the need for informed consent was waived due to minimal risk. We excluded patients who received RRT before or during ICU stay. It was difficult to ascertain the timing of RRT in relation to AKI occurrence, especially since the KDIGO guidelines assign Stage 3 for patients undergoing RRT (irrespective of urine or creatinine values). We also excluded patients who had a baseline creatinine >5 mg/dL, or did not have more than one creatinine measurements or urine output documentation.

We collected patient demographics and baseline serum creatinine. Baseline creatinine was defined as the mean of all creatinine measurements in the 180–7 days before hospital admission [28, 29]. When baseline creatinine was not available, the Modification of Diet in Renal Disease (MDRD) equation [30] was used to estimate it. We also collected diagnosis codes based on the International Classifications of Disease (ICD) 9 and 10. To identify AKI, we collected data on hourly urine outputs, serum creatinine and weight measurements over each patient’s ICU stay. AKI stages were calculated using an existing electronic implementation of the KDIGO criteria [31, 32]. This tool uses urine and creatinine criteria to define AKI. The maximum AKI stage during an encounter was used as the final AKI stage.

Feature processing and model training

We used demographics, comorbidities and admission diagnoses to build a classifier for AKI prediction. We standardized continuous features (e.g. age, height and weight) as described in the Supplementary data. Categorical features (i.e. diagnoses) were converted into binary variables based on domain knowledge (described in Supplementary data). The ICD-9/10 codes were grouped into 258 categories. Only diagnoses indicating chronic conditions or admission diagnoses were selected as features. Chronic kidney disease (CKD) was defined based on the diagnosis codes.

The dataset from Mayo Clinic cohort was split into 80% for training and 20% test. The training data were split into 6-folds. Five gradient-boosting models were trained on five of these folds. Features that were selected by all five models were retained, this resulted in 41 features.

Hyperparameters

Tree depth and minimum samples per split were optimized by grid search. Next, the effect of the number of features on model performance was evaluated. Models were trained using optimized hyperparameters and a varying number of features. Five models were trained with 56, 41, 30, 20 and 10 features, respectively. Model performance was evaluated using the AUROC, area under the precision-recall curve (AUPRC) and F1 score. The F1 score, as a combined measure of precision and recall, is used to evaluate the success of a binary classifier when one class is rare [33]. The performance of these models is shown in Supplementary data, Figure S1. The gradient-boosting model was implemented using the open-source python package Scikit-learn [34] for all the models. We used the Shapley Python package to calculate SHapley Additive exPlanations (SHAP) values, a measure of feature importance that explores the nature of the relationship between the feature and the outcome [35].

Model validation using MIMIC

In addition to internal validation, we externally validated the model using the Medical Information Mart for Intensive Care III (MIMIC-III) dataset [36]. MIMIC-III is a publicly available critical care database from Beth Israel Deaconess Medical Center, Boston containing records of all ICU admissions from 2001 to 2012. All eligible patient records were extracted using the same exclusion criteria as the Mayo Clinic dataset.

RESULTS

Following the initial screening of 131 873 ICU admissions at Mayo Clinic, 98 472 encounters met all the eligibility criteria (Supplementary data, Figure S2). There were 39 307 (39.9%) encounters that developed AKI in the Mayo Clinic group. Out of 51 801 included patients from the MIMIC-III cohort, 18 529 (46.2%) met eligibility criteria. AKI by urine criteria was met in 29 004 (29.5%) encounters in the Mayo Clinic cohort and 6757 (36.5%) encounters in the MIMIC-III cohort. AKI by creatinine criteria was met in 21 162 (21.5%) encounters in the Mayo Clinic cohort and 4798 (25.9%) encounters in the MIMIC-III cohort. In both groups, patients who developed AKI in the ICU were older, had higher weights, and had higher ICU and in-hospital mortality compared with patients who did not develop AKI (Table 1). In the Mayo Clinic cohort, patients who did not develop AKI had a median hospital stay before ICU admission of 5.8 h versus 5.9 h in patients who developed AKI. Also, patients who later developed higher stages of AKI spent less time on a general ward care before ICU admission. There were some differences between the two datasets; a more substantial proportion of patients in MIMIC-III did not have available admission weight (56% versus 10% in the Mayo Clinic cohort).

Table 1.

Baseline demographics

	Mayo clinic cohort			MIMIC-III cohort
Variable	AKI (n = 39 307)	No AKI (n = 59 165)	P-value	AKI (n = 8560)	No AKI (n = 9969)	P-value
Male, n (%)	16 372 (41.7)	24 735 (41.8)	0.63	3556 (41.5)	4176 (41.9)	0.63
African American, n (%)	567 (1.4)	806 (1.4)	0.29	572 (6.7)	668 (6.7)	0.98
Readmissions, n (%)	3098 (7.9)	3686 (6.2)	<0.001	Not available	Not available
ICU mortality, n (%)	1961 (5.0)	583 (1.0)	<0.001	954 (11.1)	332 (3.3)	<0.001
In-hospital mortality, n (%)	3588 (9.1)	1585 (2.7)	<0.001	1286 (15.0)	569 (5.7)	<0.001
Age, mean (SD), years	65.43 (16.3)	61.96 (17.12)	<0.001	66.3 (15.8)	61.1 (17.9)	<0.001
Weight, mean (SD), kg	89.14 (26.59)	82.14 (21.93)	<0.001	84.2 (23.3)	77.4 (19.2)	<0.001
Baseline serum creatinine, mean (SD), mg/dL	1.13 (0.52)	1.04 (0.43)	<0.001	1.29 (0.91)	1.08 (0.58)	<0.001
Baseline serum creatinine available	23 717 (60.3)	33 741 (57)	<0.001	2363 (27.6)	2483 (24.9)	<0.001
Reasons for ICU admission
Sepsis	2680 (6.8)	2282 (3.85)	<0.001
Heart valve surgery	2598 (6.6)	4368 (7.4)	<0.001
Myocardial infarction	2284 (5.8)	3125 (5.3)	<0.001

Of the final five models that were trained, the 30-feature model had the best trade-off between minimizing the number of features and minimizing the drop in performance. Supplementary data, Figure S1 shows the AUROC curve and AUPRC curves for the five models on the Mayo Clinic dataset. Similar results were observed in the MIMIC-III dataset.

The relative importance of features in the model, calculated using GINI (mean decrease in impurity, is a metric used in decision trees to determine the relative importance of a feature in predicting a specific outcome) importance for the 30 feature model is shown in Figure 1.

FIGURE 1:

Feature importance. Feature importance is ranked in a descending order based on GINI importance for the 30 feature model. Feature importance was calculated on the Mayo Clinic dataset.

The 30-feature model had an acceptable performance in the test dataset (20% of Mayo Clinic dataset) with an AUROC of 0.690 [95% confidence interval (CI) 0.682–0.697] (Figure 2A) and F1 score of 0.611 (Figure 2B). For comparison, the model with 41 features had only a slight improvement in performance (AUROC of 0.692 and F1 score of 0.614) (Supplementary data, Figure S1). Table 2 shows additional performance metrics for the Mayo Clinic and MIMIC datasets. At optimal threshold (chosen based on precision-recall threshold), the specificity of the model in the Mayo Clinic test dataset was 0.71 and 0.64 in the MIMIC-III dataset for ICU acquired AKI based on the comorbid medical conditions.

FIGURE 2:

Model performance in predicting AKI using (A) AUROC and (B) precision-recall for both datasets.

Table 2.

Model performance

Dataset	AUROC	AUPRC	F1 score	Accuracy	Threshold	Precision	Recall	Specificity	NPV
MayoClinic
Train	0.707	0.607	0.613	0.669	0.430	0.575	0.575	0.719	0.718
Test	0.690	0.585	0.611	0.652	0.430	0.569	0.562	0.711	0.710
MIMIC-III
All	0.656	0.602	0.634	0.581	0.347	0.583	0.583	0.642	0.642

NPV, negative predictive value.

We also used SHAP values to estimate feature importance as it additionally provides importance for feature values. As with the GINI importance, higher body mass index (BMI) at hospital admission, the presence of CKD, presence of congestive heart failure (CHF), coagulation and hemorrhagic disorders, and presence of cardiac dysrhythmias were key drivers of increased risk of ICU acquired AKI (Figure 3).

FIGURE 3:

Relationship between each feature and risk of AKI. This analysis was performed on data from the Mayo Clinic test cohort. The horizontal axis shows the relationship between the feature and risk of AKI in the ICU. A positive value means increased risk of AKI and a negative means less risk of AKI. The color indicates the value of the feature where high value (or presence of that feature in case of categorical features) is coded in red and a low value (or absence of that feature in case of categorical features) is coded in blue.

The distribution of the 30 features differed between the Mayo Clinic and MIMIC-III datasets (Supplementary data, Figure S3), also the model performed slightly worse in the MIMIC-III dataset [AUROC of 0.656 (95% CI 0.648–0.664) and F1 score of 0.634] (Figure 2). The importance of the features and their relationship with the risk of developing AKI was similar between the two datasets as well (data not shown).

DISCUSSION

In this study, we used a large, comprehensive dataset from the Mayo Clinic to develop a gradient-boosting model that only uses pre-ICU-admission information to predict AKI in patients admitted to the ICU. Our final model used 30 features, and demonstrated acceptable performance both on a held-out test set from the Mayo Clinic and an external validation set from MIMIC-III dataset. In both the Mayo Clinic and the MIMIC-III cohorts, known risk factors for AKI were found to be predictive of AKI, particularly a prior diagnosis of CKD.

The performances of currently available models vary depending on the population of interest and data resolution, and improve with increased volume and resolution of the available data. While information regarding the susceptibilities for AKI development is often available at the time of ICU admission, data on exposures, laboratory results and hemodynamics are scarce. When these data become available after a few hours in ICU, current machine learning models start performing better [23]. Using KDIGO criteria, 57% of patients met the criteria for AKI on Day 1 of ICU stay [10]. In the Mayo Clinic cohort, the median time to development of ICU-acquired AKI was 16 h. This gives a short window of opportunity for identifying high-risk AKI patients in ICU, and the first few hours of ICU are particularly critical for such determinations. Hence, having access to a model to identify high-risk patients within the first few hours of ICU admission, whether alone or in conjunction with other models, can identify high-risk patients earlier and provide better opportunities to implement appropriate preventive measures.

The use of machine learning in predicting AKI (or its prognosis) has increased in recent publications [15, 23, 25–27]. These studies are either small, do not include advanced CKD, include older patients, are not specific to AKI acquired in the ICU, or recruit patients from a single medical center. In addition, the previous models were rarely validated externally. Unlike the previous studies, we chose to focus on information that was known at the time of ICU admission and included only prevalent comorbid conditions that are easily obtained from the electronic health records. While trying to optimize precision and recall, based on the thresholds chosen, we were able to have a specificity of 71% in the Mayo Clinic cohort.

The features retained in the final model were diverse; some were comorbid medical conditions (such as CKD and CHF), others were diagnoses that were acquired during the index hospital admission (such as sepsis and respiratory failure). Similar to other studies, CKD was a common feature that predicted ICU-acquired AKI. While we did find increased BMI or weight to be associated with the risk of AKI, only Kate et al. have evaluated BMI, and they did not find an increased risk with AKI [25]. Any explanation of the association is only speculative, but it could be that fluid overload on admission was contributing to the heavier weight or higher BMI on admission. This is especially relevant as the increase in these two parameters (included as independent variables) were also associated with increased risk of AKI. Fluid overload is known to be associated with AKI [37–40]. It is challenging to make further comparison between the features retained in our study with other published studies given the nature of the features included (diagnoses versus clinical and laboratory information). However, several features in the other published studies might share a similar theme; for example, ‘shock’ and ‘septicemia’ in our study share a similar clinical significance to ‘severe infection/sepsis’ in the study by Malhotra et al. [22] along with the three other features, i.e. ‘lowest systolic pressure’, ’highest heart rate’ and ‘current weight count’ [27]. Similar to Argyropoulos et al., age was also associated with increased risk of AKI [26]. In the study by Kate et al. age was not associated with an increased risk with AKI [25]. CHF was also found to be a risk factor for ICU-acquired AKI, which is in accordance with published literature [14, 22]. The differences in the risk factors for AKI among different studies have to do with the setting AKI is being defined in and the factors investigated. In our cohort, we looked at demographic information and diagnoses; however, other studies looked at laboratory values. The interaction between the underlying comorbid conditions likely set the right milieu for AKI when fluid and hemodynamic derangement occurs later during the ICU stay. While baseline creatinine was not a significant factor in the model, the relationship between higher baseline creatinine and lower risk of subsequent AKI was intriguing. Some potential explanation is that patients with CKD often receive kidney-protective interventions like angiotensin-converting enzyme inhibitors or angiotensin receptor blocker. In the ICU, these medications are often held, and therefore patients’ serum creatinine levels decrease due to the intraglomerular hemodynamic changes. Similarly, in the mild AKI setting, serum creatinine may not increase (balances out). Another potential explanation is that patients with CKD are generally treated differently in ICU. For example, the treating team might be more cautious when considering medications with potential nephrotoxic medications (hesitation in the use of contrast, for example), and this could have resulted in our observation. This observation, however, needs to be studied more closely in future studies, as it may have additional clinical implications.

AKI in critically ill patients continues to be independently associated with mortality [6, 9, 10]. There have been multiple studies showing that nephrology services are often requested later in the course of AKI, where management is limited to providing renal replacement therapies [41]. There has been a call to adopt a more proactive role in the prevention and management of AKI [41]. The main preventive interventions are the optimization of effective blood volume, maintaining mean arterial pressure and avoiding nephrotoxins, all of which are very time sensitive [41]. These efforts are aimed at detecting such patients early in order to provide potentially preventive measures. Currently, outside of prevention and management strategies, there are no therapeutic options available for AKI. Since the use of EHR has been prevalent in the USA, several studies have demonstrated that using electronic alert systems improves patient outcomes, including lowering AKI incidence and progression [42, 43]. In a quality improvement project, Park et al. implemented an electronic alert to the primary service to consult nephrology based on an established AKI in order to minimize overlooked AKI [44]. Patients in the alert group had fewer odds of a severe AKI event than those under usual care [44]. Incorporation of risk prediction tools in electronic databases can provide a window of opportunity to detect high-risk patients that might benefit from surveillance, further directed testing with biomarkers and thus help in the early management and individualization of treatment for AKI [22]. With more advancement in technology, these risk prediction tools may utilize machine learning approach to provide an individualized risk score and allow for delivery of care through precision medicine.

Our study has several strengths. We utilized machine learning to identify risk factors for ICU-acquired AKI using information available at the time of ICU admission. This information is easily obtainable, as we used the standardized ICD-9 and -10 codes. While the trade-off of using fewer features comes at the expense of the predictive ability of the model, we were still able to obtain an acceptable AUROC that was comparable to previously published models that utilized more information. In addition, we used a more clinically relevant definition of AKI by employing both urine and creatinine criteria. Indeed, in the Mayo Clinic cohort where urine output was not missing, we found that the definition of AKI by urine criteria was more prevalent than by creatinine criteria. This increases the validity of our model as it can potentially identify patients that eventually develop AKI by urine criteria but otherwise would go undetected entirely or be detected later when creatinine levels rise. Lastly, we validated our model externally using the MIMIC-III dataset. Although there were differences between the Mayo Clinic and the MIMIC-III cohorts in terms of demographics and data availability for the staging of AKI, the prevalence of comorbidities the model used was similar. The performance of the model on data from these two hospitals shows that the model might potentially be generalizable and be applied to different cohorts.

Our study also has several limitations. First, we defined AKI at any point during the ICU stay rather than trying to predict AKI onset at a particular time. We also excluded patients who required RRT in the ICU as it was difficult to ascertain the timing of RRT in relation to AKI occurrence, especially since the KDIGO guidelines assign Stage 3 for patients undergoing RRT (irrespective of urine or creatinine values). This might not always be reproducible when performing external validation, as many patients receive RRT soon after ICU admission. In any clinical practice, including the cohorts included in this study, a percentage of patients do not have measured baseline serum creatinine levels, which could result in a bias. During the derivation and validation of our electronic tool, we showed the detection of AKI using a combination of baseline measured serum creatinine, and back-calculated creatinine using MDRD (when measured serum creatinine was not available) was able to provide a sensitivity of 88% and specificity of 96% [31]. We used MDRD back calculation for 60 mL/min/body surface area to increase the specificity of the detection of AKI. Being aware of this limitation, unfortunately, our results reflect the current practice as there are a substantial number of patients who do not have available baseline serum creatinine. Unfortunately, the information regarding the causes of CKD was not available in the majority of patients. In a systematic review using different sources, the agreement between data sources on disease etiology ranged between 59% and 89% [45]. This shortcoming is not uncommon as patients’ causes of CKD are often inferred or likely multifactorial. Lastly, although we evaluated exposure to nephrotoxic drugs prior to hospital admission, we did not include in the model as the information was not reproducible in the external validation cohort.

AKI is a complex syndrome that cannot be identified through solely relying on statistical models or biomarkers. Despite no available therapies for AKI, identifying patients at high risk for AKI might allow targeted preventive interventions that can potentially decrease the burden of the disease. With the advent and progressively higher capabilities of electronic health records, deployment of a machine-learning algorithm that can predict AKI would help provide an individualized risk score and allow for a window of opportunity where tailored intervention can be delivered.

SUPPLEMENTARY DATA

Supplementary data are available at ckj online.

FUNDING

This research did not receive any specific grant from funding agencies in the public, commercial or not-for-profit sectors.

CONFLICT OF INTEREST STATEMENT

All of the authors have disclosed that they do not have any conflicts of interest. E.G., S.L., E.S. and L.E. are employed by Philips Research North America and this work was done as a part of their duties.