Predicting Nephrotoxic Acute Kidney Injury in Hospitalized Adults: A Machine Learning Algorithm

Abstract

Rationale and Objective

Acute kidney injury (AKI) is a common complication among hospitalized adults, but AKI prediction and prevention among adults has proved challenging. We used machine learning to update the nephrotoxic injury negated by just-in time action (NINJA), a pediatric program that predicts nephrotoxic AKI, to improve accuracy among adults.

Study Design

A retrospective cohort study.

Setting and Population

Adults admitted for > 48 hours to the University of Iowa Hospital from 2017 to 2022.

Exposure

A NINJA high-nephrotoxin exposure (≥3 nephrotoxins on 1 day or intravenous aminoglycoside or vancomycin for ≥3 days).

Outcomes

AKI within 48 hours of high-nephrotoxin exposure.

Analytical Approach

We collected 85 variables, including demographics, laboratory tests, vital signs, and medications. AKI was defined as a serum creatinine increase of ≥0.3 mg/dL. A gated recurrent unit (GRU)-based recurrent neural network (RNN) was trained on 85% of the data, and then tested on the remaining 15%. Model performance was evaluated with precision, recall, negative predictive value, and area under the curve. We used an artificial neural network to determine risk factor importance.

Results

There were 14,480 patients, 18,180 admissions, and 37,300 high-nephrotoxin exposure events meeting inclusion criteria. In the testing cohort, 29% of exposures developed AKI within 48 hours. The RNN-GRU model predicted AKI with a precision of 0.60, reducing the number of false alerts from 2.5 to 0.7 per AKI case. Lowest hemoglobin, lowest blood pressure, and highest white blood cell count were the most important variables in the artificial neural network model. Acyclovir, piperacillin-tazobactam, calcineurin inhibitors, and angiotensin-converting enzyme inhibitor/angiotensin receptor blockers were the most important medications.

Limitations

Clinical variables and medications were not exhaustive, drug levels or dosing were not incorporated, and Iowa’s racial makeup may limit generalizability.

Conclusions

Our RNN-GRU model substantially reduced the number of false alerts for nephrotoxic AKI, which may facilitate NINJA translation to adult hospitals by providing more targeted intervention.

Plain-Language Summary

Nephrotoxic acute kidney injury (AKI) is common and can potentially be prevented through preemptive adjustments of medications, as demonstrated by the success of the nephrotoxic injury negated by just-in time action (NINJA) program in pediatric populations. Translation of NINJA to the adult population has been challenging, and major barriers include high alert volume in adults that can lead to high resource utilization and alert fatigue. To address this issue, we developed a machine learning model for nephrotoxic AKI in adults that reduced the number of false alerts per AKI event from 2.5 to 0.7, which can enhance future NINJA implementation in adults by allowing for a more targeted intervention with fewer alerts and more efficient resource utilization.

Acute kidney injury (AKI) is a common and widely recognized source of morbidity and mortality in hospitalized patients. The incidence of in-hospital AKI is 20%-25%,^1,2 and it is associated with significant increases in both short-term and long-term morbidity and mortality.^3,4 One major cause of AKI development is the use of nephrotoxic medications, which is a contributing factor in at least a quarter of all inpatient AKI episodes.5, 6, 7, 8 Medication classes known to be associated with an increased risk of nephrotoxic AKI include antibiotics, antifungals, antivirals, nonsteroidal anti-inflammatory drugs, bisphosphonates, calcineurin inhibitors, checkpoint inhibitors, and chemotherapeutic agents, among others.⁹ The wide variety of agents that are used to treat myriad conditions account for the substantial incidence of nephrotoxic AKI.

There is no specific therapy for nephrotoxic AKI once it has occurred.⁷ Furthermore, inpatient nephrotoxic exposures are preventable, so risk assessment and nephrotoxin reduction in high-risk patients are important for prevention of this iatrogenic AKI before it occurs. The nephrotoxic injury negated by just-in time action (NINJA) program has been shown to effectively reduce rates of nephrotoxin exposure and nephrotoxic AKI in pediatric populations. In a single-center study in children, the NINJA program resulted in a 38% reduction in nephrotoxin exposure rate and a 64% reduction in nephrotoxic AKI.¹⁰ A subsequent multicenter study observed a significant and sustained 24% lowering in rates of nephrotoxic AKI.¹¹ Our previous work demonstrated that NINJA was able to identify a high-risk adult cohort¹²; however, a major concern was the relatively low positive predictive value, which combined with the larger hospital volumes would have resulted in an alert volume 10 times greater at our adult hospital when compared with our children’s hospital. High resource utilization and alert fatigue¹³ were therefore identified as major barriers to the application of NINJA in adult populations.

The current NINJA system was developed in a pediatric population, which has less chronic disease morbidity and fewer multiproblem admissions than adult populations, so alerts were based only on medication usage and did not factor in comorbidities, laboratory values, or other clinical parameters.¹⁴ Our goal in this project was to use a machine learning algorithm and additional clinical data to predict the development of nephrotoxic AKI more accurately in adults, which can enhance future NINJA implementation in adults by allowing for a more targeted intervention with fewer alerts and more efficient resource utilization.

Methods

Study Population

We obtained local IRB approval (HawkIRB 202008447) and followed the STROBE reporting guidelines.¹⁵ All adult patients aged ≥18 years who were admitted to the University of Iowa Hospital from 2017 to 2022 who had at least one patient-day of high-nephrotoxin exposure were eligible for inclusion. Exclusion criteria were as follows: (1) admissions < 48 hours; (2) patients with a baseline estimated glomerular filtration of < 15 mL/min/1.73m² or patients on long-term dialysis before admission; (3) AKI at the time of admission; (4) patients with < 2 creatinine values during the admission; (5) absence of laboratory testing during the hospitalization; and (6) absence of vital sign data.

Nephrotoxin Exposure and AKI Definition

High-nephrotoxin exposure was defined based on the NINJA definition as receiving ≥3 nephrotoxic medications with systemic absorption in 1 day (Table 1)¹⁴ or intravenous aminoglycoside or vancomycin for ≥3 days. Iodinated contrast dye, amphotericin liposomal, and cidofovir are marked as administered for 6 days after the day of actual use and are the only agents treated in this fashion. AKI was defined using Kidney Disease Improving Global Outcomes (KDIGO) guidelines¹⁶ as a creatinine increase of ≥0.3 mg/dL or 50% from baseline serum creatinine, defined as the lowest creatinine within 6 months before hospitalization, or at the time of hospitalization in the absence of previous values.

Table 1.

Medications Within NINJA, by Class

Drug Class	Drug Name
ACEI/ARB	Captopril	Lisinopril
	Enalapril	Losartan
	Enalaprilat	Valsartan
Antibiotics	Ambisome	Piperacillin
	Amikacin	Polymixin B
	Amphotericin B	Tazobactam
	Clavulanic Acid	Ticarcillin
	Colistimethate	Tobramycin
	Gentamicin	Vancomycin
	Nafcillin
	Pentamadine
Antivirals	Acyclovir	Tenofovir
	Cidofovir	Valacyclovir
	Foscarnet	Valganciclovir
	Ganciclovir
Chemotherapies	Carboplatin	Methotrexate
	Cisplatin	Mitomycin
	Ifosfamide
Calcineurin inhibitors	Cyclosporine	Tacrolimus
Iodinated contrast dye	Diatrizoate meglumine	Iopromide
	Diatrizoate sodium	Ioversol
	Iodixanol	Ioxaglate meglumine
	Iohexol	Ioxilan
	Iopamidol
NSAID	Celecoxib	Ketorolac
	Ibuprofen	Naproxen
	Indomethacin
Others	Deferasirox	Sulfasalazine
	Lithium	Topiramate
	Mesalamine	Zoledronic acid
	Pamidronate	Zonisamide
	Sirolimus

Abbreviations: ACEi, angiotensin-converting enzyme inhibitor; ARB, angiontensin receptor blocker; NSAID, nonsteroidal anti-inflammatory drug.

Variables

Variables used to develop our machine learning algorithm included age, sex, race, patient comorbid conditions including: diabetes, liver disease, coronary artery disease, congestive heart failure, peripheral vascular disease, malignancy, and cerebrovascular disease (defined by International Classification of Diseases-10 codes on the admission record using the method of Quan et al)¹⁷; daily laboratory values (white blood cell count, hemoglobin, platelets, serum creatinine, and blood urea nitrogen; daily vital signs (highest temperature, highest daily mean arterial pressure [MAP]; lowest daily MAP); admitting service (categorized as medicine, surgery, heart and vascular, neurology or neurosurgery, and other); baseline creatinine; and daily medication administration data for the 57 nephrotoxic medications listed in Table 1.

Predicted Outcomes

Nephrotoxic AKI, defined as AKI development within 48 (primary) or 72 (secondary) hours of a high-nephrotoxin exposure.

Data Preprocessing (Time Series Analysis)

Patients missing demographic data or those without laboratory or vital sign data were excluded. For patient-days missing laboratory or vital sign values, we first forward-filled data, and then backward-filled to fill all patient-days. For patient-days with multiple laboratory or vital sign values, the median value was selected, with the exception of mean arterial pressure, in which the maximum and minimum values were recorded as separate variables.

Machine Learning Algorithm Development (Time Series Analysis)

For the primary time series analysis, we used a recurrent neural network (RNN) with gated recurrent unit (GRU) model. The model features sequential layers of GRUs, starting with 64 and followed by layers with 32 and 16 GRUs, which help the network understand how data change over time. In addition, a dense layer with 16 GRUs was included to uncover significant patterns while ensuring the model remained stable and focused on important features, and a final dense layer provided outcome predictions. To improve the model’s performance, we used the Adam optimizer with a specific learning rate of 0.0001 and a weight decay value of 1e-5. The model’s learning process was guided by the binary cross-entropy loss function, and early stopping and reduced learning rate techniques were used to prevent overfitting. The entire network was trained over 200 epochs using a defined batch size of 256 and a validation split of 20%. To determine model performance, we applied the algorithm to the independent test set, which had not been used in the training of the final model. We calculated the precision (defined as the percentage of patients with alerts that go on to develop AKI, also known as positive predictive value [PPV]), recall (the percentage of patients who developed AKI who were identified by the alert system), negative predictive value (NPV), F1-statistic (a machine learning metric combining precision and recall), and the area under the curve for both the 48-hour and 72-hour AKI prediction models.¹⁸ For the purposes of these calculations, true positives were defined as cases that generated an alert and developed AKI within the specified timeframes and true negatives were defined as cases that did not generate an alert and then did not develop AKI within the specified timeframes. False positives were cases with an algorithm alert that did not develop AKI, and false negatives were those cases without an alert who did develop AKI within the 48-hour or 72-hour timeframe. We used Python v3, Tensorflow 2.14, cuda 11.7, Shapley additive explanations, scikit-learn and pandas (The Python Software Foundation for all analyses.

Data Preprocessing (Risk Factor Importance Analysis)

The RNN-GRU model selected for the time series analysis is well-suited to longitudinal prediction but does not allow determination of relative predictor importance.¹⁹ To give an idea of the relative importance of each risk factor, we also developed an artificial neural network (ANN) model that allows for this analysis. Longitudinal data from each unique admission was compressed into a single row. Laboratory values and vital signs were divided into 2 separate variables consisting of the highest and lowest values before AKI. At least one patient-day was required to qualify as high-nephrotoxin exposure by NINJA criteria for inclusion. Finally, medications with a clear class effect, specifically angiotensin-converting enzyme inhibitors (ACEi) and angiotensin receptor blockers (ARBs), contrast agents, nonsteroidal anti-inflammatory drugs, and calcineurin inhibitors, were grouped together (Table 1).

Machine Learning Algorithm Development (Risk Factor Importance Analysis)

The ANN model incorporated sequential layers, including dense layers with varying units (256 and 128) activated by ReLU. Dropout layers (0.5) were added for robustness, and L1 and L2 regularization techniques were used for feature stability. The final dense layer, activated by sigmoid, produced binary classification outputs. For optimization, the Adam optimizer with tailored learning rates and decay was employed. Training utilized binary cross-entropy loss with early stopping and learning rate (0.001) to prevent overfitting. The ANN was trained on a subset of data (batch size 128) and Shapley additive explanations was employed to assess feature importance.

Results

Patient Characteristics and Nephrotoxins

After inclusions and exclusions, there were 37,300 patient-days that met criteria for high-nephrotoxin exposure from 14,480 patients and 18,180 unique admissions (Fig 1). Hospital characteristics of patients with a high-nephrotoxin exposure are shown in Table 2. The most encountered nephrotoxins among patients with high-nephrotoxin exposure were vancomycin (68%), iopamidol (63%), and piperacillin-tazobactam (45%). Lisinopril and acyclovir were given for at least 1 day in 15%-20% of such admissions, and ibuprofen, losartan, ketorolac, tacrolimus, and valacyclovir were each prescribed in 5%-10% of admissions. Notably, this cohort had long hospital stays (median 9 days) and a high rate of in-hospital mortality (31%) when compared with the general inpatient population.

Figure 1.

Cohort definition flowsheet.

Table 2.

Demographics, Comorbidities, Laboratory Results, Vital Signs, and Medication Usage During the Index Hospitalization for Patients With at Least One High-Nephrotoxin Exposure Day

Characteristicsa	High-Nephrotoxin Exposure
	(n = 18,180)
Age, y, median (IQR)	61 (49-70)
Female sex	7,861 (43.3)
Caucasian race	16,330 (89.9)
Body mass index, median (IQR)	28 (24-34)
Comorbid conditions
Chronic pulmonary disease	2,920 (16.1)
Diabetes mellitus	2,666 (14.7)
Congestive heart failure	2,075 (11.4)
Cerebrovascular disease	2,063 (11.4)
Peripheral vascular disease	1,366 (7.5)
Cirrhotic liver disease	1,227 (6.8)
Myocardial infarction	784 (4.3)
Prehospital serum creatinine, median (IQR)	0.7 (0.5-0.9)
Highest serum creatinine, median (IQR)	1.1 (0.8-1.5)
Highest white blood count, median (IQR)	13.9 (9.8-19.8)
Lowest white blood count, median (IQR)	6.6 (4.7-8.8)
Highest hemoglobin, median (IQR)	11.8 (10.1-13.4)
Lowest hemoglobin, median (IQR)	9.0 (7.4-10.8)
Highest platelet count, median (IQR)	319 (225-451)
Lowest platelet count, median (IQR)	184 (116-260)
Highest temperature, median (IQR)	99.8 (99.3-99.9)
Lowest mean arterial pressure, median (IQR)	65 (57-72)
Medications
Vancomycin	12,259 (67.5)
Iopamidol (contrast dye)	11,371 (62.6)
Piperacillin-Tazobactam	8,206 (45.2)
Lisinopril	3,510 (19.3)
Acyclovir	3,115 (17.1)
Ibuprofen	1,614 (8.9)
Losartan	1,606 (8.8)
Ketorolac	1,535 (8.4)
Tacrolimus	1,275 (7.0)
Valacyclovir	1,052 (5.8)
Admission service
Medicine	7,544 (41.5)
Other	4,297 (23.6)
Surgery	3,477 (19.1)
Heart and vascular	1,448 (8.0)
Neurology and neurosurgery	1,404 (7.7)
ICU admission	4,522 (24.9)
Length of stay, d, and median (IQR)	9 (5-16)
In-hospital mortality	5,563 (30.6)

Abbreviation: ICU, intensive care unit; IQR, interquartile range.

^aPresented as N (%) unless otherwise indicated.

NINJA Performance

In the testing cohort, 1,591 out of 5,555 patients (29%) who would have received a NINJA alert due to high-nephrotoxin exposure, developed AKI within 48 hours. Therefore, NINJA would have issued 2.5 false alerts per case of AKI. Although this approach has perfect sensitivity (100%), the PPV is only 0.29 (95% CI, 0.29-0.29). Similarly, at 72 hours, 1,775 (32%) patients with high-nephrotoxin exposure had developed AKI (2.1 false alerts per case of AKI).

RNN-GRU Model Performance (48 hours)

In comparison, the RNN-GRU model generated 1,705 alerts, of which 1,027 (60%) developed AKI within 48 hours. Therefore, the RNN-GRU model had a PPV of 0.60 (95% CI, 0.58-0.62), and a rate of 0.7 false alerts per case of AKI. Of the 1,591 patients who developed AKI, the model identified 1,027 correctly, giving a recall of 0.65 (0.63-0.67). The weighted F1-score was 0.78 (0.77-0.79), demonstrating good combined precision and recall. There were 564 false negatives out of 3,850 nonalerts, giving a NPV of 0.85 (0.84-0.86), and the area under the curve was 0.82 (0.81-0.83).

RNN-GRU Model Performance (72 hours)

At 72 hours, 1,775 (32%) patients with high-nephrotoxin exposure out of 5,555 had developed AKI. The 72-hour RNN-GRU model generated 2,250 alerts with a PPV of 0.57 (0.55-0.58), and a rate of 0.8 false alerts per case of AKI. Comparisons of performance characteristics for all models are given in Table 3.

Table 3.

Performance Characteristics for NINJA and the 2 Machine Learning Models

Characteristics	NINJA	RNN-GRU (48 h)	RNN-GRU (72 h)
Positive predictive value	0.29	0.60 (0.58-0.62)	0.57 (0.55-0.58)
Negative predictive value	N/A	0.85 (0.84-0.86)	0.85 (0.84-0.86)
Recall	1.0	0.65 (0.63-0.67)	0.72 (0.70-0.74)
F1-statistic	0.29	0.78 (0.77-0.79)	0.74 (0.72-0.75)
Area under the curve	N/A	0.82 (0.81-0.83)	0.81 (0.79-0.82)
Sensitivity	100%	65% (62%-67%)	72% (70%-74%)
Specificity	N/A	83% (82%-84%)	74% (73%-76%)
Positive likelihood ratio	N/A	3.8 (3.5-4.1)	2.8 (2.6-3.0)
Negative likelihood ratio	N/A	0.43 (0.40-0.46)	0.38 (0.35-0.41)
False alerts per AKI	2.45	0.66 (0.61-0.72)	0.75 (0.72-0.82)

Note: Because the NINJA model functionally assumes all high-nephrotoxic exposures will develop AKI, it has no “negative” values, rendering several metrics incalculable within this approach.

Abbreviations: AKI, acute kidney injury; NINJA, nephrotoxic injury negated by just-in time action; RNN, recurrent neural network; GRU, gated recurrent unit.

Risk Factor Importance

The most important risk factors for AKI prediction in the ANN model are given in Fig 2. Components of the complete blood count, specifically the lowest hemoglobin and highest white blood cell count, were 2 of the 3 most predictive risk factors, along with the lowest MAP. Baseline creatinine was the 4th most predictive of AKI development. Acyclovir, piperacillin-tazobactam, calcineurin inhibitors, and angiotensin-converting enzyme inhibitor/angiotensin receptor blockers (ACEi/ARBs) were the 4 most important pharmacologic risk factors.

Figure 2.

Factor analysis showing the most important prediction variables in the ANN algorithm for nephrotoxic AKI prediction. Red correlates with higher likelihood and blue with lower likelihood of AKI development. ACEi, angiotensin-converting enzyme inhibitor; ARB, angiontensin receptor blocker; ICU, intensive care unit; NSAID, nonsteroidal anti-inflammatory drug; WBC, white blood cell.

Discussion

Use of a machine learning model to predict AKI development in a cohort of patients with high-nephrotoxin exposure decreased the number of false alerts from 2.5 to 0.7 per case of AKI, which addresses a major barrier to the implementation of nephrotoxic AKI prevention programs in the adult population.

In this analysis, we applied a RNN with GRU model to an adult inpatient population to predict the development of AKI after high-nephrotoxin exposure. This model resulted in a higher PPV without a substantial decrease in the number of AKI cases identified. We then used an ANN to identify key factors for the prediction of AKI. Overall, the RNN-GRU machine learning model we developed may allow for more targeted AKI prevention efforts and greater efficiency of resource utilization.

Our machine learning model incorporates clinical variables in addition to medication data to better predict which patients with high-nephrotoxin exposure would develop AKI, with the goal of increasing the efficiency of AKI prevention programs. The highly successful NINJA program was originally developed in pediatrics and proactively identifies high-risk patients before the development of AKI.^10,11 Our recent study subsequently demonstrated that the NINJA high-nephrotoxin exposure criteria can be applied to adult inpatients to identify a population at risk of AKI.¹² However, the total number of alerts generated was more than 10-fold greater in adults than in our children’s hospital, and only 29% of patients with high-nephrotoxin exposure alerts went on to develop AKI. Our RNN-GRU machine learning model was able to greatly improve predictive accuracy by adding clinical features to NINJA high-nephrotoxin exposure criteria. Using the NINJA high-nephrotoxin exposure criteria, 2.5 false alerts were generated for each true case of AKI, compared with our RNN algorithm, which generated only 0.7 false alerts per case of AKI. The RNN model correctly identified and predicted 1,027 of 1,591 cases (65%) of AKI in our validation dataset. Thus, the number of alerts was decreased by half, while two-thirds of eventual AKI cases were still predicted by the model.

One potential downside of our model that must be considered is an increase in false negatives. Our model performed well with an NPV of 0.85, but because the previous NINJA algorithm alerted for every patient with a high-nephrotoxin exposure, there were functionally no false negatives previously. As noted above, this perfect NPV came at the expense of false positives, which led to alert fatigue and increased resource utilization. Because we anticipate that these will be the biggest barriers to program uptake, we felt that the large gains in PPV sufficiently balanced the lower NPV at present; however, this balance could be reevaluated and updated in successive iterations of the model.

The RNN-GRU model is well-suited to modeling longitudinal medical data and has a number of inherent advantages that significantly improve predictive power over other machine learning approaches and over traditional modeling techniques such as logistic regression.²⁰ However, one downside is the black-box nature of this model, which makes it impossible to determine how the model is making its predictions. To add clarity on which risk factors may be most impactful in nephrotoxic AKI prediction, we used an explanatory ANN model to evaluate predictive risk factor importance. It is important to note that the outcome and output variables used in these models are slightly different: the RNN-GRU model predicts AKI within 48 hours of high-nephrotoxin exposure, whereas the ANN model predicts nephrotoxic AKI during the course of a hospitalization. These caveats notwithstanding, the risk factor importance analysis yielded several interesting observations. First, laboratory data and vital signs were the most important features, which is likely why the RNN-GRU model’s accuracy was so much greater than NINJA high-nephrotoxin exposure criteria alone. This suggests that NINJA might benefit from formally incorporating more clinical data, ideally through the implementation of an effective machine learning model such as our RNN-GRU model. Second, the most important medications were acyclovir, piperacillin-tazobactam, calcineurin inhibitors, and ACEi/ARBs, providing direction on which medications may be most important in predicting AKI for high-risk patients.

When looking at variable importance, it is important to note that predictive importance within the model does not necessarily indicate causation. For instance, being admitted to a medicine service was predictive of AKI development, which is likely because of underlying differences in patient populations and reasons for admission not captured fully by other variables. Similarly, ACEi/ARB agents were among the most predictive medications for AKI development but may not truly induce tubular damage in the way the nephrotoxin label implies,²¹ depending on clinical circumstances. This observation has implications for future implementation of nephrotoxic AKI prevention programs in adults. For instance, it would not be advisable to transfer high-risk patients off medicine teams, and ACEi/ARB discontinuation may or may not be appropriate depending on clinical circumstances. For medications with more established nephrotoxicity, it is still important to consider whether an alternative agent is likely to provide similar benefit to avoid potential harms from inadequate treatment of underlying diseases.

NINJA has been shown to be effective in reducing rates of nephrotoxic AKI in both a single-center study and in a large multicenter validation,^10,11 Within the larger context of AKI alert systems, the impact of NINJA on clinical outcomes is an outlier. Several early trials using decision support tools in patients with reduced kidney function (due to either AKI or chronic kidney disease) demonstrated improvements in prescription patterns and appropriate medication dosing but did not examine clinical outcomes.^22,23 More recent trials of AKI alert systems have shown improvements in physician awareness and discontinuation of nephrotoxic medications, but no significant improvements in clinical outcomes such as AKI progression, dialysis, or death.24, 25, 26 Notably, alert systems to date have focused on intervention after AKI development, a point that may be too late in the clinical course to make meaningful impacts. The preventative focus of NINJA and the live communication between pharmacists, nephrologists, and primary teams may explain why this program has been able to successfully improve patient outcomes whereas other AKI alert systems have not. However, NINJA’s approach does come at the cost of a higher number of alerts and greater resource utilization, which have been major hurdles to NINJA implementation in adults. Application of our RNN-GRU algorithm addresses these issues by substantially increasing predictive accuracy, which will allow for a more targeted intervention.

Important next steps toward nephrotoxic AKI reduction in adults should include further validation of this algorithm at other institutions and among diverse patient populations. The NINJA system should then undergo prospective efficacy trials and given the growing ability of electronic medical record systems to incorporate real-time machine learning prediction algorithms,^27,28 we believe this model should be a component of these future trials. Adequate power will be needed to determine whether AKI related to certain classes of medications are more amenable to intervention and AKI prevention than others. Some medication classes, such as ACEi/ARB and iodinated contrast, might be less nephrotoxic than others,²¹ so incorporation of additional markers of tubular damage into the NINJA algorithm should also be considered.

This study has several limitations. Although we collected 18 nondrug variables, it is possible that other important clinical characteristics were not included. Similarly, we collected data on 57 nephrotoxic medications, but this is not an exhaustive list. Additional medications and classes (eg immune checkpoint inhibitors)²⁹ are likely to be added to the NINJA nephrotoxic list in the future, which would require updating of these algorithms. In addition, NINJA does not currently incorporate medication dose or drug levels,¹⁴ which may be important risk factors for AKI. As noted above, there were differences in outcomes and variables in the RNN-GRU model when compared with the ANN model, and so our feature importance may not fully capture the relative importance of these inputs within the time series model. Finally, our population reflects the University of Iowa Hospitals and Clinics population (89% Caucasian), which may limit generalizability of findings and underscores the need for validation of this model in other settings.

In conclusion, application of a RNN-GRU machine learning model was able to substantially reduce the number of false alerts for nephrotoxic AKI in adults, which may facilitate NINJA translation to adult hospitals by allowing for more targeted intervention with less resource utilization.