A Machine Learning Model to Predict Diuretic Resistance

Abstract

Key Points

• Our machine learning algorithm was able to quickly predict those at risk for diuretic resistance using common variables.

• Our machine learning algorithm could easily be implemented within electrical medical software (or as an online calculator).

• Our study provides a proof of concept/framework for the development of future machine learning models.

Background

Volume overload is a common complication encountered in hospitalized patients, and the mainstay of therapy is diuresis. Unfortunately, the diuretic response in some individuals is inadequate despite a typical dose of loop diuretics, a phenomenon called diuretic resistance. An accurate prediction model that predicts diuretic resistance using predosing variables could inform the right diuretic dose for a prospective patient.

Methods

Two large, deidentified, publicly available, and independent intensive care unit (ICU) databases from the United States were used—the Medical Information Mart for Intensive Care III (MIMIC) and the Philips eICU databases. Loop diuretic resistance was defined as <1400 ml of urine per 40 mg of diuretic dose in 24 hours. Using 24-hour windows throughout admission, commonly accessible variables were obtained and incorporated into the model. Data imputation was performed using a highly accurate machine learning method. Using XGBoost, several models were created using train and test datasets from the eICU database. These were then combined into an ensemble model optimized for increased specificity and then externally validated on the MIMIC database.

Results

The final ensemble model was composed of four separate models, each using 21 commonly available variables. The ensemble model outperformed individual models during validation. Higher serum creatinine, lower systolic blood pressure, lower serum chloride, higher age, and female sex were the most important predictors of diuretic resistance (in that order). The specificity of the model on external validation was 92%, yielding a positive likelihood ratio of 3.46 while maintaining overall discrimination (C-statistic 0.69).

Conclusions

A diuretic resistance prediction model was created using machine learning and was externally validated in ICU populations. The model is easy to use, would provide actionable information at the bedside, and would be ready for implementation in existing electronic medical records. This study also provides a framework for the development of future machine learning models.

Introduction

Volume overload is a complication of cardiovascular, kidney, and liver disease and is frequently encountered in hospitalized patients, notably in the intensive care unit (ICU) setting. When untreated or undertreated, volume overload can lead to increased end-organ dysfunction (kidney failure and heart failure), prolonged ventilator and ICU stay, and increased mortality.^1,2 Loop diuretics, including furosemide, torsemide, and bumetanide, are the mainstay of treatment in hospitalized patients.

Unfortunately, some patients have a poor response to loop diuretics, a phenomenon termed diuretic resistance.^3,4 Although a formal quantitative definition for diuretic resistance is lacking, owing to the complexities and variations in diuretic regimens, it is typically described as an inadequate amount of decongestion/diuresis despite an adequate diuretic regimen.³ Diuretic resistance is also associated with increased mortality, and proposed algorithms for its management typically involve post hoc decision making after a trial of therapy, potentially delaying the onset of effective therapies.^3,5–7

Early identification of patients at high risk of developing diuretic resistance creates a unique opportunity for intervention with potential positive prognostic implications. Patients at high risk for diuretic resistance can receive higher doses upfront and therefore potentially prevent a delay in decongestion, which may lead to earlier extubation or prevent the need for dialysis therapy for fluid overload.

A prediction model that could provide actionable information on patients identified as high risk of being diuretic resistant, with appropriate early escalation of therapy as necessary, can improve clinical decision making in this setting.³ As such, we sought to develop and externally validate a model to predict diuretic resistance in patients hospitalized in ICU settings.

Materials and Methods

Data Source and Extraction

The Medical Information Mart for Intensive Care III (MIMIC-III) and the Philips eICU databases are two large, deidentified, publicly available, and independent ICU databases in the United States.^8,9 MIMIC-III contains approximately 50,000 hospital admissions to critical care units between 2001 and 2012 at the Beth Israel Deaconess Medical Center in Boston, Massachusetts, whereas eICU contains approximately 200,000 admissions monitored by Philips Healthcare across the United States between 2014 and 2015. Commonly available data, including patient demographics, routine vitals and labs, certain drugs (including diuretic doses), and weight/urine output, were extracted from these databases. Comorbidities within the databases were coded using the International Classification of Diseases (ICD-9). These were translated into the Charlson Comorbidity Index (CCI), given its prognostic implications.^10,11

Patients who are younger than 16 years and with <24 hours of ICU data were excluded. Variables with >20% missing values were excluded, and data imputation using a highly accurate 3-tier novel technique, which incorporates supervised machine learning, was used on the remaining variables for the eICU dataset.¹² The data were partitioned into 24-hour windows from time of admission to the ICU, with each 24-hour period representing a separate datapoint. Laboratory values and vitals from the previous 24 hours were averaged while both diuretic doses and urine output from the current 24 hours were summed. Patients with <40 mg IV furosemide (or equivalent) or no IV loop diuretic doses within the current 24-hour window were excluded from the analyses.

Given that data collection from both databases was passive and that all data were deidentified in compliance with the Health Insurance Portability and Accountability Act Privacy Rule, which constitutes nonhuman subject research, no institutional or ethical approvals were required for this study.^8,9

Definitions

Loop diuretic IV equivalents were defined using a 40:20:1 ratio (in mg) for furosemide:torsemide:bumetanide, respectively, with a 1:2 IV to oral conversion for furosemide (bumetanide and torsemide kept a 1:1 ratio).³ These equivalent doses are in accordance with the pharmacologic profiles of the loop diuretics and consistent with previous studies.³ Diuretic resistance was quantitatively defined using urine output per diuretic dose, the latter being defined as 40 mg IV furosemide equivalent. The cutoff for diuretic resistance was set at ≤1400 ml/dose in 24 hours. This cutoff was selected based on a subanalysis of the ASCEND-HF trial by Ter Maaten et al.,¹³ given its associated increase in adverse outcomes at 30 days (all-cause mortality or heart failure rehospitalization). Another study with a similar cutoff (1700 ml/dose) also showed increased mortality at 180 days.¹⁴

Methods

Baseline characteristics for both the eICU and MIMIC-III databases were provided with descriptive statistics, and the chi-square and t tests were used to determine whether any significant differences between the databases existed for dichotomous and continuous variables, respectively (all continuous variables had a normal distribution). Owing to its size, the eICU database was used to create train and test datasets, set at 80% and 20% of the database, respectively. Supervised machine learning using extreme gradient boosted trees (XGBoost), a scalable end-to-end tree boosting system, was performed on the train dataset using five-fold cross-validation.¹⁵ This degree of cross-validation was selected to prevent overfitting on the training dataset. A correlation matrix was created, and final variables were chosen by feature importance score. Hyperparameter tuning for the XGBoost model was performed using Bayesian optimization, a global optimization scheme with Gaussian processes, to maximize a certain metric (e.g., precision, recall, area under the receiver operating curve, and F1 score).¹⁶ The final model was then applied on the eICU test dataset (internal testing), and model parameters such as area under the receiver operating characteristic (AUROC) curve, accuracy, Brier score, F1 score, sensitivity, specificity, positive/negative predictive values, and positive/negative likelihood ratios were calculated. This process was repeated four times, each time optimizing a different model metric using Bayesian optimization, yielding four separate models with different hyperparameters and algorithms. The models were interrogated using Shapley additive explanations (SHAP summary plot) and partial dependence plots, the former having been shown to be superior as a means of explaining machine learning models (further elaboration can also be found in the Supplemental Material).^17,18 Finally, the models were combined into a final ensemble model using the geometric mean of their generated probabilities, given that ensemble models were shown to outperform individual models in other applications of machine learning.¹⁹ This ensemble model was then optimized for specificity (to decrease “alert fatigue”) and then applied to the MIMIC-III dataset as a means of external validation/external testing. The overall scheme is depicted in Figure 1. To determine whether the ensemble model performed best, the individual models and an ensemble model with the two best models were also applied to the MIMIC-III dataset and compared. Finally, a sensitivity analysis was performed using the cutoff of 1700 ml/dose.

Figure 1.

Machine learning workflow diagram. MIMIC-III, Medical Information Mart for Intensive Care III; eICU, eICU Collaborative Research Database; SMILES: xgbooSt MIssing vaLues In timE Series; imputation method as described by Zhang et al.¹²

Data mining was performed on Google Big Query (Google Cloud Computing, Google Corporation), and all other analyses (including imputation) were performed with Python 3.7 using Google Colaboratory (Google Corporation).

Results

Study Population

Our selection criteria and patient allocation, as well as final sample sizes per database, are summarized in Figure 2, and patient baseline characteristics are summarized in Table 1. There were 30,868 and 12,671 unique days/windows that met our inclusion criteria in the eICU and MIMIC databases, respectively. Within the eICU cohort, 52% of those days were identified as diuretic resistant, whereas this was slightly less in the MIMIC cohort at 50%. There were 11,652 and 4480 unique patients encompassing 13,001 and 5059 separate admissions in the eICU and MIMIC databases, respectively. On average, patients had a median of two admissions (with the most being 46 admissions), and their length of stay ranged from 24 hours to over 130 days, with a median of 4 and 5 days for eICU and MIMIC databases, respectively. Except for sex, all baseline differences were statistically significant between both databases. Apart from comorbidities, the differences in demographics were felt to be largely clinically irrelevant and instead mostly represented small differences of large samples. Most of the population were male and White. The MIMIC-III population was more comorbid, and mortality was higher, which included in-hospital mortality and mortality at 30 and 90 days.

Figure 2.

CONSORT diagram.

Table 1.

Baseline characteristics of both eICU and MIMIC-III databases

Variable	eICU	MIMIC	Differences,a P
Total patients, n	11,652	4480
Mean age, y (SD)	67.4 (14.3)	69.0 (14.4)	<0.001
Sex, n (%)
Male	6548 (56.2)	2463 (55.0)	0.16
Female	5104 (43.8)	2017 (45.0)
Ethnicity, n (%)
White	9206 (79.0)	3366 (75.1)	<0.001b
Black	1234 (10.6)	325 (7.3)
Asian	147 (1.3)	79 (1.8)
Hispanic	296 (2.5)	137 (3.1)
Indigenous	92 (0.8)	3 (0.07)
Other/unknown	677 (5.8)	570 (12.7)
Comorbidities,c n (%)
Congestive heart failure	3197 (27.4)	2305 (51.5)	<0.001
Renal failure	1695 (14.5)	1149 (25.6)	<0.001
Liver disease	363 (3.1)	634 (14.2)	<0.001
Diabetes mellitus	1998 (17.1)	1282 (28.6)	<0.001
Cancerd	1758 (15.1)	355 (7.9)	<0.001
Peripheral vascular disease	3540 (30.4)	656 (14.6)	<0.001
Othere	4381 (37.6)	3827 (85.4)	<0.001
Total admissions, n	13,001	5059
Median admissions per patient, n (IQR; range)	2 (2; 1–46)	2 (2; 1–31)
Median length of stay per admission, h (IQR; range)	98 (130; 24–3170)	140 (218; 25–2804)
Days/windows with diuretic resistance	16,121 (52.2)	6.354 (50.1)	<0.001
Days/windows with other classes of diuretics,f n (%)
Thiazide	166 (0.5)	531 (8.4)	<0.001
Potassium sparing	856 (2.8)	407 (6.4)	<0.001
Carbonic anhydrase inhibitor	0	101 (1.6)	NA
Mortality,g n (%)
In hospital	1560 (12.0)	834 (16.5)	<0.001
At 30 days	1489 (11.5)	901 (17.8)	<0.001
At 90 days	1556 (12.0)	1351 (26.7)	<0.001

eICU, eICU Collaborative Research Database; MIMIC-III, Medical Information Mart for Intensive Care III; IQR, interquartile range.

^aChi-square for dichotomous variables and t test for continuous variables.

^bGrouped as White vs. non-White given small numbers of non-White patients.

^cCoded as per Charlson Comorbidity Score (CCI).

^dIncludes lymphoma, solid cancers, and metastatic disease, that is, multiple CCI categories compiled into one.

^eIncludes myocardial infarction, cerebrovascular accident/transient ischemic attack, dementia, chronic obstructive pulmonary disease, connective tissue disease, peptic ulcer disease, hemiplegia, and AIDS.

^fExpressed as a percent of total days (30,868 for eICU and 6354 for MIMIC-III); thiazide diuretics include hydrochlorothiazide, indapamide, and metolazone; potassium-sparing diuretics include both renal tubular epithelial sodium channel blockers (amiloride and triamterene) and mineralocorticoid receptor antagonists (eplerenone and spironolactone); carbonic anhydrase inhibitor includes acetazolamide.

^gNot cumulative.

Machine Models

The initial model derivation included 54 variables (Supplemental Material). These were initially trimmed down using a correlation matrix and feature importance scores and then further trimmed down with model experimentation. An example of feature importance scores (weight and gain), a SHAP summary plot (with detailed explanations for interpretation), and results for the derivation of the individual models are included in the Supplemental Material. With the exception of hemoglobin used in one of the models instead of hematocrit, each model consisted of the same 21 variables. These included demographics (age, female sex, and White race), comorbidities (congestive heart failure, diabetes mellitus, CKD, and liver disease), laboratory (chloride, potassium, calcium, bicarbonate, creatinine, glucose, hematocrit, platelet, and white blood cell count), and vitals (heart rate, systolic blood pressure, respiratory rate, temperature, and oxygen tension as measured by pulse oximetry). It is important to note that all comorbidities included in the model were higher in the MIMIC-III cohort; however, other than a history of congestive heart failure, these variables all had the lowest feature importance scores and thus did not contribute much to the model. Interrogation of the model using a SHAP summary plot (Figure 3) revealed the most important variables for predicting diuretic resistance to be higher serum creatinine, lower systolic blood pressure, lower serum chloride, higher age, and female sex (in that order). The other variables were much less predictive. Using a marginal effect size of around 0.5 as a cutoff, partial dependence plots of the four most important variables (Figure 4) revealed that serum creatinine >100, systolic blood pressure <130, serum chloride <107, and age older than 65 years all had predictive ability for diuretic resistance.

Figure 3.

Model SHAP summary plot.

Figure 4.

Model partial dependence plots for creatinine (s_Cr), systolic blood pressure (avg_sBP), chloride (s_Cl), and age.

Final ensemble model characteristics are summarized in Figure 5 for the AUROC curve and in Table 2 for model performance metrics. The specificity of the model on external validation was 92%, yielding a positive likelihood ratio of 3.46 while maintaining overall discrimination (C-statistic 0.69). The classification accuracy was 70% in train and test datasets, decreasing to 60% in the external validation model. This was mostly driven by false negatives (given the optimized specificity). The ensemble model outperformed both individual models and an ensemble of the two highest-performing individual models (for more information, see the Supplemental Material).

Figure 5.

Area under receiver operating characteristic curve.

Table 2.

Comparison of ensemble model performance metrics

Performance Score/Metric	eICU		MIMIC
	Train	Test	Validate	Validate (Specificity Optimized)
AUROC	0.769	0.774	0.717	0.690
Accuracy	0.696	0.701	0.644	0.599
Brier score	0.198	0.196	0.223	0.223
F1 score	0.714	0.720	0.6964	0.413
Sensitivity	72.9%	73.0%	81.4%	28.2%
Specificity	66.0%	66.9%	47.2%	91.9%
PPV	70.0%	71.0%	60.8%	77.7%
NPV	69.1%	69.0%	71.7%	56.0%
LR+	2.14	2.20	1.54	3.46
LR−	0.41	0.40	0.39	0.78

MIMIC, Medical Information Mart for Intensive Care; AUROC, area under the receiver operating curve; PPV, positive predictive value; NPV, negative predictive value; LR, likelihood ratio.

Finally, the sensitivity analysis using a 1700 ml/40 mg dose showed consistent results, with results summarized in Figure 9 and Table 1 of the Supplemental Material.

Discussion

Machine learning algorithms are increasingly being used to help predict clinical outcomes, including sepsis and AKI, with implementation of sepsis algorithms associated with decreased length of stay in hospital.^20–22 In this study, we developed and externally validated a new machine learning algorithm to predict diuretic resistance in patients admitted to the ICU. Our final model can be optimized to be highly specific (>90%), while preserving overall discrimination (C statistic 0.69), and can potentially lead to better decision making for diuretic doses in patients with volume overload.

Diuretic resistance is increasingly recognized as an independent predictor of poor outcomes, and several recent studies have examined risk factors associated with diuretic resistance.^6,23–26 In a study performed as a post hoc analysis of the PROTECT randomized trial, investigators defined diuretic response as a change in baseline weight at day 4 per 40 mg furosemide equivalent dose on the basis of the total dose of furosemide from days 1 through 3.⁶ A poor diuretic response independently predicted mortality and rehospitalization, which was subsequently confirmed in an analysis of the RELAX-AHF trial and in another prospective study in the ICU setting.^5,14,23–25 More recently, investigators used a natriuretic response prediction equation on the basis of a 2-hour spot urine sodium postdiuretic administration and found a strong association with a total of 6-hour natriuresis.²⁶ Unfortunately, none of these previous studies integrated these factors in a risk prediction model that can be used prospectively by clinicians at the bedside. Interestingly, most of these previous studies found older age, female sex, lower blood pressure (mostly systolic), higher baseline creatinine, higher urea, and occasionally lower potassium/chloride as factors commonly associated with diuretic resistance, which was similar to our findings.

With respect to the four most important variables identified in our model, there are well-established mechanistic insights that help explain their relevance physiologically. For creatinine, it is well known that CKD predicts poor diuretic response, which is largely attributable to a decrease in glomerular filtration, accumulation of uremic anions, and proteinuria.^3,4 Hypochloremia has also been implicated in both diuretic resistance and increased mortality in heart failure.^27–29 Lower systolic pressure is associated with lower perfusion pressures to the kidney and is typically a marker for more severe disease. Finally, age is a well-established predictor of poor outcomes, given that older individuals tend to be more comorbid, frailer, and typically have less physiologic reserve. There are also the effects of aging on drug pharmacokinetics.³⁰

It is important to note that we chose to model diuretic resistance as defined by a urine output of ≤1400 ml per 40 mg IV furosemide dose (or equivalent) for two reasons. The first reason is its association with increased mortality and increased rehospitalizations in two separate independent trials.^13,14 The second reason stems from the inaccuracies and inconsistencies between day-to-day weights from both MIMIC-III and eICU databases, yielding any definition with weight unreliable for our purposes. Although we do acknowledge that daily weights are typically better at capturing actual fluid balance, a previous analysis showed no significant differences between using weights and urine output in defining diuretic resistance and its associated outcomes.¹³ Furthermore, the reliability of urine output charting in the ICU is quite high given the high prevalence of catheterized patients and the low nursing-to-patient ratios. Finally, the quantitative definition of diuretic resistance was extracted from a study of patients with heart failure, and there are well-described differences that exist in decompensated heart failure, such as shifting of the diuretic response curve.³ To assess the relevance of this, a subset analysis including only patients with a history of heart failure was performed, and it did not improve model performance and discriminability.

Clinically, as an example, physicians can identify patients at high risk of diuretic resistance and consider a starting dose of 80 mg IV furosemide instead of 40 mg, decide on early sequential blockade, and perhaps even consider earlier nephrology consult and possible early CRRT if resistance is confirmed. The idea is that earlier identification and treatment would lessen the time in an overloaded state and its associated complications, such as local ischemia, and could hopefully result in a reduced length of stay within the ICU. This could have significant implications from resource utilization and economic perspectives. In addition, combining our prediction algorithm with other tools such as the natriuretic response prediction equation from a 2-hour spot urine sodium could potentially greatly improve in-hospital diuresis. From a research perspective, a randomized control trial on the effects of an algorithm-guided strategy on effectiveness of diuresis and length of ICU stay should be performed.

The machine learning ensemble model described in this paper had reasonable performance for test and validate datasets. Interestingly, sensitivity was highest in the validation cohort at the cost of specificity. Given the interest in deploying this model to electronic medical charting software, high specificity was a prespecified requirement of the model to decrease “alert fatigue.” As such, we determined that an alert threshold corresponding to at least 90% specificity would be reasonable for future implementation studies. We believe that this threshold would balance the correct identification of diuretic-resistant individuals against the potential risks of escalating therapy too quickly.

This study has other notable strengths. The databases included provide a multicenter perspective that spans nearly two decades, and although the original sample size may have been truncated due to our inclusion criteria, the number of actual samples generated from 24-hour windows provided a large database for analysis. This captured the real-world heterogeneity of diuretic resistance that exists clinically. The model also uses variables that are easily accessible and has a built-in algorithm to handle missing values. Finally, although the model generates probabilities, it ultimately produces a binary output to simplify its use at the bedside either through integration into electronic medical software or manually via an online calculator, as an example. From a software perspective, the use of Python ensures portability, compatibility, and easy integration with any existing programming platforms. This article could also serve as a proof of concept/framework for the development of other prediction tools or simply for the use of future machine learning projects/prediction models.

This study has several limitations. To begin, the retrospective nature brings about all its inherent limitations, and, although externally validated, the lack of prospective validation does limit potential applications. It is also unclear to what effect predicting loop diuretic resistance early will actually affect patient outcomes (e.g., decreased mortality and decreased length of stay). The derivation and validation were performed on an ICU population, so generalizability to medicine wards or the emergency department remains unknown. There were a large amount of missing/incomplete data that would have likely been relevant and may have improved model performance owing to the proposed pathophysiologic mechanisms underlying loop diuretic resistance.³ These include, but are not limited to serum albumin, urine studies (electrolytes, creatinine, albumin, and protein), BNP, troponin, lactate, and blood pH. There were also incomplete data on dialysis, and so a complete analysis for this was impossible yet likely relevant. Finally, each 24-hour window was treated as a separate independent datapoint, whereas the median length of stay per patient was 4 days in the eICU cohort and 6 days in the MIMIC-III cohort (not including repeated admissions).

Disclosures

T.W. Ferguson reports the following: Consultancy: Clinpredic Ltd.; Quanta Dialysis Technologies and Strategic Health Resources (Tricida Inc., Protagonist Therapeutics); Ownership Interest: Klinrisk Inc. and Palpate Health Ltd.; and Honoraria: Baxter Canada. N. Tangri reports the following: Consultancy: Marizyme, Mesentech Inc., PulseData Inc., Renibus, and Tricida Inc.; Ownership Interest: Clinpredict Ltd., Klinrisk, Marizyme, Mesentech Inc., PulseData Inc., Quanta, Renibus, and Tricida Inc.; Research Funding: Astra Zeneca Inc., Bayer, BI-Lilly, Janssen, Otsuka, and Tricida Inc.; Honoraria: Astra Zeneca Inc., Bayer, BI-Lilly, Janssen, Otsuka Pharmaceuticals, and Pfizer; Patents or Royalties: Klinrisk and Marizyme; Advisory or Leadership Role: Clinpredict, Klinrisk, and Tricida Inc.; and Other Interests or Relationships: National Kidney Foundation; Founder—Klinrisk, Clinpredict. The remaining author has nothing to disclose.

Funding

None.

Author Contributions

J.A. Mercier and N. Tangri conceptualized the study; J.A. Mercier was responsible for data curation, formal analysis, investigation, methodology, project administration, resources, software, and visualization and wrote the original draft; J.A. Mercier and N. Tangri were responsible for validation; and T.W. Ferguson and N. Tangri provided supervision and reviewed and edited the manuscript.

Data Sharing Statement

Partial restrictions to the data and/or materials apply: The data come from two MIT PhysioNet databases, MIMIC-III (https://physionet.org/content/mimiciii/1.4/) and eICU (https://physionet.org/content/eicu-crd/2.0/), which are both publicly available databases provided the following three conditions be met: (1) be a credentialed user; (2) completed required training: CITI Data or Specimens Only Research; (3) sign the data use agreement for the project. With regard to ref. 3, this explicitly prohibits sharing of the data with anyone that does not have access. Fortunately, access is quite easy and straightforward to obtain, and all the raw data are available on Google Bigquery (“the cloud”) or for download (to be assembled with SQL). The formatted data and code for the model is included in the following repository: https://github.com/jomericer/LDR_MLprediction.

Supplemental Material

This article contains the following supplemental material online at http://links.lww.com/KN9/A234.

Initial variables.

Shapley additive explanations (SHAP) values.

Initial variable selection.

Supplemental Figure 1. Feature importance plot—weight.

Supplemental Figure 2. Feature importance plot—gain.

Supplemental Figure 3. Feature importance plot—SHAP summary plot.

Individual model characteristics (eICU).

Model 1. Train and test metrics.

Supplemental Figure 4. Model 1 ROC.

Model 2. Train and test metrics.

Supplemental Figure 5. Model 2 ROC.

Model 3. Train and test metrics.

Supplemental Figure 6. Model 3 ROC.

Model 4. Train and test metrics.

Supplemental Figure 7. Model 4 ROC.

Validation/external testing.

Model 1. Validation (MIMIC).

Model 4. Validation (MIMIC)

Model 1 + 4. Ensemble validation (MIMIC).

Supplemental Figure 8. Models 1 + 4 ensemble model validation ROC (eICU and MIMIC).

Sensitivity analysis.

Supplemental Figure 9. AUC (1700 ml/40 mg model).

Supplemental Table 1. Comparison of ensemble model performance metrics (1700 ml/40 mg model).

Supplemental References