Using Machine Learning to Predict Hyperchloremia in Critically Ill Patients

Abstract

Elevated serum chloride levels (hyperchloremia) and the administration of intravenous (IV) fluids with high chloride content have both been associated with increased morbidity and mortality in certain subgroups of critically ill patients, such as those with sepsis. Here, we demonstrate this association in a general intensive care unit (ICU) population using data from the Medical Information Mart for Intensive Care III (MIMIC-III) database and propose the use of supervised learning to predict hyperchloremia in critically ill patients. Clinical variables from records of the first 24h of adult ICU stays were represented as features for four predictive supervised learning classifiers. The best performing model was able to predict second-day hyperchloremia with an AUC of 0.80 and a ratio of 5 false alerts for every true alert, which is a clinically-actionable rate. Our results suggest that clinicians can be effectively alerted to patients at risk of developing hyperchloremia, providing an opportunity to mitigate this risk and potentially improve outcomes.

I. Background

Intravenous (IV) fluids are commonly administered in critical care settings and have traditionally been considered to be safe. There is growing evidence, however, that aggressive administration of fluids even within current reference therapeutic ranges may be detrimental [1], causing organ damage and increased in-hospital mortality. Apart from the actual volume of fluid administered, one potential problem is the chloride content of many traditional fluids, such as normal saline fluid. For example, high chloride content in large-volume resuscitation [2] and in resuscitation of patients with sepsis [3] was shown to be associated with increased rates of in-hospital mortality. Conversely, the implementation of a low-chloride strategy in the ICU has been shown to be associated with a significant decrease in incidence of acute kidney injury (AKI) and renal replacement therapy [4]. Hyperchloremia has also been shown to be associated with increased rates of mortality [5] and AKI [6] in patients with sepsis. This suggests that a shift from traditional, chloride-liberal fluid administration to chloride-restrictive approaches may be desirable for the treatment and resuscitation of critically ill patients.

Critically ill patients generate thousands of data points each day and a large portion of that data is stored in electronic health records (EHRs) [7]. This rich EHR data details both the medical interventions administered during each patient’s ICU stay and the outcomes that follow. By mining data in the publicly-available Medical Information Mart for Intensive Care III (MIMIC-III) [8] cohort, we conducted a retrospective study on a broad ICU population to evaluate the associations between hyperchloremia and patient outcomes of mortality and organ dysfunction. Additionally, given the promising developments in predicting clinical events with machine learning, we propose the use of clinical data from the EHR to develop a predictive model for hyperchloremia. Such a predictive model can identify patients at high risk of developing hyperchloremia, allowing clinicians to make appropriate adjustments to chloride-related management and potentially improve outcomes. We open sourced our code at https://github.com/chloride-management/chloride-management.

II. Related Work

There have been numerous prior studies on predictive modeling using MIMIC-III data. Many of these studies were specific to sub-populations of patients, such as patients with sepsis [9], myocardial infarction [10], pneumonia [11], kidney injury [12], etc. Thus, there is still much room for predictive modeling of the general ICU population. These predictive models also typically predict outcomes, such as mortality [13], ICU readmission [14], AKI [15], [16], and other complications [17], and are able to do so with diverse areas under the receiver operating characteristic (ROC) curve that range from 0.65 to 0.9. Because these models are typically outcome-focused, there is currently a knowledge gap for predictive modeling that is focused on treatment management. This distinction is important in clinical decision-making—predicting mortality, for example, may have high prognostic value, but because there are many potential factors that may or may not prevent mortality this detail is several steps away from clinically-actionable information.

Our study aims to improve upon these previous works by using all adult patients in the MIMIC-III database—a large, generalizable ICU population. Our approach also aims to predict hyperchloremia, which is a clinically-actionable variable. Because chloride administration can be actively managed in the ICU, the information provided by our model can immediately prompt interventions to reduce chloride load and limit hyperchloremia. And because we can demonstrate associations between hyperchloremia and several key outcomes, this frame-work has the potential to improve patient outcomes if there is indeed a causal link between chloride load, hyperchloremia, and poor outcomes.

III. Methods

We extracted retrospective data from the MIMIC-III database with the goal of studying meaningful outcomes using available information from the early phase of critical illness. We first assessed the association between hyperchloremia and outcomes to establish the clinical impact of chloride on a general ICU population. Then, we fed data from the first 24 hours (“day 1”) of a patient’s ICU stay into predictive models. These models were trained to predict the likelihood that a patient will develop hyperchloremia on day 2.

A. Data Source

MIMIC-III contains over 40,000 de-identified patient records from critical care units at the Beth Israel Deaconess Medical Center, collected from 2001 to 2012. These records contain a wealth of information organized by patient, hospital admission, and ICU stay, each of which was assigned a unique ID. We wrote SQL scripts to query a local instance of this database and then extracted and aggregated data by ICU stay.

B. Inference Analysis

We first evaluated the association between hyperchloremia (serum chloride ≥ 110 mEq/L) in the first week of ICU stays and outcomes in critically ill patients. We studied several objective outcome measures: mortality during the ICU stay (ICU mortality), multiple organ dysfunction syndrome (MODS) [18] on day 7, and new AKI by day 7.

AKI stage was determined using the Kidney Disease Improving Global Outcomes (KDIGO) clinical practice guide-lines [19]. We considered new AKI to be any increase (i.e. worsening) in AKI stage at any point within the first 7 days. MODS on day 7 was considered to be present if any two Sequential Organ Failure Assessment (SOFA) [20] sub-scores had a score of 2 or above on day 7.

Multivariate logistic regression models were used to assess these associations between hyperchloremia and outcomes after controlling for age, gender, ethnicity, and SOFA score on day 1 as a measure of severity of illness on admission.

C. Prediction Modeling

1). Feature Set:

We set our modeling task to be the prediction of hyperchloremia in the second 24 hours of ICU care (day 2) using data from the first 24 hours (day 1). This time frame includes the highest number of new hyperchloremia cases in the dataset. We queried the MIMIC-III database for clinical variables from each patient’s day 1, identified for consideration based on prior literature and clinical expertise. In particular, we considered chloride-related data (serum chloride level, chloride load, fluid balance), demographics, comorbidities on admission, vitals and laboratory test results, and interventions and medications.

We excluded records of patients under the age of 18, repeat ICU stays in the same hospitalization, patients who were hyperchloremic on day 1, and patients with no measured serum chloride on day 1. Among first ICU stays of adult patients, 16,366 (32.9%) of 49,695 records were excluded because they either were hyperchloremic or had no recorded chloride levels on day 1.

Chloride load included any administration of chloride-containing fluid (e.g. normal saline, potassium chloride, etc.) and each amount was converted to milliequivalent (mEq) quantities using standard ratios. Because chloride-related events can occur at any time, we aggregated data over the first 24 hours to determine a patient’s total chloride input, maximum serum chloride, and net fluid balance on day 1. After assessing the collected data for numerical outliers, we set cutoffs to exclude nonsensical values.

Pertinent comorbidities were selected using ICD-9 codes based on the Elixhauser Index [21]. Because comorbidities in the MIMIC-III database are tied to hospital admissions and do not have timestamps, we gathered comorbidities that were assigned during past hospital admissions for each ICU stay. Thus, our predictive models only used comorbidities that were already known on admission.

Vitals, laboratory test results, interventions, and medications were collected from the first 24 hours of each ICU stay.

Though these features were selected based on prior literature and clinical expertise, they were not guaranteed to have meaningful influences on our predictive models. We thus finalized our feature set by utilizing chi-squared tests and two-sample t-tests to establish statistical significance through univariate analysis. Features that were ultimately included had significantly different distributions (p-value < 0.05) between patients who were hyperchloremic on day 2 and patients who were not.

In sum, this resulted in a total of 33,329 rows of unique ICU stays with 36 features each. These rows were then divided on a 70:30 train:test split with the testing set held out for performance evaluation.

2). Imputation of Missing Data:

Patients who had no records of chloride administration on day 1 were assumed to have no chloride input on that day. The same was assumed for fluid input/output, comorbidities on admission, and any intervention or medication we selected as a feature. Each missing vital and laboratory test result was imputed using the median of all values from all patients collected over the 24 hours prior to their discharges from the ICU (i.e. only patients who survived were included). This was considered to be a proxy of a ”healthier” state. To keep the testing set unseen, we determined medians using only the training set data.

For our prediction outcome, a patient with no serum chloride measurement on day 2 was assumed to be non-hyperchloremic. The vast majority of these patients did not have measurements after day 2 as well, and thus we assumed that clinicians had no reason to be concerned about their chloride status.

3). Machine Learning Classifiers:

We treated this scenario as a classification problem—we predicted hyperchloremia by classifying each patient as high-risk for hyperchloremia or not on day 2. We selected four classifiers to evaluate: lasso regression, random forests, XGBoost [22], and multi-layer perceptrons. All non-binary variables were standardized via medians and interquartile ranges (IQRs) derived from the training set so that each feature would have a median of 0 and an IQR of 1.

Once trained, each classifier determined the probability that a patient would develop hyperchloremia on day 2. Though the probability cutoff for classifying a patient as hyperchloremic can be arbitrarily changed, we chose the threshold that maximizes Youden’s J statistic [23] on the training set to balance precision and recall.

Because hyperchloremia on day 2 has a low prevalence in our dataset (5.84% and 6.30% in the training set and testing set, respectively), special care was taken to adjust for the imbalance. In the case of lasso regression and XGBoost we set a parameter to determine class weights based on prevalence, effectively adjusting the balance of data. For random forests and multi-layer perceptrons, we downsampled the training set by randomly removing 90% of the patients who were non-hyperchloremic on day 2. This reduced our training set to 3559 rows but increased prevalence to 38.27%, which provided an acceptable balance of training data to fit our classifiers with.

Hyperparameters were tuned using GridSearchCV from the scikit-learn [24] library. This performs an exhaustive search over a specified parameter space to maximize a metric using cross-validation on the training set. In our case, we maximized the area under the ROC curve using 4-fold cross-validation.

4). Performance Evaluation:

We evaluated classifier performance on the testing set using a variety of metrics that were sensitive to the imbalanced nature of our dataset. For each model’s testing set predictions, we determined the area under the ROC curve, precision, recall, and F₁-score. To better evaluate precision/recall trade-offs, we plotted precision-recall curves with their respective areas under the curves (AUCs).

IV. Results

A. Inference Analysis

Clinical characteristics of the population we evaluated are shown in Table I. Univariate inference analysis conducted using chi-squared tests demonstrated statistically significant differences (p-value < 0.001) in all outcomes (ICU mortality, MODS on day 7, and new AKI by day 7) when comparing patients based on presence of hyperchloremia ≥ 110 mEq/L.

TABLE I.

Clinical Characteristics and Outcomes by Hyperchloremia Status over Days 1–7

	All Patients (n = 48074)	Non-Hyperchloremic (n = 29893)	Hyperchloremic (n = 18181)	p-value
Demographics
Age, median (IQR)	65.7 (52.9–77.8)	65.0 (52.4–77.4)	66.9 (53.8–78.5)	< 0.001
Female, n (%)	21008 (43.7)	12684 (42.4)	8324 (45.8)	< 0.001
Race, n (%)
White	34407 (71.6)	21477 (71.8)	12930 (71.1)	< 0.001
Black	4620 (9.6)	3006 (10.1)	1614 (8.9)
Hispanic	1651 (3.4)	1053 (3.5)	598 (3.3)
Asian	1121 (2.3)	603 (2.0)	518 (2.8)
Other (Unreported)	6275 (13.1)	3754 (12.6)	2521 (13.9)
Comorbid Conditions, n (%)
Cancer	4576 (9.5)	3088 (10.3)	1488 (8.2)	< 0.001
Cardiovascular	26223 (54.5)	16385 (54.8)	9838 (54.1)	0.137
Diabetes	13061 (27.2)	8436 (28.2)	4625 (25.4)	< 0.001
Hepatic	4949 (10.3)	2960 (9.9)	1989 (10.9)	< 0.001
Renal	7868 (16.4)	5279 (17.7)	2589 (14.2)	< 0.001
Respiratory	10697 (22.2)	7263 (24.3)	3434 (18.9)	< 0.001
Multiple Comorbidities	40123 (83.5)	24819 (83.0)	15304 (84.2)	0.001
Outcome, n (%)
ICU Mortality	3558 (7.4)	1703 (5.7)	1855 (10.2)	< 0.001
MODS on Day 7	1890 (3.9)	781 (2.6)	1109 (6.1)	< 0.001
New AKI by Day 7	17233 (35.8)	9445 (31.6)	7788 (42.8)	< 0.001

Multivariate regression analysis demonstrated that hyperchloremia was independently associated with increased ICU mortality (adjusted odds ratio 1.376 [95% CI, 1.280–1.481]), increased rates of MODS on day 7 (1.823 [1.652–2.012]), and increased rates of new AKI by day 7 (1.680 [1.615–1.747]) after adjusting for demographic variables and severity of illness on admission (SOFA score).

B. Prediction Modeling

1). Classifier Performance:

All four classifiers performed similarly (Fig. 1 and Table II). The random forest classifier had the best quantitative performance, with AUCs of 0.80 and 0.22 for the ROC curve and precision-recall curve, respectively. Maximizing Youden’s J statistic yielded a precision of 0.1685 and an F₁-score of 0.2690. In comparison, the lasso regression classifier yielded AUCs of 0.79 and 0.18 for the ROC curve and precision-recall curve, a precision of 0.1408, and an F₁-score of 0.2370. It is important to note, however, that because classifier performance is similar at recall levels of ≥ 0.75, lasso regression may be preferable given its ease of interpretation.

Fig. 1.

Testing Set Receiver Operating Characteristic (ROC) and Precision-Recall Curves

TABLE II.

Testing Set Precision, Recall, F₁-score

Model	Precision	Recall	F1-score
Lasso Regression	0.1408	0.7492	0.2370
Multi-layer Perceptron	0.1492	0.7206	0.2472
Random Forest	0.1685	0.6667	0.2690
XGBoost	0.1447	0.7524	0.2428

Evident in Table II is a broad preference for recall over precision when the classification threshold is chosen based on Youden’s J statistic. This is desirable for our use case as we can tolerate false positive alerts while detecting a large number of patients at high risk of becoming hyperchloremic.

A precision of 0.17 for our random forest model roughly translates to a ratio of 5 false positive predictions for every 1 true positive. For our lasso regression model with a precision of 0.14, the ratio is approximately 6 to 1. This is considerably better than the baseline performance of classifying all patients as high-risk for hyperchloremia, which would have a precision equal to the prevalence of 0.06 and a ratio of 15 to 1. While this improvement certainly would not be enough to warrant high-risk interventions, it could provide reasonable justification for chloride-reducing interventions that are generally considered to be safe in ICU patients—for example, fluids with lower chloride content or early loop diuretic initiation.

V. Discussion

The results of our inference analysis show that general ICU patients with hyperchloremia have significantly increased odds of mortality, MODS on day 7, and new AKI by day 7 after adjusting for confounders. This is similar to and consistent with prior studies, which have found high chloride levels to be associated with increased mortality and low-chloride strategies to be advantageous in reducing kidney injury in certain subgroups of ICU patients [2]–[4]. Here we show this association in a large, general ICU population of 48,074 patients, which increases the generalizability of these findings.

Our classifier models were able to predict hyperchloremia on the second day of each ICU stay with good performance using data from the first day. As far as we know, these are the first predictive models for hyperchloremia. The four models we tested performed similarly, with areas under the ROC curve of approximately 0.80 and ratios of false positives to true positives of between 5 and 6 to 1. Most importantly, our results are clinically actionable. Because hyperchloremia and the quantity of chloride administered during a patient’s management in the ICU may have implications on important outcomes, clinicians can manage and adjust the amount of chloride given or enhance chloride elimination strategies, such as with the use of diuretics, to potentially improve morbidity and mortality. Because the ratio of false positives to true positives is relatively low, implementing these chloride management strategies for patients who are predicted to be high-risk would be much more efficient than doing so for all patients, which could become cumbersome or cost-prohibitive.

A. Limitations and Future Work

Our work so far has been limited to predictions for day 2. Further work would involve expanding the window of data beyond day 2 to evaluate continuous, rolling predictions. Other limitations of our study are largely defined by our selection of features from the data available in MIMIC-III. To generalize these results and evaluate potential additional features, further investigation should be conducted on other cohorts of ICU patients. With that said, there is still a wealth of information that we did not explore in MIMIC-III. For example, we did not consider including the contents of clinical notes in our study, nor did we consider longitudinal trends over the course of each patient’s first day and overall ICU stay. Such information would likely provide considerable insight into treatment developments and outcomes, and future work would include more complex models and feature engineering.

Lastly, further work should be done to determine what interventions, if any, would lead to improved outcomes, as our findings are only associations and the hypothesis that a causal link exists remains to be tested.

VI. Conclusion

Our results show that hyperchloremia during the acute phase of critical illness is independently associated with higher odds of ICU mortality, MODS on day 7, and new AKI by day 7 in general ICU patients. We demonstrate that supervised learning classifiers equipped with features from the first day of a patient’s ICU stay can effectively predict next-day hyperchloremia with clinically-actionable performance levels. We show that these classifiers would perform at a ratio of 5 to 6 false alerts for every true alert while maintaining a reasonable recall of ≥ 0.75, which is a considerable improvement given the baseline prevalence of hyperchloremia. This could provide sufficient justification for changes in patient care to prevent hyperchloremia, which could then result in improved clinical outcomes. Further work should be done to validate this model in other cohorts of ICU patients, and to evaluate the efficacy of interventions once at-risk patients have been identified.