Enhancement of Inpatient Mortality Prognostication with Machine Learning in a Prospective Global Cohort of Patients with Cirrhosis with External Validation

Abstract

Background and aims:

Cirrhosis is a major global burden requiring frequent hospitalizations with a high inpatient mortality. Traditional prognostic tools focused on inpatient mortality are affected by global disparities, which impacts timely management. We aimed to deploy machine learning (ML) approaches to enhance inpatient mortality prognostication.

Methods:

Using the prospective CLEARED cohort that enrolls cirrhosis inpatients globally, we used admission-day data to predict inpatient mortality with ML approaches versus logistic regression. Internal validation (75/25 split) and subdivision using World-Bank income status [Low/low-middle (L-LMIC), upper-middle (UMIC), high (HIC)]were performed. The ML model with the best area-under-the-curve (AUC) was externally validated in a US-Veteran cirrhosis inpatient population.

Results:

CLEARED Cohort included 7,239 cirrhosis inpatients (64% men, 56±13 years, median MELD-Na 25) from 115 centers globally. 22.5% belonged to LMICs, 41% to UMICs, and 34% to HICs. 11.1% (n=808) of patients died in-hospital. Random-Forest (RF) showed the best AUC (0.815) with high calibration, which was significantly better versus parametric logistic regression and LASSO models (AUC:0.774, p<0.001, AUC:0.787, p=0.004 respectively). RF was the ML method with the highest AUC and remained better than logistic regression regardless of country income-level; HIC (AUC:0.806), UMIC (AUC:0.867), and L-LMICs (AUC:0.768). External validation was performed in 28,670 Veterans (96% men, 67.8±10.3 years, median MELD-Na:15) with 4% (n=1158) inpatient mortality. The AUC using the CLEARED-derived RF-model was 0.859.

Conclusion:

Random forest analysis trained on a global prospective cirrhosis cohort enhances mortality prediction over traditional methods, is consistent across country income levels, and is successfully validated externally in a US-Veteran population.

Graphical Abstract

Lay Summary

We developed a machine learning model that improves prediction of inpatient death over traditional statistics across a global cirrhosis cohort, which was validated in a US-based national cirrhosis cohort.

Introduction:

Liver injury leading to cirrhosis due to alcohol, viral hepatitis, and steatotic liver disease is a major global burden characterized by frequent hospitalizations¹. Patients with cirrhosis who are hospitalized have a poor prognosis with a high risk of inpatient death or hospice transfer². However, there are wide variations across the world with regards to resources available, outpatient services, reasons for admission, and etiologies of cirrhosis, which can influence these outcomes^3–6. There is scope for improvement in prediction of these outcomes over traditional statistical regression methods with better representation^5–7. With the emergence of machine learning (ML), a higher number of variables can be studied concurrently with potentially better prediction of outcomes^{8, 9}. Enhanced prediction capabilities could help practitioners prioritize patients at higher risk for earlier interventions to potentially prevent these outcomes and triage them for liver transplant, end-of-life care, or transfer to institutions better equipped to handle these complications^{8, 10, 11}. However, current applications of ML in cohorts are restricted to either regional or single center settings and needs global representation^{12, 13}.

The Chronic Liver Disease Evolution And Registry for Events and Decompensation (CLEARED) consortium consists of inpatients with cirrhosis enrolled from all six populated continents, which gives us the opportunity to study the additional benefit of ML over traditional regression in outcome prediction across practice types and income levels³. The Veterans Health Administration (VHA) operates the largest integrated healthcare system in the United States, providing comprehensive liver care across diverse settings^14–16. These data are accessible from the Veterans Corporate Data Warehouse (VA-CDW), which includes medical records of all US Veterans and has been used to characterize cirrhosis-related and other disease outcomes^{14, 17, 18}. We hypothesized that ML methods would enhance inpatient mortality prediction over and above logistic regression across the entire CLEARED cohort, which would continue even when subdivided by country-income levels, which would in turn be externally validated in a cohort of US Veterans with cirrhosis.

Methods:

We utilized two cohorts: CLEARED consortium for model generation and the VA-CDW for external validation.

CLEARED cohort:

This global consortium consists of prospectively enrolled inpatients with cirrhosis admitted non-electively between November 2021 through May 2024 across all sites³. After local ethics board approval at each site, only patients with confirmed cirrhosis who were able to consent were enrolled. We excluded those with unclear cirrhosis diagnosis, those unable to consent, those with COVID-19, or prior transplant. We only allowed a maximum of 100 prospectively enrolled patients per site as a convenience sample to ensure diverse and equitable representation worldwide.

For all patients, a structured database was used to record demographics, cirrhosis details (etiology, prior complications, prior admissions, or infections) and comorbid conditions as previously published.³ Admission details included reason(s) for admission, medications, laboratory values, and a validated measure of cirrhosis severity (model for end-stage liver disease sodium (MELD-Na)¹⁹ on admission. Patients were followed throughout their inpatient course for mortality. Only one admission per-patient was considered. Transfer to hospice care was not counted as a mortality event if patients were not explicitly charted as dying in-hospital, due to potential disparities in hospice care and availability among centers. Data were also collected regarding other inpatient events such as intensive care unit (ICU) transfer, dialysis, mechanical ventilation, nosocomial infections, acute kidney injury (AKI), hepatic encephalopathy (HE) grade 3–4, vasopressor use/shock, as well as length of stay (LOS) for the entire stay and in the ICU. Using World Bank classifications, we classified centers into high income (HIC), upper-middle (UMIC) and low/low-middle income countries (LMICs)²⁰.

The primary outcome was inpatient mortality based on data available on the day of admission. For covariates with missing values, multiple imputation by chained equations was used to impute missing information²¹. Specifically, we used five iterations of predictive mean matching in a singular dataset, including the outcome variable as a contributor and performing this process independently in training and testing sets to avoid data leakage. This outcome variable was only used in the training set imputation.

The specific variables that required imputation were admission hemoglobin, admission WBC, admission platelets, admission albumin, and admission MELD-Na score, and the remaining variables all had complete information available. Variables with the highest missingness rates were MELD-Na (7.8%), WBC (6.1%), and platelets (5.5%). Individual components of MELD-Na score were not included in the models. Cohort characteristics were summarized and compared between the outcome groups in a univariable manner. Continuous variables were summarized using means and standard deviation (SD) or medians (IQR) depending on distribution, while categorical variables were summarized using counts and percentages of the total. Variables were compared between the groups using two sample t-tests, Wilcoxon Rank-Sum tests, or Pearson’s Chi-Squared tests, as appropriate. RStudio version 4.4.1 was used for all statistical analysis. All hypothesis tests were two-sided with statistical significance considered p<0.05.

Machine-Learning Analysis:

Four classification algorithms were examined and compared, these included standard multivariable logistic regression (LR) which included all covariates and no further interaction or non-linear effects, LASSO logistic regression, Random Forest (RFA), and Extreme Gradient Boosting (XGBoost). Both Random Forest and XGBoost²² are decision-tree based methods that aim to combine individual statistical learners in either an additive or averaged manner in order to provide strong performance while controlling for the overfitting that may occur when using only a single or small number of trees²³. Decision tree methods have potential advantages over traditional parametric models due to their ability to model interactions, nonlinearity, and other patterns without explicitly knowing these relationships, as well as their invariance to the distribution of the covariates, meaning that non-normality will not affect quality of the fit.

The full dataset was split randomly in a 75/25 ratio for training and testing each potential model. Our metric of choice to assess each method was the area-under-the receiver operating characteristic curve (AUC), which is a probability threshold-independent metric of a classifier’s discriminative performance. We evaluated each model’s AUC on the testing set, comparing the results to those obtained from traditional logistic regression using paired receiver-operating curve tests²⁴. The best model for each outcome was selected based on this test-set AUC. The XGBoost algorithm contains many hyperparameters that can be tuned in order to potentially increase performance. In this study, we compared results using both the default hyperparameter setting²⁵ and a set of hyperparameters obtained using a random search method with 60 iterations, selected using ten-fold cross-validation in the training set. The XGBoost model that performed best of these two was selected for further analysis. Random search is a method of selecting hyperparameters based on random sampling from a set of pre-specified values and ranges²⁶. This method has been shown to recover a set of hyperparameters that is in the top 5% of all potential possibilities after just 60 repetitions. The specific hyperparameters that were tuned were the learning rate (range: 0.001–0.03), regularization parameter gamma (range: 0–25), regularization parameter lambda (range: 0–25), maximum tree depth (range: 1–12), and number of trees in the ensemble (range: 25–100). These details can also be found in the Supplementary Section. For the Random Forest models, we considered the default parameters only²⁷, as RF models have been shown to provide strong results using these values^{28, 29}. For LASSO logistic regression, the hyperparameter λ controls the degree of regularization, and must also be tuned. We first standardized continuous variables by subtracting the mean and dividing by the standard deviation. Then, we used ten-fold cross-validation to select the largest value of λ, such that CV-AUC was within 1 standard error of the optimal value. This is known as “lambda-1se,” and has been shown to produce the most parsimonious model, while sacrificing only minimal gain in performance³⁰.

Variable Importance:

Within the best-performing models for each outcome, the top-15 variables using standard variable importance metrics (if XGBoost: Percentage contribution by Gain^{25, 26}, if RFA: Permutation Importance²⁵) were obtained, and the top-performing model was re-fit using these variables only, in order to create a more parsimonious model that could be easily applied in a clinical setting. In XGBoost, this variable importance measure provides the relative contribution of each feature towards the overall prediction model – a feature that separates outcome classes more effectively will be flagged as more important. In Random Forest, the permutation variable importance represents the average difference in prediction accuracy between the true included variable, and a copy of the variable where values have been randomly permuted. This has been shown to be the most reliable measurement when evaluating Random Forest models³¹. A higher value of permutation importance implies that the feature is truly contributing to the predictive accuracy of the model and is not a result of random chance alone. We chose 15 as a cutoff for the number of variables included in this model to optimize the trade-off between model complexity and parsimony.

The supplementary section includes a full data dictionary (Table S1). This analysis was performed on the entire cohort, and then a subgroup analysis was performed for HICs, UMICs, and LMICs individually, to assess any income-related differences in model performance. The final model was also compared to established scores - MELD-Na and MELD 3.0 – and performance gains discussed. As an additional subgroup analysis, final model performance was compared across alcoholic cirrhosis etiology vs. non-alcoholic etiology, HE on admission vs. no-HE on admission, and infection on admission vs. no-infection on admission (Table S5). Probability calibration in the testing set was assessed over the final ML model containing all predictors and the top-15 predictors only, by assessment of calibration slope and intercept, as well as Brier score. We compared the predicted probabilities from the machine learning model to the actual event rates. If probabilities are “correctly calibrated,” we would expect a plot of predicted vs. true rates to be a line with an intercept of 0 and a slope of 1. As an example, if the ML model predicts an inpatient death probability of 20%, we would expect that ~20% of these patients actually died. The Brier score is a metric of probability accuracy and calibration, where a lower value indicates more accurate predictions.

Finally, we assessed the clinical utility of the predicted probabilities from the final ML model by assessing positive-predictive value (PPV), negative-predictive value (NPV), sensitivity, and specificity at three cutoffs of >10%, >25%, and >50% predicted mortality risk. We discuss implications of using this model to complement clinician decision-making.

VA-CDW validation:

Veterans with cirrhosis who were admitted between 2020 and 2023 were collected to complement that timeframe of CLEARED. The Richmond VA IRB approved this protocol that waived informed consent due to de-identified records. The codes used for definition of the variables are well validated in prior studies^{14, 16, 32}. We followed them for inpatient mortality, and the best-performing model trained on the CLEARED cohort was applied. Missing values in VA-CDW data were handled in the same manner as CLEARED; only Platelets, hemoglobin, and WBC required imputation. Not all variables in the full machine-learning model were able to be collected in VA-CDW due to database limitations (see: Supplementary for complete list) – we omitted these variables from the models when comparing performance. Additionally, we assessed performance of our CLEARED-derived model which including only the top-15 predictors detailed above.

Results:

CLEARED Cohort:

Our entire cohort included 7,733 patients from 121 centers and 36 countries (Figure S7, Table S6 and S7). After excluding those who received liver transplant or who had incomplete data, 7,239 patients were included in modelling of inpatient mortality, of which 808 (11.1%) of patients died in-hospital (Figure 1A). Median time-to-death was 11 days, with an IQR of 6–21 days. Time-to-death distribution was heavily right skewed, with around one-third (33.4%) of mortality events occurring within one week, and 11.5% dying after more than 30 days.

Figure 1: Flowchart of the derivation and validation cohorts of inpatients with cirrhosis.

A: CLEARED (Chronic Liver disease Evolution And Registry for Events and Decompensation) enrolls inpatients with cirrhosis from all 6 populated continents with up to 100 patients/site, B: VA-CDW (Veterans Affairs Corporate Data Warehouse) was evaluated to include medical records from hospitalized Veterans with cirrhosis. LT: liver transplantation.

As shown in table 1, the mean age was almost 56±13 years, most (64%) were men, and the most common etiology was alcohol. 22.5% belonged to LMICs, 41% to UMICs, and 34% to HICs. Most patients had ascites and were on diuretics, and almost half had been hospitalized over the last 6 months. Almost a quarter had other cirrhosis complications, 11% were listed for transplant and almost a fifth had experienced prior infections. Almost half were on lactulose and proton pump inhibitors (PPIs), quarter on rifaximin, and 10% were on statins. Most reasons for admission were liver or infection-related, with a median admission MELD-Na of 25.

Table 1:

CLEARED Cohort Characteristics

	Total cohort (n = 7733)	Inpatient mortality
		Died (n = 808,11.1%)	Survived (n = 6431, 88.8%)	P value
Demographics
Age	55.97 (13.37)	55.28 (13.44)	56.25 (13.35)	0.051
Male Sex	4975 (64.3%)	541 (67.0%)	4105 (63.8%)	0.088
World Bank Classification	LIC: 1737 (22.5%) UMIC:3336(41.3%) HIC:2660 (34.4%)	LIC: 309 (38.2%) UMIC: 301 (37.3%) HIC: 198 (24.5%)	LIC: 1129(19.1%) UMIC: 2939 45.7%) HIC: 2263 (35.2%)	<0.001
Comorbidities
Diabetes	2284 (29.5%)	210 (26.0%)	1922 (29.9%)	0.025
Hypertension	1947 (25.2%)	194 (24.0%)	1639 (25.5%)	0.386
Hyperlipidemia	1026 (13.3%)	88 (10.9%)	877 (13.6%)	0.035
Etiology
Alcohol Use	3211 (41.5%)	390 (48.3%)	2640 (41.1%)	<0.001
MASLD	1378 (17.8%)	139 (17.2%)	1128 (17.5%)	0.850
Hepatitis B	1453 (18.8%)	91 (11.3%)	1297 (20.2%)	<0.001
Hepatitis C	809 (10.5%)	75 (9.3%)	687 (10.7%)	0.245
Autoimmune Hepatitis	423 (5.5%)	60 (7.4%)	333 (5.2%)	0.010
Cryptogenic	573 (7.4%)	71 (8.8%)	471 (7.3%)	0.156
Other	84 (1.1%)	24 (3.0%)	273 (4.2%)	0.104
Cirrhosis last 6M
Ascites	4327 (63.7%)	558 (69.1%)	4039 (62.8%)	<0.001
Variceal Bleed	2243 (29.0%)	252 (31.2%)	1860 (28.9%)	0.196
Overt HE	2150 (27.8%)	328 (40.6%)	1676 (26.1%)	<0.001
Hyponatremia	1285 (16.6%)	196 (24.3%)	971 (15.1%)	<0.001
AKI	1272 (16.4%)	212 (26.2%)	942 (14.6%)	<0.001
Hydrothorax	600 (7.8%)	59 (7.3%)	499 (7.8%)	0.697
Hospitalized	3714 (48.0%)	395 (48.9%)	3062 (47.6%)	0.519
Infections	1465 (18.9%)	214 (26.5%)	1149 (17.9%)	<0.001
Listed for LT	854 (11.0%)	109 (13.5%)	612 (9.5%)	<0.001
Medications
Beta-Blocker	2614 (33.8%)	246 (30.4%)	2159 (33.6%)	0.082
Diuretics	4063 (52.5%)	380 (47.0%)	3416 (53.1%)	0.001
Lactulose	3389 (43.8%)	482 (59.7%)	2619 (40.7%)	<0.001
Rifaximin	1959 (25.3%)	312 (38.6%)	1454 (22.6%)	<0.001
SBPPr	1062 (13.7%)	147 (18.2%)	832 (12.9%)	<0.001
Statins	763 (9.9%)	69 (8.5%)	651 (10.1%)	0.175
PPI	3366 (43.5%)	358 (44.3%)	2766 (43.0%)	0.507
HBV antivirals	1287 (16.6%)	88 (10.9%)	1147 (17.8%)	<0.001
Admission reasons
Infections	1651 (21.4%)	324 (40.1%)	1242 (19.3%)	<0.001
Anasarca	2484 (32.1%)	260 (32.2%)	2112 (32.8%)	0.735
HE	2104 (27.2%)	437 (54.1%)	1533 (23.8%)	<0.001
GI Bleed	1827 (23.6%)	196 (24.3%)	1534 (23.4%)	0.834
AKI	1558 (20.1%)	420 (52.0%)	1005 (15.6%)	<0.001
Electrolyte changes	1490 (19.3%)	300 (37.1%)	1099 (17.1%)	<0.001
HBV flare	376 (4.9%)	20 (2.5%)	339 (5.3%)	<0.001
Liver/infection unrelated	316 (4.1%)	17 (2.1%)	279 (4.3%)	0.003
Admission labs (median, IQR)
Hemoglobin (g/dl)	10.0 [8.2–11.9]	9.6 [7.7–11.5]	10.1 [8.2–11.9]	<0.001
WBC (106/ml)	6.10 [3.83–9.50]	9.20 [5.90–14.16]	5.85 [3.70–8.94]	<0.001
Albumin (g/dl)	2.86 [2.40–3.30]	2.55 [2.19–3.00]	2.90 [2.50–3.30]	<0.001
MELD-Na	25 [18–31]	31 [25–36]	24 [17–30]	<0.001

6M: six months, PPI: proton pump inhibitors, AKI: acute kidney injury, HE: hepatic encephalopathy, GI: gastrointestinal, MELD-Na: model for end-stage liver disease-sodium, MASLD: metabolic dysfunction related steatotic liver disease, WBC: white blood count, HBV: hepatitis B virus, HIC: high income countries, UMIC: upper middle-income countries, L/LMICs: low/low middle-income countries, CLEARED: Chronic Liver Disease Evolution And Registry for Events and Decompensation. Comparisons were performed using unpaired t-tests or Mann-Whitney U tests or Chi-square tests as appropriate.

The inpatient outcomes were more severe in those who died as expected (Table 2). 1407 (18.1%) were transferred to the ICU for seven common reason(s) (gastrointestinal bleeding, altered mental status, respiratory failure, sepsis, electrolyte imbalances, circulatory failure, and other non-liver conditions). Of these, the most common reasons were altered mental status, gastrointestinal bleeding, or sepsis. 11% developed grade 3–4 HE, and a similar proportion of patients needed vasopressors and mechanical ventilation. 31% developed AKI and 4% needed dialysis as an inpatient. A similar analysis was conducted and resulted in largely similar results within country income levels (Tables S2–4).

Table 2:

Inpatient course for CLEARED

	Total cohort n = 7733)	Inpatient mortality
		Died (n = 808, 11.1%)	Survived (n = 6431, 88.8%)	P value
Developed AKI?	2428 (31.4%)	605 (74.9%)	1646 (25.6%)	<0.001
Peak Creatinine	1.04 [0.75–1.81]	2.60 [1.58–3.97]	0.97 [0.72–1.50]	<0.001
Number of AKI episodes	0 [0–1]	1 [0–1]	0 [0–0]	<0.001
Need for dialysis	329 (4.3%)	153 (18.9%)	133 (2.1%)	<0.001
Need for mechanical ventilation	668 (8.6%)	371 (45.9%)	200 (3.1%)	<0.001
Need for vasopressors	839 (10.8%)	457 (56.6%)	333 (5.2%)	<0.001
HE Grade 3 or 4	884 (11.4%)	396 (49.0%)	457 (7.1%)	<0.001
Nosocomial infections	846 (10.9%)	263 (32.5%)	565 (8.2%)	<0.001
Second infections	196 (2.5%)	84 (10.4%)	96 (1.5%)	<0.001
Need for ICU?	1407 (18.2%)	459 (56.8%)	780 (12.1%)	<0.001
Reasons for ICU
Sepsis	439 (5.7%)	251 (31.0%)	164 (2.6%)	<0.001
Bleeding	474 (6.1%)	128 (15.8%)	317 (4.9%)	<0.001
Mental status	645 (8.3%)	308 (38.1%)	295 (4.6%)	<0.001
Other	380 (4.9%)	71 (8.8%)	207 (3.2%)	<0.001
Length of stay
ICU LOS	5 [3–9]	6 [3–12]	4 [2–7]	<0.001
Hospital LOS	9 [5–15]	11 [6–21]	8 [5–14]	<0.001

AKI: acute kidney injury, LOS: length of stay, ICU: intensive care unit, HE: hepatic encephalopathy, CLEARED: Chronic Liver Disease Evolution And Registry for Events and Decompensation. Comparisons were performed using unpaired t-tests or Mann-Whitney U tests or Chi-square tests as appropriate.

Key characteristics were compared in a univariable setting among the outcome groups in the entire dataset; those who died were more likely to have an alcohol etiology, less likely to have a viral etiology of cirrhosis, more likely to have prior complications such as ascites, HE/AKI, and prior hospitalizations/infection. If a patient’s primary reason for admission was HE, infection, AKI, or electrolyte imbalance, they were more likely to die inpatient compared to those with other reasons for admission. Patients on admission HE therapies, and spontaneous bacterial peritonitis prophylaxis (SBPPr) were more likely to die, while the opposite trend was seen with HBV antivirals and diuretics. Statins, beta-blockers, & PPI did not affect these outcomes.

Machine-Learning Analysis

Inpatient mortality:

For assessment of inpatient mortality, random forest analysis (RFA) achieved an AUC of 0.815 [0.785–0.844], versus 0.773 [0.738–0.807] achieved from traditional logistic regression (Table 4, Fig. 2A), which represented a 4.2-point gain in discriminative performance (p<0.001). XGBoost also performed better than LR (0.786 [0.757–0.824], 1.3-point gain), although the difference was smaller and not statistically significant. LASSO logistic regression selected 7 predictors which together achieved an AUC of 0.787 [0.755–0.819], which was 2.8-points lower than RF (p=0.004). These predictors were admission AKI, admission HE, admission infection, high income facility (vs. upper middle or low), admission MELD-Na, admission albumin, and admission WBC.

Table 4:

Machine Learning Results on Inpatient Mortality using the entire CLEARED Consortium and individual country income level groups.

Outcome	Predictors Included	Test Set AUC
Entire CLEARED cohort
Logistic Regression	69	0.773 [0.738–0.807]
LASSO	7	0.787 [0.755–0.819]
Random Forest*	69	0.815 [0.785–0.844]
XGBoost**	69	0.786 [0.757–0.824]
High Income Countries
Logistic Regression	67	0.755 [0.682–0.828]
LASSO	2	0.775 [0.700–0.850]
Random Forest*	67	0.806 [0.742–0.870]
XGBoost**	67	0.787 [0.714–0.860]
Upper middle-income countries
Logistic Regression	67	0.837 [0.782–0.891]
LASSO	8	0.867 [0.828–0.906]
Random Forest*	67	0.867 [0.823–0.911]
XGBoost	67	0.858 [0.814–0.901]
Low/low middle-income countries
Logistic Regression	67	0.717 [0.642–0.792]
LASSO	4	0.709 [0.641–0.777]
Random Forest*	67	0.768 [0.705–0.832]
XGBoost**	67	0.726 [0.657–0.795]

^*:best performing model,

^**Hyperparameter-tuned model used over default.

AUC: area under the curve, CLEARED: Chronic Liver Disease Evolution And Registry for Events and Decompensation.

Figure 2: ROC Curves and Top 15 variables on Random Forest for inpatient mortality in the entire CLEARED Consortium.

A: Comparison of receiver operating characteristic curve for the entire CLEARED consortium. The highest area under the curve was for Random Forest (Green, 0.815) followed by XGBoost (Black, 0.786), LASSO (Red, 0.787) and the lowest was for logistic regression (Blue, 0.773). CLEARED: Chronic Liver disease Evolution And Registry for Events and Decompensation B: Top 15 variables of highest importance in prediction of inpatient mortality, Red bars: variables with positive predictive contribution towards mortality, Blue bars: variables with negative predictive contribution towards mortality. HE: hepatic encephalopathy, AKI: acute kidney injury, GI: gastrointestinal, WBC: white blood cell count, MELD-Na: model for end-stage liver disease sodium, HBV: hepatitis B virus, antiviral use is for hepatitis B.

Of the top-15 important variables selected from RF (Figure 2B), Admission for AKI, HE, high MELD-Na/WBC and not being in high income country were variables most predictive of mortality, while higher albumin, hemoglobin, diuretic use on admission, viral etiology, and being in a high-income country were most protective. On subgroup analysis, these improvements from RFA over LR persisted (High-income only, 5.1-point gain using RF vs. LR, Upper-Middle Income: 3.0-point gain, Low-Income: 5.1-point gain Table 4).

Similar top 15 variables and ML results were seen for individual country income levels were consistent (Table 4, figures S1–S3).

Re-fitting of the final Random Forest model using only the top-15 predictors obtained an AUC of 0.806 [0.776–0.837], which was 0.9-points lower than the model containing all predictors and still stronger than XGB, LR, and LASSO models containing all predictors. Compared to MELD-Na alone (AUC: 0.697 [0.660–0.734], p-value for comparison: <0.001) and MELD-3.0 (AUC: 0.707 [0.669–0.744], p-value for comparison: <0.001), our model had stronger discriminative ability. On subgroup analysis of cirrhosis etiology and HE/infection on admission, we found that the model performed slightly stronger on patients with non-alcoholic cirrhosis etiology and infection on admission. Performance across those with/without HE on admission was comparable (Table S5).

Calibration curves of the final RF model on the testing set showed that the predicted probabilities of inpatient mortality were well-aligned with the actual event rates (Figure S4/5). Specifically, the full-model calibration slope/intercept was 1.17/0.253 with a Brier score of 0.086. Using only the top-15 predictors, the calibration slope/intercept was 0.970/−0.064 with a Brier score of 0.088.

On analysis of specific probability cutoffs, a predicted mortality risk of >10% had a sensitivity of 78.8%, specificity of 65.2%, PPV of 23.6%, and NPV of 95.8%. A predicted mortality risk of >25% had a sensitivity of 49.3%, specificity of 89.6%, PPV of 39.3%, and NPV of 92.8%. Finally, predicted mortality risk of >50% had a sensitivity of 14.3%, specificity of 98.9%, PPV of 63.3%, and NPV of 89.4%. This implies that patients with a predicted in-hospital mortality probability of <10% survived their inpatient admission course ~95% of the time and were low-risk, while those with >50% predicted in-hospital mortality were high-risk, dying in-hospital nearly two-thirds of the time. For example, a patient admitted from an upper-middle income facility with AKI, HE, and infection on admission, no diuretic use, WBC count of 7, platelet count of 50, albumin of 2 mg/dL, hemoglobin of 11 g/dL, MELD-Na score of 28, and no prior AKI, HE, or Hepatitis B history would have a predicted in-hospital mortality of 52.11%. Alternatively, if this patient was only admitted for infection, their risk of mortality would drop to 12.45%. Finally, if the same patient was admitted for reasons other than infection/HE/AKI/electrolyte imbalance, had a MELD-Na score of 15 (rather than 28), and were on diuretics at admission, their overall mortality risk would drop to 2.41%.

Validation in a National Cohort of Veterans

We collected admission information on 29,010 Veterans with cirrhosis between 2020 and 2023 (Figure 1). There is negligible overlap between cohorts since only one VA site with 100 patients was included in CLEARED. After excluding the 340 who received inpatient transplant, the remaining 28,670 patients were further studied (Figure 1b). The inpatient mortality rate was 4.0%, and patients were primarily male and above the age of 65 (Table 3). Those who died were more likely to have prior cirrhosis complications (HE, ascites, etc.), more severe cirrhosis disease progression (higher MELD-Na, lower albumin, etc.), more likely to be admitted for HE, AKI and electrolyte imbalance but were less likely to be on admission medications related to these complications (Table 4). Statin use was lower in those who died as well. The final random forest model, using 48 out of the 67 original covariates, attained a strong AUC of 0.859 (95% CI: [0.849–0.869]. The random forest model which was re-fit using only the top-15 variables attained a comparable AUC of 0.851 [0.840–0.862]. On subgroup analysis of cirrhosis etiology and HE/infection on admission, we found that the model performed slightly stronger on patients with no HE/infection, with comparable performance across cirrhosis etiology types (Table S5). Probabilities were well-calibrated, with a slope/intercept of 1.37/−0.05 with a Brier score of 0.036 (Figure S6). Compared to MELD-Na alone (AUC: 0.740 [0.725–0.755], p-value for comparison: <0.001) and MELD-3.0 (AUC: 0.804 [0.792–0.816], p-value for comparison: <0.001), our model was again superior in this external validation cohort. At the 10% probability threshold, NPV was high, with 98.8% of patients under this threshold surviving their hospital course. At the 50% threshold, PPV was lower than CLEARED, with 47.0% of patients over this risk threshold dying in-hospital, but this was expected due to much lower event rates within VA-CDW (4% vs. 11%).

Table 3:

CDW-VA Validation Cohort, Characteristics

	Overall (n = 28760)	Died in hospital (n = 1158, 4.0%)	Survived (n = 27,602, 96.0%)	P value
Demographics
Age	67.81 (10.25)	69.56 (10.46)	67.73 (10.23)	<0.001
Male Sex	27688 (96.3%)	1114 (96.2%)	26574 (96.3%)	0.958
Cause of Cirrhosis
Alcohol Use	10361 (36.0%)	386 (33.3%)	9975 (36.1%)	0.055
MASLD	2233 (7.8%)	67 (5.8%)	2166 (7.8%)	0.012
Hepatitis B	627 (2.2%)	30 (2.6%)	597 (2.2%)	0.382
Hepatitis C	4843 (16.8%)	134 (11.6%)	4709 (17.1%)	<0.001
Autoimmune Hepatitis	124 (0.4%)	7 (0.6%)	117 (0.4%)	0.490
Cryptogenic	3373 (11.7%)	105 (9.1%)	3268 (11.8%)	0.005
Comorbidities
Diabetes	11333 (39.4%)	454 (39.2%)	10879 (39.4%)	0.911
Hypertension	16538 (57.5%)	721 (62.3%)	15817 (57.3%)	<0.001
Hyperlipidemia	9816 (34.1%)	451 (38.9%)	9365 (33.9%)	<0.001
Cirrhosis last 6M
Ascites	2333 (8.1%)	137 (11.8%)	2196 (8.0%)	<0.001
Variceal Bleed	664 (2.3%)	31 (2.7%)	633 (2.3%)	0.452
Overt HE (any grade)	782 (2.7%)	46 (4.0%)	736 (2.7%)	0.01
Hyponatremia	1546 (5.4%)	109 (9.4%)	1437 (5.2%)	<0.001
AKI	3010 (10.5%)	269 (23.2%)	2741 (9.9%)	<0.001
Hydrothorax	69 (0.02%)	11 (0.9%)	58 (0.2%)	<0.001
Hospitalized	5826 (20.3%)	355 (30.7%)	5471 (19.8%)	<0.001
Infections	4885 (17.0%)	336 (29.0%)	4549 (16.5%)	<0.001
Medications
Beta-Blocker	8146 (28.3%)	194 (16.8%)	7952 (28.8%)	<0.001
Diuretics	9766 (34.0%)	282 (24.4%)	9484 (34.4%)	<0.001
Lactulose	3441 (12.0%)	62 (5.4%)	3379 (12.2%)	<0.001
Rifaximin	1382 (4.8%)	23 (2.0%)	1359 (4.9%)	<0.001
SBPPr	3431 (11.9%)	67 (5.8%)	3364 (12.2%)	<0.001
Statins	5838 (20.3%)	144 (12.4%)	5694 (20.6%)	<0.001
PPI	8590 (30.0%)	169 (14.6%)	8421 (30.5%)	<0.001
HBV antivirals	47 (0.2%)	2 (0.2%)	45 (0.2%)	>0.99
Admission reasons
Infections	6254 (21.7%)	710 (61.3%)	6511 (23.6%)	<0.001
Anasarca	316 (1.1%)	13 (1.1%)	303 (1.1%)	>0.99
HE	3323 (11.6%)	352 (30.4%)	2971 (10.8%)	<0.001
GI Bleed	1083 (3.8%)	59 (5.1%)	1024 (3.7%)	0.019
AKI	8229 (28.6%)	857 (74.0%)	7372 (26.7%)	<0.001
Electrolyte changes	4521 (15.7%)	340 (29.4%)	4181 (15.1%)	<0.001
Admission labs (median, IQR)
Hemoglobin (g/dl)	11.10 [9.20–12.90]	10.00 [8.40–11.90]	11.20 [9.20–12.90]	<0.001
WBC (106/ml)	6.63 [4.80–9.30]	9.91 [6.81–15.04]	6.55 [4.72–9.10]	<0.001
Platelets(106/ml)	155.00 [104.00–220.00]	159.50 [98.00–235.00]	155.00 [104.00–220.00]	0.260
Albumin (g/dl)	3.30 [2.80–3.80]	2.70 [2.30–3.20]	3.40 [2.80–3.80]	<0.001
MELD-Na	15 [11–21]	23 [17–30]	15 [10–21]	<0.001

PPI: proton pump inhibitors, AKI: acute kidney injury, HE: hepatic encephalopathy, GI: gastrointestinal, MELD-Na: model for end-stage liver disease-sodium, MASLD: metabolic dysfunction related steatotic liver disease, WBC: white blood count, HBV: hepatitis B virus, VA-CDW: Veterans affairs Corporate Data Warehouse.

Discussion:

In this large prospective inpatient cirrhosis cohort from 121 centers worldwide, we found that machine learning models, especially random forest analysis, outperformed traditional logistic regression in prediction of inpatient mortality. This pattern was similar across the major country income types although specific contributors towards these outcomes differed slightly. The random forest model was externally validated in a national cohort of Veterans with cirrhosis with strong AUC for inpatient mortality, even when simplified and streamlined using only the top-15 most predictive variables. Predicted probabilities from this ML model were effectively used to separate patients into high and low-risk groups, which highlights the potential clinical applicability in hospital settings.

Predicting mortality based on admission criteria in hospitalized patients with cirrhosis is daunting given the multiple and varied influences of outpatient management, inpatient course, availability of medications and interventions such as liver transplant³. Machine learning models have been described in this context but either in regional or local centers, or those that are limited to a priori ACLF definitions or specific etiologies^{8, 13}. Moreover, some of these models include other formulae whose constituents are either not checked in most patients or make these models very complicated. Lastly, a global perspective unencumbered from a priori definitions that considers income levels that reduces in-built biases is needed to create models with the maximum benefit^{1, 33}. Our data collected from all 6 continents represent a diverse and global contribution towards better understanding of chronic liver disease and cirrhosis outcomes.

The results consistently showed an improvement over traditional logistic regression, which is a major step forward towards refining prediction algorithms. These patterns remained consistent even when performed within the income level, further confirming the generalizability of these models. Moreover, the individual granular level of the data collected adds to the strength of the results. While the variables that contribute towards outcomes are not novel individually, their combination in the prediction of outcomes using ML techniques is. Replicating prior logistic regression analyses, we found that being in a HIC or UMIC versus LMIC is associated with a better inpatient mortality³. However, there are specific variables not included in traditional cross-sectional prognostication schemes such as MELD-Na that were important in predicting outcomes². Specifically, the contributions of HE (prior and reason for admission), inflammation and infection (WBC count, infection on admission), degree of portal hypertension (platelet count), liver function beyond MELD-Na (albumin levels), and AKI (prior history, reason for admission, electrolyte disturbances) were important and additive to MELD-Na^{2, 34}. The unique nature of the data collection was inclusion of past events and not just what was presented on admission, which adds to the real-world translation of these results. For example, even though creatinine is an integral part of MELD-Na, the admission reason being AKI, and AKI history were additional variables of importance. Similarly, admission for HE and prior HE, that also imply inclusion of pre-admission historical data were independently relevant^{35, 36}. Interestingly, diuretic use, higher hemoglobin, and prior HBV were protective. Higher hemoglobin has been shown to be good prognosticator after discharge, while it is likely that hepatitis B infected patients on antivirals were suppressed and had lower liver inflammation³⁷. Use of diuretics signifies controlled ascites, and by extension likely a priori contact with healthcare staff and better management of other outpatient cirrhosis conditions compared to the rest³⁸.

Another strength is the validation of results from the CLEARED model into an unrelated population of US-based Veterans with cirrhosis. These patients tend to be more older men who are predominantly Caucasian, and have more co-morbid conditions compared to the general US population^{17, 39}. As expected in cirrhosis, more patients admitted with liver-related and infection complications died in the VA and statin use was associated with lower mortality⁴⁰. However, the rates of HE medication use on admission was lower and associated with lower risk of inpatient mortality. This is likely because for the majority, this was their first HE episode and were likely not on these medications on admission. Moreover, unlike CLEARED, this was a database analysis and therefore case-level granularity was not possible, and certain variables available in CLEARED were not available in VA-CDW. However, despite these differences, results overall showed a good AUC (>0.85) for inpatient mortality prediction using the random forest model we studied. Prospective implementation of these results is important to improve prognostication. The link to the final Random Forest model, which allows new patient data to be entered and interpreted is now freely available at https://silveys.shinyapps.io/app_cleared/.

Although the improvement in AUC over logistic regression was statistically significant, these seem relatively minor improvements: 4.2-point improvement for inpatient mortality. However, given the complexity of these data points and difficulty in predicting these outcomes, these gains may be important in improving prognostication and determining futility. Moreover, the real-world representation of the data that includes all inpatients with cirrhosis rather than focusing on a priori restrictions related to organ failures or other definitions is a strength.

The data are limited by the low number of patients per center in CLEARED that could increase the possibility of a selection bias and reduce the generalizability of these results across larger health systems. However, the demographic and etiology profile largely reflected prior publications that included larger numbers per site. We are also limited by using a database in the case of VA-CDW. Although we were able to train and test our ML models effectively using >7200 patients, we were not able to assess more complex deep learning (DL) architectures, which may require much larger sample sizes to fit and tune a reliable model than standard ML methods^{41, 42}. Incorporating deep learning in the future could enhance cirrhosis mortality prediction even further. The codes used in VA-CDW are validated, while CLEARED represents a major swathe of tertiary care centers across the world with varying resources, disease etiologies, and availability of therapies^43–45. We relied on investigators in CLEARED to correctly identify and record prior and current cirrhosis complications, however, we gave all sites specific definitions for each cirrhosis outcome and complication and sites were led by experienced personnel. We did not focus on cause of death because it is often difficult to pinpoint one proximate factor in the complex inpatient settings. For the modelling of mortality, future research could aim to analyze the cause of death, and temporal distribution of death, namely prediction of early vs. late events, which could further improve prognostication and inform clinical decision-making. Finally, patients who received inpatient liver transplant were excluded, although overall LT rates were low (2.3% in CLEARED, 1.2% in VA-CDW). Modelling of transplant-free mortality only allowed us to develop a risk score that does not consider this competing risk of LT, which may be especially important for those who cannot be transplanted due to lack of resources. However, exclusion of LT patients could also cause selection bias, and future research could incorporate both endpoints into predictions.

We conclude that a machine learning model using random forest analysis in a prospectively enrolled global cohort of hospitalized patients with cirrhosis showed superiority over traditional logistic regression for prediction of inpatient mortality, which was externally validated in a US-based national cohort of Veterans. This machine learning model that has an equitable global representation could be beneficial in rapid prognostication of patients hospitalized with cirrhosis.

WHAT YOU NEED TO KNOW:

BACKGROUND AND CONTEXT:

• Inpatients with cirrhosis have a high rate of mortality with regional and global variations.

• Traditional statistical methods to predict mortality based on day of admission variables require enhancement to improve prognostication.

• Machine learning models have been applied in regional or single-center settings but a global approach with external validation is needed

NEW FINDINGS:

• In a prospectively recruited cohort of 7,239 patients hospitalized with cirrhosis (CLEARED) from 115 centers across all 6 populated continents, we determined that a random forest machine learning model significantly enhanced inpatient mortality prediction over other models and traditional logistic regression.

• This random forest model also showed consistently good performance and calibration on internal 75/25 validation, and when the prediction was performed within low-income, upper middle, and high-income countries within the CLEARED Cohort.

• The same model was externally validated with good performance in 28,670 hospitalized US Veterans with cirrhosis, which have a distinct demographic and cirrhosis complication profile.

LIMITATIONS:

• CLEARED consortium only allows enrollment of up to a maximum of 100 patients per site.

• Use of a database (VA-corporate data warehouse) for external validation, which could reduce granularity.

• Excluding those who received inpatient transplant, however this was <2% of the cohort.

CLINICAL RESEARCH RELEVANCE

• Results from this cohort formed with equitable global representation will increase insight into cirrhosis prognosis.

• Harmonization of multiple clinical variables into a composite machine learning model that has equivalent performance across differing country income levels should focus attention on cirrhosis management rather than individual country differences.

• Better prognostication using this publicly available model will help in timely prognostication and management of inpatients based on variable available on the day of admission.

BASIC RESEARCH RELEVANCE

• The results will drive further research into analysis of newer machine learning techniques for data analysis that includes worldwide data representation.

• Coalescing data management from differing sources worldwide will improve big data analytic techniques to improve prognostication.

Data availability:

Due to ethics board restrictions, individual level data are not available.