Predicting 30 Day Hospital Readmission Risk in a National Cohort of Patients with Cirrhosis

Abstract

Background

Early hospital readmission for patients with cirrhosis continues to challenge the healthcare system. Risk stratification may help tailor resources, but existing models were designed using small, single-institution cohorts or had modest performance.

Aims

We leveraged a large clinical database from the Department of Veterans Affairs (VA) to design a readmission risk model for patients hospitalized with cirrhosis. Additionally, we analyzed potentially modifiable or unexplored readmission risk factors.

Methods

A national VA retrospective cohort of patients with a history of cirrhosis hospitalized for any reason from January 1, 2006 to November 30, 2013 was developed from 123 centers. Using 174 candidate variables within demographics, laboratory results, vital signs, medications, diagnoses and procedures, and healthcare utilization we built a 47 variable penalized logistic regression model with the outcome of all-cause 30-day readmission. We excluded patients who left against medical advice, transferred to a non-VA facility, or if the hospital length of stay was greater than thirty days. We evaluated calibration and discrimination across variable volume and compared the performance to recalibrated pre-existing risk models for readmission.

Results

We analyzed 67,749 patients and 179,298 index hospitalizations. The 30-day readmission rate was 23%. Ascites was the most common cirrhosis-related cause of index hospitalization and readmission. The AUC of the model was 0.670 compared to existing models (0.649, 0.566, 0.577). The Brier score of 0.165 showed good calibration.

Conclusions

Our model achieved better discrimination and calibration compared to existing models, even after local recalibration. Assessment of calibration by variable parsimony revealed performance improvements for increasing variable inclusion well beyond those detectable for discrimination.

BACKGROUND

Cirrhosis carries significant morbidity and mortality due to decreased mental, physical, and biochemical function. The prevalence is estimated between 400,000 and 3,000,000 persons in the United States, and the disease causes 44,000 deaths annually.^1–5 Liver disease costs the US over $2 billion annually in direct healthcare costs,^1,6 and though the exact portion attributable to cirrhosis is challenging to calculate, a significant portion is likely due to the 150,000 hospitalizations and nearly 600,000 annual outpatient visits.^7–9 Given that 69% of hospitalized cirrhotic patients will have at least one non-elective readmission,¹⁰ reducing readmission can be a significant cost saver. Additionally, early hospital readmission is seen as a marker of poor healthcare quality^11,12 and is associated with increased mortality,¹³ increased psychosocial burden on the patient and their caregivers,^14–16 and low patient satisfaction in other conditions.¹⁷

It has been estimated that 27.1% of readmissions may be avoidable.¹⁸ Though programs to reduce readmission have been met with mixed success, the more successful interventions are often a combination of multi-pronged approaches.¹⁹ In cirrhosis, interventions such as mandatory gastroenterology consultation,²⁰ early outpatient follow-up,²¹ and specialized care delivery methods such as a “day hospital”²² may all improve readmission risk. These are resource intensive interventions and as such will require risk stratification to identify high-risk patients, an approach that has shown success for heart failure.^23,24

Clinical decision support tools that risk stratify patients in support of targeted intervention require highly accurate, generalizable risk prediction models, lest misleading risk estimates lead to inappropriate treatment choices or wasting limited resources.^25–28 Models trained on small samples often suffer from overfitting, resulting in reduced performance when applied to subsequent patients treated at the same institution or patients treated in other healthcare systems.^29–31 The majority of studies in a recent systematic review assessing cirrhosis readmission risk were limited by either being smaller, single center cohorts or large administrative database analyses.³² The former lacks the power to analyze multiple risk factors and is limited by the possibility of uncounted readmissions to other centers, which may reduce generalizability. The latter studies lack the granular detail of institutional cohorts. Moreover, the majority of these papers primarily attempted to identify risk factors, as opposed to building a risk prediction model.³² Of the analyses that did create risk prediction models, they have been limited by modest performance, sample sizes, and number of candidate predictor variables (Refer to Table 1 for a summary).^13,33–37

Table 1:

Summary of existing risk prediction models for hospital readmission among patients with cirrhosis.

Model	Study Type & No. of Subjects		No. of Candidate Variables	Candidate Variable Domains												Discrimination	Calibration
	D	V		D	X	M	P	I	R	C	L	Y	U	T	S
Berman (2011)33	447	836¥	∼ 27													C=0.57¥	n/a
Bajaj (2016)35	1343		∼ 25													C=0.64	n/a
Morales (2017)13	112		∼ 34													C=0.76	HL ns
Singal (2013)34	629	209β	∼ 30													C=0.66	HL ns
Tapper (2015)37	489	245β	∼ 22													AUC=0.69	n/a
Volk (2012)36	402		∼ 22													C=0.65	n/a
Our Model	67,749		208

Note: Study Type = Validation (V) and Development (D). If a study assessed performance in a separate validation cohort, we have reported the performance in the validation cohort. The number of candidate variables had to be inferred based on the written methods and the cohort summary table as this number was not explicitly reported. Model discrimination reported as either C-statistic or area under the curve (AUC), which can be considered equivalent. Risk Variable Domains Coding: D: Demographics; X: Medical/Surgical Hx; M: Meds; P: Inpatient procedures; I: Physical impairment; R: Risk scores; C: Cirrhosis related complications; L: Labs; Y: Psychosocial; U: Healthcare Utilization; T: Transplant status; S: Discharge disposition.

^¥Validation performed in separate study (Singal)³⁴

^βValidation cohorts were created by random train-test split of the original cohort

HL: Hosmer-Lemeshow test for goodness-of-fit.

However, even models based on large datasets require continued validation and potential updating as model accuracy may decline over time in evolving clinical environments.^38–40 Changes in the mix of patient risk factors, outcome prevalence, treatment patterns, or documentation practices may harm model calibration and lead to misaligned predicted-to-observed risk.^38,41,42 Assessing, restoring, and maintaining model calibration are critical for risk prediction tools that provide clinical decision support.^28,43,44 Only two of the studies in Table 1 attempted to assess calibration, and they used the Hosmer-Lemeshow test, which has faced criticism for being an incomplete assessment of calibration.⁴⁵

Extensive clinical data warehouses generated by electronic health record systems provide new opportunities to develop risk prediction models based on larger datasets.^25,46,47 The Department of Veterans Affairs (VA), with one of the largest clinical data warehouses encompassing over 20 million patients with granular data,⁴⁸ faces an increasing burden of chronic liver disease due to substance use disorders, chronic viral hepatitis, and increasing numbers of patients with NAFLD.⁴⁹ These dramatic increases in overall prevalence of cirrhosis at the VA necessitate targeted interventions and tailored care delivery pathways, for which risk stratification is essential. The aim of the current study was to leverage a large clinical database from the Department of Veterans Affairs to design a cirrhosis readmission risk prediction model.

METHODS

Study Cohort

We analyzed a retrospective cohort of cirrhotic patients hospitalized for any cause from among 123 medical centers in the Department of Veterans Affairs (VA) between January 1, 2006 and November 30, 2013. We included historical data from January 1, 2005 to allow for variable ascertainment and the last month in 2013 to assess for readmission from index hospitalizations during November 2013. We defined a history of cirrhosis as patients who had a cirrhosis diagnosis (based on a history of two outpatient or one inpatient) ICD-9 code (571.2 or 571.5) or an ICD-9 code for a cirrhosis complication (varices, hepatic encephalopathy, hepatorenal syndrome, or portal hypertension).⁵⁰ The use of 1 inpatient or 2 outpatient codes for identification of disease is frequently used and has been validated for patients with cirrhosis.⁵¹

Because we included all-cause hospitalizations, they did not need to have a cirrhosis or cirrhosis complication administrative code during the index hospitalization. We excluded hospitalizations after liver transplant (including the transplant hospitalization itself), if the patient was discharged against medical advice, if the patients were transferred to a non-VA facility, if the patient’s age was less than eighteen, or if the hospital length of stay (LOS) was greater than thirty days. Patients with LOS greater than thirty days were excluded because they frequently represented issues with disposition rather than severity of illness, particularly at the VA where there are fewer patient-related financial incentives for rapid hospital discharge. We further limited index hospitalizations to not having another hospitalization within the 30 days prior to admission. Refer to Figure 1 for the cohort flow diagram.

Figure 1:

Flow of patients from total number of patients before exclusion criteria to total number of patients included in study.

Though we allowed for all-cause hospitalization, we identified cirrhosis related index admissions and readmissions similarly to other studies,^52,53 as hospitalizations with discharge diagnoses identifying cirrhosis or cirrhosis related complications. Specifically, we used administrative codes to identify hepatorenal syndrome, hepatic encephalopathy, ascites, spontaneous bacterial peritonitis, hepatocellular carcinoma, portal hypertension, and “Other serious complications of chronic liver disease.” If a hospitalization had none of these codes, the hospitalization was deemed not cirrhosis related. Code sets used to determine these conditions are available in the appendix. The institutional review board and research and development committees of the Tennessee Valley Health Care System VA Medical Center, Nashville, TN approved this study.

Outcome Ascertainment

We generally followed Centers for Medicare & Medicaid Services (CMS) criteria for readmission measures.⁵⁴ Our outcome was all-cause rehospitalization within a 30-day period from the date of discharge from an index admission. Per CMS criteria, planned admissions were excluded and not counted as readmission; refer to the Appendix Figure 1 for our definition of planned admissions. If a patient had multiple unplanned admissions within 30 days of the index hospitalization, only the first one was counted. However, if the first admission within 30 days of the index hospitalization was planned, then subsequent admissions within the 30-day timeframe of the index hospitalization were not treated as readmission for that index admission. We took this approach, consistent with CMS guidance, because the subsequent admissions could be a consequence of the intervening planned admission.

Data Collection

The VA is an integrated care network that includes acute inpatient hospitals, outpatient primary care and sub-specialist clinics, outpatient pharmacies, rehabilitation facilities, long-term care facilities and domicilliaries. All VA personnel use the same EHR, Veterans Information Systems and Technology Architecture/Computerized Patient Record System (ViSTa/CPRS), for documentation and administration of clinical care,²⁰ and data from all sites is stored in the Corporate Data Warehouse and provisioned by the VA Informatics and Computing Infrastructure.⁴⁸

Predictor Variables

We initially evaluated 208 variables and excluded 34 variables with > 60% missing or < 0.2% prevalence, leaving 174 candidate risk variables from demographics, labs, diagnoses/procedures, medications, vital signs, and healthcare utilization factors. We refer the reader to the appendix for a description of all candidate variables. Our final model contained 47 variables, determined by the variable selection procedure detailed below. Except for cirrhosis-related medications (e.g., lactulose), which were coded as separate variables, all medications were represented by their corresponding VA drug class code (e.g., “cephalosporin 3^rd generation”). The VA drug class codes are available publicly through the VA National Drug File and the Unified Medical Language System.⁵⁶ Most variables were assessed at the time of discharge. Laboratory values and vital signs were assessed as the last available value prior to the discharge date and time. Medications were represented as medication classes, and both home medications and medications administered during the index hospitalization were included. Variables representing risk scores (e.g., FIB-4) were calculated from discharge lab values. Missing values for laboratory tests and vital signs were imputed using a matrix completion approach based on non-negative matrix factorization⁵⁷ using the R package NNLM.⁵⁸

Model Development

In order to obtain a more parsimonious model, we used the 174 candidate risk variables noted above in a variable selection procedure. We performed a penalized logistic regression, using the L₁ penalty (Least Absolute Shrinkage and Selection Operator — LASSO), to select a subset of the predictor variables.⁵⁹ LASSO is a form of linear regression designed to both address overfitting and provide variable selection. The LASSO approach shrinks coefficients toward the null value of 0 based on a penalty parameter, lambda, which controls the sum of the model’s coefficients. This coefficient “shrinkage” reduces the risk of overfitting the model to the particulars of the training data. Variable selection is achieved when the coefficients of weakly predictive variables are reduced to 0 and the influence of these variables is eliminated from the model. Penalized regression with LASSO is frequently superior to other common variable selection methods such as forward/backward selection in many biomedical datasets.⁶⁰ Refer to Figure 2 for a description of our overall model development workflow.

Figure 2: Description of overall model development workflow.

Our variable selection pipeline considers 208 distinct variables in our EHR and eliminates variables with very low prevalence or excessive missingness (Step 1, highlighted in figure). Subsequently we used penalized logistic regression (via LASSO) to perform variable selection (Step 2). LASSO creates separate models with increasing numbers of variables starting with the most important to least important. We selected the LASSO model that sufficiently reduced classification error while still producing a parsimonious model (Step 4). The literature models (Bajaj, Singal, and Berman) were then tailored to the VA data via logistic re-calibration (Step 5) and we compared the VAtailored version of those models with ours (Step 6).

We considered two LASSO models, one generating the lowest error and a second more heavily regularized (i.e., fewer number of variables) model yielding deviance one standard error from the former model. Since the latter afforded a more parsimonious model (174 vs. 47 variables, respectively) with similar predictive performance (AUC difference of 0.01), we selected the second model for risk prediction. This model contained 47 variables and included: healthcare utilization (6 variables), history of cirrhosis complications (4), comorbidities (4), laboratory tests (9), inpatient administered (13) and home (5) medications, and miscellaneous variables (6). A summary of the selected variables with their percent missingness is available in the Appendix Table 5. We subsequently used the 47 variables identified by the LASSO procedure in an unpenalized logistic regression model with 30-day readmission as the outcome.

Model Evaluation

We assessed overall discrimination using the Area Under the receiver operator characteristic Curve (AUC) ⁶¹ and assessed model accuracy using the Brier score, for which 0 implies perfect accuracy.⁶¹ We estimated optimism using 200 bootstrap samples and calculated confidence intervals for performance metrics.⁶² Additionally, we graphically analyzed calibration using smoothed calibration belts, which highlight the relationship between observed and predicted probabilities across the range of predictions.^63,64

Variable selection requires a tradeoff between performance and parsimony. In order to assess this, we evaluated the discrimination and calibration performance of our model across a spectrum of regularization/shrinkage under varying values of the penalty parameter, lambda, within the LASSO. For models with each variable set, we assessed model performance as a function of model complexity (as measured by the number of model variables) based on the AUC and the proportion of observations falling within calibrated ranges of the calibration belts.

Model Comparison

We compared our model against VA-data updated versions of models published by Singal,³⁴ Berman,³³ and Bajaj.³⁵ Re-calibrating a risk prediction model tailors the model to a new cohort, attempting to resolve loss of model performance due to changes in event rates and case-mix.⁴² We tailored the three published models to VA data by constructing new, separate logistic regression models using the original model components. We report the AUC and Brier scores for these VA-tailored versions of the literature models using similar procedures as discussed above. As Bajaj offers a model for both the admission and discharge timeframe, we chose to evaluate the discharge timeframe model. We defined statistical significance as non-crossing of the 95% bootstrapped confidence intervals.

Additionally, we used the outcome-specific Net Reclassification Index (NRI) to analyze our model’s improvement over existing models for two use cases: (a) identifying patients at very low risk of readmission, < 10%; and (b) finding very high risk patients, > 40% risk of readmission.^65,66 The NRI compares our primary model against each established model with values > 0 indicating improved prediction performance. The outcome specific NRI can be interpreted as the change in true positive rate for predicting readmission, or conversely the true negative rate for predicting no-readmission. All statistical analyses were performed using the R statistical programming suite, version 3.3.2.

RESULTS

Study Population

After applying inclusion and exclusion criteria, 67,749 patients were included in the study with a total of 179,298 all-cause index hospitalizations. There were 41,134 readmissions within 30 days for an unadjusted readmission rate of 23%. Males represented 97.6% of the total admissions, with an age of 60.6 ± 9.0 (mean ± SD). Caucasian patients accounted for the majority of hospital admissions (73.6%). The etiology of cirrhosis was mainly alcoholic (29.4%), viral hepatitis (11.8%), or alcoholic and viral (33.2%). In the remaining patients, the causes of cirrhosis were NAFLD (24,066, 13.4%), primary biliary cirrhosis (649, 0.4%), autoimmune hepatitis (222, 0.1%), hemochromatosis (733, 0.4%) and other/cryptogenic (20,302, 11.3%). The average MELD score at index hospitalization was 13.3 ± 5.4 with 30,907 hospitalizations (17.2%) having a MELD score ≥ 18. Refer to Table 2 for a description of the cohort.

Table 2:

Demographic, clinical and laboratory variables of included patients at first admission.

Variables	All index admissions (n=179,298 admission, 67,749 patients)
Age, mean (SD)	60.6 (9.0)
Gender (male), n (%)	175,068 (97.6)
Race, n (%)
White	132,015 (73.6)
Black	32,919 (18.4)
Asian-Hawaiian-Pacific Islander	2,885 (1.6)
American Indian-Alaskan Native	2,885 (1.6)
Unknown	8,594 (4.8)
Cirrhosis Etiology, n (%)
Alcoholic	52,736 (29.4)
Viral (Hep B and C)	21,144 (11.8)
Alcoholic and Viral	259,446 (33.2)
NAFLD	24,066 (13.4)
Hemochromatosis	733 (0.4)
Autoimmune hepatitis	222 (0.1)
Biliary Cirrhosis	649 (0.4)
Other/Cryptogenic	20,302 (11.3)
Healthcare Utilization (past 1 year), median (IQR)
ER Visits	2 (0, 4)
Inpatient Hospitalizations	2 (1,4)
Outpatient Visits	31 (16, 55)
Non-face-to-face Communication	4 (1, 10)
Congestive Heart Failure, n (%)	43,382 (24)
Diabetes Mellitus, n (%)	76.8 (42.8)
h/o Cirrhosis Complications, n (%)
Hepatic Encephalopathy	41,511 (23.1)
Varices	39,653 (22.1)
SBP	10,019 (5.6)
Ascites	63,941 (35.7)
Hepatocellular Carcinoma	18,940 (10.6)
Hepatorenal Syndrome	3,897 (2.1)
Other Sequelae of Chronic Liver Disease	17,599 (9.8)
Vitals, mean (SD)
Systolic Blood Pressure	124.1 (19.5)
Diastolic Blood Pressure	72.3 (12.2)
Labs, median (IQR)
Creatinine	0.9 (0.8, 1.2)
Blood Urea Nitrogen	15.0 (10.0, 21.6)
Sodium	137.0 (134.0, 139.0)
Potassium	4.0 (3.7, 4.3)
Total Bilirubin	1.3 (0.7, 2.3)
Albumin	2.9 (2.5, 3.4)
INR	1.3 (1.1, 1.5)
White Blood Cell	6.1 (4.4, 8.1)
Platelets	125.0 (79.0, 190.0)
Alanine aminotransferase (ALT)	33.0 (19.0, 60.0)
Aspartate aminotransferase (AST)	45.0 (26.0, 83.0)
Risk Scores
MELD, mean (SD)	13.3 (5.4)
MELD < 12, n (%)	87,912 (49.0)
MELD >= 12 and < 18, n (%)	60,479 (33.7)
MELD >= 18, n (%)	30,907 (17.2)
Disposition, n (%)
Home	164,646 (91.8)
Hospice	123 (0.1)
Hospital	1,839 (1.0)
Nursing Home	11,778 (6.6)
Other House	129 (0.1)
Unknown	776 (0.4)

SBP: Spontaneous Bacterial Peritonitis

Details of Index Admission and Readmissions

Approximately 62% and 60% of index hospitalizations and readmissions were cirrhosis related, respectively (Refer to Table 3). Ascites was the most common cause of cirrhosis related index hospitalization (32686, 18.23%) and readmission (9822, 23.88%). Hepatic Encephalopathy was associated with 10% of index hospitalizations and 15% of readmissions. Both hepatic encephalopathy and ascites had a significant increase as causes of readmission as opposed to index hospitalization. Relationships between index hospitalization diagnoses and readmission diagnoses are displayed in the flow model (Refer to Figure 3).

Table 3:

Cirrhosis related index hospitalizations and readmissions.

Condition	Index Hospitalization (N = 179,298)	Readmission (N = 41,134)
Ascites, n (%)	32,686 (18.23)	9,822 (23.88)
HCC, n (%)	12,751 (7.11)	2,589 (6.29)
HE, n (%)	17,895 (9.98)	6,305 (15.33)
HRS, n (%)	1,466 (0.82)	917 (2.23)
Portal Hypertension, n (%)	18,462 (10.30)	3,959 (9.62)
SBP, n (%)	3,874 (2.16)	1,226 (2.98)
Varices, n (%)	12,194 (6.80)	2,838 (6.90)
Other sequelae of chronic liver disease, n(%)	5,479 (3.06)	2,037 (4.95)
Possibly Cirrhosis Related,¥ n (%)	42,327 (23.61)	6,904 (16.78)
Total
Cirrhosis Related Including “Cirrhosis” ICD code, n (%)	111,053 (61.94)	24,850 (60.41)
Cirrhosis Related Excluding “Cirrhosis” ICD code, n (%)	68,726 (38.33)	17,946 (43.62)

Note: Conditions were identified using ICD codes (Refer to Appendix for relevant definitions). A single hospitalization could include multiple conditions among the discharge diagnoses.

^¥This refers to the ICD codes for Cirrhosis itself. Because the codes for cirrhosis itself, i.e. 571.2 and 571.5, may be less specific, we have identified possibly cirrhosis related admissions as hospitalizations that included “Cirrhosis” in the discharge diagnoses but no codes for the specific complications. Additionally, we calculated the total cirrhosis related admissions with and without these “Cirrhosis” codes.

Figure 3: Index hospitalization reason and readmission hospitalization reason.

For purposes of clarity, diagnoses of “Portal Hypertension” and “Other sequelae of chronic liver disease” were combined into “Other Complications.” “Possibly Cirrhosis Related” hospitalizations refer to hospitalizations using the ICD code for cirrhosis only, which may be non-specific. HE: Hepatic Encephalopathy; HCC: Hepatocellular Carcinoma; SBP: Spontaneous Bacterial Peritonitis; HRS: Hepatorenal Syndrome.

Model for 30-Day Readmission

The model’s discrimination produced an AUC of 0.670 (95% CI: 0.666 – 0.674). The calibration for our model as assessed by the Brier score was excellent 0.165 (95% CI: 0.163 – 0.166). Table 4 lists the odds ratios from the logistic regression model. In general variables reflecting poor psychosocial status, increased healthcare utilization, higher disease severity, and decompensation all were associated with higher risk of readmission. Psychosocial risk factors identified included Medicaid insurance (OR: 1.21, 95% CI: 1.09 – 1.34), drug abuse (OR: 1.07, 95% CI: 1.04 – 1.10), and psychiatric disturbance (OR: 1.11, 95% CI: 1.08 – 1.15). Among healthcare utilization predictors, number of inpatient visits, outpatient visits, and ER visits, primary care provider visits all were associated with increased risk of readmission. If the patient had surgical procedure during admission, the patient has significant lower chance of readmission (OR: 0.77, 95% CI: 0.73 – 0.80). Laboratory values suggesting worse liver disease were associated with higher risk of readmission, e.g. every 1 g/dl increase in serum albumin decreased readmission risk by 9% (OR: 0.91, 95% CI: 0.89 – 0.93).

Table 4:

Odds ratios from the logistic regression model predicting 30-day readmission for patients hospitalized with cirrhosis.

	Beta (SE)	OR (95% CI)	p value
Healthcare Utilization
Cumulative LOS	0.002 (0.000)	1.00 (1.00, 1.00)	< 0.001
Total Inpatient Visits	0.132 (0.003)	1.14 (1.13, 1.15)	< 0.001
Total Outpatient Visits	0.001 (0.000)	1.00 (1.00, 1.00)	< 0.001
Total ER Visits	0.006 (0.001)	1.01 (1.00, 1.01)	< 0.001
PCP Visits	0.000 (0.000)	1.00 (1.00, 1.00)	0.631
# of Types of PCP Visits	0.002 (0.001)	1.00 (1.00, 1.00)	0.024
h/o Cirrhosis Complications
Ascites	0.027 (0.017)	1.03 (0.99, 1.06)	0.104
Hepatic Encephalopathy	0.024 (0.016)	1.02 (0.99, 1.06)	0.146
HCC	−0.161 (0.020)	0.85 (0.82, 0.89)	< 0.001
Paracentesis	0.185 (0.019)	1.20 (1.16, 1.25)	< 0.001
Comorbidities¥
Hypothyroidism	0.057 (0.018)	1.06 (1.02, 1.10)	0.002
Fluid/Electrolyte Disorder	0.031 (0.013)	1.03 (1.01, 1.06)	0.017
Drug Abuse	0.069 (0.014)	1.07 (1.04, 1.10)	< 0.001
Psychotic Disorders	0.106 (0.016)	1.11 (1.08, 1.15)	< 0.001
Labs
Albumin	−0.096 (0.011)	0.91 (0.89, 0.93)	< 0.001
Alanine Aminotransferase	0.000 (0.000)	1.00 (1.00, 1.00)	0.125
Blood Urea Nitrogen	0.002 (0.000)	1.00 (1.00, 1.00)	< 0.001
Calcium	−0.082 (0.011)	0.92 (0.90, 0.94)	< 0.001
Hematocrit	−0.007 (0.001)	0.99 (0.99, 1.00)	< 0.001
Mean Corpuscular Volume	0.003 (0.001)	1.00 (1.00, 1.00)	< 0.001
Sodium	−0.018 (0.002)	0.98 (0.98, 0.99)	< 0.001
White Blood Cell Count	0.007 (0.002)	1.01 (1.00, 1.01)	< 0.001
Potassium	0.053 (0.013)	1.05 (1.03, 1.08)	< 0.001
Inpatient Medications
Proton Pump Inhibitor	0.046 (0.013)	1.05 (1.02, 1.07)	< 0.001
Vancomycin	−0.100 (0.022)	0.90 (0.87, 0.94)	< 0.001
Lactulose	0.051 (0.017)	1.05 (1.02, 1.09)	0.002
Midodrine	−0.244 (0.055)	0.78 (0.70, 0.87)	< 0.001
Albumin	0.237 (0.022)	1.27 (1.21, 1.32)	< 0.001
NSAIDs	0.095 (0.021)	1.10 (1.05, 1.15)	< 0.001
Aminoglycosides	0.160 (0.044)	1.17 (1.08, 1.28)	< 0.001
1st Gen Cephalosporin	−0.213 (0.030)	0.81 (0.76, 0.86)	< 0.001
Glucocorticoids	0.068 (0.023)	1.07 (1.02, 1.12)	0.003
Lincomycin Antibiotics	−0.156 (0.047)	0.86 (0.78, 0.94)	< 0.001
Extended Spectrum Penicillins	−0.082 (0.024)	0.92 (0.88, 0.97)	< 0.001
Loop Diuretics	0.042 (0.014)	1.04 (1.01, 1.07)	0.003
Benzodiazepines	0.105 (0.014)	1.11 (1.08, 1.14)	< 0.001
Home Medications
Lactulose	0.109 (0.020)	1.12 (1.07, 1.16)	< 0.001
Glucocorticoids	0.132 (0.034)	1.14 (1.07, 1.22)	< 0.001
Loop Diuretics	0.043 (0.016)	1.04 (1.01, 1.08)	0.008
K Sparing Diuretic	0.046 (0.017)	1.05 (1.01, 1.08)	0.006
Fluoroquinolones	0.093 (0.028)	1.10 (1.04, 1.16)	< 0.001
Miscellaneous
Heart Rate	0.004 (0.000)	1.00 (1.00, 1.00)	< 0.001
Medicaid Insurance	0.190 (0.053)	1.21 (1.09, 1.34)	< 0.001
Surgery During Admission	−0.268 (0.023)	0.77 (0.73, 0.80)	< 0.001
Risk Scores
Fibrosis-4 score	0.000 (0.000)	1.00 (1.00, 1.00)	0.037
LACE Score	0.015 (0.002)	1.01 (1.01, 1.02)	< 0.001
MELD	0.012 (0.001)	1.01 (1.01, 1.01)	< 0.001

Note: LACE: Length-of-Stay, Admission acuity, Charlson co-morbidity index, Emergency Department visits in prior 6 months; MELD: Model for End-Stage Liver Disease

^¥Comorbidities were defined using the Elixhauser definitions.⁸⁸

Patients with hepatocellular carcinoma (HCC) or who had surgery within 24 hours of admission were associated with decreased risk of readmission. The HCC patients in our cohort had decreased prevalence of comorbid conditions including heart failure (12.6% vs. 25.6% in HCC vs. non-HCC, p < 0.001) and chronic kidney disease (14.0% vs. 21.0%, p < 0.001). Similarly, patients receiving surgery had fewer comorbidities and had experienced fewer complications of cirrhosis, including hepatic encephalopathy (12.0% vs. 24.4% in surgical vs. non-surgical patients, p < 0.001), ascites (25.4% vs. 36.8%, p < 0.001), and varices (18.2% vs. 22.6%, p < 0.001).

Effect of Variable Number

When analyzing model performance as a function of the number of variables, we identified a plateau effect for discrimination with the AUC asymptotically approaching 0.68. The calibration also showed a shoulder effect after ~ 40 variables (Figure 4). We note that for underfit models with very few variables, calibration appears superior however the model is unable to discriminate between cases and controls due to a lack of variability in predicted values. Even after recalibrating the Bajaj, Berman, and Singal models to VA data, their calibration was particularly poor as measured by the percentage of observations that were calibrated (41.0%, 28.1%, and 35.3%, respectively).

Figure 4: Discrimination and calibration performance of 30-day readmission models as a function of the number of included variables.

The variables used at each stage depend on the “solution path” of the LASSO. Confidence intervals obtained through 5-fold cross-validation. Panel A: Change in discrimination (as measured by AUC); Panel B: Change in calibration. We measured calibration performance as the number of observations that were well calibrated. We defined well calibrated as the observed probability matching the predicted probability (within a 20% margin of error). The dotted line represents the performance of our final model (47 variables). The performance of the VA-data tailored literature models (Bajaj, Berman, and Singal) are also provided.

Comparison to Existing Models

Our model’s AUC was statistically significantly better than the Singal, Berman, and Bajaj models: 0.649 (95% CI: 0.645 – 0.653), 0.565 (95% CI: 0.562 – 0.570), and 0.577 (95% CI: 0.573 – 0.581), respectively (Refer to Table 5). The NRI demonstrated that our model, when compared to the three extant models, achieved 3 – 16 percent improvement in predicting readmission accurately for high risk patients (Figure 5). For low-risk patients, the Singal, Berman, and Singal models were poorly calibrated and did not predict that any of the patients had a low risk of readmission. Refer to the Appendix for details.

Table 5:

Discrimination and calibration results for our model versus the three comparator, literature models.

Model	AUC	Brier score
Our Model	0.670 (0.666, 0.674)	0.165 (0.163, 0.166)
Bajaj35	0.577 (0.573, 0.581)	0.175 (0.173, 0.176)
Berman33	0.565 (0.562, 0.570)	0.175 (0.174, 0.177)
Singal34	0.649 (0.645, 0.653)	0.167 (0.165, 0.168)

Figure 5: Improvement in prediction accuracy for identifying low and high probability of 30-day readmission measured by the Net Reclassification Index.

The Net Reclassification Index compares our primary model against each established model with values > 0 indicating improved prediction performance. Performance is shown for two use cases: (a) identifying patients at low risk of readmission, < 10%; and (b) finding very high risk patients, > 40% risk of readmission. Our model shows improved overall performance and outcome specific performance (readmission versus no readmission). The outcome specific NRI can be interpreted as the change in true positive rate for predicting readmission (or conversely improvement in the true negative rate for predicting no readmission).

The Brier scores for the Singal, Berman, and Bajaj models were 0.167 (95% CI: 0.165 – 0.168), 0.175 (95% CI: 0.174 – 0.177), and 0.175 (95% CI: 0.173 – 0.176), respectively. Though calibration appears similar when assessed numerically, the smoothed calibration belts (Figure 6) demonstrate differences in ranges and uncertainty in calibration. The Bajaj and Berman models appear well calibrated for admissions predicted to have a probability of readmission between 0.2 and 0.4; however, closer inspection reveals that this is primarily due to a very narrow range of predicted probabilities. Both our model and the Singal model exhibit better calibration across a wider range of predictions. Yet in comparison to the Singal model, our model’s calibration curve is either closer or similarly close to ideal calibration across the range of predictions and displays less uncertainty in this performance, particularly among higher risk observations.

Figure 6: Calibration curves for our model versus the three comparator, literature models.

Note: Perfect calibration is represented by the diagonal black line. Curves are limited to the range of predicted probabilities provided by each model. GLM: “General Linear Model,” i.e. our model.

DISCUSSION

Existing research has identified targeted interventions that help reduce readmissions amongst patients with cirrhosis. However, for these interventions to be cost effective and successful, healthcare systems must identify patients at high readmission risk. In this work, we constructed a model to predict 30-day readmission amongst patients with cirrhosis. To the best of our knowledge, this was the largest dataset with granular, clinical data used for this endeavor (67,749 patients with 179,298 admissions). Our model was significantly better at predicting readmission compared to three published models.^33–35

There is an ongoing tension in risk prediction modeling used in clinical settings between model variable parsimony and performance. Perhaps this work’s most critical finding is that sustained, adequate levels of model calibration require retaining many more variables than would be required to show only discrimination. Furthermore, recalibrating external models with fewer variables does not provide the same level of performance in a local environment. This will be especially important for identifying high risk patients, for whom the most expensive interventions would be targeted. Misleading patient-level risk estimates may lead to overconfidence, inappropriately alter treatment choices, or misappropriate limited resources.^25,28,67 The re-calibrated Bajaj, Berman, and Singal models had less than 50% of observations well calibrated.

Overall, there were several risk factors identified that were significantly associated with readmission that may guide decision making, and in some cases, may be modifiable. Unsurprisingly, increased disease severity was associated with higher risk of readmission. These findings have been corroborated in other studies, including ascites^68,69 and HE.^70,71 We also identified PPI use associated with increased risk of readmission, similar to Bajaj et al.³⁵ Our finding that patients with HCC were associated with a lower risk of readmission was also identified by Brown et al.⁷² Higher mortality among these patients and discharge to hospice or palliative care status may account for these findings.

It is essential to include modifiable risk factors whenever possible, and we favored allowing more variables in our model over further parsimony. We identified electrolyte disturbance and diuretic use to be a strongly associated with readmission. Electrolyte disturbances including hyponatremia⁷³ and hypokalemia⁷⁰ have been associated with readmission in other studies. More investigation needs to be done in how to dial in the right diuretic dose for cirrhosis as outpatients, as this is a common problem after hospitalization as diuretics are often changed during hospitalization. For example, there is exciting work in passive sensor technologies such as wearable electrolyte sensors that can measure various serum electrolyte levels measured through sweat.⁷⁴

Our study further highlights the importance of psychosocial risk factors mediating readmission risk. Drug abuse, severe psychiatric disorders, and Medicaid insurance (a surrogate for income and disability) were all strongly correlated with readmission. The VA’s patients face a higher burden of substance abuse, particularly among Iraq and Afghanistan veterans where the prevalence has been estimated at 11%.⁷⁵ Patients with cirrhosis face a high burden of depression, unemployment (44%), and financial risk.⁷⁶ Mental health comorbidities and substance abuse was a common predictor in other models of readmission risk in other diseases.⁷⁷

Historically, risk models relied on relatively few predictor variables;^33–35 however, the power of the EHR has allowed risk prediction models to be built using significantly larger cohorts with a larger candidate predictor variable pool. Routine revalidation and possible recalibration allows prediction models to be tailored to the local clinical setting.^41,78–80 All three tested models declined in discrimination and calibration performance in this cohort, despite following best practices and updating the models for VA data. Our data suggest increased accuracy using a large national dataset; however, our analysis also suggests that there might be a ceiling effect for predictive accuracy using all relevant EHR variables within a system, at least for patients with cirrhosis.

The EHR allows automated risk calculation and integration of predictive analytics into the clinical workflow for decision support. Several studies have shown improved outcomes embedding more complicated prediction models, automatically calculated by the EHR, within routine care. ^81–83 Numerous studies have described interventions to reduce hospital readmission, though effective interventions tend to be complex and involve patients in self-care.⁸⁴ Medicare initiatives targeting multifactorial improvements in post-discharge transitions of care have been shown to be effective in community hospital settings.⁸⁵ Examples of post discharge interventions include more intensive medical monitoring or telemonitoring, early post-discharge visits and phone calls, and team-based approaches.⁸⁶ Tapper et al. have shown that using checklists and standard protocols for postdischarge treatment of patients with encephalopathy or subacute bacterial peritonitis effectively reduced readmission rates in decompensated cirrhotics.⁸⁷ Careful consideration of patient risk and appropriate interventions to prevent readmissions continues to remain a challenge.

Our analysis has limitations. First, our cohort is largely comprised of male patients and may not generalize to a population with a greater proportion of female patients, particularly as female patients may face specific psychosocial risk factors that impact readmission risk. In terms of applicability, the current model can be applied within the VA EHR system at any facility. Regardless, none of the variables we used in our model are specific to the Department of Veterans Affairs. Validating model performance in other healthcare systems would be essential future work for testing the concept that this comprehensive group of variables would provide greater predictive accuracy.

Second, the transition to ICD-10 in the US (and its use in the rest of the world) will require revalidation for codes without a one-to-one mapping. Third, the VA clinical data warehouse captures information regarding all inpatient admissions to the VA and all admissions outside the VA system for which the VA pays. However, there will be a minority of patients who seek inpatient care outside the VA system, which would lead to underestimation of readmission. Fourth, the structured data only identified 123 hospitalizations where the patient was discharged to hospice care, which is likely heavily underestimated. As discussed previously, this ascertainment bias may have led to the association of reduced readmission for HCC patients. Finally, the utility of the model will require prospective validation at multiple sites within the VA system, first for corresponding accuracy and second to see if it can be used to target appropriate interventions to help reduce short term readmissions.

Conclusion

In summary, this study identified a high 30-day readmission rate in patients with advanced liver disease. To our knowledge, this is the largest study building a readmission risk model using granular clinical data for patients with cirrhosis. Our evaluation of published readmission risk models indicated all would require recalibration to be used in a new healthcare setting and some may require fully rebuilding the model. Predicting readmission for patients with cirrhosis is challenging. The current model is a step towards personalized medicine and advancing high performing predictive analytics deployed within routine care and describes the use of a robust modeling method that brings us closer to the promise of big data.