An Automated Model Using Electronic Medical Record Data Identifies Patients With Cirrhosis at High Risk for Readmission

Abstract

BACKGROUND & AIMS

Patients with cirrhosis have 1-month rates of readmission as high as 35%. Early identification of high-risk patients could permit interventions to reduce readmission. The aim of our study was to construct an automated 30-day readmission risk model for cirrhotic patients using electronic medical record (EMR) data available early during hospitalization.

METHODS

We identified patients with cirrhosis admitted to a large safety-net hospital from January 2008 through December 2009. A multiple logistic regression model for 30-day rehospitalization was developed using medical and socioeconomic factors available within 48 hours of admission and tested on a validation cohort. Discrimination was assessed using receiver operator characteristic curve analysis.

RESULTS

We identified 836 cirrhotic patients with 1291 unique admission encounters. Rehospitalization occurred within 30 days for 27% of patients. Significant predictors of 30-day readmission included the number of address changes in the prior year (odds ratio [OR], 1.13; 95% confidence interval [CI], 1.05–1.21), number of admissions in the prior year (OR, 1.14; 95% CI, 1.05–1.24), Medicaid insurance (OR, 1.53; 95% CI, 1.10 –2.13), thrombocytopenia (OR, 0.50; 95% CI, 0.35– 0.72), low level of alanine aminotransferase (OR, 2.56; 95% CI, 1.09 – 6.00), anemia (OR, 1.63; 95% CI, 1.17–2.27), hyponatremia (OR, 1.78; 95% CI, 1.14 –2.80), and Model for End-stage Liver Disease score (OR, 1.04; 95% CI, 1.01–1.06). The risk model predicted 30-day readmission, with c-statistics of 0.68 (95% CI, 0.64 – 0.72) and 0.66 (95% CI, 0.59 – 0.73) in the derivation and validation cohorts, respectively.

CONCLUSIONS

Clinical and social factors available early during admission and extractable from an EMR predicted 30-day readmission in cirrhotic patients with moderate accuracy. Decision support tools that use EMR-automated data are useful for risk stratification of patients with cirrhosis early during hospitalization.

Cirrhosis affects more than 5.5 million patients and costs nearly $4 billion annually.¹ The burden of cirrhosis is anticipated to increase over the next 10 years, beyond the 1 million hospitalizations per year currently attributed to liver disease.² Unfortunately, our episodic system of health care delivery is suboptimal for patients with chronic disease, such as cirrhosis.³ Patients with cirrhosis remain at high risk for frequent readmissions, with more than one third of patients being rehospitalized within 1 month of discharge.⁴

Readmission within 30 days of hospitalization has emerged as a focus of quality improvement and payment reform.^5,6 The Center for Medicare and Medicaid Services estimates that more than 13% of readmissions may be avoidable and will reduce reimbursements to hospital systems with excess readmissions.⁷ Although interventions such as careful discharge planning can prevent readmissions, these programs are challenging to sustain because of competing financial needs.⁸

Hospitals and health systems across the United States are in the midst of an unprecedented implementation of electronic medical records (EMRs).⁹ The opportunities that these systems might afford to improve the care of patients with cirrhosis have not been explored fully.³ The use of real-time EMR data available on admission, instead of administratively coded data from discharge (which often is not verified until 2–3 mo after discharge), can facilitate early identification of high-risk patients, allowing implementation of interventions during a hospitalization to reduce readmission. Although low-risk patients could be considered for early discharge, high-risk patients might be triaged to specialized hospital services, intensive case management, and earlier clinic visits after discharge. Furthermore, decision support systems that incorporate EMR-based risk models may help allocate limited resources. Electronic predictive models are particularly attractive because they may be introduced quickly and cheaply, after implementation of an EMR. Although several studies have attempted early risk stratification of patients with chronic diseases, such as congestive heart failure, to identify subjects for targeted interventions during hospitalization, this area has not been evaluated extensively in patients with cirrhosis.^10–13

We hypothesized that an automated model using both clinical and nonclinical factors, available within 48 hours of admission, could stratify readmission risk among patients with cirrhosis, drawing on the explanatory power of social, behavioral, and utilization factors. In this study, we constructed an electronic model of 30-day readmission risk, using present-on-admission data, which commonly is available to hospitals with an EMR.

Methods

Study Population

We conducted a retrospective cohort study of patients with cirrhosis admitted to Parkland Memorial Hospital between January 2008 and December 2009. As the sole safety-net hospital for Dallas County, Parkland Hospital cares for a large portion of cirrhotic patients in the area. Patients initially were identified using International Classification of Diseases, 9th revision (ICD-9) codes: 571.2 (alcoholic cirrhosis), 571.5 (nonalcoholic cirrhosis), 572.3 (portal hypertension), 572.2 (hepatic encephalopathy), 456.0 to 456.2 (esophageal varices with and without bleeding), 789.5 (ascites), 789.59 (nonmalignant ascites), 567.23 (spontaneous bacterial peritonitis), and 572.4 (hepatorenal syndrome). The EMR was reviewed by one author (A.G.S.), who was blinded to readmission outcome, to confirm cirrhosis diagnosis, defined as consistent histology or a cirrhotic-appearing liver on imaging with portal hypertension (ascites, encephalopathy, varices, or splenomegaly with thrombocytopenia) (see the Supplementary Appendix for details).¹⁴ The author was provided with a list of patients, identified by the ICD-9 code case-finding algorithm, but not provided with admission dates, or informed about readmission status. The author reviewed the pathology data, imaging studies, and laboratory data to determine the presence of cirrhosis, but did not access progress notes or encounters for this analysis. Patients were excluded if they died during the index (ie, initial) hospitalization, died within 30 days after discharge, or underwent liver transplantation. An index hospital encounter was defined as any nonelective admission, without prior admission within 30 days of the index admission date. The nature of admission (elective vs nonelective) is coded electronically, so exclusion of elective admission was possible using automated data. For patients with multiple admissions, each was considered as a separate admission if more than 30 days had elapsed between hospitalizations.

Outcome Variables

Our primary outcome was any-cause rehospitalization, excluding elective admissions, to any of the 136 hospitals in the Dallas–Fort Worth area within 30 days of the index hospitalization. We determined readmission to any of the hospitals using a probabilistic linkage service available through the Dallas–Fort Worth Hospital Council, a regional information sharing initiative (see the Supplementary Appendix for details). Although readmission data returned de-identified, we confirmed validity of the linkage by comparing readmissions with the index institution with an indicator variable precoded in the dataset, resulting in 100% agreement.

A secondary outcome was all-cause mortality within 90 days of discharge. Patients who died during the index hospitalization were excluded, but we included patients who died after discharge. We identified deceased patients by reviewing the last encounter in the EMR after discharge; patients without an encounter 90 days after discharge were classified using the Social Security Death Index.

Predictor Variables

Candidate risk factor variables for the electronic model had to meet 3 criteria: capable of extraction from the EMR, routinely available within 48 hours of hospitalization, and available to most hospitals with a basic EMR, as judged by consensus of an interdisciplinary team. A conceptual framework of readmission based on literature review and clinical expertise was developed.¹⁵

Clinical data were collected, using structured data and ICD-9 codes, including demographics (age, sex, race), alcohol and tobacco history, etiology of cirrhosis, history of decompensation (ascites, hepatic encephalopathy, or variceal bleeding), hepatocellular carcinoma, extrahepatic cancer, comorbid conditions, and reason for index hospitalization. Laboratory values of interest included white blood cell count, platelet count, sodium level, creatinine level, albumin level, aspartate aminotransferase level, alanine aminotransferase (ALT) level, bilirubin level, and international normalized ratio (INR) at admission. The Model for End-stage Liver Disease (MELD) score, which correlates with 90-day survival,¹⁶ also served as a measure of illness at admission.

Markers of social, behavioral, and utilization activity available using electronic data sources on presentation were recorded (Table 1). These variables were hypothesized to represent measures of social instability or low socioeconomic status (eg, number of address changes in the prior year and insurance status), risky health behavior (eg, history of cocaine), or health care use patterns (eg, number of emergency room visits in the prior year) that could influence readmission risk. In addition, we identified the presence of depression and anxiety using ICD-9 CM codes because they have been linked to negative health behaviors. We also investigated 30-day readmission as a predictor for 90-day mortality. All variables were extracted from the hospital’s EMR (EPIC Systems Corporation, Verona, WI) and related electronic databases using business intelligence software (Crystal Reports, Walldorf, Germany) that accesses structured data from the relational database. No variables required manual extraction from patient charts. For categoric variables with missing data, a missing category was created, and a readmission rate for the missing group was pooled into the appropriate reference group. For continuous variables, a multiple regression imputation algorithm was used.

Table 1.

Population Characteristics

Variable	Derivation cohort (n = 968)	Validation cohort (n = 323)
Age, y	52.5 ± 10.1	52.6 ± 10.8
Male sex, n (%)	656 (67.7)	214 (66.2)
Race/ethnicity, n (%)
Non-Hispanic Caucasian	318 (32.9)	100 (31.0)
Hispanic Caucasian	390 (40.3)	138 (42.7)
Black	215 (22.2)	73 (22.6)
Other	45 (4.6)	12 (3.7)
Insurance, n (%)
Medicaid	273 (28.2)	95 (29.4)
Medicare	131 (13.5)	53 (16.4)
Charity	292 (30.2)	84 (26.0)
Other	272 (28.1)	91 (28.2)
Single marital status, n (%)	654 (68.0)	209 (65.5)
English as primary language, n (%)	710 (73.8)	235 (73.6)
White blood cell count, ×109/L	5.8 ± 3.6	5.5 ± 2.9
Hematocrit, %	33.9 ± 5.6	34.9 ± 5.6
Platelet count, ×109/L	103.5 ± 71.1	96.7 ± 60.7
Sodium level, mEq/L	132.7 ± 5.1	132.7 ± 5.0
Creatinine level, mg/dL	1.5 ± 2.1	1.4 ± 2.0
AST level, U/L	89.9 ± 104.9	93.8 ± 195.9
ALT level, U/L	49.3 ± 66.8	47.3 ± 65.7
Bilirubin level, mg/dL	3.2 ± 4.2	2.9 ± 3.2
MELD score	15.3 ± 6.1	15.3 ± 5.9
History of HIV/AIDS, n (%)	51 (5.2)	12 (3.7)
Length of index hospitalization, d	7.3 ± 6.0	5.2 ± 5.6
Number of admissions in the year before index hospitalization	1.0 ± 1.6	1.1 ± 1.8
Number of address changes in the year before index hospitalization	1.0 ± 1.8	1.1 ± 1.9
Active alcohol use, n (%)	160 (16.5)	56 (17.3)
Active cocaine use, n (%)	52 (5.3)	11 (3.4)
Residence in high-risk census tract, n (%)	149 (15.3)	53 (16.4)
Presented to emergency room between 6 PM and 6 AM, n (%)	648 (66.9)	222 (68.7)
Admitted to medicine service, n (%)	915 (94.5)	291 (90.0)
History of leaving AMA, n (%)	34 (3.5)	3 (0.9)

AMA, against medical advice; AST, aspartate aminotransferase.

Derivation of the Electronic Readmission Model

The electronic readmission model was constructed in 4 stages. First, univariate relationships were examined through logistic regression with a threshold significance of a P value of less than .20. To protect against overfitting, the maximum number of variables for the model was estimated using shrinkage analysis. Next, candidate variables were ranked by P value using bootstrapping with replacement in 1000 multiple logistic regression iterations. Final model variables were selected on the basis of conceptual and statistical significance and did not exceed the number of variables permitted by shrinkage analysis. In the last stage, model coefficients and confidence intervals (CIs) were determined using bootstrapping with replacement in logistic regressions containing the final variables. Statistical significance was defined as a P value of less than .05. To account for the effects of patients with multiple index admissions, we applied robust variance-covariance matrix estimates to all models.

The model was validated using several approaches. Model discrimination was assessed by the c-statistic using receiver operating characteristic curve analysis. Model calibration was evaluated using the Hosmer-Lemeshow chi-square goodness-of-fit test. We internally validated the model by splitting the sample into derivation (75%) and validation (25%) cohorts. The model was constructed in the derivation sample (n = 629 patients; 968 unique admissions) and tested on the validation (n = 209 patients; 323 unique admissions) cohort. By using cut-off points determined by derivation subsamples, 5 risk categories were created based on quintiles of predicted risk and graphically assessed in the validation cohort. To assess the association between predicted risk and time-to-readmission, Kaplan–Meier curves were constructed for stratified risk groups.

Assessing and Comparing Model Performance

We compared our electronic readmission model with a readmission model by Berman et al.¹⁷ Significant predictors of 30-day readmission in their model included discharge MELD score (odds ratio [OR], 1.06; 95% CI, 1.02–1.09), diabetes (OR, 1.78; 95% CI, 1.07–2.95), and male sex (OR, 1.73; 95% CI, 1.03–2.89). We were unable to assess another model⁴ because not all variables could be extracted electronically from the EMR. The incremental discriminative performance of the models was assessed using the c-statistic and integrated discrimination improvement (IDI) index. The IDI is capable of detecting smaller incremental improvements in risk assessment than the c-statistic. Analyses were conducted using STATA 10.0 (College Station, TX) and RTREE (available:http://peace.med.yale.edu/pub). The University of Texas Southwestern Medical Center Institutional Review Board approved the research protocol.

Results

Study Population

A total of 836 patients with cirrhosis met inclusion criteria and had 1291 unique index admissions during the study period. Fifty-nine patients died during hospitalization and 30 were excluded because of death within 30 days after discharge. An additional 47 encounters were elective admissions and therefore were excluded. The median time to readmission was 12 days, with 109 (9.4%) patients readmitted within 1 week, and 347 (26.9%) readmitted within 30 days of discharge. Seventy-eight percent of readmissions occurred at Parkland Hospital, and 22% were readmitted to another hospital in the Dallas–Fort Worth area.

Clinical characteristics of the cohort are shown in Table 1. The mean age of patients was 52.5 years (range, 19–90 y), with the majority being younger than 55 years old. The cohort was predominantly male (67.6%) and single (67.3%). Our population was racially diverse, with 22.9% African Americans, 32.5% non-Hispanic Caucasians, and 40.3% Hispanic Caucasians. More than one fourth of patients were uninsured, 40.6% had Medicaid, and 16.0% had private health insurance. The average length of hospitalization was 5.9 days, with patients having an average of 1.0 admission in the prior year. The median MELD score was 14, with nearly three fourths of patients having MELD scores greater than 11.

Predictors of Readmission

Significant predictors of rehospitalization in simple logistic regression included male sex, sodium level, ALT level, aspartate aminotransferase level, bilirubin level, creatinine level, MELD score, white blood cell count, hematocrit level, INR, albumin level, platelet count, alcohol use, cocaine use, comorbid conditions (congestive heart failure, coagulation disorders, history of malignancy, electrolyte disorders, hypertension, and renal failure), marital status, insurance status, liver disease etiology, number of admissions in the prior year, number of address changes in the prior year, time of presentation to the emergency room, and primary admission service (medicine vs surgery). Because bilirubin, creatinine, and INR were components of MELD, they were not included separately in multiple logistic regression. Linear shrinkage analysis did not show overfitting. A number of demographic, health behavior, and utilization predictors of readmission remained significant in multiple logistic regression (Table 2). The number of admissions in the year before the index hospitalization (continuous) was a predictor of readmission (OR, 1.14; 95% CI, 1.05–1.24). Several socioeconomic variables including the number of address changes in the prior year (OR, 1.13; 95% CI, 1.05–1.21) and Medicaid insurance (OR, 1.53; 95% CI, 1.10 –2.13) also were associated with readmission, after adjusting for markers of disease severity including MELD and hyponatremia.

Table 2.

Multiple Logistic Regression of Predictors for 30-Day Readmission

Variablea	30-d readmission risk	OR (95% CI)b	P value
Number of address changes in the year before index hospitalization   (continuous)		1.13(1.05–1.21)	.001
Zero address changes	24%
1 address change	28%
≥2 address changes	30%
Number of admissions in the year before index hospitalization (continuous)		1.14(1.05–1.24)	.002
Zero admissions	20%
1 admission	25%
≥2 admissions	35%
Medicaid insurance	33%	1.53(1.10–2.13)	.012
Platelet count		0.50 (0.35–0.72)	<.001
<77,000/μL	28%
77,000–112,000/μL	18%
>112,000/μL	30%
ALT level		2.56 (1.09–6.00)	.031
≤9 U/L	48%
>9 U/L	26%
Hematocrit		1.63(1.17–2.27)	.004
≤30%	37%
>30%	22%
Sodium level		1.78(1.14–2.80)	.012
≤130 mEq/L	40%
>130 mEq/L	25%
MELD score (continuous)		1.04(1.01–1.06)	.004

^aAll variables were collected within 48 hours of admission from the EMR.

^bORs greater than 1.0 are associated with a higher risk of 30-day readmission, and ORs less than 1.0 are associated with a lower risk of readmission.

Electronic Model for 30-Day Readmission

The electronic readmission model produced c-statistics of 0.68 (95% CI, 0.64–0.72) and 0.66 (95% CI, 0.59–0.72) in derivation and validation cohorts, respectively. There was no evidence of lack of fit (P = .94). The electronic model was capable of stratifying patients across a wide range of risk, with high concordance between the derivation and validation cohorts. Patients in higher risk categories were readmitted earlier over the 30-day period after discharge (Figure 1). Although low-risk patients were readmitted in less than 20% of cases, 45% of high-risk patients were readmitted within 30 days (P < .001). Patients in the lowest-risk category had a mean time to readmission of 27.7 days, compared with 22.3 days in the highest-risk group (P < .001) (Figure 2).

Figure 1.

Thirty-day readmission rates stratified by readmission model risk quintile.

Figure 2.

Time to readmission stratified by readmission model risk quintile. Patients in the lowest-risk category had a significantly longer time to readmission than those in the highest-risk group (P < .001).

The model by Berman et al¹⁷ showed a fair level of discrimination for 30-day readmission, with a c-statistic of 0.57 (95% CI, 0.54–0.61). However, our electronic readmission model significantly exceeded the predictive capability of this previous model, with an IDI of 0.06 (95% CI, 0.04–0.07) (Table 3).

Table 3.

Comparison of Model Performance for 30-Day Readmission and 90-Day Mortality

	30-d readmission (n = 1291)		90-d mortality (n = 1334)
Model	C-statistic (95% CI)	IDI (95% CI)	C-statistic (95% CI)	IDI (95% CI)
MELD score	0.59 (0.55–0.62)	Reference	0.65 (0.60–0.70)	Reference
Indiana readmission model17,a	0.57 (0.54–0.61)	−0.01 (−0.01 to 0.00)	0.62 (0.57–0.67)	−0.01 (−0.02 to 0.00)
PHHS readmission model	0.67(0.64–0.71)	0.07 (0.05–0.08)	0.66 (0.61–0.71)	0.04 (0.02–0.06)
Indiana mortality model17,b	—	—	0.73 (0.68–0.77)	0.03 (0.02–0.05)
PHHS mortality modelc	—	—	0.75 (0.72–0.79)	0.03 (0.02–0.05)

PHHS, Parkland Health and Hospital System.

^aincludes MELD score at discharge, presence of diabetes, and male sex.

^bincludes MELD score at discharge, sex, age, and 30-day readmission.

^cincludes MELD score at admission, platelet count, hematocrit, sodium, Medicare insurance status, and 30-day readmission.

Predictors of 90-Day Mortality

After excluding patients who died during the index hospitalization, the 90-day mortality rate was 10.3% among 1334 unique index admissions. On multiple logistic regression, 30-day readmission was associated significantly with 90-day mortality (OR, 1.92; 95% CI, 1.31–2.81) (Supplementary Table 1). The 90-day mortality rate for patients readmitted within 30 days was 17.3%, compared with 7.9% in those without readmission (P < .001). Our mortality model predicted 90-day mortality with good accuracy, with a c-statistic of 0.75 (95% CI, 0.72–0.79). Our mortality model exceeded the predictive capability of a prior 90-day mortality model¹⁷ that had a c-statistic of 0.73 (95% CI, 0.68–0.77), as well as that of MELD score, which had a c-statistic of 0.65 (95% CI, 0.60–0.70) (Table 3), although this did not reach statistical significance.

Discussion

We found that early rehospitalizations among patients with cirrhosis are common, with 27% of patients readmitted within 1 month of discharge. Our model predicted 30-day readmission with moderate accuracy. Our model is unique in that it assessed patients early during hospitalization, used only electronic data, and was able to stratify patients into high-risk and low-risk groups for readmission. Although low-risk patients were readmitted in less than 20% of cases, 45% of high-risk patients were readmitted within 30 days. Finally, the ability to use data from an EMR to predict readmission has major implications in the current health care environment.

A systematic review found that most readmission models typically only achieve modest predictive capability, with c-statistics rarely exceeding 0.6.¹⁸ Our electronic readmission model produced a c-statistic of 0.66, significantly exceeding the predictive capability of prior models in cirrhosis. Although a previous readmission model achieved a c-statistic of 0.65, this model has not been externally validated.⁴ A strength of our study was that our readmission model was validated in a cohort separate from the derivation cohort. Validation is an important aspect of predictive model development, given that model performance generally is higher in derivation data sets than in validation sets.¹⁹

If further refined to have better accuracy, our readmission risk model could be used for an EMR decision support tool to aid providers with real-time assessments of readmission risk. Accurate assessment of readmission risk may facilitate targeted application of resource-intensive interventions, such as specialized hospital services, intensive case management, and earlier clinic visits after discharge, for high-risk patients.

This study examined social, behavioral, and economic factors when predicting readmission risk. We found that several markers of social and economic instability, such as Medicaid insurance and address changes, were associated with readmission. We believe this in part explains why our model was able to outperform prior readmission risk models. Current risk adjustment models may not sufficiently define patients’ social or environmental risk to allow safety-net institutions to compete effectively in pay-for-performance environments. Risk-adjustment models that account for complex social and behavioral factors could help identify institutions that excel in the care of these difficult-to-treat patients.^20,21 Although these data are absent from claims data, these factors may be extracted from EMRs.

Surprisingly, we found that low ALT levels and higher platelet counts were associated with readmission risk. Although low ALT levels have been correlated with poorer prognosis, our data suggest that these patients may be at higher risk of readmission.^3,4 It is important to note that although predictive models may be used to provide insight into causality of the studied outcome, this is not a requirement.²² Noncausal predictive factors can be surrogates for other markers, with tumor markers as predictors of cancer progression being the most common example. Patients with platelet counts between 77,000 and 112,000/μL were less likely to be readmitted, although this may be a surrogate for other factors (eg, patients with higher platelet counts were more likely to be admitted for ascites) (9% vs 17%; P < .001). Further studies are necessary to confirm the prognostic value of these clinical characteristics.

We recognize the limitations of our study. Our study was performed in a single large safety-net hospital system, and our results may not be generalizable across all health care settings. In addition, our model’s predictor variables were extracted from the EMR and prone to coding errors. Specifically, variables such as behavioral and social variables may not be captured accurately using administrative data. Although we confirmed our algorithm for identifying cirrhosis, we did not perform a chart review to confirm the accuracy of any other included variables. Furthermore, there are potential unmeasured confounders and missing data given our study’s retrospective nature. Finally, our analysis did not fully account for any autocorrelation of repeat admissions among patients. However, we believe our study’s strengths—including the racially and socioeconomically diverse population, our inclusion of social and economic variables, and that we captured readmissions to any of 136 surrounding hospitals—outweigh its limitations. Most importantly, our study used data that can be extracted from an EMR at admission to identify high-risk patients, allowing for earlier implementation of interventions.

In conclusion, hospital readmissions among patients with cirrhosis are common. Clinical and social factors, easily retrievable from an EMR at admission, can identify patients at highest risk for readmission with moderate accuracy. Our data suggest that decision support tools using automated data from an EMR may help risk-stratify patients early during hospitalization. Future studies are needed to determine whether an automated electronic predictive model can facilitate early interventions to reduce readmission rates.

Supplementary Material

01

Supplementary Table 1. Multivariate Analysis of Predictors for 90-Day Mortality

Abbreviations used in this paper

ALT

alanine aminotransferase

CI

confidence interval

EMR

electronic medical record

ICD-9

International Classification of Diseases, 9th revision

IDI

integrated discrimination improvement

INR

international normalized ratio

MELD

Model for End-stage Liver Disease

OR

odds ratio