A Public-Private Partnership Develops and Externally Validates a 30-Day Hospital Readmission Risk Prediction Model

Abstract

Introduction: Preventing the occurrence of hospital readmissions is needed to improve quality of care and foster population health across the care continuum. Hospitals are being held accountable for improving transitions of care to avert unnecessary readmissions. Advocate Health Care in Chicago and Cerner (ACC) collaborated to develop all-cause, 30-day hospital readmission risk prediction models to identify patients that need interventional resources. Ideally, prediction models should encompass several qualities: they should have high predictive ability; use reliable and clinically relevant data; use vigorous performance metrics to assess the models; be validated in populations where they are applied; and be scalable in heterogeneous populations. However, a systematic review of prediction models for hospital readmission risk determined that most performed poorly (average C-statistic of 0.66) and efforts to improve their performance are needed for widespread usage.

Methods: The ACC team incorporated electronic health record data, utilized a mixed-method approach to evaluate risk factors, and externally validated their prediction models for generalizability. Inclusion and exclusion criteria were applied on the patient cohort and then split for derivation and internal validation. Stepwise logistic regression was performed to develop two predictive models: one for admission and one for discharge. The prediction models were assessed for discrimination ability, calibration, overall performance, and then externally validated.

Results: The ACC Admission and Discharge Models demonstrated modest discrimination ability during derivation, internal and external validation post-recalibration (C-statistic of 0.76 and 0.78, respectively), and reasonable model fit during external validation for utility in heterogeneous populations.

Conclusions: The ACC Admission and Discharge Models embody the design qualities of ideal prediction models. The ACC plans to continue its partnership to further improve and develop valuable clinical models.

Introduction

Curbing the frequency and costs associated with hospital readmissions within 30 days of inpatient discharge is needed to improve the quality of health care services [1-3]. Hospitals are held accountable for care delivered through new payment models, with incentives for improving discharge planning and transitions of care to mitigate preventable readmissions [4,5]. Consequently, hospitals must reduce readmissions to evade financial penalties by the Centers for Medicare & Medicaid Services (CMS) under the Hospital Readmissions Reduction Program (HRRP) [6]. In 2010, Hospital Referral Regions (HRRs) in the Chicago metropolitan area had higher readmission rates for medical and surgical discharges when compared with the national average [7], and were among the top five HRRs in Illinois facing higher penalties [8].

Although penalizing high readmission rates has been debated since the introduction of the policy [9], there has been consensus on the need for coordinated and efficient care for patients beyond the hospital walls to prevent unnecessary readmissions. Augmenting transitions of care during the discharge process and proper coordination between providers across care settings are key drivers needed to reduce preventable readmissions [10-12]. Preventing readmissions must be followed up with post-discharge and community-based care interventions that can improve, as well as sustain, the health of the population to decrease hospital returns. While several interventions have been developed that aim to reduce unnecessary readmissions by improving the transition of care process during and post-discharge [13-17], there is a lack of evidence on what interventions are most effective with readmission reductions on a broad scale [18].

One approach to curtailing readmissions is to identify high risk patients needing effective transition of care interventions using prediction models [19]. Ideally, the design of prediction models should offer clinically meaningful discrimination ability (measured using the C-statistic); use reliable data that can be easily obtained; utilize variables that are clinically related; be validated in the populations in which use is intended; and be deployable in large populations [20]. For a clinical prediction model, a C-statistic of less than 0.6 has no clinical value, 0.6 to 0.7 has limited value, 0.7 to 0.8 has modest value, and greater than 0.8 has discrimination adequate for genuine clinical utility [21]. However, prediction models should not rely exclusively on the C-statistic to evaluate utility of risk factors [22]. They should also consider bootstrapping methods [23] and incorporate additional performance measures to assess prediction models [24]. Research also suggests that prediction models should maintain a balance between including too many variables and model parsimony [25,26].

A systematic review of 26 hospital readmission risk prediction models found that most tools performed poorly with limited clinical value (average C-statistic of 0.66), about half relied on retrospective administrative data, a few used external validation methods, and efforts were needed to improve their performance as usage becomes more widespread [27]. In addition, a few parsimonious prediction models were developed after this review. One was created outside the U.S. and yielded a C-statistic of 0.70 [28]. The other did not perform external validation for geographic scalability and had a C-statistic of 0.71 [29]. One of the major limitations of most prediction models is that they are mostly developed using administrative claims data. Given the myriad of factors that can contribute to readmission risk, models should also consider including variables obtained in the Electronic Health Record (EHR).

Fostering collaborative relationships and care coordination with providers across care settings is needed to reduce preventable readmissions [18]. Care collaboration and coordination is central to the Health Information Technology for Economic and Clinical Health (HITECH) Act in promoting the adoption and meaningful use of health information [30]. Therefore, health care providers should also consider collaborating with information technology organizations to develop holistic solutions that improve health care delivery and the health of communities.

Advocate Health Care, located in the Chicago metropolitan area, and Cerner partnered to create optimal predictive models that leveraged Advocate Health Care’s population risk and clinical integration expertise with Cerner's health care technology and data management proficiency. The Advocate Cerner Collaboration (ACC) was charged with developing a robust readmission prevention solution by improving the predictive power of Advocate Health Care’s current manual readmission risk stratification tool (C-Statistic of 0.69), and building an automated algorithm embedded in the EHR that stratifies patients at high risk of readmission needing care transition interventions.

The ACC developed their prediction models taking into consideration recommendations documented in the literature to create and assess their models’ performance, and performed an external validation for generalizability using a heterogeneous population. While previous work relied solely on claims data, the ACC prediction models incorporated patient data from the EHR. In addition, the ACC team used a mixed-method approach to evaluate risk factors to include in the prediction models.

Objectives

The objectives of this research project were to: 1) develop all-cause hospital readmission risk prediction models for utility at admission and prior to discharge to identify adult patients likely to return within 30-days; 2) assess the prediction models’ performance using key metrics; and 3) externally validate the prediction models’ generalizability across multiple hospital systems.

Methods

A retrospective cohort study was conducted among adult inpatients discharged between March 1, 2011 and July 31, 2012 from 8 Advocate Health Care hospitals located in the Chicago metropolitan area (Figure 1).

Figure 1.

Geographic Location of 8 Advocate Health Care Hospitals

An additional year of data prior to March 1, 2011, was extracted to analyze historical patient information and prior hospital utilization. Inpatient visits thru August 31, 2012, were also extracted to account for any readmissions occurring within 30 days of discharge after July 31, 2012. Encounters were excluded from the cohort if they were observation, inpatient admissions for psychiatry, skilled nursing, hospice, rehabilitation, maternal and newborn visits, or if the patient expired during the index admission. Clinical data was extracted from Cerner’s Millennium® EHR software system and Advocate Health Care’s Enterprise Data Warehouse (EDW). Data from both sources was then loaded into Cerner’s PowerInsight® (PIEDW) for analysis.

The primary dependent variable for the prediction models was hospital readmissions within 30-days from the initial discharge. Independent variables were segmented into 8 primary categories: Demographics and Social characteristics, Hospital Utilization, History & Physical Examination (H&P), Medications, Laboratory Tests, Conditions and Procedures (using International Classification of Diseases, Ninth Revision, Clinical Modification codes ICD-9 CM), and an Exploratory Group (Figure 2).

Figure 2.

ACC Readmission Risk Prediction Conceptual Model

Risk factors considered for analysis were based on literature reviews and a mixed-method approach using qualitative data collected from clinical input. Qualitative data were collected from site visits at each Advocate Health Care hospital through in-depth interviews and focus groups with clinicians and care mangers, respectively. Clinicians and care mangers were asked to identify potential risk factors that caused a patient to return to hospital. Field notes were taken during the site visits. Information gleaned was used to identify emerging themes that helped inform the quantitative analyses.

All quantitative statistical analyses were conducted using SAS® version 9.2 (SAS Institute). Descriptive and inferential statistics were performed on the primary variable categories to identify main features of the data and any causal relationships, respectively. The overall readmission rate was computed using the entire cohort. For modeling, one consecutive encounter pair (index admission and readmission encounter) was randomly sampled from each patient to control for bias due to multiple admissions. Index encounters were restricted to a month prior to the study period’s end date to capture any readmissions that occurred within 30 days (Figure 3).

Figure 3.

Multiple Readmission Sampling Methodology

To develop and internally validate the prediction models, the cohort was then split into a derivation dataset (75%) and a validation dataset (25%). Model fitting was calculated using bootstrapping method by randomly sampling two-thirds of the data in the derivation dataset. The procedure was repeated 500 times and the averaged coefficients were applied to the validation dataset. Stepwise logistic regression was performed and predictors that were statistically significant using a p-value ≤ 0.05 were included in the model. Two predictive models were developed: one at admission and one prior to discharge using readily available data for the patient. The admission prediction included baseline data available for a patient once admitted to the hospital. The discharge prediction model was more comprehensive, including additional data that became available prior to discharge.

The performance of each prediction model was assessed by 3 measures. First, discrimination ability was quantified by sensitivity, specificity, and the area under the receiver operating characteristic (ROC) curve, or C-statistic that measures how well the model can separate those who do and do not have the outcome. Second, calibration was performed using the Hosmer-Lemeshow (H&L) goodness-of-fit test, which measures how well the model fits the data or how well predicted probabilities agree with actual observed risk, where a p-value > 0.05 indicates a good fit. Third, overall performance was quantified using Brier’s score, which measures how close predictions are to the actual outcome.

External validation of the admission and discharge prediction models were also performed using Cerner’s HealthFacts® data. HealthFacts® is a de-identified patient database that includes over 480 providers across the U.S. with a majority from the Northeast (44%), having more than 500 beds (27%), and are teaching facilities (63%). HealthFacts® encompasses encounter level demographic information, conditions, procedures, laboratory tests, and medication data. A sample was selected from HealthFacts® data consistent with the derivation dataset. The fit of both prediction models was assessed by applying the derivation coefficients, then recalibrating the coefficients with the same set of predictors and the coefficients by using the HealthFacts® sample. The performance between models was then compared.

Results

A total of 126,479 patients comprising 178,293 encounters met the cohort eligibility criteria, of which 18,652 (10.46%) encounters resulted in readmission to the same Advocate Health Care hospital within 30 days. After sampling, 9,151 (7.25%) encounter pairs were defined as 30-day readmissions. Demographic characteristics of the sample cohort are characterized in Table 1.

Table 1. Demographic Characteristics of the Sample Cohort.

Demographic Characteristics	30 Day Readmission		No Readmission
		n=9,151 (7.25%)		n=117,328 (92.75%)
Age	µ = 66.01		µ = 57.65
Gender
Female	5,045	(55.13)			70,917	(60.44)
Male	4,106	(44.87)			46,411	(39.56)
Race
Caucasian	5,737	(62.69)			71,796	(61.19)
African American	2,357	(25.76)			26,446	(22.54)
Hispanic	648	(7.08)			10,867	(9.26)
Other	409	(4.47)			8,219	(7.01)
Language
English	6,851	(94.26)			141,624	(93.29)
No English	417	(5.74)			10,187	(6.71)
Marital Status
Married	3,771	(41.21)			58,159	(49.57)
Not Married	5,380	(58.79)			59,169	(50.43)
Employment Status
Employed	991	(10.83)			23,073	(19.67)
Not Employed	4,930	(53.87)			46,973	(40.04)
Unknown	3,230	(35.30)			47,282	(40.30)
Insurance Type
Commercial	2,828	(30.90)			56,286	(47.97)
Medicare	5,118	(55.93)			44,187	(37.66)
Medicaid	778	(8.50)			8,352	(7.12)
Self-pay	378	(4.13)			6,751	(5.75)
Other	49	(0.54)			1,752	(1.49)

The ACC Admission Model included 49 independent predictors: Demographic, Utilization, Medications, Labs, H&P, and Exploratory variables. The ACC Discharge Model included 58 independent predictors comprising all the aforementioned variables plus Conditions, Procedures, Length of Stay (LOS), and Discharge Disposition. The variables included in the ACC Admission and Discharge Prediction Models are presented in Table 2.

Table 2. ACC Admission and Discharge Prediction Models’ Variables.

Variables	Admission Model (n=49)	Discharge Model (n=58)
Demographics	✓	✓
Utilization	✓	✓
Lab Tests	✓	✓
Exploratory	✓	✓
H&P	✓	✓
Medications	✓	✓
Conditions		✓
Procedures		✓
Length of Stay		✓
Discharge Disposition		✓

Assessment of the ACC Admission Model’s performance yielded C-statistics of 0.76 and 0.75, H&L goodness-of-fit tests of 36.0 (p<0.001) and 23.5 (p=0.0027), and Brier Scores of 0.062 (7.6% improvement from random prediction) and 0.063 (6.6% improvement from random prediction) from the derivation and internal validation datasets, respectively. Assessment of the ACC Discharge Model’s performance yielded C-statistics of 0.78 and 0.77, H&L goodness-of-fit tests of 31.1 (p<0.001) and 19.9 (p=0.01), and Brier Scores of 0.060 and 0.061 (9.1% improvement from random prediction) from the derivation and internal validation datasets, respectively. The average C-statistic for the ACC Admission Model was 0.76 and for the Discharge Model it was 0.78 after the 500 simulations in derivation dataset, resulting in a small range of deviation between individual runs.

External validation of the ACC Admission and Discharge model resulted in C-statistics of 0.76 and 0.78, H&L goodness-of-fit tests of 6.1 (p=0.641) and 14.3 (p=0.074), and Brier Scores of 0.061 (8.9% improvement from random prediction) and 0.060 (9.1% improvement from random prediction) after recalibrating and re-estimating the coefficient using Healthfacts® data, respectively. The ACC Admission and Discharge Models’ performance measures are represented in Table 3.

Table 3. ACC Admission and Discharge Prediction Model’s Performance Measures.

Dataset	Performance Measures	Admission Model	Discharge Model
Derivation (n=94,859)	Discrimination
	C-statistic	0.76	0.78
	Calibration
	Hosmer-Lemeshow goodness-of-fit test (p-value)	36.0 (p<0.001)	31.1 (p<0.001)
	Overall Performance
	Brier Score (% improvement)	0.062 (7.6%)	0.060 (9.1%)
	Bootstrapping
	500 simulations average (min. to max.)	0.76 (0.75 to 0.76)	0.78 (0.77 to 0.78)
Internal Validation (n=31,619)	Discrimination
	C-statistic	0.75	0.77
	Calibration
	Hosmer-Lemeshow goodness-of-fit test (p-value)	23.5 (p=0.003)	19.9 (p=0.01)
	Overall Performance
	Brier Score (% improvement)	0.063 (6.6%)	0.061 (9.1%)
External Validation (Without Recalibration) (n=6,357)	Discrimination
	C-statistic	0.69	0.71
	Calibration
	Hosmer-Lemeshow goodness-of-fit test (p-value)	216.9 (p<0.001)	156.3 (p<0.001)
	Overall Performance
	Brier Score (% improvement)	0.065 (2.5%)	0.064 (4.0%)
External Validation (With Recalibration) (n=6,357)	Discrimination
	C-statistic	0.76	0.78
	Calibration
	Hosmer-Lemeshow goodness-of-fit test (p-value)	6.1 (p=0.641)	14.3 (p=0.074)
	Overall Performance
	Brier Score (% improvement)	0.061 (8.9%)	0.060 (9.1%)

The probability thresholds for identifying high risk patients (11%), was determined by balancing the tradeoff between sensitivity (70%) and specificity (71%) by maximizing the area under ROC curves for the prediction models (Figure 4).

Figure 4.

ROC Curves for ACC Admission & Discharge Model

Discussion

We observed several key findings during the development and validation of our ACC Admission and Discharge Models. Both our all-cause models performed reasonably better than most predictive models reviewed in the literature used to identify patients at risk of readmission [27-29]. Both our models yielded a C-statistic between 0.7 and 0.8 during derivation, internal validation, and external validation after recalibration—a modest value for a clinical predictive rule. When comparing C-statistics between the Admission (C-statistic of 0.76) and Discharge Models (C-statistic 0.78), the Discharge Model’s discrimination ability improved because Conditions and Procedures, LOS, and Discharge Disposition variables were included; which helped further explain a patient’s readmission risk since medical conditions and surgical procedures accounts for immediate health needs, LOS represents severity of illness, and discharge disposition to a post-acute setting that doesn’t meet their discharge needs could result in a return to the hospital. We also observed the same C-statistic for our ACC Discharge Model on the development and external validation sample, suggesting that it performs well both in the intended population and when using a heterogeneous dataset. Our ACC Discharge Model also had a somewhat higher C-statistic during derivation when compared to the C-statistic observed during internal validation (C-statistic of 0.77), which is typically higher when assessing predictive accuracy using the derivation dataset to develop the model [21].

Our ACC Admission and Discharge Models also demonstrated reasonable model fit during external validation after recalibrating the coefficient estimates. A non-significant H&L p-value indicates the model adequately fits the data. However, caution must be used when interpreting H&L statistics because they are influenced by sample size [31]. Our models did not demonstrate adequate model fit during derivation and internal validation due to a large sample. Yet during external validation with a smaller sample, the H&L statistics for both the ACC Admission and Discharge Model improved to a non-significant level. Since the H&L statistic is influenced by sample size, the Brier Score should also be taken into account to assess prediction models because it captures calibration and discrimination features. The closer the Brier Score is to zero, the better the predictive performance [24]. Both our prediction models had low Brier Scores, with the ACC Discharge Model’s 0.06 representing consistent percent (9.1%) improvement over random prediction during derivation, internal validation, and external validation after recalibration.

There was concern that too many independent variables would increase the possibility of building an over-specified model that only performs well on the derivation dataset. Thus, validating a comprehensive model using an external dataset to replicate the derivation results would be challenging. Our findings indicate that our model’s performance slightly diminished during the ACC Admission Model’s external validation when compared with the more comprehensive ACC Discharge Model. When we externally validated our ACC Admission Model, the C-statistic was 0.74 on the development dataset, but reduced to 0.66 when using the initial derivation coefficients on the external dataset, and then increased to 0.70 after recalibrating the coefficients. The C-statistic for our ACC Discharge Model decreased from 0.78 on the development dataset, to 0.71 using the unchanged derivation coefficients, and then increased back to 0.78 after recalibration using the external validation sample dataset. It is expected to see performance decrease from derivation to validation, but our models had no more than 10% shrinkage from derivation to the validation results [32]. We further tested our ACC Admission Model using only baseline data available for a patient (e.g., demographic and utilization variables). The C-statistic for a more parsimonious admission model was 0.74 on the development dataset, decreased to 0.66 when using the derivation coefficients on the external dataset, but then increased to 0.70 after recalibrating the coefficients. Our findings suggest that including additional variables in the model is more likely to generalize better in comparison with a parsimonious model during external validation post-recalibration.

Overall, our Admission and Discharge Models’ performance indicates modest discrimination ability. While other studies relied on retrospective administrative data, our models incorporated data elements from the EHR. We utilized a mixed-method approach to evaluate clinically-related variables. Our models were internally validated in the intended population and externally validated for utility in heterogeneous populations. Our Admission Model offers a practical solution with data available during hospitalization. Our Discharge Model has a higher level of predictability according to the C-statistic and improved performance according to the Brier Score once more data is accessible during discharge.

Creating a highly accurate predictive model is multifaceted and contingent on copious factors, including, but not limited to, the quality and accessibility of data, the ability to replicate the findings beyond the derivation dataset, and the balance between a parsimonious and comprehensive prediction model. To facilitate external validation, we discovered that a compromise between a parsimonious and comprehensive model was needed when developing logistic regression prediction models. We also found that utilizing a mixed-method approach was valuable and additional efforts are needed when selecting risk factors that are of high-quality data, easily accessible, and generalizable across multiple populations. We also believe that bridging statistical acumen and clinical knowledge is needed to further develop decision support tools of genuine clinical utility, by soliciting support from clinicians when the statistics does not align with clinical intuition.

Limitations

Our findings should be considered under the purview of several limitations. There might be additional research conducted on readmission risk tools developed after the systematic review performed by Kansagara. Additional readmission risk prediction models were developed [33,34], but they did not publish their performance statistics to help us compare our prediction models.

Our readmission rate was limited to visits occurring at the same hospital. Readmission rates based on same-hospital visits can be unreliable and dilute the true hospital readmission rate [35]. One promising approach is using a master patient index (MPI) to track patients across hospitals. Data is being linked across hospitals and to outside facilities through MPI and claims data. Using our own method to create a MPI match, we performed some preliminary analysis and were able to identify 5% more readmissions across the other Advocate Health Care hospitals, increasing the readmission rate by approximately 1%. We also assessed the utility of claims data to match the other hospitals with Millennium® encounters to gauge a more representative readmission rate. The claims data allowed us to track approximately 8% more readmissions, increasing the readmission rate by approximately 1%. Overall, using both approaches we were able to identify a more representative readmission rate that increased from 10.46% to 12.5%. We are currently working to see how this impacts our models’ performance.

Data captured through EHRs is growing, but are incomplete with respect to data relevant to hospital readmission prediction and the lack of standard data representations limits generalizability of predictive models [36]. As a result, we could not include certain data elements into our models due to data quality issues, a large percentage of missing data, and since some of the information is difficult to glean. Therefore, we could not include social determinants identified by clinicians and care managers during qualitative interviews such as social isolation (i.e., living alone) and living situation (e.g., homelessness) known to be salient factors and tied to hospital readmissions [37,38]. Initially, we only mined a single source for this information in the EHR. However, new data sources have been identified in the EHR and the utility of these risk factors are currently being assessed in our prediction models. Additional factors are also being considered in our models such, as functional status [37,39], medication adherence and availability of transportation for follow-up visits post-discharge [40].

Our prediction models do not distinguish between potentially preventable readmissions (PPR) [41,42]. We did perform some preliminary analysis and found that the overall PPR readmission rate for Advocate Health Care in 2012 was about 6% of all admissions. We estimated around 60% of all readmissions were deemed avoidable. This is higher than the median proportion of avoidable readmissions (27.1%), but falls within the range of 5% to 79% [43]. We plan to further assess PPR methodology and test our models’ ability to recognize potentially avoidable readmissions to help intervene where clinical impact is most effective.

Our initial analysis plan proposed to include observation patients (n=51,517) in the entire inpatient cohort. We performed some preliminary analysis and found the overall readmission rate increased to 10.72%, but the C-statistics for our Admission and Discharge Models reduced to 0.75 and 0.77, respectively. Our models’ discrimination ability probably diminished due to improved logic needed in making a distinction in situations where observation status changes to inpatient and vice-versa. Further assessment of observation patients is needed to better understand their importance in an accountable care environment.

Steps are underway to mitigate limitations and continue to improve the clinical utility of our readmission risk prediction models. Data is being linked across hospitals and to outside facilities through MPI and claims data. Additional data sources in the EHR that encompass social determinants and other risk factors were identified are being assessed for use in our models. Also, we are researching potentially preventable readmissions so that the models can focus on cases where clinical impact is most needed.

Conclusions

The ACC Admission and Discharge Models exemplify design qualities of ideal prediction models. Both our models demonstrated modest predictive power for identifying high-risk patients early during hospitalization and at hospital discharge, respectively. Performance assessment of both our models during external validation post-recalibration indicates reasonable model fit and can be deployed in other population settings. Our Admission Model offers a practical and feasible solution with limited data available on admission. Our Discharge Model offers improved performance and predictability once more data is presented during discharge. The ACC partnership offers an opportunity to leverage proficiency from both organizations to improve and continue in the development of valuable clinical prediction models, building a framework for future prediction model development that achieves scalable outcomes.