Risk Prediction Models for Hospital Readmission: A Systematic Review

Abstract

Context

Predicting hospital readmission risk is of great interest to identify which patients would benefit most from care transition interventions, as well as to risk-standardize readmission rates for purposes of hospital comparison.

Objective

To summarize validated readmission risk prediction models, describe their performance, and assess suitability for clinical or administrative use.

Data Sources

MEDLINE, CINAHL, and Cochrane Library through March 2011, EMBASE through August 2011, and hand search of reference lists.

Study Selection

Dual review to identify English language studies of prediction models tested with medical patients, with both derivation and validation cohorts.

Data Extraction

Data were extracted on the population, setting, sample size, follow-up interval, readmission rate, model discrimination and calibration, type of data used, and timing of data collection.

Results

Of 7,843 citations reviewed, 30 studies of 26 unique models met criteria. The most common outcome used was 30-day readmission; only one model specifically addressed preventable readmissions. Fourteen models relying on retrospective administrative data could be potentially used for standardization of readmission risk and hospital comparisons; of these, nine were tested in large US populations and had poor discriminative ability (c-statistics 0.55 – 0.65). Seven models could potentially be used to identify high-risk patients for intervention early during a hospitalization (c-statistics 0.56 – 0.72), and five could be used at hospital discharge (c-statistics 0.68 – 0.83). Six studies compared different models in the same population and two of these found that functional and social variables improved model discrimination. Though most models incorporated medical comorbidity and prior utilization variables, few examined variables associated with overall health and function, illness severity, or social determinants of health.

Conclusions

Most current readmission risk prediction models, whether designed for comparative or clinical purposes, perform poorly. Though in certain settings such models may prove useful, efforts to improve their performance are needed as use becomes more widespread.

Introduction

An increasing body of literature attempts to describe and validate hospital readmission risk prediction tools. Interest in such models has grown for two reasons. First, transitional care interventions may reduce readmissions among chronically ill adults.^1-3 Readmission risk assessment could be used to help target the delivery of these resource-intensive interventions to the patients at greatest risk. Ideally, models designed for this purpose would provide clinically relevant stratification of readmission risk and give information early enough during the hospitalization to trigger a transitional care intervention, many of which involve discharge planning and begin well before hospital discharge. Second, there is interest in using readmission rates as a quality metric. Recently, the Centers for Medicare & Medicaid Services (CMS) began using readmission rate as a publicly reported metric, with plans to lower reimbursement to hospitals with excess risk-standardized readmission rates.⁴ Valid risk adjustment methods are required for calculation of risk-standardized readmission rates which could, in turn, be used for hospital comparison, public reporting, and reimbursement determinations. Models designed for these purposes should have good predictive ability; be deployable in large populations; use reliable data that can be easily obtained; and use variables that are clinically related to, and validated in, the populations in which use is intended.⁵

This systematic review was performed to synthesize the available literature on validated readmission risk prediction models, describe their performance, and assess their suitability for clinical or administrative use.

Methods

Data sources and searches

We searched Ovid MEDLINE, CINAHL, and the Cochrane Library (Central Trial Registry, Systematic Reviews, and Abstracts of Reviews of Effectiveness) from database inception through March 2011, and EMBASE through August 2011, for English-language studies of readmission risk prediction models in medical populations. All citations were imported into an electronic database (EndNote X2, Thomson Reuters, New York, NY). Appendix A provides the search strategies in detail.

Study selection

Seven investigators reviewed the citations and abstracts identified from electronic literature searches. Full-text articles of potentially relevant references were retrieved for further review. Each article was independently assessed by two reviewers using the eligibility criteria shown in Appendix B. Eligible articles were published in English and evaluated the ability of statistical models to predict hospital readmission risk. Because a set of predictive factors derived in only one population may lack validity and applicability,⁶ we included only studies of models that were tested in both a derivation and validation cohort, even if these results were presented in separate papers. We did not pre-specify the method of validation, nor did we exclude studies in which the derivation and validation cohorts were drawn from the same population (i.e., split-half validation). We did not limit studies by diagnosis within medical populations, but we excluded studies focused on psychiatric, surgical, and pediatric populations as factors contributing to readmission risk might be considerably different in these patient groups,. Finally, we excluded studies from developing nations as these were unlikely to provide directly applicable results.

Data extraction and quality assessment

From each study, we abstracted the following: population characteristics, setting, number of subjects in the derivation and validation cohorts, utilization outcome, readmission rate, range of readmission rates according to predicted risk, and model discrimination. To facilitate a high-level comparison of predictor variables, we grouped final model variables into one of six categories (medical comorbidity, mental health comorbidity, illness severity, prior utilization, overall health and function, and sociodemographic/social determinants of health).⁷

To characterize the practical utility of each model, two reviewers abstracted from each study the type of data used and the timing of data collection. Disagreements between reviewers about these classifications were resolved through group discussion. Data type consisted of administrative, primary (e.g., survey, chart review), or both. Regarding timing, we classified a model as using real-time data if the variables would be available on or shortly after index hospital admission, and as using retrospective data if the variables would not be available early during a hospitalization. For example, a model using prior healthcare utilization and data from patient surveys conducted early during a hospitalization would be classified as using real-time data, while a model using index hospital length of stay or index hospital discharge diagnostic codes would be classified as using retrospective data. Because of coding delays, models relying on administrative codes from index hospital admission were considered retrospective.

We report the c-statistic, with 95% confidence interval when available, to describe model discrimination. The c-statistic, which is equivalent to the area under the receiver operating characteristic curve, is the proportion of times the model correctly discriminates a pair of high- and low-risk individuals.⁸ A c-statistic of 0.5 indicates the model performs no better than chance; a c-statistic of 0.7 to 0.8 indicates modest or acceptable discriminative ability, and a threshold of greater than 0.8 indicates good discriminative ability.^{9, 10} If the c-statistic was not reported, we abstracted other operational statistics such as sensitivity, specificity and predictive values for representative risk score cut-offs when available. Model calibration is the degree to which predicted rates are similar to those observed in the population. To describe model calibration we report the range of observed readmission rates from the predicted lowest to highest risk groupings.

To guide our methodologic assessment of included studies, we adapted elements – including cohort definition, follow-up, adequacy of prognostic and outcome variable measurement, and validation method – from a prognosis study quality tool and clinical decision rule assessment tool (Appendix C).^{6, 11}

Data synthesis

The included studies were too heterogenous to permit meta-analysis. Therefore, we qualitatively synthesized results, focusing on model discrimination, the populations in which the model has been tested, practical aspects of model implementation, and the types of variables included in each model.

Results

From 7,843 titles and abstracts, 286 articles were selected for full-text review (Figure available as online supplement). Of these, 30 studies of 26 unique models across a broad variety of settings and patient populations met our inclusion criteria (Table 1). Most (N=23) studies were based on US healthcare data. The remainder were from Australia (2 studies), England (2), Ireland (1), Switzerland (1), or Canada (1). Fourteen studies included only patients at least 65 years of age. Of these, seven relied solely on Medicare administrative data. Four studies used VA data.

Table 1. Characteristics of validated readmission risk prediction models.

Study	Population	Setting	No. of patients, derivation cohort	No. of patients, validation cohort*	Utilization outcome†	Actual readmission rate (% of patients)		Range of readmission rates according to predicted risk (validation cohort)	Model discrimination (c-statistic‡ unless specified otherwise)
						Derivation cohort	Validation cohort
Models relying on retrospective administrative data
Anderson, 198522	Medicare patients(excluded ESRD pts), 1974-1977	US, general population	21043	10522	60-day readmissions	NR	NR	4 – 40 (lowest to highest decile)§	NR
Bottle, 200612	Inpatients, 2000-2001	England, general population	∼1373755║	∼1373754║	12-month readmissions	9.80 overall		---	All patients: 0.72 Patients with ambulatory care sensitive conditions¶: 0.75 All patients (12 month deaths excluded): 0.70
CMS model, AMI Krumholz 200816	Medicare AMI patients ≥ 65 yr, 2005-2006	US, general population	100465	100285	30-day readmissions	18.9	19.2	8. 0 – 33.0 (lowest to highest decile)	0.63
CMS model, CHF Krumholz, 200815	Medicare CHF patients ≥ 65 yr, 2003-2004	US, general population	283919	283528	30-day readmissions	23.6	23.7	15.0 - 37.0 (lowest to highest decile	0.6
CMS model, Pneumonia Krumholz, 200817	Medicare pneumonia patients ≥ 65 yr, 2005-2006	US, general population	226545	226706	30-day readmissions	17.4	17.5	9.0 – 31.0 (lowest to highest decile)	0.63
Halfon, 200613	All hospitalizations in year 2000	Switzerland, general population	65740	66069	30-day potentially avoidable readmissions	5.1	5.2	---	Nonclinical: 0.67 Charleson based: 0.69 SQLape: 0.72
Hammill, 201118	CHF registry patients ≥ 65 yr, 2004-2006	US. general population	24163#		30-day readmissions	21.9 overall		Claims-only: 14.4 – 32.7 (lowest to highest decile) Claims-clinical: 13.5 – 33.9	Claims-only: 0.59 Claims-clinical: 0.60
Holloway, 199053	Medical, neurologic, surgical, and geriatric inpatients, 1981-1982	US, single VA hospital	2970	unclear	30-day readmissions	22.0 overall		---	NR
Holman, 200514	Medical, surgical, psychiatric inpatients, 1989-1997	Western Australia, general population	326,456	5289 (asthma) 5265 (AMI)	30-day readmissions	NR	NR	---	Asthma 0.71 AMI 0.64
Howell, 200954	General medical inpatients with ambulatory care sensitive condition¶ 2005-2006	Queensland, Australia, general population	13207	4492	12-month readmissions	45.5	45.1	LR+ readmission for risk scores 50, 70, 80: 2.04, 3.11, 7.02 (overall range 0 – 100)	0.65
Naessens, 199223	Inpatients ≥ 65 yr, 1980, 1985, and 1987	US, general population in a single county	5854	randomly selected10% of derivation cohort	60-day mortality/readmissions	20.8 overall		15.6 – 36.0 (lowest to highest quartile)	HCFA alone 0.59 (SE=0.01) HCFA + COMPLEX 0.61 (SE=0.01)
Philbin, 199919	CHF inpatients, 1995	US, multicenter in a single state	21227	21504	CHF readmissions within calendar year	21.3 overall		9.8 – 45.4 (lowest to highest ninth)	Simple scoring system: 0.60 Weighted scoring system: 0.61
Silverstein, 200820	Inpatients ≥ 65 yr, 2002-2004	US, multicenter in a single city	19528	9764	30-day readmissions	11.7 overall		---	0.65 (same for both Elixhauser and HRDES methods)
Thomas, 199621	Medicare inpatients ≥ 65 yr, 1989-1991	US, multicenter in a single state	12 different cohorts based on diagnosis; range 1163-14590		15-, 30-, 60-, and 90-day readmissions	3 - 40** overall		---	among 8 medical conditions and 4 time periods, c statistic ranged from 0.55-0.61
Models using administrative data in real time
Amarasingham, 201024	CHF patients, 2007-2008	US, single center	1029	343	30-day readmissions	24.1 overall		12.2 – 45.7 (lowest to highest quintile)	0.72 (0.70-0.75)
Billings, 200725	Patients eligible for mandatory Medicaid managed care enrollment, 2000-2004	US, general population in a single city	∼35000║	∼35000║	12-month readmissions	NR	NR	NR (inpatient costs ranged 23,687 – 44,385 for risk scores 50-90, overall range 0 – 100)	Risk scores range 0-100 Using risk score 50+, Sens 58%, Spec 74%, PPV 69.5%, LR+ 2.23
PARR model Billings, 200626	Inpatients with an ambulatory care sensitive reference condition¶ 2002-2003	England, general population	10% of hospital episodes for all England	A second 10% sample of hospital episodes for all England	12-month readmissions	NR	NR	---	0.69
Models using retrospective primary data collection
Coleman, 200427	Medicare inpatients ≥ 65 yr, 1997-1998	US, general population	700	704	30-day “complicated care transitions”††	21.9	25.0	---	administrative data model: 0.77colspan="11"administrative + self-report data: 0.83
Krumholz, 200030	Medicare CHF patients ≥ 65 yr, 1994-1995	US, multicenter in a single state	1129	1047	180-day readmissions	50.0	47.0	All-cause: 26.0 – 59.0 CHF: 9.0 – 31.0 (lowest to highest tertile)	Number of risk factors associated with readmission risk (P<0.0001). 0 risk factors: 26% 3-4 risk factors: 59%
Morrissey, 200328	Medical inpatients ≥ 65 yr, 1997-1998	Ireland, single rural hospital	487	732	12-month readmissions	40.7	29.0	---	0.70
Smith Index (original) Smith, 198555	Medical inpatients, 1979-1980	US, single county hospital	1007	499	90-day readmissions	16.9	NA	7.3 – 38.0 (lowest to highest octile)	Sens 59.0%, Spec 69.3%, PPV 29.9% LR+ 1.92
Smith Index validation Smith, 198856	Medical inpatients, 1985	US, single county hospital	502 (control) 499 (intervention)		Readmissions/month/patient (mean 180 days f/u)	NA	10.0	0.07 – 0.18 (lowest to highest tertile)	NR
Smith Index validation Smith, 199657	Medical inpatients ≥ 45 yr, 1988-1990	US, single VA hospital		662 (validation)	90-day readmissions	NA	20.1	---	0.66
Van Walraven, 201029	Medical and surgical inpatients	Canada, multicenter	4812 patients — split derivation/internal validation	1M patients from Discharge Abstract Database for external validation	30-day readmissions	7.3	7.3	0 – 42.9 (scores 0 – 17, footnote – corresponding to expected probability of readmission/death of 2.0 – 34.6%)	0.68 (0.65-0.71)
Models using primary data collected in real time
Burns, 199158	Medical inpatients ≥ 65 yr, 1987	US, single VA hospital	134	34	60-day readmissions	30.6 overall		---	NR
Evans, 198859	Medical, neurologic, and surgical inpatients over a 6 week period	US, single VA hospital	532	177	Composite of 60-day readmission, nursing home placement, or LOS longer than expected per mean LOS of DRG	21.0 overall (60-day readmissions)		% high-care users: 34.7 – 91.7 (lowest to highest eighth)	Risk score range 0-8Score >= 3: Sens 0.60, Spec 0.76, LR+ 2.5Score >= 4: Sens 0.42, Spec 0.93, LR+ 6
Hasan, 200931	Medical inpatients, 2001-2003	US, multicenter	7287	3659	30-day readmissions	17.5	17.4	5.9 – 28.9 (lowest to highest quartile)	0.61
PRA (original) Boult, 199332	Non-institutionalized Medicare patients ≥ 70 yr, 1984	US, general population	2942	2934	4 year readmissions	28.4	NA	26.1 (score 0-3) – 41.8 (score 4+)	0.61 (SE=0.01)
PRA validation Allaudeen, 201134	Medical inpatients ≥ 65 yr, 5 week period in 2008	US., single academic center	NA	159	30-day readmissions	NA	32.7	---	PRA 0.56 (0.44-0.67) Prediction by physician 0.58-0.59 (0.46-0.70) Prediction by non-physician provider 0.50-0.55 (0.38 – 0.67)
PRA validation Novotny, 200833	Medical inpatients, 2005-2007	US, single academic center	1077		41-day readmissions	NA	14.0	---	PRA score 0.53 cutpoint, LR+ 1.67

Abbreviations: DRG denotes Diagnosis Related Group; LR+, Positive Likelihood Ratio; NA, Not Applicable; NR, Not Reported; PARR, Patients at Risk for Re-hospitalization algorithm; PRA, Probability of Repeated Admissions; SE, Standard Error.

^*The most recent validation cohort is listed if a study had multiple validation cohorts.

^†Unplanned, all-cause readmissions unless otherwise specified

^‡Validation cohort values for the c-statistic are listed if a study provided c-statistic values for both validation and derivation cohorts. 95% confidence interval is provided in parentheses, if reported.

^§Approximate values of data presented in a bar graph.

^║The total number of subjects was divided equally between the derivation and validation cohorts, but the exact numbers were not specified.

^¶Reference conditions such as congestive heart failure, chronic obstructive pulmonary disease, diabetes, and asthma, for which timely and effective case-management has the potential to reduce the risks of readmission.

^#Used bootstrap method for internal validation, no separate validation cohort

^**Reports 15-, 30-, 60-, and 90-day readmission rates for 12 different conditions

^††At least one transfer from lower to higher intensity care environment

Total sample size ranged from just 173 patients to more than 2.7 million. The outcome of 30-day readmission was reported most commonly, though some models chose other follow-up intervals ranging from 14 days to 4 years. Among 21 studies reporting a c-statistic, values ranged from 0.55 – 0.83 (Table 1), but only six studies reported a c-statistic above 0.70 indicating modest discriminative ability. Performance was similar between studies using split-sample validation methods (n=21, c-statistic range 0.59-0.75), and those that used external validation methods (n=9, c-statistic range 0.53-0.83). Among models that analyzed the relationship between risk categories and actual readmission rates, a substantial gradient in readmission rate was present between patients at the lowest vs. the highest risk level. For example, among six models using 30-day readmission as an outcome, the lowest and highest risk groups differed by 20.4 to 34.5 percentage points in their actual readmission rates.

Models relying on retrospective administrative data

Fourteen models were based on retrospective administrative data and could potentially be used for hospital comparison purposes (Table 1). Most of these included medical comorbidity and prior utilization variables, but few considered mental health, functional status and social determinant variables (Table 2). The three models with c-statistics ≥ 0.70 were developed and tested in large European or Australian cohorts. One examined the risk of two or more unplanned readmissions for all hospitalized patients in England, including pediatric and obstetric patients, for one calendar year.¹² A Swiss study of potentially preventable readmissions is described in greater detail below.¹³ An Australian model incorporating over 100 medical comorbidities and administrative social determinant variables performed at a modest level in asthma patients, but poorly in myocardial infarction patients.¹⁴

Table 2. Variables considered by studies in evaluating the risk of readmission.

Variable	Included in final model in (N) studies	Evaluated but not included in (N) studies	Not considered* in (N) studies
Medical comorbidities
Specific diagnoses or comorbidity index	(24) 12-21, 23, 25-32, 53, 54, 57-59	(0)	(3) 22, 24, 55
Mental health comorbidities
Mental illness	(9) 13-15, 17, 18, 24, 25, 54, 59	(4) 16, 20, 26, 58	(11) 19, 22, 23, 28-32, 53, 55, 57
EtOH/substance use	(11) 13-15, 17-19, 24-26, 53, 54	(5) 16, 20, 28, 57, 59	(8) 22, 23, 29-32, 55, 58
Illness severity
Illness severity index	(1) 24	(1) 58	(19) 12, 13, 15-18, 20, 23, 26, 28-32, 53-55, 57, 59
Lab findings	(4) 18, 30, 55, 57	(1) 28	(15) 12, 13, 15-17, 20, 23, 26, 29, 31, 32, 53, 54, 58, 59
Other†	(4) 2, 3, 20, 2	(4) 18, 30, 57, 59	(11) 15-17, 20, 26, 28, 29, 31, 32, 54, 55
Prior utilization
Hospitalizations	(14) 12, 13, 22, 24-28, 30-32, 54, 58, 59	(1) 29	(10) 15-20, 23, 53, 55, 57
ER visits	(4) 25, 29, 55, 57	(1)24	(17) 15-20, 22, 23, 26, 28, 30-32, 53, 54, 58, 59
Clinic visits/Missed clinic visits	(3) 24, 25, 32	(0)	(19) 15-20, 22, 23, 26, 28-31, 53-55, 57-59
Index hospital length of stay	(4) 19, 21, 29, 31	(3) 30, 53, 58	(15) 15-18, 20, 22-24, 26, 28, 32, 54, 55, 57, 59
Overall health and function
Functional status; ADL dependence; mobility	(2) 27, 57	(6) 29-32, 58, 59	(14) 15-20, 22-24, 26, 28, 53-55
Self-rated health, quality of life	(3) 27, 31, 32	(2) 28, 57	(17) 15-20, 22-24, 26, 29, 30, 53-55, 58, 59
Cognitive impairment	(7) 15-18, 28, 57, 59	(5) 20, 31, 32, 54, 58	(9) 19, 22-24, 26, 29, 30, 53, 55
Visual or hearing impairment	(1) 27	(1) 32	(21) 15-20, 22-24, 26, 28-32, 53-55, 57-59
Sociodemographic factors
Age	(19) 12-18, 20-23, 25-27, 32, 53, 54, 57, 59	(7) 19, 24, 29-31, 55, 58	(1) 28
Gender	(15) 12-18, 20-26, 32	(8) 19, 29-31, 53-55, 58	(1) 28
Race/ethnicity	(7) 12, 14, 19, 20, 22, 25, 26	(8) 24, 30-32, 54, 55, 57, 58	(8) 15-18, 23, 28, 29, 53
Social determinants of health
SES/income/employment status	(5) 12, 14, 24, 25, 54	(7) 20, 26, 31, 32, 57-59	(10) 15-19, 22, 23, 28, 29, 53
Insurance status‡	(6) 19, 20, 24, 27, 31, 53	(1) 57	(5) 30, 32, 55, 58, 59
Education	(0)	(4) 28, 31, 32, 58	(17) 15-20, 22-24, 26, 29, 30, 53-55, 57, 59
Marital status/# of people in home	(4) 24, 28, 31, 59	(6) 29, 32, 53, 54, 57, 58	(11) 15-20, 22, 23, 26, 30, 55
Caregiver availability, other social support	(2) 32, 57	(1) 31	(19) 15-20, 22-24, 26, 28-30, 53-55, 57-59
Access to care/rurality	(5) 19, 22, 31, 53, 54	(2) 20, 29	(14) 15-18, 23, 24, 26, 28, 30, 32, 55, 57-59
Discharge location (home, NH)	(2) 19, 20	(1) 53	(18) 15-18, 22-24, 26, 28-32, 54, 55, 57-59

^*Six studies did not report candidate variables and only reported the final model.^{12-14, 21, 25, 27}

^†Examples include use of telemetry, shock, planned vs emergent index hospitalization, heart rate, ejection fraction.

^‡This category is not relevant to studies of Medicare patients^{15-18, 23} and non-US studies.^{12, 13, 28, 29, 54}

The nine large population-based or multicenter US studies generally had poor discriminative ability (c-statistics 0.55 – 0.65). The CMS used a methodologically rigorous process to create three models for congestive heart failure, acute myocardial infarction, and pneumonia admissions based on Hierarchical Condition Categories, which are groups of related comorbidities.^15-17 All three models showed relatively poor ability to predict 30-day all cause readmissions (c-statistics 0.61, 0.63, and 0.63, respectively). A recent study evaluating the CMS heart failure model, and an older heart failure model fared similarly (c-statistics 0.59 and 0.61, respectively).^{18, 19} The other four US models have limited generalizability: one captured readmissions to one medical center only,²⁰ and the others were developed over two decades ago.^21-23

Models using real-time administrative data

Three administrative data-based models were designed to identify high-risk patients in real-time to potentially facilitate targeted interventions. A model with modest discriminative ability (c-statistic 0.72, 95% CI 0.70-0.75) examined 30-day heart failure readmissions in a single urban US health system with a large socioeconomically disadvantaged population.²⁴ It incorporated variables from an automated electronic medical record system, including numerous social factors such as number of address changes, census tract socioeconomic status, history of cocaine use, and marital status. The only study focused specifically on Medicaid enrollees used a 0 to 100 risk score for 12-month readmissions and found patient cost profiles varied widely with risk score.²⁵ Finally, a British model used prior utilization and comorbidity data, and also controlled for observed to expected readmission rates for the admission hospital, but predictive ability remained modest (c-statistic 0.69).²⁶

Models incorporating primary data collection

Nine models incorporated survey or chart review data and could potentially be used for clinical intervention purposes, though five used data unlikely to be available early during a hospitalization. The best performing of these used administrative comorbidity and prior utilization data (c-statistic 0.77) along with functional status data (c-statistic 0.83) from the Medicare Beneficiaries Survey to predict a composite outcome of readmissions and nursing home transfers.²⁷ The survey was not routinely administered during index hospitalization and it is unclear to what extent the use of retrospective survey data affects the predictive ability of the model. Similarly, a medical record study in Ireland retrospectively applied a nine-item questionnaire, including items such as discharge polypharmacy, and performed modestly well (c-statistic 0.70).²⁸ A simple Canadian model used medical comorbidities up through index hospital discharge along with index hospital length of stay and prior utilization (c-statistic 0.68, 95%CI 0.65-0.71).²⁹ Increasing scores on another four-item model of medical comorbidities, prior utilization and discharge creatinine were associated with increasing readmission rates in heart failure patients.³⁰

Four models incorporated primary data collected in real-time. Only two of these models have been tested in contemporary populations, the others having been conducted more than two decades ago. One survey-based model developed at six academic hospitals included social determinant, comorbidity, utilization, and self-rated health variables, but had poor predictive ability (c-statistic 0.61).³¹ The Probability of Repeated Admissions (PRA) is a simple eight-item survey tool developed in older Medicare beneficiaries, but it also had poor predictive ability across several studies (c-statistic 0.56–0.61, 95% CI 0.44-0.67).^32-34

Use of variables

A comparison of the types of variables considered for, and included in, the final models can provide some information about the contribution of different types of variables to readmission risk prediction (Table 2). Nearly all studies included medical comorbidity data and many included prior utilization variables, usually prior hospitalizations. Basic sociodemographic variables such as age and gender were considered by most studies but, in many instances, these variables did not contribute enough to be included in the final model. Table 2 also highlights important gaps in model development: few studies considered variables associated with illness severity, overall health and function, and social determinants of health.

Six studies that compared the performance of different models within the same population offer further insights about the incremental value of different types of variables (Table 3). Amarasingham and colleagues found that an automated electronic medical record-based model incorporating sociodemographic factors such as drug use and housing discontinuities, was more predictive than comorbidity-based models.²⁴ Coleman and colleagues found the inclusion of variables such as functional status from survey data improved model performance slightly compared to the use of utilization and comorbidity-based administrative data alone (c-statistics 0.83 vs 0.77).²⁷

Table 3. Studies that compared models within a population.

Study and models compared	Model description	C-statistic (95% CI or SE if reported)
Halfon, 200613
Nonclinical model	Age, sex, prior utilization	0.67
Modified Charlson score based model	Charlson score60 plus prior utilization	0.69
Modified SQLape model61	Complex administrative model combining comorbidity, age, and utilization data into 49 risk categories	0.72
Hammill 201118
Claims-only model	CMS administrative heart failure model15	0.59
Claims-clinical model	CMS heart failure model + serum creatinine, serum sodium, hemoglobin, systolic blood pressure	0.60
Allaudeen, 201134
PRA*32	Age, sex, self-rated health, availability of informal caregiver, coronary disease, diabetes, hospital admission within past year, prior utilization	0.56 (0.44-0.67)
Prediction by physician	Interns, residents, and attending physicians predicted chance of readmission based on overall evaluation of patient	0.58-0.59 (0.46-0.70)
Prediction by non-physician provider	Nurses and case managers predicted chance of readmission based on overall evaluation of patient	0.50-0.55 (0.38-0.67)
Amarasingham, 201024
ADHERE mortality model	Blood urea nitrogen, creatinine, and systolic blood pressure	0.56 (0.54-0.59)
CMS heart failure model15	Complex administrative comorbidity model consisting of age, sex, and 35 hierarchical condition categories	0.66 (0.63-0.68)
Tabak mortality model62	Age, 17 lab and vital sign variables within 24 hours of hospital presentation	0.61 (0.59-0.64)
Electronic readmission model	Includes Tabak mortality score, history of depression or anxiety, single status, sex, residential stability, Medicare status, residence census tract in lowest socioeconomic quintile, history of confirmed cocaine use, history of missed clinic visit, use of a health system pharmacy, number of prior admissions, presented to emergency department between 6 am and 6 pm for index admission.	0.72 (0.70-0.75)
Coleman, 200427
Administrative model	Age, sex, prior utilization, Medicaid status, Charlson score,60 heart disease, cancer, diabetes	0.77
Administrative + self-report model	Administrative model + self-rated health, ADL assistance need, visual impairment, functional status	0.83
Naessens, 199223
Modified Health Care Financing Administration (HCFA) mortality model63	Age, sex, 16 DRG, and 8 comorbidities	0.59 (SE=0.01)
HCFA + COMPLEX	Complicated administrative model incorporating DRG based disease staging and number of body systems affected + HCFA	0.61 (SE=0.01)

Abbreviations: ADHERE denotes Acute Decompensated Heart Failure Registry; CI, Confidence Interval; CMS, Center for Medicaid and Medicare Services; COMPLEX, a measurement of comorbidity and disease severity;²³ HCFA, Health Care Financing Administration; IDI, Integrated Discrimination Improvement; PRA, Probability of Repeated Admission; SE, Standard Error.

^*Variables were obtained from chart abstraction, whereas original PRA instrument is based on patient surveys.

Other comparative studies found little difference among models.Clinical data, such as laboratory and physiologic variables, from medical records or registries did not enhance performance of claims-only CMS models.^{15-17, 28} A US study of older patients found that an intricate ICD-9 code based disease complexity system added very little discriminative ability to a poorly performing Health Care Financing Authority model.²³ A large Swiss study of potentially preventable readmission risk compared a very simple non-clinical model, a Charlson comorbidity-based model, and a more complex hierarchical diagnosis and procedures based model called SQLape, finding only slight differences among them (c-statistics 0.67, 0.69, and 0.72, respectively).¹³ Finally, Allaudeen and colleagues found internal medicine interns using a gestalt approach predicted readmissions with a similar poor level of ability as an older, established survey-based model (PRA) in a small, single center cohort.³⁴

Potentially preventable readmissions

Only one model attempted to explicitly define and identify potentially preventable readmissions.³⁵ Investigators conducted a systematic medical record review to define potentially preventable readmissions and develop an administrative data-based algorithm. A subsequent publication (described above) compared the performance of three models in predicting readmissions according to their algorithm.¹³

Discussion

In this systematic review, we found 26 readmission risk prediction models of medical patients tested in a variety of settings and populations. Several are being applied currently in clinical, research or policy arenas. Half the models were largely designed to facilitate calculation of risk-standardized readmission rates hospital comparison purposes. The other half were clinical models that could be used to identify high-risk patients for whom a transitional care intervention might be appropriate. Most models in both categories have poor predictive ability.

Readmission risk prediction remains a poorly understood and complex endeavor. Indeed, models of patient level factors such as medical comorbidities, basic demographic data, and clinical variables are much better able to predict mortality than readmission risk.^{18, 24, 29} Broader social, environmental, and medical factors such as access to care, social support, substance abuse, and functional status contribute to readmission risk in some models, but the utility of such factors has not been widely studied.

It is likely that hospital and health system-level factors, which are not present in current readmission risk models, contribute to risk.³⁶ For instance, the timeliness of post-discharge follow-up, coordination of care with the primary care physician, and quality of medication reconciliation may be associated with readmission risk.^{37, 38} The supply of hospital beds may independently contribute to higher readmission rates.³⁹ Finally, the quality of inpatient care could also contribute to risk,⁴⁰ though the evidence is mixed.⁴¹ Though the inclusion of such hospital-level factors would conceivably improve the predictive ability of models, it would be inappropriate to include them in models that are used for risk-standardization purposes. Doing so would adjust hospital readmission rates for the very deficits in quality and efficiency that hospital comparison efforts seek to reveal, and which could be targets for quality improvement interventions.

Public reporting and financial penalties for hospitals with high 30-day readmission rates are spurring organizations to innovate and implement quality improvement programs.^{42, 43} Nevertheless, the poor discriminative ability of most of the administrative models we examined raises concerns about the ability to standardize risk across hospitals in order to fairly compare hospital performance. Until risk prediction and risk adjustment become more accurate, it seems inappropriate to compare hospitals in this way and reimburse (or penalize) them on the basis of risk-standardized readmission rates. Others have reached similar conclusions,⁴⁴ and have also expressed concern that such financial penalties could exacerbate health disparities by penalizing hospitals with fewer resources.⁴⁵ Still others have argued that readmission rate is an incomplete accountability measure that fails to consider “the real outcomes of interest – health, quality of life, and value.”⁴⁶

Use of readmission rates as a quality metric assumes that readmissions are related to poor quality care and are potentially preventable. However, the preventability of readmissions remains unclear and understudied. We found only one validated prediction model that explicitly examined potentially preventable readmissions as an outcome, and it found only about one-quarter of readmissions were clearly preventable.¹³ A recent systematic review of 34 studies found wide variation in the percentage of readmissions considered preventable; estimates ranged from 5% to 79%, with a median of 27%.⁴⁷ More work is needed to develop readmission risk prediction models with an outcome of preventable readmissions. This could not only improve risk-standardization efforts, but also allow hospitals to better focus limited clinical resources in readmission avoidance programs.

As with models that are used for risk-standardization, readmission risk models that are intended for clinical use also have certain requirements and limitations. Clinical models would ideally provide data prior to discharge, discriminate high- from low-risk patients, and would be adapted to the settings and populations in which they are to be used. Very few models met all these criteria, and only one of these – a single-center study – had acceptable discriminative ability.²⁴ As with the risk-adjustment models, most of the models developed for clinical purposes had poor predictive ability, though notable exceptions suggest the addition of social or functional variables may improve overall performance.^{24, 27}

The best choice of model may depend on setting and the population being studied. The success of some models in certain populations and the lack of success of others suggest the patient-level factors associated with readmission risk may differ according to the population studied. For example, while medical comorbidities may account for a large proportion of risk in some populations, social determinants may disproportionately influence risk in socioeconomically disadvantaged populations. Our review finds, though, that very few models have incorporated such variables.

Even though the overall predictive ability of the clinical models was poor, we did find that high- and low-risk scores were associated with a clinically meaningful gradient of readmission rates. This is important given resource constraints and the need to selectively apply potentially costly care transition interventions. Even limited ability to identify a proportion of patients at risk for future high-cost utilization can increase the cost-effectiveness of such programs.^{26, 48}

Of note, very few models incorporated clinically actionable data that could be used to triage patients to different types of interventions. For example, marginally housed patients, or those struggling with substance abuse, might require unique discharge services. Relatively simple, practical models that use real-time clinically actionable data, such as the Project BOOST model, have been created, but their performance has not yet been rigorously validated.⁴⁹

Our review concurs with and adds to the findings of several other reviews that found deficiencies in the predictive abilities of risk prediction models. One recent review limited to US studies examined general risk factors for preventable readmissions, but did not search explicitly for validated models, and many of the included studies suffered from poor study design.⁵⁰ The authors suggest that, in general, measures of poor health such as comorbidity burden, prior utilization, and increasing age were associated with readmissions. Two other reviews focused on specific diagnoses and found very few readmission risk models for heart failure,⁴⁴ COPD,⁵¹ or myocardial infarction.⁵²

Our review has certain limitations. We included studies outside the United States, given that portions of US health care may resemble other countries' health systems, but applicability of models from other countries to the US may still be limited. Our classifications of data types, data collection timing, and the intended use of each model, are subject to interpretation, but we attempted to mitigate subjectivity by using a dual-review and consensus process. Finally, few studies directly compared models within the same population, and summary statistics such as the c-statistic should not be used to directly compare models across different populations.

Additional research is needed to assess the true preventability of readmissions in US health systems. Given the broad variety of factors that may contribute to preventable readmission risk, models that include factors obtained through medical record review or patient report, may be valuable. Innovations to collect broader variable types for inclusion in administrative data sets should be considered. Future studies should assess the relative contributions of different types of patient data (e.g., psychosocial factors) to readmission risk prediction by comparing the performance of models with and without these variables in a given population. These models should ideally be based on population specific conceptual frameworks of risk. Implementation of risk stratification models and their effect on work flow and resource prioritization should be assessed in a broad variety of hospital settings. Also, given that many models have limited predictive ability and may require some investment of time and cost to implement, future studies should further evaluate the relative value of clinician gestalt compared to predictive models in assessing readmission risk.

In summary, readmission risk prediction is a complex endeavor with many inherent limitations. Most models created to date, whether for hospital comparison or clinical purposes, have poor predictive ability. Though in certain settings such models may prove useful, better approaches are needed to assess hospital performance in discharging patients, as well as to identify patients at greater risk of avoidable readmission.

Figure 1. Risk Prediction Models for Hospital Readmission - Literature Flow.