International Validity of the “HOSPITAL” Score to Predict 30-day Potentially Avoidable Readmissions in Medical Patients

Abstract

IMPORTANCE

Identification of patients at high risk of potentially avoidable readmission allows hospitals to efficiently direct additional care transitions services to the patients most likely to benefit.

OBJECTIVE

To externally validate the “HOSPITAL” score in an international multicenter study to assess its generalizability.

DESIGN

International multicenter retrospective cohort study.

SETTING

9 large hospitals across 4 different countries.

PARTICIPANTS

All adult patients consecutively discharged alive from a medical department between January and December, 2011 (117,065 participants). Patients transferred to another acute care facility were excluded.

EXPOSURES

The “HOSPITAL” score includes the following predictors at discharge: Hemoglobin, discharge from an Oncology service, Sodium level, Procedure during the index admission, Index Type of admission (urgent), number of Admissions during the last 12 months, and Length of stay.

MAIN OUTCOMES AND MEASURES

30-day potentially avoidable readmission to the index hospital using the SQLape algorithm.

RESULTS

Of all medical discharges, 14.5% (n=16,992) were followed by a 30-day readmission, and 9.7% (n=11,307) were followed by a 30-day potentially avoidable readmission. The discriminatory power of the “HOSPITAL” score to predict potentially avoidable readmission was good with a C-statistic of 0.72 (95% CI 0.72-0.72). As in the derivation study, patients were classified into 3 risk categories: low (62%), intermediate (24%), and high risk (14%). The estimated proportions of potentially avoidable readmission for each risk category matched the observed proportion, resulting in an excellent calibration (Pearson goodness of fit test P=0.89).

CONCLUSIONS AND RELEVANCE

The “HOSPITAL” score identified patients at high risk of 30-day potentially avoidable readmission with moderately high discrimination and excellent calibration when applied to a large international multicenter cohort of medical patients. This score has the potential to easily identify patients in need of more intensive transitional care interventions to prevent avoidable hospital readmissions.

INTRODUCTION

Hospital readmissions are frequent, harmful and costly— 18% of Medicare patients can expect to be readmitted within 30 days in the US at a cost of more than 17 billion dollars.¹ Recent changes in healthcare policy aimed at reducing readmission have substantially increased attention to this major healthcare issue.

Although some readmissions are unavoidable, and the proportion of readmissions that are truly avoidable remains controversial, at least some appear to be preventable, therefore leaving room for improvement by hospitals and health care systems.^2,3 A recent systematic review and meta-analysis showed a relative readmission risk reduction of 0.82 from transitions interventions (p<0.01), however the most effective interventions were of high complexity.⁴ In order to most efficiently reduce hospital readmissions, hospitals need to target complex and intensive discharge interventions at those patients at high risk of potentially avoidable readmission and therefore more likely to benefit. However, prior research indicates that clinical providers are not able to accurately identify which patients are at high risk for readmission.⁵

Prediction models may provide a more efficient and accurate means to identify those patients at highest risk. Unfortunately, most existing predictions models have yielded only fair discriminative ability, do not focus on readmissions that are avoidable, are not externally validated, or are not easy to use in practice.⁶ In response to these shortcomings, we previously derived and internally validated the “HOSPITAL” score consisting of 7 readily available clinical predictors (Table 1 lists components forming the acronym). This simple prediction model accurately identified patients at high risk of potentially avoidable readmissions among more than 10,000 medical inpatients,⁷ and is innovative in several ways: it focuses on 30-day readmissions that are potentially avoidable, shows good discrimination power to accurately differentiate low from high risk patients, includes only readily available predictors, and can be calculated before discharge.

Table 1. HOSPITAL Score for 30-day Potentially Avoidable Readmissions (maximum 13 points).

Attribute	Value	Points
Low Hemoglobin level at discharge (< 12 g/dL)	yes	1
Discharge from an Oncology service	yes	2
Low Sodium level at discharge ( < 135 mmol/L)	yes	1
Procedure during hospital stay (any ICD-9 coded procedure)	yes	1
Index admission Type: urgent or emergent (non-elective)	yes	1
Number of hospital Admission(s) during the previous year	0-1	0
	2-5	2
	>5	5
Length of stay ≥ 5 days	yes	2

Before wider application of the HOSPITAL score can be recommended, it is important to understand how the score performs in populations outside those from which the score was developed. To assess its generalizability, we aimed to externally validate the HOSPITAL score in a large international multicenter cohort of medical patients.

METHODS

Study design and participants

This multicenter multinational retrospective cohort study included all consecutive adult patients discharged alive between January 1, 2011 and December 31, 2011 from the medical services of 9 hospitals in 4 different countries: the United States, Canada, Israel, and Switzerland (ICARE: International Cohort of Avoidable REadmissions). To minimize inclusion of observation stays, only hospitalizations with a length of stay of more than one day were included. We then excluded patients who were transferred to another acute healthcare facility (hospital or psychiatric hospital) or who left against medical advice.

The participating centers were: 1) University of California San Francisco Medical Center, San Francisco, CA, USA; 2) Hospital of the University of Pennsylvania, Philadelphia, PA, USA; 3) Harborview Hospital Medicine, University of Washington, Seattle, WA, USA; 4) Vanderbilt University Medical Center, Nashville, TN, USA; 5) Northwestern Memorial Hospital, Chicago, IL, USA; 6) Christiana Hospital, Wilmington, DE, USA; 7) Sheba Medical Center, Tel Hashomer, Israel; 8) William Osler Health System, Ontario, Canada; 9) Bern University Hospital, Bern, Switzerland. Of these, 7 are University hospitals and 2 are community hospitals (Christiana Hospital and William Osler Health System). All are not-for-profit hospitals.

The protocol was approved by the institutional review board of each site and also of the managing site at Brigham and Women’s Hospital/Partners Healthcare, Boston, MA, USA. The study follows the criteria from the “Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis” (TRIPOD) Initiative (checklist available upon request).⁸

Study outcome

First, all 30-day readmissions to any division of the same medical center were identified. The primary outcome of 30-day potentially avoidable readmission was then identified through the same process as in the derivation study, using the SQLape algorithm.⁷ This algorithm, based on administrative data and diagnosis codes, was initially developed in Switzerland and has been used there for now for more than 5 years for hospital benchmarking purposes.^9-11 In summary, readmissions which are foreseeable are classified as unavoidable, including readmissions for transplantation, labour and delivery, chemo- or radiotherapy, follow-up or rehabilitation treatment, specific surgical procedures, or some specific diseases deemed difficult to cure (such as multiple sclerosis and idiopathic thrombocytopenia). Then, it classifies readmissions as unavoidable if they involve a new organ system unknown to be affected during the preceding hospitalization. Conversely, complications of treatment (e.g. deep vein thrombosis, catheter associated urinary tract infection, drug-induced disorders, etc.) were considered potentially avoidable.

The unit of analysis was the hospital discharge, and each admission could be considered as both an index hospitalization and a rehospitalization when several hospital stays occurred within an interval of less than 30 days.

Predictor Variables

Information on the 7 predictor variables included in the HOSPITAL score was collected at each site using the same definition as the derivation study (Table 1).⁷ HOSPITAL is the acronym for the 7 variables included in the score: last available Hemoglobin before discharge (positive if < 12 g/dl), discharge from an Oncology service, last available Sodium level before discharge (positive if < 135 mmol/L), any Procedure performed during the hospitalization (any International Classification of Disease [ICD-9 or -10] coded procedure), Index admission Type (emergent or urgent as opposed to elective), number of Admissions in the previous 12 months, and Length of stay (positive if ≥ 5 days). ICD procedures in medical patients included for example endoscopy, hemodialysis, heart cardiac catheter, paracentesis or thoracocenthesis, transfusion, computed tomography scan or magnetic resonance imaging, or percutaneous transluminal coronary angioplasty. The scoring system ranges from 0 to a maximum of 13 points, with a risk of 30-day potentially avoidable readmission that increases with the number of points.

Due to the absence of patients with zero previous admissions in the last 12 months at 2 sites, patients with only one admission in the previous year were attributed 0 points instead of 1 as in the original score. Overall, 5.9% of hemoglobin values and 3.4% of sodium values were missing. The values were imputed as normal when missing (i.e. no points attributed for the missing variable). Other data collected included demographic information in order to describe the population.

Statistical analysis

Patients’ baseline characteristics were presented as proportions, means with standard deviation (SD), and as medians with interquartile ranges (IQR) as appropriate.

First, the HOSPITAL score was calculated for each hospital discharge. Then the risk for an admission to be followed by a 30-day potentially avoidable readmission was categorized into 3 groups according to the total number of points: low risk if 0 to 4 points, intermediate risk if 5 to 6 points, and high risk if 7 or more points. These categories were created in the derivation study for ease of interpretation, roughly corresponding to 5%, 10%, and 20% risk of potentially avoidable readmission.

The score was evaluated according to its overall performance, its discriminatory power, and its calibration. We used the Brier score to quantify the overall performance, i.e. how close predictions are to the actual outcome.^12,13 A useful prediction rule ought to have a Brier score smaller than 0.25 (the lower the better). The discriminatory power reflects the ability of the score to discriminate the cases from the non-cases based on both sensitivity and specificity (C-statistic).^14,15 The calibration shows the ability of the model to generate probabilities that match the observed rates.¹³ We used the Pearson Goodness-of-fit test to summarize this measure, and presented both observed and predicted rates for each risk group of the score (P values ≥0.05 on this test imply good fit, with higher numbers signifying better fit).

Discharges followed by an unavoidable 30-day readmission (according to SQLape classification) were grouped with the discharges not followed by any 30-day readmission (group control) for the main analysis. In a sensitivity analysis, we also evaluated the performance of the HOSPITAL score when these unavoidable readmissions were completely excluded from the analysis.

All tests were conducted as two-sided at a 0.05 level of significance. In the logistic regression, we used a fixed effect for hospital to account for possible differences in outcomes across the 9 hospitals. We also used a robust sandwich variance estimator to account for repeated admissions from the same patient within hospital.¹⁶ Analyses were performed with SAS Software, version 9.3 (SAS Institute Inc., Cary, NC).

RESULTS

During the study period, a total of 121,136 adults were discharged alive from the medical department of the 9 participating centers, and 117,065 (96.6%) discharges were included in the analysis (Figure 1). Within 30 days after discharge, 15.0% (n=17,516) of the medical patients had a readmission, and 9.7% (n=11,307) had a potentially avoidable readmission. The median time to readmission was 12.0 days (IQR 5;19). Baseline characteristics of patients are summarized in Table 2. Patients had a mean age of 60.8 years (SD 18.2), and had a median length of stay of 4 days (IQR 3;7). Overall, patients with a potentially avoidable readmission more often had an urgent or emergent index admission, were more frequently discharged from an oncology service, had a length of stay greater than 5 days, had more hospitalizations in the past year, more often had a procedure, and more often had a low hemoglobin or low sodium level at discharge. All these differences were highly statistically significant (P<0.001).

Figure 1. Study Flow Diagram.

Table 2. Baseline Characteristics of the International Study Population Stratified According to 30-day Potentially Avoidable Readmission.

	No. (%) of patients
Clinical Characteristics	Without 30-day PAR (n=112,418)	With 30-day PAR (n =11,794)
Age, mean (SD)	60.8 (18.3)	61.3 (18.0)
Male sex	53,679 (51%)	5,988 (53%)
Country
United States	70,550 (89.4%)	8,371 (10.6%)
Canada	10,221 (92.6%)	820 (7.4%)
Switzerland	8,971 (94.5%)	524 (5.5%)
Israel	16,016 (91.0%)	1,592 (9.0%)
Urgent or Emergent index admission	78,708 (74%)	9,369 (83%)
Discharge from an oncology division	6,457 (6.1%)	786 (7.0%)
Length of stay of the index admission ≥ 5 days	45,655 (43%)	6,058 (54%)
Number of hospital admissions in the past year
≤ 1	76,196 (72%)	4,826 (43%)
2-5	25,109 (24%)	4,464 (40%)
> 5	4,453 (4%)	2,017 (18%)
≥ 1 procedure during the index hospitalization	78,818 (75%)	8,781 (78%)
Low hemoglobin level at discharge (< 12 g/dL)	59,381 (56%)	8,005 (71%)
Low sodium level at discharge ( < 135 mmol/L)	13,520 (13%)	2,038 (18%)

PAR: potentially avoidable readmission

HOSPITAL score

The score ranged from 0 to 13, with a median HOSPITAL score of 4.0 (IQR 2;5). Using the HOSPITAL score, 62.4% of the patients were categorized as low risk, 23.6% as intermediate risk, and 14.0% as high risk for a potentially avoidable readmission. Risk of 30-day potentially avoidable readmission for patients with a HOSPITAL score in the low, intermediate, and high risk groups were 5.8%, 12.0%, and 22.8%, respectively.

Discriminatory power

We found that the HOSPITAL score had a C-statistic of 0.72 (95%CI 0.72-0.72), showing good discrimination. In US hospitals, the C-statistic was 0.72 (95%CI 0.71-0.72), in Canada 0.78 (95%CI 0.76-0.80), in Israel 0.68 (0.67-0.69), and in Switzerland 0.68 (95%CI 0.66-0.71). The Brier score was 0.08, signifying very good overall performance.

In a sensitivity analysis, excluding patients with unavoidable readmissions rather than combining them with patients without any readmissions did not significantly change the C-statistic (C-statistic = 0.73 [95%CI 0.73-0.74]).

Calibration

In terms of calibration, the estimated risk of potentially avoidable readmission calculated with the HOSPITAL score matched the observed proportion of potentially avoidable readmissions in each risk group: 5.8% for the low risk group, 11.9% for the intermediate, and 22.8% for the high risk group (Table 3). This is also reflected by an excellent Pearson goodness of fit test with a P-value of 0.89. When calibration is analyzed for each individual point score, calibration remains excellent except at the extreme high and low ends of the range, zero points and 11 or more points, where sample size is limited (see eTable1). The positive likelihood ratio for each point score showed a similar pattern (see eTable2).

Table 3. Observed vs. Predicted 30-day Potentially Avoidable Readmissions.

Points	Risk category	Patients in each category, n (%)	Observed proportion with PAR in the validation study, %	Estimated risk of PAR in the validation study using the HOSPITAL score, %
0-4	Low	73,031 (62%)	5.8	5.8
5-6	Intermediate	27,612 (24%)	11.9	11.9
≥ 7	High	16,422 (14%)	22.8	22.8

PAR: potentially avoidable readmission

DISCUSSION

In a large international multicenter external validation study, we found that the HOSPITAL score had good discriminative ability and excellent calibration for predicting the risk of 30-day potentially avoidable readmission. The HOSPITAL score is easy-to-use and can be calculated before discharge, which makes it a practical tool for identification of patients at high risk for preventable readmission and the timely administration of high intensity interventions designed to improve transitions of care.

A recent systematic review comparing existing readmission scores pointed out the lack of validated, reliable, and generalizable score stratification models for readmission.⁶ Most scores developed to date have exhibited poor performance, lacked geographical validation, were developed using small sample size, or are complex. Among the most frequently cited score is the LACE score.¹⁷ Derived and validated in large populations in Ontario, Canada, the score showed a C-statistic of 0.68 in Ontario and 0.70 in an external validation study in Singapore.¹⁸ It has however never been validated in other countries, including the US. Also, it requires users to calculate another score, the Charlson comorbidity index, that estimates the 1-year mortality of the patient based on billing coded comorbidities, practically available only after discharge.

In comparison, the HOSPITAL score applies 7 readily available predictors. Importantly, it can be calculated before the discharge of the patient, in time for interventions to be started in the hospital. It showed good performance in this large international multicenter study, which is to our knowledge the first prediction model for readmission to be so widely validated across an international cohort of patients and one of the few that follows current methodological recommendations.¹⁹ The size of the validation cohort is also one of the largest. Moreover, the HOSPITAL score was designed specifically to focus on readmissions that are potentially avoidable, i.e., those that are theoretically most likely to benefit from interventions. This is of critical importance as hospitals and health systems face decisions about how to best deploy limited resources, yet there has been a dearth of such tools.²⁰ In a recent study among nearly 20,000 medical patients in Denmark, the HOSPITAL score was significantly better than the LACE score for identifying patients at risk of 30 day readmission (P<0.001).²¹

With a C-statistic of 0.72, the HOSPITAL score performs similarly to other commonly used prediction models. As a comparison, the well-known CHADS2 and CHADS2-VASC scores for stroke risk stratification in atrial fibrillation, and the HAS-BLED score for bleeding risk on anticoagulation all have a C-statistic of 0.68 or less in external validation studies.^22-25

This study has several limitations. First, by design the model focuses on a medical population, and it may not be generalizable to other patient populations such as surgical patients. Second, additional predictors could have been used to improve the model’s discrimination (e.g. functional status, socioeconomic status), however we purposefully chose to use a smaller number of readily available predictors to increase clinical applicability and ease of use. Third, we should note that 30 days may not be the most appropriate time frame to judge preventable readmissions. However, we chose it because it has become the standard for research studies and for hospitals given its use by CMS in its Readmission Reduction Program. Fourth, the variables cannot and should be not used to guide readmission reduction interventions. The variables included are predictors and not necessarily modifiable risk factors. For example, we would not presume that transfusing a patient with a hemoglobin < 12 g/dL would be indicated to lower their readmission risk. Fifth, we acknowledge that transitional care interventions are most effective when implemented as early in the hospitalization as possible, and that the score cannot always identify high-risk patients early since it includes length of stay and lab values at discharge. However, some patients can be identified as high risk at the time of admission (e.g., based on admission type, service, and number of prior hospitalizations), while initially lower risk patients may be monitored and then identified as high risk prior to discharge, once they have had a procedure or the length of stay reaches 5 days. Intensive interventions could then be started at that time. Further research would be needed to determine the effects of interim low sodium and hemoglobin levels on readmission risk given their ability to normalize (or not) by the time of discharge.

Last, we acknowledge that the outcome of interest in this study is “potentially avoidable readmissions” and not “avoidable readmissions.” The SQLape algorithm identifies a population with unplanned readmissions that are likely related to the index admission (e.g., same organ system, possible complications of therapy). This population is enriched with patients whose readmissions are truly preventable, but is neither 100% sensitive nor specific for identifying such patients, assuming such patients could ever be identified with certainty. The extent to which readmissions are preventable is controversial. The advantage of tools such as the HOSPITAL score is that we can use it to begin to answer the more relevant question of which patients are most likely to benefit from interventions designed to prevent readmissions. Predicting potentially avoidable readmissions (as opposed to all readmissions) is an important step along this progression. This work is particularly important given recent incentives to reduce readmissions and several recent unsuccessful efforts to reduce them.²⁶ Future intervention studies should therefore be adequately powered to identify subgroups, such as those identified by the HOSPITAL score, most likely to benefit.

Conclusions

The HOSPITAL score had good to excellent ability to identify patients at high risk of 30-day potentially avoidable readmission when applied to a large international cohort of medical patients. The HOSPITAL score is the first risk prediction score to focus on potentially avoidable as opposed to all-cause readmissions, using readily available predictors at the time of discharge. This study externally validated it in a large cohort in 4 countries. This score has the potential to reliably identify patients in need of more intensive transitional care interventions to prevent hospital readmissions.

Figure 2. Receiving Operating Characteristic Curve of the HOSPITAL Score.