Risk of 30-Day Hospital Readmission Among Patients Discharged to Skilled Nursing Facilities: Development and Validation of a Risk-Prediction Model

Abstract

Objectives:

Patients discharged to a skilled nursing facility (SNF) for postacute care have a high risk of hospital readmission. We aimed to develop and validate a risk-prediction model to prospectively quantify the risk of 30-day hospital readmission at the time of discharge to a SNF.

Design:

Retrospective cohort study.

Setting:

Ten independent SNFs affiliated with the postacute care practice of an integrated health care delivery system.

Participants:

We evaluated 6,032 patients who were discharged to a SNF for postacute care after hospitalization.

Measurements:

The primary outcome was all-cause 30-day hospital readmission. Patient demographics, medical comorbidity, prior use of health care, and clinical parameters during the index hospitalization were analyzed by using gradient boosting machine multivariable analysis to build a predictive model for 30-day hospital readmission. Area under the receiver-operator curve (AUC) was assessed on out-of-sample observations under 10-fold cross-validation.

Results:

Among 8,616 discharges to a SNF from January 1, 2009, through June 30, 2014, 1,568 (18.2%) were readmitted to the hospital within 30 days. The 30-day hospital readmission prediction model had an AUC of 0.69, a 16% improvement over risk assessment using the Charlson Comorbidity Index alone. The final model included length of stay, abnormal laboratory parameters, and need for intensive care during the index hospitalization; comorbid status; and number of emergency department and hospital visits within the preceding 6 months.

Conclusion:

We developed and validated a risk-prediction model for 30-day hospital readmission in patients discharged to a SNF for postacute care. This prediction tool can be used to risk-stratify the complex population of hospitalized patients who are discharged to SNFs to prioritize interventions and potentially improve the quality, safety, and cost-effectiveness of care.

Introduction

Many older adults require skilled nursing facility (SNF) placement for ongoing care and monitoring after hospitalization. In the United States, approximately 20% of Medicare beneficiaries require SNF care, with 1.7 million people receiving care from SNFs (2.4 million stays) in 2014 alone.¹ SNF patients are highly complex because of multiple comorbidities, frailty, and functional dependence,² and they are at risk for repeat emergency department (ED) visits and hospitalizations.^3–7 Hospital readmissions, in particular, are harmful,⁸ costly, and potentially preventable.^{7, 9} Unplanned readmissions are considered a quality metric¹⁰; therefore, reducing readmissions from SNFs is a priority for patients, health care providers, and payers.

Several high-intensity interventions have been studied to reduce hospital readmissions from SNFs.^11–14 Patients are vulnerable to adverse events during the transition between hospital discharge and admission to a SNF.¹⁵ With shorter hospital stays, patients discharged to SNFs may have ongoing needs for monitoring and acute care, but mandatory physician visit requirements at SNF have not changed.¹⁶ With limited resource availability and the volume of patients requiring SNF care, identification of patients with highest risk of readmission is critical.

We conducted this study to 1) determine risk factors for 30-day hospital readmission in patients discharged to SNFs from all medical, surgical, and subspecialty hospital services and 2) develop and validate a prediction model to identify high-risk subgroups in this population.

Methods

The study was reviewed and approved by the Mayo Clinic Institutional Review Board. The reporting of this study is in compliance with the TRIPOD statement (Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis).¹⁷

Study Design

We conducted a retrospective cohort study by using data from electronic health records (EHRs) and administrative claims.

Population and Setting

The study population (a convenience sample) comprised patients 55 years or older who were discharged to 1 of 10 area SNFs from either of 2 Mayo Clinic hospitals (Rochester, Minnesota) from January 1, 2009, through June 30, 2014. We defined SNFs as having 24-hour nursing coverage and being licensed in Minnesota. Our institutional Department of Family Medicine and the Division of Primary Care Internal Medicine provide postacute care to patients dismissed from these hospitals to area SNFs. One facility included in the study is a not-for-profit facility that is owned and operated by Mayo Clinic. During the study period, 2 facilities with the same ownership merged and were treated as a single facility in our analysis. Patients who did not authorize use of their medical records for research and those with a SNF admission date different from the date of hospital dismissal were excluded from the study.

Independent Variables

Patient demographics, medical comorbidity, prior use of health care and available parameters pertaining to the index hospitalization that could potentially indicate clinical complexity, were analyzed to determine the association with 30-day hospital readmission risk. Patient demographic and clinical information was ascertained from the EHR and administrative databases. We used diagnosis codes from the International Classification of Diseases, Ninth Revision, to determine medical comorbidity. For the index hospitalization, primary discharge diagnoses were ascertained and grouped by using Clinical Classification software (Agency for Healthcare Research and Quality).¹⁸ Prior health care use was assessed by the number of hospital stays and ED visits in the preceding 6 months, index hospitalization, length of stay of the index hospitalization (LOS), dismissing service, and intensive care unit (ICU) level of care during the index hospitalization; these factors were determined from the EHR and from billing data. Clinical parameters such as the Braden score,¹⁹ fall risk score,²⁰ presence of delirium (using the Confusion Assessment method²¹), and mobility were ascertained from hospital nursing and physical therapy flowsheets. Laboratory values (hemoglobin, creatinine, sodium, and potassium) obtained closest to the SNF admission date were obtained from the EHR. Mortality data were collected from the patient registration system.

Primary Outcome

The primary outcome was 30-day hospital readmission after discharge to a SNF. Hospital billing encounters were cross-matched with discharge summaries in the EHR to confirm hospital readmission. Participants who died within 30 days of the index discharge were treated as right-censored in the analysis (ie, death was not considered a readmission event unless readmission occurred before death).

Statistical Analysis

Univariate summaries for all predictor variables going into the model for the groups of patients with and without 30-day readmissions were produced for descriptive purposes after removing missing values. Variables are summarized with categorical cutoffs for descriptive display only; they were treated as continuous variables for analysis.

To build a multivariate predictive model that was sufficiently robust for use in clinical practice, we considered approaches that could accommodate missing data.²² We decided on the gradient boosting machine (GBM),²³ via the “gbm” package in R version 3.2.3.^23–25 GBM is a robust, all-purpose prediction algorithm that works well for most clinical applications. It can handle a mix of discrete and continuous predictors and allows for missing values of the predictors. The use of GBM requires the specification of tuning parameters (ie, N= number of trees in the ensemble and S=shrinkage), which were chosen via a grid search by using 10-fold cross-validation. A 20-point grid was used for N, equally spaced from 100 to 800. A 20-point grid was also used for S, from 0.0001 to 0.1, equally spaced on a log scale. For model derivation, we included patients 55 years or older. The best model was selected by comparing the area under the receiver-operator curve (AUC) constructed on out-of-sample cross-validation results. The patient encounters were implicitly treated as independent (even those involving the same patient) when fitting the GBM. We believed that any overrepresentation bias introduced with this assumption would matter less for prediction than for inference. Our goal was to have good prediction for a given (future) encounter, and the cross-validation set was representative of a set of future encounters. Thus, the predictive performance via cross-validation was the most relevant measure (regardless of the assumptions used when training the model). Because of the nonlinearity and interactions involved in a GBM model, we also constructed main effect plots by grouping the responses for each variable into deciles and plotting the sample proportions to assess variable importance. The estimated main effect curves were evaluated by a smoothing spline fit to the resulting probabilities from GBM for each observation in the dataset across a given variable.

As expected, because of the high complexity of patients discharged to SNFs and their many interrelated risk factors for readmission, the predictor set included many highly correlated variables (eg, number of previous ED visits, number of previous inpatient visits, individual comorbidities, Charlson Comorbidity Index [CCI]). After assessing variable importance via standard approaches, including permutation importance and GBM-specific metrics,^{23, 26} we concluded that the most informative approach was to group similar variables and then evaluate model performance (ie, AUC) under conditions in which only a given group was included, as well as when all variables except those from the given group were included. Variable groupings were chosen purely by intuition and interpretability and had no impact on model fit or model performance.

For comparison, baseline models were constructed using the CCI ²⁵ and LOS. All models were assessed for predictive performance via 10-fold cross-validated AUC. The hold-out sets selected during cross-validation were a random sample from the data and were not used in any manner to fit the model.

Results

We identified 6,032 patients who were admitted to a SNF (8,616 admissions) directly after hospital discharge during the study period. Mean (SD) age at SNF admission was 78.1 (9.8) years; 3,703 (61%) were female and 5,781 (96%) were white. Nearly half of all 8,616 admissions (53%) were from surgical services. The 30-day mortality rate was 392/8,616 (4.5%). The overall 30-day hospital readmission rate was 18.2% (n=1,568). Table 1 and Supplemental Tables 1–2 show baseline characteristics and univariate summaries of patients who did and did not have hospital readmission within 30 days of discharge to a SNF. Variables are summarized with categorical cutoffs for descriptive display only and were treated as continuous variables in the GBM model. The first column provides the grouping where the predictor was placed to assess variable importance.

Table 1.

Characteristics of Admissions Discharged to SNFs for Postacute Care^a

Category	Predictor Variable	Variable Value Breakdown	Cohort Count (Number of Admissions)	Hospital Readmission, Number (%)
Demographics	Age, y	<55	276	66 (23.9)
		≥55 to <65	700	141 (20.1)
		≥65 to <75	1,919	334 (17.4)
		≥75 to <85	2,970	520 (17.5)
		≥85	2,751	507 (18.4)
	Sex	Female	5,353	876 (16.4)
		Male	3,263	692 (21.2)
	Race	White	8,298	1,514 (18.3)
		Nonwhite	212	45 (21.2)
		Unknown or missing	106	9 (8.5)
	Marital status	Married	4,041	711 (17.6)
		Other	4,575	857 (18.7)
Comorbidity	Charlson	>6	1,602	441 (27.5)
	Comorbidity Index (count of diseases)	≤6	7,014	1,127 (16.1)
Prior health care use	ED visits in past 6 months	0	5,287	774 (14.6)
		1–3	3,004	678 (22.6)
		≥4	325	116 (35.7)
	Hospital stay in the past 6 months	0	5,271	723 (13.7)
		1–3	3,094	743 (24.0)
		≥4	251	102 (40.6)
Hospital events	Duration of index hospitalization	≤3 d	3,081	360 (11.7)
		4–7 d	3,517	631 (17.9)
		8–14 d	1,389	362 (26.1)
		≥15 d	629	215 (34.2)
	ICU admission during index hospitalization	Yes	2,534	705 (27.8)
		No	6,082	863 (14.2)
	Delirium (Confusion Assessment Method)	Positive	1,120	310 (27.7)
		Negative	7,424	1,247 (16.8)
	Braden score (pressure ulcer risk)	High risk	5,385	1,145 (21.3)
		Low risk	3,169	413 (13.0)
	Fall risk	High risk	3,658	814 (22.3)
		Low risk	4,881	743 (15.2)
Advanced directives	Advanced directive on file	Yes	4,948	933 (18.9)
		No	3,668	635 (17.3)
Laboratory testsc	Creatinine (Female, 0.6–1.1 mg/dL; male, 0.8–1.3 mg/dL)	Normal	5,413	867 (16.0)
		Abnormal	2,799	658 (23.5)
		N/A	404	43 (10.6)
	Hemoglobin (Female, 12.0–15.5 g/dL; male, 13.5–17.5 g/dL)	Normal	787	122 (15.5)
		Abnormal	7,057	1,316 (18.7)
		N/A	772	130 (16.8)
	Potassium (3.6–5.2 mmol/L)	Normal	7,631	1,382 (18.1)
		Abnormal	688	148 (21.5)
		N/A	297	38 (12.8)
	Sodium (135–145 mmol/L)	Normal	6,832	1,204 (17.6)
		Abnormal	1,484	327 (22.0)
		N/A	300	37 (12.3)

Abbreviations: ED, emergency department; ICU, intensive care unit; N/A, not available; SNF, skilled nursing facility.

^aThe study included 6,032 patients who had 8,616 SNF admissions.

^bDetails about mobility are provided in Supplemental Table 2.

^cReference values are shown.

Figure 1 displays the main effect plots for a sample of the individual predictor variables. The changes in the curve over the range of values show how readmission risk changed globally across that variable.

Figure 1.

Main Effect Plots for 9 Predictor Variables. For each set of rectangles, the widths correspond to consecutive deciles of the original data and heights correspond to the empirical readmission rate for observations falling within that range. The red curve in each plot is a smoothing spline fit to gradient boosting machine predictions for each observation across the respective variable. ED indicates emergency department.

Readmission Prediction Model

Factors that we considered most important to the risk calculation by the GBM model included LOS, abnormal laboratory parameters, and ICU stay during index hospitalization; number of ED visits and hospital stays in past 6 months; and medical comorbidity. Supplemental Figures 1 and 2 show examples of risk calculation in high-and low-risk patients, respectively.

When comparing the GBM model (AUC, 0.699) with the CCI or LOS alone (AUC, 0.587 and 0.579, respectively) for predicting 30-day readmission, we found that the GBM model had a 16% relative improvement, thus demonstrating the superiority of a multiple-predictor approach (Figure 2). The step-like quality of the CCI receiver operating characteristic (ROC) curve is due to the discrete nature of the CCI, which ranged from 0 to 21 in our sample. A more clinically relevant measure than AUC for assessing performance in practice may be provided by a particular point on the ROC curve, ie, the rate of detected readmissions for a given false-positive rate, ie, 1−specificity. Sensitivities (with 95% CIs) for GBM, CCI, and LOS for false-positive rates of 0.2 and 0.3 (corresponding to GBM-predicted risk cutoffs of 0.24 and 0.20, respectively) are provided in Table 2. Thus, the GBM model had better sensitivity and specificity for predicting 30-day hospital readmission.

Figure 2.

Performance results for the GBM model, Charlson Comorbidity Index, and Length of Stay were compared by using corresponding ROC curves on out-of-sample observations via 10-fold cross-validation, along with 95% pointwise confidence bands (obtained via 1,000 bootstrap samples). AUC indicates area under the receiver-operator curve; CCI, Charlson Comorbidity Index; GBM, gradient boosting machine; LOS, length of stay; ROC, receiver operating curve.

Table 2.

AUC and Sensitivity of the Gradient Boosting Machine model, Charlson Comorbidity Index (CCI), and Length of Stay

Model	AUC (95% CI)	Sensitivity at 20% FPR (95%CI)	Sensitivity at 30% FPR (95% CI)
Gradient boosting machine	0.70 (0.68–0.72)	0.45 (0.42–0.48)	0.58 (0.55–0.61)
Charlson comorbidity index	0.59 (0.57–0.60)	0.29 (0.27–0.31)	0.43 (0.40–0.45)
Length of stay	0.58 (0.56–0.60)	0.31 (0.29–0.39)	0.47 (0.44–0.49)

Abbreviations: AUC, area under receiver operator curve, FPR, false-positive rate.

Predictor Variable Importance

Groupings used to assess variable importance are listed as “Predictor Variable” in the second column of Table 1. Figure 3 shows the relative importance of model variables by examining the effect of including or excluding groups of similar variables on the model AUC. Therefore, when used alone, LOS and CCI were the most influential, whereas age and documentation of an advanced directive were not. When evaluated in conjunction with other predictors, removing worse mobility and high-risk Braden score made little difference, but removing LOS or CCI had a deleterious effect, signifying their importance in predicting the overall readmission risk.

Figure 3.

Variable Group Importance. The red axis is a variable group’s individual AUC, ie, the performance of GBM when fit using only that group of variables. The blue axis is the change in AUC when GBM is fit to all variables vs when GBM is fit to all variables except those in the given variable group. The change quantifies how much model performance is diminished if a given variable group is excluded. Both axes have been scaled by a factor of 100 to improve readability, and the bars represent 95% CIs, calculated via 2,000 bootstrap samples. AUC indicates area under the receiver-operator curve; ICU, intensive care unit; LOS, length of stay.

Discussion

Preventing hospital readmission for patients receiving postacute care in a SNF may be facilitated by risk-stratifying this highly complex and multimorbid patient population so that intensive and resource-limited monitoring and interventions can be delivered to the patients with highest risk. Because of the interrelated and interdependent nature of the many factors that affect readmission risk for these patients, developing an accurate risk-prediction model with traditional regression methods may be challenging. Models predicting risk of 30-day readmission from the SNFs are limited. One study evaluated the HOSPITAL score, originally developed to predict potentially preventable 30-day readmissions in the general population, to predict all-cause readmissions in a population of medical patients discharged to SNFs.^{27, 28}

In this retrospective cohort study, we analyzed data from 6,032 patients receiving postacute care after hospitalization, with direct admission to SNFs from medical, surgical, or specialty services. We applied the GBM method to develop a model to predict 30-day all-cause hospital readmission that is robust to missing data and thus usable in routine clinical practice. The predictive model includes LOS, ICU-level care, and abnormal laboratory values during the index hospitalization; degree of medical comorbidity; and number of ED visits and additional hospitalizations in the preceding 6 months. The model had an AUC of 0.69, which was a 16% improvement compared with using comorbidity alone, and had adequate sensitivity and specificity.

Several parameters during the index hospitalization were significant predictors of 30-day readmission in our model. A longer LOS was the most influential predictor. Prolonged hospital stay indicates a greater disease burden, more potential complications, higher care needs, or a combination of factors; these elements may persist upon discharge to a SNF. Other studies have reported similar findings.^27–29 ICU-level care during the index hospitalization was another strong predictor of 30-day hospital readmission and also suggests greater clinical complexity and instability. Few prior studies included ICU care as a predictor of 30-day hospital readmission, and all were conducted in limited clinical situations.^{25, 30, 31} This is the first study to confirm the importance of ICU care, irrespective of the cause for index hospitalization. Finally, consistent with other studies,^{27, 32, 33} abnormal hemoglobin, creatinine, sodium, and potassium values before discharge from the hospital were also important predictors of readmission in our model

Greater comorbidity is associated with higher hospital readmission rates among community-dwelling patients,^{3, 34, 35} and we showed that clinical complexity also was a strong predictor of readmission risk for patients receiving SNF care. Previous studies of the effect of comorbidity on readmissions from SNFs have focused on patient groups with specific conditions.^{3, 29, 33, 36} Our results suggest that although greater comorbidity indicates risk of readmission in patients discharged to SNFs, use of comorbid status alone may not be sufficient to discern the highest-risk patients in this complex population. In examining our main effect plots (Figure 1), we found that the CCI curve plateaued, indicating diminishing risk impact after an index value of 6. The model performance was enhanced by including variables reflecting the patient’s acute care needs and fluctuating clinical course such as LOS, ICU stay, and abnormal laboratory parameters.

Similarly, greater prior use of health care reflects greater disease burden and unmet health care needs that may persist after discharge to a SNF. Frequent hospital stays and ED visits in the preceding 6 months were an important predictor of 30-day hospital readmission from the SNF in our study. This finding has been confirmed in older, community-dwelling patients and in other SNF settings.^{29, 35, 37}

Many traditional risk factors for readmission from the community, such as patient age and mobility status during hospitalization, were not significant predictors of readmission in our cohort of patients discharged to SNFs. These differences could be attributed to our homogenously older cohort, with 88% of patients being older than 65 years. Hospitalized patients discharged to SNFs have greater functional dependence than those discharged to home, and therefore, the presence of impaired mobility loses its discriminative value in this high-risk group relative to community-dwelling patients³⁸ or to those with impaired functional status (or declining status while at a postacute-care rehabilitation facility).^{29, 33, 39–41}

This study has several limitations. It was a retrospective study from a single integrated health care delivery system in the upper Midwest with a largely white population. We could not capture hospitalizations or ED visits that occurred outside of our health care system. Patient deaths outside of our immediate vicinity may not have been indicated in our EHRs.

We could not clearly distinguish between short-term SNF residents (with expectations of dismissal) vs long-term residents. To ensure that all included admissions were for postacute care, we included only those admissions that were immediately linked to an index hospitalization at 1 of our institutional hospitals. We could not reliably determine the patients’ discharge medications or their socioeconomic status, both of which may affect their risk of readmission. To increase the generalizability of our model, we did not include dismissing diagnoses from the index hospitalization; these variables are not consistently available at the time of SNF admission, when this model would be deployed in clinical practice. The tool does not incorporate provider care processes at the SNF or facility factors that also influence patient outcomes. We aimed to include only the data that were available at the time of discharge. We did not derive the model for only those readmissions that could be classified as “avoidable” because we believed it could guide interventions for all patients.

The results of this study have practical applications for health care providers in the postacute-care SNF environment. We suggest that easily obtained administrative and clinical parameters from the hospital stay, prior use of health care, and medical comorbidity could be used to identify patients with a high risk of 30-day hospital readmission, and identified patients could receive tailored and prioritized interventions. These results highlight the importance of active information sharing between health systems and provider groups to include important clinical parameters such as those identified in the model. Further studies are needed to confirm these findings in other SNF and postacute-care settings. Additional research is also needed to determine whether interventions based on patients’ risk status can impact outcomes

Conclusion/Relevance

We applied an advanced GBM algorithm to EHR and clinical data to develop an accurate model predicting the risk of all-cause 30-day hospital readmission among patients discharged from medical, surgical, and subspecialty services to SNFs for postacute care. The model includes index hospitalization LOS, ICU-level care, several abnormal laboratory values, medical comorbidity, and measures of prior health care use. These results may be useful for hospitals and SNF providers in identifying patients’ risk of 30-day hospital readmission at the time of transition of care from the hospital to the SNF. Further studies are needed to validate this model in different SNF settings and geographic locations and through larger national and claims databases.

Supplementary Material

1

Supplemental Figure 1. Prototype Example of a High-Risk Patient. The figure shows the probability of readmission and the effect of some variables that contribute to the risk of 30-day readmission in an individual patient. The y-axis in individual graphs indicates the probability of readmission, and the x-axes display the number of inpatient hospitalizations in the prior 6 months, length of stay (LOS) (in days) of the index hospitalization, the number of chronic conditions (Charlson index), and the values of creatinine (mg/dL), potassium (mmol/L), and hemoglobin (g/dL). The red line indicates the value of the respective variable for that particular patient.

Supplemental Figure 2. Prototype Example of a Low-Risk Patient. The figure shows the probability of readmission and the effect of some variables that contribute to the risk of 30-day readmission in an individual patient. The y-axis in individual graphs indicates the probability of readmission, and the x-axes display the number of inpatient hospitalizations in the prior 6 months, length of stay (in days) of the index hospitalization, the number of chronic conditions (Charlson index), and the values of creatinine (mg/dL), potassium (mmol/L), and hemoglobin (g/dL). The red line indicates the value of the respective variable for that particular patient.

Abbreviations

AUC

area under the receiver-operator curve

CCI

Charlson Comorbidity Index

ED

emergency department

EHR

electronic health record

GBM

gradient boosting machine

ICU

intensive care unit

LOS

length of stay of the index hospitalization

ROC

receiver operating characteristic

SNF

skilled nursing facility