Predicting potential adverse events during a skilled nursing facility stay: a SNF prognosis score

Abstract

Background/Objectives

Older adults are increasingly discharged to skilled nursing facilities (SNFs) for post-acute care, but adverse outcomes are common. We sought to derive a risk prediction score for potential adverse outcomes in older adults transitioning to SNF from the hospital.

Design/Setting

Retrospective analysis of the 2003–11 Medicare Current Beneficiary Survey (MCBS).

Participants

Previously community-dwelling Medicare beneficiaries who were hospitalized and discharged to SNF for post-acute care (N=2043).

Measurements

Risk factors included demographic characteristics, comorbidities, health status, hospital length of stay, prior SNF stays, SNF size and ownership, treatments received, physical function, and active signs or symptoms at time of SNF admission. The primary outcome was a composite of undesirable outcomes from the patient perspective, including hospital readmission during the SNF stay, long SNF stay (≥100 days), and death during the SNF stay.

Results

Of the 2043 previously community-dwelling beneficiaries who were hospitalized and discharged to a SNF for post-acute care, 589 (28.8%) experienced one of the three outcomes, with readmissions (19.4%) most common, followed by mortality (10.5%) and long SNF stay (3.5%). A risk score including five factors (Barthel Index, Charlson-Deyo comorbidity score, hospital length of stay, heart failure diagnosis, and presence of an indwelling catheter) demonstrated very good discrimination (C-statistic 0.75), accuracy (Brier score 0.17), and calibration for observed and expected events.

Conclusion

Older adults frequently experience potentially adverse outcomes in transitions to SNF from the hospital; this novel score could be used to better match resources with patient risk.

INTRODUCTION

Older adults are increasingly discharged to skilled nursing facilities (SNFs) for post-acute care¹, but frequently experience potentially adverse outcomes during their SNF stay – readmissions to the hospital, long stays that exceed their ability to pay, or even death.^2–4 Recent value-based payment reforms – such as bundles for acute and post-acute care, and penalties for readmissions from SNFs - have intensified the emphasis on better delineating which patients are at high risk for adverse outcomes.^5,6 Matching intensity of interventions to the needs of the patient is a cornerstone of high-value care and a key component of interventions to reduce post-hospital utilization cost-effectively.⁷ However, at this point, SNF clinicians have little evidence to identify patients likely to experience a potentially adverse outcome at the time of SNF admission.^4,8,9

An accurate model predicting post-acute care outcomes could assist clinicians, older adults, and their families in SNF with forecasting the risks or potential benefits of care to better inform decisions about interventions and plan for the future. Additionally, such a model has the potential to differentiate between patients at low risk for a potentially adverse outcome (who may require less intensive monitoring and treatment) and those at high risk (who may be at such high risk they require intensive intervention or involvement of palliative care).^5,7

Therefore, we sought to develop a model that could be used to predict potential adverse outcomes among previously community-dwelling Medicare beneficiaries being discharged from the hospital to SNF, using information that would be available around the time of SNF admission.

METHODS

Study design and setting

This was a secondary analysis of the 2003–2011 Cost and Use and Access to Care modules of the Medicare Current Beneficiary Survey (MCBS), a prospective nationally-representative cohort of the Medicare population sponsored by the Centers for Medicare and Medicaid Services. Around 12,000 Medicare beneficiaries are systematically sampled three times annually for a maximum of four years (Access to Care modules) and these surveys are matched to Medicare claims data (Cost and Use modules) and nursing home information (including the Minimum Data Set, MDS). This allows participants to be followed longitudinally across care settings, with information about their home environment as well as health care utilization. The MCBS uses a rotating-panel design, adding approximately one-quarter of the cohort annually; at the time of the analysis the 2011 data was the most recent available from the VA Information Resource Center (VIReC). From 2003–11, there were 50,417 unique beneficiaries sampled. Interviews can be completed in the community or a facility, by the beneficiary or a proxy, in English or Spanish. Patients can be dropped from the survey if they die or permanently move more than 30 miles outside their geographically sampled area. Facility interviews are completed with staff and by medical record review. The MDS is an assessment tool completed by staff on SNF admission, discharge, and at regular intervals during the stay.

Patients

We included beneficiaries age 65 or older who were community-dwelling (defined as without any previous stay in a long-term care facility; those with prior SNF stays were eligible), were hospitalized, and were discharged to a SNF after hospitalization. Acute inpatient rehabilitation, long-term acute care hospitals, and long-term care nursing homes (without skilled care) were excluded. We included data only from the second SNF stay in patients with two stays to allow assessment of the effect of a prior stay. A small number of patients with more than 2 SNF stays had these subsequent stays excluded.

Predictors

We used the first MDS assessment completed after SNF admission as a proxy for the patient’s state at time of SNF admission. If the patient had multiple MDS assessments in the first 30 days of their stay recorded, we used the most complete assessment. During the study period, two different versions of the MDS were used (MDS 2.0 and MDS 3.0), so we created a crosswalk of relevant predictor variables. In cases where the predictor was a single variable with a direct counterpart between versions, we simply crosswalked to the corresponding variable in the other assessment. For predictors captured by an instrument rather than a single variable, we crosswalked the entire instrument, using the same “triggers” for a positive finding as described in each assessment (Supplementary Table S1).

Potential predictors were chosen based on prior literature suggesting they may be linked to complex care transitions, hospital readmissions, or mortality.^2,10,11 These included patient demographics (age, education, race/ethnicity, marital status, income level), comorbidities (body mass index, Charlson-Deyo comorbidity score,¹² as well as individual chronic conditions captured on the MDS initial screening assessment by asking the patient or caregiver about the presence of particular diagnoses), self-rated general health and health compared to one year ago in addition to whether the patient and staff at the SNF think they are capable of increased functional independence, ownership type and number of beds in the SNF, hospital length of stay (log-transformed), active conditions at the time of SNF admission (such as presence of pain or delirium), whether the patient had a prior SNF stay, and treatments prescribed during the SNF stay (such as intravenous medications or inhaled oxygen). Physical function was measured using the Barthel Index which captures ability to complete activities of daily living and mobility on a level surface, stairs, and with transfers; it has been used extensively in trials of rehabilitation.^13–16 A lower score on this scale (0–100) connotes poorer physical performance.

Variables with high missingness (10% or more) or low prevalence (1 percent or less) were excluded from the initial analysis. The amount of missing data and an assessment of imputation for the predictors with missing data is described in Supplementary Figure S1; single imputation was accomplished using the MICE package in R.¹⁷ We conducted a sensitivity analysis that included variables with high missingness.

Outcomes

The primary outcome was a composite of potential adverse events occurring during the SNF stay, defined as either a hospital readmission during the SNF stay, a long SNF stay (defined as ≥100 days), or death during the SNF stay. We chose these outcomes based on relevance to patients and post-acute care payment reforms. For example, since readmission penalties will start for SNFs, we included hospital readmission during the SNF stay.⁶ Second, we included long stays (defined as ≥100 days), since Medicare benefits cease at 100 days and community discharge rates are also being targeted as a part of value-based payments for SNF.⁵ Third, we included mortality during or within the month of the SNF discharge. Mortality is systematically recorded in the MCBS as the last day of the month in which the death occurred, so some deaths included could have occurred as late as 30 days after the end of the SNF stay (assuming death on the 1^st of the month in a month with 31 days). Patients could experience more than one outcome during their stay (such as a long stay followed by death in the facility). We refer to these events as “potential” adverse events since not all readmissions, long stays, or even deaths are necessarily incongruent with patient goals.

We elected to report a composite outcome for three reasons. First, we did not feel separating the outcomes would help better identify which constitute an adverse event. Second, separating the outcomes may lead to confusion on the part of patients and clinicians since each predictive model might be different and would vary in accuracy and performance. Third, in our clinical experience, the underlying phenotype of patients most likely to have a hospital readmission, have a very long stay, or die in SNF is similar. For example, we found SNF patients who were readmitted to the hospital experienced four times the odds of mortality at 100 days compared to those not readmitted, and functional status was the most important predictor.²

Statistical analysis

Summary statistics were calculated for predictors and are summarized in Table 1; a full list of predictor variables is listed in Supplementary Table S2. All predictors that met the prevalence (>1%) and missingness (<10%) criteria were included in a starting model, and backward elimination (using progressively smaller p-value cutoffs of p≤0.25, p≤0.10, p≤0.01, p≤0.001) was used to evaluate the relative performance of progressively more parsimonious models. Given the models had very similar performance, we report the results of the model based on the smallest p-value cutoff (p≤0.001) due to its relative simplicity (characteristics of all models are listed in Supplementary Table S3).

Table 1.

Characteristics of patients in the study cohort

	All patients N=2043	Had none of outcomes N=1454	Had at least one outcome N=589
Demographics, %
Age – Years, mean (SD)	82.0 (7.4)	81.6 (7.4)	83.0 (7.2)
Male	36.0	33.6	42.1
White race	89.9	90.7	87.8
Married	33.2	32.8	34.4
Household income <$25,000	64.6	63.6	67.1
Comorbidities, %
Barthel index, mean (SD)	37.1 (22.9)	41.9 (22.2)	25.3 (20.0)
Charlson comorbidity score, mean (SD)	0.7 (1.3)	0.6 (1.1)	1.0 (1.6)
Body mass index, mean (SD)	26.1 (6.5)	26.5 (6.4)	25.0 (6.7)
Dementia	18.9	16.2	25.6
Heart failure	22.5	18.9	31.2
Hospital length of stay – days, mean (SD)	10.3 (11.0)	9.1 (9.0)	13.3 (14.5)
Active conditions at SNF admission, %
Cognitive impairment
None/borderline	66.4	70.5	56.0
Moderate	15.0	14.8	15.7
Severe	18.6	14.7	28.3
Delirium	9.4	7.2	15.0
Dyspnea	7.9	5.9	12.9
Edema	37.6	36.7	39.7
Pressure ulcer	18.0	14.4	26.7
Frequent pain	33.7	35.7	28.6
Treatments at SNF admission, %
Antianxiety or hypnotic medication	26.9	26.5	29.1
Antipsychotic medication	8.8	7.0	13.1
Indwelling catheter	2175	17.8	31.4
Feeding tube	3.4	1.2	8.9
Intravenous medications	59.2	57.6	63.2
Oxygen	33.0	29.5	41.7
SNF characteristics, %
Ownership
For-profit	63.2	61.4	67.6
Government	3.2	3.1	3.4
Not for-profit	33.1	34.9	28.7
Size (beds)
1–77	25.4	27.2	20.9
78–121	34.0	32.7	37.4
122–154	16.0	16.0	16.0
154–889	24.7	24.2	25.8

Results for other variables included in the predictive modeling are listed in Supplemental Table S2.

Accuracy of the models was assessed based on their optimism-adjusted c-statistic and Brier Score. Optimism adjustment accounts for overly positive assessments of model performance on external data due to overfitting, and was performed using the validate function in the ‘rms’ package in R, using 250 bootstrap iterations. Calibration of the models (to assess the consistency of prediction across the range of predicted values) was assessed using the calibration function in the ‘rms’ package, also using 250 bootstrap iterations.

We anticipate that the prediction model might be used in three different ways: at the best balance of sensitivity and specificity (measured by Youden’s Index), at a high level of sensitivity (to enhance the negative predictive value of a post-discharge event, identifying patients likely to do well), and at a high level of specificity (to maximize the positive predictive value and identify those nearly certain to have an event). Sensitivity, specificity, negative and positive predictive values, and positive and negative likelihood ratios at Youden’s Index and towards the extremes of sensitivity and specificity were calculated. We also created a nomogram that could be used to calculate an individual patient’s likelihood of experiencing the composite outcome. Data preparation and analyses were conducted using SAS version 9.4 and R versions 3.3.1 and 3.4.0.¹⁷ The study was approved by the Colorado Multiple Institutional Review Board.

RESULTS

Of the 2043 previously community-dwelling beneficiaries who were hospitalized and discharged to a SNF for post-acute care, 589 (28.8%) experienced at least one of the three outcomes, with readmissions (n= 396, 19.4%) most common, followed by mortality (n= 215, 10.5%) and long stay (n= 72, 3.5%). The first, second, and third quartiles of the population had predicted probabilities of 13, 24, and 41%, respectively. The minimum and maximum predicted probabilities were approximately 2 and 91%. Patients experiencing one of the endpoints appeared significantly more ill across a range of demographics, comorbidities, and treatment received.

The final model (termed the SNF Prognosis Score) included five factors: Barthel Index, Charlson-Deyo comorbidity score, hospital length of stay, a diagnosis of heart failure, and presence of an indwelling catheter. The discrimination of the model was moderate to high (optimism-corrected c-statistic 0.75, Figure 1) with a similar level of accuracy (Brier score 0.17) and high calibration when observed versus expected events were plotted (Figure 2). The model did not exhibit any regions with particularly strong biases.

Figure 1.

Receiver operating characteristic curve for final model

The receiver operating characteristic curve with labeled Youden’s Index (best trade-off of sensitivity and specificity), including corresponding values for: Se (sensitivity), Sp (specificity), PPV (positive predictive value), NPV (negative predictive value), LR+ (positive likelihood ratio), LR− (negative likelihood ratio). This figure also displays the distribution of events in a histogram at top.

Figure 2.

Calibration of SNF prognosis score

The calibration plot demonstrates observed outcomes compared to expected outcomes across the range of probabilities; perfect calibration is demonstrated by the “Ideal” dotted line while our initial results are indicated by the “Apparent” line; “Bias-corrected” refers to the performance after bootstrap resampling.

Sensitivity and specificity were moderate at the optimal value of the Youden’s index, with better negative predictive value than positive (Table 2). At the extremes of sensitivity and specificity, positive (>9) and negative (<0.2) likelihood ratios were consistent with strong prediction of outcomes. A nomogram that could be used clinically to predict an individual patient’s risk of experiencing the composite endpoint is presented in Figure 3A. For example, a patient leaving the hospital after a 9-day hospital stay (+25 points) with a Barthel Index of 30 (+70 points), 4 comorbidities (+23 points), who has heart failure (+15 points) and a urinary catheter (+13 points) has a total of 146 points – corresponding to a roughly 66% chance of sustaining one of these potentially adverse events during the SNF stay (Figure 3B). In a sensitivity analysis including variables with high missingness, there were no differences in the model selected and the C-index and Brier score were nearly identical (Supplementary Table S3).

Table 2.

Prediction model characteristics at different points on the receiver operating characteristic curve

Sensitivity	Specificity	Positive predictive value	Negative predictive value	Positive likelihood ratio	Negative likelihood ratio
0.99	0.10	0.31	0.94	1.1	0.2
0.90	0.40	0.38	0.91	1.5	0.3
0.64	0.75	0.51	0.84	2.6	0.5
0.38	0.90	0.60	0.78	2.8	0.5
0.10	0.99	0.80	0.73	9.7	0.9

Values are presented for 90 and 99 percent sensitivity and specificity, as well as at Youden’s Index where there is optimal trade-off of sensitivity and specificity, assuming equal weighting of false positives and false negatives.

Figure 3.

Nomogram for SNF prognosis score

Scoring for the continuous predictors (Barthel Index, Charlson Deyo and previous inpatient length of stay), may be determined by finding the value of the predictor on the top of the scale, and then reading the points below the line (Figure 3A). For example, a Barthel Index score of 20 would contribute 80 points to the total score. Points for indwelling catheter and/or CHF are added if the patient has the condition. Points from each of the five predictors are added and the total points are used to determine the predicted probability of the composite outcome; the example from the Results section is displayed in the bottom Figure (3B).

DISCUSSION

In this nationally-representative sample of community-dwelling Medicare beneficiaries, more than a quarter experienced a hospital readmission, long stay in SNF, or died during their SNF stay. The SNF prognosis score could accurately identify the risk of one of these events occurring among community-dwelling older adults discharged from a hospital to SNF, using five predictors available in the initial MDS assessment. This is the first tool to our knowledge that could allow clinicians to stratify SNF patients into different levels of risk.

Such a tool could be used to identify patients at low risk of an event; these patients may require less intensive services or a shorter duration of care. It could also be used to identify patients at high risk of an event, who may require a high level of monitoring or additional services. There is growing evidence that inadequate monitoring and lack of early intervention is responsible for preventable readmissions from SNF,^18–20 but the constrained environment of SNF means not all patients can or should receive more intensive interventions. Some of the high-risk patients may also benefit from a palliative care discussion, to make sure hospital readmission, potential death, or a long stay in a nursing home is compatible with their goals.²¹ In prior work, patients and caregivers often struggle with estimating benefits and risks of SNF and would welcome a tool to help with decision-making.²² We have intentionally not prescribed cut-points of risk as different patients and providers may set different thresholds according to their goals and particular situation; we see this information as supplementing a shared decision-making discussion.

From a societal perspective, this tool may lead to improved healthcare value in post-acute care, since it could better align resources with patient needs. Without such a tool, it may be difficult to identify low, medium, and high-risk populations transitioning to SNF and match resources to these risk levels, a key part of making such transitions cost-effective.⁷ SNFs shortly will face penalties for hospital readmissions and will have their 100-day community discharge rates publicly reported. Our measure has several variables in common with proposed risk adjustment for the SNF 30-day all-cause readmission measure (namely length of hospital stay, comorbidity burden, and heart failure) but the proposed risk adjustment does not incorporate measures of function or invasive treatments.²³ Incorporating these factors may improve risk adjustment.

This tool may fill a needed gap as prior risk prediction scores have either identified “complex transitions” among discharges from the hospital (not just to SNF)¹⁰ or have adapted scores derived from discharges home with less accurate prediction than the current model.²⁴ There are several studies evaluating prognosis scores for mortality in long-term nursing home residents,⁸ and identifying risk factors for hospital readmission in patients discharged to SNF², but not evaluating other outcomes in this important population. Strengths include derivation in a nationally-representative cohort of Medicare beneficiaries with the ability to use both survey and claims data to identify important risk factors for adverse outcomes. Additionally, nearly all the risk factors we identified have also been identified as associated with one or more of the composite outcomes in other cohorts.^8,25–29

These results should be interpreted in the context from which they were derived. We were surprised that dementia and prior SNF stays were not included in the final predictive model. This could be for three reasons: first, that dementia is often poorly ascertained in administrative data; second, that other factors (such as functional status) may correlate with these factors reducing their influence as predictors; and third, these factors may be more important predictors of individual outcomes (such as long stay) that could be missed in our composite endpoint.

An important caveat is that these events are not always “adverse.” A hospital readmission could be life-saving, a long stay in the SNF could result in significant improvement in functional limitations⁴, and even mortality following palliative principles may not be seen by patients or caregivers as necessarily “adverse.” These results are likely an underestimate of how common these events are in all patients discharged to SNF, since the cohort enrolled was previously community-dwelling. For example, our long-stay rate was much smaller than that in national Medicare data, which includes all patients discharged from hospital to SNF, such as those who were in long-term institutional care before the hospitalization.³⁰

Due to the relative size of the cohort, we were unable to feasibly split into a derivation and validation cohort, and the tool must be evaluated in a larger external validation cohort. We also used a composite endpoint with relevance to patients, clinicians, and health care systems but did not have sufficient events to individually evaluate each component of this endpoint. We have previously evaluated risk factors for readmission as a sole outcome in this population.² The prognosis score predicts a composite of these three outcomes, but not functional recovery per se. Several components of the tool (such as the Barthel Index) may not be as commonly used in clinical practice. The deployment of the Continuity Assessment Record and Evaluation tool may improve assessment and documentation of functional status at time of hospital discharge.³¹ We do not have access to hospital data in this sample limiting our ability to include a discharge diagnosis; we anticipate disease-specific prediction models may be of even more benefit and are an important focus for future work. We used the initial MDS assessment as a proxy for the patient’s state at SNF admission, but the dynamic nature of patient care around a transition to SNF may mean the fidelity of this assessment depending on which day it was completed is likely variable. In our sample, 74% of assessments were completed within one week and 96% by two weeks after SNF admission. SNFs intending to use this prognosis score would likely need to inquire about the relevant components of the score on admission, rather than waiting for the MDS assessment to be complete.

Next steps for evaluating the tool should include evaluating the impact of subsequent SNF stays on prognosis, identifying missing factors from larger datasets that may substantially improve the score (ie, vital sign stability before discharge), assessing the impact of in-person assessments of physical function such as gait speed or grip strength rather than more difficult to calculate scores such as the Barthel Index, deriving scores for individual components of the composite endpoints, and identifying how much a second measurement of function - in order to calculate trajectory of change - improves prognostication. Given the significant influence of function on the current score and in previous reports^25,32,33, this second measurement likely should focus on functional capacity.

Older adults frequently experience potentially adverse outcomes in transitions to SNF from the hospital, including high readmission and mortality rates^2,3 and low community discharge rates.³⁰ This novel score derived from a nationally-representative sample of Medicare beneficiaries may assist with decision-making about post-acute care in the hospital, and could be used to better match resources with patient risk for adverse events.

Supplementary Material

Supp info

Figure S1 – Imputation assessment for missing data

Table S1 – Crosswalk of Minimum Data Set 2.0 and 3.0 variables

Table S2 – List of other variables included in predictive modeling

Table S3 – Results of sensitivity analyses including varying p-value cut-offs and inclusion of variables with high missingness

Impact statement.

We certify the work is novel and are not aware of other prognosis scores for patients receiving post-acute care in a skilled nursing facility. Such as score will help clinicians match interventions to patient risk, something increasingly important as skilled nursing facilities are increasingly held responsible for patient outcomes, such as hospital readmission and community discharge rates.