Association Between 5-Star Nursing Home Report Card Ratings and Potentially Preventable Hospitalizations

Abstract

Nursing homes’ publicly reported star ratings increased substantially since Centers for Medicare & Medicaid Services’s Nursing Home Compare adopted a 5-star rating system. Our objective was to test whether the improvements in nursing home 5-star ratings were correlated with reductions in rates of hospitalization. We hypothesized that increased attention to 5-star star ratings motivated nursing homes to make changes that improved their star ratings but did not affect their hospitalization rate, resulting in a weakened association between ratings and hospitalizations. We used 2007-2010 Medicare hospital claims and nursing home clinical assessment data to compare the correlation between nursing home 5-star ratings and hospitalization rates before versus after 5-star ratings were publicly released. The correlation between the rate of hospitalization and a nursing home’s 5-star rating weakened slightly after the ratings became publicly available. This decrease in correlation was concentrated among patients receiving post-acute care, who experienced relatively more hospitalizations from best-rated nursing homes. The improvements in nursing home star ratings after the release of Medicare’s 5-star rating system were not accompanied by improvements in a broader measure of outcomes for post-acute care patients. Although this dissociation may be due to better matching of sicker patients to higher-quality nursing homes or superficial improvements by nursing homes to increase their ratings without substantial investments in quality improvement, the 5-star ratings nonetheless became less meaningful as an indicator of nursing home quality for post-acute care patients.

• What do we already know about this topic?

• Nursing homes’ publicly reported star ratings increased substantially since Centers for Medicare & Medicaid Services’s Nursing Home Compare adopted a 5-star rating system; however, whether these changes represent true improvements in nursing home quality is unknown.

• How does your research contribute to the field?

• The improvements in nursing home star ratings after the release of Medicare’s 5-star rating system were not accompanied by improvements in potentially preventable hospitalizations—a broader measure of outcomes for post-acute care patients.

• What are your research’s implications toward theory, practice, or policy?

• Our findings present empiric evidence in support of including broad-based measures of quality such as hospitalization measures in public reporting in an attempt to measure and report on multiple dimensions of health care quality to the public.

Introduction

Public reporting of health care provider quality is often promoted as an effective means to improve patient outcomes and has thus been broadly embraced by private and public payers. By measuring and then making provider quality information publicly available to patients and referring providers, public reporting aims to improve health care quality through competition. That is, the average quality of care patients receive should increase as high-quality providers gain more market share and low-quality providers work to improve their performance to avoid losing market share. Empirical evidence, however, is mixed. Although some early studies found evidence that public reporting was associated with quality gains by hospitals,¹ nursing homes,^2-4 and individual providers,⁵ these findings were not consistent across settings.

Nursing homes represent an important group of health care providers that are subject to public reporting. Most nursing homes in the United States provide 2 distinct types of services: (1) post-acute care to patients who require short-term rehabilitation or skilled nursing care after discharge from the hospital and (2) long-term care to patients with significant functional impairments who are no longer able to manage independently in the community. Nursing home care accounts for significant Medicare and Medicaid spending. In 2015, Medicare fee-for-service spending for post-acute care stays accounted for $29.8 billion,⁶ and state Medicaid programs spent $54.8 billion on long-term care in nursing homes.⁷ Despite a long history of stringent regulatory and reporting requirements, numerous studies document persistent deficits in nursing home quality. For example, the rates of avoidable hospitalizations from the worst-performing nursing homes in 2013 were almost double those from the nursing homes with the lowest rates.⁶

Nursing homes have been subject to public reporting since 2002, when the Centers for Medicare & Medicaid Services (CMS) launched the Nursing Home Compare website for Medicare- and Medicaid-certified nursing homes. The website included information on regulatory compliance, staffing measures, and 10 clinical quality measures.⁸ While evaluations found that Nursing Home Compare improved nursing home performance on some of the quality measures,^3,9 the effects were small and did not extend to performance on broader, untargeted measures of quality such as rehospitalization rates.⁹

In an effort to increase consumer use of the report cards, Nursing Home Compare adopted a 5-star rating system in December 2008. Individual measure scores were combined into 3 subscores (staffing, deficiencies, and clinical quality measures) that were in turn combined into a global rating. All 4 ratings (the global rating and the 3 sub-scores) were displayed as stars, ranging between 1 and 5 stars. By adopting a familiar and simple rating system, the 5-star Nursing Home Compare became easier for consumers to use and understand.¹⁰ In fact, consumer awareness and use of Nursing Home Compare appear to have increased since the implementation of the 5-star Report Card, with more consumers choosing 5-star facilities after the ratings became available.¹¹

There is also some evidence that nursing home awareness of their quality ratings increased under the star system. Since the 5-star rating system was implemented, nursing homes’ reported star ratings increased rapidly and substantially.¹² The proportion of nursing homes with 4- or 5-star ratings increased from 37% in 2009 to 54% in 2014.^13-15 However, there has been some skepticism about whether these changes represent true improvements in nursing home quality.

Conceptual Framework

Public reporting of health care quality aims to improve outcomes through 2 mechanisms—by giving consumers information that help them identify and choose high-quality providers and by incentivizing low-quality providers to improve. However, fundamentally improving quality of care requires significant financial resources by providers. When faced with pressure to improve quality of care, it is possible that some providers will “teach to the test”—that is make superficial changes in their care delivery that will improve their measured performance without investing in true quality improvement in a broader sense. Prior literature has found evidence of nursing homes shifting expenditures toward measured quality¹⁶ and that when nursing homes have improved their publicly reported performance, other quality measures that are not targeted by public reporting have not improved.⁶

In the worst case, teaching to the test may result in crowding out of unmeasured performance—improvements in measured performance while untargeted (and unmeasured) performance worsens. Holmstrom and Milgrom¹⁷ developed a theory of multitasking, which predicts that measuring and rewarding quality in some areas may harm quality in other areas. This is specifically the case when quality is multidimensional and when quality improvement efforts target only some dimensions of quality, which is typical in health care where it is difficult (or impossible) to measure all aspects of quality. Because quality is multidimensional, multitasking theory predicts that providers will divert resources away from unrewarded and unmeasured aspects of quality toward those that are measured and rewarded. There is some evidence of multitasking in nursing homes under public reporting, where quality of care outside the scope targeted by public reporting worsened in some cases.^18,19 Evidence also suggests that inadvertent negative consequences of public reporting on unreported quality due to multitasking occurs whether or not the unreported quality measure is a substitute for the publicly reported quality measure (eg, antipsychotic and other medication use,²⁰ physical and chemical restraint use²¹).

Potentially avoidable hospitalizations are important clinical setbacks for patients in nursing homes and have thus become a target for improvement by hospitals, nursing homes, and policymakers. Hospitalization rates capture information about many aspects of nursing home quality, as achieving low hospitalization rates often requires broad-based nursing home resources and staffing to prevent, diagnose, and treat adverse clinical developments without hospital transfer. However, until recently (2016), hospitalization rates from nursing homes were not direct targets of Nursing Home Compare.²² Thus, it is possible that in response to the 5-star rating system, nursing homes made changes that resulted in improvements in their star ratings, but that these changes were too narrow to affect a broader measure of quality such as preventable hospitalizations.

Our study was designed to measure the correlation between changes in a nursing home’s 5-star rating and changes in the broad but (until recently) unreported measure of potentially preventable hospitalization rates, focusing on this change before versus after the release of, and attention to, the 5-star rating. We hypothesized that increased attention to 5-star star ratings motivated nursing homes to make changes that weakened the association between ratings and hospitalizations, making the ratings less meaningful as a broad-based indicator of nursing home quality. No prior work has examined this question using the 5-star nursing home report card. Furthermore, because the 5-star rating is heavily weighted toward measures focused on long-stay residents, with fewer measures capturing quality of care for post-acute care patients, these effects might be larger in the post-acute care population. As nursing homes focus on long-stay quality measures (which are more numerous), post-acute care quality might be crowded out. Last, we hypothesized that, compared with nonprofit nursing homes, for-profit nursing homes would be more likely to teach to the test. To attract larger market share, for-profit nursing homes would be more susceptible to the pressures of public reporting and, to contain costs, for-profit nursing homes would be more likely to make superficial changes that narrowly improve star ratings but do not affect the underlying quality of care (eg, manipulating staffing data to improve ratings or shifting medication use from antipsychotics targeted by quality metrics to other sedating medications that have similar adverse effect profiles but were not targeted by the measures²³).

Methods

Overview of Study Design

Using 2007-2010 data from Medicare claims and nursing home clinical assessments, we measured the change in the association between a nursing home’s 5-star rating and the number of potentially preventable hospitalizations per 30 days before vs after the implementation of 5-star Nursing Home Compare report card in December 2008, controlling for nursing home and patient characteristics.

Data Sources

Patient-level data from the 100% Medicare Provider Analysis and Review (MedPAR) file, Medicare Beneficiary Summary file, and the nursing home Minimum Data Set (MDS) were used to create a longitudinal monthly record of each patient’s hospital and nursing home stays from January 2007 through September 2010. The MedPAR file includes hospital claims and was used to determine hospital admissions and discharge dates, as well as the Charlson comorbidity index.²⁴ The MDS includes detailed patient-level clinical assessment data collected at least quarterly for long-term care patients and more frequently for post-acute care patients. The MDS was used to supplement patient diagnosis information derived from hospital claims as well as to track patient admission and discharge from the nursing home. We merged these data with the publicly available Nursing Home Compare data which included the 5-star rating for each nursing home as well as the 5-star rating for each of the 3 subscores. The 5-star ratings were calculated retrospectively for the time period prior to December 2008 and updated quarterly since December 2008. We used the Online Survey Certification and Reporting (OSCAR) dataset to obtain nursing home characteristics reported annually as part of federal and state certification requirements.

Study Sample

We included all Medicare fee-for-service beneficiaries with a nursing home stay of any length from January 2007 to June 2010, excluding patients covered under Medicare Advantage during the calendar year as they have incomplete data in Medicare claims. We analyzed all nursing home stays as well as post-acute and long-term care stays separately. Post-acute care stays were distinguished from long-stay stays by the presence of a 5-day Medicare assessment in the MDS, which is used as the admission assessment for post-acute care patients in MDS and is required for Medicare billing. We assumed that post-acute care patients transitioned to long-stay if they also had at least one quarterly assessment as the Medicare benefit for post-acute care lasts only 100 days. Our final sample included 5 208 015 unique patients admitted to one of 16 046 nursing homes during the study period, representing 43 734 881 thirty-day nursing home episodes of which 7 659 917 were post-acute and 36 074 964 were long-term care.

Outcome Variable: Hospitalizations

Our primary outcome variable was nursing home hospitalization rates. For both long- and short-stay nursing home residents, low-hospitalization rates are considered a marker of nursing home quality.^25-33 To measure this, we calculated the number of potentially preventable hospitalizations per 30 nursing home days, weighted to account for the number of days the patient spent in the nursing home. One challenge in evaluating the effect of nursing home quality on hospitalizations is that the estimated effect of nursing home care on hospitalizations may be biased by the amount of time patients are exposed to that nursing home’s care.³⁴ To account for the differences in patient exposure to nursing home care, we divided each patient’s record into 30-day episodes of nursing home care from the initial admission to a nursing home through the last discharge during the study period. If a patient was hospitalized during a 30-day episode but returned to the same nursing home within 3 days of hospital discharge, then the days in the nursing home following discharge counted toward the same 30-day episode. If a patient was discharged to a new nursing home, then the day following discharge started a new 30-day episode. If a nursing home episode was less than 30 days, the number of hospitalizations per 30 days was then weighted by the length of the associated episode using the reciprocal of the fraction of 30 days the patient spent in the nursing home. Thus, 30-day nursing home exposure period was the unit of analysis. Defining the episodes as 90 days of nursing home care rather than 30 days resulted in consistent findings.

Potentially preventable hospitalizations, which represent a subset of all hospitalizations, were prespecified from claims data through a review of literature^35-38 and include the following diagnoses: angina, asthma, chronic obstructive pulmonary disease (COPD), cellulitis, congestive heart failure (CHF), dehydration, diabetes, gastroenteritis, seizure disorder, hypertension, hypoglycemia, urinary tract infection (UTI), pneumonia, and severe ear, nose, or throat infection. We used the number of total all-cause hospitalizations per 30 days as a secondary outcome, also weighted by nursing home exposure.

Independent Variables

One of our key independent variables was nursing homes’ 5-star ratings from Nursing Home Compare. Because publicly available 5-star ratings of nursing homes are only available from 2008 on, we recalculated the ratings across our entire study period (2007-2010) by applying a previously described and tested methodology.^11,39

Our other key independent variable is an indicator for whether the 30-day episode occurred before or after publication of the 5-star Nursing Home Compare report card (ie, zero before December 2008 and one after December 2008). This variable was interacted with the 5-star ratings. We created a narrow washout period around December 2008, excluding any 30-day intervals of nursing home care that included December 2008, as the effect of the 5-star report card during this transition period depends on the precise timing of the episode. We also conducted analyses widening the washout period to 90 days and 365 days, which did not affect our findings.

We controlled for patient characteristics including age, gender, race, Charlson comorbidity index,²⁴ additional patient-level clinical information from MDS assessments, including the Barthel functional index,⁴⁰ and comorbidities of pneumonia, UTI, septicemia, antibiotic resistant infection, and the presence of a pressure ulcer, oral feeding tubes, parenteral nutrition, or indwelling urinary catheter. We also controlled for Do-Not-Resuscitate and Do-Not-Hospitalize status reported in the MDS. We also controlled for time-varying nursing home characteristics including size, nursing staffing levels, occupancy, payer case mix, and chain and hospital ownership indicators from OSCAR. A 1-year look-back period was used to calculate all risk-adjustment variables.

Statistical Analysis

To test for changes in the relationship between a nursing home’s 5-star rating and its hospitalization rate after the 5-star ratings became publicly available, we estimated the following using linear regression:

$$\begin{array}{l}{Y}_{i,n,t}={\beta }_{1}Ratin{g}_{n,t}+{\beta }_{2}Pos{t}_t+{\beta }_{3}Rating\\ \times Pos{t}_{n,t}+\beta X_{i,t}+\beta N{H}_{n,t}+{\phi }_n+{\epsilon }_{i,n,t},\end{array}$$ (1)

where i indicates patients, n indicates the nursing home, and t indicates the time period of the 30-day episode (before or after December 2008). Y measures our patient-level outcomes over each 30-day nursing home exposure period (the number of potentially preventable hospitalizations or total hospitalizations per 30 days in the nursing home). These outcomes are estimated as a function of 4 dummy variables indicating the 5-star rating of the nursing home where the episode occurred, omitting a dummy for 1-star (Rating_n,t), whether the episode occurred before or after the release of the 5-star report card (Post_t), and the interaction between the two. We also include a vector of patient controls (X_i,t), a vector of time-varying nursing home characteristics (NH_n,t), and nursing home fixed effects (ϕ). The Huber White sandwich estimator was used in all regressions to account for clustering of observations within nursing homes.

Because more stars represent better quality in the 5-star quality ratings, we expect a negative coefficient for the main effect of the overall 5-star rating on the number of hospitalizations. A negative coefficient would indicate that higher-quality nursing homes have lower hospitalization rates, consistent with the view that hospitalization rates reflect broad-based quality. If our hypothesis is correct, this inverse association between 5-star ratings and hospitalizations would weaken after the implementation of 5-star Nursing Home Compare report card, and we would expect a positive coefficient on the interaction. The sum of the coefficients on the interaction and the main Post effect represents the difference in the number of potentially preventable hospitalizations per 30 days after the implementation of 5-star report card, accounting for patient and nursing home factors. We report the linear combination of ${\beta }_2+{\beta }_3$, which represents the total change in the number of hospitalizations at each star level in the post period, and test whether the linear combination of these 2 coefficients was statistically different from zero.

We repeated the analyses after stratifying by type of stay (post-acute vs long-term care) and profit status.

Statistical analyses were performed using STATA, version 14.0. The study was approved by the University of Pennsylvania Institutional Review Board and the Centers for Medicare and Medicaid Services privacy board.

Results

Table 1 describes characteristics of the 30-day nursing home episodes. Patients receiving care in nursing homes were elderly (mean age 80.2 years) and had high levels of comorbidity (mean Charlson comorbidity index 2.7) and functional disability (mean Barthel index 33.9). Patients in post-acute care were more likely to be male and white and had higher levels of comorbidity, whereas patients in long-term care had higher levels of functional disability (see also Appendix Table A1).

Table 1.

Characteristics of 30-Day Nursing Home Episodes.

	Overall sample	PAC	LTC
n	43 734 881	7 659 917	36 074 964
Female, %	68.6	63.9	69.6
Race, %
White	84.2	87.3	83.6
Black	11.7	9.3	12.2
Other	4.1	3.4	4.2
Age, mean (SD)	80.2 (11.6)	80.0 (10.7)	80.2 (11.8)
Barthel index, mean (SD)	33.9 (25.3)	36.4 (19.8)	33.4 (26.4)
Charlson comorbidity index, mean (SD)	2.7 (2.1)	3.0 (2.5)	2.6 (2.0)

Note. The combined dataset of Nursing Home Compare, the Minimum Data Set, Medicare Provider Analysis and Review, and Medicare Beneficiary Summary Files for 5 208 015 beneficiaries admitted to one of 16 046 US nursing homes between January 2007 and June 2010. The sample includes 3 549 449 PAC and 2 749 397 LTC unique patients. PAC = post-acute care; LTC = long-term care.

Table 2 shows the unadjusted number of potentially preventable hospitalizations, displayed as the number of hospitalizations per 100 patients per 30 days before and after the release of 5-star ratings. Before the release of the 5-star system, there is a consistent and monotonic relationship between the overall 5-star rating and hospitalization rates, with higher-quality nursing homes exhibiting lower hospitalization rates. This aligns with expectations that hospitalization rates are a marker of quality. After the release of the 5-star system, the gradient is reduced, with a smaller span in hospitalization rates between 1-star and 5-star nursing homes. This aligns with expectations that the ratings are becoming less meaningful over time as a broad indicator of quality. In fact, for post-acute care stays, the unadjusted rates of potentially preventable hospitalizations from 2- to 5-star rated facilities were higher after 5-star release compared with before. For the long-term care stays, there was also a monotonic relationship between nursing homes’ overall star ratings and hospitalization rates, though the gradient in hospitalization rates between 1-star and 5-star facilities was smaller than it was for the post-acute population. However, this gradient did not change substantially after the 5-star rating system was released. This suggests that star ratings retained their association with hospitalization rates over time in the case of the long-stay population.

Table 2.

Unadjusted Number of Potentially Preventable Hospitalizations per 100 Patients per 30 Days, by Star Rating.

Overall 5-star rating	All stays		PAC		LTC
	Before 5-star release	After 5-star release	Before 5-star release	After 5-star release	Before 5-star release	After 5-star release
1-star (lowest)	5.2	4.7	12.5	12.3	3.7	3.4
2-star	4.8	4.6	11.9	12.0	3.3	3.2
3-star	4.7	4.5	11.4	11.5	3.2	3.0
4-star	4.4	4.2	10.5	10.9	3.0	2.7
5-star (highest)	4.1	4.0	9.9	10.6	2.7	2.5

Note. Sample means of the unadjusted number of hospitalizations per 100 patients per 30 days by nursing home subgroup, weighted to nursing home exposure (using the reciprocal of the fraction of 30 days the patient spent in the nursing home). PAC = post-acute care; LTC = long-term care.

These observations are upheld in multivariate regression, where we found that the release of the 5-star ratings was associated with a weakening of the association between star ratings and the number of potentially preventable hospitalizations (Figure 1). For all nursing home stays, the number of potentially preventable hospitalizations was highest among 1-star nursing homes and lowest among 5-star nursing homes both before and after the 5-star release. However, the difference in potentially preventable hospitalization rates between 1- and 5-star nursing homes decreased substantially after the release of the 5-star rating system, and the change pre- to post-5-star was statistically significant for 1, 2, and 3-star facilities (where potentially preventable hospitalization rates declined by 0.27, 0.10, and 0.12 per 100 patients per 30 days, respectively). After stratifying by type of stay, we find that the results are similar in the post-acute care population, where rates of potentially preventable hospitalizations declined among 1-star facilities (by 0.52 per 100 patients per 30 days) and increased among 4- and 5-star nursing homes (by 0.28 and 0.57 per 100 patients per 30 days, respectively). These changes are consistent with the hypothesis that there was teaching to the test after the release of the star ratings, as the star ratings became less associated with hospitalization rates. In contrast, among long-term care residents the 5-star rating system remained predictive of potentially preventable hospitalization rates after the 5-star rating system was released.

Figure 1.

Adjusted difference in the number of potentially preventable hospitalizations per 100 patients per 30 days post vs pre 5-star nursing home compare report card release.

Note. The bars represent risk-adjusted hospitalization rates before and after 5-star release for each nursing home star rating. The difference between the bars listed in the table below is the sum of the coefficients on the interaction Rating × Post and the main Post effect and represents the difference in hospitalization rates attributable to the implementation of 5-star reporting, after adjusting for time trends and differences in hospitalization rates. P values refer to statistical significance of the difference between pre-5-star and post-5-star hospitalizations rates compared with zero.

*p < 0.1. **p < .01. ***p < .001.

Figure 2 depicts the changes in the relationship between star rating and number of potentially preventable hospitalizations, stratified by profit status. Overall, we observe the anticipated larger dissociation between star ratings and the number of potentially preventable hospitalizations among for-profit nursing homes compared with nonprofit nursing homes for all stays and for post-acute care stays, though most of the changes in hospitalization rates among post-acute care stays within star ratings are not statistically significant. Stratified analyses for the long-term care episodes did not reveal consistent trends (see also Appendix Table A2).

Figure 2.

Difference in the number of potentially preventable hospitalizations per 100 patients per 30 days post vs pre 5-star, stratified by profit status.

Note. The bars represent risk-adjusted hospitalization rates before and after 5-star release for each nursing home star rating. The difference between the bars listed in the table below is the sum of the coefficients on the interaction Rating × Post and the main Post effect and represents the difference in hospitalization rates attributable to the implementation of 5-star reporting, after adjusting for time trends and differences in hospitalization rates. P values refer to statistical significance of the difference between pre-5-star and post-5-star hospitalizations rates compared with zero.

*p < .1. **p < .01. ***p < .001.

We repeated all analyses using total hospitalizations as the outcome of interest. The findings were generally consistent with the potentially preventable hospitalizations results (Appendix Table A3). Appendix Table A4 reports the net difference in potentially preventable hospitalizations after vs before 5-star between 1-star and higher-star nursing homes. The findings were generally consistent when we excluded patients <65 years of age or extended the washout period to 365 days (Appendix Table A5). After stratifying our sample into stays in nursing homes that increased their 5-star rating vs those that decreased their rating over the study period, we found that the findings were concentrated among facilities that increased their star ratings (Appendix Table A6).

Discussion

We observed a dissociation between the 5-star nursing home ratings and rates of patient hospitalizations from post-acute care nursing home stays after CMS began publishing the 5-star ratings, suggesting that improvements in 5-star ratings were not consistently accompanied by improvements in this broader-based but unmeasured indicator of care quality. While hospitalization rates were generally lower overall after the implementation of 5-star report card, the inverse relationship between the number of hospitalizations and a nursing home’s 5-star ratings weakened for post-acute care patients. In some cases (among patients receiving post-acute care in 4- and 5-star nursing homes), potentially preventable hospitalization rates actually went up after the 5-star Nursing Home Compare report card was implemented. While there has been a consistent downward trend in rates of hospitalizations from nursing homes,⁴¹ our estimates accounted for these baseline trends. We also found that the dissociation was concentrated among for-profit nursing homes.

There are several possible explanations of these findings. One is that nursing homes are “teaching to the test.” We not only find some evidence of a decreased correlation between star rating and hospitalization rates after the release of the 5-star ratings, we also find some evidence suggestive of crowd-out. As 5-star ratings include more measures aimed at long-term care patients than post-acute care patients, nursing homes may focus more on long-stay quality at the expense of post-acute care quality. Indeed, hospitalization rates actually went up for post-acute care patients in 4- and 5-star nursing homes after 5-star ratings’ release. Furthermore, for-profit nursing homes that we expected a priori to be most likely to “teach to the test” had the largest dissociations between star ratings and hospitalization rates. Nursing homes’ increased attention to 5-star ratings since their release may have resulted in nursing homes making specific improvements that resulted in higher ratings rather than improving broad-based quality that was unmeasured by 5-star (such as potentially preventable hospitalizations).

It may be surprising that we did not observe an improvement in hospitalization rates among post-acute care patients over this time period, as it was a time when increasing attention was focused on high readmission rates and those rates began declining nationally.⁴¹ However, we control for secular trends in readmission rates that would otherwise be observed in this groups and identify only differences within nursing homes that are related to changes in a nursing home’s star rating. While Nursing Home Compare incorporated rehospitalization measures as part of its clinical quality indicators in 2016, hospitalization rates do not capture other aspects of quality (such as patient satisfaction, for example). Our findings present empiric evidence in support of including broad-based measures of quality such as hospitalization measures in public reporting in an attempt to measure and report on multiple dimensions of health care quality to the public. Development and inclusion of more such quality measures could further improve nursing home care.

There are other possible explanations for our findings. One is patient selection. If more complex and sicker patients were better matched to higher-star nursing homes after 5-star release, while worse-rated nursing homes selected less risky patients, then we might expect to see similar results. Although we did not observe such differences in the distribution of patients across nursing homes before vs after the 5-star ratings were released (Appendix Table A1), there may have been unobserved trends in patient case mix. However, we observed similar trends in unadjusted models that do not control for patient characteristics, suggesting that changes in case mix are unlikely to explain the dissociation observed in our study. Nevertheless, it is possible that for-profit nursing homes with higher 5-star ratings were incentivized to select high-risk patients (by partnering with hospitals willing to share their readmission incentives, for example). Selection of lower risk patients by poorly rated facilities would also explain the decrease in hospitalization rates among 1-star facilities after 5-star release.

We did not observe an analogous weakening of the correlation between the outcomes and 5-star ratings for long-term care stays. Given the differences in patient and payer characteristics between the two populations, it is possible that nursing home processes of care delivery are sufficiently different to explain this discrepancy. Within the clinical quality subcategory of ratings, quality measures used to calculate this rating differ between short-term stay and long-term care. It is possible long-term care quality measures are more relevant for broad outcomes such as hospitalizations, such that efforts to improve ratings spill over to quality more generally. Furthermore, there were more long-term care clinical quality measures than post-acute care measures in Nursing Home Compare. In 2008, for example, 7 of the 10 clinical quality measures included were long-term care measures, whereas 3 were post-acute care measures (delirium, pain, and pressure ulcers). Thus, nursing homes’ efforts to improve quality may have been shifted toward long-term care because 5-star ratings were weighted toward long-term care measures. One possible solution may be creating 2 separate 5-star ratings: one for post-acute and one for long-term care in each nursing home.

This study has several limitations. First, the observational retrospective study does not test whether the 5-star reporting system caused changes in hospitalization rates. However, it examines the important question of whether these measures are correlated with one another, as one might expect them to be, and whether the correlation changes over time. Second, although we controlled for unobserved time-invariant nursing home characteristics using fixed effects, we were unable to adjust for unobserved time-varying market and nursing home characteristics that may influence the correlation between star ratings and hospitalizations. This limits our ability to identify the specific mechanisms behind the changes in correlation. Third, we selected an unreported measure of quality—potentially preventable hospitalizations—because it is generally considered to broadly capture quality of care and is thought to be amenable to improvement through better care delivery. However, some hospitalizations may be appropriate even if categorized as potentially preventable. Furthermore, hospitalizations may be a flawed indicator of quality for long-term patients since nursing homes have an incentive to send patients back to the hospital because their per diem reimbursement from Medicare for post-acute care is greater than private pay or Medicaid long-stay reimbursement rates. This measure may also not capture other aspects of quality that may have improved through nursing home efforts to increase their ratings (such as patient satisfaction, for example).

Our findings add to the body of evidence on unanticipated consequences of public reporting. They also highlight the challenges of creating a high-stakes public reporting system that is acceptable to the facilities being measured and also reports meaningful straightforward information to the consumer. Increased scrutiny and audit of self-reported performance data may address some of these concerns of “gaming” by the providers. For staffing data, for example, CMS is incorporating payroll-based reporting to improve its quality. In its current form, the overall 5-star Nursing Home Compare rating still appears to correlate with the broader measure of hospitalization, but the correlation has been decreasing, particularly for post-acute care patients. The recent addition of a rehospitalization measure to Nursing Home Compare may alleviate this particular concern, but the challenge of incentivizing broad-based improvements will remain. Monitoring of “unreported” quality measures may be another way to address the “teaching to the test” phenomenon; however, such an approach is likely to face challenges from participating providers.

Appendix

Table A1.

30-Day Episode Characteristics Over the Entire Study Period and Stratified by Whether the Episode Occurred Before or After 5-Star Nursing Home Report Card.

	Before			After
	All	PAC	LTC	All	PAC	LTC
N	23 341 345	4 201 089	19 140 256	20 393 536	3 458 828	16 934 708
Female, %	68.7	64.3	69.6	68.5	63.6	69.5
Race, %
White	84.5	87.5	83.7	84.0	87.1	83.4
Black	11.6	9.1	12.2	11.8	9.4	12.3
Other	3.9	3.4	4.1	4.2	3.5	4.3
Age, mean (SD)	80.5 (11.4)	80.0 (10.6)	80.6 (11.6)	79.8 (11.7)	79.9 (10.8)	79.8 (11.9)
Barthel index, mean (SD)	34.5 (25.5)	36.8 (20.0)	34.0 (26.5)	33.3 (25.2)	36.0 (19.5)	32.7 (26.1)
Charlson comorbidity index, mean (SD)	2.7 (2.1)	3.0 (2.5)	2.6 (2.0)	2.6 (2.1)	3.0 (2.5)	2.6 (2.0)
Pneumonia, %	4.0	9.7	2.8	3.7	9.7	2.5
UTI in last 30 days	11.3	19.2	9.5	10.6	18.8	8.9
Pressure ulcer in past 7 days	10.6	19.8	8.5	9.5	18.5	7.7
Feeding tube	5.6	4.1	5.9	5.3	4.0	5.5
Septicemia	0.7	1.9	0.5	0.7	1.9	0.4
Parenteral nutrition	3.2	9.9	1.8	2.6	7.7	1.6
Indwelling urinary catheter	9.2	19.8	6.9	8.1	17.4	6.2
Antibiotic resistant infection	1.9	3.5	1.6	1.9	3.5	1.6
Do not resuscitate	48.2	28.8	52.5	48.8	28.1	53.0
Do not hospitalize	2.5	1.1	2.8	2.7	1.0	3.1

Note. A total of 3 409 134 unique patients had 23 341 345 episodes pre-5-star and 3 021 604 patients had 20 393 536 episodes post-5-star. For PAC, 2 098 680 unique patients had 4 201 089 episodes pre-5-star and 1 728 294 patients had 3 458 828 episodes post-5-star. For LTC, 1 890 623 unique patients had 19 140 256 episodes pre-5-star and 1 722 773 patients had 16 934 708 episodes post-5-star. PAC = post-acute care; LTC = long-term care; UTI = urinary tract infection.

Table A2.

Difference in the Number of Potentially Preventable Hospitalizations per 100 Patients per 30 Days Post vs Pre 5-Star Overall and Stratified by Profit Status.

	All stays			Post-acute care			Long-term care
		Profit status			Profit status			Profit status
	Overall	Profit	Nonprofit	Overall	Profit	Nonprofit	Overall	Profit	Nonprofit
Cumulative effect of Post and Rating × Post at each star level
One star	−0.266*** (6.21)	−0.314*** (6.55)	0.028 (0.25)	−0.524** (2.90)	−0.647** (3.28)	−0.044 (0.08)	−0.195*** (5.34)	−0.224*** (5.43)	−0.005 (0.05)
Two star	−0.102* (2.53)	−0.079 (1.63)	−0.127 (1.52)	−0.253 (1.49)	−0.180 (0.92)	−0.447 (1.21)	−0.074* (2.21)	−0.047 (1.17)	−0.101 (1.49)
Three star	−0.120** (2.98)	−0.161** (3.19)	−0.043 (0.57)	−0.045 (0.28)	−0.126 (0.64)	0.173 (0.58)	−0.158*** (4.80)	−0.189*** (4.64)	−0.122* (1.91)
Four star	−0.056 (1.38)	−0.108* (1.86)	0.015 (0.24)	0.283* (1.79)	0.138 (0.65)	0.415* (1.67)	−0.154*** (4.80)	−0.185*** (3.94)	−0.112* (2.26)
Five star	0.041 (0.59)	−0.009 (0.09)	0.128 (1.26)	0.570* (2.09)	0.623 (1.53)	0.598 (1.51)	−0.106* (1.92)	−0.148* (1.71)	−0.051 (0.70)
Constant	0.468*** (1.83)	0.659* (1.97)	0.122 (0.26)	1.015 (0.96)	0.138 (0.11)	−2.342 (1.48)	0.279 (1.31)	0.530* (1.89)	−0.270 (0.71)
Covariates	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Nursing home FE	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Patient-30-days, n	43 734 881	30 054 969	10 914 600	7 659 917	5 371 884	1 988 976	36 074 964	24 683 085	8 925 624
Nursing homes, n	16 046	11 331	4 672	15 259	10 966	4398	16 042	11 328	4671

Note. The numbers in the table are the sum of the coefficients on the interaction Rating × Post and the main Post effect, which represent the difference in the number of potentially preventable hospitalizations per 30 days after the implementation of 5-star report card compared with before, accounting for patient and nursing home factors. FE = fixed effects.

^*p < .1. **p < .01. ***p < .001. T-statistic in parentheses.

Table A3.

Difference in the Number of Total Hospitalizations per 100 Patients per 30 Days Post vs Pre 5-Star Overall and Stratified by Profit Status.

	All stays			Post-acute care			Long-term care
		Profit status			Profit status			Profit status
	Overall	Profit	Nonprofit	Overall	Profit	Nonprofit	Overall	Profit	Nonprofit
Cumulative effect of Post and Rating × Post at each star level
One star	−1.217*** (12.01)	−1.340*** (11.84)	−0.568* (2.24)	−2.475*** (5.79)	−2.871*** (6.14)	−0.226 (0.18)	−0.796*** (9.66)	−0.833*** (8.99)	−0.663** (3.22)
Two star	−0.967*** (10.54)	−0.961*** (8.71)	−0.933*** (4.74)	−2.451*** (6.36)	−2.385*** (5.39)	−2.120* (2.47)	−0.627*** (8.27)	−0.573*** (6.22)	−0.731*** (4.66)
Three star	−0.706*** (7.60)	−0.829*** (7.07)	−0.517** (3.00)	−1.456*** (3.88)	−1.768*** (3.83)	−0.594 (0.86)	−0.587*** (8.00)	−0.648*** (6.97)	−0.540*** (3.97)
Four star	−0.516*** (5.63)	−0.648*** (5.02)	−0.373* (2.52)	−0.471 (1.32)	−0.994* (2.12)	0.172 (0.29)	−0.560*** (7.99)	−0.631*** (6.24)	−0.484*** (4.49)
Five star	−0.533** (3.24)	−0.737** (3.01)	−0.498* (2.02)	−0.811 (1.18)	−0.783 (0.84)	−1.123 (1.01)	−0.556*** (4.44)	−0.775*** (3.95)	−0.451** (2.72)
Constant	15.777*** (26.20)	16.383*** (20.42)	15.008*** (13.70)	54.184*** (24.07)	56.838*** (18.56)	49.116*** (12.42)	7.403*** (15.66)	8.175*** (13.52)	−5.715*** (6.79)
Covariates	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Nursing home FE	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Patient-30-days, n	43 734 881	30 054 969	10 914 600	7 659 917	5 371 884	1 988 976	36 074 964	24 683 085	8 925 624
Nursing homes, n	16 046	11 331	4672	15 259	10 966	4398	16 042	11 328	4671

Note. The numbers in the table are the sum of the coefficients on the interaction Rating × Post and the main Post effect, which represent the difference in the number of hospitalizations per 30 days after the implementation of 5-star report card compared with before, accounting for patient and nursing home factors. FE = fixed effects.

^*p < .1. **p < .01. ***p < .001. T-statistic in parentheses.

Table A4.

Difference in the Number of Potentially Preventable Hospitalizations per 100 Patients per 30 Days Post vs Pre 5-Star at Each Star Level Compared With 1-Star.

	All stays			Post-acute care			Long-term care
		Profit status			Profit status			Profit status
	Overall	Profit	Nonprofit	Overall	Profit	Nonprofit	Overall	Profit	Nonprofit
Relative effect of Post at each star level compared with 1-star
One star	ref	ref	ref	ref	ref	ref	ref	ref	ref
Two star	0.164** (2.81)	0.235** (3.41)	−0.156 (1.13)	0.271 (1.12)	0.467* (1.70)	−0.491 (0.78)	0.121* (2.43)	0.176** (3.03)	−0.096 (0.85)
Three star	0.146* (2.47)	0.153* (2.17)	−0.072 (0.53)	0.479* (1.97)	0.522* (1.86)	0.129 (0.21)	0.037 (0.74)	0.035 (0.59)	−0.118 (1.00)
Four star	0.210*** (3.54)	0.206** (2.71)	−0.013 (0.10)	0.807** (3.35)	0.786** (2.69)	0.370 (0.62)	0.041 (0.83)	0.039 (0.61)	−0.107 (0.98)
Five star	0.307*** (3.75)	0.304** (2.61)	0.099 (0.65)	1.094** (3.33)	1.270** (2.80)	0.554 (0.82)	0.089 (1.35)	0.076 (0.79)	−0.046 (0.38)

^*p < .1. **p < .01. ***p < .001. T-statistic in parentheses.

Table A5.

Sensitivity Analyses: Difference in the Number of Potentially Preventable Hospitalizations Post vs Pre 5-Star.

	All stays of patients >65 years of age			Washout period = 365 days
		Stay type			Stay type
	Overall	PAC	LTC	Overall	PAC	LTC
Cumulative effect of Post and Rating × Post at each star level
One star	−0.220*** (4.80)	−0.426* (2.23)	−0.169*** (4.34)	0.025 (0.30)	−0.082 (0.24)	−0.036 (0.54)
Two star	−0.062 (1.45)	−0.234 (1.32)	−0.045 (1.27)	0.078 (1.06)	0.186 (0.62)	−0.020 (0.32)
Three star	−0.097* (2.28)	0.010 (0.06)	−0.154*** (4.43)	0.075 (1.01)	0.314 (1.10)	−0.094 (1.53)
Four star	−0.035 (0.82)	0.328* (2.01)	−0.147*** (4.32)	0.070 (0.96)	0.424 (1.55)	−0.144* (2.33)
Five star	0.078 (1.08)	0.667* (2.40)	−0.088 (1.55)	0.229* (1.89)	1.083* (2.31)	−0.076 (0.79)
Constant	−0.003 (0.01)	−3.870** (3.41)	−0.221 (0.93)	0.653 (1.47)	0.123 (0.08)	0.354 (0.96)
Covariates	Yes	Yes	Yes	Yes	Yes	Yes
Nursing home FE	Yes	Yes	Yes	Yes	Yes	Yes
Patient-30-days, n	39 531 280	7 059 694	32 471 586	17 864 004	3 250 314	14 613 690
Nursing homes, n	16 022	15 239	16 018	16 020	15 177	16 012

Note. The numbers in the table are the sum of the coefficients on the interaction Rating × Post and the main Post effect, which represent the difference in the number of hospitalizations per 30 days after the implementation of 5-star report card compared with before, accounting for patient and nursing home factors. PAC = post-acute care; LTC = long-term care; FE = fixed effects.

^*p < .1. **p < .01. ***p < .001. T-statistic in parentheses.

Table A6.

Sensitivity Analyses: Difference in the Number of Potentially Preventable Hospitalizations Post vs Pre 5-Star for Nursing Homes With Decreasing or Increasing Stars Post vs Pre 5-Star.

	Decreased star level			Increased star level
		Stay type			Stay type
	Overall	PAC	LTC	Overall	PAC	LTC
Cumulative effect of Post and Rating × Post at each star level
One star	−0.141* (1.93)	−0.731* (2.33)	0.016 (0.26)	—	—	—
Two star	0.158* (2.20)	0.110 (0.37)	0.156* (2.45)	0.065 (0.79)	−0.234 (0.67)	0.144* (2.14)
Three star	0.011 (0.13)	−0.293 (0.84)	0.026 (0.38)	−0.015 (0.87)	0.110 (0.40)	−0.075 (1.28)
Four star	−0.061 (0.51)	−0.192 (0.39)	0.045 (0.46)	0.059 (0.87)	0.371 (1.47)	−0.026 (0.45)
Five star	—	—	—	0.172* (1.78)	0.961** (2.61)	−0.025 (0.31)
Constant	1.204* (2.30)	3.416* (1.65)	0.162 (0.37)	0.803 (1.28)	0.638 (0.26)	0.306 (0.61)
Covariates	Yes	Yes	Yes	Yes	Yes	Yes
Nursing home FE	Yes	Yes	Yes	Yes	Yes	Yes
Patient-30-days, n	10 774 299	1 842 826	8 931 473	13 916 593	2 479 109	11 437 484
Nursing homes, n	8371	7874	8370	10 882	10 358	10 875

Note. The numbers in the table are the sum of the coefficients on the interaction Rating × Post and the main Post effect, which represent the difference in the number of hospitalizations per 30 days after the implementation of 5-star report card compared with before, accounting for patient and nursing home factors. PAC = post-acute care; LTC = long-term care; FE = fixed effects.

^*p < .1. **p < .01. ***p < .001. T-statistic in parentheses.