Changes in Consumer Demand Following Public Reporting of Summary Quality Ratings: An Evaluation in Nursing Homes

Abstract

Objective

Limited consumer use of health care report cards may be due to the large amount of information presented in report cards, which can be difficult to understand. These limitations may be overcome with summary measures. Our objective was to evaluate consumer response to summary measures in the setting of nursing homes.

Data Sources/Study Setting

2005–2010 nursing home Minimum Data Set and Online Survey, Certification and Reporting (OSCAR) datasets.

Study Design

In December 2008, Medicare converted its nursing home report card to summary or star ratings. We test whether there was a change in consumer demand for nursing homes related to the nursing home's star rating after the information was released.

Principal Findings

The star rating system was associated with a significant change in consumer demand for low‐ and high‐scoring facilities. After the star‐based rating system was released, 1‐star facilities typically lost 8 percent of their market share and 5‐star facilities gained over 6 percent of their market share.

Conclusions

The nursing home star rating system significantly affected consumer demand for high‐ and low‐rated nursing homes. These results support the use of summary measures in report cards.

Health care “report cards” are designed to improve the performance of health care markets by enabling consumers to identify and choose high‐quality providers and, by making demand more elastic to changes in quality, giving providers incentives to improve their quality so they can increase demand for their services (Berwick et al. 2003). Despite the face validity of this approach, the evidence that providers that receive a high report card rating are rewarded with an increased market share (or, conversely, that providers that receive a low report card rating lose market share) is mixed (Marshall et al. 2000; Fung et al. 2008). Research spanning health plans (Wedig and Tai‐Seale 2002; Jin and Sorensen 2006; Chernew et al. 2008; Dranove and Sfekas 2008), hospitals (Baker et al. 2003; Cutler et al. 2004), cardiac surgeons (Dranove and Sfekas 2008), and nursing homes (Grabowski and Town 2011; Werner et al. 2012) has found that highly ranked providers do not consistently gain market share.

This inconsistent response of consumers to public reporting has led to efforts to increase consumer usability of report cards. One such effort has focused on making report card information more understandable to consumers. Research has found that consumers have difficulty processing the large number of quality metrics that are often included in report cards (Schultz et al. 2001; Peters et al. 2007) and understanding the relationship between these quality metrics and a provider's overall quality (Sibbald et al. 1996; Hibbard and Jewett 1997). Additional research has found that patients often prefer summary scores which decrease the cognitive burden of using report card information (Schultz et al. 2001; Palsbo and Kroll 2007) and are more likely to be interpreted correctly (Hibbard et al. 2001; Peters et al. 2007).

With these insights in mind, patient‐oriented report cards are increasingly moving toward using summary measures to display providers' overall quality information, which combine multiple measures into one or more summary measures. The Centers for Medicare and Medicaid Services (CMS) uses summary or star‐based measures for Medicare Advantage, Medicare Part D, hospital experience of care, and nursing homes. In addition, they recently announced that they would be moving to star‐based measures for home health care and dialysis centers (Centers for Medicare and Medicaid Services 2014, 2015).

Despite the face validity of using summary measures in report cards, and their corresponding popularity, it is unknown whether this shift has made consumers more responsive to report card information. Our objective was to test consumer response to the introduction of a summary, star rating system by CMS in nursing homes.

Setting

Over 1.5 million people reside in U.S. nursing homes at a cost of over $120 billion per year (Kaiser Family Foundation 2007). Broadly, nursing homes serve two populations—long‐stay and postacute residents. Long‐stay residents are typically chronically ill individuals who spend the remainder of their lives (2 years on average) in a nursing home receiving nonskilled or compensatory care. Their care largely consists of assistance with activities of daily living such as bathing, dressing, eating, toileting, and walking. Eighty‐seven percent of nursing homes also provide skilled, rehabilitative care to individuals following an acute care hospital episode. Postacute care is aimed at a healthy discharge to the community with an average length of stay of 25 days (Banaszak‐Holl et al. 1997; Mor et al. 2003).

Despite the vulnerability of this population, the large number of people at risk for poor outcomes if quality of care is low, and the large number of health care dollars devoted to nursing home care, quality of care in nursing homes has long presented a policy challenge (Institute of Medicine 1986). Recent efforts to improve nursing home quality have focused on publicly reporting quality information. In 2002, CMS released Nursing Home Compare (NHC), a web‐based guide detailing quality of care at over 17,000 Medicare‐ or Medicaid‐certified nursing homes (Centers for Medicare and Medicaid 2002). It included 10 clinical quality measures, 6 of which were measures of quality for long‐stay residents with chronic care needs and 4 of which were measures of quality for patients in postacute care with skilled nursing needs, as well as information on staffing and rates of regulatory deficiencies.

Although the website was actively promoted to consumers with the hope that consumers would use this information to help choose a nursing home, there is little evidence that they did. Among long‐stay nursing home residents, Grabowski and Town (2011) found that the release of nursing home quality information had no discernible effect on market share for nursing homes. Among postacute residents, Werner et al. (2012) found a statistically significant effect of quality ratings for pain control on nursing home demand, but the size of the effect was negligible.

In June 2008, CMS announced it would make significant changes to the NHC rating system by introducing a 5‐star rating system. Starting on December 18, 2008, CMS began publicly rating each nursing home with a “star” rating ranging 1–5 stars. This gives consumers a “snapshot” or simplified look at nursing home quality using a graphical representation (i.e., stars). Nursing home star ratings are based on quality in three domains: health inspections (based on scope and severity of health deficiencies found at state inspection and number of repeat visits needed to confirm the correction of deficiencies), staffing (based on case mix‐adjusted measures of total nursing hours per resident day and RN hours per resident day), and quality measures (based on 10 clinical quality measures).

To assign each nursing home a star rating, CMS first calculates star ratings for these three domains based on specific criteria defined by CMS (Centers for Medicare and Medicaid Services 2010). Then, an overall star rating is calculated by combining the star ratings for the three domains by taking the health inspection results and adjusting the overall rating up or down slightly depending on the staffing and quality measures results.

The website prominently displays each nursing home's overall star rating and the star rating for each of the three domains that make up the overall rating. An example of this information is displayed in Figure 1. Although it is still possible to find the individual quality measures on the NHC site, the star ratings appear first and much more prominently, overshadowing the individual measures they combine. At its implementation, the 5‐star rating system was one of the largest experiments using a summary rating in a public health care report card and today remains one of only a few examples of large‐scale summary rating systems in the United States.

Figure 1.

Example of the Star Ratings Available for Nursing Homes on the Nursing Home Compare Website

Since the launch of the 5‐star rating system, nursing home star ratings have significantly improved, with the percent of nursing homes rated as being 5 star growing from 11.8 to 24.1 percent and the percent of 1‐star facilities declining from 22.7 to 10.5 percent (Abt Associates Inc. 2014). There have been simultaneous concerns raised about the 5‐star rating system, including the credibility of the ratings (Thomas 2014) and that they might increase disparities in nursing home care (Konetzka et al. 2015). No direct empirical evidence is available about whether star ratings affected consumer demand for nursing homes.

Methods

Conceptual Framework and Overview of Empirical Approach

Report cards may affect consumers' use of nursing home in several ways. Most simply, if consumers use the quality information in report cards to inform their choice of nursing homes, report cards will increase demand for highly ranked nursing homes. However, numerous other factors also affect nursing home choice. These include distance from home, size, and whether the nursing home is not for profit (Pesis‐Katz et al. 2013). Additional factors also affect the nursing home to which people go, including the availability of beds, the cost of the nursing home, and who the payer is. Finally, the information in report cards may be used by numerous people who affect nursing home decisions, including the nursing home residents themselves and also their agents, including families, caregivers, and health care workers such as hospital discharge planners and social workers.

Our empirical strategy is to compare the relationship between nursing home demand and nursing home 5‐star ratings before and after these ratings were publicly released. We assume that nursing home demand is driven by consumers (those admitted to the nursing home and/or their agents). We measure each nursing home's 5‐star rating in both the pre‐ and postreporting period, which enables us to control for the correlation between knowledge of the nursing home market through other pathways (market learning) and report card quality. We control for other factors that might drive demand, including nursing home characteristics that are used as signals of quality, distance to a nursing home, and bed availability. We then estimate the report card effect by testing for changes in the correlation between consumer demand and report card scores once this information is publicly disseminated.

Data

Our primary data source is the nursing home Minimum Data Set (MDS) 2.0 from January 2005 to June 2010. The MDS contains detailed clinical data collected at admission and at regular intervals for every resident in a Medicare‐ or a Medicaid‐certified nursing home, allowing us to observe virtually every nursing home admission in the United States over our study period. These data are collected and used by nursing homes to assess needs and develop a plan of care unique to each resident and are used by CMS to calculate the clinical quality measures included in NHC.

We also use the Online Survey, Certification and Reporting (OSCAR) dataset from 2005 to 2010. The OSCAR dataset contains the results of state certification inspection surveys conducted at all nursing facilities participating in the Medicare and Medicaid programs at least once every 15 months. It includes information on nursing home characteristics in addition to staffing and regulatory deficiencies issued during state inspections. OSCAR is the source of information CMS uses for two of the quality measures included in NHC and in the 5‐star rating: nurse staffing intensity and number of regulatory deficiencies.

Study Sample

Our unit of observation is a nursing home admission. We include admissions to all Medicare‐ and Medicaid‐certified nursing homes in the United States that are also included in the 5‐star rating on NHC, defining admissions by the existence of an MDS admissions assessment. Fewer than 2 percent of nursing homes with missing survey information do not have a 5‐star rating on NHC. Among these nursing homes, we take a 20 percent random sample of all nursing home admissions regardless of payer or whether the admission is for postacute or long‐term care. We then construct a choice‐level dataset for all admissions included in our study sample. For each nursing home admission, we define the set of feasible choices of nursing homes (the choice set) as all nursing homes within a fixed driving distance radius around their home—within 30 miles of home in nonrural areas and 50 miles of home in rural areas (using the center of their zip code of residence as a proxy for home). In our final sample, we exclude admissions that did not choose a nursing home within their choice set (9.4 percent of admissions, 85 percent of which were admissions for postacute care) and admissions with only one nursing home within their choice set (<1 percent of admissions). We identify postacute care admissions as those for whom the reason for admission in MDS was a Medicare 5‐day assessment (the code used for Medicare prospective payment for postacute care in nursing homes.

Dependent Variable

Our dependent variable is a binary variable indicating whether an individual chose a particular nursing home, which equals one for the chosen nursing home and zero of all nonchosen nursing homes in the choice set.

Key Independent Variables

To test for changes in nursing home demand with the release of the 5‐star ratings, we use the interaction of our two key independent variables: a variable identifying the timing of the release of the 5‐star ratings and a variable measuring each facility's 5‐star rating. We first define the time period when the 5‐star ratings were publicly available with a pre‐postindicator equal to 1 if a person was admitted to a nursing home on or after December 18, 2008, when the star measures were released publicly (and 0 otherwise).

Second, we use facility‐level report card scores of the overall 5‐star rating. We construct this 5‐star rating for each nursing home over the entire study period based on CMS specifications. As per these specifications, star ratings are updated monthly, allowing nursing homes to change star ratings over the study period. To ensure our calculated 5‐star ratings were accurate, we benchmarked them to the published ratings in 2009 and 2010. We found that our calculated ratings were highly correlated with published ratings, with the correlation coefficient ranging from 0.87 to 0.92 across the 5‐star ratings. In addition, to ensure that using the calculated ratings in place of the published ratings did not affect results of our analyses, we tested the correlation between nursing home demand and both the calculated ratings and the published ratings in 2009 and 2010 and found that the correlation was the same regardless of which ratings were used.

Covariates

In all regressions we control for covariates that might affect nursing home demand. We include several facility‐level characteristics: ownership, total number of beds, occupancy rates, percent of Medicare days, and percent of Medicaid days in each nursing home. We also include a patient‐level variable: the driving distance from an individual's home to each nursing home in his or her choice set, as distance is commonly believed to be an important factor in nursing home demand.

Statistical Model

We test for changes in the correlation between consumer demand and report card scores using a McFadden's choice (conditional logit) model (McFadden 1973), where each individual (or his/her agent) selects the nursing home from the full set of nursing homes within his or her market, selecting the one that maximizes his or her utility:

$$Choic{e}_{i,j,t}=\mathit{\beta }post{2008}_t+\mathit{\gamma }\sum _2^{5}sta{r}_{j,t-1}+\mathit{\delta }post{2008}_t\ast \sum _2^{5}sta{r}_{j,t-1}+{\mathit{X}}_{i,j,t}+{\mathit{\epsilon }}_{i,j,t}$$

In this model, the dependent variable is individual i's “choice” of nursing home (or the nursing home to which the individual was admitted) given all j nursing homes in choice set in month t. This is estimated as a function of a pre‐postvariable indicating the release of the 5‐star information (which drops out of the regression as it does not vary within choice set), a series of four indicators for the five possible overall star ratings (where 1 star is the omitted category), the interaction between the two, and covariates (driving distance and facility characteristics). We lag the star ratings by 1 month from the release date to match the timing of the star rating to the likely timing of nursing home choice. Coefficients in McFadden choice models are typically interpreted as changes in utilities (McFadden 1973; Werner et al. 2012). In this regression, the coefficient on the interaction (δ) represents the change in utility from choosing a facility with a given star rating after the star rating is publicly released (or the log‐odds of being admitted to a facility with a given star rating after the rating is publicly released). We hypothesize that this will be positive for 5‐star facilities relative to 1‐star facilities.

As a robustness check, to test whether any changes in market share that we observe are due to a simple time trend over the entire study period (rather than a change in market share in 2008 when the 5‐star rating system was released), we test for changes in nursing home demand using a false implementation date. This allows us to test whether there was an underlying trend in market learning that might explain any change in nursing home demand we see in late 2008. To do so, we redefine the study period to be entirely within the pre‐5‐star period, defining the preperiod to be 2005 to December 17, 2006, and the postperiod to be December 18, 2006, to December 17, 2008 (just before the release of the 5‐star data). We then use the same empirical setup described above to test for changes in consumer demand for nursing home based on the unpublished 5‐star data.

We also test whether changes in the correlation between the 5‐star rating and nursing home demand could likely be explained by within‐nursing‐home effects rather than between‐nursing‐home effects. That is, whether changes in nursing home demand are driven by an increase in the number of available 5‐star nursing homes (and thus at least in part provider driven) rather than a shift in market share toward 5‐star facilities (and thus at least in part consumer driven). While both within and between effects would result in increased number of admissions to 5‐star facilities, they point to different mechanisms by which public reporting is working and have different policy implications. To explore this difference, we use an intention‐to‐treat framework by using the first publicly reported 5‐star rating available for each nursing home throughout the postrelease period which, in effect, prevents an increase in the number of providers with 5‐star ratings in the postperiod (which is required for a within‐nursing‐home effect). As long as nursing home star ratings increase on average in the reporting period (which they do), any observed changes in nursing home admissions using the nursing home's first star rating can be attributed at least in part to a between‐nursing‐home effect.

Finally, to aid in interpretation, we convert the beta coefficients from the conditional logit model (which represent utilities) to market shares based on assumptions of a typical market. The conversion is based on the following formula where the share of consumers i admitted to nursing home j is:

$$Shar{e}_{i,j}=\frac{\text{exp}(\mathit{\beta }{X}_{i,j})}{{\sum }_n\text{exp}(\mathit{\beta }{X}_{i,j})}$$

where X _i,j is a vector of characteristics of nursing home j observed by consumer i, β is a vector of estimated regression coefficients, and the summation is over all n the nursing homes the consumer could have chosen. The market in our simulation is one where the market share of nursing homes at each star rating in the preperiod is equal (20 percent), there is one nursing home of each type, and all other X characteristics of nursing homes in this simulated market are fixed at the mean levels across all nursing homes. We then estimate the market share in the postperiod using the β vector of coefficients on the interaction term between each 5‐star rating and the “post” indicator. The change in the market shares for each star rating is estimated from the pre‐postdifference in market share.

Results

We include a total of 16,147 nursing homes and 2,316,649 nursing home admissions between 2005 and 2010, 92 percent of whom are admitted to postacute care. Compared to 1‐star nursing homes, 5‐star nursing homes were less likely to be part of a chain (63.5 percent of 1‐star nursing homes vs. 39.2 percent of 5‐star nursing homes), smaller (85 beds vs. 125 beds), more likely to be not for profit (43.7 percent vs. 13.8 percent), had higher percent of Medicaid residents (31.8 percent vs. 16.5 percent), and slightly lower occupancy rates (82.0 percent vs. 79.7 percent). Seven percent of all nursing homes drop out of our sample (or exit the market) during our study period.

As shown in Figure 2, the percentage of admissions to nursing homes ranked as 4 or 5 star increased beginning in early 2009, shortly after the 5‐star rankings were released in December 2008. Simultaneously, the percentage of admissions to 1‐star nursing homes declined.

Figure 2.

Percent of Nursing Home Admissions in Each Star Category. (The vertical line represents the release of the 5‐star report card in December 2008)

The results from the conditional logit model confirm the descriptive results (Table 1). The table displays the log‐odds of admission to a nursing home at each star level in the pre‐2008 period (the coefficients on the noninteracted star ratings) and the marginal change in log‐odds in the post‐2008 period (the coefficients on the interactions between the star ratings and the post‐2008 dummy variable). When exponentiated, these coefficients represent the odds of admission to a nursing home at each star level and can be interpreted in terms of their direction and statistical significance. However, because the magnitude of these coefficients is difficult to interpret, we subsequently convert these results into changes in market share using simulation (Table 3 below).

Table 1.

The Log‐Odds of Being Admitted to a Nursing Home with 2, 3, 4, and 5 Stars before (Coefficients on the Uninteracted Terms) and after the Star‐Based Ratings Were Released in 2008 (Coefficients on the Interaction Terms), All Compared to Being Admitted to a 1‐Star Facility

	All Admissions	PAC Admissions	LTC Admissions
2 stars	−0.051*** (0.003)	−0.055*** (0.003)	−0.031*** (0.009)
3 stars	−0.083*** (0.003)	−0.089*** (0.003)	−0.077*** (0.009)
4 stars	−0.182*** (0.003)	−0.192*** (0.003)	−0.160*** (0.010)
5 stars	−0.403*** (0.004)	−0.426*** (0.004)	−0.296*** (0.014)
post2008*2 stars	0.023*** (0.005)	0.024*** (0.005)	0.030* (0.018)
post2008*3 stars	0.017*** (0.005)	0.019*** (0.005)	0.012 (0.019)
post2008*4 stars	0.017*** (0.005)	0.021*** (0.005)	−0.052** (0.019)
post2008*5 stars	0.078*** (0.007)	0.081*** (0.007)	0.077*** (0.026)
N	180,148,037	164,741,202	15,406,835

Coefficients from choice models such as these are typically interpreted as changes in utility. Robust standard errors in parentheses. All regressions control for the driving distance between the nursing home resident's home zip code and the nursing home, the nursing home's ownership (for profit, not for profit, or government), total number of beds, occupancy rates, whether it is hospital based, percentage of Medicare patient days, and percentage of Medicaid patient days.

Differences between post2008*5 stars and all other interactions are statistically significantly different from zero (p < .001).

*p < .10, **p < .05, ***p < .01.

Table 3.

Simulated Changes in Market Share with the Release of the 5‐Star Report Card

	Pre‐5‐Star Market Share	Post‐5‐Star Market Share	Absolute Change in Market Share (Percentage Point Change)	Relative Change in Market Share (Percent Change)
1 star	20%	18.38%	−1.62	−8.1%
2 star	20%	20.46%	0.46	2.3%
3 star	20%	19.89%	−0.11	−0.5%
4 star	20%	19.98%	−0.02	−0.1%
5 star	20%	21.27%	1.27	6.4%

We assume a market with five nursing homes, each with one‐fifth of the market in the pre‐report card period. The relative change in market share remains the same if we assume a different distribution of market share in the pre‐report card period.

In column 1 of Table 1, showing all nursing home admissions over our study period, the coefficients on the noninteracted 5‐star ratings indicate that patients had higher log‐odds of being admitted to nursing homes with lower star ratings than higher star ratings in the absence of publicly reported 5‐star ratings. This is consistent with the descriptive data presented in Figure 1 showing that 5‐star facilities had fewer admissions than other facilities.

However, once the star ratings were released, this changed. The coefficient on the interaction between the post‐2008 indicator and each 5‐star rating indicator represents the effect of these ratings on being admitted a nursing home at that star level (compared to being admitted to a 1‐star nursing home) once the scores were publicly released, over and above the correlation between scores and nursing home demand before they were publicly released. We find that there was a higher log‐odds of admission to a nursing home with higher star ratings compared to 1‐star facilities after public reporting was initiated, with the largest effect for 5‐star facilities. Furthermore, when taking the linear combination of the coefficients and testing whether the difference is statistically different than zero, there is a statistically significantly higher log‐odds of admission to a 5‐star facility than to a 4‐, 3‐, or 2‐star facility. The results are similar when looking at those admitted to the nursing home for postacute care (column 2) and for long‐term care (column 3) separately.

When changing the 5‐star release date to the false date of December 2006, we find the effect of the release on nursing home demand disappears. These results can be seen in Table 2, column 1, where most of the interaction terms are small and not statistically significant, suggesting that observed changes in the log‐odds of admission by star rating in the main regression are not simply due to time trends. We then test whether the results are robust to using the first publicly available 5‐star rating throughout the post‐2008 period. As seen in Table 2, column 2, our results remain as expected for most star levels, with more consumers being admitted to higher rated nursing homes, though the magnitude of the effect is smaller than in the main regression results in Table 1. This suggests that our results are driven at least in part by between‐nursing home effects (consumers being admitted to higher rated nursing homes) rather than only within‐nursing home effects (nursing homes improving their star rating).

Table 2.

Robustness Tests of the Log‐Odds of Being Admitted to a Nursing Home with 2, 3, 4, and 5 Stars before and after the Star‐Based Ratings Were Released in 2008, Compared to Being Admitted to a 1‐Star Facility

Using a False Implementation Date (December 2006)		Using the December 2008 Star Rating throughout the Postreport Card Period
2 stars	−0.051*** (0.004)	2 stars in January 2008	−0.051*** (0.003)
3 stars	−0.072*** (0.004)	3 stars in January 2008	−0.085*** (0.003)
4 stars	−0.189*** (0.004)	4 stars in January 2008	−0.185*** (0.003)
5 stars	−0.376*** (0.006)	5 stars in January 2008	−0.406*** (0.004)
post2006*2 stars	0.006 (0.005)	post2008*2 stars	0.010** (0.005)
post2006*3 stars	−0.005 (0.005)	post2008*3 stars	0.009* (0.005)
post2006*4 stars	0.036*** (0.005)	post2008*4 stars	−0.040*** (0.005)
post2006*5 stars	−0.011 (0.007)	post2008*5 stars	0.058*** (0.007)
N	129,846,843	N	179,567,288

Coefficients from choice models such as these are typically interpreted as changes in utility. Robust standard errors in parentheses. All regressions control for the driving distance between the nursing home resident's home zip code and the nursing home, the nursing home's ownership (for profit, not for profit, or government), total number of beds, occupancy rates, whether it is hospital based, percentage of Medicare patient days, and percentage of Medicaid patient days.

*p < .10, **p < .05, ***p < .01.

Finally, we use simulation to estimate changes in market share due to changes in consumer demand. As shown in Table 3, for a market with five nursing homes (one at each star level) where each nursing home has an equal share of the market prior to the release of the 5‐star rating, we find that the report card was associated with a relative reduction in market share of 8 percent for 1‐star facilities and an increase in market share of 6.4 percent in 5‐star facilities but relatively small changes in market share for the 2‐, 3‐, or 4‐star facilities. Of note, the starting market share does not affect the simulation's prediction of the relative changes in market share. Thus, these relative changes can be used to estimate the absolute changes in market share in any market.

Discussion

We examine consumer response to the release of the nursing home 5‐star rating system. Our main findings indicate that the release of this summary rating system was associated with a significant change in consumer demand for low‐ and high‐scoring facilities for both postacute care and long‐term care admissions. Based on a simulated market, after the star‐based rating system was released, 1‐star facilities lost 8 percent of their market share and 5‐star facilities gained over 6 percent of their market share.

The magnitude of this response is substantially larger than previously documented market share responses when NHC was released in 2002 with 12 separate measures of quality (Werner et al. 2012), where a one standard deviation difference in the percentage of residents with moderate to severe pain (the quality measure with the largest effect on consumer demand) resulted in a statistically significant but small 0.2 percentage point change in market share in the average market. For the star‐based rating system we study here, a one standard deviation improvement in star ratings (from 3 stars to 5 stars) resulted in a much larger 1.4 percentage point increase in market share.

Prior research has demonstrated an association between Medicare's star ratings for Medicare Advantage plans and enrollment in those plans (Reid et al. 2013). However, this cross‐sectional analysis could not comment on whether enrollment in highly rated plans increased when the star ratings were released or compare enrollment to the period when health plan ratings were not star‐based. To our knowledge, our results are the first to document changes in consumer demand in response to a change to a summary rating system. Additionally, our analysis is able to isolate the effect of moving to a summary measure of quality, because the nonsummary, detailed quality data were available for 6 years prior to the release of the summary and continued to be available on the same website. CMS is moving toward more widespread adoption of summary measures, replacing several rating systems based on numerous individual quality metrics to ones focusing on star‐based rating systems. Our findings suggest that the transformation of these rating systems can lead to more consumer engagement with them. Increased consumer use of public report cards may in turn provide a stronger incentive for providers to deliver high‐quality (or highly rated) care.

The robust consumer response to star‐based ratings may not come as a surprise. Star‐based ratings are easy to understand and familiar to most consumers. Stars are widely used to rate products and experiences outside of health care, including on Yelp, Amazon, and numerous other online rating systems. Consumers are likely comfortable with these types of ratings and have experience incorporating them into their decision making.

While there are clear potential benefits to using summary measures in report cards, there may also be potential downsides to this approach. The welfare effects of increased consumer response to quality ratings depend on the validity of those ratings. Quality is multidimensional (Mor et al. 2003) and providers routinely specialize in the type of care they deliver (Banaszak‐Holl et al. 1997). Thus, performance on any particular quality indicator is often orthogonal to performance on other measures for a given provider, making it difficult to summarize them into a single measure of quality. When individual measures are uncorrelated with each another, there is an offsetting effect on summary measures—providers that perform very well on one metric but poorly on another will look average on a summary measure, decreasing the signal from the report card about which providers are best in a given area. This can result in loss of variation in quality across providers and increased difficulty distinguishing which providers specialize in care in a particular area. Indeed, in our analyses (not reported here) the star rating for nursing home clinical quality is negatively correlated with staffing and weakly correlated with health inspections. Additionally, many of the individual quality measures that make up the quality measure star rating are weakly or negatively correlated with each other. Based on this study, we are not able to comment on whether consumers supplement the overall quality ratings with the individual quality measures. It is possible that they incorporate both into their decision making. However, if consumers make decisions primarily based on the overall rating, it may make it more difficult for them to choose a nursing home that best suits their needs—which may not be the nursing home with the highest overall star rating.

There are several limitations of our study that are worth noting. First, because of the national roll‐out of the star rating system, we do not have a control group of nursing homes that did not have their star rating released. Without the ability to control for contemporaneous trends in nursing home demand, pre‐postdesign makes it difficult to confirm a causal relationship between the release of the nursing home star ratings and a change in consumer demand for nursing homes. However, descriptive trends and robustness checks indicate that there were no changes in nursing home demand that were correlated with star ratings until the exact date the information was released, providing some credibility to a causal relationship. Second, we are not able to directly observe whether nursing home residents or their agents accessed the website information that we study. Rather, we take a reduced‐form approach and ask whether nursing home demand changed at the same time the information was released and cannot comment on the exact pathway by which demand changed. Third, we cannot fully account for whether consumers were constrained in their demand for nursing homes by either the location of highly rated facilities (which may have been too far away to be a feasible choice) or by nursing homes that were full and could not admit new residents. While we include occupancy rates as a covariate to control for the latter, it may still be the case that access to 5‐star facilities in reality is more limited than it appears in our data. In addition, we do not test whether the changes in consumer demand that we document affected nursing homes' overall occupancy rates. In the setting of decreasing demand for nursing homes in general or shorter average lengths of stay in nursing homes, an increased number of nursing home admissions may not translate into higher occupancy rates. Finally, our analysis cannot address the underlying validity of the quality ratings themselves; our intent was to assess consumer response to the ratings as given. Indeed, recent concerns surrounding the validity of the ratings (Thomas 2014) prompted CMS to change their calculation of the star ratings and some of the underlying data used to calculate the ratings (Thomas 2015). As consumers gain more trust in the star ratings, we expect to see an even more robust demand response to this information.

While consumer response to report cards has been disappointing to date, our analysis provides important new results indicating that consumers use CMS's nursing home star ratings to choose a nursing home. Prior report cards may not have successfully conveyed the complex information required to make informed decisions in a way that is understandable to consumers. The use of summary scores may increase consumer comprehension of and response to report cards.

Supporting information

Appendix SA1: Author Matrix.