Association Between Medicare Summary Star Ratings for Patient Experience and Clinical Outcomes in US Hospitals

Abstract

Objective:

In 2015, the Centers for Medicare and Medicaid Services (CMS) released new summary star ratings for US hospitals based on patient experience. We aimed to test the association between CMS patient experience star ratings and clinical outcomes.

Methods:

We analyzed risk-adjusted data for more than 3000 US hospitals from CMS Hospital Compare using linear regression.

Results:

We found that better patient experience was associated with favorable clinical outcomes. Specifically, a higher number of stars for patient experience had a statistically significant association with lower rates of many in-hospital complications. A higher patient experience star rating also had a statistically significant association with lower rates of unplanned readmissions to the hospital within 30 days.

Conclusion:

Better patient experience according to the CMS star ratings is associated with favorable clinical outcomes. These results support the inclusion of patient experience data in the framework of how hospitals are paid for services.

Introduction

Data on patient experience (also termed “patient satisfaction”) are now utilized by the US Centers for Medicare and Medicaid Services (CMS) in determining payment to health-care organizations for taking care of patients. However, the relationship between patient experience and clinical outcomes is not entirely clear, and this has been the subject of controversy in the medical community. Some recent publications have suggested that patients’ subjective assessment of the care provided may not be credible because patients lack medical training, and attempts to optimize patients’ satisfaction may lead to lower quality of care (1 –3). A better understanding of the association between patient experience and clinical outcomes is needed.

New data on patient experience have recently emerged. In 2015, CMS released summary star ratings for US hospitals (ie, 1 star [worst] to 5 stars [best]) based on patient experience surveys. To our knowledge, no prior publication has tested whether the new CMS star ratings are associated with clinical outcomes. In this study, we tested the association between the star ratings and clinical outcomes for more than 3000 US hospitals.

Methods

The data used in this study are in the public domain and do not include patient-level identifiers. This study met criteria for exemption from institutional review board review at Cooper University Health Care.

We analyzed hospital-level data from CMS Hospital Compare (available at: https://data.medicare.gov/data/hospital-compare) (4). Patient experience star ratings reflect hospital performance on the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey and are a summary score for performance across all HCAHPS patient experience domains. The CMS methodology for calculating the star ratings is available at http://www.hcahpsonline.org/files/HCAHPS_Stars_Tech_Notes_9_17_14.pdf (5). The HCAHPS domains include, for example, patient evaluations of caregiver listening and communication, treatment with courtesy and respect, timeliness of assistance, and the quality of discharge instructions. We analyzed clinical outcomes using Medicare claims data from Hospital Compare. Specifically, we analyzed data for hospital complications and unplanned readmissions within 30 days of discharge. To facilitate comparisons across hospitals, the measures of hospital complications and 30-day readmissions in Hospital Compare are risk adjusted to account for differences in both hospital characteristics (eg, hospital size) and patient characteristics (eg, age, comorbid conditions) that could be associated with hospital complications and readmissions. Thus, the outcome measures we analyzed in this study were risk-adjusted rates reported as hospital complication and 30-day readmission “scores.” In this study, we did not analyze data for hospital mortality on the grounds that patient experience surveys are administered following hospital discharge, and thus patients who die in the hospital could not contribute patient experience data for the analysis.

We used linear regression to test the association between number of patient experience stars and outcomes. The model tested the effect that a 1-star rise in patient experience rating (eg, increasing from 3 to 4 stars) would have on each outcome. Since the star ratings for hospitals range from 1 (worst) to 5 (best), and the outcomes are the reported rates of adverse events, an inverse association (ie, negative slope) identifies an association between better experience and favorable outcomes. We also compared the rates of complications and readmissions between 1-star and 5-star hospitals using 2-sample t test. For all analyses, we used a P value of <.05 to represent statistical significance.

Results

Our analysis included data for more than 3000 hospitals. Using linear regression, we found statistically significant associations between the number of stars for patient experience and multiple clinical outcomes, including hospital complications and the rates of readmission for all of the diagnoses reported (see Table 1). Specifically, higher patient experience had a statistically significant association with lower rates of the following hospital complications: central line-associated bloodstream infection, postsurgical deep venous thrombosis, joint replacement complications, and a composite measure for all serious complications. Using linear regression, higher patient experience also had a statistically significant association with lower rates of readmission for patients with acute myocardial infarction, congestive heart failure, pneumonia, stroke, chronic obstructive pulmonary disease, joint replacement, and a composite measure for readmission among all patients. These associations are further illustrated in Figures 1 and 2, which display the results of the 2-sample t tests for hospital complications and 30-day readmissions, respectively, comparing 1-star versus 5-star hospitals. In summary, these risk-adjusted analyses found statistically significant associations between patient experience star ratings and multiple clinical outcomes, with the most consistent associations found between a higher number of stars for patient experience and lower rates of readmission to the hospital.

Table 1.

Results of a Linear Regression Testing the Association Between Centers for Medicare and Medicaid Services (CMS) Hospital Star Ratings For Patient Experience and Clinical Outcomes in the United States hospitals, Including Hospital Complications (A) and Unplanned Readmissions to the Hospital Within 30 Days of Discharge (B).^a

A	Hospital Complications	CLABSI	Postsurgical Deep Venous Thrombosis	Joint Replacement Complication	Surgical Wound Dehiscence	Iatrogenic Pneumothorax	Accidental Puncture or Laceration	All Serious Complications (Composite)
	Number of hospitals	2970	2986	2703	2627	3056	3050	3059
	Slope	−0.062	−0.206	−0.053	−0.002	−0.002	0.013	−0.018
	P value	<.0001	<.0001	<.0001	.7653	.1003	.3020	<.0001
B	Readmissions Within 30 Days	Acute Myocardial Infarction	Congestive Heart Failure	Pneumonia	Stroke	COPD	Joint Replacement	All Readmissions
	Number of hospitals	2235	3314	3347	2730	3264	2710	3536
	Slope	−0.210	−0.389	−0.257	−0.313	−0.173	−0.084	−0.264
	P value	<.0001	<.0001	<.0001	<.0001	<.0001	<.0001	<.0001

Abbreviations: CLABSI, central line-associated bloodstream infection; COPD, chronic obstructive pulmonary disease.

^a Since the star ratings for hospitals range from 1 (worst) to 5 (best), and the outcomes are the reported rates of adverse events, an inverse association (i.e. negative slope) identifies an association between better experience and favorable outcomes. Data for hospital complications (except CLABSI) and readmissions are risk-adjusted rates from Hospital Compare; CLABSI is based on the standardized infection ratio (SIR) from the Centers for Disease Control and Prevention.

Figure 1.

Results of 2-sample t tests comparing the rates of complications between hospitals that received 1-star (1*) and 5-star (5*) ratings for patient experience. The outcome measures are risk-adjusted complication rates (adjusted for hospital and patient characteristics) and reported as hospital complication “scores”. The “n” refers to the number of hospitals contributing data to the analysis. We found statistically significant associations between better experience and lower complications for central line-associated bloodstream infection (CLABSI), postoperative deep venous thrombosis (DVT), joint replacement surgery, and a composite measure for all serious complications (all serious). PTX indicates pneumothorax (iatrogenic).

Figure 2.

Results of 2-sample t tests comparing the rates of 30-day unplanned hospital readmissions between hospitals that received 1-star (1*) and 5-star (5*) ratings for patient experience. The outcome measures are risk-adjusted readmission rates (adjusted for hospital and patient characteristics) and reported as hospital readmission “scores.” The “n” refers to the number of hospitals contributing data to the analysis. AMI indicates acute myocardial infarction; CHF, congestive heart failure; COPD, chronic obstructive pulmonary disease.

Discussion

In this study, we found a significant association between better patient experience (as quantified by the CMS star ratings) and favorable clinical outcomes in US hospitals. Our results differ from the results of some previously published studies that were unable to identify such an association (6). A major limitation of prior studies was that they only tested a single distinct aspect of patient experience because, at the time, there was no accepted methodology to combine data for different domains of patient experience into an overall summary value (6). In contrast, our study used the newly developed summary score (star rating) that reflects all the HCAHPS patient experience domains combined and thus is more likely to be representative of the total patient experience in the hospital.

Multiple potential theories exist by which patient experience could affect clinical outcomes and vice versa. Health-care providers who are diligent about providing excellent patient experience may be similarly diligent about excellence in all elements of patient care. Further, it is possible for patients (even those without medical training) to detect high-quality care, and thus excellent patient experience scores may be reflecting patients’ subjective sense of the high quality of care that was provided. It is also possible that patients with the best short-term outcomes have the best subjective assessment of the care they received, and thus patient experience scores can, in effect, be a patient-reported outcome measure (PROM). The quality and effectiveness of patient care can also be highly dependent on the effectiveness of provider–patient communication, which is a major component of patient experience surveys. Similarly, excellent patient experience can inspire patient confidence in (and adherence to) a treatment plan, both in the hospital and following discharge (6 –8).

We acknowledge important limitations in this study. First, association does not necessarily represent causation. We also recognize that our study leveraged a large administrative database and thus may have found statistical significance where the clinical significance at the individual patient level is relatively small (ie, small “effect size”). However, small effect sizes can impact large numbers of patients when applied across very large populations. Thus, our findings are very important from the payer perspective (eg, Medicare). We also recognize that our results are somewhat heterogeneous in that no statistically significant association was found for 3 of the 7 hospital complications tested (surgical wound dehiscence, iatrogenic pneumothorax, and accidental laceration or puncture during surgery). However, our results were quite consistent in the associations between higher star ratings for patient experience and lower 30-day readmissions across all of the different diagnoses tested in this study. As the quality of patient experience is deeply rooted in the quality of caregiver communication, it is not unexpected to see the strongest signal of association between patient experience and outcome in the rate of unplanned hospital readmissions, as readmissions can depend on the effectiveness of communication pertaining to discharge instructions.

We also acknowledge that there are a multitude of patient-related factors in the clinical context that can affect the outcome measures in this study (eg, hospital readmission), which are unrelated to the care provided. It is also possible that patient experience can affect outcomes that are meaningful from a patient perspective but are not reflected in the outcomes assessed in our study. Thus, we submit that with this degree of potential “noise” in the clinical environment, the fact that any significant signal can be detected is an indication of meaningful association.

Finally, one could argue that testing the association between patient experience and clinical outcomes is not necessary on the grounds that providers and hospitals have an intrinsic duty to treat patients the way they want to be treated, independent of any association tested here. We strongly agree that a fundamental duty exists; however, we believe that the results of this study are important in that they underscore the value of measuring patient experience as a distinctly meaningful dimension of health-care quality. Accordingly, we also believe that our results support payer inclusion of patient experience data in the framework of payments to health-care organizations.

Author Biographies

Stephen Trzeciak is a Professor of Medicine and Head of Critical Care Medicine at Cooper University Health Care and Cooper Medical School of Rowan University.

John P Gaughan is a biostatistician at Cooper University Health Care.

Joshua Bosire is an Assistant Vice President at Cooper University Health Care.

Anthony J Mazzarelli is the Chief Physician Executive for Cooper University Health Care and an Assistant Professor of Emergency Medicine at Cooper Medical School of Rowan University.