Abstract
This study assesses the agreement of the US Centers for Medicare & Medicaid Services Hospital Compare, Healthgrades, and US News & World Report Best Hospitals on hospital performance for chronic obstructive pulmonary disease and heart failure.
A number of organizations publicly report condition-specific hospital mortality ratings. For those conditions where these ratings overlap, the perception is that the ratings often conflict with each other, making it difficult for health care stakeholders to understand which rating system to trust.1 We sought to understand the level of agreement and disagreement of condition-specific mortality ratings for individual hospitals across 3 national publicly reported rating systems.
Methods
We extracted publicly reported hospital 30-day mortality ratings for 2 common conditions—chronic obstructive pulmonary disease (COPD) and heart failure—from 3 national rating systems, including the US Centers for Medicare & Medicaid Services Hospital Compare, Healthgrades, and US News & World Report Best Hospitals. These rating systems were chosen because they rate hospitals on a national scale and issue ratings for a common set of conditions. The data used for our analysis reflect what was publicly reported in June 2017 and only includes hospitals that received a rating for a particular condition from all 3 rating systems. For purposes of comparing hospital performance across the rating systems, we categorized each rating system’s publicly reported performance categories into 3 groups (good, fair, and poor) based on how consumers would likely interpret the rating. The level of agreement or disagreement in a hospital’s ratings was calculated based on these categorizations. The Johns Hopkins Medicine Institutional Review Board determined that this study did not constitute human subjects research. All analyses were conducted using Excel 2016 (Microsoft).
Results
Our sample size for comparing the mortality ratings was 3230 hospitals for COPD and 3310 hospitals for heart failure. As outlined in Table 1, most hospitals were rated by Hospital Compare as no different than the US national rate (3088 of 3230 [95.6%] for COPD; 3088 of 3310 [93.3%] for heart failure), with the remaining hospitals rated as being statistically better or worse than the US national rate. Healthgrades rated 2373 hospitals (73.5%) as 3 stars (average) for COPD and 2035 hospitals (61.5%) as 3 stars for heart failure, with the remaining hospitals rated as 1 star (below average) or 5 stars (above average). Best Hospitals assigned a more even distribution of ratings, with the percentage of hospitals in each performance category ranging between 18.4% and 22.9%.
Table 1. Distribution of Hospital Performance on 3 National 30-Day Mortality Ratings for Chronic Obstructive Pulmonary Disease (COPD) and Heart Failure.
| Publicly reported performance category | Assigned category for assessing rating agreement | Hospitals, No. (%) | |
|---|---|---|---|
| COPD (n = 3230) | Heart failure (n = 3310) | ||
| US Centers for Medicare & Medicaid Services Hospital Compare | |||
| Worse than national rate | Poor | 93 (2.9) | 77 (2.3) |
| No different than national rate | Fair | 3088 (95.6) | 3088 (93.3) |
| Better than national rate | Good | 49 (1.5) | 145 (4.4) |
| Healthgrades | |||
| 1 star | Poor | 558 (17.3) | 861 (26.0) |
| 3 stars | Fair | 2373 (73.5) | 2035 (61.5) |
| 5 stars | Good | 299 (9.3) | 414 (12.5) |
| US News & World Report Best Hospitals | |||
| Worst | Poor | 663 (20.5) | 681 (20.6) |
| Worse than average | Poor | 605 (18.7) | 625 (18.9) |
| Average | Fair | 597 (18.5) | 608 (18.4) |
| Better than average | Good | 625 (19.3) | 659 (19.9) |
| Best | Good | 740 (22.9) | 737 (22.3) |
In comparing individual hospital performance on the mortality ratings, we found that only 534 of 3230 hospitals (16.5%) had full agreement in their ratings across the 3 systems for COPD and only 542 of 3310 hospitals (16.4%) had full agreement for heart failure (Table 2). For most hospitals (2609 [80.8%] for COPD; 2607 [78.8%] for heart failure), we found 2 of the 3 ratings systems agreed on the rating, with the third system rating the hospital as 1 category better or worse. For both COPD and heart failure, the 3 rating systems gave discordant results for a nontrivial number of hospitals (87 hospitals [2.7%] and 161 hospitals [4.9%], respectively), where all 3 systems assigned different ratings to the same hospital or 2 of the ratings agreed at one extreme and the third system assigned the hospital a rating at the opposite extreme.
Table 2. Level of Agreement in Hospital Performance Across 3 National 30-Day Mortality Ratings for Chronic Obstructive Pulmonary Disease (COPD) and Heart Failure.
| Level of agreement based on assigned categories | Hospitals, No. (%) | |
|---|---|---|
| COPD (n = 3230) | Heart failure (n = 3310) | |
| All 3 ratings agree | 534 (16.5) | 542 (16.4) |
| 2 Ratings agree and 1 rating differs by 1 category | 2609 (80.8) | 2607 (78.8) |
| 2 Ratings agree at 1 extreme and 1 rating is opposite extreme or all 3 ratings disagree | 87 (2.7) | 161 (4.9) |
Discussion
We found for most hospitals the 3 rating systems did not fully agree on the hospital’s mortality performance for a particular condition, but there was generally moderate agreement in the ratings. As similar studies have found, ratings are often sensitive to how the rating is constructed and may not reflect true differences in performance.2,3,4 While our data set included more than 3000 hospitals for each condition, only certain types of hospitals were included in our analysis. Free-standing pediatric hospitals, federal hospitals, and prospective payment system–exempt cancer hospitals were not included in the analysis. If there are differences in how the rating systems assess the mortality performance of the excluded hospitals, this could result in a level of agreement different from what we found in our particular data set. In addition, our analysis was based on a single snapshot of data from June 2017. If we were to repeat the analysis with a different data snapshot, we could possibly find different results. Our study did not assess the methods used by each rating system, so we cannot draw conclusions about which rating system did the best job of correctly identifying high-performing and low-performing hospitals; we simply compared the agreement and disagreement of the public ratings. As mortality reflects a singular objective construct, one step might be for the mortality rating systems to come together to identify best practices and work to produce a single rating so that stakeholders can center on one truth.
References
- 1.Soelch LA, Repp AB. Hospital responses to mortality measures: a survey of hospital administrative leaders. Qual Manag Health Care. 2019;28(2):78-83. doi: 10.1097/QMH.0000000000000209 [DOI] [PubMed] [Google Scholar]
- 2.Austin JM, Jha AK, Romano PS, et al. National hospital ratings systems share few common scores and may generate confusion instead of clarity. Health Aff (Millwood). 2015;34(3):423-430. doi: 10.1377/hlthaff.2014.0201 [DOI] [PubMed] [Google Scholar]
- 3.Shahian DM, Wolf RE, Iezzoni LI, Kirle L, Normand SL. Variability in the measurement of hospital-wide mortality rates. N Engl J Med. 2010;363(26):2530-2539. doi: 10.1056/NEJMsa1006396 [DOI] [PubMed] [Google Scholar]
- 4.Silva GC, Jiang L, Gutman R, et al. Mortality trends for veterans hospitalized with heart failure and pneumonia using claims-based vs clinical risk-adjustment variables. JAMA Intern Med. 2020;180(3):347-355. doi: 10.1001/jamainternmed.2019.5970 [DOI] [PMC free article] [PubMed] [Google Scholar]
