Abstract
Objective
Virtually anything can be ranked; the US News and World Report (USNWR or US News) ranks the top 50 hospitals specializing in cardiology, heart, and vascular surgery. Here the authors propose validating the effectiveness of rankings by comparing differences among the USNWR metrics across the top 50 hospitals.
Methods
The ranking system for the top 50 hospitals specializing in cardiology, heart, and vascular surgery was derived from 16 variant scores. Each hospital’s scores were collected from the USNWR. Hospitals were categorized into quintiles consisting of 10 institutions (1–10, 11–20, etc). An analysis of variance/χ2 comprehensive statistical analysis was run alongside a Wilcoxon/Kruskal–Wallis test to compare statistical outcomes. A significant threshold was deemed to be P < 0.05.
Results
Significant differences were noted between quintiles for advanced technologies (P = 0.05), US News specialty score (P < 0.001), number of patient referrals (P = 0.004), and expert opinion (P < 0.001). Non-statistically significant differences were found among patient experience, public transparency, Society of Thoracic Surgery transparency, American College of Cardiology transparency, recognition as a magnet hospital, and nursing staffing. Interestingly, a large variance was noted in the average number of referrals between the first quintile (13,371) and the last (6690).
Conclusions
Expert opinion plays a critical role in the reputation of the USNWR’s top 10 hospitals in cardiology, heart, and vascular surgery. Although many have argued about the merits of USNWR hospital rankings, taken together, rankings fill a strong customer demand and are sticky.
Keywords: US News and World Report, Cardiology, Center for Medicare and Medicaid Services, Heart and Vascular Surgery, Ranking
Introduction
The basic idea of evaluating hospital performance has long garnered immense interest. For a variety of pecuniary and nonpecuniary reasons, the general objective was to compare hospitals according to some aggregated dimensions. After all, health care is a rare purchase and accordingly, patients often seek prestigious hospitals. Consequently, rankings have become an important vein for annual patient visits, reimbursement rates, and funding allocation. Invariably, and not surprisingly, heart surgery and cardiology rankings are an attractive concept, from patients to clinicians to hospital administration. There are decades-old challenges over the utility of the various metrics used in rankings. So it is worth exploring the clinical indicators in these hospital ranking systems. Quality measures should ideally have a clear and causal explanation. Generally, one of the most challenging aspects of quality measurement is obtaining quantitative data. The purpose of the present article was to examine critically the most popularized existing hospital ranking system, that of the US News and World Report (USNWR or US News), to assess its validity and to derive insights about the top cardiology and heart surgery programs.
Methods
Utilizing the USNWR 2022–2023 Cardiology, Heart, and Vascular Surgery Best Hospital rankings, 16 variables were analyzed to determine the top 50 hospitals. In this analysis, the authors divided the overall group (n = 50) into quintiles; each cohort comprising 10 hospitals. The 16 critical categories included in this evaluation encompassed diverse aspects, such as US News specialty score, patient experience, public transparency, 3-day survival, discharging patients, intensives, Society of Thoracic Surgery (STS) transparency, American College of Cardiology (ACC) transparency, advanced technologies, patient services, trauma center, recognized as magnet hospital, current American Hospital Association (AHA) responder, number of patients, nurse staffing, and expert opinion. Guided by the USNWR, the methodology underlying these rankings integrated 3 fundamental components: structure, process, and outcomes. This approach was reflected in the assignment of critical percentages to the overall ranking distributing importance as follows: outcomes (37.5%), structure (30%), process/expert opinion (24.5%), patient experience (5%), and public transparency (3%). An analysis of variance (ANOVA) was used to compare continuous variant scores, and a χ2 test was used to compare the categorical variant scores among the 5 quintile groups to one another. A significant threshold was deemed to be P < 0.05 and was applied to each of the 16 categories and quintiles to ensure a reliable evaluation of this report.
In the second phase of the analysis, the 50 hospitals, subdivided into quintiles, underwent a comprehensive division process using data sourced from the Centers for Medicare and Medicaid Services (CMS) specifically focusing on heart attack, heart failure, and stroke death rates. These metrics were quantified as percentages and categorized into the same 5 groups previously established in the USNWR rankings. Mean, median, and range values were calculated for each variable. Additionally, an overall average death rate encompassing all 3 variables was derived for each of the individual hospitals. Due to data availability constraints, 2 hospitals were excluded from the heart attack death rate category, whereas only one hospital was excluded from the heart failure and stroke death rate categories. Following the compilation of average death rates for all hospitals, the groupings were restructured based on percentiles from lowest to highest facilitating a direct comparison with the original USNWR rankings.
A total of 50 hospitals stratified into 16 distinct categories were compared via an ANOVA/χ2 comprehensive statistical analysis along with a Wilcoxon/Kruskal–Wallis test. Statistical significance was established at a P value of < 0.05, revealing notable distinctions in 4 of the 16 categories examined.
This project was exempted from IRB approval by Temple University School of Medicine.
Results
Significant variations were seen in advanced technologies (P = 0.05), US News specialty score (P < 0.001), number of patient referrals (P = 0.004), and expert opinion (P < 0.001). Conversely, 7 categories exhibited no statistically significant differences among the quintiles encompassing patient experience (P = 0.65), public transparency (P = 0.54), STS transparency (P = 0.54), ACC transparency (P = 0.40), trauma center (P = 0.20), recognized as magnet hospital (P = 0.43), and nurse staffing (P = 0.53) (Table 1). Furthermore, 4 categories displayed uniform responses across all 50 hospitals and individual quintiles in the study. These categories included current AHA responder, intensives, discharging patients, and 30-day survival. Notably, the most significant variance observed pertained to the total number of referred patients between hospitals. In the first quintile, 13,371 referred patients were accounted for, whereas in the last quintile, there was only a total of 6690. This shares a substantial divergence in overall patient referrals of the hospital spectrum just within these 5 groups.
Table 1:
US News and World Report variable analysis: Examination of 16 distinct categories for quintile comparison
| Variable | Total (N = 87) | Group 1 (N = 10) | Group 2 (N = 10) | Group 3 (N = 10) | Group 4(N = 10) | Group 5(N = 10) | ANOVA/ χ2 test | Wilcoxon/ Kruskal–Wallis test |
|---|---|---|---|---|---|---|---|---|
| US News specialty score, n | 50 | 10 | 10 | 10 | 10 | 10 | < 0.0001a | < 0.0001a |
| Mean (SD) | 66.0 (9.2) | 81.2 (8.6) | 68.0 (1.3) | 62.9 (1.4) | 60.1 (0.7) | 57.7 (0.5) | ||
| Median (IQR) | 62.6 (59.8, 69.2) | 80.5 (73.6, 84.8) | 67.8 (66.8, 69.2) | 62.6 (61.9, 63.3) | 60.1 (59.8, 60.9) | 57.7 (57.4, 58.0) | ||
| Median (range) | 62.6 (57.1, 100.0) | 80.5 (72.6, 100.0) | 67.8 (66.5, 70.2) | 62.6 (61.6, 66.2) | 60.1 (58.9, 61.1) | 57.7 (57.1, 58.8) | ||
| Patient experience, n (%) | 0.65 | 0.65 | ||||||
| 2 | 3 (6.0) | 0 (0.0) | 0 (0.0) | 1 (10.0) | 1 (10.0) | 1 (10.0) | ||
| 3 | 7 (14.0) | 1 (10.0) | 1 (10.0) | 1 (10.0) | 1 (10.0) | 3 (30.0) | ||
| 4 | 30 (60.0) | 6 (60.0) | 7 (70.0) | 6 (60.0) | 8 (80.0) | 3 (30.0) | ||
| 5 | 10 (20.0) | 3 (30.0) | 2 (20.0) | 2 (20.0) | 0 (0.0) | 3 (30.0) | ||
| Public transparency, n (%) | 0.54 | 0.54 | ||||||
| 2 | 2 (4.0) | 0 (0.0) | 1 (10.0) | 0 (0.0) | 1 (10.0) | 0 (0.0) | ||
| 3 | 48 (96.0) | 10 (100.0) | 9 (90.0) | 10 (100.0) | 9 (90.0) | 10 (100.0) | ||
| 30-d survival, n (%) | ||||||||
| 5 | 50 (100.0) | 10 (100.0) | 10 (100.0) | 10 (100.0) | 10 (100.0) | 10 (100.0) | ||
| Discharging patients, n (%) | ||||||||
| 5 | 50 (100.0) | 10 (100.0) | 10 (100.0) | 10 (100.0) | 10 (100.0) | 10 (100.0) | ||
| Intensives, n (%) | ||||||||
| Yes | 50 (100.0) | 10 (100.0) | 10 (100.0) | 10 (100.0) | 10 (100.0) | 10 (100.0) | ||
| STS transparency, n (%) | 0.54 | 0.54 | ||||||
| No | 2 (4.0) | 0 (0.0) | 1 (10.0) | 0 (0.0) | 1 (10.0) | 0 (0.0) | ||
| Yes | 48 (96.0) | 10 (100.0) | 9 (90.0) | 10 (100.0) | 9 (90.0) | 10 (100.0) | ||
| ACC transparency, n (%) | 0.40 | 0.40 | ||||||
| Yes | 49 (98.0) | 10 (100.0) | 9 (90.0) | 10 (100.0) | 10 (100.0) | 10 (100.0) | ||
| Yes | 1 (2.0) | 0 (0.0) | 1 (10.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | ||
| Advanced technologies, n (%) | 0.05a | 0.05a | ||||||
| 5 | 12 (24.0) | 0 (0.0) | 1 (10.0) | 4 (40.0) | 2 (20.0) | 5 (50.0) | ||
| 6 | 38 (76.0) | 10 (100.0) | 9 (90.0) | 6 (60.0) | 8 (80.0) | 5 (50.0) | ||
| Patient services, n (%) | ||||||||
| 8 | 50 (100.0) | 10 (100.0) | 10 (100.0) | 10 (100.0) | 10 (100.0) | 10 (100.0) | ||
| Trauma center, n (%) | 0.20 | 0.20 | ||||||
| No | 9 (18.0) | 1 (10.0) | 4 (40.0) | 2 (20.0) | 0 (0.0) | 2 (20.0) | ||
| Yes | 41 (82.0) | 9 (90.0) | 6 (60.0) | 8 (80.0) | 10 (100.0) | 8 (80.0) | ||
| Recognized as magnet hospital, n (%) | 0.43 | 0.43 | ||||||
| 0 | 4 (8.0) | 0 (0.0) | 0 (0.0) | 1 (10.0) | 1 (10.0) | 2 (20.0) | ||
| 1 | 46 (92.0) | 10 (100.0) | 10 (100.0) | 9 (90.0) | 9 (90.0) | 8 (80.0) | ||
| Current AHA responder, n (%) | ||||||||
| Yes | 50 (100.0) | 10 (100.0) | 10 (100.0) | 10 (100.0) | 10 (100.0) | 10 (100.0) | ||
| Number of patients, N | 50 | 10 | 10 | 10 | 10 | 10 | 0.004a | 0.05a |
| Mean (SD) | 8807.6 (4434.2) | 13,370.5 (5587.0) | 7798.9 (3491.1) | 8265.0 (3736.0) | 7913.4 (3456.6) | 6690.1 (2698.3) | ||
| Median (IQR) | 8234.0 (5422.0, 12,190.0) | 13,061.0 (8293.0, 17,714.0) | 7690.5 (4830.0, 10,150.0) | 8357.0 (4162.0, 12,286.0) | 8054.5 (5918.0, 9269.0) | 6244.0 (4637.0, 7253.0) | ||
| Median (range) | 8234.0 (2468.0, 22,840.0) | 13,061.0 (5340.0, 22,840.0) | 7690.5 (2468.0, 12,797.0) | 8357.0 (3653.0, 13,575.0) | 8054.5 (3143.0, 15,349.0) | 6244.0 (3168.0, 12,325.0) | ||
| Nurse staffing, n | 50 | 10 | 10 | 10 | 10 | 10 | 0.53 | 0.48 |
| Mean (SD) | 2.4 (0.4) | 2.6 (0.3) | 2.5 (0.4) | 2.3 (0.5) | 2.3 (0.3) | 2.3 (0.5) | ||
| Median (IQR) | 2.4 (2.1, 2.7) | 2.5 (2.4, 2.7) | 2.4 (2.2, 2.5) | 2.1 (2.0, 2.8) | 2.4 (2.2, 2.6) | 2.2 (2.0, 2.7) | ||
| Median (range) | 2.4 (1.4, 3.2) | 2.5 (2.0, 3.1) | 2.4 (2.0, 3.2) | 2.1 (1.7, 3.2) | 2.4 (1.6, 2.7) | 2.2 (1.4, 2.8) | ||
| Expert opinion, n | 50 | 10 | 10 | 10 | 10 | 10 | < 0.0001a | < 0.0001a |
| Mean (SD) | 3.9 (6.0) | 11.8 (9.7) | 3.3 (2.1) | 1.5 (1.0) | 2.1 (1.9) | 0.9 (0.7) | ||
| Median (IQR) | 1.8 (1.0, 3.9) | 8.3 (5.7, 11.6) | 2.5 (1.7, 4.5) | 1.2 (1.0, 1.8) | 1.7 (0.8, 2.2) | 0.7 (0.5, 1.4) | ||
| Median (range) | 1.8 (0.0, 30.5) | 8.3 (3.6, 30.5) | 2.5 (1.2, 7.5) | 1.2 (0.0, 3.4) | 1.7 (0.6, 6.8) | 0.7 (0.2, 2.5) |
Statistically significant.
ACC, American College of Cardiology; AHA, American Hospital Association; ANOVA, analysis of variance; IQR, interquartile range; SD, standard deviation; STS, Society of Thoracic Surgery; USNWR/US News, US News and World Report.
This critical analysis of significant findings that distinguish the top 50 hospitals in this study revolved around key factors focusing specifically on expert opinion and patient volume. Expert opinion, a pivotal criterion, targets an institution’s ability to deliver exceptional care to challenging cases. This aspect constitutes the process component that holds substantial weight in the overall institutional ranking at 24.5%. The expert opinion as explained by the USNWR is deemed by nomination from board-certified specialists. The rankings for the year 2022–2023 were a collective analysis from the preceding 3 years of physician surveys and equal weight is assigned to each year in determining the final score for expert opinion. Regarding the structural element, the consideration of patient volume carries a weight of 6.67% within the overall ranking and constitutes 30% of the entire hospital evaluation. Structure specifically assesses hospital resources that directly impact overall patient care and the hospital environment. Lastly, patient volume is derived from medical and surgical discharges in the cardiology and heart surgery grouping based on specific Medicare Severity Diagnosis Related Group submissions for CMS reimbursement.
The secondary analysis of CMS data shown in Table 2 employed both ANOVA and Kruskal–Wallis testing, with a significance level set at P < 0.05. Heart attack, heart failure, stroke, and average death rates were subjected to statistical examination, categorized both by quintiles and overall grouping (total n = 50, with each quintile n = 10). For heart attack death rate, the ANOVA yielded a highly significant value of < 0.001, indicating a notable difference among quintiles. However, heart failure (P = 0.086) and stroke death rates (P = 0.22) did not exhibit significant differences among quintiles based on ANOVA analysis. Conversely, when considering the average death rate across all 3 categories, a significant P value of 0.0003 was obtained, indicating discernible distinctions among quintiles of the top 50 hospitals, particularly concerning death rates from heart attack, heart failure, and stroke. The highest death rate was found in the 4th group, namely, hospitals ranked 31–40 with USNWR, compared to the other groups (Table 2).
Table 2:
Comprehensive Centers for Medicare and Medicaid Services data analysis: Top 50 hospitals assessments for 3 mortality variables (Heart Attack, Heart Failure, Stroke) and Composite Mean
| Data | Group | P values | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Variable | NMiss | Overall (N = 50) | 1 (N = 10) | 2 (N = 10) | 3 (N = 10) | 4 (N = 10) | 5 (N = 10) | ANOVA | Kruskal–Wallis test |
| Heart attack DR, n | 2 | 48 | 10 | 9 | 10 | 10 | 9 | < 0.0001a | 0.0003a |
| Mean (SD) | 11.2 (1.2) | 9.7 (0.8) | 11.6 (0.4) | 11.0 (1.1) | 12.1 (1.2) | 11.8 (0.8) | |||
| Median (IQR) | 11.4 (10.4, 12.1) | 9.7 (9.2, 10.1) | 11.7 (11.4, 12.0) | 11.0 (10.4, 11.6) | 12.6 (11.0, 12.9) | 12.0 (11.0, 12.4) | |||
| Median (range) | 11.4 (8.9, 13.3) | 9.7 (8.9, 11.5) | 11.7 (10.9, 12.1) | 11.0 (9.2, 12.9) | 12.6 (9.9, 13.3) | 12.0 (10.4, 12.7) | |||
| Heart failure DR, n | 1 | 49 | 10 | 10 | 10 | 10 | 9 | 0.086a | 0.049a |
| Mean (SD) | 8.7 (1.5) | 7.5 (1.1) | 8.9 (0.9) | 8.9 (1.5) | 9.2 (1.4) | 9.1 (2.1) | |||
| Median (IQR) | 8.7 (8.0, 9.4) | 7.5 (6.8, 8.4) | 9.0 (8.4, 9.5) | 8.9 (8.3, 9.7) | 8.9 (8.1, 10.1) | 8.3 (8.0, 9.3) | |||
| Median (range) | 8.7 (5.5, 14.5) | 7.5 (5.5, 9.2) | 9.0 (6.910.1) | 8.9 (5.5, 11.0) | 8.9 (7.4, 11.5) | 8.3 (7.7, 14.5) | |||
| Stroke DR, n | 1 | 49 | 10 | 10 | 10 | 10 | 9 | 0.22 | 0.33 |
| Mean (SD) | 12.2 (1.9) | 11.2 (1.9) | 12.6 (1.6) | 11.7 (2.2) | 13.0 (1.7) | 12.2 (1.8) | |||
| Median (IQR) | 12.3 (10.8, 13.3) | 11.1 (10.2, 12.8) | 12.4 (11.8, 12.7) | 11.6 (10.1, 12.5) | 13.3 (11.7, 14.2) | 11.5 (11.2, 12.6) | |||
| Median (range) | 12.3 (8.0, 16.2) | 11.1 (8.0, 13.4) | 12.4 (10.2, 15.8) | 11.6 (8.8, 16.2) | 13.3 (10.5, 15.8) | 11.5 (10.0, 15.4) | |||
| Average DR, n | 1 | 49 | 10 | 10 | 10 | 10 | 9 | 0.0003a | 0.002a |
| Mean (SD) | 10.7 (1.1) | 9.5 (1.0) | 11.1 (0.8) | 10.5 (1.0) | 11.4 (0.9) | 11.0 (0.9) | |||
| Median (IQR) | 10.7 (10.1, 11.3) | 9.6 (8.8, 10.2) | 10.9 (10.6, 11.3) | 10.6 (9.7, 11.4) | 11.2 (10.7, 12.0) | 10.7 (10.4, 11.4) | |||
| Median (range) | 10.7 (7.8, 12.9) | 9.6 (7.8, 10.8) | 10.9 (9.6, 12.4) | 10.6 (9.2, 12.3) | 11.2 (10.2, 12.7) | 10.7 (10.1, 12.9) | |||
Statistically significant.
ANOVA, analysis of variance; CMS, Centers for Medicare and Medicaid Services; DR, death rate; IQR, interquartile range; SD, standard deviation.
Subsequently, by utilizing average percentages, the top 50 hospitals were reorganized by the variable that encompassed all 3 death rates (heart failure, heart attack, and stroke) to ascertain any disparities compared with the top quintile (1–10) of hospitals as per USNWR rankings. This is shown in Table 3. The analysis revealed a restructured ranking order of (5, 3, 8, 2, 4, 25, 29, 13, 24, and 29), suggesting that post-adjustment, only 5 out of the top 10 hospitals remained within the first quintile.
Table 3:
US News and World Report rankings compared to Centers for Medicare and Medicaid Services metrics rankings.
| Institution name | Institution location | USNWR ranking | CMS Metrics ranking |
|---|---|---|---|
| Cleveland Clinic | OH | 1 | 15 |
| Mayo Clinic, Rochester | MN | 2 | 4 |
| Cedars-Sinai Medical Center | LA | 3 | 2 |
| NewYork-Presbyterian Hospital, Columbia University and Weill Cornell | NY | 4 | 5 |
| New York University Langone Hospitals | NY | 5 | 1 |
| Mount Sinai Hospital | NY | 6 | 18 |
| Massachusetts General Hospital | MA | 7 | 11 |
| Northwestern Memorial Hospital | IL | 8 | 3 |
| Stanford Health Care, Stanford Hospital | CA | 9 | 13 |
| Brigham and Women's Hospital | MA | 10 | 27 |
| University of California, Los Angeles Medical Center | CA | 11 | 35 |
| Vanderbilt University Med Center | TN | 12 | 46 |
| Houston Methodist Hospital | TX | 13 | 8 |
| The University of Texas Southwestern Medical Center | TX | 14 | 28 |
| Lenox Hill Hospital at Northwell Health | NY | 15 | 31 |
| North Shore University Hospital at Northwell Health | NY | 16 | 26 |
| Hospitals of the University of Pennsylvania, Penn Presbyterian Medical Center | PA | 17 | 22 |
| Johns Hopkins Hospital | MD | 18 | 37 |
| Keck Medicine of University of Southern California | CA | 19 | 45 |
| Heart Institute at Baylor St Luke’s Medical Center | TX | 20 | 20 |
| University of California, San Diego Health, Cardiovascular Institute | CA | 21 | 17 |
| Saint Luke's Mid America Heart Institute | KA | 22 | 38 |
| Corewell Health William Beaumont University Hospital, Royal Oak | MI | 23 | 29 |
| Mayo Clinic, Phoenix | AZ | 24 | 9 |
| Rush University Medical Center | IL | 25 | 6 |
| University of Michigan Health Frankel Cardiovascular Center | MI | 26 | 40 |
| Scripps Memorial Hospital La Jolla | CA | 27 | 44 |
| MedStar Heart and Vascular Institute at MedStar Washington Hospital Center | DC | 28 | 10 |
| St Francis Hospital and Heart Center | NY | 29 | 7 |
| CentraCare, St Cloud Hospital | MN | 30 | 32 |
| University of Alabama at Birmingham Hospital | AL | 31 | 47 |
| University of California, Davis Medical Center | CA | 32 | 41 |
| Montefiore Medical Center | NY | 33 | 30 |
| Cleveland Clinic Hillcrest Hospital | OH | 34 | 23 |
| Duke University Hospital | NC | 35 | 36 |
| Ohio State University Wexner Medical Center | OH | 36 | 34 |
| University of California, San Francisco Medical Center | CA | 37 | 24 |
| University Hospitals Harrington Heart and Vascular Institute | OH | 38 | 42 |
| Barnes-Jewish Hospital | MO | 39 | 16 |
| University of Pittsburgh Medical Center Presbyterian Shadyside | PA | 40 | 48 |
| Mount Sinai Morningside and Mount Sinai West Hospitals | NY | 41 | 19 |
| Advocate Christ Medical Center | IL | 42 | 33 |
| Hackensack University Medical Center at Hackensack Meridian Health | NJ | 43 | 14 |
| Mayo Clinic, Jacksonville | FL | 44 | 12 |
| University of Chicago Medical Center | IL | 45 | 21 |
| Morristown Medical Center | NJ | 46 | 25 |
| University of Kansas Hospital | KS | 47 | 43 |
| Virginia Commonwealth University Medical Center | VA | 48 | 39 |
| NewYork-Presbyterian Brooklyn Methodist Hospital | NY | 49 | 50 |
| Baylor Scott & White The Heart Hospital | TX | 50 | 49 |
CMS, Centers for Medicare and Medicaid Services; USNWR, US News and World Report.
Discussion
In the current climate of performance and competition, hospitals are ranked by external metrics to measure quality. It is widely acknowledged that the USNWR hospital ranking is an important ingredient in choosing a hospital. Health care providers and patients have embraced the USNWR hospital rankings as the unanimous best per se. Traditionally, hospital ranking is conceptualized as having a causal impact on patient referrals. Each year, USNWR rates and publishes a list of the top 50 US hospitals and different specialties. There are 3 dimensions that capture the quality of care: structure (staff, equipment, and environment), process (interactions between patients and the health care system for diagnosis, treatment, and experience), and outcomes.1,2 Each dimension is complementary. On theoretical grounds, the 3 variables can be traced back to Donabedian’s model.1,2 Of the top 50 hospitals, the top 20 at the pinnacle are recognized as honor-roll hospitals.3
Beginning with graduate education, dissenting views have shaken the tenet of the USNWR rankings. The authors know surprisingly little of the drivers for top hospital rankings. This does not mean, however, that rankings are not based on rigor. Growing discussion has called into question the top 50 USNWR cardiology, heart, and vascular surgery hospitals, most notably illuminating the distinction between honor- vs non–honor-roll hospitals. From this perspective, this paper explores the linkage of the USNWR top 50 cardiology and cardiac and vascular surgery hospitals. Accordingly, the authors compared the variables used by USNWR to rank the top 50 hospitals.
The impact of USNWR rankings for undergraduate and graduate schools has been the center of recent controversy. Yale Law School withdrew from the USNWR ranking, despite being ranked first for many years.4,5 Afterward, Harvard, Stanford, Georgetown, Columbia, and Berkeley followed.4 A parallel and emerging trend occurred in January 2023, when 9 of the nation’s top-ranked medical schools announced that they would no longer participate in the USNWR ranking system.6 The schools cited philosophical differences with the system’s emphasis on factors such as standardized test scores.6 Moreover, George Q Daley, Dean of Harvard Medical School, wrote on January 17, 2023: “My concerns and the perspectives I have heard from others are more philosophical than methodological, and rest on the principled belief that rankings cannot meaningfully reflect the high aspirations for educational excellence, graduate preparedness, and compassionate and equitable patient care that the authors strive to foster in their medical education programs.”7 Moreover, Columbia University Vagelos College of Physicians and Surgeons’ Dean Katrina Armstrong made the case that rankings “perpetuate a narrow and elitist perspective on medical education.”6
Another sector where this debate is extremely relevant is hospital rankings. Empirical, as well as theoretical, researchers have argued and supported that top hospitals position themselves for increased publicity and marketing.3,8–10 The negative effect stems from the perception that rankings are used to promote name recognition, patient referrals and volume, third-party payments, and financial rewards.4,11,12 This is based on the idea that ranked hospitals cite the USNWR ratings in most (61%) of their direct-to-patient advertising.13,14 Of the top 20 USNWR hospitals, 100% advertise their ranking on their website, 70% on the primary landing page.1 In fact, the top 50 hospitals are displayed on the USNWR’s website and advertised to the public.
In 2006, Williams et al examined 744 US hospitals for cardiology and cardiac surgery.4,15 The authors found that only 23 of the top 50 hospitals demonstrated better-than-average performance, and 9 of the top 50 demonstrated significantly worse performance.4,15 Having said this, Wang et al compared 30-day mortality and readmission rates for patients 65 years or older who were hospitalized for acute myocardial infarction, heart failure, and coronary artery bypass grafting.16 Top-ranked hospitals had lower 30-day mortality, but similar or higher readmission rates.16 Another contribution from Mehta in 2019 showed the relationship between USNWR hospital rankings and actual outcomes for major cancers.3 Although the authors of that paper found lower mortality at the top 50 USNWR hospitals vs non–top-ranked hospitals, hospitals within the top 50 USNWR rankings had comparable outcomes.3 None of the 3 studies provided clear-cut results.
Debates continue as some observers have concerns with USNWR hospital rankings.4,5 Identifying USNWR methodology issues are important from several perspectives, which are not limited to the following. First, the out-of-date data used in the rankings may not represent the entire patient population.4,17 In 2022, the “procedure and conditions” metrics accounted for a large part of the rank, but some of the procedural data was 7 years old at that time.4,17 Moreover, the disproportionate number of Medicare patients used in rankings does not reflect the whole population.4 Second, “expert opinion” relies on random clinicians to rank hospitals in which they presumably have direct knowledge of the care in the hospital.4 Nearly 85% of the hospital process dimension is expert opinion, and nearly 15% is survey accounts.1,18
Mortality permeates USNWR hospital rankings and is 35% of a hospital’s score.13,19 Mortality encompasses Elixhauser comorbidities and demographics and adjusts for differences in case mix between hospitals.13,19 However, Shahian et al have shown that, due to flaws in methodology and variability in assessing comorbidities accurately, hospital mortality rates do not predict the quality of care delivered.13,20 It is interesting, furthermore, that deaths can be attributed to a specialty despite patients not being cared for by that specialty.13
Although it has been long recognized that USNWR is the most commonly referenced ranking system, the question remains: What factors matter for USNWR? The results show that each hospital’s expert opinion score is crucial. In light of the fact of the critical role of expert opinion, a central question is: Is there a better ranking system than USNWR? Having said that, further research should be aimed at discriminating clinical and financial factors. For instance, incorporating clinical factors and fiscally efficient health care delivery.
Conclusion
Top law and medical schools withdrawing from the USNWR rankings have ignited a great deal of interest in investigating hospital rankings. Comparing the top 50 hospitals for cardiology, heart, and vascular surgery is certainly a vantage point for USNWR methodology. Results mostly confirm that the expert opinion variable plays a critical role in the reputation of the top 10 hospitals. Concerning patient referrals, the authors underline that higher-tier hospitals receive more referrals than lower-tier hospitals. Overall, empirical evidence does not support significant differences among the top 50 hospitals regarding patient management, nurse staffing, patient experience, and overall satisfaction. The explanations for this are a range of possible answers that need further inquiry. Although many have argued about the merits of USNWR hospital rankings, taken together, rankings fill a strong customer demand and are sticky.
Footnotes
Author Contributions: Alexandra Kats, BA—writing original draft; Gordon H Morewood, MD, George Moser, DNP, Eric Wilkens, MD, Abul Kashem, MD, and Yoshiya Toyoda, MD—formal analysis and methodology; Huaqing Zhao, PhD—formal analysis, methodology, investigation, and writing original draft; Suyog Mokashi, MD, MBA—writing original draft, formal analysis, methodology, writing reviewing and editing, and supervision.
Conflicts of Interest: None declared
Funding: None declared
Data-Sharing Statement: The data underlying this article will be shared upon reasonable request to the corresponding author.
References
- 1.Daskivich TJ, Gewertz BL. Campaign reform for US News and World Report rankings. JAMA Surg. 2023;158(2):114–115. 10.1001/jamasurg.2022.4511 [DOI] [PubMed] [Google Scholar]
- 2.Donabedian A. Evaluating the quality of medical care. 1966. Milbank Q. 2005;83(4):691–729. 10.1111/j.1468-0009.2005.00397.x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Mehta R, Merath K, Farooq A, et al. U.S. News and World Report hospital ranking and surgical outcomes among patients undergoing surgery for cancer. J Surg Oncol. 2019;120(8):1327–1334. 10.1002/jso.25751 [DOI] [PubMed] [Google Scholar]
- 4.Sinson G, Kolinski J, Alme C, Singh S. Is it time to follow the lawyers: Should hospitals extract themselves from US News & World Report rankings? Am J Med Qual. 2023;38(3):160–161. 10.1097/JMQ.0000000000000116 [DOI] [PubMed] [Google Scholar]
- 5.Yale Law School News. Accessed 22 2022. https://law.yale.edu/yls-today/news/dean-gerken-whyyale-law-school-leaving-us-news-world-report-rankings
- 6.Shaw G. Medical schools exit from the US News & World Report ranking system. Neurology Today. 2023;23(6):1–11. 10.1097/01.NT.0000924528.56050.29 [DOI] [Google Scholar]
- 7.HMS Community . HMS Withdraws From U.S. News & World Report Rankings. Harvard Medical School. 2023. Accessed 24 March 2024. https://hms.harvard.edu/news/hms-withdraws-us-news-world-report-rankings
- 8.Malik AT, Xie JJ, Drain JP, Yu E, Khan SN, Kim J. The association of “U.S. News & World Report” hospital rankings and outcomes following anterior cervical fusions: Do rankings even matter? Spine. 2021;46(6):401–407. 10.1097/BRS.0000000000003913 [DOI] [PubMed] [Google Scholar]
- 9.Huesch MD, Currid-Halkett E, Doctor JN. Public hospital quality report awareness: evidence from National and Californian internet searches and social media mentions, 2012. BMJ Open. 2014;4(3). e004417. 10.1136/bmjopen-2013-004417 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Yaraghi N, Wang W, Gao GG, Agarwal R. How online quality ratings influence patients’ choice of medical providers: Controlled experimental survey study. J Med Internet Res. 2018;20(3). e99. 10.2196/jmir.8986 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Gambhir S, Daly S, Grigorian A, et al. Association of US News & World Report top ranking for gastroenterology and gastrointestinal operation with patient outcomes in abdominal procedures. JAMA Surg. 2019;154(9):861–866. 10.1001/jamasurg.2019.2327 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Pope DG. Reacting to rankings: evidence from “America’s Best Hospitals.” J Health Econ. 2009;28(6):1154–1165. 10.1016/j.jhealeco.2009.08.006 [DOI] [PubMed] [Google Scholar]
- 13.Mendu ML, Kachalia A, Eappen S. Revisiting US News & World Report’s hospital rankings—Moving beyond mortality to metrics that improve care. J Gen Intern Med. 2021;36(1):209–210. 10.1007/s11606-020-06002-x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Larson RJ, Schwartz LM, Woloshin S, Welch HG. Advertising by academic medical centers. Arch Intern Med. 2005;165(6):645–651. 10.1001/archinte.165.6.645 [DOI] [PubMed] [Google Scholar]
- 15.Williams SC, Koss RG, Morton DJ, Loeb JM. Performance of top-ranked heart care hospitals on evidence-based process measures. Circulation. 2006;114(6):558–564. 10.1161/CIRCULATIONAHA.105.600973 [DOI] [PubMed] [Google Scholar]
- 16.Wang DE, Wadhera RK, Bhatt DL. Association of rankings with cardiovascular outcomes at top-ranked hospitals vs nonranked hospitals in the United States. JAMA Cardiol. 2018;3(12):1222–1225. 10.1001/jamacardio.2018.3951 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Olmsted MG, Powell R, Murphy J, et al. Methodology: U.S. News & World Report 2022-23 Best Hospitals: Specialty Rankings. Accessed 22 November 2022. https://health.usnews.com/media/best-hospitals/BH_Methodology_2022-23
- 18.Olmsted MG, Powell R, Murphy J, et al. Methodology: U.S. News & World Report 2021-22 Best Hospitals: Specialty Rankings. 2021. Accessed 24 March 2024. https://health.usnews.com/media/best-hospitals/BH_Methodology_2021-22
- 19.FAQ: How and Why we Rank and Rate Hospitals. Accessed 18 November 2019. http://health.usnews.com/health-care/best-hospitals/articles/faq-how-and-why-we-rank-andrate-hospitals
- 20.Shahian DM, Wolf RE, Iezzoni LI, Kirle L, Normand S-L. Variability in the measurement of hospital-wide mortality rates. N Engl J Med. 2010;363(26):2530–2539. 10.1056/NEJMsa1006396 [DOI] [PubMed] [Google Scholar]