Rankings versus reality in pancreatic cancer surgery: a real-world comparison

Abstract

Background

Patients are increasingly confronted with systems for rating hospitals. However, the correlations between publicized ratings and actual outcomes after pancreatectomy are unknown.

Methods

The Massachusetts Division of Health Care Finance and Policy Hospital Inpatient Discharge Database was queried to identify pancreatic cancer resections carried out during 2005–2009. Hospitals performing fewer than 10 pancreatic resections in the 5-year period were excluded. Primary outcomes included mortality, complications, median length of stay (LoS) and a composite outcomes score (COS) combining primary outcomes. Ranks were determined and compared for: (i) volume, and (ii) ratings identified from consumer-directed hospital ratings including the US News & World Report (USN), Consumer Reports, Healthgrades and Hospital Compare. An inter-rater reliability analysis was performed and correlation coefficients (r) between outcomes and ratings, and between rating systems were calculated.

Results

Eleven hospitals in which a total of 804 pancreatectomies were conducted were identified. Surgical volume correlated with overall outcome, but was not the strongest indicator. The highest correlation referred to that between USN rank and overall outcome. Mortality was most strongly correlated with Healthgrades ratings (r = 0.50); however, Healthgrades ratings demonstrated poorer correlations with all other outcomes. Consumer Reports ratings showed inverse correlations.

Conclusions

The plethora of publicly available hospital ratings systems demonstrates heterogeneity. Volume remains a good but imperfect indicator of surgical outcomes. Further systematic investigation into which measures predict quality outcomes in pancreatic cancer surgery will benefit both patients and providers.

Introduction

Increasing interest in the quality of health care has led to the development of rating systems to evaluate the quality of care and outcomes of hospitals, physicians and managed care plans.^1–3 The earliest efforts to publicly report quality data describing health providers or facilities on a wide scale occurred in the mid-1980s when the Health Care Financing Administration (HCFA), now Centers for Medicare and Medicaid Services (CMS), published nationwide hospital-specific mortality data.⁴ Controversy ultimately ended the national programme, but efforts by media and state governments continued. The growth of the Internet has accelerated the availability of ratings. A study conducted by the Pew Research Center demonstrated that 36% of Internet users are seeking information about hospitals or other medical facilities online.⁵ The study noted that these medical queries are the fastest growing category of search.⁵ Public and private entities have responded by developing rating systems that are proposed to aid consumers in making informed decisions about their health care choices.

Among the more broadly available rating systems are those of the US News & World Report® (USN), Consumer Reports® (CR), Healthgrades® and Hospital Compare (HC), which have variable histories and strategies. ‘America's Best Hospitals’, published by USN, a weekly news magazine with a national audience, was instigated in 1990 and is now the longest running annual report card on hospital quality.⁶ Subsequently in 2012, CR began publishing annual rankings based on patient outcomes, experience and hospital practices, emphasizing safety as representing a large part of its determinations.⁷ Healthgrades, founded in 1998, describes itself as comprised of ‘healthcare quality experts … creating the standard of healthcare quality’.⁸ According to its website, each year over 200 million hits are generated on Healthgrades by people researching a new doctor or hospital. The US government launched its hospital ratings system, HC, in 2005, providing for the first time a nationally standardized and validated measure of hospital quality to the public.⁹

Hospitals frequently use these ratings in advertising, thereby increasing the influence of the ratings on public perceptions of hospital quality.^10,11 However, it is unclear whether these ratings appropriately identify high-quality hospitals for complex surgery, such as pancreatic surgery. Previous research has demonstrated that hospital quality ratings commonly provide conflicting and unreliable results.^12,13 One study demonstrated that hospitals identified by popular rating systems had lower mortality rates after pancreatic cancer surgery compared with hospitals that were not ranked.¹⁴ No studies to date have addressed the question of how well media rankings correlate with actual outcomes other than mortality following pancreatic surgery for cancer, or with resection volume.

The objective of the current study was to determine how well the ratings produced by the various systems correlate with actual outcomes following pancreas surgery for cancer. The Massachusetts Division of Health Care Finance and Policy Hospital Inpatient Discharge Database (MAHIDD) for 2005–2009 was utilized to compare mortality, complications, hospital length of stay (LoS) and a composite outcomes score (COS) for the highest-volume hospitals in the state. The present authors hypothesized that there might be dissonance between actual outcomes and rankings by popular rating systems.

Materials and methods

Database

The MAHIDD was queried to identify pancreatic resections performed for cancer during 2005–2009. The database includes case mix and charge data for all inpatients discharged from acute care hospitals in Massachusetts. It contains comprehensive patient-level information, including sociodemographic and clinical data.¹⁵ The Massachusetts Department of Public Health (DoPH) and the University of Massachusetts Medical School institutional review boards approved this study.

Patient and hospital selection: diagnosis and procedure codes

The MAHIDD was used to search for all discharge records that identified patients with the principal diagnosis of malignant pancreatic cancer submitted to surgical resection. Selected diagnosis and procedure codes were sourced from the International Classification of Diseases, Ninth Revision (ICD-9). Hospitals performing fewer than 10 pancreatic resections within the 5-year period were excluded.

Outcome analysis

The measured outcomes for this study included inpatient mortality, postoperative complications, LoS and the COS. Major postoperative complications were defined by specific diagnoses, with codes based on their validation as true complications, rather than as comorbidities, according to a method described by Lawthers et al.¹⁶ The postoperative hospital LoS was calculated by subtracting the number of days between admission and the first operation from the total LoS. Mortality was defined as death from any cause before patient discharge. The COS is an aggregated score derived from mortality, complications and LoS ranks.

Hospital quality ratings

Individual hospital ranks and ratings from each respective website within a 1-week interval in September 2012 were collected. The various approaches of each rating agency are summarized in Table 1.

Table 1.

Summary table of rating systems

	US News & World Report	Consumer Reports	Healthgrades	Hospital Compare
Database	MedPAR	MedPAR	MedPAR	CMS Certification and Survey Provider Enhanced Reporting System (CASPER)
	AHA annual survey	AHA		Medicare enrolment and claims
		State departments of health		Hospital Consumer Assessment of Healthcare Providers and Systems survey
Proprietary measure	Index of hospital quality	Safety score	Star ratings	N/A
Quality measures	Volume	Avoiding infections	30-day mortality	Timely and effective care
	Use of technology	Avoiding readmissions	180-day mortality	Patient satisfaction
	30-day mortality	Communication		Readmission
Quality measures	Patient safety	Appropriate use of scanning		Mortality
	Reputation survey	Avoiding complications		Complications
Ranks	Yes	Yes	No	No

MedPAR, Medicare provider analysis and review; AHA, American Hospital Association; CMS, Centers for Medicare and Medicaid Services; N/A, not available.

US News & World Report

Ranks from the US News & World Report Best Regional Hospitals in Massachusetts were used directly from its website.¹⁷ Using publicly available data sourced from the Medicare Provider Analysis and Review (MedPAR) database and the American Hospital Association (AHA) database, the company used a proprietary formula to determine the index of hospital quality (IHQ) score on which its hospital rankings were based.¹⁸ The IHQ score was derived from structural, process and outcomes measures, as well as a reputation score assessed through an annual survey, as documented in its methodology report.^19,20

Consumer Reports

Consumer Reports rankings were also obtained directly from the company's website.²¹ Source data are drawn from the CMS, state departments of health and the AHA.²² In constructing its ratings, CR states that its measures are compiled from outcomes (i.e. avoiding infections, readmissions and complications), experience and hospital practices (i.e. appropriate use of scanning and use of electronic health records). These measures are then combined to create a safety score, which is used to rank hospitals.

Healthgrades

Healthgrades does not provide explicit ranks, but, rather, ratings based on performance data extracted from 3 years of recent MedPAR data. The company uses a proprietary formula to predict mortality rates during hospitalization, and at 30 days and 180 days following discharge for over 28 common procedures and conditions. As Healthgrades does not apply a specific category for pancreatic cancer surgery, the ‘Gastrointestinal Surgeries and Procedures’ category was used for the analysis. According to Healthgrades, the mortality rate predicted for each hospital was compared with the mortality rate observed over the same time period. Hospitals are given a three-star Healthgrades rating if their ‘actual performance (observed mortality) does not differ significantly from the predicted rate’.²³ Hospitals with statistically significant differences between observed and expected mortality rates are divided by Healthgrades into two groups according to whether they did or did not exceed predicted performance. Up to 10% of those that exceed predicted performance are assigned five stars to reflect better than expected outcomes. Similarly, up to 10% of those with significantly worse performance are assigned one star to reflect poorer than expected outcomes. Those with three stars are rated by Healthgrades as achieving expected outcomes.²³

Hospital Compare

Hospital Compare is part of the CMS Hospital Quality Initiative and is the only rating system that is government-based and not for profit. It uses a number of quality measures that are agreed upon by hospital industry leaders and public sector stakeholders such as the Joint Commission, the National Quality Forum and the Agency for Healthcare Research and Quality.²⁴ These include timely and effective care, readmissions, complications and deaths, use of medical imaging, and surveys of patients' hospital experiences. Data are compiled from a number of sources, mainly Medicare, the Centers for Disease Control, and the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey conducted by hospitals. The ‘Timely and Effective Surgical Care’ and ‘Patient Experience’ ratings are the most relevant and complete parameters and are therefore chosen to determine ranks.

Statistical analysis

Primary outcomes included mortality, complications, median LoS and COS. The COS is an aggregated score combining mortality, complications and LoS ranks for each facility. Identical values or rank ties were assigned a rank equal to the average of their positions in the ascending order of the values. Ranks were determined and compared with: (i) volume of pancreatic cancer resections, and (ii) ratings identified from consumer-directed Massachusetts hospital ratings (e.g. USN & World Report, CR, Healthgrades and HC). An inter-rater reliability analysis was performed and Spearman correlation coefficients (r) were calculated between outcomes and ratings, and between rating systems. According to widely recognized standards, the absolute value of r was interpreted as indicating the strength of the correlation. The closer the absolute value of r is to 1, the stronger the relationship. The closer r is to 0, the weaker the relationship. A general rule of thumb for interpreting the size of a correlation coefficient is provided in Table 2.²⁵

Table 2.

Interpreting the size of a correlation coefficient

Size of correlation	Interpretation
0.00–0.19 (0.00 to −0.19)	Very weak (positive/negative) correlation
0.20–0.39 (−0.20 to −0.39)	Weak (positive/negative) correlation
0.40–0.59 (−0.40 to −0.59)	Moderate (positive/negative) correlation
0.60–0.79 (−0.60 to −0.79)	Strong (positive/negative) correlation
0.80–1.00 (−0.80 to −1.00)	Very strong (positive/negative) correlation

Results

Eleven hospitals that met volume-related criteria and accounted for a total of 804 pancreatectomies were identified. A total of 76% (n = 804) of all pancreatectomies performed in the state were carried out at these top-volume hospitals. Mortality ranged from 0% to 5.1%, complication rates from 9% to 32% and median LoS from 7 days to 11 days. Spearman correlation coefficients (r) between outcomes and rating systems were tabulated as shown in Table 3.

Table 3.

Spearman correlation coefficients between outcomes, volumes and rating systems

	Mortality	P-value	Complications	P-value	LoS	P-value	COS	P-value	Volume	P-value
Volume	0.24	0.480	0.26	0.433	0.47	0.141	0.39	0.232	–	–
USN	0.41	0.207	0.44	0.180	0.61	0.046	0.54	0.087	0.90	0.0002
CR	−0.49	0.129	−0.32	0.340	−0.34	0.313	−0.41	0.209	−0.29	0.386
HC	0.42	0.195	0.26	0.449	0.38	0.246	0.35	0.288	0.21	0.545
Healthgrades	0.50	0.118	0.37	0.267	0.31	0.350	0.44	0.176	0.34	0.301

LoS, length of stay; COS, composite outcomes score; USN, US News & World Report; CR, Consumer Reports; HC, Hospital Compare.

US News & World Report

Of all the rating systems including volume, USN rankings had the highest correlation with all outcome indicators with the exception of mortality. Complications, LoS and COS had correlation coefficients of 0.44, 0.61 and 0.54, respectively. Mortality was moderately correlated with USN rank (r = 0.41), and slightly better correlated with Healthgrades (r = 0.50) and HC (r = 0.42) rankings.

Healthgrades

Healthgrades ratings were found to have the second strongest correlation with volume (r = 0.34) after USN ratings and the highest among the rating systems with mortality (r = 0.50). This is not surprising as Healthgrades ratings are primarily based on a 30-day and 180-day mortality metric. Healthgrades ratings were weakly and moderately correlated with complications and COS, with Spearman coefficient factors of 0.37 and 0.44, respectively, second only to the correlations of USN ratings.

Hospital compare

Hospital Compare ratings were weakly correlated with volume and other outcome indicators, with the exception of mortality (r = 0.42), for which the HC rating showed the second strongest correlation after that of Healthgrades (r = 0.50). Hospital Compare ratings correlated with volume at 0.21, complications at 0.26, LoS at 0.38 and COS at 0.35.

Consumer Reports

Consumer Reports was the only ratings system to display negative relationships with all outcome categories, including volume.

Volume

Volume remained only weakly or moderately correlated with all outcome indicators compared with other rating systems. Among all outcome indicators, volume was most correlated with LoS (r = 0.47), which was second to USN (r = 0.61). Mortality and complications were weakly correlated, with coefficients of 0.24 and 0.26, respectively. When all outcome indicators were aggregated to the COS, volume was only weakly correlated (r = 0.39).

Volume rankings were weakly or moderately correlated with outcomes (mortality, complications, LoS and COS). Healthgrades ratings were most strongly correlated with mortality. Hospital Compare ratings were less correlated and CR ratings were inversely correlated with all outcomes. Of all the ratings systems evaluated, including volume, USN ranks were most highly associated with outcome indicators.

Discussion

This study of four widely available hospital rating systems demonstrates that these services failed to show consistent agreement when analysed in a single state and compared with true outcomes following pancreatectomy. Among the institutions studied, hospitals that were ranked most highly in one system were often ranked very poorly in another. Heterogeneity among ratings may reflect variations in the weighting methodology of each respective service. As state DoPH data indicate, although surgical volume is correlated with outcomes, it is not the best indicator of surgical outcomes following pancreatectomy. Notably, CR ranks were inversely related to all outcomes. Of the various rating systems, USN was most correlated with surgical outcome indicators.

The association of volume and outcomes has been recognized for at least three decades and volume is often used as a proxy for surgical quality.^26,27 However, the assessment of true quality in surgery is complex.²⁸ The results of this study demonstrate that although volume is associated with outcomes, it is an imperfect indicator. In a seminal paper, Donabedian proposed that the discussion of quality in health care should be framed by three measures: structure; process, and outcomes.²⁹ Although most of the rating systems utilize aspects of these concepts, they differ in their approaches. Consumer Reports ratings showed the greatest divergence from observed outcomes of pancreatic resection. This may be explained by the fact that the outcome measures emphasized by CR included ‘avoiding bloodstream infections’, ‘avoiding surgical site infections’, ‘avoiding readmissions’ and ‘avoiding complications’. Patient experience and hospital practices were also heavily factored into the CR safety score. Volume was not factored into its ratings. By contrast, USN placed heavy emphasis on mortality, survival, volume and the reputation of the hospital. Its reliance on these aspects brings its rating system into better alignment with volume and surgical outcome measures. This example highlights the potential confusion a patient might encounter when seeking information about hospitals performing pancreatectomies for cancer using widely available rating systems.

The results of this study build upon other recent reports querying the validity of rating systems for different diagnoses. Krumholz et al. investigated whether Healthgrades ratings of hospitals for mortality in acute myocardial infarction accurately discriminated among hospitals for performance based on the process of care and outcomes.³⁰ They found that ratings poorly discriminated between any two individual hospitals for process of care or mortality rates.³⁰ Osborne et al. studied the ability of rating systems to identify high-quality hospitals with reference to cardiovascular and oncological procedures and found no significant difference in mortality rates between top-rated hospitals and other hospitals with similar volumes of patient throughput.³¹

This study has several limitations that must be considered. This analysis examined 11 hospitals in Massachusetts, which accounted for 76% of all pancreatectomies performed in the state. However, the relatively small sample size precluded the achievement of enough statistical power for Spearman correlation coefficients. In addition, the limitations of the database meant that only a few of the metrics, including mortality, complication rates and LoS, that define surgical quality were available for use. Outcomes that may be equally important include readmission rates, receipt of postoperative adjuvant therapy, costs and longterm survival.³² For future studies, the linkage of the current inpatient database to other administrative records, such as outpatient archives, and access to a unique identifier, will allow for the longitudinal analysis required to develop a more robust quality score. Furthermore, the use of administrative data is subject to potential deficiencies, including the provision of insufficient clinical information and confusion concerning complications and pre-existing conditions.³³ Finally, the ranking systems used in this study, because of the nature of the comparative local database, were all based in the USA and the present findings may not be generalizable to non-US ranking systems.

Despite these limitations, the present analysis is the first to examine the correlations among multiple rating systems and to assess various domains of surgical quality, including mortality, hospital LoS, complications and a COS after pancreas surgery performed for cancer. Four widely available rating systems were evaluated and demonstrably significant variations in the correlations between rating systems and actual surgical outcomes were identified.

Conclusions

In the current era, a push to make health care more transparent has resulted in the proliferation of public rating systems. This study demonstrates that hospital rating systems show significant variation. In their current state, hospital rating systems have the potential to confound rather than to clarify true quality. Furthermore, surgical volume remains a good but imperfect indicator of surgical outcomes following pancreatectomy. Further systematic investigation into which measures truly predict quality outcomes in pancreatic cancer surgery will benefit both patients and providers.

Conflicts of interest

None declared.