Considering Potential Benefits and Consequences of Hospital Report Cards: What Are the Next Steps?

Over the past decade, there has been significant growth in the prevalence and ramifications of health care provider report cards. These provider performance evaluations are believed to increase transparency for patients and caregivers, enable evaluation of best practices, enhance quality assurance, and improve quality of care if report cards motivate patients to choose high-performing providers or cause providers to improve performance to compete on quality (Berwick, James, and Coye 2003; Galvin and McGlynn 2003; Romano and Zhou 2004; Hibbard, Stockard, and Tusler 2005; Fung et al. 2008; Bundorf et al. 2009; Werner et al. 2009; Werner, Stuart, and Polsky 2010).

However, there are also concerns that patients may be adversely affected by increased use of report cards, especially when report cards are used to determine compensation or choice of providers. Report cards can be misleading to patients and payers because of inadequate risk adjustment, limited relationship between easily observed process measures and patient outcomes, the potential for providers to artificially improve outcome measures by “teaching to the test,” skimping on important areas of care that are not measured, or selecting healthier patients or those perceived as less of a threat to measured performance (Dranove, Shanley, and Simon 1992; Iezzoni 1997; Mullen, Frank, and Rosenthal 2010; Nicholas and Dimick 2011; Nicholas, Dimick, and Iwashyna 2011; Werner et al. 2011).

Report Cards in Kidney Transplantation

Report cards have been used for over a decade in the field of solid organ transplantation (Dickinson et al. 2008). These report cards are produced on a biannual basis by the Scientific Registry of Transplant Recipients (SRTR) and are publicly available online. In addition to risk-adjusted posttransplant outcomes for each type of solid organ transplant, online content contains center-level data including patient and donor characteristics and pretransplant outcomes. Outcomes are risk adjusted based on a national reference population and the center reports include “flags” that indicate whether center outcomes are higher or lower than expected. More recently in 2007, the Centers for Medicare and Medicaid Services (CMS) issued Conditions of Participation to transplant centers that explicitly tie the availability of public funding to results of the SRTR reports (Centers for Medicaid and Medicare Services 2009).

In the current issue of Health Services Research, White et al. (2015) evaluate the impact of report card performance under these high-powered incentives in the context of kidney transplantation. The authors evaluated the effect of biannual SRTR report cards on center volume, use of higher risk donors, and access to transplant among higher risk transplant candidates. The field of kidney transplantation offers unique insights into the role of report cards for several reasons. Kidney transplantation is both efficacious across donor and recipient risk factors and cost-effective relative to the alternative treatment modality of maintenance dialysis, thus promoting increases in transplantation rates has substantial benefit to the end-stage renal disease (ESRD) population. However, there is growing patient demand for transplantation and the primary limiting factor for increasing transplant rates is the scarcity of donor organs. The lack of availability of organs creates significant challenges concerning best policies for allocating these resources in an equitable and efficient manner. In addition, the field of solid organ transplantation has developed a well-defined, mandatory process for data collection among all transplant patients that facilitates the ability to utilize data for regulatory purposes. These characteristics contribute to the notion that organ transplantation is especially well-suited to performance evaluation and has been cited as a model for care and performance evaluation that has been applauded by regulators (Hamilton 2008, 2009). We discuss several of the key results from White et al., remaining evidence gaps, and implications for improved use of report cards in transplant and other fields.

Volume Declines in Poor-Performing Centers

A consistent finding between the current and prior studies of the effects of report cards in organ transplantation is a relative decline in transplant volume among “flagged” (i.e., low-performing) centers (Howard and Kaplan 2006; Schold et al. 2013a; Buccini et al. 2014). The current study adds insights regarding the timing of these changes relative to the reporting of flags and the effects of an isolated low performance period versus chronic low performance. These results are important and one might argue consistent with the aims of oversight policy designed to reduce utilization of low-performing centers. However, a common limitation of these studies is the inability to determine whether the decline in volume occurred because patients who would have received transplants at flagged hospitals were shifted to higher performing centers or because these candidates did not receive transplants at all (i.e., Did patients remain on dialysis or die waiting for a transplant?). This is a critical distinction as directing patients to higher performing centers may have substantial benefits while limiting availability of transplants among patients in need may produce unintended barriers and potential disparities in access to the procedure. A similar CMS policy restricting Medicare bariatric surgery patients to Centers of Excellence reduced procedure use among racial minorities, possibly because patients were unwilling to travel to preferred hospitals (Nicholas and Dimick 2013).

Limited Evidence of Strategic Patient Selection

One way that report cards may improve care delivery is to promote better matching between patients and providers. In the case of transplantation, this may occur by shifting higher risk procedures (based on recipient and/or donor characteristics) away from low-performing centers. White et al. found little evidence that center flagging led low-performing centers to avoid the sickest or highest risk patients—on average, patient acuity increased across all center performance groups during the study period. If anything, consistently low-performing centers were left with higher shares of patients with low socioeconomic status (based on insurance status), a group that has reduced outcomes independent of center selection (Woodward et al. 2008). Previous research has found that specifically, privately insured patients are significantly more likely to shift away from transplant centers with poor report card performance (Howard and Kaplan 2006; Schold et al. 2013a). If, indeed, private payers are using report cards to select transplant center networks more aggressively as compared to CMS, publicly funded patients may already be concentrated at lower performing hospitals.

Use of Higher Risk Organs

One of the findings of the present study was evidence that flagged centers were less likely to transplant organs from higher risk donors. These expanded criteria donor (ECDs) organs represent about 20 percent of the deceased donor kidney pool and while they are at relatively higher risk compared to standard criteria donor organs, they are still associated with a significant benefit to patients relative to chronic dialysis (Ojo et al. 2001; Merion et al. 2005). Avoiding use of higher risk organs represents another way that flagged centers may try to limit risk and potentially improve measured performance despite risk adjustment for certain donor characteristics. This is despite the fact that there is limited evidence that donor risk impacts measured quality of transplant centers (Schold et al. 2010).

White et al. suggest that donor organs that may offer substantial improvement in life expectancy for patients with ESRD are not utilized because of provider concern over performance oversight. However, an important limitation of this study is the lack of organ follow-up; are unused ECD kidneys from flagged centers ultimately transplanted at higher performing centers or discarded without identifying a willing center? Ultimately, transplant policies that incentivize posttransplant survival at the expense of total transplant volume (i.e., lead to declines in the utilization of donor organs) may have a net deleterious effect on the patient population as transplantation at even a poor-performing center is substantially better for patient outcomes than not receiving a transplant at all (Schold et al. 2014).

Lessons from Transplant Report Cards and Regulations

Cumulatively, the current study contributes to a growing literature demonstrating that transplant center report cards do alter provider and/or patient behavior, though it remains unclear whether these behavioral responses improve or worsen patient outcomes. While concentrating transplants at better centers may improve long-term transplant outcomes for certain recipients, these gains may not be offset by harms from limiting transplant volume or exacerbating disparities in access to care. Prior studies have shown that the risk-adjustment models used to categorize transplant center performance can lead to inaccurate classification because they do not include important patient comorbidities that are associated with mortality and graft failure and are observable to physicians making treatment decisions (Weinhandl et al. 2009; Schold et al. 2013b). This may lead physicians to avoid treating patients whose risk factors will not be captured in report card models. This potential harmful and unintended impact on access to transplantation is particularly relevant and difficult to measure given a wide spectrum of relative contraindications to the procedure that vary substantially among centers (Schold et al. 2008).

Several important knowledge gaps about the efficacy of transplant report cards remain. First, is the lost transplant volume among low-performing providers balanced by uptake at higher performing centers, and are these the same patients? Are there ways to better align risks of patients and providers to improve overall outcomes? Do organ turndowns lead to acceptance at other centers, or are these scarce resources lost to transplant candidates in need? Finally, what is the long-term effect of report cards related to costs and overall health of the transplant population?

Given the proliferation of available data and metrics to evaluate provider quality, it is highly likely that the impact and visibility of report cards will continue to expand. As such, it seems that prospective work to delineate the benefits and consequences of report cards should focus on several key aspects:

• Refining methodology to generate report cards, including leveraging data that may enhance risk adjustment and minimize systematic biases. Capturing data to minimize these biases may not only improve accurate comparisons of providers' quality of care but reduce unintended consequences associated with denied care.

• Generating complementary indicators of provider behavior that measure potential unintended consequences of performance oversight, to understand the effects of oversight and reporting policies and to discourage behaviors that are not in patients' best interest. Assessing whether the gains from moving patients to different hospitals are justified by the costs of any unintended consequences should be a dynamic process, as both of these factors can change over time.

• Implement formal processes to review the effectiveness of report cards and mechanisms to use information to identify best practices. A priori plans to evaluate and disseminate best practices should accompany the use of report cards for quality assurance. While the CMS have taken the lead on several national initiatives targeting hospitals, collaboration across payers will be important to ensure that evaluations are based on outcomes among all patients affected by a particular report card policy.

• Consider metrics that evaluate cost-effectiveness and value of care in conjunction with report cards measuring outcomes. Ultimately, in the context of health care reform, it is not enough to evaluate quality in isolation as alterations in patient selection may have downstream economic effects.

• Finally, perhaps we can borrow concepts from the field of psychology, which has observed that behavioral contingencies are a critical aspect of reinforcement (Jablonsky and DeVries 1972). Specifically, research suggests that rewards are far more effective than punishment to induce desirable behaviors and thus report cards that reward excellence may have greater capacity to improve quality of care than those that target underperformers (Kolstad and Lindkvist 2013).

In the end, there is enough accumulated convincing evidence that report cards affect the practice of medicine that they are (a) important to study and (b) to implement with great care and foresight. Rather than simply identifying systematic unintended consequences of report cards after the fact, prospective strategies to minimize these consequences and rigorously monitor the effects and utility of report cards are needed.