Measuring transplant center performance: The goals are not controversial but the methods and consequences can be

Abstract

Purpose of Review

Risks of regulatory scrutiny has generated widespread concern about increasingly risk averse transplant center behaviors regarding both donor and candidate acceptance patterns. To address potential unintended consequences threatening access to care, we discuss recent changes in regulatory metrics and potential improvements in quality oversight of transplant centers.

Recent Findings

Despite many recent changes to one-year patient and graft survival regulatory criteria, the capacity to accurately identify true underperforming centers and avoiding false positive flagging remains an area of great concern. Numerous studies have demonstrated restrictions in transplant volume and access following transplant center flagging.

Summary

Current regulatory criteria are limited in their capacity to accurately identify poorly performing centers and potentially encourage risk-averse behavior by transplant centers. Efforts to address these concerns should focus on (1) improving risk-adjustment models with better data which captures the acuity of candidate and donor risk, (2) reconsidering primary outcomes measured to assess comprehensive transplant center performance, (3) improving education to address rational or perceived disincentives, and (4) using data more effectively to share best practices.

INTRODUCTION

Health care reform remains a topic at the center of our current national attention. Attempts to measure and regulate provided care are not unique or new to the field of organ transplantation. In fact, transplant centers have long been a beacon of a healthcare model designed around heightened transparency of outcomes and regulatory oversight based on measured outcomes. Since 1984 with the passage of the National Organ Transplantation Act (NOTA), a national registry for transplant patients and organ matching was formed.[1] This established the Organ Procurement and Transplantation Network (OPTN), and the contract was awarded to the United Network of Organ Sharing (UNOS) which collects data from transplant centers, organ procurement organizations (OPOs), and histocompatibility laboratories on all transplant candidates, recipients and donors. The OPTN is responsible for providing governance and oversight to all of its member institutions, namely transplant centers and OPOs.

In 1987, The Scientific Registry of Organ Recipients (SRTR) was founded under a separate federal contract to maintain and analyze the OPTN data in conjunction with data from the Centers for Medicare and Medicaid Services (CMS) and the Social Security Death Master File. The Transplantation Amendment Act of 1990 established requirements for public reporting of transplant outcomes including program-specific transplant survival rates.[2] The first of these reports appeared in 1992. These reports have since been used by the OPTN to monitor transplant center performance and trigger reviews by the Membership and Professional Standards Committee (MPSC) when lower than expected outcomes were noted. Additionally, CMS, the largest single payer for transplant services specifically related to the End-stage Renal Disease entitlement, has historically mandated certain volume and survival requirements for transplant centers in order to maintain certification and receive reimbursement for transplant services.

Early efforts

Despite these laudable early attempts to monitor and regulate transplant centers, multiple highly-publicized unfortunate events and questionable practices generated concerns about the effectiveness of CMS and OPTN oversight. Several reports appeared in 2006 in the Los Angeles Times regarding a few California transplant centers lacking necessary staff needed to care for listed patients and bypassing patients on the list [3–5]. These stories raised grave questions about the capacity of the MPSC and CMS to oversee and regulate transplant centers. In response to these events and heightened attention to transparency and quality oversight, CMS published the Final Rule and Conditions for Approval and Re-Approval of Transplant Centers in 2007.[6] This Final rule included greatly updated Conditions of Participation (CoP) which established minimum standards for one-year post-transplant patient and graft survival as requirements for center certification and maintenance of funding. A center whose observed to expected deaths or graft failures exceeded the following thresholds would be cited for review and would be at risk for losing certification and funding: (1) Observed-Expected > 3, (2) Observed/Expected (O/E) > 1.5, and (3) a 1-sided p < 0.05 based on a rolling 2.5 year cohort. Expected rates are calculated according to detailed risk-adjustment models which include many donor and recipient characteristics as captured by the OPTN data and updated by the SRTR every 6 months. These survival rates are publicly available in the program specific reports (PSRs) reported by the SRTR every 6 months as well. The above O/E criteria were used as the original flagging criteria by both CMS and the MPSC with the noted difference of utilizing a one-tailed t-test rather than the two-tailed test reported by the SRTR.

Many in the transplant community have expressed concerns about these metrics and how they are being used and the potential impact on centers to develop risk averse behaviors. Previously, these publicly-reported metrics were used by the OPTN to trigger a peer-review process by the MPSC to evaluate if any evidence of transplant center quality deficiencies existed. The use of these survival metrics by CMS as minimum criteria for CoPs have been criticized as a “bright line test” determining whether centers will be allowed to perform transplants for Medicare and Medicaid patients. As well as CMS being the largest payer for transplants, most commercial payers have followed suit and also monitor and have criteria based on first year patient and graft survival for determination of centers of excellence and maintenance of private contracts and funding. Moreover, the ability to avoid a second bad “report card” is limited given that the 6 month reports reflect a 2.5 year rolling cohort so on average only 20% of the cohort will change each report period.

Recent updates

As of May 2016, CMS raised the O/E ratio for a “conditional” level deficiency from 1.5 to 1.85 in order to mitigate risks of losing Medicare participation based on these outcome measures alone. The 1.5 threshold is retained as a “standard” level deficiency requiring QAPI efforts, but does not immediately put Medicare participation at risk or require filing of a corrective action plan.[7] Prior to this in the fall of 2014, the MPSC adopted new flagging criteria based on Bayesian methods. Under these criteria, programs with a 75% probability of the hazard ratio (HR) for survival exceeding 1.2 or a 10% probability of the HR >2.5 would be flagged. The intent was to reduce the rate of false positive flagging in small-volume programs and improve the power to detect true positives in mid-volume programs.[8]

Related to ongoing concerns regarding the impact of flagging concerns on transplant center risk averse behavior potentially compromising patient access to transplant and robust utilization of donors, further changes to the criteria used by the MPSC are being considered. A recent proposal sponsored by the MPSC involved a multiple tier system: (1) Tier 1: programs with a >60% probability that HR>1.75 would undergo expanded program review (100% probability of MPSC review), (2) Tier 2: >60% probability that HR>1.25 will be subject to a routine program review (50% probability of MPSC review), (3) Tier 3: HR>1 will be subject to a routine review (10% probability of MPSC review). This proposal has not moved forward due to negative public comment. A detailed depiction of the old and new CMS and MPSC criteria are provided in Table 1.

Table 1.

First year patient and graft survival metrics and criteria used for regulatory monitoring of transplant centers according to SRTR, MPSC, and CMS.

	Prior	Current	Proposed
SRTR Program Specific Reports	If O/E >1 AND 2-sided p<0.05, “lower than expected survival If O/E <1 AND 2-sided p<0.05, “higher than expected survival	If beyond lower threshold of 95% CI for probability that HR>1 based on Bayesian method, “lower than expected” survival If beyond upper threshold of 95% CI for probability that HR<1 based on Bayesian method, “higher than expected” survival
MPSC (OPTN/UNOS)	O-E>3, O/E>1.5, AND 2-sided p<0.05	If probability >75% that HR>1.2 OR probability > 10% that HR>2.5*	Tier 1: If probability >60% that HR>1.75, 100% reviewed Tier 2: If probability > 60% that HR>1.5, 50% randomly reviewed Tier 3: If probability > 60% that HR>1, 10% randomly reviewed*
CMS	O-E>3, O/E>1.5, AND 1-sided p<0.05	Standard-level deficiency: O-E>3, O/E>1.5, AND 1-sided p<0.05 Conditional-level deficiency: O-E>3, O/E>1.85, AND 1-sided p<0.05

^*Current and proposed MPSC criteria are based on Bayesian methodology as utilized in current SRTR program specific reports.

Statistical accuracy

In one review of the original CMS criteria, 11% of U.S. transplant centers had a least one transplant program (kidney, liver, or heart) that failed to meet these CMS criteria, almost twice as contemporaneously flagged according to the SRTR criteria based on a 2-tailed test.[9] In a stochastic simulation of flagging risk according to the original CMS criteria, 10% of programs would be falsely flagged in a 4.5 year period with the highest rate (16%) of false flagging in high-volume centers due to random variation alone.[10] Additionally, only 32% of centers assigned as poorly-performing centers (defined as an O/E=2) were correctly flagged. According to analysis provided by the SRTR in 2014, utilization of Bayesian methods for flagging would reduce false positive flagging rates by 50% for kidney, 35% for liver, 43% for heart and 57% for lung programs. Additionally, it would continue to detect 96%, 71%, 58% and 83% respectively of true positive flagging compared with prior methods.[8] However despite early promising data regarding the Bayesian methods, a comparison of flagging according to the Bayesian methods with the CMS criteria showed approximately 33% of kidney programs performing > 10 transplants had at least one low-performance rating compared with 23% of programs based on the original CMS criteria.[11] In their analysis of transplant center PSRs from 2013–2015, they identified four-fold higher rate of low performance (LP) evaluations for small-volume centers based on first year patient survival according to Bayesian methods compared with new-CMS criteria. For large volume centers, a three-fold higher rate of LP evaluations was seen. The authors concluded that a significant number of kidney transplant programs are identified as LP according to Bayesian methods despite relatively small survival differences compared with expected. They also concluded that while Bayesian criteria were associated with significantly higher flagging rates, the new CMS criteria modestly reduced flagging compared with the old CMS criteria from approximately 8% to 6% of programs underperforming every six months.[11] A recent analysis demonstrated the newly proposed MPSC criteria performed worse in terms of sensitivity, specificity, positive predictive value, and negative predictive value at all performance levels considered.[7] The total number of MPSC reviews overall increased, and programs with greater underperformance could potentially escape review.

Regardless of the statistical approach for identifying LP centers (i.e. Bayesian or traditional “frequentist” methodology), the lack of a highly predictive model for transplant outcomes remains a concern due to imperfect data being available for risk adjustment. The risk adjustment models developed by the SRTR for one-year graft loss all have approximate concordance index of approximately 0.65.[12] Given that a value of 0.5 represents no predictive capacity beyond a random coin toss and a value of 1.0 represents perfect discrimination or predictive capacity, values in the range of 0.65 indicate that there is a significant unexplained proportion of variation in transplant outcomes that are not accounted for by currently collected data. Thus at the end of the day, regardless of whether the statistical models used to summarize these data are methodologically reasonable, the unexplained variation suggests biases may exist between centers. That is, underlying factors not accounted for in the models may differ between centers. These concerns suggest some caution should accompany the interpretation of center outcomes evaluation and the strict interpretation of low quality of care attributed to center practice.

UNITENDED CONSEQUENCES AND DISINCENTIVES

Critics of regulatory metrics have cited concerns that program specific reports being utilized in essence as report cards hinders access to transplant directly and additionally incentivizes transplant centers to become risk averse resulting in further restrictions in access for certain groups of patients.

Numerous studies have demonstrated the relationship between “flagging” or of LP evaluations and changes in transplant volume. White et al. demonstrated that transplant volume declined 38% among programs with ongoing non-compliance with CMS CoP compared with a 6% increase in those programs remaining in compliance.[13] Schold et al. identified a mean decline of 22.4 transplants annually in centers with a of LP PSR compared with an average 7.8 transplant increase annually for other centers during the same time period.[14] Of concern, almost a quarter of kidney transplant centers had LP report during this period. Most alarming is that in the two years following the publication of the 2007 CoP, there was a decline in national kidney transplant volumes.[15] LP evaluations do not impact CMS participation alone, but can impact private insurance contracts even more. Studies have demonstrated a significant decline in the proportion of patients who had private primary insurance in centers with a LP evaluation.[14] Whether the primary focus for Medicare or private insurance is strictly on access to care or not, the gravity of these issues is crystallized when one compares survival of patients at high performing centers and low performing centers compared with the markedly inferior survival in patients not transplanted at all as depicted in Figure 1.[16] Based on these data, receiving a transplant at a highly underperforming center has greatly improved survival when compared with remaining on the waiting list for end-stage renal disease patients.

Figure 1.

Patient survival after listing by transplant status and center quality on the basis of deceased donor transplantation at a transplant center with a given performance at the time of listing. Used with permission from American Society of Nephrology, 2016.[16]

Similar repercussions have been identified for liver transplant centers. Buccini et al. reviewed the SRTR reports from 2007–2012 and demonstrated an average decrease of 39.9 transplants for liver transplant centers with at least one LP report compared with a 9.3 transplant average increase for centers with no low performance reports (p<0.01).[17] They also saw significant reductions in listing (67.3 decreased candidates for LP centers vs 14.9 increased candidates for no LP centers, p<0.01). Also noteworthy from this study, there was no statistical association between transplant volume change and measured performance over the remaining study period. Thus centers that either retracted transplant services or alternatively expanded programs were just as likely to increase or decrease measured performance. This suggests that efforts to improve measured outcomes by reducing volume is not effective towards improving measured performance while limiting access to care for end-stage patients.

Risk aversion – recipient risks

Reductions in access to transplant may be magnified in those patients who represent a perceived threat to performance metrics. White et al. showed evidence that programs with ongoing LP had a higher burden of risk due to recipient factors.[13] It is not surprising then, that one strategy employed by transplant centers fearful of regulatory trouble is more restrictive listing and transplant practices. In a survey of individuals in transplant administrative roles, 81% of centers with LP evaluations in the past 3 years had increased candidate selection criteria, and 94% of centers with LP evaluations and lost contracts had increased their criteria.[15] Older patients with end-stage renal disease have received much of the attention with respect to more restrictive eligibility criteria related to the inherent higher mortality risks and potentially heavier burden of cardiovascular disease in older patients. This restricted access to relatively high risk candidates creates a conflict between the interests of the patient and perceived benefits to the transplant center. Numerous studies have documented greatly improved survival in older patients (60 years or more) compared with remaining on dialysis.[18, 19] Cardiovascular risk has been identified as an area of concern among liver transplant candidates.[20] A decline in the cardiovascular risk index among liver transplant recipients was demonstrated following implementation of the CMS CoP according to a review of the UNOS database suggesting increasing recipient risk aversion related to regulatory metric concerns.[20, 21]

Risk aversion – donor risks

Complicating access issues is potentially more restrictive donor criteria among centers fearing LP evaluation and punitive action. Concerns about the impact of high risk donors on center metrics may not be well-justified given current SRTR risk-adjustment models, but nonetheless many centers still cite donor quality as an area of concern for potential risk. Surveyed transplant center administrators also reported on increased selection criteria for donors more often in LP centers (77% of LP centers vs 31% of others, p<0.001).[15] White et al. demonstrated a 55% decrease in ECD volume (compared with the overall 38% in center transplant volume) for centers with ongoing non-compliance with CoP.[13] Similarly, Schold et al. showed a 4.7 ECD transplant decline for centers with LP evaluations compared with a 3.9 increase in centers without LP evaluations.[14] Similar trends have been demonstrated for liver transplant centers. Buccini et al. showed reduced utilization of older liver donors, donors with long cold ischemic times, and donation after cardiac death donors by liver transplant centers with LP evaluations.[17]

Avoidance of ECD or now high kidney donor profile index (KDPI) kidneys in particular further compounds the access problems facing older patients with end-stage renal disease. KPDI >85% has replaced ECD as the designation used for allocation to represent grafts with higher risks of graft failure. Unfortunately, more than half of KDPI >85% are still discarded each year, [22] despite the persistent reality that over half of patients more than 60 years old will die before receiving a deceased donor kidney transplant.[23] Furthermore, multiple studies have documented the survival benefit associated with high KDPI transplant compared with waitlist mortality including those who go on to undergo low KDPI transplant.[24, 25] Interestingly, donor acceptance patterns are also undoubtedly highly influenced by fiscal realities. Schold et al. demonstrated even greater reductions in living donor kidney transplants (LDKT) compared with ECD transplants in centers with LP evaluations as highlighted in Figure 2.[14] This is a potential reflection of a loss of private contracts after a LP higher rates of LDKT among recipient with private insurance as the primary payer.[26, 27]

Figure 2.

Change in transplant volume according to donor type after centers’ receipt of a low performance evaluation. Used with permission from Wiley Periodicals, Inc., 2016. [14]

Threat to innovation

Another critical area of concern involves the inability to inadequately risk adjust for innovative and experimental practices.[9, 20, 28] As such, heightened regulation and attention to the post-transplant survival metrics will “threaten scientific innovation and advancement.”[9] The American Society of Transplant Surgeons (ASTS) touched on this important consideration citing desensitization and ABO-incompatible kidney transplant protocols as important areas of research and improving techniques which have proven success in terms of improved survival for patients compared with no transplantation, but results in inferior outcomes compared with standard kidney transplants thus threatening center metrics given the lack of protection provided by current risk-adjustment.[9] An analysis of a multi-institutional experience with desensitization demonstrated a two-to-ten-fold increased risk in flagging in centers performing 5–20% positive cytotoxic crossmatch transplants.[29]

Noted areas of consideration for liver transplant include down-staging for hepatocellular carcinoma, transplant in recipients with human immunodeficiency virus, and utilization of smaller left lobe grafts with inflow modification techniques.[20]

Resource diversion and augmented cost burden

Increased documentation requirements and monitoring and further development of quality assurance and process improvement (QAPI) programs at transplant centers have resulted in increased costs faced by transplant centers without complementary changes in funding, and are outside allowable costs under Organ Acquisition Center provisions.[15, 30] Centers have added personnel to comply with the needed documentation and monitoring related to CMS regulations and/or have had to improve existing electronic medical record infrastructures and support, all frequently very costly endeavors.[30]

FUTURE DIRECTIONS

In order to improve the validity of flagging for poor performance, the critical issues that must be addressed include (1) gaps in risk-adjustment threatening access for higher risk patients and attenuating innovation, (2) defined metrics that are focused only on one year post-transplant survival and ignore longer term survival, pre-transplant outcomes, processes of care, or other measures of value, and (3) refinement of processes to limit the frequency of spurious flagging related to statistical noise.[10]

CUSUM

Cumulative sum (CUSUM) charts are currently available to programs through the SRTR secure site. Details of these charts have been previously published.[31] These charts are not publicly available, but provide programs with more real time assessment of current center trends in patient and graft survival. This may provide more granular opportunities for QAPI efforts. These charts are not used to flag programs but could be used to demonstrate potential recent improvements in center performance for transplant centers responding to being flagged. Conceptually, use of these data and other ‘real-time’ quality surveillance may lead to improved identification of quality deficiencies and more expedient interventions independent of regulatory influence.

COIIN

The Collaborative Innovation and Improvement Network (COIIN) is a currently underway OPTN pilot project aimed at promoting utilization of deceased donor kidneys specifically high KDPI kidneys and studying alternative methods of transplant center monitoring. The project intends to reduce risk avoidance behaviors and disincentives created by current monitoring system, test alternative data rich frameworks for monitoring, and support a more collaborative approach to process improvement. Unfortunately, it may be difficult for COIIN to achieve these goals as only MPSC action will be waived for programs participating in this program. With no real change to the data published in the biannual PSR’s during the study period, programs may still face punitive action by CMS and possible loss of private contracts.

Pre-transplant metrics

Concerns about the single-minded focus on one-year post-transplant survival prompts questions regarding access to transplant and whether these metrics may shift the mortality burden to the pre-transplant period. Options may include utilization of offer acceptance or transplant rates to evaluate programs on relative conservativeness or aggressiveness of organ utilization. However, similar issues of potential unintended consequences and inability to completely risk adjust may remain salient with these metrics. Ultimately some combination of pre- and post-transplant metrics may at least provide a more comprehensive assessment of program processes of care and outcomes.

Recent MPSC proposal

The recent proposed criteria for MPSC flagging received much negative public comment resulting in a decision to not move forward with this proposal. Early analysis raised several concerns about the new criteria including more review of programs with little underperformance, potential escape from review in programs with greater underperformance, and greater review of programs may increase number of centers developing risk averse behaviors.[7]

CONCLUSIONS

To address negative consequences threatening access to transplant related to current regulatory performance metrics, we need to (1) improve current risk-adjustment models with better data, (2) determine if we are measuring the right outcomes and consider other approaches to evaluate the comprehensive quality of centers, (3) improve education and address potential real or perceived disincentives created by transplant center report cards, and (4) use data more effectively to share best practices in the spirit of learning rather than a means to justify punitive actions.