Transplant Community Perceptions of the Benefits and Drawbacks of Alternative Quality Metrics for Regulation

Abstract

Background:

There is concern that the metrics currently used to regulate transplant centers, one-year patient and graft survival, may have adverse consequences including decreasing higher-risk donor organ acceptance and transplant volume. This raises questions about whether alternative measures would be more appropriate.

Methods:

We surveyed ASTS and AST members (n=270) to characterize perceptions of several metrics that are used for regulation, are publicly reported, or have been suggested elsewhere, regarding their effectiveness, amenability to risk adjustment, and predicted effects on volume, mortality, and waitlist size.

Results:

Respondents rated one-year patient and graft survival the most effective measure of quality of care (mean scores=7.44, 7.31 respectively, out of 10) and most amenable to risk adjustment (mean scores=6.26, 6.13, respectively). Most respondents believed alternative metrics would not impact their center’s volume, waitlist size, or one-year transplant mortality. However, some did predict unintended consequences; for example, some believed using one-year waitlist mortality, one-year mortality of patients listed, or one-year mortality of patients referred for transplant would decrease the number of transplants performed (48.6%, 46.7%, and 48.3% of respondents, respectively).

Discussion:

Despite previously published concerns with existing regulatory metrics, most participants did not believe any metrics would outperform one-year patient and graft survival.

BACKGROUND

Quality metrics are used with the goals of regulating transplant centers, informing patients seeking transplantation, and improving quality of care. In 2007, the Centers for Medicare and Medicaid Services (CMS) established Conditions of Participation for organ transplant programs.¹ This policy predicates public funding on transplant center performance evaluations.¹ One-year graft and patient survival are the only quality metrics currently used in these evaluations, which are also made publicly available by the Scientific Registry of Transplant Recipients (SRTR).²

However, there is some concern that these measures may have adverse, unintended effects on patients pursuing transplantation.^3–13 Prior studies have found a decrease in transplant volume among centers “flagged” for low performance after the CMS Conditions of Participation were introduced, reducing access to transplantation for patients in the area.^3–5,7,10 Furthermore, concern about post-transplant outcome metrics included in report cards has been cited as a deterrent to providers’ willingness to accept organs from higher risk donors.^3,5 Although receiving a transplant, even from an organ at increased risk of disease transmission (IRD), has survival benefit compared to remaining on dialysis,^6,14 IRD organs are more likely to be discarded than non-IRD organs.^14,15 Policies that unintentionally promote organ discard may have a net negative effect on all organ failure patients by decreasing overall transplant volume.⁵ In light of these concerns, the CMS recently proposed eliminating the clinical experience, outcomes, and data reporting requirements for transplant center re-approval, though this would not impact their use in quality reporting and patient decision-making.¹⁶

It is unclear that metrics currently used to regulate transplant programs – one-year patient and graft survival - contain optimal incentives for transplant quality and volume. For example, tying payer participation to post-transplant outcomes may incentivize centers to avoid transplanting patients perceived as higher-risk for adverse outcomes. Various alternative metrics have been proposed, such as patient or graft survival under longer time-frames, or pre-transplant factors such as organ acceptance rates.¹⁷ The SRTR publishes additional metrics including transplant rate and transplant volume to inform patient and payer choice of center, but these metrics are not used for regulation.

Because there is little evidence about how transplant centers would respond to different regulatory metrics, we conducted a national survey of the transplant community to characterize perceived strengths and unintended consequences associated with the current metrics and several potential alternative transplant quality metrics that are currently used for public reporting or have been used and suggested elsewhere.

METHODS

Study population and survey distribution

In partnership with the American Society of Transplant Surgeons (ASTS) and the American Society of Transplantation (AST), we conducted a national survey of transplant professionals to gauge opinions on transplant quality metrics. This study’s target population included transplant surgeons, physicians, advanced transplant professionals, researchers, administrators, and other transplant providers who were members of the ASTS and/or AST. Survey invitations came directly from the ASTS and AST and were distributed to the societies’ memberships from March-May 2017. The survey was distributed through three separate streams. The ASTS sent survey links (one invitation and up to six reminders) to the society’s listserv (N=1,531 email addresses). The AST posted the survey in the society’s “eNews” newsletter (n=6,668 subscribers). The AST also sent survey links to four AST communities of practice (COPs): Kidney Pancreas (N=360 members), Liver and Intestinal (N=409 members), Thoracic and Critical Care (N=232 members), and Transplant Administrators (N=288 members). This is a total of 9,488 possible recipients of survey invitations. However, there is a high likelihood that there is significant overlap among these groups; for example AST COP members likely also received the eNews newsletter. Furthermore, as the largest number of potential recipients were sent the survey link in a newsletter, it is impossible to know how many newsletter subscribers read the letter and saw the survey invitation.

The online survey was hosted by Qualtrics. Participation was anonymous and respondents were not compensated for their time. Because the surveys were distributed through the AST and ASTS listservs, it is unknown how many emails were correct and belonged to transplant professionals, or how many invitations were blocked by spam filters. Therefore, it is impossible to calculate a precise response rate. As survey respondents participated in a professional capacity, the survey was acknowledged as non-human subjects research by the Johns Hopkins Medicine Institutional Review Board.

Survey design

The survey was developed by a health economist and transplant surgeon with input from members of the ASTS and AST who suggested questions and areas of focus. The resulting questionnaire was pilot tested by several transplant surgeon and a statistician. Respondents scored 15 alternative quality metrics on their effectiveness in describing quality of care (a score of 1 indicated the measure was totally uninformative, and a score of 10 indicated the measure was highly descriptive), and 10 alternative quality metrics on their amenability to risk adjustment (a score of 1 indicated the measure was least amenable to risk adjustment, and a score of 10 indicated the measure was would have perfect risk adjustment) (Table 1, Appendix 1). Respondents were also asked how annual transplant volume, one-year mortality rates, and size of the waitlist would change at their center if CMS quality regulations were based on 14 alternative metrics (Appendix 1). Additionally, respondents were asked three questions about respondent and transplant center characteristics (Appendix 1).

Table 1: Alternative Quality Metrics Included in the Survey.

Three types of metrics were posed to survey respondents: metrics that SRTR reports and CMS uses for regulation, “key transplant metrics” reported by SRTR but not used for regulation, and new metrics used or recommended in other settings When asked to predict the effect of each metric on transplant volume, one-year mortality, and size of the waitlist, respondents were told to assume that all metrics would be risk-adjusted in the best way possible.

Reported by SRTR and used by CMS for regulation
One-year patient survival
One-year graft survival
Reported by SRTR as “Key Transplant Metrics” but not used for regulation
Time to transplant
One-year waitlist mortality
Novel metrics used or recommended in other settings
Three-year post-transplant patient survival
Three-year post-transplant graft survival
One-year post transplant functional status of transplant recipients
Three-year post-transplant functional status of transplant recipients
One-year mortality of all patients listed for transplant
Transplant volume
Organ refusal
Complications and readmissions
Transplant readmission rate
Post-transplant length of stay
Patient satisfaction
Quality of life among waitlisted patients
Quality of life among transplanted patients
Quality of life among referred patients

Three types of metrics were posed to survey respondents: metrics that SRTR reports and CMS uses for regulation, “key transplant metrics” reported by SRTR but not used for regulation, and new metrics used or recommended in other settings (Table 1). Respondents were likely to have encountered the alternative metrics in these settings prior to taking the survey. When asked to predict the effect of each metric on transplant volume, one-year mortality, and size of the waitlist, respondents were told to assume that all metrics would be risk-adjusted in the best way possible. Descriptive statistics were calculated using Stata 14.2/MP for Linux (College Station, Texas).

Net effect of alternative quality metrics on transplant volume and waitlist size

To assess the predicted impact of quality metrics on overall access to transplantation, we considered the predicted effects on both waitlist size and transplant volume. Looking at the predicted change in waitlist alone could be ambiguous; for instance, a decrease in waitlist size could result from an increase in transplant rate, or from more selective listing practices. Each possible combination of predictions of a given alternative metric’s effect on waitlist size and transplant volume was categorized as having a net positive, negative, or neutral effect on the overall number of transplants performed (Table 2). For example, a metric with a net negative effect was one that was predicted to decrease the number of transplants performed, either by listing fewer patients or transplanting fewer patients.

Table 2. Net effect of alternative quality metrics on transplant volume and waitlist size.

To assess the predicted impact of quality metrics on overall access to transplantation, we considered the predicted effects on both waitlist size and transplant volume. Each permutation of the predictions of each alternative metric’s effect on waitlist size and transplant volume was categorized as having a net positive, negative, or neutral effect on the overall number of transplants performed.

Predicted effect on waitlist size	Predicted effect on transplant volume	Interpretation	Net positive or negative change in number of transplants performed
Decrease	Decrease	Listing fewer patients, Transplanting fewer patients	Negative
Increase	Decrease	Transplanting fewer patients	Negative
Decrease	No change	Listing fewer patients	Negative
Increase	No change	Listing more patients, Not transplanting more patients	Negative
No change	Decrease	Transplanting fewer patients	Negative
Decrease	Increase	Transplanting more patients	Positive
Increase	Increase	Listing more patients, Transplanting more patients	Positive
No change	Increase	Transplanting more patients	Positive
No change	No change	No change	Neutral

RESULTS

Study population

In total, 270 respondents participated in the survey (Table 3). Due to the likely overlap in audiences to which the survey was distributed by the ASTS and AST, the inability to determine how many emails were correct and belonged to transplant professionals, and the unknown number of invitations blocked by spam filters, it is impossible to calculate a precise response rate. However, assuming no overlap among the ASTS, AST eNews, and AST COPs (N=9,488 possible recipients), the estimated response rate would be 2.8%. Assuming total overlap of the three groups (N=6,668 possible recipients), the estimated response rate would be 4.0%. These estimates still do not account for the unknown numbers of incorrect emails, emails blocked by spam filters, or potential recipients who did not read the newsletters.

Table 3:

Respondent and Transplant Center Characteristics

Respondent and Transplant Center Characteristics	Percent
Years worked in field of organ transplantation
≤10	32.1
11–25	44.0
>25	23.8
Role
Surgeon	53.2
Physician/Advanced Transplant Professional	22.6
Researcher/Administrator	16.7
Other	7.5
Annual Transplant Volume (all organs)
≤50	11.9
51–100	17.1
>100	71.0

Most respondents (53.2%) were transplant surgeons; 22.6% were physicians or advanced transplant professionals, 16.7% were researchers or administrators, and 7.5% had other roles. Of respondents, 44.0% had worked in the field of transplantation for 11–25 years, 32.1% had worked in the field for ≤11 years, and 23.8% had worked in the field for 25 years. Most respondents (71.0%) were from centers with an annual transplant volume (all organs) of >100, while 17.1% were from centers with an annual transplant volume of 51–100 and 11.9% were from centers with ≤50 annual transplants. This reflects the distribution of transplant centers by volume in the US; in 2016, 50% of all US centers performed >100 transplants, 22% performed 51–100 transplants, and 38% performed ≤50 transplants.¹⁸

Effectiveness at measuring quality of care and ease of risk adjustment

Respondents rated one-year patient survival as both the most effective at measuring quality of care (mean score of 7.44 on a 10-point scale) and most amenable to risk adjustment (mean score of 6.26) (Table 4, Figure 1). One-year graft survival received the second highest mean scores in both categories: a mean score of 7.31 for its effectiveness at measuring quality of care and a mean score of 6.13 for its ease of adjustment (Table 4, Figure 1). Three-year patient survival rate and three-year graft survival rates received the third and fourth highest scores (respectively) for their effectiveness at measuring quality of care and ease of adjustment (Table 4, Figure 1).

Table 4: Mean of scores assigned to alternative metrics based on their effectiveness and ease of adjustment, and mean of scores assigned to both outcomes.

Respondents were asked to rate each of the alternative metrics on a scale of 1–10 based on the metric’s effectiveness at measuring quality of care and amenability to risk adjustment. The mean of the scores given to both outcomes for each metric was calculated.

Metric	Effectiveness at measuring quality of care*	Ease of Adjustment**	Combined Mean
1-year patient survival rates	7.44	6.26	6.85
1-year graft survival rates	7.31	6.13	6.72
3-year patient survival rates	7.02	5.82	6.42
3-year graft survival rates	6.90	5.80	6.35
Transplant Volume	5.86	N/A	5.86
Complications and readmissions	6.05	5.25	5.65
Patient Satisfaction	6.38	4.65	5.52
Organ Refusal	5.45	N/A	5.45
Functional status 1-year post-transplant	6.00	4.85	5.43
Post-transplant length of stay	5.42	N/A	5.42
Time to Transplant	5.42	N/A	5.42
Transplant readmission rate	5.31	N/A	5.31
Functional status 3-years post-transplant	5.86	4.71	5.29
Waitlist mortality rates	5.05	4.98	5.02
Evaluated patient mortality rates	4.35	4.45	4.40

^*A score of 1 indicates the measure is totally uninformative; a score of 10 indicates the measure is highly descriptive

^**A score of 1 indicates the measure is the least amenable to risk adjustment; a score of 10 indicates the measure has perfect risk adjustment

N/A: Respondents were not asked to rate these 5 metrics on ease of adjustment

Figure 1: Means of scores assigned to given metrics based on their predicted effectiveness at measuring quality of care and ease of risk adjustment.

This figure includes only those metrics that were given for both the survey item on effectiveness at measuring quality of care and ease of risk adjustment (Appendix 1). The number of responses is given for each metric.

Effectiveness at measuring quality of care: A score of 1 indicates the metric is totally uninformative; a score of 10 indicates the metric is highly descriptive Ease of adjustment: A score of 1 indicates the metric is the least amenable to risk adjustment; a score of 10 indicates the metric has perfect risk adjustment

Evaluated patient mortality rate (all-cause mortality from time of waitlisting, regardless of whether patients are transplanted or not) was rated as the least amenable to risk adjustment (mean score of 4.45) and least effective at measuring quality of care (mean score of 4.35) (Table 4, Figure 1). Patient satisfaction was rated as the second least amenable to risk adjustment (mean score of 4.65) and waitlist mortality rate was rated as the second least effective at measuring quality of care (mean score of 5.06).

Predicted effects of alternative quality metrics on one-year transplant mortality

Most respondents did not think any of the alternative quality metrics would have any effect on one-year transplant mortality at their center (Figure 2). However, the metrics three-year patient and graft survival were predicted by the most participants to decrease one-year transplant mortality (25.5% and 23.8%, respectively, Figure 2). The metrics organ refusal rate and quality of life among transplanted patients were predicted by the most participants to increase one-year transplant mortality at their center (21.2%, 18.1%, respectively, Figure 2).

Figure 2: Predicted effects of alternative quality metrics on one-year transplant mortality.

Respondents were asked if they believed the alternative metrics would increase, decrease or not affect one-year transplant mortality at their center. The number of responses is given for each metric.

Predicted effects of alternative quality metrics on transplant volume

Most respondents did not think any of the alternative quality metrics would have any effect on annual transplant volume at their center (Figure 3). However, the metrics organ refusal rate and patient satisfaction were predicted by the most participants to increase annual transplant volume (32.4% and 31.1%, respectively, Figure 3). The metrics one-year mortality of all patients listed and one-year mortality of all patients referred for transplant were predicted by the most participants to decrease annual transplant volume (30.1% and 29.7%, respectively, Figure 3).

Figure 3: Predicted effects of alternative quality metrics on annual transplant volume.

Respondents were asked if they believed the alternative metrics would increase, decrease or not affect annual transplant volume. The number of responses is given for each metric.

Net effect of alternative quality metrics on transplant volume and waitlist size

When considering the predicted effects of alternative quality metrics on both transplant volume and waitlist size, there’s some indication that certain metrics would cause an overall reduction in the number of transplants performed, resulting from the net effect of changes to listing and/or transplant volume (Table 2). About half respondents (48.6%) predicted that using the metric one-year waitlist mortality (censoring for transplant) would result in a combination of effects that would indicate an overall decrease in transplants being performed (Figure 4). Likewise, about half of respondents predicted that using one-year mortality of all patients referred for transplant (48.3%), or one-year mortality of all patients listed for transplant (46.7%) would result in a net decrease in transplants performed.

Figure 4: Net effect of alternative quality metrics on overall number of transplants performed.

To assess the predicted impact of quality metrics on overall access to transplantation, we considered the predicted effects on both waitlist size and transplant volume.

Each possible combination of the predictions of a given alternative metric’s effect on waitlist size and transplant volume was categorized as having a net positive, negative, or neutral effect on the overall number of transplants performed. For instance, a metric with a net negative effect was one that was predicted to decrease the number of transplants performed, either by listing fewer patients and/ or transplanting fewer patients.

The number of responses is given for each metric.

About one-third of respondents believed that using the metric patient satisfaction would result in a combination of effects that would indicate an overall increase in transplants being performed (32.4%). Likewise, 31.5% of respondents believed using the metric organ refusal rate would result in a net increase in transplants being performed; however, nearly the same proportion of respondents (29.3%) believed using organ refusal rate would result in a net decrease in transplants being performed (Figure 3). Most respondents predicted that all other alternative metrics would result in an overall neutral effect on both waitlist size and transplant volume (Figure 4).

DISCUSSION

This national survey of transplant professionals sought to gauge opinions on current and prospective transplant quality metrics. We found that, in general, the transplant community believes currently used metrics (one-year graft and patient survival) are the most effective measures of quality of care, and are the most amenable to risk adjustment, in comparison to many alternative metrics. Participants also rated three-year patient and graft survival as the next most amenable to risk adjustment and effective measures of quality of care, suggesting that patient and graft survival in general, regardless of the timeframe, are believed to be the most reliable metrics.

Furthermore, most respondents predicted that using the other alternative metrics in place of current regulatory metrics would have no impact on transplant center volume, size of the waitlist, and one-year transplant mortality at their center. However, some believed that certain metrics would have negative or positive effects on both access to and outcomes of transplantation. Regarding positive predictions, some respondents believed that three-year post-transplant graft and patient survival would improve one-year transplant mortality (25.5% and 23.8%, respectively) and that using patient satisfaction and organ refusal rate would result in a net increase in transplants performed (32.4% and 31.5%, respectively).

The results also revealed several potential negative effects of alternative quality metrics. Some respondents believed certain metrics would reduce transplant volume and the size of the waitlist at their centers (48.6% one-year waitlist mortality censoring for transplant, 48.3% one-year mortality of all patients referred for transplant, and 46.7% one-year mortality of all patients listed). While reducing the size of the waitlist could indicate that more candidates are being transplanted, the belief that metrics would reduce both waitlist size and transplant volume suggests that the transplant community anticipates becoming more conservative in listing practices. This finding indicates that some members of the transplant community believe that lower access to transplantation might be an unintended consequence of many alternative metrics. Furthermore, some respondents believed that organ refusal rate and quality of life among transplanted patients would increase one-year transplant mortality (21.2% and 18.1%, respectively).

Our finding that the majority of respondents generally do not believe that transplant centers change behavior in response to regulatory metrics is inconsistent with studies of the impact of CMS flagging. While some unintended consequences of alternative metrics were predicted, most respondents in this study believed the alternative metrics would have no effect on access to transplantation or transplant outcomes. For instance, most respondents predicted the alternative metrics would not impact center volume, whereas previous studies have observed quality metrics to reduce transplant volume.^3,4,7,10 Likewise, most respondents were not concerned that certain metrics would increase one-year transplant mortality, a finding that contradicts both prior research on the relationship between poor performance reports and reductions in transplant volume,^3–5,7,10 and the survival benefit of transplantation.^6,14

This survey was administered to the transplant community prior to the proposed changes in CMS regulation of transplant centers. These proposals aim to decrease reporting burden by lifting requirements to enter into systems improvement agreements, or submit mitigating factors, and report clinical experience and outcomes data for program re-approval.¹⁶ These changes also aim to address the unintended consequences of the current CMS requirements, and ultimately improve access to transplantation.¹⁶ The transplant community’s perceptions of alternative quality metrics may have shifted since this recent policy change. Despite this limitation, the results of this study are still useful to inform continued discussions of optimizing transplant regulation to improve access to and outcomes of transplantation.

This study had several other limitations. First, because the survey was distributed through the listservs of the ASTS and AST and the responses were anonymous, it is impossible to determine a response rate, nor to ensure that the sample is nationally representative. Our estimated response rates range from 2.8% to 4.0%, but do not account for the unknown numbers of incorrect emails, emails blocked by spam filters, or potential recipients who did not read the AST newsletter. While our sample size of 270 is consistent with other national surveys of the transplant community (range 83–449), the response rate of these prior surveys were higher as they used a simpler sampling scheme (range 18%−46%).^19–23 Despite this, respondents to this survey reported working in centers with a wide range of annual transplant volumes (Table 2). Second, all surveys have the potential to be affected by response bias if those with stronger opinions or a particular viewpoint are more likely to respond. However, this survey seemed to capture heterogeneous opinions, as similar numbers of respondents predicted positive and negative effects of most metrics. Given the large proportion of respondents who supported the current regulatory metrics, it may be the case that respondents were more familiar and thus more comfortable with these metrics. Third, respondents to this survey may have varying levels of familiarity with risk adjustment. However, the intention of this survey was to gauge the community’s perceptions of the alternative metrics, regardless of methodological expertise. We further believe the highly educated study population was capable of understanding the survey material. We chose to use a 1–10 point scale to assess perceived amenability to risk adjustment in order to capture a wider range of opinions than a 1–5 point scale would allow. Fourth, the survey did not address several issues that would have added value. For instance, the survey did not assess predicted effects of metrics used in combination; adding hypothetical combinations to the survey would have significantly increased participant burden. The survey did not ask respondents to predict effects on waitlist mortality, which may be another factor positively or negatively affected by alternative metrics. The survey also did not capture the specific organ programs or geographic areas represented by respondents, which may have influenced responses, especially given policy changes regarding redistricting. Fifth, while respondents were told to assume the alternative metrics posed in the survey would be used for regulation, the survey did not distinguish between predicted effects of metrics used for regulation by CMS, versus those presented publicly for patient education. Sixth, about 30% of initial respondents to this survey did not complete all the questions posed, likely due to the length of the survey, particularly the questions on the predicted effects of 14 alternative metrics. However, there were no differences in demographics between those who did and did not finish the survey and the denominator of each survey item is given in Figures 1–4 to provide a better understanding of survey attrition. Lastly, further research could add to this study by addressing the perspectives of patients on the value of alternative quality metrics.

This survey found that the transplant community believes currently used metrics are the most effective measures of quality of care and amenable to risk adjustment compared to alternative metrics. While some respondents believed that certain quality metrics may have unintended negative effects, most responses do not suggest a need to change metrics to improve access to or quality of transplantation. Overall, the transplant community may not appreciate the full impact that certain regulatory quality metrics might have on access to transplantation and post-transplant outcomes.