Reported Effects of the SRTR 5-tier Rating System on US Transplant Centers: Results of a National Survey

Abstract

Background:

In the US, the Scientific Registry of Transplant Recipients (SRTR) provides publicly available quality report cards. These reports have historically rated transplant programs using a 3-tier system. In 2016, the SRTR temporarily transitioned to a 5-tier system, which classified more programs as under-performing.

Methods:

As part of a larger survey about transplant quality metrics, we surveyed members of the American Society of Transplant Surgeons and American Society of Transplantation (N = 280 respondents) on transplant center experiences with patient and payer responses to the 5-tier SRTR ratings.

Results:

Over half of respondents (n=137, 52.1%) reported ≥1 negative effect of the new 5-tier ranking system, including losing patients, losing insurers, increased concern among patients, and increased concern among referring providers. Few respondents (n=35, 13.7%) reported any positive effects of the 5-tier ranking system. Lower SRTR-reported scores on the 5-tier scale were associated with increased risk of reporting at least one negative effect in a logistic model (p<0.01).

Conclusions:

The change to a more granular rating system provoked an immediate response in the transplant community that may have long-term implications for transplant hospital finances and patient options for transplantation.

INTRODUCTION

The use of transplant quality monitoring in the United States is one of the most aggressive programs of its type. Under this system, quality monitoring reports are used to both regulate transplant programs and inform the public about transplant center performance. Various methods for monitoring transplant quality are being discussed in other countries (1–4); the development of such systems could be informed by the experiences of transplant centers in the United States.

The Scientific Registry of Transplant Recipients has been charged with providing publicly available hospital performance report cards for over a decade (5, 6). Report cards are designed to improve patient decision-making and promote quality improvement within hospitals. Beyond helping patients choose appropriate transplant hospitals, report cards also improve standardization and documentation of care (7).

However, there is significant concern that report cards might have unintended consequences on access to transplantation (3, 4, 6, 8–16). Past studies have shown that transplant hospital metrics do influence hospital behavior. For example, transplant hospitals that received regulatory flags subsequently reduced volume and use of higher risk organs (11, 14, 15). Furthermore, there are concerns that compliance metrics would stifle innovation in the field of transplantation by discouraging hospitals from experimenting with new techniques or transplanting sicker patients in light of concern that risk adjustment may not fully account for the higher probability of adverse outcomes among these patients (9, 12, 13). Report cards may also influence patient and payer behavior; if payers exclude transplant hospitals labeled as “poor performing” from their networks, patients may lose feasible access to transplantation if there are no higher quality hospitals in that geographic area (10, 17–20).

Historically, the SRTR report cards followed a 3-tier system, indicating if programs are performing worse than expected, as expected, or better than expected. The SRTR 3-tier system most recently used Bayesian methods based on the risk of 1-year death and graft failure for each transplant hospital (21, 22). This system requires observations from a large number of patients across a relatively short time period in order to reliably differentiate hospitals that were better or worse than expected with statistical significance. Partly because many centers do not achieve these volume thresholds, the vast majority (≥95%) of centers were historically classified as performing as expected, even though there was considerable heterogeneity in their estimated graft and patient survival rates (21, 22). Concerns were raised about the utility of report cards that identified few distinctions between most transplant centers (22, 23).

In response to these concerns, the SRTR launched a new, publicly available 5-tier system in 2016 that aimed to provide more granularity with regards to outcomes (22, 23). On the 5-tier scale, a score of 5 indicates a program is performing “better than expected,” 4 indicates “somewhat better than expected,” 3 indicates “as expected”, 2 indicates “somewhat worse than expected,” and 1 indicates “worse than expected.” In the 5-tier system, 145 transplant programs, across all organ types, were categorized as performing worse or somewhat worse than expected, in comparison to only 9 programs in the 3-tier system (out of 547 programs total) (24). Like the 3-tier system, the new 5-tier system estimates transplant hospital-specific risk of 1-year death or graft failure; however, unlike the 3-tier system, the 5-tier system does not use a traditional test of statistical significance to demonstrate that a transplant hospital is doing better or worse than expected. Instead, the 5-tier system ranks all transplant hospitals in a single list, based on how they performed relative to expectation, and uses newly defined cutoffs to separate transplant hospitals into five groups (25).

The 5-tier rating system became publicly available as the primary source of SRTR ratings on December 2016 and remained the primary system until February 2017, before it was rolled back and temporarily moved to a beta site. This occurred as we were preparing a broad survey funded by the Arnold Foundation about improving transplant metrics. To better understand the impact of the more granular 5-tier system on transplant centers in the U.S., and to inform the development of quality monitoring systems in other countries, we leveraged this existing national survey and added a section about experiences during the 3 months of this system.

METHODS

Study Population and Survey Distribution

We conducted a national survey of U.S. transplant professionals to gather information about current and prospective transplant quality metrics. One section of this survey was devoted to opinions about and experiences with the new SRTR 5-tier rating system. To ensure that response rates would not be influenced by a particular individual’s attitude towards the new system, participants were not informed that the survey would contain questions about the SRTR 5-tier rating system prior to participating in the study.

This survey was developed and administered in partnership with the American Society of Transplant Surgeons (ASTS) and the American Society of Transplantation (AST). Our target population included transplant surgeons, physicians, administrators, professionals and researchers. Our sample included members of ASTS and AST who were contacted by email from March – May 2017. Invitations to participate in the survey came directly from the ASTS and AST. ASTS members were sent survey links through the society’s listserv and were sent six reminders. The survey was also sent to all AST members, and four specific AST Communities of Practice (Kidney Pancreas, Liver and Intestinal, Thoracic and Critical Care, and Transplant Administrators). The AST listserv and each Community of Practice were sent three reminders. The online survey was hosted by Qualtrics. Participation was anonymous and respondents were not compensated for their time.

Survey Design

Respondents were first asked three questions on respondent and transplant hospital characteristics (Appendix 1). Respondents were then asked to report the lowest rating their hospital received for each transplant program (kidney, liver, heart, lung and pancreas) on the 5-tier scale. Respondents were given a list of four positive and four negative potential effects of the SRTR 5-tier system, and were asked if their transplant hospital experienced any of these effects. The negative effects listed in the survey were: lost insurers, lost patients, increased concern among patients, and increased concern from referring providers; the positive effects listed were: gained insurers, gained patients, decreased concern among patients, and decreased concern from referring providers. Respondents were also asked to list any other positive and negative effects of the 5-tier system on their transplant hospital and to rate the effectiveness of patient and graft survival rates at measuring quality of care on a scale of 1 to 10, a score of 10 meaning the measure is highly descriptive. As survey respondents participated in a professional capacity, the survey was designated non-human subjects research by the Johns Hopkins Medicine IRB.

Lowest SRTR score among all transplant programs

Respondents reported their hospital’s lowest SRTR score on the 5-tier scale for each transplant program. For the following analyses, we used each respondent’s lowest reported SRTR score across all organ programs; for example, if a center received scores of 4, 3, 3, 3, and 2 for their kidney, liver, pancreas, heart and lung programs respectively, their lowest score used in these analyses would be a 2.

Statistical Analyses

We assessed relationships between prevalence of at least one effect of the SRTR 5-tier system and respondent characteristics, annual center transplant volume, and lowest SRTR score using logistic regression. We explored relationships between individual negative effects of the 5-tier system and respondent characteristics, annual transplant center volume, and lowest SRTR score using chi-squared and Fisher’s exact tests. Analyses were performed using STATA 14.2/MP for Linux (College Station, Texas). Statistical significance was defined as P≤0.05.

RESULTS

Study Population

In total, 280 people responded to the questions about the SRTR 5-tier rating system. Because it is unknown if the ASTS/AST email addresses are correct, if they actually belong to transplant clinicians, and if those individuals actually receive the emails (or if they are blocked by their mail server), it is impossible to calculate a precise response rate. Most respondents (50.5%) were transplant surgeons, 22.9% were transplant physicians or advanced transplant professionals, and 18.3% were researchers or administrators (Table 1). Most (51.9%) had been working in the field of organ transplantation for 11–25 years, while 18.9% had been working in the field for ≤10 years and 29.2% for >25 years. Most respondents (69.9%) reported their transplant center’s annual volume (across all organ programs) to be >100 transplants, while 17.6% were 51–100 and 12.5% were ≤50 (Table 1). This is similar to the distribution of transplant centers by volume in the US; in 2016 50% of all US centers performed >100 transplants, 22% performed 51–100 transplants, and 38% performed ≤50 transplants (26). Participants generally viewed patient and graft survival to be effective measures of transplant quality; when participants scored patient survival and graft survival on a 10-point scale (10 meaning the outcome is a highly informative measure of quality of care), 1- and 3- year patient survival received mean scores of 7.44 and 7.02 respectively, and 1- and 3-year graft survival received mean scores of 7.33 and 6.90 respectively.

Table 1:

Respondent and Transplant Hospital Characteristics

Respondent and Transplant Hospital Characteristics	Percent	N
Years worked in field of organ transplantation
≤10	18.9	44
11–25	51.9	121
>25	29.2	68
Role
Surgeon	50.5	141
Physician/Advanced Transplant Professional	22.9	64
Researcher/Administrator	18.3	51
Other	8.2	23
Annual Transplant Volume (all organs)
≤50	12.5	35
51–100	17.6	49
>100	69.9	195
Scoring measures on effectiveness at capturing quality of care		Mean*
1-year patient survival		7.44
1-year graft survival		7.33
3-year patient survival		7.02
3-year graft survival		6.90

^*Measures were ranked on a 10-point scale with a score to 10 meaning the outcome is a highly informative measure of quality of care

Lowest SRTR Scores on the 5-tier Scale

Most respondents (n=203, 72.5%) reported SRTR scores for at least one transplant program. Of the 46 respondents reporting only one SRTR score, 67.4% reported a score for their kidney program. Twenty-five respondents (8.9%) reported scores for all five organ programs. 185 respondents reported a score for their hospital’s kidney program, 147 for their hospital’s liver program, 77 for their hospital’s pancreas program, 75 for their hospital’s heart program, and 55 for their hospital’s lung program. Overall, 5% of the scores reported in this survey were worse than expected, and 18% were somewhat worse than expected.

Nearly half of respondents (45.7%) reported that their transplant hospital’s lowest rating across all organ programs was worse or somewhat worse than expected. Of respondents, 11.8% reported that their hospital’s lowest score across all organ programs was 1, 33.9% reported their lowest score was 2, 36.5% reported their lowest score was 3, 12.8% reported their lowest score was 4, and 4.9% reported their lowest score was 5 (Figure 1). The proportion of respondents reporting scores of 1 (5%) or 2 (18%) in this survey were consistent with the proportion of centers receiving scores of 1 (5%) or 2 (21%) as reported by the SRTR (24).

Figure 1: Lowest reported SRTR scores on the 5-tier scale, among all organ programs.

Respondents reported their hospitals’ lowest SRTR score on the 5-tier scale for each transplant program. For the following analyses, we used each respondent’s lowest reported SRTR score across all organ programs. Most respondents (n=203, 72.5%) reported SRTR scores for at least one organ program. Nearly half of respondents (45.7%) reported that their center received a rating of worse or somewhat worse than expected in at least one organ program.

Reports of Negative Effects of the SRTR 5-tier rating system

Over half of respondents (n=137, 52.1%) reported at least one negative effect of the new 5-tier rating system (Figure 2). Increased concern among patients was reported by 61 respondents (23.2%) and 50 respondents (19.0%) reported increased concern among referring providers (Figure 3). Twenty-five respondents (9.5%) reported losing patients, and 28 (10.7%) reported losing insurers (Figure 3).

Figure 2: Reports of at least one negative and positive effect of the SRTR 5-tier rating system.

Survey respondents were given a list of four positive and four negative potential effects of the SRTR 5-tier system, and were asked if their transplant hospital experienced any of these effects. The negative effects listed in the survey were: lost insurers, lost patients, increased concern among patients and increased concern from referring providers; the positive effects listed were: gained insurers, gained patients, decreased concern among patients, and decreased concern from referring providers.

Over half of respondents (n=137, 52.1%) reported at least one negative effect of the 5-tier rating system, while only 35 respondents (13.7%) reported experiencing at least one positive effect of the 5-tier rating system.

Figure 3: Reports of negative effects of the SRTR 5-tier rating system.

Survey respondents were given a list of four positive and four negative potential effects of the SRTR 5-tier system, and were asked if their transplant hospital experienced any of these effects. Shown are the number of respondents who reported each of the negative potential effects: increased concern among patients, increased concern among referring providers, lost insurers and lost patients.

In addition to the four potential negative effects listed in the survey, 14 respondents said in a free-response item that the 5-tier rating system caused concern among hospital administration and leadership and 7 said the 5-tier system caused concern among their transplant providers. Other negative effects listed in the free-response were negative media attention (n=3), wasted time (n=2), and developing more risk averse practices (n=1).

Reports of Positive Effects of the SRTR 5-tier rating system

Only 35 respondents (13.7%) reported experiencing at least one positive effect of the 5-tier rating system (Figure 2). Three (1.2%) reported decreased concern among patients, 2 (1.2%) reported decreased concern among referring providers, 7 (2.7%) reported gaining patients, and 2 (0.8%) reported gaining insurers.

In addition to the four potential positive effects listed in the survey, 2 respondents said in a free-response item that they received praise from hospital administration and leadership for performance. Other positive effects included improved patient comprehension (n=2), positive media attention (n=1), and use of ratings in hospital marketing (n=1).

Association between reporting at least one negative effect and respondent and center characteristics

Having at least one program ranked as worse or somewhat worse than expected (1 or 2 on the 5-tier scale) was associated with markedly higher odds of reporting at least one negative effect of the five-tiered ratings (Odds Ratio [OR] _1.7 7.1 _28.9 p<0.01; OR _1.3 2.9 _6.2 p=<0.01 respectively; Table 2). A rating of 4 (somewhat better than expected) was associated with lower risk of a negative effect (OR _0.1 0.3 _1.0 p=0.051, Table 2). A rating of 5 (better than expected) was not associated with risk of negative effects (OR _0.10.7 _4.5 p=0.7, Table 2), however only 10 respondents reported their center’s lowest rating to be a 5. Respondent characteristics (role and years working in the field of transplantation) and annual all-organ transplant volume were not associated with risk of negative effects (p>0.1, Table 2).

Table 2:

Risk of at least one negative effect of the SRTR 5-tier system by transplant hospital and respondent characteristics

This table shows relationships between prevalence of at least one effect of the SRTR 5-tier system and respondent characteristics, annual transplant volume, and lowest SRTR score using logistic regression.

Characteristic	OR	p-value
Lowest Rating on SRTR 5-tier Scale (all organs)
1	1.7 7.1 28.9	0.006
2	1.3 2.9 6.2	0.007
3	[Reference]
4	0.1 0.3 1.0	0.051
5	0.10.7 4.5	0.7
Annual Transplant Volume (all organs)
≤50	0.10.4 1.4	0.2
51–100	0.20.5 1.2	0.1
>100	[Reference]
Respondent Role
Surgeon	[Reference]
Physician/Advanced Transplant Provider	0.3 0.8 2.2	0.6
Researcher/Administrator	0.5 1.2 2.8	0.7
Other	0.41.6 6.3	0.5
Years in Transplantation
≤10 years	0.51.2 3.0	0.7
11–25 years	[Reference]
>25 years	0.81.8 4.1	0.2

SRTR rating was associated with the four specific negative effects listed in the survey. Compared to transplant hospitals with a lowest SRTR score of 3 or 4, centers with an SRTR score of 1 or 2 were more likely to report experiencing increased concern among patients (37.6% vs. 8.8%, p=0.002), increased concern among referring providers (31.2% vs 5.9%, p=.006), lost insurers (18.3% vs. 2.9%, p=.02) and lost patients (17.2% vs. 0.0%, p=.01, Table 3).

Table 3:

Percentage of respondents reporting effects of SRTR 5-tier system by lowest reported score among all organ programs at their hospital

This table shows relationships between individual negative effects of the 5-tier system and respondent characteristics, annual transplant volume, and lowest SRTR score using Chi-squared and Fisher’s exact tests.

Effect	Lowest score among all organ programs			P- value
	1–2 (%)	3 (%)	4–5 (%)
Negative Effects
Increased concern among   patients	37.6	21.1	8.8	0.002
Increased concern among   referring providers	31.2	19.7	5.9	0.006
Lost insurers	18.3	7.0	2.9	0.02
Lost Patients	17.2	9.9	0.0	0.01
Positive Effects
Decreased concern among   patients	1.1	0.0	6.1	0.1
Decreased concern among   referring providers	1.1	0	6.1	0.1
Gain insurers	0	0	0	N/A
Gain patients	3.3	0	6.1	0.1

SRTR rating was not associated with the positive effects listed in the survey. (p≥0.1, Table 3), and no respondent who reported their SRTR ratings also reported gaining insurers (Table 3). As an alternative to lowest reported score among all programs we also conducted the same analyses using only reported scores for kidney programs, the most common and largest programs in most US transplant centers. The findings were similar (p<0.03 for all negative effects except lost insurers, for which p=0.069).

DISCUSSION

In this national survey of transplant providers and professionals, 52.1% reported that their center experienced at least one negative effect of the new SRTR 5-tier system (including lost patients, lost insurers, increased concern among patients, and increased concern from referring providers), and only 13.7% reported at least one positive effect (including gained insurers, gained patients, decreased concern among patients, and decreased concern from referring providers). Transplant hospitals with lower scores on the SRTR 5-tier scale were more likely to report negative effects, and those with higher scores were more likely to report positive effects. Although the 5-tier ratings were only used as the primary source of transplant hospital ratings for a short period, the change provoked an immediate response that may have long-term implications for transplant hospitals and patients.

While it is often assumed that patients will move from poor quality hospitals to higher quality facilities, patients may be unable or unwilling to make these switches (27–32). For example, when Medicare bariatric surgery patients were required to use a subset of hospitals designated as Centers of Excellence, rates of surgery among minority patients dropped by 17% (20). Bariatric surgery is similar to transplantation as both procedures require a number of pre and post-operative visits, making it especially likely that loss of a convenient facility could motivate a transplant candidate to de-list rather than move to another center if his insurer dropped his waitlist center.

Our findings should be interpreted in light of several limitations. This survey was distributed to listservs of ASTS and AST. Because participation was anonymous, we are unable to confirm that our survey is nationally representative. However, we obtained responses from professionals at centers with considerable variation in annual volume and number of organ programs (Table 1). Furthermore, an N of 280 is consistent with sample sizes of other recently published surveys of the transplant community (33–37). While surveys always have the potential for voluntary response bias, participants in this study were not informed that the survey would contain questions about the SRTR 5-tier system before agreeing to participate, so it is unlikely that specific experiences with the 5-tier system motivated the decision to participate. Although the majority of respondents indicated concerns with the 5-tier system, they generally supported use of risk-adjusted, post-transplant patient and graft survival as an effective measure of center quality; on a scale of 1–10 with 10 meaning the outcome is highly descriptive of quality of care, participants rated 1-year patient and graft survival rates 7.44 and 7.31 respectively. Lastly, this study asked respondents about their experiences following the change from a three- to five-tier system based on 1-year post-transplant outcomes, and may not completely generalize to other rating systems.

In this study, the center’s lowest score among all organ programs was used to assess effects of the 5-tier system. However, the impacts of a low score in one program may not affect all programs at that hospital. Our findings should be interpreted as lower bound estimates of the positive and negative consequences of the 5-tier ratings, since respondents who work exclusively in one organ program may not be familiar with the response in other organ programs. Furthermore, the proportion of respondents reporting scores of 1 (5%) or 2 (18%) in this survey were consistent with the proportion of centers receiving scores of 1 (5%) or 2 (21%) as reported by the SRTR (24). This further suggests that this study’s findings were not biased by a disproportionate number of responses by participants from centers with low ratings.

The transplant community’s report of a rapid and pronounced patient and payer response to the new ratings suggests that changes to quality monitoring and reporting systems play an important role in determining where and whether at least some patients receive transplants. In the short term, there were adverse effects of being newly classified as a “somewhat worse than expected” or “worse than expected” program. Combined with recent studies in the US documenting changes in transplant center practice patterns following payer flagging in response to poor performance on SRTR report cards, (8, 15, 38, 39) our survey results highlight the potential of report cards in determining transplant center, patient, and provider behavior. Because of the timing of our survey and the quick retreat from using the 5-star ratings, we are only able to study the short-term consequences of changing quality metrics, which may or may not be representative of the longer-term effects that would result from continued use. These results should be considered by countries discussing the implementation of similar systems of quality monitoring for regulation and public reporting so that adequate plans for short-term market disruptions can be considered and potential long-term implications can be evaluated.

Abbreviations:

AST

American Society of Transplantation

ASTS

American Society of Transplant Surgeons

OPTN

Organ Procurement and Transplantation Network

OR

Odds Ratio

SRTR

Scientific Registry of Transplant Recipients