Don’t throw your heart away: Increased transparency of donor utilization practices in transplant center report cards alters how center performance is evaluated

Abstract

Background.

Publicly available report cards for transplant centers emphasize post-transplant survival and obscure the fact that some centers reject many of the donor organs they are offered (reflecting a conservative donor acceptance strategy), while others accept a broader range of donor offers (reflecting an open donor acceptance strategy).

Objective.

We assessed how the provision of salient information about donor acceptance practices and waitlist survival rates impacted evaluation judgments of hospital report cards given by lay people and medical trainees.

Methods.

We tested five different report card formats across four online randomized experiments (n1=1003, n2=105, n3=123, n4=807) in the same hypothetical decision. The primary outcome variable was a binary choice between transplant hospitals (one with an open donor acceptance strategy and the other with a conservative donor acceptance strategy).

Results.

Report cards featuring salient information about donor organ utilization rates (transplant outcomes categorized by quality of donor offers accepted), or overall survival rates (outcomes from both waitlist and transplanted patients) led lay participants (Studies 1, 3, and 4) and medical trainees (Study 2) to evaluate transplant centers with open donor acceptance strategies more favorably than centers with conservative strategies.

Limitations.

Due to the nature of the decision, a hypothetical scenario was necessary for both ethical and practical reasons. Results may not generalize to transplant clinicians or patients faced with the decision of where to join the transplant waitlist.

Conclusions.

These findings suggest that performance evaluations for transplant centers may vary significantly based not only on what outcome information is presented in report cards, but also how the information is displayed.

Introduction

Organ transplantation necessitates numerous difficult decisions. Families must decide whether to donate organs from a recently deceased loved one. Patients in need of a transplant must decide at which transplant center they should be listed. Transplant physicians must decide whether to accept or decline an available transplant organ for a listed patient. These decisions are interconnected. For example, a patient’s choice about where to join the waitlist could be influenced by information about the likelihood of survival following a transplant and about how long they can expect to wait before the center accepts an organ offer for them. Under current Organ Procurement and Transplantation Network (OPTN) policy, patients’ survival outcomes after transplant represent the primary focus of mandatory performance evaluations that are necessary for transplant centers to maintain federal certification^1–3.

Previous literature speculates that because federal regulatory agencies primarily emphasize post-transplant morbidity and mortality when auditing program performance, transplant teams have an incentive to decline donor heart offers of marginal (meaning less-than-optimal) quality^3–6. Holding programs accountable for post-transplant patient survival may discourage the efficient allocation of a scarce and valuable resource and deter transplant teams from optimizing the chances that patients will survive to transplant at all. Although donor shortages present a critical challenge for patients in every age and organ category, this research is motivated specifically by the problem of donor organ discard in the pediatric heart transplant setting, as 44% of hearts offered to pediatric patients are ultimately discarded and another 30% are declined by all pediatric teams and allocated to adult recipients^4,7,8.

The purpose of the current study is to examine how the presentation of outcome information in report cards affects transplant center evaluations by lay people and medical trainees. Specifically, we find that some information presentation formats cause respondents to prefer a hypothetical transplant center that declines many organs over one that uses all organs offered. This finding may have implications for real patients in need of transplant organs, many of whom will turn to the publicly available report cards located on the Scientific Registry of Transplant Recipients (SRTR) website^9,10. SRTR reports have recently been updated to display not only post-transplant survival but also how transplant wait times and organ offer acceptance practices vary across programs. The current research explores how the addition of such information affects evaluations of centers.

Simpson’s paradox

Consider two hypothetical transplant centers shown in Figure 1. This hypothetical scenario leverages the classic Simpson’s Paradox, in which a trend appears in several different “strata” or groups of data, but reverses when the groups are combined^11,12. When post-transplant survival rates are stratified into two groups by quality of donor, the open and conservative centers are evenly matched on survival rates. However, when post-transplant outcomes for all donor types are combined into a single metric, the conservative center clearly presents a superior survival rate. Upon observing waitlist survival rates, a similar pattern occurs: the waitlist and stratified transplant survival rates are exactly the same across the two centers, but after combining these outcomes into a single overall metric, the open center displays a higher overall survival rate. The confounding variable in this equation is donor utilization. In our scenario, both centers receive the same number of “excellent” and “marginal” donor offers, but the conservative center boosts its overall transplant success rate by accepting many excellent but few marginal donor hearts. This selectivity results in more patients remaining on the wait list, and thus decreases overall survival, because patients who receive marginal organs have better outcomes than for those who remain on the wait list in this scenario. In contrast, the open strategy center transplants more patients by accepting all marginal organs, which decreases survival after transplant but improves overall survival computed across all listed patients.

Figure 1.

Performance of two hypothetical transplant centers: Center A (uses an open donor acceptance strategy) and Center B (uses a conservative donor acceptance strategy). Each center has 90 listed patients at the start of the year and receives 54 donor offers throughout the year. The two centers are identical in terms of the number of excellent and marginal donor organs offered, the survival rate following transplant with an excellent organ, transplant survival with a marginal organ, and wait list survival. The centers differ only in the number of marginal donor offers that they accept: Center A follows a less conservative, “open” donor heart acceptance strategy in that it accepts all 30 excellent hearts and all 24 marginal hearts. In contrast, Center B adheres to a more conservative donor acceptance strategy in that it accepts all 30 excellent hearts but only 6 of the 24 available marginal hearts. Thus, Center A is less selective in accepting donor organs than Center B.

The centers can be evaluated under several different report cards that vary on the information that is provided, but not the outcomes each center achieves. Report Card 1 exemplifies the standard report card scheme where only 1-year overall transplant survival (percentage of survivals out of all patients who were transplanted) is provided. Since donors varying in quality are all combined into the same transplant survival metric, Report Card 1 makes Center B look better than Center A. In contrast, Report Card 3 shows a 1-year transplant survival metric that is stratified by donor risk level. This presentation reveals that Center A performs the same as Center B with both excellent and marginal organs – that is, recipients at the two centers have equivalent survival rates for a given category of donor organ quality but Center A accepted and transplanted more marginal organs. Report Cards 2 and 4 incorporate the information that waitlist survival is 33% at both centers – report card 2 highlights the tradeoff between transplant survival and number of patients who remain on the wait list, while report card 4 shows both the stratified transplant survival and overall survival outcomes. Report Card 5 shows overall survival rates across all patients in each center (combining those transplanted with those on the wait list), resulting in a better score for Center A. Appendix 1 provides a detailed comparison of the report cards discussed here and demonstrates how each survival metric within the report card is computed. Although all five report cards are based on the same table of information, we predict that preference for Center A will be strongest with report cards 3, 4, and 5, weakest with report card 1, and intermediate with report card 2.

Research on practice variation in transplant centers’ donor acceptance strategies provides some clinical support for assumptions underlying our scenario. In fact, a recent analysis demonstrated that when a center-wide non-selective donor acceptance strategy was implemented, the lower donor offer refusal rates were associated with lower waitlist times but no difference in post-transplant outcomes^8,13. Thus, whereas our hypothetical scenario makes the conservative assumption that marginal organs lead to worse outcomes than excellent organs do, available evidence suggests that marginal and excellent organs may lead to equivalent outcomes.

Numerous studies in behavioral economics and decision psychology have demonstrated that framing information differently can lead to different responses^14–16. Prior research in the medical domain has demonstrated that the Simpson’s Paradox cannot be avoided by using risk-adjusted outcomes measures^17,18. Recent research on the topic of transplant report cards has examined how the components recently added to the existing SRTR performance reports influence transplant center decisions and the outcomes that patients consider most relevant in choosing a transplant center^19,20. Our study is the first, however, to examine systematically how evaluation of transplant centers is affected by including vs. excluding information about donor utilization and survival from listing.

The four studies presented here make use of the five report card conditions shown in Figure 1, with condition 1 as the control. This design affords the orthogonal manipulation of two factors: (i) whether the report cards present outcomes are stratified by the quality of donor organs the center receives and (ii) whether the report cards display overall survival rates, rather than only transplant survival rates. We hypothesize that (H1a) each of these factors will increase evaluation of open strategy centers compared to conservative centers. We also predicted (H1b) a subadditive 2×2 interaction such that preference for the open strategy center will be similar among participants who viewed both stratified transplant survival and overall survival metrics together (condition 4) compared to conditions 2 and 3, but higher relative to control participants (condition 1). In addition, we predicted (H2) that preference would be similar in condition 5 (overall survival metric only) and condition 4 (both the stratified transplant survival and overall survival). Finally, we predicted (H3) a mediation such that providing information about overall survival and stratified transplant survival metrics would each lead participants to rate the chances of “receiving any type of heart” to be more important and that this focus on getting a heart would in turn lead to a higher preference for the center that accepted more hearts.

Study 1 included all five conditions shown in Figure 1 while Study 2 included only conditions 1 and 3. Study 3 was designed to exactly replicate Study 2 using a sample of laypeople, affording a comparison between lay participants and medical trainees. Study 4 was designed as a conceptual replication of Study 1 featuring report card presentations that better represented the actual graphics used on the SRTR public reporting website.

Methods

After IRB approval, studies 1, 3, and 4 recruited lay participants from Amazon Mechanical Turk (AMT). Study 2 recruited medical students with permission from the University of Pittsburgh Medical School Research on Medical Students Committee. Protocols for Studies 1 and 2 were pre-registered at Clinicaltrials.gov (NCT0413381 and NCT04176796). All study protocols were pre-registered on Open Science Framework (Study 1: https://osf.io/agzfk/?view_only=f4ce82ae50754fc58096ca57db5c26dd; Study 2: https://osf.io/8b4pd/?view_only=64845d11ab7c4945b72e7326a982a8e0; Study 3: https://osf.io/ybfhg/?view_only=769a284c41094dc2b401aa2e89120f74; Study 4: https://osf.io/zevcp/?view_only=db7232d2dc64445f8f75cce251b826a7).

Participants and Procedures

Participants in Studies 1, 3, and 4 entered an online survey and were given a brief introduction to the donor organ allocation process that included an explanation of how transplant teams make the decision to accept or reject an organ offer (see Appendix 2). A hypothetical choice scenario featured a pair of transplant center report cards with outcome metrics corresponding to the experimental condition. Participants chose the better center and then explained their choice in a free-response text box. Participants used a 0 to 100 slider scale to rate three rationales in terms of their influence on their choice: “patients were more likely to receive an excellent donor heart at the hospital I picked”; “patients were less likely to receive a marginal donor heart at the hospital I picked”; “patients were more likely to receive any kind of donor heart at the hospital I picked”. The first and second of these items were designed to provide context for the third item which was our hypothesized mediator. Finally, participants answered demographic questions and received $1.33 in pay, plus $0.10 for correctly answering each of two attention-check questions placed throughout the survey.

In Study 2 medical trainees answered the same two attention check questions as in Study 1 but did not earn bonus pay for correct responses. All participants were entered into a raffle for multiple gift card prizes. Whereas Studies 1 – 3 used infographic stimuli, Study 4 used an information presentation format adapted from the SRTR website (Figure 2).

Figure 2.

Comparison of infographic stimuli presented in Studies 1–3 (image A) and SRTR stimuli presented in Study 4 (image B). The stimuli in the figure represent the pair of outcome tables in Condition 3 (stratified transplant survival only), with survival metrics as depicted in the infographic (A) and SRTR (B) formats. While the stimuli in Studies 1–3 featured fractions and percents to convey center survival rates, Study 4 used the SRTR quintile comparison graphics assigned to each center based on relative performance. In addition, conditions 2 and 4 featured an attention cue next to the “Getting a Transplant Faster” metric. This cue is displayed on performance reports featured on the SRTR website to inform the viewer that the “Time to Transplant” metric has the largest impact on survival after listing.

Statistical Analyses

All statistical analyses were conducted using R (version 4.0.2)²¹. In keeping with the study preregistrations, the results presented for Studies 1–3 include all participants whether or not they missed any attention check questions, but participants in Study 4 were included only if they passed both attention checks.1 Appendix 3 includes descriptive statistics of the responses to the attention check questions.

To test Hypothesis 1, we conducted a binomial logistic regression with the GLM software package²². The outcome variable was choice of transplant center (open strategy center versus conservative strategy center). The regression included tests for both the main effects and the interaction effect in our 2 (combined transplant vs. stratified transplant) × 2 (overall survival vs. no overall) design. To test Hypothesis 3, we conducted a causal mediation analysis^23,24 (see Figure 3).

Figure 3.

Hypothesized mediation models. We hypothesized that report cards providing information about overall survival (top) or stratified transplant survival (bottom) would lead to a higher preference for the open strategy center (H1), compared to a standard report card not providing this information. Further, we predicted that higher rated importance on the item “receiving any type of donor heart” as a rationale for choice of transplant center would mediate the effect of report card condition on choice of center (H3).

We included additional analyses to examine potential differences in effect sizes between study populations. In order to test whether the size of the effect of the stratified treatment condition differed between lay participants vs. medical trainees, we combined the data sets from Studies 2 and 3 in a pre-registered 2 (condition: stratified vs. control) × 2 (population: lay vs medical trainee) logistic regression analysis. In order to test the prediction that the pattern of results would be similar in Studies 1 and 4, we combined these two data sets in a 2 (transplant: stratified vs. combined) × 2 (overall: included vs. not included) × 2 (stimuli: infographic vs. SRTR) logistic regression analysis.

Results

Participant characteristics for all studies are shown in Appendix 4.

Study 1 (N=1003).

Figure 4 shows the proportion of participants selecting the open strategy center across the report card conditions. In the control condition only 36% of participants chose the center with the open donor-acceptance strategy whereas in all four of the other display conditions a majority chose it. The logistic regression analysis of the first four conditions revealed two statistically significant main effects (Table 1). First, the proportion of participants preferring the open strategy center was significantly higher in the two conditions that showed overall survival metrics than in the two conditions that did not (β=1.20, P < 0.001). Second, the proportion of participants who preferred the open strategy center was higher in the two conditions that displayed stratified transplant survival than in the two conditions that did not (β= 2.34, P < 0.001). However, the predicted interaction was not statistically significant (β = −0.53, P = 0.175). In a separate analysis, participants in condition 5 who viewed the no transplant survival + overall survival metric had significantly higher preference for the open strategy center relative to control participants (β = 3.16, P < 0.001). Further, choice of center was not significantly different between conditions 4 and 5 (P = 0.993).

Figure 4.

Choice of transplant center by report card condition – Studies 1 and 4. The bar graph displays the proportion of study participants in each report card condition who preferred the transplant center with the open donor acceptance strategy. Error bars represent 95% confidence intervals.

Table 1.

Effect of Report Card Condition on Choice of Transplant Center – Study 1.

	Choice of Open Strategy Transplant Center
	Model 1A			Model 1B			Model 1C
	β	SE	P-value	β	SE	P-value	β	SE	P-value
Predictor Variables
Intercept	−0.51	0.14	<0.001	−0.59	0.16	<0.001	−0.37	0.27	0.17
Overall Survival (1= displayed, 0=not displayed)	1.04	0.18	<0.001	1.20	0.22	<0.001	1.18	0.22	<0.001
Transplant Survival (1=stratified, 0=combined)	2.12	0.20	<0.001	2.34	0.26	<0.001	2.30	0.26	<0.001
Overall × Transplant				−0.53	0.39	0.175	−0.47	0.40	0.24
Demographic Variables
Female Gender							−0.02	0.19	0.9
Age > 40 years							0.02	0.18	0.94
Non-White Race							−0.14	0.22	0.53
College Degree							−0.25	0.19	0.2
Model Statistics
Observations	765			765			753
Log Likelihood	−379.269			−378.364			−370.407

Models 1A, 1B, and 1C include conditions 1–4 (n = 765). Intercept corresponds as to the control condition in which only the combined transplant survival metric was featured on the report card.

Results of the causal mediation analysis are shown in Table 2. Higher rated importance on the item “chances of receiving any type of donor heart” was positively associated with the likelihood of choosing the open strategy center (β = 0.027, SE = 0.001, z = 14.54, P < 0.001). The overall survival metric marginally increased rated importance of receiving any type of donor heart (Appendix 5), generating a marginally significant indirect effect of the rated importance of receiving any heart on center choice (ACME = 0.02, 95% CI = [0.00 – 0.05]). The stratified transplant survival metric significantly increased rated importance of the chances of receiving a heart (Appendix 5). The mediation analysis of Study 1 showed a significant indirect effect of the importance of receiving any heart (ACME = 0.11, 95% CI = [0.08–0.15]).

Table 2.

Causal mediation analysis – Studies 1 and 4.

				Study 1 (n=765)		Study 4 (n= 577)
Predictor	Mediator	Outcome		Estimate [95% CI]	P-value	Estimate [95% CI]	P-value
Overall metric	→ Rationale	→ Choice	ACME (Indirect Effect)	0.024 [0.00–0.05]	0.056	0.067 [0.03–0.11]	<0.001
			Average Direct Effect	0.144 [0.09–0.20]	<0.001	0.108 [0.05–0.17]	<0.001
			Total Effect	0.170 [0.11–0.23]	<0.001	0.175 [0.10–0.25]	<0.001
			Proportion Mediated	0.144 [0.00–0.29]	0.056	0.380 [0.21–0.62]	<0.001
Stratified metric	→ Rationale	→ Choice	ACME (Indirect Effect)	0.114 [0.08 – 0.15]	< 0.001	0.249 [0.21–0.32]	<0.001
			Average Direct Effect	0.267 [0.21 – 0.32]	< 0.001	0.282 [0.23–0.37]	<0.001
			Total Effect	0.381 [0.32 – 0.44]	< 0.001	0.531 [0.48–0.63]	<0.001
			Proportion Mediated	0.299 [0.23 – 0.39]	< 0.001	0.469 [0.39–0.56]	<0.001

We used the procedure reported by Imai, Keele, & Tingley (2010) and the mediation software package². This method generates 95% confidence intervals from 1000 bootstrap simulations to yield unstandardized point estimates of the average causal mediation effect (ACME), average direct effect (ADE), and total effect for each independent variable in the model. The ACME represents the indirect effect, or the effect of each condition on the outcome of hospital choice, mediated through the hypothesized mediator. Meanwhile, the ADE describes the remaining effect of information display on hospital choice (e.g. the portion not mediated through the hypothesized mediator). Together, the ACME and the ADE make up the entire effect of the information display on hospital choice, which is the total effect.

Study 2 (N=105).

In the control condition, 59% of participants chose the open strategy center compared to 96% in the stratified transplant survival condition (β = 3.26; P < 0.001, Figure 5 and Table 3). In the causal mediation analysis medical trainees who viewed the stratified metric rated the importance of “receiving any type of donor heart” significantly higher than those who viewed the control metric (Appendix 5) and in turn, higher rated importance on the item “receiving any type of donor heart” was positively associated with likelihood of choosing the open strategy center (β = 0.069, SE = 0.014, z = 4.88, P < 0.001), yielding a significant indirect effect (ACME = 0.09; 95% CI = [0.02–0.18], P = 0.01) (Table 4).

Figure 5.

Choice of transplant center by report card condition – Studies 2 and 3. The bar graph displays the proportion of medical trainees and lay participants in each report card condition who preferred the transplant center with the open donor acceptance strategy. Error bars represent 95% confidence intervals. Performance of two hypothetical transplant centers: Center A (open donor acceptance strategy) and Center B (conservative donor acceptance strategy).

Table 3.

Effect of Report Card Condition on Choice of Transplant Center – Studies 2 and 3.

	Choice of Open Strategy Transplant Center
	Model 2A			Model 3A			Model 3B
	β	SE	P-value	β	SE	P-value	β	SE	P-value
Predictor Variables
Intercept	−1.00	0.70	0.153	−0.53	0.56	0.35	−0.25	0.40	0.53
Transplant Survival (1=stratified, 0=combined)	3.26	0.84	<0.001	2.18	0.49	<0.001	2.83	0.78	<0.001
Sample (1=AMT participants, 0=medical trainees)	-	-	-	-	-	-	−0.64	0.45	0.154
Transplant × Sample	-	-	-	-	-	-	−0.72	0.92	0.44
Demographic Variables
Female Gender	1.18	0.58	0.043	0.42	0.43	0.33	0.77	0.42	0.024
Age > 30 years	2.18	1.14	0.056	−0.17	0.53	0.74	0.46	0.41	0.27
White Race	0.01	0.59	0.988	0.43	0.48	0.37	0.06	0.36	0.85
< 2 years Medical Training	0.82	0.62	0.187	-	-	-	-	-	-
Model Statistics
Observations	103			122			226
Log Likelihood	−40.213			−66.207			−114.255

Model 3B includes observations from both conditions in Study 2 (n=103) and Study 3 (n=123).

Table 4.

Causal mediation analysis – Studies 2 and 3.

				Study 2 (n=105)		Study 3 (n=123)
Predictor	Mediator	Outcome		Estimate [95% CI]	P-value	Estimate [95% CI]	P-value
Stratified metric	→ Rationale	→ Choice	ACME (Indirect Effect)	0.094 [0.02 – 0.18]	0.01	0.167 [0.07 – 0.26]	< 0.001
			Average Direct Effect	0.317 [0.16 – 0.46]	0.004	0.226 [0.08 – 0.37]	< 0.001
			Total Effect	0.411 [0.25 – 0.56]	0.002	0.393 [0.24 – 0.55]	< 0.001
			Proportion Mediated	0.227 [0.06 – 0.45]	0.01	0.421 [0.22 – 0.71]	< 0.001

ACME = Average Causal Mediation Effect. 95% confidence intervals generated from 1000 bootstrap simulations. The procedure for the mediation analysis is described in Table 2.

Study 3 (N=123).

Choice of the open center was 87% among participants who viewed the stratified transplant survival metric, compared to 48% in the control condition (β = 2.04; P < 0.001, see Table 3), similar to Study 2. Although the effect size among Study 2 medical students appears slightly larger than that among Study 3 lay participants (Figure 5), the interaction between condition and study population was not statistically significant in our combined analysis of the two studies (β = −0.75; P = 0.41, Table 3). As in Study 2, Study 3 showed a significant indirect effect of providing participants with the stratified transplant survival metric (ACME = 0.17; 95% CI = [0.07–0.26], P < 0.001) on choice of center via the rated importance of our predicted mediator variable (Table 4).

Study 4 (N=807).

Analyses included the 577 participants who met our pre-registered criteria of responding correctly to both attention check questions. In the control condition, only 23.5% of participants chose the open strategy center in comparison to a majority in all three of the other display conditions. The 2×2 logistic regression analysis (Table 5) revealed significant main effects of overall survival metric (β = −1.54, P < 0.001) and of stratified transplant survival (β = 2.35, P < 0.001). Unlike in The interaction of these two factors was opposite to the predicted direction, although it was only marginal (β = 0.98, P = 0.053). We predicted that the pattern of results would be similar in Studies 1 and 4, indicating that the format of the report card stimuli (infographic vs. SRTR) produce similar effects on choice (Figure 4). Although directionally the effect sizes were larger in Study 4 than in Study 1, the interactions testing these comparisons were only marginal (Table 5).

Table 5.

Effect of Report Card Condition on Choice of Transplant Center – Study 4.

	Choice of Open Strategy Transplant Center
	Model 4A			Model 4B			Model 4C
	β	SE	P-value	β	SE	P-value	β	SE	P-value
Predictor Variables
Intercept	−1.22	0.33	<0.001	−1.38	0.43	0.002	−0.51	0.14	<0.001
Overall Survival (1=displayed, 0=not displayed)	0.79	0.268	0.003	0.83	0.28	0.003	1.04	0.18	<0.001
Transplant Survival (1=stratified, 0=combined)	2.35	0.27	<0.001	2.36	0.28	<0.001	2.12	0.19	<0.001
Sample (1=Study 4, 0=Study 1)	-	-	-	-	-	-	−0.84	0.23	<0.001
Overall × Transplant	0.98	0.51	0.053	0.96	0.51	0.060	-	-	-
Sample × Overall	-	-	-	-	-	-	0.06	0.28	0.83
Sample × Transplant	-	-	-	-	-	-	0.56	0.30	0.057
Demographic Variables
Female Gender	−0.33	0.21	0.123	−0.33	0.21	0.119	-	-	-
Age > 40 years	0.26	0.28	0.36	0.25	0.29	0.38	-	-	-
Non-white Race	0.04	0.25	0.86	0.05	0.25	0.84	-	-	-
Correctly Recall Table Metrics	-	-	-	0.17	0.30	0.58	-	-	-
Model Statistics
Observations	565			565			1342
Log Likelihood	−276.559			−276.406			−664.82

Model 4C includes observations from conditions 1–4 in Study 1 (n=765) and Study 4 (n=577).

In the Study 4 causal mediation analysis, higher rated importance on the item “chances of receiving any type of donor heart” was positively associated with choosing the open strategy center (β = 0.04, S E= 0.002, z = 15.75, P < 0.001). Both the overall survival and stratified transplant survival metrics increased rated importance of receiving any type of donor heart relative to no overall survival metric (Appendix 5). Providing an overall survival metric yielded a significant indirect effect of the mediator on center choice (ACME=0.18, 95% CI = [0.10–0.25]), which accounted for 38% of the total effect of the overall metric on center choice (Table 2). In addition, the stratified survival metric increased rated importance of receiving any type of donor heart, yielding a significant indirect effect (ACME = 0.28, 95% CI = [0.23–0.26], P < 0.001), which accounted for almost 47% of the total effect of the stratified metric on center choice (Table 2).

Discussion

Four studies demonstrated that evaluation of transplant centers is affected by how performance information is presented. First, we found that when report cards displayed a transplant survival metric stratified by donor risk status, both lay participants and medical trainees favored the transplant center with high organ acceptance rates more strongly than they did when the report card displayed only the standard 1-year post-transplant survival metric. The comparison between Studies 2 and 3 provides evidence that the effect of the stratified vs. combined information presentation is robust across lay evaluators and those with relevant knowledge in the medical domain. Further, the comparison between Studies 1 and 4 indicates that the effect of the stratified treatment condition was observed even when participants were exposed to outcome metrics in the same format as the data available on the SRTR public reporting website.

Second, our findings demonstrated that displaying an overall survival metric increased preference for the open strategy center. When faced with a display highlighting the superior transplant survival at the conservative center, and the superior total survival at the open center, participants were more likely to choose the open center than they were when total survival was not presented. Notably, condition 5 in Study 1, which presented only overall survival, resulted in the highest proportion of lay participants (93%) choosing the open center. This finding is noteworthy because the overall survival metric highlights patient survival rates from the time of listing, which is in contrast to the traditional post-transplant survival metric that instead displays the survival rates of only patients who have received transplants. As long as the rate of transplant survival exceeds that of waitlist survival, then a center’s overall survival rate will benefit from a higher transplant rate, or in other words, from moving patients off the waitlist more quickly (e.g., by transplanting more patients per unit of time). For instance, a recent analysis of organ offer data suggests that a higher rate of donor offer acceptance in heart and lung transplant programs is strongly associated with a lower incidence of both waitlist mortality and waitlist removal, via increased transplant rate²⁵ While it is possible that a fraction of the marginal hearts offered to transplant teams are of such poor quality that their use would put patients at risk of worse outcomes from transplant than from waiting for a later subsequent offer, evidence from the transplantation literature generally opposes this idea^4,26.

Third, the mediation analysis indicated that the effect of presenting the stratified transplant survival metric on hospital choice acted by way of the perceived importance of the chances of receiving any donor heart. In all four studies, the stratified metric was positively correlated with the ratings of importance for the mediator item “chances of receiving a heart”, which was in turn associated with judgments of the open center as higher performing. Although the effect of presenting the overall survival metric showed only a marginally significant indirect effect of the hypothesized mediator in Study 1, in Study 4, the indirect effect of the mediator was indeed statistically significant when the SRTR versions of the overall survival metrics were presented.

This collection of studies brings to light the way in which transplant center evaluation is affected by the confounding role of donor utilization in patient outcomes. We demonstrate how centers that focus on accepting more donor organs and transplanting more patients can, under certain information presentation formats, be evaluated more negatively than selective centers who decline many organs because the evaluators are subject to a classic Simpson’s Paradox. The current results highlight the importance of recent changes to SRTR report cards that now include information on donor utilization and time to transplant. Study 4 indicates that such information is critical for allowing patients to identify the center with the highest survival from listing.

Although the current studies focused on how report card format influences evaluation of transplant centers by lay people and medical trainees, future studies can investigate whether report card format also influences evaluation by regulators in such a way that the behavior of transplant centers changes in order to gain higher evaluations. The report metrics currently used by regulators may incentivize transplant teams to ensure that only patients with the best chance of an optimal outcome are offered transplants. Such a strategy would entail avoiding marginal organs at the expense of waitlist patient outcomes, because neither donor organ utilization nor waitlist outcomes are displayed or evaluated in performance reports. When those outcomes are displayed, evaluators judge conservative centers more harshly, suggesting that use report cards that display donor utilization and waitlist outcomes might incentivize transplant team to accept more donors.

Limitations

The current results should be considered in the context of several limitations. We used a simplified scenario that did not include the dynamics of a listed patient receiving subsequent offers after declining an initial offer. Our sample demographics were somewhat biased towards white, younger-age participants of a higher educational status relative to transplant candidates and recipients in the US population²⁷. Although our study did not assess literacy or numeracy through validated measures, the report cards in our study did adhere to several evidence-based methods for communicating probabilistic information, such as using frequencies and pictographs^28,29. The comparison of Studies 1 and 4 indicate that our results are not specific to any one type of visual display. Further, the introduction text was oriented towards explaining the donor evaluation process in the context of heart transplant exclusively. It is unknown how participants may respond to a scenario that instead uses kidney or liver evaluation as a frame of reference. Given that all solid organ transplantation faces the problem of donor organ discard, it may be worthwhile to study alternative versions of transplant center report cards in the context of other organs as well.

Our scenario is based on the assumption that patients commonly have a choice about which transplant center they will list at, but this choice does not apply to every patient. Whereas a range of transplant center options may be possible for the patients with high SES, flexible travel ability, or residences in major metropolitan areas, other patients will be limited to only one or two nearby options. Even when many patients do not have a choice of transplant center, report card format may nevertheless affect the behavior of transplant centers.

Conclusions

In conclusion, the current studies demonstrated the impact of a stratified donor risk classification scheme and an overall survival metric on evaluations of transplant center performance. Lay participants and medical trainees viewed transplant centers with open donor utilization strategies more favorably when performance reports displayed overall patient survival or transplant survival stratified by donor quality. Public report cards for transplant centers that emphasize these outcome metrics offer a potential method to increase the acceptance rate for marginal donors and decrease waitlist time and mortality for transplant candidates, in addition to providing more transparent information on donor utilization practices for future transplant patients who are faced with the decision of where to list for transplant.

Appendix 1. Calculation of survival metrics in report cards.

Appendix Table 1.

Comparison of performance report conditions and calculation of survival metrics.

		Survival Metrics Displayed in Outcome Table
Outcome	Label	#1 Combined transplant†	#2 Combined transplant† + Overall°	#3 Stratified transplant‡	#4 Stratified transplant‡ + Overall°	#5 Overall°
Patients on waitlist, start of year	P initial	Yes	Yes	Yes	Yes	Yes
Donor offers received	D received	Yes	Yes	Yes	Yes	Yes
Donor offers accepted	D accept *	Yes	Yes	Yes, stratified*	Yes, stratified*	Yes
Patients remaining on waitlist	W remain	No	Yes	No	Yes	Yes
Patients alive post-transplant, end of year	T alive **	Yes	Yes	Yes, stratified**	Yes, stratified**	No
Patients alive on waitlist, end of year	W alive	No	Yes	No	Yes	No
Patients alive overall, end of year	P alive	No	Yes	No	Yes	Yes

Patients with no donor offer received or accepted remain on waitlist: W_remain = P_initial − D_accept^*

Overall patient survival is a combination of post-transplant and waitlist survival: P_alive = T_alive + W_alive

^*D_accept is stratified by donor quality: (i) excellent = D_accept^excellent ; (ii) marginal = D_accept^marginal

^**T_alive is stratified by donor quality: (i) excellent = T_alive^excellent ; (ii) marginal = T_alive^marginal

^†Eq. (S1.1) Transplant_combined= T_alive / D_accept

^‡Eq. (S1.2) Transplant_excellent= T_alive^excellent / D_accept^excellent

Eq. (S1.3) Transplant_marginal = T_alive^marginal / D_accept^marginal

^°Eq. (S1.4) Overall = P_alive / P_initial

Appendix 2. Full text of survey questions and response options.

Appendix Table 2.

Survey questions and response options.

Question	Response options
Introduction text: In this study you will be asked to view a series of outcome statistics for two different organ transplant hospitals. A hospital that is designated as a transplant center means that patients can register to be on the waiting list for a transplant organ at that hospital. You will begin by reading important background information about the transplant process, then you will compare two hypothetical transplant hospitals and rate which hospital would be better for patients.
Today you will see outcomes from heart transplant patients at two different transplant hospitals. Patients can join the [United Network for Organ Sharing (or UNOS)] waiting list to receive a donor heart at one of these hospitals when they have a medical condition that requires a new heart. The waiting list gives these patients the chance to be matched with a donor heart and receive a transplant.
A transplant team of surgeons, cardiologists, nurses, and other medical professionals is notified [through UNOS] when a patient on the waiting list at their hospital matches with a donor heart. If the team decides to accept the donor heart for their patient, the patient receives a transplant. If the team decides to reject the donor heart, it may be offered to other teams in the [UNOS] “sequence” of patients who match with the heart. If every team in the sequence rejects the donor heart, it may even be discarded, meaning that the heart is not transplanted at all.
Some donor hearts have excellent function, while other donor hearts have function that is less-than-optimal, but still adequate for many patients on the waiting list. [For instance, a donor heart that is less-than-optimal might: have a lower left ventricular ejection fraction (below 50%) than the average donor heart; be from a donor for whom 2 or more inotropes were being administered at the time of death; (iii) be from a ”CDC high risk” donor (a donor with potential infectious disease status; for example, a donor whose organs test positive for HIV or HCV antibodies); be from a donor with a glomerular filtration rate (GFR) of <30 mL/min per 1.73 m2; be from a donor for whom cerebrovascular accident (CVA), or stroke, was the primary cause of death.]
These adequate hearts tend to be rejected by transplant teams more often than hearts with normal function -- mainly because the adequate hearts may cause a small increase in the risk of patient complications or death after transplant surgery. For the most part, adequate hearts do show improved function as the patient recovers from their transplant surgery.
If a donor heart is turned down, there is no guarantee that another will become available. There are not enough donor hearts for all the patients waiting for a heart transplant. Some patients will not get a heart in time for transplant and die while waiting.
Bonus question 1: Which of the following describes a possible event that can occur when a waiting list patient is matched to a donor heart [through UNOS]?	(1) The patient’s transplant team can accept the donor heart, and the patient receives transplant surgery; (2) The patient’s transplant team can reject the donor heart, and the heart is offered to the next waiting-list patient in the match sequence; (3) The patient’s transplant team can reject the donor heart, and the heart is discarded (not used for any transplant); (4) All of the above events can possibly occur when a waiting-list patient is matched to a donor heart; (5) None of the above events can possibly occur because no acceptance or rejection takes place: the waiting-list patient who matches to a donor heart always receives a transplant.
Explanation of choice task: You will consider the choice between Hospital A and Hospital B. First, you will view a table of the outcome information for the patients at each hospital. Please pay attention to all the information presented so you can correctly answer the bonus questions. Next, you will indicate whether Hospital A or Hospital B is a better choice for future patients who want to be added to the waiting list for a potential transplant.
Choice scenario: The transplant teams at Hospital A and Hospital B both had 90 waiting list patients at the beginning of the year, and received 54 donor heart offers throughout the year. This means that 54 patients at each hospital matched to a heart, and the transplant team at that patient’s hospital made a decision to accept or reject the donor heart for each patient match. When teams accepted a donor heart match, the patient who matched received a transplant; when teams rejected a donor heart match, the patient who matched remained on the waitlist for the rest of the year.
Please view the legend below, then scroll to the bottom of the page to see the patient outcomes for both hospital A and hospital B. [View legend corresponding to randomly assigned outcome table]
[View randomly assigned outcome tables, see “manipulated variables” on OSF links.]
Choice: Which Hospital is a better choice for patients? Please click on one of the two tables below to indicate which hospital is the better choice.	Hospital A; Hospital B
[Which Hospital do you consider to be higher performing? Please click on one of the two tables below to indicate which transplant center is superior.]*
Free response: In your own words, why do you think patients should choose the hospital you picked? [In your own words, why do you think the hospital you chose is superior?]	[text entry field]
Process measures: There are many reasons why one transplant hospital might outperform another. Which reasons were most important in your decision? Please move the slider to indicate how much you considered each of the reasons below: Patients were more likely to receive an excellent donor heart at the hospital I picked. Patients were less likely to receive an adequate donor heart at the hospital I picked. Patients were more likely to receive any type of donor heart at the hospital I picked.	[0–100 slider for each of the choices (a), (b), (c)] 0=indicates reason was not important; 100=indicates reason was extremely important
Bonus question 2: Which transplant hospital received more donor heart offers for the patients on their waiting list?	(1) Hospital A received more donor heart offers; (2) Hospital B received more donor heart offers; (3) Hospital A and Hospital B received the same number of donor heart offers; (4) This information was not provided.
Demographic question 1: What is your gender?	Male; female; transgender; [prefer to self-describe]; prefer not to answer
Demographic question 2: What is your age?	18–19; 20–29; 30–39; 40–49; 50–59; 60–69; 70+
Demographic question 3: What is your race and ethnicity? (check all that apply)	White/Caucasian; Black/African American; Asian; Hispanic/Latino; American Indian/Pacific Islander; Two or more races; Prefer not to answer
Demographic question 4: What is your highest level of education?	Less than high school; high school; some college; Bachelor’s degree; Graduate degree; Prefer not to answer
[Demographic question 5: What year of medical school are you currently completing?]	[2nd; 3rd; 4th; Gap-year/Leave of absence; None of these; Prefer not to answer]
Final comments: Please leave any comments about the survey here:	[text entry field]

Note. Bracketed and italicized text indicates a modified version of the wording in Studies 1/3/4 was presented to the medical trainees in Study 2 (e.g. “Study 1 text [Study 2 modified text]”). A * indicates that the [Study 2 modified text]* was used in Study 3.

Appendix 3. Response accuracy for attention check questions.

Appendix Table 3.1.

Number (and percentage) or participants with 0, 1, 2, or 3 correct responses to the attention check questions.

	Number of correct responses n (% of study population)
	0	1	2	3†
Study 1 (n=1003)	50 (5.0)	245 (24.4)	708 (70.6)	-
1: Combined transplant survival	5	37	138
2: Combined transplant + Overall survival	13	54	123
3: Stratified transplant survival	9	57	122
4: Stratified transplant + Overall survival	10	51	146
5: No transplant + Overall survival	13	46	179
Study 2 (n=105)	-	16 (15.2)	89 (84.8)	-
1: Combined transplant survival		8	48
3: Stratified transplant survival		8	41
Study 3 (n=123)	7 (5.7)	24 (27.6)	82 (66.7)	-
1: Combined transplant survival	3	16	49
3: Stratified transplant survival	4	18	33
Study 4 (n=807)	13 (1.6)	80 (9.9)	225 (27.9)	489 (60.6)
1: Combined transplant survival	3	6	39	143
2: Combined transplant + Overall survival	4	35	80	95
3: Stratified transplant survival	2	12	49	137
4: Stratified transplant + Overall survival	4	27	57	114

^†Study 4 included a third attention check question that measured stimulus recall accuracy within each condition (thus, the outcome table options presented varied by condition).

Appendix Table 3.2.

Distribution of answers to attention check response options.

	% of study population
	Study 1	Study 2	Study 3	Study 4
Question 1: Which of the following can occur when a waitlist patient is matched to a donor heart?
Transplant team can reject heart; heart offered to next patient	4.7	1.9	4.9	-
Transplant team can accept heart; patient gets transplant	2.7	1.9	1.6	-
Transplant team can reject heart; heart is discarded	1.2	-	2.4	-
All of these events can occur*	90.7	96.2	90.2	91.4
None of these events can occur	0.7	-	0.8	8.6
Question 2: Which hospital received more donor heart offers for the patients on their waiting list?
Hospital A received more donor offers	10.4	3.8	-	-
Hospital B received more donor offers	11.2	6.7	-	-
Hospital A and B received the same number of offers*	74.9	88.6	70.7	76.5
This information was not provided	3.6	0.9	29.3	23.5
Question 3†: Each of the four tables below is labeled with a different set of hospital outcome metrics. Which option is formatted the exact same way as the pair of tables you viewed earlier?
Table selection matched condition-specific metrics*	-	-	-	79.6
Table selection did not match condition-specific metrics	-	-	-	20.4

^†Study 4 included a third attention check question that measured stimulus recall accuracy within each condition (thus, the outcome table options presented varied by condition).

^*indicates the correct response option

Appendix 4. Descriptive statistics of participant samples.

Appendix Table 4.

Number (and %) of participants with each demographic characteristic – Studies 1–4.

Characteristics	Study 1 (n = 1003)	Study 2 (n = 105)	Study 3 (n = 123)	Study 4 (n = 807)
Gender
Male	504 (50.1)	36 (34.3)	68 (55.3)	371 (46.0)
Female	487 (48.5)	67 (63.8)	54 (43.9)	424 (52.5)
Age group (years)
18–29	225 (22.2)	87 (82.9)	24 (19.5)	140 (17.3)
30–39	362 (36.0)	15 (14.3)	55 (44.7)	311 (38.5)
40–49	217 (21.6)	3 (2.9)	20 (16.2)	170 (21.1)
50–59	122 (12.1)	-	14 (11.4)	107 (13.3)
60+	79 (7.1)	-	9 (7.3)	79 (9.8)
Race/ethnicity
White only	827 (77.2)	69 (62.2)	86 (69.9)	600 (74.4)
Black only	86 (8.0)	5 (4.5)	9 (7.3)	71 (8.8)
Hispanic only	59 (5.5)	6 (5.4)	4 (3.3)	30 (3.7)
Asian only	56 (5.2)	26 (23.4)	15 (12.2)	69 (8.6)
Multiracial/Other	31 (3.1)	5 (4.5)	8 (6.5)	37 (4.6)
Highest level of education
Less than high school	9 (0.9)	-	-	2 (0.2)
High school	107 (10.7)	-	13 (10.5)	73 (9.0)
Some college	354 (35.2)	-	37 (30.1)	223 (27.6)
Bachelor’s degree	401 (39.9)	60 (57.1)	52 (42.3)	365 (45.2)
Graduate degree	129 (12.8)	45 (42.9)	20 (16.3)	141 (17.5)
Current medical school year
2nd	-	31 (29.5)	-	-
3rd	-	28 (26.7)	-	-
4th	-	33 (31.4)	-	-
Gap year or leave of absence	-	12 (11.4)	-	-

Note. Some participants selected the response option ‘prefer not to answer’ for demographic questions, therefore totals may not add up to 100%.

Appendix 5. Summary statistics for choice rationale measures.

Appendix Table 5.1.

Mean rated importance [and 95% CI around mean] of hypothesized mediator variable in choice of transplant center.

	Rated importance of choice rationale “receiving any donor heart” Mean [95% CI]
	Study 1	Study 2	Study 3	Study 4
1: Combined transplant survival	52.5 [47.6–57.3]	68.5 [60.7–76.3]	55.8 [48.0–63.6]	40.1 [34.5–45.7]
2: Combined transplant survival + Overall survival	64.6 [60.4–68.9]	-	-	52.3 [46.1–58.5]
3: Stratified transplant survival	78.7 [74.5–82.6]	82.2 [75.4–89.0]	76.8 [68.9–84.7]	73.5 [68.1–78.8]
4: Stratified transplant survival + Overall survival	75.5 [71.5–79.5]	-	-	80.5 [76.5–84.6]
5: No transplant survival + Overall survival	74.5 [71.3–77.7]	-	-	-

Participants rated the importance of being “more likely to receive any type of donor heart” in considering their choice of transplant center. Ratings were given using a 0–100 continuous scale from 0 = not at all important in my choice of center to 100 = extremely important in my choice of center.

Appendix Table 5.2.

Mean rated importance [and 95% CI around mean] of donor heart quality measures in choice of transplant center.

	Rated importance of choice rationales Mean [95% CI]
	Study 1	Study 2	Study 3	Study 4
Patients were more likely to receive an excellent donor heart at the hospital I picked.
1: Combined transplant survival	70.7 [66.7–74.7]	60.4 [53.1–67.7]	61.0 [53.6–68.4]	79.8 [75.7–83.9]
2: Combined transplant + Overall survival	65.6 [62.0–69.3]	-	-	73.9 [69.4–78.5]
3: Stratified transplant survival	52.7 [47.9–57.6]	47.6 [37.6–57.6]	53.7 [44.8–62.6]	61.1 [56.1–66.2]
4: Stratified transplant + Overall survival	55.7 [51.1–60.3]	-	-	54.3 [49.4–59.3]
5: No transplant + Overall survival	65.3 [62.2–68.4]	-	-	-
Patients were less likely to receive a marginal donor heart at the hospital I picked.
1: Combined transplant survival	50.7 [46.3–55.0]	46.0 [39.8–52.3]	44.8 [37.7–51.9]	50.8 [45.3–56.2]
2: Combined transplant + Overall survival	49.7 [45.6–53.8]	-	-	50.6 [45.4–55.8]
3: Stratified transplant survival	40.0 [35.0–44.9]	30.0 [22.1–37.8]	43.0 [33.5–52.5]	41.9 [36.3–44.3]
4: Stratified transplant + Overall survival	39.9 [35. –44.7]	-	-	39.1 [33.9–44.3]
5: No transplant + Overall survival	43.8 [40.2–47.4]	-	-	-

Participants rated the subjective importance of each choice rationale measure when considering the transplant center choice scenario. Ratings were given using a 0–100 continuous scale from 0 = not at all important in my choice of center to 100 = extremely important in my choice of center.