Impact of Publicly Reported Outcomes on Patient Selection for Hematopoietic Cell Transplantation

Abstract

Background:

Public reporting of healthcare outcomes can have unintended effects such as inappropriate risk aversion in patient selection.

Methods:

The Center-Specific Survival Analysis (CSA) annually assigns all HCT centers in the United States a +1, −1, or 0 score for observed outcomes that are above, below or within a center-specific predicted range of outcome. For each index year (2012-2016), centers receiving a −1 score following 0 scores in the preceding 2 years were compared with contemporaneous centers with “as predicted” outcomes (0 score). Changes in the patient population characteristics in the 3 years before vs the 3 years after the index years were compared between the NBCs and the controls. A multivariate model adjusted for baseline patient population characteristics and center volume.

Results:

No differences in patient selection behavior were identified when comparing the NBCs with the controls across 8 key patient population characteristics. For the statistically modeled (predicted) 1-year OS, reflecting a holistic measure of centers’ patient population risk, we observed no statistically significant difference in change (−0.23%, 95% CI: −1.4% to 0.9%, p=0.70). The observed overall survival increased in both NBCs and controls by 0.9% and 4.5% respectively, without statistically significant difference in change.

Conclusions

Centers receiving a −1 score were not observed to deviate significantly from patient selection trends in the HCT field. These findings suggest that public reporting of HCT outcomes in the US does not result in unintended bias against HCT for high-risk patients.

Introduction

The TRANSPLANT Act of 2021 requires public reporting of relevant clinical outcomes for all transplant centers performing allogeneic hematopoietic cell transplantation (HCT).¹ To satisfy this requirement, the Center for International Blood and Marrow Transplant Research (CIBMTR) publishes risk adjusted and unadjusted 1-year overall survival (OS) outcomes of all participating bone marrow transplant programs in the Annual Report on Hematopoietic Cell Transplant Center-Specific Survival Rates, referred to as the Center-Specific Survival Analysis (CSA).² Public reporting, such as that done by the CIBMTR, improves health care transparency and can drive improvement in clinical outcomes of interest.^3,4 However, public reporting of health outcomes can also have unintended consequences. For instance, the results of centers’ scores on the CSA have been demonstrated to impact on transplant centers’ patient volumes in the years following receipt of a low performance score.⁵ To fully appraise the impact of this health policy on centers performing HCT and the patients undergoing these potentially life-saving procedures, it is important to assess for other potential unintended adverse consequences of the publicly reported CSA.

Another possible consequence of publicly reported health outcomes may be to shift healthcare providers’ decision-making regarding patient selection. Publicly reported outcomes could systematically bias providers away from intervening on patients perceived to be at high risk for mortality due to concern for the risk of impact on a provider’s or center’s outcome score, even if those patients may otherwise be indicated for intervention. For example, surveys of medical decision making among interventional cardiologists have identified a widespread belief that risk-adjustment models for publicly reported cardiology outcomes insufficiently adjust for the highest risk patients, potentially resulting in risk avoidance behavior when interventionalists are faced with the decision of whether or not to intervene on high risk patients.^6,7 High-mortality outlier hospitals performing coronary artery bypass grafts operate on less risky patients on average following release of publicly reported mortality outcomes.⁸ Additionally, for kidney transplant physicians, “increased patient selection criteria” were reported by 81% of respondents following receipt of low performance assessments from the publicly available outcomes reports.⁹ An analogous phenomenon has been reported among HCT centers; In a review of corrective action plans submitted by 47 underperforming HCT programs identified either by self-reported lower than predicted survival or by the annual CIBMTR CSA, the most frequently proposed corrective action was “changes to patient selection criteria,” proposed by 28 programs (60%).² To adjust for differences in the populations transplanted at each transplant center and to address potential risk avoidance behavior, the CSA uses the patient population characteristics of each center’s transplant population to model their predicted survival outcomes, and compares the observed survival outcomes against these predicted survival outcomes, assigning either overperforming, underperforming or as expected performance designations. The model used for the CSA has undergone iterative refinements since the program’s inception to improve adjustment for factors that are outside the control of the HCT centers, and mitigate risk avoidance behavior. However, it is not known whether these efforts have been successful.

In this report, we have sought to quantify changes in patient selection behavior at centers performing HCT that result from receipt of a low performance score on the publicly reported CSA.

Methods

The CIBMTR is a working group comprised of more than 360 worldwide stem cell transplant centers that contribute detailed data on autologous and allogeneic HCT to a central statistical center at the Medical College of Wisconsin. To produce each publicly available annual report on transplant center-specific survival rates, the CIBMTR uses patient outcome data collected from all U.S. transplant centers performing at least 1 allogeneic HCT during a 3-year window preceding the report year. A statistical model is developed each year to adjust for essential risk factors known or suspected to influence 1-year overall survival (OS). Each transplant center’s predicted 1-year survival with 95% confidence interval (CI) is estimated from their transplanted population’s baseline characteristics using the statistical model, and compared with the observed 1-year OS at that center. Centers with an observed 1-year OS within their 95% CI receive a 0 score, indicating “as expected” outcomes. Centers with observed 1-year OS below their 95% CI receive a −1 score, and those with observed 1-year OS above their 95% CI receive a +1 score. The CSA reports are released in annually in December. Input into the methods, especially the risk factors considered, is solicited from HCT stakeholders at the annual Center-Specific Outcomes Analysis forum, allowing for iterative refinement of the methodology.¹⁰

For the present analysis, the datasets used to generate the annual CSA reports for the years 2012 to 2018 were used. We included centers performing at least 1 adult related or unrelated HSCTs in the year preceding the report. Combined pediatric/adult centers were included, while pediatric only centers were excluded. The exposure of interest was receipt of a −1 score during the years 2012-2016 with the two years preceding the −1 score having “as expected” (0) or “above expected” (+1) scores. Centers with exposure to these conditions comprised the “newly below-expected centers” (NBCs) group. The outcome of interest was the change in the characteristics of the populations transplanted at these centers during the 3-year period after the −1 score, compared to the 3 year period before receipt of the −1 score. To account for changes in the practice of the transplant field as a whole over these time periods, we formed control groups consisting of all centers performing “as expected” for each 6-year window. These control groups were defined by the receipt of a 0 score for the 6 consecutive years corresponding to the 6-year periods defined by each group of NBCs. For example, a group of NBCs defined by 0/, 0/, −1 scores for 2010, 2011, and 2012 respectively were compared with a group of control centers with consecutive 0 scores for 2010, 2011, 2012, 2013, 2014, and 2015, respectively (Figure 1). Centers that did not satisfy the criteria for the NBC group nor the control group were not included in the analysis. Centers that ceased performing HCT in the 3 years following −1 scores were also excluded.

Figure 1:

Identification of Newly Below Expect OS Centers and Control Centers

CSA: Center Specific Analysis result

Y=0: the index year of the center specific analysis.

Statistical Methods:

The pre-exposure patient population characteristics were calculated from the populations of patients transplanted at the NBCs and control centers in the index year and 2 preceding years for the key population characteristics of interest: Age > 60 years, HCT-CI > 3, Karnofsky Performance Status >=90, non-White, advanced disease, use of peripheral blood graft, use of non-matched sibling/matched unrelated donor, and use of non-myeloablative conditioning. The proportions in each category were adjusted for center volume. The post-exposure population proportions at the NBCs and control centers were calculated from the population of patients transplanted at the centers during the 3 years following the index year. The change scores in population proportions were calculated as post-exposure proportion – pre-exposure proportion. The primary outcome of this analysis was the difference in change scores in population proportions of the NBCs and the control centers, calculated as ΔNBC – ΔControl. Linear mixed models were used to model change scores for each characteristic and to estimate the difference in change scores, with NBC vs. control, index year, and pre-exposure measurements as fixed effects and using an unstructured correlation matrix to account for repeated measurements on transplant center across the index years. Due to the multiple comparisons made between the NBCs and control groups, and significance threshold of p < 0.01 was pre-specified.

Results:

Between 2012 and 2016, 24 transplant centers met the criteria for NBCs and had sufficient follow up data for inclusion in the study. The number of NBCs per index year ranged from 2 in 2016 to 9 in 2015 (Table 1). For each 3-year window preceding the index years, the median number of patients transplanted at the NBCs was 375 (range: 86-2907). In the 3-year windows following the index years, the median number of patients transplanted at NBCs was 430 (range: 69 −2500 patients). In total, 4567 patients were transplanted at all NBCs in the 3-year windows before the index years, and 4130 patients were transplanted at all NBCs during the 3-year windows after the index years. Seven centers were considered low volume centers, with fewer than 40 patients transplanted per year from 2008-2017 (Table 1). In addition to the 24 NBCs, there were 11 centers that stopped performing transplants during the 3 years after receiving a −1 report which were excluded from the analysis due to insufficient follow up data. Prior to closing, these 11 centers transplanted a median of 21 patients annually (range: 3-145), and were active for a median of 7 years (range: 1-7 years).

Table 1:

Number of Centers and Patients in Newly Below Expected OS Centers and Control Centers in Each Index Year

Report year	NBC: Pre-exposure	Control: Pre-exposure	NBC: Post-exposure	Control: post-exposure
	# of Centers (# of patients)
2012	4 (375)	35 (3931)	4 (430)	35 (4530)
2013	4 (264)	37 (4023)	4 (288)	37 (4492)
2014	5 (935)	43 (5011)	5 (843)	43 (5492)
2015	9 (2907)	39 (4751)	9 (2500)	39 (4939)
2016	2 (86)	41 (5264)	2 (69)	41 (5560)
Low volume center	7 (345)	24 (2076)	7 (296)	39 (2239)

NBC: Newly Below-Expected OS Centers

For each group of NBCs, as defined by their index years, a group of control centers were identified. The control center groups had between 35 and 43 centers. Centers could be included in the control center groups for multiple index years, resulting in significant overlap in the centers comprising the control center groups included in each 6-year window. Low volume centers accounted for 29% of NBCs (7/24), and 62% of control centers (103/165).

Statistically significant changes were evident when comparing the populations transplanted before the index years to those after the index years in both the NBCs and the control centers. Population changes at NBCs and control centers had the same directionality, either increase or decrease, for most baseline population characteristics tested, with the exception of use of non-MAC regimens. Use of non-MAC regimens decreased at NBCs (−4.11% p=0.19) while it increased at control centers (+5.01%, p=0.02). At both NBCs and control centers, the proportion of patients with HCT-CI > 3 significantly increased (+8.03% p<0.01, and +5.96% p<0.01, respectively), as did the proportion of patients with age > 60 (+5.83% p<0.01, and +8.43% p<0.01, respectively), and the proportion with advanced disease (+13.25% p<0.01, and +9.14% p<0.01, respectively). The proportion of patients with KPS ≥ 90 decreased significantly at both NBCs and control centers (−7.17% p <0.01, and −4.74% p<0.01, respectively). The non-White patients accounted for an increased proportion at both NBCs and control centers, though only the increase at control centers was statistically significant (+2.12% p<0.01) (Table 2).

Table 2:

Pre-to-Post Change in Proportions and Difference in Change

	NBC*	control*	P	NBCs (pre-to-post change)	Controls (pre-to-post change)	Difference in Change (ΔNBC – ΔControl )(95% CI)	P
Observed OS	62% (61, 64)	67% (67, 68)	<0.01	0.87% (−1.56, 3.31)	4.52% ‡ (2.94,6.10)	−3.64% (−6.56, −0.72)	0.02
Predicted OS	66% (64,67)	67% (65, 68)	0.21	3.08% ‡ (2.08, 4.08)	3.30% ‡ (2.73,3.88)	−0.23% (−1.38, 0.93)	0.69
Non-White	20% (10, 25)	23% (19, 27)	0.35	0.51% (−1.73, 2.75)	2.12% ‡ (0.96,3.28)	−1.61% (−4.14, 0.91)	0.21
HCT-CI > 3	21% (46, 56)	23% (19, 27)	0.16	8.03% ‡ (4.04, 12.02)	5.96% ‡ (3.22,8.70)	+2.07% (−2.78, 6.92)	0.40
Age > 60	51% (46, 56)	46% (43, 20)	0.68	5.83% ‡ (3.09, 8.58)	8.43% ‡ (6.45,10.42)	−2.60% (−5.94, 0.74)	0.13
Advanced Disease	43% (40, 45)	41% (39, 43)	0.26	13.25% ‡ (9.94, 16.57)	9.14% ‡ (6.88,11.40)	+4.12% (0.11, 8.13	0.04
Peripheral Blood Graft	77% (72, 82)	85% (81, 88)	0.02	−4.97% † (−9.90, −0.05)	−1.87% (−5.29,1.56)	−3.11% (−9.19, 2.97)	0.28
Non-MSD/MUD	22% (19, 26)	24% (22, 26)	0.47	2.78% (0.80, 6.35)	1.65% (0.89,4.19)	+1.12% (−3.25, 5.50)	0.61
Non-MAC	38% (32, 44)	46% (42, 51)	0.03	−4.11% (−10.30, 2.07)	5.01% † (0.77,9.24)	−9.12% (−16.69, −1.55)	0.02
KPS ≥90	66% (59, 73)	61% (56, 66)	0.26	−7.17% ‡ (−11.81, −2.53)	−4.74% ‡ (−7.36,−2.12)	−2.44% (−7.84, 2.97)	0.37

^*Proportions are adjusted for center volume

Abbreviations: HCT-CI: Hematopoietic Cell Transplantation-specific Comorbidity Index. KPS: Karnofsky Performance Score. MAC: Myeloablative Conditioning. MSD: Matched Sibling Donor. MUD: Matched Unrelated Donor. NBC: Newly Below-Expected Centers. OS: Overall Survival.

^†p < 0.05

^‡p < 0.01

For the primary outcome, the differences in the magnitude of change between the NBCs and control centers were calculated as ΔNBC – ΔControl for each center characteristic. No difference in change for any individualistic characteristic reached statistical significance, given the significance threshold of p<0.01 (table 2). In absolute terms, the population characteristics with the largest changes included the proportions of patients undergoing non-MAC regimens (−9.12% p=0.02), with advanced disease (+4.12% p=0.04), and receiving peripheral blood grafts (−3.11%, p=0.28). For some high-risk characteristics, such as HCT-CI > 3, the change in the population was numerically greater for the NBCs compared with control centers (ΔNBC – ΔControl = +2.07%, 95% CI −2.78-6.92%, p=0.40). For other high-risk characteristics, such as age > 60, the change in population was numerically greater for the control centers (ΔNBC – ΔControl = −2.60%, 95% CI −5.94-0.72%, p=0.13).

At baseline, the predicted 1-year OS was not significantly different between the NBCs (66%) and the control centers (67%). The observed 1-year OS was significantly lower in the NBCs compared to the control centers at baseline (62% and 67% respectively, p<0.01), reflecting their status as having below-expected outcomes. The predicted 1-year OS for NBCs and control centers each significantly increased when comparing the pre-index years with the post-index years (+3.08% p<0.01, and +3.30% p<0.01). For the primary outcome analysis, the similar increase in predicted 1-year OS resulted in a small ΔNBC – ΔControl, which was not statistically significant (−0.23% p=0.69). In terms of observed 1-year OS, increases were observed at both the NBCs (+0.87% p=0.48) and the control centers (+4.52% p<0.01), but only the increase at the control centers was statistically significant (Table 2). The difference in observed 1-year OS was larger than that of predicted 1-year OS, but did not reach the threshold of statistical significance (−3.64% p=0.02).

Discussion

We hypothesized that receipt of a −1 report would cause a change in patient selection behavior at the affected centers, and that the population selected for transplant at these centers would skew towards lower risk patients during the time period following the −1 report relative to the rest of the transplant centers. Our analysis, incorporating data from 9 years of allogeneic HSCT in the US, did not identify evidence of systematic change in patients selection behavior as a result of −1 reports. Although there were statistically significant changes in the characteristics of patients transplanted prior to the −1 reports to those transplanted after the −1 reports at NBCs, the changes did not statistically significantly differ from the trends in the field as a whole.

Generally, the population of patients transplanted at both the NBCs and control centers grew older, had more comorbidities, and were less likely to have KPS ≥ 90 with time, reflecting a notable shift in practices in the transplant field as a whole.¹¹ Receipt of a −1 report did not result in the NBCs significantly deviating from this overall trend. A decrease in the use of non-MAC regimens was one notable deviation from this overall trend, as the proportion of patients receiving non-MAC regimens actually decreased at the NBCs, while it increased in the control centers. Our data do not clearly offer a hypothesis for this difference, though notably the change observed at NBCs was not statistically significant, and the difference in differences between the NBCs and control centers was not statistically significant, so caution in over-interpretation is warranted.

The proportion of patients with high risk characteristics such as HCT-CI > 3 increased more in the NBCs, though not to a statistically significant degree. The same was seen with advanced disease. Despite these population changes, the change in predicted OS was similar between the NBCs and control centers. The predicted 1-year OS was generated using the annual CSA models, and commensurate increases in predicted survival were seen in both groups of centers. Because the annual CSA models are intended to aggregate the risk levels of transplant centers’ populations, we interpret this as indicative that the NBCs continued to follow the general patient selection trends in the transplant field despite their receipt of a −1 report.

This analysis adds to growing literature on the impact of public reporting on medical practice. Findings similar to this report have been observed in the context of colorectal surgery. Introduction of public reporting of colorectal surgery outcomes as part of the National Bowel Cancer Audit in the UK did not result in significant changes in the predicted OS, indicating that the public reporting did not result in risk averse behavior among surgeons which would have been expected to increase the predicted OS.¹² Receipt of a −1 report on the CIBMTR CSA has been observed to result in a decreased transplant volume at the recipient center along with an increase in the volume at nearby transplant centers.⁵ An analogous impact on center volume has been observed in kidney transplant centers receiving lower scores on the Scientific Registry for Transplant Recipients’ Program-Specific Reports, which publicly report program-specific outcomes every 6 months for solid organ transplant programs.¹³ Our results suggest that the reduction in volume in HCT centers is not due to systematic exclusion of particular high risk patients.

Our finding that patient population changes at NBCs generally follow the trends of the field as a whole is surprising in part because changes to patient selection criteria is the most frequently cited corrective action by these centers.² This could be because the specific changes that are made to patient selection criteria may not be consistent from center to center, diluting their effects in this aggregated data. Additionally it is possible that the patient selection criteria which were changed were not analyzed in the CSA. Finally, the degree to which proposed corrective action plans go on to be successfully implemented is also not known, and so it is possible that proposed changes to patient populations are not carried out.

It is necessary to acknowledge several important limitations of this analysis. Data on the socio-economic status of patients was available in the CIBMTR dataset, so the impact of receipt of a −1 report on patient selection practices by socio-economic status could not be evaluated. Public reporting of outcomes has been reported to exacerbate health access inequities in other contexts. Following introduction of publicly reported quality information for home health agencies, a decrease in use of high quality home health agency use was observed in predominantly Hispanic neighborhoods, in contract to an increase in predominantly White neighborhoods.¹⁴ The CIBMTR has begun incorporating transplant recipient median household income based on ZIP code, and this may be an important factor to monitor in the future. Statistical power was another important limitation of these analyses. Below expected outcomes (−1) reports are expected in approximately 2.5% of centers annually, resulting in a paucity of events to power the analysis. The analysis may not have had sufficient power to detect smaller biases in patient selection practices.

Overall, these results support continued public reporting of HCT outcomes, and that their use did not result in unintended exclusion of high-risk patient populations. Ongoing vigilance is needed to ensure the reports accurately capture the nuances of the patients being transplanted and to avoid bias in the future. This requires continuous updating of the outcomes model used for the public reports to reflect the latest advances in the field, and incorporation of more granular patient characteristics, such as socio-economic status. With these efforts, public reporting can remain an important tool for improving patient outcomes in the HSCT field.