A Novel Patient-Centered “Intention-To-Treat” Metric of U.S. Lung Transplant Center Performance

Abstract

Despite the importance of pre-transplant outcomes, one-year post-transplant survival is typically considered the primary metric of lung transplant center performance in the U.S.. We designed a novel lung transplant center performance metric that incorporates both pre- and post-transplant survival time. We performed an ecologic study of 12,187 lung transplant candidates listed at 56 U.S. lung transplant centers between 2006 and 2012. We calculated an “intention-to-treat” survival (ITTS) metric as the percentage of waiting list candidates surviving at least one year after transplantation. The median center-level 1-year post-transplant survival rate was 84.1%, and the median center-level ITTS was 66.9% (mean absolute difference 19.6%, 95% limits of agreement 4.3 to 35.1%). All but 10 centers had ITTS values that were significantly lower than 1-year post-transplant survival rates. Observed ITTS was significantly lower than expected ITTS for 7 centers. These data show that one-third of lung transplant candidates do not survive one year after transplantation, and that 12% of centers have lower than expected ITTS. An “intention-to-treat survival” metric may provide a more realistic expectation of patient outcomes at transplant centers and may be of value to transplant centers and policymakers.

INTRODUCTION

Lung transplantation is a lifesaving option for adults and children with end-stage lung diseases.^1,2 U.S. lung transplant centers are regulated by the Department of Health and Human Services, with their accreditation status dependent on center performance.³ The Scientific Registry of Transplant Recipients (SRTR) monitors center performance and issues publicly-available program-specific reports (PSRs) on center performance, which are scrutinized by the Department of Health and Human Services, third party payers, transplant center personnel, and potential transplant candidates, their caregivers, and their physicians.³

Data presented in publically available PSRs and their patient-friendly web-based derivatives can be driving factors when patients and physicians choose their transplant center. The 1-year post-transplant survival rate available on the SRTR website is the primary metric by which the public judges center performance.³ This metric, however, has certain limitations. Centers may be incentivized to optimize post-transplant survival rates by avoiding high risk patients, in some cases based on anecdotal or low-quality evidence,³ thereby unfairly disadvantaging patients who stand to benefit from transplantation. Furthermore, this metric does not encompass the outcomes of those who were listed but never transplanted, therefore not reflecting the actual patient experience at each center. Centers with highly publicized high post-transplant survival rates may appear attractive to patients, but some of these centers may have low transplantation rates, falsely leading patient to anticipate a “good outcome” at a particular center.

While the PSRs contain many performance metrics, none combines both pre- and post-transplant survival time. Patients are unlikely to be able to integrate transplantation rates, waiting list mortality rates, post-transplant survival rates, and transplant center volume in order to choose the center that maximizes their chances of achieving a good outcome, which may reasonably include both successful transplantation and survival one year after transplantation. In order to provide a more transparent, meaningful, and comprehensive measure of lung transplant center performance, we developed a novel, patient-centered “intention-to-treat” measure that incorporates both the probability of transplantation and the probability of patient survival one year after transplantation, which we term “intention-to-treat 1-year survival” (ITTS). We hypothesized that ITTS would be significantly and substantially underestimate 1-year post-transplant survival rates at most lung transplant centers, and that variation in ITTS across centers would not be entirely explained by the characteristics of the waiting list candidates at each center.

MATERIALS AND METHODS

Study Design, Participants, and Data Sources

We performed an ecologic study of lung transplant candidates placed on the active waiting list between January 1, 2006 and December 31, 2012 based on Organ Procurement and Transplantation Network data as of September 9, 2016. Of the 15,913 listed at 68 centers during the study period, we excluded those with follow-up time of less than one year after transplantation (n = 273), re-transplant candidates (n = 2,047), those who were simultaneously listed at multiple centers (n = 284), candidates at centers that only listed one candidate (n = 1), and those removed for the following reasons: refused transplantation (n = 70), transferred to another center (n = 44), “other” (n = 573), condition improved (n=203), transplanted at another center (n=81), living donor transplantation (n = 2), candidate removed in error (n=14), and unable to contact candidate (n=27), leaving 12,294 candidates at 68 centers. After generating the expected ITTS prediction model described below, we further excluded 12 out of the 68 centers that had fewer than 20 expected ITTS events to avoid over-interpretation of the performance of low volume centers and to permit inferential statistical hypothesis testing. These 12 centers listed 23 or fewer candidates each during the study period (n = 107 candidates in total). This left 12,187 candidates who were included in the study. The Columbia University Medical Center Institutional Review Board approved this study (IRB #AAAB5142).

Measurements

We calculated the ITTS as the number of waiting candidates who both underwent transplantation and survived at least one year after transplantation divided by the number of waiting list candidates placed on the waiting list during the study period multiplied by 100. For example, a transplant center that listed 1,000 patients during the study period, of whom 700 were transplanted and of whom 650 survived through one year would have an ITTS of 650 ÷ 1000 × 100, or 65%. This same center would have a one-year post transplant survival of 650 ÷ 700 × 100, or 93%.

Analysis approach

Candidate characteristics were summarized by means and percentages across quartiles of ITTS. We used a Pearson correlation coefficient to examine the correlation between center-level 1-year survival and ITTS.

To examine patient expectations of center performance, we compared the survival rate that patients expect to experience at each center (1-year post-transplant survival, hereafter termed “anticipated” survival) to the observed survival that patients actually experience at each center (observed ITTS) by calculating the raw observed-to-“anticipated” (O/A) ratio for each center. Standard errors and chi-square statistics were calculated for each center’s O/A survival to generate 95% confidence intervals. We also calculated the absolute differences between observed and anticipated survival, which we refer to as the “discrepancy” between observed and anticipated survival.

To compare observed and expected ITTS, we used the glmnet package in R to develop a prediction model for expected ITTS using penalized estimation and shrinkage (least absolute shrinkage and selection operator, LASSO) with 10-fold cross-validation.⁴ Candidate predictors included age, sex, ABO blood type, BMI, diagnosis, height, LAS and mechanical ventilation. All variables were retained in the model. Predicted (expected) ITTS was calculated for each center using the model coefficients. We then calculated observed-to-expected (O/E) ITTS ratios for each center. Standard errors and chi-square statistics were calculated for each center’s O/E ratio to generate 95% confidence intervals.

We used a generalized linear mixed model using the logit link function to estimate the proportion of the measured variance in ITTS explained by center-level characteristics with adjustment for age, gender, LAS, diagnosis, use of mechanical ventilation, and ABO blood type. Analyses were performed using Stata version 14.1 (StataCorp, College Station, TX) and R version 3.3.1 (R Foundation for Statistical Computing).

RESULTS

Baseline characteristics

We studied 12,187 candidates placed on the waiting list at 56 lung transplant centers during the study period, of whom 10,129 underwent transplantation (1,486 of whom died within 1 year of transplantation), 1,235 died on the waiting list, and 823 were removed from the list for clinical deterioration or medical instability. The mean age at listing was 54.5 years, 43% were women, and the mean LAS at listing was 40.0. Bilateral transplantation tended to be more common among centers with higher ITTS (Table 1). Other candidate, donor, and recipient characteristics were similar across centers with high and low ITTS and post-transplant survival rates (Tables 1 and 2).

Table 1.

Baseline characteristics of lung transplant candidates at 56 U.S. lung transplant centers across “intention-to-treat” survival quartiles

Characteristic	“Intention-to-treat” survival quartile
	<64%	64–67%	67–71%	>71%
Total number of centers	24	10	11	11
Candidate characteristics (n = 12,187)
No. of candidates	3,072	3,172	3,284	2,659
Age, years	54.5	54.4	54.7	54.5
Female	44.3	42.1	42.6	43.1
Body mass index, kg/m2	25.6	25.5	25.3	24.8
Height, cm	169.2	169.5	169.6	169.8
Diagnosis
Obstructive lung disease	33.8	32.1	31.2	32.2
Pulmonary vascular disease	3.7	5.7	4.1	2.6
Cystic fibrosis	11.2	10.9	11.9	13.4
Interstitial lung disease	51.2	51.3	53.0	51.9
Lung allocation score	42.0	41.0	41.9	34.1
Oxygen use, L/min	4.2	4.0	4.1	4.6
Mechanical ventilation	3.5	2.7	3.7	3.5
Double lung preference	78.4	76.8	76.8	83.2
ABO blood type
O	45.9	46.3	44.3	44.8
A	39.0	38.5	40.5	41.1
B	11.4	11.4	11.0	11.1
AB	3.7	3.7	4.2	3.1
Recipient & Donor Characteristics (n = 10,129)
No. of recipients	2,288	2,622	2,810	2,409
Recipient Age, years	55.4	55.3	55.5	54.8
LAS at transplant	46.3	45.1	46.1	46.4
Mechanical ventilation at transplant	5.2	4.4	11.8	4.6
Ischemic time, hours	4.8	5.0	5.2	5.4
Procedure type
Bilateral	62.5	62.5	65.8	77.0
Single	37.5	37.5	34.2	23.0
Donor age, years	33.8	34.6	34.6	34.0
Donor female sex	38.4	41.5	41.7	37.4
Donor pulmonary infection	44.3	47.5	47.2	49.9

Data are means and percentages

Table 2.

Baseline characteristics of lung transplant candidates at 56 U.S. lung transplant centers across post-transplant survival categories

Characteristic	Survival Rate Tier
	<80%	80–85%	85–90%	>90%
Total number of centers	11	18	19	8
Candidate characteristics (n = 12,187)
No. of candidates	997	4,522	5,111	1,557
Age, years	54.5	55.3	54.3	52.9
Female	47.2	41.8	42.4	46.0
Body mass index, kg/m2	25.3	25.8	24.9	25.2
Height, cm	168.4	169.7	169.8	168.7
Diagnosis
Obstructive lung disease	36.3	34.3	31.3	26.7
Pulmonary vascular disease	4.0	4.2	3.5	5.7
Cystic fibrosis	12.7	10.0	12.7	13.4
Interstitial lung disease	46.9	51.4	52.5	54.2
Lung allocation score	42.0	41.5	37.5	42.1
Oxygen use, L/min	4.3	4.1	4.2	4.4
Mechanical ventilation	3.6	3.3	3.2	3.8
Double lung preference	75.5	77.0	80.3	79.8
ABO blood type
O	47.5	44.9	45.9	43.4
A	37.3	40.1	39.4	41.2
B	11.2	11.4	10.9	11.9
AB	3.9	3.6	3.8	3.6
Recipient & Donor Characteristics (n = 10,129)
No. of recipients	758	3,785	4,308	1,278
Recipient age, years	55.4	56.1	55.0	53.6
LAS at transplantation	45.6	45.2	46.0	48.6
Mechanical ventilation at transplant	4.5	8.9	4.5	8.5
Ischemic time, hours	4.7	5.1	5.3	5.0
Procedure type
Bilateral	61.7	63.0	72.0	64.1
Single	38.3	37.0	28.0	35.9
Donor age, years	33.6	34.6	34.5	33.0
Donor female sex	39.7	42.8	38.5	36.2
Donor pulmonary infection	49.3	44.9	51.1	39.7

Data are means and percentages…

“Intention-to-treat” and post-transplant survival

The median center-level 1-year post-transplant survival rate was 84.1%, and the median center-level ITTS was 66.9%, indicating that one-third of adult lung transplant candidates never achieved one-year survival after lung transplantation. Center-level ITTS correlated with the center-level 1-year post-transplant survival rate (r = 0.58, p<0.001).

On average, there was a large discrepancy (absolute difference) between ITTS and 1-year post transplant survival. ITTS underestimated 1-year post-transplant survival rate by a mean absolute difference of 19.6% (95% limits of agreement 4.1% to 35.1%). Figure 1 shows the 1-year post-transplantation survival rate (black bars), expected ITTS (dark grey bars), and observed ITTS (grey bars) for 30 transplant centers ranked from highest to lowest post-transplant survival rates. Panel A shows the top 10 performing centers. Panel B shows mid-range performing centers. Panel C shows the lowest performing centers. Visual inspection of these plots identifies meaningful discrepancies (absolute differences) between post-transplant survival rates and observed ITTS at most centers, with the greatest discrepancies observed for higher and lower performing centers (Panels A and B) and lesser and more consistent discrepancies among mid-performing centers (Panel B).

Figure 1.

One-year post-transplant survival (black bars), expected “intention-to-treat” survival (dark grey bars), and observed “intention-to-treat” survival (light grey bars) for (A) the ten centers with the highest post-transplant survival rates, (B) ten centers with mid-range post-transplant survival rates, and (C) ten centers with the lowest post-transplant survival rate. Numbers along the x-axis represent each center’s post-transplant survival rank from highest to lowest.

Figure 2A shows the “observed-to-anticipated” (O/A) ratios for each center with 95% confidence intervals (horizontal bars). There was wide variation in O/A ratios, with most ranging from 0.5 to nearly 1.0. All but 10 centers had observed ITTS values that were significantly lower than the survival rate anticipated by patients.

Figure 2.

(A) “Observed-to-anticipated” survival ratio for each center (black squares) and 95% confidence intervals (horizontal lines). Observed survival is “intention-to-treat” survival. Anticipated survival is one-year post-transplant survival. Centers with confidence intervals that exclude a value of 1 (the vertical line) have “intention-to-treat” survival rates that are significantly different from “anticipated.” (B) “Observed-to-expected” survival for each center (black squared) and 95% confidence intervals (horizontal lines). Observed survival is “intention-to-treat” survival. Expected “intention-to-treat” survival is based on a risk prediction model that includes candidate characteristics. Centers with confidence intervals that excludes a value of 1 (the vertical line) have “intention-to-treat” survival rates that are significantly different from “expected.”

Figure 2B shows the “observed-to-expected” (O/E) ratio for each center with 95% confidence intervals. Although there was wide variation in O/E ratios (most centers falling between 0.75 and 1.25), only seven centers had an O/E ratio significantly less than 1, indicating worse than expected ITTS. Two centers had an O/E ratio significantly greater than 1 (better than expected ITTS). After controlling for patient-level factors, 57% of ITTS variability was attributable to variation across centers.

DISCUSSION

We found that a simple measure that incorporates both pre- and post-transplant survival time provides novel information about lung transplant center performance that is likely to be of relevance to patients considering lung transplantation. A substantial number of centers had ITTS values that substantially underestimated patients’ anticipated survival, with 12% of centers having ITTS values that were significantly lower than what might be expected based on age, disease severity, diagnosis, and other candidate factors. This “intention-to-treat” survival measure may be found to be of use to patients, transplant centers, and policymakers when trying to maximize the overall outcomes of patients with advanced lung diseases.

The SRTR publishes PSRs biannually for each organ-specific program at each center. PSRs serve an important role in informing performance improvement efforts and program accreditation, while at the same time the public availability of these reports can sway patients, providers, and third-party payers when selecting transplant centers.⁵ The methodology used to generate PSRs was publically reviewed and critiqued in 2012,³ and has gradually undergone revision,⁶ yet PSRs do not currently include a metric that combines both pre- and post-transplant survival. If included in PSRs, ITTS, or a similar “intention-to-treat” metric, would provide new information not currently available to patients that could transform the manner in which patients perceive lung transplant center performance.

We found that center-specific one-year post-transplant survival rates significantly overestimate ITTS by 20% on an absolute scale, with over 80% of programs’ ITTS values differing significantly from their post-transplant survival rates. Notably, we found that this discrepancy was greatest among centers with either high or low post-transplant survival rates, and lesser among mid-performing centers. The magnitude of the discrepancies at higher performing centers and the similarities of the ITTS among high and mid-performing centers suggest that, on average, overall patient outcomes at centers with high post-transplant survival rates may be quite similar to those at mid-performing centers, suggesting that ITTS may be of value to patients when selecting a transplant center.

Our data suggest that ITTS may convey novel information complementary to the one-year survival rates reported in PSRs. If centers were to be held accountable for both pre- and post-transplant survival, however, it would be critically important to avoid unfairly branding a center as “worse than expected” based on the absolute difference between ITTS and post-transplant survival. In many cases, low ITTS is likely due to issues that are either beneficial to specific patients (such as listing high LAS patients) or issues that are out of the center’s control (such as donor availability). We only superficially developed a model for expected ITTS to control for candidate factors. Yet, this model showed that most centers perform “as expected,” an observation that provides reassurance that centers are, by and large, performing well. If this metric were to be implemented, additional factors such as performance of the local organ procurement organization, and a more rigorously developed prediction model would need to be considered.

Process measures, defined as a “series of actions or functions during the delivery of patient care within the program’s system that result in an organ transplant,” are as important as outcome measures (such as ITTS) in determining transplant center performance.⁷ Indeed, U.S. transplant programs and centers are required to engage in quality assessment and performance improvement activities. These activities may include pre-transplant measures (such as the time from referral to placement on the waiting list and donor acceptance rate), transplant measures (such as ischemic time), and post-transplant measures (such as immunosuppression and documentation of follow-up visits). However, since there are no universal process measures to which all programs are held, process measures are not included in PSRs. In contrast, the Society of Thoracic Surgeons has successfully implemented two process measures (recording of clinical stage and recording of performance status) in their star rating for thoracic surgery programs.⁸ In addition to novel outcome measures, the transplant community should consider including specific process measures in the evaluation of program performance.

Our study had a number of limitations. Just as the focus on post-transplant survival has altered clinical practice by incentivizing transplantation of those expected to have better outcomes, the ITTS metric could further incentivize centers to avoid listing sick patients, particularly during critical illness or when LAS scores are high, settings in which transplantation may still be beneficial.¹ In addition, since not all participants in our study underwent transplantation, our prediction model for expected ITTS included only candidate factors, leaving us unable to control for donor factors or recipient characteristics at the time of transplantation. These factors, however, did not appear to vary meaningfully across centers with low, moderate, and high ITTS values. The exclusion of low volume centers limits the generalizability of ITTS to these centers. We also did not present variation in ITTS over time, since we hoped that combining multiple years of data together would help maintain the anonymity of centers. Finally, our study did not consider process measures and other important endpoints, such as disability and quality-of-life.⁹

In summary, we describe a new metric of lung transplant center performance that captures both pre- and post-transplant survival after lung transplantation. This more inclusive center-specific measure has the potential to provide critical information to patients and may ultimately be found to help transplant centers focus on improving the outcomes of all patients with advanced lung diseases.

ABBREVIATION LIST

ITTS

“Intention-to-treat survival” survival

LAS

Lung allocation score

O/A

Observed-to-anticipated

O/E

Observed-to-expected

PSRs

Program-specific reports

SRTR

Scientific Registry of Transplant Recipients