Development and Validation of Primary Graft Dysfunction Predictive Algorithm for Lung Transplant Candidates

Abstract

Purpose:

Primary graft dysfunction (PGD) is the leading cause of early morbidity and mortality after lung transplantation. Accurate prediction of PGD risk could inform donor approaches and peri-operative care planning. We sought to develop a clinically useful, generalizable PGD prediction model to aid in transplant decision making.

Methods:

We derived a predictive model in a prospective cohort study of subjects from 2012-2018, followed by a single-center external validation. We used regularized (lasso) logistic regression to evaluate the predictive ability of clinically-available PGD predictors and developed a user interface for clinical application. Using decision curve analysis (DCA), we quantified the net benefit of the model across a range of PGD risk thresholds and assessed model calibration and discrimination.

Results:

The PGD predictive model included distance from donor hospital to recipient transplant center, recipient age, predicted total lung capacity, lung allocation score (LAS), body mass index, pulmonary artery mean pressure, sex, and indication for transplant; donor age, sex, mechanism of death, and donor smoking status; and interaction terms for LAS and donor distance. The interface allows for real-time assessment of PGD risk for any donor/recipient combination. The model offers decision making net benefit in the PGD risk range of 10-75% in the derivation centers and 2-10% in the validation cohort, a range incorporating the incidence in that cohort.

Conclusion:

We developed a clinically useful PGD predictive algorithm across a range of PGD risk thresholds to support transplant decision making, post-transplant care, and enrich samples for PGD treatment trials.

Introduction:

Primary graft dysfunction (PGD) affects 2-30% of lung transplant procedures, is the major cause of early mortality, and is a risk factor for chronic lung allograft dysfunction (1-3). While the lung transplant community’s understanding of clinical risk factors for PGD has improved, our ability to quantify patient-level PGD risk remains poor (2-6). In an international survey of lung transplant practitioners, 94% of respondents felt a PGD prediction model would inform peri-operative management (7). An accurate, generalizable PGD prediction model could improve donor approaches, inform peri-operative planning, and enrich clinical trials.

Several recipient and donor risk factors for PGD have been identified (2, 3, 6). We previously utilized those factors to develop a simple risk prediction model including pre-transplant pulmonary diagnosis, body mass index (BMI), pulmonary hypertension, and reported donor smoking history (3, 8). This model had several limitations. First, there was no external validation. Second, it relied on data from 2002-2010, limiting applicability to contemporary candidates due to changes in allocation and greater use of extracorporeal lung support as a bridge to transplant(1). Third, the model was informative over a limited range of PGD risk thresholds (5-20%). In our survey, transplant practitioners reported “acceptable” PGD risk thresholds from 5-45%, suggesting that a model would need to be informative across a wider range.

A significant challenge in predicting PGD is center-level differences in PGD risk thresholds. The goal of any predictive model is to inform a clinical decision – in this case acceptance or deferral of a donor lung for a specific recipient, and to plan advanced therapies such as ECLS. In our prior survey there was variability in risk tolerance among transplant centers: some centers were willing to accept a PGD risk of only 5% while others reported a higher threshold of 45% (7). At low-risk thresholds, a predictive model may provide little additional information or even lead to misclassification of low-risk donor-recipient pairs. At high-risk thresholds, a predictive model may be no better than simply accepting all donor lungs regardless of estimated risk. A clinically useful PGD prediction model must therefore be informative over a broad range of thresholds.

Using the multicenter Lung Transplant Outcomes Group (LTOG) cohort study (NCT00552357), including geographically diverse centers of variable size and PGD incidence, we sought to develop a PGD prediction model with an easy-to-use publicly available interface allowing for real-time risk assessment that supports patient-level decision-making.

Methods:

Study Design and Subject Selection:

LTOG is a prospective multicenter, NIH-sponsored cohort study (3, 8-11). Consented patients age 18-80 years old transplanted at LTOG centers between 2012 and 2018 were included. All recipient, donor, and pre-operative characteristics were obtained prospectively using REDCAP online case report forms and subsequently merged with data from the United Network for Organ Sharing (UNOS). Institutional review boards approved this study at all participating centers. This study used data from the Organ Procurement and Transplantation Network (OPTN). The OPTN data system includes data on all donor, wait-listed candidates, and transplant recipients in the US, submitted by the members of the Organ Procurement and Transplantation Network (OPTN). The Health Resources and Services Administration (HRSA), U.S. Department of Health and Human Services provides oversight to the activities of the OPTN contractor.

Validation Cohort:

We validated in a contemporaneous external cohort of patients from Washington University School of Medicine. We previously utilized this cohort for external validation of BAL complement biomarkers for PGD(12). Recipient, donor and pre-operative characteristics were collected and the Institutional Review Board approved this study.

Definition of PGD:

PGD severity was graded utilizing the 2016 International Society for Heart and Lung Transplantation schema (13). PGD was defined as grade 3 (PaO₂ /FiO₂ ratio <200) at 48 or 72 hours after allograft reperfusion with radiographic abnormalities assessed by two readers with consensus, an extensively validated definition (1, 3, 11-14). Because of the confounding impact of ECMO on measured oxygenation, subjects on ECMO post-transplant were graded as PGD only in the presence of radiographic infiltrates, independent of the reported PaO₂/FiO₂ ratio, while subjects without radiographic infiltrates were excluded [13].

Statistical Methods

Prediction strategies were based on TRIPOD (15).

Predictors

Patient and donor characteristics came primarily from prospective data collection. We supplemented this primary data collection with United Network for Organ Sharing (UNOS) STAR file that was matched, using an encrypted donor identifier, to the study database. Despite use of this approach, there remained a small number of missing values for many variables of interest. We did not impute any missing values; analyses were limited to subjects with complete data.

Model development

In order to maximize clinical utility of the prediction model, we chose predictors that would be readily available at the time of a donor offer and were reasonably associated with PGD based on clinical criteria, prior work, and experience (Supplement Table 1). We reviewed the numbers of missing values and emphasized inclusion of variables with less missing-ness and greater clinical importance (Supplement Tables 2-3). For each continuous covariate, we pre-specified the use of marginal linear splines, to avoid the problems of categorizing continuous covariates and potentially unrealistic assumptions of linear associations with PGD. Splines were created using the R function “lspline”, with the option “marginal=TRUE”, and the Stata function “mkspline”, with the “marginal” option. Knots were specified before analysis and based on prior data. Knots for distance were chosen to correspond to potential changes in mode of transportation with increasing distance (Supplemental Table 2). Once programmed the location of knots remained unchanged. Marginal splines were applied to the following recipient variables: age, pulmonary arterial pressure, BMI, lung allocation score (LAS), and total lung capacity (TLC). Splines for donor factors included: age, calendar quarter of transplant, and distance from donor hospital to transplant center. Date of transplant was fit using marginal splines to avoid assuming linear trends over calendar time.

We began the fitting process with these pre-specified interaction terms: distance from donor hospital to transplant center and LAS, to account for possible correlations between recipient needs and transport distance (16).

Model fitting

Exploratory model fitting began with the inclusion of all covariates in a logistic regression model. This process left only one potential set of interaction terms: twenty terms for LAS and transport distance, modeled with splines.

The next stage involved “regularization”, a process of reducing the number of covariates and shrinking (toward the null) the remaining factors to prevent model overfitting and enhance reproducibility with new data. We used the lasso method as implemented in “glmnet” in R statistical package(17–19). We identified the final model using internal cross validation. This algorithm is a form of “penalization”, in that estimates can shrink to zero, and drop out of the model. This step reduced the number of model covariates for model-based standardization. At this stage, the model did not include transplant center.

Prediction for an individual patient

Given the variation in PGD risk across transplant centers, we devised an approach that recalibrated predictions for individual patients based on the model from all centers to the particular PGD risk at an individual transplant center. In order to do this, we removed the model intercept from the prediction model and added in a center-specific intercept. This center-specific intercept was derived by creating a random intercept model of PGD with center as a random effect and no covariates. We now estimate a patient-level risk of PGD by using this center-specific intercept in the prediction.

Overall model performance

We implemented the analytic and graphical approach of decision curve analysis (DCA) in Stata v17 (StataCorp LLC, College Station, TX)(20, 21) to assess the range of predicted risks over which use of the model would improve net benefit over the alternate decisions of considering all patients to be at high risk or all patients to be at no risk. We also estimated metrics for discrimination (the c-statistic, or area under the curve), and calibration (calibration slope, calibration-in-the-large (CITL), and the ratio of expected to observed outcomes) (22). Calibration is graphically depicted with predicted PGD on the x-axis and observed PGD on the y-axis. CITL is the x-intercept of this calibration curve and represents the extent to which predicted values are too high or too low, with the ideal value equal to zero. The calibration slope represents the slope of the plotted line, with perfect calibration slope=1.

Prediction for individual patient PGD risk at other transplant centers

Given the differences across transplant centers in standardized risks of PGD, we needed a method for allowing a user to input information about a center’s PGD risk as a way to calibrate the model to the user’s center. The user enters center volume and local PGD incidence. Once the user enters the center, patient, and donor characteristics, the application estimates a predicted probability of PGD.

To account for modeling based on splines for prediction, our application requires transformation of all continuous measures to their spline equivalents. This computation task means that users must rely on a computer-based program. We developed an “app”, using R’s shiny software, that translates userentered information to estimate a predicted PGD risk.

Results

Cohort Patient Characteristics:

During the study period, there were 3831 transplants performed at participating centers, 1527 subjects enrolled in the cohort, and 1167 with complete data (Supplemental Figure 1). Subject characteristics are presented in Table 1. PGD incidence overall was 26.3% (307/1167) and varied more than 5-fold across the centers (8.6% (6/70) to 44.7% (46/108)).

Table 1:

Study subject characteristics according to PGD status of the recipient

	PGD (307)	No PGD (860)	Total Study Population (1167)
Recipient
Sex (Male)	169 (55)	517 (60)	686 (59)
Age, years	55.0	56.8	56.3
BMI, kg/m2	25.9	24.7	25.0
Race
Asian/Asian American	14 (5)	15 (2)	29 (2)
African American	39 (13)	52 (6)	91 (8)
Caucasian	249 (81)	784 (91)	1033 (89)
Other	5 (2)	9 (1)	14 (1)
Ethnicity (Hispanic/Latino)	16 (5)	24 (3)	40 (3)
Diagnosis
COPD	48 (16)	248 (29)	296 (25)
IPF	102 (33)	260 (30)	362 (31)
Non-IPF ILD	72 (23)	146 (17)	218 (19)
CF	33 (11)	124 (14)	157 (13)
PAH	22 (7)	13 (2)	35 (3)
Other	30 (10)	69 (8)	99 (8)
Lung allocation score	52.62	46.72	48.27
Transplant Type
Single	65 (21)	258 (30)	323 (28)
Bilateral	241 (79)	601 (70)	842 (72)
Other	1 (1)	1 (1)	2 (1)
TLC, liters (Mean)	6.0	6.2	6.1
Mean PAP, mmHg	45.0	41.3	42.3
Donor
Sex (Male)	175 (57)	521 (61)	696 (60)
Age, years	37.8	35.8	36.3
Race
Asian/Asian American	15 (5)	21 (2)	36 (3)
African American	55 (18)	175 (20)	230 (20)
Caucasian	209 (68)	601 (70)	810 (69)
Other	28 (9)	63 (7)	91 (8)
Ethnicity (Hispanic/Latino)	44 (14)	102 (12)	146 (13)
Smoking History (Yes)	145 (47)	382 (44)	527 (45)
Mode of Death
Head Trauma	121 (39)	320 (37)	441 (38)
Stroke/Cerebrovascular	101 (33)	260 (30)	361 (31)
Anoxia	75 (24)	258 (30)	333 (29)
Other	10 (3)	22 (3)	32 (3)
Distance to Donor	234.0	231.7	232.3
Hospital (Nautical Miles) (Mean)
Ischemic Time (Hours) (Mean)	6.3	5.9	6.0

Continuous variables reported as mean. Categorical variables reported as n(%).

Definition of Abbreviations:

PAP: pulmonary artery pressure

PGD: primary graft dysfunction

BMI: body mass index

COPD: Chronic Obstructive Pulmonary Disease

IPF: Idiopathic Pulmonary Fibrosis

ILD: Interstitial Lung Disease

CF: Cystic Fibrosis

PAH: Pulmonary Arterial Hypertension

LAS: Lung Allocation Score

PGD Predictive Model:

Model coefficients are available in the Supplement (Supplemental Table 4). DCA describes whether the model performs better than alternatives of classifying everyone or no one as high PGD risk as well as the net benefit of the model across a range of decision thresholds. DCA performed for each center based on the final predictive model demonstrates that the model offers decision making net benefit in PGD risk threshold ranges of 10- 75% (Figure 1, Supplemental Figure 2). The predictive model offers acceptable calibration (calibration slope=1.04, CITL =−0.5) and discrimination (c-statistic=0.76) (Figure 2).

Figure 1.

Predictive performance of the model for all transplant centers. Net benefit compares the number of correct predictions (true positive predictions) with the number of cases of positive predictions among those without PGD, across all possible risk thresholds. Over a wide range of risk thresholds, the net benefit of using the prediction model (blue dotted line) is higher than alternative decision rules of treating all patients as high risk (solid black curve) or at low risk (red dot-dash line). The area between the curves represents the improved risk determination from use of the prediction model. The model demonstrates net benefit in the PGD risk threshold ranges of 10-75%.

Figure 2.

Calibration curve for prediction model for primary graft dysfunction (PGD) for all transplant centers. Ideally the calibration curve (smooth curve) should coincide with the 45 degree (dotted) line that equates expected (E) and observed (O) risk, CITL (calibration in the large) should be 0.0 and the curve slope should be 1.0. The model performance is only slightly deflected downward, suggesting a small degree upward deflection of predicted risk, a conservative result for clinical applications. The AUC =0.76 (c-statistic), indicating a reasonable level of model discrimination, the ability of the model to distinguish higher from lower PGD risk. Reasonable discrimination is also reflected in the distribution of predictions by PGD status along the x-axis, where the predictions for PGD=0 are lower than those for the patients with PGD=1.

After developing our model in LTOG, we next aimed to generalize the model to subjects at nonparticipating sites. We accounted for center-specific effects based on two factors – reported PGD incidence and annual lung transplant volume. We then developed a straightforward web-based interface allowing the user to input center, donor, and recipient characteristics to produce a predicted likelihood of PGD (Supplemental Figure 3). The PGD calculator can be found at: https://shiny.pmacs.upenn.edu/PGD_Calculator.

Validation Cohort:

Characteristics of the external validation cohort were similar to LTOG (Supplement Table 5). The incidence of PGD in the external validation cohort was lower than in LTOG (7% vs 26%). As the model utilizes site-specific inputs, it accounts for observed differences in PGD incidence between the average for the development cohort and for the validation center. The validation cohort DCA (Figure 3) shows that the predictive model demonstrates net benefit in the PGD incidence range 2-10%, incorporating the PGD incidence seen in the validation cohort. In the validation cohort, the predictive model offers reasonable calibration (calibration slope=0.5, calibration in the large =−0.6) and discrimination (c-statistic=0.66) (Figure 4).

Figure 3.

Decision curve for the validation sample (n=321). The model demonstrates net benefit in the PGD incidence range 2-10%, consistent with the observed PGD incidence at this center (7%).

Figure 4.

Calibration curve for the validation center sample. Ideally the calibration curve (smooth curve) should coincide with the 45 degree (dotted) line that equates expected (E) and observed (O) risk, CITL (calibration in the large) should be 0.0 and the curve slope should be 1.0. The model demonstrated reasonable calibration (E/O=1.6) and discrimination (c=0.66).

Discussion:

We developed a clinically useful PGD predictive algorithm that improves PGD risk prediction across a wide range of risk thresholds using readily-available pre-transplant data, based on contemporary lung transplant practices. The model may be useful in transplant decision-making and post-transplant care, and to enrich PGD prevention and treatment trials. This predictive algorithm, which exhibits good discrimination and improved net benefits, builds on prior models, identifying novel predictors of PGD, including LAS, total lung capacity, and donor distance from transplant center. The model employs innovative statistical methods to avoid overfitting and the web enabled app converts the results of a complex model into usable information for real time decision support. To address the variability of PGD across transplant centers, and to deal with issues of model calibration arising from this variability, the prognostic tool allows the user to start with the center average PGD risk and then add recipient and donor level factors to improve prediction for the individual patient.

We believe this model has important real-world utility for several reasons. First, there is demand for a clinically-useful but flexible prediction model. Our survey of lung transplant practitioners found that 75% of respondents felt that a predictive model for PGD would influence the decision to proceed with transplant while 94% believed that such a model would inform the peri-operative management plan (7). Second, the development of a publicly available user-oriented interface and the inclusion of readily-available data elements make this model clinically-useful for real-time donor selection decisions. For patients with a higher-than-average risk of PGD at that center, clinicians can now prepare patients, their families, and the care team for an increased likelihood of prolonged hospital and intensive care unit length of stay and even inform discussions of controlled institution of ECLS as a bridge to recovery (2). Third, we demonstrate net benefit of our model across a range of PGD risk thresholds. While our external validation cohort had a markedly lower PGD incidence, our use of center-specific inputs allowed our model to still demonstrate a net benefit of the prediction tool in a range of risks present in the validation center. Fourth, the inclusion of center-related PGD risk as part of the prediction tool is a key improvement over prior models given the wide variability in PGD incidence across centers. This may also account for differences in intra-operative and peri-operative management that influence between-center variability in PGD risk. Regardless of the underlying reason for across-center variability, the tool adds information for clinical decision making across the spectrum of patients at a particular center. Fifth, identification of treatments for PGD requires identifying and targeting high-risk recipients. Our model can support the design of clinical trials based on prognostic enrichment, similar to proposals for improved trial design in patients with acute respiratory distress syndrome (23).

Included recipient variables largely replicate known risk factors for PGD. Pulmonary artery pressure, BMI, transplant year, and diagnosis are well-described risk factors for PGD.(3, 11) While the ratio of donor to recipient predicted TLC is associated with PGD,(4) our model includes only recipient pTLC. We considered both recipient and donor pTLC for model inclusion, however recipient pTLC was the more robust predictor. Recipient pTLC may account for additional factors that influence PGD risk including underlying lung disease or age. Decreases in sample size when including both recipient and donor pTLC may have affected model performance. In creating a predictive model, we focused on maximizing overall model performance rather than quantifying the association between an individual variable and PGD; the inclusion or absence of a variable (e.g. donor pTLC), or the magnitude of the coefficient, should not alter our mechanistic understanding of PGD.

The LAS system, in place since mid-2005, utilizes estimates of differential survival with and without lung transplantation to calculate net transplant benefit (24). Since its implementation, the proportion of listed candidates with scores less than 35 declined while the proportion with scores above 50 increased (25). At the same time, studies demonstrate increased post-transplant mortality in the highest LAS patients overall and within diagnostic subgroups (26, 27). Transplantation of high LAS patients is associated with increase healthcare utilization, including increased post-transplant utilization of ECMO and tracheostomy, care practices strongly associated with management of severe PGD (28–30). While LAS is a non-modifiable risk factor for PGD, early prognostication of high PGD risk allows for earlier implementation of post-transplant ECLS support, which may be associated with improved outcomes given the increased mortality associated with later ECLS implementation (29).

Changes in the lung allocation system in November 2017, eliminating the donor service area as the first unit of organ allocation, altered transplant practice patterns, resulting in increased travel distances, longer organ ischemic times, and potentially increased one year post-transplant mortality (31–33). In order to account for some of these changes, we included distance from donor hospital in our models. Distance is likely a marker of multiple factors, including ischemic time, center-level differences in organ acceptance patterns related to travel, and other factors that are difficult to capture (cost, time, increased turn-down rate). Inclusion of distance from donor hospital in PGD prediction algorithms is novel because this measure is not classically reported or captured in studies, incorporates travel time, and performed well in our model. The implementation of a continuous distribution model for organ allocation may alter the importance of distance in PGD risk estimations and models may require updating (34). Different approaches to organ transport outside of the US may alter the generalizability of this finding.

There remain some limitations and room for improvement in PGD predictive modeling. The causes of between-center variability in PGD risk remain poorly understood. Our model relies predominantly on recipient factors known prior to allograft implantation with relatively few data inputs from the donor due to the absence of detailed donor information on time-varying parameters. We believe that PGD risk prediction is useful when a center is choosing whether to accept a donor offer. We therefore limited our model to variables available at that moment in time. Future work could develop prediction models that include intra-operative or post-operative variables to inform post-operative clinical management or prognosis. Development of a lung specific donor risk index incorporating dynamic and extended variables could improve our ability to safely match donors with more appropriate recipients. While the kidney donor profile index was developed to predict long-term allograft failure risk, it also improves prediction of delayed graft function, the renal corollary to PGD (35, 36). Similarly, the development of a robust scoring system for donor lungs might improve donor-recipient risk matching, enhance PGD prediction at the time of organ allocation, and more appropriately adjust expected survival models. Our model was developed and validated in patients undergoing transplant in the United States. It may not be generalizable to other populations or allocation systems. Future efforts should focus both on external validation of the proposed prognostic model and the development of data collection and sharing systems with larger and more heterogeneous populations. Expansion of existing datasets to include additional variables including surgical preferences, surgical techniques, and mode of transportation of donor organ could improve future prognostic models. Despite exhaustive efforts to fill in missing-ness, our variable selection was limited by cumulative missing-ness across relevant clinical variables. It is possible that variable missing-ness was not random, and therefore we recommend some caution in interpretation of our results. We used a lasso approach to regularization in order to allow coefficients to reach zero and thereby minimize model complexity. Future work could consider alternative approaches.

Changes in the lung allocation system may alter the population of candidates preferentially transplanted. The LAS was recently replaced with the composite allocation score (CAS). While the CAS will utilize the same recipient characteristics, its calculations differ in ways that may alter who undergoes transplantation. First, while the LAS weighted wait-list to post-transplant survival in a 2:1 ratio, the CAS will weight them equally. Second, while the LAS estimated 1-year post-transplant survival, the CAS estimates 5-year post-transplant survival.(37, 38). Third, the CAS will incorporate level of human leukocyte antigen (HLA) sensitization and extremes of recipient size, and travel and proximity efficiency in donor matching. How these allocation changes will alter who undergoes transplantation, is unknown. Future PGD prediction models will need to re-evaluate model performance in this new population and to account for changes in PGD incidence over time.

Finally, prior studies demonstrate the potential benefits of HLA matching donors and recipients on bronchiolitis obliterans syndrome-free and transplant-free survival (39–41). There is a paucity of data on the impact of HLA mismatch on PGD. A recent single center study showed increasing PGD severity with increasing HLA-DQ and eplet mismatches (42). Others identified increasing PGD risk with higher levels of pre-existing high mean fluorescent index anti-HLA antibodies (43). Total calculated percent reactive antibody levels did not significantly add to our PGD predictive models, but patient level data on specific HLA epitopes and specificities of circulating anti-HLA antibodies were not available. Future studies should attempt to incorporate the expanding knowledge of donor-recipient HLA matching into prognostic models. More discrete data on HLA epitope matching will be available from the CAS system and should be available for updates.

In conclusion, we developed a practical predictive algorithm and web-based graphical user interface for determining PGD risk based on donor and recipient characteristics. Future refinements should focus on improving donor risk determination by expanding variables, improving donor-recipient matching, and an enhanced understanding of the impact of the CAS on PGD risk. Understanding different PGD risk subgroups could improve enrichment strategies for PGD treatment trials.

Abbreviations:

PGD

primary graft dysfunction

DCA

decision curve analysis

LAS

lung allocation score

BMI

body mass index

ECLS

extracorporeal lung support

LTOG

lung transplant outcomes group

ECMO

extracorporeal membrane oxygenation

UNOS

United Network for Organ Sharing

TLC

total lung capacity

CITL

calibration in the large

KDPI

kidney donor profile index

CAS

composite allocation score

TRIPOD

transparent reporting of a multivariable prediction model for individual prognosis or diagnosis

HLA

human leukocyte antigen