The Lung Allocation Score and Other Available Models Lack Predictive Accuracy for Post-Lung Transplant Survival

Abstract

Background:

Improved predictive models are needed in lung transplantation in the setting of a proposed allocation system that incorporates longer-term post-transplant survival in the United States. Allocation systems require accurate mortality predictions to justly allocate organs.

Methods:

Utilizing the United Network for Organ Sharing database (2005–2017), we fit models to predict 1-year mortality based on the Lung Allocation Score (LAS), the Chan, et al., 2019 model, a novel “clinician” model (a priori clinician selection of pre-transplant covariates), and two machine learning models (LASSO and Random Forests) for predicting 1-year and 3-year post-transplant mortality. We compared predictive accuracy among models. We evaluated the calibration of models by comparing average predicted probability versus observed outcome per decile. We repeated analyses fit for 3-year mortality, disease category, including donor covariates, and LAS era.

Results:

The area under the cure (AUC) for all models was low, ranging from 0.55 to 0.62. All exhibited reasonable negative predictive values (0.87–0.90), but the positive predictive value for was poor (all PPV <0.25). Evaluating LAS calibration found 1-year post-transplant estimates consistently overestimated risk of mortality, with greater differences in higher deciles. LASSO, Random Forests, and clinician models showed no improvement when evaluated by disease category or with the addition of donor covariates and performed worse for 3-year outcomes.

Conclusions:

The LAS overestimated patients’ risk of post-transplant death, thus underestimating transplant benefit in the sickest candidates. Novel models based on pre-transplant recipient covariates failed to improve prediction. There should be wariness in post-transplant survival predictions from available models.

Introduction

Lung transplant (LTx) can improve quality of life and increase survival for patients with advanced lung disease. However, many potential candidates for LTx die before organs become available, with waiting list mortality (14.6%) the highest of any solid organ¹ and donor supply being the rate limiting resource in providing this effective therapy.²

Greater than 60% of LTx worldwide³ utilize the Lung Allocation Score (LAS).⁴ In designing the LAS, quantitative variables from the Organ Procurement and Transplantation Network (OPTN) Scientific Registry of Transplant Recipients (SRTR) database were selected to create disease-specific models to predict 1-year mortality without transplant, i.e. waitlist mortality, and 1-year post-transplant survival.⁵ The LAS is the difference between predicted waitlist survival (weighted double) and post-transplant survival, net transplant benefit,⁶ normalized to a range from 0–100. Lungs are prioritized to those with higher scores. Its 2005 implementation in the United States resulted in decreased waiting list mortality, increased LTx rates, and an initial small improvement in 1-year post-transplant survival.⁷

Despite the improvements seen in the LAS era, the score’s ability to allocate lungs justly remains highly dependent on its performance in predicting pre- and post-transplant outcomes. This, and an interest in broader organ sharing, have led to re-examination of the current allocation system. Recent literature suggests a need for improved predictive models. A 2021 United Network for Organ Sharing (UNOS) proposal suggested transition towards utilizing longer-term survival in order to better reflect meaningful transplant benefit.^8,9 In this study, our primary aim was to assess the accuracy of the LAS and other models for predicting 1-year post-transplant mortality. In addition to refitting the Houston Methodist model by Chan, et al.,¹⁰ we also developed and validated novel models: a “clinician” model based on a priori selection of covariates by transplant pulmonology experts and two statistical machine learning models based on Least Absolute Shrinkage and Selection Operator (LASSO) and Random Forests, respectively. Finally, at a time when the United States allocation system transitions towards estimating longer-term survival,^9,11 our secondary aim was to evaluate the predictive performance of all models by considering 3-year post-transplant outcomes, fit by disease category, with inclusion of donor covariates, or by LAS era.

Materials and Methods

Study Design and Participants

This retrospective cohort study included adult patients (≥18-years-old) from the UNOS dataset provided by the OPTN who had their first LTx between May 1, 2005, and May 1, 2017. UNOS follow-up data was available through mid-June, 2020. Patients lost to follow-up were designated as such in the UNOS dataset (n=309, “UNOS PX_STAT”) and were removed from our analyses. The University of Washington IRB approved this study (#9226). This study complied with the ISHLT ethics statement.

Variables of Interest

Our initial models considered all pre-transplant recipient covariates from the UNOS dataset. A complete list of UNOS variables is included in the online supplement (Table S-1). For the clinician model, 27 predictors of interest were selected a priori by transplant pulmonology experts (co-authors KJR, SGK, and CAM). These covariates included transplant year, forced expiratory volume in one second (FEV₁) percent predicted, forced vital capacity (FVC) percent predicted, six minute walk (SMW) distance, oxygen requirement, diagnosis, mean pulmonary arterial pressure, arterial partial pressure of carbon dioxide (PCO₂), body mass index (BMI), albumin, total bilirubin, creatinine, dialysis, mechanical ventilation at time of transplant, hospitalization status at time of transplant, sex category, education, diabetes, waitlist time, cigarette use, functional status, insurance, cytomegalovirus (CMV) status, glomerular filtration rate (GFR), steroid usage, bilateral versus single LTx, and multi-organ transplant (Table 2). Age was removed from the clinician model due to collinearity with diagnosis. We then utilized the statistical machine learning approaches, LASSO¹² and Random Forests,¹³ to identify covariates of significance from the same UNOS dataset (Table 3 and Figure 3).Statistical significance for a priori selected covariates in the clinician model was adjusted for multiple comparisons using Bonferroni correction of the p-values. Analyses were performed in R version 3.6.3.

Table 2:

Coefficients from the Clinician Model

	Unit	Odds Ratio (95% CI)	Raw P-value	Bonferroni P-value
Female		0.77 (0.70, 0.85)	<0.01	<0.01
Education	High School (Baseline)
	None	1.25 (0.42, 3.76)	0.73	1.00
	Grade School	0.93 (0.70, 1.25)	0.64	1.00
	Some College	0.90 (0.80, 1.01)	0.08	1.00
	College Graduate	0.91 (0.80, 1.03)	0.14	1.00
	Post Graduate	1.02 (0.87, 1.20)	0.81	1.00
	Unknown	1.02 (0.86, 1.21)	0.82	1.00
Insurance	Private (Baseline)
	None	0.70 (0.36, 1.37)	0.30	1.00
	Medicare	1.11 (1.01, 1.23)	0.03	1.00
	Medicaid	0.88 (0.72, 1.07)	0.19	1.00
	Other	0.79 (0.60, 1.05)	0.10	1.00
BMI	Normal (18.5–25) (Baseline)
	Low (<18.5)	1.45 (1.22, 1.73)	<0.01	<0.01
	High (>25 – 30)	1.09 (0.98, 1.22)	0.11	1.00
	Obese (>30)	1.02 (0.89, 1.17)	0.74	1.00
Diagnosis	COPD (Baseline)
	CF	0.70 (0.56, 0.87)	<0.01	0.08
	IPF	1.03 (0.87, 1.22)	0.73	1.00
	Other	1.08 (0.91, 1.27)	0.37	1.00
Functional Status	No Assistance (Baseline)
	Some Assistance	1.18 (0.93, 1.51)	0.18	1.00
	Total Assistance	0.99 (0.72, 1.35)	0.95	1.00
	Unknown	1.58 (1.22, 2.05	<0.01	0.02
Medical Condition	Not Hospitalized (Baseline)
	Hospitalized	1.32 (1.13, 1.54)	<0.01	0.03
	ICU	2.00 (1.67, 2.38)	<0.01	<0.01
Dialysis Prior to LTx	No (Baseline)
	Yes	2.12 (1.29, 3.48)	<0.01	0.15
	Unknown	0.85 (0.76, 0.94)	<0.01	0.13
Steroids	No (Baseline)
	Yes	1.06 (0.97, 1.16)	0.22	1.00
	Unknown	1.03 (0.77, 1.37)	0.86	1.00
Cigarette Use	No (Baseline)
	Yes	0.91 (0.82, 1.01)	0.09	1.00
	Unknown	1.05 (0.75, 1.45)	0.79	1.00
CMV	Negative (Baseline)
	Positive	1.00 (0.91, 1.09)	0.94	1.00
	Unknown	1.07 (0.85, 1.33)	0.58	1.00
Ventilator Use		1.38 (1.14, 1.68)	<0.01	0.05
Wait List Time	100 days	1.02 (1.00, 1.03)	0.02	0.12
Multi Organ Transplant		1.49 (0.72, 3.08)	0.29	1.00
O2 Used	1L	1.00 (0.99, 1.01)	0.73	1.00
FEV1 Percent Predicted	5%	1.00(0.98, 1.02)	0.76	1.00
FVC Percent Predicted	5%	0.99 (0.97, 1.00)	0.13	1.00
Six Minute Walk	100ft	0.98 (0.97, 1.00)	0.01	0.41
GFR	5 mL/min	0.97 (0.96, 0.98)	<0.01	0.01
LAS	1 unit	1.00 (0.99, 1.00)	0.33	1.00
Mean Pulmonary Arterial Pressure	1 mmHg	1.01 (1.00, 1.01)	0.02	1.00
Serum Albumin	1 g/dL	0.81 (0.75, 0.88)	<0.01	<0.01
Total Bilirubin	1 g/dL	1.08 (1.04, 1.11)	<0.01	<0.01
Serum Creatinine	1 mg/dL	1.02 (0.85, 1.21)	0.86	1.00
PCO2	1 mmHg	1.00 (0.99, 1.00)	0.01	0.64

The risk score from the clinician model can be calculated by taking the natural log of the odds ratios and summing over each row to get a composite score for each patient.

Table 3:

Coefficients from the LASSO Model

	Unit	Odds Ratio
Female		0.93
Ventilator		1.02
Age	1 year	1.01
Six Minute Walk	100 ft	0.99
Medical Condition	Not Hospitalized (Baseline)
	Hospitalized	1.28
	ICU	1.58
GFR	5	0.96
Serum Albumin	1 g/dL	0.77
Total Bilirubin	1 g/dL	1.10
Serum Creatinine	1 mg/dL	1.20
PCO2	Normal (Baseline)
	Low	1.00
	High	0.93
Waitlist Time	100 days	1.35
Dialysis Prior to Transplant	No (Baseline)
	Yes	1.00
	Unknown	0.92
EBV	Negative (Baseline)
	Positive	0.98
	Unknown	1.00
BMI	Normal (Baseline)
	Low	1.04
	High	1.04
	Obese	1.04
Steroid	No (Baseline)
	Yes	1.02
	Unknown	1.02
Functional Status	No Assistance (Baseline)
	Some Assistance	1.00
	Total Assistance	1.00
	Unknown	1.25

The risk score from the LASSO model can be calculated by taking the natural log of the odds ratios and summing over each row to get a composite score for each patient.

Figure 3:

The 15 most important variables (by mean decrease in Gini index) from the Random Forests model

Outcomes of Interest

In designing our new models, we used death or re-transplant within the indicated timeframe as the outcome of interest.

Model Development

A random 85% of patients were selected to form a development set to fit the clinician, LASSO, and Random Forests models. The remaining 15% formed the validation set. We felt this split balanced the tradeoff between model development and evaluation precision. We fit logistic regression models on the development set with covariates chosen using the methods denoted above. Additionally, we evaluated the Chan, et al.,¹⁰ model, which used patients age, diagnosis, BMI, diabetes, total bilirubin, GFR, cardiac index, and SMW to develop a risk score.

For binary and categorical variables, patients with missing measurements were grouped into their own category. Missing measurements for continuous variables were estimated using mean imputation.

Primary Analysis

On the validation set, we compared the clinician, LASSO, and Random Forests models to the LAS at time of transplant and the Chan, et al,¹⁰ model. Prediction performance of the five different models was assessed by the area under the receiver-operator characteristic (ROC) curve (AUC), sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) using the cutoff value that maximized combined sensitivity and specificity. Calibration of each model was evaluated by calibration slope¹⁴ using logistic regression with the predicted probability as the independent variable and the observed outcome as the dependent variable.

Secondary Analyses

We next explored various methods to improve the predictive performance of the prognostic models by repeating the primary analysis: 1. fitting separate models on each disease [cystic fibrosis (CF), chronic obstructive pulmonary disease (COPD), idiopathic pulmonary fibrosis (IPF), and other] individually; 2. including donor covariates; and 3. evaluating 3-year instead of 1-year post-transplant survival.

To account for potential changes over time, prediction performance of the five models was also analyzed by LAS era, 2005–2010, 2011–2014, and 2015–2017. Calibration of the LASSO, Random Forests, and clinician models was repeated for each LAS era as well. Notably, the patient cohort to determine the LAS was updated in 2012¹⁵ and the LAS equation was updated in 2015.¹⁶

Results

Cohort Characteristics

The UNOS dataset included 19,900 adults who received their first LTx between May 1, 2005, and May 1, 2017. Of the total cohort, 2,765 (14%) died and 171 (1%) were re-transplanted within one year. Compared to those who survived at least one year, those who died or were re-transplanted were more likely to have been male, have only a single lung transplanted, use pre-transplant mechanical ventilation, have had dialysis prior to LTx, and used pre-transplant steroids (Table 1). They were also more likely to be transplanted for IPF, hospitalized or in the intensive care unit, have high or low BMI, or required total assistance. Patients who died were older and had higher LAS, oxygen requirements, mean pulmonary arterial pressure, total bilirubin, and serum creatinine but lower SMW distance, serum albumin, and GFR.

Table 1:

Descriptive Statistics

Variable	Unit	Alive without re-transplant after 1 years (n=16,964)		Death or re-transplant within 1 years (n=2,936)		p-value
		N	%	N	%	Raw	Bonferroni
Gender	Female	7,019	41%	1,090	37%	<0.001	<0.001
Mechanical Ventilation	Yes	930	5%	314	11%	<0.001	<0.001
Number of Lungs	Both Lungs	11,604	68%	1,884	64%	<0.001	1.9e-<0.001
Multi-Organ Transplant	More than one	50	<1%	15	<1%	0.085	>0.99
Medical Condition	ICU	1,171	9%	831	13%	<0.001	<0.001
	Hospitalized	1,148	8%	595	9%
	Not Hospitalized	11,293	83%	4,862	77%
Diabetes	None	11,027	81%	4,972	79%	0.3	>0.99
	Type I	431	3%	240	4%
	Type II	1,551	11%	789	13%
	Other	531	4%	242	4%
	Unknown	72	1%	45	1%
Dialysis prior to LTx	Yes	52	<1%	46	1%	<0.001	<0.001
	No	9,399	69%	4,621	73%
	Unknown	4,161	31%	1,621	26%
Cigarette Use	Yes	8,053	59%	3,722	59%	0.86	>0.99
	No	5,252	39%	2,400	38%
	Unknown	307	2%	166	3%
Steroid	Yes	6,117	45%	3,051	49%	<0.001	<0.001
	No	7,212	53%	3,104	49%
	Unknown	283	2%	133	2%
CMV Status	Positive	7,415	54%	3,437	55%	0.23	>0.99
	Negative	5,673	42%	2,603	41%
	Unknown	524	4%	248	4%
Functional Status	No assistance	632	5%	244	4%	<0.001	<0.001
	Some assistance	9,282	68%	3,936	63%
	Total Assistance	744	5%	443	7%
	Unknown	2,954	22%	1,665	26%
Race	White	11,316	83%	5,239	83%	0.86	>0.99
	Black	1,184	9%	552	9%
	Asian	212	2%	90	1%
	Pacific Islander	10	<1%	7	1%
	American Indian	46	<1%	16	<1%
	Hispanic	804	6%	370	6%
	Multi-Racial	46	<1%	17	<1%
Insurance	Private	7,250	53%	3,198	51%	0.12	>0.99
	Medicaid	873	6%	449	7%
	Medicare	4,970	37%	2,448	39%
	Other	447	3%	165	2%
	None	72	<1%	28	<1%
Education	No High School	337	2%	161	3%	0.2	>0.99
	High School or GED	4,936	36%	2,349	37%
	Attended College	3,446	25%	1,575	25%
	Associate/Bachelor	2,742	20%	1,111	18%
	Post-College Graduate	1,217	9%	589	9%
	Unknown	918	7%	492	8%
Disease	CF	1,697	12%	702	11%	<0.001	<0.001
	COPD	4,028	30%	1,723	27%
	IPF	5,201	38%	2,588	41%
	Other	2,686	20%	1,275	20%
BMI	Very Low (<17)	317	2%	206	3%	<0.001	0.016
	Low (17–18.5)	733	5%	378	6%
	Normal (18.5–25)	5,359	39%	2,252	36%
	Overweight (25–30)	5,055	37%	2,366	38%
	Obese (>30)	2,128	16%	1,076	17%
	Unknown	20	<1%	10	<1%
		N	Mean (SD)	N	Mean (SD)
Age	Years	16,964	55.1 (13.1)	2,936	56.5 (13.1)	<0.001	<0.001
Lung Allocation Score		16,102	41.5 (14.6)	2,727	43.7 (16.5)	<0.001	<0.001
FEV1 percent predicted		16,687	38.4 (20.9)	2,853	39.7 (20.2)	0.002	0.052
FVC percent predicted		16,748	48.6 (17.5)	2,865	47.96 (18.0)	0.093	>0.99
Six Minute Walk		16,238	812.5 (407.0)	2,764	764.1 (425.0)	<0.001	<0.001
Pulmonary Arterial Pressure		15,577	27.6 (10.2)	2,652	28.5 (11.2)	<0.001	0.004
PCO2		15,064	47.9 (13.6)	2,534	47.2 (14.1)	0.017	0.49
Serum Alb		11.542	3.9 (0.6)	2,118	3.8 (0.7)	<0.001	<0.001
Total Bilirubin		16,764	0.61 (0.91)	2,892	0.78 (1.70)	<0.001	<0.001
Serum Creatinine		16,934	0.85 (0.43)	2,928	0.90 (0.43)	<0.001	<0.001
Time on Waiting List	Days	16,964	196.1 (368.0)	2,936	205.3 (409.0)	0.25	>0.99
Oxygen Requirement		16,521	5.1 (4.8)	2,841	5.8 (5.5)	<0.001	<0.001
GFR		16,964	93.4 (23.2)	2,936	89.6 (26.0)	<0.001	<0.001

Differences between binary variables was tested using a test of proportions. For categorical variables, the difference in distributions was tested using a chi-square test. For continuous variables, the difference in means was tested by a t-test. We report both the p-value for each test individually (Raw), and the Bonferroni adjusted p-value (Bonferroni).

Assessment of Diagnostic Accuracy in the Validation Cohort

While the clinician and Random Forests models showed significant improvements in AUC over the LAS (Figure 1), the AUC for all models were low, ranging from 0.55 to 0.62. All exhibited reasonable NPV (0.87–0.90), but the PPV for was poor (all PPV <0.25). Calibration plots (Figure 2) for the clinician and Random Forests models suggested that they can identify those at the highest and lowest risks of 1-year death or re-transplant with reasonable calibration, but do not discriminate patients well in the middle quantiles. The calibration slope for the LASSO model (1.45, 95% CI [1.10, 1.80]) indicated poor calibration, where risk is underestimated for high-risk patients and overestimated for low-risk patients. No significant evidence for poor calibration was seen among the clinician (0.85, 95% CI [0.67, 1.10]) and the Random Forests models (0.96, 95% CI [0.76, 1.16]).

Figure 1:

1-year survival receiver-operator curves for the LASSO (red), Random Forests (green), clinician (blue), Chan, et al.,¹⁰ (orange), and LAS (grey). Specificity (Spec), sensitivity (Sens), positive predictive value (PPV), and negative predictive value (NPV) for each model at cut-points selected to maximize the total sensitivity and specificity for each model are displayed in the table.

Figure 2:

Calibration of the LASSO, Random Forests, and clinician models at predicting 1-year survival. Patients are sorted into deciles, and each dot represents one decile. The x-axis represents the average predicted probability of death or re-transplant over each decile, while the y-axis shows the observed proportions (with standard errors) of death or re-transplant in each decile. The dotted line represents ‘perfect calibration’, where the predicted probabilities of death match the observed percentages. The density plot below each scatter plot shows the distribution of predicted probabilities of death or re-transplant for each model.

Among the 27 variables considered in the clinician model, only gender, medical condition (ICU, hospitalized, or not hospitalized), BMI, GFR, serum albumin, and total bilirubin were found to be significantly associated with death or re-transplant in the multivariable model (Table 2). The LASSO model, which considered all variables available in UNOS, selected the same six variables along with age, mechanical ventilation, SMW distance, PCO₂, waitlist time, dialysis prior to LTx, EBV, steroid usage, functional status, and serum creatinine (Table 3). Random Forests, which allows for non-linearity and interactions between covariates, identified GFR, SMW distance, and BMI to be most important in predicting death or re-transplant along with smaller importance for time on waitlist, weight, age, FVC, hemodynamics cardiac (Figure 3).

The LAS Post-Transplant Survival Measurement

LAS post-transplant survival measures were available for patients who underwent LTx after February 2015, of which there were 3,964 in the training set and 706 in the validation set. While there was no statistically significant evidence for poor calibration slopes among LASSO (1.27, 95% CI [0.29, 2.26]), clinician (0.94, 95% CI [0.35,1.53]), and Random Forests (0.90, 95% CI [0.38, 1.43]) models, the confidence intervals were wide indicating large uncertainty on their accuracy (Figure 4). The calibration slope for the LAS post-transplant survival measure (0.38, 95% CI [0.03, 0.73]) indicated poor calibration, with the LAS overestimating the risk of post-transplant mortality (Figure 4). Evaluating the LAS post-transplant survival at 3-, 6-, 9-, and 12-months demonstrated that the risk of death was overestimated at each interval (Figure 5).

Figure 4:

Calibration of the LASSO model (top left), Random Forests (top right), clinician model (bottom left), and LAS Post Transplant Survival Probability (bottom right) at predicting 1-year survival among patients who were transplanted from 2015–2017 with an available LAS post-transplant survival measure. Patients are sorted into deciles, and each dot represents one decile. The x-axis represents the average predicted probability of death or re-transplant over each decile, while the y-axis shows the observed proportions (with standard errors) of death or re-transplant in each decile. The dotted line represents ‘perfect calibration’, where the predicted probabilities of death match the observed percentages. The density plot below each scatter plot shows the distribution of predicted probabilities of death or re-transplant for each model.

Figure 5:

Calibration of the LAS Post Transplant Survival Probability at predicting 3-month (top left), 6-month (top right), 9-month (bottom left), and 1-year (bottom right) survival. Patients are sorted into deciles, and each dot represents one decile. The x-axis represents the average predicted probability of death or re-transplant over each decile, while the y-axis shows the observed proportions (with standard errors) of death or re-transplant in each decile. The dotted line represents ‘perfect calibration’, where the predicted probabilities of death match the observed percentages. The density plot below each scatter plot shows the distribution of predicted probabilities of death or re-transplant for each model.

Secondary Analysis – Disease Specific Models, Donor Covariates, Long Term (3-year) Survival, and LAS Era

The LASSO and Random Forests models re-fit on disease-specific subgroups (CF, IPF, COPD, and other) separately did not show any improvement in prediction accuracy (Figure S-1). No significant improvement in AUC was observed using disease specific models rather than fitting a single model on all patients. Adding donor covariates to the LASSO and Random Forests models did not improve AUC over using recipient covariates alone (Figures S-2 and S-3). LASSO, Random Forests, and clinician models re-fit using 3-year survival instead of 1-year survival demonstrated that longer term survival is more difficult to predict than shorter term survival, and every model considered had lower AUC on 3-year predictions than 1-year predictions (Figure S-4).

There were no differences in predictive accuracy at 1-year by LAS era: 2005–2010, 2011–2014, and 2015–2017 (Figure S-5). Calibration plots for the LASSO, Random Forests, and clinician models for each LAS era (Figure S-6) discriminated only those at the highest and lowest risks of 1-year death or re-transplant with reasonable calibration in the 2005–2010 era, but not in the 2011–2014 or 2015–2017 era.

Discussion

In this large retrospective analysis, we demonstrated that the LAS and several other models lacked a meaningful ability to accurately predict post-transplant mortality using pre-transplant covariates. The LAS overestimated patients’ risk of death (particularly for those in the highest deciles of LAS) and thus may disadvantage some waitlisted patients. Machine learning models, disease-specific models, and models with donor characteristics did not improve predictive accuracy. Because of the importance of longer-term survival after LTx, in keeping with proposed UNOS policy changes, we assessed 3-year survival models, but found the predictive accuracy was even worse than that seen in the 1-year models. Several variables individually were associated with re-transplant or death within 1-year. Unfortunately, aggregate models designed by expert selection or machine learning, even when incorporating variables with significance when isolated, were unable to meaningfully predict post-transplant mortality. The available covariates do not capture the data needed to do so. These findings are disappointing, but very relevant as the US attempts to revise its lung allocation system with a focus on prioritizing post-transplant survival.

Our findings confirm an earlier study by Gries, et al.,¹⁷ who used an ISHLT dataset and demonstrated that a model based on pre-transplant covariates poorly predicted 1-year or 5-year survival (AUC 0.553 and 0.591, respectively), the LAS had poor predictive ability at 1-year and 5-years (AUC 0.58 and 0.566, respectively), and there was no improvement when fitting these models for individual disease groups.¹⁷

In the 2019 study by Chan, et al.,¹⁰ the Houston Methodist model was predictive of 1-year mortality and was able to designate patients into risk categories, but we were unable to replicate these findings in our larger, more contemporary dataset. The Houston Methodist model was designed by Chan, et al.,¹⁰ using the UNOS dataset, containing 10533 patients who underwent LTx between 1994–2014, to identify and randomly cohort the 633 patients who underwent LTx at Houston Methodist Hospital into equal development and validation cohorts with which to form a predictive model. The Chan, et al.,¹⁰ model had an AUC for 1-year mortality of 0.74 and 0.67 for the development cohort and validation cohort, respectively. On our attempt to recreate the Chan, et al.,¹⁰ model within our development cohort, we found an AUC for 1-year mortality of only 0.59. The difference in populations and size of our cohorts may account for the difference in AUC of the original model to our recreation. Chan, et al.,¹⁰ also found that the LAS had a lower AUC for 1-year mortality, at 0.58 and 0.55 for their development and validation cohorts respectively, similar to the AUC for the LAS for our cohort (0.55).

In a recent study by Parker, et al.,⁸ the predictive accuracy of the LAS was analyzed using a Cox proportional hazard model and post-transplant survival Cox proportional hazard model which comprise the LAS. They, too, found poor calibration between predicted and observed waitlist survival, post-transplant survival, and LAS, respectively. The authors suggested that prediction could be better with updated models, specifically mentioning machine learning, and noting that the lack of donor variables may have contributed to the LAS not being effective for predicting post-transplant mortality.⁸ Unfortunately, the novel models assessed in our study did not show improved predictive accuracy despite utilizing machine learning techniques, including alternate pre-transplant recipient covariates, including available donor covariates, or stratifying by diagnosis.

We were unable to identify a model which provided better 1-year or 3-year accuracy in predicting survival. The OPTN Board of Directors established continuous organ distribution as the preferred framework to distribute all organs¹⁸ to improve transparency and equity in organ allocation. Lung was selected as the first organ to make this change, leading to the proposal of the composite allocation score (CAS) by the OPTN Lung Transplantation Committee.⁹ The Lung CAS will utilize 5-year predicted post-transplant outcomes model, rather than the 1-year predicted post-transplant outcomes model currently utilized.¹¹ The 5-year model, much like the 3-year model considered in this study, did not have better predictive accuracy than the 1-year model, but was demonstrated to have similar level of confidence as the 1-year models in the report by the Scientific Registry of Transplant Recipients.¹¹ The 5-year model allows consideration of a longer outcome period and, in the context of continuous allocation, showed greater variability across age groups than 1-year models, which may allow for stratification by age.¹¹

Numerous studies have endeavored to validate the effectiveness of the LAS as a system to allocate organs to those most in need of a donor lung and its success in predicting post-transplant mortality. Earlier studies suggested that those with higher LAS experienced worse absolute survival after transplant than patients with lower LAS.^19–24 Since its introduction, the mean LAS in the upper quartile of patients who receive transplants has steadily risen,²⁵ suggesting sicker patients being listed for transplant. Earlier studies looked at absolute survival and worse outcomes may have reflected a sicker population being listed. Later studies show that recipients with higher LAS experienced greater survival benefit than those with lower LAS and that nuances exist between various diagnostic groups.^25–27

Though several studies have shown modifiable pre-transplant variables, such as weight and albumin levels, that correlate with post-LTx mortality, our study did not find significant correlation when considering these as covariates aggregated within models.^28–33 Several scoring systems, such as the Oto-Score^34,35 and the Louisville-UNOS scale in conjunction with the LAS,³⁶ predict post-transplant outcomes including 1-year mortality of the organ recipient based on donor and organ variables. The inclusion of donor covariates in a unifying model to predict post-LTx outcomes did not improve predictive accuracy in our study.

Our study involved a thorough approach to evaluating the available predictive models for post-LTx survival in a contemporary national sample of LTx recipients. There are, however, limitations to this study. First, we are limited to variables available in the OPTN registry. The data available may be insufficient to design a model with highly predictive accuracy. There may be factors we do not capture, whether in the OPTN registry or otherwise, which impact long-term mortality (see Table S-6). Second, while our data set was up to date, we have fewer outcomes, particularly 3-year survival from the most recent years, which could have impacted our assessment of 3-year survival prediction.

Our findings serve several cautionary tales. The LAS can overestimate an individual patient’s risk of death (particularly those with the highest LAS) and potentially limit access to transplant for certain patients. There should be wariness in the clinical utility of short-term survival predictions from the LAS and other models based on pre-transplant recipient covariates, and there is no identifiable model to reliably predict medium- and long-term survival after transplant. This may be due, in part, to the limitations of which variables are available in the OPTN registry. Additionally, we must consider whether the diversity of pulmonary diagnoses leading to end-stage lung disease lend to a single, unifying model for predicting post-transplant outcomes – though models stratified by diagnosis do not perform more accurately.

Stewardship of donated organs while caring for vulnerable patients is a solemn responsibility. Maintaining equity in allocating these organs while assuring that recipients and donated organs have the best chance for long-term survival are inexorable tenets. Though implementation of the LAS improved waitlist mortality and resulted in increased transplant rates, there continues to be room for improvement. Developing accurate models to predict long-term post-transplant survival is vitally important for ethical organ allocation and scarce resource-utilization.

List of Non-Standard Abbreviations

AUC

Area Under the Curve

BMI

Body Mass Index

CAS

Composite Allocation Score

CF

Cystic Fibrosis

CMV

Cytomegalovirus

COPD

Chronic Obstructive Pulmonary Disease

FEV₁

Forced Expiratory Volume in One Second

FVC

Forced Vital Capacity

GFR

Glomerular Filtration Rate

IPF

Idiopathic Pulmonary Fibrosis

LAS

Lung Allocation Score

LASSO

Least Absolute Shrinkage and Selection Operator

LTx

Lung Transplant

NPV

Negative Predictive Value

OPTN

Organ Procurement and Transportation Network

PCO₂

Arterial Partial Pressure of Carbon Dioxide

PPV

Positive Predictive Value

SMW

Six Minute Walk

SRTR

Scientific Registry of Transplant Recipients

UNOS

United Network for Organ Sharing