Abstract
Background
Improved methods are needed to risk-stratify patients with cystic fibrosis (CF) and reduced FEV1.
Research Questions
What are the predictors of death or lung transplantation (LTx) within 2 years among patients with CF whose FEV1 ≤ 50% predicted? Do these markers similarly predict outcomes among G551D patients taking ivacaftor since 2012?
Study Design and Methods
Patients with CF, age ≥ 6 years with FEV1 ≤ 50% predicted as of December 31, 2014, were identified in a data set that merged Cystic Fibrosis Foundation and United Network for Organ Sharing (UNOS) registries. The least absolute shrinkage and selection operator (LASSO) method was applied to a randomly selected training set to select important prognostic variables. Accuracy and association of the model with death or LTx with 2 years (2-year death or LTx) were validated via logistic regression on an independent test set. Sensitivity analyses explored predictors for patients with UNOS data.
Results
FEV1 percent predicted (OR, 1.51 for 5% decrease; 95% CI, 1.27-1.81), number of pulmonary exacerbations treated with IV antibiotics (OR, 1.35; 95% CI, 1.11-1.65), and continuous or nocturnal oxygen (OR, 3.71; 95% CI, 1.81-7.59) were significantly associated with 2-year death or LTx. Our model predicted outcomes with greater sensitivity (ratio of sensitivity, 1.26; 95% CI, 1.02-1.54), ratio of positive predictive value (1.25; 95% CI, 1.05-1.51), and ratio of negative predictive value (1.04; 95% CI, 1.01-1.07) than FEV1 < 30% predicted. Among those taking ivacaftor in 2014, only FEV1 remained associated with 2-year death or LTx. For patients with UNOS data, LASSO identified additional covariates of interest, including noninvasive ventilation use, low hemoglobin, pulmonary arterial systolic pressure, supplemental oxygen, mechanical ventilation, FEV1 percent predicted, and cardiac index.
Interpretation
Among individuals with CF and FEV1 ≤ 50% predicted, FEV1 percent predicted, oxygen therapy, and number of pulmonary exacerbations predicted 2-year death or LTx. Although limited by small sample size, only FEV1 remained predictive in patients receiving highly effective modulator therapy. Additional physiologic variables could improve prognostication in CF.
Key Words: classifier model, cystic fibrosis, FEV1, predictive model
Abbreviations: AUC, area under the curve; CF, cystic fibrosis; CFFPR, Cystic Fibrosis Foundation Patient Registry; CFTR, cystic fibrosis transmembrane conductance regulator; ETI, elexacaftor/tezacaftor/ivacaftor; HEMT, highly effective CFTR modulator treatment; ISHLT, International Society for Heart and Lung Transplantation; LAS, lung allocation score; LASSO, least absolute shrinkage and selection operator; LTx, lung transplantation; NPV, negative predictive value; OPTN, Organ Procurement and Transplantation Network; PPV, positive predictive value; UNOS, United Network for Organ Sharing
Take-home Points.
StudyQuestion: What are the predictors of 2-year lung transplantation (LTx) or death without LTx among patients with cystic fibrosis (CF) with FEV1 ≤ 50% predicted, and do these markers similarly predict outcomes among G551D patients taking ivacaftor since 2012?
Results: The prognostic model included FEV1 percent predicted, number of pulmonary exacerbations, and the use of supplemental oxygen, and had greater sensitivity (rSens, 1.26; 95% CI, 1.02-1.54), positive predictive value (rPPV, 1.25; 95% CI, 1.05-1.51), and negative predictive value (rNPV, 1.04; 95% CI, 1.01-1.07) than FEV1 < 30% predicted; among those taking ivacaftor in 2014, only FEV1 remained associated with death or LTx.
Interpretation: Although the implications of new, highly effective CF therapeutics have yet to be realized, three simple clinical predictors yielded a model that performed better than the historical threshold of FEV1 < 30% predicted alone and can help inform the timing of LTx for CF.
Cystic fibrosis (CF) leads to progressive bronchiectasis and death most commonly due to respiratory failure. Lung transplantation (LTx) can extend and improve quality of life for individuals with advanced CF lung disease.1,2 The International Society for Heart and Lung Transplantation (ISHLT) recommends referral for LTx when 2-year predicted survival is less than 50%,3 but identifying individuals with CF at risk of death without LTx is difficult in practice. In fact, although FEV1 ≤ 30% predicted has long been recognized as an important prognostic predictor and LTx referral criterion in CF,3, 4, 5 a 2017 study demonstrated a median survival of > 6.5 years after reaching this threshold.6 In that study, nearly 10% of patients died in the first year after the FEV1 fell below 30% of predicted.6 Moreover, other studies have shown that many respiratory deaths in CF occur in the context of late or nonreferral for LTx,7 prompting Cystic Fibrosis Foundation referral guidelines to emphasize the concept of early referral for at-risk individuals,5 also highlighting the need for accurate tools for risk prediction.
A number of studies describe statistical models incorporating various clinical factors to predict death or LTx in CF, but have shown only marginal improvements in accuracy over FEV1 ≤ 30% predicted alone.8, 9, 10, 11, 12 A limitation of prior modeling has been the inclusion of the entire CF population, and therefore a large proportion of patients at extremely low risk of the outcome. No modeling has been performed specifically in the most relevant population of patients with low FEV1, an approach that could improve predictive performance and enhance clinical usefulness in patients being considered for transplantation. In addition, prior models have trumpeted high area under the curve (AUC) but undervalue the importance of positive predictive value (PPV) in assessing the usefulness of a prognostic model.13 Another inherent but major limitation of existing models is their development before the advent of highly effective CF transmembrane conductance regulator (CFTR) modulator treatment (HEMT). Although large-scale data are lacking and follow-up time limited, recent studies showed improvements in several clinical markers after starting elexacaftor/tezacaftor/ivacaftor (ETI) in cohorts with advanced pulmonary disease.14, 15, 16 On the basis of these early trends showing decreased rates of LTx in the setting of HEMT, it is clear that prognostication in CF lung disease must account for these important therapies. With HEMT now available for most patients with CF, it is unknown how previous prognostic models may apply. Because ETI was not approved for use in the United States until October 2019, an insufficient amount of time has elapsed to allow for the large-scale study of outcomes for patients taking ETI, and ivacaftor use for people with CFTR genotype G551D serves as a model for understanding potential effects of ETI on the larger CF population.
The primary aim of this study was to develop a 2-year prognostic model derived from CF patients with FEV1 ≤ 50% predicted. The composite end point of death or LTx is commonly used in prognostic models for CF.9,11 We chose this FEV1 threshold for the following reasons: (1) FEV1 < 50% is the threshold at which the CF Foundation guidelines recommend discussion of LTx, and a prognostic model/tool could be especially valuable for this group; and (2) the notion of initiating the LTx “process” (discussions, etc.) early is based on the fact that there are individuals with FEV1 above 40% who die without LTx, and thus could have benefited from earlier consideration and/or referral. Secondarily, to assess the generalizability to a population receiving HEMT, we evaluated the accuracy of the model among patients with CFTR genotype G551D treated with ivacaftor.
Study Design and Methods
Participants and Data Collection
Participants were patients in the CF Foundation Patient Registry (CFFPR) who were ≥ 6 years old on December 31, 2014, had not previously received a lung transplant, and were seen at a CF Foundation-accredited care center in 2014. We limited the cohort to those with FEV1 ≤ 50% of predicted. CFFPR records were merged with a United Network for Organ Sharing (UNOS) data set provided by the Organ Procurement and Transplantation Network (OPTN), as previously described.17,18 CFFPR and UNOS variables are described in the online supplement (e-Table 1). The University of Washington Institutional Review Board approved this study (#2270, IRB 1).
Model Development
As our primary analysis, we fit a penalized logistic regression model (least absolute shrinkage and selection operator; LASSO) to develop model 1 with respiratory/cardiorespiratory death or LTx within 2 years (December 31, 2016) as the outcome to select predictors of interest from all covariates available in 2014 annual CFFPR data, using the glmnet package in R version 3.6.2.19 Patients with the G551D mutation, eligible for HEMT in 2014, were held out for generalizability. A random 90% of patients without G551D were selected to form a training set, on which we developed our LASSO model (model 1). The remaining 10% of the study cohort without G551D was held out for validation. A base model, for comparison, used the covariates selected in the prognostic model published by Mayer-Hamblett et al11 refit using the training data set before evaluating its performance.
Model Validation
On the validation set, we compared model 1 (LASSO) with the base model11 refit to the 2014 data set, and to FEV1 < 30% predicted.
To facilitate comparison between model 1, the base model, and FEV1 < 30% predicted, we selected a cutoff value for model 1 and the base model to have the same specificity as FEV1 < 30% predicted on the training set. We then compared model 1 and the base model, using the selected cutoff values, with FEV1 < 30% predicted on the validation set by the ratios of sensitivity (rSens), specificity (rSpec), positive predictive value (rPPV), and negative predictive value (rNPV), using log-linear regression.20 In addition, we evaluated the full discriminatory ability of model 1 compared with FEV1 as a continuous predictor among individuals with FEV1 ≤ 50%, by testing for a difference in AUC.21 The association between covariates selected by LASSO with death or LTx within 2 years was evaluated by logistic regression on the validation set with robust SEs. A calibration curve was developed with patients sorted into deciles and showed average predicted probability of death or retransplantation over each decile, and the observed proportions (with SE) of death or retransplantation in each decile.
Evaluation among patients receiving HEMT was performed by repeating the primary analysis in patients with at least one copy of the G551D CFTR mutation.
Sensitivity Analyses: 1-Year Outcomes, Cohorts With Variable FEV1 Thresholds, and a UNOS Model
Short-term outcomes were evaluated by repeating the main analysis with restriction to events at 1 year. The primary analysis was repeated among patients (age ≥ 6 years) with FEV1 < 40%, FEV1 <30%, and for adults (age ≥ 18 years) with the three different FEV1 thresholds. For the individuals listed for LTx in 2014, a LASSO model was fit using CFFPR and UNOS variables (model 2); details may be found in the online supplement.
Results
Cohort Characteristics
In total, 3,720 patients with CF with FEV1 ≤ 50% predicted met the inclusion criteria (Fig 1). The median age of the cohort was 30.2 years, and the median FEV1 was 38.5% predicted (interquartile range, 30.9-44.5). Only 8.5% were less than 18 years of age. Of the total cohort, within 2 years 357 (9.6%) died, of respiratory/cardiorespiratory disease, without LTx; 431 (11.6%) underwent LTx; and 2,932 (78.8%) were alive without LTx (Table 1). Age and race were similar among individuals with the composite outcome of death or LTx within 2 years compared with those who survived, while survivors were more often male and had higher BMI (Table 1). When comparing characteristics between those who died vs those who underwent LTx separately, those who died were more likely to have Medicaid insurance status (55% among those who died, 47% for those who underwent LTx) and less likely to be White (91% among those who died, 97% for those who underwent LTx) (e-Table 2). Patients with UNOS data available in 2014 were older, less often on Medicaid insurance, more frequently used supplemental oxygen or noninvasive ventilation, were more often anemic, and had more frequent care episodes, hospitalizations, and pulmonary exacerbations (e-Table 2).
Figure 1.
Cohort selection from the US Cystic Fibrosis Foundation Patient Registry.
Table 1.
Baseline (2014) Patient Characteristics, by 2-Year Outcome of Cardiorespiratory Death/Lung Transplantation (LTx) or Alive Without LTx
| Variable | FEV1 ≤ 50% (n = 3,720) |
|
|---|---|---|
| Cardiorespiratory Death or LTx Within 2 Years (n = 788) | Alive Without LTx (n = 2,932) | |
| Age, median (IQR), y | 29.4 (23.2-38.5) | 30.5 (23.7-39.9) |
| Sex, female, No. (%) | 410 (52%) | 1,356 (46%) |
| Medical insurance, Medicaid, No. (%) | 425 (54%) | 1,305 (45%) |
| Race,a No. (%) | ||
| White | 741 (94%) | 2,779 (95%) |
| Black | 41 (5%) | 129 (4%) |
| Hispanic | 57 (7%) | 177 (6%) |
| Other | 9 (1%) | 51 (2%) |
| BMI, No. (%), kg/m2 | ||
| Normal (18.5-25) | 452 (57%) | 1,852 (63%) |
| Low (< 18.5) | 273 (35%) | 656 (22%) |
| High (> 25) | 54 (7%) | 332 (11%) |
| At least one copy of G551D mutation (yes), No. (%) | 13 (2%) | 114 (4%) |
| F508del, No. (%) | ||
| Homozygous | 399 (51%) | 1,398 (48%) |
| Heterozygous | 278 (35%) | 1,162 (38%) |
| None | 89 (11%) | 318 (11%) |
| Unknown | 22 (3%) | 52 (2%) |
| Oxygen therapy, continuous, No. (%) | ||
| No | 171 (22%) | 1,901 (65%) |
| Yes | 495 (63%) | 559 (19%) |
| Exacerbations only | 107 (14%) | 401 (14%) |
| Unknown | 15 (2%) | 70 (2%) |
| Noninvasive ventilation (yes), No. (%) | 194 (25%) | 160 (5%) |
| Pseudomonas aeruginosa (yes), No. (%) | 624 (79%) | 2,190 (75%) |
| Any Burkholderia species (yes), No. (%) | 55 (7%) | 125 (4%) |
| Burkholderia cenocepacia (yes), No. (%) | 12 (2%) | 23 (1%) |
| Nontuberculous mycobacterium (yes), No. (%) | 59 (7%) | 227 (8%) |
| Mycobacterium abscessus/chelonae (yes), No. (%) | 23 (3%) | 94 (3%) |
| Receiving insulin for CF-related diabetes (yes), No. (%) | 364 (46%) | 871 (30%) |
| Liver cirrhosis (yes), No. (%) | 45 (6%) | 72 (2%) |
| No. of hospitalization nights in preceding year, median (IQR) | 21 (6-47) | 6 (0-19) |
| No. of care episodes in preceding year,b median (IQR) | 3 (1-4) | 1 (0-2) |
| No. of pulmonary exacerbations treated with IV antibiotics in preceding year, median (IQR) | 2 (1-4) | 1 (0-2) |
| No. of nights on home IV antibiotics in preceding year, median (IQR) | 7 (0-30) | 0 (0-14) |
| FEV1, median (IQR), L | 1.0 (0.8-1.2) | 1.4 (1.1-1.6) |
| FEV1 percent predicted, median (IQR) | 29.2 (24.2-36.9) | 40.5 (34.0-45.5) |
| FVC percent predicted, median (IQR) | 49.6 (41.1-58.3) | 60.6 (52.4-67.9) |
| Annual decrease in FEV1 between 2012 and 2014, median (IQR), %/y | 10% (4%-16%) | 6% (1%-10%) |
| Annual change in BMI, median (IQR) | 0.0 (–0.4 to +0.4) | –0.2 (–0.5 to +0.3) |
CF = cystic fibrosis; IQR = interquartile range; LTx = lung transplantation.
Selected all that apply.
Care episodes include pulmonary exacerbations, GI complications, sinus infections, and so on; these can occur in the hospital or at home.
Predictors of Death or LTx Within 2 Years, Using LASSO (Model 1)
LASSO selected just three covariates as prognostic markers for death or LTx: FEV1 percent predicted, oxygen therapy, and pulmonary exacerbations requiring IV antibiotics. Lower FEV1 percent predicted (OR, 1.51 for 5% decrease; 95% CI, 1.27-1.81), greater number of pulmonary exacerbations in 2014 requiring IV antibiotics (OR, 1.35 for each additional event; 95% CI, 1.11-1.65), and use of continuous or nocturnal oxygen (OR, 3.71 compared with no oxygen use; 95% CI, 1.81-7.59) were significantly associated with higher odds of death or LTx. Compared with patients who did not use oxygen, those who did only during an exacerbation were associated with increased odds of death or LTx (OR, 1.53; 95% CI, 0.60-3.91), although this effect was not found to be significant in our validation set.
Applying LASSO to different cohorts of individuals with varying degrees of FEV1 impairment, or limiting the analysis to adults with the three prespecified FEV1 thresholds, yielded similar results (e-Tables 3,4). When the model was applied to the end point of death or LTx at 1 year, LASSO selected oxygen therapy, FEV1 percent predicted, and the number of hospitalizations. Both oxygen therapy and FEV1 percent predicted were selected for the 2-year model, and the number of hospitalizations was highly correlated with the number of pulmonary exacerbations (Pearson correlation coefficient, 0.88). This model returned risk estimates similar to those of our 2-year model (e-Figs 1,2).
To improve the clinical usefulness of the model, we generated a guide for computing the log-odds risk score using FEV1, oxygen therapy, and number of pulmonary exacerbations, and provided examples using potential cases (e-Table 5). Risk tolerance varies across patients and providers, with the ISHLT guidelines recommending transplantation when risk of death without LTx is > 50% within 2 years.3 The calculation of 2-year risk provides flexibility in adapting timing of transplantation for individual risk tolerance.
Assessment of Model Performance in the Validation Cohort
We selected a cut point for model 1 and the base model such that all models had the same specificity (0.87) on the training set as FEV1 < 30% predicted. Model 1 compared with FEV1 < 30% predicted had significantly higher sensitivity (0.60 vs 0.47; rSens, 1.26; 95% CI, 1.02-1.54), PPV (0.55 vs 0.44; rPPV, 1.25; 95% CI, 1.05-1.51), and NPV (0.89 vs 0.86; rNPV, 1.04; 95% CI, 1.01-1.07) (Table 2). Receiver operator characteristic curves (Fig 2) demonstrated that model 1 has better sensitivity than FEV1 percent predicted alone at each level of specificity. In contrast, the base model (Mayer-Hamblett model, refit to 2014 data) did not perform any better than FEV1 < 30% alone in sensitivity (0.46 vs 0.47; rSens, 0.97; 95% CI, 0.82-1.15), PPV (0.47 vs 0.44; rPPV, 1.08; 95% CI, 0.93-1.25), and NPV (0.86 vs 0.86; rNPV, 1.00; 95% CI, 0.98-1.02). Model 1 also had a higher AUC (0.83; 95% CI, 0.78-0.88) than either the base model (0.78; 95% CI, 0.72-0.84) or FEV1 percent predicted (0.75; 95% CI, 0.69-0.82) (Fig 2), and the AUC of model 1 was significantly higher than that of FEV1 percent predicted (P < .001). The calibration slope was not statistically different from 1.0 (e-Fig 3).
Table 2.
Ratio of Performance Metrics (Specificity, Sensitivity, Positive Predictive Value, and Negative Predictive Value) for Model 1 and the Base Model, Compared With FEV1 < 30% Predicted
| Relative Accuracy of Models to FEV1 < 30% | Genotypes Other Than G551D |
G551D |
||
|---|---|---|---|---|
| Model 1 | Base Model | Model 1 | Base Model | |
| rSpec | 1.04 (0.99-1.09) | 1.03 (0.99-1.07) | 1.04 (0.97-1.11) | 1.05 (0.97-1.14) |
| rSens | 1.26 (1.02-1.54) | 0.97 (0.82-1.15) | 0.86 (0.51-1.45) | 1.14 (0.73-1.80) |
| rPPV | 1.25 (1.05-1.51) | 1.08 (0.93-1.25) | 1.11 (0.68-1.81) | 1.40 (0.85-2.31) |
| rNPV | 1.04 (1.01-1.07) | 1.00 (0.98-1.02) | 0.99 (0.96-1.03) | 1.01 (0.98-1.05) |
For ratios, the estimates for model 1 or the base model are the numerator, and estimates for FEV1 < 30% predicted are the denominator, where a ratio of 1 indicates no difference in the chosen metrics. The base model used the covariates selected in the prognostic model published by Mayer-Hamblett et al11 (refit to 2014 data). rNPV = ratio of negative predictive value; rPPV = ratio of positive predictive value; rSens = ratio of sensitivity; rSpec = ratio of specificity.
Figure 2.
Receiver operator characteristic curve for model 1 (red) compared with FEV1 percent predicted (gray) and the base model, Mayer-Hamblett model refit to 2014 data (blue), on a randomly sampled validation cohort of patients without the G551D mutation. Specificity, sensitivity, positive predictive value, and negative predictive value are presented with a cut point determined on the training set matched on specificity to FEV1 < 30%. AUC = area under the curve; CFFPR = Cystic Fibrosis Foundation Patient Registry; NPV = negative predictive value; PPV = positive predictive value; Sens = sensitivity; Spec = specificity.
Patients Receiving HEMT
Of the total cohort, 127 (3.4%) had at least one copy of the G551D CFTR mutation, of whom 117 (92.1%) used ivacaftor in 2014. Patients with G551D had substantially better outcomes, with the proportion who either died or underwent LTx within 2 years 11.2% lower (95% CI, 5.3%-17.0%) than those without G551D, and this advantage persisted when stratified by FEV1 (Fig 3). Thirteen patients with G551D (10.2%) died or underwent LTx within 2 years, all of whom were taking ivacaftor, compared with 788 (21.1%) of the non-G551D cohort. Descriptive statistics for the G551D cohort are available in e-Table 6.
Figure 3.
Probability of death or lung transplantation in 2 years by FEV1 percent predicted, stratified by genotype. LTx = lung transplantation.
In a sensitivity analysis among patients with G551D, lower FEV1 percent predicted was associated with higher odds of death or LTx within 2 years (OR, 2.31 for 5% decrease; 95% CI, 1.30-4.09). However, in contrast to those without G551D, a greater number of pulmonary exacerbations requiring IV antibiotics (OR, 1.12 for each additional event; 95% CI, 0.72-1.73) and use of continuous or nocturnal oxygen (OR, 2.30 compared with no oxygen use; 95% CI, 0.35-15.21) were not significantly associated with higher odds of death or LTx. Among those with G551D in the validation cohort, model 1 and FEV1 < 30% predicted had similar sensitivity (rSens, 0.86; 95% CI, 0.51-1.45), PPV (rPPV, 1.11; 95% CI, 0.68-1.81), and NPV (rNPV, 0.99; 95% CI, 0.96-1.03) (Table 2). There was also no significant difference in AUC between model 1 and FEV1 percent predicted alone (P = .53) in the G551D cohort.
Individuals With UNOS Data in 2014 (Model 2)
Among 249 individuals who recorded data in UNOS in 2014, LASSO identified additional covariates of interest (Table 3). Noninvasive ventilation use (OR, 2.36; 95% CI, 0.70-7.87), low hemoglobin (OR, 1.54; 95% CI, 0.61-3.87), pulmonary arterial systolic pressure (OR, 1.13 for 5-mm Hg increase; 95% CI, 1.06-1.19), supplemental oxygen (OR, 2.07 per additional liter per minute; 95% CI, 1.15-3.72), Pco2 (OR, 1.46 for 5-mm Hg increase; 95% CI, 1.10-1.95), and mechanical ventilation (OR, 3.11; 0.49-19.60) were associated with higher odds of death or LTx, whereas FEV1 percent predicted (OR, 0.73 for 5% increase; 95% CI, 0.49-1.07) and cardiac index (OR, 0.56 per 0.5-L/min/m2 increase; 95% CI, 0.32-0.97) were associated with lower odds of death or LTx. With the addition of UNOS variables, pulmonary exacerbations were not selected by LASSO for model 2, although FEV1 and supplemental oxygen were still selected in addition to other markers not available in the CFFPR (pulmonary pressure, cardiac index, and Pco2).
Table 3.
Selected Covariates and Their Corresponding ORs, Using Logistic Regression With Death or Lung Transplantation Within 2 Years as the Outcome for Patients With UNOS Data in 2014
| Database | Variable | OR |
|---|---|---|
| CFFPR | Noninvasive ventilation, used | 2.36 (0.70-7.87) |
| Hemoglobin, low (vs normal or high) | 1.54 (0.61-3.87) | |
| FEV1 percent predicted, 5% higher | 0.73 (0.49-1.07) | |
| UNOS | PA systolic pressure, 1 mm Hg higher | 1.13 (1.06-1.19) |
| Supplemental oxygen, 1 L/min higher | 2.07 (1.15-3.72) | |
| Cardiac index, 0.5 L/min/m2 higher | 0.56 (0.32-0.97) | |
| Pco2, 5 mm Hg higher | 1.46 (1.10-1.95) | |
| Ventilator assistance, used | 3.11 (0.49-19.60) |
CFFPR = Cystic Fibrosis Foundation Patient Registry; PA = pulmonary artery; UNOS = United Network for Organ Sharing.
ROC curves for model 2 developed on the subset of individuals listed for LTx in UNOS in 2014 are shown in e-Figure 4. Among patients with data in UNOS, the lung allocation score (LAS) is a better indicator of patient outcomes than FEV1 alone (AUC, 0.85 for LAS; AUC, 0.64 for FEV1). Model 2, which includes physiologic markers from UNOS, has similar discriminatory accuracy to LAS, and shows a large improvement over model 1 in this subset of individuals. Among patients in UNOS, our modeling approaches suffer from the reverse problem as our models in CFFPR: they all do significantly better at identifying patients who die or undergo LTx (PPV, 0.85-0.92) than at identifying those who are likely to survive without LTx (NPV, 0.19-0.40).
Discussion
We developed a 2-year prognostic model for patients with CF with FEV1 ≤ 50% predicted in the United States. In a large cohort merging data from the CF Foundation and UNOS registries, LASSO selected three covariates that predicted death or need for LTx within 2 years: FEV1 percent predicted, oxygen therapy, and number of pulmonary exacerbations requiring IV antibiotics. In the overall cohort the model was superior to FEV1 < 30% predicted alone in predicting outcomes, although this was not true in the subgroup with genotype G551D receiving HEMT, which was underpowered to detect significant associations. The association of FEV1 < 30% predicted with LTx may be stronger than the physiologic effect of FEV1 alone because nearly all CF health care providers use this threshold when considering the timing of lung transplantation referral.22 The additional covariates identified in the current model could augment risk stratification by CF health care providers for patients with reduced FEV1. The subgroup analysis of individuals receiving HEMT should be interpreted with caution because of low numbers, but it is informative because it highlights the ongoing risk of LTx or death without LTx despite HEMT and the need to address timing of transplantation for those taking modulators.
Although outcomes in CF are improving, progressive lung disease remains common particularly in adults,23 with 2019 CFFPR data showing that one-quarter of those ≥ 18 years old currently have an FEV1 ≤ 50% predicted; approximately 250 patients with CF in the United States underwent LTx each year between 2016 and 2019.24 Despite being an important treatment option, determining the optimal timing of LTx referral and listing remains challenging. CF Foundation guidelines on advanced lung disease and LTx referral highlight the concepts of early introduction of LTx as a treatment option for all patients and annual discussion after reaching an FEV1 ≤ 50% predicted.5,23 Focused risk assessment is recommended in individuals with FEV1 30% to 50% predicted to identify those with markers of increased disease severity, and referral to an LTx center for all patients with FEV1 < 30% predicted.5,23 These recommendations err on the conservative side to reduce the risk of death without LTx, but it is clear that many individuals with advanced CF lung disease can survive for many years,6,25 underscoring the need for improved prognostication in this population.
FEV1 percent predicted was first identified as a prognostic marker in CF in the early 1990s,4 and it has consistently predicted mortality in many studies since.9,11,26, 27, 28, 29 Several other markers of disease severity have similarly been associated with worse outcomes among patients with CF, including those with advanced disease,6,9,29, 30, 31, 32, 33 but prognostic modeling has proven challenging, with previous approaches having had relatively low PPV and/or not clearly showing superiority over FEV1 alone.10,11 The combined outcome of LTx or death was chosen as in prior CF models,8,9,11 an approach that allows for identification of the most advanced patients for whom LTx should be considered. A study by Alaa and van der Schaar that used complex machine learning methodology identified 22 important covariates associated with death or LTx within 3 years, using 115 variables in the UK CF Registry, but highlighted FEV1 and oxygenation as having the highest PPV.34 Our study and model carry several strengths including development in a large cohort specifically with reduced FEV1, and use of a merged data set that allowed for improved outcome ascertainment for LTx and listing status. The use of LASSO allowed for evaluation of all CFFPR covariates without the need for prespecification, thereby choosing the most relevant predictors with the observed data. Simulation studies also suggest that LASSO has better prognostic accuracy than forward stepwise selection.35, 36, 37 Although sex, CF-related diabetes, BMI, and change in BMI are often considered markers of increased disease severity,5 LASSO did not choose these predictors as part of the prognostic model.
An important consideration for individuals with CF is the advent of HEMT, now available for approximately 90% of the CF population over 6 years of age.38,39 These agents will improve outcomes significantly in the overall CF population, but as of now the limited follow-up time and absence of large-scale data have precluded identification of markers of worse prognosis specifically in those treated by HEMT. Ivacaftor, which was approved in 2012 for patients with the G551D CFTR mutation, and has shown durable benefits, is widely considered the first HEMT.40 We did not include individuals with other genotypes who are now eligible for ivacaftor because only the G551D CFTR mutation had US Food and Drug Administration approval until December 29, 2014, when R117H and others were added to the label.41 The longer accrued follow-up time with ivacaftor allows our study to perform prognostic modeling in patients with CF receiving HEMT. Interestingly, and in contrast to the overall cohort, our analysis of patients with the G551D mutation and receiving ivacaftor showed that additional measures of disease severity such as oxygen therapy and exacerbation frequency did not improve 2-year prognostic ability over FEV1 percent predicted alone. However, it is important to recognize that the sample size for this sensitivity analysis was small, and only 13 patients (10.2%) with G551D and FEV1 ≤ 50% predicted had the composite outcome of death or LTx during the follow-up period. This percentage, although small, does stand as a reminder that many receiving HEMT will still need LTx, and it is very possible that larger cohorts with longer follow-up may reveal additional prognostic indicators that add to the capability of FEV1 percent predicted alone.
Our study does have limitations. Our modeling approach showed improvements in classification accuracy among patients with CF with FEV1 ≤ 50% predicted over previous prognostic models and the traditional FEV1 < 30% predicted threshold, but as with previous models also suffers from a high false positive rate. As with prior work, patients classified as high risk may be identified for LTx too early, although this does stand in accordance with the priority of avoiding death without LTx, as recommended in CF Foundation and ISHLT referral guidelines.3,5 Second, our data set did not include individuals taking ETI. ETI was approved in late 2019, and these data will not be available until more time has accrued with patients receiving this HEMT. Our analysis of G551D patients serves to mitigate this limitation and provides a glimpse into prognostic modeling for patients with CF who are receiving HEMT, although previous studies have suggested that variable selection via LASSO can be difficult in settings with a small sample size. Future studies, with longer HEMT follow-up, will be required to improve on prognostic prediction in those treated with ETI or other novel CF treatments. Third, we lacked important physiologic markers of disease progression in the overall cohort that are present in those patients captured in the UNOS transplant registry (eg, Pco2, pulmonary arterial systolic pressure, and cardiac index). Some of these covariates could improve identification of patients who necessitate earlier consideration for LTx and should be collected in the CFFPR for patients with advanced lung disease. Finally, the results of this model could be optimistic given the data-driven approach, and the model should be validated in an independent cohort, for example, Canadian, Australian, or European CF Registries.
Interpretation
In conclusion, three factors from the CFFPR predicted death or LTx for individuals with CF whose FEV1 is ≤ 50% of predicted, improving on previous models and FEV1 < 30% predicted alone: FEV1 percent predicted, oxygen therapy, and number of pulmonary exacerbations. Further work will be needed to elucidate whether, in addition to FEV1, additional markers assist with prognostication in patients receiving HEMT.
Acknowledgments
Author contributions: K. J. R. is the guarantor and will ensure that questions related to the accuracy or integrity of any part of this work are appropriately investigated and resolved. K. J. R., T. H. W., N. M.-H., C. H. G., and S. G. K. made substantial contributions to the conception and design of the work; K. J. R., T. H. W., A. L. S., J. S., S. S., and C. H. G. were responsible for acquisition of the data; K. J. R. and T. H. W. were responsible for data analysis; all co-authors made substantial contributions to the interpretation of data; K. J. R., T. H. W., and S. G. K. wrote the first draft of the manuscript, and all co-authors critically revised it for important intellectual content; all co-authors approved the final version submitted for publication and agree to be accountable for all aspects of the work.
Funding/support: K. J. R. receives funding from the Cystic Fibrosis Foundation (CFF) [RAMOS17A0, RAMOS20A0-KB] and the National Institutes of Health (NIH) [K23HL138154]. A. B. was supported by funding from the NIH [R37-CA218413]. N. M.-H. was supported by a grant from the NIH [UL1 TR002319]. C. H. G. was supported by grants from the CFF, the NIH [UM1 HL119073, P30 DK089507, U01 HL114589, UL1TR000423], and the US Food and Drug Administration [R01 FD003704, R01 FD006848].
Financial/nonfinancial disclosures: None declared.
Role of sponsors: Sponsors (grant funders) had no role in the development of the research or the manuscript.
Other contributions: The authors thank the CF Foundation for the use of CF Foundation Patient Registry data to conduct this study. In addition, the authors thank the patients, care providers, and clinic coordinators at CF centers throughout the United States for their contributions to the CF Foundation Patient Registry. The data reported here have been supplied by UNOS as the contractor for the OPTN. The interpretation and reporting of these data are the responsibility of the author(s) and in no way should be seen as an official policy of or interpretation by the OPTN or the US government.
Additional information: The e-Figures and Videos are available online under “Supplementary Data.”
Supplementary Data
References
- 1.Vock D.M., Durheim M.T., Tsuang W.M., et al. Survival benefit of lung transplantation in the modern era of lung allocation. Ann Am Thorac Soc. 2017;14(2):172–181. doi: 10.1513/AnnalsATS.201606-507OC. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Singer J.P., Katz P.P., Soong A., et al. Effect of lung transplantation on health-related quality of life in the era of the lung allocation score: a U.S. prospective cohort study. Am J Transplant. 2017;17(5):1334–1345. doi: 10.1111/ajt.14081. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Leard L.E., Holm A.M., Valapour M., et al. Consensus document for the selection of lung transplant candidates: an update from the International Society for Heart and Lung Transplantation. J Heart Lung Transplant. 2021;40(11):1349–1379. doi: 10.1016/j.healun.2021.07.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Kerem E., Reisman J., Corey M., Canny G.J., Levison H. Prediction of mortality in patients with cystic fibrosis. N Engl J Med. 1992;326(18):1187–1191. doi: 10.1056/NEJM199204303261804. [DOI] [PubMed] [Google Scholar]
- 5.Ramos K.J., Smith P.J., McKone E.F., et al. Lung transplant referral for individuals with cystic fibrosis: Cystic Fibrosis Foundation consensus guidelines. J Cyst Fibros. 2019;18(3):321–333. doi: 10.1016/j.jcf.2019.03.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Ramos K.J., Quon B.S., Heltshe S.L., et al. Heterogeneity in survival in adult patients with cystic fibrosis with FEV1 < 30% of predicted in the United States. Chest. 2017;151(6):1320–1328. doi: 10.1016/j.chest.2017.01.019. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Martin C., Hamard C., Kanaan R., et al. Causes of death in French cystic fibrosis patients: the need for improvement in transplantation referral strategies. J Cyst Fibros. 2016;15(2):204–212. doi: 10.1016/j.jcf.2015.09.002. [DOI] [PubMed] [Google Scholar]
- 8.Coriati A., Sykes J., Nkam L., Hocine M.N., Burgel P.-R., Stephenson A.L. Validation of the French 3-year prognostic score using the Canadian Cystic Fibrosis registry. J Cyst Fibros. 2019;18(3):396–398. doi: 10.1016/j.jcf.2018.10.014. [DOI] [PubMed] [Google Scholar]
- 9.Nkam L., Lambert J., Latouche A., Bellis G., Burgel P.R., Hocine M.N. A 3-year prognostic score for adults with cystic fibrosis. J Cyst Fibros. 2017;16(6):702–708. doi: 10.1016/j.jcf.2017.03.004. [DOI] [PubMed] [Google Scholar]
- 10.Frost F., Nazareth D., Shaw M., Al-Aloul M. Assessing the validity and applicability of the French 3-year prognostic score in the UK cystic fibrosis population: a national cohort study. Transpl Int. 2021;34(3):436–444. doi: 10.1111/tri.13805. [DOI] [PubMed] [Google Scholar]
- 11.Mayer-Hamblett N., Rosenfeld M., Emerson J., Goss C.H., Aitken M.L. Developing cystic fibrosis lung transplant referral criteria using predictors of 2-year mortality. Am J Respir Crit Care Med. 2002;166(12 Pt 1):1550–1555. doi: 10.1164/rccm.200202-087OC. [DOI] [PubMed] [Google Scholar]
- 12.Ramos K.J., Wai T.H., Sykes J., et al. Validation of the French 3-year prognostic score for death or lung transplant in the United States cystic fibrosis population. J. Cyst Fibros. 2022;21(3):471–474. doi: 10.1016/j.jcf.2021.08.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Saito T., Rehmsmeier M. The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS One. 2015;10(3) doi: 10.1371/journal.pone.0118432. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Burgel P.-R., Durieu I., Chiron R., et al. French Cystic Fibrosis Reference Network Study Group. Rapid improvement after starting elexacaftor-tezacaftor-ivacaftor in patients with cystic fibrosis and advanced pulmonary disease. Am J Respir Crit Care Med. 2021;204(1):64–73. doi: 10.1164/rccm.202011-4153OC. [DOI] [PubMed] [Google Scholar]
- 15.O’Shea K.M., O’Carroll O.M., Carroll C., et al. Efficacy of elexacaftor/tezacaftor/ivacaftor in patients with cystic fibrosis and advanced lung disease. Eur Respir J. 2021;57(2) doi: 10.1183/13993003.03079-2020. [DOI] [PubMed] [Google Scholar]
- 16.Bermingham B., Rueschhoff A., Ratti G., et al. Short-term effect of elexacaftor-tezacaftor-ivacaftor on lung function and transplant planning in cystic fibrosis patients with advanced lung disease. J Cyst Fibros. 2021;20(5):768–771. doi: 10.1016/j.jcf.2021.05.009. [DOI] [PubMed] [Google Scholar]
- 17.Stephenson A.L., Ramos K.J., Sykes J., et al. Bridging the survival gap in cystic fibrosis: an investigation of lung transplant outcomes in Canada and the United States. J Heart Lung Transplant. 2021;40(3):201–209. doi: 10.1016/j.healun.2020.12.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Ramos K.J., Sykes J., Stanojevic S., et al. Survival and lung transplant outcomes for individuals with advanced cystic fibrosis lung disease living in the United States and Canada: an analysis of national registries. Chest. 2021;160(3):843–853. doi: 10.1016/j.chest.2021.04.010. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Friedman J., Hastie T., Tibshirani R. Regularization paths for generalized linear models via coordinate descent. J Stat Softw. 2010;33(1):1–22. [PMC free article] [PubMed] [Google Scholar]
- 20.Pepe M.S. Oxford University Press; Oxford: 2003. Oxford Statistical Science Series, Vol. 31: The Statistical Evaluation of Medical Tests for Classification and Prediction. [Google Scholar]
- 21.Robin X., Turck N., Hainard A., et al. pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics. 2011;12(1):1–8. doi: 10.1186/1471-2105-12-77. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Ramos K.J., Somayaji R., Lease E.D., Goss C.H., Aitken M.L. Cystic fibrosis physicians’ perspectives on the timing of referral for lung transplant evaluation: a survey of physicians in the United States. BMC Pulm Med. 2017;17(1):21. doi: 10.1186/s12890-017-0367-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Kapnadak S.G., Dimango E., Hadjiliadis D., et al. Cystic Fibrosis Foundation consensus guidelines for the care of individuals with advanced cystic fibrosis lung disease. J Cyst Fibros. 2020;19(3):344–354. doi: 10.1016/j.jcf.2020.02.015. [DOI] [PubMed] [Google Scholar]
- 24.Cystic Fibrosis Foundation . Cystic Fibrosis Foundation; Bethesda, MD: 2020. Cystic Fibrosis Foundation Patient Registry 2019 Annual Data Report. [Google Scholar]
- 25.George P.M., Banya W., Pareek N., et al. Improved survival at low lung function in cystic fibrosis: cohort study from 1990 to 2007. BMJ. 2011;342:d1008. doi: 10.1136/bmj.d1008. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Liou T.G., Adler F.R., Fitzsimmons S.C., Cahill B.C., Hibbs J.R., Marshall B.C. Predictive 5-year survivorship model of cystic fibrosis. Am J Epidemiol. 2001;153(4):345–352. doi: 10.1093/aje/153.4.345. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Aaron S.D., Stephenson A.L., Cameron D.W., Whitmore G.A. A statistical model to predict one-year risk of death in patients with cystic fibrosis. J Clin Epidemiol. 2015;68(11):1336–1345. doi: 10.1016/j.jclinepi.2014.12.010. [DOI] [PubMed] [Google Scholar]
- 28.Stephenson A.L., Tom M., Berthiaume Y., et al. A contemporary survival analysis of individuals with cystic fibrosis: a cohort study. Eur Respir J. 2015;45(3):670–679. doi: 10.1183/09031936.00119714. [DOI] [PubMed] [Google Scholar]
- 29.Belkin R.A., Henig N.R., Singer L.G., et al. Risk factors for death of patients with cystic fibrosis awaiting lung transplantation. Am J Respir Crit Care Med. 2006;173(6):659–666. doi: 10.1164/rccm.200410-1369OC. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Lehr C.J., Skeans M., Dasenbrook E., et al. Effect of including important clinical variables on accuracy of the lung allocation score for cystic fibrosis and chronic obstructive pulmonary disease. Am J Respir Crit Care Med. 2019;200(8):1013–1021. doi: 10.1164/rccm.201902-0252OC. [DOI] [PubMed] [Google Scholar]
- 31.Hayes D., Jr., Tobias J.D., Mansour H.M., et al. Pulmonary hypertension in cystic fibrosis with advanced lung disease. Am J Respir Crit Care Med. 2014;190(8):898–905. doi: 10.1164/rccm.201407-1382OC. [DOI] [PubMed] [Google Scholar]
- 32.de Boer K., Vandemheen K.L., Tullis E., et al. Exacerbation frequency and clinical outcomes in adult patients with cystic fibrosis. Thorax. 2011;66(8):680–685. doi: 10.1136/thx.2011.161117. [DOI] [PubMed] [Google Scholar]
- 33.Martin C., Chapron J., Hubert D., et al. Prognostic value of six minute walk test in cystic fibrosis adults. Respir Med. 2013;107(12):1881–1887. doi: 10.1016/j.rmed.2013.10.001. [DOI] [PubMed] [Google Scholar]
- 34.Alaa A.M., van der Schaar M. Prognostication and risk factors for cystic fibrosis via automated machine learning. Sci Rep. 2018;8(1) doi: 10.1038/s41598-018-29523-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Hastie T., Tibshirani R., Tibshirani R. Extended comparisons of best subset selection, forward stepwise selection, and the lasso. arXiv.org. Preprint. posted online July 29, 2017 doi: 10.48550/arXiv:1707.08692. [DOI] [Google Scholar]
- 36.Ghosh D., Chinnaiyan A.M. Classification and selection of biomarkers in genomic data using LASSO. J Biomed Biotechnol. 2005;2005(2):147–154. doi: 10.1155/JBB.2005.147. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Tibshirani R. Regression shrinkage and selection via the lasso. J R Statist Soc B. 1996;58(1):267–288. [Google Scholar]
- 38.Middleton P.G., Mall M.A., Drevinek P., et al. Elexacaftor-tezacaftor-ivacaftor for cystic fibrosis with a single Phe508del allele. N Engl J Med. 2019;381(19):1809–1819. doi: 10.1056/NEJMoa1908639. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Heijerman H.G., McKone E.F., Downey D.G., et al. VX17-445-103 Trial Group. Efficacy and safety of the elexacaftor plus tezacaftor plus ivacaftor combination regimen in people with cystic fibrosis homozygous for the F508del mutation: a double-blind, randomised, phase 3 trial. Lancet. 2019;394(10212):1940–1948. doi: 10.1016/S0140-6736(19)32597-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Ramsey B.W., Davies J., McElvaney N.G., et al. A CFTR potentiator in patients with cystic fibrosis and the G551D mutation. N Engl J Med. 2011;365(18):1663–1672. doi: 10.1056/NEJMoa1105185. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Drugs.com Kalydeco FDA approval history. https://www.drugs.com/history/kalydeco.html
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.