Risk Prediction for Delayed Allograft Function: Analysis of The Deterioration of Kidney Allograft Function (Dekaf) Study Data

Abstract

Background.

Delayed graft function (DGF) of a kidney transplant results in increased cost and complexity of management. For clinical care or a DGF trial, it would be ideal to accurately predict individual DGF risk and provide preemptive treatment. A calculator developed by Irish et al has been useful for predicting population, but not individual risk.

Methods.

We analyzed the Irish calculator (IC) in the DeKAF prospective cohort (incidence of DGF= 20.4%) and investigated potential improvements.

Results.

We found that the predictive performance of the calculator in those meeting Irish inclusion criteria was comparable to that reported by Irish et. al. For cohorts excluded by Irish: a) in pump-perfused kidneys the IC over-estimated DGF risk; b) in simultaneous pancreas kidney (SPK) transplants, the DGF risk was exceptionally low. For all 3 cohorts, there was considerable overlap in IC scores between those with and those without DGF. Using a modified definition of DGF – excluding those with a single dialysis in the first 24 hours posttransplant - we found that the calculator had similar performance as with the traditional DGF definition. Studying whether DGF prediction could be improved, we found that recipient cardiovascular disease was strongly associated with DGF even after accounting for IC predicted risk.

Conclusions.

The IC can be a useful population guide for predicting DGF in the population for which it was intended, but has limited scope in expanded populations (SPK, pump) and for individual risk prediction. DGF risk prediction can be improved by inclusion of recipient cardiovascular disease.

INTRODUCTION

Delayed graft function (DGF) after kidney transplantation is associated with decreased patient and graft survival, an increased rate of acute rejection episodes, and increased healthcare costs.^1–13 While many definitions of DGF have been proposed, the most commonly used clinical outcome definition is the need for dialysis in the first posttransplant week.^14,15 With this definition, DGF occurs in ~25% deceased donor (DD) kidney transplants in the United States – i.e., ~ 3500 transplants/year; and with the increasing numbers of transplants from donors with high kidney donor profile index (KDPI) and/or donation after circulatory death, the rate is increasing.¹⁶ Minimizing the incidence of DGF, or its impact, could significantly improve overall DD posttransplant outcomes.

Defining a population at increased risk of posttransplant DGF, either to provide preemptive therapy (if such therapy existed) or to enrich study enrollment for a prospective randomized trial to minimize DGF, would be of significant benefit. In 2003, Irish et al, using United States Renal Data Systems Registry data, described a nomogram to help identify recipients at increased risk of DGF;¹⁷ a revised nomogram, based on more recent data and including the variables listed in Table 1a, was published in 2010.¹⁸

Table 1a:

Variables included in the Irish 2010 DGF prediction nomogram

During the course of the Deterioration of Kidney Allograft Function (DeKAF) study, 2 DeKAF centers participated in a randomized study to determine the efficacy of a new drug – given before revascularizing the kidney - to minimize the incidence and impact of DGF after a DD transplant. To enrich enrollment, the Irish formula was used to identify a subgroup at high risk for DGF. For both centers, in the subgroup screened as ineligible based on Irish formula predicted risk, the incidence of DGF was substantially higher than predicted.

To this end, we undertook a detailed analysis of data from the DeKAF prospective cohort with the goal to improve the performance of the Irish model. We first assessed the predictive performance of the Irish formula in our patient population, using the population strictly meeting Irish inclusion criteria. We then analyzed the performance of the model in 2 groups excluded by Irish: pump-perfused kidneys and simultaneous pancreas kidney (SPK) transplants. Further, given that some reports use a modified definition of DGF, defined as dialysis within the first transplant week excluding a single dialysis in the first 24 hours, we investigated the performance of the Irish calculator when this modified definition was used. We assessed model performance including other patient-specific data known at the time of transplant, to see if it could improve DGF prediction. Finally, we evaluated the patient-specific variables which improved DGF prediction in DeKAF in an external cohort. These analyses may provide guidance to investigators interested in using predictive calculators for study enrollment and highlight the important tradeoff between increasing the likelihood that an enrolled participant will have the condition of interest, such as DGF, and increasing screening cost.

MATERIALS & METHODS

DeKAF Data

Information regarding the full DeKAF study is available at www.clinicaltrials.gov (NCT00270712) and a detailed description of the DeKAF prospective cohort has been published.^{19, 20} Between October 1, 2005 and April 20, 2011 1626 DD recipients were enrolled in the DeKAF prospective cohort. Donor and recipient demographic information were collected after informed consent. Of the 1626, the presence of DGF (defined as dialysis within the first 7 days posttransplant) could be determined in all but 18 individuals (n=1608, Figure 1). Of these, 210 were excluded based on three exclusion criteria used by Irish et al. (2010): age of recipient under 16 years, pre-emptive transplant, or grafts lost on the day or day after transplant; 3 were excluded because of graft failure within the first posttransplant week; and 499 were excluded due to missing values for the recipient and donor covariates included in the Irish formula (listed in Table 1a). While warm ischemia time is a covariate in the Irish model, this value was not collected in DeKAF study. Per the default specified in the online Irish DGF prediction calculator, each individual’s warm ischemia time was imputed to be 45 for use in the Irish model. (Subsequently, the online Irish DGF calculator has been made unavailable.)

Figure 1.

Diagram of Study Cohorts and Analyses. ¹Excluding: pre-emptive transplants, recipient age<16, or grafts lost within 24 hours after transplant. ²Recipient: race, gender, BMI, previous kidney transplant, primary cause of kidney disease, peak PRA, pre-transplant transfusions, HLA mismatches, duration of dialysis pre-transplant. Donor: age, donation after cardiac death, history of hypertension, terminal creatinine, cause of death, weight, cold ischemia time.

For our analyses, we studied 4 non-exclusive cohorts: The first, the “Irish” cohort (n=478), was defined by applying all Irish exclusion criteria, which additionally excluded pump- perfused kidneys and SPK transplants. The second cohort, the “pump” cohort, (n=292) consisted of pump-perfused kidneys but met all other Irish inclusion criteria. The third cohort, the “SPK” cohort (n=72) consisted of SPK transplants but met all other Irish inclusion criteria. The fourth cohort, the “expanded” cohort (n=896), consisted of the first, second, and third cohort, but met all other Irish inclusion criteria (Figure 1).

Recipient cardiovascular disease (CVD) was defined as having a prior diagnosis of any of the following conditions: coronary artery disease requiring drug treatment or revascularization, congestive heart failure, myocardial infarction, peripheral vascular disease excluding venous disease, cerebrovascular disease requiring drug treatment or revascularization, or stroke.

External Validation Cohort

Data on all non-preemptive, DD, kidney and SPK transplantations between October 1, 2005 and April 20, 2011 (the same time period as DeKAF) was obtained from SRTR standard analysis files. The data was restricted to single organ (with the exception of pancreas) transplants from adult donors (≥16 years) to adult recipients (≥16 years) with a graft failure time greater than 7-days post-transplant. Only individuals with non-missing values for DGF and the covariates listed in Table 1a (with the exception of warm ischemia time) were included in the validation data set (n= 38,965). For this cohort, recipient cardiovascular disease (CVD) was defined as having any CVD related conditions recorded in the SRTR file: coronary artery disease (CAN_ANGINA_CAD), symptomatic peripheral vascular disease (CAN_PERIPH_VASC), or symptomatic cerebrovascular disease (CAN_CEREB_VASC).

Statistical analyses

Statistical analyses were performed in SAS 9.4 (SAS Institute Inc., Cary, NC) and R version 3.2.3. Continuous values were summarized using means and standard deviations (SD) and compared using t-tests. Categorical variables were summarized using proportions and compared using chi-squared tests or fisher’s exact tests, as appropriate.

In each cohort, the predictive accuracy of the “Irish calculator” was evaluated using receiver operating characteristic (ROC) curves and the c-index (area under the ROC curve) with 95% confidence intervals.²¹ Analyses were repeated using a modified definition of DGF as “dialysis within the first 7 days posttransplant except for a single dialysis within 24 hours posttransplant.”

We next evaluated if the Irish model prediction could be improved by addition of other variables collected in the DeKAF study (Table 1b). Each variable was evaluated as a potential predictor of DGF using logistic regression models with the Irish logit prediction as an offset. Each variable was evaluated univariately and after adjustment for clinical center. In addition, we evaluated the net reclassification improvement (NRI) for each variable based on comparing a model with coefficients for the variable and clinical centers and an offset for the Irish logit prediction to a model with just coefficients for the clinical centers and an offset for the Irish logit prediction. The NRI quantifies the likelihood of a new algorithm to provide a higher risk for individuals who have the event and a lower risk for individuals without the event compared to an old algorithm.²² The NRI was calculated using the continuous NRI measurement proposed by Pencina et al. with 95% bootstrap confidence intervals as provided by the nricens R package.²³

Table 1b:

Additional variables assessed in this paper for DGF prediction

Variables which showed improvement in predicting DGF risk in DeKAF were then incorporated into models composed of a coefficient for the variable with an offset for the Irish logit prediction. The models were evaluated in the external SRTR validation cohort using the c-index with 95% confidence intervals.

RESULTS

Description of the Three Cohorts

Data on 896 DD transplants in the DeKAF prospective cohort were available for evaluating the Irish DGF prediction model (Figure 1). Demographics of the 4 cohorts are shown in Table 2 with a comparison of those excluded due to missing values given in Supplementary Table S1. The incidence of DGF was highest in the cohort meeting Irish inclusion criteria (24.3%, n=478), and lowest in the SPK cohort (13.9%, n=72), with intermediate levels in the pump perfused (18.2%, n=292), and full expanded cohorts (20.4%, n=896; Supplemental Table S2).

Table 2:

Recipient and donor demographics and characteristics for the cohorts. Mean +/− SD and N(%) are presented.

	Irish inclusion criteria (N=478)	Pump cohort (N=292)	SPK (N=72)	Expanded cohort (N=896)	P-value: Pump vs Irish	P-value: SPK vs Irish
Recipient Demographics
Age at transplant (years)	52.3 ± 13.1	52.1 ± 12.7	45.1 ± 8.2	51.2 ± 12.7	0.8442	≤0.0001
Race					≤0.0001	0.0005
White	325 (68.0%)	133 (45.5%)	65 (90.3%)	543 (60.6%)
Black	72 (15.1%)	145 (49.7%)	3 (4.2%)	252 (28.1%)
Other	81 (16.9%)	14 (4.8%)	4 (5.6%)	101 (11.3%)
Sex					0.8248	0.0474
Female	188 (39.3%)	118 (40.4%)	19 (26.4%)	346 (38.6%)
Male	290 (60.7%)	174 (59.6%)	53 (73.6%)	550 (61.4%)
Recipient Clinical Characteristics
Previous kidney transplant	82 (17.2%)	37 (12.7%)	11 (15.3%)	132 (14.7%)	0.1171	0.8200
Primary cause of kidney disease					≤0.0001	0.0005
Diabetes	130 (27.2%)	74 (25.3%)	72 (100.0%)	308 (34.4%)
Glomerular disease	103 (21.5%)	45 (15.4%)	0 (0%)	151 (16.9%)
Hypertensive nephrosclerosis	60 (12.6%)	82 (28.1%)	0 (0%)	156 (17.4%)
Polycystic Kidney Disease	63 (13.2%)	26 (8.9%)	0 (0%)	90 (10.0%)
Other	101 (21.1%)	56 (19.2%)	0 (0%)	160 (17.9%)
Unknown	21 (4.4%)	9 (3.1%)	0 (0%)	31 (3.5%)
CVD	168 (35.1%)	85 (29.1%)	30 (41.7%)	292 (32.6%)	0.0987	0.3458
Former/Current smoker	217 (45.4%)	128 (43.8%)	32 (44.4%)	391 (43.6%)	0.8032	0.9685
Diabetes	172 (36.0%)	114 (39.0%)	72 (100.0%)	395 (44.1%)	0.4382	≤0.0001
Hypertension	415 (86.8%)	256 (87.7%)	70 (97.2%)	789 (88.1%)	0.8170	0.0186
Peak PRA (% increase)	25.1 ± 35.5	26.4 ± 33.6	15.8 ± 29.3	24.5 ± 34.1	0.6218	0.0162
Pre-transplant transfusions	163 (34.1%)	45 (15.4%)	36 (50.0%)	251 (28.0%)	≤0.0001	0.0129
Number of HLA mismatches (A, B, or DR)	3.5 ± 1.9	3.8 ± 1.8	3.9 ± 1.6	3.7 ± 1.8	0.0354	0.0608
BMI (kg/m2)	28.1 ± 5.6	29.3 ± 5.7	26.4 ± 4.4	28.3 ± 5.6	0.0030	0.0045
Duration of dialysis pre-transplant (days)	1384.8 ± 997.3	1867.4 ± 1114.0	905.7 ± 553.0	1512.6 ± 1066.8	≤0.0001	≤0.0001
Plasmapheresis prior to transplant					0.1626	0.5651
No	461 (96.4%)	291 (99.7%)	68 (94.4%)	874 (97.5%)
Yes	5 (1.0%)	0 (0.0%)	1 (1.4%)	6 (0.7%)
Simultaneous pancreas transplant	0 (0.0%)	0 (0.0%)	72 (100.0%)	98 (10.9%)
Pump-perfused kidney	0 (0.0%)	292 (100.0%)	0 (0.0%)	310 (34.6%)
Donor Demographics
Age (years)	37.3 ± 17.3	41.0 ± 16.6	29.1 ± 12.2	37.4 ± 16.9	0.0034	≤0.0001
Age greater than 60	33 (6.9%)	44 (15.1%)	0 (0.0%)	80 (8.9%)	0.0004	0.0144
Sex					0.7862	0.6288
Female	214 (44.8%)	127 (43.5%)	35 (48.6%)	389 (43.4%)
Male	264 (55.2%)	165 (56.5%)	37 (51.4%)	507 (56.6%)
Donor Clinical Characteristics
Weight (kg)	76.8 ± 24.0	77.0 ± 21.8	74.2 ± 15.1	76.8 ± 22.5	0.8992	0.2321
BMI (kg/m2)	26.4 ± 6.7	26.4 ± 6.4	24.7 ± 4.0	26.2 ± 6.4	0.9280	0.0030
BMI mismatch**	369 (77.2%)	240 (82.2%)	49 (68.1%)	695 (77.6%)	0.1445	0.1072
Cold ischemia time (hours)	16.4 ± 7.1	20.5 ± 8.5	12.7 ± 5.8	17.6 ± 7.9	≤0.0001	≤0.0001
Donation after circulatory death	33 (6.9%)	32 (11.0%)	0 (0.0%)	71 (7.9%)	0.0672	0.0144
Donor history of hypertension	94 (19.7%)	76 (26.0%)	5 (6.9%)	182 (20.3%)	0.0482	0.0141
Terminal creatinine (mg/dL)	1.0 ± 0.7	1.1 ± 0.6	0.9 ± 0.4	1.0 ± 0.6	0.1265	0.2728
Extended criteria donor					0.0061	0.0002
No	410 (85.8%)	223 (76.4%)	71 (98.6%)	752 (83.9%)
Yes	63 (13.2%)	60 (20.5%)	0 (0.0%)	126 (14.1%)
Primary cause of death					0.0708	0.1269
Anoxia	76 (15.9%)	52 (17.8%)	10 (13.9%)	143 (16.0%)
Cerebrovascular/Stroke	165 (34.5%)	104 (35.6%)	24 (33.3%)	304 (33.9%)
Head Trauma	196 (41.0%)	126 (43.2%)	37 (51.4%)	396 (44.2%)
Other	39 (8.2%)	10 (3.4%)	1 (1.4%)	51 (5.7%)

^**Absolute difference between recipient and donor BMI>2

Evaluation of Model Accuracy in the Three Cohorts

For recipients in the Irish cohort (n=478), the discrimination of the Irish calculator as assessed by the c-index was comparable to the c-index reported in Irish 2010 (0.701, 95% CI: 0.646 to 0.756; vs. 0.704 in Irish et al; Figure 2, Table 3). The median (inter quartile range, IQR) Irish calculator score was 0.320 (0.213 – 0.449) in the patients that later experienced DGF, compared with the 0.206 (0.128 – 0.316) in the patients without DGF (Figure 3a, Supplementary Table S2). The distribution of scores demonstrated substantial overlap in predicted DGF risk between the two groups (Figure 3a). More than 25% of those who experienced DGF had scores less than 0.20, the approximate median score for those who did not experience DGF. Similarly, more than 25% of participants who did not experience DGF had scores exceeding 0.3, the approximate median score for individuals experiencing DGF. Given the substantial overlap, an optimal threshold for classifying individuals at risk of DGF is unclear.

Figure 2:

ROC curves for Irish calculator implemented in the Irish cohort (N=478; red line), the pump cohort (N=292; blue line), the SPK cohort (N=72; black line) and the expanded cohort (N=896; green line).

Table 3:

Comparison of Irish model predictive accuracy (C-indices with 95% CI) by cohort. “Any DGF” is defined as any dialysis within 7 days of transplant. “Modified DGF” excludes a single dialysis within the first day post-transplant is defined as

	Irish	Pump	SPK	Expanded
Any DGF	0.701 (0.646 to 0.756)	0.730 (0.653 to 0.807)	0.706 (0.546 to 0.867)	0.683 (0.64 to 0.726)
Modified DGF	0.705 (0.648 to 0.763)	0.737 (0.652 to 0.822)	0.721 (0.549 to 0.894)	0.686 (0.64 to 0.731)

Figure 3:

Distribution of Irish scores for those who did and did not experience DGF by cohort.

When the Irish calculator was applied to the pump cohort, the performance as assessed by the c-index was slightly higher than the performance in in the Irish population (c-index=0.730; 95% CI: 0.653 to 0.807; Figure 2, Table 3). While the actual prevalence of DGF was much lower in this cohort (18.2%) compared to the Irish cohort (24.3%), the Irish calculator predicted probability of DGF was much higher with a median (IQR) score of 0.519 (0.356 – 0.610) in the patients that experienced DGF and 0.320 (0.221 – 0.442) in the patients that did not (Supplementary Table S2, Figure 3b). This indicates that the predicted DGF risk for the pump-perfused population is not directly comparable to the predicted risk of those meeting the Irish inclusion criteria. However, we do see a slight improvement in the separation between groups than was seen in the Irish cohort, with less than 25% of those who did not experience DGF having scores exceeding 0.5, the approximate median for those who had DGF, and less than 25% of those who had DGF having scores less than 0.3, the approximate median for those who did not experience DGF.

Participants in the SPK cohort had the lowest predicted probability of DGF, reflective of the low incidence of DGF (13.9%) in this population, with median (IQR) Irish calculator score of 0.166 (0.110 – 0.236) in the patients that did not experience DGF, compared with the 0.255 (0.178 – 0.280) in the patients with DGF (Supplementary Table S2). The discrimination of the Irish calculator as assessed by the c-index was comparable to the c-index from the Irish cohort (c=0.706, 95% CI: 0.546 to 0.867; Figure 2, Table 3). However, examining the distribution of scores (Figure 3c), we can see that there is near complete overlap between the predicted scores for individuals experiencing and not experiencing DGF (Figure 3c). This makes defining a threshold for those at highest risk of developing DGF infeasible.

For the expanded cohort (Irish cohort, plus the pump and SPK cohorts), the median Irish calculator score in patients that experienced DGF (0.359, IQR: 0.246 – 0.525) vs did not experience DGF (0.245, IQR: 0.151 – 0.365) was comparable to the Irish cohort (Supplementary Table S2, Figure 3), but the Irish calculator had slightly poorer predictive accuracy in this cohort (c-index= 0.683, 95% CI: 0.64 to 0.726; Figure 2, Table 3), driven by inclusion of both SPK and pump perfused kidneys in the expanded cohort. Approximately 25% of those who truly experienced DGF had scores less than 0.25, the approximate median score for those who did not experience DGF; more than 25% of participants who did not experience DGF had scores exceeding 0.35, the approximate median score for individuals experiencing DGF. Again, given the substantial overlap (Figure 3d), it is unclear what the optimal threshold is for classifying individuals at risk of DGF.

Modified definition of DGF

We next reassessed the predictive accuracy of the models using a modified definition of DGF that excludes individuals with only a single dialysis in the first 24 hours postoperatively. With this modified definition, 19 individuals (out of 116) in the Irish cohort, 10 (out of 53) in the pump cohort, 1 (out of 10) in the SPK cohort, and 30 (out of 183) in the expanded cohort were reclassified as not having DGF. For the Irish, pump, and expanded cohort, the c-indices using the modified DGF definition were nearly identical to the original definition (Table 3). Similarly, the median Irish calculator predicted probability of DGF in the patients that experienced vs did not experience DGF was very similar as when the original definition was used (Supplemental Table S3, Supplemental Figure S1). For the SPK cohort, we identify a slight, but not statistically significant, increase in c-index using the modified definition (0.721, 95% CI: 0.549 to 0.894; vs. 0.706, 95% CI: 0.546 to 0.867), but predicted probabilities of DGF were very similar as when the original definition was used.

Assessment of Additional Risk Factors for DGF

Additional potential predictors of DGF were examined (Table 1b) using the expanded cohort (Table 4). Pump perfusion was significant by univariate analysis (odds ratio=0.39, p≤0.0001), but not after adjustment of clinical center (odds ratio=0.69, p=0.1033), likely due to the majority (65.8%) of pump perfusion occurring at a single center. Smoking history and recipient age were marginally associated by univariate analysis but not after adjustment for clinical center.

Table 4:

Logistic regression odds ratios (95% CIs) and p-values for potential predictors of DGF above and beyond the Irish prediction (model offset) in the expanded cohort (N=896).

		Odds Ratio (95% CI), p-value		NRI (95% CI)
Variable	N	Model with variable and Irish logit	Model with variable and Irish logit, adjusted for clinical center
Male donor to female recipient	896	0.84 (0.53, 1.31) p=0.4382	0.88 (0.55, 1.41) p=0.5972	0.132 (0.005, 0.253)
Donor gender (male)	896	0.95 (0.67, 1.36) p=0.7934	0.98 (0.68, 1.41) p=0.9045	−0.020 (−0.166, 0.142)
Pump-perfusion	860	0.39 (0.26, 0.56) p≤0.0001	0.69 (0.44, 1.08) p=0.1033	0.207 (0.042, 0.358)
SPK	896	1.02 (0.52, 1.97) p=0.9611	0.86 (0.42, 1.74) p=0.6696	0.124 (0.032, 0.209)
ECD	878	0.88 (0.55, 1.40) p=0.5768	0.81 (0.49, 1.32) p=0.3922	−0.136 (−0.254, −0.010)
BMI mismatch	894	1.19 (0.77, 1.83) p=0.4323	1.18 (0.75, 1.84) p=0.4780	0.065 (−0.066, 0.190)
CVD	896	2.39 (1.68, 3.41) p≤0.0001	1.88 (1.30, 2.73) p=0.0009	0.390 (0.238, 0.552)
Congestive Heart failure	896	2.44 (1.20, 4.98) p=0.0140	1.92 (0.91, 4.03) p=0.0855	0.083 (0.010, 0.169)
MI	896	1.78 (0.95, 3.35) p=0.0732	1.09 (0.56, 2.11) p=0.8062	0.074 (−0.015, 0.163)
PVD	896	2.41 (1.27, 4.55) p=0.0069	1.48 (0.75, 2.91) p=0.2598	0.112 (0.026, 0.206)
Cerebrovascular disease/ Stroke	896	1.73 (0.94, 3.21) p=0.0807	1.83 (0.97, 3.45) p=0.0623	0.070 (−0.021, 0.171)
CAD	896	1.90 (1.30, 2.79) p=0.0010	1.45 (0.96, 2.17) p=0.0744	0.230 (0.094, 0.381)
Former/current smoker	892	1.55 (1.09, 2.20) p=0.0141	1.24 (0.86, 1.79) p=0.2427	0.190 (0.027, 0.356)
Recipient diabetes	896	1.05 (0.74, 1.49) p=0.7949	0.98 (0.68, 1.42) p=0.9311	−0.059 (−0.219, 0.111)
Recipient hypertension	896	1.27 (0.74, 2.19) p=0.3891	1.07 (0.60, 1.90) p=0.8174	0.012 (−0.090, 0.114)
Recipient age (minus 16)	896	1.02 (1.00, 1.03) p=0.0159	1.01 (1.00, 1.03) p=0.0872	0.208 (0.047, 0.364)
Donor BMI	894	1.02 (1.00, 1.05) p=0.0987	1.01 (0.99, 1.04) p=0.3504	0.355 (0.191, 0.522)

Among the variables investigated, only recipient CVD was strongly associated with DGF both by univariate analysis (odds ratio=2.39, p≤0.0001) and after adjusting for clinical center (odds ratio=1.88, p=0.0009) and was also associated with improved classification (NRI=0.390, 95% CI: 0.238 to 0.552). The components of the CVD composite measure most strongly associated DGF were cerebrovascular disease/stroke (odds ratio=1.83, p=0.0623; NRI=0.070, 95% CI: −0.021 to 0.171), coronary artery disease (CAD) (odds ratio=1.45, p=0.0744; NRI=0.230, 95% CI: 0.094 to 0.381), and congestive heart failure (odds ratio=1.92, p=0.0855; NRI=0.083, 95% CI: 0.010 to 0.169). However, the individual components of the CVD composite measure were not significantly associated with DGF after adjusting for clinical center.

Evaluation of CVD as Risk Factor for DGF in External Cohort

Given the strong association between CVD and DGF identified in DeKAF, we further assessed this association in an external validation cohort of DD transplants obtained from SRTR standard analysis files. A comparison of the SRTR cohort to the DeKAF expanded cohort is provided in Supplementary Table S4. Of the 38,965 transplants initially included in the analysis, only 70% (27350) had information regarding CAD, cerebrovascular disease, and peripheral vascular disease (PVD), with 15.2% (4870) recording having at least one of these conditions. This is a much lower proportion than the 32.6% in DeKAF who reporting having at least one CVD related condition (which additionally included congestive heart failure and myocardial infarction). Not surprisingly, given the difference in definition (see Methods) and scope of CVD information collected in DeKAF and SRTR, we did not find that models based on the Irish calculator plus the addition of a DeKAF-based coefficient for cerebrovascular disease/stroke, PVD, CAD, or CVD were associated with improved risk prediction in the SRTR cohort compared to the Irish calculator alone (Supplementary Table S5). However, we did find that when evaluated univariately within the SRTR cohort, CVD was significantly associated with DGF (odds ratio=1.11, p=0.0094) and improved DGF classification (NRI=0.065, 95% CI: 0.044 to 0.084; Supplemental Table S6), albeit with a more modest association than was seen in the DeKAF cohort, which included additional conditions in the definition of CVD.

DISCUSSION

Over 4 decades, there has been significant effort to lower the incidence of DGF. However, over the same interval the DD organ pool has expanded to include DD kidneys that would have been discarded 30 years ago (e.g., high KDPI). These kidneys are associated with increased risk of DGF.^{10, 12, 16} To date, there are no Food and Drug Administration (FDA) approved therapies for DGF minimization. Positive findings in recent pre-clinical studies have not translated into successful therapies in humans (reviewed in²⁴). Recently, for drug development studies, the FDA has provided guidance to industry, allowing enrichment of the study population at risk for DGF.²⁵

There have been previous attempts to improve the Irish nomogram for DGF prediction.^26–29 When compared in independent datasets, the Irish model has outperformed the others.^29–32 However, when used in recent clinical trials to enrich for those with DGF, the nomogram has been inconsistent in identifying individual recipients with DGF.

The goal of the current study was to leverage the large, detailed, multicenter DeKAF prospective database in an attempt to evaluate and improve individual DGF risk prediction. In evaluating the Irish nomogram in data restricted to meet the 2010 Irish criteria (excluding SPK and pump-perfused kidneys), we found comparable results as given in Irish et al (c-index of 0.701 vs. 0.704). This is not surprising given that the DeKAF cohort is comprised of recipients transplanted in the same era as that used for the Irish model; and that Irish inclusion and exclusion criteria were used. However, the c-index only evaluates if those who truly develop the outcome have higher predicted risk than those who do not. It does not evaluate if the predicted probability of the outcome reflects the true risk in the population. As noted above, there was a large overlap in the scores of those who did and did not have DGF (Figure 3).

For pump-perfused kidneys, the Irish nomogram discriminated the data well (c-index, 0.730), but the predicted risk of DGF was much higher than in the population receiving non-pump-perfused kidneys, even though the true prevalence of DGF was lower. This suggests that if the Irish model is applied to “pumped” kidneys there may be an over-estimation of DGF risk.

Among individuals with SPK there was a very low risk of DGF (13.9%) and low Irish DGF scores, but there was no meaningful separation between risk scores between those experiencing and not experiencing DGF. It follows that in the expanded cohort, which includes both SPK and pump-perfused kidneys, the predictive accuracy of the Irish model was less than ideal (c-index, 0.683; 95% CI: 0.64 to 0.726). These findings indicate that severe caution should be exercised if the Irish calculator is used in populations which were excluded in development of the model.

Given that the impact of a single dialysis in the first 24 hours of surgery does not pose the same clinical risk as the need for dialysis between 24 hours and 7 days posttransplant.^33–35, some new studies are defining DGF as dialysis within the first transplant week excluding a single dialysis in the first 24 hours. We repeated our analyses using this definition. For all cohorts, the results were similar if not slightly improved to those using the traditional DGF definition (Table 3, Supplemental Table S3). This indicates that it may be feasible to apply this calculator to predict DGF risk when using this modified definition.

Evaluation of CVD and other risk factors.

We also studied whether adding additional variables (6 donor; 6 recipient) (Table 1b) would improve risk prediction. Of the variables evaluated, only recipient CVD (as defined in Methods), was strongly associated with DGF risk (and improved classification as assessed by the NRI) even after adjustment for the risk of DGF predicted by the Irish nomogram. The components of the CVD composite definition most strongly associated with DGF were cerebrovascular disease/stroke, CAD, and congestive heart failure. In a single center analysis, Decruyenaere et al. also reported that recipient reduced cardiac function was associated with increased DGF risk.²⁹ Recently, other risk factors for DGF have been identified (pre-transplant use of midrodrine, pre-transplant pulmonary hypertension, and high output in an AVF)^36–38, but as this information was not collected in DeKAF, we were not able to confirm these findings in our population.

When models consisting of a DeKAF-based coefficient for CVD, CAD, PVD, or cerebrovascular disease plus the Irish calculator DGF prediction were evaluated in an external cohort obtained from SRTR standard analysis files we did not find that these models improved risk prediction compared to the Irish calculator alone. However, when evaluated just within the SRTR cohort, we did find that having at least one CVD-related condition (CAD, PVD, and/or cerebrovascular disease) was associated with DGF even after accounting for the Irish predicted DGF risk.

There are several possible explanations for why the relationship between CVD and DGF was weaker in the SRTR cohort (odds ratio=1.11, 95% CI: 1.03 to 1.21) compared to the DeKAF cohort (odd ration=2.39, 95% CI: 1.68 to 3.41, prior to adjusting for clinical center). First, the information on CVD related conditions collected in DeKAF was much more expansive (CAD, congestive heart failure, MI, PVD, cerebral vascular disease, and stroke) compared to the information collected in SRTR (CAD, PVD, and cerebrovascular vascular disease). Second, in DeKAF, CVD was systematically evaluated whereas 30% of SRTR transplants had missing data. Third, the definitions of the individual components of CVD differed (e.g. prior diagnosis of PVD excluding venous disease in DeKAF vs. symptomatic PVD in SRTR). In the SRTR cohort 15.2% of individuals had at least one CVD related condition compared to 32.6% in the DeKAF expanded cohort: 7.3% with CAD (compared to 23.1% in DeKAF), 4.7% with PVD (compared to 5.9% in SRTR), and 3.4% with cerebrovascular disease (compared to 7.0% in DeKAF; Supplementary Table S4). It is likely that a large portion of the population who have experienced CVD-related conditions are missed and/or underreported in the SRTR database. Fourth, there were differences in characteristics between the SRTR and DeKAF expanded cohort. Recipients in the SRTR cohort had a lower BMI, more HLA mismatches and were less likely to be white, have had a previous kidney transplant, diabetes, pre-transplant transfusions, SPK, and/or pump-perfusion (supplementary table S4). Donors in the SRTR cohort were older, heavier, more likely to have a history of hypertension and/or be extended criteria donors, had a higher terminal creatinine, and were more likely to have had a primary cause of death of anoxia or cerebrovascular disease/stroke compared to DeKAF donors. It is possible the association between CVD and DGF may be weaker in this population or the increased frequency of DGF in this population (25% vs. 20.4% in DeKAF expanded cohort) may already be mostly accounted for by the Irish DGF risk prediction. In addition, those with CVD related conditions in the SRTR cohort were less likely to have a prior KTX and had a lower peak PRA (supplementary table S7) compared to those without CVD (whereas we did not find differences in these characteristics in DeKAF) which may explain some of the attenuation in DGF risk for individuals with CVD.

Overall, these findings suggest that with more systematic and expanded CVD information collected by SRTR, individual DGF risk prediction may be improved.

Limitations for individual risk prediction.

The challenges of trial design for studies of interventions to prevent or minimize DGF were discussed at a 2011 FDA workshop.³⁹ To date, the Irish calculator has been shown to be useful for population risk prediction but has been problematic in predicting individual risk. Grossberg et al concluded that “on a case-by case basis” the calculator “did not contribute meaningful information”.⁴⁰ While the c-statistic indicates the tendency of a predictive algorithm to assign higher scores to those who experience the condition, compared to those that who do not, it does not indicate if the predictive algorithm provides scores consistent with the probability of the event in the population. Similarly, it does not indicate at what threshold a score should be considered an indication that one is at high risk for experiencing the condition.

Using the Irish calculator for screening requires a pre-specified cut-off value to enroll a potential recipient. Zhang et al, in a study of 711 DD recipients where DGF incidence was only 17.6%, noted that using a cut-off of 0.1 for defining a participant as high risk of experiencing DGF, resulted in the best sensitivity (82.35%) and negative predictive value (93.08%).³² Using a cut-off of 0.5 gave the best specificity (98.01%) and positive predictive value (61.54%). They suggested that the optimal cut-off was 0.2 (sensitivity, 60.78%; specificity, 75.3%).

In the population of DeKAF kidney recipients meeting the Irish inclusion criteria, DGF incidence was 24.2%. If all individuals were enrolled in a controlled randomized trial of a drug aimed at reducing DGF, regardless of predicted risk, 318 individuals would need to be enrolled to detect a 50% reduction in DGF incidence with 80% power. Instead, if the study population was enriched (using the Irish calculator) such that only individuals with a Irish calculator score greater than 0.25 were enrolled in the trial (the same threshold that was used in the randomized DGF drug trial at some DeKAF centers), only 190 individuals would need to be enrolled to have comparable power. This is because the estimated baseline rate of DGF increases from 24.3% to 36.0% in the selected study population (positive predicted value, Table 5). However, the increase in power (decrease in enrollment size) by using this threshold comes at an increased cost of screening, as 404 individuals (assuming all consented to participate) need to be screened to have an estimated 190 (47.1%) who have a predicted DGF risk exceeding the threshold of 0.25. In addition, using this screening threshold would miss enrolling an estimated 30 individuals who would later develop DGF, approximately 30% of all DGF cases in the population.

Table 5:

Diagnostic performance for various Irish prediction cut-offs in the Irish cohort (N=478). In this population the probability of having DGF is 24.3%. T+: “Likely to have DGF”. T-: Unlikely to have DGF”. D+: Truly have DGF. D-: Truly do not have DGF.

Cutoff for T+	Proportion T+	Sensitivity (T+/D+)	Specificity (T−/D−)	Positive Predictive Value (D+/T+)	Negative Predictive Value (D−/T−)	False Positive Rate (T+/D−)	False Negative Rate (T−/D+)	Proportion Correctly Classified	Youden’s index*
0	1.000	1.000	0.000	0.243	NA	1.000	0.000	0.243	0.000
0.1	0.891	0.974	0.135	0.265	0.942	0.865	0.026	0.339	0.109
0.2	0.582	0.776	0.481	0.324	0.87	0.519	0.224	0.552	0.257
0.25	0.471	0.698	0.602	0.360	0.862	0.398	0.302	0.626	0.300
0.3	0.356	0.586	0.718	0.400	0.844	0.282	0.414	0.686	0.304
0.4	0.186	0.371	0.873	0.483	0.812	0.127	0.629	0.751	0.244
0.5	0.088	0.207	0.950	0.571	0.789	0.05	0.793	0.770	0.157
0.6	0.040	0.086	0.975	0.526	0.769	0.025	0.914	0.759	0.061
0.7	0.023	0.060	0.989	0.636	0.767	0.011	0.940	0.764	0.049
0.8	0.010	0.034	0.997	0.800	0.763	0.003	0.966	0.764	0.032
0.9	0.002	0.009	1.000	1.000	0.759	0.000	0.991	0.759	0.009
1	0.000	0.000	1.000	NA	0.757	0.000	1.000	0.757	0.000

^*The difference between sensitivity and 1-specificity

Importantly, screening costs are not trivial. Deceased donor transplants are done at any time of the day, 7 days a week. If a risk calculator score is to be used as an enrollment criterion (e.g., for randomization to an experimental treatment prior to revascularization), the recipient must be fully screened and consent obtained before going to the operating room. To have coordinators available around the clock requires considerable resources.

In our study population, when using the Youden’s index (i.e. the difference between sensitivity and 1-specificity⁴¹) as suggested by Zhang et el.³², the best differentiation between those with and without DGF is achieved with a cut-off of 0.3 (Table 5). Even with this cut-off, this choice is not meaningful for individual prediction of DGF as only 40.0% of individuals who have a predicted risk exceeding this threshold actually experience DGF. If this threshold were used for study enrollment over 460 individuals would need to be screened and 164 enrolled to have 80% power to detect a 50% reduction in DGF rate.

To determine the optimal cut-point for study planning, the investigative team must carefully weigh the importance of DGF event rate, screening costs, and the specific study population in planning the study. As we have illustrated, to increase event rate, an important determinant of study power, the threshold for classification can be increased. Importantly, however, this increases the overall cost of screening, since fewer individuals will have a predicted risk exceeding this threshold, and more individuals must be screened to have a reasonable study sample size. In addition, the probability of missing individuals who truly will experience DGF increases. What is feasible in terms of enrollments at various success rates of screening depends on the number and sizes of centers in the study, the difficulty of recruiting each participant, and the estimated prevalence of the condition in the study population. Table 5 illustrates measures of diagnostic accuracy for the Irish et al 2010 model applied to the cohort of DeKAF individuals meeting the Irish inclusion criteria. This table provides guidance for investigators planning a study among a similar cohort of individuals. However, if the study population includes pump-perfused kidneys or SPKs both the Irish nomogram predictive performance and the incidence of DGF will be impacted. In that case, Supplementary Tables S8–S10 should be consulted as well.

In summary: 1) We found that the Irish calculator gave comparable performance in the DeKAF population meeting the Irish exclusion criteria as in the population analyzed by Irish et al 2010. 2) We demonstrated that the Irish model over-estimates the risk of DGF in pump-perfused kidneys. 3) When a modified DGF definition is used, excluding dialysis in the first 24 hours, the Irish model had similar performance as with the conventional definition of DGF across all 4 DeKAF sub-cohorts. 4) We found that adding recipient cardiovascular disease to the model improved the accuracy of predicting DGF. 5) Finally, we provide guidance about selecting the optimal Irish model DGF cut-point for clinical trial design.

ABBREVIATIONS

CAD

coronary artery disease

CVD

cardiovascular disease

DD

deceased donor

DeKAF

Deterioration of Kidney Allograft Function

DGF

Delayed graft function

FDA

Food and Drug Administration

IC

Irish calculator

IQR

inter quartile range

KDPI

kidney donor profile index

MI

myocardial infarction

NRI

net reclassification improvement

PVD

peripheral vascular disease

ROC

receiver operating characteristic

SPK

simultaneous pancreas kidney

SD

standard deviations