External Validation of the Kidney Failure Risk Equation and Re-Calibration with Addition of Ultrasound Parameters

Abstract

Background and objectives

Progression of CKD toward ESRD is heterogeneous. The Kidney Failure Risk Equation (KFRE) was developed to identify CKD patients at high risk of ESRD. We aimed to externally validate KFRE and to test whether the addition of predefined Duplex ultrasound markers – renal resistive index (RRI) or difference of resistive indices in spleen and kidney (DI-RISK) – improved ESRD prediction.

Design, setting, participants, & measurements

The prospective Cardiovascular and Renal Outcome in CKD 2-4 Patients—The Fourth Homburg evaluation (CARE FOR HOMe) study recruits CKD stage G2–G4 patients referred to a tertiary referral center for nephrologic care. Four hundred three CARE FOR HOMe participants enrolled between 2008 and 2012 had available RRI measurements at study inclusion; they were subsequently followed for a mean of 4.4±1.6 years. This subcohort was used to validate KFRE and to assess the added value of the ultrasound markers (new models KFRE+RRI and KFRE+DI-RISK). Model performance was assessed by log-likelihood ratio test, c-statistic, integrated discrimination improvement metrics (for study participants without subsequent ESRD [IDI _{No ESRD}] and for patients with ESRD [IDI _ESRD]), and calibration plots. If either new model improved on KFRE, we determined to validate it in an independent cohort of 162 CKD patients.

Results

KFRE predicted ESRD in CARE FOR HOMe participants with a c-statistic of 0.91 (95% confidence interval, 0.83 to 0.99). Adding RRI improved the KFRE model (P<0.001), and the KFRE+RRI model was well calibrated; however, the c-statistic (0.91 [0.83–1.00]) was similar, and overall sensitivity (IDI _{No ESRD}=0.05 [0.00–0.10]) or overall specificity (IDI _ESRD=0.00 [0.00–0.01]) did not improve. Adding DI-RISK did not improve the KRFE model. In the external validation cohort, we confirmed that the KFRE+RRI model did not outperform KFRE.

Conclusions

Routine Duplex examinations among CKD patients did not improve risk prediction for progression to ESRD beyond a validated equation.

Introduction

CKD is a major public health issue because of its high prevalence, adverse outcomes, and high cost (1,2). Of note, the majority of CKD patients have mild to moderate disease, and only a few patients ultimately require renal replacement therapy (3). Moreover, interindividual progression of CKD is very heterogeneous. Whereas some patients with mild to moderate CKD suffer a rapid decline in GFR with an early need for renal replacement therapy, others have stable kidney function for years (4). Recent KDIGO guidelines advocated the identification of CKD patients at higher risk of rapid progression, who may benefit from intensive preventive or therapeutic strategies (4).

Various models to predict CKD progression have been published (5,6); however, systemic review articles identified only two models with acceptable clinical utility and usability; namely, one from Johnson et al. and the Kidney Failure Risk Equation (KFRE) (7,8). Nevertheless, many physicians consider the accuracy and precision of current models for predicting CKD progression to be limited, which substantially hinders their integration into daily clinical routine.

Against this background, it has repeatedly been suggested that Duplex markers – particularly the renal resistive index (RRI) – may predict CKD progression (9–11). However, although elevated RRI predicted adverse renal outcome in several cohort studies (10–18), these were mostly small-sized, and RRI was not compared against established models for CKD progression. Moreover, there is concern that RRI may not provide organ-specific information on structural or functional abnormalities within the kidneys, but rather reflects systemic vascular disease. It has been postulated that calculating the difference of resistive indices in spleen and kidney (DI-RISK) may provide more specific information on renal damage; however, prospective data on the predictive value of DI-RISK are completely missing.

In the present study, we aimed to (1) externally validate KFRE to predict development of ESRD, (2) analyze whether addition of either RRI or DI-RISK improved the prediction of ESRD and, if either Duplex ultrasound marker did improve risk prediction, to (3) validate the new model in an external data set.

Materials and Methods

Data Sources

Two cohorts were used: (1) The Cardiovascular and Renal Outcome in CKD 2-4 Patients—The Fourth Homburg evaluation (CARE FOR HOMe) study recruited patients who visited the outpatient department of Saarland University Medical Center between September 2008 and November 2012 with CKD (GFR categories 2–4) (4,19,20). Excluded were pregnant women, allograft recipients, patients aged <18 years, and those who were HIV-positive, with clinically apparent infections, active malignancy and/or acute kidney injury (2). The Hannover cohort comprised CKD patients attending the outpatient department of Hannover Medical School between June 1995 and September 1999 who had at least one of the following characteristics: creatinine clearance of <75% of their normal value for age and sex, a proteinuria of 150 mg/d or more and/or hypertension, or other established CKD. In all patients, renal artery stenosis was excluded by color Doppler ultrasound.

Informed consent was obtained from all patients. Both studies adhered to the Declaration of Helsinki, and were approved by ethics committees in Saarbrücken and Hannover, Germany.

Outcome

The primary outcome was ESRD, defined as a hemodialysis session, insertion of a peritoneal dialysis catheter, or renal transplantation, whichever occurred first. In both cohorts, patients were invited for regular follow-up examination (annually for CARE FOR HOMe; at months 3, 6, and 12, and then annually for Hannover). Physicians blinded to the results of the Duplex ultrasound measurements treated the patients.

Predictors

CARE FOR HOMe was used to validate the KFRE four-variable model (referred to as model 3 in the original publication and in the following) (8), which was comprised of age (per 10 years), eGFR (per 5 ml/min per 1.73 m², according to the MDRD formula), sex, and urine albumin-to-creatinine ratio (ACR). eGFR and ACR were assessed as reported earlier (21). The KFRE prediction model formula with hazard ratios is:

where ln is the natural log function.

A detailed description of ultrasound studies applied in both cohorts has been published previously (12,19). Briefly, in CARE FOR HOMe, ultrasound was performed with a Sequoia C512 unit (Acuson, Thousand Oaks, CA), using a vector probe (model 4V1; 1–4 MHz, Acuson) by trained sonographers who were supervised by a single nephrologist with long-term experience in ultrasound studies. RRI was the mean of the measurements in three interlobar arteries along the border of the medullar pyramids in the upper, middle, and lower pole of each kidney. In the spleen, Doppler spectra were obtained from the segmental branches of the splenic artery at the entry into the parenchyma to reach comparable conditions.

Resistive indices were calculated as follows:

The difference of resistive indices in spleen and kidney was calculated as follows:

We did not measure RRI and DI-RISK in patients with atrial fibrillation, with bilateral or solitary kidney renal artery stenosis, or with bilateral hydronephrosis of grade II or higher. Furthermore, DI-RISK was not measured in patients with splenectomy, with ultrasound signs of portal hypertension (splenomegaly and/ or dilatation of the portal vein), or with hemodynamically relevant stenosis of the celiac trunk (peak systolic velocity ≥2.0 m/s).

In the Hannover external validation cohort, baseline proteinuria had been assessed as 24-hour urinary protein excretion, which was converted to ACR using the same methodology as the KFRE developers (8). Ultrasound studies were performed with an Ultramark 9 High Definition Imaging ultrasound machine (Advanced Technology Laboratories, Bothell, WA), using either a C 2–4 MHz curved array or a P 2–3 sector multifrequency transductor with a 2.5 MHz pulsed Doppler frequency. RRI was measured in segmental arteries from the upper, middle, and lower pole of each kidney, and mean RRI was calculated as the average of four to six measurements, using the same equation as in the development cohort.

Statistical Methods

Data are presented as mean±SD (for parameters with skewed distribution, median [interquartile ranges]), or as numbers (percentage), as appropriate, and compared with the t test for independent samples, or with the Fisher exact test. For regression analysis, patients were censored at death.

In CARE FOR HOMe, the only missing data were ultrasound measurements, and these were not at random. Therefore, we did not impute missing data. In the Hannover data set the only missing data were four (2.5%) ACR. We created five data sets using multiple imputations based on chained equations to these predict missing values based on the values of all other predictors. We used the mean of the imputed values in the final analysis. All metrics are presented with 95% confidence interval (95% CI) unless otherwise indicated.

Validation of KFRE.

Using all available data, we assessed the performance of KFRE on the CARE FOR HOMe cohort by measures of discrimination, namely the c-statistic and R² (Cox & Snell) (22), and by assessment of calibration by graphical means. The c-statistic is a measure of discrimination, where 1 indicates perfect discrimination between those who do and do not have the event, and 0.5 indicates the test is no better than a coin toss. R² is a measure of the proportion of variation explained and is given as a raw number and a proportion of the maximum R² for the cohort being examined.

Calibration necessarily requires censoring the data at a specific period, we chose 3 years, and we assessed calibration by plotting predicted versus actual risk for the risk of ESRD 3 years following recruitment.

Development of KFRE + Ultrasound Measurement (New Models).

We used Cox regression models to combine the KFRE index with the RRI and DI-RISK (separately) to create two new models. We assessed the incremental prognostic values of RRI and DI-RISK when added to the KFRE index by the log likelihood ratio test. We determined a priori to internally and externally validate any new model where the log likelihood ratio test statistic was <0.05. As for calibration, we predicted events within 3 years and quantified the added value of the ultrasound measurement by the difference in c-statistic and the integrated discrimination improvement metrics for patients without subsequent ESRD (IDI _{No ESRD}) and for patients with subsequent ESRD (IDI _ESRD) (23). We visualized the difference in 3-year model performance with a risk assessment plot and assessed model performance by the integrated sensitivity (IS), integrated 1-specificity (IP) along with the c-statistic (24,25).

Internal Validation of the New Models.

We internally validated the new models with bootstrapping (500 samples with replacement) to estimate the c-statistic adjusted for optimism.

External Validation of the New Models.

We assessed the predictive performance of the new models on the Hannover cohort using the same measures as described.

Results

Of the 444 patients within the CARE FOR HOMe data set, 9.2% (n=41) had no RRI measurements, and 16.7% (n=74) had no DI-RISK measurements, mainly because they fulfilled exclusion criteria for ultrasound studies. Validation of KFRE and development of the ultrasound model with RRI was performed on the 403 (KFRE+RRI) patients with RRI measures and development of the ultrasound model with DI-RISK on 370 (KFRE+DI-RISK) patients. There were 162 patients in the Hannover validation cohort.

The CARE FOR HOMe and Hannover cohorts differed in several key baseline aspects, namely age, body weight, eGFR, BMI, BP, and RRI (Table 1).

Table 1.

Comparison of baseline characteristics between the development cohort and the validation cohort

Variable	All Patients	Development Cohort (CARE FOR HOMe; n=403)	Validation Cohort (n=162)	P Value
Age (years)	60.3±15.3	64.6±12.6	49.8±16.4	<0.001
Body weight (kg)	83.3±17.6	86.4±17.5	75.7±15.3	<0.001
Body height (cm)	169.4±9.4	168.9±9.5	170.6±9.3	0.06
Gender (female)	235 (41.6%)	168 (41.7%)	67 (41.4%)	>0.99
Albuminuria (mg/g creatinine)	44 (15; 204)	32 (7; 194)	57 (32; 318)	0.61
eGFR (ml/min/1.73 m2)	55.7±32.7	45.8±16.0	80.2±47.6	<0.001
BMI (kg/m2)	29.0±5.6	30.2±5.5	26.0±4.5	<0.001
Systolic BP (mmHg)	157±25	154±23	164±26	<0.001
Diastolic BP (mmHg)	90±14	87±12	97±15	<0.001
Creatinine (mg/dl)	1.6±0.9	1.6±0.6	1.7±1.4	0.42
RRI	72±9	74±9	68±10	<0.001

Depicted are mean±SD, or counts (percentages), as appropriate. Because of skewed distribution, albuminuria is given as median (interquartile range). Conversion factors for units: serum creatinine in mg/dl to mol/L, ×88.4. Information on CARE FOR HOMe is based upon those 403 patients who had baseline renal Duplex ultrasound examinations. BMI, body mass index; RRI, renal resistive index.

Patients in CARE FOR HOMe RRI cohort were followed for a mean of 4.4±1.6 years. Among those participants who had baseline RRI measurements, 52 patients experienced ESRD. As expected, patients who progressed toward ESRD were more likely to have prevalent diabetes mellitus, to be male, to have more advanced CKD at baseline, and higher systolic BP. Moreover, they had higher RRI than patients who did not progress toward ESRD (Supplemental Table 1). There were 371 patients with follow-up of >3 years, 29 of whom developed ESRD within 3 years.

There were 162 patients in the validation cohort, 23 of whom experienced ESRD during a follow-up period of 2.8±1.4 years. Again, patients who progressed toward ESRD had more severe CKD, higher BP, and higher RRI at baseline (Supplemental Table 2).

Validation of KFRE

KFRE was predictive of ESRD in the CARE FOR HOMe cohort with a c-statistic of 0.91 (0.83–1.00) (Table 2). For predicting 3-year events it was well calibrated (Figure 1, A).

Table 2.

Development of the new models

Statistic	Baseline Model (KFRE)	New Model 1 (KFRE+RRI)	New Model 2 (KFRE+DI-RISK)	Validation cohort (KFRE+RRI)
Number of patients	403	403	370	162
Number with ESRD	52	52	49	23
Beta coefficients
KFRE indexa	0.827 (0.659–0.995)	0.861 (0.676–1.05)	0.832 (0.654–1.01)	0.811 (0.446–1.18)
RRI per 5 units		0.332 (0.149–0.515)		0.198 (–0.113–0.510)
DI-RISK per 5 units			0.223 (–0.027–0.473)
Hazard ratios
KFRE indexa	2.29 (1.93–2.71)	2.37 (1.97–2.84)	2.30 (1.92–2.74)	2.25 (1.56–3.24)
RRI per 5 units		1.39 (1.16–1.67)		1.22 (0.89–1.67)
DI-RISK per 5 units			1.25 (0.97–1.60)
R2	0.29 (37.7%)	0.31 (40.7%)	0.29 (38.4%)	0.43 (57.3%)
c-statistic	0.91 (0.83–0.99)	0.91 (0.83–1.00)	0.92 (0.83–1.00)	0.95 (0.83–1.00)
Assessment censored at 3 years
Number with ≥3 years follow up	371	371
Number with ESRD within 3 years	29	29
c-statistic (AUC)	0.91 (0.87–0.95)	0.92 (0.89–0.96)
Integrated Sensitivity	0.31 (0.24–0.38)	0.36 (0.27–0.45)
Integrated 1-Specificity	0.06 (0.05–0.07)	0.06 (0.04–0.07)
IDI No ESRD		0.05 (0.00–0.10)
IDI ESRD		0.00 (0.00–0.01)

KFRE, Kidney Failure Risk Equation; RRI, renal resistive index; DI-RISK, difference of resistive indices in spleen and kidney; AUC, area under the curve; IDI _{No ESRD}, integrated discrimination improvement metrics for patients without subsequent ESRD; IDI _ESRD, integrated discrimination improvement metrics for patients with subsequent ESRD. c-statistic is Harrel’s c-statistic. R² is presented as a proportion and percentage of the maximum possible R².

^aKFRE index from Equation 1.

Figure 1.

Calibration curves for ESRD at 3-years. (A) The Kidney Failure Risk Equation (KFRE) and (B) The Kidney Failure Risk Equation plus renal resistive index (KFRE+RRI). Dotted lines represent the calibration curve. Triangles represent the quintile mean values.

Model Development

The likelihood ratio test indicated that addition of RRI to KFRE improved the fit of the data to the observed outcomes (P<0.001). The hazard ratio for the KFRE variable did not substantially change in the new model. The hazard ratio for RRI per five units was 1.39 (1.16–1.67). More of the model’s variance is explained by the addition of RRI, reflected by a greater R². However, the addition of RRI did not appear to improve discrimination: the c-statistic of the KFRE+RRI was 0.91 (0.83–1.00), similar to that of the KFRE model.

The addition of DI-RISK did not improve the performance of the KFRE model (P=0.09). There were no differences in the c-statistics between the new model and the KFRE model (Table 2). The confidence intervals for the hazard ratio DI-RISK straddled 1 (Table 2). For these reasons, no further investigation of KFRE+DI-RISK was performed and all further reference to the new model is to KFRE+RRI.

New Model Performance at 3 Years

The performance of the new model (KFRE+RRI) was assessed at 3 years. The c-statistic was good (0.92 [0.89–0.96]); (Table 2). Figure 2 illustrates that the models had much better overall specificity (IP for the new model of 0.06; ideally zero) than sensitivity (IS for the new model of 0.36; ideally one). The new model did not improve overall sensitivity (IDI _{No ESRD}=0.05 [95% CI, 0.00 to 0.10]) or overall specificity (IDI _ESRD=0.00 [95% CI, 0.00 to 0.01]). There was no difference in c-statistic between the models (P=0.18). The risk assessment plot showed that the only improvement in risk assessment was for those already at >0.4 probability of ESRD within 3 years, for whom there was a small increase in predicted risk of the event (Figure 2). The equation to calculate individual 3-year risk is available in the Supplemental Material.

Figure 2.

Risk assessment plot illustrating added value of renal resistive index (RRI). Plotted is the calculated risk of ESRD against sensitivity (gray lines) and 1-specificity (black lines). The dotted lines are for the baseline model and the solid solid lines for the new model. Improvement of the new model over the old model is seen by separation between the dotted lines and the solid lines. For sensitivity (gray lines) this means the solid curve nearer the upper right-hand corner than the dashed curve. For 1-specificity (black lines) this means the solid curve nearer the lower left-hand corner than the dashed curve. The integrated sensitivity (IS) for each model is the area under the gray lines, the integrated 1-specificity (IP) is the area under the black lines.

The new model was well calibrated (Figure 1, B).

Internal Validation.

Internal validation of the new model showed minimal adjustment, the optimum adjusted c-statistic was 0.91 and R² was 0.41 (53%).

External Validation on the Hannover Cohort.

We calculated for each patient in the external validation cohort the KFRE index, then validated the new model with the beta coefficients from the development cohort (Table 2) for KFRE and RRI. The c-statistic was 0.95 (0.83–1.00) and R² was 0.43 (57%). The calibration was fair (Figure 3).

Figure 3.

Calibration curve in the Hannover validation cohort for ESRD at 3-years for The Kidney Failure Risk Equation plus renal resistive index (KFRE+RRI). Dotted lines represent the calibration curve. Triangles represent the quintile mean values.

Discussion

It has been repeatedly hypothesized that Duplex ultrasound parameters may allow precise identification of CKD patients at high risk for progression toward ESRD (9–11). Duplex ultrasound studies are considered particularly attractive because they are noninvasive and seemingly inexpensive, which may facilitate their integration into clinical practice.

Among various ultrasound parameters, RRI gained most interest in clinical nephrology. RRI has been suggested to closely mirror organ-specific kidney damage (26), and has predicted adverse renal outcomes in previous small- and medium-sized cohort studies (10,11,13,14), including an earlier publication from the validation cohort of the present study (12).

However, we and others have objected to the idea that RRI is an organ-specific marker: in clinical studies, RRI is more closely associated with cardiovascular risk factors and with markers of systemic atherosclerosis than with functional and structural renal markers (19,27–29). Additionally, experimental data support the assumption that RRI is more strongly associated with systemic vascular changes than with local renal damage (30–32).

This led us to propose that RRI should be adjusted for resistive indices measured in other intra-abdominal parenchymal organs accessible for Duplex ultrasound, such as the spleen (33). We hypothesized that systemic factors should affect renal and splenic resistive indices to a similar degree. If this holds true, the difference of resistive indices in spleen and kidney (DI-RISK) will eliminate extrarenal determinants and thus yield a more organ-specific ultrasound marker. Cross-sectional studies among healthy volunteers and CARE FOR HOMe study participants seemingly supported our approach, because DI-RISK measurements were independently associated with renal function, but not with markers of extrarenal vascular disease (19).

Unfortunately, to date, few attempts have been made to compare Duplex ultrasound parameters with validated prediction models for CKD progression, or to integrate Duplex ultrasound parameters into these models. This is of particular importance, because ultrasound studies are both time-consuming and operator-dependent.

In the present analyses, we confirmed that the KFRE has a good discrimination and calibration for prediction of ESRD among patients with CKD stages G2–G4. To the best of our knowledge, CARE FOR HOMe is the fourth European cohort that validates the KFRE in a mainly white population, following the Dutch multifactorial approach and superior treatment efficacy in renal patients with the aid of nurse practitioners (34) and the Scottish Grampian Laboratory Outcomes Mortality and Morbidity Study-I and Grampian Laboratory Outcomes Mortality and Morbidity Study-II (35). It extends the applicability of the KFRE to patients with less severe CKD, because CARE FOR HOMe recruited CKD stage G2–G4 patients, whereas the KFRE development and validation cohorts both included CKD stage G3–G5 patients.

Next, we identified that – when added to the KFRE model – RRI was independently associated with adverse renal outcome, but DI-RISK was not. However, although RRI improved the fit of the data to the outcome, was well calibrated, and helped explain some of the variability in CKD progression, it did not improve discrimination.

Finally, we confirmed in our validation cohort that the new model, which included RRI, did not outperform the KRFE.

We acknowledge several limitations of our study. First, we deliberately focused our analysis on the implication of adding Duplex ultrasound markers to KFRE model 3, which is, to the best of our knowledge, the most used model in clinical practice (34,35). We did not analyze other KFRE models, which require more sophisticated laboratory parameters, or the Johnson equation, which is seldom used in clinical practice. We acknowledge that, in individual patients, the more sophisticated KFRE models may allow better prediction; however, their overall performance is similar to KFRE model 3 (8). Anyway, the incremental contribution of RRI or DI-RISK to a renal prediction model may be even smaller in those comprehensive KFRE models that include a broader range of laboratory parameters and have better prognostic power than the KFRE model 3.

Second, in our validation cohort, no original data on albuminuria were available, therefore we converted proteinuria to ACR following an algorithm, as suggested by Tangri and colleagues (8). Third, both cohorts recruited mainly white participants, and we cannot provide data for other ethnic groups. Fourth, although CARE FOR HOMe recruited substantially more study participants than earlier cohort studies that analyzed Duplex ultrasound, it is still a small cohort and the confidence intervals of the discrimination statistics for the addition of RRI to KFRE are broad. This means that we cannot rule out no benefit of RRI, merely not a substantial one. Confirmation of our results in even larger cohort studies would be desirable. Fifth, the Hannover cohort is small and so only substantial differences in c-statistic would be noted. The younger ages, less-advanced CKD, and different site of RRI measurements in this cohort were factors that could, but did not, contribute to substantial differences in c-statistic. However, the different demographics and site of RRI may explain the poor calibration. Finally, our study focuses on renal end-points, and future cohort studies may assess the implication of Duplex ultrasound parameters on other outcomes, such as cardiovascular event rate and mortality.

In summary, this study failed to support the hypothesis that Duplex ultrasound markers improve risk prediction of ESRD among patients with prevalent CKD over and above that of the KFRE equation. Thus, we advocate against routine measurement of Duplex ultrasound markers when aiming to predict the risk of CKD progression.

Disclosures

None.