Validation of the Kidney Failure Risk Equation in Manitoba

Abstract

Background:

Patients with chronic kidney disease (CKD) are at risk to progress to kidney failure. We previously developed the Kidney Failure Risk Equation (KFRE) to predict progression to kidney failure in patients referred to nephrologists.

Objective:

The objective of this study was to determine the ability of the KFRE to discriminate which patients will progress to kidney failure in an unreferred population.

Design:

A retrospective cohort study was conducted using administrative databases.

Setting:

This study took place in Manitoba, Canada.

Measurements:

Age, sex, estimated glomerular filtration rate (eGFR), and urine albumin-to-creatinine ratio (ACR) were measured.

Methods:

We included patients from the Diagnostic Services of Manitoba database with an eGFR <60 mL/min/1.73 m² and ACR measured between October 2006 and March 2007. Five-year kidney failure risk was predicted using the 4-variable KFRE and compared with treated kidney failure events from the Manitoba Renal Program database. Sensitivity and specificity for KFRE risk thresholds (3% and 10% over 5 years) were compared with eGFR thresholds (30 and 45 mL/min/1.73 m²).

Results:

Of 1512 included patients, 151 developed kidney failure over the 5-year follow-up period. The 4-variable KFRE showed a superior prognostic discrimination compared with eGFR alone (area under the receiver operating characteristic curve [AUROC] values, 0.90 [95% confidence interval {CI}: 0.88-0.92] for KFRE vs 0.78 [95% CI: 0.74-0.83] for eGFR). At a 3% threshold over 5 years, the KFRE had a sensitivity of 97% and a specificity of 62%. At 10% risk, sensitivity was 86%, and specificity was 80%.

Limitations:

Only 11.7% of stage 3-5 CKD patients had simultaneous ACR measurement. The KFRE does not account for other indications for referral such as suspected glomerulonephritis, polycystic kidney disease, and recurrent stone disease.

Conclusions:

The KFRE has been validated in a population with a demographic and referral profile heretofore untested and performs well at predicting 5-year risk of kidney failure in a population-based sample of Manitobans with CKD stages 3 to 5. Thresholds of 3% and 10% over 5 years are sensitive, specific, and can be used in clinical decision making. Further testing of the 4-variable KFRE and these thresholds in clinical practice should be considered.

What was known before

In 2011, the Kidney Failure Risk Equation (KFRE) was developed and has subsequently been shown to accurately predict the progression of chronic kidney disease (CKD) to kidney failure.

What this adds

The KFRE has been validated in an unreferred population in Manitoba which includes a significant number of Indigenous Canadians.

Background

Chronic kidney disease (CKD) is a common, harmful, and treatable disease and recognized as a major public health problem.¹ The prevalence of CKD is increasing worldwide, and it affects an estimated 8% to 16% of the global population.² The proportion of Canadians with CKD ranges from 10% to 15%, representing approximately 3 million affected adults.³ In these patients, CKD leads to a higher risk of cardiovascular disease and progression to kidney failure requiring dialysis.⁴

Predicting progression to kidney failure from earlier stages of CKD is crucial for patients, health care providers, and policy makers for several reasons. First, the progression of CKD is heterogeneous, with many patients suffering from early mortality before disease progression, while others rapidly progress from early stage CKD to kidney failure in less than a few years. The former often require cardiovascular risk reduction, and more conservative kidney health care, whereas the latter require expedited preparation for dialysis and transplant in addition to management of cardiovascular risk factors.

Furthermore, given the high prevalence of CKD, estimation of progression is necessary to triage patients who would most benefit from subspecialty nephrology team care and those who can be safely managed by their primary care practitioner at a significantly lower cost. Early referral to nephrology care for higher risk patients has been shown to improve survival and reduce health care costs, through increased fistula and home dialysis use.^5,6 In addition, treatments such as angiotensin-converting enzyme inhibitors, angiotensin-receptor blockers, allopurinol, and sodium bicarbonate have been shown to slow decline in eGFR and have beneficial effects on time to kidney failure in many patients.^7-14 For these reasons, a policy to guide surveillance of vulnerable populations and a tool to predict progression to kidney failure are essential for clinicians and policy makers for choosing appropriate interventions and are important when conducting field and clinical trials.

In 2011, we developed and validated the Kidney Failure Risk Equation (KFRE), a model to predict the progression of CKD to kidney failure.¹⁵ Our model uses routinely collected laboratory and demographic data, and predicts the progression of CKD to dialysis in patients with CKD stages 3 to 5. The results are presented as a percentage of risk of progression to kidney failure in 2 or 5 years where 3% is considered low risk and 10% is considered intermediate risk over 5 years. As of 2016, the KFRE has been validated in 31 cohorts, including 721/357 participants with CKD stages 3 to 5 in more than 30 countries spanning 4 continents, and has been demonstrated to be accurate within these diverse populations.¹⁶

However, the original development population and subsequent validation cohorts were comprised entirely of patients referred to nephrologists, and certain ethnic minorities with increased risk for kidney failure, particularly Indigenous Canadians, were underrepresented.^15,16 Furthermore, absolute risk thresholds to guide decision making based on the KFRE have not been tested, and their sensitivity/specificity for predicting kidney failure is unknown. Here, we report the validation of the 4-variable KFRE in an unreferred population in Manitoba and demonstrate its potential utility using actionable thresholds as a tool for CKD surveillance and management. The 4 variables are age, sex, estimated glomerular filtration rate (eGFR), and urine albumin-to-creatinine ratio (ACR) and were selected by face validity.

Methods

Population and Data Source

This study took place in Manitoba, Canada (population 1.2 million), where all residents are covered through a universal payer health care system. All data came from the Data Repository housed at the Manitoba Centre for Health Policy at the University of Manitoba which holds population-wide, anonymized health information for Manitoba residents.^17-19 The databases used in this study included Manitoba Renal Program data, Laboratory data from Diagnostic Services of Manitoba (DSM), hospital discharge abstracts, physician visit records, and the population health registry that captures all residents in Manitoba. Individual-level information from these data sources was linked across databases using scrambled identifiers.

There are 2 major laboratory service providers in Manitoba: DSM, which serves >70% of the population, and Gamma Dynacare Inc., which serves 20% to 25% of the population. The DSM is a nonprofit corporation responsible for all of Manitoba’s public laboratory services and for rural diagnostic services. DSM operates 77 clinical laboratory sites across the province, with each site serving as a point of access for comprehensive diagnostic testing. In addition, all specialized testing is done through a DSM laboratory. This includes all urinary albumin testing, critical for the case definition of CKD. The test results are retrievable using a personal health identifier number (PHIN). Additional identifiers in DSM include patient name, gender, and date of birth, all of which facilitate linkage with additional databases if PHIN information is missing or inconsistent.

Patients were eligible to be included in our cohort if they were above the age of 18 and had their serum creatinine and urine ACR measured between October 1, 2006, and March 31, 2007. Our final sample included 1512 patients with CKD stages 3 to 5. We defined individuals as having CKD if their eGFR measurement as calculated by the Chronic Kidney Disease Epidemiology Collaboration equation²⁰ indicated poor kidney function (less than 60 mL/min/1.73 m²; Figure 1). This eGFR cutoff was necessary for this study as the KFRE was only developed for a stage 3 to 5 CKD population. None of the patients at baseline had been referred to a nephrologist. Demographics were obtained through the population health registry. Comorbid conditions were defined using physician visit records and hospital discharge abstracts. Nephrology visits were defined through physician visit records. Our outcome of kidney failure was determined by an incident entry in the Manitoba Renal Program database which tracks all chronic dialysis patients in the province. Death was censored in the analysis.

Figure 1.

Cohort selection.

Note. DSM = Diagnostic Services of Manitoba; ACR = albumin-to-creatinine ratio; eGFR = estimated glomerular filtration rate; RRT = renal replacement therapy.

Validation of the KFRE

Prediction of kidney failure

We calculated the predicted risk of kidney failure for each patient at 5 years using the 4-variable KFRE. The first measurement of eGFR and ACR during the study period was entered in the equation. Following that, we generated an area under the receiver operating characteristic curve (AUROC) for the KFRE modeled as a continuous variable. We also examined the diagnostic test performance of the KFRE as a predictor of kidney failure by evaluating sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). We used risk thresholds of 3% and 10%.

Comparison of KFRE against eGFR criteria

Most physician referral criteria base clinical decisions in CKD on eGFR. For example, guidelines variably recommend referral of patients to a nephrologist if eGFR is lower than 45 or 60 mL/min/1.73 m² and definitely if eGFR is lower than 30 mL/min/1.73 m².^21,22 We therefore generated an AUROC for eGFR alone, modeled as a continuous variable, to evaluate how well eGFR predicts kidney failure, and compared the discrimination of this model to that of the KFRE. We also compared the diagnostic test characteristics (sensitivity, specificity, PPV, and NPV) for eGFR thresholds of 45 mL/min/1.73 m² and 30 mL/min/1.73 m², and compared these with those of KFRE thresholds of 3% and 10%.

Comparison of KFRE with physician decision to initiate nephrology referral

Because preparation for renal replacement therapy is initiated by nephrology, physician decision to refer for specialty care can be considered a surrogate for physician perception of risk, and reflects an integration of multiple objective and subjective factors. Furthermore, it has been shown that primary care physicians differ in their clinical evaluations and expectations for referral for patients with progressive CKD when compared with nephrologists.²³ It is therefore relevant to ask whether this “physician judgment,” as currently applied, is better or worse at predicting kidney failure than the KFRE or eGFR. We therefore compared the sensitivity, specificity, PPV, and NPV of physician judgment, defined as the decision to initiate referral, with KFRE thresholds of 3% and 10%, and eGFR thresholds of 45 mL/min/1.73 m² and 30 mL/min/1.73 m². Referral was measured by a claim for a new outpatient consultation to a nephrologist within the study period.

Utility of KFRE as a Screening/Surveillance Tool

To evaluate the potential utility of the KFREs as a tool for CKD surveillance and early detection of progressors, we performed a retrospective evaluation of all patients who started dialysis from November 2010 to December 2011, and had a measurement of eGFR and urine ACR in the 5 years preceding the onset of kidney failure. The primary objective of this analysis was to see whether the majority of patients who develop kidney failure had a high predicted risk, as identified by the KFRE well in advance of the need for dialysis.

Results

Study Population

The Manitoba DSM cohort included 1196 patients with stage 3 CKD (mean eGFR, 47.9 mL/min/1.73 m²) and 316 patients with stage 4-5 CKD (mean eGFR, 21.3 mL/min/1.73 m²). Patients in the stage 3 and stage 4-5 groups were similar in age and sex and had a similar prevalence of diabetes, ischemic heart disease, and atrial fibrillation. Heart failure was more common in patients with CKD stages 4 to 5, and these patients were also more likely to have albuminuria. Over a 5-year follow-up period, patients with CKD stages 4-5 had more adverse events when compared with patients with CKD stage 3 (Table 1).

Table 1.

Characteristics of Patients With CKD in the Kidney Failure Risk Equation Validation Cohort by CKD Severity (Residents Aged 18+ on March 31, 2007).

	Patients with CKD by severity (October 1, 2006 to March 31, 2007)
	Mild to severe (Stage G3)	Severe or kidney failure (stage G4-G5)
	1196	316
Demographics
Age group, y
18-65	40.90%	42.10%
65+	59.00%	57.90%
Average age (SD), y	67 (13)	66 (14)
Male	50.30%	49.70%
Laboratory parameters
Urine albumin-to-creatinine ratio, mg/g
<30	42.40%	21.20%
30-300	35.10%	30.10%
≥300	22.50%	48.70%
Serum creatinine, mg/dL	1.4	2.8
Glomerular filtration rate, mL/min/1.73 m2	47.9	21.3
Seen by nephrologist in previous 5 y	31.20%	76.60%
Comorbid conditions
Diabetes prevalenceb	76.60%	73.10%
Ischemic heart disease prevalencec	37.90%	38.00%
Congestive heart failure prevalenceb	17.40%	27.90%
Atrial fibrillation prevalenceb	5.90%	5.70%
Outcomes
End-stage kidney disease prevalence	4.80%	29.80%
Mortality rate	26.30%	32.90%

Note. CKD = chronic kidney disease.

^aCKD severity based on heat map categories in chapter 2 of “Kidney Disease: Improving Global Outcomes (KDIGO) CKD Work Group: KDIGO 2012 clinical practice guideline for the evaluation and management of chronic kidney disease (2013)”.

^b2004/2005-2006/2007.

^c2002/2003-2006/2007.

Validation of the KFRE

Discrimination of the KFRE

The KFRE excellently discriminated patients who developed kidney failure within 5 years from patients who did not in the Manitoba DSM cohort (C statistic, 0.900; 95% confidence interval [CI], 0.876-0.923; Table 2).

Table 2.

Discrimination of Screening Thresholds for 5-Year Progression to ESKD.

Screening methods and risk thresholds	Sensitivity	Specificity	Positive predictive value	Negative predictive value
KFRE
>3% risk of progression to ESKD	0.967	0.619	0.220	0.994
>10% risk of progression to ESKD	0.861	0.802	0.325	0.981
eGFR
<30 mL/min/1.73 m2	0.623	0.837	0.297	0.952
<45 mL/min/1.73 m2	0.841	0.544	0.170	0.969
Prior nephrology visit	0.788	0.636	0.193	0.964

Note. Overall C statistic for KFRE: 0.900 (95% CI, 0.876-0.923), overall C statistic for eGFR: 0.784 (95% CI, 0.742-0.826), and overall C statistic for prior nephrology visit: 0.712 (95% CI, 0.677-0.747). ESKD = end-stage kidney disease; KFRE = Kidney Failure Risk Equation; CI = confidence interval; eGFR = estimated glomerular filtration rate.

Comparison with eGFR

eGFR had much more modest discrimination than did KFRE, with a C statistic of 0.78 (95% CI, 0.742-0.826; Table 2).

Comparison of risk thresholds

Using the threshold of 3% risk over 5 years, the KFRE was 97% sensitive in predicting kidney failure events. Out of 151 patients who progressed to kidney failure within 5 years, 146 had a risk greater than 3% at baseline. Specificity at this low risk threshold was 62%, and NPV was 99%. In contrast, an eGFR cutoff of 45 mL/min/1.73 m² was less sensitive and specific (84% vs 97%, and 54% vs 62%) and showed lower PPV and NPV. When the 10% risk threshold was applied, as expected, the KFRE had a lower sensitivity of 86% but a higher specificity at 80%. Again, a more specific eGFR threshold of 30 mL/min/1.73 m² showed much lower sensitivity (62% vs 86%), and lower PPV and NPV (Table 2).

Comparison with physician judgment

The KFRE outperformed physician judgment defined as having been referred to and visited a nephrologist within 5 years after April 1, 2007. Sensitivity for the 3% and 10% cutoffs was considerably better than referral to nephrology as a marker of high-risk disease (97% and 86% vs 79%). Specificity was similar for referral to nephrology at the 3% threshold but much better at the 10% threshold (62% and 80% vs 64%). The KFRE showed higher PPV and NPV than referral to nephrology at both thresholds (Table 2).

Utility of KFRE as a Screening/Surveillance Tool

Patients who begin dialysis in Manitoba in 2010-2011 and had the required data (eGFR and urine ACR) available in the preceding 5 years are described in Table 3. Their mean eGFR was 28 mL/min/1.73 m² at baseline, and they had a similar prevalence of diabetes compared with our population-based sample with stage 3-5 CKD. These progressors were much more likely to have albuminuria but had a lower rate of comorbid cardiovascular conditions such as ischemic heart disease, heart failure, and atrial fibrillation. The median predicted risk of kidney failure in this group was 57% over 5 years, and more than 94% of patients who progressed had a predicted risk that exceeded our proposed 3% threshold.

Table 3.

Characteristics and Predicted Risk Thresholds of Patients With End-Stage Kidney Failure (Residents Aged 18+ on March 31, 2009).

	Patients with ESKD (2010/2011-2011/2012)
Number of patients	166
Demographics on March 31, 2009
Age group, y
18-65	69.90%
65+	30.10%
Average age (SD), y	58 (14)
Male	54.80%
Kidney health in previous 3 y
Urine albumin-to-creatinine ratio, mg/g
<30	4.80%
30-300	15.10%
≥300	80.10%
Serum creatinine, mg/dL	2.5
Glomerular filtration rate, mL/min/1.73 m2	28
Physical health within previous 5 y
Diabetes prevalencea	78.30%
Ischemic heart disease prevalenceb	32.50%
Congestive heart failure prevalencea	16.30%
Atrial fibrillation prevalencea	3.00%
Kidney Failure Risk Equation prediction
Risk of progression to ESKD, %
<3	5.40%
3-10	10.20%
≥10	84.30%
Median risk of progression to ESKD	0.565

Note. ESKD = end-stage kidney disease.

^a2006/2007-2008/2009.

^b2004/2005-2008/2009.

Discussion

In this validation study of the KFRE in a population-based cohort with different demographic characteristics than previous validations (ie, a higher percentage of Indigenous Canadians), we have demonstrated that the KFRE performs excellently for discriminating which patients will progress to kidney failure over the next 5 years when compared with eGFR and current nephrology referral practices (ie, physician judgment). A comparison of the CKD cohort with the cohort already on dialysis illustrates that the competitive risk of death increases with age as the dialysis cohort is on average 9 years younger. Moreover, the data imply that early mortality before kidney failure among the CKD population may likely be due to cardiovascular causes given the lower incidence of congestive heart failure and ischemic heart disease in the dialysis cohort.

The KFRE uses laboratory data that are obtained routinely in patients with CKD and could be easily integrated into laboratory information system prompts or a clinic electronic medical record. In addition, the KFRE can be integrated into a passive surveillance system for detecting high-risk CKD in health systems. In particular, a risk-based cutoff of 3% may be an appropriate criterion for determining referral to nephrology, as it is highly sensitive, modestly specific, and shows superior discrimination than eGFR alone.

Previous independent investigators have validated the KFRE in other CKD populations. In the Netherlands, researchers from the MASTERPLAN study,²⁴ a randomized control trial which examined the role of nurse practitioners in reducing cardiovascular risk in CKD patients, followed a cohort of patients for 5 years and evaluated the ability of the KFRE to discriminate which of these patients would progress to kidney failure.²⁵ The study concluded that the KFRE excellently discriminated the progression to kidney failure in European CKD patients with a C statistic of 0.89.²⁴ Similarly, in New Zealand, the KFRE was validated in a study which included 25 736 individuals with type 2 diabetes.²⁶ These investigators also found that the KFRE could accurately predict the risk of kidney failure in patients with type 2 diabetes. The C statistic ranged from 0.81 to 0.88 in both cohorts.²⁷ Our findings are consistent with these previous reports but extend the validity of the equation to Manitoba’s unique population in a community-based cohort where, unlike previous KFRE validation studies, the majority of the patients were not followed by nephrologists.

Our examination of risk-based cutoffs for screening and surveillance is a novel finding in this article. Multiple jurisdictions across North America have attempted to determine criteria for nephrology referral for patients with earlier stages of CKD. Recently, in 2013, the 2009 Kidney Disease: Improving Global Outcomes (KDIGO) clinical practice guidelines have suggested an eGFR cutoff of 30 mL/min/1.73 m² for referrals to specialists to avoid the enormous volume of unnecessary consults and interventions.²² Although several cutoffs have been suggested, to our knowledge, none have been examined against the important outcome of kidney failure in prospective/retrospective analyses. Our analysis tests both common eGFR-based cutoffs (30 mL/min/1.73 m² and 45 mL/min/1.73 m²) and physician judgment (new nephrology referral) against KFRE-based cutoffs, and demonstrates the superiority of risk-based decision making. These findings are consistent in the “prospective” analysis of patients with CKD, and in the “retrospective” analysis of new dialysis starts, further strengthening our conclusions.

There are important policy implications of our findings. Manitoba has recently created a provincial laboratory information system integrated with clinical decision support, and several other jurisdictions have similar systems already in progress or in place. Our findings suggest that integration of the KFRE in these systems is feasible, likely to result in accurate risk estimates, and may improve clinical decision making. Currently, eGFR is reported routinely in these settings, and patients with a low eGFR (<60 mL/min/1.73 m²) are identified as having CKD. Referral to a nephrologist of all patients with an eGFR <60 mL/min/1.73 m² or 45 mL/min/1.73 m² is impractical, unfeasible, and unlikely to be of benefit in the majority who are at low risk of progression. In contrast, integrating the KFRE into referral systems would align treatment intensity with risk rather than kidney function. A 3% threshold is highly sensitive and more specific than eGFR alone; therefore, prospective studies are needed to verify whether its implementation can result in better resource allocation of nephrology team CKD care. The 3% threshold also has a very high NPV (>99%), providing reassurance to many CKD patients. Given its relative rareness, all CKD patients still have low pretest probability of progressing to kidney failure. As a result, the 22% PPV at the 3% threshold is low, but not unexpectedly so. Furthermore, the PPV is higher than many other tools used for triage to specialists such as early cancer screening.^28-30 There are also important implications for further research. Whether higher KFRE thresholds can be used to decide which patients really need higher intensity and more costly interdisciplinary nephrology care, which patients need education regarding dialysis modality selection, and when patients need dialysis access creation or preemptive transplantation, are all important questions that should be addressed in future analyses.

The major strengths of our analysis are (1) direct verification of the validity of the KFRE in a population-based Manitoba cohort which exhibits marked demographic differences compared with the original derivation and validation cohorts and (2) demonstration of the feasibility and potential clinical utility of integrating a prediction tool with a track record of excellent discrimination and evidence-based thresholds for kidney failure into surveillance and decision support systems.

There are also some limitations to our findings. First, we were unable to determine the exact percentage of Indigenous people in our cohort. However, we are confident our study features the highest percentage of Indigenous Canadians compared with any other KFRE validation cohort for the following reasons: (1) Manitoba’s population is ~10% Indigenous, (2) the prevalence of CKD and end-stage kidney disease is higher in predominately Indigenous communities in Manitoba, and (3) recently published research where race was reported among Manitoba dialysis cohorts ranged from 22% to 24% Indigenous on peritoneal dialysis to 32% of those on dialysis overall.^31-37 Second, only 11.7% of stage 3-5 CKD patients had simultaneous ACR measurement. Therefore, 88.2% of patients with eGFR between 10 and 60 mL/min/1.73 m² were excluded from our sample. As ACR is more likely to be measured in patients with diabetes and those with known nephropathy, our sample of patients with eGFR between 10 and 60 mL/min/1.73 m² is biased as those with high ACR were more likely to be included. However, we have previously shown that the KFRE performs well in patients with or without diabetes.^27,38 We expect that the lack of available ACR measurements is partly due to the short 6-month window and has also improved over time as ACR testing becomes more routine, particularly with the recent CKD guidelines. Furthermore, the implementation of a lab prompts system for the KFRE may also improve proper CKD screening at the primary care level by requiring both eGFR and urinary ACR to deliver a patient’s risk. Finally, although we highlighted the potential utility of KFRE-based thresholds to guide nephrology referral, other indications for referral, such as acute kidney injury, suspected glomerulonephritis, polycystic kidney disease, and recurrent stone disease, may exist in low-risk patients. Some of the referrals captured in our study may exist for these reasons, and therefore, it may not be reasonable to judge all referrals as a surrogate for perception of risk for kidney failure. As such, the KFRE thresholds should be considered and used as a decision aid in the referral process, and not considered as the sole criteria for determining need for nephrology care.

Conclusions

The KFRE performs well at predicting the 5-year risk of dialysis in a population-based sample of Manitobans with CKD stages 3 to 5. A 3% risk threshold over 5 years based on the KFRE is highly sensitive, modestly specific, and can be considered a useful lab-based criterion for determining nephrology referral. A 3% threshold is also potentially useful for providing patients with prognostic information in the kidney failure risk and has potential as a possible strategy to better align resources with risk. Future testing of the 3% threshold in clinical decision support is needed before integration of the KFRE into laboratory information systems can be recommended.