Effects of Serum Creatinine Calibration on Estimated Renal Function in African Americans: the Jackson Heart Study

Abstract

Background

The calibration to Isotope Dilution Mass Spectroscopy (IDMS) traceable creatinine is essential for valid use of the new Chronic Kidney Disease Epidemiology Collaboration (CKD-EPI) equation to estimate the glomerular filtration rate (GFR).

Methods

For 5,210 participants in the Jackson Heart Study (JHS), serum creatinine was measured with a multipoint enzymatic spectrophotometric assay at the baseline visit (2000–2004) and re-measured using the Roche enzymatic method, traceable to IDMS in a subset of 206 subjects. The 200 eligible samples (6 were excluded, 1 for failure of the re-measurement and 5 for outliers) were divided into three disjoint sets - training, validation, and test - to select a calibration model, estimate true errors, and assess performance of the final calibration equation. The calibration equation was applied to serum creatinine measurements of 5,210 participants to estimate GFR and the prevalence of CKD.

Results

The selected Deming regression model provided a slope of 0.968 (95% Confidence Interval (CI), 0.904 to 1.053) and intercept of −0.0248 (95% CI, −0.0862 to 0.0366) with R squared 0.9527. Calibrated serum creatinine showed high agreement with actual measurements when applying to the unused test set (concordance correlation coefficient 0.934, 95% CI, 0.894 to 0.960). The baseline prevalence of CKD in the JHS (2000–2004) was 6.30% using calibrated values, compared with 8.29% using non-calibrated serum creatinine with the CKD-EPI equation (P < 0.001).

Conclusions

A Deming regression model was chosen to optimally calibrate baseline serum creatinine measurements in the JHS and the calibrated values provide a lower CKD prevalence estimate.

Introduction

The best indicator of kidney function is considered to be the flow rate of filtered fluid through the kidney, or glomerular filtration rate (GFR). GFR is difficult to measure directly, and therefore is usually estimated from serum concentrations of endogenous markers. Clinical guidelines recommend reporting estimated GFR when serum creatinine is measured.¹

The equations commonly used for estimating GFR are the Modification of Diet in Renal Disease (MDRD) study equation and, more recently, the Chronic Kidney Disease Epidemiology Collaboration (CKD-EPI) equation.^2,3 Both equations apply different coefficients to the same four variables – age, sex, ethnicity and serum creatinine concentrations. The original MDRD study equation was developed using serum creatinine measured at the Cleveland Clinic Laboratory and has also been re-expressed for creatinine measured using methods traceable to Isotope Dilution Mass Spectroscopy (IDMS).⁴ The new CKD-EPI equation estimates the measured GFR more accurately than the MDRD study equation, especially in patients with a relatively well-preserved kidney function.^3,5,6 In addition, the CKD-EPI equation may provide improved cardiovascular risk prediction over the MDRD in middle-age populations,⁷ classify fewer individuals as having CKD, and more accurately categorize the risk for mortality and End Stage Renal Disease (ESRD) than the MDRD Study equation across a broad range of populations.⁸ The CKD-EPI equation is designed to be used with IDMS-calibrated serum creatinine values only.³ Recently, the same group has published two additional CKD-EPI equations: one based on cystatin C concentration and one using both cystatin C and serum creatinine concentrations.⁹

The primary objectives of this study were to calibrate serum creatinine measurements determined at the baseline examination of the Jackson Heart Study (JHS) (2000–2004) to creatinine concentrations traceable to IDMS, estimate GFR with the calibrated values using the CKD-EPI equation and the MDRD equation, and also determine CKD prevalence.

Methods

JACKSON HEART STUDY

Data for this study were collected as part of the JHS, the largest single-site, prospective epidemiologic investigation of cardiovascular disease among African Americans. The methods and overview of the JHS design have been described elsewhere.^10,11 The study was approved by Institutional Review Board of University of Mississippi Medical Center (IRB number: 1998–6004), and all patients provided written informed consent prior to participating in the study. Briefly, a total of 5,301 African Americans residing in the Jackson, Mississippi metropolitan area participated in the study. Of the 5,301 participants who underwent the baseline examination (2000–2004), 91 were excluded due to lack of serum creatinine measurements, leaving 5,210 samples for GFR estimation. The present study used a simple random sampling technique to select 206 participants with stored serum specimens from the JHS baseline examination (2000–2004) for creatinine calibration.

LOCAL LABORATORY SETTINGS

For the full JHS population (N = 5,301), biochemical testing for serum creatinine was performed at the University of Mississippi Medical Center Laboratory Reading Center by using a multipoint enzymatic spectrophotometric assay (Vitros CREA dry reaction slides on a Vitros 950 Ortho-Clinical Diagnostics analyzer, Raritan, NJ).¹²

SERUM CREATININE CALIBRATION ASSAY

For the subset of 206 participants, serum creatinine was re-measured in 2006 using the enzymatic method on a Roche Modular P Chemistry Analyzer (Roche Diagnostics Corporation, Indianapolis, IN 46250) at the University of Minnesota. In this enzymatic method, creatinine is converted to creatine by creatinase, which is then acted upon by creatinase to form sarcosine and urea. Sarcosine is then converted to hydrogen peroxide, which reacts with a chromophore to produce a colored product that is measured colorimetrically. This method has an advantage over the Jaffe method in that it is not susceptible to interferences from non-creatinine chromogens. The method is calibrated using a National Institute of Standards and Technology (NIST) standard, traceable to reference material SRM 909b (IDMS). The Roche assay showed coefficient of variation of 2.3%.

GFR ESTIMATION

We estimated GFR using the CKD-EPI equation as follows:³

$$\mathit{Estimated}\mathit{GFR}=141\times min{(\mathit{Scr}/\kappa ,1)}^{\alpha }\times max{(\mathit{Scr}/\kappa ,1)}^{-1,209}\times {0.993}^{\mathit{Age}}\times (1.018if\mathit{female})\times (1.159if\mathit{black})$$

Where Scr is IDMS-traceable serum creatinine (mg/dL), κ is 0.7 for females and 0.9 for males, α is −0.329 for females and −0.411 for males, min indicates the minimum of Scr /κ or 1, and max indicates the maximum of Scr /κ or 1.

In addition, GFR was also estimated by using the MDRD study equation with standardized creatinine values:^4,13

$$\mathit{Estimated}\mathit{GFR}=175\times {\mathit{Scr}}^{-1.154}\times {\mathit{age}}^{-0.203}\times (1.212if\mathit{black})\times (0.742if\mathit{female})$$

Where Scr is IDMS-traceable serum creatinine (mg/dL).

STATISTICAL ANALYSIS

For calibration samples, we calculated summary statistics for the original, IDMS traceable serum creatinine measurements and also the difference (IDMS - original). Five extreme outliers (difference > 3 SDs from the mean) were excluded under the premise that these outliers would not contribute useful information to the calibration, because they are believed to be caused by sample evaporation, insufficient sample mixing or other handling issues, rather than differences between serum creatinine assays.^14,15 A three-way data split strategy was used in which data were divided into three mutually exclusive datasets: a training dataset, a validation dataset and a test dataset, to select an appropriate calibration equation, estimate true error, and also assess performance of the final selected calibration equation.¹⁶ Briefly, the dataset was divided into 25% for test data and 75% for model selection and true error estimation using the holdout method, 10-fold cross validation, and leave-one-out cross validation methods.¹⁷ Four potential calibration models were considered: simple linear regression, quadratic regression, piecewise linear regression, and Deming regression, as shown below:

Simple linear regression model,

$$y_i={\beta }_0+{\beta }_1{x}_i+{\epsilon }_i,\mathit{where}{\epsilon }_i~N(0,{\sigma }^2)$$

Polynomial regression (quadratic regression),

$$y_i={\beta }_0+{\beta }_1{x}_i+{\beta }_2{x_i}^2+{\epsilon }_i,\mathit{where}{\epsilon }_i~N(0,{\sigma }^2)$$

Piecewise linear regression,

$$y_i={\beta }_0+{\beta }_1{x}_{i1}+{\beta }_2(x_{i1}-x_0)x_{i2}+{\epsilon }_i,\mathit{where}{\epsilon }_i~N(0,{\sigma }^2)$$

• x_i1 is the independent variable for subject i;

• x_i2 is the dummy variable (0, if x_i1 ≤ x₀, and 1 if x_i1 > x₀) for subject i;

• x₀ is the change point.

Deming regression,

$$\begin{array}{c}{y}_i={y_i}^{\ast }+{\epsilon }_i,\mathit{where}{\epsilon }_i~N(0,{{\sigma }_1}^2)\\ x_i={x_i}^{\ast }+{\eta }_i,\mathit{where}{\eta }_i~N(0,{{\sigma }_2}^2)\\ {y_i}^{\ast }={\beta }_0+{\beta }_1{x_i}^{\ast }\end{array}$$

Note that we are reusing parameter symbols for notational convenience (β_i and ε_i), and parameters in different models should be considered as distinct. The selected Deming regression calibration equation was applied to the test dataset, and the measured and calibrated serum creatinine values were compared with the paired t-test. Agreement was assessed with concordance correlation coefficient statistics. The results were further depicted using scatterplots and Bland-Altman plots.

The IDMS traceable creatinine measurements were calculated with the final selected Deming regression calibration equation for 5,210 subjects from JHS (2000–2004). CKD prevalence, defined as eGFR < 60 ml/min/1.73 m^2,1 based on eGFR calculated by CKD-EPI and MDRD Study equation using calibrated and non-calibrated serum creatinine concentrations was compared with McNemar’s test.

All P values were 2-tailed with statistical significance set at 0.05 and all statistical analyses were performed using SAS version 9.3 (SAS Institute Inc, Cary, NC).

Results

CREATININE CALIBRATION

Of the 205 participants with successful re-measurement of baseline serum creatinine (one sample measurement failed due to evaporation/handling loss), age range was 21.4 to 77.6 years, and 57.3% were women. Mean creatinine concentrations using the spectrophotometric method and Roche enzymatic method were 0.979 mg/dL and 0.920 mg/dL respectively (Table 1), and mean difference was −0.059 ± 0.060 mg/dL (−5.2 ± 5.3 μmol/L) after excluding five outliers.

Table 1.

Comparison of Roche Enzymatic Assay and Local Laboratory Serum Creatinine Measurements

	Complete Data (N = 205)		Excluding Outliers* (N = 200)
	Mean ± SD	Range	Mean ± SD	Range
Serum Creatinine Measured at Local Lab	1.200 ± 1.446	(0.50, 11.40)	0.979 ± 0.266	(0.50, 2.40)
Serum Creatinine by Roche Enzymatic Assay (mg/dL)	1.124 ± 1.336	(0.48, 10.60)	0.920 ± 0.255	(0.48, 2.36)
Difference in Serum Creatinine (Roche – Local Lab) (mg/dL)	−0.076 ± 0.129	(−1.09, 0.11)	−0.059 ± 0.060	(−0.28, 0.11)
P†	< 0.001		< 0.001
Percent Difference in Serum Creatinine (Roche – Local Lab)/Roche × 100, %	−6.80 ± 7.18	(−35.59, 18.03)	−6.76 ± 7.25	(−35.59, 18.03)
P‡	< 0.001		< 0.001

Note: To convert serum creatinine in mg/dL to μmol/L, multiply by 88.4.

^*The outliers were defined as, subjects with absolute difference of (Roche - Lab) > 3 SDs from the mean difference.^14,15

^†: Paired t-test of the hypothesis that the mean difference is 0.

^‡: Paired t-test of the hypothesis that the mean percent difference is 0.

For calibration model selection, quadratic regression and piecewise linear regression models generated larger mean squared errors than those from simple linear regression and Deming regression with 10-fold cross validation or leave-one-out cross validation (Table 2). We therefore focused on simple linear and Deming regressions for further consideration. Results before and after exclusion of the 5 outliers showed similar regression coefficients for linear and Deming models (Table 3). Since the latter can account for measurement errors in two creatinine measurement methods, we chose Deming regression as our final calibration model. The Deming regression showed a slope of 0.968 (95% CI, 0.904 to 1.053; P < 0.001) and intercept of −0.0248 (95% CI, −0.0862 to 0.0366; P = 0.430) with R squared 0.9527. Figure 1 shows the Roche enzymatic serum creatinine versus local laboratory creatinine measurements as well as four potential calibration models with combined training and validation datasets (150 subjects) after exclusion of outliers.

Table 2.

Error Estimates for Serum Creatinine Calibration Models

	Mean Squared Error
	Simple Linear Regression	Quadratic Regression	Piecewise Linear Regression	Deming Regression
Excluding outliers
Holdout Method	0.003482	0.003473	0.003399	0.003515
10-fold Cross-validation	0.003510	0.004253	0.003935	0.003557
Leave-one-out Cross-validation	0.003594	0.004308	0.003949	0.003654

Table 3.

Calibration Equation Parameter Estimates

	Intercept ± SE	Slope ± SE	R squared
Complete Data
Simple Linear Regression	0.0186 ± 0.00684	0.920 ± 0.00367	0.9976
Deming Regression	0.0174 ± 0.0128	0.921 ± 0.0122	0.9976
Excluding outliers
Simple Linear Regression	−0.00230 ± 0.0177	0.945 ± 0.0176	0.9533
Deming Regression	−0.0248 ± 0.0313	0.968 ± 0.0329	0.9527

Figure 1.

Roche Enzymatic serum creatinine versus local laboratory creatinine measurements as well as four potential calibration models (A. Simple linear regression; B. Quadratic regression; C. Piecewise linear regression; D. Deming regression) using combined training and validation datasets (150 subjects) after exclusion of outliers. Dotted lines represent “line of identity” and solid black lines represent different calibration models.

ASSESSMENT OF CALIBRATION USING THE TEST SET

The selected Deming regression model was then applied to the test dataset (50 subjects) to assess final performance of the calibration model. The mean difference between the measured and calibrated serum creatinine values was −0.0103 ± 0.0608 mg/dL (−0.911 ± 5.375 μmol/L) (Table 4). Calculation of the concordance correlation coefficient between measured and calibrated serum creatinine values yielded a value of 0.934 (95% CI, 0.894 to 0.960, values >0.75 indicate excellent concordance¹⁸). Figure 2A shows the Roche enzymatic serum creatinine versus local laboratory creatinine measurements as well as the final calibration model using the test dataset; and the Bland-Altman analysis shown in Figure 2B indicates excellent agreement between measured and calibrated serum creatinine values.

Table 4.

Comparison of measured and calibrated serum creatinine values using the test dataset.

	Excluding Outliers* (N = 50)
	Mean ± SD	Range
Serum Creatinine Measured at Local Lab	1.008 ± 0.261	(0.50, 2.20)
Calibrated Serum Creatinine (mg/dL)	0.951 ± 0.253	(0.46, 2.11)
Serum Creatinine by Roche Enzymatic Assay (mg/dL)	0.941 ± 0.243	(0.55, 1.92)
Difference in Serum Creatinine (Roche – Calibrated) (mg/dL)	−0.0103 ± 0.0608	(−0.185, 0.113)
P†	0.237
Percent Difference in Serum Creatinine (Roche – Calibrated)/Roche × 100, %	−1.27 ± 6.60	(−16.46, 16.49)
P‡	0.181

Note: To convert serum creatinine in mg/dL to μmol/L, multiply by 88.4.

^*The outliers were defined as, subjects with absolute difference of (Roche - Lab) > 3 SDs from the mean.^14,15

^†: Paired t-test of the hypothesis that the mean difference is 0.

^‡: Paired t-test of the hypothesis that the mean percent difference is 0.

Figure 2.

Roche Enzymatic serum creatinine versus local laboratory creatinine measurements as well as final chosen Deming regression model (A) and Bland-Altman plot of measured versus calibrated serum creatinine values (B) using test dataset (50 subjects). In panel A, dotted line represents “line of identity” and solid black line represents final calibration model. Solid black line represents mean difference, and dotted lines are ± 1.96 × SD of the difference in panel B.

EFFECT OF CALIBRATION ERROR ON GFR ESTIMATION

The mean squared error for our final calibration model was estimated to be approximately 0.0036 from the resampling validation method, which corresponded to an average calibration error of 0.06 mg/dL serum creatinine. Figure 3 shows the effect of a constant error (0.06 mg/dL) on eGFR by CKD-EPI equation with increasing serum creatinine concentration for four individuals: 40-year-old and 70-year-old African-American man and woman. The effect of calibration error decreased with increasing serum creatinine concentration, and the effect on individuals with a high serum creatinine concentration (> 3.0 mg/dL) was negligible. Thus, calibration error poses little effect on eGFR estimation.

Figure 3.

Estimated GFR for four African-American individuals (A. 40 years old, B. 70 years old) based on CKD-EPI equation with no calibrated errors and with ± 0.06 mg/dL calibrated errors in serum creatinine values traceable to IDMS.

GFR ESTIMATION OF JACKSON HEART STUDY PARTICIPANTS (2000–2004)

Using serum creatinine concentrations calibrated to the Roche (IDMS) method, the baseline prevalence of CKD in the JHS was 6.30%, compared with 8.29% when defined by eGFR using non-calibrated serum creatinine values with the CKD-EPI equation (P < 0.001) (Table 5). One hundred and four subjects classified as having CKD using non-calibrated measurements were reclassified as not having CKD using calibrated measurements; none of the subjects classified as not having CKD using non-calibrated measurements were re-classified as having CKD using calibrated measurements.

Table 5.

Classification of JHS cohort (2000–2004) based on eGFR calculated by CKD-EPI and MDRD Study equation using calibrated and non-calibrated serum creatinine concentrations.

	CKD-EPInon-cal		MDRDcal
	≥60 mL/min/1.73 m2	<60 mL/min/1.73 m2	≥60 mL/min/1.73 m2	<60 mL/min/1.73 m2
CKD-EPIcal
≥60 mL/min/1.73 m2	4778	104	4870	12
<60 mL/min/1.73 m2	0	328	23	305

When eGFR was calculated using the MDRD Study equation and calibrated serum creatinine values, the prevalence of CKD was 6.08%, which was comparable with the prevalence from CKD-EPI equation (6.30%, P = 0.063) (Table 5). Twelve subjects classified as having CKD by the MDRD Study equation were re-classified as not having CKD by CKD-EPI equation, and twenty-three subjects classified as not having CKD by MDRD Study equation were reclassified as having CKD by CKD-EPI equation.

Discussion

We generated an equation to calibrate serum creatinine concentrations to a method traceable to IDMS, estimated the error using training and validation datasets, and assessed its performance within a separate test dataset. After comparison of several regression models, the Deming regression model was selected to represent the best calibration of creatinine concentration. Applying this calibration equation followed by the CKD-EPI equation to the full JHS population, the prevalence of estimated GFR <60 mL/min/1.73m² was 6.30%, a decline from previously calculated value of 8.29%.

For calibration equation selection and true error estimation, we employed holdout method, 10-fold cross validation and leave-one-out cross validation methods to split training and validation datasets. For the holdout method, four different calibration models generate similar mean squared errors. However, since the holdout method is a single train-and-test experiment, the holdout estimate of the error rate can have a high variance and will be misleading if there is an “unfortunate” or “fortunate” split.¹⁹ This limitation can be overcome with resampling methods as cross-validation or bootstrap approach at the expense of higher computational cost. Larger error estimates from quadratic regression and piecewise linear regression using ten-fold cross validation and leave-one-out cross validation methods indicated that both models were excessively complex, having too many parameters that generally have poor predictive performance when generalizing to an independent dataset (for instance, our complete JHS data).²⁰ Simple linear and Deming regression models have similar error estimates and the regression coefficients. As for our final model, we selected the Deming regression, because this model accounted for measurement errors and was also used for serum creatinine calibration in two previously published papers.^14,15 The selected calibration model performed well when applied to the test dataset.

Although the MDRD Study equation applies the same four variables used in the CKD-EPI equation, one of the major shortcomings of the MDRD Study equation has been the systematic underestimation of GFR in patients with high GFR. When GFRs were estimated with both equations in the JHS, more individuals with high GFR (≥60 ml/min/1.73 m²) had higher eGFR calculated from CKD-EPI equation than those calculated from the MDRD Study equation (75.8% vs. 24.2%, P < 0.001). Although results from a meta-analysis showed that the CKD-EPI equation classified fewer individuals as having CKD,⁸ slightly higher CKD prevalence was detected based on eGFR from the CKD-EPI equation in our JHS cohort (2000 – 2004) (this did not reach statistical significance), and the difference may come from the ethnicity or other differences of the subjects between studies.

Our study has some limitations. First of all, the collection of serum specimens for measurement of baseline JHS creatinine concentrations occurred over a span of 5 years (2000 – 2004), during which serum creatinine levels may have drifted. However, the same laboratory, instrument and method were used to measure serum creatinine in all baseline specimens during the 5 years. Secondly, the serum creatinine values in calibration samples are all below 3 mg/dL, which may limit the usage of our calibration equation. At the baseline visit of JHS, we had very few serum creatinine measurements greater 3.0 mg/dL (0.60%, 31/5210), so it is not possible for us to derive an accurate calibration equation to accommodate individuals with high creatinine values. Furthermore, calibration error induced by small variation of the calibration equation will not affect eGFR significantly to change the CKD status for individuals with a high serum creatinine concentration (>3.0 mg/dL) (Figure 3). Thirdly, while we demonstrate differences in the prevalence of low eGFR depending on serum creatinine concentrations (calibrated versus non-calibrated) and the GFR estimation equation applied (CKD-EPI versus MDRD), we did not assess whether differences in estimated GFR classification affect risk of subsequent clinical outcomes. Finally, the actual GFR measurements are not available in this study and the reported calibration equation cannot be generalizable to other serum creatinine measurements, due to inherent technical and environmental differences.

Conclusions

In summary, the Deming regression model was the final calibration equation used to calculate IDMS traceable serum creatinine values, which was found to be essential for the correct use of the CKD-EPI equation to estimate GFR in the JHS Visit 1 (2000 – 2004). Use of this model will allow better precision in determination of eGFR for future studies of JHS.

Abbreviation

IDMS

Isotope Dilution Mass Spectroscopy

CKD-EPI

Chronic Kidney Disease Epidemiology Collaboration

GFR

glomerular filtration rate

JHS

Jackson Heart Study

CI

Confidence Interval

MDRD

Modification of Diet in Renal Disease

ESRD

End Stage Renal Disease

NIST

National Institute of Standards and Technology

SD

Standard Deviation

SE

Standard Error