Abstract
ADPKD is marked by gradual renal cyst and kidney enlargement and ultimately renal failure. Magnetic resonance-based, height-adjusted total kidney volume (htTKV) over 600 ml/m predicts the development of CKD Stage 3 within 8 years in the Consortium for Radiologic Imaging in Polycystic Kidney Disease cohort. Here we compared simultaneous ultrasound and magnetic resonance imaging to determine if ultrasound and kidney length (KL) predict future CKD Stage 3 over longer periods of follow-up. A total of 241 ADPKD patients, 15–46 years, with creatinine clearance of 70 ml/min and above had iothalamate clearance, magnetic resonance and ultrasound evaluations. Participants underwent an average of five repeat clearance measurements over a mean follow-up of 9.3 years. Ultrasound and magnetic resonance-based TKV and KL were compared using Bland-Altman plots and intra-class correlations. Each measure was tested to predict future CKD Stage 3. Relatively strong intra-class correlations between ultrasound and magnetic resonance were found for both htTKV and KL (0.81 and 0.85, respectively). Ultrasound and magnetic resonance-based htTKV and KL predicted future CKD Stage 3 similarly (AUC of 0.87, 0.88, 0.87 and 0.88 respectively). An ultrasound kidney length over 16.5 cm and htTKV over 650 ml/m had the best cut-point for predicting the development of CKD Stage 3. Thus, kidney length alone is sufficient to stratify the risk of progression to renal insufficiency early in ADPKD using either ultrasound or magnetic resonance imaging.
Keywords: Ultrasound, Magnetic Resonance Imaging, Total Kidney Volume
Introduction
Autosomal dominant polycystic kidney disease (ADPKD) is the most common hereditary kidney disorder and the fourth leading cause of renal failure in the USA accounting for 4.8% of the ESRD population1. In ADPKD, cysts develop and subsequently enlarge resulting in increased kidney size decades prior to decline in kidney function, with ESRD typically occurring in the 6th decade of life2. Recently, renal cyst burden measured as total kidney volume (TKV) determined by magnetic resonance imaging (MR) has been shown to be accurate, reproducible and able to detect small changes in TKV over a short period of time3. TKV independently and strongly predicts the future development of chronic kidney disease (CKD) stage 3 within 8 years in ADPKD with a threshold of 1047 mls or 600 ml/m when corrected for height (htTKV)3. These data have enabled the design and implementation of several randomized clinical trials (RCT) in ADPKD4 and large patient registries show that TKV in addition to age and kidney function confidently predict a 30% decline in kidney function, doubling of serum creatinine concentration and progression to ESRD3. This increasing body of evidence has led to TKV being reviewed by the Food and Drug Administration as an enrichment biomarker for inclusion in clinical trials of ADPKD4.
Although highly accurate and reproducible, TKV measurements by MR are time-consuming, expensive and not available to all patients. MR relies on specific image acquisition parameters, patient adherence, and relatively manual and time-intensive image analyses done by trained personnel. Alternative approaches to estimate TKV from MR images have been developed. In one approach a representative mid-slice surface area is multiplied by the slice thickness and the number of slices required to image the entire kidney5. Another approach utilizes the ellipsoid formula based on maximum coronal or sagittal dimensions for kidney length and axial dimensions for width and depth6. This approach has been used successfully with US to follow TKV in patients over extended periods (often decades) of time and associates with complications in ADPKD such as hypertension and renal disease progression7. However, the accuracy and precision of US are insufficiently precise to support its use to measure TKV in interventional clinical trials. Kidney length can estimate TKV5 and is easily obtained by US or MR. US is readily available in all medical centers, is the most commonly employed imaging modality worldwide for diagnosing ADPKD, and is advantageous due to its relatively low cost and lack of radiation exposure. Estimation of TKV from kidney length reduces imaging and analysis time as well as cost but it is unknown how well kidney length performs in predicting the future development of CKD in ADPKD and what the impact of the imaging modality chosen (US vs. MRI) has on this prediction model.
In the Consortium for Radiologic Imaging in Polycystic Kidney Disease (CRISP) cohort, US and MR measurements were made at the baseline visit and the first year of follow-up. As anticipated, MR was more accurate and reproducible than US6. Although US accurately approximated TKV in individuals with relatively small kidneys, the agreement between US and TKV declined as kidney size increased with US systematically overestimating TKV6. Given the greater variability of US-based TKV measurements and the reduced precision in assessing kidney cyst burden, US was deemed inferior for the purposes of imaging biomarker development, and was discontinued for the rest of the CRISP follow-up.
To assess whether US-based measurements of TKV or kidney length reliably predict future renal insufficiency in ADPKD, we compared the US and MR imaging data collected simultaneously during the baseline visit in CRISP participants, similar to previous studies evaluating htTKV measurements by MR3. We compared MR and US kidney length and determined the accuracy of MR and US-based htTKV and kidney length to predict future CKD Stage 3 measured by iothalamate clearance in ADPKD over 13 years of follow-up.
Results
The mean follow-up (i.e. time from baseline to last measured iothalamate clearance) was 9.3 years (SD 3.3 years) with a median follow-up of 10.8 years. The middle 50% of follow-up times was 8.2 to 11.3 years. A total of 202 (83.8%) and 182 (75.5%) had an iothalamate clearance measured at a minimum of 6 and 8 years after baseline, respectively. Only 23 (9.5%) did not have a measurement after 3 years. The mean number of visits per person was 6.0 (SD=1.6) with a median of 7 visits and middle 50% of 5 to 7 visits. Only 14 people (5.8%) had 3 or fewer visits. Baseline distributions are described in Table 1; 88% were white, 60% were women, and 10% were black. Average baseline htTKV by MR was 637 ml (SD ±371 ml). Average kidney length measured by US and MR for the left kidney was 16.8 ±3.9 cm and 16.3 ±3.5 cm and the right kidney was 16.2±3.9 cm and 15.8 cm±3.2 cm respectively. Bland-Altman plots demonstrated high levels of agreement between US and MR kidney length (Figures 1a–d) as did the scatter plots (Figures 2a–b). The bias, or mean paired difference (± standard deviation) of US – MR measurement was 44.2±127.6 mls for right and 44.9±147.4 mls for left kidney htTKV, 0.39±2.08 cm for right and 0.51±2.11 cm for left kidney length. These levels of bias were all statistically significant (p<0.01; paired t-test) indicating that US systematically overestimates kidney size. Intraclass correlations (ICC) reflected a relatively strong agreement between the two modalities with intraclass correlation coefficient (ICC) for both genders for the right kidney being 0.828 (0.788 to 0.869) and the left kidney being 0.827 (0.786 to 0.868). The strongest ICC was seen in men for the left kidney with an ICC of 0.849 (0.792 to 0.906) (Table 2).
Table 1.
Variable | Mean ± SD | Women N = 145 |
Men N = 96 |
---|---|---|---|
Age (yrs) | 33.8 ± 8.9 | 34.0 ± 9.0 | 33.5 ± 8.8 |
Weight (kg) | 77.0 ± 18.4 | 69.7 ± 15.8 | 88.1 ± 16.6 |
BMI (m2) | 25.9 ± 5.2 | 25.5 ± 5.6 | 26.5 ± 4.7 |
MR htTKV (ml/m) | 637.5 ± 371.8 (N=231) | 632.7 ± 360.9 (N=138) | 644 ± 389.4 (N=93) |
US htTKV (ml/m) | 719.6 ± 506.1 (N=231) | 694.0 ± 496.3 (N=139) | 758.2 ± 520.8 (N=92) |
MR TKV (ml) | 1093.9 ± 672.0 (N=229) | 1031.7 ± 620.9 (N=137) | 1186.6 ± 735.5 (N=92) |
US TKV (ml) | 1247.4 ± 896.7 (N=231) | 1151.8 ± 836.0 (N=139) | 1391.9 ± 968.2 (N=92) |
L US KL (cm) | 16.8 ± 3.9 (N=232) | 16.4 ± 3.8 (N=139) | 17.3 ± 3.9 (N=93) |
L MR KL (cm) | 16.3 ± 3.4 (N=232) | 16.1 ± 3.5 (N=139) | 16.6 ± 3.3 (N=93) |
R US KL (cm) | 16.2 ± 3.9 (N=232) | 15.9 ± 3.8 (N=139) | 16.5 ± 4.0 (N=93) |
R MR KL (cm) | 15.8 ± 3.2 (N=232) | 15.7 ± 3.3 (N=139) | 15.9 ± 3.1 (N=93) |
SBP (mm Hg) | 125.1 ± 13.6 | 122.7 ± 14.0 | 128.5 ± 12.3 |
DBP (mm Hg) | 83.4 ± 10.7 | 83.2 ± 11.0 | 83.6 ± 10.4 |
MAP (mm Hg) | 98.2 ± 10.9 | 97.4 ± 11.5 | 99.4 ± 9.7 |
Serum creatinine (mg/dL) GFR (mL/mm/1.73 m2) | 1.0 ± 0.2 98.2 ± 24.9 | 0.9 ± 0.2 99.8 ± 26.6 | 1.1 ± 0.2 95.8 ± 22.1 |
Sodium excretion (mEq/day) | 185.5 ± 87.2 | 162.0 ± 74.8 | 220.9 ± 92.8 |
Potassium excretion (mEq/day) | 56.5 ± 24.7 | 51.1 ± 21.3 | 64.7 ± 27.3 |
Albumin excretion (mg/day) | 42.4 ± 60.9 | 39.4 ± 48.2 | 47.0 ± 76.2 |
Total cholesterol (mg/dL) | 172.9 ± 37.0 | 169.2 ± 34.1 | 178.6 ± 40.6 |
HDL (mg/dL) | 49.2 ± 22.6 | 51.8 ± 12.5 | 45.3 ± 31.96 |
LDL (mg/dL) | 102.4 ± 34.3 | 97.6 ± 28.9 | 109.8 ± 40.2 |
Triglycerides (mg/dL) | 120.2 ± 95.0 | 102.4 ± 63.0 | 147.5 ± 125.0 |
Table 2.
Summary of agreement based on intra-class correlations | |||
---|---|---|---|
Gender | Side | ICC | ICC 95% CI |
Overall | Right | 0.82854 | 0.78816, 0.86893 |
Left | 0.82709 | 0.78640, 0.86779 | |
Male | Right | 0.84375 | 0.78504, 0.90246 |
Left | 0.84892 | 0.79200, 0.90585 | |
Female | Right | 0.81864 | 0.76372, 0.87357 |
Left | 0.81137 | 0.75446, 0.86827 |
Table 3 shows the classification results for each of the four imaging measures, using both the multivariable model (that adjusts for serum creatinine, BUN, urine albumin excretion, urine MCP-1, and age) and the results for each individual measurement. Results show that all four measurements are similar (Figure 3) in the multivariable model (with the an AUROC of 0.87 to 0.88). The unadjusted results were also very similar with an AUROC of 0.82 and 0.83 for US and MR htTKV and an AUROC of 0.81 and 0.78 for US and MR kidney length. The optimal cut points (which best balanced between sensitivity and specificity) were 650 and 550 ml for US and MR htTKV (based on intervals of 50 mls) and 16.5 and 16.0 cm for US and MR kidney length (based on intervals of 0.5 cm), respectively.
Table 3.
Method | AUC ROC | sensitivity | specificity | Optimal cut point |
---|---|---|---|---|
htTKV (MR) ml/m (ref 7) | 0.84 | 74.0% | 75.0% | 600 ml |
htTKV (US) ml/m | 0.87 | 79.5% | 73.2% | 630 ml |
KL (MR) cm | 0.87 | 85.4% | 92.3% | 16.7 cm |
KL (US) cm | 0.86 | 82.9% | 80.8% | 16.8 cm |
It is worth noting that the imaging measures each led to higher individual classification results than the other baseline covariates, when considered individually, which included serum creatinine (AUROC=0.75), BUN (AUROC=0.70), urine albumin (AUROC=0.63), urine MCP-1 (AUROC=0.74), and age (AUROC=0.71). The NRI and IDI also reflected improved classification with imaging measures added to the multivariable model versus the model without the imaging measurement (i.e. with the other five covariates). The NRI estimates ranged between 0.02 and 0.07, with confidence intervals crossing zero. The IDI statistics, all had lower limits to the CI above 0.0 with estimates of 0.03 to 0.05, implying that addition of all imaging measurements lead to a similar net gain of 3–5% in the predicted probability of Stage 3 CKD.
Discussion
Kidney cyst burden in ADPKD is most accurately reflected by TKV measurements obtained using MR in ADPKD. In addition, TKV is currently the strongest available predictor of future renal insufficiency in ADPKD. US measurements of TKV have been shown to be inferior to MR measurements, typically overestimating true kidney size. As well US measures of TKV demonstrate reduced accuracy and reproducibility compared to MR, particularly in larger kidneys and are not well suited to tracking disease progression over short periods of time6. However, US determinants of TKV are relevant in ADPKD and provide a quantitative assessment of cyst burden and associate with renal complications of ADPKD including hypertension, gross hematuria, proteinuria and risk for progression to ESRD2,7–18. While US may not be accurate enough to assess the impact of therapies on renal cyst burden in clinical trials, this study demonstrates that US-based htTKV can be used as a tool to predict future loss of kidney function and renal insufficiency and therefore patient risk stratification for disease progression. Importantly, kidney length is a standard component of every renal evaluation during imaging by US and can easily be performed by MR. As well, higher resolution US imaging has been developed which will make these measures even more reliable.
Comparisons of US and MR with regard to kidney length and TKV were informative. This study shows that US and MR measurements are similar in kidneys of normal to moderate size (< 17 cm). The high level of accuracy of US when kidney length is < 17 cm is most likely due to the fact that the entire kidney can be obtained in a single view. This kidney length is greater than the cut point that provided the best combined sensitivity and specificity in AUROC for predicting future renal insufficiency in ADPKD. US had a reduced agreement with MR-based TKV, overestimating TKV by 11% and 9% using ellipsoid formula and direct methodologies, respectively. When assessing the ability of kidney length to predict future CKD stage 3, US and MR kidney length demonstrated robust power (AUROC = 0.88 and 0.87 respectively) similar to baseline htTKV by MR and US (AUROC = 0.87 and 0.86 respectively), Table 3.
TKV has been established as a strong predictor of renal outcomes and complications in ADPKD patients. This prediction is stronger than and replaces PKD1 genotype, proteinuria and hypertension status, all known to associate with faster progression to renal failure. MR is known to be better than US with regard to accuracy and reproducibility, which is important for monitoring for disease progression over a short period of time as would be the case in clinical trials. The agreement between US and MR decreases as kidney size increases. US image acquisition is operator-dependent and requires appropriate measurement of depth and width for assessment of TKV, unlike kidney length which is standard part of each US study. US is readily available and does not have any restrictions in terms of metallic clips or other implanted devices. Our study indicates that prediction of future CKD requires only a measurement of kidney length for prediction of future CKD stage 3 within 8 years and that US and MR are equivalent for this purpose. Using US kidney length to predict future CKD stage 3 in ADPKD will reduce costs while providing the necessary prognostic information regarding potential outcomes and complications related to ADPKD.
Methods
241 ADPKD patients between 15 to 46 years of age with relatively intact kidney function with an estimated creatinine clearance of > 70 ml/min using the Cockcroft Gault equation participated in the CRISP study. Four centers participated in this study including 1) University of Alabama at Birmingham 2) Emory University 3) University of Kansas Medical Center and 4) Mayo College of Medicine. The study protocol was approved by the steering committee and by the Institutional Review Boards at each participating clinical center. The original description of this study is provided in detail19. Briefly, after screening and enrollment, CRISP participants underwent a detailed medical history including the year and method of diagnosis of ADPKD along with any ADPKD associated complications, prior medical conditions, hospitalizations or surgeries. Medication exposures were obtained including over-the-counter and prescription medications. At baseline, subjects completed a 24-hour urine collection for creatinine, albumin and electrolyte excretions. Glomerular filtration rate (GFR) was established using non-isotopic iothalamate clearance.
MR and US were obtained during the initial baseline study of CRISP at all 4 participating clinical centers. Subjects underwent MR imaging using a standardized protocol approved by the Steering Committee and has been published elsewhere19. MR images were obtained at each clinical site by a trained imaging specialist with the patient in supine position using a dedicated 1.5 T scanner. Images were obtained with a slice thickness of 3 mm and TKV was determined by summing cross-sectional area of T1 weighted images. Experienced sonographers using 2 to 5 Hz curvilinear phased array probes performed US following MR imaging. Kidney length was measured by the sonographer from longitudinal images obtained in a sagittal plane6. US based TKV was measured using two different techniques. First the ellipsoid formula1 was used to determine TKV. Second, the cross sectional area of the kidney was measured in transverse sections at 2 cm intervals. The sum of the cross-sectional areas was multiplied by 2 to estimate TKV6.
MR-Based Image Analysis
For kidney length, analysts used a semi-automated image analysis program developed and implemented in-house. The image analyst scrolled the MR images and identified and marked both the upper most and lower most edges of the kidney. The coronal MR images of each kidney were displayed. These edge points were usually determined using two different coronal MR slices because the longest length of kidney is on the oblique coronal plane of the kidney. Kidney length was automatically calculated from the distance between the upper and lower edge points. For MR based TKV measures, detailed information regarding the analysis is published elsewhere19.
Statistical Methods
Comparisons between US and MR htTKV and kidney length measures were initially performed separately in men and women. The level of agreement between US and MR was displayed using Bland-Altman plots and further quantified using intra-class correlations. Analyses were performed using US and MR htTKV and kidney length to predict CKD stage 3 (iothalamate clearance < 60 ml/min); CKD stage for each participant was determined based on their last iothalamate clearance measurement over the 13 years of follow-up. The prediction models for CKD stage 3 used a logistic model that adjusted for other variables including serum creatinine, BUN, urine albumin excretion, urine monocyte chemo-attractant protein-1 (MCP-1), and age as reported previously for MR-based htTKV3. Areas under the receiver operator characteristic (AUROC) curve were determined across 50 ml volume intervals to determine the both sensitivity and specificity for predicting CKD stage 3. Cut-points leading to the best balance of sensitivity and specificity were reported for each measure. Since follow-up times were variable across subjects, a sensitivity analysis was conducted as follows: the AUROC analysis was repeated using the following data sets: 1) limiting the analysis to all subjects who were followed for at least 6 years, and 2) limiting the analysis to all subjects who were followed for at least 12 years. Since results for these analyses were similar, we used all follow-up data for the analysis reported here.
Each of the other five covariates (serum creatinine, BUN, urine albumin excretion, urine MCP-1, and age) were also assessed for their prognostic ability using an (unadjusted) AUROC curve. Finally, US and MR htTKV and kidney length were assessed for their additional prognostic ability in the multivariable model above and beyond the prognostic ability of the model with all of the other five covariates. This was accomplished using the net reclassification index (NRI) and the integrated discrimination improvement (IDI) index. The NRI measures the net gain in correct classification of subjects as high versus low risk (where we used the median predicted probability of CKD stage 3 as a cut-off); the IDI measures the net improvement in the continuous predicted probabilities20. The NRI and IDI were calculated using the incrisk function in Stata 12 (which produces a confidence interval rather than a p-value).
Footnotes
Disclosure
The authors declare no competing interests.
References
- 1.Driscoll JA, Bhalla S, Liapis H, et al. Autosomal dominant polycystic kidney disease is associated with an increased prevalence of radiographic bronchiectasis. Chest. 2008;133:1181–1188. doi: 10.1378/chest.07-2147. [DOI] [PubMed] [Google Scholar]
- 2.Franz KA, Reubi C. Rate of Functional Deterioration in Polycystic Kidney Disease. Kidney International. 1983;24:276–276. doi: 10.1038/ki.1983.51. [DOI] [PubMed] [Google Scholar]
- 3.Chapman AB, Bost JE, Torres VE, et al. Kidney Volume and Functional Outcomes in Autosomal Dominant Polycystic Kidney Disease. Clinical Journal of the American Society of Nephrology. 2012;7:479–486. doi: 10.2215/CJN.09500911. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Perrone RD, Marier JF, Mouksassi MS, et al. Qualification of Total Kidney Volume as a Prognostic Biomarker for Use in Clinical Trials Evaluating Patients with Autosomal Dominant Polycystic Kidney Disease. Am J Kidney Dis. 2014;63:A119. [Google Scholar]
- 5.Bae KT, Tao C, Wang JH, et al. Novel Approach to Estimate Kidney and Cyst Volumes Using Mid-Slice Magnetic Resonance Images in Polycystic Kidney Disease. American Journal of Nephrology. 2013;38:333–341. doi: 10.1159/000355375. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.O'Neill WC, Robbin ML, Bae KT, et al. Sonographic assessment of the severity and progression of autosomal dominant polycystic kidney disease: the Consortium of Renal Imaging Studies in Polycystic Kidney Disease (CRISP) Am J Kidney Dis. 2005;46:1058–1064. doi: 10.1053/j.ajkd.2005.08.026. [DOI] [PubMed] [Google Scholar]
- 7.Gabow PA, Johnson AM, Kaehny WD, et al. Factors affecting the progression of renal disease in Autosomal Dominant Polycystic Kidney disease. Kidney International. 1992;41:1311–1319. doi: 10.1038/ki.1992.195. [DOI] [PubMed] [Google Scholar]
- 8.Gabow PA, Duley I, Johnson AM. Clinical profiles of gross hematuria in autosomal dominant polycystic kidney disease. Am J Kidney Dis. 1992;20:140–143. doi: 10.1016/s0272-6386(12)80541-5. [DOI] [PubMed] [Google Scholar]
- 9.Chapman AB, Johnson AM, Gabow PA, et al. Overt proteinuria and microalbuminuria in autosomal dominant polycystic kidney disease. Journal of the American Society of Nephrology : JASN. 1994;5:1349–1354. doi: 10.1681/ASN.V561349. [DOI] [PubMed] [Google Scholar]
- 10.Johnson AM, Gabow PA. Identification of patients with autosomal dominant polycystic kidney disease at highest risk for end-stage renal disease. Journal of the American Society of Nephrology : JASN. 1997;8:1560–1567. doi: 10.1681/ASN.V8101560. [DOI] [PubMed] [Google Scholar]
- 11.Dalgaard OZ. Bilateral polycystic disease of the kidneys; a follow-up of two hundred and eighty-four patients and their families. Acta medica Scandinavica. Supplementum. 1957;328:1–255. [PubMed] [Google Scholar]
- 12.Fick-Brosnahan GM, Tran ZV, Johnson AM, et al. Progression of autosomal-dominant polycystic kidney disease in children. Kidney Int. 2001;59:1654–1662. doi: 10.1046/j.1523-1755.2001.0590051654.x. [DOI] [PubMed] [Google Scholar]
- 13.Sharp C, Johnson A, Gabow P. Factors relating to urinary protein excretion in children with autosomal dominant polycystic kidney disease. Journal of the American Society of Nephrology : JASN. 1998;9:1908–1914. doi: 10.1681/ASN.V9101908. [DOI] [PubMed] [Google Scholar]
- 14.Seeman T, Dusek J, Vondrichova H, et al. Ambulatory blood pressure correlates with renal volume and number of renal cysts in children with autosomal dominant polycystic kidney disease. Blood Press Monit. 2003;8:107–110. doi: 10.1097/01.mbp.0000085762.28312.4a. [DOI] [PubMed] [Google Scholar]
- 15.Fick-Brosnahan GM, Belz MM, McFann KK, et al. Relationship between renal volume growth and renal function in autosomal dominant polycystic kidney disease: a longitudinal study. Am J Kidney Dis. 2002;39:1127–1134. doi: 10.1053/ajkd.2002.33379. [DOI] [PubMed] [Google Scholar]
- 16.Chapman AB, Schrier RW. Pathogenesis of hypertension in autosomal dominant polycystic kidney disease. Seminars in nephrology. 1991;11:653–660. [PubMed] [Google Scholar]
- 17.Thomsen HS, Madsen JK, Thaysen JH, et al. Volume of polycystic kidneys during reduction of renal function. Urol Radiol. 1981;3:85–89. doi: 10.1007/BF02927815. [DOI] [PubMed] [Google Scholar]
- 18.Sise C, Kusaka M, Wetzel LH, et al. Volumetric determination of progression in autosomal dominant polycystic kidney disease by computed tomography. Kidney Int. 2000;58:2492–2501. doi: 10.1046/j.1523-1755.2000.00433.x. [DOI] [PubMed] [Google Scholar]
- 19.Chapman AB, Guay-Woodford LM, Grantham JJ, et al. Renal structure in early autosomal-dominant polycystic kidney disease (ADPKD): The Consortium for Radiologic Imaging Studies of Polycystic Kidney Disease (CRISP) cohort. Kidney International. 2003;64:1035–1045. doi: 10.1046/j.1523-1755.2003.00185.x. [DOI] [PubMed] [Google Scholar]
- 20.Pencina MJ, D'Agostino RB, Sr, D'Agostino RB, Jr, et al. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Statistics in medicine. 2008;27:157–172. doi: 10.1002/sim.2929. discussion 207-112. [DOI] [PubMed] [Google Scholar]