Abstract
Acute kidney injury (AKI) occurs commonly after pediatric cardiac surgery and associates with poor outcomes. Biomarkers may help the prediction or early identification of AKI, potentially increasing opportunities for therapeutic interventions. Here, we conducted a prospective, multicenter cohort study involving 311 children undergoing surgery for congenital cardiac lesions to evaluate whether early postoperative measures of urine IL-18, urine neutrophil gelatinase-associated lipocalin (NGAL), or plasma NGAL could identify which patients would develop AKI and other adverse outcomes. Urine IL-18 and urine and plasma NGAL levels peaked within 6 hours after surgery. Severe AKI, defined by dialysis or doubling in serum creatinine during hospital stay, occurred in 53 participants at a median of 2 days after surgery. The first postoperative urine IL-18 and urine NGAL levels strongly associated with severe AKI. After multivariable adjustment, the highest quintiles of urine IL-18 and urine NGAL associated with 6.9- and 4.1-fold higher odds of AKI, respectively, compared with the lowest quintiles. Elevated urine IL-18 and urine NGAL levels associated with longer hospital stay, longer intensive care unit stay, and duration of mechanical ventilation. The accuracy of urine IL-18 and urine NGAL for diagnosis of severe AKI was moderate, with areas under the curve of 0.72 and 0.71, respectively. The addition of these urine biomarkers improved risk prediction over clinical models alone as measured by net reclassification improvement and integrated discrimination improvement. In conclusion, urine IL-18 and urine NGAL, but not plasma NGAL, associate with subsequent AKI and poor outcomes among children undergoing cardiac surgery.
Acute kidney injury (AKI) is a frequent complication of pediatric cardiac surgery and negatively effects short- and long-term outcomes.1–5 Despite decades of research, no therapy has proven effective for the prevention or treatment of human AKI. Serum creatinine, the traditional marker of renal function, only rises appreciably after a 50% loss in GFR. Serum creatinine is also affected by several nonrenal factors and on average does not peak until 1 to 3 days after cardiac surgery.2,6 Thus, our ability to detect AKI early remains inadequate. The failure of prior interventional trials to attenuate AKI in cardiac surgery patients has also been attributed, in part, to the delays in the diagnosis of AKI.7,8 It is currently believed that progress in this area will occur after the availability of newer biomarkers for early and reliable diagnosis of AKI.9 Our preclinical studies in mice and initial human studies demonstrate that urine IL-18 and urine neutrophil gelatinase-associated lipocalin (NGAL) levels are early markers of AKI.10–15 In human studies, both are elevated 24 to 48 hours before the clinical diagnosis of AKI becomes apparent. Plasma NGAL has also demonstrated encouraging early results in several small pediatric and adult studies.1,2,16,17
We conducted a large, prospective, multicenter cohort study of pediatric patients receiving surgery for congenital cardiac lesions to evaluate whether early, postoperative measures of urine IL-18, urine NGAL, and plasma NGAL could predict severe AKI and adverse patient outcomes.
RESULTS
Characteristics of the Study Cohort
We studied 311 participants (Supplementary Figure 1). 51% of the patients were below two years of age, and 55% were male. Demographic and clinical characteristics of enrolled patients were similar to the overall cardiac surgery populations at participating institutions. Most surgeries were elective (91%) and utilized cardiopulmonary bypass (CPB) (99%). The mean preoperative estimated GFR (via Schwartz equation) was 90 ml/min per 1.73 m2 (Table 1). 53 of 311 participants (17%) developed severe AKI as defined by either receipt of acute dialysis or postoperative doubling of serum creatinine during hospital stay. Five patients (1.6%) received acute dialysis and six (2%) patients died before discharge. Patients who developed severe AKI were younger, had longer cardiopulmonary bypass time and cross-clamp time (Table 1). The median time to diagnosis of severe AKI was 2 days (interquartile range [IQR], 1 to 2), and serum creatinine peaked on or after day 3 of surgery in 19% of those with severe AKI. Doubling of serum creatinine lasted for 2 days or longer in 47% of patients with severe AKI.
Table 1.
Cohort description among children by severe AKI status
| Developed Severe AKI |
||||
|---|---|---|---|---|
| All (n = 311) | Yes (n = 53) | No (n = 258) | P | |
| Age at the time of surgery, mean (SD) | 3.8 (4.5) | 2.7 (4.3) | 4.0 (4.5) | 0.02 |
| Age at the time of surgery, n (%) | <0.01 | |||
| 1 month to 2 years | 158 (51%) | 37 (70%) | 121 (47%) | |
| 2 to 13 years | 134 (43%) | 13 (25%) | 121 (47%) | |
| 13 to 18 years | 19 (6%) | 3 (5%) | 16 (6%) | |
| Male gender, n (%) | 171 (55%) | 27 (51%) | 144 (56%) | 0.5 |
| Caucasian race, n (%) | 254 (82%) | 42 (79%) | 212 (82%) | 0.6 |
| Type of surgery, n (%) | 0.3 | |||
| elective | 283 (91%) | 46 (87%) | 237 (92%) | |
| urgent | 28 (9%) | 7 (13%) | 21 (8%) | |
| Renal function | ||||
| preoperative eGFR (ml/min per 1.73 m2), mean (SD) | 90 (26) | 97 (28) | 89 (25) | 0.04 |
| preoperative eGFR (percentile), mean (SD)a | 53 (34) | 69 (32) | 49 (33) | <0.0001 |
| Preoperative medications | ||||
| ACE inhibitors, n (%) | 54 (18%) | 12 (23%) | 42 (16%) | 0.3 |
| aspirin, n (%) | 36 (12%) | 7 (13%) | 29 (11%) | 0.7 |
| beta blockers, n (%) | 11 (4%) | 3 (6%) | 8 (3%) | 0.4 |
| Operative characteristics | ||||
| RACHS-1 score, mean (SD)b | 2.4 (0.7) | 2.6 (0.7) | 2.4 (0.7) | 0.3 |
| RACHS-1 score, n (%) | <0.0001 | |||
| 1 | 18 (6%) | 0 (0%) | 18 (7%) | |
| 2 | 153 (50%) | 30 (57%) | 123 (48%) | |
| 3 | 126 (41%) | 16 (30%) | 110 (43%) | |
| 4 | 12 (4%) | 7 (13%) | 5 (2%) | |
| Not categorized | 2 | 0 | 2 | |
| CPB use, n (%) | 307 (99%) | 52 (98%) | 255 (99%) | 0.5 |
| CPB time (min), mean (SD) | 107 (62) | 157 (90) | 97 (50) | <0.0001 |
| Cross-clamp time (min), mean (SD) | 48 (46) | 71 (57) | 43 (41) | <0.001 |
| Outcomes | ||||
| length of ICU stay (days), mean (SD) | 4.1 (6.3) | 9.1 (10.9) | 3.2 (4.3) | <0.0001 |
| median (IQR) | 2 (1, 4) | 5 (3, 8) | 2 (1, 3) | |
| length of hospital stay (days), mean (SD) | 8.5 (10.8) | 17.3 (19.2) | 6.8 (7.0) | <0.0001 |
| median (IQR) | 5 (4, 8.5) | 9 (6, 16) | 5 (3, 7) | |
| dialysis or mortality, n (%) | 10 (3%) | 8 (15%) | 2 (1%) | <0.0001 |
| dialysis, n (%) | 5 (2%) | 5 (9%) | 0 (0%) | <0.0001 |
| mortality, n (%) | 6 (2%) | 4 (8%) | 2 (1%) | 0.0087 |
Severe AKI is defined as the receipt of dialysis or a doubling of serum creatinine during hospitalization. ICU, intensive care unit; ACE, angiotensin-converting enzyme.
aPercentile eGFR was calculated by quantile regression based on published normal renal function measured by nuclear medicine scan GFR in 651 children.31
bThe RACHS-1 consensus-based score system categorizes the complexity of surgery. Uncategorizable RACHS-1 scores were not included in the continuous summary of RACHS scores.
Biomarker Results and Postoperative Risk of AKI
The first postoperative samples (0- to 6-hour time point) were collected at a median of 0.5 (IQR, 0.25 to 2) hours after arrival in the ICU. In patients with and without AKI, urine IL-18, urine NGAL, and plasma NGAL concentrations peaked at the first collection but were higher in those with AKI. Urine biomarkers declined over the first 2 postoperative days, whereas plasma NGAL remained elevated on all postoperative days (Figure 1). The distribution of biomarker values at each time point by AKI status is shown in Supplementary Table 2.
Figure 1.
Urine IL-18, Urine NGAL, and Plasma NGAL peaked at the first collection (0 to 6 hours). AKI is defined by receipt of acute dialysis or doubling in serum creatinine during the hospital stay. 0 to 6 hours, day 2, and day 3 are the sample collection times after surgery. Urine IL-18 is significant at all time points except day 3. Urine NGAL is significant at all time points except the preoperative (pre-op) and day-3 time points. Plasma NGAL is NS at any time point.
Associations of the first postoperative biomarker measures, categorized into quintiles of urine IL-18, urine NGAL, and plasma NGAL, with the risk of AKI are presented in Figure 2. Quintiles of urine IL-18 and urine NGAL had a graded relationship with the risk for severe AKI, ranging from 4 to 6% in the lowest quintiles to 33 to 36% in the highest quintiles (P value for trend <0.001). After adjustment for site and for clinical variables, the fourth and fifth quintiles of urine IL-18 and the fifth quintile of urine NGAL were significantly associated with at least four-fold odds of severe AKI (Table 2). None of the quintiles of plasma NGAL were independently associated with severe AKI.
Figure 2.
Quintiles of the first postoperative Urine IL-18 and Urine NGAL had a graded relationship with the risk for severe AKI. Unadjusted P for trend is reported. Risk of AKI is the percentage of AKI cases within each quintile (Q). AKI is defined by either receipt of acute dialysis or evidence of postoperative doubling of the preoperative serum creatinine value.
Table 2.
Association of biomarkers and AKI
| Quintile (Cut Points)a | Severe AKIb |
AKI RIFLE Rc |
||||
|---|---|---|---|---|---|---|
| AKI Cases (%)d | Adjusted OR Demographics (95% CI)e | Adjusted OR Full (95% CI)f | AKI Cases (%)d | Adjusted OR Demographics (95% CI)e | Adjusted OR Full (95% CI)f | |
| Urine IL-18 (pg/ml) | ||||||
| Q1 (<16) | 4.9 | 1 (referent) | 1 (referent) | 19.7 | 1 (referent) | 1 (referent) |
| Q2 (17, 43) | 8.1 | 1.5 (0.3, 6.8) | 1.1 (0.2, 5.3) | 43.5 | 2.7 (1.2, 6.2) | 2.3 (0.9, 5.8) |
| Q3 (44,123) | 13.1 | 2.9 (0.7, 11.8) | 2.5 (0.6, 10.6) | 41.0 | 2.7 (1.1, 6.2) | 2.2 (0.9, 5.3) |
| Q4 (124, 362) | 25.8 | 6.5 (1.7, 24.8) | 4.3 (1.0, 18.1) | 50.0 | 3.5 (1.5, 8.2) | 2.4 (0.9, 6) |
| Q5 (>362) | 32.8 | 9.4 (2.5, 35.5) | 6.9 (1.7, 28.8) | 54.1 | 4.2 (1.8, 9.9) | 2.6 (1.0, 6.7) |
| Urine NGAL (ng/ml) | ||||||
| Q1 (<3) | 6.6 | 1 (referent) | 1 (referent) | 23.0 | 1 (referent) | 1 (referent) |
| Q2 (3,8) | 11.3 | 1.6 (0.4, 6.1) | 1.4 (0.4, 5.5) | 33.9 | 1.5 (0.6, 3.4) | 1.3 (0.5, 3.1) |
| Q3 (9,19) | 9.8 | 1.3 (0.3, 5.2) | 1.2 (0.3, 5.1) | 36.1 | 1.6 (0.7, 3.8) | 1.4 (0.5, 3.4) |
| Q4 (20,70) | 21.0 | 3.1 (0.9, 10.5) | 2.1 (0.5, 7.8) | 56.5 | 3.5 (1.6, 7.9) | 2.1 (0.8, 5.2) |
| Q5 (>70) | 36.1 | 6.7 (2.1, 21.7) | 4.1 (1.0, 16.3) | 59.0 | 4.0 (1.7, 9.0) | 2.1 (0.8, 5.7) |
| Plasma NGAL (ng/ml) | ||||||
| Q1 (<81) | 15.8 | 1 (referent) | 1 (referent) | 38.6 | 1 (referent) | 1 (referent) |
| Q2 (82,123) | 12.3 | 0.8 (0.3, 2.3) | 0.5 (0.2, 1.8) | 40.4 | 1.1 (0.5, 2.4) | 1.0 (0.4, 2.3) |
| Q3 (124,175) | 15.5 | 1.0 (0.3, 2.7) | 0.8 (0.2, 2.5) | 43.1 | 1.3 (0.6, 2.7) | 1.2 (0.5, 2.8) |
| Q4 (176, 259) | 19.3 | 1.3 (0.5, 3.6) | 1.3 (0.4, 3.9) | 33.3 | 0.8 (0.4, 1.9) | 0.7 (0.3, 1.7) |
| Q5 (>259) | 22.8 | 1.9 (0.7, 5.1) | 2.2 (0.7, 6.9) | 50.9 | 2.3 (1.0, 5.0) | 2.3 (1.0, 5.5) |
aThe number of patients per quintile (Q): urine IL-18, n = 61 or 62; urine NGAL, n = 61 or 62; plasma NGAL, n = 57 or 58.
bSevere AKI is defined as the receipt of dialysis or a doubling of serum creatinine during hospitalization.
cAKI RIFLE R is defined as a rise of 50% or higher in serum creatinine during hospitalization.
dPercentage AKI cases within each quintile.
eAdjusted for age (per year), gender, Caucasian race, and site.
fAdjusted for age (per year), gender, Caucasian race, site, CPB time >120 min, nonelective surgery, RACHS ≥3, and preoperative eGFR percentile and site.
Biomarker Results and Nonrenal Outcomes
The average lengths of stay in the ICU and in the hospital for the entire cohort were 4.1 days (SD, 6.3) and 8.5 days (SD, 10.8), respectively. Urine IL-18, urine NGAL, and plasma NGAL were all linearly associated with longer lengths of hospital and intensive care unit stay after adjustment for other prognostic factors (Figure 3A). The average length of mechanical ventilation for the entire cohort was 1.5 days (SD, 1.5). Quintiles of each of the three biomarkers were also associated with the duration of mechanical ventilation after surgery in multivariable analyses (Figure 3B).
Figure 3.
(A) First post-operative Urine IL-18, Urine NGAL, and plasma NGAL were all linearly associated with longer lengths of hospital and intensive care unit stay after adjustment for other prognostic factors. (B) First postoperative quintiles of each of the three biomarkers were associated with the duration of mechanical ventilation after surgery after adjustment for other prognostic factors. The adjusted P for trend is reported. Q, quintile.
Diagnostic Testing
The area under the receiver operating characteristic curve (AUC-ROC) for the first postoperative serum creatinine and for each biomarker for the detection of severe AKI is shown in Figure 4. Postoperative serum creatinine had no discriminatory value for the prediction of severe AKI. No single biomarker was outstanding for classifying severe AKI-AUC (SE), for urine IL-18 was 0.72 (0.04), whereas the AUC for urine NGAL and plasma NGAL were 0.71 (0.04) and 0.56 (0.05), respectively. The sensitivities and specificities to detect severe AKI for the fifth quintile were 38 and 84%, respectively, for urine IL-18 (>364 pg/ml), 42 and 85% for urine NGAL (>72 ng/ml), and 27 and 81% for plasma NGAL (>261 ng/ml), respectively. There was no significant variation in biomarker performance for the prediction of severe AKI among the study sites (Supplementary Table 3).
Figure 4.
Diagnostic performance of the first post-operative value of urine IL-18, urine NGAL, Plasma NGAL and Serum Creatinine for the detection of severe AKI. Receiver-operating characteristic (ROC) curves show the diagnostic performance of the four biomarkers and the table shows the performance of each biomarker at the fifth quintile and the optimal cut point. Severe AKI is defined as the receipt of acute dialysis or the doubling of serum creatinine during hospitalization. Cut, cut point; Sens, sensitivity; Spec, specificity; LR+, positive likelihood ratio; LR-, negative likelihood ratio; PPV, positive predictive value; NPV, negative predictive value.
Risk Prediction
We used the net reclassification index (NRI) and integrated discrimination improvement (IDI) to determine whether biomarkers materially impacted risk prediction of severe AKI. The NRI was highly significant and of substantial magnitude for all of the three biomarkers: 0.22 (95% CI, 0.04 to 0.41; P = 0.02) for urine IL-18, 0.17 (95% CI, 0.03 to 0.32; P = 0.02) for urine NGAL, and 0.14 (95% CI, 0.02 to 0.26; P = 0.02) for plasma NGAL (Tables 3 and 4). Thirty-one percent of participants were reclassified with the addition of urine IL-18 to the clinical model (18% for urine NGAL; 20% for plasma NGAL). The main improvement in risk prediction after the addition of each biomarker to the clinical model was seen by correctly reclassifying patients in the intermediate risk category (Tables 3 and 4). IDI results were also significant for all three biomarkers.
Table 3.
Risk classification for severe AKI, based on clinical model and biomarkers
| First Postoperative Biomarker | NRI |
IDI |
||
|---|---|---|---|---|
| Value (SE) | P | Value (SE) | P | |
| Urine IL-18 (pg/ml) | 0.22 (0.09) | 0.02 | 0.05 (0.02) | 0.006 |
| Urine NGAL (ng/ml) | 0.17 (0.08) | 0.02 | 0.03 (0.01) | 0.04 |
| Plasma NGAL (ng/ml) | 0.14 (0.06) | 0.02 | 0.05 (0.02) | 0.002 |
Severe AKI is defined as the receipt of dialysis or a doubling of serum creatinine during hospitalization. Clinical model is: age (per year), gender, white race, non-elective surgery, CPB time ≥120 minutes, RACHS ≥3, eGFR percentile, and site. NRI and IDI quantify the improvement of the biomarkers on predicting the risk of severe AKI. Comparing the clinical model to the clinical model with the biomarker, NRI considers it an improvement in reclassification if an AKI patient moves up a risk category or if a non-AKI patient moves to a lower risk category. Similarly a worse reclassification occurs if an AKI patient moves down a risk category or if a non-AKI patient moves up a risk category. Overall, NRI is the difference in the proportion of improvements in reclassification and the proportion of worse reclassifications. The IDI formula quantifies the reclassification continuously instead of categorically.
Table 4.
Risk classification for first postoperative biomarkers after addition to clinical model for severe AKI
| Clinical Model, Frequency (Row Percent) | Clinical Model + Biomarker |
Total | ||
|---|---|---|---|---|
| <10% | 10 to 25% | >25% | ||
| Urine IL-18 | ||||
| AKI participants | ||||
| <10% | 2 (50.0%) | 2 (50.0%) | 0 (0%) | 4 |
| 10 to 25% | 2 (8.7%) | 7 (30.4%) | 14 (60.9%) | 23 |
| >25% | 0 (0%) | 4 (16.0%) | 21 (84.0%) | 25 |
| Total | 4 | 13 | 35 | 52 |
| Non-AKI participants | ||||
| <10% | 107 (88.4%) | 14 (11.6%) | 0 (0%) | 121 |
| 10 to 25% | 34 (34.3%) | 47 (47.5%) | 18 (18.2%) | 99 |
| >25% | 0 (0%) | 7 (21.2%) | 26 (78.8%) | 33 |
| Total | 141 | 68 | 44 | 253 |
| Urine NGAL | ||||
| AKI participants | ||||
| <10% | 2 (50.0%) | 2 (50.0%) | 0 (0%) | 4 |
| 10 to 25% | 0 (0%) | 14 (60.9%) | 9 (39.1%) | 23 |
| >25% | 0 (0%) | 3 (12.0%) | 22 (88%) | 25 |
| Total | 2 | 19 | 31 | 52 |
| Non-AKI participants | ||||
| <10% | 114 (94.2%) | 7 (5.8%) | 0 (0%) | 121 |
| 10 to 25% | 18 (18.2%) | 70 (70.7%) | 11 (11.1%) | 99 |
| >25% | 0 (0%) | 6 (18.2%) | 27 (81.8%) | 33 |
| Total | 132 | 83 | 38 | 253 |
| Plasma NGAL | ||||
| AKI participants | ||||
| <10% | 3 (100%) | 0 (0%) | 0 (0%) | 3 |
| 10 to 25% | 0 (0%) | 15 (68.2%) | 7 (31.8%) | 22 |
| >25% | 0 (0%) | 1 (4.2%) | 23 (95.8%) | 24 |
| Total | 3 | 16 | 30 | 49 |
| Non-AKI participants | ||||
| <10% | 102 (91.1%) | 10 (8.9%) | 0 (0%) | 112 |
| 10 to 25% | 23 (24.7%) | 59 (63.4%) | 11 (11.8%) | 93 |
| >25% | 0 (0%) | 5 (16.7%) | 25 (83.3%) | 30 |
| Total | 125 | 74 | 36 | 235 |
Clinical model is: age (per year), gender, Caucasian race, nonelective surgery, CPB time ≥120 min, RACHS ≥3, eGFR percentile, and site. Severe AKI is defined as the receipt of dialysis or a doubling of serum creatinine during hospitalization.
Supplementary Analysis
When we repeated the analyses using urine creatinine corrected values for urine IL-18 (pg/mg) and urine NGAL (ng/mg), there were no meaningful changes in AUCs compared with the uncorrected results (data not shown). We also repeated the analysis by excluding the 18 patients who developed AKI on the first day of study at the time of first sample collection. The point estimates of the fifth quintile and the AUC-ROCs were not changed for any of the three biomarkers in these analyses.
When we utilized a milder definition of AKI (Risk Injury Failure Loss End-Stage Kidney Disease [RIFLE] stage R or 50% increase in serum creatinine),18 the quintiles of urine biomarkers were independently associated with the incidence of AKI in a graded fashion (p for trend <0.001 for urine NGAL and urine IL-18 and P = 0.4 for plasma NGAL; Table 2). 131 (42%) patients had RIFLE R AKI. The fifth quintile of urine IL-18 and plasma NGAL had an adjusted odds ratio (OR) of 2.6 (95% CI, 1.0 to 6.7) and 2.3 (95% CI, 1.0 to 5.5), respectively, compared with the first quintile. The fifth quintile of urine NGAL had an adjusted OR of 2.1 (95% CI, 0.8 to 5.7) compared with the first quintile. The AUC-ROCs were lower, 0.64 for urine IL-18, 0.67 for urine NGAL, and 0.53 for plasma NGAL, to predict RIFLE R AKI, compared with severe AKI. Also, combinations of two or three biomarkers in the first 2 days only modestly improved the AUCs for severe AKI (highest AUC for two biomarkers, 0.73; and highest AUC for three biomarkers, 0.72) or for RIFLE R AKI (highest AUC for two biomarkers, 0.65; and highest AUC for three biomarkers, 0.67; Supplementary Figure 2).
Finally, we examined whether the discriminatory ability of biomarkers differed by the age of the patients. There were 158 (51%) patients who were less than 2 years old, and 37 (23%) had AKI. There were no effect modification by age categories and the relationship of biomarker quintiles with the AKI outcomes in multivariable analyses. For severe AKI, the AUC-ROCs were higher for patients older than 2 years for all of the three biomarkers, but these results were not statistically significant on ROC regression analyses (Supplementary Figure 2).
DISCUSSION
In this multicenter cohort study of biomarkers in children undergoing surgery for congenital cardiac lesions, we found that urinary biomarkers were associated with severe AKI, defined as postoperative doubling of serum creatinine or requirement of dialysis. A simple, noninvasive measurement of urine IL-18 or urine NGAL collected during the first 6 postoperative hours provided risk stratification for severe AKI. A postoperative level of urine IL-18 >124 pg/ml was associated with a five-fold increase in the odds of severe AKI, whereas a level >386 pg/ml increased the odds to seven-fold. A urine NGAL of >72 ng/ml distinguished a five-fold increased odds of AKI compared with patients in the lowest quintile. The biomarkers also substantially improved risk prediction for severe AKI as seen by measures of NRI and IDI. However, they did not perform well as diagnostic tests with AUCs in moderate range. The results of the biomarkers were significantly less robust for predicting mild AKI (RIFLE R). Urine NGAL was not independently associated with mild AKI, and the fifth quintile of IL-18 increased the risk prediction by only 2.6-fold. In contrast, plasma NGAL was not independently associated with severe AKI and had a weak association with mild AKI. The combination of biomarkers did not significantly improve the AUC for prediction of AKI. Additionally, we found that the first postoperative levels of urine IL-18, urine NGAL, and plasma NGAL demonstrated graduated relationships with important clinical outcomes, including length of stay in the ICU and mechanical ventilation.
This is the first multicenter, prospective study analyzing the use of biomarkers to diagnose postbypass-associated severe AKI in pediatric patients. Children are an ideal group for studying biomarkers compared with adults, because they are less likely to have comorbidities (e.g. diabetes, chronic kidney disease) that may influence biomarker performance. AKI occurs commonly after CPB in both adult and pediatric patients and is associated with prolonged ventilation and hospitalization and increased mortality.2,5 The incidence of severe AKI after pediatric cardiac surgery remains high, around 15 to 20%,19 and highlights the need for therapeutic advances. The identification of valid, early diagnostic biomarkers is fundamental to improving therapies and, ultimately, outcomes for cardiac surgery–associated AKI. This study demonstrates that urine IL-18 and NGAL are early biomarkers associated with severe AKI and that they provide additional prognostic information with regard to length of mechanical ventilation and length of stay.
We did note that biomarkers did not perform as well in this study as in previous single center reports.10 The reasons for this are likely multifactorial. Unlike previous studies, we excluded neonatal patients and had fewer patients with surgical Risk Adjustment for Congenital Heart Surgery 1 (RACHS-1) scores of 1. Additionally, because prior studies have been single center in nature, patient management, sample selection, sample storage, and testing are all more likely to be more homogeneous than would occur in a multicenter trial with several different centers representing variation in practices. Similarly, we note that these biomarkers were not predictive of less severe degrees of AKI. Although this could be related to insensitivities of the biomarkers, an alternative explanation is that more mild forms of AKI, on the basis of small elevations in serum creatinine, may have different causal mechanisms, which may affect biomarker expressions. Biomarkers may allow discrimination of true renal ischemia from other etiologies of creatinine elevation, such as prerenal azotemia, but may get classified as false positive when compared with the current gold standard of change in serum creatinine. The study of these “biomarker-negative, serum creatinine–positive” patients is important for complete understanding of these patients and the mechanisms of injury.
Interestingly, we found that plasma NGAL had minimal to no classification ability for mild or severe AKI, as had been previously reported.1,2 The reasons for this are not completely clear. When we analyzed our results, we found that our “non-AKI” patients had much larger increases in postoperative plasma NGAL than in prior studies; this resulted in smaller differences in plasma NGAL concentrations between the AKI and non-AKI groups. One potential explanation for this discrepancy relates to the assay used for plasma NGAL measurement. This study utilized an assay that has a lowest detection limit of 60 ng/ml, whereas previous studies have used assays with a much lower detection limit.2
Urine IL-18 and NGAL are up-regulated and released by the kidney tubules during injury and are biologic intermediates in the causal mechanisms of ischemia-reperfusion injury to the kidney.20,21 Both biomarkers are present at low concentrations preoperatively, and their levels increase severalfold in patients who develop kidney injury. IL-18, a mediator of inflammation, is produced by proximal tubules and is activated by caspase-1.21 The source of urine IL-18 elevation in AKI is at least in part from injured tubules. During ischemia-reperfusion injury in mice, the gene for NGAL is significantly up-regulated in the kidney, and the protein is overexpressed in distal tubule cells.20 As an iron-transporting protein, NGAL may play a primary role in renal tubular survival and recovery and has been used therapeutically in ischemia-reperfusion injury animal models.22 With respect to plasma NGAL, the kidney itself may not represent the primary source. Rather, experimental studies indicate that AKI results in a dramatic increase in NGAL mRNA and protein expression in distant organs such as the lungs and liver, which constitute a distinct plasma pool.23 Additional contributions to the systemic NGAL pool in AKI may derive from activated neutrophils, macrophages, and other immune cells.24 Although all of these mechanisms are pertinent to humans undergoing surgeries with cardiopulmonary bypass, these additional sources of plasma NGAL may also explain the lack of predictive ability of plasma NGAL for AKI in this study.
As in some prior single center studies,25,26 we found that younger patients (age, <2 years) had a higher risk of postoperative AKI. We hypothesized that this could be related to greater injury incurred by the developing kidney, more complex surgery, longer bypass times, or several unmeasured confounders such as subtle fluid management differences. Nonetheless, on further analysis, we found that younger (<2 years old) patients had similar RACHS-1 scores, CPB time, and baseline eGFR. Thus, our hypothesis that the higher risk of AKI in younger patients is related to increased complexity of surgery was not true and did not explain the difference. Rather, it appears that the association of age with AKI may be largely related to unmeasured confounders such as the response of the immature kidney to CPB or to differences in fluid management.
Our study has several strengths. We utilized a prospective cohort design and used a rigorous protocol to collect specimens, followed by blinded measurements. Second, we enrolled a large, relatively homogenous cohort of subjects in whom the most proximate etiology for AKI would be cardiopulmonary bypass. Third, patients were enrolled from three large academic pediatric medical centers with multiple surgeons and operative techniques, allowing for validation of the biomarkers in more than one clinical setting. Finally, the biomarker assays were performed using standardized clinical laboratory platforms.
This study does have limitations. First, the definition of AKI was on the basis of elevations in serum creatinine, which raises the conundrum of using a flawed outcome variable to analyze the performance of novel biomarkers.27 Evidence of injury on kidney biopsy would be an ideal gold standard but not feasible in our large cohort. Second, our cohort had a predominance of Caucasian participants; thus, the effect of race on these biomarkers could not be studied effectively. Third, there were not enough events to evaluate the association of biomarkers with the outcomes of dialysis and death. Lastly, because patient care was left to the discretion of the treating clinician, we did not control for or investigate the effects of differences in operative or postoperative management.
In summary, early postoperative levels of urine IL-18 and NGAL, measures of kidney injury, were associated with the development of severe AKI, dialysis, length of stay, and mechanical ventilation after pediatric cardiac surgery. These biomarkers improved clinical risk prediction, but their ability to classify AKI versus no AKI were marginal. Routine early measurement of urine IL-18 and NGAL could help determine the need for ICU resources, nephrology consultation and earlier use of renal replacement therapy in postcardiac surgery patients. Long-term outcomes in this population, such as chronic kidney disease and mortality, need to be studied to understand the importance of biomarker elevations when the serum creatinine does not rise.
CONCISE METHODS
Study Population
Between July 2007 and December 2009, we prospectively enrolled pediatric patients older than 1 month but younger than 18 years of age undergoing surgery for congenital cardiac lesions at three academic medical centers in North America. Patients reported here are the pediatric subset of an adult and pediatric consortium investigating new biomarkers in AKI known as Translational Research Investigating Biomarker Endpoints in Acute Kidney Injury (TRIBE-AKI; www.yale.edu/tribeaki). Our eligibility criteria were broad and included all pediatric patients ages 1 month to 18 years undergoing cardiopulmonary bypass at each of the three centers. Exclusion criteria included history of prior renal transplantation or dialysis requirement. Participants could only be enrolled in the study once. Written informed consent was obtained from all parents or legal guardians, along with assent where appropriate. The study was approved by each institution's research ethics board. The reporting of this study follows guidelines set out in the Strengthening the Reporting of Observational Studies in Epidemiology statement (Supplementary Table 1).
Sample Collection
We collected urine and blood specimens preoperatively and daily for 3 postoperative days. The first postoperative urine and blood samples were collected soon after admission to the intensive care unit. For the first 24 hours postoperatively, urine samples were collected every 6 hours. The remaining daily blood and urine samples were obtained at the time of routine morning blood collection done for clinical care. We obtained fresh urine samples from the urimeter of the Foley catheter system or using cotton balls in patients wearing diapers. We centrifuged the samples at 3,000 × g for 10 minutes to remove cellular debris. Blood was collected in EDTA tubes and was also centrifuged to separate plasma. We aliquoted urine supernatant and plasma into bar-coded cryovials and stored the samples at −80°C until biomarker measurement. No additives or protease inhibitors were added.
AKI Biomarker Measurements
We measured the two urine biomarkers with the ARCHITECT® assay (Abbott Diagnostics, Abbott Park, IL). We measured urine creatinine by a modified Jaffe reaction. The coefficient of variation (CV) for the urine creatinine assay was 5%, whereas the CV for the NGAL and IL-18 assays were approximately 5 and 8%, respectively. We performed the measurements in two batches, about 1 year apart.
We measured plasma NGAL using the Triage NGAL immunoassay, in conjunction with the Triage Meter (Biosite Inc., San Diego, CA) in two batches 7 months apart. The Triage assay has a detection range of 60–1300 ng/ml with a CV of 10 to 15%. Personnel performing the biomarker measurements were blinded to each patient's clinical information. All of the biomarkers were measured from frozen aliquots that did not undergo any additional freeze-thaw cycles. Repeat measurement of randomly selected samples between the batches confirmed high correlation without any assay drift for all of the assays measured in two batches.
Outcome Definitions
The primary outcome was the development of severe AKI, defined by receipt of acute dialysis during the entire hospital stay or a doubling in serum creatinine from the baseline preoperative value (consistent with RIFLE18 stage I or the Acute Kidney Injury Network (AKIN)28 stage 2 AKI). All of the preoperative creatinine values were measured within 2 months before surgery. Pre- and postoperative serum creatinine levels were measured in the same clinical laboratory for each patient at all sites. Additional clinical outcomes were AKI defined by a ≥50% serum creatinine rise from baseline values (RIFLE stage R), in-hospital mortality, length of in-hospital and intensive care unit stay, and length of mechanical ventilation.
Variable Definitions
We collected preoperative characteristics, operative details, and postoperative complications using the definitions of the Society of Thoracic Surgeons (http://www.sts.org/documents/pdf/CongenitalDataSpecificationsV3_0_20090904.pdf). We utilized the RACHS-1 consensus-based scoring system to categorize the complexity of surgery.29 This method of risk stratification is a widely accepted tool for the evaluation of differences in outcomes of surgery for congenital heart disease. We estimated preoperative GFR (eGFR) using the updated Schwartz equation30 and determined eGFR percentiles on the basis of published normal renal function data in 651 children.31
Statistical Analysis
Continuous variables were compared with two-sample t test or Wilcoxon rank sum test and dichotomous variables with the chi-squared test or Fisher's exact test. To evaluate the association of each biomarker with AKI, we divided the cohort into quintiles on the basis of the first postoperative value of urine IL-18, urine NGAL, and plasma NGAL. Unadjusted linear trends between the biomarkers and AKI were assessed with the Cochran-Armitage test for dichotomous outcomes and the Jonckheere-Terpstra test for continuous outcomes and for adjusted linear trends using contrasts in logistic or linear regression, respectively. Mixed logistic regression models with random intercepts for each center were used to determine the adjusted odds ratios. We used two models adjusting for important covariates that predict AKI in the pediatric cardiac surgery setting. The first model only included patient demographics (age, gender, race, and site), and the second included patient demographics, a clinical risk factor (baseline eGFR percentiles) and operative characteristics (complexity of surgery, use, and duration of cardiopulmonary bypass). An interaction term between age (≤2 and >2) and biomarker quintile was used to determine whether the association between biomarker and severe AKI differed with the age of the patient. Quantile regression models (proc quantreg; SAS statistical software, SAS Institute Inc., Cary, NC) using intercept and log-transformed age for children younger and older than 2 years were fitted on the basis of normal pediatric renal function data for calculating pediatric eGFR percentiles. We quantified the discriminatory ability of the biomarkers with the AUC-ROC and used ROC regression to determine whether the discriminatory ability differed by age (≤2 years old versus >2 years old).32 We used logistic regression models with three-fold cross-validation to estimate biomarker combinations using a maximum of three biomarkers per model.33 The discriminatory ability of the biomarker combinations was estimated with the AUC-ROC in the validation data set.34 Urinary biomarkers were not corrected for urine creatinine in primary analyses but were corrected for creatinine in sensitivity analyses. In addition, we evaluated each biomarker's utility to predict mild AKI defined by a 50% rise in serum creatinine from baseline (RIFLE R).
To evaluate the effect of biomarkers on AKI risk prediction, we determined the NRI and the IDI.35,36 Comparing the clinical model to the biomarker and the clinical model, an improvement in reclassification is defined as the movement of an AKI patient up a risk category or of a non-AKI patient to a lower risk category. Similarly a worse reclassification occurs if an AKI patient moves down a risk category or if a non-AKI patient moves up a risk category. NRI is the difference in the proportion of improvements in reclassification and the proportion of worse reclassifications (NRI = (pupAKI − pdownAKI) − (pupnon-AKI − pdownnon-AKI), where pupAKI is the number of AKI patients moving up divided by the number of AKI patients, pdownAKI is the number of AKI patients moving down divided by the number of AKI patients, pupnon-AKI is the number of non-AKI patients moving up divided by the number of non-AKI patients, and pdownnon-AKI is the number of non-AKI patients moving down divided by the number of non-AKI patients). For the NRI analysis of the prediction of severe AKI, risk categories were defined as low (<10%), medium (10 to 25%), or high (>25%). The IDI formula quantifies the reclassification continuously instead of categorically. We performed the analyses in SAS version 9.2 (SAS Institute) and R 2.12.0 (R Foundation for Statistical Computing, Vienna, Austria).
Disclosures
None.
Acknowledgments
Drs. Parikh, Sint, and Thiessen-Philbrook had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Drs. Parikh, Devarajan, and Garg were responsible for the study concept and design. Drs. Parikh, Coca, Devarajan, Shlipak, Thiessen-Philbrook, Kim, Li, Koyner, and Garg analyzed and interpreted the data. Drs. Parikh and Krawczeski drafted the manuscript. Drs. Parikh, Coca, Devarajan, Shlipak, Thiessen-Philbrook, Edelstein, Sint, Zappitelli, Koyner, Krawczeski, and Garg critically revised the manuscript for important intellectual content. Statistical analysis was by Drs. Thiessen-Philbrook and Sint. Dr. Parikh obtained funding and supervised the study.
Dr. Edelstein and Dr. Parikh are named co-inventors on the IL-18 patent. Dr. Devarajan is the co-inventor on the NGAL patents. Dr. Parikh, Dr. Coca, and Dr. Devarajan are consultants to Abbott Diagnostics. Dr. Devarajan is a consultant to Biosite, Inc.
The research reported in this article was supported by the American Heart Association Clinical Development Award, Grant R01HL-085757 from the National Heart, Lung, and Blood Institute. The study was also supported by CTSA Grant Number UL1 RR024139 from the National Center for Research Resources. This work is Clinical Trials.gov number NCT00774137. The urine biomarker assays were donated by Abbott Diagnostics. Plasma NGAL was donated by Biosite, Inc.
The TRIBE-AKI collaborators were Uptal Patel, Madhav Swaminathan, Cary Passik, Sue Garwood, Qing Ma, and Michael Bennett.
Footnotes
Published online ahead of print. Publication date available at www.jasn.org.
See related editorial, “Blind Men and Elephants and the Biological Markers of AKI,” on pages 1578–1580.
Supplemental information for this article is available online at http://www.jasn.org/.
REFERENCES
- 1. Dent CL, Ma Q, Dastrala S, Bennett M, Mitsnefes MM, Barasch J, Devarajan P: Plasma neutrophil gelatinase-associated lipocalin predicts acute kidney injury, morbidity and mortality after pediatric cardiac surgery: A prospective uncontrolled cohort study. Crit Care 11: R127, 2007 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Mishra J, Dent C, Tarabishi R, Mitsnefes MM, Ma Q, Kelly C, Ruff SM, Zahedi K, Shao M, Bean J, Mori K, Barasch J, Devarajan P: Neutrophil gelatinase-associated lipocalin (NGAL) as a biomarker for acute renal injury after cardiac surgery. Lancet 365: 1231–1238, 2005 [DOI] [PubMed] [Google Scholar]
- 3. Pedersen KR, Hjortdal VE, Christensen S, Pedersen J, Hjortholm K, Larsen SH, Povlsen JV: Clinical outcome in children with acute renal failure treated with peritoneal dialysis after surgery for congenital heart disease. Kidney Int Suppl 108: S81–S86, 2008 [DOI] [PubMed] [Google Scholar]
- 4. Sorof JM, Stromberg D, Brewer ED, Feltes TF, Fraser CD, Jr: Early initiation of peritoneal dialysis after surgical repair of congenital heart disease. Pediatr Nephrol 13: 641–645, 1999 [DOI] [PubMed] [Google Scholar]
- 5. Zappitelli M, Bernier PL, Saczkowski RS, Tchervenkov CI, Gottesman R, Dancea A, Hyder A, Alkandari O: A small post-operative rise in serum creatinine predicts acute kidney injury in children undergoing cardiac surgery. Kidney Int 76: 885–892, 2009 [DOI] [PubMed] [Google Scholar]
- 6. Park M, Coca SG, Nigwekar SU, Garg AX, Garwood S, Parikh CR: Prevention and treatment of acute kidney injury in patients undergoing cardiac surgery: A systematic review. Am J Nephrol 31: 408–418, 2010 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Molitoris BA: Transitioning to therapy in ischemic acute renal failure. J Am Soc Nephrol 14: 265–267, 2003 [DOI] [PubMed] [Google Scholar]
- 8. Siegel NJ, Shah SV: Acute renal failure: Directions for the next decade. J Am Soc Nephrol 14: 2176–2177, 2003 [DOI] [PubMed] [Google Scholar]
- 9. American Society of Nephrology Renal Research Report. J Am Soc Nephrol 16: 1886–1903, 2005 [DOI] [PubMed] [Google Scholar]
- 10. Bennett M, Dent CL, Ma Q, Dastrala S, Grenier F, Workman R, Syed H, Ali S, Barasch J, Devarajan P: Urine NGAL predicts severity of acute kidney injury after cardiac surgery: A prospective study. Clin J Am Soc Nephrol 3: 665–673, 2008 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Ling W, Zhaohui N, Ben H, Leyi G, Jianping L, Huili D, Jiaqi Q: Urinary IL-18 and NGAL as early predictive biomarkers in contrast-induced nephropathy after coronary angiography. Nephron Clin Pract 108: c176–c181, 2008 [DOI] [PubMed] [Google Scholar]
- 12. Parikh CR, Mishra J, Thiessen-Philbrook H, Dursun B, Ma Q, Kelly C, Dent C, Devarajan P, Edelstein CL: Urinary IL-18 is an early predictive biomarker of acute kidney injury after cardiac surgery. Kidney Int 70: 199–203, 2006 [DOI] [PubMed] [Google Scholar]
- 13. Melnikov VY, Ecder T, Fantuzzi G, Siegmund B, Lucia MS, Dinarello CA, Schrier RW, Edelstein CL: Impaired IL-18 processing protects caspase-1-deficient mice from ischemic acute renal failure. J Clin Invest 107: 1145–1152, 2001 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Wagener G, Gubitosa G, Wang S, Borregaard N, Kim M, Lee HT: Urinary neutrophil gelatinase-associated lipocalin and acute kidney injury after cardiac surgery. Am J Kidney Dis 52: 425–433, 2008 [DOI] [PubMed] [Google Scholar]
- 15. Wagener G, Jan M, Kim M, Mori K, Barasch JM, Sladen RN, Lee HT: Association between increases in urinary neutrophil gelatinase-associated lipocalin and acute renal dysfunction after adult cardiac surgery. Anesthesiology 105: 485–491, 2006 [DOI] [PubMed] [Google Scholar]
- 16. Haase M, Bellomo R, Devarajan P, Schlattmann P, Haase-Fielitz A: Accuracy of neutrophil gelatinase-associated lipocalin (NGAL) in diagnosis and prognosis in acute kidney injury: A systematic review and meta-analysis. Am J Kidney Dis 54: 1012–1024, 2009 [DOI] [PubMed] [Google Scholar]
- 17. Wheeler DS, Devarajan P, Ma Q, Harmon K, Monaco M, Cvijanovich N, Wong HR: Serum neutrophil gelatinase-associated lipocalin (NGAL) as a marker of acute kidney injury in critically ill children with septic shock. Crit Care Med 36: 1297–1303, 2008 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18. Bellomo R, Ronco C, Kellum JA, Mehta RL, Palevsky P, ADQI workgroup Acute renal failure: Definition, outcome measures, animal models, fluid therapy and information technology needs: The Second International Consensus Conference of the Acute Dialysis Quality Initiative (ADQI) Group. Crit Care 8: R204–R212, 2004 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19. Zappitelli M: Epidemiology and diagnosis of acute kidney injury. Semin Nephrol 28: 436–446, 2008 [DOI] [PubMed] [Google Scholar]
- 20. Mishra J, Ma Q, Prada A, Mitsnefes M, Zahedi K, Yang J, Barasch J, Devarajan P: Identification of neutrophil gelatinase-associated lipocalin as a novel early urinary biomarker for ischemic renal injury. J Am Soc Nephrol 14: 2534–2543, 2003 [DOI] [PubMed] [Google Scholar]
- 21. Melnikov VY, Faubel S, Siegmund B, Lucia MS, Ljubanovic D, Edelstein CL: Neutrophil-independent mechanisms of caspase-1- and IL-18-mediated ischemic acute tubular necrosis in mice. J Clin Invest 110: 1083–1091, 2002 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22. Mishra J, Mori K, Ma Q, Kelly C, Yang J, Mitsnefes M, Barasch J, Devarajan P: Amelioration of ischemic acute renal injury by neutrophil gelatinase-associated lipocalin. J Am Soc Nephrol 15: 3073–3082, 2004 [DOI] [PubMed] [Google Scholar]
- 23. Xu S, Venge P: Lipocalins as biochemical markers of disease. Biochim Biophys Acta 1482: 298–307, 2000 [DOI] [PubMed] [Google Scholar]
- 24. Cowland JB, Borregaard N: Molecular characterization and pattern of tissue expression of the gene for neutrophil gelatinase-associated lipocalin from humans. Genomics 45: 17–23, 1997 [DOI] [PubMed] [Google Scholar]
- 25. Krawczeski CD, Vandevoorde RG, Kathman T, Bennett MR, Woo JG, Wang Y, Griffiths RE, Devarajan P: Serum cystatin C is an early predictive biomarker of acute kidney injury after pediatric cardiopulmonary bypass. Clin J Am Soc Nephrol 5: 1552–1557, 2010 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26. Krawczeski CD, Woo JG, Wang Y, Bennett MR, Ma Q, Devarajan P. Neutrophil gelatinase-associated lipocalin concentrations predict development of acute kidney injury in neonates and children after cardiopulmonary bypass. J Pediatr 158: 1009–1015, 2011 [DOI] [PubMed] [Google Scholar]
- 27. Haase-Fielitz A, Bellomo R, Devarajan P, Bennett M, Story D, Matalanis G, Frei U, Dragun D, Haase M: The predictive performance of plasma neutrophil gelatinase-associated lipocalin (NGAL) increases with grade of acute kidney injury. Nephrol Dial Transplant 24: 3349–3354, 2009 [DOI] [PubMed] [Google Scholar]
- 28. Mehta RL, Kellum JA, Shah SV, Molitoris BA, Ronco C, Warnock DG, Levin A: Acute Kidney Injury Network: Report of an initiative to improve outcomes in acute kidney injury. Crit Care 11: R31, 2007 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29. Jenkins KJ, Gauvreau K, Newburger JW, Spray TL, Moller JH, Iezzoni LI: Consensus-based method for risk adjustment for surgery for congenital heart disease. J Thorac Cardiovasc Surg 123: 110–118, 2002 [DOI] [PubMed] [Google Scholar]
- 30. Schwartz GJ, Muñoz A, Schneider MF, Mak RH, Kaskel F, Warady BA, Furth SL: New equations to estimate GFR in children with CKD. J Am Soc Nephrol 20: 629–637, 2009 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31. Piepsz A, Tondeur M, Ham H: Revisiting normal (51)Cr-ethylenediaminetetraacetic acid clearance values in children. Eur J Nucl Med Mol Imaging 33: 1477–1482, 2006 [DOI] [PubMed] [Google Scholar]
- 32. Janes H, Pepe MS: Adjusting for covariates in studies of diagnostic, screening, or prognostic markers: An old concept in a new setting. Am J Epidemiol 168: 89–97, 2008 [DOI] [PubMed] [Google Scholar]
- 33. Hastie T, Tibshirani R, Friedman J: The elements of statistical learning: Data mining, inference, and prediction, New York: Springer-Verlag, 2007 [Google Scholar]
- 34. Baker SG, Kramer BS, McIntosh M, Patterson BH, Shyr Y, Skates S: Evaluating markers for the early detection of cancer: Overview of study designs and methods. Clin Trials 3: 43–56, 2006 [DOI] [PubMed] [Google Scholar]
- 35. Cook NR: Statistical evaluation of prognostic versus diagnostic models: Beyond the ROC curve. Clin Chem 54: 17–23, 2008 [DOI] [PubMed] [Google Scholar]
- 36. Pencina MJ, D'Agostino RB, Sr, D'Agostino RB, Jr, Vasan RS: Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond. Stat Med 27: 157–172, 207–212, 2008 [DOI] [PubMed] [Google Scholar]