Summary
Background and objectives
The purpose of this study was to determine the accuracy of plasma neutrophil gelatinase-associated lipocalin as a marker of AKI in patients admitted from the emergency department.
Design, setting, participants, & measurements
In this prospective cohort study, patients (n=616) admitted from the emergency department from March to November of 2008 were classified according to clinical criteria as AKI, transient azotemia, stable CKD, and normal function. Plasma neutrophil gelatinase-associated lipocalin was measured serially. A logistic regression model using clinical characteristics was fitted to the data, and a second model included discretized plasma neutrophil gelatinase-associated lipocalin. Performance of the models was evaluated by Hosmer–Lemeshow goodness-of-fit test, area under the receiver operating characteristic curve, net reclassification improvement, integrated discrimination improvement, and predictiveness curve.
Results
Twenty-one percent of patients were classified as AKI; the highest median levels of plasma neutrophil gelatinase-associated lipocalin were in the AKI group (146–174 ng/ml at various time points) and increased with AKI severity (207–244 ng/ml for Acute Kidney Injury Network classification stage>2). The discriminative ability of plasma neutrophil gelatinase-associated lipocalin for AKI diagnosis (area under the curve, 0.77–0.82 at various time points) improved with higher grades of severity (area under the curve, 0.85–0.89 for AKIN>2). Plasma neutrophil gelatinase-associated lipocalin discriminated AKI from normal function and transient azotemia (area under the curve, 0.85 and 0.73, respectively). Patients were classified into three grades of AKI risk according to plasma neutrophil gelatinase-associated lipocalin levels (low, moderate [i.e., the gray zone], and high). Patients with plasma neutrophil gelatinase-associated lipocalin in the high-risk category displayed a 10-fold greater risk of AKI (odds ratio, 9.8; 95% confidence interval, 5.6 to 16.9). The addition of plasma neutrophil gelatinase-associated lipocalin to the clinical model yielded a net reclassification improvement of 94.3% and an integrated discrimination improvement of 0.122.
Conclusion
Plasma neutrophil gelatinase-associated lipocalin is an accurate biomarker for prediction of AKI in patients admitted from the emergency department. This work proposes a three-grade classification of AKI risk based on plasma neutrophil gelatinase-associated lipocalin levels.
Introduction
The incidence of AKI has been rising in recent years (1), with negative influence on overall outcomes (2–5). Community-acquired AKI is common at admission to the emergency department (ED) (4–6), and the diagnosis still largely relies on serial serum creatinine (SCr) measurements, a delayed and nonspecific marker (7–9). The identification of new AKI biomarkers that differentiate intrinsic AKI from transient reversible forms of renal dysfunction and predict outcomes is recognized as a high priority.
Plasma neutrophil gelatinase-associated lipocalin (pNGAL) has been shown to be of diagnostic and prognostic value for AKI in different settings (10–19). However, there is no consensus regarding a cutoff value above which an AKI diagnosis can confidently be made.
The primary objective of our study was to determine the accuracy of pNGAL as a marker of AKI in patients admitted from the ED. Secondary objectives were to develop a three-grade classification of AKI risk based on pNGAL values and assess the improvement in AKI risk prediction provided by pNGAL.
Materials and Methods
Patient Population
We took advantage of a previous prospective cohort study of 616 patients who presented to the ED of the Fernando Fonseca Hospital in Portugal from March to November of 2008 (20). Patients were enrolled consecutively after signing the informed consent. Exclusion criteria were age under 18 or over 80 years of age, complete anuria, established CKD stage 4 or greater (21), urinary obstruction, AKI with Acute Kidney Injury Network classification (AKIN) stage 3 at admission (defined as an increase in SCr>300% over baseline or >4 mg/dl for more than 48 hours or needing dialysis therapy), cytotoxic therapy, and predicted admission time less than 48 hours. The study protocol and consent forms were approved by the institutional ethics committee.
Biomarker Measurements
Baseline renal function by SCr, medical history, and demographic characteristics were obtained from hospital electronic records for 1–6 months before ED presentation. A prospective renal function assessment was carried out by measuring SCr, serum Cystatin C (SCysC), and pNGAL at 0, 6, 12, 24, and 48 hours (T0, T6, T12, T24, and T48, respectively) from admission (Figure 1). Plasma samples for NGAL measurement were collected in EDTA tubes and centrifuged at 2000 × g for 5 minutes, and the supernatant was quickly frozen at −80°C. The pNGAL measurement was performed using the Triage NGAL Device in a double-blinded manner as previously described (10). The CysC measurement was performed as previously reported (20).
Figure 1.
Study flow diagram. Serum creatinine (SCr) and plasma neutrophil gelatinase-associated lipocalin (pNGAL) were measured at the indicated times (Ts) until discharge (D). At the end of the study, patients were adjudicated as having AKI, transient azotemia (TAz), stable CKD, or normal renal function (NF). ED, emergency department.
Clinical Adjudication
GFR was estimated using the Modification of Diet in Renal Disease Equation (21). Patients were adjudicated to one of four diagnostic categories: normal kidney function (NF), stable CKD (sCKD), transient azotemia (TAz), or AKI (22). NF was defined as a baseline estimated GFR greater than 60 ml/min per 1.73 m2 and no increases in SCr during the hospital stay. sCKD was defined as a stably reduced estimated GFR<60 ml/min per 1.73 m2 before admission and <25% change from baseline during the hospitalization (23). TAz was defined as a new-onset increase in SCr that satisfied any grade of the Risk, Assessment, Failure, Loss, and End Stage Renal Disease (RIFLE) classification (24) but was not sustained and resolved in less than 3 days of follow-up. AKI was defined according to the RIFLE and AKIN criteria (25), requiring a new onset of at least 1.5-fold increase or ≥0.3-mg increment of SCr values from baseline that did not resolve after 3 days of follow-up. The variables in the “Multidimensional Criteria Definition” (26) were recorded to determine the kidney susceptibility stage (Supplemental Table 1). All adjudications were independently verified by three nephrologists who were blind to the study. Only in 2.4% (n=15) of the cohort was a baseline SCr not available. In these subjects, baseline renal function was calculated using previously published formulas. None of the patients without baseline SCr were classified as having AKI.
Statistical Analyses
Categorical data were presented as frequencies and percentages, and continuous variables were presented as mean or median, SD, or interquartile range (25th percentile to 75th percentile) as appropriate. Univariable analysis was done using nonparametric tests (chi-squared, Mann–Whitney U, and Kruskal–Wallis) for outliers, high variability, or skewed distributions. The 95% confidence intervals (95% CIs) were also calculated whenever appropriate.
To assess the performance of pNGAL as a diagnostic test, the cutoff points of 100 and 150 ng/ml were used (12). Sensitivity, specificity, and positive and negative predictive values were calculated as well as the likelihood ratios. For the multivariate analysis, those variables that were associated with increased risk of AKI in the univariate study (P<0.10) were included (age, kidney susceptibility stage, chronic heart failure, hypertension, cardiovascular disease, and diabetes mellitus). The logistic regression model was fitted to the data to determine the influence of pNGAL alone and also adjusted by those clinical variables that were shown to have influence on the risk of AKI. The ability of these models to predict AKI was analyzed by the Hosmer–Lemeshow goodness-of-fit test (27), and their discriminative ability was determined by the area under the receiver operating characteristic (ROC) curve (28). Area under the ROC curve (AUC) values were compared as previously described (29).
A predictiveness curve was also calculated (30). This curve is a graphical representation of the distribution of risk defined by the following conditional probability: risk (y)=P[D=1|Y=y], where D denotes the binary outcome (in our case, AKI) and Y=y is a marker value (in our case, a pNGAL value). The interpretation of this curve is straightforward: a marker that is useless assigns equal risk to all individuals, and hence, the corresponding predictiveness curve is a horizontal line at the prevalence of the disease; however, a marker that is highly informative about risk yields a predictive curve that is close to a step function.
To categorize pNGAL levels, several approaches were applied. We used the minimum P value approach, which provides the point from a grid of marker values that is associated with the minimum chi-squared P value. The resulting plot suggested a possible second cutoff point. This suggestion justified a second analysis to determine two cutoff points that discriminate pNGAL values, thus defining a gray zone. One of the methods used to construct a three-zone partition is based on likelihood ratios (31). However, in our case, the resulting zone was too wide, and therefore, a second approach based on generalized additive models was used (32). A graphical representation of the influence of pNGAL on AKI diagnosis was obtained; 95% CIs were also calculated and plotted. The zone where the 95% CIs contain the zero value was considered to be the gray zone; negative values of partial function denote low risk of AKI, and positive values identify high risk of AKI.
To determine the added value of a biomarker to preexisting risk prediction models, we determined the integrated discrimination improvement (IDI) and net reclassification improvement (NRI) (33). The NRI and IDI each consider individuals who develop and do not develop AKI separately, providing additional information not available from the AUC. The NRI quantifies the correctness of upward and downward reclassification of predicted probabilities as a result of adding a new marker to an existing model. The IDI is a measure of the improvement in prediction. The continuous or category-free NRI (cfNRI) has recently been added into these metrics for improving some of the shortcomings of the previous NRI (34). The NRI may attain a maximum value of 200% in the case where a correct reclassification occurs for all the events (100%) and all nonevents (100%).
The significance level α=5% was considered. All data were analyzed using Statistical Package for the Social Sciences for Windows 15.0 (SPSS, Inc.), R software (35), and Intercooled Stata 9.2 for Windows (StataCorp LP).
Results
Subject Characteristics
From March to December of 2008, 4742 patients were admitted from the ED into medical wards. A total of 800 consecutive patients had inclusion criteria and were enrolled. The SCr of all excluded patients is shown in Supplemental Table 2. Of the enrolled patients, 172 patients were excluded, because follow-up samples were missing; 12 patients declined to continue with their participation. In total, 616 patients consented and completed the protocol. Of those patients, 130 patients fulfilled AKI criteria (21.1%), 159 (25.8%) patients had TAz, 15 (2.4%) patients had sCKD, and 312 (50.7%) patients maintained normal kidney function (Table 1). Of note, 43 (33%) AKI patients had preexisting CKD. By multidimensional criteria (26), 67.2% of patients were in stage I, 21.4% of patients were in stage II, 4.9% of patients were in stage III, and 6.5% of patients were in stage IV (Supplemental Table 3). Discharge diagnoses of the entire cohort are shown in Supplemental Table 4.
Table 1.
Patient characteristics
| Characteristic | All Patients (616) | AKI 21.1% (130) | TAz 25.8% (159) | sCKD 2.4% (15) | NF 50.7% (312) | P Value |
|---|---|---|---|---|---|---|
| Mean age, yr (SD) | 59.1 (15.8) | 66.3 (12.2) | 58.4 (16.1) | 68.5 (11.5) | 56.0 (15.7) | <0.001a,b |
| Men (%) | 386 (62.7) | 84 (64.6) | 93 (58.5) | 10 (66.7) | 199 (63.8) | 0.65c,d |
| Nonblack (%) | 536 (87.0) | 114 (87.7) | 137 (86.2) | 15 (100.0) | 270 (86.5) | 0.56c,d |
| SCr baseline (mg/dl) | ||||||
| Median (P25–P75) | 0.8 (0.6–0.9) | 1.0 (0.7–1.2) | 0.7 (0.6–0.9) | 1.2 (1.1–1.9) | 0.7 (0.6–0.8) | <0.001b,e |
| Baseline MDRD SCr-eGFR (ml/min per 1.73 m2) | ||||||
| Median (P25–P75) | 103.68 (80.23–128.03) | 77.76 (56.33–111.82) | 104.03 (85.50–133.58) | 52.31 (45.72–63.72) | 109.34 (91.95–130.13) | <0.001b |
| Principle comorbidities (%) | ||||||
| Hypertension | 391 (63.6) | 101 (77.7) | 98 (62.0) | 12 (80.0) | 180 (57.7) | <0.001d |
| Cardiovascular disease | 219 (35.6) | 71 (54.6) | 61 (38.4) | 8 (53.3) | 79 (25.3) | <0.001d |
| Diabetes mellitus | 192 (31.3) | 55 (42.3) | 46 (29.1) | 8 (53.3) | 83 (26.8) | 0.003d |
| Chronic heart failure | 81 (13.1) | 39 (30) | 21 (13.2) | 4 (26.7) | 17 (5.4) | <0.001d |
| Chronic liver disease | 50 (8.1) | 13 (10.0) | 11 (7.0) | 0 (0) | 26 (8.4) | 0.52d |
| CKD previous diagnosis | 65 (10.6) | 43 (33.1) | 7 (4.4) | 15 (100.0) | 0 (0) | <0.001d |
| pNGAL median (ng/ml; P25–P75) | ||||||
| T0 | 84.0 (60–137) | 174.5 (94–265) | 86.0(65–140) | 137.0 (61–200) | 68.0 (60–96)f | <0.001b |
| T6 | 84.5 (60–136) | 171.5(104–277) | 85.5 (60–131) | 156.0 (77–205) | 69.0 (60–98)f | <0.001b |
| T12 | 83.0 (60–138) | 167.0 (109–253) | 94.0 (60–136) | 115.0 (78–167) | 64.0 (60–93)f | <0.001b |
| T24 | 83.0 (60–136) | 161.0 (100–249) | 90.5 (60–122) | 117.0 (87–194) | 62.5 (60–94)f | <0.001b |
| T48 | 79.0 (60–121) | 146.0 (82–234) | 79.0 (60–112) | 102.5 (68–186) | 65.00 (60–94)f | <0.001b |
| Outcome | ||||||
| All sepsis (%) | 188 (30.5) | 50 (38.5) | 60 (37.7) | 3 (20) | 75 (24) | 0.002d |
| Severe sepsis (%) | 27 (4.4) | 16 (2.3) | 7 (4.4) | 0 | 4 (1.3) | <0.001d |
| Intensive care unit admission (%) | 53 (8.6) | 18 (13.8) | 11 (6.9) | 1 (6.7) | 23 (7.4) | 0.12d |
| Mechanical ventilation (%) | 22 (3.6) | 9 (6.9) | 8 (5) | 0 | 5 (1.5) | 0.03d |
| In-hospital length of stay (SD) | 11.3 (10) | 16.9 (15.5) | 10.5 (8.4) | 8.5 (3.8) | 9.5 (6.9) | <0.001b |
| Dialysis (%) | 6 (1.0) | 6 (4.6) | 0 | 0 | 0 | — |
| Mortality (%) | 27 (4.4) | 15 (11.5) | 5 (3.1) | 0 | 7 (2.2) | <0.001d |
| AKI severity (%) | ||||||
| AKIN 0 | 486 (78.9) | 0 | 159 (100) | 15 (100) | 312 (100) | |
| AKIN 1 | 100 (16.2) | 100 (76.9) | — | — | — | |
| AKIN 2 | 16 (2.6) | 16 (12.3) | — | — | — | |
| AKIN 3 | 14 (2.3) | 14 (10.8) | — | — | — |
TAz, transient azotemia; sCKD, stable CKD; NF, normal function; SCr, serum creatinine; P25–P75, 25th percentile to 75th percentile; MDRD SCr-eGFR, GFR estimated from SCr using the Modification of Diet in Renal Disease equation; pNGAL, plasma neutrophil gelatinase-associated lipocalin; T, study time in hours; AKIN, Acute Kidney Injury Network.
For age, P<0.001 comparing AKI with TAz and NF patients; for age, P=0.50 comparing AKI with CKD.
Kruskal–Wallis test P values.
Comparing AKI with CKD patients.
Chi-squared test P values.
SCr baseline P<0.001, except between TAz and NF patients (P value=0.80).
All pNGAL P25 and P75 values of NF patients were between 60 and 97 ng/ml, respectively (a range proposed as normal in this study).
pNGAL for Differentiating AKI Patients
The highest levels of pNGAL were seen in the AKI group (median values=146–175 ng/ml at various time points), and they were significantly different from patients with NF (P<0.001 for all time points) (Figure 2A). When the combined group of AKI + TAz was examined, the values of pNGAL remained significantly different from NF (P<0.001 for all time points) (Figure 2B). Furthermore, pNGAL was able to differentiate AKI from TAz (P<0.001 for all time points) (Figure 2C and Table 1).
Figure 2.
Plasma neutrophil gelatinase-associated lipocalin (pNGAL) for AKI discrimination and diagnostic accuracy. A shows the distribution of pNGAL values by diagnostic group (i.e., AKI, transient azotemia [TAz], stable CKD [sCKD], or normal renal function [NF]). Levels of pNGAL discriminated AKI from TAz and NF (P<0.001 for all study times). All values of pNGAL are means and 95% confidence intervals (95% CIs) displayed at different time points (Ts) of the study. B compares pNGAL values in NF with values in the combined group of AKI plus TAz (P<0.001 for all study times). C shows the relative differences, through time, of pNGAL and SCr in AKI patients. Statistically significant differences for pNGAL were found (P=0.01), whereas no differences were detected for SCr (P=0.16). Dif indicates the percentage of change observed from each time point to the previous one: Dif1, (T6−T0)/T0; Dif2, (T12−T6)/T6; Dif3, (T24−T12)/T12; Dif4, (T48−T24)/T24. At 12 hours, pNGAL reached the highest level and the highest percentage of increase from 6 hours.
The median levels of pNGAL in sCKD patients (102.5–156 ng/ml) were not significantly different from AKI, except at the 12-hour time point, when pNGAL levels peak in AKI patients (P=0.16, P=0.33, P=0.03, P=0.35, and P=0.17 at each time point, respectively). However, patients with higher grades of AKI (AKIN>2) showed significantly higher pNGAL levels (median values ranging from 207 to 244 ng/ml at various time points), differentiating severe AKI from sCKD (P=0.001, P=0.002, P<0.001, P=0.001, and P=0.004 at each point, respectively).
In patients with AKI, when the temporal trends for pNGAL and SCr were analyzed, significant differences between the five study points were found for pNGAL (P=0.01), whereas SCr did not show differences through the study points (P=0.16) (Figure 2C).
pNGAL for AKI Severity and Susceptibility
Higher levels of pNGAL were associated with more severe AKI by AKIN classification (median values ranging between 69 and 75, 125.5 and 148, 168 and 195, and 301.5 and 328.5 ng/ml for AKIN stages 0, 1, 2, and 3, respectively; P<0.001 for each stage). Median pNGAL levels were significantly higher in those patients who developed AKI on CKD compared with those patients who developed AKI with no previous CKD (208–242 versus 118–138 ng/ml, P<0.001).
Discriminative Ability of pNGAL
To assess the discriminative ability of pNGAL for AKI diagnosis, ROC curves were generated. The AUCs for AKI prediction were 0.77, 0.81, 0.82, 0.79, and 0.78 at different study times (Table 2). The results improved with higher grades of severity (AKIN>2 grade). The AUC for discriminating between AKI and NF was 0.85 (95% CI, 0.81 to 0.90) at the T12 time point. To discriminate AKI from TAz, the AUC attained was 0.73 (95 CI, 0.68 to 0.79).
Table 2.
Area under the receiver operating characteristic (ROC) curves at different time points
| n | T | AKI | AKIN>2a | ||
|---|---|---|---|---|---|
| AUC | CI | AUC | CI | ||
| 616 | 0 | 0.77 | 0.72 to 0.82 | 0.88 | 0.83 to 0.93 |
| 598 | 6 | 0.81 | 0.76 to 0.85 | 0.87 | 0.82 to 0.92 |
| 601 | 12 | 0.82 | 0.77 to 0.87 | 0.89 | 0.84 to 0.94 |
| 586 | 24 | 0.79 | 0.74 to 0.84 | 0.88 | 0.82 to 0.93 |
| 537 | 48 | 0.78 | 0.73 to 0.83 | 0.85 | 0.77 to 0.93 |
T, five study times in hours; AKIN; AUC, area under the ROC curves; CI, 95% confidence interval.
Patients in stage>2 of AKIN classification (reflecting severe AKI).
The performance of pNGAL was analyzed using a predetermined cutoff of 100 or 150 ng/ml (Table 3). At a cutoff of 150 ng/ml, pNGAL was less sensitive but more specific at all time points. For the prediction of AKIN>2, the negative predictive values exceeded 98% for both cutoffs at all time points.
Table 3.
Plasma neutrophil gelatinase-associated lipocalin (pNGAL) performance with predetermined cutoff points
| Times | Sensitivity | Specificity | LR+ | LR− | PPV | NPV |
|---|---|---|---|---|---|---|
| All AKI | ||||||
| Cutoff 100 ng/ml | ||||||
| T0 | 71.0 | 70.4 | 2.4 | 0.4 | 39.0 | 90.6 |
| T6 | 77.4 | 69.6 | 2.5 | 0.3 | 40.0 | 92.2 |
| T12 | 80.2 | 70.9 | 2.8 | 0.3 | 42.3 | 93.1 |
| T24 | 74.8 | 70.6 | 2.6 | 0.3 | 40.4 | 91.3 |
| T48 | 64.4 | 73.3 | 2.4 | 0.4 | 40.4 | 88.1 |
| Cutoff 150 ng/ml | ||||||
| T0 | 53.1 | 86.0 | 3.8 | 0.5 | 50.4 | 87.3 |
| T6 | 54.0 | 86.5 | 4.0 | 0.5 | 51.1 | 87.8 |
| T12 | 54.0 | 88.8 | 4.8 | 0.5 | 56.2 | 87.9 |
| T24 | 56.1 | 86.6 | 4.2 | 0.5 | 52.7 | 88.1 |
| T48 | 48.3 | 92.1 | 6.1 | 0.6 | 63.3 | 86.4 |
| −AKIN≥2a | ||||||
| Cutoff 100 ng/ml | ||||||
| T0 | 93.3 | 64.5 | 2.6 | 0.1 | 11.9 | 99.5 |
| T6 | 96.6 | 62.7 | 2.6 | 0.1 | 11.7 | 99.7 |
| T12 | 96.6 | 63.1 | 2.6 | 0.1 | 11.7 | 99.7 |
| T24 | 92.9 | 63.8 | 2.6 | 0.1 | 11.4 | 99.4 |
| T48 | 84.6 | 67.5 | 2.6 | 0.2 | 11.7 | 98.9 |
| Cutoff 150 ng/ml | ||||||
| T0 | 80.0 | 80.7 | 4.2 | 0.2 | 17.5 | 98.7 |
| T6 | 72.4 | 80.7 | 3.8 | 0.3 | 16.0 | 98.3 |
| T12 | 79.3 | 82.9 | 4.6 | 0.3 | 19.0 | 98.8 |
| T24 | 78.6 | 80.5 | 4.0 | 0.3 | 16.8 | 98.7 |
| T48 | 69.2 | 85.9 | 4.9 | 0.4 | 20.0 | 98.2 |
LR, likelihood ratio; PPV, positive predictive value; NPV, negative predictive value; T, study times; AKIN, Acute Kidney Injury Network classification.
−AKIN≥2, analysis of pNGAL in patients classified as AKIN≥2.
pNGAL Cutoffs for AKI Risk Categories
The minimum P value approach was applied to identify a specific cutoff point for the population under study, but a single cutoff point that would clearly define the patient who developed AKI could not be determined (Figure 3A). Using univariable analysis, a multivariable generalized additive model was fitted to the data at T12, while retaining those variables that kept statistical significance in the univariable study (pNGAL, age, kidney susceptibility, and cardiovascular disease) as independent variables. The resulting function that reflects the shape of the association between pNGAL and the risk of AKI was analyzed, and a gray zone was determined (Figure 3B). A pNGAL value<97 ng/ml was considered as low risk of AKI, and a value>133 ng/ml was considered as high risk. A pNGAL value between these two cutoffs was considered to belong to the gray zone. Using the low-risk category as a reference, the odds ratios (ORs) for the gray zone and high-risk categories on univariable logistic regression model were 4.6 (95% CI, 2.5 to 8.6) and 13 (95% CI, 7.8 to 21.8), respectively. In a multiple logistic regression model, patients with pNGAL in the gray zone had a fourfold risk of AKI (OR, 4.4; 95% CI, 2.3 to 8.6); those patients in the high-risk category showed a 10-fold higher risk of developing AKI (OR, 9.8; 95% CI, 5.6 to 16.9) (Table 4). Presence of cardiovascular disease was associated with an OR of 2.2 (95% CI, 1.3 to 3.7), and high grades of susceptibility were associated with an OR of 3.8 (95% CI, 2.0 to 7.0). Considering this risk prediction model, a very high performance was obtained (Hosmer–Lemeshow P value=0.96), with an excellent discriminative ability with AUC=0.84 (95% CI, 0.80 to 0.88). However, using only clinical variables for risk prediction, worse results were obtained for both predictive and discriminative performances (Hosmer–Lemeshow P value=0.63; AUC=0.75; 95% CI, 0.70 to 0.80). A P value<0.001 was obtained for the comparison of the two AUCs (29).
Figure 3.
Determination of a gray zone for plasma neutrophil gelatinase-associated lipocalin (pNGAL). A shows the resulting plot of minimum P value approach. Chi-squared values measuring the association between AKI and dichotomized pNGAL are plotted as a function of different pNGAL cutoff points. The shape of the plot (a bimodal curve for pNGAL values under 200 ng/ml) suggests the existence of a gray zone. B shows the functions estimated by the generalized additive models considering pNGAL concentrations at T12 of the study (the best point as a biomarker performance). Each vertical line on the x axis represents a 12-hour pNGAL value for individual patients with AKI. The zone where the confidence intervals contain the zero value was considered to be the gray zone, corresponding to 97 ng/ml on the lowest point and 133 ng/ml on the highest point.
Table 4.
Multiple logistic regression model for AKI risk
| Variables | OR | CI | P Value |
|---|---|---|---|
| pNGAL low-/no-risk category | |||
| pNGAL in gray zone | 4.4 | 2.3 to 8.6 | <0.001 |
| pNGAL high-risk category | 9.8 | 5.6 to 16.9 | <0.001 |
| Age (yr) | 1.1 | 1.0 to 1.2 | 0.04 |
| CVD | 2.20 | 1.3 to 3.7 | 0.003 |
| Susceptibility | 3.80 | 2.0 to 7.0 | <0.001 |
Plasma neutrophil gelatinase-associated lipocalin (pNGAL) at 12 hours at low risk (<97 ng/ml), gray zone (97–133 ng/ml), and high risk (>133 ng/ml) analyzed together with the clinical variables: age, cardiovascular disease (CVD), and susceptibility. OR, odds ratio; CI, 95% confidence interval.
A predictiveness curve approach was used to further study the distribution of individual risk of AKI based on their clinical risk and pNGAL levels (30). Analysis of the two predictiveness curves revealed that pNGAL contributes to a better performance of the clinical model, identifying 65% of patients with a risk of AKI less than 21% (line of reference) and 35% of patients with a risk of AKI greater than 21% (Figure 4). In fact, the model that includes pNGAL assigns lower risks to patients with lower estimated probabilities of having AKI and higher risks to patients with higher estimated probabilities of having AKI.
Figure 4.
Predictiveness curve. Graphical representation of the distribution of risk by plasma neutrophil gelatinase-associated lipocalin (pNGAL), age, cardiovascular disease, and kidney susceptibility. The binary outcome is AKI, and the marker value is pNGAL at T12 of the study (the best point as a biomarker performance). The horizontal line of reference corresponds to the prevalence of the disease (21%). The x axis represents the risk percentiles. The y axis represents the probability of identifying the disease. CVD, cardiovascular disease.
To further analyze the effect of adding pNGAL to the clinical model, category-free NRI and IDI were calculated. Considering the proportion of patients with correct reclassification, the addition of pNGAL to the clinical model (age, cardiovascular disease, and kidney susceptibility) improved the predicted risk in 52% of patients with AKI and reduced the predicted risk in 42% of patients without AKI, resulting in an excellent overall cfNRI of 94%. The amplitude of these changes was significant, with a good overall IDI of 0.122 (0.072–0.185) (Table 5).
Table 5.
Net reclassification for model improvement with addition of plasma neutrophil gelatinase-associated lipocalin (pNGAL) and serum Cystatin C (SCysC)
| AKI (21.1%) | +pNGAL | pNGAL + SCysC |
|---|---|---|
| Goodness-of-fit reference | 0.628 | 0.961 |
| Goodness-of-fit reference + B | 0.961 | 0.921 |
| AUC of reference | 0.75 (0.70–0.80) | 0.84 (0.80–0.88) |
| AUC of reference + B | 0.84 (0.80–0.88) | 0.87 (0.81–0.90) =0.002 |
| P value (AUCs difference) | <0.001 | |
| cfNRI, % (CI) | ||
| cfNRI events | 52.4 (29.4 to 69.4) | 40.3 (25.5 to 56.8) |
| cfNRI nonevents | 41.9 (33.4 to 58.3) | 55.7 (43.0 to 68.9) |
| cfNRI | 94.3 (76.7 to 112.4) | 96.1 (74.8 to 119.0) |
| IDI (CI) | ||
| IDI events | 0.10 (0.06 to 0.15) | 0.10 (0.04 to 0.18) |
| IDI nonevents | 0.03 (0.01 to 0.04) | 0.03 (0.01 to 0.05) |
| IDI | 0.12 (0.07 to 0.19) | 0.12 (0.05 to 0.22) |
The reference is the multiple regression model with age, cardiovascular disease, and susceptibility now improved with pNGAL and SCysC at T12 study time. Hosmer–Lemeshow goodness-of-fit test was used to study the calibration of the model. B, biomarker; AUC, area under the receiver operator characteristic curve; cfNRI, category-free net reclassification improvement (cfNRI events + cfNRI nonevents); CI, 95% confidence interval; IDI, integrated discrimination improvement.
Comparison of pNGAL and SCysC
For discrimination of AKI, we have previously shown that the performance of SCysC peaks at the T12 study point in the same cohort as reported herein, with an AUC of 0.88 (20). In the present study, the AUC for pNGAL at the same T12 time point was 0.82. When the performance of pNGAL together with SCysC was analyzed at the T12 study point, we obtained an AUC of 0.88 (0.85–0.92). Therefore, there was no increase in the biomarker performance in terms of AKI discrimination by ROC analysis. However, the risk prediction for AKI improved when SCysC was added to the model containing pNGAL and clinical factors, with excellent overall cfNRI and overall IDI (Table 5).
Discussion
The majority of studies on pNGAL has been performed in well controlled settings of hospital-acquired AKI, such as cardiac surgery (10,18,36), contrast administration (37), or critically ill patients (11,13–16). The purpose of this study was to determine the accuracy of pNGAL as a marker of AKI in patients admitted from the ED, where patients showing small increases of SCr are frequently misdiagnosed. In this study, most AKI patients were admitted with AKI but not identified as having AKI on admission. Early recognition of these patients may lead to a better renal prognosis (38,39). In our study, pNGAL allowed us to confidently distinguish AKI patients from non-AKI patients. Moreover, pNGAL specifically discriminated patients with AKI from those patients with TAz (40,41). The predictive performance of pNGAL depends on the severity of AKI, with increasing AUC values at higher AKIN classes.
We studied five different points from admission to better understand how the biomarker would perform through 48 hours of evolution. We showed differences in trends between the changes in SCr and pNGAL, with the highest peak of pNGAL at 12 hours representing the best performance point as a biomarker. Therefore, we used the 12-hours value to evaluate pNGAL prognostic ability.
There is no consensus regarding the cutoff value above which AKI diagnosis can confidently be made. Physicians often use dichotomization because of the clinical advantages and conveniences that come with this procedure. However, overestimation of measures of effect, loss of information, loss of power, and low external validity of the obtained cutoff point are some of the well known problems inherent to this approach (33). Using the experience of cardiac biomarkers (42,43), we constructed a three-zone partition for pNGAL values and discriminated patients into low-, moderate- (gray zone), and high-risk AKI categories. Clinically, when pNGAL concentrations are in the gray zone, we propose the recognition of risk factors that are independent predictors of AKI, including age, CKD, and comorbidities like CVD. Thus, patients with these risk factors may be considered at high risk of AKI, even when pNGAL levels are in the gray zone. Indeed, in our multivariable model, associating these risk factors with the newly proposed pNGAL categorization boosted the discriminative and predictive ability of AKI diagnosis.
As in other studies (15,44,45), we have shown that pNGAL reflects severity and progression of CKD. Patients with previous CKD were at highest risk of presenting with AKI, which was reliably detected by pNGAL. We could not detect a relationship between lower levels of pNGAL and an early phase of the disease, which was reported for troponin in the early phase of acute myocardial infarction (46). The relation between the three-zone partition for pNGAL and clinical outcomes also needs to be further investigated.
There are two basic statistical approaches to biomarker evaluation (47). One approach models the risk of disease as a function of the biomarker. The second model emphasizes that the value of a disease marker is in the fraction of diseased subjects detected (48). Our data showed that pNGAL has a good discriminative ability and better performance in the risk prediction model proposed (Hosmer–Lemeshow P value=0.96; AUC=0.84). When the predictiveness curve approach was applied, pNGAL was a better predictor of AKI if associated with clinical variables of risk. Although the predictiveness curve relates to classification performance measures, it also displays essential information about risk that is not displayed by the ROC curve (49). The NRI and IDI better identified those patients who did not have AKI than those patients who did have AKI. The addition of pNGAL to the model resulted in a significant change in the goodness of fit and the AUC between the reference and the new model with the biomarker. Interestingly, both cfNRI events and no events behaved similarly in percentage of increase, resulting in a markedly positive overall cfNRI (94%). The value of the IDI, which takes into account the size of those changes, was also good, indicating that the addition of pNGAL to the reference model changed the calculated risk significantly. In other words, the contribution of pNGAL to a more accurate diagnosis of AKI is clearly visible and highly significant. Furthermore, the predicted risks for patients with AKI and no AKI were correctly reclassified with further improvement in overall cNRI and IDI when SCysC was added to the model along with pNGAL.
The values of pNGAL reported in patients with CKD have been widely variable. Bolignano et al. (45) reported mean serum NGAL levels of 515 ng/ml in a cohort of patients with CKD, which exceed the pNGAL levels in patients with AKI, CKD, or AKI reported herein. Possible reasons for the discrepant results might include differences in the biosamples (serum versus plasma) and differences in the assays used (ELISA versus Triage NGAL Device). In the present study, patients at higher stages of AKIN (AKIN>2) showed significantly higher pNGAL levels, easily differentiating severe AKI from stable CKD at all time points. Nevertheless, we acknowledge that the small number of patients with stable CKD (only 2.4% of our cohort) precludes confident conclusions from being made regarding the ability of pNGAL to reliably differentiate AKI from CKD.
This study has other limitations. First, SCr (and not a clinical end point) was used to define the different patient categories of the study. Second, most patients with AKI diagnosis were in the risk category, with smaller groups in injury and failure groups. Therefore, subgroup analysis should not be overinterpreted. Third, we excluded patients with anuria, urinary obstruction, CKD stage 4 or greater, and AKIN stage 3 at admission. Finally, it is a single, although prospective, cohort study, and the results must be confirmed in larger multicenter studies.
In summary, this study identifies pNGAL as an accurate AKI biomarker in patients admitted from the ED. We also propose a three-grade classification of AKI based on pNGAL levels that should be complemented with clinical data and perhaps, additional biomarkers to more accurately identify AKI patients.
Disclosures
P.D. is a coinventor on patents submitted regarding the use of neutrophil gelatinase-associated lipocalin as a biomarker of kidney injury.
Supplementary Material
Acknowledgments
We would like to acknowledge the help and support of the Emergency Department physicians and nursing staff at Hospital Fernando Fonseca. We are also grateful to J. Calado from Hospital Curry Cabral, F. Buinho from Hospital Cruz Vermelha, and J. P. Loureiro from Hospital Garcia de Orta for their help with the data analysis. We thank Alere, Inc. for donating the Triage NGAL Device and kits used in the measurements reported herein.
Funding for this project was from the Portuguese Nephrology Society and the Fernando Fonseca Hospital (K.S.). Funding for A.L.P. was partially sponsored by national funds through Fundação Nacional para a Ciência e Tecnologia, Portugal, Project PEst-OE/MAT/UI0006/2011. P.D. is supported by National Institutes of Health Grants R01 DK069749 and P50 DK096418.
Alere, Inc. did not play any role in the design, execution, analysis, or reporting of the study.
Footnotes
Published online ahead of print. Publication date available at www.cjasn.org.
This article contains supplemental material online at http://cjasn.asnjournals.org/lookup/suppl/doi:10.2215/CJN.12181212/-/DCSupplemental.
References
- 1.Xue JL, Daniels F, Star RA, Kimmel PL, Eggers PW, Molitoris BA, Himmelfarb J, Collins AJ: Incidence and mortality of acute renal failure in Medicare beneficiaries, 1992 to 2001. J Am Soc Nephrol 17: 1135–1142, 2006 [DOI] [PubMed] [Google Scholar]
- 2.Radhakrishnan J, Kiryluk K: Acute renal failure outcomes in children and adults. Kidney Int 69: 17–19, 2006 [DOI] [PubMed] [Google Scholar]
- 3.Coca SG, Yusuf B, Shlipak MG, Garg AX, Parikh CR: Long-term risk of mortality and other adverse outcomes after acute kidney injury: A systematic review and meta-analysis. Am J Kidney Dis 53: 961–973, 2009 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Waikar SS, Liu KD, Chertow GM: Diagnosis, epidemiology and outcomes of acute kidney injury. Clin J Am Soc Nephrol 3: 844–861, 2008 [DOI] [PubMed] [Google Scholar]
- 5.Ali T, Khan I, Simpson W, Prescott G, Townend J, Smith W, Macleod A: Incidence and outcomes in acute kidney injury: A comprehensive population-based study. J Am Soc Nephrol 18: 1292–1298, 2007 [DOI] [PubMed] [Google Scholar]
- 6.Hsu CY, McCulloch CE, Fan D, Ordoñez JD, Chertow GM, Go AS: Community-based incidence of acute renal failure. Kidney Int 72: 208–212, 2007 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Murray PT, Devarajan P, Levey AS, Eckardt KU, Bonventre JV, Lombardi R, Herget-Rosenthal S, Levin A: A framework and key research questions in AKI diagnosis and staging in different environments. Clin J Am Soc Nephrol 3: 864–868, 2008 [DOI] [PubMed] [Google Scholar]
- 8.Solomon R, Segal A: Defining acute kidney injury: What is the most appropriate metric? Nat Clin Pract Nephrol 4: 208–215, 2008 [DOI] [PubMed] [Google Scholar]
- 9.Pickering JW, Frampton CM, Endre ZH: Evaluation of trial outcomes in acute kidney injury by creatinine modeling. Clin J Am Soc Nephrol 4: 1705–1715, 2009 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Dent CL, Ma Q, Dastrala S, Bennett M, Mitsnefes MM, Barasch J, Devarajan P: Plasma neutrophil gelatinase-associated lipocalin predicts acute kidney injury, morbidity and mortality after pediatric cardiac surgery: A prospective uncontrolled cohort study. Crit Care 11: R127, 2007 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Wheeler DS, Devarajan P, Ma Q, Harmon K, Monaco M, Cvijanovich N, Wong HR: Serum neutrophil gelatinase-associated lipocalin (NGAL) as a marker of acute kidney injury in critically ill children with septic shock. Crit Care Med 36: 1297–1303, 2008 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Haase M, Bellomo R, Devarajan P, Schlattmann P, Haase-Fielitz A, NGAL Meta-analysis Investigator Group : Accuracy of neutrophil gelatinase-associated lipocalin (NGAL) in diagnosis and prognosis in acute kidney injury: A systematic review and meta-analysis. Am J Kidney Dis 54: 1012–1024, 2009 [DOI] [PubMed] [Google Scholar]
- 13.Constantin JM, Futier A, Perbet S, Roszyk L, Lautrette A, Gillart T, Guerin R, Jabaudon M, Souweine B, Bazin JE, Sapin V: Plasma neutrophil gelatinase-associated lipocalin is an early marker of acute kidney injury in adult critically ill patients: A prospective study. J Crit Care 25: 176.e1–176.e6, 2010 [DOI] [PubMed] [Google Scholar]
- 14.de Geus HR, Bakker J, Lesaffre EM, le Noble JL: Neutrophil gelatinase-associated lipocalin at ICU admission predicts for acute kidney injury in adult patients. Am J Respir Crit Care Med 183: 907–914, 2011 [DOI] [PubMed] [Google Scholar]
- 15.Haase-Fielitz A, Bellomo R, Devarajan P, Bennett M, Story D, Matalanis G, Frei U, Dragun D, Haase M: The predictive performance of plasma neutrophil gelatinase-associated lipocalin (NGAL) increases with grade of acute kidney injury. Nephrol Dial Transplant 24: 3349–3354, 2009 [DOI] [PubMed] [Google Scholar]
- 16.Kümpers P, Hafer C, Lukasz A, Lichtinghagen R, Brand K, Fliser D, Faulhaber-Walter R, Kielstein JT: Serum neutrophil gelatinase-associated lipocalin at inception of renal replacement therapy predicts survival in critically ill patients with acute kidney injury. Crit Care 14: R9, 2010 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Aghel A, Shrestha K, Mullens W, Borowski A, Tang WH: Serum neutrophil gelatinase-associated lipocalin (NGAL) in predicting worsening renal function in acute decompensated heart failure. J Card Fail 16: 49–54, 2010 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Krawczeski CD, Woo JG, Wang Y, Bennett MR, Ma Q, Devarajan P: Neutrophil gelatinase-associated lipocalin concentrations predict development of acute kidney injury in neonates and children after cardiopulmonary bypass. J Pediatr 158: 1009–1015, 2011 [DOI] [PubMed] [Google Scholar]
- 19.Shapiro NI, Trzeciak S, Hollander JE, Birkhahn R, Otero R, Osborn TM, Moretti E, Nguyen HB, Gunnerson K, Milzman D, Gaieski DF, Goyal M, Cairns CB, Kupfer K, Lee SW, Rivers EP: The diagnostic accuracy of plasma neutrophil gelatinase-associated lipocalin in the prediction of acute kidney injury in emergency department patients with suspected sepsis. Ann Emerg Med 56: 52–59, 2010 [DOI] [PubMed] [Google Scholar]
- 20.Soto K, Coelho S, Rodrigues B, Martins H, Frade F, Lopes S, Cunha L, Papoila AL, Devarajan P: Cystatin C as a marker of acute kidney injury in the emergency department. Clin J Am Soc Nephrol 5: 1745–1754, 2010 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Levey AS, Bosch JP, Lewis JB, Greene T, Rogers N, Roth D, Modification of Diet in Renal Disease Study Group : A more accurate method to estimate glomerular filtration rate from serum creatinine: A new prediction equation. Ann Intern Med 130: 461–470, 1999 [DOI] [PubMed] [Google Scholar]
- 22.Nickolas TL, O’Rourke MJ, Yang J, Sise ME, Canetta PA, Barasch N, Buchen C, Khan F, Mori K, Giglio J, Devarajan P, Barasch J: Sensitivity and specificity of a single emergency department measurement of urinary neutrophil gelatinase-associated lipocalin for diagnosing acute kidney injury. Ann Intern Med 148: 810–819, 2008 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.National Kidney Foundation : K/DOQI clinical practice guidelines for chronic kidney disease: Evaluation, classification, and stratification. Am J Kidney Dis 39[Suppl 1]: S1–S266, 2002 [PubMed] [Google Scholar]
- 24.Bellomo R, Ronco C, Kellum JA, Mehta RL, Palevsky P, Acute Dialysis Quality Initiative workgroup : Acute renal failure—definition, outcome measures, animal models, fluid therapy and information technology needs: The Second International Consensus Conference of the Acute Dialysis Quality Initiative (ADQI) Group. Crit Care 8: R204–R212, 2004 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Mehta RL, Kellum JA, Shah SV, Molitoris BA, Ronco C, Warnock DG, Levin A, Acute Kidney Injury Network : Acute Kidney Injury Network: Report of an initiative to improve outcomes in acute kidney injury. Crit Care 11: R31, 2007 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Mehta RL, Chertow GM: Acute renal failure definitions and classification: Time for change? J Am Soc Nephrol 14: 2178–2187, 2003 [DOI] [PubMed] [Google Scholar]
- 27.Hosmer DW, Lemeshow S: Applied Logistic Regression, New York, John Wiley & Sons, 1989 [Google Scholar]
- 28.Hanley JA, McNeil BJ: The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143: 29–36, 1982 [DOI] [PubMed] [Google Scholar]
- 29.DeLong ER, DeLong DM, Clarke-Pearson DL: Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics 44: 837–845, 1988 [PubMed] [Google Scholar]
- 30.Huang Y, Sullivan Pepe M, Feng Z: Evaluating the predictiveness of a continuous marker. Biometrics 63: 1181–1188, 2007 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Coste J, Pouchot J: A grey zone for quantitative diagnostic and screening tests. Int J Epidemiol 32: 304–313, 2003 [DOI] [PubMed] [Google Scholar]
- 32.Hastie T, Tibshirani R: Generalized Additive Models, New York, Chapman & Hall, 1990 [Google Scholar]
- 33.Pencina MJ, D’Agostino RB, Sr., Steyerberg EW: Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Stat Med 30: 11–21, 2011 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Pencina MJ, D’Agostino RB, Sr., D’Agostino RB, Jr., Vasan RS: Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond. Stat Med 27: 157–172, discussion 207–212, 2008 [DOI] [PubMed] [Google Scholar]
- 35.R Development Core Team: R: A Language and Environment for Statistical Computing, 2008. Available at: http://www.R-project.org Accessed February 2013
- 36.Mishra J, Dent C, Tarabishi R, Mitsnefes MM, Ma Q, Kelly C, Ruff SM, Zahedi K, Shao M, Bean J, Mori K, Barasch J, Devarajan P: Neutrophil gelatinase-associated lipocalin (NGAL) as a biomarker for acute renal injury after cardiac surgery. Lancet 365: 1231–1238, 2005 [DOI] [PubMed] [Google Scholar]
- 37.Hirsch R, Dent C, Pfriem H, Allen J, Beekman RH, 3rd, Ma Q, Dastrala S, Bennett M, Mitsnefes M, Devarajan P: NGAL is an early predictive biomarker of contrast-induced nephropathy in children. Pediatr Nephrol 22: 2089–2095, 2007 [DOI] [PubMed] [Google Scholar]
- 38.Kolhe NV, Stevens PE, Crowe AV, Lipkin GW, Harrison DA: Case mix, outcome and activity for patients with severe acute kidney injury during the first 24 hours after admission to an adult, general critical care unit: Application of predictive models from a secondary analysis of the ICNARC Case Mix Programme database. Crit Care 12[Suppl 1]: S2, 2008 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Kiers HD, Griesdale DE, Litchfield A, Reynolds S, Gibney RT, Chittock D, Pickkers P, Sweet DD: Effect of early achievement of physiologic resuscitation goals in septic patients admitted from the ward on the kidneys. J Crit Care 25: 563–569, 2010 [DOI] [PubMed] [Google Scholar]
- 40.Uchino S, Bellomo R, Bagshaw SM, Goldsmith D: Transient azotaemia is associated with a high risk of death in hospitalized patients. Nephrol Dial Transplant 25: 1833–1839, 2010 [DOI] [PubMed] [Google Scholar]
- 41.Hoste EA, Kellum JA: AKI severity class doesn’t tell all: The case for transient AKI. Nephrol Dial Transplant 25: 1738–1739, 2010 [DOI] [PubMed] [Google Scholar]
- 42.Potocki M, Breidthardt T, Reichlin T, Hartwiger S, Morgenthaler NG, Bergmann A, Noveanu M, Freidank H, Taegtmeyer AB, Wetzel K, Boldanova T, Stelzig C, Bingisser R, Christ M, Mueller C: Comparison of midregional pro-atrial natriuretic peptide with N-terminal pro-B-type natriuretic peptide in the diagnosis of heart failure. J Intern Med 267: 119–129, 2010 [DOI] [PubMed] [Google Scholar]
- 43.Lee HM, Kerr D, O’H Ici D, Kelly AM: Clinical significance of initial troponin I in the grey zone in emergency department chest pain patients: A retrospective pilot study. Emerg Med J 27: 302–304, 2010 [DOI] [PubMed] [Google Scholar]
- 44.Bolignano D, Lacquaniti A, Coppolino G, Campo S, Arena A, Buemi M: Neutrophil gelatinase-associated lipocalin reflects the severity of renal impairment in subjects affected by chronic kidney disease. Kidney Blood Press Res 31: 255–258, 2008 [DOI] [PubMed] [Google Scholar]
- 45.Bolignano D, Lacquaniti A, Coppolino G, Donato V, Campo S, Fazio MR, Nicocia G, Buemi M: Neutrophil gelatinase-associated lipocalin (NGAL) and progression of chronic kidney disease. Clin J Am Soc Nephrol 4: 337–344, 2009 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.van Kimmenade RR, Pinto YM, Januzzi JL, Jr.: Importance and interpretation of intermediate (gray zone) amino-terminal pro-B-type natriuretic peptide concentrations. Am J Cardiol 101[3A]: 39–42, 2008 [DOI] [PubMed] [Google Scholar]
- 47.Moons KGM, Harrell FE: Sensitivity and specificity should be de-emphasized in diagnostic accuracy studies. Acad Radiol 10: 670–672, 2003 [DOI] [PubMed] [Google Scholar]
- 48.Pepe MS, Janes H, Longton G, Leisenring W, Newcomb P: Limitations of the odds ratio in gauging the performance of a diagnostic, prognostic, or screening marker. Am J Epidemiol 159: 882–890, 2004 [DOI] [PubMed] [Google Scholar]
- 49.Pepe MS, Feng Z, Huang Y, Longton G, Prentice R, Thompson IM, Zheng Y: Integrating the predictiveness of a marker with its performance as a classifier. Am J Epidemiol 167: 362–368, 2008 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.