NATIONAL SURGICAL QUALITY IMPROVEMENT PROGRAM UNDERESTIMATES THE RISK ASSOCIATED WITH MILD AND MODERATE POSTOPERATIVE ACUTE KIDNEY INJURY

Abstract

Objective

In a single-center cohort of surgical patients we assessed the association between postoperative change in serum creatinine (sCr) and adverse outcomes and compared the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP)’s definition for acute kidney injury (NSQIP-AKI) with consensus RIFLE (Risk, Injury, Failure, Loss, and End-stage Kidney) and KDIGO (Kidney Disease: Improving Global Outcomes) definitions.

Design

Retrospective single center cohort.

Setting

Academic tertiary medical center.

Patients

27,841 adult patients with no previous history of chronic kidney disease undergoing major surgery.

Intervention

RIFLE defines AKI as change in sCr greater than or equal to 50% while KDIGO uses 0.3 mg/dl change from the reference sCr. Since NSQIP defines AKI as sCr change > 2mg/dl, it may underestimate the risk associated with less severe AKI.

Measurements

The optimal discrimination limits (ODL) for both percent and absolute sCr changes were calculated by maximizing sensitivity and specificity along the receiver operating characteristic (ROC) curves for postoperative complications and mortality.

Main Results

Although prevalence of RIFLE-AKI was 37%, only 7% of RIFLE-AKI patients would be diagnosed with AKI using the NSQIP definition. In multivariable logistic models patients with RIFLE or KDIGO-AKI had a 10 times higher odds of dying compared to patients without AKI. The ODLs for change in sCr associated with adverse postoperative outcomes were as low as 0.2 mg/dl while the NSQIP discrimination limit of 2.0 mg/dl had low sensitivity (0.05 – 0.28).

Conclusion

Current ACS NSQIP definition underestimates the risk associated with mild and moderate AKI otherwise captured by the consensus RIFLE and KDIGO criteria.

Introduction

Severe acute kidney injury (AKI) requiring renal replacement therapy (RRT) is a well-recognized risk factor for hospital mortality (1). With the introduction of the Risk, Injury, Failure, Loss, and End-stage Kidney consensus AKI definition (RIFLE-AKI), which has standardized the description of less severe acute changes in renal function, the adverse effects of small serum creatinine (sCr) changes have begun to be systematically studied (2, 3). Among surgical patients, the association between small postoperative sCr changes and short and long-term mortality has emerged in the literature (4-8).

The RIFLE defines three grades of AKI severity based on at least a 50% change in sCr relative to the reference sCr (RsCr) (9) and the recent consensus Kidney Disease: Improving Global Outcomes (KDIGO) guidelines have expanded the AKI criteria to include changes as small as 0.3 mg/dl (10). However, the implementation of the consensus AKI definition in the surgical guidelines and the literature has been slow (11). The American College of Surgeons Committee on Trauma defines AKI after trauma as a sCr above 3.5 mg/dl, but in a multicenter trauma study only 15% of all RIFLE-AKI trauma patients had a sCr greater than 3 mg/dl (12). The American College of Surgeons–National Surgical Quality Improvement Program’s (ACS NSQIP), the largest prospective surgical database, defines postoperative AKI as a postoperative rise in sCr greater than 2 mg/dl or as the acute need for RRT (13). However, in a single-center study of 10,000 postoperative patients, 90% of RIFLE-AKI patients would not fulfill NSQIP-AKI criteria as their postoperative change in sCr was less than 2 mg/dl (6). Not surprisingly, a study using the 2005–2006 ACS NSQIP dataset reported an AKI prevalence of only 1% with an eightfold increase in 30-day mortality (14). Hence the NSQIP-AKI definition may underestimate the occurrence of AKI in patients with small postoperative sCr changes as defined by the RIFLE or KDIGO classification. Furthermore, the association between adverse outcomes and longitudinal sCr changes considered as a continuous value rather than AKI categories based on predefined cut-offs has not been studied previously in this population.

In a large single-center cohort of patients with no history of chronic kidney disease (CKD) undergoing major surgery we assessed the association between any postoperative change in sCr level and adverse outcomes to determine the optimal discriminatory cut-offs and to compare the consensus AKI definitions with the NSQIP-AKI definition in this cohort.

Patients and Methods

Data source

Using the University of Florida (UF) Integrated Data Repository we assembled training and validation cohorts by integrating perioperative clinical, administrative and laboratory databases at the UF and Shands Hospital. (Supplemental Digital Content (SDC) Methods). The validation cohort was used for validating the performance of logistic regression models developed in the training cohort in order to increase the internal validity and replicability of the results. The training and validation cohorts included all adult patients admitted to the hospital for longer than 48 hours following any type of operative procedure between January 1, 2000 and December 31, 2008 and between January 1, 2009 and November 30, 2010, respectively. The study was designed and approved by the UF Institutional Review Board.

Patient population

We identified 47,801 patients fulfilling the inclusion criteria for the training cohort and 13,299 patients for the validation cohort. After excluding patients with CKD stage five on admission (n=1136), those with a reference estimated glomerular filtration rate (ReGFR) < 65 ml/min/1.75 m² (n=5888), transplant, trauma and burn patients (n=8405) and those with less than two sCr measurements (n=5642), the final training cohort included 27,841 patients with a ReGFR ≥ 65 ml/min/1.75 m². We classified all surgeries as cardiothoracic surgery, non-cardiac general and vascular surgery, neurologic surgery and specialty surgeries (orthopedic, gynecological, ear-nose-throat, urology and plastic surgeries) (SDC Methods). The final validation cohort comprised 7,083 patients with reference eGFR ≥ 65 ml/min/1.75 m2 and a distribution of surgical procedures similar to the training cohort.

Definition of AKI

Changes in measured sCr levels during hospitalization were calculated both as the maximum absolute change and maximum percent change from RsCr in the first thirty postoperative days. Need for RRT was determined using hospital charges for RRT in the billing database. We applied three AKI definitions using sCr changes only without urine output criteria: RIFLE (RIFLE-AKI), KDIGO (KDIGO-AKI) and NSQIP (NSQIP-AKI). RIFLE defines AKI using at least a 50% sCr change from a reference sCr (RsCr) (9), KDIGO uses either 0.3 mg/dl increase within 48 hours or 50% increase above RsCr (10) and NSQIP defines AKI as a sCr increase greater than 2 mg/dl from the preoperative value or as acute RRT requirement within 30 days of the operation (13). For all calculations we defined RsCr either as the minimum of the sCr values available within six months prior to admission including the admission day sCr (used for the results reported in the manuscript) or as the minimum of the sCr values available within seven days prior to admission including the admission day sCr (used for sensitivity analyses) (15). Patients with RIFLE-AKI were stratified according to the maximum RIFLE class reached during the hospital admission using the largest percent change in sCr during hospitalization. RIFLE-R corresponds to a 50% change in sCr, RIFLE-I to a doubling in sCr and RIFLE-F to a tripling in sCr compared to RsCr. Renal outcome at the time of discharge was evaluated by comparing the discharge sCr to the RsCr. Complete renal recovery existed if the sCr returned to a level less than 50% above RsCr, whereas partial renal recovery existed if there was a persistent increase in sCr more than 50% above RsCr but no need for RRT. No renal recovery implied there was a need for RRT at the time of hospital discharge.

Covariates and outcomes

Patient survival status was determined using hospital discharges and the Social Security Death Index to measure hospital mortality, 90-day mortality and hospital mortality rates (number of deaths per 100 patient-months) to adjust for length of stay. In addition, we used postoperative complications as outcome variables as previously described (SDC, Table 1) (16, 17). For the definition of sepsis we followed the selection criteria developed by the Agency for Healthcare Research and Quality for the patient safety indicators “Postoperative Sepsis” (PSI-13) (18) using International Classification of Diseases, Ninth Revision, Clinical Modification ICD-9-CM) codes. Organ failure associated with sepsis was identified by adding ICD-9-CM codes for acute organ dysfunction to the sepsis diagnosis (19, 20). For each patient in the cohort the exact dates were used to calculate the duration of mechanical ventilation (MV) and RRT. Long-term MV was defined as a requirement for MV greater or equal to 14 consecutive days. The presence of underlying comorbidities was identified by ICD-9-CM codes as described by Elixhauser et al (21). In addition, we calculated the Charlson-Deyo comorbidity score and grouped patients into score categories of 0, 1-2, 3-4 and ≥5 (22). For each subject we also determined whether discharge records contained ICD-9-CM diagnostic codes for “ARF” (584.XX or 997.5).

Statistical analysis

The analytical plan followed the STROBE recommendations for observational cohort studies (23) and was performed using SAS software (v.9.2, Cary, N.C.) by AB, MB, TOB and CEH (SDC, Methods). Univariable logistic regression models were fit to assess the association between sCr change as a continuous measurement (percent change or absolute change in sCr from RsCr) and each outcome variable (hospital mortality and post-operative complications) separately. Receiver operating characteristic (ROC) curve analysis was performed by calculating sensitivity and specificity for sCr changes. The optimal discrimination limits (ODL) for both percent and absolute sCr changes were calculated by maximizing sensitivity and specificity along the ROC curves for each outcome. Sensitivity and specificity of the NSQIP-AKI discrimination limit (2.0 mg/dl) was obtained from ROC curve fit for absolute changes in sCr. Area under the curve (AUC) with 95% confidence intervals (CI) was used to assess model fit.

We constructed separate multivariable logistic regression models for each outcome using different definitions of AKI as the main independent covariate. NSQIP-AKI, RIFLE-AKI and KDIGO-AKI were entered in each model as class variables while absolute change in sCr and percent change in sCr were entered as continuous variables. Each model was adjusted for age (continuous variable), gender (female versus male), race (African-American versus others), admission comorbidities (Charlson-Deyo comorbidity score as a continuous variable), elective operative status (emergent surgery versus routine elective surgery), day of admission (weekend versus weekday), surgery type (reference specialty surgery) and primary insurance (reference private insurance). We selected explanatory variables based on their significance in prior univariable analysis and previously reported associations in the literature. Model diagnostics included testing for collinearity (variance inflation factor and tolerance), interactions and model fit (Hosmer-Lemeshow goodness of fit). Multiple ROC curve comparison analyses were performed to assess the predictive performance of univariable and multivariable models with different AKI definitions for each outcome in both training and validation cohorts and between the cohorts. AUC (95% CI) comparisons were made using the DeLong test, a nonparametric method that exploits the mathematical equivalence of the AUC to the Mann-Whitney U-statistic (24). We performed sensitivity analyses by comparing the effect of a) different definitions for RsCr and b) inclusion of patients with missing values for sCr and c) omission of different covariates on model fit for each outcome.

Results

Prevalence and mortality rates for RIFLE-AKI and NSQIP-AKI

The prevalence of RIFLE-AKI among 27,841 adult surgical patients with ReGFR ≥ 65 ml/min/1.75 m² was 37% (10,228/27,841). Mild and moderate AKI (stages Risk and Injury) comprised the majority of all RIFLE-AKI: 58% (5959/10,228) RIFLE-R and 25% (2578/10,228) RIFLE-I with only 17% (1691/10,228) with severe RIFLE-F AKI. Only 3% (701/27,841) of all patients were classified with AKI using NSQIP definition. No patients with mild RIFLE-AKI were classified as having AKI using the NSQIP definition and only 40% of patients with severe AKI (RIFLE-F) would fulfill NSQIP criteria for AKI. Overall 93% of all patients with RIFLE-AKI were not classified as having AKI using the NSQIP definition (Table 1).

Table 1.

Clinical characteristics and postoperative outcomes for all patients stratified by RIFLE-AKI stages.

	No AKI (n=17,613)	All AKI (n=10,228)	P1	RIFLE-R (n=5959)	RIFLE-I (n=2578)	RIFLE-F (n=1691)	P2
AKI by NSQIP definition, n (%)		701 (7)		0 (0)	3 (0.1)	698 (41)	<0.001
ARF diagnostic code1 in discharge summary, n (%)		1021 (10)		114 (2)	166 (6)	741 (44)	<0.001
Renal replacement therapy, n (%)		332 (3)		0 (0)	0 (0)	332 (20)	<0.001
AKI onset in the first 72 hours postoperatively, n (%)		7330 (72)		4197 (70)	1948 (76)	1185 (70)	<0.001
AKI onset between day 3-7 postoperatively, n (%)		2515 (25)		1666 (28)	511 (20)	338 (20)
AKI onset after postoperative day 7, n (%)		383 (3)		96 (3)	119 (5)	168 (10)
Complete Renal Recovery, n (%)		7491 (73)		5107 (86)	1707 (66)	677 (40)	<0.001
Partial or No Renal Recovery, n (%)		2735 (27)		852 (14)	871 (34)	1012 (60)
Baseline Characteristics
Age (years), Mean (SD)	54 (15)	56 (15)	<0.001	55 (15)	56 (15)	56 (15)	NS
Female sex, n (%)	9017 (51)	5094 (50)	0.03	3081 (52)	1300 (50)	713 (42)	<0.001
African-American ethnicity, n (%)	2174 (12)	1473 (14)	<0.001	794 (13)	390 (15)	289 (17)	<0.001
Rural area residency	5411 (33)	3132 (33)	NS	1843 (33)	794 (33)	495 (32)	NS
Annual Income (1000$), Median (25th, 75th)	35 (30, 41)	34 (29, 41)	NS	34 (29, 41)	34 (28, 41)	33 (28, 41)	NS
Primary Insurance			<0.001				0.003
Medicare	6740 (38)	4632 (45)		2655 (45)	1195 (46)	782 (46)
Medicaid	1808 (10)	1360 (14)		746 (13)	368 (14)	246 (15)
Private	7993 (45)	3709 (36)		2222 (37)	895 (35)	592 (35)
No insurance	1072 (6)	527 (5)		336 (5)	120 (5)	71 (4)
Weekend admission	1568 (9)	1484 (15)	<0.001	804 (13)	414 (16)	266 (16)	0.002
Weekend discharge	4026 (23)	1685 (16)	<0.001	1028 (17)	403 (16)	254 (15)	0.04
Emergency admission	5247 (30)	5135 (50)	<0.001	2677 (45)	1412 (55)	1046 (62)	<0.001
Initial admission to medicine service	1154 (7)	2093 (20)	<0.001	889 (15)	568 (22)	636 (38)	<0.001
Charlson’s Comorbidity Score, Median (25th, 75th)	1 (0, 2)	2 (1, 3)	<0.001	1 (0, 3)	2 (1, 3)	2 (1, 3)	0.05
0	7226 (41)	2474 (24)	<0.001	1642 (28)	568 (22)	264 (16)	<0.001
1-2	6920 (39)	4804 (47)		2748 (46)	1234 (48)	822 (48)
3-4	1734 (10)	1771 (17)		879 (15)	501 (19)	391 (23)
≥ 5	1715 (10)	1170 (12)		685 (11)	271 (11)	214 (13)
Postoperative Outcomes
Hospital mortality, n (%)	98 (0.6)	771 (8)	<0.001	162 (3)	168 (7)	441 (26)	<0.001
90-day mortality, n (%)	457 (3)	1103 (11)	<0.001	343 (6)	273 (11)	487 (29)	<0.001
Sepsis, n (%)	91 (0.5)	929 (9)	<0.001	156 (3)	260 (10)	513 (30)	<0.001
Severe Sepsis, n (%)	42 (0.2)	830 (8)	<0.001	117 (2)	216 (8)	497 (29)	<0.001
Postoperative surgical infections, n (%)	639 (4)	805 (8)	0.001	380 (6)	250 (10)	175 (10)	0.001
Mechanical wound complications, n (%)	700 (4)	981 (10)	<0.001	486 (8)	276 (11)	219 (13)	<0.001
Procedural complications, n (%)	782 (4)	1102 (11)	<0.001	532 (9)	297 (12)	273 (16)	<0.001
Mechanical ventilation, n (%)	2251 (13)	4393 (43)	<0.001	2045 (34)	1247 (48)	1101 (65)	<0.001
Days on mechanical ventilation,	2 (1,2)	4 (2, 14)	<0.001	2 (2, 7)	6 (2, 16)	12 (3, 29)	<0.001
Long-term mechanical ventilation, n (%)	367 (2)	1431 (14)	<0.001	423 (7)	446 (17)	562 (33)	<0.001
Pulmonary complications, n (%)	729 (4)	1584 (15)	<0.001	668 (11)	475 (18)	441 (26)	<0.001
Cardiovascular complications, n (%)	512 (3)	775 (8)	<0.001	356 (6)	237 (9)	182 (11)	<0.001
Neurological complications, n (%)	294 (2)	369 (4)	<0.001	186 (3)	117 (5)	66 (4)	0.004
Gastrointestinal complications, n (%)	407 (2)	513 (5)	0.05	268 (5)	147 (6)	98 (6)	0.02

¹International Classification of Diseases, Ninth Revision, Clinical Modification diagnostic codes 584.XX or 997.5.

P¹ Comparing no RIFLE-AKI versus all RIFLE-AKI patients.

P² Comparing RIFLE-AKI classes to each other.

Although baseline demographic characteristics of patients with RIFLE-AKI were similar to those with no AKI, RIFLE-AKI more often emerged as an early complication (AKI onset within the first 72 postoperative hours among 72% of patients) of emergent surgical cases and for those patients admitted on weekends (Table 1). Postoperative complications were significantly more prevalent among patients with RIFLE-AKI and proportional to the AKI severity. Only 3% (332/10,228) of all patients with RIFLE-AKI required RRT. Close to two thirds of all RIFLE-AKI patients had complete recovery at hospital discharge.

Hospital and 90-day mortality were significantly higher among patients with AKI compared to patients with no AKI, regardless of the definition (Table 2). Mortality rates were four to ten times higher in patients with RIFLE-AKI compared to patients with no AKI, and varied depending on the type of surgical procedure from 10 to 14 deaths per 100 patient-months (100 PM). Although patients with NSQIP-AKI had high mortality rates (25 to 34 deaths per 100 PM), they comprised only a small fraction of all patients and only 20% of all deaths occurred in this group. Furthermore, when the RIFLE-AKI definition was applied to the 97% of the cohort not fulfilling NSQIP-AKI criteria, the 35% additional patients identified as having RIFLE-AKI had a two to five fold increase in mortality rates compared to patients with no RIFLE-AKI with an additional 821 deaths in the 90-day period (Table 3). Although the excess in mortality rates for patients with no NSQIP-AKI was highest among patients with severe RIFLE-F AKI, the most death events were missed among patients with mild (RIFLE-R, 343 deaths) and moderate RIFLE-AKI (RIFLE-I,272 deaths).

Table 2.

Prevalence and hospital mortality rates for patients with RIFLE-AKI compared to NSQIP-AKI, stratified for surgical services.

	RIFLE-AKI	NSQIP-AKI	No RIFLE-AKI	No NSQIP-AKI
	Prevalence, n (%)	Prevalence, n (%)	Prevalence, n (%)	Prevalence, n (%)
	Hospital mortality, n (%)	Hospital mortality, n (%)	Hospital mortality, n (%)	Hospital mortality, n (%)
	90-day mortality, n (%)	90-day mortality, n (%)	90-day mortality, n (%)	90-day mortality, n (%)
	Mortality rate/100 PM1	Mortality rate/100 PM	Mortality rate/100 PM	Mortality rate/100 PM
All patients (n=27841)	10228 (37%)	701 (3%)	17613 (63%)	27140 (98%)
	771 (8%)	269 (38%)	98 (0.6%)	600 (2%)
	1103 (11%)	282 (40%)	457 (3%)	1278 (5%)
	11 (10, 12)	31 (27, 35)	2 (2, 3)	6 (5, 6)
Surgical Service
Cardiothoracic surgery (n=4292)	2415 (56%)	291 (7%)	1877 (44%)	4001 (93%)
	210 (9%)	111 (38%)	18 (1%)	117 (3%)
	265 (11%)	112 (38%)	54 (3%)	207 (5%)
	11 (12, 13)	31 (26, 37)	3 (2, 5)	6 (5, 7)
Non-Cardiac General and Vascular Surgery (n=8532)	3324 (39%)	215 (3%)	5208 (61%)	8317 (97%)
	220 (7%)	94 (44%)	8 (0.2%)	134 (2%)
	329 (10%)	103 (48%)	107 (3%)	333 (4%)
	10 (9, 12)	34 (28, 42)	1 (1, 2)	4 (4,5)
Neurologic surgery (n=5526)	1809 (33%)	49 (1%)	3717 (67%)	5477 (99%)
	218 (12%)	26 (53%)	61 (2%)	253 (5%)
	297 (16%)	24 (49%)	186 (5%)	459 (8%)
	14 (12, 16)	29 (28, 42)	6 (5, 8)	16 (9, 12)
Specialty Surgeries (n=9491)	2680 (28%)	146 (2%)	6811 (72%)	9345 (98%)
	123 (5%)	38 (26%)	11 (0.1%)	96 (1%)
	212 (8%)	43 (29%)	110 (2%)	279 (3%)
	10 (8, 12)	25 (18, 34)	1 (1, 2)	4 (3, 5)

¹Mortality rate are calculated as number of deaths per 100 person-months (PM) (95% Confidence Intervals)

Hospital mortality, 90-day mortality, and hospital mortality rates are significantly different between patients stratified by AKI using both RIFLE-AKI and NSQIP criteria.

Table 3.

Prevalence of RIFLE-AKI stages and mortality rates among 27,140 patients without AKI per NSQIP definition.

	Prevalence N (%)	Hospital Mortality N (%)	90-day Mortality N (%)	Mortality rate/100 PM1 (95% CI)
All patients without NSQIP-AKI (n=27140)
No RIFLE-AKI	17613 (65%)	98 (0.6%)	457 (3%)	2.5 (2.0, 3.0)
RIFLE-AKI, all stages
RIFLE-AKI Risk	5959 (22%)	162 (3%)*	343 (6%)*	6.0 (5.2, 7.0)*
RIFLE-AKI Injury	2575 (9%)	167 (6%)*	272 (11%)*	8.5 (7.3, 9.9)*
RIFLE-AKI Failure	993 (4%)	173 (17%)*	206 (21%)*	12.7 (10.9, 14.7)*

¹Mortality rate calculated as number of deaths per 100 person-months (95% Confidence Interval)

^*p-value<0.05 for comparing RIFLE-AKI stages to No RIFLE-AKI.

Optimal discrimination limits for postoperative changes in serum creatinine

In addition to using a predefined cut-off to define AKI we tested the association between any postoperative changes in sCr considered as a continuous variable and the occurrence of adverse hospital outcomes in both cohorts. Univariable logistic regression models demonstrated a significant association between both percent and absolute changes in sCr and all outcomes (Table 4, Figure 1A-B). Optimal discrimination limits (ODL) for percent change in sCr in relationship to different outcomes ranged from 1.38 to 1.70 (38% to 70% change from reference sCr) with AUC from 0.62-0.92. The ODL for absolute change in sCr ranged from 0.21 mg/dl to 0.48 mg/dl with AUC from 0.59-0.89. Sensitivity and specificity for the NSQIP-AKI cut-off of 2.0 mg/d was determined along the ROC curve for absolute change in sCr model for each outcome. Although the specificity of NSQIP-AKI cut-off was high, for all outcomes sensitivity remained low (0.05 – 0.28) and all NSQIP-AKI models had significantly lower AUC (95% CI) in comparison to models using percent or absolute change in sCr (P<0.05). When applied in the validation cohort all univariable models had unhanged or significantly higher AUC (95% CI).

Table 4.

Optimal discrimination limits for the percent and absolute change in serum creatinine associated with adverse postoperative outcomes for training and validation cohorts.

Outcome	Percent change in sCr Model (ratio)	Absolute change in sCr Model (mg/dl)
Hospital mortality
Optimal Discrimination limit (Sensitivity, specificity)	1.68 (0.80, 0.80)	0.31 mg/dl (0.81, 0.75)
NSQIP-AKI cut-off1 (Sensitivity, specificity)		2 mg/dl (0.28, 0.98)
Training AUC2 (95% CI)	0.87 (0.86, 0.88)	0.86 (0.84, 0.87)
Validation AUC2 (95% CI)	0.85 (0.82, 0.88)	0.83 (0.80, 0.87)
90-day mortality
Optimal Discrimination limit (Sensitivity, specificity)	1.66 (0.61, 0.77)	0.40 mg/dl (0.59, 0.77)
NSQIP-AKI cut-off1 (Sensitivity, specificity)		2 mg/dl (0.18, 0.98)
Training AUC2 (95% CI)	0.75 (0.73, 0.76)	0.73 (0.71, 0.74)
Validation AUC2 (95% CI)	0.79 (0.76, 0.83)	0.77 (0.73, 0.81)
Sepsis
Optimal Discrimination limit (Sensitivity, specificity)	1.70 (0.85, 0.80)	0.32 mg/dl (0.82, 0.76)
NSQIP-AKI cut-off (Sensitivity, specificity)		2 mg/dl (0.24, 0.98)
Training AUC (95% CI)	0.89 (0.88, 0.90)	0.86 (0.85, 0.87)
Validation AUC (95% CI)	0.90 (0.88, 0.92)	0.87 (0.84, 0.89)
Severe sepsis
Optimal Discrimination limit (Sensitivity, specificity)	1.70 (0.90, 0.80)	0.32 mg/dl (0.87, 0.75)
NSQIP-AKI cut-off (Sensitivity, specificity)		2 mg/dl (0.27, 0.98)
Training AUC (95% CI)	0.92 (0.91, 0.93)	0.89 (0.88, 0.90)
Validation AUC (95% CI)	0.93 (0.91, 0.94)	0.90 (0.88, 0.92)
Postoperative surgical infections
Optimal Discrimination limit (Sensitivity, specificity)	1.38 (0.67, 0.54)	0.28 mg/dl (0.56, 0.61)
NSQIP-AKI cut-off (Sensitivity, specificity)		2.0 mg/dl (0.06, 0.97)
Training AUC (95% CI)	0.64 (0.62, 0.65)	0.61 (0.60, 0.63)
Validation AUC (95% CI)	0.66 (0.63, 0.89)	0.63 (0.60, 0.66)
Mechanical wound complications
Optimal Discrimination limit (Sensitivity, specificity)	1.50 (0.58, 0.65)	0.23 mg/dl (0.60, 0.60)
NSQIP-AKI cut-off (Sensitivity, specificity)		2.0 mg/dl (0.07, 0.97)
Training AUC (95% CI)	0.65 (0.64, 0.67)	0.63 (0.62, 0.65)
Validation AUC (95% CI)	0.65 (0.62, 0.68)	0.63 (0.60, 0.66)
Procedural complications
Optimal Discrimination limit (Sensitivity, specificity)	1.48 (0.59, 0.65)	0.23 mg/dl (0.61, 0.61)
NSQIP-AKI cut-off (Sensitivity, specificity)		2.0 mg/dl (0.07, 0.98)
Training AUC (95% CI)	0.66 (0.64,0.67)	0.65 (0.63, 0.66)
Validation AUC (95% CI)	0.68 (0.65, 0.71)	0.67 (0.64, 0.70)
Mechanical ventilation
Optimal Discrimination limit (Sensitivity, specificity)	1.43 (0.71, 0.69)	0.23 mg/dl (0.70, 0.68)
NSQIP-AKI cut-off (Sensitivity, specificity)		2 mg/dl (0.14, 0.98)
Training AUC (95% CI)	0.76 (0.76, 0.77)	0.75 (0.75, 0.76)
Validation AUC (95% CI)	0.78 (0.77, 0.80)**	0.77 (0.76, 0.78)**
Long term mechanical ventilation6
Optimal Discrimination limit (Sensitivity, specificity)	1.65 (0.70, 0.78)	0.31 mg/dl (0.64, 0.76)
NSQIP-AKI cut-off (Sensitivity, specificity)		2 mg/dl (0.16, 0.98)
Training AUC (95% CI)	0.80 (0.79, 0.81)	0.75 (0.74, 0.77)
Validation AUC (95% CI)	0.84 (0.82, 0.86)**	0.79 (0.76, 0.81)**
Pulmonary complications
Optimal Discrimination limit (Sensitivity, specificity)	1.45 (0.70, 0.65)	0.23 mg/dl (0.71, 0.61)
NSQIP-AKI cut-off (Sensitivity, specificity)		2 mg/dl (0.1, 0.98)
Training AUC (95% CI)	0.73 (0.72, 0.74)	0.71 (0.70, 0.72)
Validation AUC (95% CI)	0.73 (0.70, 0.75)	0.71 (0.69, 0.73)
Cardiovascular complications
Optimal Discrimination limit (Sensitivity, specificity)	1.41 (0.67, 0.60)	0.23 mg/dl (0.68, 0.60)
NSQIP-AKI cut-off (Sensitivity, specificity)		2 mg/dl (0.10, 0.97)
Training AUC (95% CI)	0.68 (0.66,0.69)	0.68 (0.66, 0.69)
Validation AUC (95% CI)	0.66 (0.63, 0.69)	0.65 (0.62, 0.68)
Neurological complications
Optimal Discrimination limit (Sensitivity, specificity)	1.61 (0.45, 0.75)	0.21 mg/dl (0.56, 0.58)
NSQIP-AKI cut-off (Sensitivity, specificity)		2 mg/dl (0.05, 0.97)
Training AUC (95% CI)	0.62 (0.60, 0.64)	0.59 (0.57, 0.61)
Validation AUC (95% CI)	0.64 (0.60, 0.69)	0.60 (0.55, 0.64)
GI complications
Optimal Discrimination limit (Sensitivity, specificity)	1.46 (0.57, 0.63)	0.26 mg/dl (0.58, 0.60)
NSQIP-AKI cut-off (Sensitivity, specificity)		2 mg/dl (0.05, 0.97)
Training AUC (95% CI)	0.63 (0.61, 0.65)	0.62 (0.60, 0.63)
Validation AUC (95% CI)	0.65 (0.61, 0.69)	0.64 (0.59, 0.68)

¹NSQIP sensitivity and specificity were found along the ROC curve for absolute change in sCr model applying the NSQIP-AKI cut-off of 2.0 mg/dl change from RsCr.

²Training AUC (95% CI) and validation AUC (95% CI) indicate area under receiver operator curve with 95% confidence intervals for each univariable model applied in training and validation cohort, respectively.

^**p-value < 0.05 for difference in AUC (95% CI) between the models fit to training cohort and validation cohort.

Figure 1.

Univariate receiver operating curves with discrimination limits for A. Percent change in sCr and in-hospital mortality (represented as ratio) and B. Absolute change in sCr and in-hospital mortality. Sensitivity and specificity of the NSQIP-AKI discrimination limit (2.0 mg/dl) was obtained from ROC curve fit for absolute changes in sCr.

Multivariable logistic regression model comparison for different definitions of AKI

We constructed five multivariable logistic regression models for hospital mortality as an outcome using different definitions for AKI (NSQIP-AKI, RIFLE-AKI, KDIGO-AKI, absolute change in sCr and percent change in sCr) as the main independent variable (see Statistical Analysis). Although all models showed that AKI had a significant association with hospital mortality, the model using the NSQIP-AKI definition had a significantly lower AUC compared to all other definitions (AUC 0.80, 95% CI 0.79-0.82, P<0.05 for difference) (Table 5). Patients with either RIFLE-AKI or KDIGO-AKI had 10 times higher odds of dying compared to patients without AKI. For each 10% change is sCr the odds of dying increased by 5% (95% CI 1.04-1.06, P<0.001, AUC 0.87) while for every 0.1 mg/dl change in sCr the odds increased by 8% (95% CI 1.08-1.09, P<0.001, AUC 0.87). Using the same approach we constructed multivariable logistic regression models for each postoperative complication as an outcome. Both RIFLE-AKI and KDIGO-AKI were associated with significantly higher odds for each of the tested complications. Changes in sCr as low as 10% or 0.1 mg/dl were associated with increased odds for postoperative complications, with the strongest association observed for sepsis and mechanical ventilation. AUC of models using each of the above definitions were significantly higher than AUC of models using the NSQIP-AKI definition for all tested outcomes (P<0.05). When applied in the validation cohort all multivariable models had unchanged or significantly higher AUC (95% CI). The sensitivity analyses examining the effect of two methods of assigning the RsCr value demonstrated no significant change in model fit for each outcome regardless of which method was used to define the RsCr (SDC Table 2). Similarly, inclusion of patients with missing values for sCr and omission of different covariates did not affect model fit for any of the outcomes (data not shown).

Table 5.

Comparison of the association between different AKI definitions and adverse outcomes in multivariable logistic regression models for training and validation cohorts

	RIFLE-AKI Model (compared to no RIFLE-AKI)	Percent change in sCr Model (for every 10% change)	KDIGO-AKI Model (compared to no KDIGO-AKI)	Absolute Change in sCr Model (for every 0.1 mg/dl change)
Hospital mortality*
OR (95% CI)	10.67 (8.59, 13.26)	1.05 (1.04, 1.06)	10.04 (7.94, 12.68)	1.08 (1.08, 1.09)
Training AUC (95% CI)	0.84 (0.84, 0.86)	0.87 (0.86, 0.89)	0.84 (0.83, 0.85)	0.87 (0.87, 0.89)
Validation AUC (95% CI)	0.86 (0.84, 0.88)	0.87 (0.87, 0.90)	0.85 (0.83, 0.88)	0.87 (0.85, 0.89)
90-day mortality*
OR (95% CI)	3.26 (2.90, 3.67)	1.03 (1.03, 1.04)	2.94 (2.60, 3.33)	1.06 (1.06, 1.07)
Training AUC (95% CI) 1	0.81 (0.80, 0.82)	0.82 (0.81, 0.83)	0.80 (0.79, 0.81)	0.82 (0.81, 0.83)
Validation AUC (95% CI)1	0.84 (0.82, 0.86)	0.85 (0.83, 0.87)	0.83 (0.81, 0.86)**	0.85 (0.83, 0.87)
Sepsis*
OR (95% CI)	15.97 (12.81, 19.91)	1.06 (1.05, 1.06)	15.49 (12.17, 19.72)	1.09 (1.08, 1.09)
Training AUC (95% CI)	0.83 (0.83, 0.84)	0.87 (0.86, 0.88)	0.82 (0.81, 0.83)	0.84 (0.83, 0.86)
Validation AUC (95% CI)	0.86 (0.85, 0.88)**	0.87 (0.85, 0.89)	0.85 (0.83, 0.87)**	0.85 (0.83, 0.87)
Severe sepsis*
OR (95% CI)	31.02 (22.66, 42.47)	1.06 (1.05, 1.06)	30.84 (21.70, 43.84)	1.10 (1.09, 1.10)
Training AUC (95% CI)	0.85 (0.84, 0.86)	0.89 (0.88, 0.9)	0.84 (0.83, 0.85)	0.87 (0.86, 0.88)
Validation AUC (95% CI)	0.88 (0.86, 0.90)**	0.89 (0.86, 0.91)	0.87 (0.85, 0.88)**	0.87 (0.84, 0.89)
Postoperative surgical infection*
OR (95% CI)	2.21 (1.97, 2.47)	1.02 (1.01, 1.02)	2.14 (1.91, 2.40)	1.03 (1.02,1.03)
Training AUC (95% CI)	0.67 (0.66, 0.69)	0.66 (0.64, 0.67)	0.67 (0.65, 0.68)	0.65 (0.64, 0.67)
Validation AUC (95% CI)	0.68 (0.65, 0.72)	0.67 (0.64, 0.7)	0.68 (0.65, 0.72)	0.67 (0.64, 0.7)
Mechanical wound complications*
OR (95% CI)	2.39 (2.15, 2.66)	1.02 (1.01, 1.02)	2.26 (2.03, 2.51)	1.03 (1.02, 1.03)
Training AUC (95% CI)	0.65 (0.65, 0.67)	0.63 (0.68, 0.65)	0.65 (0.64, 0.67)	0.63 (0.61, 0.64)
Validation AUC (95% CI)	0.66 (0.63, 0.69)	0.64 (0.61, 0.67)	0.66 (0.63, 0.69)	0.64 (0.61, 0.67)
Procedural complications*
OR (95% CI)	4.42 (4.13, 4.73)	1.02 (1.02, 1.02)	2.55 (2.30, 2.83)	1.03 (1.03, 1.04)
Training AUC (95% CI)	0.65 (0.64, 0.66)	0.63 (0.61, 0.64)	0.65 (0.63, 0.66)	0.63 (0.61, 0.64)
Validation AUC (95% CI)	0.68 (0.65, 0.71)	0.66 (0.63, 0.7)	0.68 (0.65, 0.72)	0.66 (0.63, 0.69)
Mechanical ventilation*
OR (95% CI)	9.59 (8.58, 10.73)	1.06 (1.06, 1.06)	4.29 (4.01, 4.59)	1.09 (1.08, 1.09)
Training AUC (95% CI)	0.82 (0.82, 0.83)	0.81 (0.81, 0.82)	0.82 (0.81, 0.82)	0.80 (0.80, 0.81)
Validation AUC (95% CI)	0.84 (0.83, 0.85)**	0.84 (0.83, 0.85)**	0.83 (0.82, 0.84)**	0.82 (0.81, 0.83) **
Long term mechanical ventilation*
OR (95% CI)	7.53 (6.67, 8.52)	1.06 (1.05, 1.06)	6.15 (5.43, 6.97)	1.08 (1.07, 1.08)
Training AUC (95% CI)	0.81 (0.80, 0.82)	0.81 (0.80, 0.82)	0.79 (0.78, 0.80)	0.79 (0.77, 0.8)
Validation AUC (95% CI)	0.83 (0.81, 0.85)	0.83 (0.81, 0.85)	0.81 (0.79, 0.83)	0.80 (0.78, 0.83)
Pulmonary complications*
OR (95% CI)	3.85 (3.49, 4.23)	1.03 (1.02, 1.03)	3.68 (3.33, 4.07)	1.05 (1.04, 1.05)
Training AUC (95% CI)	0.72 (0.71, 0.73)	0.70 (0.68, 0.71)	0.71 (0.70, 0.72)	0.69 (0.67, 0.70)
Validation AUC (95% CI)	0.72 (0.70, 0.74)	0.70 (0.67, 0.72)	0.71 (0.69, 0.73)	0.69 (0.66, 0.71)
Cardiovascular complications*
OR (95% CI)	2.24 (1.98, 2.53)	1.01 (1.01, 1.02)	2.38 (2.09, 2.70)	1.03 (1.02, 1.03)
Training AUC (95% CI)	0.75 (0.73, 0.76)	0.74 (0.73, 0.75)	0.75 (0.74, 0.76)	0.74 (0.72, 0.75)
Validation AUC (95% CI)	0.74 (0.72, 0.77)	0.74 (0.71, 0.76)	0.74 (0.71, 0.77)	0.74 (0.71, 0.76)
Neurological complications*
OR (95% CI)	2.63 (2.21, 3.13)	1.02 (1.01, 1.02)	2.51 (2.11, 2.98)	1.04 (1.03, 1.04)
Training AUC (95% CI)	0.85 (0.84, 0.87)	0.85 (0.84, 0.86)	0.85 (0.84, 0.87)	0.85 (0.84, 0.86)
Validation AUC (95% CI)	0.86 (0.84, 0.89)	0.85 (0.82, 0.88)	0.86 (0.83, 0.89)	0.84 (0.82, 0.88)
GI complications*
OR (95% CI)	2.56 (2.23, 2.94)	1.01 (1.01, 1.02)	2.51 (2.18, 2.89)	1.03 (1.02,1.04)
Training AUC (95% CI)	0.75 (0.73, 0.76)	0.73 (0.71, 0.75)	0.75 (0.73, 0.76)	0.73 (0.71, 0.75)
Validation AUC (95% CI)	0.78 (0.74, 0.81)	0.76 (0.73, 0.80)	0.78 (0.75, 0.81)	0.76 (0.73, 0.80)

For detailed models description see Statistical Analysis.

¹Training AUC (95% CI) and validation AUC (95% CI) indicate area under receiver operator curve with 95% confidence intervals for each model applied in training and validation cohort, respectively.

^*p-value < 0.05 for difference in AUC (95% CI) between the models fit for each listed AKI definition compared to NSQIP-AKI model.

^**p-value < 0.05 for difference in AUC (95% CI) between the models fit to training cohort and validation cohort.

Discussion

When applied to a large single-center cohort of postoperative patients, the ACS NSQIP definition for AKI underestimates the prevalence of AKI defined by the consensus RIFLE criteria as it captures only 7% of all RIFLE-AKI patients. Importantly, after applying the RIFLE-AKI definition to the 97% of patients not fulfilling NSQIP-AKI criteria, the 35% additional patients identified with RIFLE-AKI had a fivefold increase in mortality rates compared to patients with no RIFLE-AKI with 80% of all 90-days postoperative deaths occurring among these patients. Hence the risk associated with less severe and more prevalent stages of AKI is not accounted for in the current ACS NSQIP database. In addition, for the first time our study demonstrates that the risk-adjusted association between postoperative change in sCr and various adverse clinical outcomes is continuous and observed at even lower cut-offs and in this cohort validates the new KDIGO-AKI definition that expands the RIFLE criteria to include changes in sCr as low as 0.3 mg/dl (10).

The ACS NSQIP is a prospective and validated database that quantifies 30-day risk-adjusted surgical outcomes for patients undergoing major surgical procedures using sampling performed by a trained clinical reviewer applying predefined criteria (25). It has its roots in the Veterans Affairs National Surgical Quality Improvement Program established in the 1990s and has proven to be an important tool for comparative measurement of the quality of surgical care leading to improvement of surgical outcomes (25-28). The ACS NSQIP defines postoperative AKI as “progressive renal insufficiency” defined by a rise in sCr of >2 mg/dl from a preoperative value with no requirement for RRT” or as “acute renal failure” defined by the acute postoperative requirement for RRT, both within the 30 days of the operation (13). In contrast, the RIFLE defines AKI as a change in sCr greater than or equal to 50% from the RsCr (9). For patients with normal kidney function preoperatively, a 50% increase from reference sCr would result in a change in sCr in the range of 0.3 to 0.5 mg/dl only. For the majority of elderly patients and women whose reference sCr is expected to be less than or equal to 0.8 mg /dl, even a threefold increase in sCr after surgery might still be below the NSQIP cut-off of 2 mg/dl.

Prior to the introduction of the RIFLE consensus criteria in 2004, the reported prevalence of any hospital-acquired AKI, including perioperative AKI, varied significantly from 1 to 31% reflecting the incoherent criteria used to define AKI (29, 30). Since then RIFLE has been validated in studies involving more than 500,000 patients from different patient populations and an analysis of pooled data demonstrates the stepwise increase in relative risk for hospital and long-term death with increasing AKI severity compared to non-AKI patients (2, 6, 12). Only a fraction of the studies using RIFLE-AKI have included perioperative patients, often without clear discrimination between those who required intensive care unit admission and those who did not (31, 32). In addition, the prevalence of RIFLE-AKI has been largely dependent on the type of surgery and varies from 25% for trauma patients (12) to 50% for patients undergoing aortic surgeries or liver transplant (32-34) and is consistent with our report. Nonetheless, the prevalence of RIFLE-AKI is significantly higher than the reported prevalence of AKI defined by the guidelines of the major surgical societies (35, 36) or public surgical databases like ACS NSQIP (12, 37).

Not surprisingly, studies utilizing the ACS NSQIP dataset to describe the epidemiology of postoperative AKI suffer the limitation of this definition, and provide a limited picture of AKI as a serious but rare complication of surgery. Among 152 244 operations in the 2005–2006 ACS NSQIP dataset, Keheterpal et al reported an AKI prevalence of 1% with an eightfold increase in 30-day mortality (14). In a cohort of 84,730 patients from the ACS NSQIP dataset who underwent inpatient general and vascular surgery, Ghaferi et al reported an AKI prevalence between 1.2% - 1.7% (38). Our study, in contrast to these reports, demonstrates that when properly defined AKI is one of the most prevalent perioperative complications associated with other adverse postoperative events, an increase in resource utilization, and both short and long term mortality (2, 6, 12, 39). This is true not only for the most severe AKI that requires RRT but also for the whole spectrum of AKI severity including only small changes in sCr. The most recent KDIGO consensus expanded the RIFLE criteria to include sCr changes as small as 0.3 mg/dl (10). Our work corroborates these findings and extends them to demonstrate that the association between sCr change and adverse postoperative outcomes including mortality occurs through the whole continuum of the postoperative changes in sCr.

As with any other retrospective observational analysis, our study is limited by the fact that causal inference cannot be made and we cannot exclude bias from unmeasured factors although we attempted to control for selection bias with multivariable statistical methods and risk adjustment. The major strength of our study was the ability to use longitudinal changes in sCr to define AKI rather than relying on previously established cut-offs (40). We recognize that the addition of urine output criteria would increase the sensitivity for AKI diagnosis but even sCr criteria in both consensus definitions outperformed NSQIP classification in identifying patients with adverse outcomes. We assessed comorbidities and some of the complications using previously validated criteria (21, 41). This approach relies on accurate coding leading to potential risk underassessment. Although errors and variance in the data may exist, we have assumed that these are randomly distributed and should not lead to significant bias in our conclusions. Although this is a single institution report, it comprises a large cohort of patients with morbidity and mortality that is comparable to that reported in the literature for the same procedures, making it unlikely that the difference in outcomes was due to surgical technique solely. The internal validity of analyses is assured by our use of the methodology recommended by the STROBE guidelines including the use of a validation cohort and sensitivity analyses addressing missing sCr data, selection bias and effect of RsCr (23).

Our analyses addressed the relationship between all-cause sCr change and subsequent adverse events. The etiology of AKI after major surgery is multifactorial and the strong association between AKI and hospital mortality was demonstrated regardless of the etiology of AKI (11). Although the ischemia-reperfusion model has been used to describe postoperative AKI, recent laboratory and clinical evidence suggest that the inflammatory milieu with associated immunological danger signals and G1 cell-cycle arrest in renal tubular cells may be important key factors leading to AKI and perhaps in creating the observed long-term consequences of AKI (42, 43). The strong association between sepsis and AKI has been increasingly recognized and AKI has been recently proposed as a major factor that perpetuates organ dysfunction in sepsis, leading to persistent inflammation and immunosuppression in ICU patients (44-46).

We suggest updating the ACS NSQIP definition of AKI to incorporate the well-validated and internationally accepted consensus RIFLE or KDIGO definitions as has already been done by the Society of Thoracic Surgeons (47). Until the common and false perception of AKI as a rare complication that “patients die with but not of” is changed, there will be no impetus to improve the diagnosis, treatment, and ultimately the prevention of AKI (48). Like surgical infections and sepsis, postoperative AKI is a disease process for which ownership regarding diagnosis and prevention should be in the hands of the intensivists, surgeons and anesthesiologists who are the primary caregivers in the operating rooms, trauma bays and ICUs. The profound physiological changes encountered in surgical patients occur there, and it is in those places that we can prevent and modify this potentially catastrophic complication by affecting processes related to our practices of surgery and surgical critical care.

In conclusion, applying the NSQIP definition for AKI using changes in sCr, in a large single-center cohort of surgical patients, failed to identify 93% of RIFLE-AKI patients who experienced 80% of all 90-day deaths. Furthermore, postoperative sCr changes as low as 0.2 mg/dl or 10% changes from RsCr were associated with increased hospital morbidity and mortality.