Model to Predict Mortality in Critically Ill Adults with Acute Kidney Injury

Abstract

Summary

Background and objectives

Acute kidney injury (AKI) requiring dialysis is associated with high mortality. Most prognostic tools used to describe case complexity and to project patient outcome lack predictive accuracy when applied in patients with AKI. In this study, we developed an AKI-specific predictive model for 60-day mortality and compared the model to the performance of two generic (Sequential Organ Failure Assessment [SOFA] and Acute Physiology and Chronic Health Evaluation II [APACHE II]) scores, and a disease specific (Cleveland Clinic [CCF]) score.

Design, setting, participants, & measurements

Data from 1122 subjects enrolled in the Veterans Affairs/National Institutes of Health Acute Renal Failure Trial Network study; a multicenter randomized trial of intensive versus less intensive renal support in critically ill patients with AKI conducted between November 2003 and July 2007 at 27 VA- and university-affiliated centers.

Results

The 60-day mortality was 53%. Twenty-one independent predictors of 60-day mortality were identified. The logistic regression model exhibited good discrimination, with an area under the receiver operating characteristic (ROC) curve of 0.85 (0.83 to 0.88), and a derived integer risk score yielded a value of 0.80 (0.77 to 0.83). Existing scoring systems, including APACHE II, SOFA, and CCF, when applied to our cohort, showed relatively poor discrimination, reflected by areas under the ROC curve of 0.68 (0.64 to 0.71), 0.69 (0.66 to 0.73), and 0.65 (0.62 to 0.69), respectively.

Conclusions

Our new risk model outperformed existing generic and disease-specific scoring systems in predicting 60-day mortality in critically ill patients with AKI. The current model requires external validation before it can be applied to other patient populations.

Introduction

Acute kidney injury (AKI) requiring dialysis occurs in 1% to 4% of hospitalized patients and, in the setting of multisystem organ failure, is associated with exceptionally high morbidity and mortality (1–3). To date, there is no specific treatment for AKI. Moreover, neither differences in modality (intermittent hemodialysis versus continuous renal replacement therapy [CRRT]) nor intensity (higher Kt/V_urea and/or hemofiltration or hemodiafiltration volume versus lower) of therapy have been shown to favorably influence survival in randomized clinical trials (4–6).

Risk adjustment and mortality prediction are critically important for comparing outcomes across interventions and health systems. Over the past three decades, severity-scoring systems, which capture various combinations of physiologic derangement, organ dysfunction, and pre-existing comorbidity, have been developed and applied to predict outcomes in the intensive care unit (ICU) (7–9). Disease-specific scores have also been developed in critically ill patients with AKI. However, with changes in clinical characteristics of hospitalized patients with AKI, updated or new models are required to apply to contemporary cohorts (10–12).

The Veterans Affairs/National Institutes of Health (VA/NIH) Acute Renal Failure Trial Network (ATN) study (ClinicalTrials.gov, NCT00076219) was a multicenter randomized clinical trial of intensive versus less intensive renal support in critically ill patients with AKI conducted between November 2003 and July 2007 at 27 VA- and university-affiliated medical centers (5,13). Using data from all randomized patients, we aimed to develop a new AKI-specific predictive model and to examine the relative performance of commonly used generic and AKI-specific severity-of-illness scoring systems.

Study Population and Methods

Study Participants

The ATN study enrolled 1124 adults in critical care units who had AKI attributable to acute tubular necrosis, as well as sepsis or at least one additional nonrenal organ system failure. Specific inclusion and exclusion criteria and a detailed description of the study cohort are available elsewhere (5,13). Briefly, subjects with a clinical diagnosis of AKI requiring dialysis due to acute tubular necrosis were eligible for inclusion in the ATN study; patients with chronic kidney disease (defined as premorbid serum creatinine >2 mg/dl in men and >1.5 mg/dl in women) or prior kidney transplantation were excluded.

Explanatory Variables

In developing the new predictive model, we initially considered 117 clinically relevant variables, including demographic, clinical, and laboratory information at time of initiation of renal replacement therapy (RRT), corresponding to the time of entry into the ATN study. Variables that captured redundant physiologic parameters, had prevalence rates below 5%, or were treatment related were excluded; out of the remaining 53 candidate variables, 49 were missing data in fewer than 5% of subjects, and the remaining four variables were missing data in fewer than 15%. Two subjects were excluded due to missing laboratory data. All missing categorical data were related to medical history. Missing data were more common among patients consented by surrogate. Prior studies have demonstrated that when elements of medical history are unknown by surrogates, and there is no record of a condition being present, the condition is usually absent. For this reason, we viewed these missing data elements as informative and were imputed as “No,” with a probability of 0.9, and “Yes,” with a probability of 0.1. Missing data for continuous variables were imputed using regression-based maximum likelihood methods (14).

Model Building

We summarized demographic, clinical, and laboratory data as mean (SD) for continuous variables and frequency for discrete variables. We applied univariate logistic regression to each variable for the initial model selection with 60-day all-cause mortality as the dependent variable (589/1122), and subsequently evaluated for linear, quadratic, and cubic fit. Variables with P ≤ 0.1 on univariate analysis, or that were selected a priori based on clinical grounds, were included in the final multivariable logistic regression model. We retained variables in the multivariable model using backward selection with P ≤ 0.05. We a priori evaluated several multiplicative interactions based on clinical grounds. The coefficients from the fitted logistic regression model were used to develop an interactive Web-based tool to calculate the probability of 60-day mortality automatically.

Risk Score Construction

We also constructed a paper-based scoring system (integer risk score), derived from the above-mentioned risk model, to facilitate manual calculation. We chose a constant that, when multiplied by the parameter estimates of categorical variables in the model, would result in integers (score points) and minimize residuals. Subsequently, we multiplied coefficients of continuous variables by the constant, which then represents the increase in risk score for each unit increase of an individual continuous variable.

Model Performance

We assessed model discrimination using the area under the receiver operating characteristic (ROC) curve, the concordance index (C-statistic). We applied a bootstrap procedure (500 draws with replacement) to obtain parameter estimates for the C-statistic. Model optimism was subsequently assessed by comparing the bootstrap adjusted and the original model C-statistic (15). We assessed model calibration using the Hosmer–Lemeshow goodness-of-fit chi-squared test. To compare results of the new model to older generic and disease-specific scores, we compared model performance metrics for the Sequential Organ Failure Assessment (SOFA), Acute Physiology and Chronic Health Evaluation II (APACHE II), and Cleveland Clinic Foundation (CCF) scores applied to ATN study database (7–10). Model comparisons were carried out using nonparametric method proposed by DeLong (16).

Results

Adequate data were available on 1122 (99%) randomized subjects, with mean age 59.7 ± 15.3 years; 70.6% were male; 74.4% were Caucasian. The mean baseline serum creatinine concentration (sCr) before developing AKI was 1.1 ± 0.4 mg/dl. The main contributing etiologic factors for AKI were ischemia in 871 (81%), nephrotoxic agents in 286 (28%), and sepsis in 579 (55%); AKI was multifactorial in 626 (59%) subjects. Three hundred fifty-five (32%) subjects were treated with intermittent hemodialysis and 746 (66%) with either continuous veno-venous hemodiafiltration or sustained low-efficiency hemodialysis as their initial dialytic modality; 16 (1.4%) patients never received RRT. The overall ICU, in-hospital, and 60-day mortality rates were 44%, 50%, and 53%, respectively. Table 1 outlines the basic demographic and clinical characteristics at time of initiation of RRT.

Table 1.

Demographics and clinical characteristics on day of initial renal replacement therapy

Parameter	All Patients (n = 1124)
Age (years)	59.7 ± 15.3
Male	793 (71)
Race/ethnicity
Caucasian	835 (74)
African American	179 (16)
Hispanic	77 (7)
other	16 (3)
Premorbid renal function
serum creatinine (mg/dl)	1.1 ± 0.4
eGFR (ml/min per 1.73 m2)a
≥60	686 (61)
45 to 59	237 (21)
30 to 44	124 (11)
Etiology of AKI
ischemic	871 (77)
nephrotoxic	286 (25)
multifactorial	626 (56)
Postsurgery	463 (41)
Sepsis	579 (52)
Mechanical ventilation	905 (81)
Oliguria	877 (78)
SOFA score
respiratory	2.3 ± 1.1
coagulation	1.4 ± 1.2
liver	1.4 ± 1.3
cardiovascular	2.3 ± 1.7
central nervous system	2.5 ± 1.4
totalb	13.8 ± 3.9
Cleveland Clinic Score	12.1 ± 3.4
APACHE II	26.4 ± 7.3

Continuous data are presented as mean ± SD, and categorical data as number of patients (%). eGFR, estimated glomerular filtration rate; SOFA, Sequential Organ Failure Assessment; APACHE II, Acute Physiology and Chronic Health Evaluation; AKI, acute kidney injury; BUN, blood urea nitrogen.

^aeGFR was calculated using four-variable MDRD estimation equation.

^bAll patients assigned score of 4 for renal component of SOFA score.

Predictive Risk Model

We identified 21 nonredundant factors independently associated with 60-day all-cause mortality (Table 2). The bootstrap estimated C-statistics of the logistic regression model was 0.85 (0.83 to 0.88) and was not statistically different from the original model (P-value = 0.32). The model also had good calibration, as shown in Figure 1 (Hosmer–Lemeshow goodness-of-fit P = 0.76). From an a priori selection of multiplicative interactions, only the interaction of mechanical ventilation with high fraction of inspired oxygen requirement was statistically significant (P = 0.001) and, hence, included in the model. Patient location before index hospitalization and ICU length-of-stay before RRT initiation were independently associated with mortality; however, these were excluded from the final model due to concerns about the influence of local practice patterns and trial-related selection bias that might compromise external validity. Using the coefficients from the fitted model, we developed a computer-based calculator (http://rcc.simpal.com/RCEval.cgi?RCID=0iaLo3) for automated calculation of the probability of mortality within 60 days.

Table 2.

Predictive risk model for 60-day mortality in critically ill patients with AKI requiring dialysis^a

Parameter	Coefficient	OR	P-value
Chronic health condition
Age (per year)	0.04	1.04	<0.01
Chronic hypoxemia	0.73	2.07	0.07
Cardiovascular diseaseb	0.27	1.31	0.03
Malignancyc	0.99	2.68	<0.01
Immunosuppressive therapyd	−0.44	0.65	0.02
Present illness
Ischemic AKI	0.16	1.17	<0.05
Postsurgery	−0.45	0.64	0.08
At RRT initiation
Heart rate (beats/min)e	0.01	1.007	<0.03
Mean arterial pressure (mmHg)e	−0.02	0.983	<0.02
Urine volume (mL/day)e,f	−0.14	0.867	0.02
Mechanical ventilation	1.03	2.793	<0.01
FiO2≥0.6e	3.07	21.6	<0.01
Interaction of mechanical ventilation & FiO2 ≥0.60g	−1.53	0.216	<0.01
Arterial pHe	−2.64	0.07	0.06
Arterial oxygen partial pressure (mmHg)e,f	−0.66	0.52	<0.05
Serum creatinine (mg/dl)	−0.18	0.834	0.01
Serum bicarbonate (mmol/L)e	−0.03	0.968	0.02
Serum phosphate (mg/dl)e	0.02	1.05	0.05
Serum albumin (g/dl)e	−0.34	0.710	0.02
Total bilirubin (mg/dl)f	0.36	1.439	<0.01
INRe,f	0.54	1.719	0.01
Platelet count (k/μl)f	−0.36	0.695	<0.01

AKI, acute kidney injury; RRT, renal replacement therapy; FiO₂, fraction of inspired oxygen; INR, international normalized ratio; OR, odds ratio; CI, confidence interval.

^aMultivariable logistic regression was used to derive model coefficients.

^bHistory of angina, documented myocardial infarction, or congestive heart failure.

^cSolid tumor with or without metastases, leukemia, or lymphoma.

^dPositive HIV status, AIDS, nonrenal organ transplantation, or immunosuppressive therapy.

^eWithin 24 hours of RRT initiation.

^fLog-transformed variable.

^gDue to negative interaction, the risk of mortality in a mechanically ventilated subject with FiO₂ ≥0.60 is lower compared to a non-ventilated subject with FiO₂ ≥0.60, and higher compared to a ventilated subject with FiO₂ <0.60.

Figure 1.

Calibration of risk model for 60-day mortality. Horizontal axis represents actual observed 60-day mortality, whereas vertical axis is the predicted probability using the multivariable logistic regression model. Broken line indicates the line of identity (ideal prediction model). Circles represent subgroups of our cohort.

Integer Risk Score

To facilitate manual calculation of 60-day mortality risk, we mapped the coefficients from the final risk model to integer scores for the 21 included parameters (Table 3). The risk score is calculated as the arithmetic sum of the points for each of these parameters (range 0 to 59 points). Figure 2 illustrates the relation between the total sum of score points for an individual subject and probability of 60-day all-cause mortality. The integer risk score had slightly lower discriminative capacity compared with the nonintegerized risk model, with an area under the ROC of 0.80 compared with 0.85 (Figure 3). An elevated oxygen requirement (fraction of inspired oxygen [FiO₂] ≥0.60) was present in 488 (43%) of patients at initiation of RRT and contributed the greatest weight to the point score.

Table 3.

Integer risk score for prediction of 60-day mortality in critically ill patients with AKI requiring dialysis^a

Parameters	Score Points
	0	1	2	3	4	5	12	14
Chronic health condition
age	≤50	51 to 56	57 to 62	63 to 68	69 to 74	>74
chronic hypoxemia	No			Yes
cardiovascular diseaseb	No	Yes
malignancyc	No					Yes
immunosuppressive therapyd	Yes		No
Present illness
ischemic AKI	No	Yes
postsurgery	Yes		No
At RRT initiation
heart rate (beats/min)e	≤96	97 to 126	>126
mean arterial pressure (mmHg)e	≥86	73 to 85	61 to 73	<61
urine volume (ml/day)e	≤501	500 to 151	150 to 41	<41
Mechanical ventilation and FiO2:
mechanical ventilation and FiO2 <0.60e	No					Yes
no Mechanical ventilation and FiO2 ≥0.60e	No							Yes
mechanical ventilation and FiO2 ≥0.60e	No						Yes
arterial pHe	>7.3	7.2 to 7.3	<7.2
arterial oxygen partial pressure (mmHg)e	>96	≤96
serum creatinine (mg/dl)	>4.7	3.7 to 4.7	<3.7
serum bicarbonate (mmol/L)e	>25	18 to 25	<18
serum phosphate (mg/dl)e	≤3	>3
serum albumin (g/dl)e	≥2.5	2.4 to 1.9	<1.9
total bilirubin (mg/dl)	≤1	1 to 1.9	1.9–3.7	>3.7
INRe	≤1.2	1.3 to 1.8	>1.8
platelet count (k/μl)	≥165	165 to 91	90 to 50	<50

AKI, acute kidney injury; FiO₂, fraction of inspired oxygen; INR, international normalized ratio; RRT, renal replacement therapy.

^aRisk score minimum (0), maximum (59); total score is calculated as the arithmetic sum of the points for the individual parameters.

^bHistory of angina, documented myocardial infarction, or congestive heart failure.

^cSolid tumor with or without metastases, leukemia, or lymphoma.

^dPositive HIV status, AIDS, nonrenal organ transplantation, or immunosuppressive therapy.

^eWithin 24 hours of RRT initiation.

Figure 2.

Predicted versus observed 60-day all-cause mortality per integer risk score. Black, predicted mortality; gray, observed mortality.

Figure 3.

Area under receiver operating characteristic (ROC) curves for the new risk score (0.85), integer risk score (0.80), APACHE II (0.68), total SOFA (0.69), and Cleveland Clinic score (0.65). APACHE, Acute Physiology and Chronic Health Evaluation; SOFA, Sequential Organ Failure Assessment. Green, risk model; black, integer risk score; red, SOFA score; yellow, APACHE II score; blue: Cleveland Clinic score.

Comparison with Previous Predictive Models

Figure 3 shows the ROC curves for our new risk-prediction model, as well as for APACHE II, SOFA, and CCF scores. The area under the ROC curve was 0.68 (0.64 to 0.71) for the APACHE II, 0.69 (0.66 to 0.73) for the SOFA, and 0.65 (0.62 to 0.69) for the CCF scores, as compared with values of 0.85 (0.83 to 0.88) for the nonintegerized risk model and 0.80 (0.77 to 0.83) for the integerized risk score. Both areas under the ROC curve values (0.80 and 0.85) were statistically better compared with the three previous models (P < 0.001). Higher values of both generic severity illness scores and the CCF score were associated with 60-day mortality (Table 1). Of note, the APACHE II score is validated for use at ICU admission (7), whereas the SOFA score is validated for sequential use each ICU day (9), and the CCF score (10), as the current model, was developed for use at initiation of RRT in the ICU.

Discussion

Using a large multicenter cohort of critically ill patients with AKI who required RRT we have developed a multivariable risk model and a scoring system for predicting 60-day mortality. Our new risk model and scoring system demonstrated good discrimination and calibration, and outperformed previously published generic and AKI-specific scoring systems.

We examined the performance of two widely used generic severity-of-illness scoring systems: the APACHE II score, which integrates components of both acute and chronic organ dysfunction, and the SOFA score, which assesses the presence and severity of organ dysfunction. The application of generic severity scores in specialized cohorts, such as patients with AKI, has been problematic not only because of relatively poor discrimination but also because of suboptimal calibration (11,12,17). The use of external benchmarks—score systems developed in a population with different clinical characteristics from the target population—may underperform for several reasons. For instance, databases used to build general severity scores consist of patients with ICU length of stay less than 3 days, whereas our cohort's average predialysis ICU stay was almost 1 week (18). Another pitfall is the choice of in-hospital mortality as the primary outcome; hospital or long-term care facility transfer practices and short follow-up times may underestimate the strength of the association with mortality (19). These limitations may be particularly relevant in patients with AKI, where RRT and/or recovery span over weeks rather than days (2). Moreover, patients with severe illness typically have an eventful ICU course, where the relation between a risk factor determined at baseline and outcome is likely to be nonlinear, may change over time, and/or reverse direction. For example, sCr and urine output are established components of various risk scores, as it is in AKI-specific models—including our new score; however, the correlation is positive (or U shaped) in generic severity scores (7,9) but reverses direction in AKI-specific scores (10,12). Rather than being a marker for the extent of impaired kidney function, in a population where all subjects have severe AKI, a lower sCr might reflect underlying inflammation and/or malnutrition, hypervolemia with dilution, and/or other complications in the setting of multisystem organ failure (12,20,21).

The performance of AKI-specific scoring systems have been comparable to generic severity of illness scores (11,12). Many of the AKI-specific scores are obsolete, or at least outdated, due to the inevitable change in patient characteristics, disease epidemiology, medical technology, and standards of care over time. Although these limitations could be partially remedied via recalibration or structural model revisions (adding new variables) (22), we took advantage of a large, heterogeneous, multicenter patient cohort, namely, patents with AKI requiring RRT enrolled in the ATN study, to develop a new model. The large sample size ensures adequate patient numbers in high-mortality risk groups and better accuracy. In addition, the timing of the score application at initiation of RRT has previously been shown to improve predictive capacity compared with time of onset of AKI or renal consultation (12,20). We also favored the use of physiologic parameters in our model, rather than treatment- or diagnosis-based variables. The former are likely to be more objective and less prone to variations in practice patterns in time and space, hence potentially improving external validity.

The new score contains factors reflecting chronic health, such as chronological age, underlying respiratory disease, cardiovascular disease, and malignancy. The absence of baseline kidney function as an independent predictor of outcome has been noted in previous reports; however, the present cohort excluded most patients with moderate to severe chronic kidney disease, limiting our ability to assess the impact of impaired baseline renal function on outcomes (12,20,23). Respiratory complications, reflected by a requirement for a high FiO₂ and the need for mechanical ventilation, had the largest combined point allocation. This rather large impact attributed to pulmonary organ failure is likely partially confounded by volume-driven morbidity, which is highly prevalent in subjects with renal failure. Consistent with previous literature, postoperative status (i.e., “surgical” rather than “medical” AKI) was associated with a more favorable prognosis, most likely because patients with the most severe chronic conditions are generally not considered as candidates for elective surgery (selection bias) (20,24). The same selection bias also applies to the subgroup of subjects who used immunosuppressive medications, which included nonrenal solid organ transplant recipients and connective tissue disease, in contrast to subjects with compromised immune system due to malignancy.

This study has several important strengths. The sample was derived from many sites across the United States and was diverse in terms of age, gender, race/ethnicity, and cause of AKI. We carefully incorporated objective clinical variables that were tracked as part of a randomized clinical trial; data entry was checked for accuracy. We bootstrapped the sample to derive parameter estimates that would be less likely to be overfit to the derivation sample. The prediction accuracy achieved by both the risk model and the scoring system suggests strong internal validity of our results. The derivation of the paper-based scoring system from the new score model was accompanied by a notable loss in discriminative capacity. The latter compromise in predictive accuracy could be circumvented by using the Web-based calculator for the risk model. Finally, we compared the current risk model and scoring system to two generic and one disease-specific predictive models and found superior results with the current model.

There are several important limitations to the study presented here. Although the data were generated from 27 centers, recruitment was predominantly in tertiary-care academic medical centers. The participation of non-VA centers, on the other hand, provided better representation of non-Caucasian and female subjects. While we would anticipate excellent model performance if applied to nontertiary care hospitals, the distribution of health conditions in tertiary care hospitals is different than in smaller hospitals or in hospitals to which patients are less likely referred. However, it is noteworthy that none of the variables selected for inclusion in the multivariable model were unique to tertiary care hospitals (e.g., recipients of bone marrow transplantation or extracorporeal membrane oxygenation). The new scoring system was developed specifically for critically ill patients with AKI requiring RRT, and external validity is further limited by the inclusion/exclusion criteria of the ATN study. Patients with moderate to advanced baseline chronic kidney disease were excluded. However, only a small percentage (7.6%) of subjects were excluded based on that particular reason, and 39% had estimated GFR of less than 60 ml/min per 1.73 m². AKI due to causes other than ATN were excluded; therefore, the model would not be expected to perform well among patients with other causes of AKI (e.g., prerenal azotemia, urinary tract obstruction, acute glomerulonephritis, or tubulointerstitial nephritis). Certain data elements were excluded due to a relatively high percentage of patients with missing measurements; one such variable is serum lactate, which has been shown to be a robust predictor of mortality in the critically ill (11). The combination of very low serum bicarbonate and acidemia included in our score is likely to capture most of the variation related to elevated serum lactate concentrations. In the ATN study, 724 patients had one hemodialysis treatment or up to 24 hours of CRRT before randomization. The majority of data used for developing our score were collected before any prerandomization RRT; however, few parameters used in our score, such as serum bicarbonate and phosphate, and urine output, may have been collected 24 hours into prerandomization therapy. It is unlikely, however, that any parameter that is rapidly modified by RRT will influence mortality at 60 days. This limitation is also inherent in many generic severity-of-illness scores, including the APACHE II and III scores, whose points are based strongly on physiologic (and to a lesser extent laboratory) variables that can often be remedied within the first ICU day.

In summary, we developed a novel risk model and a scoring system that accurately predicts 60-day all-cause mortality in critically ill patients with AKI requiring RRT. The model can be used for risk adjustment and possibly to stratify prospective subjects in upcoming clinical trials of AKI requiring RRT. Risk-prediction models always demonstrate their best calibration in the population in which they are generated; further validation of the model in other cohorts of patients with AKI will be required before it can be widely applied to clinical practice.

Disclosures

None.