Abstract
Rationale & Objective
Risk factors for acute kidney injury (AKI) in the hospital have been well studied. Yet, risk factors for identifying high-risk patients for AKI occurring and managed in the outpatient setting are unknown and may differ.
Study Design
Predictive model development and external validation using observational electronic health record data.
Setting & Participants
Patients aged 18-90 years with recurrent primary care encounters, known baseline serum creatinine, and creatinine measured during an 18-month outcome period without established advanced kidney disease.
New Predictors & Established Predictors
Established predictors for inpatient AKI were considered. Potential new predictors were hospitalization history, smoking, serum potassium levels, and prior outpatient AKI.
Outcomes
A ≥50% increase in the creatinine level above a moving baseline of the recent measurement(s) without a hospital admission within 7 days defined outpatient AKI.
Analytical Approach
Logistic regression with bootstrap sampling for backward stepwise covariate elimination was used. The model was then transformed into 2 binary tests: one identifying high-risk patients for research and another identifying patients for additional clinical monitoring or intervention.
Results
Outpatient AKI was observed in 4,611 (3.0%) and 115,744 (2.4%) patients in the development and validation cohorts, respectively. The model, with 18 variables and 3 interaction terms, produced C statistics of 0.717 (95% CI, 0.710-0.725) and 0.722 (95% CI, 0.720-0.723) in the development and validation cohorts, respectively. The research test, identifying the 5.2% most at-risk patients in the validation cohort, had a sensitivity of 0.210 (95% CI, 0.208-0.213) and specificity of 0.952 (95% CI, 0.951-0.952). The clinical test, identifying the 20% most at-risk patients, had a sensitivity of 0.494 (95% CI, 0.491-0.497) and specificity of 0.806 (95% CI, 0.806-0.807).
Limitations
Only surviving patients with measured creatinine levels during a baseline period and outcome period were included.
Conclusions
The outpatient AKI risk prediction model performed well in both the development and validation cohorts in both continuous and binary forms.
Index Words: acute kidney injury, ambulatory, renal failure, risk prediction, outpatient
Graphical abstract
Plain-Language Summary.
We used electronic health record data from a multisite health care system to develop a predictive model for acute kidney injury (AKI) occurring and managed in the outpatient setting, or outpatient AKI, over an 18-month period among a primary care–receiving population. The continuous form of the model was transformed into 2 separate binary forms for 2 distinct purposes: identifying high-risk patients for closer clinical monitoring or for research in outpatient AKI. The continuous model was subsequently externally validated using national electronic health record data from the Veterans Health Administration and showed good performance. The performance of the binary tests in the validation cohort was also described. This model, in its various forms, may be used to identify high-risk patients for outpatient AKI.
Acute kidney injury (AKI) occurs frequently among hospitalized patients,1 including as many as half of critically ill patients.2 AKI also occurs in the ambulatory setting, with only a subset of patients subsequently admitted to a hospital.3, 4, 5, 6, 7, 8 Regardless of the setting, patients with AKI are at an increased risk of adverse outcomes, including the progression of chronic kidney disease (CKD), end-stage kidney disease, cardiovascular disease, and death.8, 9, 10, 11, 12, 13, 14 Among hospitalized patients, the risk factors for AKI have become increasingly well recognized, and prediction models for patients in the hospital have been published.15, 16, 17, 18, 19, 20, 21 Yet, there has been no such examination of the predictability of AKI in the ambulatory setting or support tool developed to assess AKI risk in outpatients. A substantial proportion of AKI cases may occur in the outpatient setting and be managed by outpatient providers,6,8 and although the short-term mortality risk is lower, the long-term mortality risk appears to be similar for hospitalized patients with AKI and those not hospitalized at the time of AKI after statistical adjustment.7 Therefore, identifying patients at high risk for outpatient AKI is important. This may lead to improved pre- or post-AKI care in the ambulatory setting and research in preventive interventions in this under-recognized group.
We sought to develop and externally validate a risk prediction model for AKI occurring and managed in the outpatient setting using covariates widely available in electronic health records (EHRs). Such a predictive model could be used to both define and subsequently identify “high-risk” patients for clinical interventions (eg, avoidance of nonsteroidal anti-inflammatory drugs) or recruitment into clinical trials.
Methods
Study Population
A model-development cohort was derived from the M Health Fairview health system (of the greater Minneapolis, Minnesota area), beginning with its current EHR adoption November 29, 2000, through June 1, 2018. A baseline period was defined by the receipt of at least 2 primary care encounters over at least 18 months (to allow both an adequate number of encounters and time to document comorbid conditions and obtain laboratory and vital sign measurements). All baseline periods ended by December 1, 2016, to allow the completion of a subsequent 18-month outcome period for all participants.
An external validation cohort was derived from national Veterans Affairs (VA) data via VA Informatics and Computing Infrastructure. Owing to a longer history of VA EHR use, the validation cohort was limited to patients with a second qualifying primary care encounter after December 31, 2009. Data were accessed through January 1, 2019 (with all baseline periods ending by July 1, 2017).
Inclusion criteria included at least one outpatient-measured serum creatinine value before the end of the baseline period and also at least one from the outcome period. Participants were aged 18-90 years and without baseline end-stage kidney disease, kidney transplantation, or CKD stage 5 (Figs 1 and 2).
Figure 1.
Patient study eligibility flow diagram for the development cohort, derived from the M Health Fairview health care system electronic health record. Abbreviations: AKI, acute kidney injury; eGFR, estimated glomerular filtration rate.
Figure 2.
Patient study eligibility flow diagram for the validation cohort, derived from the Veterans Affairs health care system electronic health record. Abbreviations: AKI, acute kidney injury; eGFR, estimated glomerular filtration rate.
Baseline Characteristics
Baseline laboratory- and vital signs–based covariates were defined by the final measurement during the baseline period. The baseline estimated glomerular filtration rate (eGFR) and proteinuria have been identified as independent risk factors for inpatient AKI.20,22 For this study, eGFR was treated as a continuous variable and independently calculated by the Modification of Diet in Renal Disease Study equation23 using the “175” coefficient.24 The Modification of Diet in Renal Disease Study equation was selected as it is used in the VA EHR,25,26 whereas the eGFR equation used by M Health Fairview was not consistent across the study timeframe.
To avoid widespread missing or old data, proteinuria was defined as the most recent spot albuminuria, total proteinuria, or semiquantitative urinalysis-albumin concentration. In the event of concurrent measurements or uninterpretable results, spot albuminuria was the first choice (because of precalculated ratios) and urinalysis was last. “Severe” proteinuria was defined as spot albuminuria or total proteinuria of ≥300 mg/g or urinalysis-albumin concentration of “≥300” or equivalent.27 “Moderate” proteinuria was 30-300 mg/g or urinalysis-albumin concentration of “30-300” or equivalent. Proteinuria below these thresholds was considered “normal.”
Owing to assumed nonrandomness in missing data, binary “missingness” covariates were introduced to coincide with laboratory measurements and vital signs.8,21 This allowed modeling AKI risk on both laboratory measurements/vital signs and whether these covariates had been measured. When missing, the original covariate was entered as zero. Occasional nonphysiologic values were treated as missing (Table S1). Laboratory measurements and vital signs were included according to the risk factors identified in inpatient AKI models: serum sodium, albumin, hemoglobin A1c,20 calcium,19 hemoglobin, and systolic and diastolic blood pressures (BPs).18 Serum potassium level was also included because of widespread availability on metabolic panels.
AKI was divided into 2 mutually exclusive categories because of potential differences between inpatient and outpatient AKIs (Fig S1).7 Cases of AKI that developed within 7 days preceding hospitalization, the so-called community-acquired AKI,4, 5, 6, 7,28 were categorized with cases of hospital-acquired AKI as “inpatient AKI.” This was done because of the similar mortality risk observed by others after either community-acquired AKI and hospital-acquired AKI and a dissimilar mortality pattern after AKI without hospitalization.7 To avoid missing cases of inpatient AKI that developed before admission, inpatient AKI was defined as an increase in inpatient-measured creatinine level of ≥50% above the most recent outpatient-measured creatinine level from 7-365 days before admission or, if none existed, the most recent creatinine level from any setting more than 365 days prior. Baseline inpatient AKI was defined before the end of the baseline period.
Outpatient AKI was defined by a ≥50% increase in outpatient-measured creatinine level compared with a moving baseline. The moving baseline creatinine level was defined as the average of the 3 most recent outpatient-measured creatinine values 25-365 days prior. If no baseline outpatient creatinine was available during that window, the most recent creatinine measurement >365 days prior was used. Creatinine values between 0 and 25 days prior were not included in the calculation of the moving baseline to avoid missing cases of slowly peaking AKI (which otherwise would result in the inclusion of increasing creatinine measurements in the moving baseline). This was similar to prior definitions for AKI originating in the outpatient setting.5,8 A creatinine increase of ≥0.3 mg/dL over 48 hours or ≥50% over ≤7 days29,30 was not used as outpatient laboratory measurements are rarely obtained with that frequency. Outpatient AKI was defined as a baseline risk factor when occurring before the end of the baseline period.
Other comorbid diseases were defined by ≥2 International Classification of Diseases-9 and 10 and/or Current Procedural Terminology billing codes. These included diabetes mellitus, hypertension, cardiovascular disease (defined as the composite of coronary artery disease, peripheral vascular disease, congestive heart failure, or stroke), hypertension, cancer, and liver disease. The race was defined as “Black” if ever documented as Black in the EHR. Smoking status was defined as “ever smoker” or “never smoker.” The hospitalization history was defined by any hospitalization during which serum creatinine level was measured to exclude nonmedical/surgical hospitalizations.
Study Outcome
The study outcome was outpatient AKI occurring during a fixed 18-month outcome period starting at the end of the baseline period. The same definition for outpatient AKI was again used. An 18-month outcome period was chosen to allow the inclusion of patients seeking care as infrequently as yearly (plus an additional 6 months for allowable tardiness), as lacking at least one outcome period creatinine measurement was a study exclusion criterion.
Statistical Analyses
Logistic regression was used to assess the risk of outpatient AKI during the outcome period. Restricted cubic splines were initially applied to all continuous variables to assess nonlinearity. Nonlinear and interaction terms specified a priori were included only in the subsequent variable selection if both were statistically significantly associated with outpatient AKI and if meaningful nonlinear fitting or effect modification of AKI risk was observed on visual inspection.
The final model was developed using backward stepwise elimination of covariates with N = 1,000 bootstrap sampling based on the Akaike information criterion. If either a continuous or paired missingness covariate was selected to remain in the model, the pair was forced into the model. The risk equation was subsequently applied to the validation cohort and assessed by C statistic for discrimination and Hosmer-Lemeshow goodness-of-fit test and visually assessed by decile of predicted AKI risk.
To demonstrate the clinical significance of predicting outpatient AKI, time-to-mortality after the outpatient AKI period was determined in the subsets of patients in the highest and lowest deciles of predicted AKI risk. Death was defined in the VA EHR. Censoring was defined by the final encounter, vital sign, or laboratory measurement. Crude analyses for mortality predicted by outpatient AKI were performed using the log-rank test. Unadjusted and adjusted hazard ratios were calculated using Cox proportional hazards modeling. Covariates for adjustment were age, sex, race, smoking history, eGFR category (eGFR of ≥60, 45-59, 30-44, or 15-29 mL/min per 1.73 m2),31 baseline inpatient and outpatient AKI history, cardiovascular disease, diabetes mellitus, hypertension, liver disease, cancer, and hospitalization history.
The original model was subsequently transformed into 2 binary tests based on potential uses: (1) identification of high-risk research patients or (2) identification of patients warranting additional clinical monitoring or interventions. Each positive-negative cut-point in the continuous AKI risk prediction was selected by weighting the utility or cost of true and false positive and negative cases based on perceived risks and benefits.32 The maximum value of the resulting “total utility” was calculated in the development cohort and the optimal cut-point applied to the validation cohort. The research test was defined by setting the utility of true positives at 100 times the cost of false positives to promote an adequate number of AKI outcomes for well-powered research. This was balanced by weighting the utility of true negatives at 10 times the cost of false negatives owing to the risk and financial cost of study inclusion incurred from those not truly at high risk. The clinical test was similarly defined by setting the utility of true positives at 20 times the cost of false positives to promote diagnosing cases of AKI, but without the strict concern for statistical power, and the utility of true negatives at 1/20th the cost of false negatives due to an assumed lower risk and cost incurred from unnecessary clinical monitoring than a research study.
The Institutional Review Boards of the University of Minnesota and Minneapolis VA approved this study (applications 1502M63126 and 4739-B, respectively). Informed consent was waived due to deidentified data. Cohorts were defined in SQL (Microsoft). Statistical analyses were performed in R (R Foundation for Statistical Computing).
Results
Cohort Characteristics
There were 152,371 patients in the development cohort, of whom 4,611 experienced outpatient AKI during the 18-month outcome period. In the validation cohort, 115,744 of 4,864,576 patients experienced outpatient AKI. The median ages were 55 and 63 years in the development and validation cohorts, respectively. The median baseline serum creatinine values were similar: 0.9 and 1.0 mg/dL for the development and validation cohorts, respectively. Baseline characteristics and their linear associations with outpatient AKI for both cohorts are shown in Tables 1 and 2. The distribution of “most recent” proteinuria measurements used to define baseline proteinuria is shown in Table S2.
Table 1.
Baseline Characteristics of Model-Development Cohort, Overall and by Outpatient Acute Kidney Injury
| Baseline Characteristic | All Patients (N = 152,371) | With Outpatient AKIa (N = 4,611) | Without Outpatient AKIa (N = 147,760) | P valueb |
|---|---|---|---|---|
| Age, y, median (IQR) | 55.1 (43.1-66.7) | 60.7 (48.3-71.8) | 55.0 (43.0-66.6) | <0.001 |
| Male, N (%) | 69,799 (46%) | 1,789 (39%) | 68,010 (46%) | <0.001 |
| Black, N (%) | 9,185 (6.0%) | 365 (7.9%) | 8,820 (6.0%) | <0.001 |
| Ever smoker, N (%) | 66,388 (44%) | 2,254 (49%) | 64,134 (43%) | <0.001 |
| Final creatinine level of the baseline period, mg/dL, median (IQR) | 0.9 (0.8-1) | 0.9 (0.7-1.1) | 0.9 (0.8-1) | 0.03 |
| eGFRc stage, N (%) | <0.001 | |||
| eGFR, ≥60 | 127,976 (84%) | 3,269 (71%) | 124,707 (84%) | |
| Stage 3A, eGFR, 45-59 | 17,732 (12%) | 779 (17%) | 16,953 (12%) | |
| Stage 3B, eGFR, 30-44 | 5,518 (3.6%) | 401 (8.7%) | 5,117 (3.5%) | |
| Stage 4, eGFR, 15-29 | 1,145 (0.8%) | 162 (3.5%) | 983 (0.7%) | |
| Most recent proteinuria elevated, N (%)d | 12,245 (8.0%) | 715 (16%) | 11,530 (7.8%) | <0.001 |
| Most recent proteinuria not elevated, N (%)d | 78,138 (51%) | 2,402 (52%) | 75,736 (51%) | 0.26 |
| Missing, N (%) | 61,988 (41%) | 1,494 (32%) | 60,494 (41%) | <0.001 |
| Baseline proteinuria is “moderate,” N (%)d | 10,514 (6.9%) | 550 (12%) | 9,964 (6.7%) | <0.001 |
| Baseline proteinuria is “severe,” N (%)d | 1,731 (1.1%) | 165 (3.6%) | 1,566 (1.1%) | <0.001 |
| Baseline history of outpatient AKIe, N (%) | 4,336 (2.8%) | 492 (11%) | 3,844 (2.6%) | <0.001 |
| Baseline history of inpatient AKIf, N (%) | 766 (0.5%) | 98 (2.1%) | 668 (0.5%) | <0.001 |
| Number creatinine values at baseline, median (IQR) | 2 (1-4) | 3 (1-6) | 2 (1-4) | <0.001 |
| Diabetes mellitus, N (%) | 25,696 (17%) | 1,353 (29%) | 24,343 (17%) | <0.001 |
| Hypertension, N (%) | 65,787 (43%) | 2,487 (54%) | 63,300 (43%) | <0.001 |
| Cardiovascular disease, N (%)g | 19,511 (13%) | 1,113 (24%) | 18,398 (13%) | <0.001 |
| Coronary artery disease, N (%)g | 12,833 (8.4%) | 700 (15.2%) | 12,133 (8.2%) | <0.001 |
| Congestive heart failure, N (%)g | 3,951 (2.6%) | 382 (8.3%) | 3,569 (2.4%) | <0.001 |
| Stroke, N (%)g | 4,221 (2.8%) | 224 (4.9%) | 3,997 (2.7%) | <0.001 |
| Peripheral vascular disease, N (%)g | 3,789 (2.5%) | 272 (5.9%) | 3,517 (2.4%) | <0.001 |
| Liver disease, N (%) | 4,758 (3.1%) | 269 (5.8%) | 4,489 (3.0%) | <0.001 |
| Cancer, N (%) | 13,443 (8.8%) | 559 (12%) | 12,884 (8.7%) | <0.001 |
| Systolic blood pressure, mm Hg, median (IQR) | 124 (114-135) | 126 (114-138) | 124 (114-135) | <0.001 |
| Missing, N (%) | 6,166 (4.0%) | 279 (6.1%) | 5,887 (4.0%) | <0.001 |
| Diastolic blood pressure, mm Hg, median (IQR) | 76 (69-82) | 74 (67-82) | 76 (69-82) | <0.001 |
| Missing, N (%) | 6,244 (4.1%) | 281 (6.1%) | 5,963 (4.0%) | <0.001 |
| Sodium level, mmol/L, median (IQR) | 140 (138-142) | 140 (138-142) | 140 (138-142) | <0.001 |
| Missing, N (%) | 5,901 (3.9%) | 142 (3.1%) | 5,759 (3.9%) | 0.005 |
| Potassium level, mmol/L, median (IQR) | 4.1 (3.9-4.4) | 4.1 (3.9-4.5) | 4.1 (3.9-4.4) | 0.29 |
| Missing, N (%) | 3,205 (2.1%) | 95 (2.1%) | 3,110 (2.1%) | 0.84 |
| Calcium level, mg/dL, median (IQR) | 9.2 (9.0-9.5) | 9.2 (8.9-9.5) | 9.2 (9.0-9.5) | <0.001 |
| Missing, N (%) | 7,050 (4.6%) | 191 (4.1%) | 6,859 (4.6%) | 0.11 |
| Hemoglobin, g/dL, median (IQR) | 14.0 (12.9-15.0) | 13.3 (11.9-14.4) | 14.0 (13.0-15.0) | <0.001 |
| Missing, N (%) | 27,023 (18%) | 650 (14%) | 26,373 (18%) | <0.001 |
| Albumin level, g/dL, median (IQR) | 4.1 (3.9-4.4) | 4 (3.7-4.3) | 4.1 (3.9-4.4) | <0.001 |
| Missing, N (%) | 43,216 (28%) | 1,153 (25%) | 42,063 (29%) | <0.001 |
| Hemoglobin A1c in percentage, median (IQR) | 6.1 (5.6-7.0) | 6.5 (5.8-7.6) | 6.1 (5.6-7.0) | <0.001 |
| Missing, N (%) | 108,953 (72%) | 2,770 (60%) | 106,183 (72%) | <0.001 |
| Baseline history of hospitalization, N (%) | 16,222 (11%) | 1,004 (22%) | 15,218 (10%) | <0.001 |
Note: Conversion factors for units: creatinine in mg/dL to μmol/L, ×88.4; calcium in mg/dL to mmol/L, ×0.2495.
Abbreviations: AKI, acute kidney injury; eGFR, estimated glomerular filtration rate; IQR, interquartile range.
Outcome of AKI occurring and managed in the outpatient setting over an 18-month period.
Two sample t test or χ2 test, as applicable, applied to those with outpatient AKI versus those without.
eGFR, by the Modification of Diet in Renal Disease Study equation,23 in mL/min per 1.73 m2.
Proteinuria defined as the most recent of spot albuminuria, spot total proteinuria, or semiquantitative albuminuria by urinalysis, derived from Table 7 of the Kidney Disease Improving Global Outcomes 2012 Clinical Practice Guideline for the Evaluation and Management of Chronic Kidney Disease.27 “Severe” = spot albuminuria or total proteinuria ≥300 mg/g or albuminuria by urinalysis of “300” or equivalent. “Moderate” = spot albuminuria or total proteinuria ≥30 mg/g or albuminuria by urinalysis of “30” or equivalent. “Normal” or not elevated if otherwise measured.
Baseline history of AKI occurring and managed in the outpatient setting.
Baseline history of AKI managed in the inpatient setting regardless of setting of onset.
Cardiovascular disease defined as the composite of coronary artery disease, congestive heart failure, stroke, or peripheral vascular disease as nonmutually exclusive categories.
Table 2.
Baseline Characteristics of Model Validation Cohort, Overall and by Outpatient Acute Kidney Injury
| Baseline Characteristic | All Patients (N = 4,864,576) | With Outpatient AKIa (N = 115,744) | Without Outpatient AKIa (N = 4,748,832) | P valueb |
|---|---|---|---|---|
| Age, y, median (IQR) | 63.3 (52.4-70.2) | 63.9 (56.8-70.6) | 63.2 (52.3-70.2) | <0.001 |
| Male, N (%) | 4,492,178 (92%) | 106,449 (92%) | 4,385,729 (92%) | <0.001 |
| Black, N (%) | 815,167 (17%) | 23,457 (20%) | 791,710 (17%) | <0.001 |
| Ever smoker, N (%) | 3,466,441 (71%) | 90,231 (78%) | 3,376,210 (71%) | <0.001 |
| Final creatinine of the baseline period, mg/dL, median (IQR) | 1.0 (0.9-1.1) | 1.0 (0.8-1.2) | 1.0 (0.9-1.1) | <0.001 |
| eGFRc stage, N (%) | <0.001 | |||
| eGFR, ≥60 | 4,080,191 (84%) | 87,880 (76%) | 3,992,311 (84%) | |
| Stage 3A, eGFR, 45-59 | 572,480 (12%) | 15,938 (14%) | 556,542 (12%) | |
| Stage 3B, eGFR, 30-44 | 177,056 (3.6%) | 8,129 (7.0%) | 168,927 (3.6%) | |
| Stage 4, eGFR, 15-29 | 34,849 (0.7%) | 3,797 (3.3%) | 31,052 (0.7%) | |
| Most recent proteinuria elevatedd, N (%) | 623,912 (13%) | 29,135 (25%) | 594,777 (13%) | <0.001 |
| Most recent proteinuria not elevatedd, N (%) | 3,797,304 (78%) | 77,568 (67%) | 3,719,736 (78%) | <0.001 |
| Missing, N (%) | 443,360 (9.1%) | 9,041 (7.8%) | 434,319 (9.1%) | <0.001 |
| Baseline proteinuria is “moderate”d, N (%) | 569,567 (12%) | 23,851 (21%) | 545,716 (12%) | <0.001 |
| Baseline proteinuria is “severe”d, N (%) | 54,345 (1.1%) | 5,284 (4.6%) | 49,061 (1.0%) | <0.001 |
| Baseline history of outpatient AKIe, N (%) | 322,564 (6.6%) | 20,484 (18%) | 302,080 (6.4%) | <0.001 |
| Baseline history of inpatient AKIf, N (%) | 119,253 (2.5%) | 9,184 (7.9%) | 110,069 (2.3%) | <0.001 |
| Number creatinine values at baseline, median (IQR) | 8 (4-17) | 12 (5-24) | 8 (4-17) | <0.001 |
| Diabetes mellitus, N (%) | 1,311,503 (27%) | 51,945 (45%) | 1,259,558 (27%) | <0.001 |
| Hypertension, N (%) | 3,024,529 (62%) | 90,739 (78%) | 2,933,790 (62%) | <0.001 |
| Cardiovascular diseaseg, N (%) | 1,259,851 (26%) | 45,748 (40%) | 1,214,103 (26%) | <0.001 |
| Coronary artery diseaseg, N (%) | 1,006,004 (21%) | 34,841 (30%) | 971,163 (21%) | <0.001 |
| Congestive heart failureg, N (%) | 221,907 (4.6%) | 14,361 (12%) | 207,546 (4.4%) | <0.001 |
| Strokeg, N (%) | 148,727 (3.1%) | 5,739 (5.0%) | 142,988 (3.0%) | <0.001 |
| Peripheral vascular diseaseg, N (%) | 271,887 (5.6%) | 12,792 (11%) | 259,095 (5.5%) | <0.001 |
| Liver disease, N (%) | 142,127 (2.9%) | 7,114 (6.1%) | 135,013 (2.8%) | <0.001 |
| Cancer, N (%) | 816,634 (17%) | 26,079 (23%) | 790,555 (17%) | <0.001 |
| Systolic blood pressure, mm Hg, median (IQR) | 129 (120-138) | 131 (120-142) | 129 (120-138) | <0.001 |
| Missing, N (%) | 18,296 (0.4%) | 517 (0.4%) | 17,779 (0.4%) | <0.001 |
| Diastolic blood pressure, mm Hg, median (IQR) | 77 (70-84) | 76 (68-84) | 77 (70-84) | <0.001 |
| Missing, N (%) | 21,091 (0.4%) | 593 (0.5%) | 20,498 (0.4%) | <0.001 |
| Sodium level, mmol/L, median (IQR) | 139 (138-141) | 139 (137-141) | 139 (138-141) | <0.001 |
| Missing, N (%) | 11,946 (0.2%) | 325 (0.3%) | 11,621 (0.2%) | 0.01 |
| Potassium level, mmol/L, median (IQR) | 4.2 (4.0-4.5) | 4.2 (3.9-4.5) | 4.2 (4.0-4.5) | <0.001 |
| Missing, N (%) | 33,930 (0.7%) | 816 (0.7%) | 33,114 (0.7%) | 0.76 |
| Calcium level, mg/dL, median (IQR) | 9.3 (9.0-9.6) | 9.3 (9.0-9.6) | 9.3 (9.0-9.6) | <0.001 |
| Missing, N (%) | 259,971 (5.3%) | 4,733 (4.1%) | 255,238 (5.4%) | <0.001 |
| Hemoglobin, g/dL, median (IQR) | 14.5 (13.5-15.4) | 13.8 (12.5-14.9) | 14.5 (13.5-15.4) | <0.001 |
| Missing, N (%) | 77,238 (1.6%) | 1,614 (1.4%) | 75,624 (1.6%) | <0.001 |
| Albumin level, g/dL, median (IQR) | 4.1 (3.8-4.3) | 4.0 (3.6-4.2) | 4.1 (3.8-4.3) | <0.001 |
| Missing, N (%) | 330,596 (6.8%) | 5,750 (5.0%) | 324,846 (6.8%) | <0.001 |
| Hemoglobin A1c in percentage, median (IQR) | 5.9 (5.5-6.5) | 6.1 (5.6-7.2) | 5.9 (5.5-6.5) | <0.001 |
| Missing, N (%) | 1,212,012 (25%) | 20,102 (17%) | 1,191,910 (25%) | <0.001 |
| Baseline history of hospitalization, N (%) | 1,069,398 (22%) | 44,850 (39%) | 1,024,548 (22%) | <0.001 |
Note: Conversion factors for units: creatinine in mg/dL to μmol/L, ×88.4; calcium in mg/dL to mmol/L, ×0.2495.
Abbreviations: AKI, acute kidney injury; eGFR, estimated glomerular filtration rate; IQR, interquartile range.
Outcome of AKI occurring and managed in the outpatient setting over an 18-month period.
Two sample t test or χ2 test, as applicable, applied to those with outpatient AKI versus those without.
eGFR, by the Modification of Diet in Renal Disease Study equation,23 in mL/min per 1.73 m2.
Proteinuria defined as the most recent of spot albuminuria, spot total proteinuria, or semiquantitative albuminuria by urinalysis, derived from Table 7 of the Kidney Disease Improving Global Outcomes 2012 Clinical Practice Guideline for the Evaluation and Management of Chronic Kidney Disease.27 “Severe” = spot albuminuria or total proteinuria ≥300 mg/g or albuminuria by urinalysis of “300” or equivalent. “Moderate” = spot albuminuria or total proteinuria ≥30 mg/g or albuminuria by urinalysis of “30” or equivalent. “Normal” or not elevated if otherwise measured.
Baseline history of AKI occurring and managed in the outpatient setting.
Baseline history of AKI managed in the inpatient setting regardless of setting of onset.
Cardiovascular disease defined as the composite of coronary artery disease, congestive heart failure, stroke, or peripheral vascular disease as nonmutually exclusive categories.
Model Development
Linear versus restricted cubic spline fitting for continuous covariates were compared (Table S3). The majority of interaction terms were not included in the variable selection (Table S4). The final model included age, sex, smoking history, outpatient AKI history, eGFR, proteinuria, cardiovascular disease, diabetes mellitus, hypertension, liver disease, serum sodium, potassium, hemoglobin A1c, albumin, hemoglobin, systolic and diastolic BP, and hospitalization history. Included interaction terms were age and hospitalization history, sex and systolic BP, and sex and diastolic BP. The model is shown in Table S5. The resulting model performed well in the development cohort with a C statistic of 0.717 (95% CI, 0.710-0.725).
Model Validation
The model performed well in the validation cohort, with C statistic of 0.722 (95% CI, 0.720-0.723). The Hosmer-Lemeshow goodness-of-fit test suggested appreciable differences between the predicted and observed risk of outpatient AKI in the validation cohort (P < 0.001). The performance of the model in the validation cohort is shown in Fig 3 by decile of predicted AKI risk and did not suggest overfitting of the model.
Figure 3.
Observed risk versus mean predicted risk of outpatient acute kidney injury in the 18 months after a primary care baseline period by decile of predicted risk in the validation cohort (N = 4,864,576). Observed and mean predicted risks by deciles are 0.5% versus 0.9% for the 1st decile, 0.8% versus 1.3% for the 2nd decile, 1.0% versus 1.6% for the 3rd decile, 1.2% versus 1.9% for the 4th decile, 1.5% versus 2.2% for the 5th decile, 1.9% versus 2.6% for the 6th decile, 2.3% versus 3.1% for the 7th decile, 2.9% versus 4.0% for the 8th decile, 4.0% versus 5.4% for the 9th decile, and 7.7% versus 12.1% for the 10th decile, respectively. N = 486,458 for odd-numbered deciles and 486,457 for even-numbered deciles. Abbreviation: AKI, acute kidney injury.
Clinical Significance of the Model
In the lowest decile of predicted AKI risk in the validation cohort, there were 5,149 deaths among 450,160 patients with documented follow-up after the AKI period (73 deaths were among 2,321 patients with outpatient AKI). Median follow-up after the AKI period was 3.7 years. Kaplan-Meier curves reflected greater mortality among those experiencing outpatient AKI in the uniformly low-AKI risk group (P < 0.001) (Fig 4). The unadjusted and adjusted hazard ratios for death after outpatient AKI were 2.81 (95% CI, 2.23-3.54) and 2.54 (95% CI, 2.02-3.20), respectively.
Figure 4.
Kaplan-Meier survival curves for (A) the lowest decile of predicted outpatient acute kidney injury (AKI) risk (N = 450,160) and (B) the highest decile of predicted outpatient AKI risk (N = 481,058) in the validation cohort, beginning at the end of the 18-month AKI period, among those not lost to follow-up before the end of the AKI period, and censored on final encounter, vital sign, or laboratory measurement.
In the highest decile of AKI risk, there were 191,892 deaths among 481,058 patients with 4.5 years median follow-up (17,452 deaths were among 36,973 patients with outpatient AKI). Kaplan-Meier curves also displayed greater mortality among those with outpatient AKI than those without (P < 0.001) (Fig 4). The unadjusted and adjusted hazard ratios for death after outpatient AKI were 1.36 (95% CI, 1.33-1.38) and 1.59 (95% CI, 1.56-1.61), respectively.
Model Performance as 2 Binary Tests
The performance of the original model when transformed into research-oriented and clinically oriented tests is shown in Table 3. A “positive” result for the research test, derived from the development cohort, was ≥9.5% risk, resulting in a positive predictive value of 0.096 (95% CI, 0.094-0.097) and a negative predictive value of 0.980 (95% CI, 0.980-0.980) in the validation cohort. A “positive” result for the clinical test was ≥4.5% risk, resulting in a positive predictive value of 0.058 (95% CI, 0.058-0.059) and a negative predictive value of 0.985 (95% CI, 0.985-0.985).
Table 3.
Performance in the Validation Cohort (Unless Otherwise Specified) of 2 Binary Tests Produced From the Risk Prediction Model for Outpatient Acute Kidney Injury in the 18 Months After a Primary Care Baseline Period
| Statistic | Research Test | Clinical Test |
|---|---|---|
| Predicted risk cut-pointa | ≥9.5% | ≥4.5% |
| Sensitivity | 0.210 (95% CI, 0.208-0.213) | 0.494 (95% CI, 0.491-0.497) |
| Specificity | 0.952 (95% CI, 0.951-0.952) | 0.806 (95% CI, 0.806-0.807) |
| Positive predictive value | 0.096 (95% CI, 0.094-0.097) | 0.058 (95% CI, 0.058-0.059) |
| Negative predictive value | 0.980 (95% CI, 0.980-0.980) | 0.985 (95% CI, 0.985-0.985) |
| Positive likelihood ratio | 4.34 (95% CI, 4.29-4.39) | 2.55 (95% CI, 2.53-2.57) |
| Negative likelihood ratio | 0.83 (95% CI, 0.83-0.83) | 0.63 (95% CI, 0.62-0.63) |
| Percent of development cohort “negative” by the test (percentile of cut-point) | 96.7% | 85.5% |
| Percent of validation cohort “negative” by the test (percentile of cut-point) | 94.8% | 80.0% |
| P value for predicting outpatient AKIb | <0.001 | <0.001 |
Abbreviations: AKI, acute kidney injury; CI, confidence interval.
Threshold defining a “positive” test, as selected based on model performance in the development cohort.
By χ2 test, for the association between outpatient AKI and test positivity.
Discussion
We developed and externally validated a (continuous) risk prediction model for AKI occurring and managed in the outpatient setting. The continuous form of our model was also transformed into 2 binary tests with different applications: one for identifying patients for closer clinical monitoring or intervention and another for identifying research participants for randomized trials. In the continuous form, the model performed equally well in both the development and external validation cohorts (C statistic, ∼0.72), supporting generalizability. This performance was similar to that observed in inpatient AKI risk models, which have generally produced C statistics ranging from 0.7-0.8.18, 19, 20,33
The model also performed well in its binary forms. The clinical test, as designed, offers primary care providers a tool with which they can determine which patients may most benefit from closer clinical monitoring for outpatient AKI, which may lead to improved preventive care, prognostication, and post-AKI care. The sensitivity and specificity of the clinical test indicate that approximately one-half of cases of outpatient AKI would be identified by this test, and only 1 in 5 of those who would not experience outpatient AKI over 18 months are falsely identified as high-risk patients. Eighty percent of patients are deemed low-risk by this test.
The research test, with a higher positivity threshold identifying the 5% most at-risk patients, could alternatively serve to identify patients for clinical trials of outpatient AKI prevention. Although only 1 in 5 cases of outpatient AKI is captured by the research test, approximately 10% of those who test “positive” would have the outcome (compared with 2%-3% in the general population). Should clinical providers or researchers weigh the risks or benefits of being labeled “high-risk” differently than we did for either test, our model (when applied to a population) could be reconfigured with alternate weights.
Our study results stand in contrast to prior risk prediction models focused on AKI among hospitalized patients. Several risk factors known to associate with inpatient AKI overlapped with the risk factors identified in this study of outpatient AKI, although other novel risk factors were identified. In the study by Malhotra et al,18 like our study, CKD, heart disease, liver disease, hypertension, and anemia were identified as predictive of AKI. Others have identified albuminuria, diabetes mellitus, hypoalbuminemia, and hyponatremia as postsurgical AKI risk factors.20 Other known predictors of AKI that are typically limited to the inpatient setting, including acidemia, mechanical ventilation or hypoxia, or sepsis,15, 16, 17, 18,34 were not included in our model for variable selection.
Novel predictors of AKI in the outpatient setting identified included hospitalization history, smoking history, and serum potassium levels. Surprisingly, hypotension has not been frequently identified as a risk factor for predicting inpatient AKI.18 We identified a greater impact on AKI risk by diastolic BP over the physiologic range observed than by systolic BP, although both were selected for model inclusion. We also recognized interactions between age and prior hospitalizations and between sex and systolic and diastolic BP on their association with outpatient AKI risk. Also identified was the predicted risk of outpatient AKI associated with lacking a baseline laboratory or vital sign measurement, which, to our knowledge, has not been previously reported for AKI in any setting. The study by Tomasev et al21 did not provide risk estimates for individual covariates. Although lacking these prior measurements may be a general marker of wellness or a marker of infrequent or incomplete care by which outpatient AKI may be missed, not all missingness covariates predicted a reduced risk of observing outpatient AKI. Rather, lacking a prior proteinuria, hemoglobin A1c, or serum albumin measurement increased the predicted risk. Not identified in our model as risk factors for outpatient AKI were race, cancer, or serum calcium, despite being identified in studies of inpatient AKI.17,19 Although prior outpatient AKI was identified as a risk factor for subsequent outpatient AKI, inpatient AKI was unexpectedly not identified as an outpatient AKI risk factor.
Our model did display differences between observed and predicted risk in the validation cohort. This was not unexpected owing to our study’s statistical power. Although validation cohorts are typically recommended to contain at least 100 cases and 100 noncases,35,36 ours far exceeded those figures. Despite this, a reasonable visual agreement was seen between observed and predicted risk for outpatient AKI. Our model did typically overestimate the AKI risk in the validation cohort, which may be due to important differences between the 2 cohorts not captured by the covariates included in our study. Nonetheless, the model performed well in its continuous and binary forms in the validation cohort.
Although not the focus of this study, we observed that outpatient AKI was more common than inpatient AKI, consistent with our prior work8 and highlighting the potential impact of predicting outpatient AKI. In this study, we showed that the mortality risk was not fully captured by the presence of AKI risk factors (despite likely overlap in risk factors) and was heightened by the occurrence of outpatient AKI. On an absolute scale, this effect was greatest in the highest AKI risk group. On a relative scale, the occurrence of outpatient AKI impacts the mortality risk greatest among patients who, other than AKI, are the most well.
Additional strengths of this study include large cohort sizes and a lack of excluding patients for or imputation of missing data. They also include the use of objective variables in the EHR that could be used by both clinicians and researchers to identify high-risk patients.
Study limitations include its retrospective design. Additional cases of outpatient AKI not reflected in the EHR may have been missed. Cases of inpatient AKI that persisted into the outpatient setting may also have been misclassified in some cases. Cases of progressive CKD may have been incorrectly categorized as AKI if creatinine had been measured only infrequently. We were unable to retrospectively determine which creatinine measurements used isotope-dilution mass spectrometry–traceable methods and which did not. Our study design also restricted the cohorts to those patients surviving the 18-month AKI period. Early mortality after outpatient AKI is reassuringly low.7 Although some cases of outpatient AKI soon followed by death may have been missed, this design avoided bias from classifying those who died before AKI into the “no AKI” group. Finally, medications were not considered in this study, in part, because of the limited accuracy of outpatient medication lists outside managed care organizations.
Our outpatient AKI risk prediction model performed well in both our development and validation cohorts. The risk predicted by our model is calculable from readily available vital signs, laboratory tests, and diagnosed comorbid conditions in addition to incorporating the predictive quality of lacking these measurements. Such a tool can be adapted into a binary test to categorize patients as high-risk or not based on weighing risks and benefits and could be used to guide clinical monitoring or identify patients for research. Future studies will be needed to prospectively validate our model and to determine whether interventions in patients with an elevated risk of outpatient AKI can reduce the incidence of AKI or lead to a reduction in the mortality associated with outpatient AKI.
Article Information
Authors’ Full Names and Academic Degrees
Daniel Murphy, MD, MS, Scott Reule, MD, MS, David Vock, PhD, and Paul Drawz, MD, MHS, MS.
Authors’ Contributions
Study design: DM, SR, DV, and PD; data analysis/interpretation: DM and PD. Each author contributed important intellectual content during manuscript drafting or revision and accepts accountability for the overall work by ensuring that questions pertaining to the accuracy or integrity of any portion of the work are appropriately investigated and resolved.
Support
This research was supported by National Institutes of Health grant UL1TR002494.
Financial Disclosure
The authors declare that they have no relevant financial interests.
Acknowledgements
The authors thank Luke Bicknese, BS, for producing the model-development dataset and Maxwell Leither, MD, MS, for preliminary statistical code.
Disclaimer
These contents do not necessarily represent the views of the US Government or Department of Veterans Affairs. These contents are solely the responsibility of the authors and do not necessarily represent the views of National Institutes of Health.
Prior Publication
The data presented here originally constituted the Master’s thesis for DM with oversight by PD, SR, and DV.
Peer Review
Received May 12, 2021, as a submission to the expedited consideration track with 2 external peer reviews. Direct editorial input from the Statistical Editor and the Editor-in-Chief. Accepted in revised form August 18, 2021.
Footnotes
Complete author and article information provided before references.
Figure S1: Study definitions of outpatient acute kidney injury, inpatient acute kidney injury, and the moving baseline creatinine calculations for each.
Table S1: Accepted physiologic laboratory or vital sign values.
Table S2: Most recent proteinuria test to define baseline proteinuria by cohort.
Table S3: Restricted cubic spline fitting of continuous covariates.
Table S4: Interaction terms considered for variable selection in model development.
Table S5: Risk prediction model for outpatient acute kidney injury in the 18 months following a primary care–defined baseline period.
Supplementary Material
Figure S1; Tables S1-S5
References
- 1.Hsu C.Y., McCulloch C.E., Fan D., Ordonez J.D., Chertow G.M., Go A.S. Community-based incidence of acute renal failure. Kidney Int. 2007;72(2):208–212. doi: 10.1038/sj.ki.5002297. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Sileanu F.E., Murugan R., Lucko N., et al. AKI in low-risk versus high-risk patients in intensive care. Clin J Am Soc Nephrol. 2015;10(2):187–196. doi: 10.2215/CJN.03200314. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Wonnacott A., Meran S., Amphlett B., Talabani B., Phillips A. Epidemiology and outcomes in community-acquired versus hospital-acquired AKI. Clin J Am Soc Nephrol. 2014;9(6):1007–1014. doi: 10.2215/CJN.07920713. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Kaufman J., Dhakal M., Patel B., Hamburger R. Community-acquired acute renal failure. Am J Kidney Dis. 1991;17(2):191–198. doi: 10.1016/s0272-6386(12)81128-0. [DOI] [PubMed] [Google Scholar]
- 5.Soto K., Campos P., Pinto I., et al. The risk of chronic kidney disease and mortality are increased after community-acquired acute kidney injury. Kidney Int. 2016;90(5):1090–1099. doi: 10.1016/j.kint.2016.07.018. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Holmes J., Rainer T., Geen J., et al. Acute kidney injury in the era of the AKI e-alert. Clin J Am Soc Nephrol. 2016;11(12):2123–2131. doi: 10.2215/CJN.05170516. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Sawhney S., Fluck N., Fraser S.D., et al. KDIGO-based acute kidney injury criteria operate differently in hospitals and the community-findings from a large population cohort. Nephrol Dial Transplant. 2016;31(6):922–929. doi: 10.1093/ndt/gfw052. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Leither M.D., Murphy D.P., Bicknese L., et al. The impact of outpatient acute kidney injury on mortality and chronic kidney disease: a retrospective cohort study. Nephrol Dial Transplant. 2019;34(3):493–501. doi: 10.1093/ndt/gfy036. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Jones J., Holmen J., De Graauw J., Jovanovich A., Thornton S., Chonchol M. Association of complete recovery from acute kidney injury with incident CKD stage 3 and all-cause mortality. Am J Kidney Dis. 2012;60(3):402–408. doi: 10.1053/j.ajkd.2012.03.014. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Bucaloiu I.D., Kirchner H.L., Norfolk E.R., Hartle J.E., II, Perkins R.M. Increased risk of death and de novo chronic kidney disease following reversible acute kidney injury. Kidney Int. 2012;81(5):477–485. doi: 10.1038/ki.2011.405. [DOI] [PubMed] [Google Scholar]
- 11.Wu V.C., Wu C.H., Huang T.M., et al. Long-term risk of coronary events after AKI. J Am Soc Nephrol. 2014;25(3):595–605. doi: 10.1681/ASN.2013060610. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Chawla L.S., Amdur R.L., Shaw A.D., Faselis C., Palant C.E., Kimmel P.L. Association between AKI and long-term renal and cardiovascular outcomes in United States veterans. Clin J Am Soc Nephrol. 2014;9(3):448–456. doi: 10.2215/CJN.02440213. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Wald R., Quinn R.R., Luo J., et al. Chronic dialysis and death among survivors of acute kidney injury requiring dialysis. JAMA. 2009;302(11):1179–1185. doi: 10.1001/jama.2009.1322. [DOI] [PubMed] [Google Scholar]
- 14.Hobbs H., Bassett P., Wheeler T., et al. Do acute elevations of serum creatinine in primary care engender an increased mortality risk? BMC Nephrol. 2014;15:206. doi: 10.1186/1471-2369-15-206. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Coritsidis G.N., Guru K., Ward L., Bashir R., Feinfeld D.A., Carvounis C.P. Prediction of acute renal failure by "bedside formula" in medical and surgical intensive care patients. Ren Fail. 2000;22(2):235–244. doi: 10.1081/jdi-100100868. [DOI] [PubMed] [Google Scholar]
- 16.Hoste E.A., Lameire N.H., Vanholder R.C., Benoit D.D., Decruyenaere J.M., Colardyn F.A. Acute renal failure in patients with sepsis in a surgical ICU: predictive factors, incidence, comorbidity, and outcome. J Am Soc Nephrol. 2003;14(4):1022–1030. doi: 10.1097/01.asn.0000059863.48590.e9. [DOI] [PubMed] [Google Scholar]
- 17.Chawla L.S., Abell L., Mazhari R., et al. Identifying critically ill patients at high risk for developing acute renal failure: a pilot study. Kidney Int. 2005;68(5):2274–2280. doi: 10.1111/j.1523-1755.2005.00686.x. [DOI] [PubMed] [Google Scholar]
- 18.Malhotra R., Kashani K.B., Macedo E., et al. A risk prediction score for acute kidney injury in the intensive care unit. Nephrol Dial Transplant. 2017;32(5):814–822. doi: 10.1093/ndt/gfx026. [DOI] [PubMed] [Google Scholar]
- 19.Matheny M.E., Miller R.A., Ikizler T.A., et al. Development of inpatient risk stratification models of acute kidney injury for use in electronic health records. Med Decis Making. 2010;30(6):639–650. doi: 10.1177/0272989X10364246. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Park S., Cho H., Park S., et al. Simple postoperative AKI risk (SPARK) classification before noncardiac surgery: a prediction index development study with external validation. J Am Soc Nephrol. 2019;30(1):170–181. doi: 10.1681/ASN.2018070757. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Tomasev N., Glorot X., Rae J.W., et al. A clinically applicable approach to continuous prediction of future acute kidney injury. Nature. 2019;572(7767):116–119. doi: 10.1038/s41586-019-1390-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Grams M.E., Sang Y., Ballew S.H., et al. A meta-analysis of the association of estimated GFR, albuminuria, age, race, and sex with acute kidney injury. Am J Kidney Dis. 2015;66(4):591–601. doi: 10.1053/j.ajkd.2015.02.337. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Levey A.S., Bosch J.P., Lewis J.B., Greene T., Rogers N., Roth D. A more accurate method to estimate glomerular filtration rate from serum creatinine: a new prediction equation. Modification of diet in Renal Disease Study Group. Ann Intern Med. 1999;130(6):461–470. doi: 10.7326/0003-4819-130-6-199903160-00002. [DOI] [PubMed] [Google Scholar]
- 24.Levey A.S., Coresh J., Greene T., et al. Using standardized serum creatinine values in the modification of diet in renal disease study equation for estimating glomerular filtration rate. Ann Intern Med. 2006;145(4):247–254. doi: 10.7326/0003-4819-145-4-200608150-00004. [DOI] [PubMed] [Google Scholar]
- 25.Hall R.K., Wang V., Jackson G.L., et al. Implementation of automated reporting of estimated glomerular filtration rate among Veterans Affairs laboratories: a retrospective study. BMC Med Inform Decis Mak. 2012;12:69. doi: 10.1186/1472-6947-12-69. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Wang V., Maciejewski M.L., Hammill B.G., et al. Recognition of CKD after the introduction of automated reporting of estimated GFR in the Veterans Health Administration. Clin J Am Soc Nephrol. 2014;9(1):29–36. doi: 10.2215/CJN.02490213. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Kidney Disease: Improving Global Outcomes (KDIGO) CKD Work Group KDIGO 2012 clinical practice guideline for the evaluation and management of chronic kidney disease. Kidney Int Suppl. 2013;3(1):1–150. [Google Scholar]
- 28.Wang Y., Wang J., Su T., et al. Community-acquired acute kidney injury: a nationwide survey in China. Am J Kidney Dis. 2017;69(5):647–657. doi: 10.1053/j.ajkd.2016.10.034. [DOI] [PubMed] [Google Scholar]
- 29.Bellomo R., Ronco C., Kellum J.A., Mehta R.L., Palevsky P. Acute Dialysis Quality Initiative w. Acute renal failure—definition, outcome measures, animal models, fluid therapy and information technology needs: the Second International Consensus Conference of the Acute Dialysis Quality Initiative (ADQI) Group. Crit Care. 2004;8(4):R204–R212. doi: 10.1186/cc2872. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Mehta R.L., Kellum J.A., Shah S.V., et al. Acute kidney injury network: report of an initiative to improve outcomes in acute kidney injury. Crit Care. 2007;11(2):R31. doi: 10.1186/cc5713. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Go A.S., Chertow G.M., Fan D., McCulloch C.E., Hsu C.Y. Chronic kidney disease and the risks of death, cardiovascular events, and hospitalization. N Engl J Med. 2004;351(13):1296–1305. doi: 10.1056/NEJMoa041031. [DOI] [PubMed] [Google Scholar]
- 32.Liao H.F., Cheng L.Y., Hsieh W.S., Yang M.C. Selecting a cutoff point for a developmental screening test based on overall diagnostic indices and total expected utilities of professional preferences. J Formos Med Assoc. 2010;109(3):209–218. doi: 10.1016/S0929-6646(10)60044-7. [DOI] [PubMed] [Google Scholar]
- 33.Thakar C.V., Arrigain S., Worley S., Yared J.P., Paganini E.P. A clinical score to predict acute renal failure after cardiac surgery. J Am Soc Nephrol. 2005;16(1):162–168. doi: 10.1681/ASN.2004040331. [DOI] [PubMed] [Google Scholar]
- 34.Peres L.A., Wandeur V., Matsuo T. Predictors of acute kidney injury and mortality in an intensive care unit. J Bras Nefrol. 2015;37(1):38–46. doi: 10.5935/0101-2800.20150007. [DOI] [PubMed] [Google Scholar]
- 35.Vergouwe Y., Steyerberg E.W., Eijkemans M.J., Habbema J.D. Substantial effective sample sizes were required for external validation studies of predictive logistic regression models. J Clin Epidemiol. 2005;58(5):475–483. doi: 10.1016/j.jclinepi.2004.06.017. [DOI] [PubMed] [Google Scholar]
- 36.Collins G.S., Ogundimu E.O., Altman D.G. Sample size considerations for the external validation of a multivariable prognostic model: a resampling study. Stat Med. 2016;35(2):214–226. doi: 10.1002/sim.6787. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Figure S1; Tables S1-S5