Skip to main content
BMJ Open logoLink to BMJ Open
. 2017 Mar 8;7(3):e013511. doi: 10.1136/bmjopen-2016-013511

Predicting AKI in emergency admissions: an external validation study of the acute kidney injury prediction score (APS)

L E Hodgson 1,2, B D Dimitrov 1, P J Roderick 1, R Venn 2, L G Forni 3,4
PMCID: PMC5353262  PMID: 28274964

Abstract

Objectives

Hospital-acquired acute kidney injury (HA-AKI) is associated with a high risk of mortality. Prediction models or rules may identify those most at risk of HA-AKI. This study externally validated one of the few clinical prediction rules (CPRs) derived in a general medicine cohort using clinical information and data from an acute hospitals electronic system on admission: the acute kidney injury prediction score (APS).

Design, setting and participants

External validation in a single UK non-specialist acute hospital (2013–2015, 12 554 episodes); four cohorts: adult medical and general surgical populations, with and without a known preadmission baseline serum creatinine (SCr).

Methods

Performance assessed by discrimination using area under the receiver operating characteristic curves (AUCROC) and calibration.

Results

HA-AKI incidence within 7 days (kidney disease: improving global outcomes (KDIGO) change in SCr) was 8.1% (n=409) of medical patients with known baseline SCr, 6.6% (n=141) in those without a baseline, 4.9% (n=204) in surgical patients with baseline and 4% (n=49) in those without. Across the four cohorts AUCROC were: medical with known baseline 0.65 (95% CIs 0.62 to 0.67) and no baseline 0.71 (0.67 to 0.75), surgical with baseline 0.66 (0.62 to 0.70) and no baseline 0.68 (0.58 to 0.75). For calibration, in medicine and surgical cohorts with baseline SCr, Hosmer-Lemeshow p values were non-significant, suggesting acceptable calibration. In the medical cohort, at a cut-off of five points on the APS to predict HA-AKI, positive predictive value was 16% (13–18%) and negative predictive value 94% (93–94%). Of medical patients with HA-AKI, those with an APS ≥5 had a significantly increased risk of death (28% vs 18%, OR 1.8 (95% CI 1.1 to 2.9), p=0.015).

Conclusions

On external validation the APS on admission shows moderate discrimination and acceptable calibration to predict HA-AKI and may be useful as a severity marker when HA-AKI occurs. Harnessing linked data from primary care may be one way to achieve more accurate risk prediction.

Keywords: GENERAL MEDICINE (see Internal Medicine)


Strengths and limitations of this study.

  • This large study is one of the few external validations of an acute kidney injury (AKI) prediction rule in general medical and surgical hospital populations, including patients without a baseline serum creatinine (SCr).

  • Stringent inclusion criteria (stay at least one night, first admission, repeat SCr performed), excluded a large proportion of ‘low-risk’ patients, recognised as such by the clinical team.

  • Single-centre cohort, in the same geographical area to the derivation site cautions against generalisability.

  • Medical histories relied on previously coded events on the hospital database and are thus likely to underestimate disease prevalence.

  • Reliance on a baseline SCr that could have last been performed a number of months ago, means that a proportion of the patients considered to have hospital-acquired AKI may have already fulfilled the criteria for a change in SCr prior to admission, that is, have community-acquired AKI.

Introduction

Acute kidney injury (AKI) is a clinically diagnosed syndrome defined as an acute increase in serum creatinine (SCr) and/or a reduction in urine volume.1 One in five adults worldwide experience AKI during a hospital admission and despite significant developments in hospital care, mortality associated with AKI remains high.2–7 Deficits in the recognition and subsequent management of patients who have developed AKI have been shown,8 9 with recent studies suggest around 60–79% of AKI cases are from the community and can be flagged at admission.10–15 However, a significant proportion of AKI develops in hospital (hospital-acquired AKI, HA-AKI) and systematically highlighting patients at the highest risk of HA-AKI is of great interest.16 17 Given that a continuum of injury exists before loss of excretory kidney function can be measured with standard laboratory tests (ie, SCr), one method that could be used to highlight patients at an earlier stage before injury is the use of prediction models.18–21 Most prediction research involves derivation and internal validation, however, external validation is crucial to address overfitting and generalisability.22–26 The majority of HA-AKI prediction models focus on high-risk specialist groups such as cardiac surgery with few studies in acute medicine or surgery.15 27–29 One such model is the acute kidney injury prediction score (APS), derived in acute medicine patients in a single UK centre which uses a combination of comorbidities and acute physiological variables derived from a hospital electronic data system (see online supplementary appendix table A1 for variables). Discrimination assessed by the area under the receiver operating characteristic curve (AUROC) to predict HA-AKI within 7 days was 0.72 in derivation and 0.76 in an internal validation cohort (without a baseline SCr).15 The primary aim of this study was to externally validate the APS in a general medical population with a known baseline SCr. The presence of a known baseline accurately accounts for patients with chronic kidney disease (CKD) and therefore allows more confident conclusions to be drawn as to whether acute deterioration in renal function has already occurred at admission (community-acquired AKI, CA-AKI). Secondary aims were validations of the APS in (1) a general medical population without a known baseline SCr and (2) general surgical populations again with and without a known baseline SCr. Published guidance for reporting were followed.22

supplementary appendix

bmjopen-2016-013511supp_appendix.pdf (238.2KB, pdf)

Methods

A retrospective observational cohort external validation study of the APS (see online supplementary appendix table A1 for variables and weightings) was performed on the adult medical and surgical units of St Richard's Hospital site of Western Sussex Hospitals NHS Foundation Trust (WSHFT), for the period March 2013 to February 2015. WSHFT is an 870-bed Trust on the South Coast of England, with a combined annual emergency department attendance over 150 000 and 60–80 acute medical or surgical admissions per 24-hour period. St Richard's is a separate hospital site from Worthing hospital where in 2011 the APS was derived on general medical patients. There was no cross-site contamination of staff, though catchment populations for the two sites, 20 miles apart, are similar. Surgery at the Trust includes general surgery, urology and trauma and orthopaedic, but does not include major trauma, neurosurgery, cardiac, major vascular or transplantation. (see online supplementary appendix table A2 for clinical and demographic values for the derivation and presented external validation cohorts).15

At admission, all inpatients (acute medicine and elderly care, emergency and elective surgery) routinely had physiological observations measured and entered via handheld systems into the clinical data software system (Patientrack Sydney, NSW, Australia). Previous International Statistical Classification of Diseases 10th revision (ICD-10) electronically coded history (heart failure, liver disease and diabetes mellitus) were retrieved and CKD was defined as an estimated glomerular filtration rate (eGFR)<60 mLs/min prior to admission (in those with an available baseline SCr) using a National algorithm that has been shown to perform well.30 Kidney disease: improving global outcomes (KDIGO) criteria for AKI was employed (SCr increase of ≥1.5 from the admission value or ≥26.5 μmol/L within a rolling 48 hours during the first 7 days of the patients first admission in the study period). Patients were included if they were ≥18 years of age, stayed at least one night in the medical or surgical division over the 24-month period (2013–2015) and had at least one SCr repeated.

Exclusion criteria:

  • patients with AKI on admission (defined using KDIGO change from baseline SCr or absolute SCr value ≥354 μmol/L);

  • patients moved directly from the accident and emergency department (A&E) department to the intensive care unit (ICU) (as neither area uses the Patientrack data system);

  • aged<18 years;

  • obstetrics and gynaecology admissions;

  • discharged without spending a night in hospital.

Patients were followed-up until discharge from hospital, or death, during the inpatient spell. The analysis was performed for the patients' first admission where more than one occurred in the study period. As all patients admitted had an APS calculated automatically by the Patientrack© system before any outcome had occurred, there were no missing demographic or physiological data at admission. As history relied on previous coded events (from attending hospital), potentially such data could have been missing. None of the researchers involved in analysis of the data were involved in the management of the patients. The research team members responsible for data analysis had access only to the fully anonymised individual-level data and were blinded to other patient data, as well as to the components of the calculated scores in the hospital information system. Since there is no consensus on how to determine what counts as an adequate sample size in such studies, all available (19 276) hospital cases for the period 2013–2015 were included in the analysis.22

Following logistic regression analysis, discrimination was assessed by the area under the receiver operating characteristic curve (AUROC),22 and calibration by the Hosmer-Lemeshow (H-L) test, where a significant p<0.05, may indicate poor fit of a rule or model, though this has limitations in larger datasets.31 32 Calibration was additionally assessed graphically by plotting predicted probabilities (x-axis) against the observed event rate (y-axis) of the outcome and deriving a linear function (an intercept of zero and a slope of one indicating perfect calibration).22 Predictive values and likelihood ratios were calculated to further inform on the way model performance could impact on clinical workload. Following extraction, all data were fully anonymised on Microsoft Excel and analyses performed on SPSS V.22.

Results

From an initial 19 276 patient episodes in medicine (including elderly care) and surgery (see figure 1) over a 2-year period 12 554 episodes were analysed (n=7170 in medicine and n=5384 in surgery) after excluding patients with AKI on admission (n=782) or with no repeat SCr while in hospital (n=5940). Over a quarter of patients (n=3329) had no baseline SCr. Table 1 summarises the four groups in terms of clinical and demographic data. Incidence of HA-AKI was:

  • medical patients with a known SCr baseline 8.1%;

  • medical patients without a known baseline SCr 6.6%;

  • surgical patients with a known baseline SCr 4.9%;

  • surgical patients with no known baseline were 4%.

Figure 1.

Figure 1

Consort study flow chart. HA-AKI, hospital-acquired acute kidney injury; SCr, serum creatinine.

Table 1.

Clinical and demographics data of the four cohorts

Medicine with baseline SCr (n=5034)
Medicine no baseline SCr (n=2136)
Surgery with baseline SCr (n=4191)
Surgery no baseline SCr (n=1193)
HA-AKI HA-AKI 8.1% (n=409) No HA-AKI (n=4625) p Value HA-AKI 6.6% (n=141) No HA-AKI (n=1995) p Value HA-AKI 4.9% (n=204) No HA-AKI (n=3987) p Value HA-AKI 4% (n=49) No HA-AKI (n=1144) p Value
Mortality* 21.5% (n=88) 4.5% (n=209) <0.001 23% (n=32) 4% (n=72) <0.001 5.9% (n=12) 0.4% (n=15) <0.001 8.2% (n=4) 0.9% (n=10) 0.002
LOS 12 (6–21) 6 (3–12) <0.001 11.5 (5–20) 5 (2–11) <0.001 8 (4–14) 3 (2–5) <0.001 7 (3–17) 3 (2–6) <0.001
APS 4 (3–5) 3 (2–4) <0.001 3 (3–4) 2 (1–3) <0.001 3 (2–4) 2 (0–3) <0.001 2 (0–3) 0 (0–2) <0.001
Age 84 (77–89) 79 (67–86) <0.001 85 (75–91) 74 (55–85) <0.001 77 (69–84) 69 (56–78) <0.001 69 (53–83) 52 (37–67) <0.001
RR ≥20 28% 23% 0.034 24% 19% 0.139 8% 5% 0.063 12% 6% 0.055
<Alert AVPU 1.7% 1% 0.197 0.8% 0 0.619 n=0 0.2% 1 n=0 n=2 1
CKD 29% 10% <0.001 14% 3% <0.001
Diabetes 25% 21% 0.059 16% 10% 0.066 21% 15% 0.021 10% 5% 0.187
Heart failure 37% 18% <0.001 28% 9% <0.001 9% 2% <0.001 n=3 n=8 0.009
Liver disease 2% 0.8% 0.026 1% 1% 0.185 0.5% 0.4% 0.593 n=0 n=4 1
NEWS 2 (1–4) 1 (0–3) <0.001 2 (0–3) 1 (0–3) 0.024 1 (0–2) 1 (0–2) 0.376 1 (0–2) 1 (0–1) 0.209

*refers to mortality being ‘in-patient mortality’. Bold indicates P<0.05. APS, acute kidney injury (AKI) prediction score; AVPU, consciousness scale of best response: Alert; Vocal, Pain, Unresponsive, CKD, chronic kidney disease; HA-AKI, hospital-acquired AKI (within 7 days of admission); LOS, length of stay (days) in hospital; Mortality, inhospital; NEWS, national early warning score; RR, respiratory rate (per minute); SCr, serum creatinin.

The medical cohort were significantly older than the surgical cohort (78 years (65–86) vs 67 (51–77), respectively, p≤0.001). There was also a higher frequency of morbidity, particularly heart failure in the medical population who also had longer hospital stays. Inpatient mortality was increased in those with HA-AKI across all four groups, but in absolute terms this was greatest in medical patients where in those with a baseline SCr the observed mortality was 21.5% in those who developed HA-AKI died versus 4.5% in those without HA-AKI (p≤0.001).

For the primary analysis in medical patients with a baseline SCr the AUROC was 0.65 (95% CI 0.62 to 0.67) and the H-L p=0.064. In those patients without a baseline SCr the AUROC was 0.71 (0.67 to 0.75), H-L p=0.014. In surgical patients with baseline SCr: AUROC 0.66 (95% CI 0.62 to 0.69) and H-L p=0.093; Surgery without SCr baseline: AUROC 0.67 (0.58 to 0.75) and H-L p=0.664 (see figure 2).

Figure 2.

Figure 2

Area under the receiver operating characteristic curves for APS to predict HA-AKI. A: medicine with known baseline SCr 0.65 (95% CI 0.62 to 0.67); B: medicine no baseline SCr 0.71 (95% CI 0.67 to 0.75); C: Surgery with known baseline SCr 0.66 (95% CI 0.62 to 0.70); D: surgery without a baseline SCr 0.67 (95% CI 0.58 to 0.75). APS, acute kidney injury (AKI) prediction score; HA-AKI, hospital-acquired AKI; SCr, serum creatinine.

Calibration plots (figure 3) for the association between predicted probabilities and observed event rates in medical and surgical cohorts (with a baseline SCr) demonstrated agreement at low probability rates while at higher rates calibration deviated in the medical cohort, though the number of events were small. Table 2 compares the predicted rates of HA-AKI (from the original derivation study cohort) compared with observed rates in the validation medical cohort (with baseline SCr), grouped according to APS admission score.15 The table shows in the validation cohort 4% of patients scoring 0–2 APS points developed HA-AKI, compared with 28% of cases scoring ≥7 points with an OR of 4.7 (95% CI 3.1 to 7.2).

Figure 3.

Figure 3

Calibration plots of predicted probabilities versus observed rates of HA-AKI. Predicted probabilities versus observed rates (HA-AKI) at each level of the APS score in the medical (TOP) and surgical (BOTTOM) cohorts with a known baseline creatinine. APS, acute kidney injury (AKI) prediction score; HA-AKI, hospital-acquired AKI.

Table 2.

Risk range for HA-AKI by APS in validation study versus original derivation study

Validation
Derivation study
APS points HA-AKI OR* HA-AKI OR*
0–2 4% 0.4 (0.3 to 0.5) 3% 0.29 (0.17 to 0.49)
3–4 8% 2.2 (1.6 to 2.9) 7% 2.3 (1.3 to 4.1)
5–6 14% 2.3 (1.8 to 2.9) 19% 4.4 (2.7 to 7.1)
≥7 28% 4.7 (3.1 to 7.2) 33% 6.7 (2.5 to 18.3)

*OR (95% CIs).

APS, acute kidney injury (AKI) prediction score; HA-AKI, hospital-acquired AKI.

At a cut-off of five points on the APS for medical patients with baseline SCr sensitivity was 34% (30–39%), specificity 82% (81–83%), positive predictive value (PPV) 16% (13–18%), negative predictive value (NPV) 94% (93–95%), positive likelihood ratio (PLR) 2.1 (1.8–2.4) and negative likelihood ratio (NLR) 0.79 (0.73-0.84). Lowering the cut-off to a score of three points increased sensitivity to 82% (78–86%), with a reduction in specificity to 37% (36–39%), with PPV 10% (9–12%) and NPV 96% (95–97%), PLR 1.3 (1.25–1.38) and NLR 0.48 (0.39–0.59).

In the medical cohort with a known baseline SCr, HA-AKI was associated with significantly higher inpatient mortality using a cut-off of five points on the APS: 28% of patients who developed HA-AKI died versus 18% with a score <5 points (OR 1.8 (95% CI 1.1 to 2.9), p=0.015). This was also found in patients without a baseline SCr (mortality 42% vs 19%, p=0.029).

Discussion

This study represents one of the few external validations of a prediction model for HA-AKI—the APS—for acute general medical and surgical cohorts. Moderate discrimination and satisfactory calibration were found in cohorts with and without a known baseline SCr. External validation addresses optimism and generalisability.22–24 33 34 Discrimination, assessed by AUROC analysis was inferior to the derivation study (0.65 in the comparable medical with baseline SCr vs 0.72, though 95% CIs crossed) a finding commonly found reflecting overfitting from derivation design or modelling (sample size and selection of variables) and clinical factors such as case mix.23 33 35 However, the observed rates of HA-AKI (8.1% vs 7.2% in the derivation) and associated inpatient mortality were similar in those with the outcome (20.5% vs 21.5%) and those without the outcome (3.5% vs 4.5%) in derivation and validation cohorts, respectively. The validation cohort was older and had higher rates of diabetes and heart failure whereas in contrast, rates of CKD and liver disease were lower. The two physiological observations included in the APS (reduced level of consciousness and tachypnoea) were also less commonly abnormal in the validation cohort (see online supplementary appendix table A2 for summary).

Different methods were used in the two studies, for example, to define CKD (eGFR >1 and <6 months prior to admission in derivation vs national baseline in this external validation). It is unclear to what extent this would have affected the results, though patients could conceivably have already developed AKI in the community. For example, if the last available SCr was a number of months ago, HA-AKI may be overestimated and CA-AKI underestimated. The validation uses the pragmatic national baseline definition that will be applicable in clinical practice and allows automation in an electronic hospital system. The original APS study had relatively few events per variable, as well as high weighting for two infrequent variables (liver disease and conscious level) that may account for reduced model performance in validation. In the larger validation population these variables, though still significant, were rarely present. Furthermore, the derivation cohort included patients who had more than one admission, while the external validation cohort included only first admission during the study period. Patients with recurrent admissions may have been prone to recurrent AKI episodes, and in particular a prior episode of AKI could be incorporated to improve model prediction. This is an area of future interest.

However, currently employed statistical markers of performance also have limitations. Correctly predicting a future event is more complex than diagnostic prediction models, partly due to the time elapsed between prediction and outcome, with a significant stochastic element and therefore accurate risk stratification is often the best that can be achieved. AKI, defined by a nominal change in SCr, may be the manifestation of a number of acute precipitants (eg, sepsis, vomiting) affecting a susceptible host (eg, a patient with heart failure and reduced physiological reserve), making accurate prediction at a single time period (hospital admission) with high discrimination, impossible.36 37 Given there is no consensus on what constitutes acceptable discrimination, descriptors such as ‘poor’ or ‘good’ to a particular AUROC are often ascribed, but the AUROC has shortcomings including a narrow focus on accuracy that may not incorporate information on consequences.38–40 For example, if a false-negative result is more harmful than a false-positive result, a model with much greater specificity, but slightly lower sensitivity than another would have a higher AUROC, but may be a poor clinical choice.41 With the assumption of uniform distribution of risk, the maximum AUROC is 0.83.42 In a perfectly calibrated model in a population with an average 10-year risk of 10% with relatively little spread, risk is centred around 10%, and the maximum AUROC is 0.63.40 Though calibration is often overlooked, it is crucial in risk prediction,40 43 and in this validation the calibration plots suggest adequate fit between predicted probabilities and observed frequencies of clinical value. More nuanced alternatives are yet to be widely used.41 44 45

Potential clinical use of the APS

First, using risk ranges, an APS score of 0–2 points (encompassing 36% all medial patients with a baseline SCr) was associated with a low risk of HA-AKI (4%), while a score ≥7 points had a 28% risk.

Second, at five APS points (18% of patients) PPV was relatively low (16% (13–18%)), however, the high NPV 94% (93–95%) suggests an ability to identify those at low risk (as rule out) and importantly even if HA-AKI developed patients (with an APS <5 points) were significantly less likely to die than those with a higher APS.

Third, with only seven variables the APS is relatively simple to calculate. Fourth, all variables can potentially be automatically calculated in (the increasing number of) hospitals with electronic records containing clinical coding and physiological observation systems.

Fifth, the score in this validation performed with similar discrimination in two other cohorts—general surgery and in patients without a baseline SCr.

Finally, the score has potential as an aid memoire to variables associated with HA-AKI, consistently found on systematic review of other AKI prediction models.46

Alternative models

Appraising other models can inform whether a model is worth further investigating or implementing, or highlighting well developed, validated alternatives. In the field of HA-AKI prediction only one other model for general emergency medical and surgical admissions from the UK study (to predict HA-AKI at 72 hours) has external validation evidence, with an AUROC of 0.67 in derivation and 0.71 in external validation.29 The variables included (age, previous admissions, admission diagnostic category, seven laboratory parameters, Charlson comorbidity index and proteinuria) may be difficult to automate and inclusion of admission diagnoses is a shortcoming if use is required at the earliest opportunity. Two older studies, one a retrospective cohort and the other including 27 variables have not been externally validated.27 28 Importantly all three models may be difficult to calculate at the bedside. (see supplementary appendix table A3 for a summary of these models with a corresponding Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) guidance reporting score.22

A large number of postsurgical models have been derived principally in cardiac and transplantation surgery, although the outcome is often the use of renal replacement therapy with its inherent flaws and indeed when KDIGO AKI definitions have been used in external validations AUROCs range from 0.61 to 0.74.47–51 In contrast-induced AKI several models have been developed, however, most use specialty specific variables limiting extrapolation to a general population.46 In patients with acute heart failure the single externally validated model produced an AUROC of 0.65 in both external studies.52 53 The only externally validated general surgical model derived in the USA had an AUROC of 0.66 when transported to a Chinese population.54 Finally the UK study in Trauma and Orthopaedics performed external validation in the same publication, reporting an AUROC of 0.70.55 Thus, in a number of fields externally validated prediction models for HA-AKI are relatively rare and have moderate discriminatory performance.

Study strengths

This large study is one of the few external validations in general populations.

In England, for example, general medicine, surgery and orthopaedics are the three most common divisions under whom patients are admitted.56 Though few studies break down HA-AKI by specialty, two UK teaching hospital studies suggested over half AKI episodes were in these general specialties.57 58 Stringent inclusion criteria (stayed at least one night, repeat SCr performed), would have excluded a large proportion of ‘low-risk’ patients, recognised as such by the clinical team, making the group in question of clinical relevance. Using only the first admission during the study period avoids for the effects of recurrent admissions. It is likely those with multiple attendances would be at higher risk of HA-AKI as well as scoring higher on the APS which takes into account age and (coded) comorbidities, associated with an increased risk of readmissions.59–62 Model updating could be assess the effects of a prior episode of AKI, for example. While discrimination of a single admission APS score was only moderate, this represents a single time point and from an outcome that could occur up to 7 days later. If, for example, a physiological deterioration occurred following admission (with tachypnoea or reduced conscious level) this would be reflected in a higher APS. Patients without a known baseline SCr (a quarter of the medical cohort) pose particular problems: first, CKD contributes to the APS and second assessment of whether the AKI was community or hospital acquired in nature is more difficult. Such patients in general would be lower risk than those with recent available results as the latter would be more likely to have been having tests to monitor chronic disease or prior illnesses. The study demonstrated that in this cohort the APS had similar performance to those with a known baseline SCr.

Study limitations

The single-centre nature of the cohort cautions against generalisability. Also although data were collected in a prospective nature by the electronic hospital record, past medical histories relied on previously coded events on the hospital database, thus they are likely to underestimate disease prevalence. This could be addressed by (planned) improved linkage with primary care databases that could also incorporate information on medications. Moreover, the validation site is in the same geographical area to the derivation site and though overall a more affluent population bears similarities (elderly population on the South East coast of England) that may not make the results widely generalisable. Furthermore, relying on a baseline SCr that could have last been measured a number of months ago means that a proportion of the patients considered to have HA-AKI may well have already fulfilled the criteria for a change in SCr prior to admission, that is, have CA-AKI, which could also be true of those without a baseline SCr.

Future directions

Prediction of AKI remains of key importance given its association with a high risk of mortality and in survivors, high morbidity.16 Further external validation of prediction models, with updating where necessary is desirable.22 Unfortunately, as AKI is a syndrome reflecting diverse underlying pathophysiological states (usually) imposed on a host with chronic disease, any model at a single time point, is unlikely to predict with high discrimination. Improving models will require comprehensive records (medical and medications through primary care linkage) with fluctuating physiological trends (and potentially laboratory parameters), mirroring the traditional clinical approach to prognosis, with adherence to enablers of clinical uptake.63 64 Complex models may enable personalised risk stratification, however, manual inputting or use of a calculator without automation are barriers to bedside use. A balance between comprehensive risk factor inclusion, bedside usability and automated functionality must thus be struck. Drawbacks of using urine output and SCr known to be unpredictably affected by diverse inputs suggest that the employment of more refined biomarkers as predictors and markers of significant AKI are required.65 Finally, no AKI prediction models have undergone impact analysis to assess whether implementation can improve outcomes and this is an area of interest. As no single intervention has been found to improve outcome in AKI, a model would most likely to be used as a part of a systematic alert to risk and initiation of enhanced monitoring in the appropriate clinical location with avoidance of iatrogenic harm.

Conclusion

The APS to predict HA-AKI in an external validation study of general medical and surgical patients performed with moderate discrimination and acceptable calibration. The prediction rule could help identify at admission those patients at higher risk of developing HA-AKI, in order to prevent its occurrence and avoid, or at least mitigate associated significant complications.

Footnotes

Contributors: All authors contributed to the paper in accordance with ICMJE recommendations and agree to be accountable for all aspects of the work. LEH contributed to study design, data collection, data analysis, writing up of the paper and final approval of the version submitted. BDD contributed to study design, data analysis, writing up of the paper and final approval of the version submitted. PJR contributed to study design, writing up of the paper and final approval of the version submitted. RV contributed to study design, data analysis, writing up of the paper and final approval of the version submitted. LGF contributed to study design, data analysis, writing up of the paper and final approval of the version submitted.

Funding: This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

Competing interests: All authors declare that the results presented in this paper have not been published previously in whole or part. All authors have completed the Unified Competing Interest form at http://www.icmje.org/coi_disclosure.pdf and declare that all have no relationships with companies that might have an interest in the submitted work in the previous 3 years; their spouses, partners, or children have no financial relationships that may be relevant to the submitted work; and have no non-financial interests that may be relevant to the submitted work.

All authors had full access to all of the data (including statistical reports and tables) in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. The lead author affirms that the manuscript is an honest, accurate, and transparent account of the study being reported and no important aspects of the study have been omitted.

Ethics approval: Ethical approval was given by NHS Research Ethics Committee London—South East (REC reference 13/LO/0884).

Provenance and peer review: Not commissioned; internally peer reviewed.

Data sharing statement: No additional data are available.

References

  • 1.KDIGO Clinical Practice Guideline for Acute Kidney Injury. Kidney Int 2012;(Suppl 2):1–136. 10.1038/kisup.2012 [DOI] [Google Scholar]
  • 2.Ventilation with lower tidal volumes as compared with traditional tidal volumes for acute lung injury and the acute respiratory distress syndrome. The acute respiratory distress syndrome network. N Engl J Med 2000;342:1301–8. 10.1056/NEJM200005043421801 [DOI] [PubMed] [Google Scholar]
  • 3.Angus DC, Barnato AE, Bell D et al. A systematic review and meta-analysis of early goal-directed therapy for septic shock: the ARISE, ProCESS and ProMISe Investigators. Intensive Care Med 2015;41:1549–60. 10.1007/s00134-015-3822-1 [DOI] [PubMed] [Google Scholar]
  • 4.Uchino S, Bellomo R, Bagshaw SM et al. Transient azotaemia is associated with a high risk of death in hospitalized patients. Nephrol Dial Transplant 2010;25:1833–9. 10.1093/ndt/gfp624 [DOI] [PubMed] [Google Scholar]
  • 5.Coca SG, Peixoto AJ, Garg AX et al. The prognostic importance of a small acute decrement in kidney function in hospitalized patients: a systematic review and meta-analysis. Am J Kidney Dis 2007;50:712–20. 10.1053/j.ajkd.2007.07.018 [DOI] [PubMed] [Google Scholar]
  • 6.Chertow GM, Burdick E, Honour M et al. Acute kidney injury, mortality, length of stay, and costs in hospitalized patients. J Am Soc Nephrol 2005;16:3365–70. 10.1681/ASN.2004090740 [DOI] [PubMed] [Google Scholar]
  • 7.Wang HE, Muntner P, Chertow GM et al. Acute kidney injury and mortality in hospitalized patients. Am J Nephrol 2012;35:349–55. 10.1159/000337487 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.National confidential enquiry into patient outcome and death. Adding insult to injury: a review of the care of patients who died in hospital with a primary diagnosis of acute kidney injury (acute renal failure) 2009. http://www.ncepod.org.uk/2009report1/Downloads/AKI_report.pdf. http://www.ncepod.org.uk/2009report1/Downloads/AKI_report.pdf: NCEPOD.
  • 9.Aitken E, Carruthers C, Gall L et al. Acute kidney injury: outcomes and quality of care. QJM 2013;106:323–32. 10.1093/qjmed/hcs237 [DOI] [PubMed] [Google Scholar]
  • 10.Wonnacott A, Meran S, Amphlett B et al. Epidemiology and outcomes in community-acquired versus hospital-acquired AKI. Clin J Am Soc Nephrol 2014;9:1007–14. 10.2215/CJN.07920713 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Schissler MM, Zaidi S, Kumar H et al. Characteristics and outcomes in community-acquired versus hospital-acquired acute kidney injury. Nephrology (Carlton) 2013;18:183–7. 10.1111/nep.12036 [DOI] [PubMed] [Google Scholar]
  • 12.Wang Y, Cui Z, Fan M. Hospital-acquired and community-acquired acute renal failure in hospitalized Chinese: a ten-year review. Ren Fail 2007;29:163–8. 10.1080/08860220601095918 [DOI] [PubMed] [Google Scholar]
  • 13.Obialo CI, Okonofua EC, Tayade AS et al. Epidemiology of de novo acute renal failure in hospitalized African Americans: comparing community-acquired vs hospital-acquired disease. Arch Intern Med 2000;160:1309–13. 10.1001/archinte.160.9.1309 [DOI] [PubMed] [Google Scholar]
  • 14.Xu X, Nie S, Liu Z et al. Epidemiology and clinical correlates of AKI in Chinese hospitalized adults. Clin J Am Soc Nephrol 2015;10:1510–18. 10.2215/CJN.02140215 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Forni LG, Dawes T, Sinclair H et al. Identifying the patient at risk of acute kidney injury: a predictive scoring system for the development of acute kidney injury in acute medical patients. Nephron Clin Pract 2013;123:143–50. 10.1159/000351509 [DOI] [PubMed] [Google Scholar]
  • 16. National Institute for Health and Care Excellence (2013a) NICE Clinical Guideline 169: Acute kidney injury: Prevention, detection and management of acute kidney injury up to the point of renal replacement therapy. http://tinyurl.com/najylbl. [Google Scholar]
  • 17.Mehta RL, Cerdá J, Burdmann EA et al. International Society of Nephrology's 0by25 initiative for acute kidney injury (zero preventable deaths by 2025): a human rights case for nephrology. Lancet 2015;385:2616–43. 10.1016/S0140-6736(15)60126-X [DOI] [PubMed] [Google Scholar]
  • 18.Braitman LE, Davidoff F. Predicting clinical states in individual patients. Ann Intern Med 1996;125:406–12. 10.7326/0003-4819-125-5-199609010-00008 [DOI] [PubMed] [Google Scholar]
  • 19.Feinstein AR. “Clinical Judgment” revisited: the distraction of quantitative models. Ann Intern Med 1994;120:799–805. 10.7326/0003-4819-120-9-199405010-00012 [DOI] [PubMed] [Google Scholar]
  • 20.Concato J, Feinstein AR, Holford TR. The risk of determining risk with multivariable models. Ann Intern Med 1993;118:201–10. 10.7326/0003-4819-118-3-199302010-00009 [DOI] [PubMed] [Google Scholar]
  • 21.Christensen E. Prognostic models including the Child-Pugh, MELD and Mayo risk scores--where are we and where should we go? J Hepatol 2004;41:344–50. 10.1016/j.jhep.2004.06.005 [DOI] [PubMed] [Google Scholar]
  • 22.Moons KG, Altman DG, Reitsma JB et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med 2015;162:W1–73. 10.7326/M14-0698 [DOI] [PubMed] [Google Scholar]
  • 23.Altman DG, Vergouwe Y, Royston P et al. Prognosis and prognostic research: validating a prognostic model. BMJ 2009;338:b605 10.1136/bmj.b605 [DOI] [PubMed] [Google Scholar]
  • 24.Moons KG, Kengne AP, Grobbee DE et al. Risk prediction models: II. External validation, model updating, and impact assessment. Heart 2012;98:691–8. 10.1136/heartjnl-2011-301247 [DOI] [PubMed] [Google Scholar]
  • 25.Toll DB, Janssen KJ, Vergouwe Y et al. Validation, updating and impact of clinical prediction rules: a review. J Clin Epidemiol 2008;61:1085–94. 10.1016/j.jclinepi.2008.04.008 [DOI] [PubMed] [Google Scholar]
  • 26.Reilly BM, Evans AT. Translating clinical research into clinical practice: impact of using prediction rules to make decisions. Ann Intern Med 2006;144:201–9. 10.7326/0003-4819-144-3-200602070-00009 [DOI] [PubMed] [Google Scholar]
  • 27.Drawz PE, Miller RT, Sehgal AR. Predicting hospital-acquired acute kidney injury--a case-controlled study. Ren Fail 2008;30:848–55. 10.1080/08860220802356515 [DOI] [PubMed] [Google Scholar]
  • 28.Matheny ME, Miller RA, Ikizler TA et al. Development of inpatient risk stratification models of acute kidney injury for use in electronic health records. Med Decis Making 2010;30:639–50. 10.1177/0272989X10364246 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Bedford M, Stevens P, Coulton S et al. Development of risk models for the prediction of new or worsening acute kidney injury on or during hospital admission: a cohort and nested study. Southampton, UK: Queen's Printer and Controller of HMSO; 2016. [PubMed] [Google Scholar]
  • 30.Sawhney S, Fluck N, Marks A et al. Acute kidney injury-how does automated detection perform? Nephrol Dial Transplant 2015;30:1853–61. 10.1093/ndt/gfv094 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Kramer AA, Zimmerman JE. Assessing the calibration of mortality benchmarks in critical care: the Hosmer-Lemeshow test revisited. Crit Care Med 2007;35:2052–6. 10.1097/01.CCM.0000275267.64078.B0 [DOI] [PubMed] [Google Scholar]
  • 32.Hosmer DW, Lemeshow S. Applied logistic regression. New York: Wiley, 1989:187–215. [Google Scholar]
  • 33.Justice AC, Covinsky KE, Berlin JA. Assessing the generalizability of prognostic information. Ann Intern Med 1999;130:515–24. 10.7326/0003-4819-130-6-199903160-00016 [DOI] [PubMed] [Google Scholar]
  • 34.Collins GS, de Groot JA, Dutton S et al. External validation of multivariable prediction models: a systematic review of methodological conduct and reporting. BMC Med Res Methodol 2014;14:40 10.1186/1471-2288-14-40 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35.Altman DG, Royston P. What do we mean by validating a prognostic model? Stat Med 2000;19:453–73. [DOI] [PubMed] [Google Scholar]
  • 36.Coggon DI, Martyn CN. Time and chance: the stochastic nature of disease causation. Lancet 2005;365:1434–7. 10.1016/S0140-6736(05)66380-5 [DOI] [PubMed] [Google Scholar]
  • 37.Cook NR. Statistical evaluation of prognostic versus diagnostic models: beyond the ROC curve. Clin Chem 2008;54:17–23. 10.1373/clinchem.2007.096529 [DOI] [PubMed] [Google Scholar]
  • 38.Angus DC, Clermont G, Kramer DJ et al. Short-term and long-term outcome prediction with the Acute Physiology and Chronic Health Evaluation II system after orthotopic liver transplantation. Crit Care Med 2000;28:150–6. 10.1097/00003246-200001000-00025 [DOI] [PubMed] [Google Scholar]
  • 39.Halligan S, Altman DG, Mallett S. Disadvantages of using the area under the receiver operating characteristic curve to assess imaging tests: a discussion and proposal for an alternative approach. Eur Radiol 2015;25:932–9. 10.1007/s00330-014-3487-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40.Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation 2007;115:928–35. 10.1161/CIRCULATIONAHA.106.672402 [DOI] [PubMed] [Google Scholar]
  • 41.Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making 2006;26:565–74. 10.1177/0272989X06295361 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42.Gail MH, Pfeiffer RM. On criteria for evaluating models of absolute risk. Biostatistics 2005;6:227–39. 10.1093/biostatistics/kxi005 [DOI] [PubMed] [Google Scholar]
  • 43.Altman DG. Prognostic models: a methodological framework and review of models for breast cancer. Cancer Invest 2009;27:235–43. 10.1080/07357900802572110 [DOI] [PubMed] [Google Scholar]
  • 44.Pencina MJ, D'Agostino RB Sr, D'Agostino RB Jr et al. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 2008;27:157–72; discussion 207-12 10.1002/sim.2929 [DOI] [PubMed] [Google Scholar]
  • 45.Steyerberg EW, Vickers AJ, Cook NR et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 2010;21:128–38. 10.1097/EDE.0b013e3181c30fb2 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46.Silver SA, Shah PM, Chertow GM et al. Risk prediction models for contrast induced nephropathy: systematic review. BMJ 2015;351:h4395 10.1136/bmj.h4395 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 47.Birnie K, Verheyden V, Pagano D et al. Predictive models for kidney disease: improving global outcomes (KDIGO) defined acute kidney injury in UK cardiac surgery. Crit Care 2014;18:606 10.1186/s13054-014-0606-x [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 48.Wong B, St Onge J, Korkola S et al. Validating a scoring tool to predict acute kidney injury (AKI) following cardiac surgery. Can J Kidney Health Dis 2015;2:3 10.1186/s40697-015-0037-x [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 49.Thakar CV, Arrigain S, Worley S et al. A clinical score to predict acute renal failure after cardiac surgery. J Am Soc Nephrol 2005;16:162–8. 10.1681/ASN.2004040331 [DOI] [PubMed] [Google Scholar]
  • 50.Mehta RH, Grab JD, O'Brien SM et al. Bedside tool for predicting the risk of postoperative dialysis in patients undergoing cardiac surgery. Circulation 2006;114:2208–16; quiz 08 10.1161/CIRCULATIONAHA.106.635573 [DOI] [PubMed] [Google Scholar]
  • 51.Ng SY, Sanagou M, Wolfe R et al. Prediction of acute kidney injury within 30 days of cardiac surgery. J Thorac Cardiovasc Surg 2014;147:1875–83, 83 e1 10.1016/j.jtcvs.2013.06.049 [DOI] [PubMed] [Google Scholar]
  • 52.Breidthardt T, Socrates T, Noveanu M et al. Effect and clinical prediction of worsening renal function in acute decompensated heart failure. Am J Cardiol 2011;107:730–5. 10.1016/j.amjcard.2010.10.056 [DOI] [PubMed] [Google Scholar]
  • 53.Wang YN, Cheng H, Yue T et al. Derivation and validation of a prediction score for acute kidney injury in patients hospitalized with acute heart failure in a Chinese cohort. Nephrology (Carlton) 2013;18:489–96. 10.1111/nep.12092 [DOI] [PubMed] [Google Scholar]
  • 54.Xing XZ, Wang HJ, Huang CL et al. Two acute kidney injury risk scores for critically ill cancer patients undergoing non-cardiac surgery. World J Emerg Med 2012;3:278–81. 10.5847/wjem.j.1920-8642.2012.04.007 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 55.Bell S, Dekker FW, Vadiveloo T et al. Risk of postoperative acute kidney injury in patients undergoing orthopaedic surgery-development and validation of a risk score and effect of acute kidney injury on survival: observational cohort study. BMJ 2015;351:h5639 10.1136/bmj.h5639 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 56. Health and Social Care Information Centre. Hospital Episode Statistics. http://www.hesonline.nhs.uk/Ease/servlet/ContentServer?siteID=1937.
  • 57.Porter CJ, Juurlink I, Bisset LH et al. A real-time electronic alert to improve detection of acute kidney injury in a large teaching hospital. Nephrol Dial Transplant 2014;29:1888–93. 10.1093/ndt/gfu082 [DOI] [PubMed] [Google Scholar]
  • 58.Selby NM, Crowley L, Fluck RJ et al. Use of electronic results reporting to diagnose and monitor AKI in hospitalized patients. Clin J Am Soc Nephrol 2012;7:533–40. 10.2215/CJN.08970911 [DOI] [PubMed] [Google Scholar]
  • 59.Raven MC, Billings JC, Goldfrank LR et al. Medicaid patients at high risk for frequent hospital admission: real-time identification and remediable risks. J Urban Health 2009;86:230–41. 10.1007/s11524-008-9336-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 60.Ko M, Lee Y, Chen C et al. Prevalence of and Predictors for Frequent Utilization of Emergency Department: A Population-Based Study. Medicine (Baltimore) 2015;94:e1205 10.1097/MD.0000000000001205 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 61.Longman JM, I Rolfe M, Passey MD et al. Frequent hospital admission of older people with chronic disease: a cross-sectional survey with telephone follow-up and data linkage. BMC Health Serv Res 2012;12:373 10.1186/1472-6963-12-373 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 62.Kirby SE, Dennis SM, Jayasinghe UW et al. Patient related factors in frequent readmissions: the influence of condition, access to services and patient choice. BMC Health Serv Res 2010;10:216 10.1186/1472-6963-10-216 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 63.Kawamoto K, Houlihan CA, Balas EA et al. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005;330:765 10.1136/bmj.38398.500764.8F [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 64.Bates DW, Kuperman GJ, Wang S et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003;10:523–30. 10.1197/jamia.M1370 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 65.Uchino S. Creatinine. Curr Opin Crit Care 2010;16:562–7. 10.1097/MCC.0b013e32833ea7f3 [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

supplementary appendix

bmjopen-2016-013511supp_appendix.pdf (238.2KB, pdf)


Articles from BMJ Open are provided here courtesy of BMJ Publishing Group

RESOURCES