Assessing the accuracy of ICD-10 coding for measuring rates of and mortality from acute kidney injury and the impact of electronic alerts: an observational cohort study

Abstract

Background

The application of a uniform definition for acute kidney injury (AKI) is vital to advance understanding and management of AKI. International Classification of Diseases (Tenth Revision) (ICD-10) coding is frequently used to define AKI, but its accuracy is unclear. The aim of this study was to determine whether ICD-10 coding is a reliable method of monitoring rates and outcomes of AKI in inpatients compared with biochemically defined AKI, and whether electronic alerts (e-alerts) for AKI affect ICD-10 AKI coding.

Methods

An observational cohort study of all 505 662 adult admissions to acute hospitals in two Scottish Health Boards [National Health Service (NHS) Tayside and NHS Fife] from January 2013 to April 2017 was performed. AKI e-alerts were implemented in NHS Tayside in April 2015. Sensitivity, specificity, positive and negative predictive values of ICD-10 coding for AKI compared with biochemically defined AKI using the Kidney Disease: Improving Global Outcomes definition and relative risk of 30-day mortality in people with ICD-10 and biochemically defined AKI before and after AKI e-alert implementation were performed.

Results

Sensitivity of ICD-10 coding for identifying biochemically defined AKI was very poor in both health boards for all AKI (Tayside 25.7% and Fife 35.8%) and for Stages 2 and 3 AKI (Tayside 43.8% and Fife 53.8%). Positive predictive value was poor both for all AKI (Tayside 76.1% and Fife 45.5%) and for Stages 2 and 3 AKI (Tayside 45.5% and Fife 36.8%). Measured mortality fell following implementation of AKI e-alerts in the ICD-10-coded population but not in the biochemically defined AKI population, reflecting an increase in the proportion of Stage 1 AKI in ICD-10-coded AKI. There was no evidence that the introduction of AKI e-alerts in Tayside improved ICD-10 coding of AKI.

Conclusion

ICD-10 coding should not be used for monitoring of rates and outcomes of AKI for either research or improvement programmes.

INTRODUCTION

Acute kidney injury (AKI) is associated with adverse patient outcomes including increased length of hospital stay, mortality and future development of chronic kidney disease [1–6]. However, there has previously been no universally accepted definition of AKI and so establishing its true incidence has been difficult [7–9]. The Kidney Disease: Improving Global Outcomes (KDIGO) definition for AKI was developed in 2012 [10]. This has been universally adopted and has led to a greater understanding of the epidemiology and adverse outcomes associated with AKI. There is a worldwide recognition of the need to improve AKI care, with initiatives such as the global World Kidney Day providing a platform to increase the recognition and awareness of AKI [11] and with many countries launching national improvement programmes, including the National Health Service (NHS) England ‘Think Kidneys’ campaign [12] and the Scottish Patient Safety Programme AKI national collaborative [13]. Electronic alerts (e-alerts) for AKI, which are based on the KDIGO definition, have been implemented in many hospitals worldwide to facilitate earlier recognition.

Despite this, much research and many improvement programmes still use hospital International Classification of Diseases (Tenth Revision) (ICD-10) discharge coding to identify AKI [14–17]. ICD-10, first published in 1994 [18], does not account for current AKI definitions, with the code N17 (acute renal failure) commonly being used despite not distinguishing between different AKI stages or including a biochemical definition [19].

The aim of this study was to determine whether ICD-10 coding is a reliable method of identifying cases of AKI and measuring AKI-associated mortality compared with AKI defined biochemically using the KDIGO definition and whether ICD-10 coding rates changed with the introduction of AKI e-alerts in NHS Tayside in 2015.

MATERIALS AND METHODS

Study design

The study design was an observational cohort study. We included all patients aged ≥18 years and not receiving renal replacement therapy (RRT) who were admitted to an NHS acute hospital in Tayside or Fife health boards between 1 January 2013 and 30 April 2017. Apart from some elective surgery, all inpatient healthcare and biochemistry laboratory testing in the two health boards studied is provided by the NHS. There are approximately 40 000 medical admissions, 16 000 surgical admissions and 2800 intensive care admissions per year to NHS Tayside. NHS Fife has 34 000 medical admissions, 13 000 surgical admissions and 2400 intensive care admissions per year.

Data sources

Data were provided by the Health Informatics Centre (HIC) at the University of Dundee [20], linking data from the Community Health Index (CHI) patient register; Scottish Morbidity Record of hospital admissions excluding psychiatric admissions (SMR01); NHS biochemistry results; General Register Office (GRO) national death registration; and the Scottish Renal Registry [21]. Data on ICD-10 discharge coding were obtained from SMR01, data on biochemically defined AKI were obtained from the laboratory results database and data on mortality were obtained from the GRO death database and CHI database. The Scottish Renal Registry [21] was used to identify those who were receiving RRT (chronic dialysis or transplant) prior to admission.

Definitions

ICD-10-coded AKI was defined as a patient having N17 (acute renal failure) recorded as a discharge diagnosis. Biochemical AKI was defined using the NHS England algorithm based on the KDIGO definition (Supplementary data, Appendix) [10, 12]. Baseline was taken as the median creatinine level in the period between 8 and 365 days prior to the index creatinine measurement or, if not available, the lowest level in the period 0–7 days prior to the index measurement or the lowest level between 0 and 2 days prior to the index measurement. In the absence of any baseline measure, an increase of >26 µmol/L in creatinine level in a 48-h window was also labelled as AKI Stage 1.

AKI Stage 1 was defined as an increase in serum creatinine of ≥26.4 mmol/L or an increase of 1.5–1.9 times baseline. AKI Stage 2 was defined as an increase in serum creatinine to 2–2.9 times the baseline value. AKI Stage 3 was defined as an increase in serum creatinine to ≥3 times the baseline value or serum creatinine of ≥354 mmol/L or initiation of RRT. Where there were multiple AKI episodes identified in biochemistry data, the highest stage of AKI during the admission was used.

Statistical analysis

All analyses were performed using Stata (Version 14) and IBM SPSS (Version 22) software.

Validity and predictive value of ICD-10 coding for AKI

The sensitivity, specificity, and positive and negative predictive values of ICD-10 code N17 compared with biochemically defined AKI were calculated with 95% confidence intervals (CIs). These values were calculated for the overall study period as well as for before and after the introduction of the AKI e-alerts. We separately compared ICD-10 coding with all biochemically defined AKI and for AKI Stages 2 and 3 combined. To examine whether ascertainment of less severe AKI using ICD-10 coding changed overtime, the proportion of total AKI consisting of AKI Stage 1 was also reported for pre- and post-intervention periods for ICD-10-coded cases that were also biochemically defined.

Mortality rates of coded versus biochemically defined AKI

Thirty-day mortality was calculated for people with ICD-10-coded AKI and biochemically defined AKI, defined as the number of deaths as a proportion of the number of AKI cases from date of admission. Data were tested for normality using the Shapiro–Wilk test [22]. The relative risk (RR) of death before and after the introduction of the AKI e-alerts was calculated for coded and biochemically defined AKI, with 95% CIs reported for the results.

Interrupted time series and segmented regression analysis

An interrupted time series with segmented regression analysis [23] was used to assess whether there were changes in the proportion of admissions coded as having AKI associated with the introduction of the e-alerts in NHS Tayside (April 2015) compared with changes in NHS Fife at the same time (where e-alerts were not introduced). Monthly rates of ICD-10-coded AKI were defined as the number of patients coded as having AKI as a proportion of all patients aged ≥18 years admitted to hospital each month. Rates were plotted over time, and the functional form of the relationship before and after the intervention was assessed for linearity. The Kolmogorov–Smirnov test [24] and the Shapiro–Wilk test [22] were used to check data were normally distributed. Segmented regression analysis was used to examine changes associated with the introduction of e-alerts. The Durbin–Watson statistic was used to explore first-order autocorrelation [25], with adjustment for autocorrelation using lag terms as required. The changes in level and trend were reported with 95% CIs, and the effect size of the intervention at 24 months was converted into a relative percentage difference between the predicted and the actual ICD-10-coded AKI rates.

Ethical considerations

Anonymized record linkage was performed according to HIC Standard Operating Procedures (SOPs). The Tayside Research Ethics Committee does not require submission of individual studies that follow SOPs. The research protocol was reviewed and approved by the University of Dundee, School of Medicine Research Ethics Committee.

RESULTS

There were 240 227 eligible hospital admissions in NHS Tayside and 265 435 eligible hospital admissions in NHS Fife between 1 January 2013 and 30 April 2017 (Figure 1). The demographic characteristics of NHS Tayside and NHS Fife cohorts were similar for age, sex and deprivation status. The mean (standard deviation [SD]) age for those included for analysis was 63 (19) in Tayside and 61 (18) in Fife. Tayside’s cohort was 53.9% female, comparable to 54.9% female in Fife. In Tayside, 34% of the cohort resided in the two most deprived Scottish Index of Multiple Deprivation (SIMD5) quintiles, compared with 43% in Fife.

FIGURE 1.

Flow chart showing the derivation of cohorts from the Tayside and Fife regions.

Over the course of the study, there were 20 967 episodes of biochemically defined AKI (13 638 AKI Stage 1 and 7239 Stages 2 and 3) and 7068 cases of N17 ICD-10-coded AKI in NHS Tayside. In NHS Fife, there were 17 454 episodes of biochemically defined AKI (11 029 AKI Stage 1 and 6425 Stages 2 and 3) and 9386 cases of ICD-10-coded AKI.

There were 413 cases of coded AKI in Tayside that fell into a set of ICD-10 codes used by Mansfield et al. [14] to define AKI, and 633 cases in Fife (Supplementary data, Table S1). However, our analysis focused on the N17 code only.

Missing data

There were 7983 patients in Tayside and 9504 patients in Fife for whom demographic data from SIMD5 could not be obtained.

Assessing the predictive value of ICD-10 coding for AKI

In both health boards over the whole period studied, sensitivity of ICD-10 coding compared with gold standard biochemically defined AKI was poor for all stages of AKI (25.7% in Tayside and 35.8% in Fife) and for Stages 2 and 3 AKI (43.8% in Tayside and 53.8% in Fife). Positive predictive values were moderate or poor for all AKI (76.1% in Tayside and 45.5% in Fife) and for Stage 2 and 3 AKI (45.5% in Tayside and 36.8% in Fife). Specificity and negative predictive value over the whole period studied were both consistently high (≥97.7% and ≥93.3%, respectively) (Table 1).

Table 1.

Sensitivity, specificity, and positive and negative predictive values of ICD-10 code N17 for biochemically defined AKI

Health board	Sensitivity (95% CI)	Specificity (95% CI)	Positive predictive value (95% CI)	Negative predictive value (95% CI)
Over study period all stages
Tayside	25.7 (25.0–26.3)	99.2 (99.2–99.3)	76.1 (75.1–77.0)	93.3 (93.3–93.4)
Fife	35.8 (35.1–36.5)	98.7 (98.7–98.8)	66.5 (65.6–67.4)	95.6 (95.6–95.7)
Over study period Stages 2 and 3
Tayside	43.8 (42.7–45.0)	98.3 (98.3–98.4)	45.5 (44.5–46.5)	98.2 (98.2–98.3)
Fife	53.8 (52.6–55.0)	97.7 (97.7–97.8)	36.8 (36.0–37.6)	98.8 (98.8–98.9)

In Tayside, there was an increased proportion of coded AKI consisting of AKI Stage 1 (N = 5378; 76.1% of total ICD-10-coded AKI) following the introduction of the AKI e-alerts [pre-intervention: 38.1%; post-intervention: 42.4%; +4.3% (1.6–6.9)] but a non-significant reduction in those with biochemically defined AKI [pre-intervention: 65.6%; post-intervention: 64.4%; −1.3% (−2.6 to 0.04)]. In NHS Fife, where e-alerts were not introduced, there was a similar increase in the proportion of coded AKI consisting of AKI Stage 1 (N = 6242; 66.5% of total ICD-10-coded AKI) for the same time frame [pre-intervention: 41.3%; post-intervention: 48.3%; +6.9% (4.5–9.4)], though there was a small increase in those with biochemically defined AKI [pre-intervention: 62.1%; post-intervention: 64.6%; +2.5% (1.0–3.9)].

In Tayside, with N17-coded AKI, sensitivity was 23.7% (22.9–24.5%) before the introduction of the alerts and 27.9% (27.1–28.8%) after the introduction of the alerts. For more severe AKI, pre-intervention sensitivity was 42.6% (41.1–44.2%) and post-intervention was 45.2% (43.5–46.9%) (Table 2). In NHS Fife, sensitivity of N17 was 33.2% (32.3–34.1%) before the introduction of the alerts in Tayside and 39.1% (38.0%–40.2%) after the introduction of the alerts. For more severe AKI stages, pre-intervention sensitivity was 51.4% (49.8–53.0%) and post-intervention sensitivity was 57.1% (55.2–59.0%) (Table 2).

Table 2.

Sensitivity [% (95% CI)] before and after Tayside e-alert introduction of ICD-10 code N17 for biochemically defined AKI

Health board	Sensitivity before Tayside e-alerts	Sensitivity after Tayside e-alerts	Difference in sensitivity
Tayside
All stages	23.7 (22.9–24.5)	27.9 (27.1–28.8)	4.27 (4.27–4.28)
Stages 2 and 3	42.6 (41.1–44.2)	45.2 (43.5–46.9)	2.57 (2.56–2.58)
Fifea
All stages	33.2 (32.3–34.1)	39.1 (38.0–40.2)	5.87 (5.87–5.88)
Stages 2 and 3	51.4 (49.8–53.0)	57.1 (55.2–59.0)	5.69 (5.68–5.71)

^aThere was no intervention in NHS Fife, with AKI e-alerts only implemented in Tayside during the period of analysis. Pre- and post-interventions are for comparison with observed differences in NHS Tayside.

Mortality rates after admission with ICD-10-coded versus biochemically defined AKI

Thirty-day mortality was high irrespective of how AKI was identified (Table 3). Thirty-day mortality after date of admission for ICD-10-coded AKI was 19.8 and 16.0% before and after e-alert introduction in NHS Tayside and 18.7 and 16.8% in the same periods in NHS Fife. For those with biochemically defined AKI, 30-day mortality after date of admission was 18.2 and 18.1% in NHS Tayside before and after e-alert introduction and was 18.5 and 19.6% in the same periods in NHS Fife. In both health boards, 30-day mortality rates were lower in ICD-10-coded AKI in the period after NHS Tayside introduced e-alerts [NHS Tayside RR 0.81 (0.73–0.90) and NHS Fife RR 0.90 (0.82–0.98)]. However, there were no differences in 30-day mortality rates in people with biochemically defined AKI [NHS Tayside RR 1.00 (0.94–1.06) and NHS Fife RR 1.06 (0.99–1.12)].

Table 3.

Mortality (%) over study period and relative risk (95% CI) in post- versus pre-intervention periods^a

	Tayside		Fife
	Pre-intervention	Post-intervention	Pre-intervention	Post-intervention
N17 ICD-10-coded AKI: 30-day mortality
Total	3535	3533	4644	4742
30-Day mortality (N)	700	567	870	798
% Mortality	19.8	16.0	18.7	16.8
Relative risk (95% CI)	0.81 (0.73 to 0.90)		0.90 (0.82 to 0.98)
Biochemical AKI: 30-day mortality
Total	11 243	9724	9829	7625
30-Day mortality (N)	2046	1763	1823	1495
% Mortality	18.2	18.1	18.5	19.6
Relative risk (95% CI)	1.00 (0.94–1.06)		1.06 (0.99–1.12)

^aThis analysis included all cases of ICD-10-coded AKI.

Interrupted time series and segmented regression analysis of change in ICD-10-coded AKI after the introduction of e-alerts

The pre- and post-intervention rates were plotted over time, as shown in Figure 2. Over the course of the study, 31/1000 of NHS Tayside inpatients were coded with ICD-10 code N17 compared with 35/1000 in NHS Fife. Results of segmented regression analysis (Table 4) for NHS Tayside showed that there was no significant change in level at the time of the introduction of the e-alerts and the change in trend indicated that coded AKI rates had in fact fallen since the introduction of the alerts compared with prior trends [β₁: 0.36 (0.15–0.57), β₂: −0.28 (−4.41 to 3.84) and β₃: −0.39 (−0.69 to −0.08)]. The absolute change at 24-months post-intervention was −9.5 (−17.2 to −1.8) and the relative change was −27.0% (−42.9% to −11.0%), with the overall effect being a flattening of the previous rising trend. Results for NHS Fife demonstrated that there was no change in either level or trend over the course of the study [β₁: 0.40 (0.10 to 0.69), β₂: −2.61 (−7.82 to 2.60) and β₃: −0.29 (−0.64 to 0.05)].

FIGURE 2.

Rates of biochemically defined (red) and ICD-10-coded AKI (blue) before and after the implementation in NHS Tayside of AKI e-alerts (vertical line, e-alerts are not implemented in NHS Fife but changes at the same time point are modelled).

Table 4.

Segmented regression analysis of interrupted time-series data to determine whether coded rates of AKI are changing overtime

Health board	Coefficient (95% CI)	P-value
Tayside
Baseline trend ${\beta }_1$	0.36 (0.15 to 0.57)	0.00
Level change ${\beta }_2$	−0.28 (−4.41 to 3.84)	0.89
Trend change ${\beta }_3$	−0.39 (−0.69 to −0.08)	0.02
Fife
Baseline trend ${\beta }_1$	0.40 (0.10 to 0.69)	0.01
Level change ${\beta }_2$	−2.61 (−7.82 to 2.60)	0.32
Trend change ${\beta }_3$	−0.29 (−0.64 to 0.05)	0.09

Level change: immediate change in AKI rates due to intervention.

Trend change: change in slope gradient pre- and post-interventions.

DISCUSSION

This large observational study of more than 500 000 inpatients showed that using ICD-10 codes to monitor rates and outcomes from AKI is unreliable and potentially misleading, with unacceptably poor sensitivity and moderate positive predictive value compared with biochemically defined AKI. This was true for all AKI and the subset of more severe Stages 2–3 AKI. We anticipated that the introduction of the AKI e-alerts in NHS Tayside in April 2015 would lead to an increase in the rates of ICD-10-coded AKI because of increased recognition and awareness by clinicians. However, this was not the case, with similar patterns of coding over time observed in both NHS Tayside (where e-alerts were introduced) and NHS Fife (where they were not). Incidence is therefore seriously underestimated using ICD-10-defined AKI, and mortality additionally falsely appears to reduce over time because of increased ICD-10 coding of less severe biochemically defined cases.

The consistently low sensitivity of ICD-10 code N17 for biochemically defined AKI has significant implications for monitoring the impact of the growing number of programmes to improve AKI care. Using ICD-10 code N17 to monitor outcomes from AKI improvement programmes is likely to significantly underestimate true AKI rates, and as the sensitivity varies by AKI stage and over time, evaluation of changes in incidence and severity are at high risk of bias. The results also suggest that there is unlikely to be a simple way to improve the accuracy of ICD-10 coding for AKI given that the coding rates were largely unaffected by the e-alerts, a system designed to improve recognition and awareness of AKI.

The findings also show that observed reductions in mortality are at high risk of ascertainment bias. The mortality rate of ICD-10-coded AKI appears to be reducing over time, but this is likely due to a changing denominator where more cases of Stage 1 AKI are coded over time. This increasing proportion of less severe AKI cases being coded makes it difficult to assess the true impact of improvement efforts on mortality rates using ICD-10-coded AKI.

Several studies have assessed the validity of administrative database coding for AKI [26–32]. Vlasschaert et al.’s [33] review in 2011 drew on a range of studies that mostly used ICD-9 codes and concluded that sensitivity to detect AKI was poor overall (median 29%) and positive predictive values were variable, comparable to our findings. Hwang et al. [31] assessed the validity of ICD-10 code N17 specifically, and this study was published after the Vlasschaert review. They found sensitivity of 37.4% for people presenting to the emergency department and 61.6% for those admitted to hospital [31]. However, Hwang’s study only included those aged ≥66 years and, unlike our study, the reference standard used was not based upon the most recent KDIGO AKI definition [10], which includes less severe AKI cases. This would be consistent with the higher sensitivity (43.8% in Tayside and 53.8% in Fife) for Stages 2 and 3 AKI observed in this study. None of the studies examining sensitivity and other performance measures examined change over time in the context of AKI e-alert introduction. A recently published study by Campbell et al. [34], which looked to determine the extent of under-reporting of AKI using the ICD-10-AM (Australian Modification) in four Australian hospitals, found there to be a poor sensitivity of the ICD-10-AM codes for picking up AKI, improving for more severe AKI cases [34], which supports the findings of our study. Siew and Davenport. [35] reviewed the rising trends of AKI incidence and explored possible causes, discussing that the rising trends in coded AKI reported in some studies may be explained by an increased awareness of AKI and changes in coding practices, in particular when discharge codes can be used as part of healthcare reimbursement [35]. This is an interesting comparison to make with our study, where significant increases in AKI coding rates were not noted over time, in a region where ICD-10 coding is not used for healthcare reimbursement. Waikar et al. had previously alluded to the possibility that a decline in mortality could in part be explained by increased coding of less severe AKI [26] and Sawhney and Fraser [36] also discussed the easier recognition of milder AKI as an explanation for falling mortality in some studies [36], but the limitations of using coding data to study AKI mortality has not been fully addressed in previous studies. ICD-10 will be replaced by ICD-11, which will differentiate between different AKI stages and include a biochemical definition [37], but whether this will lead to improved coding of AKI remains unclear. While utilizing biochemistry data to measure AKI for research and improvement would be ideal, access to such datasets can be complex and challenging.

Strengths of this study are its large size, examination in two health boards serving distinct populations where coding practices could plausibly vary and the use of a robust quasi-experimental design to examine the impact of AKI e-alerts on coding practices [23, 38]. When examining the performance of ICD-10 coding to identify AKI, we defined AKI using the KDIGO definition [10].

However, there were some weaknesses. While being able to compare NHS Tayside and NHS Fife is an important strength of this study, there are also factors that differ between the regions that were difficult to account for within the analysis. Within Tayside, over the last few years, there have been several initiatives introduced to improve AKI recognition and care. In addition to the introduction of the e-alerts, the first Scottish health board to do so, AKI management guidelines have been redeveloped along with an educational video aimed at junior doctors and nursing staff [39]. An educational lanyard card was developed to provide an easily accessible reminder about risk factors, essential tests and management steps for AKI, and both the video and the lanyard card were used at junior doctor induction and as part of AKI Awareness Week [39]. These efforts to increase awareness and management of AKI in Tayside were not paralleled with the Fife care system, and this may have had an impact on the results.

Throughout Scotland, ICD-10 coding is based upon discharge diagnoses stated within the discharge letters, often by junior members of the team. An administrative coder is then responsible for using the discharge diagnoses provided to select the administrative code that best aligns itself with the diagnoses provided. This process is therefore limited by the accuracy of the discharge letter provided and the experience of the coding staff. Additionally, there are considerable differences in the financial incentives to ensure accurate ICD-10 coding among Scotland, England and other regions such as the USA and Canada. The introduction of the Payment by Results scheme for NHS England, which is based on ICD-10 coding, is suggested to have led to an improvement in coding accuracy in a systemic review [40]. Similarly, regions like the USA and Canada use ICD-10 coding for reimbursement and allocation of resources, and therefore it is plausible that there is an increased drive in these regions to ensure accurate coding practices are upheld. However, the same study also notes that there were no considerable differences between Scotland and England [40], despite the Payment by Results system not being used in Scotland. Additionally, different regions can modify the World Health Organisation ICD-10 codes for use in their own region, such as ICD-10-CA (Canada) [41], ICD-10-CM (Clinical Modification, USA) [42] and ICD-10 Fifth Edition (UK) [43]. It is difficult to say what impact these variabilities may have on the generalizability of the results obtained in this study.

CONCLUSIONS

There are two main implications of this study. First, based on these findings and previous research, ICD-10 coding is not sufficiently robust to use to identify AKI for either research or improvement programmes.

Secondly, AKI e-alerts did not have any obvious impact on AKI coding. Although the primary purpose of e-alerts is to improve AKI detection and management, this was unexpected. However, it is consistent with the evidence that AKI e-alerts have limited effect on patient outcomes [44]. There is a need for more research evaluating the impact of e-alerts and other interventions in this population.

SUPPLEMENTARY DATA

Supplementary data are available at ckj online.