Creating a High-Specificity Acute Kidney Injury Detection System for Clinical and Research Applications

To the Editor:

Although acute kidney injury (AKI) is common and early interventions may be beneficial, recognition of AKI remains a challenge.¹ By allowing earlier detection, AKI alerts in electronic medical records (EMR) have been shown to improve outcomes.^2,3 However, alerts can lead to “alert fatigue” and negatively impact workflows,⁴ especially when there are high false-positive rates. We have previously reported improved mortality, decreased length of stay, and reduced need for dialysis with implementation of an electronic AKI alert.² Our alert fired when an admission serum creatinine level (Scr) was increased 50% or more from the lowest level recorded for that patient in the previous year or if Scr increased by 0.3 mg/dL. or more within any 52-hour window (48 hours plus 4 hours to account for variation in testing times) during the hospitalization. However, our approach, which was used until 2015, resulted in a false-positive rate >10%.² We now report on refinements to the alert firing threshold to decrease the false-positive rate.

As part of an approved, ongoing quality improvement project, we collected data from EMR for all adults (age >18 years) admitted across the 17 hospitals in our health care system between January 2016 and March 2018. Our only exclusion criteria were pre-existing kidney failure treated by kidney replacement therapy or receiving dialysis on admission. We stratified patients according to whether they developed AKI defined by KDIGO criteria for Scr⁵ as well as a clinical diagnosis for AKI (ICD-10 code).

To reduce false-positive rates, we made 2 modifications to the computer logic for AKI detection. The first ‘was to use the median, instead of the lowest value, of Scr for the patient in EMR for last 12 months for reference Scr (for the 75% of patients with prior values). The second modification was an added requirement that, in addition to an increase of 0.3 mg/dL or more from previous Scr in last 52 hours, the rise had to be 0.3 mg/dL above the reference Scr to activate the alert (Fig S1). We then calculated the sensitivity and specificity of the alert both by using ICD-10 codes and by clinical adjudication as described in Item S1.

We identified 337,380 eligible patient admissions during the study period (Table 1). The alert fired in 70,033 (20.8%) of these patients and 39,494 (11.7%) were also associated with an ICD-10 code. The alert rates fell from 25.3% in 2015 to 20.8% in 2016–2018 with more stringent criteria (adjusted OR, 0.64 [95% CI, 0.62–0.66]; P< 0.001). The clinical diagnosis of AKI (new or old AKI alert together with an ICD code), however, remained stable (OR, 1.001 [95% Cl, 0.997–1.006]; P=0.6).

Table 1.

Baseline Characteristics and Unadjusted Outcomes

Characteristic	Alert & AKI Code (n = 39,494)	No Alert & No AKI Code (n = 251,785)	Alert & No AKI Code (n = 30,539)	No Alert & AKI Code (n = 15,615)
Age, ya	69.2 ± 15.6	62.2 ± 18.1	68.3 ± 16.6	70.8 ± 15.6
Female sex	18,069 (45.8%)	133,266 (52.9%)	17,781 (58.2%)	6,504 (41.7%)
Race
African American	4,756 (12.0%)	24,271 (9.6%)	2,881 (9.4%)	2,190 (14.0%)
White	32,896 (83.3%)	217,159 (86.3%)	26,367 (86.3%)	13,055 (83.6%)
Other	226 (4.4%)	963 (3.5%)	74 (2.9%)	83 (2.8%)
Type of patient
Medical	28,202 (71.4%)	152,001 (60.4%)	19,668 (64.4%)	12,664 (81.1%)
Surgical	11,292 (28.6%)	99,784 (39.6%)	10,871 (35.6%)	2,951 (18.9%)
Sepsis	8,469 (21.4%)	8,899 (3.5%)	2,762 (9.0%)	1,828 (11.7%)
Cardiothoracic surgery	1,429 (3.6%)	8,776 (3.5%)	1,636 (5.4%)	262 (1.7%)

^aValues given as mean ± SD.

We randomly selected 100 cases from each of the 4 strata defined by the new alert status and presence of ICD-10 codes. We then reviewed them for clinically adjudicated AKI. Among the 100 cases reviewed out of 39,494 patients where both the alert fired and the ICD-10 code was present, all cases were confirmed to have clinically adjudicated AKI. Of 15,615 patients with an ICD-10 code for AKI but without an alert, none of the 100 patient records reviewed had clinically adjudicated AKI. Among the 100 cases reviewed out of 251,732 where neither the alert fired nor the ICD-10 code for AKI was present, we found no cases of clinically adjudicated AKT. Finally, among the 100 patient charts reviewed out of the 30,539 cases where the alert fired but there was no associated ICD-10 code for AKI, we found that in 68 cases, AKI was not confirmed on clinical review, indicating 68% false-positives in this subgroup and corresponding to a 6.3% [(25+0)/400] false alert rate overall, when clinically adjudicated AKI is used as the gold standard. Assuming this rate is representative of alerts where an ICD-10 code was absent, the theoretical specificity would be 92.7% (see Table 2 for calculations). Thus, as shown in Table 2, despite making the alerting criteria more stringent, theoretical sensitivity appeared to remain at 100%. We call this theoretical sensitivity because we did not review all cases.

Table 2.

Sensitivity and Specificity of Alert for AKI

	Clinically Adjudicated Gold Standard
	AKI	Not AKI	Total
New alert logic
Positive alert	58	25	83
Negative alert	0	317	317
Total	58	342	400

Two by two table for alert results (positive or negative) comparing clinical adjudication of a sample of 400 patients. New alert logic sensitivity = 58/(58+0) = 100%; specificity = 317/(317+25) = 92.7% Sensitivity and specificity calculated by weighting the reviewed cases to account for proportions in the entire cohort (detailed methodology in Item S1).

Thus, we show that the use of median values for baseline Scr and excluding normalization of Scr after a decrease improves alert specificity and results in lower alert rates while still maintaining 100% sensitivity. These improvements provide better agreement with clinically adjudicated AKI and should therefore improve performance of clinical decision support systems as well as epidemiologic studies involving electronic health record data.