Abstract
Objective
To determine whether electronic health record alerts for acute kidney injury would improve patient outcomes of mortality, dialysis, and progression of acute kidney injury.
Design
Double blinded, multicenter, parallel, randomized controlled trial.
Setting
Six hospitals (four teaching and two non-teaching) in the Yale New Haven Health System in Connecticut and Rhode Island, US, ranging from small community hospitals to large tertiary care centers.
Participants
6030 adult inpatients with acute kidney injury, as defined by the Kidney Disease: Improving Global Outcomes (KDIGO) creatinine criteria.
Interventions
An electronic health record based “pop-up” alert for acute kidney injury with an associated acute kidney injury order set upon provider opening of the patient’s medical record.
Main outcome measures
A composite of progression of acute kidney injury, receipt of dialysis, or death within 14 days of randomization. Prespecified secondary outcomes included outcomes at each hospital and frequency of various care practices for acute kidney injury.
Results
6030 patients were randomized over 22 months. The primary outcome occurred in 653 (21.3%) of 3059 patients with an alert and in 622 (20.9%) of 2971 patients receiving usual care (relative risk 1.02, 95% confidence interval 0.93 to 1.13, P=0.67). Analysis by each hospital showed worse outcomes in the two non-teaching hospitals (n=765, 13%), where alerts were associated with a higher risk of the primary outcome (relative risk 1.49, 95% confidence interval 1.12 to 1.98, P=0.006). More deaths occurred at these centers (15.6% in the alert group v 8.6% in the usual care group, P=0.003). Certain acute kidney injury care practices were increased in the alert group but did not appear to mediate these outcomes.
Conclusions
Alerts did not reduce the risk of our primary outcome among patients in hospital with acute kidney injury. The heterogeneity of effect across clinical centers should lead to a re-evaluation of existing alerting systems for acute kidney injury.
Trial registration
ClinicalTrials.gov NCT02753751.
Introduction
Acute kidney injury—an abrupt decline in the kidney’s filtration function—is a common occurrence in patients in hospital and is associated with nearly a 10-fold increase in inpatient mortality.1 2 3 In survivors, acute kidney injury is associated with increased risks of chronic kidney disease, end stage kidney disease, and mortality.4 5 Despite the strong association with important outcomes, the syndrome (which is asymptomatic) is often missed in the course of clinical care; it occurs in roughly 15% of patients in hospital and yet is recorded in the electronic health record in fewer than half of patients with acute kidney injury.6 7 8 9 10 Best practices for acute kidney injury, such as routine urinalysis and measurement of urine output, occur infrequently, and even repeat measurement of creatinine is not universal. These deficiencies provide a reason for alerting providers about the presence of acute kidney injury.11
With the assumption that increased recognition of acute kidney injury will improve care of these patients and thus improve clinical outcomes, many health systems in the United States, and the National Health Service in the United Kingdom, have introduced an electronic alert for acute kidney injury as part of routine clinical care.12 13 14 Data on the efficacy of these alerts are limited, however, and definitive evidence of their benefit for patient outcomes remains absent. Additionally, as acute kidney injury is a heterogeneous condition, lacking proven strategies for its mitigation, an alert might prompt actions that are unnecessary or even harmful.
The effectiveness of alerting probably depends on a combination of patient, provider, and disease factors, and might differ in different care environments, thus requiring rigorous evaluation. We conducted a multicenter randomized clinical trial of usual care versus usual care plus pop-up acute kidney injury alerts delivered through an electronic health record. The alert was designed to reach multiple providers and was implemented using an identical protocol across a diverse set of hospitals. We hypothesized that alerts would influence provider behavior and improve clinical outcomes.
Methods
Extended details of the methods are shown in a supplementary web appendix.
Trial design and oversight
A detailed description of the trial design and rationale has been previously published.15 Briefly, we conducted a patient level, parallel group, randomized controlled trial of health record electronic alerts versus usual care in adults with acute kidney injury at six hospitals in the Yale New Haven Health System in Connecticut and Rhode Island, US. The trial was conducted in accordance with the principles of the Declaration of Helsinki, and two institutional review boards associated with the six study hospitals approved the study, which was deemed minimal risk. Patients were enrolled, and a requirement for informed consent was waived, as the alert was deemed unlikely to affect patient welfare and informing patients of their diagnosis of acute kidney injury would contaminate the usual care group. An external data and safety monitoring board performed four independent assessments during the trial, including one formal interim analysis at 50% recruitment. These analyses were not stratified by hospital. Hospitals were given guidelines to terminate the study if the P value associated with interim analysis was less than or equal to 0.001 for efficacy or less than or equal to 0.005 for harm. The hospitals also had the authority to terminate the study of their own accord regardless of those recommendations. This trial was registered with clinicaltrials.gov under registration number NCT02753751 on 25 April 2016, before study initiation and patient enrolment.
Patients
Inpatient adults aged 18 or older with acute kidney injury, defined by the Kidney Disease: Improving Global Outcomes (KDIGO) criteria as an increase in creatinine 0.3 mg/dL (26.5 μmol/L) within 48 hours or 1.5 times the lowest measured creatinine within the previous seven days of admission to hospital, were eligible.16 Owing to missing and inaccurate urine output data, urine output was not used to classify acute kidney injury. Patients with a history of end stage kidney disease (based on International Classification of Disease, ninth and 10th revision (ICD-9 and ICD-10) codes, a dialysis order in the past year, or an initial hospital creatinine greater than 354 μmol/L were excluded automatically by the detection algorithm. We subsequently excluded patients whose admission date was before the inception of alerts at a study hospital, whose first alert occurred after hospital discharge, who had been enrolled in a previous study, and who were enrolled during a two week period in which alerting ceased owing to an upgrade of the electronic health record system (fig S1).
Randomization and masking
Eligible patients were identified using an acute kidney injury detection algorithm built into our best practice alert within the Epic electronic health record system. When the chart is opened, the algorithm automatically assesses the patient record for inclusion and exclusion criteria. If all criteria are met, the patient is automatically enrolled into the trial and randomized. Simple randomization was performed within Epic using an internal random number rule with concealed allocation, as our electronic health record system cannot perform permutated block randomization or import external randomization lists. Randomization was performed at the patient rather than the provider level, as inpatients at participating hospitals are often cared for by many providers who change throughout their hospital stay. All study investigators were blinded to patient randomization status until enrollment was complete.
Intervention
The intervention was an automated, electronic, pop-up alert which fired whenever the patient’s electronic chart (Epic Systems, Verona, WI) was opened (fig 1). Alerts were displayed only to individuals who had authority to change or enter new orders on behalf of the patient—hereafter referred to as “providers”—which included interns, residents, fellows, attending physicians, nurse practitioners, and physician’s assistants. Alerts were displayed each time the chart was opened, provided the patient continued to meet criteria for acute kidney injury. If the provider agreed or disagreed with the presence of acute kidney injury, the alert was suppressed for 48 hours for that provider. If multiple providers used the electronic health record to care for the same patient, each of them separately received the alert whenever they opened the patient chart. Patients randomized to the usual care group generated “silent” alerts, which did not display to providers but were tracked, allowing the study team to observe which type of providers opened the chart and how often.
Fig 1.
Screenshot of the acute kidney injury (AKI) alert. Creatinine 1 mg/dL=88.42 μmol/L.
The alert contained an option to add acute kidney injury to the patient’s problem list, and also a link to an acute kidney injury order set (supplemental fig S2); the set included options for blood and urine testing and kidney imaging but were limited to tests and procedures considered minimal risk (that is, intravenous fluid administration was not included). Providers at each of the study hospitals received education about acute kidney injury and the alert system before study roll out during departmental conferences. The educational program explained the definitions of acute kidney injury, the functioning of the alert, and the ethics of the study but did not provide suggested practices of care.
Outcomes
The primary outcome was a composite of inpatient acute kidney injury progression (defined as an increase in acute kidney injury stage), receipt of dialysis, or death within 14 days of randomization. Prespecified secondary outcomes included the components of the primary outcomes and frequency of various practices of care for acute kidney injury. These practices included administration of contrast, fluids, or a nephrotoxic agent (a non-steroidal anti-inflammatory drug, angiotensin converting enzyme inhibitor/angiotensin receptor blocker, or aminoglycoside), ordering a urinalysis, documentation of acute kidney injury, monitoring of creatinine and urine output, and ordering a kidney consultation. An assessment of each hospital’s alert effects was also prespecified.
Statistical analysis
We present descriptive statistics as median (interquartile range) for continuous variables and proportions for categorical variables. For the primary analysis, and all comparisons of categorical variables between the intervention and control group, we used the Mantel-Haenszel test, accounting for each hospital site as a stratum. We used the Mantel-Haenszel approach to obtain the pooled relative risks across hospital strata without adjusting for other baseline factors.
As a sensitivity analysis, we used modified Poisson generalized estimating equations with a robust variance estimator to present the relative risk estimates, adjusting for the following characteristics assessed at the time of randomization: age, sex, race, creatinine, blood urea nitrogen, white blood cell count, heart rate, respiratory rate, systolic and diastolic blood pressure, chronic heart failure, hypertension, diabetes, malignancy, intensive care unit status, modified sequential organ failure assessment (mSOFA) score, Elixhauser comorbidity score, and hospital.17 We used the Van Elteren test to compare continuous variables across the intervention groups, accounting for hospital strata.
To compare time to event between study groups, we used Cox proportional hazard regression with intervention as the independent variable, stratified by hospital, with censoring at 14 days after randomization. For individual hospital analyses, Kaplan-Meier curves were generated and log rank tests were used. Patients discharged before 14 days without an outcome of interest were assumed to be free of that outcome at 14 days. Death was treated as a censoring event in analyses where death was not the outcome. We used Schoenfeld residuals to examine the proportional hazards assumption in Cox models; there were no violations. We specifically amended the protocol to include a test for site heterogeneity on 4 May 2018—that is, two months after the initial patient enrollment and before we had started alerting in any of the hospitals in which adverse outcomes were ultimately detected. This new protocol was based on ongoing executive committee calls and the hypothesis that the alert effect might differ by hospital. No data were used to drive this decision.
Our preliminary data suggested that 24.5% of patients would experience the composite outcome. We considered a relative 20% reduction in this outcome (to 19.6) was clinically significant and would represent an acceptable threshold to justify adding an interruptive alert to the clinician workflow. A sample size of 2512 in each arm of the study achieves 90% power to detect this degree of change, but given the potential for contamination across study arms (whereby clinicians “learn” to identify and care for acute kidney injury over the course of the trial), which would tend to bias the results toward the null, we inflated the sample size by 20% to a total of 6030. Although this sample increase does not deal with the bias toward the null, it does allow for increased detection of a smaller effect size.
Statistical analyses were conducted in Stata version 15.1 (College Station TX), SAS version 9.4 (SAS Institute, Cary, NC), and R (RStudio version 1.2.5033 (R version 3.5.3), Boston, MA), and P≤0.04 was considered statistically significant for the primary outcome (to account for the interim analysis). We do not report P values for secondary and exploratory analyses, except for safety outcomes, but report 95% confidence intervals in all cases. The widths of the 95% confidence intervals have not been adjusted for multiplicity, and inferences drawn from these might not be reproducible.
Patient and public involvement
No patients were involved in setting the research question or the outcome measures, nor were they involved in developing plans for recruitment, design, or implementation of the study. No patients were asked to advise on interpretation or writing up of results. Patient involvement was not sought owing to the limited public understanding of acute kidney injury and a lack of definitive treatment.
Results
From 29 March 2018 to 14 December 2019, 6030 individuals met enrollment criteria and were randomized (supplemental fig S1). Those patients generated a total of 226 316 alerts (including silent alerts in the usual care arm), which reached 3632 unique providers, a median (interquartile range) of 7 (7-13) for each patient. Characteristics of the study population at the time of randomization are shown in table 1 (stratified analyses, supplemental tables 1-6). The median (interquartile range) age of patients was 71 (59-82) years, 2882 (48%) were female, and 946 (16%) self-identified as black. The majority (n=4564 (76%)) were admitted to a medical service, and 1923 (32%) were in an intensive care unit at the time of randomization.
Table 1.
Patient characteristics at the time of randomization. Data are presented as median (interquartile range) or count (percent)
| Variables | Usual care (n=2971) | Alert (n=3059) |
|---|---|---|
| Demographics: | ||
| Median (IQR) age (years) | 71.3 (59.4-81.6) | 71 (59-81.7) |
| Female sex | 1440 (48.5) | 1442 (47.1) |
| Black | 462 (15.6) | 484 (15.8) |
| Hispanic | 289 (9.7) | 331 (10.8) |
| Hospital admission characteristics: | ||
| Medical admission | 2280 (76.7) | 2284 (74.7) |
| Patient in the ICU | 961 (32.3) | 962 (31.4) |
| Patient in the emergency department | 108 (3.6) | 93 (3.0) |
| Patient in the ward | 1902 (64.0) | 2004 (65.5) |
| Hospital 1 (urban, teaching) | 1238 (41.7) | 1277 (41.7) |
| Hospital 2 (urban, teaching) | 599 (20.2) | 649 (21.2) |
| Hospital 3 (urban, teaching) | 456 (15.3) | 479 (15.7) |
| Hospital 4 (suburban, teaching) | 292 (9.8) | 275 (9.0) |
| Hospital 5 (suburban, non-teaching) | 278 (9.4) | 276 (9.0) |
| Hospital 6 (suburban, non-teaching) | 108 (3.6) | 103 (3.4) |
| Comorbidities: | ||
| Chronic kidney disease | 1127 (37.9) | 1163 (38.0) |
| Congestive heart failure | 1307 (44.0) | 1351 (44.2) |
| COPD | 1015 (34.2) | 1049 (34.3) |
| Diabetes mellitus | 1197 (40.3) | 1287 (42.1) |
| Hypertension | 2434 (81.9) | 2510 (82.1) |
| Malignancy | 4712 (15.9) | 460 (15.0) |
| Depression | 687 (23.1) | 655 (21.4) |
| Liver disease | 397 (13.4) | 458 (15.0) |
| Laboratory values (median (IQR))*: | ||
| eGFR at admission (mL/min/1.73 m2) | 55.1 (34.2-84.4) | 55.8 (36.6-83.5) |
| Creatinine (mg/dL)† | 1.5 (1.1-2) | 1.5 (1.2-2) |
| Nadir creatinine (mg/dL)†‡: | 1.1 (0.8-1.5) | 1.1 (0.8-1.5) |
| Sodium (mmol/L) | 138 (135-141) | 138 (135-141) |
| Potassium (mmol/L) | 4.2 (3.8-4.6) | 4.2 (3.8-4.6) |
| Chloride (mmol/L) | 102 (98-106) | 102 (98-106) |
| Bicarbonate (mmol/L) | 24 (21-27) | 24 (21-27) |
| Anion gap (mmol/L) | 12 (10-14) | 12 (10-15) |
| Blood urea nitrogen (mg/dL)† | 28 (18-42) | 28 (19-40) |
| White blood cell count (×1000/μL) | 9.8 (7.2-13.5) | 9.8 (7.2-13.8) |
| Hemoglobin (g/dL) | 10.7 (9-12.3) | 10.5 (8.9-12.3) |
| Platelet count (×1000/μL) | 202.5 (149-266) | 200 (144-266) |
| Exposures before acute kidney injury: | ||
| Contrast in prior 72 hours | 586 (19.7) | 623 (20.4) |
| Cardiothoracic surgery in prior 72 hours | 110 (3.7) | 135 (4.4) |
| ACE inhibitor/ARB in prior 72 hours | 626 (21.1) | 670 (21.9) |
| NSAID in prior 72 hours | 403 (13.6) | 388 (12.7) |
| PPI in prior 72 hours | 684 (23.0) | 677 (22.1) |
| Study timing (median (IQR)): | ||
| Time from admission to randomization (hours) | 50.7 (28.8-104.3) | 50.1 (29.8-101.0) |
| Time from AKI to randomization, hours | 0.5 (0.2-1.3) | 0.5 (0.2-1.2) |
| Alert distribution (median (IQR)): | ||
| Unique providers reached | 7 (4-13) | 7 (4-13) |
ACE=angiotensin converting enzyme; AKI=acute kidney injury; ARB=angiotensin receptor blocker; COPD=chronic obstructive pulmonary disease; eGFR=estimated glomerular filtration rate; ICU=intensive care unit; IQR=interquartile range; NSAID=non-steroidal anti-inflammatory drug; PPI=proton pump inhibitor.
Values at randomization unless otherwise specified.
Blood urea nitrogen 1 mg/dL=0.357 mmol/L; creatinine 1 mg/dL=88.42 μmol/L.
Values in the 48 hours before randomization.
Patients randomized to acute kidney injury alerts were more likely to receive an order for intravenous fluids (38.4% alert arm v 34.8% usual care arm, absolute difference 3.8% (95% confidence interval 1.4% to 6.2%)), urinalysis (17.0% v 14.9%, 1.9% (0.1% to 3.7%)), and to have serum creatinine measured (87.2% v 85.2%, 1.8% (0.1% to 3.6%)) within 24 hours of randomization (table 2; stratified analyses, supplemental tables 7-12). Randomized patients were also more likely to have documentation of acute kidney injury in the problem list (70.0% alert arm v 63.0% usual care arm, absolute difference 7.0% (95% confidence interval 4.6% to 9.3%)) during the hospital admission. No significant differences were found in the frequency of intravenous contrast administration or recording of urine output between the arms at 24, 48, or 72 hours after randomization.
Table 2.
Process outcomes stratified by study arm. Unless otherwise specified, outcomes reflect proportion completed within 24 hours of randomization. Results for usual care and alert groups are shown as number (percent) unless otherwise specified; absolute differences are shown as percent (95% confidence interval) unless otherwise specified
| Variables | Usual care (n=2971) | Alert (n=3059) | Absolute difference |
|---|---|---|---|
| Diagnostic and therapeutic investigation: | |||
| Intravenous fluid drip order | 1034 (34.8) | 1174 (38.4) | 3.8 (1.4 to 6.2) |
| Intravenous fluid bolus order | 339 (11.4) | 397 (13.0) | 1 (−0.4 to 2.3) |
| Urinalysis | 444 (14.9) | 519 (17.0) | 1.9 (0.1 to 3.7) |
| Urine output measurement | 2130 (71.7) | 2242 (73.3) | 1.7 (−0.6 to 4) |
| Subsequent creatinine measurement (28 hours) | 2532 (85.2) | 2666 (87.2) | 1.8 (0.1 to 3.6) |
| Kidney consultation (within 14 days) | 710 (23.9) | 727 (23.8) | 0.1 (−2.1 to 2.2) |
| Nephrotoxin use: | |||
| Contrast exposure | 102 (3.4) | 115 (3.8) | 0.3 (−0.6 to 1.2) |
| Aminoglycoside exposure | 19 (0.6) | 14 (0.5) | * |
| Aminoglycoside exposure among those already receiving aminoglycoside (n=40) |
8/20 (40.0) | 5/20 (25.0) | * |
| NSAID exposure | 166 (5.6) | 144 (4.7) | −1 (−2 to 0.1) |
| NSAID
exposure among those already receiving NSAIDs (n=600) |
129/312 (41.3) | 103/288 (35.8) | −6.7 (−14.4 to 1.0) |
| ACEi/ARB exposure | 425 (14.3) | 424 (13.9) | −0.1 (−1.8 to 1.6) |
| ACEi/ARB
exposure among those already receiving ACEi/ARB (n=1093) |
326/532 (61.3) | 331/561 (59.0) | −2 (−7.8 to 3.8) |
| Administrative: | |||
| AKI documentation (at end of encounter) | 1871 (63.0) | 2141 (70.0) | 7.0 (4.6 to 9.3) |
| Median (IQR) hospital direct costs ($) | 10 300 (5400-21 900) | 10 600 (5400-22 800) | 400 (229.7 to 1029.7) |
| Median (IQR) hospital total costs ($) | 19 100 (9900-42 500) | 20 100 (10 200-43 600) | 740.5 (287.6 to 1768.7) |
ACEi=angiotensin converting enzyme inhibitor; ARB=angiotensin receptor blocker; AKI=acute kidney injury; NSAID=non-steroidal anti-inflammatory drug; PPI=proton pump inhibitor.
Too few events for reliable estimates.
Primary outcome: progression of acute kidney injury, dialysis, or death
The primary outcome occurred in 653 (21.3%) of 3059 patients with an alert and in 622 (20.9%) of 2971 patients receiving usual care (relative risk 1.02, 95% confidence interval 0.93 to 1.13, P=0.67; table 3 and stratified analyses supplemental tables 13-18). No evidence of a change was seen in the relative effect across intervention groups over time (P=0.90 for interaction). Time to event analyses for the composite outcome and for each component and each hospital are shown in supplemental figures S3-S6 and were of similar magnitude.
Table 3.
Primary and secondary outcomes of interest.* Results for usual care and alert groups are shown as number (percent) unless otherwise specified; absolute difference is shown as percent (95% confidence interval) except for duration of AKI and maximum AKI stage achieved, in which odds ratios (95% confidence interval) are reported
| Variables | Usual care (n=2971) | Alert (n=3059) | Absolute percentage difference |
|---|---|---|---|
| Primary outcome: | |||
| AKI progression,† dialysis or death at 14 days | 622 (20.9) | 653 (21.3) | 0.6 (−1.4 to 2.7) |
| Secondary outcomes: | |||
| AKI progression† at 14 days | 461 (15.5) | 487 (15.9) | 0.5 (−1.3 to 2.3) |
| Dialysis at 14 days | 93 (3.1) | 106 (3.5) | 0.3 (−0.6 to 1.1) |
| Death at 14 days | 265 (8.9) | 272 (8.9) | −0.3 (−1.7 to 1.1) |
| Discharge to home | 1494 (50.3) | 1503 (49.1) | −1.4 (−4 to 1.1) |
| Discharge to inpatient or home hospice | 165 (5.6) | 173 (5.7) | 0.06 (−1.1 to 1.2) |
| Median length of stay (IQR; days after randomization) | 4.2 (2.1-9.3) | 4.3 (2.2-9.2) | 0.1 (−0.2 to 0.3) |
| Duration of AKI (days): | 0.9 (0.8 to 1) | ||
| <2 | 2108 (71.0) | 2239 (73.2) | |
| 2-<7 | 722 (24.3) | 700 (22.9) | |
| ≥7 | 141 (4.7) | 120 (3.9) | |
| Max AKI stage achieved: | 1 (0.9 to 1.1) | ||
| 0 | 5 (0.2) | 6 (0.2) | |
| 1 | 2302 (77.5) | 2376 (77.7) | |
| 2 | 403 (13.6) | 412 (13.5) | |
| 3 | 261 (8.8) | 265 (8.7) |
Eleven individuals who were randomized never developed AKI, owing to laboratory errors in creatinine measurement, allowing patients to meet AKI criteria, which were subsequently revised lower after enrollment.
Progression of acute kidney injury (AKI) was defined as achieving a higher Kidney Disease: Improving Global Outcomes (KDIGO) AKI stage than the one present at the time of randomization, dialysis, or both.
Assessment of heterogeneity of treatment effect
Evidence of heterogeneity of treatment effect was found across the six hospitals (P=0.05; fig 2). Some sites suggested a trend towards a protective effect of alerts, such as hospital 3 (relative risk 0.80, 0.62 to 1.04, P=0.09), but others showed potential harm from alerts. In a post hoc analysis, this heterogeneity was not accounted for by patient characteristics, but by whether the hospitals were stratified as “teaching” or “non-teaching.” In particular, there appeared to be harm from alerts at the two non-teaching hospitals in the study (n=765), where 24.3% of patients in the alert arm compared with 16.3% in the usual care arm met the primary outcome (relative risk 1.49, 1.12 to 1.98, P=0.006). No such effect was found at the four teaching hospitals (0.97, 0.87 to 1.07, P=0.54), with P=0.006 for interaction comparing teaching with non-teaching hospitals.
Fig 2.
Primary and secondary outcome events, stratified by hospital type. Error bars are 95% confidence intervals of the observed proportion of events. AKI=acute kidney injury
The differences in the primary outcomes observed in the non-teaching versus teaching hospitals seemed to be driven by death as opposed to dialysis or progression of the acute kidney injury. We noted evidence of heterogeneity of the association between alerts and death across the enrolled hospitals (P=0.05). Again, alerts were associated with a significantly higher risk of death at 14 days in the non-teaching hospitals, where 59 (15.6%) patients in the alert group versus 33 (8.6%) patients receiving usual care met this outcome. The relative risk of death was 1.82 (1.22 to 2.72, P=0.003) in the non-teaching hospitals compared with a relative risk of 0.89 (0.74 to 1.06, P=0.18) at the teaching hospitals (P=0.001 for interaction).
Secondary outcomes and process measures
The frequency of kidney consultations was similar between the two arms. Within 14 days of randomization, 23.8% of the alert group and 23.9%of the usual care group had received a consultation (absolute difference 0.1% (95% confidence interval −2.1% to 2.2%)). No evidence of heterogeneity of this effect was seen across the study hospitals (P>0.99). Similarly, in time to event analysis, no increased rate of kidney consultations was found across the two arms (hazard ratio 1.00 (95% confidence interval 0.90 to 1.11). No significant difference was seen in the rate of discharge to hospice overall or in the non-teaching hospitals.
Subgroup analyses
Prespecified subgroup analyses are shown in figure 3. The effect of alerting was largely similar between medical and surgical patients, ICU and non-ICU patients, by age, race, and sex, and across baseline creatinine values.
Fig 3.
Prespecified subgroup analyses show similar alert effect across a diverse array of patient characteristics. Diamonds reflect relative risk, with bars showing 95% confidence interval. Creatinine 1 mg/dL=88.42 μmol/L. ICU=intensive care unit
Sensitivity and post hoc analyses
We examined the adjusted relative risk of the primary outcome accounting for a variety of baseline factors individually and stratified into teaching and non-teaching hospitals (supplemental table 19). The adjusted relative risks were broadly similar, although the adjusted risk of death in the non-teaching hospitals attenuated to 1.58 (95% confidence interval 1.08 to 2.31) and in the teaching hospitals to 0.85 (0.72 to 1.01).
We also performed a formal mediation analysis to determine if certain actions taken after alerting might have mediated the adverse outcomes in the non-teaching hospitals (supplemental table 20).18 The use of intravenous fluid (in drip or bolus form, in high chloride or more physiologic form) did not appear to mediate the adverse effect of the alert. Other proxies of volume overload (respiratory rate, oxygen saturation, and the change in these parameters) similarly did not show significant effect mediation. Exposures to medication, kidney consultations, and the percent of alerts that went to attending versus other providers also showed no significant effect mediation. Finally, the burden of other electronic health record alerts (such as those for sepsis or interactions between drugs) did not appear to mediate the deleterious effects of alerts for acute kidney injury.
We also examined the risks of two components of our primary outcome, death and progression of acute kidney injury, in the period two years before the start of this trial in each hospital (fig S7). Broadly, these risks were similar to those of the control group in the randomized portion of the study. For example, two years before our study started, the risk of death at 14 days after acute kidney injury in the non-teaching hospitals was 8.6%, similar to the risk of death for the control group of 8.6%, and significantly lower than the risk of death of 15.6% for the intervention group.
Discussion
Principal findings
In this multicenter randomized trial of patients with acute kidney injury, the provision of an electronic health record alert to providers had no effect on the risk of progression of acute kidney injury, dialysis, or death. Substantial heterogeneity of effect was seen across study hospitals, however, with signals for harm emerging in the two non-teaching hospitals. We cannot be sure whether the observed differences in effect were due to the presence of teaching services or other practices that might differ between the hospitals in our study. The fact that some process measures (such as orders for intravenous fluids and urinalysis) occurred more often in the alert arm suggests that the alerts were being noticed by providers and that they could change provider behavior, but the magnitude of these changes was relatively small.
Reconciliation of those findings with the fact that the risk of death was higher in the alert arm in the non-teaching institutions is difficult. This increased risk might have been due to random chance, although the P value for the alert:death association of 0.003 suggests that chance is unlikely. Furthermore, adjustment for a variety of baseline factors did not ablate the observed association, suggesting that imbalance in baseline covariates is insufficient to explain the association. Additionally, in our prior, single center, acute kidney injury alert trial (conducted at a medical center not involved in this study), we noted a numerically higher, although statistically insignificant, risk of death in the alert versus the usual care arm. The risk of death at 14 days in that trial was 7.1% in the usual care group and 7.7% in the alert group (P=0.58).19 To date, that study is the only previous trial of acute kidney injury alerts that employed randomization at the patient level. Finally, we examined historical death rates of acute kidney injury in the study hospitals and found them to be quite similar to the rates seen in the usual care arm of our trial. Taken together, these data suggest that there might be true harm associated with the provision of alerts for acute kidney injury in certain situations. In view of the secondary nature of these analyses, the results should be considered hypothesis generating, and any attribution to the “non-teaching” nature of the hospital is premature.
As alerts of this type are in widespread clinical use, we sought to determine if any factor downstream of randomization drove these results. We speculated that an increase in fluid administration in the alert group might lead to harm through volume overload, but accounting for fluid administration did not attenuate the observed effect. We further considered that certain types of providers (eg, medical doctors v physician’s assistants) might respond to acute kidney injury differently, but there was no indication that the effect changed according to the type of provider receiving the alert. Furthermore, we assessed whether alert fatigue might be to blame for poorer outcomes in the alert group, but accounting for the frequency of other electronic health record alerts did not change the observed association with the alert for acute kidney injury. Possibly, while no single factor mediated the adverse effect in every patient, a constellation of factors resulted in the observed effect. We are left without a satisfying unifying explanation for the potential harm. Alerts might distract providers from other, more clinically important, patient care. Alternatively, in the larger teaching institutions, systems exist to protect patients from certain harms (like inappropriate drug dosing), which are not present in the smaller hospitals (regardless of academic status). Finally, we should consider that alerts could compel providers to do something, even if it is not clinically indicated, perhaps owing to fears of medicolegal consequences if they do not act.
Comparisons with other studies
Alerts of all kinds, from electronic to auditory to visual, have proliferated in medical care with the best of intentions under assumptions that risk is minimal.20 Notably, alerting systems for acute kidney injury have already been widely adopted across various health systems. The published literature suggests that alerts are often effective in changing processes of care, but their effect on clinical efficacy has been mixed.21 For example, a stepped wedge randomized trial of 24 059 episodes of acute kidney injury in the UK found a decreased length of stay, but no change in overall mortality risk between the control and intervention periods.22 In contrast, a before/after study of a clinical decision support system for acute kidney injury among 64 512 patients found a significant reduction in mortality risk from 10.2% to 9.4% from before to after the intervention.23 This design could not account for secular trends in the care of patients with acute kidney injury, although the mortality risk in patients with non-acute kidney injury did not change over the study period.
This trial cannot show a clear efficacy of alerts, but the phenomenon of “alert fatigue”—a tendency to disregard even important alerts owing to their abundance—is increasingly recognized.24 25 26 27 The economic costs of alerts might be minimal after the initial alert build, but the costs of provider attention, burnout, and potential overtreatment might be substantial and clearly deserve additional study.28 This trial was conducted under a waiver of informed consent, based on the prevailing assumption that informational alerts for acute kidney injury pose minimal risk to the patient and the recognition that obtaining the consent of those randomized to the control group would invalidate the study results. Unfortunately, for the alert deployed in this study, the assumption of minimal risk proved incorrect. These results do illustrate, however, that clinical trials of seemingly benign interventions, such as alerts, require rigorous evaluation and should not be implemented without robust evidence of safety and efficacy.
Strengths and limitations of this study
Limitations of the study include randomization at the patient level, allowing for providers potentially to learn how to better identify acute kidney injury throughout the course of the trial. This contamination would tend to bias the results toward the null hypothesis that the alerts did not affect the primary outcome, but an analysis accounting for the duration that the alerts had been in effect at each hospital found no evidence of this. Furthermore, our previous research has shown that patient level factors, such as baseline creatinine value and sex, contribute to alert benefit.29 Additionally, the alert was largely informational. Although the alert provided a link to an acute kidney injury order set, it provided no patient-specific recommendations (such as identifying current nephrotoxic drug treatment)—a key feature of the successful alert for pediatric acute kidney injury piloted by the nephrotoxic injury negated by just-in-time action (NINJA) program, and by certain “care bundles” for acute kidney injury that have been used in the UK.22 30 The heterogeneity of acute kidney injury might require more personalized interventions. Furthermore, the alert was sent only to certain care providers, notably excluding nurses and pharmacists, who might specifically benefit from receipt of the alert. The alert did not use urine output criteria, as many ward patients do not have reliable estimates of urine output. This might have meant that the study population was not fully representative of all patients with acute kidney injury. Finally, the alert was conducted across six hospitals within a single large health system in northeastern US, limiting generalizability. However, the hospitals represent a diverse mix of patients and care models, suggesting that these results might apply more broadly.
The strengths of the study include its large size with an ability to detect clinically meaningful changes in hard endpoints, its enrollment across a diverse group of hospitals, and the use of a fully automated enrollment and randomization system that could pave the way for future trials of alerting and clinical decision support systems.
Conclusions
In this large, randomized, multicenter study, we found that, overall, the provision of an automated, electronic alert for acute kidney injury had a modest effect on process measures and no effect on the risk of progression of acute kidney injury, dialysis, or death. Substantial heterogeneity of effects across hospitals was found, with the concerning finding of a significantly increased risk of death in the alert group in non-teaching hospitals. This study argues against the implementation of informational alerts for acute kidney injury and for a reconsideration of the alerts currently used.
What is already known on this topic
Acute kidney injury occurs in 15% of patients in hospital and is associated with poor outcomes, yet fewer than 50% of these results are documented in the electronic health record, and best practices for acute kidney injury are performed inconsistently
Limited data from randomized trials exist to illustrate the clinical benefit of alerts for acute kidney injury or demonstrate their efficacy on patient outcomes
What this study adds
This randomized controlled trial found only a modest effect of electronic alerts on process methods for acute kidney injury and no overall effect on the risks of death, dialysis, or disease progression in patients in hospital with acute kidney injury
The heterogeneity of effect of the alert across hospitals argues for a more rigorous evaluation of safety and efficacy of both current and future alerting systems for acute kidney injury
Acknowledgments
We thank the Joint Data Analytics Team at Yale for their help in data acquisition and Epic integration of our acute kidney injury detection algorithm and alert.
Web extra.
Extra material supplied by authors
Web appendix: Supplementary methods
Web appendix: Supplementary figures and tables
Contributors: The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted. All authors meet ICMJE recommendations for authorship: Have made substantial contributions to the conception or design of the work (FPW, MM, YY, TA, HF, AXG, JHG, MH, SL, SGM, DGM, PMP, CRP, MS, JT, UU), or to the acquisition, analysis, or interpretation of data for the work (FPW, MM, YY, CP, EM, TA, AB, HF, AXG, JHG, MH, SL, FL, HL, SGM, DGM, PMP, CRP, MS, JT, UU); drafted the work or revised it critically for important intellectual content (all authors); gave final approval of the version to be published (all authors); agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved (all authors). FPW is the guarantor.
Funding: This work was supported by grants NIH P30DK079310 and R01 DK113191 to FPW. The funding sources had no involvement in study design; in collection, analysis, and interpretation of data; in writing of the manuscript; or in the decision to submit for publication. All authors have full access to all data in the study and take responsibility for the integrity of the data and the accuracy of its analysis.
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare the following support for the submitted work: PMP is a consultant for Baxter and receives grant support from Dascena and BioPorto. AXG is supported by the Dr Adam Linton Chair in Kidney Health Analytics and a Clinician Investigator Award from the Canadian Institutes of Health Research. HF is a consultant for the Kyowa Kirin Corporation, and is the editor in chief of the American Journal of Kidney Disease. JT reports grants and personal fees from Sequana Medical, grants and personal fees from BMS, personal fees from Astra Zeneca, personal fees from Novartis, grants and personal fees from five laboratories, personal fees from Cardionomic, personal fees from Bayer, grants and personal fees from Boehringer Ingelheim, personal fees from MagentaMed, grants from Otsuka, personal fees from Renalguard, grants and personal fees from Sanofi, grants and personal fees from FIRE1, grants from Abbott, personal fees from WL Gore, and personal fees from Windtree therapeutics outside the submitted work. SGM receives grant support from the American Heart Association and the Patterson Trust Fund. CRP is on the advisory board of RenalytixAI and owns equity in the same; serves on the data and safety monitoring board of Genfit Pharma and is supported by NIH grants R01085757 and UO1DK-082185. MH has received consulting fees from Abbvie. DGM receives grant support from the NIH/NIDDK (K23DK117065). FPW reports grant support from NIDDK R01DK113191 and P30DK079210.
Ethical approval: This research was approved by the Yale School of Medicine institutional review board (ID 1604017596) and the Bridgeport Hospital institutional review board (IDs 051802, 041801, 071808) under a waiver of informed consent.
Data sharing: A deidentified participant dataset with an associated data dictionary can be found at https://doi.org/10.5061/dryad.4f4qrfj95.
The corresponding author affirms the honest, accurate, and transparent account of the reported research and that no important or relevant aspects of the study have been omitted. Any discrepancies from the study as originally planned have been explained.
Dissemination to participants and related patient and public communities: There are no plans to disseminate the results of the research to individual study participants, but the results of this study will be disseminated as medical manuscripts. An associated press release will be made, and all manuscripts will be publicly available on PubMed Central. The study data have been made publicly available at the above website.
Provenance and peer review: Not commissioned; externally peer reviewed.
References
- 1. Andrikos E, Tseke P, Balafa O, et al. Epidemiology of acute renal failure in ICUs: a multi-center prospective study. Blood Purif 2009;28:239-44. 10.1159/000231986 [DOI] [PubMed] [Google Scholar]
- 2. Bagshaw SM, George C, Bellomo R, ANZICS Database Management Committee . Early acute kidney injury and sepsis: a multicentre evaluation. Crit Care 2008;12:R47. 10.1186/cc6863 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Chawla LS, Amdur RL, Shaw AD, Faselis C, Palant CE, Kimmel PL. Association between AKI and long-term renal and cardiovascular outcomes in United States veterans. Clin J Am Soc Nephrol 2014;9:448-56. 10.2215/CJN.02440213 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Coca SG, Singanamala S, Parikh CR. Chronic kidney disease after acute kidney injury: a systematic review and meta-analysis. Kidney Int 2012;81:442-8. 10.1038/ki.2011.379 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Grams ME, Rabb H. The distant organ effects of acute kidney injury. Kidney Int 2012;81:942-8. 10.1038/ki.2011.241 [DOI] [PubMed] [Google Scholar]
- 6. Aitken E, Carruthers C, Gall L, Kerr L, Geddes C, Kingsmore D. Acute kidney injury: outcomes and quality of care. QJM 2013;106:323-32. 10.1093/qjmed/hcs237 [DOI] [PubMed] [Google Scholar]
- 7. James MT, Dixon E, Roberts DJ, et al. Improving prevention, early recognition and management of acute kidney injury after major surgery: results of a planning meeting with multidisciplinary stakeholders. Can J Kidney Health Dis 2014;1:20. 10.1186/s40697-014-0020-y [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Lewington AJ, Cerdá J, Mehta RL. Raising awareness of acute kidney injury: a global perspective of a silent killer. Kidney Int 2013;84:457-67. 10.1038/ki.2013.153 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Macedo E, Bouchard J, Soroko SH, et al. Program to Improve Care in Acute Renal Disease Study . Fluid accumulation, recognition and staging of acute kidney injury in critically-ill patients. Crit Care 2010;14:R82. 10.1186/cc9004 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Wilson FP, Bansal AD, Jasti SK, et al. The impact of documentation of severe acute kidney injury on mortality. Clin Nephrol 2013;80:417-25. 10.5414/CN108072 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Hoste EA, Kashani K, Gibney N, et al. 15 ADQI Consensus Group . Impact of electronic-alerting of acute kidney injury: workgroup statements from the 15(th) ADQI Consensus Conference. Can J Kidney Health Dis 2016;3:10. 10.1186/s40697-016-0101-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Lachance P, Villeneuve PM, Rewa OG, et al. Association between e-alert implementation for detection of acute kidney injury and outcomes: a systematic review. Nephrol Dial Transplant 2017;32:265-72. 10.1093/ndt/gfw424 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Park S, Baek SH, Ahn S, et al. Impact of electronic acute kidney injury (AKI) alerts with automated nephrologist consultation on detection and severity of AKI: a quality improvement study. Am J Kidney Dis 2018;71:9-19. 10.1053/j.ajkd.2017.06.008 [DOI] [PubMed] [Google Scholar]
- 14. Selby NM. Electronic alerts for acute kidney injury. Curr Opin Nephrol Hypertens 2013;22:637-42. 10.1097/MNH.0b013e328365ae84 [DOI] [PubMed] [Google Scholar]
- 15. Mutter M, Martin M, Yamamoto Y, et al. Electronic Alerts for Acute Kidney Injury Amelioration (ELAIA-1): a completely electronic, multicentre, randomised controlled trial: design and rationale. BMJ Open 2019;9:e025117. 10.1136/bmjopen-2018-025117 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Kidney Disease: Improving Global Outcomes (KDIGO) Acute Kidney Injury Work Group. KDIGO Clinical Practice Guideline for Acute Kidney Injury. Kidney Int 2012;2(Suppl.):1-138. [Google Scholar]
- 17. Zou G. A modified Poisson regression approach to prospective studies with binary data. Am J Epidemiol 2004;159:702-6. 10.1093/aje/kwh090 [DOI] [PubMed] [Google Scholar]
- 18. Imai K, Keele L, Tingley D. A general approach to causal mediation analysis. Psychol Methods 2010;15:309-34. 10.1037/a0020761 [DOI] [PubMed] [Google Scholar]
- 19. Wilson FP, Shashaty M, Testani J, et al. Automated, electronic alerts for acute kidney injury: a single-blind, parallel-group, randomised controlled trial. Lancet 2015;385:1966-74. 10.1016/S0140-6736(15)60266-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20. Yu DT, Seger DL, Lasser KE, et al. Impact of implementing alerts about medication black-box warnings in electronic health records. Pharmacoepidemiol Drug Saf 2011;20:192-202. 10.1002/pds.2088 [DOI] [PubMed] [Google Scholar]
- 21. Sahota N, Lloyd R, Ramakrishna A, et al. CCDSS Systematic Review Team . Computerized clinical decision support systems for acute care management: a decision-maker-researcher partnership systematic review of effects on process of care and patient outcomes. Implement Sci 2011;6:91. 10.1186/1748-5908-6-91 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22. Selby NM, Casula A, Lamming L, et al. An organizational-level program of intervention for AKI: a pragmatic stepped wedge cluster randomized trial. J Am Soc Nephrol 2019;30:505-15. 10.1681/ASN.2018090886 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23. Al-Jaghbeer M, Dealmeida D, Bilderback A, Ambrosino R, Kellum JA. Clinical decision support for in-hospital AKI. J Am Soc Nephrol 2018;29:654-60. 10.1681/ASN.2017070765 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24. Cykert S. The benefits and concerns surrounding the automation of clinical guidelines. N C Med J 2015;76:235-7. 10.18043/ncm.76.4.235 [DOI] [PubMed] [Google Scholar]
- 25. Footracer KG. Alert fatigue in electronic health records. JAAPA 2015;28:41-2. 10.1097/01.JAA.0000465221.04234.ca [DOI] [PubMed] [Google Scholar]
- 26. Beeler PE, Bates DW, Hug BL. Clinical decision support systems. Swiss Med Wkly 2014;144:w14073. [DOI] [PubMed] [Google Scholar]
- 27. Bryant AD, Fletcher GS, Payne TH. Drug interaction alert override rates in the meaningful use era: no evidence of progress. Appl Clin Inform 2014;5:802-13. 10.4338/ACI-2013-12-RA-0103 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28. Xie M, Johnson K. Applying human factors research to alert-fatigue in e-prescribing. AMIA Annu Symp Proc 2007;1161. [PubMed] [Google Scholar]
- 29. Biswas A, Parikh CR, Feldman HI, et al. Identification of patients expected to benefit from electronic alerts for acute kidney injury. Clin J Am Soc Nephrol 2018;13:842-9. 10.2215/CJN.13351217 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30. Goldstein SL, Mottes T, Simpson K, et al. A sustained quality improvement program reduces nephrotoxic medication-associated acute kidney injury. Kidney Int 2016;90:212-21. 10.1016/j.kint.2016.03.031 [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Web appendix: Supplementary methods
Web appendix: Supplementary figures and tables