Association between e-alert implementation for detection of acute kidney injury and outcomes: a systematic review

Abstract

Background. Electronic alerts (e-alerts) for acute kidney injury (AKI) in hospitalized patients are increasingly being implemented; however, their impact on outcomes remains uncertain.

Methods. We performed a systematic review. Electronic databases and grey literature were searched for original studies published between 1990 and 2016. Randomized, quasi-randomized, observational and before-and-after studies that included hospitalized patients, implemented e-alerts for AKI and described their impact on one of care processes, patient-centred outcomes or resource utilization measures were included.

Results. Our search yielded six studies (n = 10 165 patients). E-alerts were generally automated, triggered through electronic health records and not linked to clinical decision support. In pooled analysis, e-alerts did not improve mortality [odds ratio (OR) 1.05; 95% confidence intervals (CI), 0.84–1.31; n = 3 studies; n = 3425 patients; I²=0%] or reduce renal replacement therapy (RRT) use (OR 1.20; 95% CI, 0.91–1.57; n = 2 studies; n = 3236 patients; I²=0%). Isolated studies reported improvements in selected care processes. Pooled analysis found no significant differences in prescribed fluid therapy.

Conclusions. In the available studies, e-alerts for AKI do not improve survival or reduce RRT utilization. The impact of e-alerts on processes of care was variable. Additional research is needed to understand those aspects of e-alerts that are most likely to improve care processes and outcomes.

INTRODUCTION

Acute kidney injury (AKI) complicates the hospital course of 13–18% of patients [1] and up to 60% of those admitted to an intensive care unit (ICU) [2, 3]. AKI can modify patient outcomes and contribute to greater resource utilization and higher healthcare costs.

Consensus statements by experts have recommended early personalized investigations, monitoring and management for AKI [4, 5]. The impact of these recommendations, which focus largely on harm avoidance, remains to be determined. One challenge in evaluating the impact of these and other care processes is a failure of early recognition of AKI [6].

In 1994, Rind et al. [7] proposed a computer algorithm to automatically track serum creatinine (SCr) changes and once a threshold was reached, send an alert to the responsible team. Recently, a number of studies have evaluated ‘alerts’ for detection of AKI [8]. Alerts have generally been triggered by detecting changes in SCr and/or urine output. However, the impact of these alerts on care processes and patient outcomes has been inconsistent [9–13]. This would imply that the benefit for implementing an electronic alert (e-alert) for detecting AKI remains uncertain. Indeed, the Acute Disease Quality Initiative recently convened a consensus meeting focused on big data applications for AKI [14], and highlighted the evidence care gap in the rigourous evaluation of e-alerts for AKI [15, 16]. Accordingly, we performed an evidence synthesis to describe the spectrum of e-alert systems for AKI detection and to specifically assess their impact on care processes, outcomes and resource use.

MATERIALS AND METHODS

The protocol for this systematic review was registered with PROSPERO (CRD42016033033) [17]. We conducted this systematic review following the PRISMA-P guideline (available at www.systematicreviewsjournal.com/content/4/1/1). The research questions following the PICO format are detailed in Supplementary Appendix 1.

Data source and searches

Our search strategy was developed in consultation with an expert research librarian (R.M.F.) and peer-reviewed by a second research librarian [18] (Supplementary Appendix 2).

Study selection

We included randomized, quasi-randomized, observational and before-and-after studies of hospitalized patients (i.e. emergency department [ED] and outpatients were excluded). Studies had to implement an e-alert using a clear operational definition for AKI and describe its impact on one or more care processes, patient-centred outcomes or resource utilization measures. Two authors (P.L., P.-M.V.) independently identified potentially eligible articles by an initial review of abstracts. This was followed by full-text review for potentially relevant studies fulfilling predefined eligibility criteria. Disagreements were resolved through discussion with a third reviewer (S.M.B.).

Data extraction and quality assessment

Data were abstracted from relevant studies by the two same authors using a standardized electronic data form. Data extracted included publication-related information, study design and quality assessment. Data on patient demographics, comorbidity and case-mix, along with care process, patient and health resource-related outcomes were extracted. We also captured detailed descriptors of the e-alert. E-alert disruptiveness was rated using an a priori established intrusiveness scale [17] (Supplementary Appendix 3). Study quality was rated using the Modified Downs and Black checklist [19].

Outcomes

• (i)  Primary patient-centred outcome was all-cause mortality, as defined by each study. Secondary outcomes were peak SCr, progression of AKI, proportion of patients fulfilling criteria for KDIGO stage 3 (or equivalent), receipt of renal replacement therapy (RRT) and recovery.

• (ii) Primary process of care outcome was nephrotoxin dose adjustment or discontinuation. Secondary outcomes were changes in monitoring frequency, investigations and management (i.e. medication review, medical record documentation of AKI, fluid prescription, vasopressors or diuretics use, nephrology consult).

• (iii) Primary health services use outcome was hospital length of stay. Secondary outcomes were ICU admission, length of stay and readmission.

Data synthesis and analysis

The primary analysis was mixed narrative and meta-analytic where feasible. Data were summarized and pooled to generate effect estimates of the impact of e-alerts on available outcomes. Statistical heterogeneity was assessed and quantified for each pooled summary estimate using Cochran’s Q statistic and the I² statistic, respectively [20]. Pooled analyses used random effects models and reported odds ratios (OR) with 95% confidence intervals (CI) for categorical variables and weighted mean differences with 95% CI for continuous variables, respectively. Subgroup analyses for categorical variables or meta-regression for continuous variables were considered to assess for possible sources of heterogeneity according to: AKI definition, setting, study design, study quality and alert intrusiveness. Publication bias was assessed by visual inspection of a funnel plot. The strength of recommendations derived from each study and pooled analysis were evaluated using the GRADE system (clinicalevidence.bmj.com/x/set/static/ebm/learn/665072.html). Analyses were performed using STATA (version 14; Stata Corp, College Station, TX, USA).

RESULTS

Our search yielded 5302 articles (Figure 1). Of 43 articles reviewed in full text, six fulfilled eligibility (five studies [11, 12, 21–23] and one abstract [24]). Details of studies not fulfilling eligibility are described in Supplementary Appendix 4. Of included studies, one was a randomized trial [12], three were time series [11, 21, 23], one was a before-and-after [22] and one used a historical control [24] (Table 1).

Figure 1.

Flow chart for study inclusion.

Table 1.

Details of publication and design of included studies

Study	Journal	Design	Context		Number of patients
			Country	Settinga
Rind [21]	Arch Intern Med	Time series	USA	Mix	922
McCoy [22]	Am J Kidney Dis	Before-and-after	USA	Mix	1659
Colpaert [11]	Crit Care Med	Time series	Belgium	ICU	951
Selby [23]	Clin J Am Soc Nephrol	Time series	UK	Mix	4159
Moran [24]	J Am Soc Nephrol	Historical cohort	UK	N/A	189
Wilson [12]	Lancet	Randomized controlled trial	USA	Mix	2393

^aMix, ICU and ward.

N/A = not available.

Most studies had good quality reporting [24] (Supplementary Appendix 5). Supplementary Appendix 6 reports study quality according to the GRADE system. All but one study [22] reported patient-related outcomes, four studies reported on process of care outcomes [11, 12, 21–22] and four reported data on health resources use [11, 12, 21, 23] (Table 2).

Table 2.

Summary of outcomes evaluated across included studies

Study	Patient-related outcomes			Process of care outcomes						Health service use
	Receipt of RRT	Worsening AKI or change in creatinine	Mortality	Time to drug dose adjustment	Medication review for nephrotoxins (discontinuation/ avoidance)	Documented AKI in the medical record	Nephrology consult	Follow-up creatinine	Administration of fluid, vasopressors or diuretic therapy	Hospital LOS	ICU LOS	ICU admission	ICU readmission
Rind [21]	No	Yes	Yes	Yes	No	No	No	No	No	Yes	No	No	No
McCoy [22]	No	No	No	Yes	Yes	No	No	No	No	No	No	No	No
Colpaert [11]	Yes	Yes	Yes	Yes	Yes	No	No	No	Yes	No	Yes	No	No
Selby [23]	No	Yes	Yes	No	No	No	No	No	No	Yes	No	No	No
Moran [24]	No	No	Yes	No	No	No	No	No	No	No	No	No	No
Wilson [12]	Yes	Yes	Yes	No	Yes	Yes	Yes	Yes	Yes	Yes	No	No	No

LOS, length of stay.

N/A = not available.

Design of e-alerts

The design, algorithm and implementation of e-alerts were available for each study except one [24] (Table 3). All studies issued alerts near real-time following detection (≤1 h). Most used algorithms for changes in SCr only for detection and e-alert triggering. Colpaert et al. [11] used changes in both SCr and urine output. Most e-alerts targeted ‘attending’ physicians; except in the study by Wilson et al. [12], where e-alerts targeted residents/interns, nurse practitioners and pharmacists.

Table 3.

Summary of details of each e-alerts used and settings across studies

Study	Trigger for e-alert	Timing	Target	Mode of transmission	Generation	Intrusivenessa	Integration of CDS
							Diagnostic recommendations	Mechanisms for harm avoidance	Follow-up recommendations	Format of CDS provided
Rind [21]	Creatinine increase by ≥44 µmol/L for patient receiving nephrotoxins; ≥50% or >177 µmol/L for patients on renally excreted medications	Instantaneous	Physician	E-mail	Automated	2	No	No	No	N/A
McCoy [22]	Creatinine increase by ≥44 µmol/L in 48 h	Instantaneous	Physician	EHR	Automated	2 and 3	No	Yes	No	Integrated into e-alert
Colpaert [11]	New or increase in RIFLE category	Instantaneous	Physician	Text message (SMS)	Automated	2	No	No	No	Integrated into e-alert
Selby [23]	Creatinine increase by ≥50% from baseline	Instantaneous	Physician	EHR and telephone call	Semi-automated	2 and 3	Yes	Yes	Yes	Referred/linked to internal website
Moran [24]	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A
Wilson [12]	Creatinine increase by ≥26 µmol/L in 48hrs or ≥50% within 7 days	Batched every 1 h	Intern or resident; and nurse practitioner or pharmacist	Text message (SMS) and e-mail	Automated	2	Yes	Yes	Yes	Linked to external website

^aBased on our intrusiveness scale (see Supplementary Appendix 1).

N/A = not available.

Algorithms used multiple modes to transmit e-alerts. One study used e-mail [21], one used text messages to an ICU-specific mobile [11] and one used both text messages, including a Simple Mail Transfer Protocol (SMTP) protocol to send messages from email to mobile phones [12]. McCoy et al. [22] had the e-alert target physician order entry in the electronic health records (EHRs). Selby [23] had the e-alerts, based on SCr thresholds, verified by a clinical chemist, who then issued the e-alert via the hospital EHR and direct to attending physicians by telephone for those classified with moderate-severe AKI (KDIGO stages 2–3). Most e-alerts were minimally disruptive (intrusiveness score ≤2). However, Selby evaluated the impact of increasing their e-alert intrusiveness (score 3). McCoy et al. [22] also evaluated passive and intrusive e-alerts. A passive e-alert was generated upon login to the EHR. If no acknowledgement had occurred upon logout, an intrusive e-alert was generated.

Strategies for education and implementation were variable. Wilson et al. [12] provided a brief educational session during grand rounds about implementing the e-alert; however, details on attendance were unavailable and there was no description of whether content focused on AKI management. Selby [23] provided inter-professional information sessions along with focused training to different specialties and on selected hospital wards. Prior to implementation, the investigators further engaged in a multifaceted education strategy that included: a focused education programme, a web-based tool, and face-to-face and AKI teaching sessions [25]. Colpaert et al. [11] provided a one-time information session on AKI management to trainees prior to implementation that was integrated with their teaching curriculum. Three studies provided no description of whether pre-implementation education was provided [21, 22, 24].

Content of e-alert

The integration of clinical decision support (CDS) varied (Table 3). Wilson et al. [12] included an electronic link embedded in their e-alert to the KDIGO clinical practice guidelines [5]. McCoy et al. [22] included recommendations to discontinue or dose-adjust nephrotoxic medications. In Selby [23], the e-alert was coupled with a care bundle that included diagnostic, therapeutic and follow-up recommendations. Two studies provided no CDS [11, 21].

Outcomes

Studies were heterogeneous in size, population and settings (Table 1 and Supplementary Appendix 7). Rind et al. [21] included 922 patients in a mixed ICU/ward setting. Time to modification and/or discontinuation of medications was shorter among those receiving e-alerts (75.9 h for e-alert versus 97.5 h for control, P < 0.0001). This was driven primarily by adjustment in renally excreted medications. Of 562 patients in the final analysis, no clinically significant differences kidney function or worsening AKI was evident between groups. There were no differences in hospital stay, mortality or in total pharmacy or hospital costs.

McCoy et al. [22] focused on the impact of an e-alert to modify prescription of nephrotoxic and renal-excreted medications through the use of two e-alerts, as mentioned above. In total, 1659 patients were evaluated in a mixed ICU/ward setting. E-alerts were associated with an increased rate of interventions within 24 h (dose adjustment 36.2 responses/100 events versus 46.4 responses/100 events, P < 0.001; medication discontinuation 33.9 responses/100 events versus 55.9 responses/100 events, P < 0.001). This was primarily driven by the interruptive rather than passive e-alert.

Colpaert et al. [11] performed a phased before-and-after study in an ICU setting evaluating an automated e-alert, integrating both changes in SCr and urine output. During the intervention phase, e-alerts were associated with an increase in the proportion of patients receiving fluids (23.0% intervention versus 4.9% pre-intervention and 9.2% post-intervention, P < 0.01), diuretics (4.2% versus 2.6% and 0.8%, P < 0.001) and vasopressors (3.9% versus 1.1% and 0.8%, P < 0.001). In addition, the time to receive any intervention was significantly shorter during the intervention phase (mean 19.0 ± 17.4 min for intervention versus 28.8 ± 17.6 min for pre-alert and 29.2 ± 17.3 min for post-alert, P < 0.001). Among patients with RIFLE-Risk, the e-alert was associated with a greater proportion recovering kidney function. There were no differences in worsening AKI, receipt of RRT, SCr at ICU discharge, ICU length of stay or mortality.

The study by Moran et al. [24] was published in abstract form only. The investigators described no difference in mortality associated with implementation of an e-alert for both community-acquired and hospital-acquired AKI.

Wilson et al. [12] performed a single-centre randomized trial evaluating the implementation of an e-alert for AKI. In total, 2393 patients from mixed medical/surgical wards and ICU settings were enrolled. There was no difference in the primary endpoint, a composite of maximum change in SCr, receipt of RRT and/or death at 7 days [% change in SCr: e-alert 0.0% (0.0–18.4%) versus control 0.6% (0.0–17.5%), P = 0.81; RRT: 7.2% versus 5.9%, P = 0.18; death: 5.9% versus 5.1%, P = 0.40; composite, P = 0.88]. There were no differences in secondary outcomes, with the exception of a more tests ordered at 48 h. In a planned subgroup analysis of surgical wards, there were greater nephrology consultations (12.0% versus 5.0%, P = 0.01) and RRT utilization (6.0% versus 3.0%, P = 0.03).

Selby [23] performed a time-series analysis of e-alert implementation, where four 6-month blocks were sequentially evaluated following the phased introduction of an education package, care bundle, and linkage of the e-alert and care bundle (along with making the alert more disruptive). There was incremental improvement in 30-day survival during their phased implementation (76.3% period 1 versus 79.2% periods 2 and 3 versus 80.5% period 4, P = 0.007). No differences were found in recovery or duration of hospitalization. Specific evaluation of the alert (as opposed to the combined interventions) was not performed.

In pooled analysis, e-alert implementation showed no reduction in mortality (OR 1.05; 95% CI, 0.84–1.31; n = 3 studies; n = 3425 patients; I²=0%) (Figure 2) or reduction in proportion receiving RRT (OR 1.20; 95% CI, 0.91–1.57; n = 2 studies; n = 3236 patients; I²=0). There were no differences in use of fluid therapy (OR 2.18; 95% CI, 0.46–10.31; n = 2 studies; n = 4378 patients; I²=99%), although this result was derived from only two studies that encompassed the divergent results from the studies by Colpaert et al. and Wilson et al. There were insufficient data to perform pooled analysis on whether e-alerts modified progression to stage 3 AKI, proportion receiving aminoglycosides or duration of hospitalization. There were insufficient studies to perform detailed subgroup analyses or meta-regression.

Figure 2.

Pooled effect estimate for the impact of AKI e-alerts on mortality. M–H, Mantel–Haenszel.

DISCUSSION

Our systematic review found that e-alerts are considerably heterogeneous in design, variably implemented and seldom include clear direction for decision support. We found that e-alerts have focused on creatinine-based algorithms for detection of AKI with few exceptions (i.e. urine output not feasible or available), generally utilized automated alerting (i.e. via EHR), predominantly targeted physicians (i.e. attending or resident trainee) and have been minimally disruptive to workflow.

Our review also implies that e-alerts for AKI, as designed and implemented across these studies, do not significantly improve patient-centred outcomes or lead to improved utilization of health services, although in some settings (but not others) they do appear to modify processes of care. Our review highlights the important gap in knowledge related to the efficacy and effectiveness of e-alert implementation in the AKI domain. Accordingly, no decisive inferences are possible about whether e-alert implementation for the detection of AKI significantly improves care processes or patient and health resource outcomes.

Context with prior studies

E-alert systems are increasingly being implemented across hospitalized settings concomitant with the broader integration of EHRs. Alerts have commonly focused on identification of drug interactions and medication adverse events [26]. In ICU settings, e-alerts have focused on the early detection of new episodes of sepsis [27], lung injury [28] and optimization of glycaemic control [29]. The findings from these e-alert studies have been mixed. Selected evaluations of e-alerts have shown promising impact for improving care processes and outcomes, such as compliance with prescription of deep venous thrombosis prophylaxis and reduced rate of thromboembolism [30, 31]. Alternatively, others have not shown improvement (i.e. time to antibiotics, fluid administration) in an ICU setting [27]. There are likely important sources for this disparity in effectiveness. These not only relate to variability in study design, but also the complexity of the disease or condition being detected, the context-specific setting in which the e-alert is deployed, along with subtle differences in e-alert design, implementation and CDS integration.

Algorithm design for AKI detection

The studies in our review nearly all used e-alert algorithms to detect AKI based solely on changes in SCr. This is logical given that SCr is widely measured and available across hospital settings. Conversely, urine output, which is also integral to the KDIGO consensus definition for AKI, was seldom incorporated, due largely to intermittent or sparse capture, particularly in non-ICU settings.

The e-alert developed by Selby used a tiered verification step in the algorithm. A clinical chemist confirmed the initial AKI detection prior to issuing the e-alert, which may be likely to reduce false positives and improve overall accuracy. However, this step in the algorithm could contribute to delays for issuing e-alerts when compared with alerts that are fully automated. No data were available on the timeline between electronic detection of AKI, verification by the clinical chemist, and issue of the e-alert. Therefore, it is uncertain whether there was any trade-off between accuracy and delay in e-alert transmission. In a related ward-based quality improvement programme aimed at AKI prevention, Goldstein et al. [32] showed the benefit of an intermediate step in the alerting process involving a ward-based pharmacist ordering follow-up SCr tests in response to e-alerts for hospitalized children prescribed nephrotoxic medications. This e-alert focused on children at-risk for AKI and included context-specific decision support. Impressively, the study showed a 64% decrease in AKI incidence and a 36% decrease in nephrotoxin exposure. These findings emphasize how changes in healthcare provider behaviour associated with integration of an alert may be fundamental to success.

Mechanisms for alerting

All studies generally sent the e-alert either real time or rapidly (≤1 h) to providers. However, there have been no specific data to guide on the optimal ‘time’ for communicating e-alerts to providers and such analysis may be at risk for bias due to variability across studies for when providers ‘acknowledge’ the alert.

In addition, the method for e-alert delivery is important and likely context specific. Some studies used the EHR or e-mail to issue e-alerts. When the e-alert is sent instantaneously to EHR or e-mail, it may conceivably take hours or greater after being generated before the provider is truly notified. This could relate to logistical issues like time of day or day of week or for selected services (i.e. surgical programmes) with significant cross-coverage by various providers, this may disrupt the timely communication of urgent e-alerts. Alternative algorithms direct e-alerts to a conventional pager or a mobile device that would conceivably deliver the e-alert instantaneously; however, they may direct the e-alert remote from the bedside. In the end, the studies in our review did not present data on the time from when the e-alert was generated to acknowledgement by providers and whether this impacted outcomes [11, 12, 21–24]. This may be a gap in our understanding on the implementation of e-alerts. To address this, we suggest that future work evaluates the deployment of e-alerts across multiple platforms (EHR, pager, etc.), concomitant with e-alert characteristics (i.e. appearance, content) across selected contexts, particularly where there may be susceptibility to delayed recognition.

The relationship between the alert intrusiveness and likelihood of alert fatigue is uncertain. Among the various published e-alerts, the degree of intrusiveness was generally low. In these circumstances, there may be a propensity for providers to overlook or override e-alerts [33]. Data from two studies support the notion that compliance may be improved when e-alerts are intrusive [22, 34]. McCoy et al. [22] found a more intrusive e-alert was associated with significantly greater compliance compared with a passive e-alert. Similarly, in a retrospective cohort study, Paterno et al. [34] showed that increasing tiers of intrusive e-alerts for drug–drug interactions improved compliance. These data would support a tiered approach to the intrusiveness of e-alerts, stratified by the urgency and/or severity of AKI.

Implementation methodology

The process of implementation may be a major determinant of success and a potential confounder in studies where e-alerts were not found to be beneficial. Indeed, there is an argument to evaluate e-alerts using a quality improvement approach and to capture data on how and in which settings e-alerts are most effective. Limited data have described the process of implementation and strategies for sustainability for e-alerts. In Xu et al. [25], providing multifaceted education was associated with an improvement in provider satisfaction and confidence in their ability to diagnose and manage AKI. In a survey of healthcare providers targeted by e-alerts for AKI in the trial by Wilson et al., only 69% of 98 respondents approved of continued use of the e-alert. Approval was highly correlated with the perception among respondents that the e-alerts translated into improvements in patient care; however, it was notable that approval decreased over time [35]. These findings reinforce the importance of deploying a rigourous process of implementation, including education and feedback to healthcare providers to ensure engagement and broad adoption.

An important consideration prior to e-alert implementation is what impact an AKI-specific e-alert may have in the context of ‘competing’ alerts that may exist within particular clinical settings or the EHR.

CDS content of e-alerts

The decision-support content of e-alerts varied widely. Guidelines for e-alerts for drug–drug interactions have recommended integrating clear and concise decision support [36]. E-alerts that provide directed context-specific management guidance may improve compliance along with translating into more appropriate investigations, monitoring and interventions, in particular among those not considered as experts in AKI (i.e. non-nephrologist). Emerging evidence in support of directed CDS is derived from a prospective evaluation of e-alert implementation paired with an AKI-specific care bundle, where compliance with the bundle was associated with improved outcome [9, 23].

Implications for policy, providers and research

Based on the studies included in our review, one implication may be that e-alert implementation alone may not contribute to broad improvements for the care of hospitalized patients with AKI. We would suggest, however, that if e-alerts are to be implemented, it should occur with rigour, and alerts should ideally be linked and integrated with care processes, and iteratively evaluated. Further evaluation is needed to understand the ideal populations, settings and context in which AKI e-alerts are most likely to improve the reliability and quality of care and outcomes. Larger scale implementation of AKI e-alerts may repeatedly prove ineffective unless alerts are tailored to the local site and context-specific care needs, and result in effective actions [37, 38]. Importantly, we strongly believe that e-alert implementation, considering the lack of benefit noted across the studies in our review, should consider integration of simple low-risk decision support aimed at harm avoidance (i.e. withdrawal or modification of potential nephrotoxins). Supplementary Appendix 8 summarizes our recommendations for AKI e-alerts.

Limitations

Our study has limitations. First, while the methodological quality of the included studies was good, there were few studies that fulfilled our eligibility and only one study was a randomized trial. Secondly, the process of e-alert implementation was variable and infrequently described, and few studies captured information on provider response to alerts. Thirdly, there was significant heterogeneity in the setting, alert format and in the targeted providers across studies. We were unable to perform significant pooled analyses for all outcomes. Accordingly, we presented a narrative summary of the included studies. For example, the pooled analysis for fluid therapy included only two studies, one of which was in an ICU setting with relatively small sample. Future studies should aim to rigourously evaluate how implementing an e-alert influences provider behaviour (i.e. stratified by setting/target and process of care indicators), and modifies patient outcomes and health services use. It is plausible that specific variation (i.e. customization) to an e-alert may be needed for different settings and care providers.

Conclusion

The available evidence shows that e-alerts for AKI do not improve survival or reduce RRT utilization. The impact of alerts on processes of care appears variable, reflecting differences in alert type, degree of integration with healthcare processes and the context in which they are applied. There is a significant gap in the knowledge related to the e-alerts in the AKI domain. Therefore, before drawing firm inferences about their efficacy and effectiveness, additional research is needed. Future work should focus on understanding those aspects of e-alerts that are most likely to improve care processes and outcomes.

SUPPLEMENTARY DATA

Supplementary data are available online at http://ndt.oxfordjournals.org.