A Trial of In-Hospital, Electronic Alerts for Acute Kidney Injury: Design and Rationale

Abstract

Background

Acute Kidney Injury (AKI) is common in hospitalized patients, increases morbidity and mortality, and is under-recognized. To improve provider recognition we previously developed an electronic alert system for AKI. To test the hypothesis that this electronic AKI alert could improve patient outcome, we designed a randomized controlled trial to test the effectiveness of this alert in hospitalized patients. The study design presented several methodologic, ethical, and statistical challenges.

Purpose

To highlight the challenges faced and the solutions employed in the design and implementation of a clinical trial to determine whether the provision of an early, electronic alert for AKI would improve outcomes in hospitalized patients. Challenges included how to randomize the delivery of the alert system, and the ethical framework for waiving informed consent. Other methodologic challenges included the selection and statistical evaluation of our study outcome, a ranked-composite of a continuous covariate (creatinine) and two dichotomous outcomes (dialysis and death), and the use of the medical record as a source of trial data.

Methods

We have designed a randomized trial to assess the effectiveness of an electronic alert system for AKI. With broad inclusion criteria, and a waiver of informed consent, we enroll and randomize virtually every patient with AKI in our hospital.

Results

As of March 31, 2014, we have enrolled 2373 patients of 2400 targeted. Pre-alert data demonstrated a strong association between severity of AKI and inpatient mortality with a range of 6.4% in those with mild, stage 1 AKI, to 29% among those with stage 3 AKI (p<0.001). We judged that informed consent would undermine the scientific validity of the study and present harms that are out of proportion to the very low risk intervention.

Conclusions

Our study demonstrates the feasibility of designing an ethical randomized controlled trial of an early electronic alert for AKI without obtaining informed consent from individual participants. Our study outcome may serve as a model for other studies of AKI, insofar as our paradigm accounts for the effect that early death and dialysis have on assessment of AKI severity as defined by maximum achieved serum creatinine.

Introduction and Background

Acute Kidney Injury (AKI) is defined as an abrupt loss of kidney function characterized by retention of nitrogenous waste products, dysregulation of electrolyte concentrations, and perturbations in extracellular volume status. AKI is common, occurring in up to 5–10% of hospitalized patients, and carries with it a substantially increased risk of in-hospital and long-term mortality^{1, 2}. In a study that examined United States hospital discharges among patients with an administrative claim for AKI, the inpatient mortality rate was greater than 20%³. Several other studies document similar risks, even among patients with very small elevations in serum creatinine concentration - a standard marker of kidney function ^4–7.

Despite the myriad physiologic disturbances that follow AKI, the condition is defined operationally by either an increase in the serum concentration of creatinine or a decrease in urine output. In 2012 the Kidney Disease Improving Global Outcomes group published consensus criteria for staging AKI that have been broadly adopted in the research community⁸. These definitions, which we utilize in our trial, are enumerated in Table 1.

Table 1.

KDIGO Creatinine Staging Criteria for AKI⁸

Stage	Serum Creatinine Concentration
1	1.5 to < 2 times baseline over 7 days OR ≥ 0.3mg/dl absolute increase over 2 days
2	2 to < 3 times baseline over 7 days
3	3 times baseline OR increase to ≥ 4.0 mg/dl OR initiation of renal replacement therapy

International guidelines for the treatment of AKI focus on appropriate management of drug dosing, avoidance of further nephrotoxic exposures, and careful attention to fluid and electrolyte balance⁸. For severe AKI, the institution of dialysis also may be required to achieve these management goals. Early nephrologist involvement also may improve outcomes in AKI⁹. Without appropriate provider recognition of AKI, however, none of these measures can be taken, and patient outcomes may suffer. Regrettably, AKI is frequently not appreciated or documented by clinicians¹⁰, despite its substantial cost, morbidity and mortality burden. In a study of University of Pennsylvania Health System patients with severe AKI (as defined by a doubling in serum creatinine), we found that only 43% had documentation of AKI in the medical record sufficient to generate a discharge diagnosis code. Adjusted for admission type and severity of illness, AKI documentation was associated with decreased mortality (odds ratio 0.63, 95% confidence interval 0.47 to 0.84, p=0.002)^{11, 12}.

The hypothesis that failure of providers to recognize or document AKI leads to adverse outcomes led us to consider the use of an electronic alert system to inform providers of AKI. While the electronic storage of medical data makes implementation of such systems increasingly feasible, no randomized studies exist examining electronic alerting for AKI, and few other studies are available to assess the potential benefit of such a system^13–16.

Methods

Study Design and Rationale for Design Decisions

We designed a single-center, randomized, controlled trial to evaluate the effectiveness of an electronic alert system for AKI. Patients are unaware of random assignment, i.e., to the alert intervention or to no intervention or of the intervention itself. All outcomes are assessed under blinded conditions. Providers are, of necessity, not blinded to the intervention.

Participants

We consider for enrollment all inpatients at our large, urban, academic medical center who develop AKI defined by the stage 1 creatinine criteria of the Kidney Disease: Improving Global Outcomes guidelines (Table 1). A computerized algorithm that we developed for this study is used to identify study participants. The algorithm examines, on an hourly basis, all serum creatinine values reported by the central laboratory in the electronic medical record. Each creatinine value is compared to prior values generated for the same patient. AKI is defined based on comparison of each creatinine value to antecedent values during the prior seven days (for 50% increase criterion) or two days (for 0.3mg/dl increase criterion). When this comparison meets AKI criteria, the study definition of AKI is satisfied, the patient is automatically enrolled and undergoes randomization (see examples, Figure 1). We define the "alert" creatinine as C1 and the prior creatinine to which it was compared as the baseline creatinine, or C0.

Figure 1. Time-Course of Study Entry.

Three patients who meet criteria for Acute Kidney Injury (AKI) on day 4.

–Patient 1 by 50% increase in serum creatinine over ≤7 days, Patient 2 by 0.3 mg/dL increase in serum creatinine increase over ≤48 hours, and Patient 3 by both criteria. Assessment of the combined primary outcome is limited to the 7 days after meeting AKI criteria.

Exclusion Criteria

Patients admitted to hospice services or observation units are not eligible for the trial, as the implementation of early treatment for AKI was not expected to impact their care. Patients with an initial hospital creatinine ≥ 4.0 mg/dl are excluded due to a lack of consensus definitions of AKI in this population. Patients who were enrolled during a prior episode of AKI are excluded to maintain independence of observations.

Intervention

Patients are randomized to the active intervention or to a no-contact control group. The intervention is a single electronic alert informing the relevant provider of the presence of AKI in the given patient. Our primary considerations in choosing the delivery method were rapidity of alerting and ability to translate the intervention into other health systems and conditions. Given its immediacy and generalizability, we utilize a short messaging system based text-paging approach. In our health system, each patient is electronically assigned a primary in-house provider (such as an intern or nurse practitioner) who acts as the clinical contact point for that patient. The electronic system sends a text page to this person and to the hospital unit pharmacist. This alert approach reaches only two individuals, but they are the two most intimately involved in the day-to-day care of the patient, particularly in the areas of order entry and review. A similar alert for acute respiratory distress syndrome is already used in the intensive care units of our health system^{17, 18}.

Alert Frequency

Alerts are delivered only once per study participant. While we considered repeated alerts (for example, daily during the duration of AKI), we were concerned that they might become onerous and lead to alert fatigue, a condition in which providers no longer pay attention to frequently repeated alerts ^19–21. In addition, one-time alerting represents a uniform intervention for all patients in the alert arm.

Alert Text

Although guidelines for the evaluation and management of patients with AKI exist⁸, there have been no randomized trials demonstrating the beneficial effect of any particular therapy after AKI onset. As such, we use broad language in the alert, as its purpose is to increase provider awareness rather than to increase the use of any particular diagnostic or therapeutic strategy. The text-page reads: "AKI ALERT: Patient [Initials], [Room Number], has been identified as having Acute Kidney Injury based upon the latest serum creatinine concentration. Please consider diagnostic and therapeutic options." This is followed by a second page reading "Please Note: THIS ALERT DOES NOT FIRE FOR ALL PATIENTS WITH AKI. For more information, visit [study website]."

Randomization

Patients are stratified at enrollment into four mutually-exclusive strata based on whether they are on a surgical or medical service and whether or not they are in the intensive care unit at the time of the alert. We generated randomization tables for each stratum using a permuted block randomization method²². Allocation concealment is maintained as the alert process is completely automated; only the computerized system has access to the randomization lists. Other studies of hospital alerts have randomized at the provider level in an effort to avoid contamination of the intervention^{23, 24}. A provider-level approach is feasible when the outcome is provider-based (such as rates of ordering certain contraindicated medication combinations), but becomes very problematic when the outcome of interest is a clinical change in patient status. We deemed provider-based randomization to be infeasible because patients at our institution are cared for by multiple providers who may change throughout the course of their stay. Under a provider-based randomization paradigm, nearly all patients would be expected to have at least one provider who received an AKI alert.

Primary Outcome

The primary outcome is a composite of relative maximum change in creatinine, dialysis, and death within 7 days after randomization. We hypothesized that making providers aware of AKI early in the clinical course would lead to corrective actions (such as volume resuscitation), the avoidance of further nephrotoxic insults, and early diagnostic evaluation, all of which may mitigate the severity or adverse consequences of AKI. Measuring the severity of an AKI episode is challenging, given the time-varying nature of serum creatinine concentration. An AKI episode begins with a defined serum creatinine increase (which triggers randomization). The creatinine may continue to rise over time based on, 1) the severity of AKI and 2) the duration of time since the onset of AKI. Other minor factors such as muscle mass, volume of distribution of creatinine, and severity of illness also may play a role in the creatinine increase²⁵. Designers of many studies have opted to use the "peak achieved serum creatinine" within a defined time frame as a marker of AKI severity^26–28. While attractive because of its simplicity, this approach would not account for the competing risk of mortality, as death may occur while the creatinine concentration is rising. A similar effect would be seen if the patient were dialyzed before the creatinine had peaked. Using peak serum creatinine, therefore, underestimates the severity of AKI in the two most significantly affected groups: those who die or who require dialysis.

To integrate the clinically relevant outcomes of change in creatinine, dialysis, and death, we created a rank-based primary outcome for this trial. Under this framework, all patients will be ranked based upon their maximum achieved relative change in creatinine within seven days of randomization. Maximum relative change will be calculated as maximum achieved creatinine divided by C1. Relative change (versus absolute change) was chosen because the same relative change in creatinine (from 0.5 – 1.0 mg/dl or from 1.0 – 2.0 mg/dl for example) corresponds to the same percentage reduction in glomerular filtration rate. Participants who receive dialysis within this time period (but do not die) receive a rank higher than the highest relative change in creatinine achieved by all participants who neither receive dialysis nor die. Participants who die receive a rank higher than the dialysis rank. A visual representation of this ranking schema appears as Figure 2.

Figure 2. Schema of our rank-based composite endpoint.

Participants are ranked initially based on peak relative change in serum creatinine concentration. At the end of 7 days, the outcome for patients who received dialysis but have not died is ranked worse than all patients who neither received dialysis nor died. The outcome for patients who died is ranked worst.

Secondary Outcomes of interest

We are particularly interested in process measures that might change in response to AKI alerts. Secondary outcomes are listed in Table 2. Given the number of outcomes assessed, any clinically or statistically significant findings will be considered hypothesis-generating and assessed directly in future trials.

Table 2.

Secondary Outcomes of Interest

Endpoint and Definitions	Data Source
Mortality
7-day mortality	Hospital record
Inpatient mortality	Hospital record
30-day mortality	Social Security Death Master File
Dialysis
7-day dialysis	Order entry system
Inpatient dialysis	Order entry system
Discharge on dialysis	Social work records
Renal Failure
Progression to stage 2 AKI	Laboratory values
Progression to stage 3 AKI	Laboratory values
Hospitalization
Hospital-free days post AKI	Hospital record
ICU-free days post AKI	Hospital record
Readmission rate and Cost
30-day readmission rate	Hospital record
Cost of hospitalization	Billing records
Process
Contrast administration	Order entry system
Fluid administration	Order entry system
Aminoglycoside administration	Order entry system
NSAID administration	Order entry system
ACE-Inhibitor / ARB administration	Order entry system
Renal consult	Direct chart review
Urinalysis order	Order entry system
Renal ultrasound order	Order entry system
Subclavian catheter placement	Procedure codes
Provider Awareness
Chart documentation of AKI	Billing records, direct chart review

Subgroup Analyses

The effect of the alert may differ based on patient and provider characteristics. To reduce alert fatigue, clinicians may wish to target alerts to high-risk patients or to providers who are at risk of failure to recognize acute kidney injury. We have pre-specified several subgroups of interest and justified our choices in Table 3.

Table 3.

Planned Subgroup Analyses and Justification

Subgroup of Primary Interest	Justification
Surgical patients	Risk of under-documentation33
Patients with low baseline creatinine	AKI occurs when creatinine in "normal range"*
Women	Lower rate of creatinine increase after AKI12
Black Race	Higher rate of creatinine increase after AKI12
Elderly Patients	Lower rate of creatinine increase after AKI12
Intensive Care Unit Patients*	AKI may be overlooked in setting of multiple clinical problems
Absolute versus relative AKI creatinine criteria on study entry.	Clinicians may be less likely to recognize relative increases vs. absolute increases

• Medical versus / surgical status and ICU versus non-ICU correspond to four mutually-exclusive randomization strata.

Contamination

As providers are not randomized, there is a risk of contaminating the intervention. A provider who receives an AKI alert about one patient may prompted to look for AKI in another patient. The alerts also may serve to educate individual providers about AKI criteria, thereby improving their ability to detect AKI over time. Given our randomization process, this level of contamination is unavoidable. We will pursue two avenues to measure it. First, from July 15 to September 17, 2013 we ran the alert algorithm without issuing alerts to providers. We captured a cohort of 770 patients with AKI who will serve as a pre-trial “baseline” to assess outcomes and process measures. A substantial decrease in outcome severity in the control group post-activation would suggest contamination. Second, we will examine the effect size between control and intervention groups over time. If substantial "education" and enhanced recognition of AKI is occurring across the entire population of providers at the institution, we would expect the effects associated with the alert to decrease over the duration of the study. Contamination is minimized by having the alert fire only once per patient, limiting a given provider’s exposure to the intervention.

Medical Record as a Source of Trial Data

While electronic medical records (EMRs) are used frequently in clinical trials to identify potential trial candidates, we intend to use the electronic medical record as our main source of trial data, including our primary outcome data. Real-time data collection is limited to the data provided to investigators by the alert system itself: Patient identification number, C0 and time, C1 and time, and primary providers. Upon patient discharge, retrospective data collection begins.

Our institution has created a central data processing unit, the Penn Data Store, whose mission is to improve the quality of patient care and facilitate research. This unit integrates data from multiple sources, including the laboratory system, electronic order entry system, and a patient demographic database. The Penn Data Store also contains International Classification of Diseases-9 comorbidity information and discharge disposition. We have utilized the Penn Data Store in the past to create the University of Pennsylvania Health System-AKI cohort, a retrospective study of more than 6000 patients who developed AKI in our health system, which characterizes the longitudinal experience of patients with AKI from admission to discharge via laboratory, medication, and procedure information¹². As part of our trial design efforts, we have held extensive discussions with the Penn Data Store team, piloted data extraction techniques, and engaged in quality assurance activities to ensure that all data obtained in this manner are accurate.

Statistical Analysis

As our primary outcome is ranked-based, we will use non-parametric testing to compare the intervention and control groups. This approach obviates the need to choose arbitrary "weights" for each outcome. In the rank-testing paradigm, we specified that dialysis is worse than any relative loss of kidney function, and that death is worse than dialysis, but we did not need to decide how much worse. Though non-parametric tests will provide a p-value for the difference in our rank-based outcome, they do not adequately describe the data. As such, we will report the individual components of the outcome (mean change in creatinine, proportion dialyzed, proportion who died), but provide a p-value for the overall analysis.

Power and Sample Size Considerations

Our primary analysis will use a nonparametric rank sum test, with a substantial number of tied values as those who die, or are referred for dialysis, will automatically be assigned the two highest ranks. Calculating a sample size that provides high power for differences in such an analysis is necessarily complex. We used the method described by Zhao et al, as implemented in the program PASS 12, to calculate sample size by dividing possible outcomes into six categories and estimating the expected proportion falling into each category using distributions of outcomes observed retrospectively during trial development (PASS 12, NCSS Inc, Kaysville UT).²⁹ Estimates are obtained by simulation; we specified 5000 simulations. We estimate that with 1200 patients per arm we will have at least 90% power to detect a downward shift in the outcome rankings in the intervention group that we judged would represent a clinically significant improvement. This shift would include, for example, a reduction in number of deaths from 10% to 9%, in dialysis from 5% to 4%, and an increase in the proportion of patients with no further creatinine increase beyond C1 from 50% to 56%. A table showing power for possible shifts in proportions is shown in supplemental table 1. These power estimates are conservative, because our analysis will use the actual values for each individual rather than categorizing these values, thereby utilizing more information. Thus the trial is powered to detect important and feasible reductions in severity of outcomes, and to allow for rational subgroup analyses outlined in Table 3. While subgroup analysis is a secondary goal, and power will obviously be more limited than in the primary analysis, we will retain the ability to note large shifts in the distribution of outcomes.

Ethical Issues

The ethics of the study required careful consideration. Outcomes will be measured across a population of hospitalized patients, but the study includes neither direct intervention on patients nor any contact between study investigators and patients. United States federal guidelines regarding waiver of consent require that 1) the research pose no more than minimal risk to the subject, 2) the waiver not adversely affect the rights and welfare of the subject, 3) the research could not be practicably carried out without a waiver, and 4) whenever appropriate, the subjects be provided with additional pertinent information after participation. The trial investigators concluded that this study met all criteria to qualify for a waiver of informed consent.

Waiver of informed consent and study procedures must be evaluated in terms of the consequences for patient autonomy and the potential for harm (i.e. violation of non-maleficence).^{30, 31} As we designed this study, we worked closely with the University of Pennsylvania Institutional Review Board to ensure that our approach was considered ethical and also would result in broadly applicable knowledge. In this study, the consequences regarding patient autonomy from eschewing informed consent should be contrasted with the potential procedures for obtaining consent from patients. Because AKI episodes are unpredictable, obtaining informed consent from patients would involve either prospectively informing all admitted patients about the study or finding all patients with AKI and enrolling them as rapidly as possible. Enrolling all hospitalized patients would be inefficient for both patients and study personnel, since approximately 5% of hospitalized patients experience AKI². This approach also would involve risks of loss of confidentiality for a large number of patients who never will develop AKI. Obtaining informed consent from patients with AKI near the time of the intervention would delay the alert; one of our core hypotheses is that an alert delivered at the moment of detection will allow for the clinician rapidly to assess and intervene on AKI. Furthermore, the very act of consenting patients for the study would serve as an AKI alert of sorts. Patients would be placed in the untenable position of withholding information from their providers which they may perceive as important for their providers to know.

The risks of harm also appear to be minimal. Enhanced provision of information about renal function to a provider cannot cause any direct harm to a patient. It is conceivable that providers may pay less attention to serum creatinine changes if they (falsely) believe that they will receive an alert for all patients with AKI. To address this concern, we include a warning in each text page explaining that the AKI alert system does not activate for all patients with AKI in order to reduce the risk of provider complacency.

Communication with Providers and Patients at Study Completion

Providers’ perception of the benefit of the alert is being assessed continuously during the trial via an online survey. In addition, a summary of trial results will be presented to all hospital departments, along with a justification for termination of the AKI alert program at the end of the study or continuance for all inpatients.

While patients are not and can not be informed of their participation at enrollment, we conceivably could inform them of their participation after discharge. This is problematic for several reasons. First, it may not be feasible to contact all participants after discharge given 1) the high (near 10%) inpatient mortality in the population studied, 2) the high proportion of patients discharged to other care facilities, and 3) the presence of inaccurate contact information in the medical record. In addition, we expect that many trial patients never will be told that they experienced AKI because of provider failure to recognize the condition, or because the condition was transient and clinically insignificant in the broader context of their care. Informing a patient who has recovered from their hospital stay that they experienced AKI, a condition familiar to few non-clinicians, may cause undue stress and burden to the patient. Thus, we elected not to inform patients of their participation in the study, even after discharge.

In summary, the investigators believed that the potential disadvantages of informed consent and post-study notification of participation outweighed the advantages. These issues were presented to the University of Pennsylvania Institutional Review Board which reviewed and approved this study.

Pre-Intervention Data

Study population

We analyzed 12 months of retrospective data to determine how many patients would be eligible for enrollment at our institution. Over that time period, 2,363 patients met inclusion criteria. The mean (standard deviation) C0 and C1 in that cohort were 1.18 (0.73) mg/dl, and 1.69 (0.96) mg/dl respectively. The mean peak creatinine achieved within 7 days was 1.98 (1.29) mg/dl. Most patients (79%) never achieved an AKI stage higher than 1, with 14% achieving stage 2 AKI and 6% achieving stage 3 AKI. The inpatient mortality rate was 8.8%. This rate varied significantly with peak achieved AKI stage: 6.4% mortality occurred among those with stage 1, 13.8% with stage 2, and 29% among those with stage 3 AKI (chi-square p<0.001).

Current Status

As of March 31^st, 2014, 2373 patients had been enrolled. Of these 1308 (55%) were medical patients, 1065 (45%) were surgical patients, and 715 (30%) were in an intensive care unit at the time of enrollment. The mean (standard deviation) C0 is 1.10 (0.73) mg/dl and the mean (standard deviation) C1 is 1.55 (0.84) mg/dl. Median (intra-quartile range) length of stay is 10 (6 – 17) days. The inpatient mortality was 9.4%. Mortality was greater among those who were in an intensive care unit at enrollment (20.4% vs 4.6%, p<0.001). Participants who died during the hospitalization had a higher C0 and C1 value than those who did not, mean (standard deviation) for C0 1.25 (0.75) vs. 1.09 (0.73) mg/dl, p=0.002, mean (standard deviation) for C1 1.73 (0.80) versus 1.53 (0.84), p<0.001.

Discussion

Acute kidney injury dramatically increases the risk of morbidity and mortality among hospitalized patients, even when the injury causes only small changes in serum creatinine. We designed a single-center trial to test whether drawing attention to AKI through electronic alerts will improve outcomes in patients with this condition. The development of this trial was challenging in several domains. First was the need to create a novel, electronic alert system tailored to this particular problem. Second was the choice of a primary outcome that was both feasible and clinically relevant. The statistical methodology to evaluate that outcome appropriately across the two arms of the trial required careful consideration of the impact of mortality and dialysis on the time-varying covariate of serum creatinine concentration. Third, the ethical implications of enrolling patients in a trial without informed consent had to be carefully considered by investigators and the responsible institutional review board.

If the AKI alerts are successful in limiting the magnitude of the creatinine increase, dialysis, and death in this single-center study, further evaluation of AKI alerts in larger and more diverse populations will be warranted. Multi-center studies would afford the opportunity to determine the generalizability of our findings and also would provide enough patients to power evaluation of important outcomes like dialysis, mortality, and renal recovery as individual endpoints. Given the current lack of treatments for AKI that can be easily implemented across diverse clinical populations, a simple, cost-effective, and (after the initial development process) fully-automated alert system would be a highly attractive method to improve patient care. In addition, as ongoing AKI research identifies potential novel therapeutics, such an alert system could facilitate efficient identification of patients for study and, ultimately, for timely, targeted treatment.

The trial may demonstrate that electronic alerts are ineffective for a variety of reasons. First, the alerts may be providing no new information to clinicians. Second, even if the alert improves AKI recognition, it may not follow that clinical management of the patient changes. Finally, even if management in the alert group differs from that in the control group, there may be no effect of these changes on our primary outcome.

Limitations

Our study is limited by the use of a composite outcome that includes a biochemical outcome (severity of AKI as measured by serum creatinine concentration), though we will assess the relevant clinical outcomes (death and dialysis) individually in secondary analyses. There are ample data to show that severity of AKI is strongly associated with these clinical outcomes ^{6, 12, 32}. A further limitation is the performance of this study within a single hospital. Underlying rates of AKI recognition and treatment approaches may differ substantially from hospital to hospital, thus modifying the benefit of an alert system. Finally, outcome assessment is performed passively. Participants in the alert arm of the trial may have more frequent creatinine measurements and thus a higher opportunity to capture the true “peak” creatinine compared to those in the no-contact arm, biasing our results away from the null hypothesis. In secondary analyses we will examine the effect of the intervention adjusted for number of creatinine measurements in the follow-up period to account for this effect.

Lessons Learned

The design of this trial posed unique statistical and ethical challenges. Choosing an endpoint that would capture AKI severity in terms of serum creatinine concentration while accounting for the competing effects of dialysis and death on creatinine increase required the development of an unusual, rank-based primary outcome. The ethical conduct of this study, in which informed consent would undermine both feasibility and scientific validity, required assurance to the institutional review board that the risk of any substantial harm to either patients or providers was extremely low. Finally, the use of the electronic medical record as our primary source of study data required extensive pilot testing and dedicated quality assurance efforts to ensure that all outcomes were assessed properly. The end result of these efforts is a trial that will assess the effectiveness of electronic alerting in a broad patient population.