Renal medication-related clinical decision support (CDS) alerts and overrides in the inpatient setting following implementation of a commercial electronic health record: implications for designing more effective alerts

Abstract

Objective

To assess the appropriateness of medication-related clinical decision support (CDS) alerts associated with renal insufficiency and the potential/actual harm from overriding the alerts.

Materials and Methods

Override rate frequency was recorded for all inpatients who had a renal CDS alert trigger between 05/2017 and 04/2018. Two random samples of 300 for each of 2 types of medication-related CDS alerts associated with renal insufficiency—“dose change” and “avoid medication”—were evaluated by 2 independent reviewers using predetermined criteria for appropriateness of alert trigger, appropriateness of override, and patient harm.

Results

We identified 37 100 “dose change” and 5095 “avoid medication” alerts in the population evaluated, and 100% of each were overridden. Dose change triggers were classified as 12.5% appropriate and overrides of these alerts classified as 90.5% appropriate. Avoid medication triggers were classified as 29.6% appropriate and overrides 76.5% appropriate. We identified 5 adverse drug events, and, of these, 4 of the 5 were due to inappropriately overridden alerts.

Conclusion

Alerts were nearly always presented inappropriately and were all overridden during the 1-year period studied. Alert fatigue resulting from receiving too many poor-quality alerts may result in failure to recognize errors that could lead to patient harm. Although medication-related CDS alerts associated with renal insufficiency had previously been found to be the most clinically beneficial alerts in a legacy system, in this system they were ineffective. These findings underscore the need for improvements in alert design, implementation, and monitoring of alert performance to make alerts more patient-specific and clinically appropriate.

INTRODUCTION

Adverse drug events (ADEs), defined as medication-related injuries, occur in approximately 1.5 million inpatients every year in the United States (US).¹ Some studies suggest ADEs account for 5%–17% of hospital admissions.^2–5 Of the 1.5 million ADEs occurring in the US annually, approximately 400 000 are considered preventable. Patients with renal insufficiency are at high risk for ADEs due to drug accumulation secondary to reduced renal elimination. They are also at higher risk of medication induced kidney damage than patients without preexisting kidney injury. Polypharmacy and comorbidities also increase the risk of ADEs for this vulnerable population.^6–10 Given their association with increase in hospital length of stay, costs, and morbidity and mortality, efforts have been made to reduce the frequency of ADEs.^11–14

One method for preventing ADEs is implementing medication-related clinical decision support (CDS); this can be especially effective in patients with renal insufficiency.¹⁵ Overall, medication-related CDS systems have been shown to reduce medication errors by 81%, although these data came from legacy, homegrown applications.^8,16 Even though electronic health records (EHRs) at most hospitals include some form of CDS, most of this CDS currently is vendor-developed, and often does not address medication use in patients with renal insufficiency. It has been reported that one-third of patients receive inappropriate doses based on renal function.¹⁰

Medication-related CDS represents an effective way to reduce errors, and ADEs.^17–19 However, this impact may be decreased or even extinguished if too many clinically inappropriate alerts are given.^20–22 This problem represents an important one in informatics today, as EHRs are now broadly implemented, and almost all are vendor-developed. One study reported approximately 60% of overrides of alerts were appropriate and the override rates varied based on type.²³ Duplication medication alerts were appropriately overridden 98.0% of the time, drug allergy alerts were appropriately overridden 96.5% the time, non-formulary medication alerts were appropriately overridden 82.5% of the time, drug–drug interaction alerts were appropriately overridden 26.4% of the time, age-based medication substitution alerts were appropriately overridden 26.4% of the time and medication-related CDS alerts associated with renal insufficiency were appropriately overridden 2.2% of the time. Many other studies have found higher override rates.^23,24 Alert fatigue introduces the risk of missing critical alerts that may compromise patient safety.^25–28 A study performed at our institution using a legacy, homegrown EHR system found that inappropriately overridden CDS alerts were associated with an increased risk of ADEs.²⁹

Few studies have evaluated the quality of CDS alerts in commercial systems by measuring frequency and appropriateness of alert overrides or ADEs resulting from overrides of alerts. The objective of this study was to analyze the appropriateness of medication-related CDS alerts associated with renal insufficiency, assess appropriateness of alert overrides, and assess the potential harm from overriding alerts.

MATERIALS AND METHODS

Study population

This study was performed at Brigham and Women’s Hospital, Boston, MA, a large urban academic medical center that uses a leading vendor EHR system in the US. We performed a retrospective evaluation of medication-related CDS alerts associated with renal insufficiency in adult inpatients (including the ICU and step-down units) from 05/2017 to 04/2018. All patients ≥ 18 years who had a medication-related CDS alert associated with renal insufficiency were included. Data for total number of alerts and alert override frequency was collected. There were 2 types of medication-related CDS alerts associated with renal insufficiency at our medical center. The first type, intended to convey a recommendation to adjust the medication dose, stated: “Specific dosing guidelines are not available for this patient’s level of renal impairment.” This alert fired for specific renally cleared medications such as gabapentin and potassium chloride when a patient did not have a recent serum creatinine value. The second type of alert stated: “This drug is not recommended for this patient’s level of renal impairment (CrCl 0–30).” This alert fired for some medications such as Nitrofurantoin and potassium chloride but only if the patient had a recent serum creatinine indicating renal function was below normal range but not necessarily impaired to a level at which the drug should be avoided. The alerts only consider serum creatinine levels drawn during the patient’s current hospital admission. For the second type of medication-related CDS alert associated with renal insufficiency, CrCl ranges were provided based on dosing guidelines for the specific medication recommended by our medication knowledge base, a third-party provider (First Databank, South San Francisco, CA). For example, if a physician prescribed nitrofurantoin, the alert stated, “This drug is not recommended for this patient’s level of renal impairment (CrCl 0–60).” However, this range of CrCl did not always correlate with the patient’s actual renal impairment. For example, a patient with a slightly lower than normal CrCl of 80mL/min would still receive an alert stating “This drug is not recommended for this patient’s level of renal impairment (CrCl 0–60)” as the alert fired based on drug prescribed not the patient’s actual level of renal impairment. An example of a medication for which either type of alert could present is potassium chloride. Orders for this medication in patients without a recent serum creatinine level would get a “dosing guidelines not available” alert and for patients with a recent serum creatinine indicating reduced renal function would get a “this drug not recommended” warning. In addition, the same patient could potentially receive the first type of warning initially and the second type of warning after a serum creatinine was obtained. The only time a medication-related CDS alert associated with renal insufficiency would remain silent is when the patient had a normal CrCl. For example, if a physician ordered potassium chloride and the patient had an estimated CrCl of 100ml/min, there would be no alert. The data on each alert trigger was obtained from our institution’s enterprise data warehouse. All data were collected with approval from Brigham and Women’s Hospital Institutional Review Board.

Chart reviews alert and overrides and ADEs

Random samples of 300 cases of the first type of alert “specific dosing guidelines are not available for this patient’s level of renal impairment” and 300 cases of the second type of alert “this drug is not recommended for this patient’s level of renal impairment,” stratified by level of renal impairment range provided in the alert, were reviewed for appropriateness. The renal alert strata were CrCl < 30, CrCl 31–60, and CrCl > 60. Data collected included the alert, the triggering medication, and the patient’s creatinine clearance if overridden.

The alerts were independently reviewed by 2 clinical pharmacists (1 with significant experience with renal transplant patients and 1 with significant experience in medication safety) based on 3 criteria: 1) alert appropriateness based on the triggering medication, 2) appropriateness of the override, and if the medication was administered; 3) occurrence of an ADE. Final criteria for alert trigger appropriateness and alert override appropriateness were discussed between the 2 reviewers and modified until a consensus was reached and are included in Figure 1 and Figure 2. Criteria considered for determining appropriateness included: comfort measure only (ie, hospice), chronic vs acute renal insufficiency, strong medication contraindication, 1-time dose, laboratory monitoring, and dialysis. Information on these characteristics was either extracted from the enterprise data warehouse or obtained by manual review of patient EHRs. Chronic renal insufficiency was defined as abnormalities of kidney structure or function, present for over 3 months with implications for health, based on cause, glomerular filtration rate (GFR) category and albuminuria category based on KDIGO guidelines. Acute renal insufficiency was defined as increase in SCr by ≥ 0.3 mg/dl within 48 hours or increase in SCr ≥ 1.5 times baseline which is known to have occurred within the prior 7 days or urine volume < 0.5ml/kg/h for 6 hours. Criteria for appropriateness of override were adapted based on previously published data, including guidelines as well as clinical experience (Figure 2).^23,30 Overridden alerts for medications that were not administered were included in the analysis of whether the alert trigger was appropriate and whether the alert override was appropriate but were excluded for the evaluation of ADEs as the medication never reached the patient. Inappropriate overrides for medications that did not reach the patient were considered potential ADEs. If different medications triggered unique alerts in the same patient, both alerts were reviewed and included in the dataset. However, only 1 alert per medication was allowed. Thus, multiple alerts triggered by the same medication for the same patient were excluded. Alerts for medications given for diagnostic procedures, and anesthesia in the OR, were also excluded, as there is typically no alternative and are 1-time doses. Baseline patient characteristics were collected for demographics.

Figure 1.

Criteria for appropriateness of alert trigger.

Figure 2.

Criteria for appropriateness of alert override.

To evaluate for ADEs, charts were reviewed for all patients in the random sample in which the medication was administered regardless of override appropriateness. The inter-rater agreement for appropriateness of alert triggers, appropriateness of override, and presence of ADE was determined with a Cohen’s k statistic. Disagreements were resolved by discussion between the 2 reviewers. If consensus could not be achieved, a third reviewer (a physician with significant experience in medication and patient safety, and in some cases, a nephrologist) was consulted.

ADE evaluation

An ADE was defined as an injury occurring from use of the triggering medication. We included instances where the patient’s serum creatinine increased between 10% and100% from baseline, which was considered a significant ADE. Baseline serum creatinine was the most recent inpatient serum creatinine prior to alert firing. An increase in serum creatinine could indicate further renal decline, and could be a result of overriding a drug or dose that was inappropriate given the patient’s baseline renal function. If serum creatinine increased more than 2-fold compared to baseline, this was classified as a serious ADE. Another type of ADE included electrolyte disturbances such as hyperkalemia, hyponatremia, hypermagnesemia, hypercalcemia and metabolic acidosis. These were defined by the reference ranges of the lab utilized at our institution. Hyperkalemia was defined as potassium levels above 5.5 mEq/L, with a level of above 6.0 considered moderate and anything above 7.0 mEq/L considered severe. Hyponatremia was defined as sodium levels below 135 mEq/L, with a level of 120–130 considered moderate hyponatremia and anything under 120 severe. Hypermagnesemia was defined as magnesium level above 2.2 mg/dL, with a level above 3 as moderate and anything above 7 mg/dL was considered severe. Hypercalcemia was defined as calcium level above 10mg/dL, with a level of 11.5 as moderate and anything above 12 severe. Metabolic acidosis was defined as a bicarbonate level below 15mEq/L and an anion gap greater than 12. The period of evaluation for ADE started after the alert override and continued for the remainder of the patient’s admission. Only ADEs that were specific to the overridden alert were included in analysis. Data relevant to override appropriateness and ADEs such as laboratory reports, medication orders, and patient notes documented by nurse or providers were abstracted and summarized by the reviewers. This was done so the ADES could be classified as either preventable or nonpreventable. ADEs were classified as preventable if there was an error associated with the ADE (which aligned with the appropriateness of an override). Study personnel had undergone training based on guidance developed by the Center for Excellence for Patient Safety Research at Brigham and Women’s Hospital, which has been described and used in previous studies.²⁷ The rates of inappropriate alert triggers, appropriate alert triggers, inappropriate overrides, appropriate overrides, and number of ADEs were calculated.

RESULTS

Our total study population consisted of 10 263 patients who received a medication-related CDS alert associated with renal insufficiency during the study period (Table 1). The 600 randomly selected alerts we reviewed occurred in 532 unique patients.

Table 1.

Patient demographics

Gender	Number (%)
Female	5464 (53.2)
Male	4779 (46.8)
Race	Number
White	8089 (78.8)
Black/African	1099 (10.7)
Other	360 (3.5)
Hispanic/Latino	233 (2.2)
Asian	235(2.3)
Unknown	247(2.4)
Mean Age
70 (19–104)
Location of Alerts	Number
Non-ICU Alerts	35 940 (85.2)
ICU Alerts	6255 (14.8)

Frequency of medication-related CDS alerts associated with renal insufficiency

There were 37 100 “specific dosing” guideline alerts, and 100% of these were overridden (Table 2). There were 5095 “this drug is not recommended” alerts and 100% were also overridden (Table 2).

Table 2.

Renal alert overrides

	“Specific dosing guidelines are not available for this patient’s level of renal impairment” Alerts (%)	“This drug is not recommended for this patient’s level of renal impairment” Alerts (%)
Number of Alerts	37 100	5095
Appropriate Alert Triggersa	12.5%	29.6%
Inappropriate Alert Triggersa	87.5%	70.4%
Number of Overridden Alerts	37 100 (100)	5095 (100)
Appropriate Overridesa	90.5%	76.5%
Inappropriate Overridesa	9.5%	23.5%

^aFrom a sample of 300 charts reviewed for dose alerts and 300 charts reviewed for drug not recommended alerts.

Specific dosing guideline alerts (“specific dosing guidelines are not available for this patient’s level of renal impairment”)

The random sample of 300 alerts reviewed represented 0.81% of the total number of alerts triggered. Of these alerts, 5 (1.7%) alerts associated with medications given for diagnostic procedures were excluded. The remaining 295 alerts were included in the analysis. The top 5 drugs that triggered alerts (Table 3) were gabapentin, vancomycin, potassium chloride, furosemide, and calcium gluconate. The Cohen’s k statistic for appropriateness of alert trigger was 0.96 (CI: 0.91–1.0), and for appropriateness of alert override was 0.92 (CI: 0.84–1.0), both showing a high level of agreement between the 2 reviewers. Alert triggers were classified as appropriate in 12.5% of cases (n = 38), and inappropriate in 87.5% of cases (n = 257) (Table 2). Alerts were determined to be overridden appropriately 90.5% of the time (n = 267) and inappropriately 9.5% of the time (n = 28). Overall, 81.7% (n = 241) of the medications reached the patient.

Table 3.

Top 5 medications triggering specific dosing guideline alerts

Total sample drug name	Total number of alerts N = 37 100 (%)
Gabapentin	5080 (13.7)
Vancomycin	3105 (8.4)
Potassium Chloride	3809 (10.3)
Furosemide	2146 (5.8)
Calcium Gluconate	2050 (5.5)

This drug not recommended alerts (“this drug is not recommended for this patient’s level of renal impairment”)

The random sample of 300 alerts reviewed represented 5.9% of the total alerts triggered. Of these, 23 (7.7%) alerts associated with medications given for diagnostic procedures were excluded, leaving 277 alerts in the analysis. The top 5 drugs that triggered alerts were potassium chloride, cholecalciferol, chlorothiazide sodium, magnesium oxide, and desmopressin (Table 4). The Cohen’s k statistic for appropriateness of alert trigger was 0.96 (CI: 0.92 –0.99) and for appropriateness of override was 0.96 (CI: 0.92–1.0). Alert triggers were classified as appropriate in 29.6% of cases (n = 82) and inappropriate in 70.4% of cases (n = 195) Table 2) Alert overrides were determined to be appropriate 76.5% of the time (n = 212) and inappropriate 23.5% of the time (n = 65). Overall, 68.2% (n = 189) of the medications reached the patient.

Table 4.

Top 5 medications that triggered “This drug is not recommended for this patient’s level of renal impairment alert”

Total sample drug name	Number of alerts N = 5095 (%)
Potassium Chloride	1894 (37.2)
Cholecalciferol	478 (9.4)
Chlorothiazide Sodium	377 (7.4)
Magnesium oxide	328 (6.4)
Desmopressin	274 (5.4)

Adverse drug events

The Cohen’s k statistic for presence of ADEs was 0.99 (CI: 0.94–1.0). Only 5 ADEs were identified; all were significant. Of these, 4 of the 5 ADEs were considered preventable, and 1 was judged nonpreventable. In 1 case, fenofibrate was administered to the patient at a dose that was too high based on their renal function. This resulted in a significant increase in creatinine kinase (CK) and worsening renal function and the drug had to be stopped. In another case, a patient’s furosemide dosage was increased from 20 mg to 160mg abruptly, leading to worsening kidney function. In a third case, the patient received a single dose of magnesium, when the patient’s magnesium levels were already within normal range, resulting in a markedly elevated magnesium level of 5.5 and stopping of the medication. In addition, a patient with acute kidney injury received chlorothiazide which resulted in worsening kidney function. In the case judged nonpreventable, the patient received calcium acetate for high phosphate levels, resulting in elevated calcium levels even though the phosphate levels remained elevated.

DISCUSSION

We evaluated medication-related CDS alerts associated with renal insufficiency delivered in our vendor EHR and found that all these alerts were overridden, in contrast to the performance of alerts in the past which had been quite effective and had been demonstrated to improve dosing accuracy and decrease length of stay.¹⁵ In this study, nearly 9 out of 10 alerts were classified as inappropriate, and therefore the majority of alert overrides were considered appropriate.

The override rate was much higher than previous studies, but the appropriateness of alert overrides was consistent with previous studies.^30,31 These data demonstrate that there is substantial room for improvement in our current process in several areas, including the monitoring approach, accuracy of alerts, and how suggestions are delivered. We have shared these data with the operational team who has made improvements in alerts for selected medications including vancomycin and furosemide, 2 frequently used medications with high risk of harm if dosed inappropriately. The improvements include decreasing the CrCl threshold at which an alert would fire to reduce the number of alerts shown to providers.

We found 5 ADEs in the cases we reviewed, 4 of which were preventable. Clearly, this issue needs to be addressed to reduce the frequency of medication-related harm. This is very unlikely to be a local issue, as the vendors evaluated both for EHR and medication knowledge base are 2 of the most widely implemented in the US.

There are several potential reasons why our current medication-related CDS alerts associated with renal insufficiency are less effective than they had been in the legacy homegrown system:

• No automatic calculation of level of renal function (CrCl or estimated GFR)

• No automatic dose adjustment

• No recommendations for alternatives

• No consideration of patient specific parameters, such as dialysis

In the past, the EHR calculated the patient’s estimated GFR and, if a nephrotoxic or renally excreted medication was ordered, had the application default to the appropriate dose and frequency given that patient’s kidney function, so that it was easy for providers to select the correct dose for the individual patient. However, our current vendor EHR does not allow tailoring of dosage based on a patient’s level of renal function. Furthermore, in the current application, multiple alerts fire at once with most being low-value, and the new system gives the provider the ability to override all alerts at once. This ability to override all alerts at once allows physicians to skip reviewing many alerts which we did not previously allow, although we were much more selective about delivering them.

By analyzing the renal insufficiency triggers, we showed that most alert triggers were in fact inappropriate. Alerts fired even when the patient did not have renal insufficiency or if the dose had already been adjusted for renal function. The alerts do not consider inpatient lab values for renal function that are drawn on the same day as the alert. Some common examples of alerts that were considered inappropriate were those for low dose aspirin, potassium chloride, and vancomycin. Other examples include alerts for single doses of electrolytes or diuretics when the patient is being monitored and alerts firing when dose had been adjusted appropriately for renal function of the patient. Even when the patient did have renal insufficiency, the alerts did not provide insightful or helpful information to providers, which is consistent with previous studies that have identified that better design and implementation are needed for renal medication CDS.³² One alert simply stated that, “This drug is not recommended for this patient’s level of renal impairment,” which was triggered for a drug that would require dosing adjustment based on a recent serum creatinine available for the patient. The second alert stated “Specific dosing guidelines are not available for this drug based on patient renal impairment,” which was triggered for a drug that required renal dose adjustment when no recent serum creatinine was available. Neither of these alerts provided physicians with any further guidance. These characteristics violate basic human factors principles around alerting.³² Since nearly all these alerts were inappropriate, it is not surprising that clinicians developed alert fatigue. Physicians become desensitized to important renal insufficiency alert triggers, as most alert triggers become white noise. This also leads to inappropriate alert overrides which could lead to patient harm. Inappropriate alert overrides have the potential of causing harm, although in this study we only identified a few ADEs that occurred in the inappropriate alert overrides that were administered to the patient. Overrides that presented significant risk may have been more often intercepted before reaching the patient. In addition, inadequate documentation of some types of ADEs in the chart may have prevented identification of some ADEs that occurred during the hospital admission, resulting in underestimation of ADE incidence.

One way to improve renal CDS alert triggers is to create alert triggers that offer recommendations for alternative medications to physicians. A previous study conducted at BWH using the hospital’s in-house developed legacy, homegrown EHR system showed that when physicians were provided with alternatives, high warning alert triggers were accepted 100% of the time.^33–35 Lastly, the alert triggers do not currently consider patient-specific parameters, such as dialysis or end of life care. Alert triggers should take these factors into consideration before firing to reduce alert burden presented to providers.

This study has several limitations. We retrospectively evaluated the alerts; therefore, it is hard to tell why a physician may have overridden the alert and challenging to identify all patient harm. Identification of ADEs requires adequate documentation in the EHR, which does not always occur. We also only evaluated ADEs that were documented during hospitalization and did not follow up on patients after discharge, where they could have remained on the drug and developed ADEs that take longer to manifest. This study was done at 1 site, so these results might not be generalizable. Our institution utilizes an EHR and knowledge base that may not be used at other institutions. Alert logic from knowledge bases are based on set criteria but may be tailored to institution-specific needs. However, given that the EHR we use is among the most commonly adopted, we believe that our situation applies to a significant number of other institutions with a similar commercial EHR.³¹

CONCLUSION

We evaluated renal CDS alert triggers in a newly implemented commercial EHR and found that all were overridden. Most alerts were appropriately overridden, though this was the case because the alert frequency was so high—this problem of over-alerting is a critical issue in informatics today and highlights the need to redesign medication-related CDS alerts associated with renal insufficiency. Even with a high rate of appropriately overridden alerts we discovered 4 ADEs out of 5 that could have been prevented if the alerts fired more appropriately. Future studies should focus on application of human factors principles in redesigning and implementing medication-related CDS alerts associated with renal insufficiency, with a goal of reducing the number of inappropriate alerts presented to providers by making alerts more patient-specific and clinically appropriate. Specifically, these alerts should automatically calculate GFR, suggest an adjusted dose or frequency for that patient, recommend alternatives when appropriate, and consider other patient-specific parameters, such as whether or not the patient is on dialysis.

FUNDING

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

AUTHOR CONTRIBUTIONS

SNS and DLS made substantial contributions to the conception of the design the work and interpretation of data; DWB supervised the research design and contributed to the writing and revision of the manuscript. KGA, MGA, and SNS assisted in the data collection and interpretation. All authors give approval for the final version to be published and agree to be accountable for all aspects of the work, ensuring that any questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

CONFLICT OF INTEREST STATEMENT

Dr. Bates consults for EarlySense, which makes patient safety monitoring systems. He receives cash compensation from CDI (Negev), Ltd, which is a not-for-profit incubator for health IT startups. He receives equity from ValeraHealth which makes software to help patients with chronic diseases. He receives equity from Clew which makes software to support clinical decision-making in intensive care. He receives equity from MDClone which takes clinical data and produces deidentified versions of it. He receives equity from AESOP which makes software to reduce medication error rates. He receives research funding from IBM Watson Health. He serves as a Visiting Professor at Stavanger University. The other authors have no conflicts of interest to declare.