Drug Interaction Alert Override Rates in the Meaningful Use Era

Summary

Background

Interruptive drug interaction alerts may reduce adverse drug events and are required for Stage I Meaningful Use attestation. For the last decade override rates have been very high. Despite their widespread use in commercial EHR systems, previously described interventions to improve alert frequency and acceptance have not been well studied.

Objectives

(1) To measure override rates of inpatient medication alerts within a commercial clinical decision support system, and assess the impact of local customization efforts. (2) To compare override rates between drug-drug interaction and drug-allergy interaction alerts, between attending and resident physicians, and between public and academic hospitals. (3) To measure the correlation between physicians’ individual alert quantities and override rates as an indicator of potential alert fatigue.

Methods

We retrospectively analyzed physician responses to drug-drug and drug-allergy interaction alerts, as generated by a common decision support product in a large teaching hospital system.

Results

(1) Over four days, 461 different physicians entered 18,354 medication orders, resulting in 2,455 visible alerts; 2,280 alerts (93%) were overridden. (2) The drug-drug alert override rate was 95.1%, statistically higher than the rate for drug-allergy alerts (90.9%) (p < 0.001). There was no significant difference in override rates between attendings and residents, or between hospitals. (3) Physicians saw a mean of 1.3 alerts per day, and the number of alerts per physician was not significantly correlated with override rate (R2 = 0.03, p = 0.41).

Conclusions

Despite intensive efforts to improve a commercial drug interaction alert system and to reduce alerting, override rates remain as high as reported over a decade ago. Alert fatigue does not seem to contribute. The results suggest the need to fundamentally question the premises of drug interaction alert systems.

1. Introduction

Computerized practitioner (or provider) order entry (CPOE) systems with integrated clinical decision support (CDS), intended to reduce preventable adverse drug events (ADEs), are increasingly prevalent due their inclusion into the Medicare and Medicaid EHR Incentive Program requirements [1, 2, 3]. Most of these systems create pop-up alerts at the point of order entry generated by checking for drug-drug and drug-allergy interactions, two functionalities required for Stage 1 Meaningful Use attestation by both eligible professionals and hospitals [4].

Order checking alerts interrupt workflow to prompt a change in therapy. Clinicians persistently override the majority of alerts despite attempts to improve their relevance, raising questions about the alerts’ continued clinical utility, safety benefit, and acceptance by ordering practitioners [5, 6]. High override rates have continued at many institutions despite efforts to improve alert relevance [7, 8]. This has raised concern for “alert fatigue”, in which excessive quantities of alerts may desensitize providers to such warnings. According to this theory, providers are exposed to an overwhelming number of both relevant and irrelevant alerts resulting in information overload, leading to more frequent overrides of both appropriate and clinically irrelevant alerts. Alert fatigue has been suspected to contribute to inappropriate prescribing, but its direct influence on override rates is not well described [9]. Physicians may become disenchanted with CDS because of negative experiences with medication alerts.

Some efforts to improve alert acceptance have been met with success, especially when excessive alerts for low risk interactions are removed from the provider’s workflow [10, 11]. Our medical center removed alerts for conditions with low patient risk, improved accuracy of the patient medication lists on which alerts rely, and worked with vendors to modify alert rules for specific patient groups. Prior studies reporting success with similar measures were conducted in locally-developed EHRs in the outpatient setting, rather than with commercial EHRs in the high-volume prescribing environment of inpatient care [12]. Most of the evidence for alert improvements also comes from institutions with customized CDS systems, and may not be applicable to the more common commercial systems in use at many hospitals, which have not been well studied [13]. The need to understand alert improvement within the confines of commercially available inpatient systems used in the care of most Americans is important in the era of Meaningful Use incentives, which reward implementation regardless of alert performance or patient outcomes.

2. Objectives

We sought to measure alert override rates on inpatient medication orders entered by physicians using a commercial electronic medical record and clinical decision support system. We also examined whether the type of alert, physician experience level, or specific hospital site affected override rates. We hypothesized that current rates in our hospitals would be significantly lower than historical rates from other studies due to our ongoing quality improvement processes. Further, we hypothesized that physicians who saw higher quantities of alerts would have higher override rates, as predicted by the theory of alert fatigue.

3. Methods

3.1 Setting

The University of Washington healthcare system (UW Medicine) includes two primary teaching hospitals. UW Medical Center (UWMC) is a 450-bed university hospital with a focus on advanced subspecialty care. UWMC provides standard acute care, solid organ transplantation, and cardiac catheterization facilities as well as medical, cardiac, oncologic, and neonatal intensive care and has 15,137 surgical cases annually. Harborview Medical Center (HMC) is a 413-bed county hospital and level 1 trauma center providing indigent and tertiary care. HMC has 100 of its beds distributed among the Medical, Cardiac, Neurologic, Trauma/Surgery, Burn, and Pediatric intensive care units, and has 14,872 surgery cases annually. As of 2010, HMC had about 62,000 annual Emergency Department visits, compared with 23,000 for UWMC. Both hospitals have about 19,000 admissions per year. The two hospitals share a common pool of approximately 1,000 residents and fellows, supervised by a UW Medicine faculty of about 1,800 attending physicians.

Since 2003, UW Medicine has used Cerner Millennium as its core inpatient electronic medical record at both hospitals. The system is used across all inpatient and emergency departments for CPOE and CDS, results review, electronic documentation, support of physician sign out and rounding workflow, electronic medication administration records, and ongoing adoption of bar code medication administration. Epic Systems’ products are used for admission, discharge, and transfer processing and for the master patient index. Epic Systems EpicCare was implemented in 1997 as the electronic medical record for outpatient primary and later specialty care. EpicCare is used for outpatient CPOE, electronic documentation by all specialties, clinic workflow and messaging, and to support a patient portal. A clinical data repository (MINDscape) provides access to legacy data, and is used for a combined view of inpatient and outpatient documentation and test results.

Medication orders are checked for both drug-drug and drug-allergy interactions at the point of entry by comparison with Cerner’s Multum knowledge base. Each pairwise interaction has a default severity from “minor” to “major”, where “major” interactions carry the highest risk of adverse events. Pharmacy staff can see alerts for interactions of all severity levels during order processing, but only major interactions generate interruptive alerts to the physician during order entry and are recorded within the EHR for later review. Outpatient prescriptions are handled by a separate EHR, with a different order checking system, and thus were not included in the current study.

3.2 Knowledge base customization

A panel of physicians, pharmacists, and information technology staff has met monthly since 2008 to review all major drug-drug interaction alerts. The panel uses the operational classification scheme described by Hansten and Horn to reassign major drug-drug interactions to lower severity levels, and thus decrease interruptive alerting [14]. Candidate interactions are identified by review of internal alert override data, and also by direct discussion with providers and team pharmacists. Monthly review is necessary due to frequent vendor updates of the interaction knowledge base.

Reclassification is based on the expert opinion of multiple reviewing pharmacists, but also considers published data on interaction severity and seeks consistency among interactions with similar pharmacologic effects [15]. If an interaction has unclear severity, the full panel uses input from the relevant providers to help assess clinical impact and assign an appropriate level. The new severity level is then entered into the interaction database, and the interaction is flagged to prevent future vendor updates from overwriting changes. All interactions were reviewed before the implementation of CDS, causing 65% of all major interactions to be reclassified to lower levels [15]. As an example, the macrolide and warfarin interaction was high severity by default, but was reclassified to moderate severity due to our institution’s extensive anticoagulation monitoring orders, which lower the risk of harm. Other interactions, such as olanzapine and non-parenteral benzodiazepines, were reclassified due to a lack of published evidence for harm and the need to combine them for therapeutic effect. All of these changes are applied to the interaction database at both hospitals simultaneously, and affect all departments.

A similar review of the allergy cross-reactivity rules before implementation found that most were appropriate. While a few allergy alerts were removed, further review is not routinely performed.

3.3 Design and scope

We retrospectively reviewed data for major medication alerts generated at UWMC and HMC between Monday, June 10 2013 at 00:00 and Thursday, June 13 2013 at 23:59. This continuous 96 hour order entry period provided adequate data and excluded dates of resident and faculty turnover. The study period fell at the end of the academic year to ensure that all providers had several months experience with the order entry system. All inpatient medical and surgical services were represented, including the medical and surgical acute care floors, intensive care units, emergency department, observation unit, and dedicated subspecialty services such as Cardiology and Neurosurgery. Because many of our ED patients are admitted, the ED and inpatient encounters are combined within the EHR to facilitate continuity of care for critically ill patients. A clinical pharmacist wrote custom queries using EHR functions and Cerner’s proprietary database programming language (CCL) to extract alerts meeting the inclusion criteria. Alerts were included if they occurred within the study time period, were handled by a physician, and occurred on an inpatient. There were no exclusion criteria.

Each alert record contained the interaction type, timestamp, triggering order, interacting medication, hospital, provider name, provider credentials, provider practice level (resident versus attending), and provider-selected override reason. As configured by the vendor, the interruptive alert window prevents further ordering action until the provider selects a single override reason from a coded list (▶ Figure 1). The alert window and choices for the provider are identical for both drug-drug and drug-allergy alerts: “Allergy: Previously Received”, “Allergy: Not True Allergy”, “Allergy: Desensitized”, “Provider Approved”, “Not Applicable”, “Interaction: Indicated”, or “Clinical Pharmacist Notified”. Presence of a reason indicated an alert override, while its absence indicated cancellation of the order (▶ Figure 1).

Fig. 1.

Interruptive drug-allergy alert window within the Cerner CPOE interface. Workflow can only resume after providers cancel the alerted order or override the alert with selection of a coded reason (Figure with permission: © 2013 Cerner Corporation. All Rights Reserved.)

3.4 Analysis

Data extracted included alerts associated with orders written by physicians, pharmacists, and registered nurses. Medication orders are typically entered by physicians, or by pharmacists who order on behalf of authorized providers. There are very few nurse practitioners assisting the surgery and ER teams, and registered nurses do not accept verbal medication orders. Pharmacists frequently enter dosage adjustments or therapeutic substitutions for medications. Pharmacists often override alerts based on discussions with the medical team rather than new alert information. Pharmacists see all severity alerts and do not benefit from reductions in high severity alerting. Because of these confounding factors, we focused on physicians alone to assess the impact of alert reclassification.

Records were filtered to include only alerts associated with credentials of MD (Doctor of Medicine), DO (Doctor of Osteopathic Medicine), DDS (Doctor of Dental Surgery), or MBBS (Bachelor of Medicine, Bachelor of Surgery) (▶ Figure 2). Medication orders may trigger multiple interactions and generate several alerts within a single window. These alerts cannot be overridden or accepted independently. Because the override of multiple alerts represented a single decision, these coincident alerts were included as a single override in the main analysis. Since each individual medication interaction had the ability to trigger an alert we counted each interaction separately during the analysis of triggering medication frequency.

Fig. 2.

Alert record processing before analysis. Excluded records were duplicates, alerts viewed by non-physicians, or alerts which could not be accepted independently from another in the same window.

Override rates of alerts within different categories were compared with Pearson’s chi-squared contingency tables and adequate group numbers. To search for evidence of alert fatigue, simple linear regression was used to correlate individual physicians’ alert quantity and override rate; significance was measured by 2-tailed F test. Statistical significance was predefined as α = 0.05. All analysis was conducted using Microsoft Excel™.

This study was approved by the University of Washington Human Subjects Division.

4. Results

During the four day audit period providers entered 43,287 medication orders, but only 18,354 were entered directly by physicians. The four day audit period captured 2,455 distinct alerts, involving 461 distinct physicians, for an overall alerting rate of 13.4% (2,455 alerts per 18,354 orders) (▶ Table 1). Drug-drug and drug-allergy alerts were equally represented, as were both hospitals, and 80% of alerts were triggered by resident physicians.

Table 1.

Physicians’ critical medication alert characteristics and override rates. P values are for difference between categories using Pearson’s chi square test.

Category	Total	% of all alerts	Override number	Override rate (%)	P
Unique alerts	2455	100	2280	93
Interaction type
Drug-drug	1153	47	1097	95	< 0.001
Drug-allergy	1302	53	1183	91
• Exact match	123	9	94	76	< 0.001
• Cross reaction	1179	48	1089	92
Physician level
Attending	480	20	454	95	0.11
Resident	1975	80	1830	93
Hospital
HMC	1200	49	1111	92	0.25
UWMC	1255	51	1175	94

Override rates were high in all categories, but the drug-drug alert override rate was significantly higher than that for drug-allergy alerts (95.1% vs. 90.9%, p < 0.001). There was no significant difference in override rates between the two hospital sites (p = 0.25), or between attending and resident physicians (p = 0.11). About 20% of all alerts occurred as multiples within the same alert window. Multiple alerts had a similar override rate (92%) as single alerts.

A majority of drug-drug and drug-allergy interaction pairs were overridden at every occurrence. For example, the olanzapine-lorazepam drug interaction alert occurred 92 times and was overridden every time. Most triggering medication classes had drug-drug override rates greater than 90% when grouped by class (▶ Table 2). Override rates for drug-allergy alerts were similarly high among all classes, with the notable exception of antibiotics. Opioids accounted for the majority of allergy alert triggers; drug-drug alert triggers were more diverse.

Table 2.

Overrides of alerts triggered by selected drug classes. Numbers indicate quantity of overrides as a fraction of all alerts for the class, with override rate in parentheses. “Psychiatric” medications include antipsychotics, antide-pressants, and benzodiazepines.

Drug class	Drug-allergy alert overrides		Drug-drug alert overrides
Opioids	1140 / 1227	93%	144 / 150	96%
Other analgesics	109 / 117	93%	81 / 83	98%
Antibiotics	121 / 172	70%	43 / 45	96%
Diuretics	87 / 90	97%	53 / 53	100%
Antithrombotics	21 / 23	91%	78 / 80	98%
Psychiatric	4 / 5	80%	469 / 485	97%

Most overriding physicians entered the reason “Provider Approved” for both alert types, and 18% of drug-drug alert override reasons inappropriately used an override indication for allergies. Only 9% of allergy alerts (123/1302) were triggered by exact matches between the ordered medication and the allergy. Most (76% or 94/123) of these exact allergy alerts were overridden. The remainders of allergy alerts were triggered by cross-reactivity with other drugs.

Physicians triggered a median of 3 alerts over the 4 day period, with a mean of 1.3 alerts per day, while entering an average of 10 medication orders per day. Fewer than 5% of individuals viewed more than 4 alerts per day. Individual override rates varied widely, even among physicians with identical alert quantities. There was no significant association between physicians’ alert quantity and their override rate (R² = 0.03, p = 0.41, ▶ Figure 3).

Fig. 3.

Physician override rate does not correlate with viewed alert quantity. Markers denote mean override rates among each set of physicians with the same alert quantity. R2 = 0.03, p = 0.41.

5. Discussion

Drug-drug interaction alerting with commercial EHR software is no longer limited to early adopters. It is now used in over 85% of U.S. hospitals [16]. While it has been known for over a decade that drug interaction and allergy alerts were met with a high override rate, the importance of this study lies in the finding that commercial EHRs as federally required to meet core Meaningful Use objectives have not managed to improve this issue. Physicians at our institution are unhappy with, and increasingly jaded by, decision support features that were intended to provide safety and to demonstrate the benefits of decision support. We are the first to report a lack of improvement in decision support acceptance (as measured by overrides) after Meaningful Use requirements took effect.

University of Washington made a strong effort to improve CDS and its acceptance based on suggestions in the literature using local expertise to modify the default alerts within the constraints of our commercial system.[11, 17, 18, 19, 20] The alert window in our EHR is interruptive, concise, and requires a coded override reason rather than free text. Only interactions in the highest tier of severity generate interruptive alerts. Regular review of our commercial interaction knowledge base by a multidisciplinary task force has reclassified many “major” interactions to lower tiers, reducing the number of alerts. These interventions have not reduced the rate at which physicians override high risk medication alerts, suggesting that the notion that drug-drug alerts provide value may be flawed and that further incremental changes may not be sufficient to improve alert specificity and physician compliance.

There has been no apparent decline in override rates of CDS alerts in the 15 years of reported literature. Our current rates are at or above the rates in previous studies, including those at our regional VA hospital [7, 8]. Similar override rates of over 90% have been found in drug dose alerts [21]. The high override rates are especially notable in light of ARRA incentive programs and Meaningful Use requirements, which are increasing utilization of EHRs with clinical decision support nationwide. The persistent disregard for alerts raises concern that they may hinder clinical workflow and decision making, rather than reduce the substantial costs of ADEs [3, 22].

The low number of alerts per physician makes it less likely that quantity-based alert fatigue is a major contributor to dismissal of alerts. We found no evidence that physicians who saw more alerts were more likely to dismiss them. The lack of correlation between alert quantity and override rate also supports prior evidence that reducing the number of alerts will not appreciably increase physician acceptance or reduce overrides [23]. In contrast to one prior study, our override rate did not vary with physicians’ trainee status, and ordering behavior appeared similar among resident and attending physicians [5]. The similar rate between two hospitals with a shared staff and EHR also implies that unique patient demographics or medication use patterns are not influencing overrides.

The frequent overrides of most alert types including exact allergy matches, and the often inappropriate (unmatched to the alert) override reasons suggest that physicians dismiss alerts without assessing the specific risk of each alert. As Hayward et al. describe, alerts appear “too late”, when the prescribing decision has been made and negotiated with the patient. Consequently, alerts do not fit correctly into the workflow and are likely to be ignored by providers [24].

Alert overriding might also result from an ignorance of interaction significance that precludes meaningful risk assessment. Physicians are aware of inaccurate allergy charting and interaction data, which can reduce the perceived interaction risk relative to known clinical benefit. This study was limited to physicians. However, we acknowledge that pharmacists, nurses, and other providers may also suffer from alert fatigue or other behavioral barriers to effective alert response and warrant study in systems where they direct therapeutic choices.

The response to alert reclassification has been disappointing at the University of Washington in that override rates remain high. Further refinements might improve physician compliance and facilitate quality improvement. Creating a visual distinction between drug-allergy alerts and drug-drug alerts might increase provider attention and compliance. Changing the available list of override reasons is unlikely to improve quality, as providers almost always selected the first option regardless of appropriateness.

We propose a crowdsourcing approach and suggest that vendors should make it simpler for providers to flag inappropriate alerts for review and indicate why an alert is unhelpful. Our improvement team is limited to revising interactions one pair at a time, rather than across an entire drug class [25]. Vendor added classes would allow for class-level customization and would expedite removal of irrelevant alerts, freeing up expertise for detailed review of interactions with unclear safety implications [26]. While we work with our vendors to make changes, we suspect that even these changes may not increase the acceptance of interruptive alerts.

Many alert overrides are appropriate with such a nonspecific alerting system. However, we did not review the overridden alerts for appropriateness in this study. There is yet no evidence for an optimal override rate. Many physicians report these alerts unhelpful and disruptive despite the steps taken to improve alert relevance and usability [27]. While the overall risk of ADEs is low, our very high override rates could increase the absolute number of inappropriate overrides, raising the ADE risk [5, 28].

High drug-allergy override rates suggest that these alerts may not be preventing harm from true drug allergies, where a single dose of medication may cause serious reactions. Like many institutions, we struggle to achieve accurate allergy charting, especially when differentiating intolerance from life-threatening reactions. Attempts at institution of coded entry of allergic reactions were met with limited success.

Antibiotic allergy alerts had the highest compliance rate. Physicians may be more acceptant of antibiotic allergy alerts because they have experience with true allergic reactions in this class, and therefore place more value and more trust in documented antibiotic allergies. While other drug classes may be less likely to cause true allergies, some patients do experience them. The innate distrust of all listed allergies in the class may contribute to inappropriate overrides. More specific allergy encoding with drug intolerances coded separately and excluded from interruptive allergy or cross-reactivity alerts might reduce inadvertent dismissal of appropriate warnings [28].

Most of our allergy alerts were triggered by narcotic allergies, or by cross-reactions of unknown significance [29, 30]. Physicians override cross-reaction alerts more often than exact allergy matches, and likely view the former with a healthy skepticism that is only enforced by daily false alarms. Our review panel did not remove allergic cross-reactions because cross-reactions represent a variable risk to each patient. Still, removal of cross-reaction alerts that outnumber exact allergy alerts by a factor of 10 and are almost always overridden or have little evidence base would likely improve allergy alert compliance.

This study has several limitations. We examined only medication alert override rates and associated metadata, over a short time span, within a single vendor’s clinical decision support system. We did not assess alert or override appropriateness, review adverse events, or examine behavior outside of the alert window, and could not determine whether alert fatigue or other factors increased the fraction of inappropriate overrides. Some physicians may override alerts, but later cancel their orders, leading to falsely high estimates of noncompliance with the alert recommendation. Our high baseline override rates may also have masked any difference in rates due to alert fatigue, or any other sources of variability between physicians.

6. Conclusions

Our analysis of medication order alerts shows that override rates remain as high as more than a decade ago, and before the Meaningful Use program. It is of great concern that these high override rates persist despite five years of local effort to improve the drug interaction database and reduce alert frequency according to best practices outlined in the literature. We demonstrated this lack of improvement within a popular commercial decision support system, similar to those used in many hospitals nationwide. With Meaningful Use mandating the use of drug-drug interaction alerts, our study results question the value of this requirement for certified EHRs. Further, our experience illustrates the substantial obstacles when trying to customize a commercial product to local needs.

Fortunately, the number of alerts per physician is already approaching what we believe to be a reasonable level. Variations in alert quantity among physicians have no measurable impact on override rates. Taken together, these findings suggest that the currently recommended methods to reduce alert quantity and improve relevance will not be enough to improve physician acceptance of interruptive alerts. CDS should be welcomed and embraced as helpful by providers, however, this is not the case today, and the absence of progress should receive national attention on the utility of such mandatory systems.