Medication-related clinical decision support alert overrides in inpatients

Abstract

Objective

To define the types and numbers of inpatient clinical decision support alerts, measure the frequency with which they are overridden, and describe providers’ reasons for overriding them and the appropriateness of those reasons.

Materials and Methods

We conducted a cross-sectional study of medication-related clinical decision support alerts over a 3-year period at a 793-bed tertiary-care teaching institution. We measured the rate of alert overrides, the rate of overrides by alert type, the reasons cited for overrides, and the appropriateness of those reasons.

Results

Overall, 73.3% of patient allergy, drug-drug interaction, and duplicate drug alerts were overridden, though the rate of overrides varied by alert type (P < .0001). About 60% of overrides were appropriate, and that proportion also varied by alert type (P < .0001). Few overrides of renal- (2.2%) or age-based (26.4%) medication substitutions were appropriate, while most duplicate drug (98%), patient allergy (96.5%), and formulary substitution (82.5%) alerts were appropriate.

Discussion

Despite warnings of potential significant harm, certain categories of alert overrides were inappropriate >75% of the time. The vast majority of duplicate drug, patient allergy, and formulary substitution alerts were appropriate, suggesting that these categories of alerts might be good targets for refinement to reduce alert fatigue.

Conclusion

Almost three-quarters of alerts were overridden, and 40% of the overrides were not appropriate. Future research should optimize alert types and frequencies to increase their clinical relevance, reducing alert fatigue so that important alerts are not inappropriately overridden.

BACKGROUND AND SIGNIFICANCE

Medication errors have been highlighted as an important contributor to patient harm, and error rates range from 5% to 20%, depending on the patient care area.^1–11 When harm is caused by a medication, it is called an adverse drug event (ADE), regardless of whether the harm is preventable. ADEs associated with errors are considered preventable ADEs. Medication-related clinical decision support (CDS) has been shown to improve patient safety and quality of care by preventing medication errors.^12–17 While key system functionalities, such as electronic problem lists, medication lists, patient allergy lists, and laboratory test results, must be implemented and used to gain these improvements,^18,19 prior work suggests that their use has been highly variable.²⁰ Providers can typically accept or override CDS alerts. Overrides can be appropriate or inappropriate, and can occur when providers reject or cancel an alert and proceed with the action under question. Evaluating physicians’ responses to CDS alerts is key to achieving their reported benefits to patient safety and quality of care.

Since 2011, the federal government has provided financial incentives to hospitals for achieving the core objectives of meaningful use of electronic health records (EHRs).²¹ Meaningful use involves not only using certified EHRs, but using them as a tool to improve quality, safety, and efficiency, engage patients and family members, improve care coordination and public health, and maintain privacy. The core objectives for the meaningful use incentives include the use of a computerized physician order entry system with medication-related CDS that has, at a minimum, enabled alerts for patient allergies, drug-drug interactions, and formulary substitutions.²¹

When the threshold for alerting is too low, alert overload can cause providers to miss or override important alerts.^22–27 While tiering of alerts with hard stops for only the most severe alerts can help with alert fatigue,²⁸ there have been reports of unintended consequences of hard-stop alerts, such as delays in treatment.²⁹ Improved knowledge around alert and override rates, the reasons providers cite for overriding alerts, and the appropriateness of those reasons are critical to improving alert delivery, local and federal policies around meaningful use, and criteria for certification of EHRs.

Alert override rates have been reported from 50% to >90% in inpatients^22,30–39 and 33% to >90% in outpatients.^23,25,40 While our group has previously reported on the appropriateness of overrides in the outpatient setting,⁴¹ and drug allergy alerts⁴² and formulary substitutions⁴³ in the inpatient setting, the literature on the appropriateness of other types of alert overrides, such as drug-drug interactions, duplicate drugs, and age- and renal-based dose adjustments, in the inpatient setting is sparse.^44,45 The aims of this study were to define the types and numbers of clinical decision support alerts delivered in the inpatient setting, determine the rate at which they are overridden, and describe providers’ reasons for overriding them and the appropriateness of those reasons.

MATERIALS AND METHODS

Study site

This study included data from Brigham and Women’s Hospital (Boston, Massachusetts, USA), a 793-bed tertiary-care teaching affiliate of Harvard Medical School. At the time of this study, all electronic orders were entered through the Brigham Integrated Clinical Information System, a comprehensive homegrown electronic medical record. This system provides clinical, administrative, and financial computing functions and includes a patient-specific CDS that is usually presented to providers in the form of medication alerts at the time of prescribing. The CDS provides 3 levels of alerts: level 1 alerts involve hard stops, where the user has to either discontinue the order or discontinue the interacting medication; level 2 alerts are interruptive and require the user to give a reason if he or she decides to override the alert; and level 3 alerts are information only and do not require any user action. Our study included level 2 alerts triggered by drug allergies, drug-drug interactions, renal- and age-based medication substitutions, duplicate medications, and nonformulary medications. The renal-based medication substitution alerts were based on a calculation of the patient’s creatinine clearance using weight, height, age, sex, and most recent creatinine level.

Study design

After obtaining institutional review board approval, we conducted a cross-sectional observational study of alerts generated over a 3-year period from 2009 to 2012. We obtained a count of alert overrides and the providers’ coded reasons for overrides at the time when they ordered the medications. Provider types included attending physicians, house staff, and nonphysicians with prescribing authority, such as nurse practitioners. We included the following 6 types of alerts provided by our CDS system: patient allergy alerts, drug-drug interaction alerts, duplicate drug alerts, age-based recommendations, renal recommendations, and formulary substitutions.

Appropriateness criteria

To develop criteria for the appropriateness of overrides for each alert type in the inpatient setting, our group used chart and literature review to iteratively modify our previously used and validated appropriateness framework⁴¹ as necessary until consensus was reached on all criteria. In general, alert overrides were considered appropriate if the reason indicated by the provider was allowable in our framework and could be verified on chart review. For example, if a physician ordered a medication and an allergy alert was displayed, the possibly appropriate override reasons included “patient has taken previously without reaction,” “low risk cross sensitivity,” or “will monitor,” based on data showing that monitoring is beneficial to patients.^46,47 Our team verified the listed reasons for overrides. These verifications of appropriateness involved extensive chart review, including, for example, checking for required monitoring per our criteria, checking notes from outside physicians to look for previously tolerated drug combinations, and checking nurses’ notes and medication administration records to look for medications that were held or refused. A group of 6 trained clinicians – 2 physicians (PEB, TE) and 4 pharmacists (DLS, MGA, QLH, OD) – then analyzed random samples of at least 200 alert overrides per alert category to assess whether the overrides were appropriate. We assumed an override rate of 10% and aimed for a target precision of ±5% (95% CI: 5-15%), which suggested reviewing 150 overrides, and we rounded up to a sample size of 200, for greater precision and to enable subsidiary analyses. Each sample of overrides was analyzed for appropriateness independently by 2 clinicians, and interrater reliability was calculated for override appropriateness. Disagreements were resolved by discussion and consensus among the group of clinician reviewers.

Outcomes

Our primary outcome was the rate of alert overrides in the inpatient setting. Our secondary outcomes were the reasons cited for overrides when the medications were ordered and the appropriateness of the overrides.

Analysis

We compared alert override rates and the appropriateness of overrides by alert type. Comparisons are presented as counts with percentages, and P-values were calculated using the chi-square test. Counts, percentages, and chi-square analyses were conducted using Microsoft Excel 2011 v14.5.5 (Redmond, WA, USA). Interrater reliability analyses were conducted using SAS 9.3 (SAS Institute, Cary, NC, USA).

RESULTS

A total of 337 921 alerts were overridden during our study period. We were able to obtain data to calculate the override rate for 156 283 (46.2%) of the overrides, including patient allergy alerts (81.9% overridden), drug-drug interaction alerts (68.2% overridden), and duplicate medication alerts (51.9% overridden), as shown in Table 1.

Table 1.

Breakdown of alert overrides

	Total alerts	Alert overrides
Alert type	N (%)	N (%)	Three most common reasons for override
Patient allergy	131 615 (61.7)	107 812 (81.9)	Patient took previously without allergic reaction (57.4%) Physician aware (17.2%) Low risk of cross-sensitivity, will monitor (12.3%)
Drug-drug interaction	37 579 (17.6)	25 616 (68.2)	Will monitor as recommended (54.0%) Will adjust dose as recommended (15.5%) Patient had already tolerated combination (15.4%)
Duplicate drug	44 059 (20.7)	22,855 (51.9)	Combination therapy indicated (55.7%) Onetime dose (33.0%) Not duplicate therapy (10.2%)
Total	213 253 (100.0)	156 283 (73.3)

While the overall override rate for these alert categories was 73.3%, override rates varied significantly by alert type (P < .0001). We did not have access to the number of nonformulary and age- and renal-based dose adjustment alerts, so these were excluded from our calculation of override rate. We also excluded drug-drug interaction alerts related to appropriate prophylactic anticoagulant use in patients with epidurals. While these alerts may contribute to alert fatigue, they were overwhelmingly overridden.

We evaluated the appropriateness of random samples of alert overrides for each of the 6 alert types represented in our sample of 337 921 overrides: duplicate medication alerts (98.0% appropriate), drug allergy alerts (96.5% appropriate), nonformulary medication alerts (82.5% appropriate), drug-drug interaction alerts (62.0% appropriate), age-based medication substitution alerts (26.4% appropriate), and renal-based medication substitution alerts (2.2% appropriate). The appropriateness rates are shown in Table 2. While an overall average of 61.3% (median 72.3%) of overrides were appropriate, the percentage of overrides that were appropriate varied significantly by alert type (P < .0001). Interrater reliability for assessment of alert appropriateness was very good (κ = 0.96, 95% CI: 0.95-0.97).

Table 2.

Appropriateness of overrides by alert type

Alert type	Sample size reviewed for appropriateness	Appropriate override (%)
Duplicate drug	200	98.0
Patient allergy	200	96.5
Formulary substitution	206	82.5
Drug-drug interaction	250	62.0
Age-based suggestion	208	26.4
Renal suggestion	1033	2.2
Total	2097	61.3

Providers cited many different reasons for overriding the alerts; the most common overall reason was that the patient had previously tolerated the medication. The reasons cited for alert overrides varied by alert type, as did the most common reasons cited for overrides in each category (Table 1). The most common reason given for overriding drug-drug interaction alerts was that the provider “will monitor as recommended” (54.0%). In these cases, the provider overrode the alert and ordered the medication, stating that he or she would monitor for the expected reaction. If the patient was monitored as stated, for example, with an electrocardiogram for prolonged QT interval or INR for medications that interact with Coumadin, then the override was considered appropriate. If we could not verify in the patient’s chart that the patient was monitored as indicated in the override reason, then the override was considered inappropriate. The most common reason cited for overriding duplicate medication alerts was “combination therapy is indicated” (55.7%).

DISCUSSION

We found that almost three-quarters of patient allergy, drug-drug interaction, and duplicate drug alerts in our sample were overridden, and the override rate varied significantly by alert type. About 60% of overrides were appropriate, and the rates of appropriateness also varied by alert type. For example, renal- and age-based medication substitution alert overrides had low appropriateness rates, while duplicate drug, patient allergy, and formulary substitution alerts were largely appropriately overridden, suggesting that the numbers of these alerts could be reduced in the inpatient setting to prevent alert fatigue.

The overall override rate in this study is higher than the rate we reported in 2014 for our ambulatory practices, where about 50% of alerts were overridden.⁴¹ There are several possible reasons for the higher override rate in the inpatient setting. First, the threshold for alerting may be lower in the inpatient setting than the outpatient setting, leading to a greater number of appropriate overrides. Second, the computerized physician order entry system with CDS was implemented in the inpatient setting in 1993, 9 years prior to the outpatient setting. The numbers of alerts and overrides tend to increase over time,^24,30 which may have led to the higher override rates in our inpatient population. Third, the inpatient patient population may be more medically complex than the outpatient population, thus generating more alerts and alert fatigue. Finally, providers may feel more comfortable overriding alerts in the inpatient setting, where patients are more closely monitored than in outpatient practices.

Our results are consistent with existing literature, which reports inpatient override rates ranging from 50% to >90%.^22,30–38 Also, the proportion of overrides that were appropriate is similar to what is reported in the literature.⁴⁴ Dekarske and colleagues⁴⁴ found that appropriateness increased from 50–60% to 70–80% when customized lists and/or free-text override reasons were allowed.

Of the renal-based medication substitution alerts that were overridden, only 2% were appropriately overridden. For age-based medication substitution alerts, about one-quarter of the overrides were appropriate. This is consistent with literature showing very low appropriateness rates for overrides of renal-based medication substitution alerts in other settings.⁴¹ There is some evidence that a small number of providers and a small number of drugs may account for a large fraction of overrides,^48–50 suggesting that a focused intervention targeting primarily these providers and medications has the potential to improve medication safety.

Our study has several limitations. First, our results are based on data from a large, academic, tertiary health care center with an internally developed clinical decision support system, and our findings may not be generalizable to nonteaching hospitals or those with other clinical decision support systems. While allowable customization and critical values that generate alerts may differ between institutions with various decision support systems, nearly all available systems use similar alert categories, such as patient allergies, drug-drug interactions, duplicate drugs, and renal-based substitutions, and many of the lessons learned will be applicable within these domains, regardless of who developed the system. Second, our outcome was the rate of inappropriate overrides, not patient harm. While evaluation of patient harm would be helpful, the sample size required would be extremely large, because observed patient harm is rare. Errors or inappropriate overrides that do not lead to observed patient harm are near misses with the potential for harm, and it is important to evaluate these events regardless of whether they lead to patient harm. Third, we were not able to obtain the number of alerts for the following alert categories: formulary substitution, age-based medication substitution, and renal-based medication substitution. Thus, our overall override rate excludes these categories, as shown in Table 1. Fourth, some decisions regarding the appropriateness of overrides may be equivocal. For example, the use of a relatively contraindicated drug in the elderly may be appropriate in certain instances. However, our interrater reliability for assessment of alert appropriateness was very good, suggesting that a large number of incorrect assessments is unlikely. Finally, we did not evaluate the clustering of CDS overrides or their appropriateness by provider. This is an important area for future research and interventions aimed at targeting providers with inappropriately high override rates.

CONCLUSION

In summary, providers overrode almost three-quarters of the alerts in our sample, and 40% of the overrides were not appropriate. While some of the alerts, such as formulary substitutions, had little clinical impact, others had greater implications for patient safety, such as age- and renal-based medication substitution alerts. Despite warning of potential significant patient harm, these categories of alert overrides were not appropriate >75% of the time. While unnecessary alerts should simply be turned off, future research should also be done to optimize alert types and frequencies in order to increase their clinical relevance, with targeted interventions aimed at reducing alert fatigue so that important alerts are not inappropriately overridden. Importantly, override rates should be reevaluated for measurable reductions after targeted interventions to reduce alert fatigue are implemented.

Funding

This work was supported by Agency for Healthcare Research and Quality (AHRQ, Rockville, MD, USA) grant numbers 1U19HS021094-01 and 1K08HS024764-01. The content is solely the responsibility of the authors and does not necessarily represent the official views of the AHRQ. PEB was supported by the Swiss National Science Foundation, grant number P2BSP3_148619. OD was supported by the Fondation Saint-Luc (Brussels, Belgium).

Competing Interests

All authors declare no financial relationships with any organizations that might have an interest in the submitted work in the previous 3 years, except DWB as described here. DWB is a co-inventor on Patent No. 6029138 held by Brigham and Women’s Hospital on the use of decision support software for medical management, licensed to the Medicalis Corporation. He holds a minority equity position in the privately held company Medicalis, which develops web-based decision support for radiology test ordering. He serves on the board for S.E.A. Medical Systems, which makes intravenous pump technology. He consults for EarlySense, which makes patient safety monitoring systems. He receives equity and cash compensation from QPID, Inc., a company focused on intelligence systems for electronic health records. He receives cash compensation from CDI (Negev), Ltd., which is a not-for-profit incubator for health IT startups. He receives equity from Enelgy, which makes software to support evidence-based clinical decisions. He receives equity from ValeraHealth, which makes software to help patients with chronic diseases. He receives equity from Intensix, which makes software to support clinical decision-making in intensive care. He receives equity from MDClone, which takes clinical data and produces deidentified versions of it. DWB’s financial interests have been reviewed by Brigham and Women’s Hospital and Partners HealthCare in accordance with their institutional policies.

Contributors

KCN wrote the statistical analysis plan, conducted the statistical analysis, and drafted and revised the paper. She is a guarantor. DLS designed data-collection tools, reviewed overrides for appropriateness, and assisted with data analysis, drafting, and revising the paper. MGA, PEB, QLH, OD, TE, ERS, JMF, and MS reviewed overrides for appropriateness and assisted with drafting and revising the paper. SPS designed data-collection tools and assisted with drafting and revising the paper. NM entered, cleaned, and analyzed the data and assisted with drafting and revising the paper. STH assisted with data analysis and drafting and revising the paper. DWB designed data-collection tools, monitored data collection, and revised and approved the paper. All authors gave their approval for the final version to be published.