Optimizing Drug-Drug Interaction Alerts Using a Multidimensional Approach

DDI alerts were identified and refined by using an interdisciplinary advisory group, metrics analysis, and surveys of clinicians’ perceived value to optimize alerts.

Abstract

OBJECTIVES:

Excessive alerts are a common concern associated with clinical decision support systems that monitor drug-drug interactions (DDIs). To reduce the number of low-value interruptive DDI alerts at our hospital, we implemented an iterative, multidimensional quality improvement effort, which included an interdisciplinary advisory group, alert metrics, and measurement of perceived clinical value.

METHODS:

Alert data analysis indicated that DDIs were the most common interruptive medication alert. An interdisciplinary alert advisory group was formed to provide expert advice and oversight for alert refinement and ongoing review of alert data. Alert data were categorized into drug classes and analyzed to identify DDI alerts for refinement. Refinement strategies included alert suppression and modification of alerts to be contextually aware.

RESULTS:

On the basis of historical analysis of classified DDI alerts, 26 alert refinements were implemented, representing 47% of all alerts. Alert refinement efforts resulted in the following substantial decreases in the number of interruptive DDI alerts: 40% for all clinicians (22.9–14 per 100 orders) and as high as 82% for attending physicians (6.5–1.2 per 100 orders). Two patient safety events related to alert refinements were reported during the project period.

CONCLUSIONS:

Our quality improvement effort refined 47% of all DDI alerts that were firing during historical analysis, significantly reduced the number of DDI alerts in a 54-week period, and established a model for sustained alert refinements.

Electronic health records (EHRs) integrate clinical decision support (CDS) within computerized prescriber order entry (CPOE) systems to provide clinicians with intelligently filtered, person-specific information at appropriate times to improve health care delivery.¹ Active interruptive CDS presents unsolicited information and requires a clinician’s response to continue.² Excessive interruptive CDS alerts can lead to alert fatigue, possibly resulting in clinicians ignoring clinically relevant alerts.^3,4 Alert fatigue can also be affected by inferior interface design and limitations in knowledge bases.^5,6

To reduce the potential for alert fatigue, consensus groups recommend reducing the number of ineffective alerts by analyzing alert metrics and perceived satisfaction of alerts.^7,8 Current alert metrics that are easily retrievable from the EHR provide limited insight into the clinician’s perspective.⁹ Alert frequency (the number of times an alert is presented during a given period) cannot differentiate between clinically appropriate and inappropriate alert presentations. Alert override rate (the number of continued actions that generated an alert divided by the total number of alerts) contains both justified and unjustified overrides.⁷ Determining if an override is justified or unjustified requires detailed analysis of alert and patient data.^9,10 Recently, a drug-drug interaction (DDI) CDS workgroup recommended combining clinician feedback and perceptions of alert systems with current alert metrics to target alerts for deactivation and tracking the performance of alert system improvements.⁷

One mechanism for alert system improvements is the guided review of an interdisciplinary clinician panel,^11–16 which can evaluate DDI alert frequencies both individually^11,15 and by drug class.^12–14 Organizing DDI alerts into drug-class and class-class categories allows for a large number of individual DDI alerts to be evaluated for clinical effectiveness.^12–14 DDI alerts that are identified as having little clinical value can be completely suppressed or selectively filtered according to patient-specific factors by using contextual awareness.¹² One analysis of hospital-wide DDI alerts revealed that 25% of alerts could be improved with contextually aware filtering.¹⁷

Although systematically evaluating DDI alert satisfaction is recommended, only 1 known survey instrument has been developed and evaluated psychometrically.¹⁸ The developers of this tool recommended that it be used as part of comprehensive efforts to assess clinicians, including alert data and metrics.¹⁸ However, we did not find published studies in which this tool was used to guide DDI alert improvement efforts.

In this quality improvement (QI) report, we describe amultidimensional approach to improve DDI alert effectiveness at St. Jude Children’s Research Hospital (St. Jude), with the goal of reducing the frequency of DDI alerts per 100 medication orders by 20% in 1 year. Our efforts comprised evaluation by an interdisciplinary advisory group, alert metric analysis, and assessment of clinician perceptions of DDI alert value with a validated survey.

Methods

Setting

St. Jude is a 78-bed hospital with integrated outpatient clinics and treats children with cancer, blood disorders, and related life-threatening diseases. Since 2010, St. Jude has used a fully implemented EHR system with CPOE (Millennium; Cerner Corporation, North Kansas City, MO) for all aspects of inpatient and outpatient care.¹⁹

Context

The EHR system primarily generates DDI alerts using a commercial knowledge base (Cerner Multum, Denver, CO). During initial CPOE implementation in 2008, alert fatigue was considered, and only “major” and “major contraindicated” DDI alerts were presented to clinicians. Duplicate therapy alerts were not presented. When the EHR system detects a DDI, a pop-up window interrupts clinician workflow. To proceed, it must be acknowledged through an override (which requires an override reason) or acceptance (ie, removing the offending order; Supplemental Fig 3). With oversight from the Pharmacy and Therapeutics (P&T) Committee, the DDI alert database has been intermittently modified according to clinician recommendations, review of alert data, and literature reviews. All medication orders entered by a midlevel practitioner (eg, nurse practitioner [NP]) require an attending cosignature, and alerts are generated at order entry and at cosignature.

Project Design

The alert advisory group (AAG), which was established to provide oversight and guidance, met routinely to guide, improve, and review project results (Supplemental Information). To align with St. Jude’s quality and safety structure, the AAG reported routinely to the P&T Committee, which authorized the AAG to implement refinements and report them retrospectively (Supplemental Information).

The AAG identified new alert-refinement processes and opportunities over time. Monthly DDI alert database updates received from the vendor were evaluated for clinical value, especially to determine if alerts should be presented to clinicians. Also, the value of DDI alert presentation for each clinician role (eg, prescribing, dispensing and/or verifying, and attending cosignature) was evaluated.

Interventions

DDI alert refinements that were identified during the project included both suppression (ie, complete deactivation of an alert) and contextual-awareness filtering of alerts (ie, presenting alerts to clinicians that were based on patient-specific data within the EHR).⁸ Throughout the project, input was sought from clinicians about additional alerts for potential refinement. Refinement strategies were operationalized with custom alerts and tools and filters from our EHR vendor.

The administration of a validated satisfaction survey to clinicians who encountered interruptive DDI alerts provided a source of DDI alert refinements.¹⁸ Survey data were collected in the following 2 formats: in-person at departmental meetings and e-mail invitation. In both formats, project leaders engaged with respondents by explaining the project’s purpose and encouraging clinicians to send alert system–improvement recommendations.

Measures

To facilitate the analysis of a large number of DDI alerts, DDI alert medications were classified (when applicable) into class-class and drug-class categories of DDI alerts. The frequency of interruptive medication CDS alerts received by clinicians was determined by calculating the number of alerts per 100 medication orders. The override rate for DDI alerts was calculated by dividing the number of alerts that were overridden by the total number of alerts that were fired. DDI alerts were the most frequent type of interruptive medication alerts and selected as the initial focus of alert refinements.

As a balancing measure to detect any unintended consequences from alert refinements, we tracked patient safety event reports related to DDIs during the project. Clinicians were also encouraged to report any unexpected outcomes of alert modifications.

Analysis

The average alert frequency and alert override rate were calculated for each class-class and drug-class category, ranked, and evaluated for possible CDS refinements. Weekly DDI alert frequency was analyzed by using a statistical process control (SPC) chart (u-chart) that was created with SPC for Excel (version 5.0.1.6; SPC for Excel, Cypress, TX). Weekly DDI alert frequency was measured by dividing the number of DDI alerts per week by the number of 100 medication orders each week to generate an alert per 100 orders metric. Upper and lower control limits for the u-chart were calculated by using prerefinement data to detect a significant change in frequency due to refinement efforts or other special causes. The average center line was shifted when a special cause was detected after initiating improvement efforts. The software evaluated the following standard rules to identify special cause: 1 point beyond the control limits, 2 out of 3 consecutive points in the outer one-third section and within control limits, 4 out of 5 consecutive points in the middle one-third section and within control limits, ≥7 consecutive points on 1 side of the average, ≥7 consecutive points trending up or down, 8 consecutive points that are not contained within the inner one-third section and are within control limits, 15 consecutive points that are contained in the inner one-third section and within control limits, and 14 consecutive points alternating up and down.

The percentage of DDI alerts due to the requirement of an attending cosignature was calculated by dividing the number of DDI alerts that were received by midlevel practitioners and would require an attending cosignature by the total number of alerts.

The total number of patient safety event reports that were related to DDI alerts and any unexpected outcomes of alert modifications were calculated.

Ethical Considerations

This project was reviewed by the St. Jude Institutional Review Board and found to be a QI project.

Results

Alert Refinement

To identify alerts for potential refinement, the AAG analyzed the frequency of alerts from June 2, 2012, to June 2, 2015, and decided to target DDIs for refinement (Supplemental Fig 4). We sorted the 106 528 DDI alerts into 282 class-class or drug-class categories. Then, we selected the top 25 class-class or drug-class categories with high alert frequency and override rates for potential refinement. During the project, the average override rate for alerts was 93.9%. In Table 1 and Fig 1, we present alert-refinement strategies that were used for specific DDI alert categories and the refinement date. We implemented a total of 26 alert refinements during the project period, which represents 46.8% of all alerts that fired during the analysis period (Table 1). We used alert suppression for 16.5% of all DDI alerts that fired and introduced contextual awareness to refine 30.2% of all DDI alerts that fired during the time frame.

TABLE 1.

DDI Alert Refinements by the AAG During February 2015–February 2017

Category	Frequency, %	Override Rate, %	Refinement Date	Alert Refinement Strategy	AAG Clinical Rationale for Alert Refinement
Potassium and potassium-sparing diuretics	9.2	95.9	December 14, 2015	Alert only when previous potassium level was >4.8 mEq/L or when no potassium level has been obtained in 7 d.	Serum potassium monitored routinely. Risk of unnoticed hyperkalemia present when patient is not monitored or has high serum potassium at initiation.
Potassium ACEI and/or ARB	6.7	95.8	December 14, 2015	Alert only when previous potassium level was >4.8 mEq/L or when no potassium level has been obtained in 7 d.	Serum potassium monitored routinely. Risk of unnoticed hyperkalemia present when patient is not monitored or has high serum potassium at initiation.
Posaconazole and H2 receptor blocker or PPI	4.1	95.0	December 14, 2015	Alert only fires when posaconazole solution is ordered.	Absorption affected by gastric pH only for oral solution dosage form of posaconazole.
Potassium and anticholinergics	2.2	94.4	December 14, 2015	Alert only fires when anticholinergic is ordered with extended-release potassium.	Risk of gastric damage due to delayed gastric emptying of anticholinergics only for extended-release dosage forms of potassium.
Fluoroquinolones and steroids	1.2	95.9	December 14, 2015	Suppress	This extended a previous suppression of an individual DDI alert before the AAG formation to the class-class level.
Iohexol (intrathecal only) and lidocaine	1.1	90.8	December 14, 2015	Suppress (suppression of all Iohexol intrathecal DDI alert interactions that occurred in July 2016)	Iohexol is not administered via the intrathecal route in our patient population.
Methotrexate and P-glycoprotein inhibitors	0.7	98.3	December 14, 2015	Suppress	Methotrexate doses are monitored and adjusted via pharmacokinetic monitoring.
Diuretics and aminoglycosides	0.7	92.9	December 14, 2015	Suppress	Benefits of both medications outweigh risk of ototoxicity and nephrotoxicity. Patients are evaluated for both toxicity risks during evaluation before ordering.
NSAID and NSAID	3.4	90.1	March 9, 2016	Alert only fires for 2 inpatient NSAID orders.	Alert fired inappropriately for outpatient NSAID orders with inpatient ones. Alert only relevant when 2 active inpatient NSAID orders are present.
CNS depressant and CNS depressant	0.5	89.0	March 9, 2016	Suppress	Potential increased risk of CNS depression monitored closely in our patient population.
Methotrexate and PPI	3.6	90.3	August 26, 2016	Alert only fires with high-dose methotrexate (doses >300 mg/m2).	Elimination of methotrexate at high doses is impaired by PPIs.
5HT3 antagonist and serotonin modulator	6.3	95.3	September 12, 2016	Suppress	Majority of patient population receives serotonin antagonist. Low risk of serotonin syndrome that will be detected from inpatient and frequent outpatient clinic monitoring.
Midazolam and CYP3A4 inhibitor	1.6	94.4	September 12, 2016	Suppress on the basis of analysis of all CYP3A4 substrates and inhibitor interactions.	CYP3A4 inhibition clinically insignificant for this medication for our patient population.
Fentanyl and CYP3A4 inhibitor	0.4	97.8	September 12, 2016	Suppress on the basis of analysis of all CYP3A4 substrates and inhibitor interactions.	CYP3A4 inhibition clinically insignificant for this medication for our patient population.
Methadone and CYP3A4 inhibitor	0.3	94.3	September 12, 2016	Suppress on the basis of analysis of all CYP3A4 substrates and inhibitor interactions.	CYP3A4 inhibition clinically insignificant for this medication for our patient population.
Oxycodone and CYP3A4 inhibitor	0.3	92.5	September 12, 2016	Suppress on the basis of analysis of all CYP3A4 substrates and inhibitor interactions.	CYP3A4 inhibition clinically insignificant for this medication for our patient population.
Citalopram and CYP3A4 inhibitor	0.1	98.6	September 12, 2016	Suppress on the basis of analysis of all CYP3A4 substrates and inhibitor interactions.	CYP3A4 inhibition clinically insignificant for this medication for our patient population.
Salmeterol and CYP3A4 inhibitor	0.1	97.3	September 12, 2016	Suppress on the basis of analysis of all CYP3A4 substrates and inhibitor interactions.	CYP3A4 inhibition clinically insignificant for this medication for our patient population.
Escitalopram and CYP3A4 inhibitor	0.1	95.4	September 12, 2016	Suppress on the basis of analysis of all CYP3A4 substrates and inhibitor interactions.	CYP3A4 inhibition clinically insignificant for this medication for our patient population.
Itraconazole and CYP3A4 inhibitor	0.1	95.9	September 12, 2016	Suppress on the basis of analysis of all CYP3A4 substrates and inhibitor interactions.	CYP3A4 inhibition clinically insignificant for this medication for our patient population.
Oxycodone and CYP3A4 inhibitor	0.1	91.4	September 12, 2016	Suppress on the basis of analysis of all CYP3A4 substrates and inhibitor interactions.	CYP3A4 inhibition clinically insignificant for this medication for our patient population.
Fluticasone and CYP3A4 inhibitor	0.1	94.8	September 12, 2016	Suppress on the basis of analysis of all CYP3A4 substrates and inhibitor interactions.	CYP3A4 inhibition clinically insignificant for this medication for our patient population.
Olanzapine and benzodiazepines	1.5	96	November 21, 2016	Suppress	DDI occurs only with intramuscular formulations of both drugs. Olanzapine is not orderable via intramuscular route at our institution.
Deferasirox and NSAIDs	1.3	97	November 21, 2016	Suppress for all NSAIDs (previously on suppressed for ibuprofen).	This extended a previous suppression of an individual DDI alert before the AAG formation to the class-class level.
Tacrolimus and spironolactone	1	97	November 21, 2016	Alert only when previous potassium level was >4.8 mEq/L or when no potassium level has been obtained in 7 d.	Serum potassium monitored routinely. Risk of unnoticed hyperkalemia present when patient is not monitored or has high serum potassium at initiation.

None of the refinements above were detailed as DDIs that should always be active in pediatric EHRs.²⁰ ACEI, angiotensin-converting enzyme inhibitor; ARB, angiotensin II receptor blocker; CNS, central nervous system; CYP3A4, cytochrome P450 3A4; PPI, proton pump inhibitor.

FIGURE 1.

SPC u-chart of DDI alerts per 100 medication orders per week. Dates of AAG alert refinements are indicated by arrows, and individual refinements are detailed in Table 1. Upper control limit and lower control limit were determined by using weekly data before AAG refinement efforts (from February 1, 2015, to December 20, 2015). 5SL, 5 σ limit; −5SL, negative 5 σ limit.

After multiple alert refinements from December 2015 to September 2016, the SPC u-chart revealed a significant (40%) decrease from 22.9 to 13.7 DDI alerts per 100 medication orders per week (Fig 1). This decrease was sustained for 4 months after the end of the QI project (Fig 1). The override rate also decreased significantly during this time period (Supplemental Fig 5). During the project period, DDI alerts to clinicians at cosignature represented 69% of all DDI alerts encountered by physicians. The AAG approved the suppression of DDI alerts presented to physicians at cosignature, and the refinement was implemented in August 2016. We observed the following significant decreases in DDI alert frequency for each clinician role over the project period: 29% for pharmacists, 21% for NP and/or physician assistant, and 82% for attending physicians (Fig 2).

FIGURE 2.

Run chart used to compare DDI alert frequency by clinician role. Averages were compared before and after the final round of refinements for each role. Pharmacists decreased from 10.4 to 7.4 DDI alerts per 100 orders. NPs and physician assistants decreased from 3.4 to 2.7 DDI alerts per 100 orders. Attending physicians decreased from 6.5 to 1.2 DDI alerts per 100 orders. These decreases were statistically significant for each clinician role.

We monitored automatic vendor-provided updates to the DDI alert content database, which proved to be an important intervention. An update on May 18, 2016, led to an increase in DDI alert severity for a commonly used drug (sulfamethoxazole-trimethoprim) in our patient population and in multiple drugs and drug classes that can increase serum potassium. We found a significant increase in alerts to clinicians that resulted from this May 2016 update (Fig 1). The AAG did not agree with the knowledge base vendor’s rationale for increasing alert severity for St. Jude patients, and these alerts were suppressed by early August 2016.

Patient Safety Event Reports

During the project, 2 safety events related to DDI alert refinements were reported to the hospital’s event reporting system. First, pantoprazole was ordered and administered to a patient who was receiving high-dose methotrexate. The alert had been refined and functioned properly, but the alert text was ignored and overridden. We revised the alert text to more clearly explain that coadministration of these medications is contraindicated. Second, oral potassium was administered to a patient who was receiving olanzapine, which prompted an event report. Using a vendor tool, we refined this alert to only present with relevant potassium dosage forms, but the tool did not function. Thus, we created a new custom alert to provide an alert that would include all relevant dose forms of potassium. Finally, 1 event was informally communicated after the AAG approved suppressing DDI alerts in the vendor-supplied DDI database that fired when prescribing 2 nonsteroidal anti-inflammatory drugs (NSAIDs). It was recommended to create a new custom alert after observing a patient with active inpatient orders for 2 NSAIDs. Thus, we implemented a custom alert that interrupts clinicians when an inpatient NSAID order is placed for a patient with an existing active NSAID order.

Discussion

Our QI efforts reduced the weekly number of interruptive DDI alerts per 100 medication orders by 40% by both suppressing and modifying alerts, which exceeded our original goal of a 20% reduction in 1 year. We identified potential low-value alerts through alert analysis and interaction with clinicians when administering the satisfaction survey. We used these best practices to improve our hospital’s DDI alert system, and our project was also the first known effort to employ a construct-validated clinician survey on DDI alert quality.¹⁸ To our knowledge, our QI effort is the first to demonstrate significant reductions in DDI alert frequency by combining the following 3 unique methods: systematic alert data extraction and review, a multidisciplinary panel, and use of a validated survey instrument.

Alert fatigue can negate the potential patient safety benefits of interruptive CDS, and it is a widely discussed concern.^{3,4,7,9,10,14,21,22} Previous research has reported effective techniques for categorizing historical interruptive DDI alert data for critical review, organizing at the drug or drug category level, and calculating frequencies of alert occurrence.^11–16 Interdisciplinary panels can review an institution’s alert data and safely identify alerts to deactivate or modify.^7,8

Using a validated survey provided broad clinician insights and helped fulfill previous recommendations to use multiple sources of information to evaluate interruptive alert quality.^7,9,23,24 Although the response rate was too low to provide a meaningful representation of clinician alert satisfaction, the act of administering the survey in-person during staff meetings was valuable. By explaining the project objectives face-to-face and engaging clinicians with the audience response devices, we generated support for the project and created an opportunity for end users to offer novel ideas to improve the alert system.

The need for increased use of contextually aware filtering has been discussed repeatedly in previous research, and our efforts addressed these calls by including this technique in 6 of the 17 modified alerts.^{7,9,14,25–27} As previously reported, classifying DDI alerts by drug class increased the efficiency of our efforts and identified more DDI alerts for refinement than in our previous efforts to analyze metrics for individual DDI alerts, and it identified that only 25 alert groups were responsible for 46% of alert firings.^11–16,21 Refinement of these alert groups resulted in a significant decrease in the frequency of DDI alerts across multiple clinician roles (Fig 2). Importantly, our experience was consistent with previous efforts showing that refining a relatively small number of drug-drug and drug-class alerts can encompass a large percentage of all occurring DDI alerts.¹²

Limitations of our project included our balancing measure, the survey response rates, and applicability to other institutions. Although voluntary event reporting is used in similar efforts, it is not able to reliably and comprehensively capture all harm from medications, including DDIs.^15,28,29 The low response rates were due to poor response rates from the online response collection efforts. Future efforts to collect online responses should involve a broad awareness campaign and the use of multiple recruitment tactics, such as printed promotional materials, announcements during staff meetings, e-mailed invites from senior leadership, and possibly incentives for participation. Importantly, the engagement generated from discussing the project, the AAG, and administering the survey during these meetings was valuable in generating support and eliciting ideas to improve the alert system. Our comprehensive approach to DDI alert management is resource intensive and may not be replicable in institutions with more limited resources to devote to the project. To help others apply what we have learned, our AAG charter and guidance for replication and adaptation is provided in the supplement (Supplemental Information). Key ingredients to our success that may be replicated with fewer resources include development of a close partnership between clinical leaders and information services and the use of SPC to interpret alert data.

Another limitation (which is broader than our project) is the lack of alert metrics to easily identify potentially inappropriate DDI alerts.^23,24,30 Although the frequency and override rate can be easily calculated, their ability to reveal alert appropriateness without further clinician review is poor.^9,10 Alert frequency and override rate do not provide insight into an alert’s clinical use because clinicians may still value some alerts that occur frequently but are still overridden. Also, override rates can be falsely high in instances in which it is easier to override the DDI alert for a new medication and immediately discontinue the old medication than to acknowledge the alert, discontinue the old medication, and then reenter the new medication. Override rates and reasons for overriding can also vary across departments within a single institution and across alert types.^22,31,32 Metrics that can provide more information on appropriateness and value, such as alert adherence rate,^9,10 require increased clinician insight and cannot be easily obtained from an automated report. Other metrics have been recently introduced but need additional refinement.^23,24

Conclusions

We demonstrated that multiple alert-refinement methods within a single QI effort decreased DDI alert frequency. We created a sustainable model for future alert-refinement efforts at our hospital, which may be applied more broadly (Supplemental Information). Further research on easily obtainable alert metrics that provide a clear understanding of alert value and clinician responses will help to improve the efficiency of future alert-improvement efforts.

Glossary

AAG

alert advisory group

CDS

clinical decision support

CPOE

computerized prescriber order entry

DDI

drug-drug interaction

EHR

electronic health record

NP

nurse practitioner

NSAID

nonsteroidal anti-inflammatory drug

P&T

Pharmacy and Therapeutics

QI

quality improvement

SPC

statistical process control