Evaluation of Clinical Relevance of Drug–Drug Interaction Alerts Prior to Implementation

Abstract

Introduction  Drug–drug interaction (DDI) alerts are often implemented in the hospital computerized provider order entry (CPOE) systems with limited evaluation. This increases the risk of prescribers experiencing too many irrelevant alerts, resulting in alert fatigue. In this study, we aimed to evaluate clinical relevance of alerts prior to implementation in CPOE using two common approaches: compendia and expert panel review.

Methods  After generating a list of hypothetical DDI alerts, that is, alerts that would have been triggered if DDI alerts were operational in the CPOE, we calculated the agreement between multiple drug interaction compendia with regards to the severity of these alerts. A subset of DDI alerts ( n  = 13), with associated patient information, were presented to an expert panel to reach a consensus on whether each alert should be included in the CPOE.

Results  There was poor agreement between compendia in their classifications of DDI severity (Krippendorff's α: 0.03; 95% confidence interval: –0.07 to 0.14). Only 10% of DDI alerts were classed as severe by all compendia. On the other hand, the panel reached consensus on 12 of the 13 alerts that were presented to them regarding whether they should be included in the CPOE.

Conclusion  Using an expert panel and allowing them to discuss their views openly likely resulted in high agreement on what alerts should be included in a CPOE system. Presenting alerts in the context of patient cases allowed panelists to identify the conditions under which alerts were clinically relevant. The poor agreement between compendia suggests that this methodology may not be ideal for the evaluation of DDI alerts. Performing preimplementation review of DDI alerts before they are enabled provides an opportunity to minimize the risk of alert fatigue before prescribers are exposed to false-positive alerts.

Background and Significance

Drug–drug interactions (DDIs) are an important cause of preventable morbidity and mortality in the hospital setting. 1 A DDI occurs when medications that are taken concurrently interfere with the anticipated effect of one another, potentially resulting in an adverse drug event. 2 3 To minimize the risk of DDI-related adverse drug events in hospital patients, DDI alerts are embedded into computerized provider order entry (CPOE) systems. 4

There is limited research demonstrating that DDI alerts are effective in reducing DDIs and subsequent patient harm. 5 A persistent and widespread problem appears to be that DDI alerts are frequently ignored and overridden by prescribers. 6 7 8 This is likely due to the large number of false-positive DDI alerts presented to prescribers, resulting in alert fatigue. Research has shown that prescribers could experience between 20 and 145 DDI alerts per 1,000 prescriptions, 9 10 and that even 20 DDI alerts per 1,000 medication orders can result in alert fatigue. 10 Thus, reducing the quantity of alerts alone is not sufficient to ensure alerts are read and acted upon. The clinical relevance of alert information should also be evaluated to help guide alert inclusion.

To determine which alerts to include and exclude in CPOE requires evaluation of their clinical relevance. Typically, this involves a review of alerts which are frequently overridden by users. 11 Alerts that rarely result in a medication order being changed are viewed to be of limited value. However, this methodology requires DDI alerts to be operational in a CPOE before being evaluated.

Interestingly, evaluation of DDI alerts is rarely done prior to implementation in a system. 12 13 Instead, DDI alerts, often part of the “out-of-the-box” vendor functionality of CPOE, are implemented to meet Meaningful use or accreditation requirements. 14 15 However, evaluation and the subsequent removal of low-quality alerts following implementation of DDI alerts has been shown to be a complex and challenging task. 11 16 17 For example, in one study, frequently overridden DDI alerts were reviewed by an expert panel, but the panel could not agree that any of the 86 DDI alerts identified should be removed from the system. 11

Two approaches available to organizations wishing to evaluate their alerts prior to implementation in a CPOE are drug compendia and expert panel review. A common approach involves comparing the agreement between multiple drug interaction compendia on the severity of detected interactions. 18 19 High agreement between compendia, regarding the severity of DDI alerts, could indicate that the alert is clinically relevant. 18 19

In other studies, clinical experts have been tasked with reviewing alerts, with the aim of excluding irrelevant alerts from the system. 20 21 However, this approach is sometimes problematic, as poor agreement between panelists regarding the clinical relevance of alerts has been found. 11 17 22 23 This may be because panelists are often required to rate alerts independently, with no opportunity to discuss their decisions, and alerts are often presented in the absence of the clinical context in which they are triggered, possibly making it difficult for panelists to conceptualize clinical relevance. In previous studies, panelists have been required to indicate whether an alert is relevant or not, and rarely given the opportunity to identify the contexts in which they believe a specific DDI alert to be clinically relevant. 11 Only triggering DDI alerts in the presence of particular context factors has been suggested as a method of reducing false-positive alerts. 24 For example, if a particular DDI is harmful in newborns but not adults, such as the coadministration of ceftriaxone and calcium, then the patient's age is a context factor that could be used to dictate whether to trigger the DDI alert. 24

At our study hospital, a decision was made to evaluate DDI alerts prior to implementation in a CPOE. An audit of “hypothetical” DDI alert numbers revealed that DDI alert rate would be high (147 DDI alerts per 1,000 medication orders) and would likely contribute to the development of alert fatigue. Thus, this provided a strong rationale to evaluate DDI alert quality in terms of clinical relevance before implementation into a CPOE.

Objective

The objective of this study was to trial two commonly used methods (compendia review and an expert panel) to assess clinical relevance of DDI alerts before implementation. The results of the study would inform which DDI alerts to implement and under what context factors they should trigger.

Methods

Study Setting

This study was conducted in a 379-bed public teaching hospital in Australia. The hospital uses the CPOE MedChart (referred to herein as MedChart). MedChart allows electronic prescribing, review, and administration of medications. 25 Several computerized alerts are operational in the system including allergy alerts, therapeutic duplication alerts, and local messages (e.g., reminders about antibiotic restrictions). At the time of the study, DDI alerts were not enabled.

This study was approved by the hospital's human research ethics committee.

Identification of DDI Alerts for Testing Clinical Relevance

To generate DDI alerts, medications for a sample of patients ( n  = 78) were entered into MedChart's training environment and all hypothetical DDI alerts were noted. The sample consisted of all patients that were discharged over two consecutive days. The patients were from seven specialties including: neurology, gastroenterology, infectious diseases, geriatrics, oncology, cardiology, and cardiothoracic surgery.

The DDI compendium utilized by MedChart is MIMS, and as the hospital planned to implement only DDI alerts of the highest severity, only severe DDI alerts were subsequently assessed for quality. 26 In MIMS, severe DDI alerts warn against interactions between medications that may be life-threatening or cause permanent damage. 26

Assessing Quality of DDI Alerts Using Compendia

The severe DDI alerts were entered into three other drug interaction compendia: Stockley's Drug Interactions, Micromedex, and YouScript. These compendia were selected based on high usage and reputation. 9 27 28 29

The severity rating of each drug pair that triggered a DDI alert was reviewed in the three compendia (see Appendix A ). Agreement between the three compendia and MIMS was assessed using Krippendorff's α. An α of 1 indicates perfect agreement between compendia, a value of 0 no agreement, and a value of –1 indicates an inverse agreement between compendia. 30 IBM SPSS Statistics Version 23 was used for analysis.

Appendix A. Severity ratings from multiple drug compendia.

Comparative ranking	MIMS drug interaction checker	Micromedex	YouScript	Stockley's Drug Interactions
SEVERE	Severe: The interaction between these medications may be life-threatening or may cause permanent damage. These medications are not usually used concurrently; medical intervention may be required	Contraindicated: Drug pairs are contraindicated for concurrent use	Contraindication: This drug has an interaction that is contraindicated in the product insert due to the potential for a severe or life-threatening reaction. This combination should not be administered together	Severe: Interactions that may totally incapacitate a patient or result in permanent detrimental effect. Can be life-threatening
		Major: Potentially life-threatening interactions and/or require medical intervention to minimize serious adverse effects	Major Clinical Impact: This drug has an interaction that may result in severe clinical effects or large changes in drug levels. The risks of the interaction generally outweigh the benefits of prescribing the drug
MODERATE	Moderate: These medications may interact resulting in the potential deterioration of the patient's condition. The patient should be monitored for the possible manifestations of the interaction. Medical intervention or a change in therapy may be required	Moderate: The interaction may result in exacerbation of the patient's condition and/or require an alteration in therapy	Moderate Clinical Impact : This drug has an interaction that may result in substantial clinical effects or moderate changes in drug levels. Changes in therapy, such as making dose adjustments or prescribing alternatives, may be warranted	Moderate: For interactions that could cause considerable distress or partial incapacitation of patients. Unlikely to be life-threatening
MINOR	Minor: Clinical effects of the interaction are limited and may be bothersome but would not usually require a major change to therapy. The patient should be monitored for the possible manifestations of the interaction	Minor: The interaction would have limited clinical effects. Manifestations may include an increase in the frequency or severity of the side effects but generally would not require a major alteration in therapy	Minor Clinical Impact: This drug has an interaction that may result in minor clinical effects or small changes in drug levels. The benefits of prescribing the drug generally outweigh the risks of the interaction. Major changes in therapy are not expected, although minor dose adjustments may be appropriate	Mild: For interactions that are unlikely to result in an effect or that if an effect was to occur it would be mild and unlikely to incapacitate the majority of patients
	Caution: The interaction may occur based on the mechanism of action of the coadministered medicines. Be alert for increased or decreased effect, depending on the combination of medicines		Minimal Clinical Impact: This drug may be associated with clinically insignificant and/or favorable interactions. No change in therapy is necessary

Note: Compendia used nonstandardized severity terminology. This table shows how these inconsistent labels were grouped for comparison across compendia.

Assessing Quality of DDI Alerts Using an Expert Panel

The panel consisted of five health care professionals: two clinical pharmacologists, two senior clinical pharmacists, and one geriatrician. Six patient cases were randomly selected and presented to the panel. These cases would have triggered 13 different hypothetical severe DDI alerts if enabled in a CPOE. Short case presentations were delivered to the panel, including information on medical history and progress during admission.

Panel members were asked to review the patient's medication charts and were provided with supporting information including: the relevant hypothetical DDI alert, how frequently the alert would fire in our sample of 78 patients if enabled, the literature summaries from the four compendia, as well as the alerts severity rankings from each compendium. For each of the DDI alerts, panelists individually decided whether the alert should be included in MedChart. Then, all panelists presented their independent view as a prelude to a general open discussion. During the discussion, the panelists explained how and why they determined whether an alert should or should not be included in a CPOE and attempted to form a consensus. The panel was encouraged to discuss context factors that related to the patient (i.e., age), the medications (i.e., dose and route of administration), and the organization (i.e., whether the prescriber was a junior medical officer). Consensus was defined as a minimum of four of the five panelists reaching agreement, after open discussion. Panelists were informed that it was not necessary for a consensus opinion to be reached.

Results

Alert Quality Using Compendia

A total of 147 DDI alerts were triggered by 45 unique DDI drug pairs in the 78 hospital patients. These 45 drug pairs were entered into Stockley's Drug Interactions, Micromedex, and YouScript. In total, 8 of the 45 unique drug pairs (18%) were ranked as severe in all four compendia. These 8 drug pairs accounted for 15 of the 147 (10%) alerts. Statistical analysis confirmed that there was poor agreement between drug compendia on the severity classification of DDIs. The Krippendorff's α was 0.03 with a 95% confidence interval of –0.07 to 0.14.

The six drug pairs that would have triggered the highest number of DDI alerts, using the MIMS interaction module, are shown in Table 1 . Oxycodone and oxycodone/naloxone was the drug pair that triggered the most alerts in our sample, but this interaction was only classed as a severe DDI by MIMS.

Table 1. Top six drug pairs resulting in DDI alerts.

Drug pair	Number of DDI alerts triggered (% of total)	Compendia that ranked DDI as severe
Oxycodone and naloxone/oxycodone	13 (9)	1/4
Amiodarone and furosemide	10 (7)	2/4
Amiodarone and tacrolimus	9 (6)	3/4
Morphine and naloxone/oxycodone	9 (6)	3/4
Itraconazole and tacrolimus	7 (5)	3/4
Amiodarone and warfarin	7 (5)	2/4

Abbreviation: DDI, drug–drug interaction.

Alert Quality Using an Expert Panel

The panelists reached consensus after open discussion on 12 of the 13 alerts triggered in the patient cases ( Table 2 ). Nine were recommended for inclusion in the system, but for four, it was suggested that these should trigger only in certain clinical contexts. These context factors are shown in Table 2 .

Table 2. Findings from the expert panel.

Drug pair	Panel's response after open discussion	Key context factors identified
Furosemide and gentamicin	Include
Amiodarone and domperidone	Include
Amiodarone and warfarin	Include
Ondansetron and domperidone	Include
Amiodarone and ondansetron	Include
Heparin and salicylates	Include in certain clinical contexts	• IF patient is older than 66 y old THEN trigger • IF renally impaired THEN trigger • IF hyperkalemic THEN trigger
Methadone and ondansetron	Include in certain clinical contexts	• IF no ECG performed THEN trigger • IF dose (ondansetron) > 16 mg THEN trigger • IF dose (methadone) > 80 mg THEN trigger • IF route of ondansetron is parenteral THEN trigger
Enoxaparin and warfarin	Include in certain clinical contexts	• IF INR is elevated THEN trigger • IF junior medical officer prescribing THEN trigger • IF renal impairment THEN trigger
Temazepam and olanzapine	Include in certain clinical contexts	• IF patient is older than 75 THEN trigger; • IF more than 5 h between administrations THEN do not trigger • IF STAT dose THEN do not trigger
Lorazepam and olanzapine	No consensus
Amiodarone and bisoprolol	Exclude	• The alert is warning for the desired effect (i.e., bradycardia)
Aspirin and selective serotonin receptor inhibitors	Exclude	• Well-known DDI with likely low significance in the average person • Common combination that is taken without adverse drug events
Ramipril and spironolactone	Exclude	• Indications outweigh the risks of DDI

Abbreviations: DDI, drug–drug Interaction; ECG, electrocardiogram; INR, international normalized ratio.

Discussion

This study used two common methods to evaluate the clinical relevance of DDI alerts before implementation into a CPOE. In assessing the quality of alerts using compendia, only a small number of hypothetical alerts (10%) were classed as severe by all four compendia. The very poor agreement found between compendia with respect to classifications of severity, brings into question the usefulness of compendia alone to determine which alerts should be included or excluded in a CPOE. This poor agreement between compendia is echoed in other studies 18 31 32 and is due to different, nontransparent DDI prediction models used to classify the severity of interactions. 20 The poor agreement is concerning, as it is common for prescribers to only consult one compendium when reviewing risk of adverse drug events.

When a subset of DDI alerts were presented to an expert panel in the context of individual patient cases and panelists were given the opportunity to discuss their views, we found high agreement between panelists with respect to whether alerts should be included in a CPOE. Of the 13 alerts presented, panelists reached a consensus on 12 of the alerts. Panelists agreed that three should be excluded and four included only in certain contexts. Previous studies have reported moderate to poor agreement between panelists when determining the importance of DDI alerts. 11 22 23 The high agreement observed in our study could be attributed to the methodology adopted, which differed from the approach taken in other studies. 22 23 First, the panelists were encouraged to openly discuss the reasoning for their recommendations, while other studies have utilized an independent and noncollaborative approach to the assessment of alert relevance. 22 23 Second, alerts in this study were presented in the context of particular patient cases. Consideration of the patient and their clinical context is a crucial element in determining clinical relevance. The panel members were able to consider the context factors impacting on the probability DDI resulting in an adverse effect. 24 For example, the panel agreed that the increased risk of bleeding due to the coadministration of enoxaparin and warfarin was only clinically relevant when the patient's international normalized ratio (INR) (a laboratory indicator for bleeding risk) was elevated. Incorporating INR information into the alerting system would ensure that when triggered, the DDI alert would be relevant to the prescriber's decision.

Research has shown that only triggering alerts in the presence of relevant context factors could reduce overall alert burden significantly. 24 Our current evaluation demonstrates that presenting alerts in conjunction with patient context and allowing for open discussion, not only facilitated agreement between panelists on clinical relevance of DDI alerts but allowed the identification of context factors for improving alert specificity.

This study had several limitations. The expert panel was limited by time and was only able to review a sample of 13 alerts that triggered in 6 patient cases. Level of experience, profession, and expertise of panel members may have impacted results; however, we attempted to minimize these influences by including a range of different professionals (i.e., pharmacists, clinical pharmacologists, and geriatricians) and providing each panelist with an opportunity to share their view. Finally, we reassured the panelists that reaching a consensus was not necessary.

Although utilizing an expert panel is more resource-intensive and time-consuming than inputting medications into compendia and reviewing agreement, we suggest presenting a large sample of alerts to the expert panel before implementation. This would ensure that alerts likely to cause the highest burden to prescribers are clinically relevant. An evaluation of alert burden and alert relevance postimplementation would reinforce the effectiveness of this approach to minimize the risk of alert fatigue.

Conclusion

In this study, we assessed the clinical relevance of DDI alerts in terms of clinical relevance prior to their implementation in a CPOE. We found drug compendia to be unreliable in their classification of DDI alerts, but the expert panel was highly consistent in their assessments of clinical relevance and in their identification of context factors, most likely because alerts were presented in the context of specific patient cases, and experts were permitted to share their clinical knowledge and discuss any differences in opinion. Although more resource-intensive, we recommend expert panel review as an effective approach for assessing clinical relevance of DDI alerts prior to alert implementation, to minimize the risk of alert fatigue before prescribers are exposed to alerts.

Clinical Relevance Statement

Hospitals across the world are utilizing CPOE systems and, commonly, enable extra functionalities such as drug–drug interaction alerts. However, due to overexposure, alert fatigue is a common problem. This article identifies a method that may reduce alert rate by improving clinical relevance of alerts before they are implemented in a CPOE.

Multiple Choice Question

What did this study do differently, that potentially resulted in panelists having higher agreement on whether DDI alerts should be included or excluded from the CPOE?

1. Blinded the panelists to each other's answers to keep views unbiased.

2. Blinded the panelists to the clinical context of the patient to ensure that the results were generalizable.

3. Allowed the panelists to discuss their reasoning openly with each other with respect to the clinical relevance and important context factors with each DDI alert.

4. Used only panelists from one specialty so that they would reach consensus.

Correct Answer: The correct answer is option c. Allowing for open discussion gave way to fruitful debate between the panelists and was successful at helping them come to agreement on 12 of the 13 DDI alerts they reviewed.

Funding Statement

Funding This study was funded by National Health and Medical Research Council Program Grant APP1054146.