Improving the specificity of drug-drug interaction alerts: Can it be done?

Abstract

Purpose

Inaccurate and nonspecific medication alerts contribute to high override rates, alert fatigue, and ultimately patient harm. Drug-drug interaction (DDI) alerts often fail to account for factors that could reduce risk; further, drugs that trigger alerts are often inconsistently grouped into value sets. Toward improving the specificity of DDI alerts, the objectives of this study were to (1) highlight the inconsistency of drug value sets for triggering DDI alerts and (2) demonstrate a method of classifying factors that can be used to modify the risk of harm from a DDI.

Methods

This was a proof-of-concept study focused on 15 well-known DDIs. Using 3 drug interaction references, we extracted 2 drug value sets and any available order- and patient-related factors for each DDI. Fleiss’ kappa was used to measure the consistency of value sets among references. Risk-modifying factors were classified as order parameters (eg, route and dose) or patient characteristics (eg, comorbidities and laboratory results).

Results

Seventeen value sets (56%) had nonsignificant agreement. Agreement among the remaining 13 value sets was on average moderate. Thirty-three factors that could reduce risk in 14 of 15 DDIs (93%) were identified. Most risk-modifying factors (67%) were classified as order parameters.

Conclusion

This study demonstrates the importance of increasing the consistency of drug value sets that trigger DDI alerts and how alert specificity and usefulness can be improved with risk-modifying factors obtained from drug references. It may be difficult to operationalize certain factors to reduce unnecessary alerts; however, factors can be used to support decisions by providing contextual information.

Key Points

• Consistency of drugs among drug interactions value sets is limited.

• Incorporating contextual factors, such as patient characteristics and order parameters, may improve drug interaction alerts.

• The findings suggest that a central authority and new technology may be needed to maintain drug value sets and that expanding clinical decision support logic to include contextual factors will be crucial to improving drug interaction alert specificity.

Clinical decision support (CDS) has the potential to prevent harmful drug-drug interactions (DDIs); however, greater than 90% of DDI alerts are overridden (ie, ignored or dismissed).¹ Despite clinicians deeming most DDI alerts as “not clinically significant,” more than half of alert overrides are inappropriate.^2,3 Inappropriate overrides result in approximately 6 times more adverse drug events compared to appropriate overrides.⁴ Numerous irrelevant DDI alerts can desensitize clinicians, whereby important patient safety alerts are overridden, a phenomenon called alert fatigue.⁵ Excessive override rates and ubiquitous adverse drug events are evidence that conventional medication alerts are not meeting clinician information needs or CDS implementer expectations.^6-9

Conventional DDI alerts are triggered or invoked by the presence of 2 interacting drugs. While increasing alert specificity or usefulness is possible by incorporating patient factors (eg, comorbidities and laboratory results) and drug order factors (eg, route and dose) in CDS logic,^10,11 alerts largely do not account for contextual factors that can eliminate the need of an alert or affect the risk of certain DDIs.¹² For instance, a drug interaction between the HMG Co-A reductase inhibitor atorvastatin and the protease inhibitor fosamprenavir may not be clinically significant if the dose of atorvastatin is below 20 mg per day. In addition to the lack of contextual factors in CDS logic, triggering of DDI alerts depends on 2 separate drug groups (ie, value sets).¹³ Value sets have been characterized as lists of codes and terms that define a clinical concept.¹⁴ Drug knowledge bases frequently define interactions in terms of value sets based on certain criteria, such as mechanism of action for a monoamine oxidase inhibitor (MAOI) value set. Evidence for a drug interaction is often limited to a few drugs in a value set,¹⁵ with the potential for an interaction inferred to additional drugs.¹⁶ Generalizing evidence to members of a drug class is an unstandardized approach that can lead to variation among value sets that trigger DDI alerts, resulting in reduced CDS effectiveness and inconsistent outcomes across institutions.¹⁷ For example, fluvastatin and rosuvastatin, which would both be included in an HMG Co-A reductase inhibitor value set, are metabolized by the cytochrome P450 (CYP) 2C9 isozyme and will likely not be clinically relevant to an interaction between an HMG Co-A reductase inhibitor and a CYP3A4 inhibitor. While CDS implementors have some ability to control CDS logic and value sets, institutions often rely on third-party systems and drug information knowledge bases (eg, MedKnowledge [First DataBank, Inc.], Multum [Cerner Corporation], and Medi-Span [Wolters Kluwer Health]) to generate DDI alerts.⁶

CDS implementers have employed multiple strategies to improve DDI alerts. Individual alert override rates have been used to identify and filter irrelevant or ineffective alerts.¹⁸ Another strategy is to create a list of low-priority DDIs and downgrade alerts to noninterruptive mechanisms.¹⁵ Additionally, severity thresholds have been increased so that only high-priority alerts are shown to clinicians.^15,19 While the process by which drug alerts are filtered or modified is often accomplished with user consensus,^20-24 certain institutions do not have the ability to customize CDS logic or do not have access to terminologists and subject matter expertise to create and maintain value sets^25,26; therefore, these approaches to improving DDI alerts are less explored. Toward understanding the need for curated value sets and how contextual factors can be used to increase the specificity and usefulness of DDI alerts, the objectives of this study were to (1) highlight the inconsistency of drug value sets for triggering DDI alerts and (2) demonstrate a method of classifying factors that can be used to modify the risk of harm from a DDI.

Methods

As a proof-of-concept study, we focused on 15 high-priority DDIs identified by the Office of the National Coordinator (ONC) for Health Information Technology (Table 1).²⁷ We analyzed DDI information in 3 references: Lexicomp (UpToDate, Inc.), Micromedex (IBM Corporation), and Hansten and Horn’s The Top 100 Drug Interactions.²⁷ Drugs in each interacting value set and factors that can affect the risk or severity of the DDI were extracted. Drug interaction references provide a list of drug names (ie, value sets) for each DDI (ie, drug A and drug B), and we mapped DDIs among references to the ONC list. As such, for value sets with a single drug A (eg, atazanavir), multiple interacting drugs were often listed in each reference. The appendix provides a complete list of drugs for each value set across references. Contextual factors were extracted from the narrative monograph and classified as either order parameters or patient characteristics.

Table 1.

Office of the National Coordinator List of High-Priority Drug-Drug Interactions

	Drug A	Drug B
1.	Amphetamine & Derivatives	MAOIs
2.	Atazanavir	PPIs
3.	Febuxostat	Azathioprine & Mercaptopurine
4.	SSRIs	MAOIs
5.	Irinotecan	Strong CYP3A4 Inhibitors
6.	Narcotic Analgesics	MAOIsa
7.	Tricyclic Antidepressants	MAOIs
8.	High-risk QT Prolonging Agents	High-risk QT Prolonging Agents
9.	Ramelteon	Strong CYP1A2 Inhibitors
10.	Strong CYP3A4 Inducers	Protease Inhibitors
11.	HMG Co-A Reductase Inhibitors	CYP3A4 & Protease Inhibitors
12.	CYP3A4 & Protease Inhibitors	Ergot Alkaloids & Derivatives
13.	Tizanidine	CYP1A2 Inhibitors
14.	Tranylcypromine	Procarbazine
15.	Triptans	MAOIs

Abbreviations: MAOI, monoamine oxidase inhibitor; SSRI, selective serotonin reuptake inhibitor; CYP3A4, cytochrome P450 isozyme 3A4; CYP1A2, cytochrome P450 isozyme 1A2; PPI, proton pump inhibitor.

Drugs.

Items on the ONC list of high-priority DDIs were defined in terms of individual drugs (eg, irinotecan), drug classes (eg, selective serotonin reuptake inhibitors), or mechanism of action (eg, strong CYP3A4 inhibitors).²⁸ We measured the consistency of drugs in each value set among the ONC list and the 3 drug interaction references. The value set consistency was calculated using Fleiss’ kappa, which is used for measuring nominal scale agreement among many raters or, for the purpose of this analysis, drug interaction references. Measure categories were as follows: almost perfect (0.81-1.00), substantial (0.61-0.80), moderate (0.41-0.60), fair (0.21-0.40), and slight (0.0-0.2).^29,30 We chose Fleiss’ kappa due to familiarity with Cohen’s kappa, a method to measure agreement between 2 raters.

Factors.

Drug monographs for each DDI combination were manually extracted from the 3 references. Factors that affect DDI risk or severity were annotated by a single author (T.R.) with pharmacotherapy and DDI experience. Factor categories were initially derived iteratively with a bottom-up approach across the extracted monographs. Collation of derived categories aligned with the Fast Healthcare Interoperability Resources (FHIR) standard (Health Level Seven International [HL7], Ann Arbor, MI; all FHIR resources discussed here are trademarked by HL7).³¹ A second clinician with DDI and data standards expertise (A.R.) reviewed the resulting resources and categories. Factors in the extracted monographs were first annotated according to order or patient categories. Then, order and patient factors were classified as timing, dose, and route and as condition, observation, and medication, respectively.

FHIR resources.

MedicationRequest is a FHIR resource that is used for inpatient and outpatient settings to order or request medications. The resource includes elements for timing, dose, and route of medication orders. Three FHIR “event” resources (Condition, Observation, and Medication) relate to patient characteristics. Timing factors are relevant when time between the administration of 2 interacting drugs can modify the risk of DDI occurrence. Dose factors are relevant when the dose of either interacting drug affects the potential DDI harm severity. Route of administration is important when the route can modify the risk of a DDI, such as lack of systemic absorption from topical, ophthalmic, or otic agents. Condition factors include patient comorbidities and acute problems that may affect the severity of DDIs. Observation factors are any relevant measurements (eg, laboratory test results and vital signs) that modify the risk or severity of a DDI. Medication factors include other medications the patient is taking that may increase or decrease the severity or risk of a DDI, as well as medications in a specific value set that pose less or more interaction risk. We tracked the number of drug interaction references that provided a similar factor in the DDI monograph. Table 2 provides an example of the monograph and annotations for the SSRIs/MAOIs DDI.

Table 2.

Example of Drug Monograph Extraction and Annotation for SSRI/MAOI Interaction

Source	Annotation
Lexicomp (UpToDate, Inc.)
“The washout period should probably be at least 1-2 weeks (2 weeks for vilazodone, 3 weeks for vortioxetine, 5 weeks for fluoxetine), depending on the half-life of the agent being discontinued. In general, furazolidone, linezolid, and procarbazine will pose less risk than isocarboxazid, phenelzine, or tranylcypromine. Caution is still advised. Selegiline administered in high oral dosages (e.g., more than 10 mg/day of tablet/capsule; more than 2.5 mg/day of orally disintegrating tablet) or transdermally will exhibit nonselective MAOI activity. MAOI Exceptions Linezolid; Methylene Blue; Tedizolid”	Timing: The washout period should probably be at least 1-2 weeks Dose: Selegiline administered in high oral dosages Route: Transdermally, will exhibit nonselective MAOI activity Medication: Furazolidone, linezolid, and procarbazine will pose less risk than isocarboxazid, phenelzine, or tranylcypromine and MAOI exceptions (linezolid, methylene blue, and tedizolid)
Micromedex (IBM Corporation)
“At least 14 days (preferably 18-20) should elapse after stopping an MAOI before starting a serotonergic drug. At least 5 weeks should after stopping fluoxetine before starting an MAOI. If possible use an alternative to the serotonergic drug in patients on linezolid or MAO-B inhibitors. Tedizolid does not appear to be a MAOI to a clinically important degree, so is unlikely to increase the risk of serotonin syndrome. Also, depending on the antibiogram, one could consider alternative antibiotics such as vancomycin or telavancin for linezolid”	Timing: At least 14 days should elapse after stopping an MAOI Medication: Tedizolid does not appear to be an MAOI to a clinically important degree
Hansten and Horn’s The Top 100 Drug Interactions27
“Wait at least 14 days after discontinuing an MAOI intended to treat psychiatric disorders before initiating fluoxetine. Wait at least 5 weeks after discontinuing fluoxetine before initiating therapy with an MAOI intended to treat psychiatric disorders”	Timing: Wait at least 14 days after discontinuing an MAOI

Results

Overall, the 15 high-priority DDIs on the ONC list were identified in the 3 drug interaction references. A discrepancy was that one reference separated the Tranylcypromine/Procarbazine and Triptans/MAOI DDIs into separate value sets. For instance, certain triptans were included in the broader serotonergic value set. For these DDIs, the consistency was calculated among the 2 other drug references and the ONC list.

Drugs.

The 15 DDIs encompassed 30 drug value sets and 682 drugs. The mean number of drugs per value set was 23 (median, 13). Of the 23-drug average per value set, an average of 3.8 drugs were the same across references (Figure 1). Table 3 provides the total number of drugs in each value set, the calculated value set agreement, and a listing of drugs in value sets for which there was perfect consensus among references. For instance, the Amphetamine & Derivatives value set included 17 unique drugs and had a significant Fleiss’ kappa of 0.16 (P = 0.04); amphetamine, dextroamphetamine, lisdexamfetamine, and methamphetamine were identified in the ONC list and the 3 drug interaction references. Seventeen value sets (56%) had nonsignificant agreement across the drug interaction references. Further, of the significant value sets, the average Fleiss’ kappa was moderate (0.45; standard deviation [SD], 0.33). Value sets with perfect agreement (Fleiss’ kappa = 1) included Irinotecan, Tizanidine, and Azathioprine & Mercaptopurine. Two value sets had moderate agreement (Febuxostat, P = 0.03; and PPIs, P < 0.001), 4 had fair agreement (MAOIs [Table 3, DDI 1], P = 0.006; MAOIs [Table 3, DDI 4], P = 0.006; MAOIs [Table 3, DDI 7], P = 0.001; and Protease Inhibitors, P < 0.001), and 4 had slight agreement (Amphetamine & Derivatives, P = 0.04; SSRIs, P = 0.008; strong CYP1A2 Inhibitors, P = 0.002; and MAOIs [Table 3, DDI 6], P = 0.02).

Figure 1.

Depiction of value set inconsistency across drug references.

Table 3.

Numbers of Medications and Classes in Value Sets, Value Set Consistency (κ), and Individual Drugs With Perfect Agreement Among References

DDI	Drug A	Drug B
1	Amphetamine & Derivatives (n = 17, κ = 0.16*) Amphetamine, dextroamphetamine, lisdexamfetamine, methamphetamine	MAOIs (n = 12, κ = 0.24*) Isocarboxazid, phenelzine, selegiline, tranylcypromine
2	Atazanavir (n = 6, κ = 0.11) Atazanavir	PPIs (n = 11, κ = 0.49*) Esomeprazole, dexlansoprazole, lansoprazole, omeprazole, pantoprazole, rabeprazole
3	Febuxostat (n = 2, κ = 0.47*) Febuxostat	Azathioprine & Mercaptopurine (n = 2, κ = 1.00*) Azathioprine, mercaptopurine
4	SSRIs (n = 23, κ = 0.17*) Citalopram, escitalopram, fluoxetine, fluvoxamine, paroxetine, sertraline	MAOIs (n = 12, κ = 0.24*) Isocarboxazid, phenelzine, selegiline, tranylcypromine
5	Irinotecan (n = 1, κ = 1.00*) Irinotecan	Strong CYP3A4 Inhibitors (n = 37, κ = 0.16*) Clarithromycin, indinavir, itraconazole, ketoconazole, nelfinavir, ritonavir, saquinavir, telithromycin, voriconazole
6	Narcotic Analgesics (n = 43, κ = –0.01) Fentanyl, meperidine, methadone, tapentadol, tramadol	MAOIs (n = 13, κ = 0.20*) Isocarboxazid, phenelzine, selegiline, tranylcypromine
7	Tricyclic Antidepressants (n = 12, κ = 0.06) Clomipramine, imipramine	MAOIs (n = 12, κ = 0.29*) Isocarboxazid, phenelzine, selegiline, tranylcypromine
8	High-risk QT Prolonging Agents (n = 114, κ = –0.06) Arsenic trioxide, disopyramide, dofetilide, ibutilide, procainamide, quinidine, sotalol, thioridazine	High-risk QT Prolonging Agents (n = 114, κ = –0.06) Arsenic trioxide, disopyramide, dofetilide, ibutilide, procainamide, quinidine, sotalol, thioridazine
9	Ramelteon (n = 37, κ = –0.11) Ramelteon	Strong CYP1A2 Inhibitors (n = 16, κ = –0.06) Ciprofloxacin
10	Strong CYP3A4 Inducers (n = 22, κ = –0.05) Carbamazepine, rifampin	Protease Inhibitors (n = 22, κ = 0.39*) Atazanavir, darunavir, fosamprenavir, indinavir, lopinavir, nelfinavir, ritonavir, saquinavir, tipranavir
11	HMG Co-A reductase inhibitors (n = 8, κ = –0.16) Lovastatin, simvastatin	CYP3A4 and protease inhibitors (n = 41, κ = 0.08) Indinavir, nefazodone, nelfinavir, ritonavir, saquinavir
12	CYP3A4 & Protease Inhibitors (n = 34, κ = –0.23) Indinavir, nelfinavir, ritonavir, saquinavir	Ergot alkaloids & derivatives (n = 7, κ = 0.07) Dihydroergotamine, ergotamine
13	Tizanidine (n = 1, κ = 1.00*) Tizanidine	CYP1A2 Inhibitors (n = 18, κ = –0.05) Ciprofloxacin
14	Tranylcypromine (n = 13, κ = 0.11) Isocarboxazid, phenelzine, rasagiline, safinamide, tranylcypromine, selegiline	Procarbazine (n = 13, κ = 0.11) Isocarboxazid, phenelzine, rasagiline, safinamide, tranylcypromine, selegiline
15	Triptans (n = 7, κ = 0.14) Sumatriptan, rizatriptan, zolmitriptan	MAOIs (n = 12, κ = –0.12) Isocarboxazid, phenelzine, tranylcypromine

Abbreviations: CYP1A2, cytochrome P450 isozyme 1A2; CYP3A4, cytochrome P450 isozyme 3A4; MAOI, monoamine oxidase inhibitor; PPI, proton pump inhibitor; SSRI, selective serotonin reuptake inhibitor. The symbol κ denotes Fleiss’ kappa statistic.

^a P < 0.05.

Factors.

From 14 of the 15 DDIs (93%), a total of 33 contextual factors were extracted (Table 4). The DDI with no identified factors was CYP3A4 & Protease Inhibitors/Ergot Alkaloids. Most factors (22 of 33, or 67%) were order related, with timing factors extracted for 10 of 15 DDIs (67%). Furthermore, timing factors were consistent across references (ie, 3 of 3 or 2 of 2 sources were in agreement) for 8 of the 10 DDIs (80%). Most timing factors were associated with MAOIs; the references provided recommendations to wait at least 14 days after stopping an MAOI before starting an interacting medication. Dose factors were available for 9 DDIs (60%). Compared to timing, dose factors were less consistent across the references. Two of the 9 DDIs (22%) had dose considerations that were consistent across the references. Most dose recommendations were nonspecific increases or decreases when initiating a medication that changes metabolism. Route of administration factors were identified for 3 of 15 DDIs (20%). The route factors were specific to the transdermal administration of the MAOI selegiline.

Table 4.

Order and Patient Factors by Potential DDI Across Drug Interaction Databases^a

		Order Factors			Patient Factors
DDI	Drug A/Drug B	Timing	Dose	Route	Condition	Observation	Medication
1	Amphetamine & Derivatives/MAOIs	3/3
2	Atazanavir/PPIs	3/3	3/3		2/3
3	Febuxostat/Azathioprine & Mercaptopurine		2/3
4	SSRIs/MAOIs	3/3	1/3	1/3			2/3
5	Irinotecan/Strong CYP3A4 inhibitors	3/3					1/3
6	Narcotic Analgesics/MAOIs	3/3		1/3			2/3
7	Tricyclic Antidepressants/MAOIs	3/3	1/3	1/3			3/3
8	High-risk QT Prolonging Agents	1/3	1/3		3/3	1/3	1/3
9	Ramelteon/Strong CYP1A2 Inhibitors						1/3
10	Strong CYP3A4 Inducers/ protease inhibitors	1/3	3/3
11	HMG Co-A Reductase Inhibitors/CYP3A4 and protease inhibitors		2/3				2/3
12	CYP3A4 and protease inhibitors/ergot alkaloidsb
13	Tizanidine/CYP1A2 inhibitors		1/3
14	Tranylcypromine/procarbazine	2/2
15	Triptans/MAOIs	2/2	1/2				1/2
	Total instances of agreement	10	9	3	2	1	8

Abbreviations: MAOI, monoamine oxidase inhibitor; PPI, proton pump inhibitor; SSRI, selective serotonin reuptake inhibitor; CYP3A4, cytochrome P450 isozyme 3A4; CYP1A2, cytochrome P450 isozyme 1A2.

^aData are fraction of evaluated references in agreement (eg, 2/3 indicates a factor was found in 2 of 3 references).

^bNo factors identified.

Patient factors were available with 9 DDIs (60%). Most medication factors were specific drug exclusions from the value set due to low risk of the interaction. High-risk QT-prolonging agents had patient considerations in all 3 categories; however, only the condition factor (eg, comorbidities) was consistent across all 3 drug interaction references. The condition factor for the Atazanavir/PPIs DDI included treatment experience (ie, atazanavir treatment experienced vs naïve), age, sex, and heart disease. Observation factors included laboratory measures (eg, hypokalemia). The medication factor for High-Risk QT Prolonging Agents was related to CYP3A4 substrates. These substrates increase the blood concentration of certain QT-prolonging agents and was an additive risk for prolonging heart rate–corrected QT (QTc) intervals.

Discussion

This study demonstrates the importance of increasing the consistency of drug value sets that trigger DDI alerts and how alert specificity and usefulness can be improved with risk-modifying factors. Despite the recognition and familiarity of the chosen DDIs, it was surprising that less than half (43%) of the 30 drug value sets had a significant consistency among drug references and that the average agreement was moderate (Table 3). Further, most DDIs (93%) had at least 1 potential risk-modifying factor and several (40%) had more than 2 factors (Table 4). Incorporating contextual factors in CDS logic can not only improve the specificity of alerts but also make alerts more useful by providing contextual information about DDI risk factors. A feasible approach to address risk-modifying factors and curated value sets may be through emerging health information technology standards such as HL7’s FHIR, Clinical Quality Language (CQL), and CDS Hooks.^32-36 CDS implementers can use these standards and web-based technology to bridge limited electronic health record functionality with CDS logic that incorporates contextual factors and external value sets.

The low consistency of drug value sets aligns what others have found regarding variability of CDS among institutions. Fung et al²⁴ identified large differences in the number and agreement of drug pairs across 3 drug knowledge bases (ie, First DataBank, Micromedex, and Multum) that are used to generate CDS alerts. Among 8.6 million unique drug pairs, only 5% of them were found in all the 3 knowledge bases. Cornu et al³⁷ found that while high-priority DDI alerts were generally available across knowledge bases, severity levels (eg, “moderate” and “severe”) varied considerably. Severity levels contribute to whether certain alerts are active and the type of alert that was shown to clinicians.²² Often severity levels are preset by knowledge vendors, but healthcare institutions can modify these settings. Our analysis complements previous studies by providing agreement of drugs within value sets and contextual factors that can be used to reduce false-positive alerts or make alerts that are more clinically relevant.

While the findings suggest that alert specificity can be improved by adding CDS logic to account for factors related to the drug order or patient characteristics, there are caveats to operationalizing contextual factors with DDI alerts. Not all DDIs are amenable to modifying factors. Certain DDIs pose the same risk across patients and medication order parameters. Empiric evidence of DDIs is limited. The evaluated DDIs were not selected based on the potential for factors to modify the risk or severity of the DDI. Other DDIs may be more or less amenable to increasing alert specificity. Further, most factors we identified were nonspecific conditions or exposure thresholds. These factors may not be granular enough to filter DDI alerts; however, the factors may be useful for clinicians to determine DDI risk. For example, heart disease is specified as a patient factor for QTc-prolonging interactions. Would this factor be operationalized as heart failure, atrial fibrillation, or a previous myocardial infarction? Or perhaps all three? It is unclear how this factor could be used to filter an alert, but it could be used to present patient-specific information to clinicians. Finally, DDI contextual factors lack a standardized taxonomy to support implementation in CDS logic.

Limitations.

This study has limitations. The selected DDIs, while identified as high-priority DDIs by ONC, may not be representative of DDIs across institutions. Additionally, we did not search for contextual factors in drug knowledge bases that are used for electronic health record alerts, as these knowledge bases are not easily referenceable. Finally, we did not empirically test how factors and curated value sets would increase alert specificity.

Conclusion

The goal of this proof-of-concept analysis was to better understand the need for curated value sets that are used in triggering DDI alerts and demonstrate how contextual factors can be used to increase the specificity and usefulness of DDI alerts. The findings highlight that the consistency of drug value sets is limited for a list of well-known DDIs. Encouraging, however, was the potential to use contextual factors in CDS logic. While many identified factors may not provide granular information to eliminate certain DDI alerts, the factors can provide useful information for clinicians to decide whether to override an alert. These findings can generalize to other DDIs, suggesting that a central authority and new technology may be needed to maintain drug value sets and that expanding CDS logic will be crucial to improve DDI alert specificity.

Appendix—Drugs included in the value sets

DDI	Drug A	Drug B
1.	Amphetamine & Derivatives	MAOIs
	amphetamine, dextroamphetamine, lisdexamefetamine, methamphetamine, benzphetamine, atomoxetine, diethylpropion, methylphenidate, pseudoephedrine, dexmethylphenidate, dopamine, ephedrine, isometheptene, mazindol, metaraminol, phenylephrine, tapentadol	isocarboxazid, phenelzine, selegiline, tranylcypromine, procarbazine, rasagiline, safinamide, methylene blue, syrian rue, moclobemide, linezolid, furazolidone
2.	Atazanavir	PPIs
	atazanavir, fosamprenavir, indinavir, ledipasvir, nelfinavir, rilpivirine	esomeprazole, dexlansoprazole, lansoprazole, omeprazole, pantoprazole, rabeprazole, revaprazan, cimetidine, famotidine, nizatidine, ranitidine
3.	Febuxostat	Azathioprine & Mercaptopurine
	febuxostat, allopurinol	azathioprine, mercaptopurine
4.	SSRIs	MAOIs
	citalopram, escitalopram, fluoxetine, fluvoxamine, paroxetine, sertraline, desvenlafaxine, duloxetine, milnacipran, venlafaxine, vilazodone, vorioxetine, dapoxetine, fentanyl, levomilnacipran, meperidine, methadone, tapentadol, tetrabenazine, tramadol, trazodone, cyclobenzaprine, dextromethorphan	isocarboxazid, phenelzine, selegiline, tranylcypromine, procarbazine, rasagiline, safinamide, methylene blue, linezolid, syrian rue, moclobemide, furazolidone
5.	Irinotecan	Strong CYP3A4 Inhibitors
	irinotecan	clarithromycin, indinavir, itraconazole, ketoconazole, nelfinavir, ritonavir, saquinavir, telithromycin, voriconazole, boceprevir, cobicistat, darunavir, nefazodone, posaconazole, telaprevir, amiodarone, amprenavir, diltiazem, erythromycin, fluconazole, idelalisib, verapamil, mifepristone, ceritinib, ciprofloxacin, cyclosporine, dasabuvir, delavirdine, dronedarone, imatinib, tipranavir, lapatinib, lomitapide, ombitasvir, paritaprevir, quinupristin, troleandomycin
6.	Narcotic Analgesics	MAOIs
	fentanyl, meperidine, methadone, tapentadol, tramadol, alfentanil, benzhydrocodone, buprenorphine, codeine, hydrocodone, nalbuphine, oxycodone, oxymorphone, remifentanil, sufentanil, butorphanol, dextromethorphan, dihydrocodeine, heroin, hydromorphone, levorphanol, meptazinol, morphine, opium, paregoric, pentazocine, citalopram, cyclobenzaprine, desvenlafaxine, dextromethorphan, duloxetine, escitalopram, fluvoxamine, levomilnacipran, levomilnacipran, milnacipran, paroxetine, sertraline, tetrabenazine, trazodone, venlafaxine, vilazodone, vortioxetine	isocarboxazid, phenelzine, selegiline, tranylcypromine, procarbazine, rasagiline, safinamide, methylene blue, linezolid, furazolidone, moclobemide, syrian rue, tedizolid
7.	Tricyclic Antidepressants	MAOIs
	clomipramine, imipramine, amitriptyline, amoxapine, desipramine, doxepin, nortriptyline, protriptyline, trimipramine, dosulepin, lofepramine, melitracen	isocarboxazide, phenelzine, selegiline, tranylcypromine, procarbazine, rasagiline, safinamide, syrian rue, moclobemide, methylene blue, linezolid, furazolidone
8.	High-risk QT Prolonging Agents	High-risk QT Prolonging Agents
	arsenic trioxide, disopyramide, dofetilide, ibutilide, procainamide, quinidine, sotalol, thioridazine, amiodarone, bepridil, flecainide, mesoridazine, pimozide, terfenadine, vandetanib, anagrelide, astemizole, bedaquiline, chloroquine, chlorpromazine, cilostazol, ciprofloxacin, citalopram, cisapride, donepezil, dronedarone, droperidol, erythromycin, escitalopram, fluconazole, haloperidol, halofantrine, lapatinib, levomethadyl, levofloxacin, methadone, moxifloxacin, ondansetron, oxaliplatin, pentamidine, quinine, sparfloxain, abiraterone acetate, acetylcholine, ajmaline, azithromycin, aripiprazole, atazanavir, azithromycin, aliskiren, clarithromycin, cocaine, ceritinib, celecoxib, clofazimine, clozapine, delamanid, domperidone, desflurane, dronedarone, dolasetron, efavirenz, fluoxetine, foscarnet, ganciclovir, gatifloxacin, grepafloxacin, hydroxychloroquine, hydroxyzine, ibogaine, ivabradine, lenvatinib, ketoconazole, lansoprazole, lumefantrine, lopinavir, levomepromazine, lomefloxacin, milrinone, nilotinib, nelfinavir, norfloxacin, olanzapine, omeprazole, papaverine, panobinostat, pazopanib, probucol, perphenazine, posaconazole, propafenone, propofol, roxithromycin, sulpiride, saquinavir, sertraline, solifenacin, sulfamethoxazole/trimethoprim, sumatriptan, sunitinib, toremifene, terlipressin, terodiline, tacrolimus, technetium tc 99m tetrofosmin, telitromycin, thiothixene, tramadol, trazodone, trifluoperazine, vernakalant, vasopressin, voriconazole, n-acetylprocainamide	arsenic trioxide, disopyramide, dofetilide, ibutilide, procainamide, quinidine, sotalol, thioridazine, amiodarone, bepridil, flecainide, mesoridazine, pimozide, terfenadine, vandetanib, anagrelide, astemizole, bedaquiline, chloroquine, chlorpromazine, cilostazol, ciprofloxacin, citalopram, cisapride, donepezil, dronedarone, droperidol, erythromycin, escitalopram, fluconazole, haloperidol, halofantrine, lapatinib, levomethadyl, levofloxacin, methadone, moxifloxacin, ondansetron, oxaliplatin, pentamidine, quinine, sparfloxain, abiraterone acetate, acetylcholine, ajmaline, azithromycin, aripiprazole, atazanavir, azithromycin, aliskiren, clarithromycin, cocaine, ceritinib, celecoxib, clofazimine, clozapine, delamanid, domperidone, desflurane, dronedarone, dolasetron, efavirenz, fluoxetine, foscarnet, ganciclovir, gatifloxacin, grepafloxacin, hydroxychloroquine, hydroxyzine, ibogaine, ivabradine, lenvatinib, ketoconazole, lansoprazole, lumefantrine, lopinavir, levomepromazine, lomefloxacin, milrinone, nilotinib, nelfinavir, norfloxacin, olanzapine, omeprazole, papaverine, panobinostat, pazopanib, probucol, perphenazine, posaconazole, propafenone, propofol, roxithromycin, sulpiride, saquinavir, sertraline, solifenacin, sulfamethoxazole/trimethoprim, sumatriptan, sunitinib, toremifene, terlipressin, terodiline, tacrolimus, technetium tc 99m tetrofosmin, telitromycin, thiothixene, tramadol, trazodone, trifluoperazine, vernakalant, vasopressin, voriconazole, n-acetylprocainamide
9.	Ramelteon	Strong CYP1A2 Inhibitors
	ramelteon, caffeine, duloxetine, tasimelteon, tizanidine, acebrophylline, acenocoumarol, agomelatine, alosetron, aminophylline, asenapine, bromazepam, clomipramine, clozapine, cyclobenzaprine, dacarbazine, flutamide, fluvoxamine, lidocaine, melatonin, mexiletine, mirtazapine, olanzapine, pimozide, pirfenidone, pomalidomide, propranolol, ramosetron, rasagiline, ropinirole, ropivacaine, stiripentol, theophylline, thiothixene, tizanidine, trifluoperazine, tacrine	ciprofloxacin, fluvoxamine, mexiletine, enoxacin, amiodarone, atazanavir, cimetidine, tacrine, zileuton, ticlopidine, deferasirox, methoxsalen, rucaparib, stiripentol, thiabendazole, vemurafenib
10.	Strong CYP3A4 Inducers	Protease Inhibitors
	carbamazepine, rifampin, phenytoin, st. john’s wort, apalutamide, bosentan, enzalutamide, fosphenytoin, lumacaftor, mitotane, primidone, rifabutin, rifapentine, barbiturates, phenobarbital, dabrafenib, dexamethasone, efavirenz, etravirine, lumacaftor, nivirapine, oxcarbazepine	atazanavir, darunavir, fosamprenavir, indinavir, lopinavir, nelfinavir, ritonavir, saquinavir, tipranavir, amprenavir, boceprevir, delavirdiune, etravirine, dasabuvir, elbasvir, ombitasvir, maraviroc, rilpivirine, simeprevir, sofosbuvir, telaprevir, daclatasvir
11.	HMG Co-A Reductase Inhibitors	CYP3A4 & Protease Inhibitors
	lovastatin, simvastatin, atorvastatin, cerivastatin, fluvastatin, pitavastatin, pravastatin, rosuvastatin	indinavir, nefazodone, nelfinavir, ritonavir, saquinavir, atazanavir, boceprevir, clarithromycin, cobicistat, darunavir, itraconazole, ketoconazole, saquinavir, telithromycin, telaprevir, voriconazole, amiodarone, amprenavir, idelalisib, lopinavir, posaconazole, aprepitant, ceritinib, cyclosporine, danazol, diltiazem, delavirdine, dronedarone, dasabuvir, erythromycin, fluconazole, fluvoxamine, imatinib, lapatinib, lomitapide, ombitasvir, paritaprevir, ranolazine, ticagrelor, tipranavir, verapamil
12.	CYP3A4 & Protease Inhibitors	Ergot Alkaloids & Derivatives
	indinavir, nelfinavir, ritonavir, saquinavir, atazanavir, boceprevir, clarithromycin, cobicistat, darunavir, itraconazole, ketoconazole, telaprevir, telithromycin, voriconazole, conivaptan, mifepristone, nefazodone, posaconazole, amiodarone, aprepitant, ceritinib, cyclosporine, delavirdine, diltiazem, dronedarone, imatinib, lapatinib, ombitasvir, paritaprevir, verapamil, erythromycin, grapefruit, dasbuvir, lomitapide	dihydroergotamine, ergotamine, ergonovine, methylergonovine, ergoloid mesylates, methysergide, bromocriptine
13.	Tizanidine	CYP1A2 Inhibitors
	tizanidine	ciprofloxacin, fluvoxamine, mexiletine, enoxacin, zileuton, amiodarone, atazanavir, cimetidine, ethinyl estradiol, tacrine, ticlopidine, deferasirox, methoxsalen, rucaparib, stiripentol, thiabendazole, vemurafenib, propafenone
14.	Tranylcypromine	Procarbazine
	isocarboxazid, phenelzine, rasagiline, safinamide, tranylcypromine, selegiline, methylene blue, linzolid	procarbazine, moclobemide, furazolidone, syrian rue, tedizolid
15.	Triptans	MAOIs
	sumatriptan, rizatriptan, zolmitriptan, almotriptan, eletriptan, frovatriptan, naratriptan	isocarboxazid, phenelzine, tranylcypromine, methylene blue, moclobemide, procarbazine, rasagiline, safinamide, selegiline, linezolid, syrian rue, tedizolid

Disclosures

This work was supported by the National Institutes of Health (grant numbers R01 LM011838, R21 HS023826, R01 HS025984, and R01 AG062499). Outside of the submitted work, Dr. Kawamoto reports having received honoraria from and engaged in consulting, sponsored research, writing assistance, licensing, or codevelopment in the past 3 years with McKesson InterQual, Hitachi, Pfizer, Klesis Healthcare, RTI International, Mayo Clinic, the University of California at San Francisco, MD Aware, and the ONC for Health Information Technology (via Security Risk Solutions) in the area of health information technology (IT). Dr. Kawamoto is also a former unpaid board member of the nonprofit HL7 health IT standard development organization, is currently an unpaid member of the US Health Information Technology Advisory Committee, and has helped develop a number of health IT tools that may be commercialized to enable wider impact. The other authors have declared no potential conflicts of interest.

Additional information

Drs. Boyce and Malone contributed to the study design. Dr. Romero contributed to the analysis and data synthesis. Drs. Wright, and Liu, Del Fiol, and Kawamoto contributed to substantive revisions. All authors reviewed the manuscript.