Abstract
Objective
The study sought to determine frequency and appropriateness of overrides of high-priority drug-drug interaction (DDI) alerts and whether adverse drug events (ADEs) were associated with overrides in a newly implemented electronic health record.
Materials and Methods
We conducted a retrospective study of overridden high-priority DDI alerts occurring from April 1, 2016, to March 31, 2017, from inpatient and outpatient settings at an academic health center. We studied highest-severity DDIs that were previously designated as “hard stops” and additional high-priority DDIs identified from clinical experience and literature review. All highest-severity alert overrides (n = 193) plus a stratified random sample of additional overrides (n = 371) were evaluated for override appropriateness, using predetermined criteria. Charts were reviewed to identify ADEs for overrides that resulted in medication administration. A chi-square test was used to compare ADE rate by override appropriateness.
Results
Of 16 011 alerts presented to providers, 15 318 (95.7%) were overridden, including 193 (87.3%) of the highest-severity DDIs and 15 125 (95.8%) of additional DDIs. Override appropriateness was 45.4% overall, 0.5% for highest-severity DDIs and 68.7% for additional DDIs. For alerts that resulted in medication administration (n = 423, 75.0%), 29 ADEs were identified (6.9%, 5.1 per 100 overrides). The rate of ADEs was higher with inappropriate vs appropriate overrides (9.4% vs 4.3%; P = .038).
Conclusions
The override rate was nearly 90% for even the highest-severity DDI alerts, indicating that stronger suggestions should be made for these alerts, while other alerts should be evaluated for potential suppression.
Keywords: patient safety, electronic prescribing, health information technology, medication safety, quality of care
INTRODUCTION
More than 1.5 million medication errors resulting in patient injury occur per year in the United States.1 Efforts to improve patient safety have included use of electronic health records (EHRs) with computerized provider order entry (CPOE) systems that include clinical decision support (CDS) systems that provide reminders and warnings to assist with medication prescribing.2,3 Utilizing CPOE with adequate CDS systems may reduce preventable adverse drug events (ADEs).4,5 CDS alerts warning of potential drug-drug interactions (DDIs) (ie, change in the effects of a drug when given together with a second drug, resulting in potential harmful consequences) are included in most CPOE systems, although the included alerts and their presentation vary widely among systems and institutions.6
When providers receive too many CDS alerts they can develop “alert fatigue,” in which they begin to ignore or override alerts, resulting in potential patient harm.7 Van der Sijs et al8 reviewed literature from 1980 to 2004 regarding overriding of medication safety alerts in CPOE systems and found that alerts were overridden in 49%-96% of cases. High override rates ranging from 60% to 91% have been shown for DDI alerts in both outpatient and inpatient settings.9–12 Although CPOE with CDS has been shown to prevent errors in the medication ordering process, the impact of specific types of alerts on reducing ADEs or how often overriding alerts results in ADEs has not been adequately studied. A recent evaluation of several types of CDS overrides in the intensive care unit (ICU) found that overrides that occurred with a commercial CPOE system were common, and often appropriate, and ADEs occurred more commonly in alerts that were inappropriately overridden.13
Our institution formerly utilized an in-house–developed EHR that incorporated a tiered DDI system that included hard stops that did not allow providers to proceed with ordering the highest-severity DDIs.14 In May 2015, we transitioned to a commercial EHR, with a system that eliminated hard stops, which meant that providers could override all DDI alerts, even those with potential to cause serious harm. We conducted this study to determine the frequency and appropriateness of overridden high-priority DDI alerts in the new system and determine if inappropriately overridden alerts were associated with ADEs.
MATERIALS AND METHODS
This retrospective, observational study evaluated CDS alerts, alert overrides, and resultant ADEs for high-priority DDIs during a 1-year period (April 1, 2016, to March 31, 2017), which began approximately 1 year after the implementation of a commercial EHR. The medication database used as the core for the rule set was First Databank (First Databank, South San Francisco, CA).
Patients 18 years of age or older with inpatient or outpatient orders during the study period for medications on the high-priority DDI list developed for this study were included. The high-priority DDI list included the entire list of highest-severity DDI alerts (“hard stops”) from the legacy system, which closely followed published recommendations.15 The list also included approximately 20 additional DDIs considered high priority based on the authors’ clinical and research experience6,11 and a literature search for studies identifying high risk DDIs.16–23 We considered frequency of use of medications and potential clinical severity of the interaction when identifying and arriving at consensus on additional interactions to study. Examples of these additional high-priority DDIs include combinations of amiodarone and digoxin, lithium and thiazide diuretics, valproic acid and carbapenems, and warfarin and sulfonamides.
We obtained data for number of alerts, frequency of alert overrides, and documented reasons for overrides, if available, for the drugs on the high-priority DDI list. Alerts that presented more than once for the same order (duplicate alerts) were excluded. An alert was defined as “overridden,” if the order was continued and completed after an alert was presented to the ordering provider, even if the order was later cancelled and the medication not administered to the patient. The reason for override options in the CPOE system included: “benefit outweighs risk,” “per protocol,” “inaccurate warning,” “does not apply to patient,” “patient tolerated before,” and “will monitor.” A reason was not captured if all DDIs were overridden as a group after alerts were presented by selecting “override all alerts.”
Three of the additional high-priority DDI alerts that we originally selected to study were filtered out during our study period (and thus not shown to providers), so override rates were not available for these interactions. We included data for the total numbers of these alerts to assess frequency of prescribing of DDI pairs on the high-priority list.
The primary outcome was the frequency of appropriate overrides for the overridden DDI alerts. Secondary outcomes included the incidence of ADEs associated with the overrides and the relationship between override appropriateness and ADEs.
Evaluation of appropriateness
We reviewed patient medical records for all highest-severity DDI alert overrides (n = 193) and for a stratified (by DDI alert type) random sample of additional high-priority DDI overrides (n = 371) to get an equal number of alerts reviewed for each alert type (36 alerts for most alert types). Alerts with a frequency of <35 alert overrides (5 DDI alerts) included an evaluation of all overridden alerts. Appropriateness of override was evaluated independently by a clinical pharmacist and a student pharmacist utilizing appropriateness criteria previously developed by a multidisciplinary group that were further refined by group consensus after reviewing new types of alerts.10
All overrides of alerts that were highest severity (ie, previously hard stops in the in-house EHR system) were considered inappropriate. For the additional high-priority alert overrides reviewed, actions such as dose reductions to adjust for a DDI, documentation of appropriate monitoring, or documentation that the patient was already tolerating the combination were considered in determining appropriateness of overrides. For DDIs resulting in potential corrected QT interval (QTc) prolonging, baseline QTc, number of additional QTc-prolonging drugs, and whether a single dose was used were considered in determining appropriateness of overrides. Disagreements on assessment of appropriateness were discussed and resolved by discussion between the 2 reviewers. A third experienced reviewer was consulted if consensus was not achieved.
ADE evaluation
For overridden alerts reviewed for appropriateness that resulted in both medications associated with the DDI administered to the patient (inpatients), or documented concurrent use (outpatients), 2 reviewers reviewed medical records independently to evaluate for occurrence of ADEs. The reviewers were the same as those who had reviewed for appropriateness of override; however, they conducted evaluation for ADEs after their assessment of appropriateness was complete for all overrides and were blinded to how they had categorized the appropriateness of override when evaluating for ADE. Severity of identified ADEs (significant, serious, life threatening, or fatal) was assessed and categorized based on the Brigham and Women’s Hospital Center for Patient Safety Research and Practice’s methodology.24 The reviewers recorded any supporting documentation of an ADE specific to the DDI from the chart, including notes from providers, laboratory values, and electrocardiography results. Prolonged QTc in patients with ventricular pacing were not rated as ADEs because of difficulty in interpreting risk in those patients. If an ADE was noted more than once in the same patient, the ADE was only counted once.
The interrater agreement for both override appropriateness and ADE occurrence was determined by calculating a Cohen’s kappa. Initial kappa scores were 0.54 (95% confidence interval, 0.47-0.61) for appropriateness and 0.53 (95% confidence interval, 0.40-0.67) for ADEs, which are in the moderate range. After discussion and strict adherence to appropriateness criteria, we obtained 100% agreement between the 2 reviewers.
Statistical analysis
Descriptive statistics were used to summarize patient demographics and alert characteristics including frequencies of individual alerts and overrides, classification of override appropriateness, and number and description of ADEs. A chi-square test was used to compare the rate of ADEs by appropriateness of override, with a significance level set at P < .05.
RESULTS
Override rates
Figure 1 includes details on total numbers of alerts and overrides. During the study period, 20 045 alerts occurred for DDIs on the high-priority list. We further evaluated the 16 011 (79.9%) alerts presented to providers. Of these, 15 318 (95.7%) were overridden. Overrides occurred for 193 of 221 (87.3%) of the highest-severity alerts and for 15 125 of 15 790 (95.8%) of the additional high-priority DDI alerts.
Figure 1.
Alert screening and inclusion.
The override rate for DDI alerts ranged from 83.3% to 96.6% among the different DDI pairs (Table 1). More than half of the alerts presented to providers were for ondansetron–QT-prolonging agents (n = 10 251).
Table 1.
Override rates for high-priority DDI alerts
| Specific DDI | Total alerts (excluding filtered alerts) | Overridden alerts |
|---|---|---|
| Ondansetron–QT-prolonging agents | 10 251 | 9897 (96.5) |
| Simvastatin (>20 mg)–amlodipine | 1962 | 1886 (96.1) |
| Other QT-prolonging agents (includes 1 level 1 DDI with 21 overrides)a | 1519 | 1450 (95.5) |
| Warfarin–sulfonamides | 757 | 706 (93.3) |
| Amiodarone–QT-prolonging agents | 713 | 689 (95.1) |
| Amiodarone; dronedarone–digitalis glycosides | 465 | 442 (95.1) |
| Tizanidine–ciprofloxacina | 116 | 100 (86.2) |
| Lithium–thiazide diuretics | 42 | 36 (85.7) |
| Valproic acid–carbapenem antibiotics | 42 | 40 (95.2) |
| Abatacept–TNF-blocking agentsa | 28 | 24 (85.7) |
| Dofetilide–thiazide diureticsa | 23 | 20 (87.0) |
| Ramelteon–fluvoxaminea | 14 | 13 (92.9) |
| Atazanavir; nelfinavir–proton pump inhibitorsa | 11 | 10 (90.0) |
| Efalizumab/natalizumab–immunosuppressants/immunomodulatorsa | 6 | 5 (83.3) |
| Filtered alertsb | ||
| 5HT-1D agonists (triptans)–SSRIs; SNRIs | 40 | 0 (0) |
| Amiodarone–warfarin | 11 | 0 (0) |
| Warfarin–metronidazole; tinidazole | 11 | 0 (0) |
| Total | 16 011 | 15 318 (95.7) |
5HT-1D: 5-hydroxytryptamine (serotonin) receptor 1D; DDI: drug–drug interaction; SNRI: selective norepinephrine reuptake inhibitor; SSRI: selective serotonin reuptake inhibitor; TNF: tumor necrosis factor.
Highest-severity alert.
Overridden alerts not included for filtered alerts (includes cancelled order).
Although we did not further review 3 alerts that were not shown to providers, the total occurrence of these alerts including those that were filtered out were as follows: 5-hydroxytryptamine (serotonin) receptor 1D agonists (triptans) and selective serotonin reuptake inhibitors or selective norepinephrine reuptake inhibitors (n = 2517), amiodarone–warfarin (n = 952) and warfarin and metronidazole or tinidazole (n = 397).
Appropriateness of overrides and ADEs
Patient demographics and additional patient and alert characteristics for the sample of alerts in which medical records were reviewed for override appropriateness are included in Table 2. The 564 overrides reviewed (all 193 highest-severity DDIs, stratified random sample of 371 additional high-priority DDIs) occurred in 505 unique patients.
Table 2.
Patient demographics of alerts reviewed for appropriateness (n = 505 unique patients, 564 alerts) and proportion of total alerts reviewed by type
| Age, y | 61.0 ± 15.7 (20-96) |
|---|---|
| Male | 256 (50.7) |
| Alerts by service | |
| Outpatient | 177/564 (31.4) |
| Inpatient | 387/564 (68.6) |
| ICU | 78 (20.2) |
| Non-ICU | 309 (79.8) |
| Proportion of alerts reviewed by type | |
| Highest-severity alerts | 193/193 (100) |
| Additional high-priority alerts | 371/15 125 (2.5) |
Values are mean ± SD (range), n/n (%), or n (%).
ICU: intensive care unit.
Override appropriateness was 45.4% overall (256 of 564 charts reviewed), 0.5%(n = 1 of 193) for highest-severity DDI alerts, and 68.7% (n = 255 of 371) for additional high-priority alerts. All highest-severity alert overrides were initially considered inappropriate, as noted in the Materials and Methods, but after review of medical records, it was determined that one override, in which an additive immune suppressant corticosteroid was ordered to be given “swish and spit” rather than ingested, was appropriate. Override appropriateness for the additional high-priority alerts was 62.5% in the outpatient setting and 70.2% in the inpatient setting.
Both medications associated with the DDI were administered to the patient in 423 of the 564 (75.0%) eoverridden DDI alerts reviewed, including 116 (60.1%) of the highest-severity DDIs and 307 (82.7%) of the additional high-priority DDIs. Reviewers identified 29 ADEs (6.9% of overridden DDIs administered, 5.1 ADEs per 100 overrides) occurring in unique patients, 9 (7.8%) ADEs with highest-severity DDIs, and 20 (6.5%) with additional high-priority DDIs. There were 20 (9.4%) ADEs identified for the 213 medication pairs administered for alerts previously determined to be inappropriately overridden, and 9 (4.3%) ADEs for 210 medication pairs administered for alerts previously determined to be appropriately overridden (Table 3). An ADE was more likely to occur with inappropriately overridden alerts than with appropriately overridden alerts (χ2 = 4.318, P = .038). A list of total ADEs and ADEs with appropriate and inappropriate overrides is listed for each DDI in Table 3.
Table 3.
Override appropriateness and adverse drug events for administered medications
| Drug interaction pairs | ADEs with appropriate overrides (n = 210) | ADEs with inappropriate overrides (n = 213) | Total ADEs (rate per 100 overrides) |
|---|---|---|---|
| Highest-severity DDIs | |||
| Dofetilide–QT-prolonging agents | 0 | 4 | 4 (19.0) |
| Dofetilide–thiazide diuretics | 0 | 3 | 3 (15.0) |
| Tizanidine–ciprofloxacin | 0 | 2 | 2 (2.0) |
| Abatacept–TNF-blocking agents | 0 | 0 | 0 (0) |
| Ramelteon–fluvoxamine | 0 | 0 | 0 (0) |
| Atazanavir; nelfinavir–proton pump inhibitors | 0 | 0 | 0 (0) |
| Efalizumab; natalizumab–immunosuppressants; immunomodulators | 0 | 0 | 0 (0) |
| Additional high-priority DDIs | |||
| Amiodarone–QT-prolonging agents | 2 | 5 | 7 (20.0) |
| Ondansetron–QT-prolonging agents | 2 | 0 | 2 (5.7) |
| Sotalol–QT-prolonging agents | 0 | 2 | 2 (12.5) |
| Methadone–QT-prolonging agents | 0 | 1 | 1 (2.9) |
| Ciprofloxacin–QT-prolonging agents | 0 | 1 | 1 (2.9) |
| Escitalopram–QT-prolonging agents | 2 | 0 | 2 (5.7) |
| Warfarin–sulfonamides | 1 | 1 | 2 (5.6) |
| Lithium–thiazide diuretics | 2 | 0 | 2 (5.6) |
| Simvastatin (>20 mg)–amlodipine | 0 | 1 | 1 (2.8) |
| Amiodarone or dronedarone–digitalis glycosides | 0 | 0 | 0 (0) |
| Valproic acid–carbapenem antibiotics | 0 | 0 | 0 (0) |
| Total | 9 | 20 | 29 (5.1) |
Values are n (%).
ADE: adverse drug event; DDI: drug–drug interaction; TNF: tumor necrosis factor.
Five ADEs occurred in outpatients, 8 in patients in the ICU setting, and 16 in non-ICU inpatients. ADEs included QTc prolongation (n = 22), bleeding and elevated international normalized ratio (n = 2), lithium toxicity (n = 2), hypotension (n = 2), and elevated creatine kinase (n = 1) (Table 4). Severity of the ADEs included 2 rated as significant (minor bleeding and elevated creatine kinase), 26 serious (QTc >500 ms [n = 21], symptomatic hypotension [n = 2], lithium toxicity [n = 2], elevated international normalized ratio of 5.9 with hemoptysis), and 1 life-threatening (QTc increase from 400 ms baseline to 709 ms). We did not note differences in severity of ADE between highest-severity or additional high-priority alerts or between outpatients and inpatients.
Table 4.
Adverse drug reactions by drug–drug interaction
| DDI | ADEs | ADE details and classification |
|---|---|---|
| Highest-severity alerts | ||
| Dofetilide–QT-prolonging agents | 4 | QTc prolongation |
| Severity: serious (n = 4) | ||
| Location: inpatient medicine (n = 3), ICU (n = 1) | ||
| Dofetilide–thiazide diuretics | 3 | QTc prolongation |
| Severity: serious (n = 3) | ||
| Location: outpatient clinics (n = 2), Non-ICU inpatient (n = 1) | ||
| Tizanidine–ciprofloxacin | 2 | Hypotension: BP decrease to 80/48 mm Hg in one patient and to 96/53 mm Hg in another patient with symptoms and monitoring noted |
| Severity: serious (n = 2) | ||
| Location: non-ICU inpatient (n = 2) | ||
| Additional high-priority alerts | ||
| Amiodarone–QT-prolonging agents | 7 | QTc prolongation |
| Severity: serious (n = 6); life-threatening (n = 1); QTc increase from 400 ms to 709 ms in an ICU patient) | ||
| Location: ICU (n = 5), non-ICU inpatient (n = 2) | ||
| Ondansetron–QT-prolonging agents | 2 | QTc prolongation |
| Severity: serious (n = 2) | ||
| Location: non-ICU inpatient (n = 2) | ||
| Sotalol–QT-prolonging agents | 2 | QTc prolongation |
| Severity: serious (n = 2) | ||
| Location: non-ICU inpatient (n = 2) | ||
| Warfarin–sulfonamides | 2 | Minor bleeding, therapeutic INR (n = 1) |
| Hemoptysis with elevated INR (5.9) (n = 1) | ||
| Severity: significant (n = 1), serious (n = 1) | ||
| Location: outpatient (n = 1), non-ICU inpatient (n = 1) | ||
| Escitalopram–QT-prolonging agents | 2 | QTc prolongation |
| Severity: serious (n = 2) | ||
| Location: ICU (n = 1); non-ICU inpatient (n = 1) | ||
| Lithium–thiazide diuretics | 2 | Elevated lithium level with symptoms (1 patient with level increase from 0.2 to 1.43 mEq/L and complained of nausea (normal range, 0.5-1.3 mEq/L); second patient with level increase from 0.78 to 1.36 mEq/L and symptoms of neurodecline (possibly due to lithium noted in chart) |
| Severity: serious (n = 2) | ||
| Location: outpatient (n = 2) | ||
| Methadone–QT-prolonging agents | 1 | QTc prolongation |
| Severity: serious (n = 1) | ||
| Location: non-ICU inpatient (n = 1) | ||
| Ciprofloxacin–QT-prolonging agents | 1 | QTc prolongation |
| Severity: serious (n = 1) | ||
| Location: non-ICU inpatient (n = 1) | ||
| Simvastatin (>20 mg)– amlodipine | 1 | Elevated CK (201 baseline to 535) |
| Severity: significant (n = 1) | ||
| Location: ICU (n = 1) | ||
ADE: adverse drug event; CK; creatine kinase; DDI: drug–drug interaction; ICU: intensive care unit; INR: international normalized ratio; QTc: corrected QT interval; TNF: tumor necrosis factor.
DISCUSSION
During this study period, providers overrode nearly all CDS alerts for high-priority DDIs, putting patients at risk for harm because of receiving dangerous combinations of medications. This finding is consistent with high override rates reported in the literature,8–13 which remains a widespread problem despite descriptions of several attempts to improve alert effectiveness and acceptability.14,25–31 Interventions have included making alerts more patient specific, tiering them based on severity, selecting some alerts for suppression to reduce alert burden, and tailoring alerts to provider role. These interventions have shown some benefits in alert acceptance, but improvements with greater impact that reduce alert fatigue and improve patient outcomes are needed. Calls have been made for innovative techniques to achieve better results, and we agree that this is urgently needed.32,33
All but one of the overrides for the highest-severity alerts in our study were classified as inappropriate; however, for the additional high-priority alerts, more than two-thirds of the overrides were considered appropriate. Our data suggest that many alerts that were considered “high priority” should be reviewed to see if they should be turned off or shown only when clinically appropriate based on patient specific factors. However, for the highest-severity DDIs, stronger intervention appears to remain necessary. There is very rarely a good clinical reason to give the drugs in the highest-severity group together.
Most of the overrides we analyzed resulted in medication administration, even for the highest-severity DDIs, which were associated with a higher frequency of ADEs than were lower severity DDIs. Relatively few studies have assessed the frequency of harm, compared with the number that assess the frequency of alerts; thus, these findings add to the data assessing the potential harm associated with alert overrides.
While the highest-severity alerts were overridden less often than the additional high-priority alerts, the override rate of 87.3% in these alerts for drug-drug combinations that were formerly not allowed to be ordered together and most of the medications reaching the patient is concerning. However, the total number of these highest-severity alerts that were presented to providers during the 1-year period was small in comparison with the additional high-priority alerts (221 vs 15 790), indicating that attempts to order these highest-risk combinations were rare. The ADE rate of 7.8% in patients who received these potentially dangerous medication combinations was lower than we expected, with only 2 cases of hypotension and 7 cases of QTc prolongation to >500 ms with none resulting in life-threatening arrhythmias, although in all these instances, there were reasonable therapeutic alternatives. It is possible that we missed some ADEs with our manual retrospective chart review, or that we did not have enough patients receiving these combinations to detect additional ADEs. In addition, the overrides for these medications that resulted in medication administration may have occurred in patients determined by the prescribers to be at lower risk or with greater ability to be monitored closely.
The override rate of 95.7% overall and 95.8% of our additional high-priority DDIs and our finding that the most common reason for alert overrides of “NULL” (no reason listed) is likely due to “alert fatigue” experienced by providers presented with too many alerts, who could select the option “override all alerts,” rather than evaluate each alert individually. Investigators at our institution noted that in the first several months after implementation of the commercial EHR, interruptive alerts for all DDIs were 6 times more frequent than before implementation, which was likely a contributor to alert fatigue.34
ADEs in the additional high-priority DDI group included mostly QTc prolongations to >500 ms and one >700 ms, with none resulting in life-threatening arrhythmias. We noted in chart reviews that most of these were noted as being monitored closely, with adjustments or discontinuations for one of the QTc-lowering medications often occurring. The one ADE that was rated as life-threatening was a patient who was being monitored closely in an ICU. For the QTc issues, risk is likely highest for patients not being monitored. Other identified ADEs of bleeding, increased lithium levels and increased creatine kinase, were not rated as life-threatening. As with the highest-severity alerts, the rate of 6.5% for ADEs found may also be an underestimate due to missed information detected with our retrospective chart review.
Interestingly, alerts for ondansetron and the risk of QTc prolongation accounted for more than half of the total alerts received (n = 10 362 alerts). As ondansetron is often given as a single dose perioperatively, or as an “as-needed” medication, this alert may need to be further evaluated at our institution. Other single-dose or as-needed medications could also be evaluated to see if any adjustment or tailoring of the alerts is recommended. As ADEs were more common in alerts that were inappropriately overridden, considering more stringent requirements to enter override reasons for contraindicated medication combinations or some other method to improve alert acceptance could be considered. Some changes to the overall DDI alerts have already occurred,34 with additional alerts being filtered in some cases, and more recently, some previously filtered alerts being shown to providers (amiodarone–warfarin and warfarin–metronidazole). We plan to share additional information learned from this study with EHR content administrators at our institution.
Limitations
Although we presented data from both the inpatient and outpatient populations, this study was conducted at a single center, which may limit its generalizability. The list of DDIs we included was based on previous studies and expert opinion, but other institutions may categorize the severity of these DDIs differently. Others may not agree with our designation of all except the one noted exception of the highest-severity alert overrides as inappropriate. The reason for our designation was that a multidisciplinary group of experts at our institution had felt that these alerts were severe enough that they would have previously been unable to be overridden. We did not encounter situations for these highest-severity alerts that we deemed changed their overrides to be appropriate, except in the one instance mentioned. However, it could be argued that in selected instances, such as orders for prn or single doses given perioperatively, or for lower doses, that the overrides could have been appropriate.
The potential for missing data in our retrospective chart review limited our ability to completely assess appropriateness and detect all ADEs. This likely resulted in an underestimation of ADEs, especially in outpatients who may not have documented ADEs that occurred outside the hospital setting. For outpatients, DDIs were counted as administered to the patient if the patient had an active prescription for both medications during the study period, but unlike inpatients who had doses documented as given in the EHR, it could not be confirmed if all doses were taken in the outpatient setting.
When determining the interrater agreement for appropriateness and for ADE, the moderate initial kappa scores were mostly due to inconsistency in strictly applying the appropriateness criteria for QTc-prolonging drugs. However, after discussing the disagreements and making minor adjustments to the criteria such as not counting patients with uninterpretable QTc due to implanted defibrillators with pacemakers as having inappropriate overrides, we achieved 100% final agreement.
Last, when assessing ADEs for those with prolonged QTc after administration of the DDI medications, even though provider notes and electrocardiography interpretations as “possible drug effect” for prolonged QTc were used to support documentation of ADEs, it is possible that there were additional potential etiologies of the prolonged QTc in those assessed as having ADE.
CONCLUSION
We found that most high-priority DDI alerts were overridden, including nearly 90% of the highest-severity alerts. Our findings suggest an urgent need to implement steps to address and improve the effectiveness of these alerts. A first step is to evaluate and potentially turn off some of the less important alerts and offer stronger warnings for the most important ones. Newer alternatives include addressing alert fatigue by using innovative techniques such as machine learning frameworks to limit presentation of alerts to that are most clinically relevant.35 We found that ADEs were more likely to occur in patients who received medications in which the alerts were inappropriately overridden compared with those appropriately overridden, though the observed level of harm related to this was less than we expected.
Author Contributions
All authors contributed to the conception and design of the study. AW and DLS conducted data collection and preparation; and HE and MGA conducted chart reviews for override appropriateness and ADEs, and led data analysis and interpretation of data and results. HE and MGA led drafting and writing the manuscript and revising it critically, with all coauthors commenting on drafts. All authors gave their approval of the final version for submission for publication.
Ethics Approval
This study was reviewed and approved by the Partners HealthCare System Institutional Review Board (Protocol #2016-P-001197; Brigham and Women’s Hospital).
Acknowledgments
We acknowledge the participation of PharmD students who assisted with retrieval of drug information used for this project, including Nicholas Donnelly and Hyungwon Kim.
Conflict of interest
HE, MGA, AW, and DLS have no competing interests. DWB consults for EarlySense, which makes patient safety monitoring systems; and for AESOP, which is using machine learning to identify patients getting medications that were not intended for them; receives cash compensation from CDI (Negev), Ltd, which is a not-for-profit incubator for health IT startups; receives equity from ValeraHealth, which makes software to help patients with chronic diseases; from Clew, which makes software to support clinical decision making in intensive care; and from MDClone, which takes clinical data and produces deidentified versions of it; and will be receiving research funding from IBM Watson Health. DWB’s financial interests have been reviewed by Brigham and Women’s Hospital and Partners HealthCare in accordance with their institutional policies.
REFERENCES
- 1.Institute of Medicine. Preventing Medication Errors: Quality Chasm Series. Washington, DC: The National Academies Press; 2007. [Google Scholar]
- 2. Bates DW, Gawande AA.. Improving safety with information technology. N Engl J Med 2003; 348 (25): 2526–34. [DOI] [PubMed] [Google Scholar]
- 3. Bennett JW, Glasziou PP.. Computerised reminders and feedback in medication management: a systematic review of randomised controlled trials. Med J Aust 2003; 178 (5): 217–22. [DOI] [PubMed] [Google Scholar]
- 4. Nuckols TK, Smith-Spangler C, Morton SC, et al. The effectiveness of computerized order entry at reducing preventable adverse drug events and medication errors in hospital settings: a systematic review and meta-analysis. Syst Rev 2014; 3 (1): 56. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Leung AA, Keohane C, Amato M, et al. Impact of vendor computerized physician order entry in community hospitals. J Gen Intern Med 2012; 27 (7): 801–7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. McEvoy DS, Sittig DF, Hickman TT, et al. Variation in high-priority drug-drug interaction alerts across institutions and electronic health records. J Am Med Inform Assoc 2017; 24 (2): 331–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Agency for Healthcare Research and Quality. PS Net. Patient safety primer: alert fatigue. https://psnet.ahrq.gov/primer/alert-fatigue Accessed October 4, 2019.
- 8. van der Sijs H, Aarts J, Vulto A, Berg M.. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006; 13 (2): 138–47. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Isaac T, Weissman JS, Davis RB, et al. Overrides of medication alerts in ambulatory care. Arch Intern Med 2009; 169 (3): 305–11. [DOI] [PubMed] [Google Scholar]
- 10. Nanji KC, Slight SP, Seger DL, et al. Overrides of medication-related clinical decision support alerts in outpatients. J Am Med Inform Assoc 2014; 21 (3): 487–91. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Slight SP, Seger DL, Nanji KC, et al. Are we heeding the warning signs? Examining providers’ overrides of computerized drug-drug interaction alerts in primary care. PLoS One 2013; 8 (12): e85071. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Nanji KC, Seger DL, Slight SP, et al. Medication-related clinical decision support alert overrides in inpatients. J Am Med Inform Assoc 2018; 25 (5): 476–81. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Wong A, Amato MG, Seger DL, et al. Prospective evaluation of medication-related clinical decision support over-rides in the intensive care unit. BMJ Qual Saf 2018; 27 (9): 718–24. [DOI] [PubMed] [Google Scholar]
- 14. Paterno MD, Maviglia SM, Gorman PN, et al. Tiering drug–drug interaction alerts by severity increases compliance rates. J Am Med Inform Assoc 2009; 16 (1): 40–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. Phansalkar S, Desai AA, Bell D, et al. High-priority drug–drug interactions for use in electronic health records. J Am Med Inform Assoc 2012; 19 (5): 735–43. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Juurlink DN, Mamdani M, Kopp A, Laupacis A, Redelmeier DA. Drug-drug interactions among elderly patients hospitalized for drug toxicity. JAMA 2003; 289 (13): 1652–8. [DOI] [PubMed] [Google Scholar]
- 17. Marengoni A, Pasina L, Concoreggi C, et al. Understanding adverse drug reactions in older adults through drug-drug interactions. Eur J Intern Med 2014; 25 (9): 843–6. [DOI] [PubMed] [Google Scholar]
- 18. Dechanont S, Maphanta S, Butthum B, Kongkaew C.. Hospital admissions/visits associated with drug-drug interactions: a systematic review and meta-analysis. Pharmacoepidemiol Drug Saf 2014; 23 (5): 489–97. [DOI] [PubMed] [Google Scholar]
- 19. Bucşa C, Farcaş A, Cazacu I, et al. How many potential drug-drug interactions cause adverse drug reactions in hospitalized patients? Eur J Intern Med 2013; 24 (1): 27–33. [DOI] [PubMed] [Google Scholar]
- 20. Banda JM, Callahan A, Winnenburg R, et al. Feasibility of prioritizing drug-drug-event associations found in electronic health records. Drug Saf 2016; 39 (1): 45–57. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21. Juarez-Cedillo T, Martinez-Hernandez C, Hernandez-Constantino A, et al. Clinical weighting of drug-drug interactions in hospitalized elderly. Basic Clin Pharmacol Toxicol 2016; 118: 298–305. [DOI] [PubMed] [Google Scholar]
- 22. Scheife RT, Hines LE, Boyce RD, et al. Consensus recommendations for systematic evaluation for drug-drug interaction evidence for clinical decision support. Drug Saf 2015; 38 (2): 197–206. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23. Tilson H, Hines LE, McEvoy G, et al. Recommendations for selecting drug-drug interactions for clinical decision support. Am J Health-Syst Pharm 2016; 73 (8): 576–85. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24. Bates DW, Cullen DJ, Laird N, et al. Incidence of adverse drug events and potential adverse drug events. Implications for prevention. JAMA 1995; 274 (1): 29–34. [PubMed] [Google Scholar]
- 25. Hussain MI, Reynolds TL, Zheng K.. Medication safety alert fatigue may be reduced via interaction design and clinical role tailoring: a systematic review. J Am Med Inform Assoc 2019; 26 (10): 1141–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26. Daniels CC, Burlison JD, Baker DK, et al. Optimizing drug-drug interaction alerts using a multidimensional approach. Pediatrics 2019; 143 (3): e20174111. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27. Pirnejad H, Amiri P, Niazkhani Z, et al. Preventing potential drug-drug interactions through alerting decision support systems: a clinical context-based methodology. Int J Med Inform 2019; 127: 18–26. [DOI] [PubMed] [Google Scholar]
- 28. Beeler PE, Emmanuel E, Markus S, Jurg B.. Negligible impact of highly patient-specific decision support for potassium-increasing drug-drug interactions-a cluster- randomized controlled trial. Swiss Med Wkly 2019; 149: w20035. [DOI] [PubMed] [Google Scholar]
- 29. Horn J, Ueng S.. The effect of patient-specific drug-drug interaction alerting on the frequency of alerts: A pilot study. Ann Pharmacother 2019; 53 (11): 1087–92. [DOI] [PubMed] [Google Scholar]
- 30. Baysari MT, Zheng WY, Li L, et al. Optimizing computerized decision support to transform medication safety and reduce prescriber burden: study protocol for a mixed-methods evaluation for drug-drug interaction alerts. BMJ Open 2019; 9 (8): e026034. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31. Bakker T, Klopotowska JE, Eslami S, et al. The effect of ICU-tailored drug-drug interaction alerts on medication prescribing and monitoring: protocol for a cluster randomized stepped-wedge trial. BMC Med Inform Dec Mak 2019; 19: 159. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Bakken S. Need for innovation in electronic health record-based medication alerts. J Am Med Inform Assoc 2019; 26: 901–2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33. Tolley CL, Slight SP, Husband AK, et al. Improving medication -related clinical decision support. Am J Health-Syst Pharm 2018; 75 (4): 239–46. [DOI] [PubMed] [Google Scholar]
- 34. Wright A, Aaron S, Seger D, et al. Reduced effectiveness of interruptive drug-drug interactions alerts after conversion to a commercial electronic health record. J Gen Intern Med 2018; 33 (11): 1868–76. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35. Liu N, Chen CB, Kumara S.. Semi-Supervised learning algorithm for identifying high-priority drug-drug interactions through adverse event reports. IEEE J Biomed Health Inform 2019; 24: 57–68. [DOI] [PubMed] [Google Scholar]