Abstract
Background
Drug–drug interaction (DDI) alerts target the co-prescription of two potentially interacting medications and are a frequent feature of electronic medical records (EMRs). There have been few controlled studies evaluating the effectiveness of DDI alerts. This study aimed to determine the impact of DDI alerts on rates of DDIs and on associated patient harms.
Methods
Quasi-experimental controlled pre–post study in five Australian hospitals. Three hospitals acted as control hospitals (EMR with no DDI alerts) and two as intervention (EMR with DDI alerts). Only DDI alerts at the highest severity level (defined as ‘major contraindicated’) were switched on at intervention hospitals. These alerts were not tailored to clinical context (ie, patient, drug). A total of 2078 patients were randomly selected from all patients (adult and paediatric) admitted to hospitals 6 months before and 6 months after EMR implementation. A retrospective chart review was performed by study pharmacists. The primary outcome was the proportion of admissions with a clinically relevant DDI. Secondary outcomes included the proportions of admissions with a potential DDI and with DDI-related harm.
Results
Potential DDIs were identified in the majority of admissions (n=1574, 74.7%) and clinically relevant DDIs identified in half (n=1026, 48.7%). DDI alerts were associated with a reduction in the proportion of admissions with potential DDIs (adjusted OR (AOR)=0.38 (0.19, 0.78)) but no change in clinically relevant DDIs (AOR=1.12 (0.68, 1.84)) or in DDI-related harm (AOR=2.42 (0.47,12.31)). 199 DDIs (76 at control and 123 at intervention hospitals) for 35 patient admissions were associated with patient harm, and 2 patients experienced severe DDI-related harm pre-EMR implementation.
Discussion
Implementation of DDI alerts, without tailoring alerts to clinical context, is unlikely to reduce patient harms from DDIs. Organisations should reconsider implementation of DDI alerts in EMRs where significant tailoring of alerts is not possible. Future research should focus on identifying safe, efficient and cost-effective ways of refining DDI alerts, so expected clinical benefits are achieved, and negative consequences of excessive alerting are minimised.
Keywords: Health services research; Medication safety; Decision support, computerized
WHAT IS ALREADY KNOWN ON THIS TOPIC.
WHAT THIS STUDY ADDS
Our study showed that implementation of DDI alerts, without tailoring alerts to clinical context, did not reduce clinically relevant DDIs or harm to patients from DDIs. DDI-associated patient harms were rare, and potential DDIs were not good indicators of risk to patients from DDIs
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY
The findings suggest that organisations should reconsider the implementation of simple DDI alerts in EMRs, where significant tailoring of alerts is not possible.
Background
Drug–drug interaction (DDI) alerts target the co-prescription of two potentially interacting medications and are a frequent feature of electronic medical records (EMRs). Their inclusion in EMRs is recommended by government and safety bodies (eg, the US Government’s Meaningful Use Program).1 2 However, a large number of studies have shown that DDI alerts lead to clinician frustration, are frequently overridden by users, and contribute to alert fatigue.3,11
There have been limited controlled studies evaluating the effectiveness of DDI alerts. Two studies assessed the implementation of a single customised DDI alert on the concurrent prescription of two medications and showed that the introduction of an alert was associated with fewer orders of the targeted drug pairs.12 13 In reality, however, a single DDI alert is not implemented in isolation, and organisations make entire databases of DDI alerts available to users, stratified by severity level. DDI alerts are also just one alert type of many included in EMRs,14 and clinicians are typically presented with a range of other alerts, such as allergy and dose range warnings. The result is a significant alert burden to clinicians, and consequently, an increased risk of alert fatigue, leading to safety-critical alerts being missed. Increasingly, alert fatigue is emerging as a key contributor to incidents resulting in medication errors and subsequent patient harm.15,17
A recent call to focus research efforts on the development and evaluation of advanced clinical decision support, rather than simple alerts,18 assumes a strong evidence base exists for simple alert inclusion and exclusion. There is an expectation among both management and clinicians that simple DDI alerts will reduce DDIs and associated patient harms, and this expectation, along with low tolerance for patient and institutional risk, are critical drivers for alert implementation.19 However, to date, no controlled evaluations assessing the impact of a set of DDI alerts on DDIs have been undertaken. Importantly, there have also been no controlled studies examining the impact of simple DDI alerts on DDI-related patient harm. This study aimed to fill these significant evidence gaps. We aimed to determine the impact of DDI alerts on rates of DDIs and on associated patient harms.
Method
Study design
This was a quasi-experimental controlled pre–post study using retrospective patient record review data. This design was chosen because randomisation of hospitals to control and intervention groups was not possible, and two participating sites had implemented an EMR (one with and one without DDI alerts) at the time of study commencement.
Settings and intervention
Sites were five hospitals in two Australian states, New South Wales and Queensland, see online supplemental appendix 1. Three hospitals acted as control hospitals and implemented their EMR (Cerner Millennium or DXC Technology MedChart) for medication management without DDI alerts in place, and two acted as intervention hospitals (both with Cerner Millennium) and made DDI alerts available to prescribers in the EMR.
In intervention hospital EMRs, DDI alerts were interruptive and triggered at the point of medication order entry (see figure 1). To bypass alerts, prescribers were required to enter an override reason into the alert screen. No hard-stop DDI alerts that prevent the prescriber from continuing with their order were included in the EMR. The intervention hospitals used the Oracle Health Multum DDI knowledge-base (https://www.oracle.com/health/service-lines-departments/pharmacy/#rc30p5) for DDI detection and had only the highest level of alert severity (major-contraindicated) operational. This included approximately 7500 DDI alerts.
Figure 1. Example drug–drug interaction alert.
Sample
Patients were randomly selected from all patients who were in study hospitals during a 1 week period 6 months before and 6 months after EMR implementation (dates appear in online supplemental appendix 2). All inpatients were eligible for inclusion across all hospital wards, except those who visited the ED but were not admitted to inpatient wards, and those in wards where a different EMR system with variable DDI alerts was in use (ie, the intensive care unit (ICU) and oncology department). If inpatients were transferred from the ward to ICU, their ward admission was included up until the point of transfer. Those whose hospital length of stay was greater than 3 months were also excluded. Anaesthetic charts were also excluded from chart review due to the variability in the documentation of anaesthetic medications across hospitals.
Outcome measures
Our primary outcome measure was the proportion of patient admissions with a clinically relevant DDI. Secondary outcomes included the proportion of admissions with a potential DDI and the proportion of admissions with DDI-related harm. Definitions of outcome measures appear in table 1 and a flow diagram depicting the data collection procedure appears in figure 2.
Table 1. Study outcome measures.
| Measure | Definition |
|---|---|
| Potential DDIs | Two or more drugs interacting with each other in such a way that the effectiveness or toxicity of one or more drugs is potentially altered.* |
| Clinically relevant DDIs† | Two or more drugs interacting with each other in such a way that the effectiveness or toxicity of one or more drugs is highly likely to be altered when taking into account individual patient factors (eg, age, sex, diagnosis, comorbidities) and medication order factors (eg, dose, route). |
| Patient harm | Identification of harm was based on symptoms and investigations recorded in the patient record. Harm constituted ‘impairment of structure or function of the body and/or any deleterious effect arising there from, including disease, injury, suffering, disability and death, and may be physical, social or psychological’.51 |
Potential DDIs were those classified as moderate and severe by Stockley’s Interaction Checker.
Primary outcome measure.
DDIs, drug–drug interactions.
Figure 2. Data collection procedure (pDDIs=potential DDIs, cDDIs=clinically relevant DDIs). *Stockley’s Interactions Checker22 is an authoritative international source of drug interaction information. DDIs, drug–drug interactions.
Data collection procedure
Data collection was undertaken by a team of clinical pharmacists (RQ, MM, DD, LF, AAS), who had 7–22 years of experience working as clinical pharmacists and were independent from study hospitals. Based on a review of relevant literature and consultation with pharmacists and clinical pharmacologists, a data collection guideline was developed, including detailed guides for determining whether a DDI was clinically significant.20 All pharmacists received training on the application of the guideline and undertook several practice rounds where admissions were reviewed and classified, and then classifications discussed among the study pharmacy team. Prior to formal data collection commencing, inter-rater reliability testing was performed to ensure all pharmacists were consistent in their classification of clinically significant DDIs (Cohen’s kappa=0.8).20 After achieving a minimum Cohen’s kappa score of 0.8, indicating a strong level of agreement,21 pharmacists then collected data independently and only brought difficult cases to the broader data collection team for discussion.
For all patients, pharmacists recorded patient demographics and details of all medication orders that were prescribed during an admission into a Microsoft Access database. In the post-EMR period, medication data were extracted from EMR systems, rather than manually extracted from records. Each patient’s medication orders that were active on the same calendar date were then entered into Stockley’s Interactions Checker,22 an authoritative international source of drug interaction information, to identify potential DDIs. Based on the severity classifications used by the Stockley’s checker, potential DDIs of the two highest severity levels (ie, severe and moderate) were recorded into the study database and underwent further review. Pharmacists completed a detailed audit of patients’ medical records to determine whether potential DDIs were clinically relevant, taking into account patient factors, such as age, sex, kidney function and medication order factors, such as dose and route. They also considered whether the benefit of prescribing the combination outweighed any risks to patients. online supplemental appendix 3 outlines all contextual factors considered when determining whether or not a DDI was clinically relevant.20 Each potential DDI was rated as very unlikely, unlikely, possible or likely to be a clinically relevant DDI, with the latter two categories classified as clinically relevant DDIs.
For all clinically relevant DDIs where medications were administered to patients, an in-depth review of patient records was performed to identify any evidence of possible harm to the patient resulting from DDIs (eg, abnormal test result, administration of an antidote). This information was extracted from records and collated into ‘patient case studies’,23 which were then presented to an expert panel of two clinical pharmacologist physicians, independent from study hospitals, who determined whether these possible harms constituted actual patient harm resulting from the DDI. The panel were blinded to group (control vs intervention) and period (pre vs post). After being trained on the use of the rating tools, the two clinical pharmacologists separately rated each case for severity of harm (no harm, minor, moderate, serious or severe)24 and likelihood that the patient harm was a result of the DDI (unlikely, possible, probable or certain)25, with the latter two likelihood categories constituting actual DDI-related harm to patients, see online supplemental appendix 4. The clinical pharmacologists then met to discuss each case and agree on the ratings via a consensus process.
Sample size
The initial sample size calculation was based on the study by Vonback et al26 which used comparable methodology to our planned study. They reported that 56% of patients experienced at least one potential DDI during their hospital stay. We estimated a 25% or greater change in potential DDIs to be clinically relevant. To detect this change, the estimated sample size was 2800 patient admissions (with 90% power and 5% level of significance). However, data collection was significantly impacted by COVID-19. We recalculated the sample size based on data collected in our study up until May 2020. The proportion of admissions with one or more clinically relevant DDIs was 44%. Assuming a 35% reduction in the proportion of admissions with a clinically relevant DDI, the total sample size required with 80% power at 5% significance level (two-sided test) was 1740 patient admissions.
Statistical analysis
We conducted an intention-to-treat analysis. Descriptive summaries of patient demographics and admission characteristics were presented for each study period for control and intervention groups. To examine the impact of the implementation of DDI alerts, a generalised estimating equation approach with an interaction term between the study group and period was applied to investigate changes in the primary outcome, that is, proportion of patient admissions with a clinically relevant DDI in the intervention group, relative to the changes in the control group. The adjusted rates and relative rate changes were estimated after adjusting for other relevant patient demographics, including sex, age group and number of drugs, with consideration of patient level cluster. The same modelling approach was applied to investigate the two secondary outcomes. All statistical analyses were performed using SAS V.9.4.
Approval and protocol deviations
The pharmacist’s ability to collect data from hospital records was significantly impacted by COVID-19, requiring the original study scope, as reported in our protocol,27 to be reduced. The Strengthening the Reporting of Observational Studies in Epidemiology checklist appears in online supplemental appendix 5.
Role of the funding source
The National Health and Medical Research Council (NHMRC) played no role in study design, collection, analysis and interpretation of data, in writing of the report or in the decision to submit this manuscript. Employees of eHealth NSW and eHealth QLD contributed to study design, the writing of the report and in the decision to submit this paper for publication, and their contributions have been noted in the author contribution and acknowledgement sections.
Results
Patient characteristics
In total, 1170 patient medical records were reviewed in control sites and 908 in intervention sites. As shown in table 2, there were differences across intervention and control hospitals in both patient sex and age distributions. Control sites had a greater proportion of female patients (55.9% vs 49.0%) and fewer patients>64 years of age (39.9% vs 42.6%) compared with intervention sites.
Table 2. Patient and admission characteristics pre and post introduction of an electronic medical record (EMR) in control and intervention hospitals.
| Control | Intervention | All | |||||
|---|---|---|---|---|---|---|---|
| Pre-EMR N (%) |
Post-EMR with no DDI alerts N (%) |
Pre-EMR N (%) |
Post-EMR with DDI alerts N (%) |
Overall N (%) |
Control N (%) |
Intervention N (%) |
|
| Patients, N (row %) | 593 (50.7) | 577 (49.3) | 454 (50.0) | 454 (50.0) | 2078 (100.0) | 1170 (56.3) | 908 (43.7) |
| Sex | |||||||
| Female | 341 (57.5) | 313 (54.2) | 224 (49.3) | 221 (48.7) | 1099 (52.9) | 654 (55.9) | 445 (49.0) |
| Male | 252 (42.5) | 264 (45.8) | 230 (50.7) | 233 (51.3) | 979 (47.1) | 516 (44.1) | 463 (51.0) |
| Age (years) | |||||||
| 0–64 | 398 (67.1) | 352 (61.0) | 259 (57.0) | 262 (57.7) | 1271 (61.2) | 750 (64.1) | 521 (57.4) |
| 65+ | 195 (32.9) | 225 (39.0) | 195 (43.0) | 192 (42.3) | 807 (38.8) | 420 (35.9) | 387 (42.6) |
| Admissions | |||||||
| Number of admissions, N (row %) | 597 (50.3) | 589 (49.7) | 461 (50.0) | 461 (50.0) | 2108 (100.0) | 1186 (56.3) | 922 (43.7) |
| Number of orders, median (IQR) | 9 (4–18) |
9 (3–17) |
14 (6–25) |
14 (6–25) |
11 (4–21) |
9 (3–18) |
14 (6–25) |
| Number of drugs, median (IQR) | 7 (3–13) |
7 (2–14) |
10 (5–18) |
11 (5–18) |
9 (4–16) |
7 (3–13) |
10 (5–18) |
| Number of pDDIs/admission Median (IQR) |
4.0 (0.0–14.0) |
4.0 (0.0–13.0) |
8.0 (1.0–25.0) |
10.0 (1.0–28.0) |
5.0 (0.0–19.0) |
4.0 (0.0–13.0) |
9.0 (1.0–26.0) |
| Number of cDDIs/admission Median (IQR) |
0.0 (0.0–3.0) |
0.0 (0.0–3.0) |
1.0 (0.0–6.0) |
1.0 (0.0–6.0) |
0.0 (0.0–4.0) |
0.0 (0.0–3.0) |
1.0 (0.0–6.0) |
cDDI, clinically relevant drug–drug interaction; pDDI, potential drug–drug interaction.
30 patients were admitted multiple times during study periods. In total, 1058 admissions were reviewed in the pre-period and 1050 admissions in the post-period. As shown in table 2, the median number of medication orders prescribed per admission was 11, with fewer medication orders per admission in control hospitals than intervention hospitals (9 vs 14).
Impact of DDI alerts on rate of clinically relevant DDIs
In total, 10 011 clinically relevant DDIs were identified in our sample, and 48.7% (n=1026) of admissions experienced one or more clinically relevant DDIs. Just over a quarter of potential DDIs were determined to be clinically relevant (26.9%; 95% CI: 26.4% to 27.3%).
The change in prevalence of clinically relevant DDIs pre and post EMR in the control hospitals (with no DDI alerts) was similar to the change in intervention hospitals (with DDI alerts). That is, after adjusting for sex, age group and number of drugs, there was no difference in the proportion of admissions with a clinically relevant DDI pre and post EMR across control and intervention hospitals (adjusted OR=1.12, p=0.665) (table 3).
Table 3. Admissions with a potential DDI, clinically relevant DDI or DDI-related harm pre and post EMR in control and intervention hospitals.
| Control | Intervention | Relative changes | ||||||
|---|---|---|---|---|---|---|---|---|
| Pre-EMR n/N(%) | Post-EMR with no DDI alerts n/N(%) | Pre-EMR n/N(%) | Post-EMR with DDI alerts n/N(%) |
OR (95% CI) | P value | AOR (95% CI) | P value | |
| Admissions with a pDDI | 420/597 (70.4) |
411/589 (69.8) |
379/461 (82.2) |
364/461 (79.0) |
0.83 (0.55 to 1.26) |
0.388 | 0.38 (0.19 to 0.78) |
0.008 |
| Admissions with a cDDI | 256/597 (42.9) |
250/589 (42.4) |
257/461 (55.7) |
263/461 (57.0) |
1.07 (0.76 to 1.52) |
0.691 | 1.12 (0.68 to 1.84) |
0.665 |
| Admissions with harm* | 8/597 (1.3) |
3/589 (0.5) |
11/461 (2.4) |
13/461 (2.8) |
3.15 (0.65 to 15.21) |
0.153 | 2.42 (0.47 to 12.31) |
0.288 |
AOR=adjusted OR; AORs were estimated using the same generalised estimating equation models adjusting for sex, age group (<65, 65+) and number of drugs.
ORs=ORs, estimated from the generalised estimating equation models accounting for patient-level cluster with adjustment for and the interaction between study group and period.
Harms were those determined to be plausible and certain by the physician panel.
cDDI, clinically relevant drug–drug interaction; DDI, drug–drug interaction; EMR, electronic medical record; pDDI, potential drug–drug interaction.
Impact of DDI alerts on rate of potential DDIs
In total, 37 299 potential DDIs were identified in our sample, and the median number of potential DDIs/admission was five (table 2). 74.7% (n=1574) of admissions experienced one or more potential DDIs. The change in prevalence of potential DDIs pre and post EMR in the control hospitals was less than the change in intervention hospitals. That is, after adjusting for sex, age group and number of drugs, DDI alerts were associated with a decrease in the proportion of admissions with a potential DDI (table 3).
Impact of DDI alerts on DDI-related patient harms
In total, 199 DDIs in 35 patient admissions (34 patients) were associated with patient harm. Thus, 0.5% (199/37 299) of potential DDIs and 2.0% (199/10 011) of clinically relevant DDIs resulted in patient harm. As shown in table 4, 76 DDIs were associated with patient harm at control hospitals (57 pre and 19 post-EMR) and 123 DDIs at intervention hospitals (33 pre and 90 post-EMR). The majority of DDI-related harm identified was of minor or moderate severity, with 46 and 14 cases of serious and severe harm respectively.
Table 4. Number of clinically relevant DDIs associated with harm pre and post EMR in control and intervention hospitals*.
| Severity of harm | Control | Intervention | Overall | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Pre-EMR | Post-EMR with no DDI alerts | All | Pre-EMR | Post-EMR with DDI alerts | All | |||||||||
| n |
% of cDDIs (95% CI) |
n | % of cDDIs (95% CI) | n | % of cDDIs (95% CI) | n | % of cDDIs (95% CI) | n | % of cDDIs (95% CI) | n | % of cDDIs (95% CI) | n |
% of cDDIs (95% CI) |
|
| Minor | 19 | 0.8 (0.5 to 1.3) |
0 | 19 | 0.4 (0.3 to 0.7) |
3 | 0.1 (0.0 to 0.3) |
40 | 1.3 (0.9 to 1.7) |
43 | 0.8 (0.6 to 1.0) |
62 | 0.6 (0.5 to 0.8) |
|
| Moderate | 21 | 0.9 (0.6 to 1.4) |
6 | 0.3 (0.1 to 0.6) |
27 | 0.6 (0.4 to 0.9) |
27 | 1.1 (0.7 to 1.5) |
23 | 0.7 (0.5 to 1.1) |
50 | 0.9 (0.7 to 1.1) |
77 | 0.8 (0.6 to 1.0) |
| Serious | 5 | 0.2 (0.1 to 0.5) |
13 | 0.6 (0.4 to 1.1) |
18 | 0.4 (0.3 to 0.7) |
1 | 0.0 (0.0 to 0.2) |
27 | 0.8 (0.6 to 1.2) |
28 | 0.5 (0.3 to 0.7) |
46 | 0.5 (0.3 to 0.6) |
| Severe | 12 | 0.5 (0.3 to 0.9) |
0 | 12 | 0.3 (0.2 to 0.5) |
2 | 0.1 (0.0 to 0.3) |
0 | 2 | 0.0 (0.0 to 0.1) |
14 | 0.1 (0.1 0.2) |
||
| Total | 57 |
2.5 (2.0 to 3.3) |
19 |
0.9 (0.6 to 1.5) |
76 |
1.8 (1.4 to 2.2) |
33 |
1.3 (0.9 to 1.8) |
90 |
2.8 (2.3 to 3.4) |
123 |
2.1 (1.8 to 2.6) |
199 | 2.0 (1.7 to 2.3) |
Severity based on the harm associated with medication errors classification (HAMEC).24
Harms were those determined to be plausible and certain by the physician panel.
cDDIs, clinically relevant drug–drug interactions; DDIs, drug–drug interactions; EMR, electronic medical record.
Online supplemental appendix 6 shows the DDI pairs associated with severe patient harm. Two patients experienced the 14 DDIs that resulted in severe patient harm. These severe harms occurred pre-EMR implementation. Both patients who experienced severe harm had pharmacodynamic DDIs. One patient was prescribed two antiplatelet agents and experienced a major bleed. Another was prescribed six drugs that caused central nervous system (CNS) depression. The patient became oversedated requiring supplementary oxygen and overnight observation in the ICU.
Patients in the intervention hospitals experienced a similar change in prevalence of DDI-related harm pre and post EMR to patients in the control hospitals. As shown in table 3, after adjusting for sex, age group and number of drugs, there was no difference in prevalence of DDI-related harm pre and post EMR across control and intervention hospitals.
Discussion
This was the first controlled study to examine the impact of the introduction of a set of DDI alerts on DDIs and associated patient harms. We found that although DDI alerts, at the ‘major contraindicated’ level of severity, were associated with a reduction in potential DDIs, they were not associated with reductions in rates of clinically relevant DDIs or in DDI-related harms. Importantly, although 75% of patient admissions had one or more potential DDIs, only 26.9% of these DDIs were determined to be clinically relevant and only 0.5% of potential DDIs were associated with patient harm. This demonstrates that potential DDIs are not good indicators of risk to patients from DDIs.
Although this pre–post study was designed to determine if DDI alerts were effective, not why or why not, our complementary work can shed some light on explaining the results observed here. Our exploratory research with prescribers,628,30 and a large number of international studies,1031,33 highlight the challenges associated with the design of effective and usable DDI alerts. Consistent with our findings here, many DDI alerts are viewed as not clinically relevant by end-users, and as a result, alert fatigue sets in.34 A prerequisite for DDI alerts to be effective in reducing patient harms resulting from DDIs is that they are read and acted on by clinicians, behaviours unlikely to occur when alert fatigue is being experienced.28 However, our study also showed that even if current alerts are read and acted on, this is unlikely to lead to reductions in DDI-related harms, given the low rate at which DDI-related harms occurred. We found only two patients in our sample who experienced severe DDI-related harm. For both patients, the interactions were pharmacodynamic, which most clinicians can recognise if they know the mechanism of action of the drugs; in contrast to pharmacokinetic interactions, which require more specialised knowledge, access to reference texts or interaction software.35 36 One of the patients experienced severe harm from taking six concurrent CNS depressants, highlighting the clinical importance of drug interactions beyond drug pairs, which are not captured by current DDI alerts.
The small but significant reduction we observed in patients with a potential DDI following DDI alert implementation, in the absence of a corresponding decline in clinically significant DDIs, could suggest that DDI alerts resulted in clinicians not prescribing combinations that were in fact safe for patients. This has been identified as a potential unintended consequence of DDI alert implementation.13 Further research is needed to determine the frequency with which DDI alerts may result in inappropriate responses, and the flow-on effects for patients.
Our findings add to the growing body of literature highlighting that improvements to alert specificity are critical for reducing frequent interruptions to prescriber workflow and improving the effectiveness of alerts to prevent errors.37 38 Acknowledging the potential negative impact that large numbers of alerts can have on prescribers, the Office of the National Coordinator of Health Information Technology Safety Assurance Factors for EHR Resilience guides recommend that all hospitals have a process in place to review interactions so that only the most significant alerts are presented to clinicians.39 However, alert customisation is highly resource-intensive,40 requiring the establishment of multidisciplinary groups to review and make decisions on DDI alert inclusion,41 with positive impacts of this process not guaranteed.42
A recent large cluster randomised stepped-wedge trial across nine ICUs in the Netherlands examined the effect of tailoring potential DDI alerts to the ICU setting on the administration of high-risk drug combinations.43 In that study, tailoring alerts involved a modified Delphi procedure with an expert panel of ICU doctors and pharmacists recruited to identify DDIs that were clinically relevant for the ICU context. Implementing these tailored DDI alerts, by either removing irrelevant alerts or implementing an initial set of alerts, resulted in a 12% decrease in the number of high-risk drug combinations being administered.43 Although this study did not assess DDI-related patient harm, the results suggest that targeting alerts to the context of use may be of benefit. Further research is needed to determine the cost-effectiveness and impact to users (eg, override rates) of tailoring alerts to context.
As an alternative to manual and consensus-based approaches to alert optimisation, researchers and healthcare organisations are increasingly turning to data-driven approaches to improve alert specificity.44 For example, machine learning has been used to predict when medication alerts might be ignored, based on patient and provider characteristics.45 Algorithms to drive DDI alerts have also been developed to incorporate relevant patient-specific and drug information,46 and calls have been made to incorporate more relevant data into algorithms (eg, heart rate) and use large language models to recommend alternative drugs to clinicians.18 These approaches show promise and warrant further investigation, but a recent qualitative study identified a range of significant barriers to healthcare organisations implementing tailored DDI alerts.47 Technical barriers relating to hardware and software were identified (eg, high maintenance requirements), but also barriers related to the users (eg, training) and the organisation (eg, governance).47 To overcome these barriers, the authors suggest demonstrating value from the implementation of tailored DDI alerts.47 Our study contributes to the evidence base for tailored DDI alerts by demonstrating limited value from implementation of simple DDI alerts.
Limitations
This study employed retrospective chart review, so was limited by the information contained in medical records. In particular, low levels of harm may have reflected limited investigation of and documentation of harms in medical records. Our retrospective chart review could not identify whether clinicians considered the risks of DDIs when deciding to prescribe interacting drugs. Our study was powered to detect a 35% reduction in the proportion of admissions with a clinically relevant DDI. Although there was no indication that alerts impacted rates of DDIs (with the observed trend in the opposite direction to that expected), we would need to collect a larger sample to detect smaller changes in clinically relevant DDIs. Our study was also not powered to detect changes in DDI-related harm. Study pharmacists were not blinded to the site (intervention vs control). We used Stockley’s Interaction Checker for DDI identification as it is considered to be the clinical gold standard and has been used as a comparison point for other reference sources.48 49 However, this differed from the DDI knowledge base operating in our sites’ EMR systems, and there may have been differences in potential DDIs as identified by Stockley’s and the DDIs flagged in alerts to clinicians.50 Although we expected DDI alerts to impact clinically relevant DDIs, regardless of the method we used to identify DDIs, we acknowledge the inconsistent DDI knowledge bases as a study limitation. Finally, our study did not collect information on prescribers, such as their roles or experience levels, and excluded ICU patients, who are possibly at higher risk of clinically relevant DDIs.
Conclusion
This controlled study revealed that implementation of a set of DDI alerts, at the highest severity level available within an EMR, is unlikely to reduce harm to patients from DDIs. The majority of potential DDIs, flagged by a typical alert system, were not clinically relevant, creating high levels of noise for end-users and fostering conditions ripe for alert fatigue. DDI-related harms are infrequent. These findings suggest that organisations should reconsider the implementation of simple DDI alerts in EMRs, where significant tailoring of alerts is not possible. Future research should focus on identifying safe, efficient and cost-effective ways of refining DDI alerts, including ways of effectively capturing drug interactions beyond drug pairs, so that expected clinical benefits are achieved, and negative consequences resulting from excessive alerting are minimised.
Supplementary material
Acknowledgements
We would like to thank Ruby Samson, Donna Dorrington, Peter Kennedy, Selvana Awad, Cameron Ballantine, Annim Mohammad, Young Ku and Peter Gates for their support and assistance with this research.
Footnotes
Funding: This study was funded by National Health and Medical Research Council (APP1134824)
Provenance and peer review: Not commissioned; externally peer reviewed.
Patient consent for publication: Not applicable.
Data availability free text: De-identified individual patient data are available for future use on the condition that appropriate ethical approval is obtained.
Ethics approval: Ethics approval was obtained from one participating site’s Human Research Ethics Committee (Reference number: 8/02/21/4.07), and site-specific governance approval obtained from all sites.
Correction notice: This article has been corrected since it was first published online. In the original article, affiliations 6 and 12 were duplicates. In addition to this, the acknowledgements statement has been updated.
Data availability statement
Data are available upon reasonable request.
References
- 1.Centre for Disease Control and Prevention Meaningful use. 2017. https://www.cdc.gov/ehrmeaningfuluse/introduction.html Available.
- 2.Healthcare Information and Management Systems Society Electronic medical record adoption model 2018. 2018. https://www.himssanalytics.org/emram Available.
- 3.Phansalkar S, van der Sijs H, Tucker AD, et al. Drug-drug interactions that should be non-interruptive in order to reduce alert fatigue in electronic health records. J Am Med Inform Assoc. 2013;20:489–93. doi: 10.1136/amiajnl-2012-001089. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Wright A, Aaron S, Seger DL, et al. Reduced Effectiveness of Interruptive Drug-Drug Interaction Alerts after Conversion to a Commercial Electronic Health Record. J Gen Intern Med. 2018;33:1868–76. doi: 10.1007/s11606-018-4415-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Pirnejad H, Amiri P, Niazkhani Z, et al. Preventing potential drug-drug interactions through alerting decision support systems: A clinical context based methodology. Int J Med Inform. 2019;127:18–26. doi: 10.1016/j.ijmedinf.2019.04.006. [DOI] [PubMed] [Google Scholar]
- 6.Baysari MT, Westbrook JI, Richardson KL, et al. The influence of computerized decision support on prescribing during ward-rounds: are the decision-makers targeted? J Am Med Inform Assoc . 2011;18:754–9. doi: 10.1136/amiajnl-2011-000135. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.van der Sijs H, Aarts J, Vulto A, et al. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc. 2006;13:138–47. doi: 10.1197/jamia.M1809. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Humphrey K, Jorina M, Harper M, et al. An Investigation of Drug-Drug Interaction Alert Overrides at a Pediatric Hospital. Hosp Pediatr. 2018;8:293–9. doi: 10.1542/hpeds.2017-0124. [DOI] [PubMed] [Google Scholar]
- 9.Paterno MD, Maviglia SM, Gorman PN, et al. Tiering drug-drug interaction alerts by severity increases compliance rates. J Am Med Inform Assoc. 2009;16:40–6. doi: 10.1197/jamia.M2808. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Simpao AF, Ahumada LM, Desai BR, et al. Optimization of drug-drug interaction alert rules in a pediatric hospital’s electronic health record system using a visual analytics dashboard. J Am Med Inform Assoc. 2015;22:361–9. doi: 10.1136/amiajnl-2013-002538. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Felisberto M, Lima GDS, Celuppi IC, et al. Override rate of drug-drug interaction alerts in clinical decision support systems: A brief systematic review and meta-analysis. Health Informatics J. 2024;30 doi: 10.1177/14604582241263242. [DOI] [PubMed] [Google Scholar]
- 12.Strom BL, Schinnar R, Bilker W, et al. Randomized clinical trial of a customized electronic alert requiring an affirmative response compared to a control group receiving a commercial passive CPOE alert: NSAID--warfarin co-prescribing as a test case. J Am Med Inform Assoc. 2010;17:411–5. doi: 10.1136/jamia.2009.000695. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Strom BL, Schinnar R, Aberra F, et al. Unintended effects of a computerized physician order entry nearly hard-stop alert to prevent a drug interaction: a randomized controlled trial. Arch Intern Med. 2010;170:1578–83. doi: 10.1001/archinternmed.2010.324. [DOI] [PubMed] [Google Scholar]
- 14.Page NJB, Baysari MT, Westbrook JI. Selection and use of decision support alerts in electronic medication management systems in Australian hospitals: a survey of implementers. Pharmacy Practice and Res . 2019;49:142–9. doi: 10.1002/jppr.1479. [DOI] [Google Scholar]
- 15.Baysari MT. Finding fault with the default alert. AHRQ WebM&M; 2013. [Google Scholar]
- 16.Grissinger M. Medication errors involving overrides of healthcare technology. Pa Patient Saf Advis. 2015;12:141–8. [Google Scholar]
- 17.Coroners Court of New South Wales Inquest into the death of paul lau. 2018. http://www.coroners.justice.nsw.gov.au/Documents/Findings%20Paul%20Lau%20Final.pdf Available.
- 18.Amaral A-B, Cuthbertson BH. The efficiency of computerised clinical decision support systems. Lancet. 2024;403:410–1. doi: 10.1016/S0140-6736(23)02839-8. [DOI] [PubMed] [Google Scholar]
- 19.Baysari MT, Van Dort BA, Stanceski K, et al. Is evidence of effectiveness a driver for clinical decision support selection? A qualitative descriptive study of senior hospital staff. Int J Qual Health Care. 2023;35:mzad004. doi: 10.1093/intqhc/mzad004. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Li L, Baker J, Quirk R, et al. Drug-Drug Interactions and Actual Harm to Hospitalized Patients: A Multicentre Study Examining the Prevalence Pre- and Post-Electronic Medication System Implementation. Drug Saf. 2024;47:557–69. doi: 10.1007/s40264-024-01412-w. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb) 2012;22:276–82. [PMC free article] [PubMed] [Google Scholar]
- 22.Baxter K, Preston C. Pharmaceutical Press; 2019. Stockley’s interaction checker.http://www.medicinescomplete.com/ Available. [Google Scholar]
- 23.Mumford V, Raban MZ, Li L, et al. Developing a process to measure actual harm from medication errors in paediatric inpatients: From design to implementation. Br J Clin Pharmacol. 2024;90:1615–26. doi: 10.1111/bcp.16052. [DOI] [PubMed] [Google Scholar]
- 24.Gates PJ, Baysari MT, Mumford V, et al. Standardising the Classification of Harm Associated with Medication Errors: The Harm Associated with Medication Error Classification (HAMEC) Drug Saf. 2019;42:931–9. doi: 10.1007/s40264-019-00823-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.World health organization - uppsala monitoring centre The use of the who-umc system for standardized case causality assessment. 2013. http://www.who.int/medicines/areas/quality_safety/safety_efficacy/WHOcausality_assessment.pdf Available.
- 26.Vonbach P, Dubied A, Krähenbühl S, et al. Prevalence of drug-drug interactions at hospital entry and during hospital stay of patients in internal medicine. Eur J Intern Med. 2008;19:413–20. doi: 10.1016/j.ejim.2007.12.002. [DOI] [PubMed] [Google Scholar]
- 27.Baysari MT, Zheng WY, Li L, et al. Optimising computerised decision support to transform medication safety and reduce prescriber burden: study protocol for a mixed-methods evaluation of drug-drug interaction alerts. BMJ Open. 2019;9:e026034. doi: 10.1136/bmjopen-2018-026034. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Baysari MT, Tariq A, Day RO, et al. Alert override as a habitual behavior - a new perspective on a persistent problem. J Am Med Inform Assoc. 2017;24:409–12. doi: 10.1093/jamia/ocw072. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Baysari MT, Zheng WY, Van Dort B, et al. A Late Attempt to Involve End Users in the Design of Medication-Related Alerts: Survey Study. J Med Internet Res. 2020;22:e14855. doi: 10.2196/14855. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Lowenstein D, Zheng WY, Burke R, et al. Do user preferences align with human factors assessment scores of drug-drug interaction alerts? Health Informatics J. 2020;26:563–75. doi: 10.1177/1460458219840210. [DOI] [PubMed] [Google Scholar]
- 31.Phansalkar S, Zachariah M, Seidling HM, et al. Evaluation of medication alerts in electronic health records for compliance with human factors principles. J Am Med Inform Assoc. 2014;21:e332–40. doi: 10.1136/amiajnl-2013-002279. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Russ AL, Chen S, Melton BL, et al. A Novel Design for Drug-Drug Interaction Alerts Improves Prescribing Efficiency. The Joint Commission Journal on Quality and Patient Safety. 2015;41:396–AP1. doi: 10.1016/S1553-7250(15)41051-7. [DOI] [PubMed] [Google Scholar]
- 33.Tolley CL, Slight SP, Husband AK, et al. Improving medication-related clinical decision support. Am J Health Syst Pharm. 2018;75:239–46. doi: 10.2146/ajhp160830. [DOI] [PubMed] [Google Scholar]
- 34.Van Dort BA, Zheng WY, Baysari MT. Prescriber perceptions of medication-related computerized decision support systems in hospitals: A synthesis of qualitative research. Int J Med Inform. 2019;129:285–95. doi: 10.1016/j.ijmedinf.2019.06.024. [DOI] [PubMed] [Google Scholar]
- 35.Ko Y, Malone DC, Skrepnek GH, et al. Prescribers’ knowledge of and sources of information for potential drug-drug interactions: a postal survey of US prescribers. Drug Saf. 2008;31:525–36. doi: 10.2165/00002018-200831060-00007. [DOI] [PubMed] [Google Scholar]
- 36.Glassman PA, Simon B, Belperio P, et al. Improving recognition of drug interactions: benefits and barriers to using automated drug alerts. Med Care. 2002;40:1161–71. doi: 10.1097/00005650-200212000-00004. [DOI] [PubMed] [Google Scholar]
- 37.Duke JD, Bolchini D. A successful model and visual design for creating context-aware drug-drug interaction alerts. AMIA Annu Symp Proc . 2011;2011:339–48. [PMC free article] [PubMed] [Google Scholar]
- 38.Seidling HM, Klein U, Schaier M, et al. What, if all alerts were specific - estimating the potential impact on drug interaction alert burden. Int J Med Inform. 2014;83:285–91. doi: 10.1016/j.ijmedinf.2013.12.006. [DOI] [PubMed] [Google Scholar]
- 39.USA: 2016. The office of the national coordinator for health information technology. safety assurance factors for ehr resiliance (safer) guide: contingency planning. [Google Scholar]
- 40.Phansalkar S, Desai AA, Bell D, et al. High-priority drug-drug interactions for use in electronic health records. J Am Med Inform Assoc. 2012;19:735–43. doi: 10.1136/amiajnl-2011-000612. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.McGreevey JD, III, Mallozzi CP, Perkins RM, et al. Reducing Alert Burden in Electronic Health Records: State of the Art Recommendations from Four Health Systems. Appl Clin Inform. 2020;11:001–12. doi: 10.1055/s-0039-3402715. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.Bryant AD, Fletcher GS, Payne TH. Drug interaction alert override rates in the Meaningful Use era: no evidence of progress. Appl Clin Inform. 2014;5:802–13. doi: 10.4338/ACI-2013-12-RA-0103. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Bakker T, Klopotowska JE, Dongelmans DA, et al. The effect of computerised decision support alerts tailored to intensive care on the administration of high-risk drug combinations, and their monitoring: a cluster randomised stepped-wedge trial. Lancet. 2024;403:439–49. doi: 10.1016/S0140-6736(23)02465-0. [DOI] [PubMed] [Google Scholar]
- 44.Graafsma J, Murphy RM, van de Garde EMW, et al. The use of artificial intelligence to optimize medication alerts generated by clinical decision support systems: a scoping review. J Am Med Inform Assoc. 2024;31:1411–22. doi: 10.1093/jamia/ocae076. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Liu S, Kawamoto K, Del Fiol G, et al. The potential for leveraging machine learning to filter medication alerts. J Am Med Inform Assoc. 2022;29:891–9. doi: 10.1093/jamia/ocab292. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.Horn J, Ueng S. The Effect of Patient-Specific Drug-Drug Interaction Alerting on the Frequency of Alerts: A Pilot Study. Ann Pharmacother. 2019;53:1087–92. doi: 10.1177/1060028019863419. [DOI] [PubMed] [Google Scholar]
- 47.Zhang T, Gephart SM, Subbian V, et al. Barriers to Adoption of Tailored Drug-Drug Interaction Clinical Decision Support. Appl Clin Inform. 2023;14:779–88. doi: 10.1055/s-0043-1772686. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48.Reis AMM, Cassiani SHDB. Evaluation of three brands of drug interaction software for use in intensive care units. Pharm World Sci. 2010;32:822–8. doi: 10.1007/s11096-010-9445-2. [DOI] [PubMed] [Google Scholar]
- 49.Vonbach P, Dubied A, Krähenbühl S, et al. Evaluation of frequently used drug interaction screening programs. Pharm World Sci. 2008;30:367–74. doi: 10.1007/s11096-008-9191-x. [DOI] [PubMed] [Google Scholar]
- 50.Fung KW, Kapusnik-Uner J, Cunningham J, et al. Comparison of three commercial knowledge bases for detection of drug-drug interactions in clinical decision support. J Am Med Inform Assoc. 2017;24:806–12. doi: 10.1093/jamia/ocx010. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51.WHO Final technical report for the conceptual framework for the international classification for patient safety. 2009
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
Data are available upon reasonable request.