Abstract
Introduction
Medication organizations across the USA have adopted electronic health records, and one of the most anticipated benefits of these was improved medication safety, but alert fatigue has been a major issue.
Objective
We compared the appropriateness of medication-related clinical decision support alerts triggered by two commercial applications: EPIC and Seegnal’s platform.
Methods
This was a retrospective comparison of two commercial applications. We provided Seegnal with deidentified inpatient, outpatient, and inpatient genetic electronic medical record (EMR)-extracted datasets for 657, 2731, and 413 patients, respectively. Seegnal then provided the alerts that would have triggered, which we compared with those triggered by EPIC in clinical care. A random sample of the alerts triggered were reviewed for appropriateness, and the positive predictive value (PPV) and negative predictive value (NPV) were calculated. We also reviewed all the inpatient and outpatient charts for patients within our cohort who were receiving ten or more concomitant medications with alerts we found to be appropriate to assess whether any adverse events had occurred and whether Seegnal’s platform could have prevented them.
Results
Results from EPIC and the Seegnal platform were compared based on alert load, PPV, NPV, and potential adverse events. Overall, compared with EPIC, the Seegnal platform triggered fewer alerts in the inpatient (1697 vs. 27,540), outpatient (2341 vs. 35,134), and inpatient genetic (1493 vs. 20,975) cohorts. The Seegnal platform had higher specificity in the inpatient (99 vs. 0.3%; p < 0.0001), outpatient (99 vs. 0.3%; p < 0.0001), and inpatient genetic (97.9 vs. 1.2%; p < 0.0001) groups and higher sensitivity in the inpatient (100 vs. 68.8%; p < 0.0001) and outpatient (88.6 vs.78.3%; p < 0.0001) groups but not in the inpatient genetic cohort (81 vs. 78.5%; p = 0.11). We identified 16 adverse events that occurred in the inpatient setting, 11 (69%) of which potentially could have been prevented with the Seegnal platform.
Conclusions
Overall, the Seegnal platform triggered 94% fewer alerts than EPIC in the inpatient setting and 93% fewer in the outpatient setting, with much higher sensitivity and specificity. This application could substantially reduce alert fatigue and improve medication safety at the same time.
Supplementary Information
The online version of this article (10.1007/s40264-021-01048-0) contains supplementary material, which is available to authorized users.
Key Points
| The Seegnal platform triggered fewer alerts in all settings. |
| Lower alert rates would reduce alert burden and alert fatigue and improve safety. |
Introduction
Adverse drug events (ADEs) represent a substantial patient safety issue and occur in approximately 1.5 million inpatients every year in the USA [1]. Some studies suggest ADEs account for up to 5–17% of hospital admissions [2–5]. Of the 1.5 million ADEs occurring in the USA annually, approximately 400,000 are considered preventable. Medical errors are among the top ten causes of death in the USA, cause 6–10% of hospital admissions, and result in costs estimated to be more than $US1000 per patient per year [5]. ADEs are defined as untoward medical occurrences associated with the use of drugs that may or may not have been preventable, whereas medical errors are defined as preventable adverse events. More than 20% of patients treated with multiple medications (polypharmacy) experience major drug-related problems such as adverse side effects, lack of efficacy, and unnecessary hospitalizations [6]. One way to reduce the frequency of ADEs is to utilize clinical decision support (CDS) systems.
Overall, CDS systems have been shown to reduce medication errors by 81% in the inpatient setting, although these data came from internally developed applications with highly tuned decision support [6, 7]. Even though nearly all electronic health records (EHRs) at most hospitals include some form of CDS, nearly all this CDS is vendor developed and often does not address patient-specific factors. CDS represents an effective way to reduce errors and ADEs [8–10]. However, this impact may be decreased or even extinguished if too many clinically inappropriate alerts are given [11–13]. One study reported that approximately 60% of overrides of alerts were appropriate and that override rates varied based on type. For example, override rates were 98% for duplicate medication alerts, 96.5% for drug allergy alerts, 82.5% for nonformulary medication alerts, 26.4% for age-based medication substitution alerts, and just 2.2% for renal alerts [14]. Many other studies have found even higher override rates [15, 16].
Alert fatigue introduces the risk of missing critical alerts that may compromise patient safety [17–20]. A study performed at our institution using a legacy home-grown EHR system found that inappropriately overridden CDS alerts were associated with an increased risk of ADEs [21]. Nearly all major US healthcare organizations have now switched to commercial EHRs, with EPIC and Cerner being the market leaders, and organizations generally contract for medication decision support with a third-party company, with three companies controlling nearly all of the market: FDB, Medispan, and Multum (which is owned by Cerner). In such cases, the EHR imposes some logic on top of the linked database, and the combined capacities compose the CDS solution.
In this study, we calculated the positive predictive value (PPV), negative predictive value (NPV), sensitivity, and specificity, and overall alert load generated by the Seegnal platform compared with those generated by EPIC as implemented within our healthcare system.
Methods
Technology Assessed
Seegnal (Tel Aviv, Israel) has developed an application called the Seegnal platform. This platform is based on a multitier mechanism that relies on proprietary, real-time algorithm-derived alert personalization, Hansten & Horn’s knowledge-based clinical drug interaction evaluations, the University of Washington’s pharmacokinetic and pharmacogenetic databases, and FDB data. The Seegnal platform assesses a wide scope of relevant drug-related problem data including drug–drug interactions, drug allergy, drug dosing, drugs in pregnancy/lactation, drugs and smoking, drug–kidney/liver function interactions, pharmacokinetic drug interactions, drug–food/herbals/vitamin interactions, and pharmacogenetic interactions. The Seegnal platform simultaneously assesses multiple patient-specific parameters (e.g., drug dose/administration route, renal function, laboratory values, electrocardiogram, smoking), and medication-specific parameters (e.g., potency, dose–response curves, time to steady state, accumulative dosing, combined effects of multiple [more than two] medications) in real time to determine the need to trigger an alert or bypass the potential alert for the specific patient. The Seegnal platform also has an innovative user interface that enables the provider at the point of care to detect, prioritize, and choose an appropriate alternative. The Seegnal platform’s user interface was not assessed in this evaluation. To date, one study has been conducted using patient-specific parameters to alert on drug interactions compared with a customized vendor-based CDS database [22]. No study has directly evaluated the alert load produced by the Seegnal platform compared with those of other vendor-based CDS.
Study Site and Data
This study was performed at Brigham and Women’s Hospital (BWH), Boston, MA, USA, a large urban academic medical center that uses a leading vendor EHR system, EPIC (Verona, WI, USA). BWH provided Seegnal with deidentified inpatient, outpatient, and inpatient genetic electronic medical record-extracted datasets of 657, 2731, and 413 patients, respectively. The dataset contained 56 predefined structured (coded) parameters that were potentially documented in the patients’ EHR during their hospitalization period and/or outpatient visits between 3 June 2018 and 9 June 2018. The dataset for inpatients with genetic data contained all data files during any inpatient admission between 1 January 2017 and 31 December 2019 as only a small portion of patients had genetic data available. The inpatient genetic records were identified by Partners Healthcare Biobank. The purpose of the Partners Biobank is to help researchers understand how health is affected by genes, lifestyle, and environment. Patients who consent to participate in the Partners Biobank contribute a DNA sample for genetic testing, including pharmacogenomics testing. During our data pull, we identified patients with mutations at cytochrome P450 (CYP)-3A5. Examples of parameters extracted were patients’ medication list with start and end dates, dose, frequency, route of administration, comments if any, active problem list, smoking status, pregnancy status, and various laboratory results such as potassium, creatinine, liver function tests, and so on. Specific patient modifiers/factors/data elements extracted from the EHR are listed in the electronic supplementary material.
Seegnal then ran the retrospective dataset against the Seegnal platform. The output from the Seegnal platform provided the alerts that would have been triggered in only the Seegnal platform, in both the Seegnal platform and EPIC, and only in EPIC as utilized within BWH. The alerts were then classified into predefined categories and reviewed for appropriateness to the specific patients. Two pharmacists independently reviewed a random sample of up to 100 alerts per each of the predefined categories (977 inpatient alerts, 808 inpatient genetic alerts, and 938 outpatient alerts, for a sum of 2723 alerts) but were blinded as to whether the alert originated from only the Seegnal platform, both the Seegnal platform and EPIC, or only EPIC.
The types of alerts reviewed included disease–drug contraindication, dosing, drug–drug interaction, duplicate therapy, geriatric, most probable side effects, renal dosing, pregnancy, pharmacokinetic drug interactions, and pharmacokinetic drug interactions with the patient parameter of smoking. The pharmacists (SS, DS) then compared their review; for any differences, two additional reviewers (DB, a doctor with extensive experience in patient safety, and JH, a pharmacist with extensive experience with drug–drug interactions) made the final decision as to whether or not the alert was appropriate.
Analysis
Once the random sample alerts were evaluated, we then calculated the PPV (true positive/(true positive + false positive)), NPV (true negative/(true negative + false negative)), specificity (true negative/(true negative + false positive)), sensitivity (true positive/(true positive + false negative)), and overall interruptive alert load generated by the Seegnal platform compared with those of EPIC as it functions at BWH. True positives were alerts considered appropriate for the patient, false positives were alerts that were not appropriate for the patient, true negatives were alerts we agreed should not be triggered, and false negatives were alerts that should have triggered but did not. Examples of each are provided in Table 1. Analysis was performed in SAS v. 9.4 (SAS Institute, Cary, NC, USA), and significance was set using a two-sided p value of < 0.05.
Table 1.
Examples of true positive, false positive, true negative, and false negative alerts
| Example type | Alert type | Interaction synopsis |
|---|---|---|
| True positive—agree with alert | Drug–drug interactions | Ondansetron + ciprofloxacin + patient’s corrected QT interval is 550 |
| False positive—disagree with alert | Renal | Cephalexin 250 mg capsule every 6 h and creatinine clearance is 43 |
| True negative—agree not to alert | Drug disease | Acute kidney failure, unspecified on patient’s active problem list + acetaminophen 500 mg tablet but patient's creatinine clearance is normal |
| False negative—disagree not to alert | Dosing | 87-year-old patient ordered to start zolpidem 10 mg daily in the outpatient setting |
True Positive (TP) – agree with alert
False Positive (FP) – disagree with alert
True Negative (TN) – agree not to alert
False Negative (FN) – disagree not to alert
We excluded evaluation of the appropriateness of the genetic alerts returned from the Seegnal platform from our final inpatient and genetic sample because we only had data for one pharmacogenetic marker (CYP3A5), so very few alerts fired. We evaluated all the other alerts for the inpatient genetic cohort. We also reviewed all the inpatient/inpatient genetic (n = 69) and outpatient (n = 65) charts for patients within our cohort who were receiving ten or more concomitant medications with alerts we found to be appropriate to assess whether any adverse events had occurred and whether the Seegnal platform could have prevented them.
Results
Patient Demographics
The inpatient cohort consisted of 657 patients with 665 admissions; eight patients had two admissions. The inpatient genetic cohort consisted of 413 patients with 413 visits. The outpatient cohort consisted of 2731 patients with 2749 visits; 18 patients had two visits during the study period. Demographics of all patients can be found in Table 2.
Table 2.
Patient demographics
| Demographic | Inpatient cohort | Inpatient genetic | Outpatient |
|---|---|---|---|
| Mean age, years | 61 | 58 | 55 |
| Sex | |||
| F | 328 (49.9) | 232 (56.2) | 1831 (67.0) |
| M | 329 (50.1) | 181 (43.8) | 900 (33.0) |
| Race | |||
| American Indian or Alaska Native | 4 (0.6) | 0 | 2 (0.1) |
| Asian | 23 (3.5) | 12 (2.9) | 108 (4.0) |
| Black or African American | 67 (10.2) | 156 (37.8) | 358 (13.1) |
| Declined | 5 (0.8) | 8 (1.9) | 58 (2.1) |
| Hispanic or Latino | 17 (2.6) | 30 (7.3) | 364 (13.3) |
| Native Hawaiian or other Pacific Islander | 0 | 0 | 2 (0.1) |
| Other | 33 (5.0) | 23 (5.6) | 210 (7.7) |
| Unavailable | 6 (0.9) | 1 (0.2) | 40 (1.5) |
| White or Caucasian | 502 (76.4) | 183 (44.3) | 1589 (58.2) |
| Ethnicity | |||
| Hispanic | 51 (7.8) | 61 (14.8) | 576 (21.0) |
| Non-Hispanic | 594 (90.4) | 344 (83.3) | 1673 (61.3) |
| Unavailable | 12 (1.8) | 8 (1.9) | 482 (17.6) |
Data are presented as N (%) unless otherwise indicated
Summary of Main Outcomes
In the inpatient setting, the Seegnal platform triggered 1697 alerts, which was 94% less than the EPIC platform, which triggered 27,540 alerts (Table 3). Sensitivity was 100% for the Seegnal platform and 68.8% for EPIC. In the inpatient genetic group (Table 3), the Seegnal platform triggered 1493 inpatient alerts and EPIC triggered 20,975. The sensitivities were similar: 81% for the Seegnal platform and 78.5% for EPIC. In the outpatient setting (Table 3), the Seegnal platform triggered 93% fewer alerts and sensitivity was 88.6%, whereas sensitivity was 78.3% for EPIC.
Table 3.
Inpatient alerts, inpatient genetic alerts, and outpatient alerts
| Cohort | The Seegnal platform | EPIC (control group) | p value |
|---|---|---|---|
| Inpatient | |||
| Number of patients | 657 | 657 | NA |
| Number of admissions | 665 | 665 | NA |
| Interruptive alerts | 1697 | 27,540 | NA |
| Alert load | 6% | 100% | NA |
| True positives | 1422 | 979 | NA |
| False positives | 275 | 26,561 | NA |
| True negatives | 26,359 | 72 | NA |
| False negatives | 0 | 444 | NA |
| PPV | 83.81% | 3.55% | < 0.0001 |
| NPV | 100% | 14.01% | < 0.0001 |
| Sensitivity | 100% | 68.8% | < 0.0001 |
| Specificity | 99% | 0.3% | < 0.0001 |
| Inpatient genetic | |||
| Number of patients | 413 | 413 | NA |
| Number of admissions | 413 | 413 | NA |
| Interruptive alerts | 1493 | 20,975 | NA |
| Alert load | 7% | 100% | NA |
| True positives | 1068 | 1034 | NA |
| False positives | 425 | 19,941 | NA |
| True negatives | 19,748 | 232 | NA |
| False negatives | 250 | 284 | NA |
| PPV | 71.51% | 4.93% | < 0.0001 |
| NPV | 99% | 44.98% | < 0.0001 |
| Sensitivity | 81% | 78.5% | 0.11 |
| Specificity | 97.9% | 1.2% | < 0.0001 |
| Outpatient | |||
| Number of patients | 2731 | 2731 | NA |
| Number of admissions | 2749 | 2749 | NA |
| Interruptive alerts | 2341 | 35,134 | NA |
| Alert load | 7% | 100% | NA |
| True positives | 1924 | 1700 | NA |
| False positives | 407 | 33,424 | NA |
| True negatives | 33,134 | 117 | NA |
| False negatives | 248 | 472 | NA |
| PPV | 82.54% | 4.84% | < 0.0001 |
| NPV | 99% | 19.84% | < 0.0001 |
| Sensitivity | 88.6% | 78.3% | < 0.0001 |
| Specificity | 99% | 0.3% | < 0.0001 |
NA not applicable, NPV negative predictive value, PPV positive predictive value
Inpatient and Outpatient Chart Review
We identified 16 adverse events that occurred in the inpatient setting. Of these, the Seegnal platform potentially would have prevented 11 (68.8%). Examples included patients who developed hypotension, bradycardia, QTc prolongation, increased combined side effects, and renal dysfunction. One adverse event involved a patient with a history of nausea and vomiting from oral potassium chloride, which might have been prevented if it had been documented in a structured format under allergies/intolerances instead of being included in the free-text chart notes. Overall, reviewers judged that four (25.0%) adverse events were nonpreventable, so no CDS could have prevented them, and one adverse event was possibly preventable. Of the 11 preventable adverse events that did occur, nine were judged serious and two were considered significant (Table 4). We found no adverse events in the outpatient setting.
Table 4.
Description of adverse events identified by chart review
| ADE description | Potentially preventable with the Seegnal platform | ADE severity |
|---|---|---|
| Intermittent delirium due to olanzapine and lorazepam while in ICU | Yes, using the most probable side effects mode | Serious |
| Excessive daytime somnolence and delirium due to multiple sedating medications (trazodone, haloperidol, quetiapine) | Yes, using the most probable side effects mode | Serious |
| Hyponatremia that improved after eslicarbazepine dose was lowered | Yes, this drug can cause hyponatremia, and the platform monitors patients’ laboratory results and would have recommended dose reduction when 1200 mg was first given | Significant |
| QTc on admission was 519 and went up to 614 after receiving ciprofloxacin and ondansetron | Yes, even though the MD received the DDI alert for ciprofloxacin and ondansetron, the platform would have alerted that QTc levels were already high when medication was ordered | Serious |
| Coagulopathy, most likely secondary to interaction between amiodarone and warfarin | Yes, even though the MD received the DDI alert for amiodarone and warfarin, the platform would have alerted once the patient's laboratory results started showing signs of coagulopathy | Serious |
| Increased tacrolimus levels due to DDI with voriconazole | Yes, even though the MD was aware of the DDI between tacrolimus and voriconazole, the platform would have alerted once tacrolimus levels started to increase | Significant |
| Decreased blood pressure to 84/44 and heart rate to 50s while receiving amlodipine, lisinopril, and metoprolol ER | Yes, by monitoring patient parameters and knowing the patient was on multiple medications that could decrease blood pressure and heart rate | Serious |
| Increased QTc for 5 days—had DDI alert for ciprofloxacin and tacrolimus | Yes, the platform would have identified all medications that increase QTc and alerted when QTc started to increase | Serious |
| Hypotension due to epidural | Yes, the platform would have alerted once the patient's blood pressure started to decrease | Serious |
| Bradycardia for 2 days due to propranolol | Yes, the platform would have alerted once the patient’s heart rate started decreasing | Serious |
| Increase QTc due to fluconazole; hypotension and mental status changes due to multiple medications | Yes, the platform can identify all medications that increase QTc and cause hypotension and mental status change and also alert based on the patient's specific parameters (QTc, blood pressure) | Serious |
| Contrast (Omnipaque)-induced nephropathy. Renal function was ok prior to receiving contrast | No, this ADE was not preventable because the patient’s renal status was ok prior to receiving the contrast | Serious |
| Severe chemotherapy-induced mucositis requiring TPN | No, this ADE was not preventable | Serious |
| Postoperative hypotension, thought to be a combination of hypovolemia and perioperative ACEI use | No, this ADE was not preventable, as it occurred during surgery. Patient was taken from the OR to the PACU then SICU in setting of postoperative hypotension requiring phenylephrine infusion | Life threatening |
| Naloxone needed after surgery | No, this ADE was not preventable; it occurred during surgery | Serious |
| Nausea due to oral KCL—admitted for hypokalemia with history of not tolerating oral KCL due to nausea and vomiting | Maybe; patient had a history of this ADE at admission, and the platform would have alerted if it was documented in a structured format instead of just notes | Significant |
ACEI angiotensin-converting enzyme inhibitor, ADE adverse drug event, DDI drug–drug interaction, ER extended release, ICU intensive care unit, KCL potassium chloride, MD physician, OR operating room, PACU post-anesthesia care unit, QTc corrected QT interval, SICU surgical intensive care unit, TPN total parenteral nutrition
Discussion
We evaluated the alerting performance of the Seegnal platform compared with that of the EPIC combination and found that the Seegnal platform would have triggered nearly twentyfold fewer alerts, with higher sensitivity and higher specificity in both the inpatient and the outpatient setting. At the same time, the Seegnal platform potentially could have prevented a small number of adverse events associated with issues for which EPIC did not alert.
The high alert burden was consistent with other recent literature reporting very high override rates, mainly because of the inappropriate triggering of alerts [15, 19–21]. Not only did the Seegnal platform have a much lower alert load but it also had higher specificity, higher sensitivity, better PPV, and better NPV than EPIC in both the inpatient and the outpatient setting. It achieved this by utilizing patient-specific parameters that can reduce inappropriate alerting and diminish alert fatigue.
EHRs linked with CDS have been demonstrated to prevent many types of medication errors in an internally developed application [23]. However, nearly all systems implemented in the USA are now commercial, and other studies suggest these are not yet as effective as some of the earlier implementations [21]. The potential benefit of CDS is diminished by inappropriate alerting [11–13]. Studies have also shown that CDS can have unintended consequences. One study found that CDS contained incorrect or misleading CDS content or out-of-date content [23]. Another study also found alert fatigue and user unfriendly systems with limited ability to customize databases or output to be major problems [24]. The frequency of inappropriate alerts leads clinicians to ignore nearly all alerts, even those that can lead to patient harm. The Seegnal platform employs a CDS knowledge base that is patient specific and flexible and can be tailored to an institution’s specific requirements. This reduces the inappropriate alert load and alert fatigue. Overall, the Seegnal platform outperformed our current CDS system.
Other countries have taken a different approach for medication-related decision support. Notably, the Dutch use a database called G-Standaard, which is issued monthly by Z-Index, an organization owned by the Royal Dutch Association for the Advancement of Pharmacy [25]. This database has been refined to reduce some of the false-positive alerts but is not used outside the Netherlands [26].
Many studies have found that, when alerts are overridden inappropriately, the chance of harm to a patient increases significantly [17–21]. This study identified 16 adverse events that occurred in the inpatient setting. A retrospective chart review found that 11 of the 16 alerts would have potentially been prevented by the Seegnal platform, whereas the current system failed to catch these.
This study has several limitations. Because the design was retrospective, many of the features of the Seegnal platform, for example using electronic medication administration record data to assess the number of medication doses a patient had received, could not be assessed. The tool might perform better in real time with access to these additional features. The Seegnal platform has not been subjected to the customization/configuration of a healthcare facility, which could also improve or worsen performance. As the genetic information available within our biobank was limited at the time of data extraction, we were unable to fully evaluate the pharmacogenetic decision support. We did not measure how clinicians would respond to suggested alternatives to the identified drug-related problems; that will need to be assessed prospectively. Specifically, the platform should be tested in real time to assess the alert load reduction, user interface, and performance characteristics.
Conclusion
We evaluated two commercial applications regarding their performance around medication-related CDS and found that the Seegnal platform substantially outperformed EPIC in alert load, specificity, sensitivity, PPV, and NPV in both the inpatient and the outpatient setting. It would reduce the alert load by nearly twentyfold. The performance of the Seegnal platform is attributed mostly to its ability to personalize alerting because of its real-time consideration of up to hundreds of patient-specific and medication-specific factors. It could also help notify physicians of a small number of new potential ADEs, and—based on the chart review—69% of these potentially could be prevented. This application should be further evaluated in real time, and further evaluation of pharmacogenetic alerts would also be beneficial.
Supplementary Information
Below is the link to the electronic supplementary material.
Author contributions
SNS and DLS made substantial contributions to the conception of the data analysis and interpretation of the data. SNS and DLS contributed to writing and revision of the manuscript. DWB supervised the research design and contributed to the writing and revision of the manuscript. JRH assisted with revision of the manuscript. JMF contributed to data collection. All authors give approval for the final version to be published and agree to be accountable for all aspects of the working in answering questions related to the accuracy or integrity of any part of the work.
Declarations
Funding
This research was supported by Seegnal (Sayvon, Israel), which had no role in the development of the manuscript.
Conflict of interest
John Horn is a consultant to Seegnal US, Inc. and Urovant Sciences, Inc. and is co-author and publisher of The Top 100 Drug Interactions: A Guide to Patient Management. Dr. Bates consults for EarlySense, which makes patient safety monitoring systems; receives cash compensation from CDI (Negev), Ltd, which is a not-for-profit incubator for health information technology startups; receives equity from ValeraHealth, which makes software to help patients with chronic diseases; receives equity from Clew, which makes software to support clinical decision making in intensive care; receives equity from MDClone, which takes clinical data and produces deidentified versions of it; receives equity from AESOP, which makes software to reduce medication error rates; and receives research funding from IBM Watson Health. SN Shah, DL Seger, and JM Fiskio have no conflicts of interest that are directly relevant to the content of this article.
Ethics approval
This was approved by Partners Healthcare IRB.
Consent to Participate
Not Applicable.
Consent for Publication
Not Applicable.
Availability of Data and Material
Not Applicable.
Code Availability
Not Applicable.
References
- 1.Institute of Medicine . Evidence-based medicine and the changing nature of health care: 2007 IOM annual meeting summary. Washington, DC: National Academies Press; 2008. [PubMed] [Google Scholar]
- 2.Beijer HJ, de Blaey CJ. Hospitalizations caused by adverse drug reactions (ADR): a meta-analysis of observational studies. Pharm World Sci. 2002;24(2):46–54. doi: 10.1023/A:1015570104121. [DOI] [PubMed] [Google Scholar]
- 3.Lazarou J, Pomeranz BH, Corey PN. Incidence of adverse drug reactions in hospitalized patients: a meta-analysis of prospective studies. JAMA. 1998;279:1200–1205. doi: 10.1001/jama.279.15.1200. [DOI] [PubMed] [Google Scholar]
- 4.Budnitz DS, Pollock DA, Weidenbach KN, et al. National surveillance of emergency department visits for outpatient adverse drug events. JAMA. 2006;296(15):1858–1866. doi: 10.1001/jama.296.15.1858. [DOI] [PubMed] [Google Scholar]
- 5.Makary MA, Daniel M. Medical error- the third leading cause of death in the US. BMJ. 2016;353:i2139. doi: 10.1136/bmj.i2139. [DOI] [PubMed] [Google Scholar]
- 6.Corsonello A, Pedone C, Corica F, et al. Concealed renal insufficiency and adverse drug reactions in elderly hospitalized patients. Arch Intern Med. 2005;165:790–795. doi: 10.1001/archinte.165.7.790. [DOI] [PubMed] [Google Scholar]
- 7.Bates DW, Leape LL, Cullen DJ, et al. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA. 1998;280:1311–1316. doi: 10.1001/jama.280.15.1311. [DOI] [PubMed] [Google Scholar]
- 8.Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med. 2003;163(12):1409–1416. doi: 10.1001/archinte.163.12.1409. [DOI] [PubMed] [Google Scholar]
- 9.Nuckols TK, Smith-Spangler C, Morton SC, et al. The effectiveness of computerized order entry at reducing preventable adverse drug events and medication errors in hospital settings: a systematic review and meta-analysis. Syst Rev. 2014;3:56. doi: 10.1186/2046-4053-3-56. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Shamliyan TA, Duval S, Du J, Kane RL. Just what the doctor ordered. Review of the evidence of the impact of computerized physician order entry system on medication errors. Health Serv Res. 2007;43:32–35. doi: 10.1111/j.1475-6773.2007.00751.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Wolfstadt JI, Gurwitz JH, Field TS, et al. The effect of computerized physician order entry with clinical decision support on the rates of adverse drug events: a systematic review. J Gen Intern Med. 2008;23(4):451–458. doi: 10.1007/s11606-008-0504-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Bates DW, Teich JM, Lee J, et al. The impact of computerized physician order entry on medication error prevention. J Am Med Inform Assoc. 1999;6:313–321. doi: 10.1136/jamia.1999.00660313. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Lin CP, Payne TH, Nichol P, Hoey PJ, Anderson CL, Gennari JH. Evaluating clinical decision support systems: monitoring CPOE order check override rates in the Department of Veterans Affairs’ Computerized Patient Record System. J Am Med Inform Assoc. 2008;15:620–626. doi: 10.1197/jamia.M2453. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Nanji KC, Slight SP, Seger DL, et al. Overrides of medication-related clinical decision support alerts in inpatients. J Am Med Inform Assoc. 2018;25(5):476–481. doi: 10.1093/jamia/ocx15. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Nanji KC, Slight SP, Seger DL, et al. Overrides of medication-related clinical decision support alerts in outpatients. J Am Med Inform Assoc. 2014;21(03):487–491. doi: 10.1136/amiajnl-2013-001813. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Kuperman GJ, Bobb A, Payne TH, et al. Medication-related clinical decision support in computerized provider order entry systems: a review. J Am Med Inform Assoc. 2007;14(01):29–40. doi: 10.1197/jamia.M2170. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Payne TH, Nichol WP, Hoey P, Savarino J. Characteristics and override rates of order checks in a practitioner order entry system. Proc AMIA Symp. 2002;2002:602–606. [PMC free article] [PubMed] [Google Scholar]
- 18.Ancker JS, Edwards A, Nosal S, Hauser D, Mauer E, Kaushal R, with the HITEC Investigators Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Med Inform Decis Mak. 2017;17:36. doi: 10.1186/s12911-017-0430-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Weingart SN, Toth M, Sands DZ, Aronson MD, Davis RB, Phillips RS. Physicians' decisions to override computerized drug alerts in primary care. Arch Intern Med. 2003;163(21):2625–2631. doi: 10.1001/archinte.163.21.2625. [DOI] [PubMed] [Google Scholar]
- 20.Hsieh TC, Kuperman GJ, Jaggi T, et al. Characteristics and consequences of drug allergy alert overrides in a computerized physician order entry system. J Am Med Inform Assoc. 2004;11(06):482–491. doi: 10.1197/jamia.M1556. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Wong A, Amato MG, Seger DL, et al. Evaluation of medication-related clinical decision support alert overrides in the intensive care unit. J Crit Care. 2017;39:156–161. doi: 10.1016/j.jcrc.2017.02.027. [DOI] [PubMed] [Google Scholar]
- 22.Horn J, Ueng S. The effect of patient-specific drug-drug interaction alerting on the frequency of alerts: a pilot study. Ann Pharmacother. 2019;53(11):1087–1092. doi: 10.1177/1060028019863419. [DOI] [PubMed] [Google Scholar]
- 23.Bates DW, Leape LL, Cullen DJ, et al. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA. 1998;280(15):1311–1316. doi: 10.1001/jama.280.15.1311. [DOI] [PubMed] [Google Scholar]
- 24.Ash JS, Sitting DF, Campbell EM, Guappone KP, Dykstra RH. Some unintended consequences of clinical decision support systems. AMIA Annu Symp Proc. 2007;2008:26–30. [PMC free article] [PubMed] [Google Scholar]
- 25.The G-Standaard: the medicines standard in healthcare. KNMP.nl apothekersorganisatie. https://www.knmp.nl/producten/gebruiksrecht-g-standaard/informatie-over-de-g-standaard/the-g-standaard-the-medicines-standard-in-healthcare. Accessed 20 Dec 2020.
- 26.Coleman JJ, van der Sijs H, Haefeli WE, et al. On the alert: future priorities for alerts in clinical decision support for computerized physician order entry identified from a European workshop [published correction appears in BMC Med Inform Decis Mak. 2013;13:122]. BMC Med Inform Decis Mak. 2013;13:111. 10.1186/1472-6947-13-111 [DOI] [PMC free article] [PubMed]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.