Skip to main content
PMC home page
Journal of the American Medical Informatics Association : JAMIA logoLink to Journal of the American Medical Informatics Association : JAMIA
. 2020 Nov 8;28(1):167–176. doi: 10.1093/jamia/ocaa230

How effective are electronic medication systems in reducing medication error rates and associated harm among hospital inpatients? A systematic review and meta-analysis

Peter J Gates 1,✉, Rae-Anne Hardie 1, Magdalena Z Raban 1, Ling Li 1, Johanna I Westbrook 1
PMCID: PMC7810459  PMID: 33164058

Abstract

Objective

To conduct a systematic review and meta-analysis to assess: 1) changes in medication error rates and associated patient harm following electronic medication system (EMS) implementation; and 2) evidence of system-related medication errors facilitated by the use of an EMS.

Materials and Methods

We searched Medline, Scopus, Embase, and CINAHL for studies published between January 2005 and March 2019, comparing medication errors rates with or without assessments of related harm (actual or potential) before and after EMS implementation. EMS was defined as a computer-based system enabling the prescribing, supply, and/or administration of medicines. Study quality was assessed.

Results

There was substantial heterogeneity in outcomes of the 18 included studies. Only 2 were strong quality. Meta-analysis of 5 studies reporting change in actual harm post-EMS showed no reduced risk (RR: 1.22, 95% CI: 0.18–8.38, P = .8) and meta-analysis of 3 studies reporting change in administration errors found a significant reduction in error rates (RR: 0.77, 95% CI: 0.72–0.83, P = .004). Of 10 studies of prescribing error rates, 9 reported a reduction but variable denominators precluded meta-analysis. Twelve studies provided specific examples of system-related medication errors; 5 quantified their occurrence.

Discussion and Conclusion

Despite the wide-scale adoption of EMS in hospitals around the world, the quality of evidence about their effectiveness in medication error and associated harm reduction is variable. Some confidence can be placed in the ability of systems to reduce prescribing error rates. However, much is still unknown about mechanisms which may be most effective in improving medication safety and design features which facilitate new error risks.

Keywords: medication error, prescribing error, administration error, patient harm, health information technology

INTRODUCTION

In an effort to rapidly amplify and focus activities leading to improvements in medication safety, in 2017 the World Health Organization (WHO) announced the third Global Patient Safety Challenge (GPSC) as “Medication Without Harm.”1 To meet the WHO’s target of reducing the prevalence of medication errors causing severe patient harm by half within 5 years, it is essential for hospitals to introduce interventions with clear evidence of effectiveness. The Institute of Medicine has promoted the adoption of electronic medication systems (EMS) (also called electronic prescribing systems or computerized provider order entry systems) for well over a decade, and, in the US, incentive payments have been available for the meaningful use of electronic health records.2

Systematic reviews of the impact of EMS across inpatient settings have reported significant reductions in medication error rates,3–15 as well as improvements to practitioner workflow and adherence to guideline-based care.16–21 However, these reviews have typically focused only on prescribing errors and omitted administration errors. Existing systematic reviews have also failed to draw attention to system-related errors (SREs), which have been cited to be frequent with electronic systems yet difficult to detect.22 Moreover, reviews have consistently highlighted the low quality of individual studies.10,23

Several systematic reviews have assessed the impact of EMS on medication errors that cause actual patient harm (ie, preventable adverse drug events or pADEs) or potential harm to patients. The assessed outcomes have included reported rates of pADEs;4–6,10,13,17,24–26 average error severity ratings (describing potential harm);11 average lengths of patient stay (LOS);15,18,21,27 and/or hospital mortality.6,15,18,24,27 However, these reviews often include studies relying upon incident reports, which have been shown to underestimate error rates.28

Further, many previous reviews have only included studies that were specific to a ward type (eg, ICU). Only 3 reviews have provided a meta-analysis of study results, and all focused on intensive care settings.5,6,15 Two of these 3 reviews reported no significant change in measures of medication-related patient harm.6,15 The remaining review included studies published prior to 2014 and reported a significant reduction in the rate of pADEs among adult inpatients (pooled risk ratio 47%; based on 6 studies, 2 of which were conducted prior to 2000).5

Synthesized evidence about the impact of introducing EMS on the rate of harmful medication errors across hospital inpatient groups (eg, for those on different wards, with different clinical conditions or age ranges) is unclear from the available evidence in systematic reviews. This highlights the need to collate quality research to determine how effective these systems may be at contributing to the current WHO GPSC of Medication Without Harm. Thus, our aims were to conduct a systematic review and meta-analysis of studies to assess: 1) changes in the rates of medication errors and any associated patient harm following the introduction of an electronic medication system; and 2) evidence of medication errors facilitated by the use of an EMS.

MATERIALS AND METHODS

Search strategy

We searched the Embase and Medline (via Ovid), and the Cumulative Index to Nursing and Allied Health Literature (CINAHL; via EBSCOhost) databases for English-language literature published between January 2005 and March 2019. Our search strategy used a combination of MeSH terms and keywords related to the intervention (computerized provider/physician order entry and clinical decision support [CDS]) and outcomes of interest (medication errors, LOS, mortality, and patient harm). We also “hand-searched” the reference lists of full-text articles that were assessed for potential inclusion. The complete search strategy was guided by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement, including registration with PROSPERO (registration number CRD42018080778), and is provided in Supplementary Appendix SA.

Study selection

We executed our search strategy on 25 March 2019 and found 2936 citations. After removing duplicates, 2 researchers (PG, RH) screened the titles and abstracts of citations. All potential articles identified from each independent assessment were assigned to full-text review. The same 2 researchers then reviewed the full-text articles and discussed reasons for inclusion or exclusion and met to discuss disagreements and gain consensus. Multiple reports of the same study were assessed for unique data. No single report of multiple studies was found. The study selection process is presented in Figure 1.

Figure 1.

Figure 1.

Open in a new tab

Study selection process.

We defined an electronic medication system (EMS) as a computer-based system for electronic prescribing and/or administration of medicines. Systems with or without detailed CDS and those integrated or not with larger hospital electronic health record systems were included. All types of study designs conducted in single or multiple wards were included if they reported on: changes in medication error rates and/or medication error-related harm before and after the introduction of an EMS in hospitals that previously used paper-based ordering; or cross-sectional studies of a single hospital that randomly allocated patients to a ward using paper-based ordering or to an EMS.

Studies were required to report data in at least 1 of 4 outcomes:

  1. rate of medication errors (ME), defined as the combination of error subtypes including, at a minimum, prescribing errors (PE) and medication administration errors (MAE);

  2. rate of PE, where the study definition of PE included a minimum of the 2 most frequent clinical error types of wrong dose and wrong time;29

  3. rate of MAE, where the study definition of MAE included multiple types of administration errors that met Shawahna et al’s30 definition of: “any deviation from the prescriber’s medication order as [directly] observed or [indirectly identified] on the patient’s chart, manufacturers’ preparation/administration instructions, or relevant institutional policies”;

  4. rate of potential or actual patient harm associated with ME, PE, and/or MAE. These studies reported on harm outcomes including error severity ratings (typically associated with potential harm, for example the Index published by the National Co-ordinating Council for Medication Error Reporting and Prevention [NCC-MERP]31), number of pADEs (actual harm), LOS, and/or mortality rate (actual harm).

Exclusion criteria were:

  1. studies solely using incident reports to identify errors (due to the known underreporting of errors by incident reports).28 Studies identifying PE by chart review and MAE by observation were included;

  2. studies where EMS was introduced in conjunction with an additional intervention at the same time which may have confounded results. Some examples of these excluded studies were EMS implemented in parallel with an automated unit-dose trolley,32 or dispensing system,33 patient barcoding,34 and education-based stewardship programs.35 We included studies which introduced system training or workflow optimizations to compliment EMS implementation and describe these where relevant in Supplementary Appendix SB;

  3. studies that were highly selective or specialized. For example, studies which focused only on specific medications,36,37 or medication groups such as chemotherapy;38 as well as studies of a specific medication error type;39

  4. studies only available as abstracts, posters, editorials, or opinion pieces;

  5. studies conducted outside of inpatient settings (eg, outpatients, simulation, or primary care clinics);

  6. studies without data from a paper-based setting (eg, the introduction of clinical decision support to an existing electronic system).

Data extraction

We extracted study details including study author(s), year of publication, country in which the study was conducted, type of hospital, study design, full description of the EMS, duration, and timing of data collection and system implementation. For studies that measured medication errors and/or prescribing errors, we extracted error definitions, number of patients, patient days, orders, and administrations, as well as number of errors or error rates at baseline and intervention. For studies that measured administration errors, we also extracted number of opportunities for error (OE, defined as all doses administered plus any doses omitted that could be classified as either correct or incorrect), and errors per OE, at baseline and intervention. For studies that measured mortality, we extracted number of patient deaths in hospital at baseline and post-EMS. For studies that measured LOS, we extracted mean or median LOS before and after EMS. For studies that measured patient harm, we extracted the number of preventable adverse drug events, or mean severity ratings, before and after intervention. We also extracted any quantitative or qualitative data regarding SREs. Specifically, quantitative data included the rate, or proportion, of prescribing errors, or administration errors, that were reportedly facilitated by the EMS. Qualitative data included information presented about the types of SRE identified or specific descriptive accounts of these errors. As the majority of EMS operate with differing levels of functionality and CDS, from basic alerts for duplicate orders to patient-specific algorithms,40 we extracted all information describing the EMS and CDS provided in the reviewed articles. As shown in Table 1, the CDS was classified using the language used by study authors as: “absent” (3 studies), “basic” (2 studies), “rudimentary” (1 study), “moderate” (2 studies) and “extensive” (1 study). In addition, some studies did not specifically describe the level of CDS but indicated that the EMS had capabilities for alert notifications (9 studies). Details of the type of EMS and CDS in the reviewed studies is provided in Supplementary Appendix SB.

Table 1.

Characteristics of the included studies

Author (year) Country Hospitals, Study wards Patient population EMS vendor Study description of CDSa Data collection Outcomes reported Quality rating
Al-Sarawi et al (2009)41 Australia 3 Public, Multiple-ward NR EPAS Alertsb PCR PE 2: Weak
Armada et al (2014)42 Spain 1 Teaching, ICU Adult, Cardiac FarmaTools Dominion “Moderate”a PCR PE, LOS, Mo, Hm, SRE 6: Moderate
Colpaert et al (2006)43 Belgium 1 Teaching, ICU Adult Centricity Critical Care Clinisoft “Moderate”a PCR PE, LOS, Hm 10: Strong
Han et al (2016)44 USA 1 Teaching, ICU Adult EPIC Systems Corp None Obs. Mortality, LOS 7: Moderate
Hernandez et al (2015)45 France 1 Teaching, General Adult, Orthopaedic NR, Commercial Alertsb Obs. PE, MAE 6: Moderate
Hinjosa-Amaya et al (2016)46 Mexico 1 Teaching, General Adult + Paediatric NR NR RCR ME 7: Moderate
Holdsworth et al (2007)47 USA 1 Public, Multiple ward Paediatric Eclipsys System 2000 ”Extensive”a PCR Hm 8: Moderate
Jheeta and Franklin (2017)48 UK 1 Teaching, General Adult, Elderly NR Alertsb Obs. MAE 2: Weak
Joy et al (2012)49 USA 1 Public, Multiple ward Adult + Paediatric EPIC Systems Corp NR PCR ME, Hm 5: Moderate
Jozefczyk et al (2013)50 USA 1 Teaching, ICU Paediatric, Neonatal McKesson’s Horizon Expert Orders ”Rudimentary”a RCR ME 4: Weak
Mills et al (2017)51 UK 1 Public, Multiple ward Adult NR, Commercial Alertsb RCR LOS, PE, SRE 4: Weak
Oliveros et al (2016)52 Spain 1 Teaching, General NR Prescriwin “Basic”a RCR MAE, SRE 4: Weak
Pontefract et al (2018)53 UK 3 NR, General Adult NR, Commercial Variedc PCR PE 4: Weak
Shawahna et al (2011)54 Pakistan 1 Teaching, Multiple ward Adult + Paediatric NR None PCR PE 5: Moderate
Van Doormaal et al (2009)55 Netherlands 2 Teaching, General Adult iSoft Medicator + Theriak “Basic”a PCR LOS, ME, Hm 8: Moderate
Walsh et al (2008)56 USA 1 Public, Multiple ward Paediatric Sunrise Clinical Manager Alertsb PCR ME, Hm, SRE 8: Moderate
Warrick et al (2011)57 USA 1 Public, ICU Paediatric ICIP Alertsb PCR PE 4: Weak
Westbrook et al (2012)58 Australia 2 Teaching, General Adult CMPO + iSoft MedChart Alertsb PCR PE, SRE 10: Strong
Open in a new tab

Abbreviations: CDS, Clinical Decision Support; CMPO, Cerner Millennium Power Orders; EMS, Electronic Medication System; EPAS, The Enterprise Patient Administration System; General, the study included only general wards; Hm, Harm; Multiple-ward, study included both general and intensive care wards; ICU/ED, the study included only intensive care or complicated patient wards (eg, oncology, neonate); ICIP, Intellivue Clinical Information Portfolio; PCR, Prospective Chart Review (charts were assessed during the normal course of daily work); PE, Prescribing Error; Public, the hospital was nonteaching, RCR, Retrospective Chart Review (archived charts were assessed in retrospect); LOS, length of stay; Mo, mortality; NR, not reported; Obs., direct observations conducted pre- and post-EMS implementation; SRE, quantitative reporting of system-related errors.

a

The terms “basic,” “rudimentary,” “moderate,” and “extensive” are taken verbatim from study author. For a complete description of the EMS and CDS, see Supplementary Appendix SB.

b

Many studies did not provide a specific description of CDS, though they did mention some alert functionality, such as potential drug–allergy and drug–drug interactions.

c

This study assessed the impact of 5 levels of CDS. For a complete description, please see Supplementary Appendix SB.

Quality assessment

Two researchers (PG, RH) independently assessed the methodological quality of the included studies and met to discuss disagreements and gain consensus. This quality assessment comprised application of a combination of a previously described 10-point Methodological Quality Assessment59 and the Effective Public Health Practice Project (EPHPP) quality assessment tool for quantitative studies.60 These tools were selected for their applicability to both randomized and nonrandomized studies and suitability for use in systematic reviews through expert judgement15 or previous use.18,19,59,61 We adapted these tools to best reflect the complexity of conducting these “real-world” intervention studies and removed the criterion requiring control groups as—while highly desirable—opportunities for obtaining control groups may be very limited for real-world studies of this type of organization-wide IT intervention.62,63 Thus, our quality assessment focused on 6 components: 1) allocation bias, 2) unit of allocation bias, 3) data collection methods, 4) completeness of the samples, and 5) assessment of confounders. A further component, 'clarity of reporting', was added in order to score how clearly the electronic system being assessed, and the patient population, were reported. Using these 6 components, each with a score of 0–12, we describe study quality as weak (score 0–4), moderate (score 5–8), and strong (score 9–12). Full details of this assessment by individual study is presented in Supplementary Appendix SC.

Statistical analysis

Authors of articles to be included in the meta-analyses, but containing insufficient information (n = 18), were sent emails requesting additional data to be provided, and, in 4 cases, data were received. Meta-analyses were performed if there were at least 3 studies with sufficient data reported in published studies or provided by authors, using random effects models to pool results examining the impact of EMS on medication error rates and associated harm. Risk ratios and their 95% confidence intervals (CIs) were calculated. To be conservative, the Knapp-Hartung approach64 was applied to account for heterogeneity between studies. Forest plots were used to present meta-analyses results. Between-study heterogeneity was evaluated using I2 statistics.65 The potential for publication bias for each meta-analysis was assessed by inspection of funnel plots and statistical tests based on weighted linear regression of the intervention effect on its standard error.66 Sensitivity analyses were conducted by grouping studies by their quality rating, and also for studies with and without CDS, provided there were at least 3 studies with sufficient information. All statistical tests were 2-sided and were evaluated at a significance level of 0.05. Analyses were carried out using R version 3.6.1.67

RESULTS

Study characteristics

Eighteen studies met our inclusion criteria.41–58 Characteristics of the included studies are provided in Table 1. Study publication dates ranged from 2006 to 2019. All studies used a before–after design, except for 2 interrupted time series without control,48,55 and 1 controlled cross-sectional trial.43 One before–after study included a control group.58 Seven studies were rated as being of weak quality,41,48,50–53,57 9 moderate,42,44–47,49,54–56 and 2 studies were rated as strong.43,58 Individual study results for each outcome are presented in Supplementary Appendix SD.

Changes in medication error (ME) rates pre- and post-EMS

Five studies assessed the impact of EMS on the incidence of ME,44,46,49,50,56 with 4 studies rated to be of moderate quality.44,46,49,56 Differences in the reported numerators and denominators across studies prevented meta-analysis examining the association between the introduction of EMS and ME rates. A significant reduction in the rate of ME was reported in 2 of the 4 moderate quality studies.46,49 One study, conducted in a pediatric hospital, reported no significant change in error rates at 9 months post-EMS, and 1 study, conducted in the intensive care wards of an adult hospital, reported a significant increase in ME rates at 3 months post-EMS.44

Changes in prescribing error (PE) rates pre- and post-EMS

Ten studies assessed the impact of EMS on PE rates, with a range of denominators reported across studies.41–43,45,51,53–55,57,58 Heterogeneity in error rate reporting and study settings prevented meta-analysis of the association between EMS and rates of PE. EMS, with or without CDS, was associated with a significant reduction in PE rate in all but 1 study. That study was rated to be of weak quality and found no significant change in PE per 100 orders.57 Only 1 study compared before–after changes in a control group.58 That study identified no statistically significant change in the PE rate in the control group.

Changes in medication administration error (MAE) rates pre- and post-EMS

Three studies assessed the impact of EMS on MAE rates, each assessing the rate of errors per OE. All were included in our meta-analysis.45,48,52 Two studies reported a significant reduction in MAE rates after the intervention.45,52 Overall, the meta-analysis showed a significant reduction in MAE rates post-EMS implementation (pooled RR: 0.77, 95% CI: 0.72–0.83, P = .004) (Figure 2). There was little evidence of heterogeneity among these studies (P < .73, I2<0.1%). There was evidence of publication bias (P = .5).

Figure 2.

Figure 2.

Open in a new tab

Relative risk of a medication administration error following the introduction of EMS.

Changes in the rate of ME, PE, or MAE with potential for harm or causing harm pre- and post-EMS

Nine studies assessed the impact of introducing an electronic medication system (EMS) on the rate of ME, PE, or MAE with potential for harm or causing actual patient harm using a range of denominators.42–44,47,49,54–56,58 Five studies assessing the impact of EMS on the proportion of ME (n = 2)44,49 and PE (n = 3)42,54,55 to cause actual harm were included in the meta-analysis. All 5 studies classified actual harm severity using the NCC-MERP index.31 Two studies reported results at 1 and 3 months post-EMS,42,49 however; as there were no statistically significant changes in harm across these periods, the results from both were combined for this analysis. The 5 studies were found to be significantly heterogeneous (heterogeneity between studies: I2=97.0%, P < .001). There was no evidence of publication bias (P = .8). Overall, the meta-analysis results indicated that the introduction of EMS had no significant effect on patient harm (pooled RR: 1.22, 95% CI: 0.18–8.38, P = .8) (Figure 3). Sensitivity analysis was conducted for 3 studies of moderate quality,44,49,55 and there was no significant effect (pooled RR: 0.83, 95% CI: 0.01–78.26, P = .8).

Figure 3.

Figure 3.

Open in a new tab

Relative risk of actual patient harm related to medication errors following the introduction of EMS.

Results were mixed in the 4 studies not included in the meta-analysis. One of 2 moderate quality studies assessing the rates of patients experiencing at least 1 pADE47,56 reported a significant reduction post-EMS.47 One strong quality study reported a reduction in the rate of potentially harmful PE per admission58 The remaining study assessed the rate of PE that reached the patient with the potential to cause any harm, was rated to be of strong quality, and reported a significant reduction post-EMS, from 1.0 per 100 orders to 0.15 per 100 orders.43

Changes in length of stay and mortality pre- and post-EMS

Differences in statistical parameters (ie, mean, median) and patient groups across studies prevented meta-analysis examining the association between introduction of EMS and LOS. Six studies assessed the association between an EMS and LOS; 5 in adult hospitals,42–44,51,55 and 1 in a pediatric hospital.47 Overall, there was little evidence of change up to 1 year post-EMS implementation in the adult hospitals. The largest moderate quality study, reported a significant change in the LOS on general wards in 2 adult hospitals 2 months post-EMS.55 That study also reported the highest baseline average LOS, with large variation across patients. The only pediatric study included over 1000 patients in each group, was rated as moderate quality, and found no significant reduction in mean LOS 15 months post-EMS using a logistic regression model which included all variables found to be significantly different between the 2 study periods.47

Only 2 studies assessed the association between an EMS and hospital mortality rates, both in adult ICUs.42,44 The first, a small study of poor quality, found no significant reduction in mortality 3 months post-EMS with “moderate” CDS.42 The second, a large, moderate quality study, showed a significant increase in mortality in the first 4 months following system introduction using data adjusted for patient mortality risk, though reported an overall significant reduction in mortality at 5 months postintervention.44

System-related errors (SREs) introduced post-EMS

Twelve studies provided qualitative examples of SREs,41–43,45,48,49,51–53,56–58 though only 5 reported on SRE rates.42,51,52,56,58 We report the full qualitative data extraction in Supplementary Appendix SD. Regarding the quantitative data, 3 studies reported rates of SRE as 0.4% of prescribed drugs,42 1.6 SREs per 1000 patient-days,56 and 0.57 SREs per admission.58 Four studies reported the percentage of medication errors that were related to system use,51–53,58 which ranged from 1.2% to 34.8% of all errors, with heterogeneity in study quality and in the hospitals and EMS implemented.

The 5 studies providing quantitative data also assessed SREs in relation to the overall effect of EMS on medication error rates.42,51,52,56,58 These studies assessed the extent to which the SRE associated with the introduction of EMS moderated any identified change in the rate of medication errors following EMS introduction. Three weak quality studies reported an overall significant decrease in medication error rates despite new errors associated with system use and advocated that the impact of new errors can be minimized with additional training and adherence to EMS use guidelines.42,51,52 One moderate quality study of a pediatric hospital post-EMS reported no significant change in the rate of ME and that the rate of SRE was low and unlikely to have moderated this lack of effect.56 The remaining study, of strong quality, reported a significant reduction in procedural prescribing error rate despite new SRE.58 That study also reported a significant reduction in the clinical error rate, though no statistically significant change was found when new SRE were included. However, the overall proportion of prescribing errors classified as “potentially serious” was lower post-EMS.

DISCUSSION

This systematic review and meta-analyses examined the extent to which introducing an electronic medication system (EMS) in hospital wards which previously used paper-based medication ordering was associated with changes in ME rates (including both PE and MAE at a minimum) and any associated potential or actual patient harm associated with those errors. Although half of the 18 studies published between January 2005 and March 2019 assessed potential or actual patient harm, the impact of EMS on patient harm remains unclear as studies were heterogeneous and results mixed. Meta-analysis found no evidence of a significant overall effect on patient harm related to medication error rates (combining data on ME, PE, and MAE), with studies showing positive, negative, and no change following the introduction of EMS. In contrast, there was consistent evidence across 14 studies that changing from a paper-based medication system to an EMS is likely to reduce medication error rates, particularly prescribing errors.38,41–43,45,46,49–51,53–55,58,68Meta-analysis results also indicated a significant reduction in administration error rates, though only 3 studies were included in this analysis.45,48,52 The very few studies that measured the relationship between changes in medication error rates and the introduction of SREs, found that new SREs are likely to marginally limit overall reductions in medication errors. Those studies reporting SREs consistently reported that system implementations which included ongoing support and the flexibility to adapt systems to respond to user input could reduce the overall rate of SRE and improve the safety benefits of these systems.

Consistent with other interventions designed to reduce the rate of medication error, such as the participation of a clinical pharmacist, or the use of checklists and guidelines,69,70 the jury is still out as to whether introducing EMS will significantly impact rates of medication-related harm. These results support observations of a continuing trend where decisions to implement health information technology such as EMS are based on motivators other than research evidence of effectiveness.71 The lack of consistent evidence of effects on patient harm may be explained in several ways. Firstly, previous reviews,10,23 and the present review, have identified an absence of high-quality studies. We identified that the use of quasi-experimental study designs with no control group, inadequate reporting of data collection methods, and failure to report between group differences and consider potential confounding variables were the main contributors to poor study quality. Given that actual patient harm is reasonably infrequent,47,56 studies with large samples are required to assess the impact on harm. We identified only 1 strong quality study which reported outcomes on actual patient harm.47

Secondly, EMS represented in the review may not have significantly impacted rates of medication-related patient harm due to CDS functionality. CDS in EMS is an important feature shown to be effective in targeting particular types of medication errors72 yet was limited in many systems studied and poorly described by others. Only 4 studies reported CDS functionality beyond simple system alert notifications.42,43,47,53 One of these studies was included in our meta-analysis, though that study did not report any actual patient harm before or after system implementaion.42 Among the 4 studies with advanced CDS, 3 assessed changes to the rate of prescribing error and all reported a significant reduction,42,43,53 and 3 assessed actual patient harm, and, excluding the study which found no harm, the remaining 2 studies reported a significant reduction postimplementation.43,47 One study assessed 3 hospitals which introduced 2 different EMS with differing levels of CDS to paper-based ordering and measured the change in the rate of prescribing errors.53 In that study, although the authors reported a significant reduction in prescribing errors that had targeted CDS, no linear association was found between CDS functionality and error reduction, with different results generated across the 2 sites with the same system. These authors concluded that the manner of system implementation may have significantly moderated this association.53

The successful adoption of EMS requires significant adaption of work practices, acceptance of innovation, and good organizational fit, and the methods of implementation and ongoing improvements in response to user-needs can significantly impact system effectiveness.72–76 We found that details of the implementation methods were only briefly reported in studies or entirely omitted (see Supplementary Appendix SB for details), thus making it difficult to determine how these factors may moderate system effectiveness. Ignoring the importance of the organizational and social context in which EMS is to be implemented may have significant impact on the system effectiveness.73 Review of factors associated with successful system implementation77 and studies included in this review have identified the importance of adequate system training44,49,52,53 and the use of support staff.44 Though the level of training or support that would be considered adequate is unclear.

Finally, interpretation of results relating to reductions in medication-related patient harm should consider the overall mix of medication error types included in the calculation of rates pre- and post-EMS. EMS implementation brings immediate benefits in terms of the standardization of orders, almost eliminating “procedural” errors such as illegible orders, incomplete orders, incorrect units, etc. These errors are highly prevalent at baseline, yet usually of low potential severity. Consequently, comparison of the rates of medication errors causing harm pre- and post-EMS should report medication-related harm rates for “clinical’ and “procedural” medication errors separately.

Our results highlight the difficulties faced by those measuring the effectiveness of interventions designed to reduce medication error-related harm and meet the goals of the WHO Global Patient Safety Challenge.78 This review identified several ways to improve future research as others have called for.79 In particular, outcomes should be reported as error rates: medication errors per order, or if this is not feasible, per 1000 patient days; the proportion of errors with the potential to cause patient harm; the proportion of patients who experienced actual harm; and the level of harm severity. Actual harm resulting from medication errors is understudied and should be an important focus of future studies.

Limitations

This review focused on evidence of the impact of EMS on patient harm and medication errors; however, we acknowledge that the use of electronic systems in hospital settings are more widely transformative and provide other benefits that we did not assess.16–21 In addition, all systematic reviews are limited by the number and quality of the included studies. We utilized 2 quality assessment tools to describe the quality of included study methods as weak, moderate, or strong.59,60 We found only 2 strong quality studies. In addition, pooling study outcomes was limited by the small number of studies and variation in the chosen denominator to measure outcomes and in the chosen methods for describing patient harm. Our review was limited by the poorly reported, heterogenous, and often short follow-up periods of less than 7 months across studies postimplementation of EMS. This is important, as factors that would impact on the outcomes we assessed, such as changes in staff perceptions of or proficiencies in using these systems, required system improvements or the development of workarounds that would not be captured.76 A strength of this review was its wide criteria for study inclusion which were not limited by patient group or hospital setting. Moreover, by excluding studies using incident reports, the included studies were less likely to underreport error rates. Finally, there was heterogeneity in the definitions of medication error and harm across studies; however, we attempted to address this limitation by including only studies that met our minimum criteria.

CONCLUSION

Our review identified evidence indicating that introducing EMS is likely to significantly reduce rates of medication errors, particularly prescribing errors. However, evidence of changes in the rate of harmful medication errors is less clear. Further, insufficient studies have investigated the effects on medication administration error rates. We identified few methodologically strong studies, particularly of advanced systems with substantive clinical decision support. Future controlled trials should be powered to be able to detect changes in medication errors resulting in harm but also need to focus on the identification of specific mechanisms and system features most effective in improving medication safety along with those that may facilitate new error risks.

EMS systems have been used in many hospitals around the world for well over 2 decades. The limited and poor-quality effectiveness evidence base is somewhat surprising given the size and costs associated with these interventions and their extensive reach in terms of potentially impacting the care of all hospital inpatients. The methodological challenges in conducting studies of this type is 1 reason for this deficit, but this situation is also likely a reflection of complacency regarding the necessity for evaluation of large-scale information technology (IT) systems.71 Many still view these systems as IT interventions despite their integral role and desired impact on clinical care. While few would argue with the great potential of clinical IT systems, it is critical that we assess their actual impact in real-world settings and identify system, user, and organizational factors which will allow this potential to be realized in terms of improved clinical care now and into the future.

FUNDING

This work was supported by the National Health and Medical Research Council, grant numbers [APP1094878, APP1143941, and APP1174021] and by Macquarie University, grant number [MQRDG IRIS # 9201601531].

AUTHOR CONTRIBUTIONS

All authors have participated in the concept and design or analysis and interpretation of data and drafting or revising of the manuscript. All have approved the manuscript as submitted and agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

CONFLICT OF INTEREST STATEMENT

None declared.

Supplementary Material

ocaa230_Supplementary_Data
Click here for additional data file. (240.4KB, zip)

REFERENCES

  • 1. Donaldson LJ, Kelley ET, Dhingra-Kumar N, et al. Medication without harm: WHO’s third global patient safety challenge. Lancet 2017; 389 (10080): 1680–1. [DOI] [PubMed] [Google Scholar]
  • 2. Wager KA, Lee FW, Glaser JP.. Health Care Information Systems: A Practical Approach for Health Care Management. San Francisco, CA: Jossey-Bass; 2017. [Google Scholar]
  • 3. Kaushal R, Shojania KG, Bates DW.. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. JAMA Intern Med 2003; 163 (12): 1409–16. [DOI] [PubMed] [Google Scholar]
  • 4. Ammenwerth E, Schnell-Inderst P, Machan C, et al. The effect of electronic prescribing on medication errors and adverse drug events: a systematic review. J Am Med Inform Assoc 2008; 15 (5): 585–600. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5. Nuckols TK, Smith-Spangler C, Morton SC, et al. The effectiveness of computerized order entry at reducing preventable adverse drug events and medication errors in hospital settings: a systematic review and meta-analysis. Syst Rev 2014; 3 (1): 56. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6. van Rosse F, Maat B, Rademaker CM, et al. The effect of computerized physician order entry on medication prescription errors and clinical outcome in pediatric and intensive care: a systematic review. Pediatrics 2009; 123 (4): 1184–90. [DOI] [PubMed] [Google Scholar]
  • 7. Georgiou A, Williamson M, Westbrook JI, et al. The impact of computerised physician order entry systems on pathology services: a systematic review. Int J Med Inform 2007; 76 (7): 514–29. [DOI] [PubMed] [Google Scholar]
  • 8. Conroy S, Sweis D, Planner C, et al. Interventions to reduce dosing errors in children: a systematic review of the literature. Drug Saf 2007; 30 (12): 1111–25. [DOI] [PubMed] [Google Scholar]
  • 9. Lapkin S, Levett-Jones T, Chenoweth L, et al. The effectiveness of interventions designed to reduce medication administration errors: a synthesis of findings from systematic reviews. J Nurs Manag 2016; 24 (7): 845–58. [DOI] [PubMed] [Google Scholar]
  • 10. Ranji SR, Rennke S, Wachter RM.. Computerised provider order entry combined with clinical decision support systems to improve medication safety: a narrative review. BMJ Qual Saf 2014; 23 (9): 773–80. [DOI] [PubMed] [Google Scholar]
  • 11. Reckmann MH, Westbrook JI, Koh Y, et al. Does computerized provider order entry reduce prescribing errors for hospital inpatients? A systematic review. J Am Med Inform Assoc 2009; 16 (5): 613–23. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12. Rinke ML, Bundy DG, Velasquez CA, et al. Interventions to reduce pediatric medication errors: a systematic review. Pediatrics 2014; 134 (2): 338–60. [DOI] [PubMed] [Google Scholar]
  • 13. Shamliyan TA, Duval S, Du J, et al. Just what the doctor ordered. Review of the evidence of the impact of computerized physician order entry system on medication errors. Health Serv Res 2007; 43 (1p1): 32–53. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14. Stultz JS, Nahata MC.. Computerized clinical decision support for medication prescribing and utilization in pediatrics. J Am Med Inform Assoc 2012; 19 (6): 942–53. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15. Prgomet M, Li L, Niazkhani Z, et al. Impact of commercial computerized provider order entry (CPOE) and clinical decision support systems (CDSSs) on medication errors, length of stay, and mortality in intensive care units: a systematic review and meta-analysis. J Am Med Inform Assoc 2017; 24 (2): 413–22. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16. Chaudhry B, Wang J, Wu S, et al. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med 2006; 144 (10): 742–52. [DOI] [PubMed] [Google Scholar]
  • 17. Eslami S, Keizer NFd, Abu-Hanna A.. The impact of computerized physician medication order entry in hospitalized patients—a systematic review. Int J Med Inform 2008; 77 (6): 365–76. [DOI] [PubMed] [Google Scholar]
  • 18. Garg AX, Adhikari NK, McDonald H, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 2005; 293 (10): 1223–38. [DOI] [PubMed] [Google Scholar]
  • 19. Hunt DL, Haynes RB, Hanna SE, et al. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA 1998; 280 (15): 1339–46. [DOI] [PubMed] [Google Scholar]
  • 20. Niazkhani Z, Pirnejad H, Berg M, et al. The impact of computerized provider order entry systems on inpatient clinical workflow: a literature review. J Am Med Inform Assoc 2009; 16 (4): 539–49., [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21. McKibbon A, Lokker C, Handler SM, et al. The effectiveness of integrated health information technologies across the phases of medication management: a systematic review of randomized controlled trials. J Am Med Inform Assoc 2012; 19 (1): 22–30. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22. Westbrook JI, Baysari MT, Li L, et al. The safety of electronic prescribing: manifestations, mechanisms, and rates of system-related errors associated with two commercial systems in hospitals. J Am Med Inform Assoc 2013; 20 (6): 1159–67. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23. Weir CR, Staggers N, Laukert T.. Reviewing the impact of computerized provider order entry on clinical outcomes: the quality of systematic reviews. Int J Med Inform 2012; 81 (4): 219–31. [DOI] [PubMed] [Google Scholar]
  • 24. McKibbon KA, Lokker C, Handler SM, et al. Enabling medication management through health information technology (Health IT). Evid Rep Technol Assess 2011; (201): 1–951. [PMC free article] [PubMed] [Google Scholar]
  • 25. Wolfstadt JI, Gurwitz JH, Field TS, et al. The effect of computerized physician order entry with clinical decision support on the rates of adverse drug events: a systematic review. J Gen Intern Med 2008; 23 (4): 451–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26. Gates PJ, Meyerson SA, Baysari MT, et al. Preventable adverse drug events among inpatients: a systematic review. Pediatrics 2018; 142 (3): e20180805. [DOI] [PubMed] [Google Scholar]
  • 27. Thompson G, O’Horo JC, Pickering BW, et al. Impact of the electronic medical record on mortality, length of stay, and cost in the hospital and ICU: a systematic review and meta-analysis. Crit Care Med 2015; 43 (6): 1276–82. [DOI] [PubMed] [Google Scholar]
  • 28. Westbrook JI, Li L, Lehnbom EC, et al. What are incident reports telling us? A comparative study at two Australian hospitals of medication errors identified at audit, detected by staff and reported to an incident system. Int J Qual Health Care 2015; 27 (1): 1–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29. Gates PJ, Meyerson SA, Baysari MT, et al. The prevalence of dose errors among paediatric patients in hospital wards with and without health information technology: a systematic review and meta-analysis. Drug Saf 2019; 42 (1): 13–25. [DOI] [PubMed] [Google Scholar]
  • 30. Shawahna R, Masri D, Al-Gharabeh R, et al. Medication administration errors from a nursing viewpoint: a formal consensus of definition and scenarios using a Delphi technique. J Clin Nurs 2016; 25 (3-4): 412–23. [DOI] [PubMed] [Google Scholar]
  • 31.National Coordinating Council for Medication Error Reporting and Prevention (NCCMERP). NCC MERP Index for categorizing medication errors. 2018. http://www.nccmerp.org/types-medication-errors Accessed January 5, 1996
  • 32. Velez-Diaz-Pallares M, Delgado-Silveira E, Emilia Carretero-Accame M, et al. Using healthcare failure mode and effect analysis to reduce medication errors in the process of drug prescription, validation and dispensing in hospitalised patients. BMJ Qual Saf 2013; 22 (1): 42–52. [DOI] [PubMed] [Google Scholar]
  • 33. Alonso AH, Gonzalez CGR, Saez MS.. Information technology and automation in hospitals: Strategies and experience in a tertiary hospital in Spain. EJHP Pract 2011; 17 (4): 26–31. [Google Scholar]
  • 34. Franklin BD, Jacklin A, Barber N.. The impact of an electronic prescribing and administration system on the safety and quality of medication administration. Int J Pharm Pract 2008; 16 (6): 375–9. [Google Scholar]
  • 35. Zucker J, Mittal J, Jen SP, et al. Impact of stewardship interventions on antiretroviral medication errors in an urban medical center: a 3-year, multiphase study. Pharmacotherapy 2016; 36 (3): 245–51. [DOI] [PubMed] [Google Scholar]
  • 36. Elsaid K, Truong T, Monckeberg M, et al. Impact of electronic chemotherapy order forms on prescribing errors at an urban medical center: results from an interrupted time-series analysis. Int J Qual Health Care 2013; 25 (6): 656–63. [DOI] [PubMed] [Google Scholar]
  • 37. Griffey RT, Lo HG, Burdick E, et al. Guided medication dosing for elderly emergency patients using real-time, computerized decision support. J Am Med Inform Assoc 2012; 19 (1): 86–93. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38. Aziz MT, Ur-Rehman T, Qureshi S, et al. Reduction in chemotherapy order errors with computerised physician order entry and clinical decision support systems. Health Inform Manag J 2015; 44 (3): 13–22. [DOI] [PubMed] [Google Scholar]
  • 39. Galanter WL, Didomenico RJ, Polikaitis A.. A trial of automated decision support alerts for contraindicated medications using computerized physician order entry. J Am Med Inform Assoc 2005; 12 (3): 269–74. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40. Wright A, Sittig DF, Ash JS, et al. Clinical decision support capabilities of commercially-available clinical information systems. J Am Med Inform Assoc 2009; 16 (5): 637–44. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41. Al-Sarawi F, Polasek TM, Caughey GE, et al. Prescribing errors and adverse drug reaction documentation before and after implementation of e-prescribing using the Enterprise Patient Administration System. J Pharm Pract Res 2019; 49 (1): 27–32. [Google Scholar]
  • 42. Armada ER, Villamañán E, López-de-Sá E, et al. Computerized physician order entry in the cardiac intensive care unit: effects on prescription errors and workflow conditions. J Crit Care 2014; 29 (2): 188–93. [DOI] [PubMed] [Google Scholar]
  • 43. Colpaert K, Claus B, Somers A, et al. Impact of computerized physician order entry on medication prescription errors in the intensive care unit: a controlled cross-sectional trial. Crit Care 2006; 10 (1): R21. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 44. Han JE, Rabinovich M, Abraham P, et al. Effect of electronic health record implementation in critical care on survival and medication errors. Am J Med Sci 2016; 351 (6): 576–81. [DOI] [PubMed] [Google Scholar]
  • 45. Hernandez F, Majoul E, Montes-Palacios C, et al. An observational study of the impact of a computerized physician order entry system on the rate of medication errors in an orthopaedic surgery unit. PLoS One 2015; 10 (7): e0134101. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46. Hinojosa-Amaya JM, Rodríguez-García FG, Yeverino-Castro SG, et al. Medication errors: electronic vs. paper-based prescribing. Experience at a tertiary care university hospital. J Eval Clin Pract 2016; 22 (5): 751–4. [DOI] [PubMed] [Google Scholar]
  • 47. Holdsworth MT, Fichtl RE, Raisch DW, et al. Impact of computerized prescriber order entry on the incidence of adverse drug events in pediatric inpatients. Pediatrics 2007; 120 (5): 1058–66. [DOI] [PubMed] [Google Scholar]
  • 48. Jheeta S, Franklin BD.. The impact of a hospital electronic prescribing and medication administration system on medication administration safety: an observational study. BMC Health Serv Res 2017; 17 (1): 547. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 49. Joy A, Davis J, Cardona J.. Effect of computerized provider order entry on rate of medication errors in a community hospital setting. Hosp Pharm 2012; 47 (9): 693–9. [Google Scholar]
  • 50. Jozefczyk KG, Kennedy WK, Lin MJ, et al. Computerized prescriber order entry and opportunities for medication errors: comparison to tradition paper-based order entry. J Pharm Pract 2013; 26 (4): 434–7. [DOI] [PubMed] [Google Scholar]
  • 51. Mills PR, Weidmann AE, Stewart D.. Hospital electronic prescribing system implementation impact on discharge information communication and prescribing errors: a before and after study. Eur J Clin Pharmacol 2017; 73 (10): 1279–86. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 52. Oliveros NV, Teresa GC, Menendez-Conde CP, et al. Effect of an electronic medication administration record application on patient safety. J Eval Clin Pract 2017; 23 (4): 888–94. [DOI] [PubMed] [Google Scholar]
  • 53. Pontefract SK, Hodson J, Slee A, et al. Impact of a commercial order entry system on prescribing errors amenable to computerised decision support in the hospital setting: a prospective pre-post study. BMJ Qual Saf 2018; 27 (9): 725–36. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 54. Shawahna R, Rahman NU, Ahmad M, et al. Electronic prescribing reduces prescribing error in public hospitals. J Clin Nurs 2011; 20 (21-22): 3233–45. [DOI] [PubMed] [Google Scholar]
  • 55. van Doormaal JE, van den Bemt PMLA, Zaal RJ, et al. The influence that electronic prescribing has on medication errors and preventable adverse drug events: an interrupted time-series study. J Am Med Inform Assoc 2009; 16 (6): 816–25. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 56. Walsh KE, Landrigan CP, Adams WG, et al. Effect of computer order entry on prevention of serious medication errors in hospitalized children. Pediatrics 2008; 121 (3): e421–7. [DOI] [PubMed] [Google Scholar]
  • 57. Warrick C, Naik H, Avis S, et al. A clinical information system reduces medication errors in paediatric intensive care. Intensive Care Med 2011; 37 (4): 691–4. [DOI] [PubMed] [Google Scholar]
  • 58. Westbrook JI, Reckmann M, Li L, et al. Effects of two commercial electronic prescribing systems on prescribing error rates in hospital in-patients: a before and after study. PLoS Med 2012; 9 (1): e1001164. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 59. Brenner SK, Kaushal R, Grinspan Z, et al. Effects of health information technology on patient outcomes: a systematic review. J Am Med Inform Assoc 2016; 23 (5): 1016–36. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 60.Effective Public Health Practice Project. Quality Assessment Tool for Quantitative Studies. 1998. https://merst.ca/ephpp/ Accessed August 11, 2019.
  • 61. Jerant AF, Hill DB.. Does the use of electronic medical records improve surrogate patient outcomes in outpatient settings? J Fam Pract 2000; 49 (4): 349–57. [PubMed] [Google Scholar]
  • 62. Greenhalgh T, Russell J.. Why do evaluations of eHealth programs fail? An alternative set of guiding principles. PLoS Med 2010; 7 (11): e1000360. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 63. Lilford RJ, Foster J, Pringle M.. Evaluating eHealth: how to make evaluation more methodologically robust. PLoS Med 2009; 6 (11): e1000186. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 64. Knapp G, Hartung J.. Improved tests for a random effects meta-regression with a single covariate. Stat Med 2003; 22 (17): 2693–710. [DOI] [PubMed] [Google Scholar]
  • 65. Higgins JP, Thompson SG, Deeks JJ.. Measuring inconsistency in meta-analyses. BMJ 2003; 327 (7414): 557–60. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 66. Egger M, Smith GD, Schneider M, et al. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997; 315 (7109): 629–34. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 67.R Development Core Team. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing; 2013. [Google Scholar]
  • 68. Franklin BD, O’Grady K, Donyai P, et al. The impact of a closed-loop electronic prescribing and administration system on prescribing errors, administration errors and staff time: a before-and-after study. Qual Saf Health Care 2007; 16 (4): 279–84. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 69. Maaskant JM, Vermeulen H, Apampa B, et al. Interventions for reducing medication errors in children in hospital. Cochrane Database Syst Rev 2015; (3): CD006208. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 70. Manias E, Williams A, Liew D.. Interventions to reduce medication errors in adult intensive care: a systematic review. Br J Clin Pharmacol 2012; 74 (3): 411–23. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 71. Rigby M, Magrabi F, Scott P, et al. Steps in moving evidence-based health informatics from theory to practice. Healthc Inform Res 2016; 22 (4): 255–60. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 72. Page N, Baysari MT, Westbrook JI.. A systematic review of the effectiveness of interruptive medication prescribing alerts in hospital CPOE systems to change prescriber behavior and improve patient safety. Int J Med Inform 2017; 105: 22–30. [DOI] [PubMed] [Google Scholar]
  • 73. Tsiknakis M, Kouroubali A.. Organizational factors affecting successful adoption of innovative eHealth services: a case study employing the FITT framework. Int J Med Inform 2009; 78 (1): 39–52. [DOI] [PubMed] [Google Scholar]
  • 74. Westbrook JI, Baysari MT.. Nudging hospitals towards evidence-based decision support for medication management. Med J Austr 2019; 210 (S6): S22–4. [DOI] [PubMed] [Google Scholar]
  • 75. Zheng WY, Richardson LC, Li L, et al. Drug-drug interactions and their harmful effects in hospitalised patients: a systematic review and meta-analysis. Eur J Clin Pharmacol 2018; 74 (1): 15–27. [DOI] [PubMed] [Google Scholar]
  • 76. Baysari MT, Hardie R-A, Lake R, et al. Longitudinal study of user experiences of a CPOE system in a pediatric hospital. Int J Med Inform 2018; 109: 5–14. [DOI] [PubMed] [Google Scholar]
  • 77. Mayo-Smith MF, Agrawal A.. Factors associated with improved completion of computerized clinical reminders across a large healthcare system. Int J Med Inform 2007; 76 (10): 710–6. [DOI] [PubMed] [Google Scholar]
  • 78. Sheikh A. Realising the potential of health information technology to enhance medication safety. BMJ Qual Saf 2020; 29 (1): 7–9. [DOI] [PubMed] [Google Scholar]
  • 79. Talmon J, Ammenwerth E, Brender J, et al. STARE-HI–Statement on reporting of evaluation studies in Health Informatics. Int J Med Inform 2009; 78 (1): 1–9. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

ocaa230_Supplementary_Data
Click here for additional data file. (240.4KB, zip)

Articles from Journal of the American Medical Informatics Association : JAMIA are provided here courtesy of Oxford University Press

RESOURCES