The Effect of Electronic Prescribing on Medication Errors and Adverse Drug Events: A Systematic Review

Abstract

The objective of this systematic review is to analyse the relative risk reduction on medication error and adverse drug events (ADE) by computerized physician order entry systems (CPOE). We included controlled field studies and pretest-posttest studies, evaluating all types of CPOE systems, drugs and clinical settings. We present the results in evidence tables, calculate the risk ratio with 95% confidence interval and perform subgroup analyses for categorical factors, such as the level of care, patient group, type of drug, type of system, functionality of the system, comparison group type, study design, and the method for detecting errors. Of the 25 studies that analysed the effects on the medication error rate, 23 showed a significant relative risk reduction of 13% to 99%. Six of the nine studies that analysed the effects on potential ADEs showed a significant relative risk reduction of 35% to 98%. Four of the seven studies that analysed the effect on ADEs showed a significant relative risk reduction of 30% to 84%. Reporting quality and study quality was often insufficient to exclude major sources of bias. Studies on home-grown systems, studies comparing electronic prescribing to handwriting prescribing, and studies using manual chart review to detect errors seem to show a higher relative risk reduction than other studies. Concluding, it seems that electronic prescribing can reduce the risk for medication errors and ADE. However, studies differ substantially in their setting, design, quality, and results. To further improve the evidence-base of health informatics, more randomized controlled trials (RCTs) are needed, especially to cover a wider range of clinical and geographic settings. In addition, reporting quality of health informatics evaluation studies has to be substantially improved.

Introduction

Medical errors are an important factor that influences the quality of patient care. According to Barach et al., nearly 100,000 individuals per year in the US die of preventable medical errors. ¹

Medication errors have been identified as a major type of medical errors. The Council of Europe ² and the British Department of Health ³ defined medication errors as “any preventable event that may cause or lead to inappropriate medication use or patient harm … .” The Institute of Medicine reports that a hospital patient can expect on average to be subjected to more than one medication error per day. ^4,5 Medication errors can lead to adverse drug events (ADEs) that are defined as “any response to a drug that is noxious and unintended… .” ³ A report from the Institute of Medicine that was published in 1999 stated that annually in the US 7,000 deaths can be associated with medication errors. ⁶

In light of these figures, it is not surprising that the British Department of Health recommends the wider use of electronic prescribing to reduce the risk of medication errors. ³ Electronic prescribing is supported by Computerized Physician Order Entry (CPOE) systems. The term CPOE refers to a variety of computer-based systems for ordering medications, which share the common features of automating the medication ordering process. ⁷ The CPOE systems can range from systems that only provide a list of possible medications that the physician can then choose from, to systems providing varying levels of decision support, including checks of drug-drug interactions, drug-allergy contraindications, or checks of prescriptions concerning the patient's recent laboratory results. All those checks lead to alerts and reminders given to the ordering physician in case problems are detected.

In the scientific literature, many studies investigate the effects of electronic prescribing on the quality of patient care. Many studies showed a reduction in medication errors (such as Bates et al., 1999), ⁸ while others described negative effects such as the facilitation of medication error risks ⁹ or an increase in mortality. ¹⁰ The available reviews on this topic seem to not come to an unambiguous conclusion on the overall effects of electronic prescribing, concluding that published evidence is often insufficient. ^11–16 None of the published reviews has—to our knowledge—tried to quantitatively summarize the available evidence on the effects, and to systematically investigate the factors contributing to those effects.

Therefore, we performed a systematic and quantitative review to determine the effect of electronic prescribing on the risk of medication errors and adverse drug events (ADEs), and analyzed factors for those effects. This review should inform the health informatician of the effects that can be expected of electronic prescribing, and on the factors that influence the success of such systems. We see this review as a contribution to support evidence-based health informatics. ¹⁷

Methods

Identification of Trials

We searched MEDLINE (1966 to April 2006) and EMBASE (1976 to April 2006) for studies on electronic prescribing that evaluated its effect on medication errors and ADEs. We combined MeSH terms, such as “Medical Order Entry Systems,” “prescriptions, drug,” “drug therapy, computer-assisted” with general search terms such as “order entry,” “CPOE,” “POE,” “order communication,” “prescription system,” “drug prescription,” “prescribing,” “ordering,” and “computerized reminders.” Those search terms were combined with terms that searched for evaluation studies that were taken from an earlier literature survey. ¹⁸ The complete query is available upon request.

We also searched the Cochrane Database of Systematic Reviews and examined the reference lists of the relevant reviews in order to find further studies. Finally, we performed a hand search of the Journal of the American Medical Informatics Association (1994–2006), of the International Journal of Medical Informatics (1997–2006), and of the Methods of Information in Medicine (1990–2006). We completed this by hand-searching the references of retrieved study papers. We did not restrict the search to any single language.

Inclusion Criteria and the Selection of Studies

Intervention: We included studies wherein the intervention was electronic prescribing. As an intervention, we included all the computer-based application systems to order drugs that are used at the point of care. We included electronic prescribing systems independent of the level of decision support that they provided (e.g., with or without alerts on drug-drug interaction), and for all types of drugs. We excluded the studies that used electronic prescribing only for ordering diagnostic tests or therapeutic procedures.

Control: We only included studies wherein either an electronic prescribing system was compared with handwritten ordering, or where an electronic prescribing system with a more sophisticated functionality (e.g., a drug-drug interaction alert) was compared with another less sophisticated system.

Population and setting: We included studies where physicians were the primary users of the electronic prescribing system. We excluded studies where other groups (e.g., nurses, pharmacists) were the primary users. We included all clinical settings such as outpatient care, inpatient care, and intensive care. We included all patient groups.

Study design: We included randomized controlled trials, non-randomized controlled trials, before-after trials, as well as time-series analysis with multiple measurements. We only included field studies and excluded all lab studies and simulation studies.

Outcomes: We included studies that evaluated the effect of electronic prescribing on medication errors, potential ADEs, and ADEs. We defined medication error as all errors in the process of ordering, transcribing, dispensing, administering, and monitoring medication. This included an inappropriate drug, dosing, frequency, route, or timing (when related to patient safety), involving problems such as illegible or unsigned orders, and problems related to drug-allergy, drug-drug, drug-lab, and other interactions. Potential ADEs were defined as a medication error with significant potential to harm a patient that may or may not actually reach a patient. Adverse drug events (ADE) were defined as patient injuries resulting from drug use. We excluded studies where medication errors or ADEs were not the primary focus of the study, and studies that were still ongoing. We excluded papers if groups were definitely not comparable. If the data reporting was unclear, we contacted the authors and requested further information.

Data Extraction and Study Quality Assessment

We extracted data from the text, tables, and graphs of the original publications. Two reviewers (EA and CM) examined the data and reached consensus after discussion. In addition, one reviewer (PSI) independently reviewed all the extracted data. All of the cases of discordant data were resolved by discussion.

To detect medication error rates and ADE rates, we used the definition that is provided in each paper (please see Table 3, available online at www.jamia.org, for the definition of medication error and ADE for each respective paper).

When no absolute numbers were provided for medication error or ADE, we calculated these numbers based on the given data (e.g., if the frequency of ADEs was only given per 1.000 patient-days, the absolute ADE number could be calculated from the number of patient-days). To determine the study size, we used the number of orders as an observation unit of the analysis. If the number of orders was not available, the number of patients or patient-days was used. If multiple data were reported in a study (e.g., in time-series analysis), we used the data of the last reported measurement.

We classified the functionality of the CPOE system either as

• • no decision support: selection of drugs from a list, information on available doses and on costs, access to drug monographs, no further decision-support;

• • limited decision support: evidence-based patient-specific recommendation of a drug, dosing, frequency etc.; or

• • advanced decision support: at least some drug-allergy, drug-drug interaction, drug-lab, or other patient-specific alerts.

All results were reported in systematic evidence tables. The study quality was assessed by using a checklist (please see Table 1, available online at www.jamia.org), which was developed based on a 16-item assessment tool by the German Scientific Working Group Technology Assessment for Health Care that was applied independently by two reviewers (PSI, EA). Differences in judgment were then solved by discussion.

Statistical Analysis

For each study, the risk ratio (RR) with its 95% confidence interval (CI) was calculated by comparing medication error rates, potential ADE rates, and ADE rates between the intervention and comparison group. If available, the number of orders was used as the denominator. Otherwise, the number of patient-days or the number of patients was used as the denominator.

We used a graphical approach based on forest plots to perform sub-group analysis and arranged studies by increasing risk ratios within subgroups. Subgroups were a priori defined as potentially relevant such as the clinical setting (inpatient, outpatient, or intensive care), patient group, type of drugs, type of system (home-grown or commercial), functionality (no, limited, or advanced decision support), study design, and method to detect errors.

All the analyses were performed with the software package STATA 9.2 (StataCorp, College Station, Texas, USA).

Results

Overall, we identified 172 evaluation studies and 15 systematic reviews. From the 172 evaluation studies, 27 studies met all the inclusion criteria and were, therefore, included in a detailed review. ^8,19–44 For details, see Figure 1, available online at www.jamia.org. We obtained additional data from eleven authors.

Characteristics of the Trials

▶ shows the general characteristics of all the analysed 27 studies.

Table 2.

Table 2 Characteristics of 27 Studies on Electronic Prescribing

Paper	Time of study	Name of hospital	Clinical setting	Level of care	Patients included in study	Drugs included in study	Name of system	Type of system	Electronic prescribing introduced since when	Description of functionality	Type of functionality	Control situation
Bates (1998)	1993–1995	Brigham and Women's Hospital, Boston, USA	Six intensive, medical and surgical units in a 700-bed academic tertiary-care hospital	Normal care	No restrictions	No restrictions	rICS	Home grown	Electronic prescribing introduced as part of BICS since 1993 with increasing functionality	CPOE system with default doses, display of lab results, consequent orders, limited drug-allerg, dru-drug interaction checking and drug-laboratory checking	Advanced DSS	Orders were handwritten in order books
Bates (1999)	1992–1997	Brigham and Women's' Hospital, Boston, USA	One intensive and two general medical care units in a 700-bed academic tertiary-care hospital	Normal care	No restrictions	No restrictions	BICS	Home grown	Electronic prescribing introduced as part of BICS since 1993 with increasing functionality	(Period 3): CPOE system with display of lab values, drug-allergy checking, redundant medication checking, drug-drug interaction checking, drug-lab checking, order sets	Advanced DSS	Orders were handwritten in order books
Bizovi (2002)	1999–2000	Oregon Health & Science University, USA	Emergency department of a tertiary care teaching hospital	Normal care	Emergency care patients	No restrictions	EmSTAT, CyberPlus Corporation (in use since 1997)	Commercial	Electronic prescribing was introduced in the emergency department in 2000	Prescription writing module with display of patient name, age, sex, patient's weight	No DSS	Handwritten prescriptions
Chertow (2001)	1997 - 1998	Brigham and Women's Hospital, Boston, USA	Medical, surgical, neurology, and obstetrics and gynaecology services of 720-bed urban tertiary care teaching hospital	Normal care	Patients with renal insufficiency	Renally cleared or nephrotoxic medications	BICS	Home grown	Electronic prescribing introduced as part of BICS since 1993 with increasing functionality	Decision-support system that calculates estimated creatinine clearance and proposes a recommended dose and frequency for the ordered drug	Limited DSS	CPOE without special advice for patients with renal insufficiency with regard to dose and frequency
Colpaert (2006)	2004	Ghent University Hospital, Belgium	22-bed intensive care unit at tertiary care hospital	Intensive care	Intensive care patients	No restrictions	Centricity Critical Care Clinisoft, GE Healthcare Europe	Commercial	Electronic prescribing was implemented in the intensive care unit 10 months before beginning of study	ICU with incorporated CPOE: Protocol-based recommendation on drug, dose and frequency, alerts to important drug-drug interactions and drug-related complications, highlighting of patient allergy status	Advanced DSS	Paper-based ordering
Cordero (2004)	2001–2002	Ohio State University Medical Center, USA	Neonatal intensive care units	Intensive care	Very-low-birth-weight infants	Caffeine and Gentamicin	Invision 24, Siemens Medical Solutions	Commercial	Electronic prescribing was introduced to selective services in February 2000 and introduced to the study units in March 2002	CPOE with best-practice based order sets, drug-allergy checks, drug-drug interactions, order duplication, corollary orders, weight-based dosage, maximum dosage, drug-route restriction	Advanced DSS	All order were entered manually in the chart kept at the patient bedside
Evans (1998)	1992–1995	LDS Hospital, Salt Lake City, Utah, USA	12 bed shock trauma respiratory intensive care unit	Intensive care	Intensive care patients	Antiinfectives	HELP	Home grown	The antiinfectives management program was implemented in the study unit in 1994	Computerized antiinfectives-management program with advice on antiinfective regiment (drug, dosing, dosing interval) based on clinical data and available evidence, drug-allergy checks, drug-drug interactions, information on toxicity and costs	Advanced DSS	No antiinfectives-management program was used
Feldstein (2006)	2000–2004	Kaiser Permanente Northwest, a health maintenance organization, USA	8 primary care clinics with 126 primary care clinicians	Normal care	No restrictions	Warfarin	EpicCare EMR, Epic Systems ∗ (in use since 1996)	Commercial	Warfarin alerts were implemented in January 2003	CPOE system with alerts on five warfarin-interacting drugs	Advanced DSS	CPOE without alerts
Fraenkel (2003)	1995–1998	Royal Brisbane Hospital, Queensland, Australia	12-bed adult general intensive care unit in a 790-bed tertiary referral teaching hospital	Intensive care	Intensive care patients	No restrictions	CareVue, Philips Medical Systems	Commercial	Intensive care system with electronic prescribing was introduced on study unit in November 1996	Medication administration record module with pre-defined dose ranges, drug scheduling and drug preparation notes, prompt for drug administration	No DSS	Paper-based documentation
Galanter (2005)	2002–2003	University of Illinois Hospital and Medical Center, USA	Not further described	Normal care	Patients with renal insufficiency	Renally cleared or nephrotoxic medications	Millenium and Discern Expert, Cerner Corporation	Commercial	Alerts were implemented during study period	CPOE system that calculates the creatinine clearance and checks for potentially contraindicated drugs	Advanced DSS	CPOE without alert on potentially contraindicated drugs for patients with renal insufficiency
Gandhi (2005)	1999–2000	Four Boston adult primary care practices affiliated with an academic medical center, USA	4 primary care practices	Normal care	Outpatients over Age 18	No restrictions	One commercial, one home-grown system	Mixed	No further information provided	Computerized prescribing with printed prescriptions, required fields and non-mandatory default doses, without any automatic checks	No DSS	Hand-written prescriptions
Igboechi (2003)	1999–2002	Metropolitan Hospital Center, New York, USA	Inpatient and outpatient services of 350-bed acute tertiary care hospital	Normal care	No restrictions	No restrictions	Ulticare System Database, Per Se Technologies	Commercial	Electronic prescribing was introduced in June 2001 after recent upgrade to a new clinical system	CPOE system with drug-allergy alerts, drug interaction alerts, dosage range checks, lab values, and drug information resource	Advanced DSS	Handwritten orders
Kim (2006)	2001–2004	John Hopkins Children's Center, Baltimore, USA	Paediatric oncology	Normal care	Children with neoplastic disorders	Chemotherapeutica	RxTFC Pharmacy Information Systems, GE Medical Systems Information Technologies	Commercial	Electronic prescribing was introduced as part of the available pharmacy system in 2002	CPOE system with dosing support and drug-allergy checking ∗	Advanced DSS	Paper-based order entry
King (2003)	1993–1999	Children's Hospital of Eastern Ontario, Canada	Three medical and two surgical wards of a tertiary care paediatric hospital	Normal care	Children	No restrictions	Sunrise Clinical Manager, Eclyps	Commercial	Electronic prescribing was introduced, with periodic upgrades, on the two intervention wards in 1996	CPOE without decision support (no further information provided)	No DSS	Handwritten order entry
Kirk (2005)	2003	National University Hospital, Singapore	Paediatric unit in a university teaching hospital	Normal care	Paediatric patients (< 16 years of age)	Paracetamol and promethazine	Not specified	Home grown	Existing clinical decision support system (in use since 2000) system was modified in March 2003 to also support electronic prescribing for paediatric departments	Computer calculated dosing based on weight	Limited DSS	Traditional computer-based prescription, no dosing advice
Koide (2000)	1994–1996	University of Tokyo Hospital, Japan	1040-bed tertiary care and teaching hospital	Normal care	Patients with psoriasis	Etretinate, a medication indicated for psoriasis	Medical information system (in operation since 1973)	Unknown	Electronic prescribing in operation since 1988, alerts for etretinate added in 1995	Alert system monitors ALT values and AST values	Limited DSS	CPOE without alerts
Maurer (2003)	1999–2000	Centre Hospitalier Le Raincy-Montfermell, Montfermell, France	General Pneumology Unit with 60 beds	Normal care	Pneumology Patients	No restrictions	Excel-based application	Home grown	Electronic prescribing introduced in January 2000	Prescription sheet with drug list and recommended dosing based on weight and renal function	Limited DSS	Handwritten prescriptions
Mitchell (2004)	2002	Southmead Hospital, Bristol, UK	General surgery	Normal care	No restrictions	No restrictions	Clinical Manager 3.0A, iSoft UK PLC	Commercial	Electronic prescribing introduced in general surgery as 3-months-pilot in 2002	Comparing unit with electronic medication administration record and electronic prescribing with areas of hand-written prescriptions	No DSS	Paper-based ordering
Mullett (2001)	1998–1999	Primary Children's Medical Center, University of Utah, USA	26-bed paedriatric intensive care unit of a 232-bed primary children's medical center	Intensive care	Children	Antiinfective	HELP	Home grown	Pediatric antiinfectives decision support tool was implemented on intensive care units in January 1999	Computerized antiinfectives-management program with dosing recommendations based on evidence and clinical data	Limited DSS	Handwritten orders
Oliven (2005)	2001 ∗	Bnai-Zion Medical Center, Haifa, Israel	Two 44-beds general medical wards in an acute care university hospital	Normal care	No restrictions	No restrictions	Unknown	Home grown	Electronic prescribing was introduced three years before the study	CPOE system with administration reminders, reminders for blood-drug-level test or further monitoring, extended drug information, treatment guidelines, drug-drug interaction, drug-allergy, drug-disease, drug-lab alerts, access to blood test results	Advanced DSS	Handwritten drug orders
Pestotnik (1996)	1988–1994	LDS Hospital, Salt Lake City, Utah, USA	520-bed acute-care referral hospital	Normal care	No restrictions	Surgical antibiotics	HELP	Home grown	Surgical antibiotics management modules were developed since 1985	Antibiotic management program with patient-specific guideline-based clinical decision support	Advanced DSS	No antibiotic management program available, computerized ordering mainly related to nursing orders and supply orders ∗
Peterson (2005)	2001–2002	Brigham and Women's' Hospital, Boston, USA	Medical, surgical, neurology, and gynaecology services of tertiary care academic hospital	Normal care	Elderly patients (65 years and older)	Psychotropic medication	BICS	Home grown	Decision-support for psychotropic medication was developed and implemented for the study	CPOE with evidence-based geriatric-specific recommendations for dosing and substitution	Limited DSS	CPOE with dosing suggestions and recommendations for substitutions
Potts (2004)	2001–2002	Vanderbilt Children's Hospital, Nashville, USA	20-bed paediatric critical care unit in a tertiary-care children's hospital	Normal care	Children	No restrictions	WizOrder (developed in 1994)	Home grown	CPOE was implemented in critical care unit in December 2001	CPOE with decision support such as drug allergy alerts, dose checking, drug interaction alerts, clinical pathways-based order sets, links to drug monographs, drug-lab alerts	Advanced DSS	No CPOE
Shulman (2005)	2001–2002	University College Hospitals London, Middlesex hospital, London, UK	Intensive Care Unit	Intensive care	Intensive care patients	No restrictions	QS 5.6 Clinical Information System, GE Healthcare	Commercial	QS with electronic prescribing was introduced in April 2002	CPOE component without decision support	No DSS	Hand-written drug charts
Spencer (2005)	2002–2003	University of North Carolina, Chapel Hill, USA	Two general medicine floors of a 688-bed academic hospital	Normal care	No restrictions	No restrictions	Siemens Medical Solutions, Malvern, PA	Commercial	Siemens CPOE system was implemented on two study units in October 2002 resp. January 2003	Patient allergy screening was partly in use ∗	Advanced DSS	No CPOE
Tamblyn (2003)	1997–1998	General Practicioners in Quebec, Canada	107 primary care physicians	Normal care	Elderly patients (66 years and older)	No restrictions	Unknown	Commercial	Electronic prescribing was introduced to intervention group physicians two months after randomization	Health-record software which documented medication prescribed, offering drug-disease contraindications and information on previous drugs	Advanced DSS	No computerized decision support
Teich (2000)	1991–1995 ∗	Brigham and Women's' Hospital, Boston, USA	720-bed urban academic medical center	Normal care	Adults	No restrictions	BICS	Home grown	Electronic prescribing introduced as part of BICS since 1993 with increasing functionality	CPOE system with dosing recommendation, drug-allergy, drug-drug interactions, duplicate medications, possible alternative medication, consequent order recommendation	Advanced DSS	Paper-based ordering

^∗ this information was provided by the author upon request.

The majority of studies were conducted in the United States and in normal inpatient care units. Most studies were unspecific concerning the included patients and concerning the ordered drug. Half of the studies evaluated a commercial system, and half evaluated a home-grown, self-developed system. All of the systems offered a display of basic patient data (name, age), a list of drugs from which a provider can choose, and a list of potential doses for a chosen drug. Half of the studies used a system with advanced decision support, the others used systems with no or limited decision support. In most trials, electronic prescribing was compared to handwritten ordering.

Table 3, available online at www.jamia.org, shows the further study characteristics of the 27 studies, showing that most of them were before-after studies (including time-series analysis), with only two studies being randomized controlled trials.

Of those 27 studies, 15 only evaluated medication errors, two were only on ADE, and 10 reported both on medication errors and ADEs. Table 4, available online at www.jamia.org, shows the evidence table for outcomes.

Effect on Medication Errors

Twenty-five studies reported on the risk of medication errors (▶). Twenty-three of these studies showed a significant relative risk reduction, with a risk ratio between 0.01 and 0.87. This indicates a relative risk reduction for medication errors of 13% to 99%. One study reported a statistically significant increase of 26% for the risk of medication errors (Spencer 2005). ⁴² In this study, there was no connection between the CPOE system and the pharmacy system, so that orders had to be entered twice, in turn leading to transcription errors in the intervention group. In addition, the study authors only analysed those errors that were reported by the staff members themselves (voluntary reporting system)—this constitutes a potential detection bias. The one study (Mitchell 2004) ³⁵ showing inconclusive results (RR = 1.02, 95%-CI: 0.88; 1.19) was one of the smallest studies in this review, with only 320 orders analysed.

Figure 2.

Risk ratios of 25 studies analyzing the effect of electronic prescribing on medication errors.

Effect on Potential Adverse Drug Events

Nine studies reported on the risk of potential ADE (▶). Six of these studies showed significant relative risk reduction, with a risk ratio between 0.02 and 0.65. This indicates a relative risk reduction for potential ADEs of 35% to 98%.

Figure 3.

Risk ratios of nine studies analyzing the effect of electronic prescribing on potential ADEs.

Two studies showed inconclusive effects (Bizovi 2002 and Gandhi 2005). ^20,28 Both of these studies were rather small studies. The remaining also very small study by Mitchell 2004 showed a significant increase in the risk for potential ADEs. ³⁵

Effect on Adverse Drug Events

Seven studies reported the risk of ADE (▶). In one of them (Shulman 2005), ⁴¹ no events occurred in the intervention or in the comparison group and, therefore, this study was excluded from further analysis.

Figure 4.

Risk ratios of seven studies analyzing the effect of electronic prescribing on ADEs. Shulman was excluded, because there was no event in either the intervention group or the comparision group.

Four of the six remaining studies showed a significant relative risk reduction for ADEs, with a risk ratio between 0.16 and 0.70. This indicates a relative risk reduction for ADEs of 30% to 84%. One study (Mullett 2001) ³⁶ showed a not statistically significant relative risk reduction of 13%, and the remaining study (Bates 1998) ¹⁹ showed a small, not statistically significant increase of 9%.

Sub-group Analysis

A graphical sub-group analysis was only performed for those 25 studies that focused on medication errors.

The sub-group analysis comparing medication error reduction by commercial systems versus home-grown systems indicated a higher relative risk reduction by home-grown systems (▶).

Figure 5.

Sub-group analysis of the effect of home-grown versus commercial systems on the relative risk reduction of medication errors (n=25).

The sub-group analysis for those seven studies that compared advanced electronic prescribing with limited electronic prescribing and those 18 studies that compared electronic prescribing with handwritten ordering showed a higher relative risk reduction for the comparison made with handwritten ordering (▶).

Figure 6.

Sub-group analysis of the type of comparison group on the relative risk reduction of medication errors (n=25).

The 14 studies with an advanced decision-support reported a higher relative risk reduction than the 11 studies with limited or no decision-support (▶). Most studies with limited support, however, were compared to computer-based ordering, whereas most studies with advanced decision-support or without decision-support were compared to handwritten ordering.

Figure 7.

Sub-group analysis of the effect of the level of functionality on the relative risk reduction of medication errors (n=25).

The sub-group analysis comparing different methods to detect errors showed higher relative risk reductions with the manual chart review of prescriptions (11 studies) than with the automatic database analysis of prescriptions (6 studies) (please see Figure 8, available online at www.jamia.org).

The other graphical sub-group analysis did not indicate differences between groups: The risk ratios between groups seem to be similar for the level of care (normal care versus intensive care), patient groups included (elderly, children, unspecific), type of drugs included (specific or unspecific), or study design (before-after trial, controlled trial, RCT).

Study and Reporting Quality

The reporting quality was poor for most of the studies. The details are shown in Table 5, available online at www.jamia.org. For example, many studies did not clearly specify the inclusion or exclusion criteria of the participating institutions or patients, did not report on the baseline characteristics of participants or as to whether the comparison and intervention groups were treated similarly (with the exception of the intervention), or did not comment on missing values or drop-outs.

In addition, only two studies were randomized trials, and few studies had comparison groups recruited over the same period. Only in half of the studies, the outcome seemed to be measured validly and reliably, however, the measurement was mostly not blinded. Nearly all the studies used adequate hypothesis tests and reported on the statistical precision of the main outcomes, but only less than half of the studies attempted to adjust for confounding or clustering. Most studies were single centre studies, so that the generalisability of the results to other centres, especially of a different type, is unclear.

Discussion

Main Findings

Thwenty-three of the 25 studies that analysed the effects on the medication error rate showed a significant relative risk reduction of 13% to 99% for medication errors. Six of the nine studies that analysed the effects on potential ADEs showed a significant relative risk reduction for potential ADEs of 35% to 98%. Four of the seven studies that analysed the effect of electronic prescribing on ADEs showed a significant relative risk reduction for ADEs of 30% to 84%. These findings, that are in line with results from other reviews, indicate that electronic prescribing can substantially reduce the risk for medication errors, potential ADEs, and ADEs. 12,13,16,45–47 The low number of studies focussing on ADE may be due to the fact that medication errors can be detected more easily than ADEs. ¹⁶

From a clinical perspective, the observed effect sizes (e.g., up to 99% reduction of medication errors, up to 84% reduction of ADEs) seems to be rather large. However, the medication errors and ADEs are just surrogate outcomes, and are not necessarily directly related to changes in the patient-relevant medical outcomes. ⁴⁸ The actual improvements in medical outcomes (e.g., reduction in mortality rates or hospitalisation days) have not yet been sufficiently analysed by quantitative, controlled trials. ^15,45,48 One exception is the study by Han et al. ¹⁰ that observed an increase in mortality after the introduction of a CPOE system (these heavily discussed findings could not be reproduced later on by others such as Del Baccaro et al. ⁴⁹ ). Altogether, more systematic trials looking at patient-relevant medical outcomes should be conducted in the future. Furthermore, more studies should also be conducted to analyse the costs for any benefit that may be obtained. For this review, we concentrated on medication errors and ADEs, as a sufficient amount of published evidence i s available.

Results of the Sub-group Analysis

The included studies showed substantial heterogeneity (see their detailed description in ▶ as well as in Table 3, available online at www.jamia.org). This reflects the diversity of electronic prescribing systems and their respective use.

The sub-group analysis indicated that the level of care, as well as the included patient groups and drugs did not affect the overall risk ratios. The graphical presentation, however, indicated that home-grown systems show a higher relative risk reduction than commercial systems. This is not surprising, as home-grown systems can be modified and adapted easier and more quickly to the local needs. Rigby ⁵⁰ calls this “alpha sites” and argues that those sites are atypical concerning larger technical, emotional, and financial support.

Also, studies comparing handwritten prescribing with electronic prescribing seem to show a higher relative risk reduction than those studies comparing groups with a different level of electronic ordering. This is also not surprising, as certain error types such as illegible orders, that are often present in handwritten prescribing, are completely removed by electronic prescribing.

Electronic prescribing systems with advanced decision support seemed to show a higher relative risk reduction compared to those with limited or no decision support. It is here that the patient-specific alerts seem to best support the quality of prescribing.

Finally, different methods for detecting medication errors seem to show a different relative risk reduction. Using a manual chart review, relative risk reduction seems to be highest. However, chart review can hardly be blinded, and reviewers may be biased towards CPOE.

Study Quality

Overall, the reporting quality and study quality of the included studies were not always satisfactory.

One major problem was that many studies did not provide sufficient information in order to adequately assess the comparability of the intervention and comparison groups (e.g., by not reporting baseline characteristics, by not reporting about adjunct initiatives such as training sessions or about drop-outs etc.). This made it difficult to analyse as to whether the differences found between the study groups really originated from electronic prescribing, or from other factors. Subsequently, about two-thirds of the studies did not attempt to adjust for possible confounding factors. Then, many studies also used designs of lower level of evidence such as before-after trials, here it is unclear whether any context such as staffing or workflow of the study departments may have changed over time, in turn influencing the observed effects. The same is true for any non-randomized allocation of clinicians and/or patient to study groups. This all affects the validity of the analysed studies. We dealt with this by describing in detail the individual study quality (see Table 5, available online at www.jamia.org), and by excluding those studies that seem to have non-comparable groups. ^51,52

There were other limitations of the included studies: Four studies analysed the data from voluntary error reporting systems (e.g., nurses and physicians voluntary reported on errors), which may underestimate the real number of errors and ADE in those studies (detection bias). Other studies retrospectively analysed routinely collected information on medication errors and ADEs. Here, it is partly unclear as to how they assessed the quality of this routine data. Overall, only a few studies provided information on the objectivity and reliability of the instruments that were used to (manually or automatically) detect errors.

Summarizing, the reporting and study quality were not always adequate. This must be taken into account when interpreting the results of the studies. Efforts should be made to improve the reporting quality and study design and analyses of evaluation studies. Publication guidelines that are comparable to CONSORT ⁵³ or STROBE ⁵⁴ should be developed, taking into account the often quasi-experimental design of the health informatics evaluation studies. One example for such publication guidelines for health informatics evaluation studies are the STARE-HI guidelines (Standards for Reporting on Evaluation Studies in Health Informatics) that will shortly be published. ⁵⁵

Limitations of the Review

Our review may present some limitations.

First, the published trials on electronic prescribing differ substantially in their respective terminology. We used many different Keywords to search for trials; searched in several databases as well as for non-English papers; and applied hand-searching. Nevertheless, we may have overlooked some individual studies. In addition, there is the possibility of publication bias, as we have only included published studies.

Second, the definition of the measured outcome variable differed between the studies. For example, some studies included the legibility of orders in the definition of medication, and others did not. This can affect the measured medication error rate.

Third, when several end-points were reported (e.g., in a time-series analysis), we only included the final endpoint. This may overestimate the positive effects of electronic prescribing, as the intermediate results partly show less (or even negative) effects during an early introduction period. In addition, our analysis relied on the assumption that one order could have maximal one error. However, this was not explicitly reported by several studies.

Fourth, our categories that were used to stratify the sub-groups might have been too crude. For example, the study design was very heterogeneous and the diversity might not have been represented adequately by the categories we chose. A multivariate sub-group analysis controlling for the multiple factors simultaneously was not performed because of the limited number of studies. Then, further factors such as the organisation of medication processes in the various settings, age and computer knowledge of users, management support, or the quality and usability of the CPOE system may further affect the effects. Information on those items was not included in most studies and could, therefore, not be used for sub-group analysis.

Fifth, the rather large percentage of home-grown systems in our review may have biased this review towards more positive results. Moreover, 8 of the studies included came from just two sites: the Brigham and Women's Hospital, Boston, MA with the BICS system, and the LDS Hospital, Salt Lake City, Utah, with the HELP systems. Therefore, we cannot exclude a selection bias, with the observed effects not being representative for general routine clinical settings. Some studies on BICS and HELP may not be completely independent of each other (we therefore excluded Evans et al., 1994), ⁵⁶ but when they used different outcome criteria or addressed different patient populations, we decided to include them.

Sixth, we were unable to use meta-analysis to pool the effect sizes, given the large heterogeneity among the studies. Meta-analysis would help to quantify the overall effect, and has already been performed e.g., for computer-assisted management of anticoagulant therapy ⁵⁷ or for clinical reminder systems for preventive care, ⁵⁸ but needs a sufficient number of homogeneous studies. We therefore conducted a graphical sub-group analysis—even this, however, should be viewed with care, given the tremendous variability in study design and in the methodological rigor of the included studies.

Seventh, most of the evaluated systems have only been implemented shortly before or during the study, and sometimes only as pilots in selected areas. This may have led to erroneous estimations concerning the long-term effects of electronic prescribing. More long-term studies seem to be needed.

Finally, researchers such as Berger ¹⁵ note that electronic prescribing systems could increase the number of medication errors and ADEs when not appropriately designed and implemented. There is, in fact, increasing evidence from (mostly qualitative) research pointing to risks with electronic prescribing by CPOE systems. Koppel et al., ⁹ e.g., described an increase in medication errors risks by CPOE. Campbell et al. ⁵⁹ identified categories of unintended consequences of CPOE. This (qualitative) evidence is, however, not (yet) reflected in the results of quantitative controlled evaluation studies and therefore also not in our review. The discussion of the reasons for this (such as a possible publication bias) is outside the scope of the present paper. For a more detailed discussion see Ammenwerth et al. ⁶⁰

Implication and Conclusion

After having been envisioned by Morris Collen in 1970 (“Physicians should enter their medical orders directly into the computer”), ⁶¹ electronic prescribing systems have now been in routine use for approximately 15 years. Our review is, to our knowledge, the first that quantitatively calculates and compares the risk ratios for each evaluation study. We also conducted a sub-group analysis, hereby analysing factors that influence the effects of electronic prescribing; and we analysed and discussed carefully the heterogeneous and often weak quality of published studies.

Our results indicate that electronic prescribing seems to be a useful intervention for reducing the risk of medication errors and ADEs. We found that there is especially good evidence for a positive effect of electronic prescribing offering advanced decision support functionality in hospital settings. Less evidence is available for systems outside the U.S., for primary care settings, and for specific commercial systems. Therefore, more studies are necessary.

The study quality was often weak, with many before-after trials and an unclear comparability of the study groups. More randomized controlled trials from more sites are needed to further improve the evidence and to identify the setting that those systems are most useful in. Researchers planning comparable studies are advised to focus early on the issue of study quality and study validity, to avoid as many sources of bias as possible (see also Brender, 2002 ⁶²⁾ . A larger number of high-quality studies for certain settings and patient groups would also allow for the use of meta-analysis to pool the effect sizes. In addition, the reporting quality of health informatics evaluation studies has to be substantially improved; here, the initiatives for publication guidelines such as STARE-HI ⁵⁵ are urgently needed. All this should help to increase our knowledge on electronic prescribing and globally increase the evidence-base of health informatics. ¹⁷

Systematic reviews of quantitative trials can indicate which effect can be obtained from an intervention. They do not show, however, how to implement electronic prescribing. It seems obvious that the quality of the implementation process has an impact on the success of a CPOE implementation. ⁶³ Insufficient implementation planning, or systems that are not integrated into the general information systems, may lead to negative effects on the process and even the outcome of care. ^64–67 Negative effects of electronic ordering were identified only in two studies in our review, even when increasing evidence from qualitative research points to possible adverse effects of electronic prescribing. Each implementation should be carefully monitored concerning an increase in medication errors and ADEs, and more qualitative research should be conducted to assess the reasons for the respective success or failure, and to guide further successful implementation processes.