Evaluation of Outpatient Computerized Physician Medication Order Entry Systems: A Systematic Review

Abstract

This paper provides a systematic literature review of CPOE evaluation studies in the outpatient setting on: safety; cost and efficiency; adherence to guideline; alerts; time; and satisfaction, usage, and usability. Thirty articles with original data (randomized clinical trial, non-randomized clinical trial, or observational study designs) met the inclusion criteria. Only four studies assessed the effect of CPOE on safety. The effect was not significant on the number of adverse drug events. Only one study showed a significant reduction of the number of medication errors. Three studies showed significant reductions in medication costs; five other studies could not support this. Most studies on adherence to guidelines showed a significant positive effect. The relatively small number of evaluation studies published to date do not provide adequate evidence that CPOE systems enhance safety and reduce cost in the outpatient settings. There is however evidence for (a) increasing adherence to guidelines, (b) increasing total prescribing time, and (c) high frequency of ignored alerts.

Introduction

A 1999 Institute of Medicine report estimated that about 80,000 people are hospitalized and 7,000 die annually in the United States due to medication errors in the inpatient setting. ¹ Of these errors, 32–69% are definitely or possibly preventable. From an economic point of view, hospital costs of preventable adverse drug events were estimated at $2 billion per year. Similar reports in other countries show that medication errors indeed have important impact on mortality, morbidity, and cost of care. ^2,3 Medication errors have been defined as any error in the medication process, which comprises taking history, ordering, pharmacy management, administration management, and surveillance. A medication error may or may not result in patient harm. Adverse drug events (ADEs) are usually considered to include both medication errors that result in harm (preventable ADEs) and adverse drug reactions (ADRs), which are considered unpreventable. ⁴

Although failure to monitor patients ⁵ and work pressure ⁶ have been reported as possible causes of medical errors, adverse outcomes are more commonly associated with the problems during the medication process. ⁷ The ordering step is crucial in the process: it is the point at which the physician’s thoughts are transformed to decisions which trigger a series of actions, ultimately resulting in the patient receiving the medication. ⁷

The Institute of Medicine and other important stakeholders have identified Computerized Physician Order Entry (CPOE) or Electronic Prescription (EP) as a key to reduce medication errors and improve safety. ^1,8 In this article, the term CPOE refers specifically to medication ordering.

In an outpatient setting, patient information is often scattered among various paper and electronic information systems, rendering the identification of adverse drug events and other outcomes very difficult. ⁹ This complicates the evaluation of CPOE systems in the outpatient setting. Not surprisingly, work on evaluation of CPOE systems has focused on the inpatient setting where CPOE systems have been shown to decrease medication errors, especially when the CPOE systems include decision support. ^10–16 However, most prescriptions occur in outpatient settings and, due to less control in this environment, more errors might be expected to occur with outpatient prescriptions. A meta-analysis suggested that in 1994, more than one million outpatients in the United States experienced an ADE that required admission to the hospital. ¹⁷ Of these ADEs 106,000 were fatal, placing them between the fourth and sixth leading causes of death, although these projections may have been somewhat overestimated. ^17,18

The main objective of this systematic review was to identify and summarize published studies of outpatient CPOE systems that evaluated one of six aspects: safety; cost and efficiency; adherence to guidelines; alerts; time; and satisfaction, usage, and usability.

For the purposes of this review, an outpatient CPOE system is a computer-based system that allows clinicians to enter medication orders directly for outpatients or primary care patients. In this context, a decision support system (DSS) is any system designed to directly aid a health professional in decision-making during medication ordering.

Methods

We searched for relevant English language articles based on keywords in title, abstract, and MeSH terms, using Ovid MEDLINE^® (1950 to March 31, 2006) Ovid MEDLINE In-Process^®, and EMBASE^® (1980 to March 31, 2006). The final literature search was performed on March 31, 2006.

▶ shows the search strategy used to identify the relevant articles. In the first part (A), we applied keywords without quotes and MeSH terms pertaining to electronic prescription. These terms cover old and new ways to refer to CPOE systems. In the second part (B), we searched for medication related terms to identify studies that address prescribing. In the third part (C), we searched for terms related to outpatient care. The results of these three parts were combined using the Boolean operator “and.” Searching was supplemented by scanning bibliographies from identified articles.

Figure 1.

Keywords and MeSH terms used in the search strategy (words in bold are MeSH terms) and the search flow.

Two reviewers individually examined all titles and abstracts. Discrepancies among the two reviewers were resolved by consensus involving a third reviewer. Articles were selected if they reported original data from a study in an outpatient setting and if one of their main objectives concerned evaluation (a) of a CPOE system for medication ordering and/or (b) of a DSS used during the medication ordering. Study designs include clinical trials as well as observational studies such as questionnaire surveys on satisfaction, and simulation studies. All studies reporting on alerts, reminders, and DSS which are not part of the CPOE system and which are not triggered during medication order entry were excluded. Opinion papers, letters, and evaluation studies of vaccination ordering systems and stand-alone programs for ordering specific drugs such as Warfarin or Digoxin were excluded.

From the selected papers, the same two reviewers extracted data on the demographics (such as number of physicians, number of patients, and duration of study), study design, outcome measures, and results. Discrepancies between these two reviewers were again resolved by consensus involving the third reviewer.

We listed all reported outcome measures and then categorized them into outcome groups. The measured effects were classified as being one of: statistically significant positive effects; demonstrated positive effects (when the authors report a positive effect but without reporting statistical significance); a mix of statistically significant and demonstrated positive effects; no effect (when reported as such by the authors, with or without statistical arguments); statistically negative effects; and, finally, a mix of positive, absence of, and negative effects. To get insight into the heterogeneous nature of these evaluation studies we classified them according to the hierarchy of study designs developed by the University of California San Francisco Stanford Evidence-Based Practice Center and implemented by Kaushal et al. ¹⁶ in their review (▶).

Table 1.

Table 1 Hierarchy of Study Designs

Level	Study design	Description
I	Randomized Controlled Trial (RCT)	A study in which people are allocated at random (by chance alone) to receive one of several clinical interventions. One of these interventions is the standard of comparison or control. The investigator controls the exposure to the intervention.
II	Non-Randomized Controlled Trial	A study in which people are allocated to receive one of several clinical interventions. One of these interventions is the standard of comparison or control. The investigator controls the exposure to the intervention but allocation of people is not based on chance. It includes interrupted time series and before-after studies.
III	Observational study with controls	A study in which individuals are observed or certain outcomes are measured without a specific attempt to affect the outcome (the investigator does not control the exposure to the intervention, e.g., the use of a CPOE). The intent is to observe how exposure to risk factors (implemented CPOE) influences the outcome of interest. Includes cross-sectional studies to estimate the prevalence of the outcome of interest or the prevalence of exposure to intervention or both; cohort (longitudinal) studies with control in which individuals who are exposed to the intervention are followed for a defined length of time and the effects of the intervention on the exposed group is compared to a group that was not exposed; and case control studies in which a comparison of exposure to the CPOE in a group of individuals with the outcome of interest (cases) is compared to those without the outcome of interest (controls).
IV	Observational study without controls	Includes cohort studies without controls or case series.

Results

Searching the online databases resulted in 1,032 articles from Ovid MEDLINE^®, Ovid MEDLINE In-Process^®, and EMBASE^® after removing duplicates. Initial screening of titles and abstracts rendered 52 articles eligible for further full text review. Five additional articles were identified by reviewing bibliographies, yielding a total of 57 articles for full-text review. Based on the full-text review, 19 studies were excluded because they turned out not to address a specific CPOE system or an outpatient setting. One study was excluded because the effects could not be attributed to the CPOE system alone. Two studies were excluded because they only predicted, instead of measuring, the effects. Finally, five studies were excluded because evaluation was not a main objective of the study, leaving 30 articles for detailed analyses. Two of the 30 papers reviewed were published in MEDLINE-indexed conference proceedings. We contacted the authors of these studies, and, to our knowledge, they were not published in a journal before this review was completed. The 30 studies are listed in Table 2 (available as a JAMIA on-line data supplement at www.jamia.org).

For all these studies, we categorized the main outcome measures in six main groups: medication safety; cost and (organizational) efficiency; adherence to guidelines; alerts and appropriateness of alerts; time; and satisfaction, usage and usability. Most papers evaluated multiple outcome measures.

▶ shows a summary of all measured effects of the thirty CPOE evaluation articles. Controlled study designs were also described by the effects of CPOE. Articles may have two or more listings in the table because they can describe studies with multiple outcome measures and study designs.

Table 3.

Table 3 Frequency of All Selected Articles According to Outcome Categories and Study Design

Outcome Category	Study Designs with Control										ODWC*
	Specific Study Design	Total	Positive Effect			No Effect	Negative Effect			Mixed Effect
			Demonstrated	Statistically	Mix		Demonstrated	Statistically	Mix
Safety	RCT	1				1 21
	non-RCT	1			1 22
	OWC	2	1 19			1 20
Cost and Efficiency	RCT	4				3 21,28,33			1 29
	non-RCT	5		2 31,25		1 23	1 24			1 30
	OWC	3	1 32	2 26,27
Adherence to Guideline	RCT	4			1 37	3 28,29,33
	non-RCT	4		3 25,35,38		1 30
	OWC	3	2 34,36			1 32
Alert	RCT										6 19,39–43
	non-RCT
	OWC
Time	RCT	1						1 44			1 46
	non-RCT	2						2 31,45
	OWC
Satisfaction, Usage and Usability	RCT	3				2 28,29		1 21			9 25,30,32,33,36,43,44,46,48
	non-RCT	2					1 23	1 35
	OWC	1	1 47
Total	RCT				1	9		2	1		16
	non-RCT			5	1	2	2	3		1
	OWC		5	2		2

ODWC = Observation design without control; OWC = observational with controls.

Controlled Studies are also described by the effects of CPOE.

Below we describe the effects per outcome group for all included studies.

Medication Safety

Only four studies assessed the effect of CPOE, all with a DSS, on safety. ^19–22 One retrospective observational study showed that there were no ADEs found in a set of randomly selected cases in which the physician accepted the alert on drug allergy or on high severity drug interaction (n = 67). ¹⁹ However, among the randomly selected cases in which alerts were ignored there were 3 ADEs found (n = 122, p = 0.55). Since the number of cases (n = 189) was limited, these results did not amount to a significant difference. Another prospective cohort study could not show a statistically significant difference in number of ADEs and preventable-ADEs between computerized and manual prescription systems. ²⁰ An RCT showed that there was no significant difference in the actual number of clinically relevant drug interactions between a control group and the intervention group which received alerts on interactions. However, usage of the system in this study was optional and very low (2.8% of medication orders were prescribed by CPOE). ²¹

A recent non-RCT study showed that the provider did not complete the medication order of 18 high-volume and high-risk medications when an alert for an abnormal rule-associated laboratory result was displayed (p = 0.03). This study did not show a statistically significant reduction in percentage of definite or probable ADEs (p = 0.23). ²²

Cost and (Organizational) Efficiency

Twelve studies evaluated the impact of CPOE on the physician’s office expenses, medication costs and (organizational) efficiency. ^21,23–33

Two studies, both non-RCT, showed that there was no significant effect on medication cost when using a price comparison module ²³ or prescription cost information ²⁴ but another study with the same study design showed that a CPOE system could reduce the cost of medication by suggesting equally effective but cheaper drugs. ²⁵ One recent cohort study, resulting in two articles, showed that clinicians who used electronic prescribing, with or without receiving a list of prewritten prescriptions and patient specific diagnostic information, had significantly lower prescription costs than those in the control group. ^26,27 Rotman et al. performed an RCT and evaluated the cost effect of a CPOE system including an interaction alert module. ²¹ There was neither an effect on the number of clinically relevant interactions nor a significant effect on the cost of prescribing drugs. Two other RCTs showed that a CPOE system with a DSS did not affect the number of emergency department visits significantly. ^28,29 At the same time, one of these two studies showed that there was no significant effect on the cost ²⁸ while the other showed that the cost significantly increased. ²⁹ One study, a non-RCT, evaluated the effect of a CPOE system on the physician’s office expenses and showed that these had increased. ³⁰ Three studies, one observational, one non-RCT, and one RCT, evaluated the effect of electronic prescription on physician office resources and showed a reduction in pharmacist interventions for prescriptions. ^30–32 One RCT showed there was no statistically significant effect on consultation rate. ³³

Adherence to Guidelines

Eleven studies evaluated the impact of CPOE with a DSS on the adherence to a guideline or another standard. 25,28–30,32–38 Among these, four studies showed that there was a significant positive effect on adherence; 25,35,37,38 two studies showed a positive effect without reporting on statistical significance; ^34,36 and five studies 28–30,32,33 did not find a significant difference between the control and the intervention group. Among the studies with a positive effect on adherence to guidelines, one used an RCT design, ³⁷ three used a non-RCT design ^25,35,38 and two were observational studies ^34,36 whereas in the studies without any effect three were RCTs, ^28,29,33 one was a non-RCT ³⁰ and one was observational. ³²

Alerts and Appropriateness of Alerts

Six observational studies ^19,39–43 assessed the impact of CPOE on the produced, accepted and ignored alerts from two points of view: system weakness and user response. Fernando et al. performed an observational study and showed important weaknesses in generating alerts in four commonly used commercial systems in Britain’s GP offices. ³⁹ None of them was able to generate all 18 predefined established alerts for contraindicated drugs and hazardous drug-drug combinations. ³⁹

The remaining five studies addressed user response to alerts. Four studies showed that most of the alerts (from 55% to 91.2%) were ignored by the physicians. ^19,40–42 Two studies showed that “clinically irrelevance” was the main reported reason for overriding alerts. ^41,43

Time

Three studies ^31,44,45 one RCT and two non-RCTs, showed that the total time for direct and indirect patient care increased due to the introduction of the CPOE system.

Another observational study showed that physicians did not believe that electronic prescription was more time consuming than hand-written prescription. ⁴⁶

Satisfaction, Usability, and Usage

Fifteen studies evaluated the impact of a CPOE system on user satisfaction and system usage and usability. Among them, five observational studies 25,30,36,43,44 showed that after the introduction of the CPOE system, the majority of users were satisfied with the system and they believed that the system is usable. Although the environment, questionnaire, and target groups were different in these studies, the majority of users believed that CPOE improved drug management and quality of care. Three others, ^21,23,35 one RCT and two non-RCT, showed that user satisfaction and usability decreased. Two other RCTs showed that patient satisfaction did not change significantly after introducing CPOE with decision support. ^28,29 There was a wide variability in the degree of CPOE usage. Four studies showed that of all prescriptions, 2.8% to 90%, were entered electronically 21,32,46,47 and another study showed that the levels of system usage were low. ³³ However, another study showed there was continued improvement in system usage and usability. ⁴⁷ Finally, a simulation study showed the relationship between usage of two decision support models and the complexity of the cases. ⁴⁸ During prescribing, physicians were more willing to use on-demand decision support as the clinical situation became more complex while for simple cases the reminder-based support was appropriate.

Discussion and Recommendations

We have identified and described the results of 30 papers on evaluation of CPOE systems in outpatients. The number of such evaluation studies has clearly increased since 2002 (only 10 articles out of 30 before 2002). We used extensive search criteria in order to capture the different ways a CPOE system is referred to in the published literature. However, the following are two limitations of our search. First, because we only addressed studies in which evaluation formed a main objective, we could have missed some studies with a limited evaluation focus. Second, we may have missed some studies that have targeted outpatient CPOE systems in specific specialties such as oncology and pediatrics.

To our knowledge this is the first review exclusively dedicated to the evaluation of CPOE systems in outpatient settings. Existing reviews of CPOE system evaluation studies focused on inpatients, ^13,16,49 where advantages of these systems have been reported. Recently, Chaudhry et al. reviewed the impact of health information technology on quality, efficiency, and cost of medical care in inpatient and outpatient settings ⁵⁰ but only two papers of our review appeared there. Another recent review article focused on overriding drug safety alerts in CPOE, which forms only one aspect of CPOE system evaluation. ⁵¹ In contrast, we provide a comprehensive characterization of outpatient studies including description of the study design (with level of evidence), methods, materials, and results.

In spite of the efforts made to enhance safety by introducing CPOE, only four studies evaluated safety in an outpatient setting. Three of them did not show significant reduction in the number of ADEs. Moreover, only one study showed a significant reduction in the number of errors. Recently some observational studies in inpatients described how new medication errors were associated with the use of the CPOE system itself. ^52–54 No such studies were found in the outpatient setting, but one should be aware of the possible existence of such associations in outpatients as well.

A plausible explanation for the low number of ADE-related evaluation studies is the difficulty of obtaining quantitative data on ADEs and the high cost associated with chart reviewing in the face of incomplete [and scattered] patient data. ⁵⁵ The availability of an electronic patient record or a CPOE is only part of the solution because there should also be an infrastructure interconnecting information residing in laboratory and radiology information systems with information residing in the outpatient clinic or GP offices. However, we believe that by focusing on specific patient groups, high risk drugs, simple errors and typical ADEs, one can evaluate the effect of CPOE systems on ADEs with a reasonable level of effort. Another approach would be to identify the few currently existing health care settings with comprehensive paperless systems in place in order to perform evaluation studies with errors and ADEs as outcomes.

Studies on alerts show that alerts were largely ignored by physicians. This does not necessarily mean that safety is compromised; alerts should not be used as proxies for the number of errors or ADEs. Many alerts are not applicable to the patient at hand or they are not clinically important. A valid, although not a new, insight is that the provision of non-patient specific advice is a considerable weakness of CPOE systems, which may lead to low user response and inattention. Five out of the six studies on alerts used the observational study design which has a lower evidence level. Future qualitative and quantitative studies are necessary to show the reasons for overriding alerts and whether redesigning the system is effective in reducing unnecessary alerts and clinician overrides.

Another stated potential benefit of CPOE systems is the reduction of medication cost. There is some indication, although only from two studies with non-RCT design, that advice on equally effective but cheaper drugs and evidence-based messages are more effective at reducing costs than simply displaying a list of drugs with their prices. It is likely that this effect will become more pronounced if the suggestions become patient specific and also target specific expensive groups of medications such as antidepressants and antihypertension medications. ²⁶

Niinimaki and Forsstrom ⁵⁶ described recommendations for standardization and evaluation of CPOE systems in outpatients. They suggested evaluating technical as well as medical facets of these systems. None of the selected papers in our study evaluated technical facets such as data security and reliability of data transfer solutions, client interface, technical functionality, the checking mechanism for dangerous drug dosage, etc. Technical facets such as the user interface are important as they influence the way users perceive and interact with the system. Such aspects deserve more research attention in the future.

In the selected articles various study designs were used to evaluate CPOE systems in outpatients. The results obtained by non-randomized studies were more likely to report statistically significant improvement in the outcome measures than RCTs. This is possibly an indication that non-randomized studies might be biased.

Conclusion

In spite of the cited merits of enhancing safety and reducing costs, published evaluation studies do not provide adequate evidence that CPOE systems provide these benefits in outpatient settings. A possible explanation is the small number of such studies conducted to date and the relatively weak study designs used. In contrast, there is more evidence on the ability of CPOE systems to increase adherence to guidelines in outpatient settings. We hence hypothesize that cost reduction can be achieved when guidelines are specifically geared towards this goal and that safety can be improved when guidelines are made more patient-specific. A second conclusion of our study is that there is much to be gained in insight when more direct outcome measures on safety are included. However, this is hard to achieve due to the nature of the scattered patient information and the non-controllable environment in outpatients. A third related conclusion is that although more studies with a randomized controlled design are welcome to demonstrate and confirm the effects of CPOE on the medication process outcomes in outpatients, there are serious difficulties in conducting them in outpatient settings. Hence we believe that there is also room for new and nontraditional methodologies and case studies for addressing the impact of information technology interventions in dynamic environments such as health care.

In sum, we found that CPOE systems seem to support adherence to guidelines which have the potential to influence costs and safety. To date, there is, however, little evidence about improving safety as measured by medical errors and ADEs. Focusing on outpatient subgroups and specific drugs merits more attention in the future. The field would also benefit from efforts to standardize evaluation studies, such as the standard proposed in Bell et al. ⁵⁷ which aims at facilitating comparisons among studies. Finally, standards for CPOE system requirements and functionality, such as those pertaining to providing alerts, merit more attention as they could facilitate the design and implementation of such systems in the future.