The Impact of Computerized Physician Order Entry on Medication Error Prevention

Abstract

Background: Medication errors are common, and while most such errors have little potential for harm they cause substantial extra work in hospitals. A small proportion do have the potential to cause injury, and some cause preventable adverse drug events.

Objective: To evaluate the impact of computerized physician order entry (POE) with decision support in reducing the number of medication errors.

Design: Prospective time series analysis, with four periods.

Setting and participants: All patients admitted to three medical units were studied for seven to ten-week periods in four different years. The baseline period was before implementation of POE, and the remaining three were after. Sophistication of POE increased with each successive period.

Intervention: Physician order entry with decision support features such as drug allergy and drug-drug interaction warnings.

Main outcome measure: Medication errors, excluding missed dose errors.

Results: During the study, the non-missed-dose medication error rate fell 81 percent, from 142 per 1,000 patient-days in the baseline period to 26.6 per 1,000 patient-days in the final period (P < 0.0001). Non-intercepted serious medication errors (those with the potential to cause injury) fell 86 percent from baseline to period 3, the final period (P = 0.0003). Large differences were seen for all main types of medication errors: dose errors, frequency errors, route errors, substitution errors, and allergies. For example, in the baseline period there were ten allergy errors, but only two in the following three periods combined (P < 0.0001).

Conclusions: Computerized POE substantially decreased the rate of non-missed-dose medication errors. A major reduction in errors was achieved with the initial version of the system, and further reductions were found with addition of decision support features.

Medication errors occur commonly in hospitals.^{1, 2, 3, 4, 5} While most such errors are minor, a small proportion result in an injury or adverse drug event (ADE). In an earlier report, we found that about 1 in 100 medication errors actually results in an ADE, although about 7 per 100 had the potential to do so.⁵ Although most medication errors have little potential for harm, they are undesirable and do cause substantial extra work.⁵

Adverse drug events are important both for patients and hospitals. While most ADEs are minor—rashes and diarrhea, for example—some are serious, and a few even result in death.^{6, 7, 8} These events are also costly: Two recent studies independently estimated their costs to be more than $2,000 per event.^{8, 9} Nationally, the cost of ADEs occurring in hospitals has been estimated at $2 billion.^{8, 9}

One intervention that has substantial potential for improving the medication ordering process is computerized physician order entry (POE), in which physicians write orders online.¹⁰ Physician order entry can improve ordering by ensuring complete, unambiguous, and legible orders. Also, the computer can assist the physician at the time of ordering by suggesting appropriate doses and frequencies, displaying relevant laboratory data, and screening orders for allergies and drug-drug and drug-laboratory interactions.

We recently reported the results of a study of the impact of POE on serious medication errors—errors that either had the potential for harm or actually resulted in an ADE.¹⁰ However, the impact of POE on all medication errors—both minor and serious—is important and was not addressed in that study, and the impact of POE was measured at only one stage in its development. Therefore, we performed a trial to evaluate the impact of POE on the medication error rate and also collected error rates at several stages—for a base-line period before POE and then for three separate time periods after implementation. Specific goals were to compare the medication error rates over time; to evaluate which types of medication errors were most likely to be affected; and to evaluate the errors not prevented with the most mature version, to guide future improvements.

Methods

Study Site and Participants

This study took place at Brigham and Women's Hospital, an academic tertiary-care hospital with approximately 700 beds. The study was performed on three medical units (two general care medical units and one medical intensive care unit) over four and a half years. Data were collected in four separate time periods, all seven to ten weeks long. All evaluations took place in October, November, and December, except for the period 3 analysis, which was delayed until March 1997 because of staffing shortages in the pharmacy. The study periods were as follows: baseline, 51 days, Oct-Nov 1992; period 1, 68 days, Oct-Dec 1993; period 2, 49 days, Nov-Dec 1995; and period 3, 52 days, Mar-Apr 1997. Participants were all patients admitted to a study floor during a study period.

An internally developed information system (BICS) manages the hospital's administrative, financial, and clinical data.¹¹ The system includes clinical results reporting and a computer-based event detection application, which uses rules to detect a wide variety of events.¹² Computerized physician order entry is another application within BICS which came online in May 1993.^{13, 14} All orders are written using this application. Approximately 16,000 orders are written daily, 40 percent for medications. The BICS POE application checks each order for completeness and ensures that certain parameters (e.g., names of medications) come from standard lists. Suggested doses and frequencies are offered for medication orders. Entered orders are screened for problems, such as drug allergies and drug-drug interactions, and the system presents these problems to the physician immediately, when appropriate. At the time of the study, orders were printed out on the floor and carried manually to the pharmacy; a direct electronic link has subsequently been completed.

Interventions

In the baseline period, orders were hand written in order books in the traditional fashion. Subsequent study periods took place after the introduction of POE and serial improvements to it (▶).

Table 1.

System Characteristics By Period

Period	System Characteristics
Baseline	Orders written on paper
	No automated decision support
Period 1	POE in place:
	Complete orders required Medication name, dose, and frequency selection from lists Relevant laboratories displayed Decreased transcription Rudimentary drug-allergy checking Redundant medication checking Rudimentary drug-drug interaction checking Notification for several drug-laboratory problems Many orders entered using preapproved order sets
Period 2	Improved drug allergy checking
Period 3	Improved potassium ordering
	Improved drug-drug interaction checking

Period 1 began five months after the implementation of POE. During this period, most basic system features were in place, but relatively little decision support was provided. Specifically, all orders required specification of the dose, the route, and the frequency of administration. Doses were ordered primarily from hospital-approved standard lists, so that egregious dosing errors were much less likely; medication names and frequencies were also selected from lists. Relevant laboratory results were displayed on the ordering screen. Transcription was greatly decreased but not eliminated; transcription to a paper medication administration record was still required. A simple system was in place to detect drug-allergy interactions for the drug families to which patients are most commonly allergic (e.g., penicillins and sulfa drugs). Duplicate orders (multiple orders for the same drug) generated a warning as well. Clinicians were notified about a small number of life-threatening drug-drug interactions and drug-laboratory checks by paging them after the fact. Many orders were entered through predefined order sets.

In period 2, the most important change was introduction of a comprehensive allergy warning system that included cross-sensitivity checking. By period 3, rules for the use of potassium chloride were significantly revised. All orders for amounts of intravenous potassium higher than 20 milliequivalents were amended to include explicit notation that it be administered in divided doses, the number of available sliding scales was reduced, and individualized scales were restricted. In addition, a more sophisticated drug-drug interaction checking system was implemented, which checked for approximately 100 pairs of serious interactions and also notified the physician at the time of ordering, instead of after the fact.¹⁵ Orders were also electronically communicated to the pharmacy (in periods 1 and 2, they were printed out on the floor and hand-carried to the pharmacy).

Main Outcome Measures

The main outcome measure of the study was the non-missed-dose medication error. We defined medication errors as errors in the process of ordering, dispensing, or administering a medication, regardless of whether an injury occured or whether the potential for injury was present. Missed-dose errors were errors in which doses of medications were not available to nursing personnel at the time they were needed for administration; in most instances, the doses were eventually given. Because missed-dose errors tend to have less potential for harm than other errors and we did not expect them to be significantly decreased by the implementation of POE, we excluded them from the primary outcome. Applications that might be expected to decrease the frequency of missed dose errors are direct electronic transmission of orders to and from the pharmacy and integration of a computerized medication administration record. As noted earlier, direct electronic order transmission to the pharmacy was not in place in period 3, and we are still developing a computerized medication administration record.

As secondary outcomes, we collected data on adverse drug events (both nonpreventable and preventable)⁷ and potential adverse drug events, defined as errors with the potential for harm that did not result in an injury. Potential adverse drug events included errors that were intercepted before the medication reached the patient and non-intercepted errors that did reach the patient but did not cause injury. An example of the latter would be inadvertent administration of penicillin to a patient with a known penicillin allergy, who did not react. Serious medication errors were defined as those associated with a preventable ADE or a potential ADE; non-intercepted serious medication errors include only the preventable ADEs and non-intercepted potential ADEs.

Case Finding

Medication errors, potential ADEs, and ADEs were detected in three ways: 1) pharmacists reported any prescribing errors, potential ADEs, or ADEs that they identified during the dispensing process, and reports were also solicited from nurses through daily visits by the study investigator; 2) a trained reviewer evaluated all medication sheets received by the pharmacy; and 3) the study case investigator reviewed all charts daily on weekdays for evidence of medication errors or ADEs.⁷ The bulk of medication errors were identified by evaluating the medication sheets, so that the detection process differentially detected ordering errors. The chart review included a careful reading of the progress notes in each chart and a more detailed investigation if the investigator identified indications of an ADE (e.g., major bleeding, new confusion, unanticipated intensive care unit transfer, or cardiac arrest). People reporting incidents were assured of anonymity; it was emphasized that we viewed errors and incidents as results of system flaws rather than as human failings.

Classification

All incidents were evaluated as to whether they represented medication errors. The medication errors were then classified by type: dose error (overdose, underdose, missed dose, wrong dose form, dose omitted), route error (incorrect route, wrong route, route omitted), frequency error (incorrect frequency, frequency omitted), substitution (wrong drug given, wrong patient received drug), drug-drug interaction, inappropriate drug, illegible order, known allergy to drug, drug not available (nonformulary and not readily attainable), avoidable delay in treatment, and preparation error.

Incidents suspected of being ADEs or potential ADEs were evaluated by two independent reviewers, who classified each incident into one of four categories: ADEs, potential ADEs, medication orders with little potential for harm, and no error or ADE. Potential ADEs were categorized as intercepted or not intercepted. All ADEs and potential ADEs were classified according to severity as life-threatening, serious, or significant.⁷ Preventability was classified using the categories of definitely preventable, probably preventable, probably not preventable, and definitely not preventable, and in the analysis this four-point scale was collapsed into preventable and not preventable. Reliability for judgments made using this approach has previously been reported⁷; for judgments about whether an incident was an ADE, kappas were 0.81 to 0.98; for preventability kappa was 0.92; and for severity kappas were 0.32 to 0.37.

Analysis

Rates of non-missed-dose medication errors were calculated using both the number of patient-days in the period and the number of admissions. Comparisons between periods were made using the chi-squared test for trend using StatXact. The test for trend assumes an ordering of the categories and was used because of the serial improvements in the system over the four periods.

Results

During the study, the number of admissions in the three units varied among the study periods from 379 to 492, patient-days varied from 1,704 to 2,619, and medication orders varied from 10,070 to 15,025 (▶). The number of medication orders per admission remained relatively constant over the four periods, although the number of medication orders per patient-day was higher in the last two periods compared with the first two, reflecting shorter inpatient length of stay in later periods.

Table 2.

Comparison Across Periods

	Baseline	Period 1	Period 2	Period 3
Duration in days:	51	68	49	51
Patient-days:	1,704	2,619	1,784	1,878
Admissions:	379	492	471	475
Medication orders:	10,070	15,025	13,139	14,352
Medication orders/patient-days:	5.91	5.74	7.36	7.64
Medication orders/admission:	26.6	30.5	27.9	30.2

The main outcome, the non-missed-dose error rate per 1,000 patient-days, fell from 142 during baseline to 51.2 in period 1, then rose to 74.0 in period 2, and then fell again to 26.6 in period 3 (P < 0.0001, ▶). The largest differences were between the baseline and period 1 (a 64 percent drop), when POE was implemented, and between periods 2 and 3 (also a 64 percent drop), when the screens were changed to minimize the likelihood of potassium chloride errors (▶).

Table 3.

Medication Error and Event Rates, By Period

	Baseline	Period 1	Period 2	Period 3	P Value
Non-missed-dose medication errors (n)*:	242	134	132	50	-
Non-missed-dose error rate/1,000 pt-days	142	51.2	74.0	26.6	0.0001
Non-missed-dose error rate/admission	0.64	0.27	0.28	0.11	0.0001
Missed-dose medication errors (n):	288	500	400	617	-
Missed-dose error rate/1,000 pt-days	169	191	224	329	0.0001
Missed-dose error rate/admission	0.76	1.02	0.85	1.30	0.0001
Non-intercepted potential ADEs (n):	8	4	1	0	-
Non-intercepted potential ADEs/1,000 pt-days	4.7	1.5	0.6	0	0.0006
Intercepted potential ADEs (n):	27	82‡	106‡	1	-
Intercepted potential ADEs/1,000 pt-days	15.8	31.3	59.4	0.5	0.15
Preventable ADEs (n):	5	15	2	2	-
Preventable ADE rate/1,000 pt-days	2.9	5.7	1.1	1.1	0.05
Non-intercepted serious medication errors (n)†:	13	19	3	2	-
Non-intercepted serious medication error rate/1,000 pt-days	7.6	7.3	1.7	1.1	0.0003
Nonpreventable ADEs (n):	20	24	17	16	-
Nonpreventable ADE rate/1,000 pt-days	11.7	9.2	9.5	8.5	0.33
Total ADEs (n):	25	39	19	18	-
Total ADE rate/1,000 pt-days	14.7	14.9	10.7	9.6	0.09
Note: Pt-days indicates patient days; ADE, adverse drug event.

^*Includes the errors associated with potential ADEs and preventable ADEs.

^‡Of the 82 intercepted potential ADEs in period 2, 77 were potassium chloride errors; and in period 3, of the 106 intercepted potential ADEs, 101 were potassium chloride errors. In period 4, these errors were essentially eliminated by revising the potassium chloride ordering screens.

^†Preventable ADEs and non-intercepted potential ADEs combined.

Figure 1.

Event rates by period. Top, the non-missed-dose medication error rate per 1,000 patient-days, by period. This rate dropped 64 percent between baseline and period 1, then climbed 45 percent between periods 1 and 2, and finally fell 64 percent between periods 2 and 3; the overall decline was 81 percent (P < 0.0001). Middle, the missed dose error rate per 1,000 patient-days across periods, which climbed significantly during the study (P < 0.0001). Bottom, the non-intercepted serious medication error rate per 1,000 patient-days, which fell during the study (P = 0.0003). Non-intercepted serious medication errors were defined as either non-intercepted potential adverse drug events (ADEs) or preventable ADEs.

In contrast, the missed-dose error rate climbed substantially over the study period (▶), almost doubling from 169 per admission in the baseline period, to 329 in period 3 (P < 0.0001).

The non-intercepted serious medication error rate (the combination of preventable ADEs and non-intercepted potential ADEs) fell over the four periods, from 7.6 per 1,000 patient-days in the baseline period to 7.3 per 1,000 patient-days in period 1 and to 1.7 and 1.1 in periods 2 and 3, respectively (▶, P = 0.0003). The rate of preventable ADEs was highest in period 1, at 5.7 events per 1,000 patient-days, although it fell in the following two periods.

An obvious concern was whether the increase in preventable ADEs in period 1 was due to POE. We therefore examined them more closely by evaluating the case descriptions, in particular to assess whether they were caused by POE. None appeared to have been caused by POE. Of the 15 preventable ADEs, 3 might have been prevented by changes made in period 2 or 3, while 12 would not; most of the 12 were due to use of multiple sedating drugs, not addressed by POE.

The rate of intercepted potential ADEs climbed substantially from baseline to periods 1 and 2; it rose from 15.8 per 1,000 patient-days at baseline to 31.3 in period 1 and 59.4 in period 2 (P = 0.15) before falling to 0.5 in period 3 (▶). These increases in errors were largely related to POE's initial structure for potassium chloride orders, which made it easy to order large doses of intravenous potassium without explicitly specifying that it be given in divided doses (i.e., not more than 20 milliequivalents at a time). Standard nursing practice is to give it in divided doses, and all of these errors were intercepted by nurses so that no patient was injured. These potassium ordering errors accounted for 77 of the 82 intercepted potential ADEs in period 1 and for 101 of the 106 in period 2. Once the potassium ordering screen was changed to include this stipulation, the rate of intercepted potential ADEs fell to 0.5 per 1,000 patient-days in period 3. Only one intercepted event was identified in period 3.

Examination of the severity of ADEs and potential ADEs across the periods (▶) showed that in the baseline period all five errors associated with preventable ADEs were serious or life-threatening, and six of the eight errors associated with non-intercepted potential ADEs also fell within those categories. In periods 2 and 3 combined, there were only two such serious or life-threatening errors that were not intercepted.

Table 4.

Severity of Adverse Drug Events (ADEs) and Potential ADEs, by Period^*

	Baseline	Period 1	Period 2	Period 3
ADEs, not preventable:	20	24	17	16
Life-threatening	0 (0%)	2 (8%)	2 (12%)	0 (0%)
Serious	3 (15%)	4 (17%)	6 (35%)	6 (38%)
Significant	17 (85%)	18 (75%)	9 (53%)	10 (62%)
ADEs, preventable:	5	15	2	2
Life-threatening	1 (20%)	4 (27%)	1 (50%)	0 (0%)
Serious	4 (80%)	5 (33%)	0 (0%)	0 (0%)
Significant	0 (0%)	6 (40%)	1 (50%)	2 (100%)
Potential ADEs, not intercepted:	8	4	1	0
Life-threatening	1 (13%)	0 (0%)	0 (0%)	0 (0%)
Serious	5 (63%)	1 (25%)	1 (100%)	0 (0%)
Significant	2 (25%)	3 (75%)	0 (0%)	0 (0%)
Non-intercepted serious medication errors†:	13	19	3	2
Life-threatening	2 (15%)	4 (21%)	1 (33%)	0 (0%)
Serious	9 (69%)	6 (32%)	1 (33%)	0 (0%)
Significant	2 (15%)	9 (47%)	1 (33%)	2 (100%)
Potential ADEs, intercepted:	27	82	106	1
Life-threatening	3 (11%)	78 (95%)	99 (93%)	0 (0%)
Serious	12 (44%)	2 (2%)	4 (4%)	0 (0%)
Significant	12 (44%)	2 (2%)	3 (3%)	1 (100%)

^*Percentage totals may not add to 100, because of rounding.

^†Non-intercepted serious medication errors are the sum of preventable ADEs and non-intercepted potential ADEs.

As expected, the non-missed-error rate was higher in the intensive care unit than in the general care units; it also fell more in the intensive care units than in the general care units (▶). In contrast, the missed-dose error rate did not display a consistent pattern in the intensive care unit, but in the general care units it rose substantially over the period of the study.

Table 5.

Medication Error Rate, By Unit Type Across Periods

	Baseline	Period 1	Period 2	Period 3	P Value
Patient-days:	1,704	2,619	1,784	1,878	-
ICU (n)	403	736	296	597	-
Non-ICU (n)	1,301	1,883	1,488	1,281	-
Non-missed-dose errors:
ICU (n)	100	71	47	21	-
ICU rate/1,000 pt-days	248	96.5	159	35.2	0.0001
Non-ICU (n)	142	63	85	29	-
Non-ICU rate/1,000 pt-days	109	33.5	57.1	22.6	0.0001
Missed-dose errors:
ICU (n)	127	185	119	172	-
ICU rate/1,000 pt-days	315	251	402	288	0.09
Non-ICU (n)	161	315	281	445	-
Non-ICU rate/1,000 pt-days*	124	167	189	347	0.0001
Note: Pt-days indicates patient-days; ICU, intensive care unit.

^*This rate increased over the study.

Evaluation By Error Type

The results by type of error show that the rates for most subtypes fell over the course of the study (▶). Dose errors were the most frequent subtype, and these actually remained high in periods 1 and 2. However, most dose errors in periods 1 and 2 (86 and 89 percent, respectively) were potassium chloride ordering errors, and this rate fell substantially in period 3 after changes were made in potassium chloride ordering and usage practices. The next most common types of errors, frequency errors and route errors, both fell significantly between baseline and period 3 (P < 0.0001). Some of the rarer errors such as known allergy errors actually have the greatest potential for harm. There were ten known allergy errors in the first period, and only two in the following three periods combined (P < 0.0001). In period 1, approximately 50 orders per day hospital-wide were canceled because of a computer warning about drug allergies; in period 2, this number rose to 80 per day.

Table 6.

Non-missed-dose Medication Errors, By Type and Period

	Baseline (n = 10,070)	Period 1 (n = 15,025)	Period 2 (n = 13,139)	Period 3 (n = 14,352)	P* Value
Dose errors	81 (47.5)	90† (34.3)	114† (63.9)	40† (21.3)	0.03
Frequency errors	43 (25.2)	4 (1.5)	2 (1.1)	4 (2.1)	0.0001
Route errors	25 (14.7)	5 (1.9)	6 (3.3)	4 (2.1)	0.0001
Substitution errors	12 (7.0)	3 (1.1)	3 (1.7)	0 (0)	0.0001
Documented allergy	10 (5.9)	1 (0.4)	1 (0.6)	0 (0)	0.0001
Inappropriate drug	7 (4.1)	3 (1.1)	1 (0.6)	0 (0)	0.002
Avoidable delay	7 (4.1)	0 (0)	0 (0)	0 (0)	0.003
Drug-drug interaction	2 (1.2)	0 (0)	1 (0.6)	0 (0)	0.19
Inadequate follow-up	1 (0.6)	0 (0)	0 (0)	0 (0)	0.17
Other	54 (31.7)	28 (10.7)	4 (2.2)	2 (1.1)	0.0001
Total	242 (142.0)	134 (51.2)	132 (74.0)	50 (26.6)	0.0001
Note: The number of occurrences of each error is shown, followed by the rate per 1,000 patient-days in parentheses.

^*The P value was determined by the chi-squared test for trend across the four periods.

^†Of these errors, 77 of 90 (86 percent) in period 1 and 101 of 114 (89 percent) in period 2 were potassium chloride errors, whereas none of the 40 dose errors in period 3 was a potassium chloride error.

Analysis of Errors Missed in Period 3

We evaluated the errors still occurring in period 3 to see whether additional changes to POE could prevent similar errors in the future. Of the 50 errors in period 3, 48 could potentially be prevented with additional changes to POE. These changes fell into nine categories, but changes in two might prevent the majority. Eighteen errors were due to improper use of a “multiple routes” option, in which the physician could indicate more than one route for the medication, depending on the patient's condition (e.g., morphine sulfate IV/PO). These were considered errors if for one of the routes the dose would be inappropriate or if one route would not be feasible for the medication—for example, a drug that should not be given intravenously. The second large group (n = 13) of errors might be prevented with route restrictions for some oral medications, such as sustained-release preparations and gelatin capsules, which should not be ordered to be given by enterostomy.

Discussion

More than 80 percent of non-missed-dose medication errors were eliminated by computerized POE. Three quarters of this reduction was achieved with a relatively simple system, which structured the entry of orders and included rudimentary order checking. Reductions were seen in a broad array of error types and in both general care and intensive care units. The number of non-intercepted serious medication errors—those with the potential to cause injury—also fell significantly.

Many other evaluations have demonstrated benefits of POE systems^{10, 16} and computerized decision support^{17, 18, 19, 20} for improving the quality and efficiency of care. For example, Tierney et al.¹⁶ demonstrated that implementation of a POE system on a medical service resulted in a decrease in average length of stay of 0.89 days and a 12.7 percent reduction in charges. Recently, Evans et al.¹⁹ found that computer-assisted decision support for ordering antibiotics in one intensive care unit resulted in substantially lower costs and improved quality of care, in part by improving the appropriateness of drug dosing and decreasing the number of allergic reactions.

In another evaluation of the same POE system evaluated in this study, using serious medication errors as the primary outcome and data from 1995, we found that this system prevented 55 percent of serious medication errors. The current study evaluated the impact of POE on all medication errors and included 1997 data, after the system had undergone additional refinements.¹⁰ The additional refinements probably account for some of the difference in effect size between the current study (an 88 percent reduction for serious medication errors between the baseline and period 3) and the 55 percent effect noted above. Order entry resulted in improvements both because of additional structuring of orders and because it allowed checking of orders for problems such as allergies and drug interactions.

The increase in the number of intercepted potential ADEs that occurred post-POE during periods 1 and 2 illustrates the potential that any change, especially a systems change with profound effects such as POE, has for causing new errors, even though this particular error was always intercepted and the overall effect was clearly positive.

The implications for designers of computerized ordering systems are clear. Physician order entry guides physicians toward particular ordering paths, which physicians are likely to follow. Thus, it is vitally important that the suggested paths and parameters are correct and are revised continually. Implementation of POE made it easier for users to order potassium sliding scales and large intravenous doses but did not initially include safeguards about the rate of intravenous potassium. While “all nurses should know” that potassium should not be given too rapidly intravenously, this was not explicitly stated in the entered order. Although this problem was detected in period 1, it was not fixed until period 3. During this interval, the program changes were in a long queue of changes to be made in the system and never quite reached the top. After implementation of POE, a host of valuable quality changes were quickly sought by practitioners, such as guidelines for drugs and implementation of critical pathways. This example underscores the need for significant programming resources after the introduction of POE.

Even the last version of POE we tested in this study was far from mature; we continue to refine it and add improvements. For example, this version of the application still did not include guided dose algorithms or dose adjustments for renal insufficiency, two changes that a prior study suggested might have the largest impact on adverse event prevention.²¹ An outstanding issue that we are still addressing is the problem of multiple sedating drugs; a simple algorithm will not deal with this problem, since many drugs are involved, and often the issue is that several of these drugs are given in close temporal proximity.

Other automation and system strategies will be necessary to further reduce the number of medication errors, in particular the number of missed-dose errors, which rose substantially during this study. Missed-dose errors have relatively low potential for harm⁵ but are costly because they cause additional work for providers. One estimate was that each missed-dose error causes about 15 minutes of extra work, primarily for nursing and pharmacy personnel.⁵ There are many possible causes of the increase in the missed-dose rate. Patient acuity and the number of transfers rose over the study period, while pharmacy staffing decreased. Also, the pharmacy satellites were closed as part of a programmatic change to deploy pharmacists on floors to take advantage of their clinical skills (although this probably resulted in better patient care overall).²² Furthermore, until the final period of the study, drug orders were not communicated electronically to the pharmacy but were printed out and hand carried. Another issue is that nurses had increased drug delivery expectations, because POE made orders immediately visible to them. Eventually, moving to point-of-care delivery devices linked with bar-coding and computerization of the medication administration record could reduce the number of missed doses.^{22, 23}

One limitation of this study is that we evaluated only three units at one tertiary care institution, so the impact could be different in other settings. However, Raschke et al.²⁰ found important benefits of computerized medication alerts in a community hospital. Another limitation of this study is that it was a time series study, not a randomized trial. Because of the complexity of introducing POE, a randomized trial was not feasible. However, it is unlikely that unmeasured temporal effects accounted for the very large effect seen, which appeared immediately after the introduction of the intervention and persisted for the duration of the study. Also, the fourth period of the study was conducted later in the year than the first three, so that the house officers had more experience, although in prior evaluations we saw little evidence of a temporal correlation with error rate. A further limitation is that our detection methodology was better for detecting errors in ordering than errors in medication administration. One method used to assess the frequency of medication errors is the direct observation approach, which finds about one medication error per patient per day, and most of these are administration errors.^{1, 2} Ordering errors appear more often than administration errors to result in patient injuries, although both are important.⁷ Another limitation is that we did not formally assess the reliability of our medication error detection approach in this study, although we did assess the reliability of our assessment of events, and this reliability was good.

We conclude that computerized POE resulted in a very large decrease in the frequency of non-missed-dose medication errors, the errors that are most likely to harm patients. Systems such as these have the potential to both fix and cause problems, and require evaluation. The reductions occurred because order entry both structured orders and facilitated the checking of them. Further reductions should be possible with additional decision support and refinement of the system. Such systems should be used more widely.