Relationship between medication event rates and the Leapfrog computerized physician order entry evaluation tool

Abstract

Objective

The Leapfrog CPOE evaluation tool has been promoted as a means of monitoring computerized physician order entry (CPOE). We sought to determine the relationship between Leapfrog scores and the rates of preventable adverse drug events (ADE) and potential ADE.

Materials and methods

A cross-sectional study of 1000 adult admissions in five community hospitals from October 1, 2008 to September 30, 2010 was performed. Observed rates of preventable ADE and potential ADE were compared with scores reported by the Leapfrog CPOE evaluation tool. The primary outcome was the rate of preventable ADE and the secondary outcome was the composite rate of preventable ADE and potential ADE.

Results

Leapfrog performance scores were highly related to the primary outcome. A 43% relative reduction in the rate of preventable ADE was predicted for every 5% increase in Leapfrog scores (rate ratio 0.57; 95% CI 0.37 to 0.88). In absolute terms, four fewer preventable ADE per 100 admissions were predicted for every 5% increase in overall Leapfrog scores (rate difference −4.2; 95% CI −7.4 to −1.1). A statistically significant relationship between Leapfrog scores and the secondary outcome, however, was not detected.

Discussion

Our findings support the use of the Leapfrog tool as a means of evaluating and monitoring CPOE performance after implementation, as addressed by current certification standards.

Conclusions

Scores from the Leapfrog CPOE evaluation tool closely relate to actual rates of preventable ADE. Leapfrog testing may alert providers to potential vulnerabilities and highlight areas for further improvement.

Background and significance

Computerized physician order entry (CPOE) systems are widely promoted as means to reduce rates of medication errors and adverse drug events (ADE).^1–4 However, experience has shown that these systems require appropriate customization and monitoring to achieve the desired safety benefits.^5–7

The importance of customization and monitoring is underscored by the recommendations made by the National Quality Forum and the requirements of ‘meaningful use’ under the American Recovery and Reinvestment Act.^{7 8} However, directly evaluating safety outcomes from CPOE—by measuring changes in the rates of preventable ADE and potential ADE—is an arduous and expensive process.^{1 9–12} Therefore, for practical reasons, most hospitals seeking to evaluate the effectiveness of a CPOE system are limited to indirect, surrogate measures.

To this effect, the Leapfrog Group has developed an independent, inexpensive, and standardized tool for assessing the performance of a hospital's CPOE system by using simulation cases. In essence, the Leapfrog CPOE evaluation tool estimates the potential benefit of a CPOE system by testing how it handles a variety of dangerous medication ordering scenarios.^{1 8 13} Accordingly, performance scores are presumed to be linked to actual outcomes.¹

Objective

The Leapfrog CPOE evaluation tool, presently the only instrument of its kind, has been quickly adopted into practice for monitoring purposes.^{8 13 14} However, it still remains uncertain whether Leapfrog performance scores are related to outcomes in real-world settings as empirical evidence is currently lacking.⁸ Addressing this evidence gap, we sought to determine the relationship between test scores and actual rates of preventable ADE and potential ADE.

Materials and methods

We performed a cross-sectional study to compare the rates of preventable ADE and potential ADE with scores reported by the Leapfrog CPOE evaluation tool. This study was conducted independently of the Leapfrog Group and was approved by the institutional review boards at each hospital site.

Study setting and participants

In a recent study conducted at five community hospitals in Massachusetts,² we assembled a cohort of 1000 adult patients (aged ≥18 years) admitted between October 1, 2008 and September 30, 2010. In total, 200 patients were selected from each site using simple random sampling. These study sites were chosen because they were felt to be reasonable representatives of typical small to medium sized community hospitals, each with 100–300 inpatient beds.² Approximately 6 months before cohort assembly, each hospital independently selected, implemented, and customized a vendor CPOE system. A total of two vendors was selected. The identities of the commercial vendors cannot be published because of contractual non-disclosure agreements between the sites and vendors. These CPOE systems, however, are among the top five most frequently used systems in the state.

Leapfrog CPOE evaluation tool

During the last phase of our study, each site's CPOE system was evaluated using the ‘adult inpatient’ version of the Leapfrog CPOE evaluation tool.¹⁵ This tool, developed independently from our study hospitals, was first introduced in 2006 by the Leapfrog Group, a coalition of healthcare purchasers dedicated to improving the safety and quality of healthcare.⁸ The Leapfrog CPOE evaluation tool examines how a system with clinical decision support is able to intercept a variety of potentially dangerous medication order using a variety of simulated clinical scenarios. A test bank consisting of over 130 adult test orders designed to identify CPOE system vulnerabilities was created based on an expert panel with industry experience and literature review, focusing on the types of ordering errors that would most likely result in ADE.⁸ Since its creation, this test bank has been periodically reviewed, updated, and modified.

For our study, in accordance with the Leapfrog methodology,¹⁵ a random list of 10 simulated test patients (each with a specific demographic background, and a list of medical comorbidities and medications) along with 50 unique medication orders was downloaded from the master test bank for each study site. The local chief medical informatics officer then entered these test orders for these simulated patients into the CPOE system and noted any guidance provided by decision support. Of note, in order to protect the content of the test bank (and to prevent intentional ‘gaming’ of the system), the Leapfrog Group does not permit publication or comparison of the list of scenarios between sites.

On completion of the test, scores were automatically generated and reported by the tool (ranging from 0% to 100%)—in which higher scores indicate greater success at intercepting potentially harmful orders. These weighted scores reflect both the potential severity of an ADE (classified as life-threatening, severe, significant, or not significant) and its likely frequency of occurrence (scored on a three-point scale of most frequent, less frequent, or least frequent).^{1 8} In general terms, a potentially lethal event, although possibly rare in occurrence, would be assigned a high score; on the other hand, a less clinically severe event, but one that occurs at a very high frequency, would likewise be given a high score. The potential severity of an ADE was determined by clinical judgment, and the likely frequency of an ADE that may result from a problematic medication order was estimated using published studies, the gray literature, and automated surveillance methods.^{8 9 16 17} Although the methodological development of the scoring system has previously been published by the Leapfrog Group,⁸ the specific scenarios and weights are not publically available.

Outcome ascertainment

The primary outcome of our study was the rate of preventable ADE. These events pose tangible threats to patient safety and are also the principal outcome the Leapfrog tool was designed to predict. The secondary outcome was the composite rate of preventable ADE and potential ADE. Both of these are potentially preventable with CPOE technology.^{1 18}

We defined an ADE as any drug-related injury. ADE were considered preventable if they arose from errors in the medication use process (eg, prescribing penicillin to a patient with a known history of penicillin allergy resulting in anaphylaxis). Furthermore, medication errors with the potential to cause injury, but not actually resulting in injury (‘near misses’), were classified as potential ADE (eg, a 100-fold overdose of insulin, but intercepted by a nurse). All potential ADE stemmed from errors and were, therefore, preventable in nature.

We determined the actual frequency of preventable ADE and potential ADE in our study population by reviewing physician orders, medication lists, laboratory reports, admissions histories, progress and consultation notes, discharge summaries, and nursing flow sheets to identify the occurrence of preventable and potential ADE.² Data were abstracted by trained research nurses and each case was then independently reviewed by two investigators (AAL, CK, MA, SRS, MC NK, BC and GS), blinded to hospital site and prescribing physician. Disagreements were resolved by consensus. Events were systematically classified according to standardized definitions and methodology, as in previous studies.^{2 19–21} As earlier reported, we had excellent interrater agreement (κ score 0.89; 95% CI 0.85 to 0.92) when classifying the type of event for this cohort.²

These observed outcomes were considered to be the ‘gold standard’, which the Leapfrog scores were compared against. To facilitate comparisons, events were subclassified according to the published Leapfrog categories:^{8 15} therapeutic duplication, single and cumulative dose limits, allergies and cross-allergies, contraindicated route of administration, drug–drug interactions, drug–food interactions, drug–diagnosis interactions, contraindication from dose limits based on age and weight, contraindication from dose limits based on laboratory studies, contraindication from dose limits based on radiology studies, and corollary orders (see supplementary appendix, available online only, for detailed descriptions and examples; see supplementary etable S1, available online only). These 11 categories reflect different types of potentially harmful medication errors. Medication incidents with the potential for harm, but not falling into any of the preceding categories, were classified as ‘other’ in our study (eg, prescribing the wrong pharmaco-equivalent dose when substituting a medication for another within the same drug class). Of note, the Leapfrog CPOE evaluation tool additionally evaluated measures of system efficiency according to ‘cost of care’ and ‘nuisance orders’.⁸ ‘Cost of care’ is largely related to detecting redundant laboratory testing, whereas ‘nuisance orders’ relate to system-generated warnings for clinically inconsequential orders. These two categories were excluded from our outcome ascertainment as neither have the potential for injury (eg, repeat laboratory testing for serum thyroid stimulating hormone levels within 1 h—while not particularly beneficial—would not conceivably result in an ADE either).

Analysis

The basic unit for the analysis was the hospital site. Of the observed clinical outcomes, rates of preventable ADE and potential ADE were expressed as the number of events per 100 admissions. Descriptive rates and proportions were reported overall and according to each predefined prevention category. These were then compared with Leapfrog scores. Scores were automatically generated and reported by the online tool according to each prevention category.¹⁵ An overall performance score was then calculated as the mean score across all categories, excluding ‘cost of care’ and ‘nuisance orders’ (as these were systematically excluded from our study). Data were subsequently fit using Poisson regression with medication event rates as the dependent variable and Leapfrog scores as the independent variable. We tested for model fit and a scale parameter was used to correct for overdispersion, using the square root of the Pearson χ² divided by the total degrees of freedom.²² Relative effect measures were expressed as rate ratios (RR) and absolute effect measures as rate differences (RD), along with 95% CI. Analysis was performed using SAS (V.9.3).

Results

Observed events according to Leapfrog category

The characteristics of the five sites were previously described in detail.² During the study interval, a total of 645 preventable medication events occurred, a composite of 70 preventable ADE and 575 potential ADE (table 1). Of these, the majority were related to excessive dosing, either by therapeutic duplication (56.3%) or by exceeding dosing limits (4.3%). Many were related to inappropriate prescribing based on laboratory studies (7.0%), age and/or weight considerations (4.0%), patient diagnosis (2.6%), and documented allergy (1.6%).

Table 1.

Leapfrog score and observed outcomes according to category

Prevention category	Site 1			Site 2			Site 3			Site 4			Site 5
	Score (%)	Observed rate (no./100 adm.)		Score (%)	Observed rate (no./100 adm.)		Score (%)	Observed rate (no./100 adm.)		Score (%)	Observed rate (no./100 adm.)		Score (%)	Observed rate (no./100 adm.)
		Prev. ADE	Prev. ADE and pot. ADE		Prev. ADE	Prev. ADE and pot. ADE		Prev. ADE	Prev. ADE and pot. ADE		Prev. ADE	Prev. ADE and pot. ADE		Prev. ADE	Prev. ADE and pot. ADE
Therapeutic duplication	100.0	1	97	100.0	1	61	100.0	4	49	100.0	0	45	100.0	0	111
Single and cumulative dose limits	0.0	1	8	33.3	0	2	33.3	0	6	57.1	1	2	57.1	0	10
Allergies and cross allergies	100.0	0	1	100.0	1	5	100.0	1	2	100.0	0	1	100.0	0	1
Contraindicated route of administration	66.7	0	0	66.7	0	1	66.7	0	0	66.7	0	0	100.0	0	0
Drug–drug and interactions	75.0	2	2	100.0	1	4	40.0	0	0	16.7	0	0	16.7	0	0
Drug–food interactions	0.0	0	0	0.0	0	0	0.0	0	0	0.0	0	0	50.0	0	0
Drug–diagnosis interactions	0.0	5	11	0.0	0	0	25.0	1	2	0.0	2	4	0.0	0	0
Contraindication: dose limits based on age and weight	0.0	2	4	0.0	0	0	0.0	0	0	25.0	2	2	25.0	1	20
Contraindication: dose limits based on laboratory studies	25.0	4	8	50.0	5	19	75.0	2	6	0.0	2	8	0.0	0	4
Contraindication: dose limits based on radiology studies	0.0	0	0	0.0	0	0	0.0	0	1	0.0	0	0	0.0	0	0
Corollary orders	0.0	1	2	25.0	0	0	0.0	0	0	50.0	0	0	0.0	0	0
Cost of care*	0.0	–	–	0.0	–	–	0.0	–	–	0.0	–	–	50.0	–	–
Nuisance orders*	0.0	–	–	0.0	–	–	0.0	–	–	50.0	–	–	50.0	–	–
Other	–	7	85	–	0	20	–	8	16	–	11	16	–	4	9
Overall†	33.3	23	218	43.2	8	112	40.0	16	82	37.8	18	78	40.8	5	155

*Adverse events routed in ‘cost of care’ and ‘nuisance orders’ were not measured.

†Overall site-wide performance was calculated as the mean score across all categories excluding ‘cost of care’ and ‘nuisance orders’.

ADE, adverse drug event; adm, admissions; Prev., preventable; pot., potential.

Of the hundreds of drugs evaluated, only a handful was responsible for the majority of events. The most frequent offenders were acetaminophen, acetaminophen-containing products, and opioids, accounting for 30.0%, 22.8%, and 7.4% of all events, respectively—mostly because of therapeutic duplication or excessive dosing. A similar pattern was seen across all five sites. Notably, ‘single and cumulative dose limits’ and ‘drug–diagnosis interactions’ were two other categories in which the five hospitals scored poorly and significant numbers of medication events were observed. Many problems with dosing were related to ordering excessive cumulative amounts of medication (24/28; 85.7%), mostly because of incorrect doses or frequencies, and sometimes because of the absence of stop limits (eg, potassium 20 mEq intravenously every hour without a cumulative dose restriction). Infrequently, overdoses resulted from a single dose of medication (4/28; 14.3%). These cases were related to misplacement of a decimal place, incorrect weight-based calculations, and rarely because the wrong route was specified (eg, unfractionated heparin 5000 units was ordered intravenously twice a day instead of subcutaneously). Furthermore, inappropriate prescribing based on patient diagnosis also accounted for a significant portion of events. Half of these were related to the use of antihypertensive medications and negative chronotropic drugs in the setting of known hypotension and bradycardia (8/17; 47.1%). Other examples of medication events included the continued use of nephrotoxic drugs in the setting of acute renal failure, prescriptions for thiazolidinediones for patients with congestive heart failure, and the ongoing use of psychotropic medications for those with delirium.

Observed events not defined by Leapfrog category

Nearly a quarter (146/645) of all preventable medication events did not fall into any of the predefined Leapfrog categories. Most of these were related to ordering errors (72.6%), followed by errors in transcription (23.3%) and administration (4.1%).

Of the errors arising from the ordering stage, some could have been averted with basic CPOE alone (ie, 32 incomplete orders). Other errors were potentially preventable with advanced decision support (ie, two errors arose from incorrect dose conversions when substituting a medication with another from the same drug class, two from under-dosing, one from an incorrect ‘sound-alike’ medication, and one from delayed treatment of hyperkalemia). Changes to physician training may have prevented as many as 30 events (eg, writing an order for the wrong patient). Finally, improper medication reconciliation accounted for 38 errors, 10 of which eventually resulted in ADE.

Relationship between performance scores and observed event rates

We found that the overall Leapfrog performance scores were highly related to the primary outcome (table 2). A 43% relative reduction in the rate of preventable ADE was predicted for every 5% increase in Leapfrog performance scores (RR 0.57; 95% CI 0.37 to 0.88). In absolute terms, four fewer preventable ADE per 100 admissions were predicted for every 5% increase in overall Leapfrog scores (RD −4.2; 95% CI −7.4 to −1.1). The majority of the variability in preventable ADE rates between sites was explained by the Leapfrog scores alone (R² 0.89 and 0.90 for the models predicting relative and absolute rate differences, respectively).

Table 2.

Relationship between predicted performance and observed outcomes

Outcome	Rate ratio* (95% CI)	p Value	Rate difference† (95% CI)	p Value
Primary
Preventable ADE	0.57 (0.37 to 0.88)	0.01	−4.2 (−7.4 to −1.1)	0.01
Secondary
Preventable ADE and potential ADE	0.71 (0.43 to 1.17)	0.18	−19.3 (−52.9 to 14.4)	0.26

*Rate ratio represents the predicted relative change in the event rate for every 5% absolute increase in the Leapfrog performance score.

†Rate difference represents the predicted absolute change in the event rate per 100 admissions for every 5% absolute increase in the Leapfrog performance score.

ADE, adverse drug event.

In contrast, the relationship between Leapfrog performance scores and the secondary outcome was not statistically significant, although the point estimates suggested a reduction in the composite rate of preventable ADE and potential ADE overall. For every 5% increase in Leapfrog scores, a possible relative rate reduction of 29% was predicted (RR 0.71; 95% CI 0.43 to 1.17), which corresponded with a possible reduction of 19 preventable ADE and potential ADE per 100 admissions (RD −19.3; 95% CI −52.9 to −14.4), although these associations were not statistically significant.

Discussion

We assessed the relationship between Leapfrog performance scores and rates of preventable ADE and potential ADE. We found that even a 5% improvement in Leapfrog scores was associated with a relative rate reduction in preventable ADE by nearly a half, and an absolute reduction of four preventable ADE per 100 hospital admissions. Therefore, our findings support the use of the Leapfrog tool as a convenient means of evaluating and monitoring actual CPOE performance after implementation.¹⁴ Current certification approaches require that functionalities such as drug–drug interaction and drug-allergy testing to be present, but have not yet required the inclusion of other specific categories tested by the Leapfrog tool.¹

Previous work has suggested that most of the safety benefits of CPOE are rooted in the accompanying clinical decision support.²³ Our findings, therefore, should not be inherently surprising when considering the purpose of the Leapfrog tool, its development, and application. The Leapfrog CPOE evaluation tool tests how systems handle a wide variety of potentially dangerous medication ordering errors across a wide range of decision support categories, thus exposing problems at the point of order entry.^{5 8 15 23} Accordingly, this study reinforces the association between medication ordering processes, as measured by the Leapfrog tool, and distal outcomes.^{18 20} Therefore, in face of the growing pressure placed on organizations to monitor their CPOE systems,^{7 8} the Leapfrog tool appears to be an attractive, efficient, and validated alternative to the traditional methods of chart review, incident reporting, or direct observation, which may be labor-intensive, expensive, insensitive, and ineffective.^9–12 Moreover, previous work has suggested that CPOE performance is more likely to be reported as favorable when assessments are internally conducted compared to more objective external evaluations.²⁴ Consequently, the fact that the Leapfrog Group serves as an independent board further adds to the face validity of the tool.

Our findings, however, also suggest that some of the simulated Leapfrog scenarios may not fully reflect common medication practices encountered in routine care. This may partly explain why we were unable to detect a significant association between Leapfrog scores and the composite outcome of preventable ADE and potential ADE. For example, we found that therapeutic duplication of acetaminophen and acetaminophen-containing products accounted for approximately half of all potential ADE. Indeed, others have also confirmed that supratherapeutic dosing of acetaminophen is common,^25–27 associated with liver enzyme derangements,^{26 28} and can be potentially fatal.²⁹ However, when evaluating each hospital's ability to handle therapeutic duplication, the Leapfrog tool still scored all five hospitals with 100% in this prevention category. Indeed, the use of CPOE has previously been associated with significant rates of duplicate medication orders, even in the presence of clinical decision support designed to identify therapeutic duplication, as these types of orders are common and some may even be intentional.³⁰ As such, clinical decision support for duplicate or additive medication orders may only play a limited role in reducing risk as many of the system-generated alerts are ignored or overridden by users. Therefore, separate and complementary systems are likely to be needed to catch duplication errors before they reach the patient, which may include updating existing medication databases, developing safer policies for order entry (eg, reviewing current medications before entering new orders), and revising protocols likely to result in duplicate orders.³⁰ Furthermore, we found that approximately half of preventable ADE did not fall into any of the predefined decision support categories. Therefore, while interrelated, the relationship between Leapfrog scores and distal events remains imperfect, suggesting that the tool can still be further refined.

Our study has limitations. Although 1000 patients were enrolled, our data were still limited to five community hospitals. Therefore, our study estimates may be imprecise. Nonetheless, the qualitative interpretation of our findings remain stable within the boundaries of statistical significance; as Leapfrog scores improve, the predicted rate of preventable ADE decrease. Second, to avoid overfitting our statistical models, we intentionally did not adjust for other factors. Even so, our univariable models were able to explain nearly all the variability in preventable ADE rates observed. Finally, we were unable to detect an association between Leapfrog scores and the secondary, composite outcome of preventable ADE and potential ADE. Admittedly, the major disadvantage to using a composite endpoint arises from trying to pool heterogeneous outcomes.³¹ While interrelated, the relationship between potential ADE and preventable ADE is loose, as only a small fraction of medication errors typically result in distal injury.¹⁸ Even though we did not detect a significant relationship in our composite outcome, the strong and positive finding for our primary outcome is much more clinically relevant. Preventable ADE, unlike potential ADE, pose tangible threats to patient safety. In contrast, potential ADE, while undesirable, do not cause real injury, but are rather ‘near misses’.

Conclusion

In conclusion, the Leapfrog CPOE evaluation tool is an attractive, systematic, and standardized method for monitoring CPOE. Performance scores relate to rates of preventable ADE. Leapfrog testing may alert providers to potential vulnerabilities and highlight areas for improvement in order to achieve the desired safety benefits and complement certification—as all the participating hospitals already had implemented certified products. Possible refinements to the Leapfrog tool may include expansion of the predefined decision support strategies and updates to the current therapeutic duplication scenarios, as these changes may further improve the ability of the Leapfrog tool to predict the occurrence of preventable ADE and potential ADE.