Evaluating Clinical Decision Support Systems: Monitoring CPOE Order Check Override Rates in the Department of Veterans Affairs' Computerized Patient Record System

Abstract

Objective

To measure critical order check override rates in VA Puget Sound Health Care System's computerized practitioner order entry (CPOE) system and to compare 2006 results to a similar 2001 study.

Design

Analysis of ordering and order check data gathered by a post-hoc logging program. Use of Pearson's chi-square contingency table test comparing results from this study and the earlier study.

Measurements

Factors measured were total number of orders, frequency of order check types, frequency of order check overrides by order check type and comparisons of these results with previous results.

Results

A total of 37,040 orders generated 908 (2.5%) critical order checks. Drug-drug critical alert override rate was 74/85 (87%) in 2006 compared to 95/108 (88%) in 2001 (X ²=0.04, df=1, p=0.85). The drug-allergy override rate was 341/420 (81%) compared to 72/105 (69%) in 2001 (X ²=7.97, df=1, p=0.005). In 2001, 0.25% (105/42,621) orders generated a drug-allergy order check compared to 1.13% (420/37,040) in 2006 (X ²=238.45, df=1, p<0.0001).

Conclusion

Override rates of critical drug-drug and drug-allergy order checks remain high at VA Puget Sound Health Care System including significant increases in drug-allergy order checks. We recommend that monitoring override rates be regular practice in clinical computing systems and conclude that qualitative research should be carried out to better understand how physicians interact with decision support at the point of ordering.

Introduction

Clinical information systems that include computerized practitioner order entry (CPOE) have the potential to reduce medical errors and improve patient safety. ¹ This result has put increasing pressure on hospitals to adopt such systems. More specifically, automated clinical decision support systems found within most CPOE systems have been shown to contribute to error reduction by providing recommendations and checking for potential medication allergies, interactions or overdosing. ^2,3 These warnings, known as order checks, ^∗ display warnings to the practitioner at the time orders are entered. However, many studies show that order checks do not necessarily result in the cancellation of an order by the practitioner, even though the warning given by the decision support system is deemed “high severity.” ^4–7 When a practitioner receives an order check but continues with the order, the order check is considered “overridden.” In some systems, for the highest severity or “critical” order checks, an explanation for continuing the order, known as an “override reason” must be entered by practitioners to justify their actions. High override rates may be an indication that a high proportion of order checks generated by the decision support systems are not clinically relevant.

At VA Puget Sound Health Care System, health care practitioners have used a CPOE system including order checking since 1997. ⁸ In 2001, we reported that 88% of critical drug-drug and 69% of critical drug-allergy order checks were being overridden. Some 29% of the critical drug-drug overrides occurred when one of the medications was in topical form. ⁶ Based in part on the findings from our 2001 study, the CPRS order check rules were changed to reduce topical medication order checks because 28 of 31 (90.3%) critical drug-drug order checks involving topical medications were overridden. ⁶ In this follow-up study, we analyzed a set of orders from two 3-day periods in 2006 to re-assess override rates and compare them to our previous 2001 study. Because of the rule changes accounting for topical medications, we expected override rates would decrease. Our re-assessment was also motivated by our knowledge that many factors can influence override rates making prediction of override rates difficult.

Setting

We analyzed orders from VA Puget Sound Health Care System which consists of two primary and tertiary care facilities, the Seattle Division located in Seattle, and the American Lake Division in Tacoma, Washington and a system of community based outpatient clinics. There are 313 acute care beds and 131 nursing home care unit beds. In 2006 there were 8,000 admissions and 624,764 outpatient visits. The VA of Puget Sound is an active teaching hospital, affiliated with the University of Washington School of Medicine, training over 500 residents, interns and students each year.

The Computerized Patient Record System (CPRS) component of the larger Veterans Information System Technology Architecture (VISTA) has been used by VA Puget Sound since 1997 for note entry, results review and order entry. ^8,9 In 2005, 3,857,131 orders were entered at VA Puget Sound with an average of 10,567 orders per day. All inpatient units use CPRS except the chemotherapy units on the Bone Marrow Transplant unit. While CPRS can be customized by each clinical facility, the overall development and engineering is carried out nationally with new versions, fixes and upgrades distributed to local VA facilities. The majority of data used by CPRS for its drug-allergy and drug-drug order checking is controlled by the national drug file (NDF) maintained by the National Drug File Support Group (NDF Support Group). This group also classifies order checks as “critical” or “significant.” To be classified as critical, the interaction must be identified in the manufacturer's black box warning, or be well documented in the literature to cause significant sequelae. Significant drug interactions do not meet the critical drug-drug interaction criteria but are still thought to have substantial clinical importance. ¹⁰ Our study was only concerned with critical order checks. Changes to the NDF are made at the national level and implemented locally. In addition, local VA hospitals can maintain their own data in a local drug file which may also generate order checks, and the NDF Support Group is notified of all locally entered interactions for possible inclusion at the national level. ¹⁰ The VA's NDF Support Group added 268 critical drug-drug interactions and removed 7 between 2001 and 2006 and these were adopted by VA Puget Sound.

In addition to system data changes, new software features were introduced in the interval since the first study including direct provider entry of adverse drug events as well as the ability to add non-VA medications (over-the-counter, prescription, and herbal). Non-VA medications with entries in the NDF would increase the scope of items available to the order check logic.

Methods

Orders can be entered by physicians, nurses, pharmacists, other practitioners and clerical staff under the direction of practitioners at several entry points in CPRS. The ordering data we analyzed includes order activity from Wednesday, January 4, 2006 14:11 to Friday, January 6, 2006 15:46 (Period 1) and from Monday, January 9, 2006 08:41 to Wednesday, January 11, 2006 10:30 (Period 2). The gap in the order activity between Period 1 and Period 2 was due the system engineers preferring to run our order logging program (described in the Methods section) when there was adequate system support in case the program adversely affected CPRS. We were only concerned with orders generated through direct practitioner entry in the ordering package. Orders entered through the Pharmacy, Lab or Radiology packages were excluded in our analysis.

Evaluating the use of an installed, integrated CPOE system such as the VA's CPRS can be complex. To measure or monitor a specific aspect of system use that was not initially designed to record data for retrospective analysis requires a post-hoc logging system. Our study falls into this category: We wished to measure the number of orders that were started, had an order check, and then never finished or signed (presumably because of the order check). We define override rate as the percentage of distinct orders receiving a high severity, critical order check that are signed.

In CPRS, orders cancelled prior to signature are treated differently from signed orders. Orders are temporarily saved in the archival database and assigned an order number. When a practitioner attempts to sign the order, final order checks are triggered. If the order is signed, the order and order number become permanently archived in the database. If the order is cancelled at any point before signing, then the record is eventually deleted and the order number may be re-used at a later date. Because of this system behavior, archived order records do not contain orders cancelled prior to signing. ^†

Thus, we found that the most practical way to measure the override rate at the VA Puget Sound was to collect and analyze data via a logging system that captured calls to CPRS order checking logic and then compare this log with records from the archived database containing signed orders. The CPRS does not readily provide a method for performing post-hoc logging of order data. Therefore, in 2001 we developed a local logging method for our original study. However, because this method had been discontinued by the time of this follow-up study, it was necessary to develop a new logging process using an existing debugging utility built by CPRS developers. This utility outputs information about order checking from orders entered using the CPRS ordering interface shown in ▶. ▶ Next, we provide some details about the data analysis required to identify and understand entries from this logging system.

Figure 1.

Medication selection in the CPRS ordering interface. The practitioner clicks the circled “Accept Order” button after selecting items for the order triggering the order checking functionality and the “Order Checking” window shown.

Figure 2.

Order signing in the CPRS ordering interface. Practitioners must enter a free text justification for overriding orders with critical, high severity (Severity 1) order checks. Attempting to sign the order triggers order checking functionality. The order checks for this order are shown in the “Order Checks” window.

Using our data linking heuristics, we processed 247,767 lines of logged data, representing approximately 38,926 orders (∼6.4 lines per order). Applying our exclusion rules, we ultimately analyzed a data set of 37,040 orders that were started during our two study periods. Of these orders 2,444 triggered at least one order check, and 812 orders triggered critical order checks. We used a chi-square contingency table test to compare results from the 2001 and 2006 studies. Our study was approved by the University of Washington and VA Puget Sound Human Subjects Divisions and the VA Puget Sound Research & Development Committee.

Results

The orders in our study data generated 8 different types of critical order checks, listed in ▶. Drug-Allergy order checks represented the largest number of high severity order checks, with Drug-Drug order checks, “No allergy assessment” and metformin order checks comprising larger portions of the remaining order checks.

Table 1.

Table 1 Breakdown of High Severity Order Checks and Corresponding Override Rates

Order Check Type	Total distinct Orders w/Order Check	Orders Signed	Override Rate %
Drug-Drug Interaction	85	74	87.1
Drug-Allergy Interaction	420	341	81.2
Clozapine appropriateness	22	22	100
Procedure uses intravenous contrast media - abnormal biochem result/no creatinine results within 30 days	121	98	81.0
Metformin - no serum creatinine	47	44	93.6
Patient has no allergy assessment	122	111	91.0
Patient allergic to contrast media	6	4	66.7
Procedure uses intravenous contrast media and patient is taking metformin	21	14	66.7
Total distinct orders with at least 1 critical order checks (Some orders may have more than 1 high severity order check)	812	687	84.6

“Total distinct Orders w/Order Check” is the number of individual orders triggering a specific order check type. Orders could have more than one type of critical order check or several of the same type.

Some orders contained more than one of the same type of order check. For example, there were 118 distinct critical drug-drug order checks for 85 distinct orders. We use the number of distinct orders (85) that contain a specific type of order check in our override calculation because as seen in ▶, CPRS shows all order checks generated for an order in a single window. Thus, the practitioner's workflow is interrupted to the same extent whether there is one order check or many. For the few orders containing multiple types of critical order checks, we counted them once for each type. Of the 85 distinct orders containing at least 1 critical drug-drug interaction, 74 (87.1%) of those orders were signed despite the critical drug-drug order check while 11 orders were cancelled, Of the 420 orders triggering critical drug-allergy order checks, 341 (81.2%) resulted in a signed order and 79 orders were cancelled.

▶ summarizes our findings in comparison to our earlier 2001 results. In 2001, 42,621 orders generated 215 (0.5%) critical order checks while in 2006, 37,040 orders generated 908 (2.5%) critical order checks. Some 85 orders contained at least 1 drug-drug order-check and 74 those orders (87%) were signed by the practitioner. This represents a slight decrease from the 2001 results where 95 of 108 drug-drug order checks (88%) were overridden. ^††

Table 2.

Table 2 Results for Total Order Checks Generated, Drug-Drug and Drug-Allergy Order Checks Comparing 2001 to 2006.

Total High Severity Order Checks	2001	2006	p
Total orders	42,621	37,040
Number of high severity order checks	215	908
% of total orders with high severity order checks	215/42,621 0.5%	908/37,040 2.5%	<0.0001
Drug-Drug Order Checks
Total orders with order checks	(108)	85
Signed orders	95	74
% Overridden	(95/108 88%)	74/85 87%	0.85
% of total orders with order checks	(108/42,621 0.25%)	85/37,040 0.23%	0.49
Drug-Allergy Order Checks
Total orders with order checks	105	420
Signed Orders	72	341
% Overridden	72/105 69%	341/420 81%	0.005
% total orders with order checks	105/42,621 0.25%	420/37,040 1.13%	<0.0001

For drug-drug order checks, the results are in parentheses because in the 2001 study, order checks were counted rather than distinct orders.

In 2001, 0.25% (105/42,621) orders generated a drug-allergy order check. In 2006, this percentage increased to 1.13% (420/37,040). Of these orders, 341 (81%) with drug-allergy checks were overridden compared with 72 of 105 (69%) in 2001.

Pearson's chi-square contingency table test shows that overall there has been a statistically significant change in the rate of critical order checks from 2001 to 2006 (X ²=536.95, df=1, p<0.0001). We could not conclude there was a significant change in drug-drug order checks (X ²=0.47, df=1, p=0.49) nor their associated override rates (X ²=0.04, df=1, p=0.85). However, there has been a significant change in both the percentages of drug-allergy order checks (X ²=238.45, df=1, p<0.0001) and override rates (X ²=7.97, df=1, p=0.005).

A secondary goal of our study was to assess the impact of system changes related to topical medication order checks. In our previous study, 25.9% (28/108) of drug-drug order checks were overridden because one of the medications was a topical medication. This was determined by analyzing the narrative text and searching for the terms -“topical,” “top,” “oint,” “ointment” or “shampoo”. ⁶ We repeated this search of override reasons in our study and did not find any corresponding entries. We also verified with CPRS developers that order checking logic now considers whether one of the two interaction medication is topical. However, even with these order checks eliminated, we still have what is generally considered a high override rate of 87%. It is possible that there are still clinically relevant order checks involving topical medication in CPRS that we were unable to detect, but their impact on order check overrides has decreased.

Discussion

As a result of our work to date, we have learned several lessons that are more broadly applicable to decision support systems and especially to ordering systems that provide critical order checks to practitioners placing orders. First, the rules and logic that govern orders checks should be understandable, editable and maintainable by system operators and users. The CPRS order check rules are created centrally and meant to serve many local VA hospitals. This model is similar for commercial systems that contain knowledge bases and rule sets meant to serve a wide range of customers. Advantages of commercial knowledge bases include their comprehensiveness and ability to draw on a larger pool of expert opinion, but a drawback is their high sensitivity, resulting in frequent order checks. ¹¹ We believe that the VA's centralized order check development model exhibits this same trade-off. Individual VA hospitals have the technical ability to customize or adjust rules, but doing so is a significant undertaking without easy-to-use tools and well-defined organizational processes. We agree with Kuperman et al. who recommend that drug knowledge base creators need to provide the necessary tools to understand, customize and share rule information and that organizations need to create policy and procedure infrastructure to support the use of these tools. ¹²

Second, system behavior should be easily monitored, and ease of evaluation and the development of built-in evaluation tools should play a more significant role in system design. As we have documented, it is particularly difficult to retrieve information about cancelled orders from the VA's CPRS system. Yet without this information, we cannot measure override rates, and thus cannot assess how often users overrule CPRS drug-drug and drug-allergy interaction rules. Because there is no native order check evaluation tool, it was necessary to develop our own local evaluation method in 2001 and again in 2006 using different techniques. Although CPRS is used at all VA hospitals, it would be a challenge to reproduce our study at other sites without a significant effort because of the variation in local implementations and available human and technical resources.

Finally, we agree with Abookire et al. that system behavior in general should be periodically evaluated, especially when there are significant changes in rules about order checks or in ordering policies or software feature changes. ⁴ Clinical decision support systems are expensive, complex systems that must be tightly integrated with other hospital information systems. Without periodic evaluation, it is difficult to know how these systems are actually being used and monitoring may alert system operators to the unexpected impact of changes in the environment. Clearly, if there are major changes in the design and features of an order check system such as those suggested by other researchers (categorized override reasons, tighter integration with the maintenance of patient allergy lists, suppression of renewal order checks for previously tolerated medications) or in our case the addition of non-VA medications and changes in topical medications, evaluation of ordering and override rates would be warranted. ^4,5,7 Less obvious, perhaps, is that indirect changes such as changes in patient population, house staff, or system policies could also have unexpected effects on order checking and must be monitored as well.

In the post-analysis of these results compared to 2001, we noted the statistical increase in the overall rate of high severity order checks from 0.5% to 2.5%. This is in part due to the introduction of new critical order check types such as “No patient allergy assessment” and possible changes in logic of previously existing order check types. We also speculated that new VA Puget Sound allergy policies might have contributed to the much higher number of drug-allergy order checks. Previously only pharmacists could enter patient allergies, but a new policy permits practitioners, nurses and dieticians to enter allergies as well. However, the ability to remove allergies is limited to pharmacists. In addition, during the period between the two studies, VA Puget Sound began standardizing allergy data by disallowing free text allergy entry and matching existing free text allergies with drug file allergies and removing any unmatched entries. Any new locally standardized allergy terms are submitted to a national data standardization process to be added to the national drug file. We did not control for these factors in our study design, but we think it likely that these new policies were unanticipated contributors to changes in order check behavior.

As Van Der Sijs et. al concluded in their review of drug safety order check studies, error factors can unwittingly originate at many levels from the individual to the organization, and maintaining both the high sensitivity and specificity of order checks is one of the challenges of decision support systems. ³ Frequent order check evaluation with supporting system and environmental knowledge could help system operators adjust and improve their decision support systems before problems such as distrust or order check fatigue becomes an issue. Override rates would presumably be one component of such an evaluation, but because they only measure the final step in practitioner order entry, other evaluation methods such as behavior observation and work analysis should also be utilized to paint a richer picture of order checks and ordering behavior.

Our study is similar to other quantitative studies that have reported override rates that are generally considered high. 4,7,13,14 However, our purpose was not only to show current override rates at VA Puget Sound. We wished to demonstrate and discuss issues regarding local monitoring of practitioner order check override rates in a centrally developed CPOE system as part of on-going quality assurance.

There is significant interest in the medical informatics community in improving CPOE systems such as the VA's CPRS system by reducing override rates through the elimination of clinically irrelevant order checks. Shah et al. report a higher practitioner acceptance of order checks when only a subset of the original drug database was used and when only the most critical order checks required practitioner action before signing, and Weingart and colleagues recommend that clinically irrelevant order checks be suppressed. ^7,14 Current VA order check logic is based in large part on VA drug classes that often group pharmacologically unrelated medications and is believed to contribute to unacceptably high override rates for CPRS that studies such as ours continue to show. To address this source of clinically irrelevant order checks, the VA has purchased a proprietary database that includes drug-drug and drug-allergy interaction order checking based on more specific chemical structure rather than broad drug classes and offers new features such as dosage checking. In addition, the VA is exploring other features such as expanded laboratory finding order checks and incorporating co-existing problems and patient characteristics including age, gender, and potential for pregnancy.

It is worth noting, that while we did not qualitatively evaluate the clinical relevance of the order checks in our data set, when examining other studies, a surprisingly large percentage of order checks appear to be clinically relevant compared to the corresponding override rates. This suggests that if a decision support system has a high override rate, it does not necessarily follow that a high percentage of order checks are clinically irrelevant. In the study by Weingart et al, 41% of drug-drug and 24% of drug-allergy order checks were deemed inappropriate, leaving the majority of order checks, in fact, appropriate. ⁷ However, the same study measured 89% drug-drug and 91% drug-allergy override rates implying that many clinically relevant order checks were being overridden. Similarly, in a study by Hsieh that reported an 80% override rate of drug-allergy order checks, 55% of the override reasons fell into the “Aware/Will Monitor” category indicating that the majority of these order checks may have been clinically relevant as well. In our case, although we report a very high override rate and show that it has stayed high over time, we do not believe that it should necessarily be a goal to reduce this rate without considering other factors of practitioner work.

Many overridden order checks may be clinically relevant or there may be a wide variation in perceived clinical relevance of order checks, as Spina et al conclude. ¹⁵ Certainly, we take seriously the problems associated with high override rates: informatics research has appropriately focused on practitioner acceptance of decision support systems, and a system that includes many order checks that force the practitioner to respond can be disruptive and perceived as a nuisance. However, one recent study of perceptions of CPRS orders checks suggests that practitioners may be more accepting of “false positive” order checks than previously reported and furthermore found that these false positives may have a neutral to positive impact. ¹⁶

It is important to remember that 15% of the order checks in our study resulted in a cancelled order that presumably enhanced patient safety. Override rates themselves should not be used as the only gauge of system performance because these numbers do not indicate the practitioners' decision-making process. They are specific, recordable, yes/no decision points that may not accurately reflect the complexities of such a process. In the analysis of our log data, we observed that some overrides occurred following the cancellation of an initial order with the same orderable item and order check possibly indicating that the practitioner thoroughly considered the order check before re-entering the order and overriding the order check. For other orders, practitioners entered override reasons containing only a space or period character possibly indicating that the practitioner barely looked at the order check before overriding or felt it was a nuisance not worthy of explanation. More research is needed to better understand ordering and order check behaviors and their relationship to information needs, decision making system quality, and ultimately patient outcomes.

Limitations

Because CPRS does not save all cancelled orders, we used a prospective logging system to capture orders as they are being entered. For this study, we were unable to use the same logging methods from our 2001 study although the underlying system, CPRS, and the study measurements (orders, order checks) were the same. As we have discussed above, within each order check type we chose to analyze the order override rate because the practitioner is presented with a single interruptive window containing all of the order checks. In our comparison to previous results that analyzed order checks separately, we acknowledge the possibility that the previously reported drug-drug order check override rate may be slightly lower than the order override rate. This highlights the difficulty in using retrospective analysis of CPRS orders to determine the relevancy of individual order checks if several order checks appear on the same screen and the order is signed with a single override reason. We believe this supports our recommendation that qualitative work be carried out in parallel with quantitative order check analysis to analyze user behavior or order check effectiveness.

Also, we sampled orders at different times during the year. For this study, we analyzed orders over 6 days in early January, excluding the intervening weekend whereas the 2001 study analyzed orders entered during a continuous week in early August (August 1, 2001 through August 8, 2001). ⁶ It is possible that varied ordering practitioner (primarily house staff) experience influenced the results.

It is challenging to identify which factor or combination of factors, both technical and social, may have contributed to new system behaviors including significantly higher drug-allergy order check and override rates. Our study did not control for many possible changes in the environment so we cannot say with certainty the cause of the increases we report. We speculate that the addition of non-VA medications or changes in hospital policy supporting more comprehensive allergy documentation may have affected override rates and we recommend that further research be done to determine whether this is the case.

Conclusion

We have used new techniques to analyze override rates at VA Puget Sound Health Care System for the CPRS order entry system to determine if they have changed since 2001. Because our original data gathering method was unavailable to us and also because CPRS does not retain all cancelled orders, we used an existing debug program in CPRS to prospectively log all order activity during our study period. Our results show that drug-drug override and drug-allergy override rates at VA Puget Sound remained high from 2001 to 2006 with significant increases in drug-allergy order checks.

Many possible factors affect override rates and at VA Puget Sound, we have identified several possibilities including policy changes and changes in rule bases and drug files. We find it interesting that a notable reason for overrides in 2001, topical medications, had been addressed by 2006 via system upgrades leading us to believe that other unanticipated factors may have played even more significant a role in the high override rates. Monitoring of decision support systems, including override rates, contributes to improved understanding of their use, can help detect unanticipated changes in system behavior and should be a regular practice in clinical computing systems. However, we also believe that looking simply at the override rate itself is not necessarily a measure of the usefulness of order checks and should only be one factor when evaluating system changes or making system improvements. We argue that to better evaluate order check systems, qualitative, observational work should be carried out in parallel with quantitative order check monitoring to better understand clinical decision making and the interactions physicians have with information and decision support systems. Such studies should help both enumerate the factors that affect override decisions, as well as help assess when an order check is clinically valuable, independently of whether or not that check is overridden. If our goal is to improve such decision support systems, we should aim to improve the delivery of clinically relevant, useful information, rather than simply aim to reduce the override rate.