Optimizing the acceptance of medication-based alerts by physicians during CPOE implementation in a community hospital environment

Abstract

There are significant challenges surrounding the implementation of CPOE systems in community hospitals without a mandate for system use. One of these is to incorporate clinical decision support such as medication-based alerts, which are key to benefit realization, but can be perceived adversely by busy physicians trying to learn a new system, without negatively affecting usability and adoption. We describe a phased, iterative approach to incorporating medical-based alerts into a successful CPOE deployment. During the first 8 months after system activation, we continuously monitored the frequency of alert presentation, the frequency of “positive” responses to the alerts and physician satisfaction with each of these. Responses included alterations in filtering strategies, changes in individual drug profiles; changes in user display options; and the addition or withdrawal of entire alert categories in order to decrease maximize value and acceptance. With this strategy we were able to document order removal or modification rates of 21–66% for 6 alert categories while still achieving CPOE adoption of 75–78% during this period.

Introduction

Memorial Health Services (MemorialCare) is a multi-hospital system, compromised of 6 not for profit hospitals in Los Angeles and Orange County. All hospitals have voluntary medical staff, with a very small number of employed physicians, mainly in non-patient care positions (mainly quality improvement and medical staff leadership). We are in the midst of a multi-year rollout of an inpatient-focused electronic health record across the enterprise. A primary goal of this initiative is our goal of maximizing clinical quality through the general adoption of Computerized Physician Order Entry (CPOE) across all facilities. Positioning the enterprise for success in this challenging endeavor was the subject of a multi-pronged effort over three years prior to the initial hospital go-live in June, 2006.

One of the principal goals of the project was to maximize patient safety and clinical quality. One of the significant targets in this process was medication orders. The decision to implement this system was heavily predicated on the ability of CPOE to intervene real-time in the order-related decision-making process and maximize the quality and effectiveness of the subsequent orders, as others have described in the informatics literature.¹ Prior to the implementation of the current integrated system, medication orders were entered into a legacy order entry system, then printed in the pharmacy and reentered into a disparate pharmacy system. The high frequency of translational errors associated with this non-integrated approach (1–2%) was of great concern and is a subject of another paper in preparation from our group.² In that legacy system, physicians or nurses entering orders into the legacy system did not see any medication-based alerts; pharmacists saw only drug-drug interaction and allergy alerts, and the later was contingent on the re-entry of that information from the order entry system in use on the units into the pharmacy system.

The new integrated system allowed us to present a variety of medication-based alerts to physicians that we believed would result in an improved quality of medication ordering. However, the nature of the implementation in a community-based hospital system without a mandate for physician system use, required a thoughtful approach designed to maximize physician usability and acceptance. If physicians found the order entry process cumbersome, it was clear that CPOE would not be adopted and our efforts to improve the quality of their ordering practices with real-time alerts would fail. This has been well described previously in academic and community hospital implementations.^3–6 In a separate paper, we describe a physician governance strategy designed to maximize system adoption. This strategy focused on involving physicians actively and from the outset in system design and deployment strategies, maximizing efforts in readiness assessment and change management education and aggressively supporting physicians throughout the implementation period ⁷

Strategic Decisions

Among the key design decisions made by the physician steering committee in their attempt to improve order quality while minimizing were the following:

• At implementation only “key” medication-based alerts would be deployed to maximize the perception of value and minimize alert fatigue.

• The number and category of alerts would be increased as system acceptance and CPOE utilization became routine.

• Pharmacists agreed to use the same “base” alert level as chosen by physician steering committee, so physicians would rarely be contacted asynchronously about alerts they had not seen when they originally entered the order. The physician steering committee determined that it would be a major disincentive for physician adoption of the CPOE system if they determined that doing so would not prevent them from further pharmacy staff follow-up.

• All licensed physicians and their agents would view the same alerts. Although this guaranteed their value regardless of whether physicians or nurses were entering the orders and might be interpreted as a disincentive to CPOE, we determined that it was important for physicians or other clinicians entering the same orders to see the same alerts to avoid any confusion about their importance or relevance.

Our goal was to define a process that resulted in alerts causing the reconsideration or deletion of the relevant medication orders more than 25–30% of the time. Although an upper limit of acceptance was not determined, it was clear from the literature that a figure significantly less than 100% was appropriate if we were not to miss a significant number of opportunities for clinically relevant interventions⁸. We were also concerned about the absolute frequency of alert presentation regardless of whether the alerts were accepted or not, but did not determine specific numerical targets given our lack of experience.

In order to maximize their acceptability, the physician steering and medication management teams agreed that the alerts used at the time of activation should focus on those which would be easy to understand, yet have the potential to affect a large number of problematic orders. This resulted in our focus on medication-related orders for several reasons. First the clinical content was embedded in the drug knowledgebase in use, and would not require further research on our part. Second, these rules categories were well-known and would likely not be a subject of debate themselves, as home-grown clinical content might. Finally the risk of inappropriate medication orders was well appreciated by the medical staff.

Therefore, the categories chosen for initial implementation were Drug Allergy (including inactive ingredients), Drug-Drug interactions (critical only), Duplicate Medication Checking and Pregnancy and Lactation related alerts. These alerts appeared to have the highest “value to noise” ratios, based upon integrated and workflow testing and physician and pharmacist assessment. Based upon other pre-go-live testing it was agreed not to deploy alerts related to IV incompatibilities or dose range related alerts. The former decision was made because of the difficulty in managing alerts related to the solutions in which drugs were reconstituted. Dose range checking was temporary deferred until all drug records could be reviewed and the ranges made as specific as possible. A preliminary review of the ranges found in the commercial drug knowledge-based indicated that the overlap between standard dosing and doses commonly given was substantial.

Alert categories which were not further evaluated following steering committee discussions included therapeutic duplications and food-drug interactions, the former because of our perception that this would result in a large number of non-accepted alerts and the later because of an assessment that using the prescribed diet as the basis for such alerts was problematic since there was no way to know what foods a patient was actually ingesting. As with all similar decisions, the overriding concern was that acceptance of a medication alert strategy would be adversely affected if an entire alert category was perceived to be a nuisance.

Because of the method by which the drug knowledgebase interacted with the order entry system, the process by which alert categories in use were fine-tuned depended heavily on a filtering process that allowed us to display the alerts believed to be most likely to result in changes in medication order decision-making. Continued fine tuning of medication orders associated with what was perceived to be inappropriate or inefficient alert presentations has continued throughout. The physicians and pharmacists met regularly and quickly agreed to filter medication-based alerts that physician and pharmacy leads agreed had excessive noise to value ratios. For example, an overwhelming number of duplicate alerts (> 1600 per week) led to a plethora of physician complaints and a rapid decision (within the first week after go-live) to limit such duplicate medication alerts to pharmacists only since there was no straightforward method to filter them. This decision was made despite the fact that 28% of these alerts resulted in orders being reconsidered or deleted.

Data Extraction and Analysis

The relevant data included the frequency of alert presentation by category, the actions taken on alert presentation (alert overridden, order deleted, alert cancelled & order further reconsidered or modified) and the CPOE utilization statistics. These were captured in appropriate master files within the clinical system database and extracted using pre-existing programming scripts provided by the product vendor. The data were then imported into Excel spreadsheets for further manipulation and presentation.

Special note is made of the category of “alert cancelled”. When presented with an alert, the clinical choices to remove the order from the queue or override the alert are clearly understood. The decision to cancel the alert is somewhat problematic since the ultimate outcome is not clear-cut. We have displayed this category with “orders removed” because the clinician at a minimum reconsidered the order before deciding to modify or proceed with it. This in itself was seen as a value-added feature, even if after reconsideration or discussion, perhaps with another physician or pharmacist, the ordering clinician elected to proceed and override the ‘re-displayed” alert.

Results

During the first 6 months after go-live, we were surprised at the number of alerts which were presented related to medication orders, and the frequency with which the orders were removed from the queue or the order cancelled and re-considered. A total of 46,000 medication-order related alerts that were displayed to clinicians during this period, had one of those two results (almost 280 per day). The frequency with which alerts resulted in order removal or reconsideration varied, as expected, from a low of 32.7% for those related to pregnancy to 66% for drug-drug interactions (Figure 1)

Figure 1.

First 6 months after activation: overridden alerts, and those for which the order was removed or the alert cancelled and the order reconsidered.

Of note, while 80% of the allergy alert displays resulted in the order being removed, the majority of alerts in all other categories resulted in alert cancellation and order reconsideration. Because of this, during the first 6 months, we changed the level of approximately 1000 drug interactions (up or down) to increase value. Furthermore, because of the unexpectedly high frequency of duplicate medication alerts (> 250 per day) the display of such alerts was removed from the physician view and limited to pharmacy only. Interestingly, the rate at which such alerts resulted in alert cancellation and order reconsideration varied little between the physician and pharmacy domains (23 to 28%).

Because of the rapid adoption of CPOE and the acceptance of the frequency of alert presentation, at 2 months into the activation, high dose alerts were added to the repertoire. At the same time, the display of severe drug-drug interactions to the critical interactions previously seen.

Unfortunately, despite our best efforts to ‘fine-tune” the dose ranges provided in the drug knowledgebase to the highest risk situations, the frequency with which such alerts results in action has continued to be low (2.2% orders removed, 13% alerts cancelled and order reconsidered). Subsequently we have further adjusted dose ranges on almost 1000 additional drug records to eliminate this apparent excess of unnecessary dose alerts. This has resulted in a small increase in the rate at which such alerts result in order removal or reconsideration (21%), although this is still below are target range of 25–30%.

The overall frequency of alert presentation has decreased somewhat, but in months 7 and 8 post activation, 7300 orders (125 per day) were affected by medication based alerts – either order removal or alert cancellation and order reevaluation (Figure 2).

Figure 2.

Alert management in months 7 & 8 after activation- the effect of further iterative and filtering modifications.

The frequency of order modification related to drug interactions has continued in the acceptable range (66%), but the frequency with which these alerts are displayed is a continuing source of MD frustration. Hence filtering has been increased to focus on the highest-risk interactions (all critical, but only some serious interactions, as defined in the drug knowledgebase).

As we continue to iteratively modify alert display options, tracking the frequency of alerts and the actions taken upon them, we have been able to show that the majority of these actions are taken by physicians rather than pharmacists with ongoing rates of order removal or cancellation of 17–53% (Table 1). Only drug-drug interaction related alerts typically result in orders being removed (deleted from the order queue), but we contend that the process of canceling and reevaluating the order is an additional value-added feature.

Table 1.

Rates at which medication-based alerts result in order removal or cancellation during the 7^th and 8^th month after activation

Type	% Orders removed or cancelled	% Actions taken by Physician	Action taken
Drug-Drug Interaction	53.1%	54%	95% cancelled
Allergy	37.0%	77%	80% removed
Dose Alert	17.2%	49%	52% removed
Pregnancy	27.1%	55%	90% cancelled
Lactation	45.7%	46%	90% cancelled

The major concern that has driven this focus on alert monitoring has, of course, been its effect on physician use of the order entry system and adoption rates. We have monitored the rates of CPOE since the inception of the project and have predicated modifications of the clinical decision support capabilities upon these continued high rates of adoption. We have therefore been gratified to see that CPOE adoption rates have remained steady despite the regular “insertion” of new alert categories and new drug-based alerts within already deployed categories.

Physician CPOE rates are calculated as CPOE orders (the user entering the order <u>is</u> the authorizing provider) plus standard protocol orders divided by all orders placed (Figure 4). Although protocol and other standing orders are a small percentage of total orders they match our reasons for using CPOE and do not require clinical decision support processing. Since these orders are pre-defined and approved they improve the timing of care, reduce the variation of clinical practice and eliminate ambiguous orders as do CPOE orders.

Figure 4.

Physician CPOE during the first 8 months of system use at MemorialCare showing persistently high rates of physician adoption of CPOE (73–78%). The rates of pre-existing POE utilization using a legacy system are also shown (25–30%)

Ongoing Challenges and Concerns

There are a number of drugs, such as amiodarone, which the drug database documents as having severe or contraindicated interactions with other drugs (such as warfarin or levofloxacin) with which it is commonly administered. This has been an ongoing frustration, but a simple answer is elusive since filtering is limited to categories of interaction severity and not individual drug combinations.

Low rates of “effective” pregnancy-related alerts appear to be related to the fact that physicians are in the patient record ordering medications before nursing completes their admission assessments and documents patient age and non-pregnant status. This can be especially annoying when the patient is elderly. A new application feature is planned to enable the physician to easily document non-pregnant status on such patients as part of the physician admission note and we are hopeful that this may solve this problem.

Our attempts to better effect the implementation of dose range alerts have been discouraged by challenges with prn (as necessary) medications. The current version of our system calculates the dose to be used in daily dose range checking as the total daily dose that “would” be administered if the drug was to be given at the maximum frequency ordered, no matter how unlikely that may be (i.e. by multiplying a single dose amount by 8, for a medication to be given every 3 hours, as necessary for pain). This has resulted in many dose range alerts of dubious value, and will need to be better addressed if more effective use of dose range alerts is to be effected.

Finally, we do intend to redeploy duplicate drug alerts to physicians in the near future but additional work is needed by the physician steering committee and pharmacy staff to be certain that these are perceived to be of value and not as an annoyance that could affect overall physician satisfaction and adoption rates.

Conclusion

Achieving and maintaining excellent rates of physician adoption of CPOE while implementing real-time medication order-based alerts is a significant, but not insurmountable, challenge in any hospital setting^9–10 and can be particularly challenging in a community hospital with a voluntary medical staff. With ongoing attention to the rates at which alerts lead to changes or reconsiderations of medication orders and the frequency with which such alerts appear, physician usage can be maintained and improved medication order entry achieved¹¹. However, this process is not limited to the initial design and implementation process, but carries through the post-go-live period as new problems and challenges are identified. It is clear that success will demand ongoing attention to these issues for a long, perhaps indefinite, period of time.

Figure 3.

Filtering of alerts during months 7 and 8 after activation, 67% of total are filtered (75 – 80% of drug-drug interaction, pregnancy & lactation alerts).