Improving Acceptance of Computerized Prescribing Alerts in Ambulatory Care

Abstract

Computerized drug prescribing alerts can improve patient safety, but are often overridden because of poor specificity and alert overload. Our objective was to improve clinician acceptance of drug alerts by designing a selective set of drug alerts for the ambulatory care setting and minimizing workflow disruptions by designating only critical to high-severity alerts to be interruptive to clinician workflow. The alerts were presented to clinicians using computerized prescribing within an electronic medical record in 31 Boston-area practices. There were 18,115 drug alerts generated during our six-month study period. Of these, 12,933 (71%) were noninterruptive and 5,182 (29%) interruptive. Of the 5,182 interruptive alerts, 67% were accepted. Reasons for overrides varied for each drug alert category and provided potentially useful information for future alert improvement. These data suggest that it is possible to design computerized prescribing decision support with high rates of alert recommendation acceptance by clinicians.

Computerized prescribing applications that embed clinical decision support systems (CDSS) within computerized provider order entry reduce medication error rates both by structuring prescriptions and by checking them for potential problems such as drug interactions, allergies, and other issues.^{1,2,3,4,5,6,7,8,9} If a potential problem is found, the CDSS can provide clinicians with real-time alerts, allowing the clinician to make appropriate changes before the prescription is finalized.

While computerized prescribing applications are commercially available (either as stand-alone applications or as part of an electronic medical record), these systems may not be as effective for improving safety if clinicians override clinically important alerts. When the threshold for alerting is set too low, clinicians are inundated with alerts of low clinical significance, leading to high override rates and the potential to override even important alerts.^10,11,12 In one inpatient study, Payne et al.¹⁰ found an 88% override rate for drug interaction alerts, and a 69% override rate for drug-allergy alerts. Similarly, Weingart et al¹¹ found ambulatory physicians overrode 91% of drug-allergy alerts, and 89% of high-severity drug-drug interaction alerts.

Many CDSS use commercial knowledge bases to drive their alerting. These knowledge bases are often highly inclusive, placing more emphasis on breadth of coverage than on clinical relevancy or severity of adverse events.¹³ However, this approach can have serious consequences. If too many alerts are delivered, in addition to missing important alerts, clinicians may refuse the application altogether due to disruptions in workflow.^14,15 When designing knowledge bases for CDSS, care must be taken to display alerts judiciously and to maintain the right balance between useful alerting and overalerting.

Our objective was to improve clinician acceptance of drug alerts by designing a selective set of clinically significant drug alerts for the ambulatory care setting and minimizing workflow disruptions by designating only critical to high-severity alerts to be interruptive to clinician workflow. In addition, we required prescribers to supply reasons for overriding alerts to better understand their rationale. This article reports on (1) the extent our alert design minimized workflow interruptions, (2) the clinician accept rate of our selective alerts, (3) the specific types of alerts clinicians accepted most frequently, and (4) the reasons clinicians gave for overriding alerts.

Methods

Setting

This study included clinicians at 31 adult primary care practices affiliated with Brigham and Women's Hospital (BWH) and Massachusetts General Hospital (MGH), two Boston teaching hospitals in the Partners HealthCare System. The sites included nine academic hospital-based clinics, 17 off-site clinics, and five community health centers. The prescribing medical staff included 701 clinicians, composed of 224 attending physicians, 249 resident physicians, 35 nurse practitioners, and 193 ancillary staff including nurses and medical assistants.

LMR and Computerized Prescribing

The Longitudinal Medical Record (LMR) is a Partners-developed electronic medical record implemented in 2000. The LMR has features including note writing, access to laboratory, radiology, and pathology reports and computerized prescribing, which permits clinicians to enter prescriptions directly into the computer. Each prescription is placed sequentially in the LMR and constitutes a separate order for drug contraindication checking by the CDSS. The current medication list is continually updated in real time as new medications are entered or discontinued. The computerized prescribing application ensures the completeness of all prescriptions by providing clinicians with a coded dictionary of medication names, available strengths including default doses, and required fields for drug dose, number of pills or units dispensed, and refills. Prescriptions cannot be signed and completed until all relevant information has been entered by the prescriber. Additionally, the application ensures legibility by printing or electronically faxing typed prescriptions to pharmacies.

Knowledge Base Creation

We created a knowledge base containing the drug contraindications to be used in our computerized prescribing CDSS. A physician and pharmacist expert panel was convened to review potential duplicate drug class, drug-disease, drug-drug, drug-lab, and drug-pregnancy contraindications including those available through First DataBank, Hansten's, U.S. Food and Drug Administration resources, preexisting locally available knowledge bases, and peer-reviewed literature in order to assess clinical importance. Only alerts clinically relevant to the ambulatory setting were included in the knowledge base. Drug-allergy alerts had previously been implemented in the LMR and did not undergo the same knowledge base intervention and thus were not part of this study. Based on the severity of the adverse event, likelihood of event occurrence, strength of the supporting clinical evidence including clinical studies, case reports, and expert panel experience, and whether the combination was ever clinically indicated, the expert panel placed each alert into one of three clinical severity tiers. All decisions were reached through iterative discussion and consensus agreement. If drugs were outside the scope of the expert panel, then individual clinical experts were consulted. Level 1 alerts indicate a fatal or life-threatening interaction, such as the combination of erythromycin and diltiazem, which increases the risk of ventricular arrhythmias. Level 2 alerts indicate an undesirable interaction with the potential for serious injury, such as the use of metformin in a patient with a serum creatinine greater than 1.4 mg/dL, which increases the risk of lactic acidosis. Level 3 alerts indicate the possibility of an undesirable interaction in which a drug should only be used with caution or may require increased monitoring, such as the combination of warfarin and levofloxacin that may result in an elevated prothrombin time. The entire review process took approximately four months to complete.

Prescribing Alerts

We designed computerized alerts for the selected drug contraindications and all alerts were implemented at all 31 adult primary care sites in this study. The CDSS uses data from the LMR about each patient's active medication list at time of medication ordering, problem list, laboratory results, and demographics to identify potential contraindications (▶). When a clinician begins an order for a contraindicated medication, the alert appears as an on-screen warning identifying the contraindication. A single medication can generate multiple alerts displayed on a single screen, each requiring a separate clinician action.

Table 1.

Examples of Drug Alert Contraindications

Contraindication	Example
Duplicate drug class	A clinician orders captopril, and lisinopril is already on the patient's medication list.
	A clinician orders diazepam, and lorazepam is already on the patient's medication list.
Drug-drug	A clinician orders isocarboxazid, and meperidine is already on the patient's medication list.
	A clinician orders sildenafil, and nitroglycerin is already on the patient's medication list.
Drug-lab	A clinician orders metformin, and the patient's creatinine is >1.4 mg/dL.
	A clinician orders hydrochlorothiazide, and the patient's potassium is < 3.0 mEq/L.
Drug-disease	A clinician orders sumatriptan, and coronary artery disease is on the patient's problem list.
	A clinician orders lovastatin, and hepatic disease is on the patient's problem list.
Drug-pregnancy	A clinician orders isotretinoin, and the patient has a positive pregnancy test within the past nine months.
	A clinician orders warfarin, and the patient has a positive pregnancy test within the past nine months.

With Level 1 alerts, clinicians could not proceed with the prescription without either eliminating the contraindication or in the case of drug-pregnancy alerts, indicating that the patient was not pregnant or of child-bearing age (▶). With Level 2 alerts, clinicians could proceed if they provided any override reason (▶). The clinician could either choose an override reason from a preselected group of coded responses or type a response into a free-text box. The default was to cancel the order. If the clinician selected an override reason stating he or she would discontinue the contraindicated preexisting drug or diagnosis identified by the alert, these actions were facilitated through the automatic generation of a button that the clinician could click to discontinue the drug or disease. For drug pregnancy alerts, if the clinician chose the coded response stating that the patient was not pregnant, then subsequent Level 2 drug-pregnancy alerts would only be generated if a new positive human β-chorionic gonadotropin, which indicated pregnancy was recorded in the laboratory results. Level 3 alerts were displayed for clinician viewing on the top of the computer screen in red letters (▶). Level 3 alerts were specifically designed to minimize clinician interruptions while still conveying clinical information.

Figure 1.

Example of an interruptive Level 1 drug-drug contraindication alert presented to a clinician ordering erythromycin when diltiazem is already on the patient's medication list. Alert requires the clinician to cancel or modify order before continuing with prescription.

Figure 2.

Example of an interruptive Level 2 duplicate drug class alert presented to a clinician ordering diazepam when lorazepam is already on the patient's medication list. Alert requires clinician to cancel order or provide override reason before continuing with prescription.

Figure 3.

Example of a noninterruptive Level 3 drug-drug contraindication alert presented to a clinician ordering levofloxacin when warfarin is already on patient's medication list. Alert is presented in red letters in upper left-hand corner of screen for clinician's viewing. No additional action required before prescription completion.

Data Collection

All alerts were presented to clinicians at each site. Data were electronically collected each time a clinician entered a prescription that triggered an alert during the six-month period between August 5, 2004, and January 5, 2005. A file was created for each drug alert that included the patient's name, medical record number, clinician user's name and practice location, name of medication that generated the alert, date, alert type, severity level, and clinician action including override reasons when applicable.

The study was approved by the Institutional Review Board of Partners HealthCare System.

Definitions

We defined alerts as “interruptive” if they required a user action before the prescription could be completed and were assigned Level 1 or Level 2. We defined alerts as “noninterruptive” if they were in the Level 3 severity tier since they did not require a user action before prescription completion.

We defined the clinician action “cancel order” as the clinician aborting the attempted prescription. The action “modify order” was defined as the clinician indicating from the override response list that he or she would discontinue or hold the contraindicated preexisting drug or diagnosis identified by the alert. We chose to call these override reasons “modify order” because these actions would eliminate the drug contraindication. Therefore, the clinician was considered to have “accepted alert” with an action of either “cancel order” or “modify order.” An “alert override” was defined as the clinician choosing to continue with the prescription with an override reason that would not eliminate the drug contraindication.

Statistical Analysis

Descriptive statistics were used to summarize the alerts triggered in each drug alert level and category, the alerts clinicians accepted and overrode overall and in each drug alert category, and the override reasons in each drug alert category. Analyses were conducted using Microsoft Excel 2000 and SAS statistical software, version 8 (SAS Institute Inc., Cary, NC).

Status Report

The final knowledge base contained 1,444 drug contraindication rules including 192 duplicate drug class, 326 drug-disease, 351 drug-drug, 255 drug-lab, and 320 drug-pregnancy rules. The distribution of alerts was 2% Level 1 alerts, 63% Level 2 alerts, and 35% Level 3 alerts.

We evaluated all 18,115 drug alerts generated during our study period. These included 5,182 (29%) interruptive Level 1 or 2 alerts and 12,933 (71%) noninterruptive Level 3 alerts. The majority of interruptive alerts were in the duplicate drug class category (3,875), followed by drug-drug interactions (1,078; ▶).

Table 2.

Alerts by Type

Contraindication	No.	Interruptive (Levels 1 and 2) No. (%)	Noninterruptive (Level 3) No. (%)
Duplicate class	3,875	3,875 (100)	N/A*
Drug-drug	4,625	1,078 (23)	3,547 (77)
Drug-lab	4,536	92 (2)	4,444 (98)
Drug-disease	43	19 (44)	24 (56)
Drug-pregnancy	5,036	118 (2)	4,918 (98)
All	18,115	5,182 (29)	12,933 (71)

^*No alerts of this type in knowledge base.

Among the 5,182 interruptive drug alerts presented, the order was canceled in 993 (19%) and modified in 2,482 (48%), resulting in a 67% accept rate. Accept rates varied considerably among different alert categories. The highest accept rate was observed in the duplicate drug class category (77%), followed by drug-disease alerts (53%). The lowest accept rate of 10% was seen among the drug-pregnancy alerts (▶).

Table 3.

Accept and Override Rates for Interruptive Alerts

		No. (%) of Alerts Accepted*
Contraindication	No.	No. (%) of Orders Canceled‡	No. (%) of Orders Modified§	Total No. (%) of Alerts Accepted	No. (%) of Alerts Overridden†
Duplicate class (Level 2 only)	3,875	681 (18)	2,284 (59)	2,965 (77)	910 (23)
Drug-drug	1,078	254 (24)	197 (18)	451 (42)	627 (58)
Level 1	13	4 (31)	9 (69)	13 (100)	0 (0)
Level 2	1,065	250 (23)	188 (18)	438 (41)	627 (59)
Drug-lab (Level 2 only)	92	37 (40)	0 (0)	37 (40)	55 (60)
Drug-disease (Level 2 only)	19	9 (47)	1 (5)	10 (53)	9 (47)
Drug-pregnancy	118	12 (10)	0 (0)	12 (10)	106 (90)
Level 1	16	2 (13)	0 (0)	2 (13)	14 (88)
Level 2	102	10 (10)	0 (0)	10 (10)	92 (90)
Total	5,182	993 (19)	2,482 (48)	3,475 (67)	1,707 (33)

^*Accepted alert: clinician canceled or modified order.

^†Overridden alert: clinician chose to continue with prescription with an override reason that would not eliminate the drug contraindication.

^‡Canceled order: clinician aborted the attempted prescription.

^§Modified order: clinician indicated override reason that would eliminate the drug contraindication.

Of the 1,707 overridden alerts, 245 had no override reason because the clinician chose “Other” from the coded responses, but then left the free-text box blank. Among the remaining 1,462 overridden alerts, the reasons for override varied by alert type (▶).

Table 4.

Most Frequent Override Reasons Provided by Clinician

Contraindication	Override Reasons (%)
Duplicate class	Transitioning from one medication to other (42)
	Patient is on long-term therapy with combination (21)
	Short-term or as-needed dosing (7)
Drug-drug	Will monitor as recommended (49)
	Patient has already tolerated combination (21)
	Will adjust dose as recommended (14)
Drug-lab	Will monitor/manage as recommended (67)
	More recent lab result available that warrants use (18)
Drug-disease	Patient has tolerated this drug in the past (56)
	New evidence supports therapy of this type (22)
Drug-pregnancy	Patient is not pregnant (90)
	Patient is not of child-bearing potential (5)

Duplicate Drug Class

Of the 3,875 interruptive duplicate drug class alerts, 2,965 (77%) were accepted (▶), predominantly via the order modifying action of discontinuing the preexisting medication. The medication classes that generated an alert were analgesic (29%), psychiatric (26%), gastrointestinal (19%), cardiac (17%), and endocrine (9%). The accept rates were high in all medication classes, ranging from 86% for endocrine to 61% for psychiatric medications. Override reasons included the patient was “transitioning from one drug to the other” (42%), the patient was “on long-term therapy with combination” (21%), the patient was being placed on combination for a short-term or as-needed basis only (7%), the drug was ordered as per “advice from a consultant” (5%), or as per “MD orders” (2%), and “new evidence” exists for use (2%).

Drug-Drug Interactions

Of the 1,078 interruptive drug-drug interaction alerts, 13 Level 1 alerts were generated, all of which required the clinician to either cancel the order or discontinue the previous medication. These 13 Level 1 drug-drug contraindications included three for sildenafil and isosorbide mononitrate, three for gatifloxacin and levofloxacin, two for linezolid and methylphenidate, and one each for isocarboxazid and amphetamine/dextroamphetamine, eplerenone and spironolactone, linezolid and Sinemet, linezolid and methylphenidate, and fluoxetine and selegiline. The remaining 1,065 drug-drug contraindication alerts were Level 2 alerts, of which 438 (41%) were accepted (▶), representing 250 order cancels and 188 order modifications. Override reasons included the clinician would monitor the patient (49%), the patient had previously tolerated the medication (21%), the clinician would “adjust dose as recommended” (14%), and “no reasonable alternatives” (4%).

Drug-Lab

Among the 92 interruptive drug-lab alerts, 37 (40%) were accepted (▶). Of the overridden alerts, there were 37 (67%) in which the clinician stated he or she would “monitor/manage as recommended” and the appropriate laboratory test was performed in 28 (76%). Another 10 alerts (18%) were overridden with the clinician stating there was a “more recent lab result available” (likely performed at outside facilities and not available in the LMR), and six alerts (11%) generated based on renal function were overridden with the clinician stating the patient was on dialysis.

Drug-Disease

Among the 19 interruptive drug-disease alerts, 10 (53%) were accepted (▶), including nine where the clinician canceled the order and one where the clinician modified the order by choosing “discontinue preexisting diagnosis.” The 19 alerts included 12 hepatic disease contraindication alerts, seven of which were accepted; five seizure disorder contraindication alerts, three of which were accepted; and two coronary artery disease contraindication alerts, both of which were overridden. Reasons for alert overrides were the patient had “tolerated the medication in the past” (56%), and there was “new evidence” for use of the medication (22%), “advice from a consultant” (11%), and “no reasonable alternatives” (11%).

Drug-Pregnancy

Among 118 interruptive drug-pregnancy alerts, 16 (14%) were Level 1 (▶). These Level 1 alerts included 11 for isotretinoin, four for leflunomide, and one for misoprostol. The order was canceled in two cases. For the remaining 14, the clinician indicated either the “patient is not pregnant” or “patient is not of child-bearing potential.” Among 102 Level 2 alerts, 10 (10%) were accepted. Override reasons included “patient is not pregnant” (93%), “advice from a consultant” (1%), “no reasonable alternative” (1%), patient has “tolerated” in past (1%), and medication is for short-term/as-needed use only (1%).

Discussion

We found high user acceptance of ambulatory computerized prescribing alerts when using a selective knowledge base and minimizing workflow interruptions. By implementing tiered alerts, we limited alert burden by assigning 71% of triggered alerts to a noninterruptive display mode. Clinicians accepted the more selective interruptive alerts two-thirds of the time. Acceptance rates differed substantially by alert type. Additionally, we identified override reasons that have the potential to further improve the quality of alerts. Specific lessons learned from our implementation of ambulatory alerts are summarized in Table 5 (available as a JAMIA online supplement at www.jamia.org).

Our study of an ambulatory computerized prescribing alert application has several unique features compared to previously reported studies. In this study, we strove to be highly selective about which alerts to display. This is especially important because clinicians may override extremely important alerts with adverse clinical consequences if they are confronted with too many interruptive alerts. In addition, previous studies have only evaluated a binary outcome of canceling the order or overriding the alert and have not evaluated whether the alert modified other clinician actions.^10,11 In our study, we examined in detail why clinicians continued with an alerted prescription and what actions they took as a consequence of the alert. In many instances, although the clinician continued ordering an alerted medication, he or she also eliminated the potential contraindication (facilitated by the CDSS) by discontinuing the preexisting medication or removing an inaccurate diagnosis. Other times, although the contraindication persisted, the alert achieved its intended effect by altering clinician behavior (i.e., ordering extra monitoring). Thus, even when a clinician continued ordering an alerted prescription, the alert may have appropriately modified subsequent actions, which is important to assess when fully evaluating the impact of a CDSS.

A key difference between this study and the Weingart et al.¹¹ study, in which ambulatory physicians accepted only 11% of high-severity drug interaction alerts, is the knowledge base used to generate the alerts. Whereas the Weingart et al. study used an inclusive commercial knowledge base, we started with a commercial knowledge base, but then modified it to create a subset of only the most clinically relevant contraindications. Furthermore, by categorizing the alerts into severity tiers, we were able to only interrupt clinicians for contraindications with the highest clinical severity. We believe the high clinician acceptance of our alerts was achieved by presenting the clinicians with fewer but more meaningful alerts. The question still remains as to where the optimal specificity for alerts lies. It is certainly possible that our system now underalerts and misses some important alerts, but we felt it was initially more important to maximize alert acceptance and gain clinician confidence in the system. More research is needed to find the optimal balance between over- and underalerting.

One of the common concerns among clinicians is that drug alert systems will generate inappropriate alerts.^16,17 In the study by Weingart et al.,¹¹ physician reviewers judged one-third of generated alerts to be inappropriate. Similarly, our previous experience with drug-allergy alerts highlighted that the vast majority of allergy alert overrides were clinically appropriate and did not lead to adverse drug events.¹⁸ Presenting clinicians with inaccurate alerts can erode their faith in the system and make it more likely for them to ignore subsequent alerts.^15,19,20,21 We did find some override reasons suggesting inappropriate alerts. For example, there were cases where the clinician stated new evidence existed for use of the medication despite the displayed contraindication. Additionally, we found instances in which the clinician indicated a disease identified for the drug-disease alerts was incorrectly entered on a problem list, or more recent labs were available than the ones used to generate the drug-lab alerts. The problem of incomplete information was especially prominent with the Level 2 drug-pregnancy alerts in which virtually all the override reasons stated the patient was not pregnant. It is often difficult for the computer to verify a patient's pregnancy status based on laboratory values alone, since usually there is not a repeat test performed after a miscarriage or delivery. Since there is no repeat test, the most recent positive pregnancy test triggers an alert. This emphasizes the need for better tracking of pregnancy status in clinical systems. More broadly, these findings of inaccurate alerts underscore the need to keep drug alert knowledge bases up-to-date with current clinical literature, the need to maintain accurate clinical documentation within an electronic medical record, and the need to ensure optimal linkage of CDSS to all clinical data repositories of patient information. These measures can help achieve more credible alerts.

The override reasons we captured suggest that clinicians often deviate from recommendations for good clinical reasons and we believe this information is worth capturing for subsequent evaluation and revision of alerts. Clinician override reasons should be validated before changes to an alert knowledge base are made. We did find occasions when clinicians overrode the alerts without providing a reason. While these instances were a lost opportunity to understand the clinician's reason for override, we recognize that these omissions may occasionally be necessary for clinical expediency. Given the potential value of this information for future alert improvement, CDSS should be designed to most effortlessly capture the reason for clinician overrides and minimize omissions of override reasons.

This study had several limitations. Although we were able to collect a large number of drug alerts, there were several categories of alerts, including drug-disease, for which there were low numbers of alerts. However, this is because many of the alerts are for rare issues; we still were able to identify patterns of clinician behavior. Also, we were unable to evaluate whether the noninterruptive alerts were meaningful to clinicians or why certain alert categories were more highly accepted than others. These are both important areas for future research. Another limitation is that our study was performed within a single health care system using one outpatient prescribing system. Thus, it is possible that our results reflect certain features particular to our system and may not be generalizable. However most computerized prescribing systems have alerts, and the goal of this study is to influence future design of alerts by demonstrating the benefit of using selective and tiered knowledge bases. Although we created an internal review process to determine relevance and level assignment of drug contraindications, there is currently no gold standard for this process. In the future, creation of a validated approach would be valuable. We do acknowledge that the creation of a customized knowledge base requires substantial institutional time and resources that may be limited in many organizations. A possible solution could be a national repository of knowledge information that would gather this information for public sharing with vendors and local organizations.

Conclusion

The lessons learned from the results of our implementation provide valuable information for subsequent design efforts. By interrupting clinicians for only those drug contraindications with the highest clinical importance, it is possible to achieve fewer interruptive alerts and high clinician acceptance of alerts, especially when taking into account subsequent clinician actions that eliminate the drug contraindication. Additionally, systems should strive to reduce inaccurate alerts by maintaining accurate clinical documentation and improving the linkage of patient information from all clinical data repositories. Furthermore, it is useful to collect the reasons clinicians override alerts in order to understand what impact the alert had on their subsequent actions and to aid future CDSS design. Finally, further research is needed to determine the best balance between under- and overalerting, as well as the clinical impact of these systems.