Turning Off Frequently Overridden Drug Alerts: Limited Opportunities for Doing It Safely

Abstract

Objectives

This study sought to identify opportunities to safely turn off frequently overridden drug–drug interaction alerts (DDIs) in computerized physician order entry (CPOE).

Design

Quantitative retrospective analysis of drug safety alerts overridden during 1 month and qualitative interviews with 24 respondents (18 physicians and 6 pharmacists) about turning off frequently overridden DDI alerts, based on the Dutch drug database, in a hospital setting. Screen shots and complete texts of frequently overridden DDIs were presented to physicians of internal medicine, cardiology, and surgery and to hospital pharmacists who were asked whether these could be turned off hospital-wide without impairing patient safety, and the reasons for their recommendations.

Results

Data on the frequency of alerts overridden in 1 month identified 3,089 overrides, of which 1,963 were DDIs. The category DDIs showed 86 different alerts, of which 24 frequently overridden alerts, accounting for 72% of all DDI overrides, were selected for further evaluation. The 24 respondents together made 576 assessments. Upon investigation, differences in the reasons for turning off alerts were found across medical specialties and among respondents within a specialty. Frequently mentioned reasons for turning off were “alert well known,” “alert not serious,” or “alert not needing (additional) action,” or that the effects of the combination were monitored or intended. For none of the alerts did all respondents agree that it could be safely turned off hospital-wide. The highest agreement was 13 of 24 respondents (54%). A positive correlation was found between the number of alerts overridden and the number of clinicians recommending to turn them off.

Conclusion

Although the Dutch drug database is already a selected reduction from all DDIs mentioned in literature, the majority of respondents wanted to turn off DDI alerts to reduce alert overload. Turning off DDI alerts hospital-wide appeared to be problematic because of differences among physicians regarding drug-related knowledge and of differences across the hospital in routine drug monitoring practices. Furthermore, several reasons for suppression of alerts could be questioned from a safety perspective. Further research should investigate when each of the following might help: changes in alert texts; new differential alert triggers based on clinician knowledge or specialty; and nonintrusive alert presentation so long as serum levels and patient parameters are measured and stay within limits.

Introduction

Computerized physician order entry (CPOE) systems frequently include integrated decision support components. The generation of medication-related alerts depends on whether information (on drug–drug interactions or dose levels) is present in the CPOE system's knowledge base and whether the system can use this information (alerting features). Drug knowledge bases are often overly inclusive, with alerts for every potentially dangerous situation mentioned in the literature. ^1–4 An overly inclusive database may make CPOE systems generate excessive numbers of drug safety alerts, causing clinicians to ignore even important alerts and to override them, potentially impairing patient safety. ^3,5 The most important reason listed by physicians for overriding alerts is alert fatigue, which often occurs because some alerts do not relate to serious outcomes, because many alerts are irrelevant, and because a given alert may appear repeatedly. To reduce alert fatigue and to improve patient safety, irrelevant and nonurgent alerts should be suppressed or displayed in a noninterruptive manner. ⁵ However, turning off alerts can also impair patient safety if performed without careful error management. ^5,6 This study attempted to identify situations in which frequently overridden drug alerts within a CPOE system might potentially be suppressed in some manner, while at the same time maintaining safety.

Research questions included:

• 1 What reasons do hospital clinicians give when they are asked whether drug safety alerts can be safely turned off hospital-wide?

• 2 Do different specialties differ in their opinions and considerations on this question?

• 3 Do residents and specialists differ in their opinions and considerations regarding turning off drug safety alerts?

• 4 Does the desire to turn off a drug safety alert change if more information about the alert is presented?

• 5 Which frequently overridden drug safety alerts can be safely turned off hospital-wide?

Background

Error management has three components: prevention, visible notification of potential and real errors, and mitigation of the effects of errors. ^7,8 Drug safety alerting systems provide visible notification of potential errors during the order entry process, with the goal of averting such errors. To limit the incidence of potentially dangerous prescribing errors, alerts should be generated in all critical situations; high sensitivity is strived for. Alerting per se does not automatically prevent all critical errors because cognitive overload induced by overactive alerting systems is itself a known cause of errors. ⁹ Alarms that are installed on a “better safe than sorry” basis are likely to make responses to them less rather than more reliable. ¹⁰ High numbers of low-importance and irrelevant alerts are common causes of “alert fatigue.” ⁵ The importance or relevance of an alert is not absolute, but rather situation-dependent. An alert may become irrelevant in a hospital where monitoring of serum drug levels or clinical effect-related patient parameters occurs routinely, whereas it may be relevant for the general practitioner who does not routinely monitor such parameters in outpatient settings.

The current study attempted to identify opportunities to turn off inpatient drug-related alerts safely. Feldstein et al. ¹¹ stated that clinicians should not be able to control the display of safety alerts because those who need alerts the most would turn them off. It seems desirable to consult physicians of different specialties before turning off alerts because this may reveal important considerations for the improvement of computerized decision support systems (CDSS). Another consideration is that uninformed suppression of drug alerts could result in legally actionable negligence claims when harm to patients occurs that might have been prevented. Kuperman et al. ⁴ pleaded for research targeting an improved understanding of how to employ commercial knowledge bases to create CDSS that are well accepted by practicing clinicians.

In their viewpoint article, Miller et al. ¹ argued for a U.S. national standard for drug interaction information that could be locally customized, and included: (1) generic names of interacting drugs, (2) a brief human-readable but computable standard set of descriptions for the clinical nature of the interactions, (3) an indication of the strength of the evidence base for the interaction/effect on a five-category scale, (4) a four-category scale for the seriousness of interaction/effects, and (5) a frequency listing on a logarithmic scale of how often each severity reaction has been reported to occur.

In The Netherlands, such a national drug database exists, although some small differences from Miller et al.'s proposed criteria are discernable. The Dutch seriousness index has six categories (A through F) instead of four, and the evidence index has the same number of categories but ranges from zero to four instead of one to five. ¹² A seriousness index and evidence index are combined in an alphanumeric code. Information on the frequency of adverse events often cannot be presented because of the lack of interaction studies. In the Dutch drug database (also known as “G-standard”) combinations of drugs mentioned in the literature as causing drug–drug interactions (DDIs) are categorized as yes/yes (interacting and requiring action), yes/no (interacting but requiring no action), and no/no (not interacting, requiring no action). ¹² Sixty-four percent of the DDIs were categorized as yes/yes DDI, automatically generating a DDI alert in the Dutch CPOE systems. ¹² DDIs with the label yes/no normally do not generate alerts, but such alerts can optionally be enabled. The national Dutch drug database does contain some additional information desirable for optimizing alert specificity, such as sequence indications that indicate an alert is relevant when new drug A is added to an existing regimen containing drug B, but not if new drug B is added to existing drug A (for example, starting an angiotensin-converting enzyme inhibitor in a patient using diuretics may cause severe hypotension and should be performed with low doses, whereas a patient chronically taking angiotensin-converting enzyme inhibitors can start with diuretics without such precautionary measures). Several CPOE systems lack the ability to use these indications for sequence-dependent alerting.

In the Netherlands, the Royal Dutch Association for the Advancement of Pharmacy generates dedicated alert texts (as well as background information) for general practitioners, community pharmacies, and hospitals. The alert texts consist of information about the potential adverse reaction (e.g., rising serum level, hypotension) and a recommendation for how to address the alerting condition, followed by extra information such as clinical consequences, mechanism, or literature references. Text wording may be modified based on comments from clinical users about the texts.

The investigators hypothesized that alerts with a low level of seriousness or alerts to initiate what is already routinely performed monitoring would generate considerable agreement regarding alert suppression (i.e., turning them off). Furthermore, investigators expected that surgical and nonsurgical specialties would come to different decisions (because of differences in perceived importance of drugs) and that within-specialty differences would be small. Finally, investigators hypothesized that presentation of more information about the alert will result in a decision change in less familiar alerts.

Methods

Setting

The Erasmus Medical Center (Erasmus MC) in Rotterdam, The Netherlands, comprises a 1,237-bed academic medical center consisting of 3 hospitals, a 800-bed general hospital, a pediatric hospital, and an oncology clinic. The current study was performed in the general hospital. In that hospital, a CPOE system for medication ordering was introduced in December 2001. Since March 2005, all inpatient wards excluding the intensive care units have used the CPOE system Medicatie/EVS by iSOFT (Leiden, The Netherlands). ¹³ Physicians and midwives exclusively enter medication orders. At present, nurses are not legally allowed to prescribe drugs and therefore do not enter medication orders via CPOE. The system requires complete orders containing drug name, dosage form, strength, drug dose, frequency, start date, and start time. During order entry, medications can be selected from the pharmacy database listing stock held on the ward and in the hospital pharmacy, or from the national drug database. It is also possible to select preformed, standardized orders from predefined order sets, or to enter free text prescriptions. ¹³ The CPOE system generates intrusive (stopping user workflow) drug safety alerts for DDIs, for overdosages, and for duplicate orders. ▶ shows how an alert is shown to the user: both interacting drugs, their dosage regimens, and an explanation including a recommendation are given.

Figure 1.

Example of a screenshot of a DDI. DDI alert presented to a physician ordering Trisporal (itraconazole; new order) when Prograf (tacrolimus; current order) is already on the patient's medication list. The screenshots presented to the respondent also include a single information sentence put above the alert, which contains the DDI database code, the evidence index, seriousness index, and the number of times the alert has been overridden in 1 month.

The complete alert text can only be read if the user scrolls down to the bottom. In Medicatie/EVS version 2.20, which has been used in this study, alerts can always be overridden without giving a reason. Overridden alerts are routinely logged for pharmacy review. Free text prescriptions do not generate drug safety alerts. The CPOE system cannot use clinical chemistry data or existing patient drug serum levels to either generate or suppress alerts. Medicatie/EVS version 2.20 allows only for hospital-wide turning off alerts. The knowledge base for drug safety alerting in the system makes use of the national “G-standard,” which is updated monthly and can be customized according to local requirements.

Participants

All medical ward coordinators (specialists) in internal medicine and cardiology were included as participants, as well as all registered hospital pharmacists working in the front office of the hospital pharmacy. Other specialists and residents known to be active users of the program and willing to participate were included to create equal sample sizes of six respondents for each specialty (internal medicine, cardiology, surgery) and hospital pharmacy. In total, 18 physicians and 6 pharmacists were recruited for the study, without using financial or other incentives. Although hospital pharmacists do not receive drug safety alerts in real time themselves, but only view overridden alerts, they were included as they are generally responsible for CPOE implementation and drug safety, including checks on overrides and turning off alerts.

Data Collection

The total number of overridden drug safety alerts was analyzed for 1 month (October 2005) in the general hospital of Erasmus Medical Center. DDIs overridden more than 10 times per month were selected for further evaluation. Those DDIs without an alphanumeric code were excluded because seriousness was thought to be an important consideration in specialists' decisions whether to turn off. ¹⁴ The DDIs concerning drug administration time were excluded as well because it was proposed to direct these alerts to other people in the workflow. ¹⁵ As overriding may have different causes and reasons that cannot be detected from quantitative analysis, ⁵ the study conducted qualitative interviews of prescribing physicians. Printed screenshots of the 24 remaining overridden DDI alerts were presented to the respondents, who were asked whether this DDI could be turned off, hospital-wide, without impairing patient safety. They were also asked to provide their reasons for this decision. Override rate, DDI name, database code, and alphanumeric code were also shown to respondents (▶). After they had assessed the DDI alerts, the respondents were then asked the same question again after being presented with the complete alert text (▶). All interviews were conducted by the first author using an interview protocol. Interviews lasted 14 to 43 min and were audiotaped.

Figure 2.

Translated example of a complete text of a DDI. The boxed text can be observed at a glance; the rest of the alert text can only be read if it is scrolled down to the bottom.

Analysis

Interviews were transcribed verbatim and analyzed. The number of alerts that respondents recommended to be turned off hospital-wide were counted and related to specialty, job status, and alert type. The number of decision changes due to the presentation of the complete alert text was calculated.

Every recommendation to turn off an alert was coded manually with one or two relevant keywords representing the main reason for the respondent's opinion (▶). These reasons were derived from items used for the classification of DDIs in the Dutch drug database, ¹² and referred to in literature. ⁵ To this list were added themes emerging from the interviews. Reasons for turning off an alert were analyzed as a whole, by drug safety alert, by specialty, and by function of the person recommending the action. Statistical tests were performed using SPSS version 15 (SPSS Inc., Chicago, IL). Correlation analysis was used to examine the strength and direction of linear relationships between variables. Spearman's rank order correlation (rho) was used as a nonparametric test to calculate the strength of the relationship.

Table 1.

Table 1 Response Analysis: Reasons and Number of Times Mentioned

Reason	Explanation of Reason	Number of Times Reasons Mentioned by Respondents
		Per Specialty				Overall	Resulting in Turning On or Off
		Internal Medicine	Cardiology	Surgery	Hospital Pharmacy		On	Off
Seriousness 12	DDI is mentioned to be serious, not serious, clinically relevant, or irrelevant; the letter of the alphanumeric code is mentioned.	39	30	20	67	156	115	41
Evidence 12	The evidence of the DDI is mentioned or the number of the alphanumeric code.	7	0	1	4	12	8	4
Risk patients 12	Risk factors making the DDI relevant are mentioned.	3	4	0	1	8	8	0
Incidence 12	The incidence of adverse events due to the DDI is mentioned.	8	6	3	0	17	5	12
No action 12	The respondents mention the alert does not need any action or that they never perform any action.	7	9	11	6	33	0	33
Text 12	The information in the alert text is mentioned (recommendations to adjust doses, to measure serum levels, monitor patient parameters, or prescribe alternative drugs).	15	24	29	27	95	95	0
Number 5	The quantity or number of alerts (generated or overridden) is mentioned.	3	5	4	10	22	15	7
Knowledge 5	The fact that the alert is known or unknown is mentioned.	72	40	27	20	161	104	57
Specialty 5	The fact that only specialists are prescribing a specific drug or the combination of drugs is mentioned.	1	11	8	4	24	0	24
Urgency 5	The rapidity of the adverse effect is mentioned.	3	2	0	8	13	4	9
Monitoring 5	The fact that effects are monitored or serum levels measured is mentioned.	7	12	6	22	47	0	47
Intentional	The fact that the drugs are intentionally combined because of a desired effect of the DDI or the fact that they are generally combined for other reasons is mentioned.	11	9	1	17	38	0	38
Hospital-wide	The fact that the drugs are prescribed by two or more different specialties is mentioned.	2	14	0	9	25	25	0

Each alert assessment resulted in one to two main reasons per respondent for the opinion whether to turn off.

Results

A total of 3,089 drug safety alert overrides occurred in October 2005. This comprised 1,963 DDIs (64%), 684 overdosage alerts (22%), and 442 duplicate order alerts (14%). In the DDI category, a total of 86 different individual alerts were overridden. Of those, 32 alerts occurred more than 10 times. Eight alerts were excluded because their relevance had not been assessed completely, and they had not yet been assigned an alphanumeric code, or because the interaction referred to administration time. The study used the remaining 24 individual alerts for its assessments (▶). These alerts accounted for 72% of all overridden DDI alerts (1,413 of 1,963). High-level alerts (E/F 25%), medium-level alerts (C/D 54%) and low-level alerts (A/B 21%) were present.

Table 2.

Table 2 Alerts, Number of Respondents Agreeing to Turn Off the Alert, and Reason Most Often Mentioned

DDI Database Code	DDI Name	Seriousness Index	Evidence Index	Number of Alert Overrides in 1 Month	Number of Clinicians Agreeing to Turn Off Alerts					Reason Most Often Mentioned
					Int (n = 6)	Card (n = 6)	Surg (n = 6)	Pharm (n = 6)	Total (n = 24)
1066	Potassium and potassium-saving diuretics	F	3	41	4	1	0	6	11 (46%)	Knowledge (7)
0035	ACE inhibitors and potassium-saving diuretics	F	2	112	1	2	0	6	9 (38%)	Knowledge (10)
0280	Beta-blockers and verapamil/diltiazem	E	3	12	1	2	1	3	7 (29%)	Knowledge (6) Intentional (6)
5096	QT interval prolonging drugs erythromycin/clarithromycin/voriconazole	E	3	10	1	0	0	0	1 (4%)	Seriousness (11)
3395	Statins (simvastatin/atorvastatin) and verapamil/diltiazem	E	3	10	0	1	0	0	1 (4%)	Text (9)
5088	QT interval prolonging drugs and QT interval prolonging drugs (except erythromycin, clarithromycin, voriconazole)	E	1	154	0	0	0	0	0 (0%)	Seriousness (12)
0531	Coumarins and amiodarone/propafenone	D	3	47	3	4	0	5	12 (50%)	Monitoring (12)
0299	Nonselective beta-blockers and insulin	D	3	18	3	4	0	4	11 (46%)	Knowledge (7)
0019	ACE inhibitors and diuretics	D	3	312	2	2	0	3	7 (29%)	Knowledge (6) Intentional (6)
3921	Haloperidol and enzyme inductors	D	3	14	1	0	1	3	5 (21%)	Knowledge (7)
1155	Diuretics and NSAIDs	D	3	41	2	0	0	2	4 (17%)	Seriousness (14)
0027	ACE inhibitors and NSAIDs	D	3	30	1	0	0	2	3 (13%)	Seriousness (14)
0124	Digoxin and amiodarone	D	3	16	0	1	0	1	2 (8%)	Seriousness (9) Knowledge (9)
1465	Tacrolimus and enzyme inhibitors	D	3	27	0	0	0	0	0 (0%)	Seriousness (11)
3360	NSAIDs (except COX-2 inhibitors) and selective 5HT reuptake inhibitors/trazodone	C	4	15	0	0	1	1	2 (8%)	Knowledge (9)
0272	Beta-blockers and NSAIDs	C	3	37	5	3	0	4	12 (50%)	Knowledge (9)
2046	NSAIDs (except COX-2 inhibitors) and corticosteroids	C	3	83	2	1	1	2	6 (25%)	Knowledge (9)
0736	Coumarins and NSAIDs	C	3	29	2	1	0	2	5 (21%)	Knowledge (6)
0310	Nonselective beta-blockers and beta-adrenergic agonists	C	3	12	2	2	0	1	5 (21%)	Text (8)
0302	Selective beta-blockers and insulin	B	3	160	3	4	0	6	13 (54%)	Knowledge (9)
3964	Beta-blockers and oral hypoglycemic drugs	B	3	57	3	3	1	4	11 (46%)	Knowledge (8)
1228	AT receptor antagonists and diuretics	B	3	87	2	3	0	3	8 (33%)	Knowledge (5) Intentional (5)
0078	Alpha-blocking drugs (for benign prostate hyperplasia) and beta-blockers/calcium channel blockers	B	3	78	1	0	0	4	5 (21%)	Knowledge (8)
0345	Calcium channel blockers and CYP3A4 inhibitors	B	3	11	3	0	0	2	5 (21%)	Knowledge (10)
Total				1,413	42	34	5	64	145

The DDIs are ordered according to the seriousness index. () = number of respondents mentioning this reason.

5HT = serotonin; ACE = angiotensin-converting enzyme; AT = angiotensin; Card = cardiology; COX-2 = cyclooxygenase-2; CYP3A4 = cytochrome P450 3A4; DDI = drug–drug interaction; Int = internal medicine; NSAID = nonsteroidal anti-inflammatory drug; Pharm = hospital pharmacy; Surg = surgery.

The 24 respondents included 4 specialists and 2 residents in internal medicine and cardiology, 4 registered hospital pharmacists, and 2 residents in hospital pharmacy. For surgery, only 2 specialists were available for interviews because attending surgeons never prescribe on inpatient wards and only supervise residents. Therefore, 2 specialists and 4 (final year) residents were recruited. The 24 respondents together made 576 assessments.

Quantitative Analysis of Recommendations to Turn Off Alerts

The number of respondents agreeing that a given alert could be turned off hospital-wide is presented in ▶. There were no alerts that all clinicians agreed could be turned off safely. However, a significant positive correlation of 0.44 (α = 0.05) was found between the number of overridden alerts and the number of physicians recommending alerts to be turned off. This suggests that an increase of alert overrides increases the number of physicians advising to turn off alerts. No correlation was found between the level of seriousness and the number of respondents agreeing to turn off alerts hospital-wide. For three alerts, at least 50% of the 24 respondents recommended suppression hospital-wide.

Six clinicians (four for surgery and two for cardiology) did not want to turn off any alerts. Specialists agreed four alerts should not be turned off hospital-wide, whereas several residents thought these could be turned off safely. Hospital pharmacists always made their decisions for the entire hospital, but many physicians reported that they could not make this decision for colleagues outside their specialty. Several decisions to turn off alerts could only be made for their own specialty (19%, 12%, and 94% of the decisions of internists, cardiologists, and surgeons, respectively), whereas others were made for the entire hospital (81%, 88%, 6%, respectively).

Specialties differed in the number of alerts they thought could be turned off hospital-wide. Internal medicine recommended more alerts be turned off than cardiology and cardiology more than surgery. Internists agreed on turning off four alerts for their own specialty, which would result in a mean reduction of overridden alerts of 19% for their specialty. The residents asked for turning off for their specialty more often than the specialists. Eight residents made 83 requests for turning off, whereas 10 specialists asked 76 times.

Five times, respondents could not make a decision whether to turn off an alert with the limited information on the printed screenshot, but they were able to do so with the complete text presented in the second part of the study. The request to turn off an alert changed in 14 assessments after presentation of the complete text (2.4% of total). Hospital pharmacists changed their opinion more often (7 times) than internists (3 times), cardiologists (0), and surgeons (4). In 63 assessments (11%), respondents spontaneously commented negatively on the length, content, and sequence of the complete text presented.

Qualitative Analysis of Recommendations to Turn Off Alerts

Qualitative analysis showed that at the beginning of their interviews, three respondents mentioned a reason for not turning off any alerts hospital-wide. A cardiologist said that the question about hospital-wide turning off was useless and bad because residents early in their training do not have the appropriate knowledge. The surgeons said that the drug and DDI knowledge of residents and specialists in surgery was too low and therefore every DDI should be shown. These three respondents were excluded from further qualitative analysis of reasons because they did not mention reasons for single alerts. One internist was reserved about alert suppression and favored frequent alerting, saying: “I prefer having a bit too many alerts than too few.”

Reasons for suppression of alerts and the number of times they were mentioned are presented in ▶. Two new themes emerged in the interviews. The first theme was about drugs that are combined intentionally by the same specialist because the effect of the DDI is advantageous in a specific patient group, whereas the combination might cause harm in others (e.g., intended bradycardia due to beta-blockers combined with verapamil or diltiazem, prescribed by a cardiologist). The second theme that emerged was a combination of drugs that are generally prescribed individually within each of two or more specialties (e.g., alpha-blockers for benign prostate hyperplasia by urology and beta-blockers by internal medicine) that might cause problems if the alert is suppressed. The internist may not focus on the possible harmful effects of combination of a known beta-blocker with a rather unknown alpha-blocking drug the internist never prescribes.

“Knowledge” and “seriousness” were the most frequently mentioned reasons for not turning off alerts, followed by “text.” Reasons used for the classification and presentation of DDIs in the “G-standard” (seriousness, evidence, risk factors, incidence, action, text) were mentioned about as often as more context-specific reasons (knowledge, intentional, monitoring). Risk factors, incidence, and evidence were not mentioned very often, nor were the number of alerts.

Thirty-three times (6%) respondents mentioned that alerts were not acted upon or did not need any action, whereas all alerts were categorized as yes/yes interactions in the Dutch drug database.

Results Viewed per Alert

Respondents rated four alerts that could result in increased risk of torsades de pointes, myopathy, and nephrotoxicity as unknown and serious, with adverse effect preventable by following the recommendation given, and recommended unanimously that these 4 alerts should not be turned off. The interaction alerts due to QT-interval prolongation and liver enzyme inhibition were not directly deducible from the pharmacological group of drugs and therefore perceived as largely unknown and useful.

At least 50% of the respondents stated that three specific alerts could be suppressed hospital-wide because related effects were either monitored regularly by measuring the international normalized ratio, well known and not serious, or irrelevant (in the case of short-term treatment with nonsteroidal anti-inflammatory drugs). Respondents very often characterized frequently overridden sequence-dependent alerts as false positives, for example, when a diuretic was added to therapy with an angiotensin-converting enzyme inhibitor or to an angiotensin receptor antagonist. Internists (and cardiologists also) frequently prescribed such combinations, were aware of these alerts, and asked to turn them off. The drug combination of potassium-saving diuretics with potassium was said to be known, intentional, and always based on low potassium levels and therefore not useful. The low-level B-alerts were perceived as serious eight times (7%). On only one occasion was a high-level E-alert described as not serious.

Results Viewed per Specialty

Surgeons gave a lower average number of different reasons for alert suppression or retention (4.5) than did hospital pharmacists (8), internists (7.3), and cardiologists (8). Surgeons relatively often mentioned “text” as a reason not to turn off and “no action” as a reason to turn off alerts. Hospital pharmacists relatively often mentioned “seriousness,” “number,” “monitoring,” and “intentional,” whereas “knowledge” was hardly considered. Internists very often mentioned “knowledge” as a reason for suppression, whereas cardiologists referred more to “specialty-specific” prescribing. Physicians generally included their own experience in their considerations. Internists and cardiologists often asked to turn off the DDIs having to do with the serum potassium level because these levels are measured routinely for inpatients. Surgeons admitted they do not regularly measure these levels and prefer these high-level F-alerts to be shown.

Results Viewed by Job Status (Residents versus Specialists)

The number of alerts recommended to be suppressed per respondent was higher for residents than for specialists; this difference was highest for surgeons. Residents more often mentioned “no action,” “only prescribed by specialists,” and “low incidence” as reasons for alert suppression than did specialists. One surgical resident did not understand the text of the sequence-dependent alerts well, considered the administration of drugs was out of the control of physicians, and thought these alerts therefore irrelevant. However, the alert was about problems arising when the patient had previously been using a diuretic for a while.

Discussion

This study attempted to identify opportunities to safely turn off (suppress) drug alerts hospital-wide. Nevertheless, the respondents rating alerts across specialties as well as within one specialty differed substantially in their recommendations and reasons for suppression of drug safety alerts, even when only medical specialties were taken into account. The same sorts of differences occurred for residents and specialists. Opinions on whether to suppress alerts changed minimally when more information was presented. Hospital-wide suppression was deemed not feasible.

Unexpected Study Results

The study results surprised the investigators in several regards. First, one-quarter of respondents (all physicians) recommended not turning off any alerts hospital-wide, either because the alerts did not bother them, or because they feared that a perceived lack of knowledge among residents and surgeons would lead to errors that alerts could prevent. This surprised the investigators because a major motivation for the study was the high frequency of physicians' complaints about DDI alert overload prior to the study. Differences in preferences and reasoning were observed between as well as within specialties. These results suggest to the investigators that alert presentation might improve if it is customized to specialty and job status. This is in line with previous recommendations in literature. ¹⁴

Second, no positive correlation could be observed between the nationally determined level of DDI seriousness ratings and the number of respondents stating that the alert should be suppressed. Seriousness was very often mentioned as an important consideration in the decision about whether to suppress an alert, but several times the respondents' perceived seriousness did not correspond with the national seriousness index. Physicians may perceive alerts as not serious because frequent monitoring in the hospital setting provides direct feedback about whether harm is imminent. However, the ability to monitor serum levels or patient parameters was only mentioned 47 times (8%) as an important reason to suppress an alert, whereas the majority of the alerts have effects that can be assessed by measuring serum levels, heart rate and rhythm, or blood pressure.

Third, respondents cited the number of alerts being overridden only 22 times (4%) as an important consideration for alert suppression, and the literature supports that other factors are more important for the perceived usefulness of alerts. ¹⁴ However, the current study observed a positive correlation between the number of overrides for a given alert and the number of physicians recommending that the specific alert be turned off. A possible explanation for this correlation is that frequently shown alerts resulted in a learning effect ¹⁶ and were characterized by respondents as “alert well known” instead of citing the number of alerts overridden or generated.

Fourth, presentation of the complete texts rarely resulted in opinion changes, but spontaneously prompted negative comments on text content, sequence, and length. It is said that drug safety alerts should not be lengthy, but clear and concise to be helpful, with links to supporting evidence. ¹¹ Users do not review alert texts prior to inclusion in the Dutch drug database, although their comments are welcomed and sometimes acted on. The investigators advise having clinician-users review potential DDI alert texts prior to introducing them into practice.

Error Management

Several times, physicians did not rate seriousness of an alert correctly (i.e., according to the national categorization). They rated some alerts as not serious (and thus candidates for suppression) when they had never seen the adverse reactions, or when the physicians generally had not taken any actions upon presentation. Furthermore they did not consider risk factors. These results suggest that physicians cannot always envisage all potential adverse events of drug combinations and that structural assessments, as well as better education of the users about the alphanumeric codes, risk factors, and DDI incidence rates would probably help.

This study shows that many physicians used considerations that are questionable from a safety perspective, like “effects intended,” “only prescribed by specialists,” “no action needed,” or “alert well known,” and that they hardly considered risk factors. When a drug combination is intentionally prescribed by physicians in one specialty, it does not imply that other specialties will prescribe it safely (or that they will never prescribe it), so such alerts cannot be safely turned off hospital-wide. Many residents mentioned they would never act on an alert (“no action”) and it could therefore be turned off. This reason is given far less frequently by specialists and is problematic from a safety perspective, but is in line with the observation that those who need alerts most would turn them off. ¹¹ Lack of clinician end-user knowledge can be a good reason not to turn off alerts. It is questionable as a reason for knowledge-specific or specialty-specific alert suppression, ¹⁴ because a recent British study indicated that 57% of prescribing errors were due to incorrectly executing an appropriate plan, because clinicians were busy, or had been interrupted during routine tasks. ^17,18 Lack of attention, distraction, and forgetfulness, rather than a lack of knowledge, have been cited as frequent causes of errors. ¹⁹ Therefore, even turning off alerts for experts would carry some safety concerns.

Alerts in Medicatie/EVS appear as pop-ups. ¹³ Literature suggests that these intrusive alerts should only be used for the most severe clinical indications, ²⁰ when the situation requires remedial action before the prescription becomes complete. ³ Nonintrusive presentations can take the form of sidebars ²¹ or as nonintrusive text messages on the ordering screen. ^3,22 The DDIs in this study all have been categorized as interacting and requiring action (yes/yes DDIs). ¹² It is not clear whether nonintrusive alerts would induce alert fatigue or not, and whether they would result in the preferred action required to prevent adverse events. Further research must occur to assess the cognitive burden of various forms of alerts on the user. For example, it might be the case that when yes/yes DDI alerts whose effects can be measured via serum levels are turned off, or are shown nonintrusively, the CPOE system should also incorporate clinical rules that not only check if serum levels are within the therapeutic window, but also if these levels have been ordered and measured.

Strengths and Weaknesses of the Study

The current study had several unique features. Whereas other studies have focused on turning off alerts of a commercial knowledge base after iterative consensus based discussions by an expert panel, ^2,3 this study took the consensus based knowledge base ¹² as a starting point for further customization. Other studies analyzed override reasons for specific patients, ^3,6,23 whereas this study focused on considerations for hospital-wide alert suppression, and included assessment of perceived usefulness of alerts for physicians. The qualitative part of the study design helped to identify new reasons for alert suppression, and initiated a dialogue about unclear answers. The interviews in this study were performed one-to-one to prevent individual opinions changing under the influence of a group of respondents. The study included six experts per specialty and revealed within-specialty differences in recommendations and reasons. By contrast, expert panels generally include a smaller number of experts per specialty. This study revealed many unexpected results and gave an insight into what direction future research on alert suppression might follow; specifically, investigation of safe mechanisms for specialty-specific or knowledge-specific alert suppression and investigation of how to optimally word the alert text.

The current study also had limitations. The study examined only 24 individual drug–drug interaction alerts, and sought opinions from only three medical subspecialties. Including the pharmacists, the study obtained 576 person-alert assessments. The alerts accounted for the majority of overrides in an October 2005 sample (24 alerts accounted for 72% of all overrides and 60% of the alerts were overridden by the 3 specialties). Overriding was nearly equally common among the three medical specialties: 22%, 17%, and 20% for internal medicine, cardiology, and surgery respondents, respectively. In a March 2006 follow-up, the selected alerts and medical specialties still accounted for 67% and 58 % of the overrides, respectively. The percentage of overridden DDI alerts compared to the total number of overridden alerts was relatively constant (64% and 61% for October 2005 and March 2006, respectively). The majority of the 24 selected alerts (58%) are also frequently encountered in Dutch community pharmacies. ²⁴

More specialists than residents were included in this study because in our opinion the responsibility for turning off alerts for the entire hospital cannot put on the shoulders of residents; specialists should make this decision. However, residents prescribe more, are more likely to suffer alert fatigue, and perhaps are therefore more willing to turn off alerts than specialists. To eliminate a learning effect as much as possible, only final-year residents were included in this study. ²⁵

Only respondents willing to participate were recruited, which may have resulted in selection bias. The respondents, however, represented a large variety of opinions and arguments. It is therefore unlikely that inclusion of other respondents would have resulted in different conclusions.

Conclusions

Overly inclusive drug databases for CPOE drug safety alerting can cause alert fatigue and can impair patient safety. Turning off (suppressing) alerts is a potential mechanism to reduce alert fatigue, and may be safe for alerts irrelevant in certain specific clinical contexts. The drug database used in Dutch CPOEs was not overly inclusive, but investigators observed before, during, and after the study that many physicians complained about too many alerts and asked for selective suppression of alerts.

This study attempted to identify opportunities to turn off DDIs hospital-wide safely, but the results suggest that this may not be feasible. None of the study participants unanimously agreed that hospital-wide suppression of a specific alert could occur safely. Within one hospital, knowledge about drug–drug interactions and their sequelae and routine monitoring practices differed considerably across specialties, and also between specialists and residents. These observations suggest that alert suppression might be studied and implemented in a specialty-specific or knowledge-specific manner. Furthermore, in their recommendations to turn off DDI alerts, respondents frequently cited reasons that are questionable from a safety perspective. The national seriousness index for an alert and the number of clinicians recommending its suppression were not correlated. In contrast, the study found a positive correlation between the number of alerts overridden and the number of clinicians recommending the suppression of the alert. The latter finding should be examined in a larger-scale study.

The investigators concluded that hospital-wide DDI alert suppression is not feasible. Future research should examine the potential effectiveness of sequence-specific DDI alerting, of methods to optimize alert texts, approaches for knowledge-specific and specialty-specific alert suppression (or alternatively, using nonintrusive alert presentation), and methods to provide safety during alert suppression, such as implementing concomitant clinical rules that check whether serum levels or patient parameters are indeed measured and stay within limits.