Tiering Drug–Drug Interaction Alerts by Severity Increases Compliance Rates

Abstract

Objective

Few data exist measuring the effect of differentiating drug–drug interaction (DDI) alerts in computerized provider order entry systems (CPOE) by level of severity (“tiering”). We sought to determine if rates of provider compliance with DDI alerts in the inpatient setting differed when a tiered presentation was implemented.

Design

We performed a retrospective analysis of alert log data on hospitalized patients at two academic medical centers during the period from 2/1/2004 through 2/1/2005. Both inpatient CPOE systems used the same DDI checking service, but one displayed alerts differentially by severity level (tiered presentation, including hard stops for the most severe alerts) while the other did not. Participants were adult inpatients who generated a DDI alert, and providers who wrote the orders. Alerts were presented during the order entry process, providing the clinician with the opportunity to change the patient's medication orders to avoid the interaction.

Measurements

Rate of compliance to alerts at a tiered site compared to a non-tiered site.

Results

We reviewed 71,350 alerts, of which 39,474 occurred at the non-tiered site and 31,876 at the tiered site. Compliance with DDI alerts was significantly higher at the site with tiered DDI alerts compared to the non-tiered site (29% vs. 10%, p < 0.001). At the tiered site, 100% of the most severe alerts were accepted, vs. only 34% at the non-tiered site; moderately severe alerts were also more likely to be accepted at the tiered site (29% vs. 10%).

Conclusion

Tiered alerting by severity was associated with higher compliance rates of DDI alerts in the inpatient setting, and lack of tiering was associated with a high override rate of more severe alerts.

Introduction

Since the introduction of point of care decision support within computerized provider order entry (CPOE) systems, drug–drug interactions (DDIs) have been among the most frequent alerts presented to clinicians writing medication orders. Presentation of any warning if given too frequently can result in what has been termed “alert fatigue,” and concern has risen that this causes clinicians to override or ignore clinically important warnings, or even rebel against decision support altogether. 1,2,3,4

Background

In an effort to reduce over-alerting, Partners Health Care (PHS) redesigned its DDI alerting modules so that only the most serious DDIs would require a response by the clinician, with less serious ones presented in a non-interruptive fashion, i.e., as information only. We have referred to this presentation process as “tiering,” and hypothesized that tiering would improve the compliance rate of alerts. A provider is considered to comply with or “accept” an alert when he or she selects the action recommended on the alert screen. Not selecting a recommended action is termed an “override.” In one study, we examined general effects of tiered alerting in ambulatory prescribing, ¹ which showed a compliance rate of almost two-thirds, which is much higher than other previously published studies. ^2,3 However, a limitation of this study was the lack of a control group with which to compare compliance rates.

We sought to compare compliance rates for DDI alerts in our two academic medical centers based on the manner of presentation of alerts and controlling for differences in ordering practice and workflow between the institutions. The hospitals share a common DDI knowledge base, but present alerts differently. Anecdotal experience suggests that how alerts are presented can have a major impact on compliance rates, but there are few studies comparing how alerts are presented. ⁵ During the study period at Hospital A, all alerts were interruptive and required action by the clinician in “non-tiered” presentations, whereas at Hospital B they were “tiered” according to level of severity of the interaction. This concurrent use of tiered and non-tiered presentation of DDI alerts allowed comparison of compliance rates in the two settings. This study evaluated only drug–drug interaction alerting, and we did not review other types of drug warnings, such as those for drug allergies. ⁶

Methods

Study Sites

This study examined DDI alerting on orders for adult inpatients at two academic medical centers. Both are acute care, teaching hospitals located in the same city, are part of the same enterprise and share a common adverse drug event reporting system. Hospital A has 735 beds, and Hospital B has 898. Admissions for the year following this study were 45,051 at Hospital A and 46,276 at Hospital B. Hospital B provides pediatric care, which Hospital A does not. Both provide general medical and surgical care, and specialty care, including oncology. A subset of providers travels between both sites.

Partners Health Care instituted DDI checking as a decision support component of the CPOE system at Hospital A in 1996. In the initial version, pairs of interacting drugs were stored in a knowledge base that was developed through review of empirical data, local experience, and expert opinion and compared with a commercial database. ^7,8 Severity levels are assigned by a group of physicians and pharmacists that represents the two institutions, which meets to consider and review the intervention knowledge base on an on-going basis.

When one or more potential interactions were detected on a drug being ordered, an alert screen prompted the clinician to cancel the order. Technical limitations precluded offering the ability to discontinue the pre-existing order at the time. All interactions for an order were presented on a single screen, so that when the alert was accepted and the order canceled, all potential interactions were removed at once. Likewise, overriding an alert meant that all interactions remained. An override reason was required in order to continue with the drug order; a single text box was provided for free text entry. All interactions detected were presented, and all required action by the clinician, regardless of severity of the potential interaction.

In 2003, Partners Health Care implemented an enhanced version of its DDI modules in the CPOE system at Hospital B, and in the outpatient record. The knowledge base was reviewed and revised, using a process which has been described by Shah et al. ¹ Interactions were divided into one of three severity level indicators; the presentation of each took a different form and required separate responses in a process we have described as “tiering.” Level 1 alerts are the most serious, are considered to be life-threatening, and the clinician is required either to cancel the order he or she is writing or discontinue the pre-existing drug order (a “hard stop”) (▶).

Figure 1.

Level 1 Alert, requiring one of the orders to be stopped. Since these alerts are presented at the time a drug is selected for ordering, in this example no order for lsosorbid Dinitrate has yet been written, allowing the order process to be canceled. Sildenafil has previously been ordered for this patient, however, and a discontinue order must be created, which the CPOE system will do when this option is selected.

Level 2 alerts are less serious, but still require action by the clinician in that the clinician is required to discontinue one or the other drug, or to select an override reason. Review of the reasons given for overriding alerts in the earlier system led to the creation of a pick list of frequent reasons. Multiple reasons for overriding may be selected, and the ability to add text not included in the pick list is provided (▶). Level 2 alerts are functionally very similar to the Hospital A alerting system.

Figure 2.

Level 2 Alert with Override Reasons. As with the non-tiered site alerts, the clinician may choose either to keep both orders or stop one of them, and is required to choose a reason for continuing both.

The largest proportion of alerts is in Level 3, which is the least serious (▶). In this DDI module, alerts are presented as information only, and require no action of any kind from the clinician. In an attempt to reduce “noise” from too many alerts, the presentation uses available screen real estate, so that no keystroke is needed. The intent was to have only the most serious DDIs interrupt the clinician, so she or he would be more inclined to consider the alert and accept the recommendation.

Figure 3.

Level 3 Alert. These alerts use available screen real estate and require no action at all, not even clicking an “Ok” button. The figure shows how it fits into the medication ordering screen, with an enlarged version of the alert area itself provided for clarity.

Hospital A utilized the same revised knowledge base of DDI alerts, accepting along with Hospital B the changes provided by PHS, but continued to present all alerts, regardless of severity level, in the same interruptive fashion (▶).

Figure 4.

Non-tiered DDI alert screen. All alerts for the drug being ordered appear on this screen. Scrolling through the “Warnings” box displays individual messages for each alert. The action selected (Cancel or Keep) affects all the alerts shown. Selecting “Keep” opens a text box and requires a reason to be supplied.

Alerts are presented as soon as a drug is selected for ordering, interrupting the order process. Each CPOE system acts on the provider's choice automatically by canceling the current order process or, at the tiered site, discontinuing the previously written order. If the provider re-orders one of the offending drugs while the other is still on the patient's medication list, the alert will be presented again as soon as it is selected.

Data

Data were gathered from the intervention logs at each site. These logs record the date and time of the alert, the DDI pair, the severity level, the action selected by the clinician, and override reasons if provided. Demographic data about providers and patients were obtained from our Enterprise Master Patient Index (EMPI). The vast majority of orders in the inpatient setting at both sites were written by resident physicians.

From these data, we created a database of alerts occurring at both sites for the one-year period from 2/1/2004 through 2/1/2005. We considered all drug pairs that generated alerts at the sites, and categorized severity levels according to their assignment at the tiered site. Because one institution cares for pediatric patients and the other does not, only alerts on patients greater than or equal to 18 years were included in the analysis.

To reduce bias from DDIs that fired more frequently than others, we analyzed only the first instance of a DDI alert for any particular patient, drug pair, and hospitalization. Although including all instances may have the effect of changing compliance rates, we sought to minimize the effect of differences in ordering practices between the two hospitals, such as whether alerts re-fire on renewal orders, which we considered a potential confounder to our analysis.

Outcomes of an alert were described as either “Accepted” or “Overridden.” At the non-tiered site, the user can cancel only the order being written, not the previous order. To control for differences in compliance resulting from this, we analyzed the text of override reasons. Those we found that indicated the intent to hold, discontinue, stop, or cancel the drug already ordered, we considered to be “Accepted,” even though the alert action selected was “Override.” This affected approximately 1000 Level 2 alerts.

Data were exported from the production clinical information system using Caché 5.0.8 (InterSystems, Boston, MA) into a Microsoft Access 2003 database, where the additional cleaning of the data described above was accomplished. The final query was exported to Microsoft Excel 2003, and recoded for import into SPSS. Statistical tests were performed using SPSS 15.0 (SPSS Inc., Chicago, IL).

Data included in the final data set were: date, site, severity level, interacting drug pair, action selected, acceptance, reason for override (where applicable), patient age, gender, and race, provider age, gender, and race, warning message presented, and links to alert logs and ordering sessions. Patient and provider information was appropriately de-identified. This study was approved by the Partners Institutional Review Board.

Results

Descriptive statistics for patients and providers are shown in ▶, with no statistically significant differences between the two sites with respect to age or gender. The initial data retrieved from the alert logs included more than 109,000 alerts. After removing multiples and outliers as described in the Methods, the final data set included 71,350 alerts across both sites, with 39,474 (55%) recorded at the non-tiered site and 31,876 (45%) at the tiered site. There were 157 (0.2%) Level 1 alerts (the most serious), 20,282 (28.4%) Level 2 alerts, and 50,931 (71.4%) Level 3 alerts (the least serious). At the non-tiered alert site, there was a lower proportion of Level 1 alerts (0.1% vs. 0.3%), and Level 2 alerts (26.7% vs 30.5%) and a higher proportion of Level 3 alerts (73% vs. 69%, ▶).

Table 1.

Table 1 Provider and Patient Demographics

Demographics	Total	By Site
		Tiered	Non-Tiered
Provider Gender
Female	31,768 (44.5%)	13,735 (43.1%)	18,033 (45.7%)
Male	9,582 (55.5%)	18,141 (56.9%)	21,441 (54.3%)
Provider Age
Mean (StdDev)	31 (6.6)	32 (7.7)	30 (5.4)
Patient Gender
Female	34,300 (48.1%)	14,885 (46.7%)	19,415 (49.2%)
Male	37,050 (51.9%)	16,991 (53.3%)	20,059 (50.8%)
Patient Age	63 (15.6)	64 (15.5)	63 (15.7)
Mean (StdDev)
Alert Totals	71,350	31,876	39,474

Table 2.

Table 2 Alert Severity ^∗

Level	Total Alerts		By Site
			Non-Tiered		Tiered
1	157	0.2%	58	0.1%	99	0.3%
2	20,282	28.4%	10,531	26.7%	9,731	30.5%
3	50,931	71.4%	28,885	73.2%	22,046	69.2%
Overall	71,350		39,474		31,876

^∗ Severity levels are numbered as 1 through 3, ranging from most (1) to least (3) serious.

For Level 1 alerts, 100% were accepted at the tiered site (by design since this was a “hard stop” where the provider had no alternative). However, at the non-tiered site, only 34% of Level 1 alerts were accepted (p < 0.001). Among Level 2 alerts, significantly more alerts were accepted at the tiered site compared with the non-tiered site (29% vs. 10%, p < 0.001, ▶). Level 3 response rates between the sites cannot be directly compared, because by design at the tiered site, no response was collected for Level 3 alerts.

Table 3.

Table 3 Compliance Rates

Level	By Site
	Non-Tiered ∗		Tiered∗∗			Totals
	Accepted/Total (Rate)		Accepted/Total (Rate)		p-values	Accepted/Total (Rate)
1	20/58	(34%)	99/99	(100%)	<0.001	119/157	76%
2	1164/10531	(11%)	2782/9731	(29%)	<0.001	3946/20262	19%
3	2702/28885	(9%)	N/A			2702/28885	9%
Overall	3886/39474	(10%)	2880/9820	(29%)	<0.001	6786/49294	14%

^∗ At the non-tiered site, all alerts both require a response and allow overriding.

^∗∗ At the tiered site, Level 1 alerts require stopping at least one drug, Level 2 alerts require a response when overridden, while Level 3 alerts are display-only and compliance could not be measured.

In addition to examining compliance rates by level, we also analyzed the data from the perspective of the drug pairs, grouping the alerts by drug pair (n = 456) and weighting each pair equally. We wanted to determine whether our findings could be accounted for by differences between the two sites in formularies or prescribing patterns; the results indicate that they cannot. The average compliance rate at the non-tiered site was 15.7% versus 30.9% at the tiered site. Examination of Level 2 compliance by drug pair showed some wide differences in compliance rates between the two sites. For fourteen of these pairs, compliance at the tiered site was 50–77%, whereas at the non-tiered site, it was 25% or less, and in three cases, all alerts were overridden at the non-tiered site (▶). Restricting the data to only those pairs that occurred at both places (n=169) marginally changed only the results for the non-tiered site (increasing compliance from 15.7% to 16.7%). In one example, verapamil-cyclosporine alerts at the non-tiered site were all overridden, whereas at the tiered site, the compliance rate was 60%. A second example, heparin-enoxaparin, also showed a large difference in compliance rates; at the non-tiered site, 22% of 1037 alerts were accepted, while the rate was 70% of 1091 alerts at the tiered site.

Table 4.

Table 4 Level 2 DDIs with Disparate Compliance Rates

Drug Pair	Non-Tiered	Tiered
Cyclosporine & Verapamil	0.0%	60.0%
Indomethacin & Ketorolac Tromethamine	0.0%	59.5%
Theophylline Immediate Release & Levofloxacin	0.0%	55.6%
Magnesium Hydroxide & Minocycline	10.0%	70.0%
Heparin & Drotrecogin Alfa	11.8%	50.0%
Ciprofloxacin & Sucralfate	12.5%	50.0%
Ketorolac Tromethamine & Celecoxib	15.0%	66.7%
Magnesium Hydroxide & Tetracycline	15.4%	76.9%
Sertraline & Linezolid	16.7%	52.6%
Theophylline & Levofloxacin	16.7%	50.0%
Cyclobenzaprine & Tramadol	17.6%	53.8%
Citalopram & Linezolid	21.7%	76.9%
Heparin & Enoxaparin	22.1%	70.4%
Fluoxetine & Epinephrine Continuous Infusion	25.0%	55.0%
Acetylsalicylic Acid & Drotrecogin Alfa	33.3%	60.0%

DDI = drug–drug interaction.

Discussion

We found that tiering the presentation of DDI alerts by severity level was associated with a much higher rate of compliance for interruptive alerts. In particular, we found a difference in compliance for alerts presented in the same way (non hard stop yet interruptive, and requiring an override reason), even though the underlying databases about which interactions to alert on were the same at the two sites. Additionally, we found that about two-thirds of the Level 1 alerts at the non-tiered site were overridden, which by design was not possible at the tiered site. These findings suggest that how alerts are prioritized and presented to the user may be as important as which alerts to deliver. They also suggest that it may be higher risk to present drug–drug alerts in an untiered way, since even very serious alerts often are overridden when this is done. Failure to tier resulted in substantially less recommended provider behavior.

Pairs of DDI drugs that are identified as Level 1 are considered to be high risk and have the potential for causing very serious adverse drug events to occur. Reducing this risk was a primary reason for requiring a hard stop in the tiered alerting process, something not required at the non-tiered site. During the time period we studied, only 34% of Level 1 alerts at the non-tiered site were accepted. All but two of the drug pairs producing Level 1 alerts occurred only once during the year. Although most of the time the patient does well clinically when even these drugs are used concurrently, our experts have judged that they regard this as high enough, and there is a better alternative. Increasing the Level 1 compliance rate from 34% to 100% through tiered alerting that requires a hard stop implies safer care for patients. Clinicians did not object to these hard stops at the tiered site, and they occur very infrequently, so that any individual clinician is unlikely to have had sufficient prior experience with a drug pair to make a reasoned decision without examining the evidence themselves.

Interrupting clinicians only for more serious interactions may make them more receptive to the alerts, and may be the reason why the compliance rate for tiered Level 2 alerts was almost three times higher than for non-tiered Level 2 alerts, despite a relatively similar presentation. A review of Level 2 drug pairs that caused alerts to fire at both sites demonstrated significantly higher compliance rates at the tiered site compared to the non-tiered site for many drugs. Tiering the presentation of DDI alerts almost certainly reduced alert fatigue, since Level 3 alerts (interruptive at the non-tiered site and non-interruptive at the tiered site) represented 71% of the total alerts in our data set for the year studied. At the non-tiered site, clinicians were required to respond to 28,885 Level 3 alerts, of which they accepted only 9%. By removing the need to respond to a similarly high number of Level 3 alerts (22,046) at the tiered site, the overall compliance rate rose significantly, from 10% (non-tiered) to 29% (tiered).

In a recent literature review, Van der Sijs et al. ⁹ identified difficulties in generalizing results due to a lack of standardization of alert levels, and noted that alerts frequently were inappropriate, giving this as a cause for overriding. If sensitivity and specificity of an alert are important factors in determining whether alerts are overridden, then it is reasonable to believe that they should affect the decision of whether or not to alert, and how. Other recommendations for criteria to use to decide drug interaction alerts are level of evidence, relevance, risk factors, and incidence of adverse reactions. ^10,11 Other influences on compliance rates include provider type, ² timing of alert presentation within the order process (renewals versus initial order, for example), ^2,3 and the appropriateness of alerting (e.g., for topical drugs). ^2,3,12 Many of the ideas presented in the literature we reviewed have been put into place at our sites. Topical drugs are not included in our alert knowledge base, and the tiering itself represents an effort to reduce alert fatigue by only requiring action on alerts that are clinically relevant. Determination of what interactions are clinically relevant is done by a committee of physicians and pharmacists that represents both sites, in a process described by Shah. ¹

Compliance rates for DDIs at Partners' ambulatory clinics, reported by Shah et al., ¹ differ from those in our study. Overall compliance with Level 2 alerts at the tiered site in our study was 29%, considerably less than the 41% reported by Shah for drug–drug interactions. The same knowledge base, checking service, and tiered alert presentation supports both the ambulatory clinics studied by Shah and the tiered inpatient site in our study. Proportions of alerts at each level were similar, with slightly more Level 2 alerts in the inpatient study (30.5%) than in the ambulatory one (23%). The sample sizes were considerably different; more than three times as many DDI alerts occurred in a 6 month period in the inpatient setting as compared to the ambulatory. An open question which deserves additional evaluation is what the “ideal” acceptance rate is.

We compared the two studies to determine why our respective compliance rates were different. Ambulatory sites included the option, “modify order” in the selection list of reasons for overriding the alert, and Shah included selection of this option in her definition of “accepted” alerts. The inpatient site did not offer this choice, and we limited our definition of compliance to canceling the new order or discontinuing the existing one. Differences in workflow between the inpatient and outpatient settings, the ability to monitor the patient closely while he/she was taking both drugs, and experience level of the clinicians writing the orders, may also have had an effect, but were beyond our scope to review.

Little published evidence is available regarding tiering and its effects on the problem of over-alerting. Based in part on these results, we plan to institute tiering in the non-tiered site as soon as it is technically possible. However, many current commercial applications do not use tiering. These data—although observational—strongly suggest that this should be reconsidered.

This study had several additional limitations. For example, there was significant heterogeneity in the frequency of particular drug pairs among the alerts at both sites. This may have reflected differences in prescriber practice, formularies, or content and utilization of templates for medication ordering. In which direction and to what extent these differences affected response rates is unclear. In addition, the study was conducted at only two hospitals in one integrated delivery system, and the results may not be generalizable to other systems or regions. We assumed that in all instances clinicians are selecting the correct override reason, and not just the first one or the easiest one. Since we used override reason selection to identify compliance where the selection indicated intent to modify or remove one of the drug orders, if our assumption was incorrect, the results could differ from what they are for an unknown number of alerts. Another limitation is that we did not in this study collect data about adverse events, as doing so was beyond the scope of the study, and therefore we do not know what effect tiering the alert presentation may have on adverse event rates. It is clear, however, that the risk of some specific drug–drug interactions may be overrated ¹³ —the converse may also be true for other interactions. Future studies that seek to correlate how alert severity is determined and what the compliance rates are when alert presentation is tiered with documented adverse drug event rates would help refine the presentation of alerts as well as the determination of severity levels of drug interactions.

The lack of demographic information that could be used to control for response is another limitation. We did not include any indicator of the service or location of the patient, nor did we have available the type of the provider (student, resident, fellow, attending) who was presented with the alert. These variables may correlate with the likelihood of accepting or overriding alerts, in general. As noted by Weingart, ² the level of training, experience, and responsibility toward the patient may make a difference in the response to alerts.

Conclusion

We conclude that tiering of alerts by severity was associated with a much higher likelihood of compliance with alert recommendations. Despite the limitations of this observational, “natural experiment” study design, we believe that tiering clearly appears beneficial, and the high override rate for Level 1 alerts without tiering represents a substantial safety concern. We plan to implement tiering with Level 1 hard stops across our hospital system as soon as technically feasible. We would encourage vendors and developers of CPOE systems to strongly consider implementing tiered presentation of drug-drug interaction alerts in their systems as well.