Strategies for Reducing Nuisance Alerts in a Dose Checking Application

Abstract

Commercial rule bases can be implemented to identify medication orders that fall outside recommended dosage ranges, but they are likely to produce an excessive number of nuisance and clinically insignificant alerts. Strategies for customizing commercial dosing rules can be implemented to minimize this problem. This paper describes specific strategies implemented in a dose checking application necessary for achieving a clinically acceptable alert rate.

INTRODUCTION

Medication errors are a national concern and have received substantial attention since the 1999 Institute of Medicine (IOM) report suggested that 44,000–98,000 deaths may occur annually in the U.S. from medical errors, and that more than 7,000 of these deaths were medication-related.¹ Furthermore, it has been reported that more than half of all preventable medication errors are the consequence of improper physician orders.^2–3 Although commercial vendors often provide robust sets of drug dosage rules, the effectiveness of this rule-based clinical decision support (CDS) is frequently diminished by poor positive predictive value of the rule sets.⁴ Judicious use of alerts is necessary to avoid decision support overload, which can result in oversight of clinically significant alerts.^5–6

One solution to this problem is to customize commercially available dosing rules by implementing relatively simple strategies to decrease rule sensitivity for problematic drugs. Using these strategies, attention can be focused on alerts that are of significant clinical importance.

The purpose of this paper is to identify strategies that can be implemented for reducing nuisance alerts and describe the process, challenges and impact they might have on overall performance of a dose checking application.

BACKGROUND

We previously implemented a commercial rule base (Cerner Multum, Kansas City, MO) with dose range checking rules at Barnes-Jewish Hospital (BJH), a university teaching hospital. Asynchronous alerts generated from our expert system using these rules, dubbed DoseRanger, are routed to pharmacists for resolution. Initially, only the vendor’s maximum single dose rules for 187, mostly chemotherapy drugs, were implemented based on their potential to cause dose-related toxicity. The alert rate for these drugs was a manageable 3.9%, resulting in 450 alerts per month.

In our efforts to expand the safety net to include an additional 1,251 drugs, we retrospectively evaluated maximum single dose rules for BJH and four community hospitals in the BJC HealthCare Systems. Patient specific information such as age, creatinine clearance, weight, height, and body surface area were used as parameters in the screening process. In a single month, a total of 192,669 medication orders from five hospitals were screened. Using the vendor’s rules in unmodified form, the system would have generated alerts for 9.2% of these orders, producing 17,725 alerts.

After evaluating these initial results, our domain experts determined that there were an excessive number of nuisance alerts. Successful deployment of additional drugs rules required that the alert volume be reduced to an acceptable level to avoid decision support overload, and shift focus to clinically significant alerts.

METHODS

From December 2002 (initial deployment) to December 2004 we implemented the following strategies for reducing nuisance alerts in DoseRanger:

• ▪ Dosage calculation buffer (December 2002)

• ▪ Minimum/maximum dose multiplier (June 2003)

• ▪ Creatinine clearance buffer (September 2003)

• ▪ Weight buffer (September 2003)

• ▪ Suppression of duplicate alerts (October 2004)

• ▪ Frequency multiplier (December 2004)

The dosage calculation buffer provides an ± x % margin for rounding dosages to practical values. These dosages are calculated based on patient’s weight, height, or body surface area. For instance, if an Acyclovir order of 350 mg is prescribed for a patient who weighs 69 kg and the vendor’s rule for this drug recommends a maximum of 5mg/kg for that patient population, the rule would alert because the maximum recommended dose would be 345 mg. With the dosage calculation buffer set to 3%, a maximum dose of 360.5 mg would be allowed.

The minimum and maximum dose multipliers are numeric values by which the vendor’s minimum or maximum dose limit are multiplied to establish a new dosage threshold. This relatively simple method of implementation makes the multiplier value easily changed as needed. In addition, a brief comment clarifying the modified dosage threshold is associated with each modified rule and is included with each alert.

The creatinine clearance buffers (CrCl) are percentages by which creatinine clearance thresholds are adjusted to suppress alerts that result from the CrCl being just slightly below or above a rule’s threshold. For instance, a rule with a CrCl threshold of < 25 mL/min and creatinine clearance buffer set to 20% would not be satisfied until the patient’s estimated CrCl was <= 20 mL/min.

The weight buffers are percentages by which weight thresholds are adjusted to suppress alerts that result from the patient’s weight being slightly below or above the rule’s threshold. For instance, a rule with a weight threshold of <= 70 kg and weight buffer set to 10% would not be satisfied until the patient’s weight was <= 63 kg.

The duplicate alert suppression method suppresses alerts if a previous alert had been generated within the last x day(s) for the same patient, medication, dose, route and frequency. It is common for medication orders to be re-entered into the pharmacy systems for the purpose of changing other, non dose-related information such as administration times. As a result, many of these orders would trigger alerts that from a clinician’s perspective are considered duplicates. The duplicate suppression strategy checks pharmacists’ entered responses to previously generated alerts for the same patient, drug, dose, route and frequency and if no doses were changed in response to the previous alerts, then subsequent alerts for x number of day(s) for the same patient, drug, dose, route and frequency will be suppressed. If at least one dose was changed as a result of an alert, then DoseRanger will take a conservative approach and re-alert.

The frequency multiplier represents a numeric value by which the vendor’s maximum frequency (expressed as doses per day) is multiplied to establish a new frequency. For example, if the vendor’s rule for a drug recommends a maximum frequency of Q24H, but a maximum frequency value of Q12H is desired, the frequency multiplier is set to 2 for this particular rule. DoseRanger will then take the original frequency value of Q24H, map the value to ‘doses per day’ which in this example equals 1, and multiply the ‘doses per day’ value by the frequency multiplier to get the desired frequency value. In this case, the modified frequency would be 2 doses per day (Q12H).

The dosage calculation buffer was implemented as an additional field in a database table and applies to rules that involve dosage calculations based on patient’s weight, height or body surface area. Minimum/maximum dose and frequency multipliers were implemented as additional fields (columns) in the appropriate database tables and can be configured for each individual rule. The creatinine clearance and weight buffers were implemented as numeric values (representing percentages) in a database table and apply to all rules that are based on patient’s creatinine clearance or weight values. The duplicate alert suppression strategy was implemented as a combination of a character and numeric value representing the ‘yes/no’ and number of days for previous alerts respectively and applies to all rules.

All of the above strategies were implemented with flexibility to allow different hospitals to have different parameter values in place for each strategy while preserving commercial rule base architecture for easy maintenance and updates.

RESULTS

While these strategies were implemented sequentially in our deployed system, this report retrospectively analyzes the impact of each strategy individually.

Our data set consisted of one month of medication orders for the university teaching hospital and two community hospitals. The number of active rules was slightly different for each hospital due to inactivation of some rules with a certain dose form or route. For instance, some hospitals enter large volume parenteral (LVP) orders into the pharmacy systems as entire packages rather than the actual clinical doses given to patients. Table 1 shows the total number of orders screened and rules implemented for each hospital.

Table 1.

	Hospital A	Hospital B	Hospital C
Orders	80,185	36,424	29,890
Screened Rules Active	21,742	22,116	22,286

We retrospectively ran the selected data set for each strategy individually. Figures 1 and 2 show the impact of the individual strategies on alert rate for all hospitals and for each respective hospital.

Figure 1.

Impact of Individual Strategies on Alert Rate for all Hospitals

Figure 2.

Impact of Individual Strategies on Alert Rate for each Hospital

Figure 3 shows percentages of suppressed alerts and the numbers of alerts suppressed by strategy. The impact of each strategy largely depends on the number of affected rules and order volume for each strategy. For instance, the dose multipliers strategy can affect all rules in the commercial rule base, thus having the greatest potential impact. The suppression of duplicate alerts strategy can also be applied to all rules meaning that each alert has a potential to be suppressed if a previously generated alert for the same patient, drug, dose, route, and frequency was generated within the specified number of days. Conversely, the creatinine clearance, weight and dosage calculation buffer strategies can only affect a certain number of rules, thus having the lowest potential impact. For example, the creatinine clearance and weight buffer strategies can only affect rules that have creatinine clearance and weight thresholds respectively. The dosage calculation buffer strategy can only affect rules that require dosage calculation based on patient’s weight, height or body surface area.

Figure 3.

Percentage of Suppressed Alerts (Number of Alerts) for each Hospital

DISCUSSION

Differences in alert rates by hospital are due to different patient populations, drugs used and allowable doses determined by their local domain experts.

The overall alert rate without any strategy implemented was 7.28%. This unacceptably high alert rate was reduced to a manageable 1.20% after all strategies were implemented.

The dose multiplier strategy had the greatest impact on the alert rate reducing it from 7.28% to 1.65%. These multipliers were determined by local domain experts and were highly specific to dosing practices at each hospital.

All of the other strategies had a relatively low impact on the alert rate. However, considering the volume of medication orders screened by DoseRanger, even strategies with lower percentage decreases suppressed significant number of alerts. For instance, the strategy with the lowest impact on alert rate (frequency multiplier) suppressed 115 alerts in Facility A in one month.

Clinical decision support for drug dosing is an important tool for patient safety; however, the development of large knowledge bases can result in an avalanche of alerts, warnings, and reminders if not judiciously implemented. Most commercial dose checking software is very basic and difficult to customize at the individual alert level.

While it is difficult to determine the ideal alert rate, there have been several published failures of Computerized Physician Order Entry (CPOE) systems due to decision support overload. Strategies described in this paper should be considered by software developers to reduce the number of clinically insignificant alerts.

In comparison to other approaches to solving the problem, we found that there are some vendors that provide a way to alter the allowable dose or frequency; however, the common strategy is for users to simply turn off a decision support module because it alerts too often.

Our findings are limited to:

• ▪ a university teaching hospital and two community hospitals, so results may vary in other institutions

• ▪ we performed this evaluation with only one commercial knowledge vendor for drug dosing rules

• ▪ rule modifications where determined by local experts at each hospital. Local experts at other institutions may not always agree with these rule modifications

• ▪ not all rules were tested: We tested all max single dose rules, but only max frequency and max daily rules for selected drugs.

More study is needed to answer the question “what is the ideal alert rate”.

CONCLUSION

Commercial rule bases are conservative by nature and often result in an unacceptably high volume of clinically insignificant alerts.

We propose several strategies to reduce nuisance alerts. Of these, the strategies that implement minimum/maximum dose multipliers and suppression of duplicate alerts had the greatest impact on our alert rates. This is important for those implementing clinical decision support systems in order to make the greatest impact with the least disruption in workflow.

The creatinine clearance, weight and dosage calculation buffer strategies had lower impact on the alert rates. However, even strategies with less dramatic impact on the alert rates contribute to the effort to reduce alert fatigue and CDS overload and give more credibility to the system.