A Trial of Automated Decision Support Alerts for Contraindicated Medications Using Computerized Physician Order Entry

Abstract

Background: Automated clinical decision support has shown promise in reducing medication errors; however, clinicians often do not comply with alerts. Because renal insufficiency is a common source of medication errors, the authors studied a trial of alerts designed to reduce inpatient administration of medications contraindicated due to renal insufficiency.

Methods: A minimum safe creatinine clearance was established for each inpatient formulary medication. Alerts recommending cancellation appeared when a medication order was initiated for a patient whose estimated creatinine clearance was less than the minimum safe creatinine clearance for the medication. Administration of medications in patients with creatinine clearances less than the medication's minimum safe clearance were studied for 14 months after, and four months before, alert implementation. In addition, the impact of patient age, gender, degree of renal dysfunction, time of day, and duration of housestaff training on the likelihood of housestaff compliance with the alerts was examined.

Results: The likelihood of a patient receiving at least one dose of contraindicated drug after the order was initiated decreased from 89% to 47% (p < 0.0001) after alert implementation. Analysis of the alerts seen by housestaff showed that alert compliance was higher in male patients (57% vs. 38%, p = 0.02), increased with the duration of housestaff training (p = 0.04), and increased in patients with worsening renal function (p = 0.007).

Conclusion: Alerts were effective in decreasing the ordering and administration of drugs contraindicated due to renal insuffiency. Compliance with the alerts was higher in male patients, increased with the duration of housestaff training, and increased in patients with more severe renal dysfunction.

Adverse drug events (ADEs) have been shown to contribute to the morbidity and mortality associated with the treatment of disease as well as the cost of care.^1,2,3 A substantial portion of these ADEs are preventable, with estimates in the literature ranging from 20% to 69%.^4,5,6,7 Preventable ADEs are often the result of medication errors.⁸ Thus, efforts to reduce medication errors have the ability to lower the rate of ADEs substantially and improve the overall delivery of health care.

Information and knowledge offered to the clinician to facilitate the best decision and thereby reduce medication errors is termed clinical decision support (CDS). Automated CDS systems transform clinical data gathered in an electronic medical record (EMR) as well as expert-based or evidence-based practice guidelines into useful patient-specific knowledge to assist clinical decision making. Recent findings show that computerized physician order entry (CPOE) in conjunction with basic CDS such as Drug–Allergy and Drug–Drug interaction checking decreases the likelihood of serious medication errors,^9,10 although limited by clinician noncompliance with alert recommendations.^11,12,13

Clinical decision support designed to reduce medication errors associated with renal dysfunction should be an efficacious intervention because the risk for error is high, and the CDS systems should be able to accurately assess a patient's renal function. One of the earliest reports of such a system was published by Rind et al.¹⁴ This study was not performed with CPOE and real-time alerts but demonstrated the utility of automated alerts to improve dosing in renal insufficiency as well as decrease the use of contraindicated medications. More recently, Chertow et al.¹⁵ published a study on real-time decision support delivered during CPOE and designed to improve dosing of renally excreted medications. The clinicians were provided a highlighted dosing suggestion as the default order when attempting to order. The intervention was shown to be beneficial but did not completely eliminate poor dosing due to clinician noncompliance with the dosing recommendations.

In a recent study by Oppenheim et al.,¹⁶ the intervention was designed to identify completed medication orders with inappropriate dosing and to suggest to clinicians that the order may require modification. This intervention was also found to be beneficial but, again, did not reduce all dosing errors because clinicians often did not comply with the alerts. These examples both show that CDS has great potential to reduce dosing errors in renal dysfunction; however, clinician noncompliance can limit the efficacy of these efforts.

We implemented and analyzed automated alerts designed to reduce the use of contraindicated drugs in patients with renal insufficiency. Our goals were twofold: evaluate the utility of these alerts as well as analyze the factors that may play a role in noncompliance with alert recommendations among housestaff. Understanding and overcoming alert noncompliance issues will be important in designing improved CDS systems and safer drug administration processes.

Methods

CPOE and CDS Environment

The University of Illinois Hospital and Medical Center utilizes a commercially available EMR (Millennium®, Cerner Corporation) which is used as the primary source of presentation of all results and orders to clinicians. All medication and laboratory orders are placed using CPOE, predominantly by housestaff. Nurses and pharmacists sometimes place medication orders based on a physician's verbal order. Although infrequent, attending physicians can directly place a medication order using CPOE. The commercially available automated CDS (Discern Expert®, Cerner Corporation) has been previously described.^17,18,19

Development of Alerts

At each instance of a newly reported serum creatinine level, an estimated creatinine clearance (CrCl_est) was determined using the Cockcroft-Gault equation.²⁰ Although usually available, there were instances in which entry of height was neglected or occurred after the serum creatinine level was reported. In these situations a normalized CrCl_est was determined based on only gender, serum creatinine level, and age.²¹ Determinations of CrCl_est were only made at the time of a newly reported serum creatinine level. The CrCl_est was used because it is a better estimate of renal function than the serum creatinine level,²² whereas the gold standard, a 24-hour creatinine clearance, is rarely available.

A group of pharmacists reviewed the entire inpatient hospital formulary and chose a safe CrCl_est for all drugs deemed to be potentially contraindicated for patients with renal insufficiency. The drugs along with their safe CrCl_est are shown in ▶.

Table 1.

Potentially Contraindicated Drugs with Safe Renal Function Cutoffs

Safe Creatinine Clearance	Drugs*
CrClest > 10 mL/min	Acetazolamide, Ifosfamide Methotrexate, Pancuronium, Gold sodium thiomalate, Spironolactone
CrClest > 30 mL/min	Alendronate, choline magnesium salicylate, Demeclocycline, diclofenac, ibuprofen, indomethacin, ketorolac, naproxen, rofecoxib, salsalate, tolmetin
CrClest > 40 mL/min	Sotalol
CrClest > 50 mL/min	Chloral hydrate, Chlorpropamide, Metformin, Nitrofurantoin, Penicillamine, Phenazopyridine, Probenecid, Ribavirin
CrClest > 80 mL/min	Auranofin

^*Medications available on the hospital formulary at the time of the study that were potentially contraindicated owing to renal insufficiency.

The alerts were designed to trigger when a clinician attempted to order one of the potentially contraindicated drugs for a patient whose most recent CrCl_est was less than the corresponding safe level for the drug. The clinician received a “pop-up” alert suggesting that they not proceed with the order due to the patient's renal insufficiency. In the alert message, the clinician is provided the most recent CrCl_est, the established “safe” CrCl_est value for each drug, and a pager number for the on-call pharmacist if the clinician had any questions regarding the alert. A picture of this type of alert is shown in ▶.

Figure 1.

Example of a contraindication alert. This alert is displayed to the ordering clinician upon initiation of an order for a contraindicated medication only when the most recent CrCl_est is below a previously determined safe value. Clicking “OK” does not automatically discontinue the medication order. This example is for an order for the drug metformin in a mock patient with a CrCl_est of 34.3 mL/min.

Determination of Likelihood of Drug Administration before and after Alerts Implementation

To examine changes in the administration of contraindicated drugs in patients with renal insufficiency, a historical cohort was established for the four-month period before alert implementation. During this four-month control period, CrCl_est were available in the EMR for clinician review; however, no alerts were generated. During the subsequent 14-month study period not only were CrCl_est available for review, but alerts were also generated when a medication that was contraindicated due to renal insufficiency was ordered.

The control cohort was electronically mined for instances in which the alerts would have been generated if active, i.e., attempts to order a drug when the patient's most recent CrCl_est is less than the safe CrCl_est for the given drug. For all the instances in both the control and study periods, the electronic medical record was reviewed for record of administration of at least one dose of the medication associated with the order in question.

Determination of Alert Compliance and Characteristics of the Patient and Clinician

A log of all alerts is automatically recorded by the software. For each contraindication alert over the 14-month study period, the EMR was reviewed to determine if the alerting order was completed or cancelled before completion. Completion of the order was considered noncompliance with the alert recommendation. The time of day, patient age, gender, creatinine level, and CrCl_est were recorded as well as the type of clinician: housestaff, pharmacist, nurse, medical student or attending physician. Only the alerts received by housestaff were included in the analysis of compliance. The start date of the individual residents' training was recorded and compared with the date of the alert to calculate the duration of training.

Statistical Analysis

For the administration of contraindicated medications, the proportions of patients receiving at least one dose in the control and study periods were compared using a χ² test. Characteristics of the control and study groups were compared using a Student's t-test for continuous variables and the χ² statistic for proportions. A p-value of 0.05 was chosen for statistical significance of all comparisons.

Analysis of the housestaff compliance with the alert recommendations was performed using multiple logistic regression. The dependent variable was the compliance with the alert. The independent variables were the time of day, patient gender, patient age, patient's renal function, and length of training of housestaff. The CrCl_est was used as the measure of renal function. Based on initial trends using bivariate analysis, the time of day was described by a dichotomous variable, either 7 am to 1 am or 1 am to 7 am. The training was categorized as either ≤365 days, or >365 days.

Because some clinicians received more than one alert, a clustering effect based on clinician was possible. This was accounted for by using the generalized estimating equations modification of logistic regression analysis.²³

Results

Over the 14-month study period, 323 alerts were generated. In the four-month control period there were 87 situations that would have generated an alert if the alerts had been implemented at that time. In both time periods the most commonly ordered drug was metformin. Features of the control and study groups are presented in ▶. The control and study periods did not differ with respect to patient gender, age, or degree of renal dysfunction. The likelihood of a patient receiving at least one dose of the contraindicated medication decreased from 89% to 47% after alert implementation (p < 0.0001), a 42% absolute reduction in administration. This reduction was almost entirely due to cancellation of the order after viewing the alert, which occurred in 41% of instances in which an alert was given.

Table 2.

Group Characteristics and Administration Likelihood

	Control Group	Study Group	p Value
Duration	4 mo	14 mo
No. of alerts or alerting situations	87	323
Age	66 ± 12 yr.	66 ± 14 yr.	NS*
Female (%)	84	75	NS*
CrClest average	39 ± 16 mL/min	37 ± 14 mL/min	NS*
Likelihood of patient receiving at least a single dose of the contraindicated medication (%)	89	47	< .0001

Characteristics of the control and study groups were compared using a student's t-test for continuous variables and the χ² statistic for proportions. The CrCl_est average is the average of the most recent CrCl_est at the time of each alert. The proportion of patients receiving a dose in the control and study periods was compared using a χ² test. A p value of 0.05 was chosen for statistical significance of all comparisons.

^*No significant difference between the value in the control and study groups (p > 0.05).

The distribution of clinicians receiving the alerts was 70% medical housestaff, 13% nurses, 11% pharmacists, 4% medical students, and 2% attending physicians. There were a total of 233 unique patients. For the 226 alerts received by housestaff, the alert compliance rate was 42%; for the remaining clinicians the compliance rate was 38% (p = 0.54).

For the remainder of the analysis of compliance, only the alerts seen by housestaff were analyzed further, investigating some of the factors affecting compliance in housestaff using multivariate analysis. For this subset of 226 alerts, the average age of the patient was 66 years, and 77% were women. The analysis looked at both patient-specific as well as non–patient-specific factors. Among the patient-specific factors evaluated, age was not found to be associated with alert compliance (p = 0.32). Patient gender, however, was found to be associated with the alert compliance rate; with compliance in female patients lower than that in male patients, 38% vs. 57% (p = 0.02). Alert compliance also decreased with improving renal function or increasing CrCl_est (p = 0.007).

The time of day was not found to be significantly associated with the likelihood of alert compliance. However, the duration of housestaff training was found to be associated with the likelihood of alert compliance. Housestaff having more than one year of training had a compliance rate of 50%, whereas those with less than one year of training had a compliance rate of 37% (p = 0.04). The relationship is shown in ▶.

Figure 2.

Compliance versus duration of housestaff training. The y-axis represents the percentage of housestaff compliant with the alert recommendation to discontinue an order for a contraindicated drug. The x-axis represents the duration of housestaff training from the beginning of their residency until the time that the alert was received in months.

Discussion

This study found that real time “pop-up” CDS alerts generated during CPOE were able to reduce the number of medication errors by decreasing both the completion of medication orders and administration of medications that were considered to be contraindicated due to renal insufficiency. This intervention, however, was limited by noncompliance with the alert recommendations as has been shown in many other trials of CDS alerts with CPOE.^{11,12,13,14,15,16,19,24,25,26}

It is interesting to note that noncompliance with alerts for renal dysfunction has been a problem regardless of the type or strategy of decision support implementation. Chertow et al.¹⁵ did not generate alerts at the time of order but rather modified the default suggested dose and frequency of renally cleared medications. This intervention was successful but limited by significant noncompliance. In the intervention reported by Oppenheim et al.,¹⁶ no warnings or dosing assistance was initially given; however, upon completion of the order, alerts were generated if a suspected dosing error occurred. This intervention was also efficacious but again limited by noncompliance. Our intervention, which generated warnings at the time of order initiation, was efficacious but was also limited by noncompliance. Compliance is, therefore, an important element to consider in CDS implementation strategies because it clearly limits efficacy. An understanding of the factors that promote or diminish compliance needs to occur to fully realize the benefits of CDS.

The problem of noncompliance with alerts for medication orders has been studied in the past.^11,12,13,27 Some of the causes determined in the earlier studies are likely related to a portion of the noncompliance observed in our study. In the earlier studies, re-ordering or renewal of orders for previously given medications has been shown to increase noncompliance.^11,12,13,27 In our study this also may have been the case as many of the potentially contraindicated medications may have been previously given to the patient either earlier in the admission or in the ambulatory setting. This seems particularly likely for the most commonly alerted medication, metformin.

Noncompliance can also be associated with a compelling indication for the medication despite the risk.¹² Such careful consideration of risk–benefit likely was not a large factor in our study as the most commonly alerted medications, metformin and nonsteroidal anti-inflammatory medications, are not life-saving inpatient medications and have safer alternatives in patients with renal dysfunction.

Noncompliance can also result when the ordering clinician has knowledge of clinical information that the CDS system does not. This relevant clinical information may not be present in the EMR or may be stored in a format not accessible to the CDS. The CDS alerts in our study were based on only the assessment of renal function and the drug being ordered. Because serum creatinine is frequently ordered during the course of inpatient care, the CDS system and clinician were both likely considering the same information at the time of medication order entry.

Our analysis looked at some additional factors that are associated with noncompliance for our contraindication alerts. An important finding in our study was that in the case of physician housestaff, increased duration of training improved the compliance to the alert recommendations. This finding was also reported by Oppenheim et al.¹⁶; however, the exact methodology of the comparison was not specified. Although our study was not designed to determine the reason for this finding, it is interesting to postulate a cause. The alert in this study suggests to clinicians that they are making a medication error based on a patient's renal insufficiency.

Our alerts provided two elements of information: the first is patient related (i.e., the presence of renal insufficiency based on a calculated CrCl_est) and the second is drug related (i.e., the drug being ordered is not safe to use in renal insufficiency). It is reasonable to assume that a clinician would comply with the alert recommendations if they understand and accept as true these two elements of information being provided.

As they advance through training, housestaff members are more likely to have been exposed to the use of the calculated CrCl_est as an estimate of renal function and have acquired knowledge of drugs that are not safe to use in patients with renal insufficiency. For the knowledgeable housestaff, the alerts may simply remind the physician of the presence of renal insufficiency in a given patient, a piece of data that may have been forgotten or never reviewed before ordering. Housestaff members with less experience may not fully understand estimates of renal function and are less likely to know which drugs are safe in patients with renal insufficiency. A mistrust or misunderstanding of the alert information, particularly the non–patient-specific element, may cause the clinician to ignore the alert recommendation. This hypothesis can be tested in future studies by surveying clinicians receiving alerts and measuring their reasons for either complying with or ignoring alerts.

An unexpected finding in our study was the relationship between patient gender and noncompliance. Housestaff was more likely to listen to the suggestion to cancel an order when contraindicated in a male patient than in a female patient. The degree of renal dysfunction was the same in the male and female patients, 37 mL/min, as was the age, 66 years (p > 0.5 in both). The average creatinine level of the female patients was, however, lower than that of the male patients; 1.7 ± 1.4 mg/dL vs. 2.8 ± 2.7 mg/dL (p = 0.0002). One possible explanation for this gender-based difference is that some clinicians may be more trusting of the serum creatinine level than the CrCl_est as determined by the Cockroft-Gault equation.²⁰ This would produce the tendency to disregard a low CrCl_est when the creatinine level is listed as normal. This situation is more likely to occur in female patients due to smaller average ideal body weight and a gender-based factor of 0.85 in the Cockroft-Gault equation²⁰; thus, the tendency to trust the creatinine level over the CrCl_est would produce a gender-based difference in compliance. This finding suggests an increased need for educating housestaff that the use of CrCl_est as an assessment of renal function is preferred over serum creatinine level.²²

The positive association between compliance and the degree of renal dysfunction is not surprising. In a study by Dexter et al.,²⁸ compliance with reminders for preventive care in inpatients showed an association with the strength of the indication. Similarly, compliance with our contraindication alerts improved with the increased risk associated with decreasing renal function.

Although the alerts were statistically beneficial, they did not completely eradicate the ordering of medications contraindicated due to renal insufficiency. Our institution chose to institute additional alerts that are triggered by evidence of noncompliance or override. We have reported on such alerts in the past, implemented for digoxin use.¹⁷ These additional alerts are triggered when a clinician completes an order for a contraindicated medication when a given patient's CrCl_est is < 67% of the threshold CrCl_est that is considered safe for the given medication. This alert is sent to both the pharmacy and the nursing unit suggesting that the drug not be administered without an explanation from the clinician as to why the alert was disregarded. We hope that these additional alerts will further decrease the use of contraindicated medications and plan on studying their efficacy.

Limitations

The institution studied is a public teaching hospital with the majority of orders place by housestaff physicians. It is not clear to what extent these findings are applicable to facilities in which attending physicians perform the majority of the medication ordering. It would not be surprising to find that attending physicians have similar behavior as related to patient gender and estimates of renal function, but this of course needs to be verified experimentally.

The study examined only one type of alert: contraindication due to renal insufficiency. This was done as actions related to this type of alert are simple to interpret: compliance (medication not administered) or noncompliance (medication administered). Outcomes associated with alerts for dosing, relative contraindications, drug side effects, or interactions are much more ambiguous to interpret. There is a continuing need for studies with a broad variety of alerts to further enhance the knowledge in this field.

Conclusions

We developed and implemented CDS alerts to decrease the use of contraindicated medications in patients with renal insufficiency. The alerts were successful in decreasing the likelihood of clinicians completing contraindicated orders and decreased the administration of these medications. The efficacy of the alerts was limited by clinician noncompliance. Housestaff was less compliant when alerted for female patients than for male patients, whereas alert compliance improved with worsening patient renal function and increasing duration of housestaff training. Due to the problem of noncompliance, it is likely that as CDS evolves, a variety of alert mechanisms and forms of decision support will be used together to ensure appropriate patient safety while allowing for clinician education and autonomy.